Language Embedding by Dual Compilation and State Mirroring: Implementing a portable Tcl/Tk-Prolog interface

<title>Language Embedding by Dual Compilation and State Mirroring:
implementing a portable Tcl/Tk-Prolog interface</title>

<h1>Language Embedding by Dual Compilation and State Mirroring:
Implementing a portable Tcl/Tk-Prolog interface</h1>

<pre>
<A HREF="http://www.sciences.umoncton.ca/infoque/PAULT.HTM">
Paul Tarau</A>
Dept. Informatique  
Universit'e de Moncton
tarau@info.umoncton.ca
<A HREF="http://www.cs.kuleuven.ac.be/deptcw-members.html">
Bart Demoen</A>
Dept. of Computer Science 
Katholieke Universiteit Leuven
bimbart@cs.kuleuven.ac.be
</pre>

<hr>
Abstract<p>
We give an abstract scheme for the embedding of two languages with
strong meta-programming, parsing and structure manipulation
capabilities.  We provide a call-back mechanism requiring each language
to compile its service requests to the other's syntax for processing
through the other's meta-programming
capabilities. State mirroring ensures
that only differential information needs to be transmitted.  We
describe a practical implementation of these principles and our
experience with the 
<A HREF="ftp://clement.info.umoncton.ca/BinProlog">
BinProlog</A>
Tcl/Tk interface. 
Compilation by each side, that targets
the other's syntax and dynamic object manipulation capabilities,
achieves in less than two hundred 
lines of code, a fully functional integration
of the two systems. The environment
is portable to any Prolog system to which it
adds powerful graphic and transparent distributed
network programming facilities.
Our approach suggests that logic programming language-processors
can be made to fit easily in a multi-paradigm programming environment
with little programming effort by using their
strong  language translation and  meta-programming
capabilities.

<p>
 Keywords:
multi-paradigm programming, visual environments for logic programming
languages, meta-programming, Prolog graphic interface.

<h2>Introduction</h2>

Integration of various programming paradigms is a necessity
in modern programming environments. 
Monolithic language implementations tend to be replaced more and more
by a combination of specialised  tools.
In particular, generic graphic processors can be connected
to various languages which do not need 
their own graphic primitives anymore.
<p>
However, when there's a large
semantic distance between the languages and they feature
radically different computational models, a low-level
interface is a painful and not always rewarding programming
task. It happens very often that when the interface is finished
one of the sides of the interface just becomes obsolete
because of
rapid evolution. This is especially true with various
windowing systems which are often machine/operating system dependent.
 <p>
The aim of this paper is to propose an unusually quick
interfacing technique which uses meta-programming capabilities
on the two sides. The basic idea is to link the two languages
through a standard client-server interface and have each of
them drive the other as an interactive agent, by generating
the appropriate code on the fly.
By mirroring the state of the objects, only state
changes have to be transmitted from one side to the other,
so that the granularity of the interaction will not become
a bottleneck. Moreover, if one side has the capacity to
react to events, this property will be inherited with
minimal programming effort by the other side.
<p>
This paper is motivated by our work on finding a simple and
portable way to add a visual programming environment to
Prolog systems.
<p>
The popular Tcl/Tk visual language by John Ousterhout
is basically
 a composite
of a shell-like interpreted programming
language (Tcl) and a high-level Motif-style graphic
package (Tk). Tcl is an untyped string-only language with
strong meta-programming facilities (i.e. an  eval primitive
and dynamic procedure creation).
<p>
After describing a general language-processor interaction
model we will report
how it has been applied to an interface between BinProlog [Tarau 93:GULP],
and Tcl/Tk and how this interface has been ported to
another Prolog system (Prolog by BIM) with minimal
programming effort.

<h2>An abstract multi-language communication model</h2>

Let L and R be two languages, t(L,R) a translation function
(compilation) from L to R, t(R,L) the reverse translation from
R to L.
Let e(L) and respectively
e(R) be the  eval operations of L and R.
Let r(L), w(L), r(R) and w(R) denote the read and write operations
of languages L and R.
<p>
An L->R processor is a process with capabilities to
<ul>
<li> execute programs written in L
<li> implement a translation function from L to R.
</ul>

We say that two language processors L->R and R->L are  connected
if the following conditions are verified:
<ul>
<li> each  translate+write operation on one side triggers exactly one
 read+eval operation on the other side,
<li> one or more   translate+write operations
can be triggered by each  eval operation.
</ul>

(Footnote: The ability to trigger more than one  translate+write
is needed for instance when more than one task has to be
initiated or more than on component of the state of one side
has to be updated in a transaction.)

<p> This
can be formalized more precisely in terms of temporal modal
operators. 
<p>
Our  write and  read
operations should not be confused with
ordinary input-output primitives. More precisely they are abstracted
from the subset of input-output operations which can be meaningfully
translated and are therefore suitable
as input for the other side's eval functions.
Thus they can be seen as carrying messages from an L->R
processor to an R->L processor.
<p>
We say that an L->R and an R->L processor
are  fairly interacting if
<ul>
<li> each write operation on one side will be 
eventually read on the other side
<li>
the behaviour of
each side is observable on the other side in terms of a sequence of successive
read and write operations
</ul>

If for two L->R and R->L language processors are  connected
and their read and write operations are queued, then
they are  fairly interacting.
<p>
We will break this fully symmetric behaviour by defining
a  slave language-processor as one that passively waits
for write operations from the other side and a  master
language-processor which reacts to a write operation
by a corresponding read operation with
its result passed to its  eval operation.
We suppose also that the user interacts only with the master.
As only the slave waits blocked on read operations, the
master is free to be `multi-threaded' or event-driven.

<h2>The BinProlog to Tcl/Tk interface</h2>

As an instance of this general scheme,
we have chosen to connect BinProlog as a `slave process'
to a Tcl/Tk  wish shell enhanced
with reactive capabilities
given by the  addinput facility which enables it to react to the presence of new input
by triggering the execution of a user specified procedure
when input becomes available.. A specialized BinProlog
toplevel is launched from the Tcl/Tk shell as a
background Unix process connected through
a bidirectional pipe.

<p>(footnote)
A modification of the Tcl/Tk event loop (
freely available from
ftp.neosoft.com)
<p>
The BinProlog process is able to generate and react to Tcl/Tk events.
The interface uses tcl7.3 with tk3.6 combined with addinput-3.6a.  It
consists of 91 lines
of Prolog and 101 lines of (commented) Tcl code.  Thanks to the
strong meta-programming capabilities of both languages the interface
boldly  compiles 
messages from one side
to evaluable representations on the
other side and calls the appropriate
 eval operation.
The interface is portable to other
Prolog systems and requires no C-programming. 
The installation of a BinProlog slave on the 
`event channel' handled by addinput is done as follows:
<pre>

proc start_prolog  
#   Opening a bidirectional pipe to BinProlog
#   under control of the addinput facility
    set f [open "|bp -q5 server.bp" r+]
    addinput $f "bpInOut  E F"

</pre>
Output from the BinProlog slave
bp server.bp activates a Tcl/Tk
procedure which executes on a  new Tk-interpreter without
disturbing normal interaction with the  wish shell.
<p>
Basically, performance is not affected by the parsing
as this is kept minimal. Moreover in systems with a parser
written in C
(as Prolog by BIM or the Koen De Bosschere's 
ISO-parser based version of BinProlog)
parsing is in itself fast enough not to dominate
the interaction cost.
Techniques as  state mirroring which will be discussed in the next
section are also used to minimize the communication overhead.
<p>
High-level primitives on the Tcl/Tk side ensure that
output to Prolog are valid Prolog terms.
On the Prolog side a special toplevel loop reads the
Tcl/Tk message, and applies its eval operation
(call/1) to it. To avoid spurious interaction due
to syntax errors or unexpected messages on each side
only lines having a reserved header (i.e. call_prolog
and respectively call_tcl )
are evaluated. However, to give the look and feel of interacting
with a Prolog system, messages from Prolog which are not
of the form "call_tcl ..." will be printed out
as such. This fits well with Tcl which is string oriented
but would not fit well with a term-oriented language like
Prolog without parser modifications.

<h2>State mirroring</h2>

State mirroring is a simple technique that consists of
duplication by a given language processor of (relevant)
state information of an interactively connected other
language processor.
<p>
In the case of Prolog, we have chosen to avoid the
representation of states by infinite forward loops
for two reasons:

<ul>
<li> unexpected failure would make the system unreliable
and hard to debug
<li> unexpected delays induced by garbage collection would make
a visual interface unpleasant to use
</ul>


Fortunately, BinProlog's blackboard turned out to be the perfect match
to mirror the state
of various Tcl/Tk objects. A library emulating BinProlog's blackboard
operations in terms of generic Prolog 
dynamic database operations
has been used to achieve
the same effect in the Prolog by BIM port of the interface.
 
<h2>Programming with the embedded environment</h2>

Using the BinProlog-Tcl/Tk interface is very simple.
Let (P) be the name of the composite program.
Suppose the user creates 2 files (P).tcl and 
(P).pl. The composite `program' can be executed
as follows:
<pre>
    $ wish
     source (P).tcl
</pre>
(P).tcl is expected to have at its end something like:
<pre>
   start_prolog
   p compile((P))
</pre>

The user will get the usual Tcl command prompt to interact with
either Tcl or Prolog. A command entered at the Tcl prompt
normally `goes to' Tcl except for the following: 

<pre>
    start_prolog      -connects to a new Prolog process
    halt_prolog       -terminates the Prolog process:   
    p (PrologTerm)  -sends a goal to Prolog for quiet evaluation
    q (PrologTerm)  -sends a query to Prolog and gets back answers
</pre>
After being activated by one of the previous (p or q) Tcl commands
a Prolog goal may `call back' to continue the dialog with Tcl/Tk using
the following Tcl command on the Prolog side:
<pre>
   call_tcl/1    -sends to Tcl a command to be immediately
                  evaluated in `background', while the Tcl shell
                  prompts and waits for Tcl or Prolog commands.
</pre>
Output with puts (Tcl) or write/1
(Prolog) goes as usual to the stdout of the Tcl shell.

<h2>A visual N-queens program</h2>

This program illustrates two simple-minded but useful features
of the interface:
<ul>

<li>  Prolog lists are sent to Tcl as Tcl-lists
<li>  Prolog simply pumps solutions one by one using their most natural
   representation through call_tcl(display_queens(L)).

<li> synchronization is done as follows:
<li> the user provides mouse events
generating write operations 
<li> the prolog process waits for input on a read operation
until asked for another answer
</ul>

Equivalently, the user
can also control the Prolog program from Tcl's command line
by typing in "p more" or "p done" messages.
<p>
The code on the Prolog side consists of a
classic N-queens program, the generic
prolog interface program server.pl
and the clause:

<pre>
qs(N):-
    queens(N,L),
    call_tcl(display_queens(L)),    sends a display request
    write('type (p done) when finished (p more) otherwise'),nl,
    in(call_prolog(done)),!.        waits for `done' or `more'

</pre>

The code on the Tcl/Tk side
consists of the generic server.tcl interface program,
routines to set up the visual display and procedures like
)~show_queen~, )~hide_queen~ which are called by the
main Tcl/Tk-side routine:

<pre>
proc display_queens qs 
  global w count
  incr count
  hide_queens
  set l 0
  foreach q $qs 
    incr l
    show_queen $w [expr $l -1] [expr $q -1]    
    
</pre>

This procedure simply counts the answers and updates the display
when a new answer comes from Prolog, triggered by the presence
of new input on the pipe.

<h2> A visual heap-measurement tool - Performance evaluation</h2>
<p>
Tcl/Tk is itself a glue language which links together
large blocks of C-code.
We have written a naive-reverse program
and measured that it executes in Tcl between
20-50 times slower than in BinProlog.
This definitely suggests that our technique, using
relatively expensive parsing and compilation operations
will not become itself an interaction bottleneck and
therefore due to the much
higher flexibility and almost instant
portability between various Prolog systems
this approach looks superior to a highly
Prolog-specific C-level integration.
<p>
As BinProlog is a much faster symbolic processor than Tcl,
we have
decided to do most of the translation
operations on the Prolog side.
We have measured about 1500 BinProlog-Tcl/Tk exchanges per second
on a Sparcstation 2 and 700-800 exchanges on a Sparcstation 1,
both serving the same Tektronix X-terminal through a network.
<p>
This confirms that the interface is fast enough for complex
real time visualisation tasks and pleasant to use in a
average Unix environment.

<h2>Porting the interface to another Prolog</h2>

It took one hour to make the interface work with ProLog by BIM.
The main change was to modify )~start_prolog~ on the
Tcl side to start a ProLog by BIM process with:

<pre>
   set f [open "|bim server.pro" r+]
</pre>

The ProLog by BIM process is initialized from
a small new toplevel )~server.pro~. The one hour work
had more to deal with the differences between the two 
Prolog systems,
than the interface itself. Moreover, code written for
one Prolog system (like the N-queens program) 
runs without changes on the other, 
as far as it is written in portable Prolog.
We think this compares favorably with
a dedicated piecewise interface written
in C for each Prolog graphic builtin
in terms of programming effort,
portability and learning curve for the user,
and, as our performance evaluation has shown,
with reasonable real-time interactivity.

<h2>Related work</h2>

The only other
Prolog with Tcl/Tk interface we know
of is that developed by
Micha Meier at ECRC as the standard
Eclipse GUI, although ongoing work on SICStus Prolog
may also lead to a such an interface in the future.
Micha Meier's interface is tightly integrated (at C-level)
and Eclipse specific (although a Sicstus port is also provided).
The tighter integration is better (in principle)
in the case
of a very low granularity communication, although
the programming effort to implement it is
more considerable than ours.
Our performance analysis
shows that global costs are dominated by process switching,
graphic manipulations and brute force processing in Prolog.
Moreover, our interface, does
not rely on a specific Tcl/Tk implementation and has
no modification to the C-code neither on the Prolog emulator
neither on the Tcl/Tk side.
Because of these reasons, we believe that
the simplicity and portability
of our design is a clear advantage,
especially in the context of rapidly evolving
backward incompatible Tcl/Tk releases.

<h2>Future work</h2>
We plan to make the interface fully Prolog-independent
and to write Prolog libraries to support various
existing Tcl/Tk extensions and tools for
distributed Tcl/Tk programming,
automatic interface generators etc.
A Tcl/Tk based generic Prolog toplevel would also
be very helpful to give the full illusion of
 being in Prolog while benefiting of
visual goodies of a Tcl/Tk shell.

<h2>Conclusion</h2>

We have outlined a generic programming framework for a seamless
integration of two programming languages which have `eval' capability
and implemented an instance of the framework as a Prolog to TCL/Tk
interface.
<p>
Our experiments show that
support for metaprogramming tools
in modern programming environment
is important not only for the expressiveness of the
language itself but also multi-paradigm language
integration.
<p>
We have shown that with a reduced programming effort
a powerful interface can be built between a Prolog system and
a state-of-the art visual environment. 
<p>
This suggests that embedding of a logic engine in a
generic multi-language
environment is competitive in functionality and performance
with the traditional approach which advocates to hardwire
in the Prolog system a set of language-specific extensions.
<p>
The code for the interface can be obtained by ftp from
clement.info.umoncton.ca and has been integrated in the
BinProlog 2.20 distribution.

<h2>Acknowledgements</h2>
Paul Tarau thanks
for support from K.U.Leuven through Fellowship F/93/36,
from NSERC (grant OGP0107411) and 
from the FESR of the Universit\'e de Moncton.
Bart Demoen thanks 
the Belgian Ministry (DPWB) for support through project IT/IF/4.
Special thanks go to the referees
for their useful comments and suggestions.

<p>
The HTML form of this document was converted from LaTeX and
kindly sent to us by:

<pre>
<A HREF="http://ultraman.pse.titech.ac.jp/~lu/index.html"> 
Dr Ming Liang LU</A>  
Tokyo Institute of Technology
Research Laboratory of Resources Utilization,
E-mail: lu@pse.titech.ac.jp
WWW: http://ultraman.pse.titech.ac.jp/~lu/index.html
</pre>