Embedding Python in Your C Programs (Linux Journal)

Posted Dec 31, 2005 0:20 UTC (Sat) by ncm (subscriber, #165) [Link]

Posted Dec 31, 2005 0:31 UTC (Sat) by dskoll (subscriber, #1630) [Link]

Tcl is a superb language to embed in C programs. The C API of Tcl is a joy to behold: Orthogonal, complete, well-documented, and extremely well thought-out. I don't know what you think of Tcl as a language, but it's become the de-facto default embedded language in a lot of EDA (Electronic Design Automation) products.

Perl is a more popular language, but its C API is atrocious. Poorly-documented, arcane and prone to easy breakage. I speak from bitter experience.

Posted Jan 1, 2006 1:13 UTC (Sun) by sumnerh (guest, #34870) [Link]

I've also found Tcl easy enough to embed, but I don't like the language at all.

Posted Jan 1, 2006 5:32 UTC (Sun) by zblaxell (subscriber, #26385) [Link]

One of Tcl's strengths is that all native datatypes have a canonical string representation, including code. This is used in the field for things like single-file applications (database + database library + application, all in one self-modifying cross-platform file) and remote code execution (one company makes a motion controller which contains a network server to which you can send fragments of Tcl code to control industrial machinery).

One of Tcl's weaknesses is that there's no such thing as a reference, and another is that if you're not extremely careful about implied conversions, you might cause all intermediate math results to go through atof() and back. If you want to do a lot of bit-level meddling with the machine state from the scripting language then you need to be careful to avoid implicit string conversions or changes in interpreter state that invalidate cached compiled code. It might be actually be painful to use Tcl in this case--I've done bit-banging code in Tcl, and it can be spectacularly slow unless the coder is careful to avoid those implied string conversions.

Another Tcl weakness is that the language is simple--much simpler than many programmers are used to. There are 11 paragraphs in the Tcl(7) man page describing the entire language syntax, but if you ever miss any trivial detail of any of them, then your program will run, but it will do something different than what you expect--often this involves accidentally executing variables as code in contexts you didn't expect.

In your case, you could store your code either as Tcl strings, or in Tcl procedures (which are still strings but only one word is used to invoke them ;-). From the Tcl C API's point of view, you can convert the strings to Tcl_Obj*, then evaluate the Tcl_Obj* which will compile the string to bytecode and cache the result relatively transparently. The API also has fast-path hooks for calling a named procedure with a vector of arguments from C.

It might be better to expose machine state through a variety of accessor functions rather than as a single Tcl data structure--that way you'll only be converting and/or copying the parts of the machine state you're actually accessing from Tcl. The "Tcl way" of doing that sort of thing is a function like "get_register_state -bx -ax" which returns a list of the contents of the "bx" and "ax" registers in that order (sorry, I forget which CPU the Intellivision uses ;-) or "get_memory_contents 0 1023" which would return a string that could be further processed by "binary scan ...".

Tcl has trigger functions on its own variable accesses. This probably isn't useful in your case, but it's interesting nonetheless.

Posted Jan 1, 2006 2:28 UTC (Sun) by sumnerh (guest, #34870) [Link]

Using different minor versions of the compiler is fine, I haven't tried different major releases but the issues should be the same as for using different majors to build a shared library and a project. Interpreters written in C++/ObjC may be different since the compatibility requirement between compiler versions are often more stringent there.

I've had no problems embedding Python into projects I build with autoconf.

As an earlier comment noted, you can avoid some of these issues by making your app an extension and calling back into it from Python/Perl code rather than vice-versa; whether that makes sense depends on your application.

Posted Jan 1, 2006 4:46 UTC (Sun) by zblaxell (subscriber, #26385) [Link]

It is a little odd that Python wants to change the behavior from the default, but it wouldn't be so odd if Python insisted on having the default while your application contains C-library-behavior-changing #defines.

Unfortunately all libraries impose these kinds of constraints on their calling applications (and even other libraries) to some degree. Libraries that happen to be embeddable scripting language interpreters simply have more constraints than most. They also tend to include a portable runtime library that runs on all the platforms the scripting language does, so they'll include their own layers on top of stdio, GUI frameworks, and so on, all of which just increase the number of points where parameters of the build environment can collide with semantics of the scripting language.

These days the scripting languages also come with the infrastructure to integrate them into application build systems, although often that means "build your application with ScriptLanguageMake" instead of whatever tool you wanted to use. Perl does a lot of source-level preprocessing and (last time I checked) pretty much implements make itself, while Tcl provides almost 200 lines of compiler flag variables that your application needs to be built with.

Auto* in my experience is particularly badly suited for scripting language modules. Last time I tried (which was admittedly many automake versions ago), I had to do things like write C programs whose sole function in life was to execve() a script in the appropriate language to finish the build process.