|
|
Subscribe / Log in / New account

Tcl helped Guile Scheme

Tcl helped Guile Scheme

Posted Oct 2, 2024 10:35 UTC (Wed) by jem (subscriber, #24231)
Parent article: Tcl/Tk 9.0 released

Not everyone was convinced of the excellence of Tcl. From the Guile Manual:

Richard Stallman, as the primary author of GNU Emacs, had a particular vision of what extension languages should be, and Tcl did not seem to him to be as capable as Emacs Lisp. He posted a criticism to the comp.lang.tcl newsgroup, sparking one of the internet’s legendary flamewars. As part of these discussions, retrospectively dubbed the “Tcl Wars”, he announced the Free Software Foundation’s intent to promote Guile as the extension language for the GNU project.


to post comments

Tcl helped Guile Scheme

Posted Oct 2, 2024 18:07 UTC (Wed) by dskoll (subscriber, #1630) [Link] (11 responses)

I don't know for sure, but I suspect part of Stallman's antipathy to Tcl was its very liberal (MIT-like) license rather than a GNU license. His technical criticisms were either off-base or have been addressed in subsequent Tcl releases.

Anyway, you can read Stallman's criticism and the responses here.

Tcl helped Guile Scheme

Posted Oct 3, 2024 7:44 UTC (Thu) by oldtomas (guest, #72579) [Link] (10 responses)

A Google-free alternative here: https://vanderburg.org/old_pages/Tcl/war/

The whole (IMO very unfortunate) episode became known as the "Tcl War", and yes, I think Stallmann's main beef at the time must have been Tcl's "corporate friendliness" (Stallmann went as far as calling Ousterhout a "parasite" for leaving Berkeley and working for Sun -- this is around 1994, so about the same time). That he pushed technical arguments to the fore must be an instance of that classical "nerd dishonesty" many of us fall into from time to time (I know I do).

On the one hand, Stallmann was a visionary: these days you can see the corporate world gutting free software of all its values and using it to curtail user's freedoms in far more subtle ways than one might have imagined back then.

On the other hand, this episode caused a lot of unnecessary grief. Most of the Tcl community is wary of the GPL and all things FSF to this days.

Oh, another thing: this radical "everything is a string" (affectionately called EIAS in Tcl lingo) made for an incredibly enjoyable C interface: Tcl functions just took argc, argv. This, of course, got lost once Tcl gained a bytecode interpreter with 8.0 (must have been as a guest to Sun, where much more paid hands were available to do that huge work). A typical evolution of languages back then (Guile did something pretty similar).

Tcl helped Guile Scheme

Posted Oct 3, 2024 8:51 UTC (Thu) by anselm (subscriber, #2796) [Link] (9 responses)

This, of course, got lost once Tcl gained a bytecode interpreter with 8.0

That was less the bytecode interpreter than it was the introduction of the Tcl_Obj types, which enabled Tcl data items to have a non-string representation and to hang on to it for extended periods of time. This meant that the interpreter could convert a variable's value from a string to a number and use the numeric value from then on, without converting it back to a string for storage and again to a number if you wanted to use it in a calculation once more.

Of course the bytecode interpreter helped a lot with running Tcl code faster, but the ability to, e.g., store lists as lists (with direct pointer-based access to individual items as opposed to having to parse a long string to locate them every time) arguably had the biggest impact on Tcl performance.

Tcl has various other interesting features, like the idea of subinterpreters, including “safe” interpreters used to run untrusted code, that other languages are still missing (the Python community is now taking a greater interest in subinterpreters, 25 years after Tcl added the feature). In a number of respects the language was way ahead of its time, and it is a pity that – except for some areas such as EDA – it never got the traction it deserved. To think that instead of coming up with a bum language like JavaScript, Netscape could simply have added Tcl to their browser boggles the mind. What a wonderful world this could be …

Tcl helped Guile Scheme

Posted Oct 3, 2024 11:28 UTC (Thu) by mgedmin (subscriber, #34497) [Link] (8 responses)

I'm not sure what you mean by the Python community's sudden interest in subinterpreters, but, to quote python.org

> CPython has supported multiple interpreters in the same process (AKA “subinterpreters”) since version 1.5 (1997).

Tcl helped Guile Scheme

Posted Oct 3, 2024 13:38 UTC (Thu) by foom (subscriber, #14868) [Link] (7 responses)

Python supported it...brokenly. Only recently is the support actually being fixed.

All of the interpreters ended up incorrectly sharing a lot of mutable python objects, due to deficiencies in the C extension module API design. This started to be addressed in Python 3.5, via addition of new APIs (which modules need to switch to, and even core python itself wasn't doing this properly until recently).

Only in 3.12 can interpreters have their own district global interpreter locks, and thus run in parallel with each-other.

Even still today, the APIs that allow running a callback into python from C application code (as distinct from calling back into Python in the context of a call from Python) are incorrect if there are multiple interpreters. You must somehow ensure you're calling into the correct Python interpreter state. But all the well-lit paths are broken and incorrect -- they assume that either you already have a python interpreter context associated with the current thread, or that there is a single "main" interpreter that you want to use. (See https://github.com/python/cpython/issues/59956)

Tcl's support for multiple (isolated) interpreters, running in parallel, was solid this entire time.

Tcl helped Guile Scheme

Posted Oct 3, 2024 20:04 UTC (Thu) by NYKevin (subscriber, #129325) [Link] (6 responses)

> You must somehow ensure you're calling into the correct Python interpreter state. But all the well-lit paths are broken and incorrect -- they assume that either you already have a python interpreter context associated with the current thread, or that there is a single "main" interpreter that you want to use. (See https://github.com/python/cpython/issues/59956)

Unfortunately, there's just not much that can be done to properly fix that API. The whole point of PyGILState is to provide a convenient way for foreign code to call into Python without having to know any details of how Python has been initialized or configured. From the perspective of this API, you could be calling into a /usr/bin/python process from a C extension, calling into an embedded interpreter from the program doing the embedding, or any number of more complicated cases. In principle, I think you can even call into this API from a ptrace-injected thread.

The obvious problem is that, if the foreign code does not know which interpreter it wants, then the API has no hope of selecting the right one in the general case. And if the foreign code does know which interpreter it wants, it can call PyThreadState_New() etc. instead of using PyGILState. That's marginally more verbose, but frankly not by very much (there would be some marginal utility in a pair of helper functions or macros similar to Py_BEGIN_ALLOW_THREADS, but each would only be two or three lines long). The larger problem is that you need to restructure your C code in such a way that you know which interpreter you want, either because you create it in the first place, or because you arrange for Python to call into you before you call into Python.

Offering this API in the first place was perhaps imprudent in retrospect, but I'm sure there are plenty of cases where it is very useful.

Tcl helped Guile Scheme

Posted Oct 3, 2024 20:19 UTC (Thu) by anselm (subscriber, #2796) [Link]

The obvious problem is that, if the foreign code does not know which interpreter it wants, then the API has no hope of selecting the right one in the general case.

Tcl addresses this by making the first parameter to most of the function calls in its C API a pointer to a Tcl_Interp structure (which contains the state of the Tcl interpreter in question). That way, it is always clear which Tcl interpreter is meant when you call Tcl from your own C code.

This structure is one of the consequences of Tcl being originally intended as an embedded extension language – the application (usually written in C or C++) would create a new Tcl interpreter via the Tcl C API and use that to run Tcl code (which would then presumably use functionality that the application provided to that interpreter in the shape of Tcl commands) –, rather than a free-standing programming language with an escape hatch into C. Later on, when Tcl acquired the capability to load dynamic shared libraries, it became more popular to provide the “application” as a shared library you'd load from tclsh or wish (a tclsh that is already linked to Tk), rather than the other way round by having a C application that brought in libtcl.so.

Tcl helped Guile Scheme

Posted Oct 4, 2024 3:12 UTC (Fri) by foom (subscriber, #14868) [Link] (4 responses)

Yes, perhaps the best way to fix the Python GILState APIs is to simply deprecate them and recommend an interpreter-aware alternative.

ISTM that a lot of the time, C code will be doing something like invoking a python function callback pointer which had been recorded in the past (potentially on a different thread). It would be "easy" for such code to record the interpreter pointer at the same time as recording the function pointer. Or, at least, it would be if there existed proper documentation of the requirement to do so, and demonstrating how to do so.

Tcl helped Guile Scheme

Posted Oct 4, 2024 18:46 UTC (Fri) by NYKevin (subscriber, #129325) [Link] (3 responses)

It is documented, albeit badly (IMHO). See https://docs.python.org/3/c-api/init.html

Unfortunately, the correct procedure is rather path-dependent and can be complicated in some edge cases, which is probably why the PyGILState functions exist. Here are some examples:

* If Python previously called into you from the same thread, then that thread would usually be managed by Python, and so you probably already have the GIL. Then there would be nothing to do. The only exception I can think of is if you called into Python first (and then it called you back), which implies that you already figured out how to acquire the GIL, so just do whatever you did again.
* If Python previously called into you from a different thread, then you can use PyInterpreterState *interp = PyInterpreterState_Get() to get the current interpreter, and then (on the new thread) call PyEval_RestoreThread(PyThreadState_New(interp)) (error checking elided). When you're done calling into Python, you will probably want to call PyThreadState_Clear(PyThreadState_Get()); PyThreadState_DeleteCurrent() to free the object created by PyThreadState_New() (and implicitly release the GIL). If you want to keep it around for multiple calls into Python, you can use PyEval_SaveThread()/PyEval_RestoreThread() to release and re-acquire the GIL in the usual manner (and you can also use the Py_BEGIN_ALLOW_THREADS convenience macro), but you should eventually free it when finished calling into Python. You need a separate thread state object for each thread that will call into Python (technically, you need one for each thread-interpreter combination, but I'm assuming you're not planning to interleave interactions with multiple interpreters on a single thread, because then you have to use PyThreadState_Swap() and the complexity really starts to make your code hard to read).

Frankly, those two bullets cover the vast majority of "real" use cases. But for full feature parity, there's a long tail of other cases to consider:

* If Python did not previously call into you at all, then you have a couple of further options. You could use the "main" interpreter, which is the first interpreter initialized for the process, as returned by PyInterpreterState_Main(), but that's really only suitable if you're not going to interact with any pre-existing Python objects and just need an arbitrary interpreter. In that case, it is probably preferable to create your own private sub-interpreter instead, preferably with its own private GIL (which you can do by acquiring the main interpreter's GIL and then using Py_NewInterpreterFromConfig(...) to create the sub-interpreter). OTOH, creating lots of tiny interpreters is probably not optimal either, so there's a certain amount of nuance here. You may need to experiment with different setups and benchmark the performance. Creating a private interpreter is most likely to be worth it if you know that the rest of the process is making extensive use of the main interpreter (or some sub-interpreter which shares the main's GIL), and least likely if you're the only code in the whole process calling into Python.
* The other possibility is that you know which interpreter you want, because you created it in the first place. But then this reduces to the "Python previously called you from a different thread" case, since creating an interpreter puts your thread into (more or less) the same state as if Python just called into it from that interpreter. It also implicitly releases the parent interpreter's GIL (if you request a separate GIL for the child interpreter), so you need not worry about doing that.
* PyGILState_Ensure() can also be called recursively if you do not know whether you hold the GIL, and so for full parity, we need to have a way to check whether you have the GIL already (to avoid self-deadlock in the case that you do have the GIL). You can use PyGILState_Check() for that, and unlike the rest of the PyGILState API, this ought to be reasonably compatible with multiple interpreters. That's because it is reading the same thread-local variable that the rest of the PyThreadState API interacts with, so it should not "care" whether you are acquiring the GIL via PyGILState or with the interpreter-aware functions. The caveat is that it is possible to clear this variable without actually releasing the GIL (e.g. with PyThreadState_Swap()), in which case this function can lie to you. So don't do that and then try to query whether the GIL is held - PyThreadState_Swap() requires holding the GIL anyway, so I'm not entirely sure there's even a use case for doing this in the first place.
* Believe it or not, we're still not done, because recursive calls to PyGILState_Ensure() can be interleaved with the rest of the PyThreadState API, including PyEval_SaveThread(), so it also has to check for that case explicitly and re-acquire the lock if it has been released since the last call. I'm... not really sure why you would do that, but apparently it is a thing you can do.
* Note also that Python's thread-local storage API does not require holding the GIL to use, and supports arbitrary (void*) types, so you can record additional information there if it is useful to do so (which is exactly how PyGILState works internally - it maintains a counter of how many times you have called into it in order to figure out when to destroy the thread state, and this system is even designed to avoid destroying manually-created thread states so that it can be safely interleaved with PyThreadState_New() and company).

Fair warning: I have read the documentation for these functions. I have not actually tried to write code against them. I did read some of CPython's source code to spot-check a few points of uncertainty, and I'm fairly sure that most if not all of the above is basically correct (in particular: PyGILState_Ensure() does indeed call PyEval_RestoreThread() to acquire the GIL if necessary, so I'm reasonably convinced that it is legal to create a new thread state from scratch and "restore" it), but it is always possible that I have missed something.

Support for sub-interpreters

Posted Oct 5, 2024 11:26 UTC (Sat) by cgm (guest, #173850) [Link] (2 responses)

Wow! That's a pretty good argument for using Tcl!

Support for sub-interpreters

Posted Oct 6, 2024 8:28 UTC (Sun) by NYKevin (subscriber, #129325) [Link] (1 responses)

Yes, but then I have to learn a new language. I can't throw a stone without hitting three Python scripts written by random people, so I have to know Python.

Support for sub-interpreters

Posted Oct 8, 2024 14:43 UTC (Tue) by dskoll (subscriber, #1630) [Link]

Tcl is a very easy language to learn. Most experienced programmers could learn it in a day or two.

But yes, given the relative popularity of languages, knowing Tcl isn't much of an advantage wrt getting work or understanding other software.

Tcl helped Guile Scheme

Posted Oct 17, 2024 5:54 UTC (Thu) by ceplm (subscriber, #41334) [Link]

And started one of longest variations on “Surely …” skeleton joke https://mcepl.fedorapeople.org/tmp/emacs-guile.jpg


Copyright © 2025, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds