Taste [LWN.net]

Taste

Posted Sep 6, 2007 20:06 UTC (Thu) by ncm (subscriber, #165)
In reply to: Taste by foom
Parent article: LinuxConf.eu: Documentation and user-space API design

At the time I was at ITA, they did go to great lengths to avoid ever triggering a GC. (Are you saying QPX runs GC cycles now?) The only integer type that could be used without accumulating garbage was 30 bits. It's one thing to know you're overflowing your ints and quite another to avoid doing it; sometimes you really need more bits. Using floating-point values did accumulate garbage. There was discussion of sponsoring a a 64-bit CMUCL port, which would have offered a bigger fixed-size integer type, and enough address space to tolerate more accumulated garbage, and (maybe?) a native floating-point type. I suppose that port, or the SBCL equivalent, is in use now. Restarting the program once a day while other servers take up the load is an extremely reliable sort of garbage collection, but you need lots of address space to tolerate much accumulated garbage.

(Log in to post comments)

Taste

Posted Sep 6, 2007 23:22 UTC (Thu) by foom (subscriber, #14868) [Link]

> (Are you saying QPX runs GC cycles now?)

Yes. With a generational collector, triggering minor GCs is not actually a terrible thing. I'm sure
QPX ran the GC while you were around as well, although, as you say, some effort was put in to try
to avoid it happening very often.

But, as it turns out, the strange things QPX did to avoid allocating new memory for objects that
need to be on the "heap" was actually *slower* than allocating the objects normally and letting
the GC do its thing.

Basically, the GC works fine, and it was a mistake to try to avoid using it. (This mistake wasn't
made because of stupidity or premature optimization, it was an optimization made for another
lisp implementation with a poor GC, and was kept around without re-assessing its necessity
perhaps as soon as should have been done.)

Of course, not allocating memory at all is going to be faster than allocating memory, but when
you do it, a garbage collector is a fine thing to have.

Taste

Posted Sep 7, 2007 0:40 UTC (Fri) by ncm (subscriber, #165) [Link]

"I'm sure QPX ran the GC while you were around"

Long experience has taught me to be very distrustful of anything a GC advocate is "sure" of.

Evidently ITA's "mistake" in avoiding GC cycles was to insist on running their application for several years before a Lisp runtime with a tolerable GC was available to them. They certainly were not lax in trying to obtain one: they used Allegro CL at the time I started, and dropped it for CMUCL while I was there. SBCL was under active development. They employed one of the primary CMUCL maintainers. (I think he would be surprised to find his competence disparaged here; he was always admirably forthcoming with me in acknowledging CMUCL's then-current and Lisp's inherent limitations.)

This exchange illustrates well some of the reasons why Lisp hasn't exactly taken the world by storm. Chief among them must be Lisp advocates still unable to understand why not.

But this is all off-topic, and I apologize again to those reading the article to learn about system call interfaces.

Taste

Posted Sep 7, 2007 2:43 UTC (Fri) by foom (subscriber, #14868) [Link]

As I said, the mistake was that nobody had properly re-tested the assumption that the GC did not
work acceptably for too long. But, attacking straw men is more fun, right? I'm sorry you have such
an acute dislike for garbage collectors that you need to make things up to prove them terrible.

Lest anyone be confused: CMUCL and SBCL's garbage collectors are essentially identical, and CMUCL
has had a generational garbage collector since...a very long time ago.

While Lisp has most certainly not taken over the world, garbage collectors have.

Taste

Posted Sep 7, 2007 23:53 UTC (Fri) by jsnell (guest, #47245) [Link]

Lest anyone be confused: CMUCL and SBCL's garbage collectors are essentially identical, and CMUCL has had a generational garbage collector since...a very long time ago.

While the code in the two GCs might be essentially identical, that doesn't really mean that their performance characteristics are. There are many important performance improvements in the memory management of sbcl which never made it back to cmucl. Some of those improvements were in the GC, others in related areas like the fast path of the memory allocation sequence. As a result of those, cmucl can take 2x the time sbcl does to run an allocation-heavy program and spend 5x as long in the GC for it [*].

But ultimately those improvements were just tweaks on a 10 year old GC that uses only concepts that are 20 year old, and which was bolted on to a compiler that doesn't really provide any support for the GC. It's not hard to imagine that newer GC designs or ones that are properly integrated into the compiler would perform even better.

[*] Those results are from the Alioth binary-trees benchmark with n=20, since I don't have any better benchmarks accessible right now. Incidentally, in the shootout results the Lisp version of this program is faster than the C one.