The Rise of Functional Languages (Linux Journal) [LWN.net]

What, again?

Posted May 1, 2007 19:52 UTC (Tue) by dcoutts (subscriber, #5387) [Link]

Ah well, I guess we should give up on Java and C# and all those other garbage-collected languages since apparently it's impossible to deal with resources.

This isn't true of course. If we have some really critical resource (say a DB connection) we can use more manual management than leaving it to the GC. And since functional languages are great for building abstractions we can neatly write functions to capture patterns of bounded resource use. One simple example is block scoping:

withDbConnection $ \db -> do
   -- 'db' var only in scope inside this 'do' block
   ...
   doSomething db
   ...
-- 'db' connection no longer accessible here, resource can be recycled

What, again?

Posted May 1, 2007 20:30 UTC (Tue) by ncm (subscriber, #165) [Link]

Yes, I did say "Stone Age".

What, again?

Posted May 1, 2007 22:05 UTC (Tue) by dark (subscriber, #8483) [Link]

What are you comparing it to? Imperative languages can do no better than
this, and many do worse by requiring you to explicitly open and close the
database connection. Separating the open and close operations leaves many
fun opportunities for forgetting one of them, especially when handling
errors.

What, again?

Posted May 1, 2007 22:25 UTC (Tue) by smitty_one_each (subscriber, #28989) [Link]

>What are you comparing it to?
Possibly, the 'Stone Age' observation is related to the cave-painting-like simplicity and obviousness of the functional code.
A Renaissance, imperative piece of code would be awash with color and strokes, not to mention the swell frame(work), driving up its market value. ;)

What, again?

Posted May 1, 2007 22:45 UTC (Tue) by ncm (subscriber, #165) [Link]

"Stone Age" refers to the impossibility of encapsulating management of the resource in the languages under discussion. One is left manually calling "close" functions, or (as in dcoutts's toy example above) trying to match resource lifetime to program control flow. "Imperative" (vs., presumably, "declarative") does not make a meaningful distinction. Any language that lacks destructors leaves the programmer needing to explicitly "open and close the database connection", or the mutex, or the socket, or what-have-you.

Traditionally Python has avoided this problem by supporting the equivalent of destructors in C-coded language extensions.

What, again?

Posted May 2, 2007 0:00 UTC (Wed) by dcoutts (subscriber, #5387) [Link]

And how do destructors work? They're either block scoped, if you allocate an object on the stack or if you allocate on the heap then you have to manually free the object (which then calls the destructor). So how was this better? You have to explicitly open and close the database connection, via explicitly allocating and freeing the object.

What, again?

Posted May 2, 2007 1:58 UTC (Wed) by njs (subscriber, #40338) [Link]

C++ actually does rather better than this (if used correctly) -- it's easy to use block scoping for most resource management (because, well, most resources of all kinds actually have lexical extent, at least in most programs when written well), and when a resource has dynamic extent, you use a smart pointer, which has explicit semantics about when it will be freed (i.e., exactly when the last reference disappears). Or you can do other fancier and situation-specific things if you want, of course, like pooled allocation.

There's no reason that other languages couldn't do this, but most GC languages are so hasty to ensure that the programmer does not *have* to worry about memory management that they remove the guarantees that the programmer needs in case they ever *want* to worry about memory management. So, in particular, most GC languages have either poor or no support for finalizers (cf. the weird interactions between Python's __del__ and the cycle collector, or the minimal guarantees Java makes about the environment in which a finalizer will be run), and make no promptness guarantees about collection. C++ is very very far from a perfect language, but it does get this part right -- you can always tell from code inspection exactly when each resource will be acquired and released.

What, again?

Posted May 12, 2007 6:55 UTC (Sat) by jwalden (guest, #41159) [Link]

when a resource has dynamic extent, you use a smart pointer, which has explicit semantics about when it will be freed (i.e., exactly when the last reference disappears).

That's all well and good for simple data, but it falls flat on its face with cyclic data. If your data is cyclic, you have to explicitly break it somehow, perhaps via forget, or else no member of the cycle is going to be deallocated. Consider also that if you do this, you also have to be careful that in breaking the cycle you have a (perhaps stack-allocated) smart pointer to yourself as well, or the action of forgetting may trigger a sequence of forgets which eventually causes the method of the object performing the deletion to delete itself! Smart pointers ease much of the common drudgery, but they are not a panacea.

As for the rest of your comment, it's true that having the ability to perform prompt finalization is useful, in some circumstances; it enables idioms like RAII. I do have to disagree with "you can always tell from code inspection exactly when each resource will be acquired and released", however, because determining variable lifetime (particularly effective lifetime, i.e. how long the value of a variable can influence future execution -- GC provides no solution to that problem) for values on the heap can be reduced to the halting problem.

What, again?

Posted May 1, 2007 20:18 UTC (Tue) by pate (guest, #10) [Link]

Yeah, that's probably why YAWS performs so badly. I think there's room for functional languages out there, maybe not for every app, or every programmer, but room nonetheless.

What, again?

Posted May 1, 2007 20:49 UTC (Tue) by peterh (subscriber, #4225) [Link]

> A mostly-functional language could be practical for engineering if it
> provided the tools necessary to abstract resource management.

Can you be specific? As another poster observed, the most-heavily used languages in the real-world are garbage collected these days (Java, C#, any scripting language), so clearly garbage collection is orthogonal to whatever issue you are raising.

As to why programming language researchers love garbage collected languages --- it's pretty simple. Most reasonable people would see some form of memory-safety as a non-negotiable language design goal. In a language with explicit deallocation, there are currently no known really good solutions for ensuring memory safety (for example, preventing the use of dangling pointers to freed space). This is a really hard program analysis problem (in general, it seems you need to understand properties like shape). A simple and almost always adequate solution --- use garbage collection.

What, again?

Posted May 1, 2007 21:38 UTC (Tue) by ncm (subscriber, #165) [Link]

As I noted earlier, functional languages may be thought of as alternatives to scripting languages. (However, it's notable that I have never had a Python program consume all available swap space.) Java and C# may be thought of as particularly cumbersome, ugly scripting languages.

As to why programming language researchers have so little effect on the world of programming -- it's pretty simple. They have no interest in what makes a difference to people solving very hard, real-world problems. Those who do have such an interest are working on C++ libraries or on improving C++ itself. This isn't because C++ is perfect, or close, but because it does offer the tools they need. To be very specific, it offers destructors.

Progress in computer languages proceeds by extracting features out of the language core, and making it possible to implement them -- or useful variations -- in libraries, instead. Often the language core needs to get more powerful to enable such expression.

A language designer who pretends that there is only one resource to be managed, and then fails to provide the tools needed to manage even that one, is a failure. It was forgiveable fifty years ago, but not today.

What, again?

Posted May 1, 2007 23:37 UTC (Tue) by peterh (subscriber, #4225) [Link]

> This isn't because C++ is perfect, or close, but because it does offer
> the tools they need. To be very specific, it offers destructors.

What, _exactly_ do you get from having destructors (over, say, java-style finalizers)? I'm looking for a precise argument here as to why the absence of this language feature makes it hard to write large, real-world programs.

Really, these days you need a very good reason to write code in C or C++. The benefits of safe languages are very compelling. There are reasons to use C/C++ and friends, they just don't apply to the vast majority of code that people write (i.e. not operating system kernels, nor that very small proportion of code where small constant-factor performance differences matter).

I'm afraid I simply don't follow your statement about "[failing] to provide the tools needed to manage that [resource]". It's difficult to understand such a claim when the current #1 language (by the TIOBE index) Java, is so heavily influenced by ideas from the programming languages research community (type-safety, garbage collection, generics (aka. type-classes), ... largely functional language concepts). Clearly a large number of programmers manage to get their work done in such an environment.

What, again?

Posted May 2, 2007 0:58 UTC (Wed) by bronson (subscriber, #4806) [Link]

What does a destructor provide that finalize() doesn't? That's easy: a definite time that the cleanup code is called. This argument is ages old. Briefly:

    for(log files) {
        File f;
        f.open(file name);
        f.quick_scan();
    }

In C++, you will only have a single fd in flight because the destructor is called and the file is closed the moment the block is exited. In Java (older versions; I haven't used Java recently), you'd typically bomb out after a few thousand iterations because you exhaust your file descriptors before a garbage collection cycle can be run.

Most resources are of such a limited quantity that you need to know with confidence exactly when they are released. Destructors provide this, finalizers don't.

For instance, do you see the problem with the following code? It's only a problem if your language uses finalizers.

   try {
     FileReader f = new FileReader(filename);
     process(f);
     r.close();
   } catch(IOException e) {
     ...
   }

What, again?

Posted May 2, 2007 1:34 UTC (Wed) by peterh (subscriber, #4225) [Link]

Yes, but you have _exactly_ the same problem in a language with explicit deallocation and destructors.

If you use finalizers to free resources, you'll have these sorts of problems. Use a free_my_resources() method for that. If you forget to call it, you'll leak the resources (but the finalizer could check for that, at least limiting the damage). If, on the other hand, you forget to call "delete" on the object, you leak the resources and the object itself. So you have to do something explicit either way, and without garbage collection the damage is worse (you have a memory leak into the bargain).

Not quite sure how that's an improvement, myself.

There is an exception here --- if your object is allocated on the stack (which Java forbids). This does get you a kind of "auto-destruction" semantics. But is this a common case?

What, again?

Posted May 2, 2007 3:42 UTC (Wed) by ncm (subscriber, #165) [Link]

It has been almost ten years since I coded a "delete" statement in C++. I can't forget to write them because I never needed to write them in the first place. How does memory get freed? Destructors free it. I don't write calls to destructors, because the compiler writes those calls itself.

That's what is meant by "encapsulating resource management in libraries": all the work is coded in the library, and then you just use it. There is no value at all in the language providing "memory-safety" when the language is powerful enough to allow it to be encoded in libraries in exactly the forms needed. A language not powerful enough to encode resource management in a library looks useless to me.

Finalizers are little more than a cruel joke, and do not merit discussion in this context.

What, again?

Posted May 2, 2007 4:41 UTC (Wed) by foom (subscriber, #14868) [Link]

...just so long as you never have circular references.

What, again?

Posted May 2, 2007 17:06 UTC (Wed) by dcoutts (subscriber, #5387) [Link]

So you are exclusively using blocked scoped resources or smart pointers. With smart pointers you either are using ref counting or single owner semantics. Single owner is very limited, it doesn't allow you to build shared structures where there are multiple ways of reaching a value. If you're using ref counting it's not too bad, though you can't ever build cyclic data structures and you have to keep track of who is holding onto references or you get space & resource leaks. This last point is true of GC'ed languages too, forgetting that some object holds a reference to some resource causes space leaks.

In my experience, essentially all the times where we have to be concerned with use of a precious resource there are very clear bounds where the resource is needed. In almost all of these cases simple block scoping of the resource is sufficient. As you say there are some cases where we want more flexible techniques like pools but again these can be implemented in any language GC'd or not.

Sure the Java example where it does not let you do any manual resource management and only closes files via a finaliser is silly. All it needs is a .close() method or something and then in a nice functional language with cheap abstractions you don't even need to remember to call .close() as (like in my earlier example) you can write functions which encapsulate the resource usage patterns like allocate, use, release.

What, again?

Posted May 2, 2007 18:26 UTC (Wed) by bronson (subscriber, #4806) [Link]

> if your object is allocated on the stack (which Java forbids)...

If you didn't notice, I'm using Java finalizers and its lack of automatic objects as an example of broken design.

> ...this does get you a kind of "auto-destruction" semantics. But is this a common case?

In C++, absolutely. It couldn't be more common.

I feel like I've answered your question but you're refusing to acknowledge it. Just because Java blew it with finalizers, that doesn't mean that destructors are broken too. Did you understand the second example in my post? I'm happy to explain further if you'd like.

What, again?

Posted May 2, 2007 1:23 UTC (Wed) by djabsolut (guest, #12799) [Link]

these days you need a very good reason to write code in C or C++

A very pointed statement. Some counter arguments: Firstly, C and C++ are two different languages. The latter just happens to contain a large subset of the former. Secondly, C++ can be a very sharp tool, but a coder can easily restrict him/herself to only use safe programming practices and idioms in C++ (one example: ignoring all the nasty parts of C). Thirdly, the "truckload of libraries" advantage of languages such as Python has been more than noticed by the folks working on the next version of C++. Many of the Boost libraries are candidates for inclusion in the next standard (in fact, there is already a few of them in the TR1 extensions of C++). Due to C++'s templates and the ability to overload operators, some of the libraries have in effect extended the language already. To put it another way: the amount of work done per line of source code has increased considerably. Fourthly, apart from numerical applications where Fortran may still be faster, it is hard to beat the speed of C++ programs.

What, again?

Posted May 2, 2007 3:31 UTC (Wed) by ncm (subscriber, #165) [Link]

Actually C++ is now beating FORTRAN soundly in numerics, too. This is a consequence of its template system, which allows domain-specific optimizations to be defined in libraries, and applied at compile time. Even back in 1995, Todd Veldhuizen wrote a C++ library that beat IBM's specialized FORTRAN by 40% on IBM's own vector machine running fluid-dynamic simulations.

The C++ numerics community has not stood still since then. A current example of such a library is VSIPL++, which can apply MPI underneath to automatically parallelize array operations. It's published under GPL, by the way, so you can find out how.

What, again?

Posted May 2, 2007 17:16 UTC (Wed) by dcoutts (subscriber, #5387) [Link]

Yeah, adding domain-specific optimisations into libraries is cool stuff and really practical. We do this with the Haskell compiler GHC which allows you to stick equational rewrite rules into a library and then the compiler applies them to users code. In effect it allows you to program the optimiser letting the library author apply domain-specific knowledge to direct how to compile user's code.

We do this for example to fuse array operations, combining mutliple passes over arrays into a single pass and eliminating intermediate temporary arrays. This allows users to write in a higher level modular style but then we can compile this down to fast low level code.

Compile-time typing good.

Posted May 2, 2007 18:10 UTC (Wed) by ncm (subscriber, #165) [Link]

Yes, Haskell has a very powerful type system. (Of course that's nothing to do with its being a functional language, so we are drifting off topic a bit, but what the hell.) VSIPL++, by the way, also fuses array operations, probably by very similar means. Recently it demonstrated a 3x FFT performance on IBM Cell vs. IBM's own library and custom compiler.

I met somebody back in Massachusetts working on something he called Eager Haskell. People laugh, but it's much faster than regular Haskell, making it practical, at least for pure computational work. Probably Darcs should be rewritten in Eager Haskell.

Somebody else demonstrated a Haskell compiler as a C++ library. It compiles (rather slowly) to fast native code.

What's practical?

Posted May 3, 2007 0:14 UTC (Thu) by shapr (subscriber, #9077) [Link]

What's practical?

Some Haskell applications that I consider to be practical are: a Quake clone Frag, a fast as Apache webserver HWS, and an operating system named House.

You mentioned darcs already, it also seems practical to me.

You also mentioned that Haskell compilers produce fast native code, so what about Haskell do you perceive to be impractical?

What's practical?

Posted May 3, 2007 18:03 UTC (Thu) by ncm (subscriber, #165) [Link]

Haskell's garbage collector and the language primitives that depend on it makes it impossible to implement the features needed to abstract management of non-memory resources. Its inability to abstract resource management also makes it impossible for exception facilities to safely centralize error handling. In this it is like other languages that have failed, and will continue to fail, to unseat C++ from serious industrial applications. That's unfortunate because C++ needs serious competition.

By "practical", in context, I meant "practical for industrial use". For pure computation Haskell (well, Eager Haskell) verges on the practical already.

Looks practial to me.

Posted May 4, 2007 0:09 UTC (Fri) by shapr (subscriber, #9077) [Link]

Haskell's garbage collector and the language primitives that depend on it makes it impossible to implement the features needed to abstract management of non-memory resources.

I think this is incorrect.
First of all, non-memory resource management is already abstracted. File handles, sockets, etc are garbage collected when they are no longer reachable.
Second of all, if you want to be able to explicitly de-allocate a resource before the GC reaches it, you can use an RAII-style wrapper, like withSocket or withHandle.

Haskell doesn't really need explicit destructor support in the language. Because they can be written in a library, they don't need to be part of the core.

Its inability to abstract resource management also makes it impossible for exception facilities to safely centralize error handling.

I'm not really sure what you mean here. Haskell exceptions can be caught anywhere further up in the scope. That means you can catch an exception thrown deep in the code and handle it in your top level code. That sounds like centralized error handling to me. Perhaps I misunderstood?

It sounds like your concern is about putting resources inside other structures (assuming proper use of those structures) and getting semi-automatic management, with an option to have more explicit management.
I've already mentioned GC and RAII-style (block scope) destructors. For other custom schemes (pools, regions, etc), you can use monads.

By "practical", in context, I meant "practical for industrial use".

Haskell is already used in the industry. What's your definition of industrial use?

For pure computation Haskell (well, Eager Haskell) verges on the practical already.

In my experience, non-strictness in Haskell dramatically simplifies the logic of a program, I wouldn't want to return to writing strict languages. Why do you think non-strictness is impractical?

Boost?

Posted May 2, 2007 10:54 UTC (Wed) by scripter (subscriber, #2654) [Link]

The upcoming inclusion of Boost in the C++ standard is admirable. This is coming at least 10 years later than it should have. On the other hand, it's taken time to standardize C++ and get the various compiler implementations up to date.

Eleven years ago, I started using CPAN. For most projects, it's not so much important what language is used as the community, the libraries and the tools that surround the language. It's no wonder that languages other than C++ have seen such dramatic use over the past decade.

I'm starting to use Boost, and it has been worthwhile.

What, again?

Posted May 2, 2007 7:43 UTC (Wed) by ms (subscriber, #41272) [Link]

As I noted earlier, functional languages may be thought of as alternatives to scripting languages. ... Java and C# may be thought of as particularly cumbersome, ugly scripting languages.

Yay. Let's talk about cars, they're everyone's favorite meaningless analogy. "A car can be though of as a an alternative to a blimp." Suddenly it's all so clear...

Cars

Posted May 2, 2007 17:18 UTC (Wed) by ncm (subscriber, #165) [Link]

There's nothing wrong with cars. Cars get lots of people where they need to go. When you need a fleet of truck-and-trailer rigs, though, or container ships, or earth-movers, a car just won't do.

Python and Perl, by this analogy, are Toyota sedans. Erlang is an SUV. They're perfectly capable, and they get the kids to soccer practice better than anything else. Just don't confuse them with what you need for heavy work. It's easy to miss how much serious work is being done in C++, just as you don't much notice cargo ship traffic. You would certainly notice if it ever stopped.

What, again?

Posted May 2, 2007 7:52 UTC (Wed) by ms (subscriber, #41272) [Link]

Most reasonable people would see some form of memory-safety as a non-negotiable language design goal.

Not only that, but after a few months with Haskell, I see the preservation of type information a design goal. Which means any language based on sequential composition is out.

   e1:t1  e2:t2
   ------------   (SEQ)
     e1;e2:t2

Agh! Seriously, Haskell is really lovely language for doing I/O work in. The strength of the type system is a massive gain. And is anyone really going to claim that currying, higher-order functions and first class functions are features you can do without?

What, again?

Posted May 12, 2007 7:15 UTC (Sat) by jwalden (guest, #41159) [Link]

And is anyone really going to claim that currying [is a feature] you can do without?

Yes. The utility of currying depends heavily on how your libraries use higher-order functions. For an example from Standard ML, consider List.map mapFun lst; in code where you're using the same mapping function often enough, currying is a natural way to simplify the code. However, if the language doesn't support this in its libraries or if there's no syntactical support (JavaScript comes to mind as one such language), currying isn't that huge a win over just hardcoding the mapping function a bunch of places.

Don't get me wrong: currying is a very nice feature. However, in my experience its utility depends heavily on how the libraries you're working with make use of higher-order functions in their interfaces and on the syntax the language provides to use it.

Erlang much?

Posted May 1, 2007 22:22 UTC (Tue) by qu1j0t3 (subscriber, #25786) [Link]

Very much an engineering driven language... and functional to boot. (As another poster mentioned, it builds the among the most reliable and scalable concurrent systems out there.)

Erlang much?

Posted May 2, 2007 11:14 UTC (Wed) by NAR (subscriber, #1313) [Link]

Erlang is indeed functional, but I'm afraid its functional features are more like nuisances than strong points. The fact that you don't have destructive updates leads to variable names like HR0, HR1, etc. to hold the value after an update (at one place I even saw HR7). Of course, all data that would be in objects in Java are stored in processes, mnesia or ets tables where there is destructive update. The dynamic type system leads to lots of type errors during development. Actually I'm pretty surprised that it works, the VM underneath Erlang is really good (huge number of processes can be run, object files can be changed on the fly) - but I think it has not much to do with the language being functional.

Bye,NAR

Erlang much?

Posted May 3, 2007 5:00 UTC (Thu) by jbw (subscriber, #5689) [Link]

In message #232640 NAR wrote:

> Erlang is indeed functional, but I'm afraid its functional features
> are more like nuisances than strong points. The fact that you don't
> have destructive updates leads to variable names like HR0, HR1,
> etc. to hold the value after an update (at one place I even saw
> HR7).

This is actually one of the key strengths of Erlang. All of the run-time upgrading that is done in Erlang crucially depends on the lack of assignments (destructive updates).

What, again?

Posted May 2, 2007 1:30 UTC (Wed) by gdt (subscriber, #6284) [Link]

As usual, language choice depends on the problem domain. A huge advantage of functional languages is that they are implicitly parallel. Since >16 core processors seem heading for the desktop within the decade the ability to do parallel programming in an application without thinking about the issues of locking and the like could be very useful.

Object oriented programming was once a academic oddity too, until its good match to GUI programming bought it into the mainstream. I expect the need to efficiently program a large but unknown number of CPU cores will pull functional programming into the mainstream.

What, again?

Posted May 2, 2007 9:21 UTC (Wed) by schaueho (guest, #45025) [Link]

> Perhaps the greatest deficit of all these languages is their inability to
> manage resources outside the mathematical domain of the language. Toss in
> some file descriptors, sockets, database connections, or locks, and
> suddenly you're back to the Stone Age. This is a curious consequence of
> LISP's original memory management primitives, CONS and garbage collection.

Are you sure you have seen any functional language in the last five years? Perhaps you should have come out of that cage before throwing rocks. Take for instance any Common Lisp implementation (randomly chosen as an example because of your lisp bashing, not because I'm a Lisp zealot): You will find file handles, socket libraries, database access etc., typically nicely integrated. In particular, the Common Lisp condition system (think of it as an extended framework for exception handling) eases handling the nitty-gritty error situations that occur in production systems.

I think your comment is really funny, because the "rise of functional languages" also has to do with Python and Ruby, which do have a strong functional influence and are typically chosen because of their easy (scripting) integration with todays technologies.

What, again?

Posted May 2, 2007 18:29 UTC (Wed) by ncm (subscriber, #165) [Link]

Yes, Common Lisp, like C, has access to file handles and sockets. However, as in C, you cannot abstract management of them. To be very precise, if you put a socket in some larger data structure, you must explicitly call some cleanup function to ensure it gets closed in a timely fashion. You cannot automate this management. You cannot write a library which invisibly embeds a socket without exposing to the user a need to call your own cleanup function. ("Finalization" is no help, and in real systems has proven actually harmful.)

Without the ability to automate such management, exception handling facilities are a cruel joke, unable to concentrate error handling in a few well-tested spots. Instead, you get "finally" clauses, hundreds or thousands of separate snippets of error handling code scattered throughout one's program, as hard to exercise as error-code returns, and as likely to be wrong.

The true cost of most languages' failure to provide the rudiments of resource management is hard to bound. It ends up touching everything.

What, again?

Posted May 3, 2007 6:34 UTC (Thu) by flewellyn (subscriber, #5047) [Link]

To be very precise, if you put a socket in some larger data structure, you must explicitly call some cleanup function to ensure it gets closed in a timely fashion. You cannot automate this management. You cannot write a library which invisibly embeds a socket without exposing to the user a need to call your own cleanup function.

AHEM. I must be imagining the existence of the WITH-OPEN-FILE macro in the Common Lisp standard, which does exactly this: automates the closing of a file, invisibly, such that the user need not be concerned with explicitly calling the CLOSE function. Within the macro body, the file is referenced by the variable you give the macro as an argument; once control leaves the body of the macro, you are guaranteed that the file has been closed. Even if code contained in the body signals an error, the file WILL close.

Under the hood, of course, the macro expands into an UNWIND-PROTECT special form, in which the code that calls the necessary OPEN function and your code to work with the file are wrapped, along with the cleanup code that CLOSEs the file, which UNWIND-PROTECT guarantees will run when the form exits, normally or otherwise.

You'll see this idiom all over the place in Common Lisp libraries of all types. For instance, sockets are not part of the standard, since it was written before Berkeley Sockets became The One True Networking Way, but every socket library I've seen for CL has a WITH-OPEN-SOCKET macro that handles things the same way. Anywhere you have a resource that needs to be handled in a body of code and then disposed of, the rule is the same: write a WITH-style macro that wraps the opening and closing code in UNWIND-PROTECT, and then call the macro intead of worrying about the explicit managment.

So you see, you CAN abstract management of non-memory resources, such that they are always correctly cleaned up. You just need proper macro support and something akin to UNWIND- PROTECT.

What, again?

Posted May 3, 2007 7:15 UTC (Thu) by ncm (subscriber, #165) [Link]

This argument has already been disposed of elsewhere. Anyhow, it is not responsive: with-open-file is no help at all when your file descriptor is placed in a data structure returned to a caller, which is the case I described above.

The argument form used here is familiar: "Lisp is Good. Therefore, Lisp lacks nothing important. Therefore either the feature you describe is not important, or Lisp really has it after all. If in fact it is important and Lisp lacks it, let me describe some other feature which it does have, which you must agree is good." Congratulations on skipping over the middle bits. I will happily concede that with-open-file is positively jim-dandy, but not that it is an adequate substitute for the capacity to abstract management of non-memory resources.

There are sound reasons that Lisp, over the course of five decades, has failed to attract significant industrial use. Rather than engaging in apologetics, it would be better to understand industrial needs and design languages to meet those needs. C++ achieved its industrial position precisely this way. A Haskell or ML variant with destructors and without the equivalent of CONS might give C++ a run for its money. However, C++ isn't standing still. C++09 (and G++-4.x, for some small x) will support concepts, which is another fundamental advance.

What, again?

Posted May 3, 2007 7:28 UTC (Thu) by flewellyn (subscriber, #5047) [Link]

Okay, fair enough. So I slightly misunderstood your point.

Your point, it seems, is "If I do X which breaks the automatic tools we have to manage resources,
then I have to manually manage the resources! This is bad!"

My response is: so don't do that, then.

But aside from that, you've spent a lot of time saying "current languages don't offer a way to
manage resources the Right Way." What is this Right Way?

Rather than engaging in polemics, it would be better to help us understand just what it is you're
looking for in a functional language that doesn't do what you want.

What, again?

Posted May 9, 2007 6:13 UTC (Wed) by ncm (subscriber, #165) [Link]

"My response is: so don't do that, then."

I.e., don't try to write industrial code in Lisp. And I don't. Which is to say, you didn't slightly misunderstand my point; you entirely missed it.

Functional or not functional is irrelevant to the discussion. It would not be especially difficult to invent a new language with a strong type system, and destructors, as powerful as C++ but about a tenth the size. Functional semantics might make it easier to specify, and to optimize. For destructors to help, it would need to be non-garbage-collected, or it would need to offer some way to indicate that an allocation is meant to be GC'd, and prevent anything with a destructor from being allocated that way.

Block scope is the most powerful organizational method we have in programming. "With-foo" macros correctly tie object lifetime to a scope. However, they tie it to the wrong scope. Frequently the correct lifetime matches the block of some caller up the chain from the function that claimed the resource. Just a little less frequently, it's some other block entirely. Lisp offers no mechanism to express this, and neither can any other language that lacks destructors.

The consequence for exception handling is a little harder to explain, but just as large in its effect on programs. In a well-constructed C++ program, cleanup code is almost all in destructors, and exercised routinely in the course of running the program. There are no "finally" clauses, and very few "catch" clauses. In other words, in a "throw", very little code is executed that would not be run anyway. Exceptions are routed to just a few easily tested collection points. Omit destructors, and cleanup code has to be put in "finally" clauses scattered throughout the program, typically difficult or even impossible to test.

What, again?

Posted May 9, 2007 6:44 UTC (Wed) by flewellyn (subscriber, #5047) [Link]

No, I did not mean "don't write industrial code in Lisp".

I meant "don't go out of your way to break your tools".

Honestly, I think I agree with the commenter who said that you present a prejudice, not an
argument.

What, again?

Posted May 3, 2007 14:13 UTC (Thu) by jschrod (subscriber, #1646) [Link]

Your rabid polemics simply show that you don't know Common Lisp -- or O'Caml, to prevent your "you are a stupid Lisp fanatic who doesn't know a thing about Real Programming(tm)" bashing that you have so eagerly thrown at the GP and at several other posters.

If one wants them, CLOS has explicit constructors with full control over object lifetime and the ability to manually code resource management as needed. That's what MOP is for.

There are quite some technical and management problems when one wants to use CL or O'Caml for application development, but automatic or explicit resource management ain't one. And this is also not the reason why such languages aren't widely used, either -- reasons to select a programming language are most often non-technical.

What, again?

Posted May 3, 2007 18:47 UTC (Thu) by ncm (subscriber, #165) [Link]

"Rabid polemics"? I don't recall using words like "rabid", "stupid", or "fanatic". Maybe I'm being confused with somebody else, or maybe somebody is taking objective analysis of computer language features personally.

Nobody has said Lisp makes constructors impossible, or even difficult. Nobody has said Lisp makes manual resource management impossible, or difficult. What is impossible in Lisp, as in other GC-dependent languages, is any equivalent to destructors, and (therefore) the abstraction capabilities they enable. It's a fundamental language-design choice: provide tools so users can build resource management in libraries, or build exactly one kind of resource management into the language core. The latter approach is favored in academic languages meant to impress professors, the former has become essential in a language meant for industrial use.

For easy problems, any language will do -- perhaps badly, but few notice. For large, hard problems, language choice can make the difference between success and failure. Lisp has had very few industrial successes despite a head start measured in decades. The reasons are worth investigating if you want to be responsible for industrial successes. If you don't, you have lots of company.

What, again?

Posted May 4, 2007 9:13 UTC (Fri) by dcoutts (subscriber, #5387) [Link]

I think I see the point you've been trying to make and we've mostly all been missing.

You're saying that if we have some library function which returns an object and that object embed but does not expose some expensive resource (db connection or whatever) then in a normal GC language we can't guarantee that the DB connection gets released in a timely manner.

This is kind of true. Note though that in C++ it relies on the user, the recipient of the object, to free that object it a timely manner. That is, it relies manual memory management everywhere. If the caller is using a local stack allocated variable that's probably going to be ok, but if they're dynamically allocating them then we're relying on the user to know when to free them.

In a sense we cannot abstract this anyway, since the caller really does need to know that the resource is expensive and that they should not hold onto them for too long or have too many allocated at once. Putting expensive objects inside other structures and claiming it's abstract doesn't really help, it's still an expensive object that somehow needs to be treated carefully.

In the same way, in a mostly GC'd language where for this expensive resource we have to use more careful block scoping or explicit resource release methods we get exactly the same lack of abstraction. Any object which embed this expensive object has to also use more careful resource allocation techniques. In either case, this cost is always imposed on the caller, you can't not know about the high cost of the object and get a well performing program.

The difference is that in a GC'd language it looks more explicit because for all the normal objects that do not embed expensive resources we get to use a less obtrusive mechanism. In C++ we have to pay the cost of manual memory management everywhere, so it doesn't look much worse to deal with the expensive object case.

What, again?

Posted May 4, 2007 16:28 UTC (Fri) by schaueho (guest, #45025) [Link]

Speaking of running code on object destruction, SBCL includes an extension called sb-ext:finalize, doing the obvious. I would be surprised to find that only SBCL should provide something like that (actually a quick goggle search tells me that there are all sorts of implementations of this, summed up in clocc). Of course, not part of the standard means not part of the language, and as you also point out, it's an additional hassle.

What, again?

Posted May 7, 2007 1:06 UTC (Mon) by ncm (subscriber, #165) [Link]

No, finalization is no substitute. In fact it is actively harmful. You have no way to know when a finalization routine will happen, whether it will ever happen, what thread it will happen in, what order a series of them will be called in, or what else may be going on when they fire off. Competent Java shops typically forbid finalization except for debugging purposes, to help catch missed manual close operations.

What, again?

Posted May 7, 2007 0:59 UTC (Mon) by ncm (subscriber, #165) [Link]

No, I'm afraid you have completely missed the point.

In fact, in C++, the recipient of the object does not need to free it in a timely manner. Libraries do not rely on manual memory management everywhere. As I have noted elsewhere in this thread, it has been many years since I coded a "delete" statement. Where is this "cost of manual memory management" I am supposed to be paying everywhere? C++ coders don't pay any such cost.

In fact you can, in C++, abstract management of scarce resources (anyway). It may be necessary for the caller to know an expensive resource is contained, but not for the caller to explicitly call a "close" or "free" function to release it. In fact, most such objects' lifetimes are bounded by some scope, but (and this is important; stop and consider carefully) not typically the scope in which they were created or claimed. It is trivial to arrange for the equivalent of a close to happen automatically in C++, but it remains entirely impossible in any typical GC language, particularly including Lisp and all its descendants.

Let me repeat that: Turing-completeness notwithstanding, this common need in industrial programming is one that Lisp cannot address at all, as a direct result of a fundamental weakness in the core language design. That fundamental weakness cannot be resolved so long as CONS and GC remain in the core language. Other languages that have adopted Lisp's CONS and GC (or their equivalent) suffer the same weakness, whatever their other strengths.

What, again?

Posted May 4, 2007 6:59 UTC (Fri) by schaueho (guest, #45025) [Link]

> Yes, Common Lisp, like C, has access to file handles and sockets.
> However, as in C, you cannot abstract management of them. To be very
> precise, if you put a socket in some larger data structure, you must
> explicitly call some cleanup function to ensure it gets closed in a
> timely fashion.

This part confirms my guess that you haven't worked with functional languages in the last five years. In Common Lisp you would write a macro that wraps around any access to your resource and ensures that whatever you need to do after some functionality gets executed (i.e. clean whatever in whatever way up). This *is* the abstraction you would provide to the user of your library/whatever. Just because it's different from what you would do in an imperative language (or make that a "language with explicit memory managment") like C doesn't mean it isn't possible.

> Without the ability to automate such management, exception handling
> facilities are a cruel joke, unable to concentrate error handling in a
> few well-tested spots. Instead, you get "finally" clauses, hundreds or
> thousands of separate snippets of error handling code scattered
> throughout one's program, as hard to exercise as error-code returns, and
> as likely to be wrong.

Take a look at the Common Lisp condition system, it will broaden your view
on "exception" handling a lot.

Posted May 9, 2007 5:37 UTC (Wed) by ncm (subscriber, #165) [Link]

1. You haven't been reading carefully.

2. You are mistaken; Lisp does in fact lack the important capability I identified. Of course it is only important in industrial work, and not in academic demonstrations. You haven't missed it. I certainly would.

3. Your attitude is common among Lisp fans, and suffices to explain why Lisp has not gained a foothold in industrial programming despite fifty years' promotion in academia.

What, again?

Posted May 3, 2007 16:03 UTC (Thu) by lysse (subscriber, #3190) [Link]

> Perhaps the greatest deficit of all these languages is their inability to manage resources outside the mathematical domain of the language. Toss in some file descriptors, sockets, database connections, or locks, and suddenly you're back to the Stone Age.

Good flamebait; of course, Genera is but one counter-example.

What, again?

Posted May 3, 2007 18:54 UTC (Thu) by ncm (subscriber, #165) [Link]

Google doesn't know about it -- unless you're talking about the Symbolics programming environment (R.I.P.)?

What, again?

Posted May 3, 2007 23:23 UTC (Thu) by lysse (subscriber, #3190) [Link]

Yes, the one that was 100% Lisp - except it was an OS, not just a programming environment; in order to move it to Alphas under OSF/1, they basically wrote a Symbolics 3600 emulator and shifted it wholesale.

(And as far as I'm concerned, that's where I leave this discussion. What you have voiced here is a prejudice, not an argument.)