LWN: Comments on "Using Common Lisp in Emacs"

Using Common Lisp in Emacs

bpearlmutter — Sat, 21 Dec 2024 15:22:15 +0000

Minor bug: you forgot to return h. You need an h after the (dolist ...).

Using Common Lisp in Emacs

jschrod — Thu, 16 Nov 2023 23:37:49 +0000

The article mentioned the statistics that roughly a third of emacs packages use cl-lisp.

Why do you claim that these 30+% of package writers are "just a vocal minority on the emacs-devel mailing list"?

What would you demand for other statistics that would prove the point valid that these developers see cl-lib as a common part of Elisp, if 30+% of developers using it is not enough?

Do you demand explicit documentation that they consider the features of cl-lib good and usable?

Using Common Lisp in Emacs

jschrod — Thu, 16 Nov 2023 23:27:08 +0000

Ok, I'm not anonymous, you can find my handle. I'm the maintainer of xindy in TeX Live which is written in CL.

I'm programming in Elisp and CL since 40 years. I don't think that "Elisp + CL" is bad.

Especially, I don't like intrusive hyperbole like "bad bad language" without any argument why that is so. Many people with decades of Lisp programming think otherwise.

IOW: Cut down in your quest to defend RMS and start to add something to the technical discussion.

Using Common Lisp in Emacs

Wol — Thu, 16 Nov 2023 23:02:46 +0000

Well, my programming language of choice is DataBASIC.

But as I say, my point is not that one language is better or worse than another, but that it shapes the way you see the world. The first languages you are taught often have a major impact on how easily you learn others. My first language was FORTRAN, and I'm sure (if you know FORTRAN), you would see the heritage even in the code I write today, 40 years on. As a simple example, when writing C I almost always use arrays, not pointers ... they may be functionally identical (near enough), but that's what seems natural to me. (And when given a C programming test by a recruitment consultant, my colleague and I got some of the highest scores the consultant had seen ...)

Cheers,
Wol

Using Common Lisp in Emacs

louai — Thu, 16 Nov 2023 22:28:54 +0000

Hi Ian,

The thing with Common Lisp is that there's a great manual to actually refer to. Of course cl-lib isn't a complete replacement for it, in the sense that it's incomplete. So I do agree that it's not a perfect situation.

I came at Emacs from the other direction - I learned Common Lisp first, and then elisp. From that perspective elisp in some ways feels "impoverished", in the same way that for someone looking at it in the opposite direction Common Lisp might appear baroque.

cl-lib fills many of those gaps with the useful delta that has accumulated over time. I think the gaps are real, because there exist various elisp modules to provide utilities that are similar to those that cl-lib provides, for example seq.el and dash.el.

There is nothing wrong with these other modules. But cl-lib has one advantage: it can actually draw from an ANSI standard that won't change. That gives the idioms that it implements, even if they aren't precise, complete implementations of what Common Lisp specifies, a certain stability and predictability. It's not going to go away, and I don't have to think about these things after I've learned them once.

For me this is a large part of the appeal of Common Lisp. The core is very stable and well specified. It has warts. It has many warts. But it's fundamentally quite sound and relatively coherent. And on top of it it's possible to implement just about anything. In some sense elisp is less stable - list manipulation libraries come and go, implementing and reimplementing the same thing in the currently fashionable style. This is because elisp is a very different thing from Common Lisp. Its core is smaller and doesn't need to be as general purpose.

I think people should be free to choose whichever approach they prefer. If you are at all a seasoned Lisp hacker you will not have problems following along. This isn't to say that everyone who contributes must be a master guru uber ninja. I'm saying there's room for both, and removing usages of cl-lib for the sake of removing usages of cl-lib isn't a very productive endeavor. And I think if you need to learn how some code that uses cl-lib or seq.el or dash.el works, you will find the overhead of learning these utilities to be smaller than the overhead of learning how the main module works. I just don't see it as an issue.

Using Common Lisp in Emacs

louai — Thu, 16 Nov 2023 20:07:21 +0000

Apologies for the self-reply, I accidentally hit publish when I meant preview. Here's an example from a personal project, to convert LEB128 octets to integers:


(declaim (inline unsigned->signed))
(defun unsigned->signed (n size)
  (logior n (- (mask-field (byte 1 (1- size)) n))))

(defun leb128->integer (chunks &optional signed)
  (loop for bits from 0 by 7
	for chunk in chunks
	for septet = (mask-field (byte 7 0) chunk)
	for result = septet then (dpb septet (byte 7 bits) result)
	finally (return (if signed
			    (unsigned->signed result bits)
			    result))))

It's a powerful tool that makes for very succinct loops. Like any powerful tool it does require some practice to use properly. I feel that is very much in the Emacs tradition though.

Using Common Lisp in Emacs

louai — Thu, 16 Nov 2023 20:01:15 +0000

Here is a detailed look at the LOOP macro. Admittedly it is divisive - some people love it, some hate it. But it's also very powerful. I'll pick an example from the link above with a couple comments:


;; List of 100 random numbers under 10000
(defparameter *random* (loop repeat 100 collect (random 10000)))

;; Iterate over *random* counting even and odd numbers, calculating
;; a total sum, and grabbing smallest and largest elements
(loop for i in *random*
   counting (evenp i) into evens
   counting (oddp i) into odds
   summing i into total
   maximizing i into max
   minimizing i into min
   finally (return (list min max total evens odds)))

Using Common Lisp in Emacs

sfink — Thu, 16 Nov 2023 18:52:48 +0000

I'm with @mti, I have a dim distant familiarity with Lisp, and the `cl-loop` version is much harder to understand.

First of all, what's the point of `with` bindings when they're outside of the loop anyway? Why should they be considered "part of the loop"? What's wrong with using `let*` to create an environment with the correct scope, given that that's what the programmer is expressing? I don't get it, and that made it harder for me to understand the `cl-loop` thing since I thought it couldn't possibly be doing the thing that seemed like a bad idea to do, so I tried to come up with some other meaning.

Second, the `for` keyword is absolutely baffling if you aren't already familiar. I would also assume it was doing some sort of triply-nested loop. If you want bindings in the loop body, wouldn't the most straightforward way to do that be to... put the bindings in the loop body?

`cl-loop`/`loop` just feels to me like it's trying too hard to be declarative or something, which creates a level-of-abstraction separation between the loop and its surroundings that feels jarring. Declarative is great. Imperative is ok. Mixing the two is a mess, and the benefits really have to pay for themselves.

I feel like I'm just not getting it. Which is unsurprising, since I've written almost no code in either CL or Elisp for multiple decades, and I'm unfamiliar with the conventions and space. So my opinion shouldn't carry much weight. But I'm offering it up as a datapoint about an external perspective.

Using Common Lisp in Emacs

mpr22 — Thu, 16 Nov 2023 18:32:07 +0000

You know the quote about "BASIC considered harmful"?

I do.

I also know people who studied computer science in the 1990s having had their first exposure to computer programming be 8-bit microcomputer BASICs; quite a few of those BASICs were arguably worse than the Dartmouth BASIC of 1975 that Dr Dijkstra was familiar with.

Their minds have never struck me as "mutilated".

Using Common Lisp in Emacs

Wol — Thu, 16 Nov 2023 17:11:34 +0000

> I honestly think the reason Lisp fans talk so glowingly about this approach is that CL is only used by hobbyists or small companies with low churn. (I do still like it a lot though, it was the second language I learned and I’d happily work with it over most mainstream languages.)

The other thing is, like me with Pick and Relational, Lisp is actually fundamentally different to other languages in many ways. Like Forth is fundamentally different. You know the quote about "BASIC considered harmful"?

The languages you know shape the way you think. For example, comparing those three languages, BASIC encourages/d long monolithic procedural code. There's no reason for it to be spaghetti, but it easily slips into it. Forth it's pretty much impossible to write monolithic code, it is structured functional code. Lisp, as its name implies, is very much data-driven structures. Those are three completely different ways of thinking, and for example comaparing C code written by a C expert with C code written by a Lisp expert who is good at C, even with them both tackling the same project their code will have a completely different feel. And quite possibly they would have difficulty understanding WHY the other one had done things a certain way, even if both ways worked very well. Lisp programmers think their approach is superior, and actually I believe the evidence bears that out. Even if writing C in the Lisp style actually results in a more efficient program!

Like me, my data tables have a completely different feel, in all likelihood, to a Relational guy. Because my Pick heritage means tables "just want" to be fourth normal form, because I do an EAR. My mindeset, the way I see data and code, is completely different to the way a Relational guy sees it. And of course, I think my way is best :-)

(The same analysis applies to natural languages. I'm multi-lingual - European languages only - and there are plenty of examples where different nations see things differently because only one language has the words to express something. Or the overtones of a direct translation give completely the wrong meaning - compare the English "borgeois" with the German "gut burgerlich", for example!)

Cheers,
Wol

Using Common Lisp in Emacs

Phantom_Hoover — Thu, 16 Nov 2023 13:02:25 +0000

I wouldn’t say you’re ‘misinformed’, this is very much my own hot take. But I have had to maintain and extend software written in a DSL by someone who was clearly bored and had since left. Fortunately it was a simple script that could be unpicked in an hour or two, and easily rewritten in the base language; if it had been substantially more complex, and used more of the DSL’s unique functionality, understanding what it was actually doing and fixing or extending it would have been extremely difficult — I’d probably still have extracted all the logic and rewritten it in the base language that I and my colleagues are familiar with.

One of the most difficult and important challenges in software design is making architectures that are flexible, extensible and maintainable by many people over many years, and language design is a fairly pure example of that. It’s insane to me that anyone thinks it’s a good choice of tool for problems like formatting a string or writing loops. Remember that CL’s format and loop DSLs are mature standards, despite all their complexity — imagine trying to work with them in the more common scenario where they’re legacy code, half finished and written by someone you can’t contact.

I honestly think the reason Lisp fans talk so glowingly about this approach is that CL is only used by hobbyists or small companies with low churn. (I do still like it a lot though, it was the second language I learned and I’d happily work with it over most mainstream languages.)

Using Common Lisp in Emacs

jem — Thu, 16 Nov 2023 06:51:27 +0000

In this specific case, the "domain" is writing loops in Lisp programs. That is, somebody thought normal Lisp syntax is not expressive enough, or gets too unwieldy for writing loops, so a totally new sub language had to be invented. Not only that, the sub language is very un-lispy, including, among other things, an infix "=" operator instead of the standard "let" form, for no obvious reason.

This reminded me of the attempt to introduce infix notation in Scheme, because prefix notation "does not feel natural". https://srfi.schemers.org/srfi-105/srfi-105.html

Using Common Lisp in Emacs

NYKevin — Thu, 16 Nov 2023 03:02:14 +0000

> The fact that some Lisp advocates seem to seriously think ‘you use macros to create a DSL to solve your problems’ is an incredible selling point frankly baffles me.

As a complete outsider to this discussion... every time I read someone's argument for Why Lisp Is Great™, DSLs are always at the top of the list. Have I been misinformed?

I don't use Lisp, so I can't judge for myself.

Using Common Lisp in Emacs

iabervon — Wed, 15 Nov 2023 17:57:31 +0000

I think that the syntax of keyword arguments in Common Lisp is very hard to follow if you learned a Lisp dialect without keyword arguments. It's a big departure from how members of an S-expression can relate to each other in R4RS Scheme, for example, where tokens in the middle of an S-expression simply don't gather up other tokens into implicit logical units.

Using Common Lisp in Emacs

louai — Wed, 15 Nov 2023 17:44:03 +0000

Of course those C++ assignments should be


    auto f = it.first;
    auto type_spec = it.second;

Using Common Lisp in Emacs

louai — Wed, 15 Nov 2023 17:41:08 +0000

Yes in Common Lisp it's just LOOP. You might write something like this:


(defun frob-type-specifiers (specifiers)
  (loop with comp-ctxt = (make-comp-cstr-ctxt)
        with h = (make-hash-table :test 'eq)
        for (f type-spec) in specifiers
        for cstr = (comp-type-spec-to-cstr type-spec)
        do (setf (gethash f h) cstr)
        finally return h))

Roughly equivalent to this C++ code with made-up types:


std::unordered_map<f_t, cstr_t>
frob_type_specifiers(const std::vector<std::pair<f_t, spec_t>> &specifiers) {
  auto comp_ctx = make_comp_cstr_ctxt();
  std::unordered_map<f_t, cstr_t> h;

  for (auto &it : specifiers) {
    auto f = it.second.first;
    auto type_spec = it.second.second;
    auto cstr = comp_type_spec_to_cstr(type_spec);
    h[f] = cstr;
  }

  return h;
}

To a trained eye the loop construct is much easier to grok since it encapsulates all the relevant variables in one place. The with clauses create variables that have block scope. The for clauses create variables that have loop iteration scope. The do clause does something on every iteration. The finally clause can be used to do something when the loop terminates.

Using Common Lisp in Emacs

mpr22 — Wed, 15 Nov 2023 17:39:14 +0000

If there's one thing having 20+ years experience in writing software has taught me, it's that very few things are actually obvious.

Using Common Lisp in Emacs

mti — Wed, 15 Nov 2023 14:05:25 +0000

OK, maybe not completely unreadable. I assumed that cl-loop was some kind of loop based on the name. But my experience from other programming languages would also lead me to assume that the two 'for' also introduced loops. So I assumed it was three nested loops. And 'finally' reminds me of finally in Java and other languages. But I don't think that is the meaning here.

So, almost, but not completely unreadable ;-)

There is a balance in how much abstraction you use at the syntax level. Short common idioms are good for the experienced developer but can be hard to understand for the occasional developer.

Or expressed in another way. If I was developing Common Lisp software I would probably use cl-loop myself (*). When occasionally looking at elisp code I would prefer that the code does not use cl-loop.

(*) I assume it is called just 'loop' in CL, 'cl-' just being a prefix used in elisp?

Using Common Lisp in Emacs

spacefrogg — Wed, 15 Nov 2023 13:01:14 +0000

The statement that cl-lib makes elisp a "mix of languages" is plain wrong. It is a set of abstractions written in elisp to mimic the API of CommonLisp. It's elisp the same way that the HTTP API is still elisp although it is talking HTTP.

What you are actually saying is: "It should not be allowed to write certain functions in elisp, because they would mimic functions in CommonLisp, which I don't like!" Well, you can do that, I suppose. I do not find this a valid opinion, though. Nobody, not you, not anyone else is telling me how to program and what functions to write to deliver a functionality. And if you want to meaningfully debug my code, you still have to understand how the functions, that I use, operate. So, forcing me to write my own abstractions, won't even help you debugging my code.

Emacs is a complex computer interface. It has to deal with all kinds of languages and also understands others than Elisp like XML or JS to provide valuable service to you. Do you want to forbid that as well just to keep a holy language clean?

People have put effort into writing emacs packages and they chose to use cl-lib, because it solved real problems for them that no other emacs developer or any programmer, including you, could be arsed to help them with.

I always find it funny when some "gray-beard developers" show up telling anyone how much easier it is reading the simple constructs of elisp and not having to read "110 page manual"s, while at the same time, newcomers, apparently, find it easier to just use the, oh so dreadful, cl-lib functions. How is it that those newcomers are so much cleverer learning new code constructs than you old big bosses?

I suspect it is you, who is completely besides the point. And I can, at the same time, completely respect the emacs maintainer's stance that they don't want do maintain code that needs cl-lib. Well, then they have to find somebody to clean up the code that exists and deter any newcomer from using cl-lib. And please, don't speak for "people". Speak for yourself. You have no idea what is "better for people" any more than I do.

Using Common Lisp in Emacs

spacefrogg — Wed, 15 Nov 2023 12:37:27 +0000

With all due respect, I don't believe a single word of "completely unreadable". You had a well enough understanding of the loop to identify it as such. In the elisp code, you literally have to first spot the dolist deep in the middle of the code block, to even identify that it is a loop, we are talking about. Otherwise it is just a stack of bindings with no context.

The loop, with all its weaknesses, I grant you that, captures the meaning of the code much better, because everything that belongs to the loop is encapsulated by it. To do such abstractions is an essential property of high-level programming languages.

I find arbitrarily selected "I reject any further abstractions from this point on" to be no contribution to any debate about computer programming. There are valid situations to reject needless abstractions, but capturing loops is certainly not one of them.

All programs invent abstractions. When those abstractions are not represented in the programming language, they are called protocols or API.

Using Common Lisp in Emacs

neggles — Wed, 15 Nov 2023 12:17:12 +0000

I'm not anonymous, and I'm basing "what people want to use" on the pretty significant pushback from other emacs devs described in this very article.

Using Common Lisp in Emacs

IanKelling — Wed, 15 Nov 2023 12:13:32 +0000

Note: This was meant to be a reply to the 1st comment in the article by louai.

Using Common Lisp in Emacs

IanKelling — Wed, 15 Nov 2023 12:09:40 +0000

> When I write elisp I routinely use cl-lib because

Your comment might as well be "I like javascript. It is better than emacs lisp. It is better because X, Y, and Z." Ok, and you are totally missed the point. You can argue that any number of other languages are better than Emacs Lisp, but that doesn't mean Emacs is better off being some mix of several languages. Yes, it is great to be fluent in many programming languages, and when they are interoperable, being able to mix them is fun. But that doesn't mean that is the best or even good way to write code for other people to use. You already learned it, so who shouldn't everyone else? Because we've got limited time and brainpower. I'm pretty sure everyone else is better off not having to slowly learn the 110 page manual of the "partial" common lisp implementation in order to modify and extend emacs https://www.gnu.org/software/emacs/manual/html_mono/cl.html. The language "emacs lisp + lots of cl-lib" seems to me a very poorly designed language with tons of redundancy, and poor choice for the use case of being the language of emacs. Even the unrealistic hypothetical idea of converting emacs into common lisp is a bad idea because elisp is better suited to the job in many ways. Imo, emacs should be doing even more to discourage cl code in common emacs packages.

Using Common Lisp in Emacs

IanKelling — Wed, 15 Nov 2023 11:41:41 +0000

Of course we had to have random unfounded RMS smear from an annonymous commenter who loses nothing.

> the reality of what end-users and other developers actually want to see/use.

"emacs lisp + common lisp" is obviously a bad bad language that emacs has good logical reasons to discourage. This is not an arbitrary RMS opinion.

Using Common Lisp in Emacs

mti — Wed, 15 Nov 2023 11:40:18 +0000

Actually for me the first version with cl-loop was completely unreadable while the version without cl-loop was mostly understandable from what I remember from the Scheme course I read 30 years ago.

When not using a language every day it is much easier to read code that uses a few simple constructs.

When using a language every day it may be different.

Using Common Lisp in Emacs

Phantom_Hoover — Wed, 15 Nov 2023 10:58:55 +0000

The code snippet you posted is in Lisp, which most Lisp programmers can be presumed to know. The snippet using loop is written in CL’s loop-macro sublanguage, which Elisp programmers are understandably entirely unfamiliar.

The fact that some Lisp advocates seem to seriously think ‘you use macros to create a DSL to solve your problems’ is an incredible selling point frankly baffles me. At best you’re reducing a variety of medium-difficulty problems into the hard problem of designing a decent computer language, and at worst… well if you asked me to maintain a legacy codebase full of ad-hoc half-finished DSLs I’d run a mile in the other direction.

Using Common Lisp in Emacs

Phantom_Hoover — Wed, 15 Nov 2023 10:47:02 +0000

Stallman’s fixation on keyword arguments in particular seems weird and obnoxious to have to work around, but it is at least consistent with his long record of fixating on trivial details and stubbornly taking a stance on them.

Using Common Lisp in Emacs

spacefrogg — Wed, 15 Nov 2023 10:33:20 +0000

I believe you know already that the loop code is equivalent to something like:

(let* ((comp-ctxt (make-comp-cstr-ctxt))
       (h (make-hash-table :test #'eq)))
  (dolist (x comp-known-type-specifiers h)
    (let* ((f (car x)
           (type-spec (cdr x))
           (cstr (comp-type-spec-to-cstr type-spec)))
      (puthash f cstr h))))

And now tell my how this, anywhere on earth, is easier to understand than the cl-loop macro. If you know anything about the cl-loop macro, you realise that "with ... =" is a let binding before the loop and "for ... =" is a let binding inside the loop. "for ... in" is obviously the loop itself. And even if you don't know cl-loop specifically. You can roughly infer the idea by just using common understanding of the English language and common use of trigger words like "with" and "for". Even elisp itself uses the with-* metaphor to signify temporary bindings. So I call this whole previous comment a straw man. You may dislike the loop macro anyway you want. I do, too. But calling it unreadable is not serving any constructive purpose.

Using Common Lisp in Emacs

jem — Wed, 15 Nov 2023 08:10:45 +0000

The original idea was to just replace the Emacs Lisp engine with Guile, while retaining Emacs Lisp as the extension language. GNU Guile supports Emacs Lisp and ECMAScript in addition to Scheme. I seem to recall there were differing opinions on whether to also allow Emacs extensions to be written in Scheme as well, if Guile had been integrated into Emacs.

Using Common Lisp in Emacs

jem — Wed, 15 Nov 2023 07:59:31 +0000

This is not just about keyword arguments. From the mailing list:

>Btw, the above is a very simple use of cl-loop. We have quite a few of much more complex ones. For example:
>(cl-loop
>  with comp-ctxt = (make-comp-cstr-ctxt)
>  with h = (make-hash-table :test #'eq)
>  for (f type-spec) in comp-known-type-specifiers
>  for cstr = (comp-type-spec-to-cstr type-spec)
>  do (puthash f cstr h)
>  finally return h)

Boy that is hard to understand.

Using Common Lisp in Emacs

epa — Wed, 15 Nov 2023 07:54:55 +0000

And then there’s replacing Emacs Lisp with Guile Scheme. That was someone’s project for a while although I think it was always viewed as a bit blue-sky.

Using Common Lisp in Emacs

jem — Wed, 15 Nov 2023 07:47:19 +0000

And the people wanting to replace Emacs Lisp with Common Lisp are not pushing their own personal ideology? Richard Stallman definitely has strong opinions on a lot of things, but here we are talking specifically about the programming language used for writing Emacs extensions.

I understand Stallman's fear that the simplicity of Emacs Lisp is at risk if it turns into a combination of two languages: Emacs Lisp and Common Lisp. Completely replacing Emacs Lisp with Common Lisp is impossible at this point.

Do you have any statistics to back up your claim that Common Lisp is "what end-users and other developers actually want to see/use"? Maybe it is just a vocal minority on the emacs-devel mailing list?

Using Common Lisp in Emacs

rsidd — Wed, 15 Nov 2023 07:45:35 +0000

And it's not even clear what's not to like. What's wrong with keyword arguments? It generally makes code easier to read.

Using Common Lisp in Emacs

neggles — Tue, 14 Nov 2023 21:32:46 +0000

Stallman is far more concerned with pushing his own personal ideology than with the reality of what end-users and other developers actually want to see/use.

"I don't like thing"
"Well, we like thing"
"But I don't like thing!"

Does an excellent job of alienating users and developers alike, and usually results in something that's good in theory but kind of mid in practice (e.g. GPLv3, which went a little too hard in a couple of aspects and shot itself in the foot by favouring ideology over reality)

Using Common Lisp in Emacs

louai — Tue, 14 Nov 2023 19:58:04 +0000

As an Emacs user I'm pretty saddened by this. In truth I think Common Lisp is in many ways superior to Emacs Lisp - there is a real spec, many implementations, and the language in general is just very rigorous and full of thoughtful little features. When I write elisp I routinely use cl-lib because it adds so many useful things - ranging from small things like default values for optional arguments in cl-defun, to larger things like cl-pushnew as mentioned in the article that simplify code. Why reinvent the wheel on these when a whole bunch of very smart people spent a lot of time figuring out good ways to it?