Stallman on GCC, LLVM, and copyleft [LWN.net]

Stallman on GCC, LLVM, and copyleft

Posted Jan 29, 2014 10:33 UTC (Wed) by Del- (guest, #72641)
In reply to: Stallman on GCC, LLVM, and copyleft by PaXTeam
Parent article: Stallman on GCC, LLVM, and copyleft

Sorry, I am still in the dark as to what nonsense you are talking about. Maybe I was unclear on something, I will try to my best ability to clarify things for you, maybe then you can tell me you mean. Please avoid the ad hominems.

Yes I said stable APIs are the back-bone of successful open projects, and I mean it. Linux has a rock stable API, as does glibc. Breaking backwards compatibility is typically seen as a bug. I just wish that had been common practise for the rest of the GNU/Linux stack, whether it is sound, gui-toolkits, process communication, etc.

When it comes to linux drivers, we are not talking about an API, but about an ABI. It is somewhat different, but as I said, there is not supposed to be an API for external use there. People are expected to include the drivers as part of the distribution. This is different.

Talking about GCC, it does make sense to share what you mean with API. For most people I believe they are occupied by linking issues, which in the case of C++ refers to the ABI implemented by GCC that was originally specified by Intel for their Itanium architecture. It made it possible for the likes of Intel to make compilers that could even link in statically compiled GCC stuff. There were issues preventing this from being rock solid with C++, but I believe it worked out reasonably well. You find a reference here: http://refspecs.linuxbase.org/cxxabi-1.83.html

In the case of nix, I can of course guess that it is a reference to the newer plugin API, for which you find documentation here: http://gcc.gnu.org/wiki/GCC_PluginAPI
By now I should have checked of course, but maybe you can share with me, is this the API you are struggling with? I haven't worked with that API myself, but I have seen GCC developers being sorry for not having it more stable. Whether they are just being modest, or whether they really have broken the API numerous times I don't know, and to be frank, I do not find you line of posting very helpful in conveying it either.

You see, the example from Octave using LLVM as a JIT library, is AFAIK, and intended usage of LLVM by an external project, and then whether the API is stable or not is a rather important issue, at least for me. I may of course be wrong, that Octave's usage was never intended for external usage of LLVM, so that we are more talking about an internal API, which in my mind is a different ball game.

to post comments

Stallman on GCC, LLVM, and copyleft

Posted Jan 29, 2014 15:43 UTC (Wed) by andreasb (guest, #80258) [Link] (2 responses)

When it comes to linux drivers, we are not talking about an API, but about an ABI. It is somewhat different, but as I said, there is not supposed to be an API for external use there.

Do you even know what these words mean? We are talking about an API. The Linux kernel internal API, which is what is relevant with regard to drivers and anything else running in kernel space, is unstable and will be changed whenever an improvement comes along that needs it.

That the ABI for kernel modules changes is so far out of the question that it is hardly noteworthy. Even a simple configuration change can change the ABI.

Stallman on GCC, LLVM, and copyleft

Posted Jan 29, 2014 19:23 UTC (Wed) by Del- (guest, #72641) [Link] (1 responses)

> Do you even know what these words mean? We are talking about an API. The Linux kernel internal API, which is what is relevant with regard to drivers and anything else running in kernel space, is unstable and will be changed whenever an improvement comes along that needs it.

Sigh, yes. Linux has two APIs, the internal and user space. APIs are at the source code level. The concept is quite general, but it often consists of calls to library routines. An ABI is an interface between binaries where one is typically a library or an operating system. In the case of linux there has been a lot of focus on that lack of an ABI towards drivers, or kernel modules in linux lingo. The lack of a stable ABI for drivers typically means that you need access to source code for the drivers in order to support upcoming kernels, since you often need to recompile. Regarding the linux drivers, most people care about the ABI, not the internal linux API, so I thought that was more relevant for the discussion. People often mix the concept of API and ABI, but as I already have said, they are not the same although they are clearly related concepts. Not sure how this confusion came about, but I guess I had one post where I was unclear.

Stallman on GCC, LLVM, and copyleft

Posted Jan 29, 2014 20:32 UTC (Wed) by PaXTeam (guest, #24616) [Link]

> In the case of linux there has been a lot of focus on that lack of an ABI towards drivers,

there's no lack of ABI, it's mostly defined by the usual GNU toolchain and some kernel side decisions and it doesn't really change, certainly much much less than the API.

> or kernel modules in linux lingo

no, not all kernel modules are drivers.

> The lack of a stable ABI for drivers typically means that

you don't know what you're talking about again. the ABI is quite stable, what is not stable is the API and no amount of recompilation will help if the source code itself is not patched to accomodate the changed API.

> Regarding the linux drivers, most people care about the ABI, not the internal linux API, so I thought that was more relevant for the discussion.

most people who care about the API actually understand the difference.

> People often mix the concept of API and ABI,

you don't say ;).

> but as I already have said, they are not the same although they are clearly related concepts.
> Not sure how this confusion came about, but I guess I had one post where I was unclear.

this will be a fun day, keep it coming! ;)

Stallman on GCC, LLVM, and copyleft

Posted Jan 29, 2014 19:23 UTC (Wed) by PaXTeam (guest, #24616) [Link] (3 responses)

> Sorry, I am still in the dark [...]

that's about the only statement you got right ;).

> Please avoid the ad hominems.

not before you learn to follow your own advice.

> Yes I said stable APIs are the back-bone of successful open projects, and I mean it.

so according to you linux and gcc are not successful.

> Linux has a rock stable API

wrong. the kernel's API has nothing to do with the syscall interface it provides to userland. the former is for kernel code, modules included, not userland and is intentionally not stable.

> Breaking backwards compatibility is typically seen as a bug.

linux does it all the time. please submit your bugreport to lkml. we all need a good laugh every now and then after all ;).

> When it comes to linux drivers, we are not talking about an API, but about an ABI.

wrong. you don't have a clue what these words mean, do you ;).

> It is somewhat different,

they're not somewhat different, but very different things. they refer to different levels of abstraction.

> but as I said, there is not supposed to be an API for external use there.

this sentence makes no sense whatsoever.

> People are expected to include the drivers as part of the distribution.

what does being in-tree vs. out-of-tree even have to do with the kernel API?

> This is different.

what is different?

> Talking about GCC, it does make sense to share what you mean with API.

are you saying that you didn't understand nix's response about said API? why did you even respond then? besides trolling, that is. and i did point you at said API, you can read the code or references to the code in that header.

> For most people I believe they are occupied by linking issues,

wrong, the API is not about 'linking issues'.

> which in the case of C++ refers to the ABI implemented by GCC

wrong, the gcc API that nix talked about has nothing to do with any C++ or other ABI. but then you already knew that difference, didn't you? ;)

> In the case of nix, I can of course guess that it is a reference to the newer plugin API

there's no gcc plugin API. there're a few plugin specific functions to register callbacks but other than that all public gcc symbols are available for plugin (ab)use. and this interface (functions and data structures, macros, etc) is not stable at all as you can see from my feeble attempt.

> By now I should have checked of course, but maybe you can share with me, is this the API you are struggling with?

i'm not the one struggling here with anything ;).

> I haven't worked with that API myself,

then on what grounds did you react to nix's statement?

> but I have seen GCC developers being sorry for not having it more stable.
> Whether they are just being modest, or whether they really have broken the API
> numerous times I don't know, and to be frank, I do not find you line of posting
> very helpful in conveying it either.

that's because you don't understand anything about this whole programming business yet it seemingly doesn't prevent you from posting nonsense after nonsense. you're still back at square one: figure out what API means, figure out what it is in the gcc world (never mind linux and others) and then figure out how stable they are and why. so do your homework before you spread your stupid claims about API stability.

> You see, the example from Octave using LLVM as a JIT library, is AFAIK, and intended usage of LLVM by an external project,
> and then whether the API is stable or not is a rather important issue, at least for me.

i can't imagine how anything programming related could be important for you ;). as for the API exposed by LLVM, it's the same deal as with gcc, all public symbols are there to take (over 40k of them at last count) and there're changes in there every day.

> I may of course be wrong, that Octave's usage was never intended for external usage of LLVM, so that we are more talking about an internal API, which in my mind is a different ball game.

what is LLVM's internal API? and what is 'external usage of LLVM'? and how's this 'external usage' different from LLVM's internal API?

Stallman on GCC, LLVM, and copyleft

Posted Jan 29, 2014 21:20 UTC (Wed) by Del- (guest, #72641) [Link] (1 responses)

> wrong. the kernel's API has nothing to do with the syscall interface it provides to userland. the former is for kernel code, modules included, not userland and is intentionally not stable.

"The kernel user-space API is the API that allows programs in user space to access system resources and services of the Linux kernel. This API, by choice, never breaks."
ref. http://en.wikipedia.org/wiki/Linux_kernel_API#Kernel_user...

> wrong, the gcc API that nix talked about has nothing to do with any C++ or other ABI.

For GCC the APIs most people are interested in are in the libs. I showed the ABI part, but there is also an API for C++ too of course, here you are: http://gcc.gnu.org/onlinedocs/libstdc++/api.html
I find the ABI story of C++ in GCC more fascinating, that's all.
A part from that I can only think of the plugin API I already gave you. Does even GCC define an internal API? AFAIK, internal APIs are typically not stable, so not sure how relevant they are for this discussion at all. The starting point for this discussion was usage of LLVM by external projects. So no, I have no clue as to what GCC API you go on about.

You'd better run along and edit wikipedia though, my patience with you is up.

Stallman on GCC, LLVM, and copyleft

Posted Jan 29, 2014 22:36 UTC (Wed) by PaXTeam (guest, #24616) [Link]

> The kernel user-space API

that's the *userland* API (syscalls), and not the kernel's API (which is used by kernel code, including modules, both in-tree or out-of-tree). which part of it is not clear to you? ;)

> For GCC the APIs most people are interested in are in the libs.

irrelevant, you were not biching about this kind of API regarding LLVM either, so don't try to change the subject. and the C++ ABI (i bet you don't know what that is either) was irrelevant here as well, noone was discussing it.

> Does even GCC define an internal API?

what is an 'internal API'? what makes it 'internal'?

> AFAIK, internal APIs are typically not stable, so not sure how relevant they are for this discussion at all.

because you were bitching about the unstable nature of the same kind of 'internal API' of LLVM as something bad. you can't have it both ways, that's what everyone's been trying to explain to you.

> The starting point for this discussion was usage of LLVM by external projects.

no it wasn't. rather it started with you making this claim:

> This sounds incredibly stupid. Stable API's is the back-bone of any successful open project.

and you were shown counter-examples (all GPL projects btw) and you still didn't recognize how idiotic your claim is. in other words, you were trolling and i'm having great fun at exposing it ;).

> So no, I have no clue as to what GCC API you go on about.

that's because you don't know what an API is. go educate yourself, it'd be a much better use of your time than trolling.

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 0:17 UTC (Thu) by nix (subscriber, #2304) [Link]

and i did point you at said API, you can read the code or references to the code in that header

Oh, if only that was all plugins had to conform to. Keeping them working would be simple! :)

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 0:15 UTC (Thu) by nix (subscriber, #2304) [Link] (12 responses)

Sorry, but in this case it *is* fairly clear that you don't know what you're talking about.

The GCC plugin API described on that page is merely a way by which dlopen()ed libraries can interpose themselves into the compiler's execution. Once there, they can (and generally must) inspect the program's intermediate representation and (possibly) modify it. The generation and manipulation of that intermediate representation is the true API that plugins must conform to, and it is *huge*. I mean, really huge, particularly if the plugin tries to change anything.

We're not just talking GENERIC and GIMPLE <ftp://gcc.gnu.org/pub/gcc/summit/2003/GENERIC%20and%20GIM...> (obsolete, of course, it is more complex now) and the lower-level RTL IR, we're not just talking the explicit dependencies expressed in the pass manager (many of which have names like 'late interprocedural optimization' where you have to know what 'late' means to understand it), we're not even talking just about the dependencies expressed in comments in the code, of which there are many -- but there are also wholly *implicit* dependencies, e.g. that a later pass implicitly expects particular properties of the representation provided by an earlier pass. Sometimes (much less often than in the past, but still true) the earlier pass doesn't even *know* it's providing those properties: they are only there by chance, and the later pass is also depending on them by chance: this is particularly likely to be true of any pass dating from the early 90s or before. That means they can be broken by chance! Worse yet, some properties are depended upon by only some machine definition files, which means they can be broken for *particular ports*, which you probably can't build.

Plus, of *course* it's continually changing: virtually anything other than a bugfix or a simple target or very simple language is going to add *some* new requirement somewhere, or perturb something somehow, and since plugins are necessarily exposed to virtually everything (not just 'can see virtually everything' but 'need to use a lot of it to be of any real use'), plugins can be, and are, perturbed by virtually everything.

(Now I'm overstating things here: lots of changes don't break any plugins at all, and lots of plugins will not be broken by a large percentage of changes that may break some plugins. But there are changes, and they are not rare, which necessarily break the lot, and possibly in quite subtle ways. This is true of the rest of the compiler as well, of course, since plugins are literally just dynamically loaded bits of the compiler: but the rest of the compiler has a 50,000+ test testsuite to make sure it doesn't regress. Plugins don't.)

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 0:51 UTC (Thu) by nix (subscriber, #2304) [Link]

which means they can be broken for *particular ports*, which you probably can't build

Or, rather, which you probably won't think to build, and can't test even if you built them.

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 1:11 UTC (Thu) by PaXTeam (guest, #24616) [Link] (2 responses)

> but the rest of the compiler has a 50,000+ test testsuite to make sure it doesn't regress. Plugins don't.

unless of course you run said regression tests with your plugins loaded :).

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 13:56 UTC (Thu) by nix (subscriber, #2304) [Link] (1 responses)

Yeah, and I'm sure the authors of major plugins do that. Basically nobody else does though, and you have to do a bunch of fiddling about with RUNTEST_FLAGS and perhaps even the .exp scripts to make it work.

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 19:11 UTC (Thu) by PaXTeam (guest, #24616) [Link]

i usually just set TEST_ALWAYS_FLAGS in site.exp to load the plugins.

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 7:27 UTC (Thu) by Del- (guest, #72641) [Link] (7 responses)

> Sorry, but in this case it *is* fairly clear that you don't know what you're talking about.

In the sense that I have not used the GCC plugin API, and cannot vouch for it's implementation. I thought I made that clear already, so I see no reason for you to fall down to Pax level in your rhetoric. I do very well know what an API is, but realise that bringing ABIs into the discussion should have been accompanied by a long explanation to avoid any possible confusion. I could of course repeat about three times more that I am sorry for any confusion caused, and that I should have thought twice and thrice how to formulate any sentence related to ABIs. Sorry.

When it comes to GCC it seems fairly well accepted that it is not a particularly modular code base, and this seems to be one of the key areas being prioritized for the next releases (making it more modular that is, I really cannot be too clear here if I want to avoid ad hominems). I believe this should enable an easier implementation of a stable and useful plugin API (I am making the assumption that you actually referred to usage of the plugin API to hook into GCCs optimizer routines in your post, terribly sorry if you did not). I appreciate your notes in this post, and find them infinitely more interesting than your first post. It also seems to portray how using GCCs plugin API can lead to similar cases as experienced by the Octave guys (I said similar not identical mind you, I really hate when I have to make this kind of comments because the whole discussion turns into an unpleasant witch hunt with strawmen on each corner).

Now please bring the discussion out of the pit hole instead of mud bathing with pax. Again, sorry for any confusion I may have caused, but this thread has continued a bit too long with a bit too many sloppy posts for me to stay sharp at every corner.

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 14:00 UTC (Thu) by nix (subscriber, #2304) [Link] (6 responses)

I do very well know what an API is,

Actually, from your comments it seemed very much like you didn't. APIs for random programs differ radically from the interfaces exposed to something like a plugin for a compiler: I was trying to describe how and why that was true. If you saw it as patronising 'mud bathing', I'm sorry.

When it comes to GCC it seems fairly well accepted that it is not a particularly modular code base, and this seems to be one of the key areas being prioritized for the next releases

Yes, but that will not help much. What's changing is e.g. that a bunch of code is moving out of macros into function pointers ('langhooks'). This will not change the fact that plugins are inspecting and possibly even changing a giant data structure manipulated by large numbers of other passes, and it will not greatly reduce the enormous number of implicit and explicit dependencies that plugins are thus exposed to. It'll make plugins more pleasant to write, definitely, but it won't mean they won't still get unavoidably broken at unpredictable intervals by changes in GCC. This is not a stable API in any real sense, nor do I see how it can ever become one. (The API by which plugins register themselves can be and is fairly stable, but that is, again, not the API plugins are really concerned with.)

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 15:46 UTC (Thu) by Del- (guest, #72641) [Link] (5 responses)

> Actually, from your comments it seemed very much like you didn't.

Yes, I can see two posts where I was very unclear, but unfortunately there is no edit button here, and I really did not get what pax was aiming at.

> This is not a stable API in any real sense, nor do I see how it can ever become one.

Thanks for the information. Sounds like the api does little more than expose the internals of GCC. Such an approach creates full flexibility for the plugins, but I do not think it s a sustainable design for a plugin API. As I said, I believe stable *external* APIs is one of the trademarks of successful open projects. But then somebody would need to design that API for GCC plugins as a stable layer between the plugins and the GCC internals.

Stallman on GCC, LLVM, and copyleft

Posted Jan 31, 2014 15:10 UTC (Fri) by nix (subscriber, #2304) [Link] (4 responses)

Sounds like the api does little more than expose the internals of GCC.

That's pretty much what it does. That's also pretty much what LLVM's does, and I have explained in other comments how I suspect this is unavoidable for anything remotely like a compiler. Most of its 'internals' are, after all, either exposed in or implied by the very IR whose exposure is the whole point of a plugin interface in the first place.

Stallman on GCC, LLVM, and copyleft

Posted Jan 31, 2014 15:46 UTC (Fri) by Del- (guest, #72641) [Link] (3 responses)

Then I am afraid the API is of little usage for anybody but GCC developers. I am not convinced it is hopeless though, there is probably a lot of functionality in GCC that can be of interest for external usage in a plugin setting, which does not force you into a global dependency spaghetti. Parsing comes to mind as something people have asked for. While IPO kind of takes in everything, I believe functionality like vectorization is more modular and local in nature. It is probably possible to split out rather high level stuff too, sacrificing the flexibility of the API for usability. For instance a high level JIT library (not sure how well it would work out for the Octave folks, but who knows) can be a good target since it is being designed as we speak.

Stallman on GCC, LLVM, and copyleft

Posted Feb 3, 2014 10:16 UTC (Mon) by nix (subscriber, #2304) [Link] (2 responses)

Yes, the sort of plugin that's likely to be more useful (and which does indeed survive GCC upgrades without major changes a lot of the time) is the sort that does only analysis or data extraction to an external destination. But if you want to actually *change* anything, well, Lisp macros this is not: expect frequent breakage.

Stallman on GCC, LLVM, and copyleft

Posted Feb 3, 2014 18:20 UTC (Mon) by PaXTeam (guest, #24616) [Link] (1 responses)

not my experience after like two dozen plugins, breakage has so far been trivial and independent of the plugin being passive or active (across gcc 4.5-4.9). i do however expect real breakage (of the kind that will force an effective rewrite) when tree/gimple/rtx enter the c++ conversion process.

Stallman on GCC, LLVM, and copyleft

Posted Feb 4, 2014 14:54 UTC (Tue) by nix (subscriber, #2304) [Link]

That's both good and bad news, I'd say... :)