Stallman on GCC, LLVM, and copyleft [LWN.net]

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 0:15 UTC (Thu) by nix (subscriber, #2304)
In reply to: Stallman on GCC, LLVM, and copyleft by Del-
Parent article: Stallman on GCC, LLVM, and copyleft

Sorry, but in this case it *is* fairly clear that you don't know what you're talking about.

The GCC plugin API described on that page is merely a way by which dlopen()ed libraries can interpose themselves into the compiler's execution. Once there, they can (and generally must) inspect the program's intermediate representation and (possibly) modify it. The generation and manipulation of that intermediate representation is the true API that plugins must conform to, and it is *huge*. I mean, really huge, particularly if the plugin tries to change anything.

We're not just talking GENERIC and GIMPLE <ftp://gcc.gnu.org/pub/gcc/summit/2003/GENERIC%20and%20GIM...> (obsolete, of course, it is more complex now) and the lower-level RTL IR, we're not just talking the explicit dependencies expressed in the pass manager (many of which have names like 'late interprocedural optimization' where you have to know what 'late' means to understand it), we're not even talking just about the dependencies expressed in comments in the code, of which there are many -- but there are also wholly *implicit* dependencies, e.g. that a later pass implicitly expects particular properties of the representation provided by an earlier pass. Sometimes (much less often than in the past, but still true) the earlier pass doesn't even *know* it's providing those properties: they are only there by chance, and the later pass is also depending on them by chance: this is particularly likely to be true of any pass dating from the early 90s or before. That means they can be broken by chance! Worse yet, some properties are depended upon by only some machine definition files, which means they can be broken for *particular ports*, which you probably can't build.

Plus, of *course* it's continually changing: virtually anything other than a bugfix or a simple target or very simple language is going to add *some* new requirement somewhere, or perturb something somehow, and since plugins are necessarily exposed to virtually everything (not just 'can see virtually everything' but 'need to use a lot of it to be of any real use'), plugins can be, and are, perturbed by virtually everything.

(Now I'm overstating things here: lots of changes don't break any plugins at all, and lots of plugins will not be broken by a large percentage of changes that may break some plugins. But there are changes, and they are not rare, which necessarily break the lot, and possibly in quite subtle ways. This is true of the rest of the compiler as well, of course, since plugins are literally just dynamically loaded bits of the compiler: but the rest of the compiler has a 50,000+ test testsuite to make sure it doesn't regress. Plugins don't.)

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 1:11 UTC (Thu) by PaXTeam (guest, #24616) [Link] (2 responses)

> but the rest of the compiler has a 50,000+ test testsuite to make sure it doesn't regress. Plugins don't.

unless of course you run said regression tests with your plugins loaded :).

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 13:56 UTC (Thu) by nix (subscriber, #2304) [Link] (1 responses)

Yeah, and I'm sure the authors of major plugins do that. Basically nobody else does though, and you have to do a bunch of fiddling about with RUNTEST_FLAGS and perhaps even the .exp scripts to make it work.

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 19:11 UTC (Thu) by PaXTeam (guest, #24616) [Link]

i usually just set TEST_ALWAYS_FLAGS in site.exp to load the plugins.

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 7:27 UTC (Thu) by Del- (guest, #72641) [Link] (7 responses)

> Sorry, but in this case it *is* fairly clear that you don't know what you're talking about.

In the sense that I have not used the GCC plugin API, and cannot vouch for it's implementation. I thought I made that clear already, so I see no reason for you to fall down to Pax level in your rhetoric. I do very well know what an API is, but realise that bringing ABIs into the discussion should have been accompanied by a long explanation to avoid any possible confusion. I could of course repeat about three times more that I am sorry for any confusion caused, and that I should have thought twice and thrice how to formulate any sentence related to ABIs. Sorry.

When it comes to GCC it seems fairly well accepted that it is not a particularly modular code base, and this seems to be one of the key areas being prioritized for the next releases (making it more modular that is, I really cannot be too clear here if I want to avoid ad hominems). I believe this should enable an easier implementation of a stable and useful plugin API (I am making the assumption that you actually referred to usage of the plugin API to hook into GCCs optimizer routines in your post, terribly sorry if you did not). I appreciate your notes in this post, and find them infinitely more interesting than your first post. It also seems to portray how using GCCs plugin API can lead to similar cases as experienced by the Octave guys (I said similar not identical mind you, I really hate when I have to make this kind of comments because the whole discussion turns into an unpleasant witch hunt with strawmen on each corner).

Now please bring the discussion out of the pit hole instead of mud bathing with pax. Again, sorry for any confusion I may have caused, but this thread has continued a bit too long with a bit too many sloppy posts for me to stay sharp at every corner.

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 14:00 UTC (Thu) by nix (subscriber, #2304) [Link] (6 responses)

I do very well know what an API is,

Actually, from your comments it seemed very much like you didn't. APIs for random programs differ radically from the interfaces exposed to something like a plugin for a compiler: I was trying to describe how and why that was true. If you saw it as patronising 'mud bathing', I'm sorry.

When it comes to GCC it seems fairly well accepted that it is not a particularly modular code base, and this seems to be one of the key areas being prioritized for the next releases

Yes, but that will not help much. What's changing is e.g. that a bunch of code is moving out of macros into function pointers ('langhooks'). This will not change the fact that plugins are inspecting and possibly even changing a giant data structure manipulated by large numbers of other passes, and it will not greatly reduce the enormous number of implicit and explicit dependencies that plugins are thus exposed to. It'll make plugins more pleasant to write, definitely, but it won't mean they won't still get unavoidably broken at unpredictable intervals by changes in GCC. This is not a stable API in any real sense, nor do I see how it can ever become one. (The API by which plugins register themselves can be and is fairly stable, but that is, again, not the API plugins are really concerned with.)

Stallman on GCC, LLVM, and copyleft

Posted Jan 30, 2014 15:46 UTC (Thu) by Del- (guest, #72641) [Link] (5 responses)

> Actually, from your comments it seemed very much like you didn't.

Yes, I can see two posts where I was very unclear, but unfortunately there is no edit button here, and I really did not get what pax was aiming at.

> This is not a stable API in any real sense, nor do I see how it can ever become one.

Thanks for the information. Sounds like the api does little more than expose the internals of GCC. Such an approach creates full flexibility for the plugins, but I do not think it s a sustainable design for a plugin API. As I said, I believe stable *external* APIs is one of the trademarks of successful open projects. But then somebody would need to design that API for GCC plugins as a stable layer between the plugins and the GCC internals.

Stallman on GCC, LLVM, and copyleft

Posted Jan 31, 2014 15:10 UTC (Fri) by nix (subscriber, #2304) [Link] (4 responses)

Sounds like the api does little more than expose the internals of GCC.

That's pretty much what it does. That's also pretty much what LLVM's does, and I have explained in other comments how I suspect this is unavoidable for anything remotely like a compiler. Most of its 'internals' are, after all, either exposed in or implied by the very IR whose exposure is the whole point of a plugin interface in the first place.

Stallman on GCC, LLVM, and copyleft

Posted Jan 31, 2014 15:46 UTC (Fri) by Del- (guest, #72641) [Link] (3 responses)

Then I am afraid the API is of little usage for anybody but GCC developers. I am not convinced it is hopeless though, there is probably a lot of functionality in GCC that can be of interest for external usage in a plugin setting, which does not force you into a global dependency spaghetti. Parsing comes to mind as something people have asked for. While IPO kind of takes in everything, I believe functionality like vectorization is more modular and local in nature. It is probably possible to split out rather high level stuff too, sacrificing the flexibility of the API for usability. For instance a high level JIT library (not sure how well it would work out for the Octave folks, but who knows) can be a good target since it is being designed as we speak.

Stallman on GCC, LLVM, and copyleft

Posted Feb 3, 2014 10:16 UTC (Mon) by nix (subscriber, #2304) [Link] (2 responses)

Yes, the sort of plugin that's likely to be more useful (and which does indeed survive GCC upgrades without major changes a lot of the time) is the sort that does only analysis or data extraction to an external destination. But if you want to actually *change* anything, well, Lisp macros this is not: expect frequent breakage.

Stallman on GCC, LLVM, and copyleft

Posted Feb 3, 2014 18:20 UTC (Mon) by PaXTeam (guest, #24616) [Link] (1 responses)

not my experience after like two dozen plugins, breakage has so far been trivial and independent of the plugin being passive or active (across gcc 4.5-4.9). i do however expect real breakage (of the kind that will force an effective rewrite) when tree/gimple/rtx enter the c++ conversion process.

Stallman on GCC, LLVM, and copyleft

Posted Feb 4, 2014 14:54 UTC (Tue) by nix (subscriber, #2304) [Link]

That's both good and bad news, I'd say... :)