Replacing AWK with Python in GCC? [LWN.net]

Replacing AWK with Python in GCC?

Posted Jul 25, 2018 18:26 UTC (Wed) by nirbheek (subscriber, #54111)
Parent article: Replacing AWK with Python in GCC?

I've never understood this mentality in open source, that when someone comes along to cleanup a horribly messed up and unmaintainable piece of crap part of the code in a particular way, suddenly the *entire community* has an opinion on how it has to be done and *none* of those people are actually volunteering to help, just volunteering an opinion.

"Oh cool, you want to fix this? I support you fully, but only if you do it the way I would've done it."

Well, guess what you're not doing it, so the choice is not "Python 3 or Python 2 or Perl or Guile", it's "Python 3 or status-quo of dumpster fire".

For god's sake it's not Java, why is there so much bikeshedding over it.

to post comments

Replacing AWK with Python in GCC?

Posted Jul 25, 2018 18:55 UTC (Wed) by smoogen (subscriber, #97) [Link] (3 responses)

I have the same reaction too, but have come to realize it is basic human things. It is no different with auto mechanics, building houses, or the various sciences. Everyone has grumbled about getting deionized water into the labs, but done nothing about it. Then when someone comes up with a pipe system, every chemist is now a plumbing expert and complain that their preferred method wasn't used. They all think they are helping make things better but it is as much about humans being hard wired to keep status quo for safety as it is "well maybe my idea I have been sitting on for 20 years could get done since you are actually going to do something about it."

Replacing AWK with Python in GCC?

Posted Jul 26, 2018 4:45 UTC (Thu) by marcH (subscriber, #57642) [Link] (2 responses)

> It is no different with auto mechanics, building houses, or the various sciences.

Software and especially software maintenance is unlike any other kind. It is very different.

Replacing AWK with Python in GCC?

Posted Jul 30, 2018 7:08 UTC (Mon) by Rudd-O (guest, #61155) [Link] (1 responses)

It isn't at all different in the sense that people suddenly become "experts" when their cheese is moved.

Replacing AWK with Python in GCC?

Posted Aug 2, 2018 22:38 UTC (Thu) by Wol (subscriber, #4433) [Link]

Actually, it isn't different at all.

If you let the engineers do things properly, it's a simple job. If you let people with no clue (like the beancounters and marketeers) interfere, then things get messy.

Sadly, it's usually the beancounters and marketeers that hold the purse-strings, so it gets done their way or - more usually - not at all (meanwhile promising customers "it's coming - it really is - it'll be here real soon now").

Cheers,
Wol

Replacing AWK with Python in GCC?

Posted Jul 25, 2018 19:58 UTC (Wed) by madscientist (subscriber, #16861) [Link] (9 responses)

There's a reason something is the status quo: it works. If you're proposing to replace something that works, it needs to be replaced in a way that doesn't cause other people pain.

Saying "guess what, your previously working environment is now working Even Better(tm), but oh by the way now you need to install a whole bunch of extra stuff onto your system before you can do the same things you were doing yesterday" is not, to everyone, actually an improvement.

I build GCC regularly. Many of the systems I do this on do not have Python 3 installed. I am NOT excited about having to go find/download/build Python 3 on all my build systems.

The GCC build system already relies (at least tangentially) on Perl. So why introduce Yet Another Interpreter Dependency? If you don't like awk (and who does?) then Perl is more than powerful and expressive enough for this job: rewrite the awk scripts in Perl and avoid adding new prerequisites for the build system.

Change has consequence and just because someone did some work doesn't mean the results have no cost and should be accepted without thought. It's not bikeshedding if the change has a major repercussion like increasing the dependency footprint of the software.

Replacing AWK with Python in GCC?

Posted Jul 25, 2018 20:03 UTC (Wed) by rahvin (guest, #16953) [Link] (4 responses)

Are you volunteering to rewrite the AWK scripts in perl?

Replacing AWK with Python in GCC?

Posted Jul 25, 2018 20:07 UTC (Wed) by epa (subscriber, #39769) [Link] (1 responses)

There is an awk to Perl translator. It used to ship as part of the Perl core and is still available.

Replacing AWK with Python in GCC?

Posted Jul 25, 2018 20:49 UTC (Wed) by smurf (subscriber, #17840) [Link]

The a2p translator is likely to create an even greater mess of unreadable code from the current awk scripts.
They are also complex enough that the translated version is unlikely to work at all, given that a2p is essentially unmaintained AFAIK.

WRT old machines without Python >= 2.7: there's always cross compilation.

Replacing AWK with Python in GCC?

Posted Jul 25, 2018 20:37 UTC (Wed) by madscientist (subscriber, #16861) [Link] (1 responses)

What I'm saying is that the complaint in the post I replied to is inappropriate for (at least) this situation. It's perfectly reasonable, and not whining or bikeshedding, to consider and discuss carefully and fully a decision like adding a huge new build dependency.

However, to answer your question, if the choice ends up being either someone comes up with a Perl implementation in which case we'd prefer to use that, or if no one does that then we're going to start requiring Python 3 as a GCC build dependency, then sure, I'd be willing to put in that effort. It does not appear that these are the options on offer, however.

Replacing AWK with Python in GCC?

Posted Jul 25, 2018 21:22 UTC (Wed) by rahvin (guest, #16953) [Link]

Maybe you should make your offer to convert these scripts to perl officially on the list so they can be considered. Mind you, they'd probably also want a commitment from you to maintain them as well.

Commenting on it here is all well and good, but the discussion is moving towards python, at least partly, because no one has come forward and said they'd move the scripts to perl. If you don't make the offer it's not there for consideration.

Replacing AWK with Python in GCC?

Posted Jul 26, 2018 4:42 UTC (Thu) by marcH (subscriber, #57642) [Link]

> rewrite the awk scripts in Perl

Replacing certainly unreadable code by likely unreadable code doesn't sound like a major upgrade IMHO, not worth the disruption.

> and avoid adding new prerequisites for the build system.

Granted.

Replacing AWK with Python in GCC?

Posted Jul 26, 2018 20:14 UTC (Thu) by MrWim (subscriber, #47432) [Link] (2 responses)

Saying "guess what, your previously working environment is now working Even Better(tm), but oh by the way now you need to install a whole bunch of extra stuff onto your system before you can do the same things you were doing yesterday" is not, to everyone, actually an improvement.

One trick they could apply is to run these scripts during make dist and include the output in the source tarball. This would mean that there are no additional dependencies required for people who just want to build GCC, only for those who want to develop GCC. This would make bring-up on new targets just as easy as now.

Of course this assumes that the process is deterministic and that the generated C files are themselves portable - but this should not be too much of an inconvenience.

You can even go further and include the generated C files in the git repo. It sounds icky I know, but the downsides of the duplication can be mitigated as long as:

The generation is deterministic - it depends only on files in the same git repo and isn't dependent on the architecture of the machine it's running on or the environment in which it is run.
You have a CI system to reject git commits with an inconsistency between the .opt and .c file
Concurrent modifications to the .opt files are fairly rare - you don't want to be dealing with merge conflicts in generated code - even if they can be resolved just by rerunning the tool on the merge commit.

I've used this approach successfully in some of my projects where I want a build dependency, but I don't want to burden other developers (including casual, drive-by ones) with the requirement to install that dependency.

Replacing AWK with Python in GCC?

Posted Jul 26, 2018 20:54 UTC (Thu) by jwilk (subscriber, #63328) [Link] (1 responses)

They already have configure (generated from configure.ac) committed to the repo, so the eww factor shouldn't be a problem.

(BTW, it's Subversion, not git.)

Replacing AWK with Python in GCC?

Posted Jul 27, 2018 21:19 UTC (Fri) by paulj (subscriber, #341) [Link]

The general approach is to have those built "source" files in the 'dist' tarball (generated by 'make dist'), so that that is buildable without dependencies on the tools needed for those built "source" files.

Replacing AWK with Python in GCC?

Posted Jul 25, 2018 21:35 UTC (Wed) by david.a.wheeler (subscriber, #72896) [Link] (1 responses)

Just fixing something probably wouldn't have raised any issues.

However, this is proposing a major new dependency in a key component. That's a big decision, and worth discussing first.

Replacing AWK with Python in GCC?

Posted Jul 25, 2018 23:12 UTC (Wed) by jhoblitt (subscriber, #77733) [Link]

As gcc is so fundamental during bootstraping, I also agree that introducing a dep on python 3 is likely raise problems on many platforms.

I am completely ignorant as to the gcc build process but is it possible for `optionlist` to be pre-computed and perhaps even included in the release tarballs? Is the .opt format so complicated that a small c/c++ parser couldn't be included as part of the build?

Replacing AWK with Python in GCC?

Posted Jul 25, 2018 21:39 UTC (Wed) by xorbe (guest, #3165) [Link] (3 responses)

I don't think anyone is denying that awk is unfavorable or difficult. It's the complication of adding dependencies to the gcc build flow, it's a valid concern. If only there were light weight scripting languages that didn't inevitably implement every feature request. Sort of like a usable awk.

Replacing AWK with Python in GCC?

Posted Jul 25, 2018 23:13 UTC (Wed) by jhoblitt (subscriber, #77733) [Link] (2 responses)

Isn't that essentially what lua is?

Lua

Posted Jul 26, 2018 4:26 UTC (Thu) by marcH (subscriber, #57642) [Link] (1 responses)

Yes - the drawbacks being as expected: much smaller community/number of people understanding it and generally smaller "ecosystem" of tools like debuggers, editors,... You can't have your cake and eat it.

Plus "light-weight" is intuitively not one of the top requirements when gcc is concerned... is it?

Now compared to awk (as opposed to Python) I suspect lua would win on practically every possible metric :-)

Lua

Posted Jul 26, 2018 6:52 UTC (Thu) by eru (subscriber, #2753) [Link]

much smaller community/number of people understanding it and generally smaller "ecosystem" of tools like debuggers, editors,.

On the other hand, Lua is very readable and easy to learn. Rather like Python in that respect.

Plus "light-weight" is intuitively not one of the top requirements when gcc is concerned... is it?

Its implementation is so small that you could include it with GCC sources without growing them with any significant percentage!

Replacing AWK with Python in GCC?

Posted Jul 26, 2018 4:38 UTC (Thu) by marcH (subscriber, #57642) [Link] (1 responses)

> and *none* of those people are actually volunteering to help, just volunteering an opinion.
> "Oh cool, you want to fix this? I support you fully, but only if you do it the way I would've done it."

No one can tell for how long the volunteer/hero-of-the-day will stick around after his initial feat for less appealing maintenance, not even himself. No one knows if/when he will have kids, a super demanding job in a brand new startup, sickness, etc. People capable of large and intense efforts tend to also be the ones getting bored the most easily. Maybe the people "volunteering an opinion" are the ones who have been taking out the garbage for decades, did you check?

> to cleanup a horribly messed up and unmaintainable piece of crap part of the code [...] status-quo of dumpster fire".

Did anyone say it was that bad? AFAIK gcc works.

> For god's sake it's not Java, why is there so much bikeshedding over it.

Good to know we can count on your bikeshedding whenever Java is considered for some random thing :-)

Kotlin

Posted Jul 26, 2018 5:02 UTC (Thu) by marcH (subscriber, #57642) [Link]

More seriously: if you ever had to suffer with Java I very highly recommend this Kotlin demo at Google I/O 2017 https://www.youtube.com/watch?v=X1RVYt2QKQE

It blew me away because it methodically removes most of Java pain points one by one - while of course keeping all the advantages.

I digress sorry; *not* suggesting Kotlin for this case.