LWN: Comments on "GNU make 4.3 released"

GNU make 4.3 released

mathstuf — Sun, 26 Jan 2020 22:49:53 +0000

> But now a makefile may choose to enable -j by default, setting me up for an unpleasant surprise.)

That's clearly a bug. `-j` without a number is basically asking for a dead machine (it means that processes are never counted and whenever make get scheduled again, it's free to fork off even more work). It needs to be combined with the `-l` flag to really be useful.

That said, I do really prefer the default of ninja to default to a `nproc + 2` parallelism level. I hope something like *that* becomes the snippet "everyone" ends up copying.

GNU make 4.3 released

felix.s — Sun, 26 Jan 2020 15:19:19 +0000

Enabling parallelism from within the makefile seems like a misfeature. The person writing the makefile is not necessarily the one who knows on what machine the project will be built and therefore will not know the optimal value for the -j option. Right now, I am using a single-core machine, on which doing parallel builds only brings counterproductive overhead; from now on I'd be forced to pass -j1 every time I use make on it, including indirectly.

The -j should only be possible to enable on the command line, for the same reason GCC refuses to honour (the equivalent of) -march=native specified in source code.

(Also, even on multicore machines on which I tried it, when I start make with plain -j, the machine becomes unusable for interactive tasks while make is running, so I usually prefer to pass -j6 anyway. But now a makefile may choose to enable -j by default, setting me up for an unpleasant surprise.)

Breakage ahead

mirabilos — Thu, 23 Jan 2020 16:16:19 +0000

Caveat 3: the Korn shell closes open file descriptors >2 upon running external programs, so if you have any shell scripts in between…

Breakage ahead

StefanBr — Thu, 23 Jan 2020 15:59:56 +0000

> The jobserver protocol isn't hard to implement manually, even if not using make.

Unfortunately, the jobserver protocol is broken.

Job tokens are passed down via inherited file descriptors -

Caveat 1: If the called program is not $(MAKE) (verbatim), the file descriptors are closed on exec (FD_CLOEXEC), and the chain ends there. You can opt-in by adding a '+' in front of your command, but then you have to do that on every elegible command. Hint - none of the common Makefile generators does that.

Caveat 2: If you happen to call make indirectly (e.g. via a script called from the Makefile), unfortunately the '--jobserver-auth=<readfd>,<writefd>' will still be in the MAKEFLAGS env var although make had just closed the corresponding file descriptors. Two things can happen:

a) The fds no longer refer to valid file descriptors - make will detect this
b) The fds have been reused in the meantime - make will find valid file descriptors. If the fd is not readable (e.g. referring to a write end of a pipe), make will start busy looping. If the fd is readable, it will spawn a new job for every byte read (now figure what will happen if it points to a 2 kByte source file ...)

Grouped targets is this version's killer feature

david.a.wheeler — Wed, 22 Jan 2020 14:42:14 +0000

I agree. I think "grouped targets" is this version of GNU make's killer feature.

My "make-booster" GNU make package here: https://github.com/david-a-wheeler/make-booster implements a mechanism to do grouped targets that works on all versions of GNU make. But it's much nicer to have the facility built-in.

Breakage ahead

mirabilos — Wed, 22 Jan 2020 02:00:46 +0000

Maybe so, but the process of building Debian packages has a well-defined way for the invoker to specify a maximum number of CPUs to use, so this must be supportable.

Breakage ahead

madscientist — Tue, 21 Jan 2020 23:48:31 +0000

You should examine GNU make's -l (lowercase L) option. This works in conjunction with -j by checking the load on the system and ensuring that it doesn't get too high. So, make will start a maximum of -j parallel commands, unless the load is higher than specified with -l.

This is all described in the manual: https://www.gnu.org/software/make/manual/html_node/Parall...

Breakage ahead

mirabilos — Tue, 21 Jan 2020 19:24:13 +0000

No, the scenario is here to have one (physical or virtual, doesn’t matter) machine with multiple CPU cores, running, for example, two instances of a Debian buildd setup (e.g. one for unstable, one for experimental, or, one for 32-bit and one for 64-bit). The actual builds all happen in isolated chroots (not VMs, so the chroots would “see” all CPUs), but must be limited to nproc/2 (in this scenario).

Another example is building stuff on a workstation while using it (with modern GUI apps that use a lot of CPU) at the same time.

Breakage ahead

josh — Tue, 21 Jan 2020 18:31:01 +0000

> Running multiple builds in parallel will need the individual builds to not take up too many CPUs…

If you're running multiple builds in parallel, consider running a jobserver. That would allow limiting the combined CPUs used by all the builds, while allowing a build to use more CPUs while another build is using less.

The jobserver protocol isn't hard to implement manually, even if not using make.

Breakage ahead

josh — Tue, 21 Jan 2020 18:24:25 +0000

Would it be possible to have a built-in option to do -j with the number of processors, to allow documentation and Makefiles to eventually use that instead of assuming the availability of programs like nproc (which doesn't necessarily exist on non-Linux)? Perhaps a -J (capital J) option?

Breakage ahead

mirabilos — Tue, 21 Jan 2020 16:41:43 +0000

Indeed ☻ don’t I know that… (followup bugfix releases within the week)

Thanks for the response and information!

GNU make 4.3 released

pm215 — Tue, 21 Jan 2020 10:18:11 +0000

Wow, the 'rules with grouped targets' feature is something I've felt was missing from Make for decades -- much appreciation for the work done to specify and implement something to fill that gap!

Breakage ahead

madscientist — Tue, 21 Jan 2020 06:06:13 +0000

We had months of pre-releases and no reports of anyone having a problem with \#. Of course, that only means that we'll get reports within the week. But some of these quoting/escaping problems are simply to problematic to live: make's syntax is already tricky enough as it is, and trying to allow both old and new behaviors is a huge mess.

Yes, -j set on the command line overrides values set in the makefile. -j1 disables parallelism. -j0 is currently an error and not accepted: "make: the '-j' option requires a positive integer argument"

Breakage ahead

mirabilos — Tue, 21 Jan 2020 00:05:35 +0000

The \# change is likely to introduce breakage… thankfully it only affects an already GNU make-specific feature, but I can imagine people having used it. (Just had a short grep through all debian/rules files I had at hand and could not find it, only one use of \# outside of gmake functions, but that doesn’t exclude the probability.)

The parallelism thing will also be fun… how to enforce an upper limit, or no parallelism… does -j0 (is that even a thing?) or -j1 (perhaps that instead… how *do* you disable parallelism?) take precedence, at least? Running multiple builds in parallel will need the individual builds to not take up too many CPUs…