Breakage ahead

Posted Jan 21, 2020 0:05 UTC (Tue) by mirabilos (subscriber, #84359)
Parent article: GNU make 4.3 released

The \# change is likely to introduce breakage… thankfully it only affects an already GNU make-specific feature, but I can imagine people having used it. (Just had a short grep through all debian/rules files I had at hand and could not find it, only one use of \# outside of gmake functions, but that doesn’t exclude the probability.)

The parallelism thing will also be fun… how to enforce an upper limit, or no parallelism… does -j0 (is that even a thing?) or -j1 (perhaps that instead… how *do* you disable parallelism?) take precedence, at least? Running multiple builds in parallel will need the individual builds to not take up too many CPUs…

Breakage ahead

Posted Jan 21, 2020 6:06 UTC (Tue) by madscientist (subscriber, #16861) [Link] (2 responses)

We had months of pre-releases and no reports of anyone having a problem with \#. Of course, that only means that we'll get reports within the week. But some of these quoting/escaping problems are simply to problematic to live: make's syntax is already tricky enough as it is, and trying to allow both old and new behaviors is a huge mess.

Yes, -j set on the command line overrides values set in the makefile. -j1 disables parallelism. -j0 is currently an error and not accepted: "make: the '-j' option requires a positive integer argument"

Breakage ahead

Posted Jan 21, 2020 16:41 UTC (Tue) by mirabilos (subscriber, #84359) [Link]

Indeed ☻ don’t I know that… (followup bugfix releases within the week)

Thanks for the response and information!

Breakage ahead

Posted Jan 21, 2020 18:24 UTC (Tue) by josh (subscriber, #17465) [Link]

Would it be possible to have a built-in option to do -j with the number of processors, to allow documentation and Makefiles to eventually use that instead of assuming the availability of programs like nproc (which doesn't necessarily exist on non-Linux)? Perhaps a -J (capital J) option?

Breakage ahead

Posted Jan 21, 2020 18:31 UTC (Tue) by josh (subscriber, #17465) [Link] (5 responses)

> Running multiple builds in parallel will need the individual builds to not take up too many CPUs…

If you're running multiple builds in parallel, consider running a jobserver. That would allow limiting the combined CPUs used by all the builds, while allowing a build to use more CPUs while another build is using less.

The jobserver protocol isn't hard to implement manually, even if not using make.

Breakage ahead

Posted Jan 21, 2020 19:24 UTC (Tue) by mirabilos (subscriber, #84359) [Link] (2 responses)

No, the scenario is here to have one (physical or virtual, doesn’t matter) machine with multiple CPU cores, running, for example, two instances of a Debian buildd setup (e.g. one for unstable, one for experimental, or, one for 32-bit and one for 64-bit). The actual builds all happen in isolated chroots (not VMs, so the chroots would “see” all CPUs), but must be limited to nproc/2 (in this scenario).

Another example is building stuff on a workstation while using it (with modern GUI apps that use a lot of CPU) at the same time.

Breakage ahead

Posted Jan 21, 2020 23:48 UTC (Tue) by madscientist (subscriber, #16861) [Link] (1 responses)

You should examine GNU make's -l (lowercase L) option. This works in conjunction with -j by checking the load on the system and ensuring that it doesn't get too high. So, make will start a maximum of -j parallel commands, unless the load is higher than specified with -l.

This is all described in the manual: https://www.gnu.org/software/make/manual/html_node/Parall...

Breakage ahead

Posted Jan 22, 2020 2:00 UTC (Wed) by mirabilos (subscriber, #84359) [Link]

Maybe so, but the process of building Debian packages has a well-defined way for the invoker to specify a maximum number of CPUs to use, so this must be supportable.

Breakage ahead

Posted Jan 23, 2020 15:59 UTC (Thu) by StefanBr (guest, #110916) [Link] (1 responses)

> The jobserver protocol isn't hard to implement manually, even if not using make.

Unfortunately, the jobserver protocol is broken.

Job tokens are passed down via inherited file descriptors -

Caveat 1: If the called program is not $(MAKE) (verbatim), the file descriptors are closed on exec (FD_CLOEXEC), and the chain ends there. You can opt-in by adding a '+' in front of your command, but then you have to do that on every elegible command. Hint - none of the common Makefile generators does that.

Caveat 2: If you happen to call make indirectly (e.g. via a script called from the Makefile), unfortunately the '--jobserver-auth=<readfd>,<writefd>' will still be in the MAKEFLAGS env var although make had just closed the corresponding file descriptors. Two things can happen:

a) The fds no longer refer to valid file descriptors - make will detect this
b) The fds have been reused in the meantime - make will find valid file descriptors. If the fd is not readable (e.g. referring to a write end of a pipe), make will start busy looping. If the fd is readable, it will spawn a new job for every byte read (now figure what will happen if it points to a 2 kByte source file ...)

Breakage ahead

Posted Jan 23, 2020 16:16 UTC (Thu) by mirabilos (subscriber, #84359) [Link]

Caveat 3: the Korn shell closes open file descriptors >2 upon running external programs, so if you have any shell scripts in between…