using coreutils in production [LWN.net]

using coreutils in production

Posted Feb 12, 2025 17:14 UTC (Wed) by garyguo (subscriber, #173367)
Parent article: Rewriting essential Linux packages in Rust

We actually use uutils in production, and I think this is an interesting story to share.

We had some scripts that invoke `nproc` to determine the parallelism to use, and we want to move this into containers. Then we realized that in containers with resource restrictions, `nproc` from coreutils still report number of total cores, but not the amount of CPUs available to it by the cgroup. It turns out that the nproc haven't received meaningful updates since its inception and obviously doesn't know what to do with cgroups.

The fix? We simply replace coreutils nproc with uutils nproc in the container. uutils nproc simply calls into Rust std's available_parallelism, which is cgroup-aware.

to post comments

using coreutils in production

Posted Feb 12, 2025 17:27 UTC (Wed) by intelfx (subscriber, #130118) [Link] (7 responses)

> We had some scripts that invoke `nproc` to determine the parallelism to use, and we want to move this into containers. Then we realized that in containers with resource restrictions, `nproc` from coreutils still report number of total cores, but not the amount of CPUs available to it by the cgroup. It turns out that the nproc haven't received meaningful updates since its inception and obviously doesn't know what to do with groups.

You might have been using a very old coreutils. I've been relying on this behavior of coreutils' nproc for a few years already, and as far as I can tell, it very much exists:


# pacman -Qo nproc
/usr/bin/nproc is owned by coreutils 9.6-2

# nproc
8

# systemd-run --pipe -p AllowedCPUs=0,1 nproc
Running as unit: run-p1960742-i1961042.service
2

using coreutils in production

Posted Feb 12, 2025 18:40 UTC (Wed) by jengelh (subscriber, #33263) [Link]

Jan 13 2010 coreutils-8.4.tar.gz.
Using 15-year old systems as a base to justify today's decisions... *sigh* that's a new low.

using coreutils in production

Posted Feb 12, 2025 19:09 UTC (Wed) by garyguo (subscriber, #173367) [Link] (4 responses)

> You might have been using a very old coreutils.

No, coreutils 9.5 still has the same issue.

`AllowedCPUs` restricts what CPUs can be seen, so nproc reports what you expected. This is true even with very old nproc. The difference is `CPUQuota`, which is what gets used for k8s resource limits.

$ systemd-run --pipe -p CPUQuota=200% nproc
32

$ systemd-run --pipe -p CPUQuota=200% uutils-nproc
2

using coreutils in production

Posted Feb 12, 2025 19:13 UTC (Wed) by intelfx (subscriber, #130118) [Link] (3 responses)

That's working as intended, then? CPUQuota=200% means 2 seconds total CPU time per second *across all cores*, so it might be argued that the optimal way to utilize that quota is still to spawn a thread/process per core and let them share the quota.

using coreutils in production

Posted Feb 12, 2025 19:36 UTC (Wed) by NYKevin (subscriber, #129325) [Link] (1 responses)

> so it might be argued that the optimal way to utilize that quota is still to spawn a thread/process per core and let them share the quota.

I'm skeptical of that. It seems like you should get preempted more often and lose some performance to context switching (as compared to the approach of dividing CPUQuota by 100% to get the optimal number of threads, so that each thread has enough quota to run for the maximum allowable timeslice without preemption).

using coreutils in production

Posted Feb 12, 2025 19:37 UTC (Wed) by intelfx (subscriber, #130118) [Link]

On the other hand, this is certainly the most energy-efficient interpretation.

Correct behaviour under a CPUQuota= setting

Posted Feb 12, 2025 19:53 UTC (Wed) by farnz (subscriber, #17727) [Link]

It depends on whether you've got bursty work, or saturate your threads.

If your workload is bursty - a lot of work for a small fraction of the period used by CPUQuota, then idle for the rest of the period - you might want a thread per core so that when work comes in, you spread across all cores, get it done, and go back to idle. If your workload saturates all possible threads - so it'll be throttled by CPUQuota - you usually want just enough threads to allow you to saturate your quota, and no more (e.g. 3 threads for a quota of 250%); doing so means that you benefit from the cache effects of holding onto a single CPU core for longer, rather than being throttled most of the time and hitting a cooler cache when you can run.

And if you're I/O bound, chances are good that you can't make good use of a large number of threads anyway, because you're spending all your time stuck waiting for I/O on one thread or on 128 threads. You might as well, in this situation, just use 2 threads and save the complexity of synchronising many threads.

I thus believe it's rare to want thread per CPU core when you have a CPUQuota control group limit.

using coreutils in production

Posted Feb 13, 2025 0:08 UTC (Thu) by sionescu (subscriber, #59410) [Link]

Interestingly, if I run that on a Fedora it returns 2 when running as root, but the total number of cores when run with --user.