Race-free process creation in the GNU C Library

Posted Sep 1, 2023 21:34 UTC (Fri) by bbockelm (subscriber, #71069)
In reply to: Race-free process creation in the GNU C Library by bluca
Parent article: Race-free process creation in the GNU C Library

> The 1980s are calling and want their problems back ;-) In 2023 and on modern Linux, file descriptors are only limited by available memory. Open as many as you want.

Oh, the youthful banter of someone who hasn't spent a few hours this week debugging issues caused by file descriptor exhaustion!

(In this case, it was due to a hypervisor that booted a VM with trivial amounts of memory, the VM kernel adjusted system-wide file descriptor limits down accordingly, then the hypervisor would hotplug another 32GB of RAM later...)

For what it's worth, I agree this _should_ have been a problem relegated to history. I want to live in the future!

Race-free process creation in the GNU C Library

Posted Sep 6, 2023 8:39 UTC (Wed) by lathiat (subscriber, #18567) [Link]

I debugged the exact same issue, with proxmox having "Memory Balooning" enabled. Despite having a "minimum memory" of 16GB, it would boot with 1GB and plug the rest in later. This gave you very low maximum number of processes on the system giving fork: retry: Resource temporarily unavailable errors inside a Docker container.

I found the following very low Default:

# systemctl show --property=DefaultTasksMax
DefaultTasksMax=981

Which you also see in cgroupfs:
find /sys/fs/cgroup -name pids.max -exec grep -H . {} ;

The systemd docs state this is set based on threads-max "Configure the default value for the per-unit TasksMax= setting. See systemd.resource-control(5) for details. This setting applies to all unit types that support resource control settings, with the exception of slice units. Defaults to 15% of the minimum of kernel.pid_max=, kernel.threads-max= and root cgroup pids.max. Kernel has a default value for kernel.pid_max= and an algorithm of counting in case of more than 32 cores. For example with the default kernel.pid_max=, DefaultTasksMax= defaults to 4915, but might be greater in other systems or smaller in OS containers."

We then find a very low /proc/sys/kernel/threads-max of 6541. According to the kernel docs "During initialization the kernel sets this value such that even if the maximum number of threads is created, the thread structures occupy only a part (1/8th) of the available RAM pages."

Despite being a pretty experience Linux performance engineer it took me a bit to find that one, as it only showed up in the cgroup limits and not in /proc/PID/limit.

Good times :)