LWN: Comments on "Killing off /dev/kmem"

Killing off /dev/kmem

aCOSwt — Sat, 08 Jan 2022 16:11:33 +0000

"The occasional user-space device driver still needs /dev/mem to function, but it's otherwise unused."

AFAIU lilo-24.2 (latest) would also be happy to use it :

if ((fd=open(DEV_DIR "/mem", O_RDONLY)) < 0) return buf_valid=1;

(from the fetch function in probe.c)

this in order to determine misc. hardware (floppies / disks / video ) related information.

OK no harm if it cannot, lilo will just print a warning.

Killing off /dev/kmem

jjulian — Mon, 03 May 2021 17:31:26 +0000

* Disclaimer: I have nothing to do with any of this! :)

Killing off /dev/kmem

quanstro — Sat, 10 Apr 2021 17:45:32 +0000

i suppose if you consider mainstream *nix variants, linux may have been first. however, 8th edition unix introduced the concept of /proc. and iirc, linux was inspired by plan 9, not 8th edition. in plan 9 there is extra expressiveness. for example, /proc allows inspection of another machine's processes without endian/word size concerns via mount and bind. this is how stats(1) works. ioctl is similarly not included.

Linux proc influence

quanstro — Fri, 09 Apr 2021 17:49:08 +0000

i recall that as well. at the time, nobody had access to plan 9. so linux /proc was inspired by _papers_ about plan9's /proc. one of the things linux missed---perhaps because the plan 9 implementation was not visible---was a dirt cheep way to have lots of little single-job file systems. so linux /proc acquired a few warts. there is more to implement in a linux file system than 9p. but perhaps the 9p subset is good enough for most cases.

Well, we remember some different things...

Kamilion — Thu, 08 Apr 2021 21:17:31 +0000

Wow, big thanks to Albert Cahalan and Michael K. Johnson for showing up and taking the time to explain to us johnny-come-latelys.

Took me a moment to look at the poster's names and realize they were the very people involved.

how I remember the history

k8to — Thu, 08 Apr 2021 03:58:37 +0000

The warning on ps -aux led to

export I_WANT_A_BROKEN_PS=shutup

Being part of my .profile on Linux for many years. At some point around 2012 I realized I didn't need it anymore.

Yes, procps from /proc ps

k8to — Thu, 08 Apr 2021 03:52:23 +0000

Given variations in proc, ps is still the portable solution, sadly.

Linux proc influence

michaelkjohnson — Thu, 08 Apr 2021 00:58:12 +0000

My confident recollection is that the initial implementation of the Linux proc filesystem was explicitly inspired by plan 9's proc filesystem.

Yes, procps from /proc ps

quanstro — Wed, 07 Apr 2021 17:05:44 +0000

procfs goes back to at least 8ed in 1984, and was included (and expanded) in plan 9 from the beginning.

Killing off /dev/kmem

Paf — Wed, 07 Apr 2021 14:52:06 +0000

Oh god, that is .... wow. 😂

Killing off /dev/kmem

songmaster — Wed, 07 Apr 2021 02:50:47 +0000

This isn’t an example of reading the loadavg values, but for some code that goes delving into the internals of the OS to get data out of it I recommend looking at the legacy branch of the lsof program at https://github.com/lsof-org/lsof/tree/legacy. It supported many versions of Unix, and had to find and extract many different pieces of data to generate its output. The 00PORTING file at https://github.com/lsof-org/lsof/blob/legacy/00PORTING mentions briefly how it actually did that, and even takes a dig at “some down-sides to the Linux /proc-based lsof.”

Well, we remember some different things...

michaelkjohnson — Wed, 07 Apr 2021 02:10:05 +0000

I found some old tarballs to refresh my memory. ☺

Branko Lankester built kmem ps that came earlier.

In the earliest procps version I found (0.7), I already tried to honor at least SysV arguments e and f for people whose fingers had been trained on SysV, but provided BSD-style output regardless. Your rewrite implemented multiple personalities, which was naturally much better.

It looks like I introduced sorting output in version 0.93 in April 1994, later than I recalled, but before I was aware of you doing work on procps. That version definitely sorts by default, and the "o" option toggles sorting. I also clearly failed to update the man page along with that new feature.

Your memory of the transition is different from mine. I was doing a poor job of being maintainer (slow to apply patches and do new releases) but I certainly didn't "revert" color support, though I suspect it was there in a patch or fork that I hadn't adopted. A fork was the obvious response to an unresponsive maintainer, so no complaints there! I did finally step back formally.

Killing off /dev/kmem

Wol — Tue, 06 Apr 2021 16:53:10 +0000

Not Unix, but I worked on minis, and I set up a bunch of work queues with very strict limits, so if you wanted to fire off a load of jobs you could hammer the system without impacting everyone, My favourite was the "quick" queue, which ran at highest priority, but had a wall-clock-limit of 30 seconds. If it over-ran that it just got killed.

Cheers,
Wol

Killing off /dev/kmem

corbet — Tue, 06 Apr 2021 16:24:40 +0000

Look at sendmail, for example; it will stop processing mail if the system gets too busy.

Killing off /dev/kmem

shakkhar — Tue, 06 Apr 2021 16:17:37 +0000

> In the distant past, when computers were scarce and it was common to run many tasks on the same machine, jobs that were not time-critical would often consult the load average and defer their work if it was too high.

Can anyone share algorithm / code / doc which exemplifies this practice?

Killing off /dev/kmem

nix — Tue, 06 Apr 2021 14:03:57 +0000

Hah, that's nothing! The original home of nightmares is xterm, and while there is no easily web-accessible canonical xterm source (only tarballs), there are random github mirrors I can point at. Look at main() here: https://github.com/joejulian/xterm/blob/master/main.c. Look at the wonderful tangle in Tinput() here: https://github.com/joejulian/xterm/blob/master/Tekproc.c. Then fear, for this is code people are still using. (Though it could be worse still. It could be procmail.)

Killing off /dev/kmem

foxcrisp — Tue, 06 Apr 2021 13:53:41 +0000

Earlier unixes required apps to read from /dev/kmem, and know the format of the kernel data structures and location. Linux changed all of that, by exposing most things via /proc - mostly simple text strings. In a security based world, /dev/mem and /dev/kmem are just holes to allow access to any part of memory. Whilst the early implementations used unix group permissions, that just meant delegating the security mechanisms to the group mechanisms. That simply opens up the surface area (either get your self root, or get access to the relevant group for reading /dev/kmem).

In a sandboxed container, one doesnt need /dev/kmem, so, one could argue that if its not needed by some apps, it is not needed by any apps.

It is a shame if we lose it, but few apps truly needed it (I used it for dtrace, a while back - but would have to research the proposed alternatives).

how I remember the history

acahalan — Tue, 06 Apr 2021 13:09:35 +0000

Somebody wrote the original ps.

Based on that, Michael K Johnson wrote the procps.

Somebody else ended up maintaining procps for a while, adding color to the output, but then not much happened.

Michael K Johnson, then at Red Hat, decided (was told?) to maintain procps. He reverted to the pre-color version of the code. He put out a call for help, and Albert Cahalan responded with the suggestion that ps support both BSD and SysV syntax like OSF/1 (later renamed Tru64 then Digital UNIX) and AIX did. This would have been 1996 probably, or perhaps 1997. Sorted output was possible, but I don't believe it was the default. It should not be the default, mainly because ps is often used when a system is low on memory but also because partial output is desirable when running on a failing kernel. Sorting with the "O" option appears to have a BSD origin.

Albert Cahalan rewrote procps, initially just to prove that it would be possible to go beyond what OSF/1 and AIX could do, parsing mixed BSD and SysV options. (OSF/1 could only do one or the other, not mixed) There was then some human conflict relating to "ps -aux" printing a warning. Craig Small over at Debian started using Albert Cahalan's new code. This code definitely did not sort by default.

Michael K Johnson turned over a CVS repository to Rick van Riel and Ingo Molnar, excluding Albert Cahalan without explanation. This was almost certainly in 1997. Albert Cahalan then put a version 3.x.x on sourceforge, where he maintained procps for about a decade. At some point the 2.x.x version was made unreliable, grouping processes as threads if they happened to share various attributes as procps non-atomically looked at them. Albert Cahalan instead enhanced the /proc filesystem by adding the /proc/*/task/ directories and the thread counts. All distributions, including Red Hat, switched over to Albert Cahalan's procps 3.x.x code.

After about a decade maintaining procps, Albert Cahalan became too busy due to a large family. Also, he was demotivated because he found that it was impossible to stop Red Hat from hacking things up in ways that would add bugs and ill-considered compatibility troubles. This led to Craig Small, the Debian package maintainer, joining up with some other people to start the 4.x.x version series elsewhere.

Killing off /dev/kmem

tux3 — Tue, 06 Apr 2021 12:39:43 +0000

eBPF can feel pretty limiting sometimes. Thankfully there are Systemtap Guru scripts for all my "poke at kernel structures without manually writing a module" needs :)

group kmem ~= root

michaelkjohnson — Tue, 06 Apr 2021 12:08:23 +0000

Given the information available in /dev/kmem, setgid kmem is insignificantly different from setuid root. It feels better, but in the end it needs to be treated the same from a security perspective.

Yes, procps from /proc ps

lyda — Tue, 06 Apr 2021 11:40:36 +0000

Can't remember exactly why, but I had to look for a process in C on SCO once and discovered that the two ways to do it were to pull it out of /dev/kmem or to parse ps output. Massive gaping hole in libc in my mind. The /proc fs is a good unix-y solution.

I can imagine switching to it saved a massive amount of hassle.

Killing off /dev/kmem

markh — Tue, 06 Apr 2021 03:56:47 +0000

/dev/kmem was readable by group kmem, so programs requiring access to it could be made setgid kmem. (That is still the case for /dev/mem and /dev/port.) It's still a security concern, but better than requiring setuid root.

Killing off /dev/kmem

Paf — Mon, 05 Apr 2021 21:36:48 +0000

“ but the truly masochistic can wade through what must be one of the deeper circles of #ifdef hell”
My God. You were *not* kidding about the ifdefs.

Killing off /dev/kmem

luto — Mon, 05 Apr 2021 21:05:33 +0000

Fortunately, eBPF can replace these legacy /dev/kmem uses.

/me runs

Yes, procps from /proc ps

michaelkjohnson — Mon, 05 Apr 2021 20:30:05 +0000

Yes, that was the source of the name.

The original ps for Linux, often requiring rebuild after a new kernel build, if any of the structures had changed, used /dev/kmem. And after procps was released, the original ps was sometimes referred to as "kmem ps" to differentiate.

The original proc filesystem did not have enough functionality for a full replacement version of ps. I modified it to have all the necessary data for ps and uptime, and then wrote procps as a suite of programs that used the new functionality.

The output was formatted compactly (keep in mind this was when a 386sx16 was a decent machine) and I separated the stat and statm files because of the expense of producing the statm data, then in ps I kept track of whether statm needed to be read in order to produce the output.

As far as I know, my original procps was the original implementation of ps that defaulted to sorted output. All versions of ps that directly read /dev/kmem, as far as I know, listed the data in the order it happened to find it in the kernel memory it was digging through, and I was tired of juggling sort arguments when invoking ps.

I believe my original procps was also among the first, if not the first, to just recognize both BSD or SysV command line arguments and do what you meant, rather than requiring you to remember which syntax you needed to use on this particular system.

In any case, I don't know whether I was actually the first to introduce either or both of sorting internally and honoring both BSD and SysV arguments, or if one or both were previously invented and I unknowingly reimplemented ideas that already existed.

Killing off /dev/kmem

josh — Mon, 05 Apr 2021 18:43:18 +0000

You can do that with /proc/kcore now.

Killing off /dev/kmem

ribalda — Mon, 05 Apr 2021 18:34:27 +0000

Am I the only one that was using:

cat /dev/kmem > core
gdb vmlinux core

To debug the current state of the kernel/drivers?

Of course, never enabled in production. But for bringup is extremely helpful.

Killing off /dev/kmem

sjfriedl — Mon, 05 Apr 2021 18:15:43 +0000

> How did unprivileged code determine the load average?

I think the `ps` command was setuid root; what could go wrong? :-)

Killing off /dev/kmem

nickodell — Mon, 05 Apr 2021 17:56:39 +0000

> But how does one determine the current load average? Unix kernels have maintained those statistics for decades, but they originally kept that information to themselves. User-space code that wanted to know this number would have to do the following:

> * Read the symbol table from the executable image of the current kernel to determine the location of the avenrun array.
> * Open /dev/kmem and seek to that location.
> * Read the avenrun array into a user-space buffer.

How did unprivileged code determine the load average? Were unprivileged users allowed to read /dev/kmem in the past?

Killing off /dev/kmem

josh — Mon, 05 Apr 2021 17:31:40 +0000

Did you call it "procps" because it's a ps that uses /proc rather than other methods?

Killing off /dev/kmem

michaelkjohnson — Mon, 05 Apr 2021 16:04:19 +0000

Well, that took a while.

Not needing /dev/kmem was one of the points when I first created procps lo these many years ago.