Rustaceans at the border [LWN.net]

Rustaceans at the border

Posted Apr 14, 2022 19:24 UTC (Thu) by atnot (guest, #124910)
Parent article: Rustaceans at the border

I think Kent Overstreet hinted at another good point in the message:

> We already have large swaths of code being imported from userspace into the kernel as-is - e.g. zstd [...]

Presumably, this is in reference to this event last November:

> Hi Linus,
>
> I am sending you a pull request to add myself as the maintainer of zstd and
> update the zstd version in the kernel, which is now 4 years out of date,
> to the latest zstd release. [...]
(https://lore.kernel.org/all/20211109013058.22224-1-nickrt...)

It sounds like it would already be pretty desirable in general to create a more formal, more automated version of the process of making vetted external dependencies available to the kernel, not just for Rust, but also for C.

to post comments

Rustaceans at the border

Posted Apr 16, 2022 1:39 UTC (Sat) by flussence (guest, #85566) [Link] (12 responses)

The kernel's got a quarter of a gigabyte of binary blob sitting in the amdgpu part of the tree already, I think they've surrendered their authority to be picky at this point.

Rustaceans at the border

Posted Apr 16, 2022 1:52 UTC (Sat) by davmac (guest, #114522) [Link] (11 responses)

Where is that in the tree exactly? (In 5.15.32 The whole of the amdgpu directory has 45M, the containing amd directory totals 328M but most of that seems to be include files)

Rustaceans at the border

Posted Apr 17, 2022 10:05 UTC (Sun) by khim (subscriber, #9252) [Link] (10 responses)

Said include files are precisely these “binary blobs” people are talking about.

They include hundreds of thousands of flags (maybe millions by now?) which do god-knows-what and are not documented anywhere. The code which toggles these is, essentially, a decompiled binary blob.

This is still kind-of better than what nVidia is doing (you can execute that mess on the RISC-V without binary translator, e.g.) but only marginally so.

You couldn't even prove that said code doesn't affect the kernel! The comments which are there strongly hints that code does what it is supposed to do, but you have absolutely no way to verify if said comments are true or not!

What's worse: it's not clear if or how one can improve that situation: AMD releases new architectures too fast for anything else. If there would even be someone who may decide to actually verify what all that code is doing and AMD would be supportive of that effort… I'm not even sure they have the required information internally! They are literally doing the best effort they could, this does not mean we understand what the heck is happening there!

Rustaceans at the border

Posted Apr 18, 2022 5:55 UTC (Mon) by nhaehnle (subscriber, #114772) [Link] (9 responses)

You haven't actually pointed at anything?

Last I checked, our binary blobs were in the firmware repository, not the kernel. IIRC the kernel contains a pre-assembled copy of the shader trap handler, but the trap handler source is right next to it and LLVM's assembler can be used to rebuild a binary.

The header situation is unfortunate, but it is the closest thing to the hardware team's source of truth that could be open sourced. It's also decidedly not binary blobs -- there is no code there, just register definitions.

Rustaceans at the border

Posted Apr 19, 2022 5:58 UTC (Tue) by flussence (guest, #85566) [Link] (8 responses)

Nvidia produces a PNG with bundled libpng shim and gets flamed mercilessly for decades over it.

AMD throws a massive obfuscated XPM over the wall, and gets praised by the tech splogs as a messiah of FOSS for giving gcc oh so much valid input to chew on.

Intel, for all their faults, just gives people the damn SVG.

Rustaceans at the border

Posted Apr 19, 2022 6:12 UTC (Tue) by mjg59 (subscriber, #23239) [Link] (7 responses)

What's obfuscated about the AMD driver? https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/... is a huge file, but it reads exactly like I'd expect a set of register definitions to read - there's just a *lot* of registers.

Rustaceans at the border

Posted Apr 19, 2022 20:53 UTC (Tue) by zdzichu (subscriber, #17118) [Link] (6 responses)

What's the point in including definitions of all the registers, while actual code uses only some of them? It's obfuscating which registers are important. On the other hand, the actual code is not obfuscated...

Rustaceans at the border

Posted Apr 20, 2022 3:48 UTC (Wed) by ssokolow (guest, #94568) [Link] (5 responses)

Are you really arguing in favour of less documentation?

The Andrew Schulman Programming Series ("Undocumented DOS", "DOS Internals", "The Undocumented PC", "Undocumented Windows", "Windows Internals", etc.) would like a word with you.

Rustaceans at the border

Posted Apr 25, 2022 19:41 UTC (Mon) by flussence (guest, #85566) [Link] (4 responses)

And on the other end there's the OOXML specification...

Rustaceans at the border

Posted Apr 29, 2022 14:09 UTC (Fri) by ssokolow (guest, #94568) [Link] (3 responses)

The problem with the OOXML specification isn't over-documentation. It's rampant NIH combined with "Good time to clear out code debt? What's that?"

For example, the XLS to XLSX migration would have been a great time to ditch Excel 1.0's date format which assumes 1900 was a leap year. Instead, they baked it into XLSX.

Combine that with how OOXML embeds a million other things that ODF farms out to other specs (eg. SVG for vectors) and it's easy to see why OOXML is such a mess. They pushed a million implementation details of a legacy codebase into the spec document and then politicked it into being a "standard" to make it difficult for others to implement. (As I remember, they fronted the membership fees to get a bunch of their allies added to the voting body just in time for the vote.)

Rustaceans at the border

Posted Apr 29, 2022 16:00 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (2 responses)

> The problem with the OOXML specification isn't over-documentation.

As if "do X like Excel 2003 did" was anything but under-documentation.

Rustaceans at the border

Posted Apr 29, 2022 19:11 UTC (Fri) by ssokolow (guest, #94568) [Link]

If they have some of that, then that's also a problem. I was referring to how it's made needlessly difficult by things like these snips quoted from the post I linked:

For legacy reasons, an implementation using the 1900 date base system shall treat 1900 as though it was a leap year. [Note: That is, serial value 59 corresponds to February 28, and serial value 61 corresponds to March 1, the next day, allowing the (nonexistent) date February 29 to have the serial value 60. end note] A consequence of this is that for dates between January 1 and February 28, WEEKDAY shall return a value for the day immediately prior to the correct day, so that the (nonexistent) date February 29 has a day-of-the-week that immediately follows that of February 28, and immediately precedes that of March 1.

date1904 (Date 1904)

Specifies a boolean value that indicates whether the date systems used in the workbook starts in 1904.
A value of on, 1, or true indicates the date system starts in 1904.
A value of off, 0, or false indicates the workbook uses the 1900 date system, where 1/1/1900 is the first day in the system.
The default value for this attribute is false.

(That particular example is their "TIFF has little-endian and big-endian versions" moment. An XLSX file contains a flag that indicates whether the dates are serializations of Excel 1.0 for Windows's in-memory format or Excel 1.0 for Mac's in-memory format, which have different epochs and different bugs... because apparently, in true Microsoft fashion, they were made by different teams who at least didn't communicate and probably saw each other as competitors.)

Rustaceans at the border

Posted Apr 29, 2022 20:20 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

People keep referring to that, but in reality it's just a couple of relatively unimportant formatting directives that you can safely ignore in practice. OOXML is _really_ overspecified.

Rustaceans at the border

Posted Apr 17, 2022 0:58 UTC (Sun) by developer122 (guest, #152928) [Link]

I'd also like to point out, nearly 100% of the code in the kernel already gets downloaded from an external repository: Linus' git.

If there were actually the proper infrastructure in place, useful rust crates could be pulled down as sub-gits or whatever along with firmware and everything else. Whether it's a good idea to auto-mirror new versions from crates.io, or how dependencies should be handled is an open question, but downloading a ton of code when you start to build a kernel isn't a new idea at all.