|
|
Log in / Subscribe / Register

McIntyre: Firmware - what are we going to do about it?

[Posted April 19, 2022 by corbet]

Steve McIntyre argues that Debian needs to rethink its approach to non-free firmware.

Today, a user with a new laptop from most vendors will struggle to use it at all with our firmware-free Debian installation media. Modern laptops normally don't come with wired ethernet now. There won't be any usable graphics on the laptop's screen. A visually-impaired user won't get any audio prompts. These experiences are not acceptable, by any measure.

to post comments

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 7:14 UTC (Tue) by developer122 (guest, #152928) [Link] (25 responses)

A point brought up that I don't think is often properly addressed: A lot of devices have had onboard firmware longer than drivers have been uploading firmware onto them. They just stopped shipping the firmware on-device, meaning it now needs to be uploaded.

There's a *strong* contingent in Free Software would prefer it one way vs the other: that uploading firmware onto a device is bad, but it's OK if the devices is running the firmware off of internal storage. Then it's "just hardware." Even better if it's a ROM because then it's especially "just hardware."

This is absurd. The fact that firmware is unchangeable is not a benefit to the user. It only means that bugs cannot be patched and Free replacements cannot be developed and distributed.

The bugaboos of A) users being somehow forced by a manufacturer to upgrade the firmware or B) that they might somehow download a backdoored firmware later have not come to pass. A) is simply not going to happen on any system running a Free OS to begin with. You need the user's cooperation, and even then many companies are actually not proactive *enough* about shipping firmware patches. As for B), why in the world would you trust the original firmware more?

But perhaps even more absurd is the notion that it's bad to upload firmware to the device rather than load from onboard flash. As if the firmware will somehow give the Free OS cooties for uploading the inert data. What this really is is a concerted effort to push the problem of firmware out of sight and out of mind.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 7:34 UTC (Tue) by jorgegv (subscriber, #60484) [Link] (18 responses)

I think the main problem today is the artificial distinction made between "software" and "firmware".

We unconsciously think of our current computers as a Big CPU connected to some little hardware "bricks", and then decide that the software inside those bricks is somehow inferior, give It a different name, and apply different policies to It.

But the reality is that our laptop is indeed a network of hundreds of computers, each of them running their own software. And free software principales should be applied to all of them, not just to "the Big one".

Since we have made the distinction and we apply the principles to one software class, but not quite to the other one, hardware manufacturers have shifted to shipping software of the latter class.

Being completely coherent is a tough Ride, as Debian is showing us...

I was shocked a few years ago when my Dell laptop offered me to upgrade the firmware of its USB controller... and It took about ten times longer than upgrading the BIOS!

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 7:50 UTC (Tue) by mjg59 (subscriber, #23239) [Link]

In the past, hardware vendors shipped firmware in ROM or in flash. Now, it's uploaded at runtime. I don't think that the category of software that hardware vendors have shipped has meaningfully changed, it's just that now it's more visible to us. Is it hypocritical to treat firmware as if it's not software? Sure. Is it hypocritical to pretend that firmware that's shipped with the hardware instead of the OS is somehow more free? I think so.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 9:02 UTC (Tue) by joib (subscriber, #8541) [Link] (13 responses)

Going the other, why are we treating HW different than software then? HW is just software (Verilog / VHDL / whatever) that has been compiled and "printed out", right? Particularly so for things like higher-level HDL's and FPGA's.

To answer my first rhetorical question, because it would be immensely impractical. In reality we would have no free HW on which to run our free SW. So we're already compromising on our "purity". Non-free firmware is just one fairly small step along the continuum. And arguably, one which alienates people from free software ending up in a net loss for software freedom.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 11:18 UTC (Tue) by excors (subscriber, #95769) [Link] (11 responses)

I don't think there's really a "continuum" here:

Developing a patch for regular application software, starting with C and ending up with a usable binary, takes maybe a few minutes and costs nothing.

Developing a patch for an ASIC, starting with Verilog and ending up with a physical chip, takes maybe six months and costs a million dollars.

Developing a patch for firmware, on devices that allow installing firmware updates over some host-accessible interface (which is probably all of the interesting ones - anything that doesn't require occasional bug fixes isn't doing any complicated work), takes maybe a few minutes and costs nothing.

There's many orders of magnitude difference in the practicality of updating software and hardware, so it makes a lot of sense to treat them differently in terms of demanding users be given the tools and permissions to modify them. Firmware used to be somewhere in the middle in practicality, when it was in ROM and was much harder to update than software but much easier than hardware. But nowadays it's in flash or RAM and it's exactly the same level of practicality as software, because it _is_ software, just with a funny name for historical reasons.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 12:00 UTC (Tue) by mfuzzey (subscriber, #57966) [Link] (6 responses)

I do think there is a continuum.

The "regular application in C" and "ASIC in Verilog" examples you mention are the two extremes.

For firmware the problem is that the source code is not available most of the time (non free firmware). So, in practice, "patching" the firmware requires understanding the existing firware well enough to do that (either as a binary partch or implementing a complete open source equivalent) without access to the original source code. This is possible (by reverse engineering) but is significantly harder than for regular application software and requires lots of time and experience (though if you have that it can mostly be done fairly cheaply).

Then there is the FPGA case which is similar to ASIC from the tools perspective but does not have the problem of huge cost. The problem today is that open source FPGA toolchains are not as advanced as the proprietary equivalents though things are improving.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 13:38 UTC (Tue) by farnz (subscriber, #17727) [Link]

Not forgetting the other axis on which there's a continuum; microcontrollers and FPGAs usually have configurable pin-outs, controlled by the firmware you load onto them. If you make a mistake with the configurable pin-out, you can cause the magic smoke to escape the chip, rendering it useless.

This is more of a risk with FPGAs than with microcontrollers, because FPGA bitstreams typically include the pin to I/O assignment as part of the configuration process, where microcontrollers have pin to I/O assignment done in code, and if you don't execute code to change the I/O assignment, you don't leave the "safe" reset state.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 14:48 UTC (Tue) by excors (subscriber, #95769) [Link] (3 responses)

I think there are two separate issues: software vs firmware, and having source code access/permissions vs not. Currently there's a strong correlation between those, but they need to be considered independently when discussing whether Free Software principles should apply to firmware.

(There's also a third issue of requiring code to be signed with a key that regular users can't use or replace, which seems very common for firmware but can apply to software too (e.g. dm-verity on Android) so that's probably a separate discussion.)

Reverse-engineering and patching a firmware binary that runs on some coprocessor isn't fundamentally any harder than reverse-engineering and patching a binary kernel module that does equivalent work on the main CPU. Or if you have source code for both, they're still just as easy as each other. The classification as 'firmware' isn't what makes the difference.

From a Free Software perspective, why is it so much more important to let users modify code running on the main CPU than the code running on other chips (which may well have more processing power and memory than the main CPU, and be running their own full OS)? I like the description that "our laptop is indeed a network of hundreds of computers" because that makes it clear that the CPU _isn't_ special or privileged, it's just one component of many. (Arguably the laptop's physical boundary isn't even that special - my CPU is interacting with my GPU firmware which it previously downloaded from the internet, and it's also interacting with the proprietary Python code on LWN's web server, and LWN sends back some proprietary JavaScript to run on my CPU, and why should I care more or less about the GPU than about LWN?)

So, I don't think it's a coherent philosophy to say that Free Software principles apply to software but not firmware, because that's a meaningless boundary. It would be coherent to apply it to all code (both software and firmware, and arguably all the internet services you use) but not hardware, because there's a meaningful technical boundary in the unavoidably prohibitive cost of modifying hardware. It would also be coherent to accept a mix of Free and non-Free code, while strongly preferring Free code, without being bothered about what chip the code is running on. The latter seems like the only practical option that doesn't ignore the reality of modern computer design.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 19:09 UTC (Tue) by mfuzzey (subscriber, #57966) [Link] (2 responses)

>From a Free Software perspective, why is it so much more important to let users modify code running on the main CPU than the code running on other chips?

Because code running on main CPU affects other code running on the main CPU far more than firmware does.

For example consider a GPU.

If it requires a closed source blob running on the main CPU (let's assume in user space to avoid getting into the legality or otherwise of closed source kernel modules) then that is going to restrict the users practical freedom in several ways:

* If they want to run Wayland and the blob only supports X11 out of luck.

* If they want to improve security of the system but using ASLR (that requires libraries be compiled PIC) out of luck if the blob isn't compiled that way.

* If they want to switch to a different processor architecture (Risc-V?) out of luck if there's no suitable blob version.

On the other hand none of those issues apply to any *firmware* that GPU may require, provided it works and is redistribuable the user can do all the above with a closed firware blob and an open userspace driver.

Of course if the firmware is buggy then the user can't fix that (easilly) in a blob.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 7:59 UTC (Wed) by marcH (subscriber, #57642) [Link] (1 responses)

You're basically saying that the communication and dependencies between processes running on the same CPU are tighter than processes running on different CPUs. This is generally true but it's not always true and it does not have to be true. You can have a very elaborate protocol in memory shared between the "software" and the "firmware"CPUs that creates a strong dependency between the two sides.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 20:40 UTC (Wed) by khim (subscriber, #9252) [Link]

That's not a theoretical concern, BTW. Just look on Asahi Linux.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 16:39 UTC (Tue) by rgmoore (✭ supporter ✭, #75) [Link]

For firmware the problem is that the source code is not available most of the time (non free firmware). So, in practice, "patching" the firmware requires understanding the existing firware well enough to do that (either as a binary partch or implementing a complete open source equivalent) without access to the original source code. This is possible (by reverse engineering) but is significantly harder than for regular application software and requires lots of time and experience (though if you have that it can mostly be done fairly cheaply).

I'm not sure how true this is. Reverse engineering an application isn't easy, either. Look at how much work has gone into, for example, Libre Office in an attempt to reverse engineer and reproduce other office suites. It's relatively easy to modify an application once that front-end effort of creating a work-alike copy has been done, but getting to that point is a ton of effort. The big difference- and IMO one of the differences between free software and free firmware- is that the applications aren't tied nearly as tightly to particular hardware.

It's fairly standard to run the same software, without a recompile, on generation after generation of the same CPU architecture, or to port with a recompile from one architecture to another. It's less common for firmware to last across generations of the same architecture, and basically unheard of to port from one architecture to another. Just as a practical matter, that means we're a lot more likely to have to throw firmware away and start over from scratch than application software.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 18:05 UTC (Tue) by ejr (subscriber, #51652) [Link]

Documenting software and its interfaces such that it can be modified typically requires a single manual.

Documenting firmware and all its interfaces such that it can be modified requires at least multiple manuals.

Documenting an ASIC and all its interfaces requires at least a bookshelf.

And IME the cost of developing and maintaining documentation is not linear in its size.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 20:44 UTC (Tue) by chris_se (subscriber, #99706) [Link]

> Developing a patch for regular application software, starting with C and ending up with a usable binary, takes maybe a few minutes and costs nothing.

Only if the application is simple enough. I've never even looked at the source code of e.g. Firefox or LibreOffice in any meaningful amount of detail, and to be completely frank: for a simple enough peripheral it'd probably be way less work for me to modify a schematic in KiCad and order it via one of these PCB prototyping services that also include assembly than to understand enough of the abstractions and concepts behind the source code of large enough software projects I haven't looked at before in order to be able to change something there.

I don't disagree that modifying hardware requires more effort than modifying software of a similar complexity, but there's tons of hardware out there that's way, way simpler in design than a lot of widely used software packages.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 7:48 UTC (Wed) by marcH (subscriber, #57642) [Link] (1 responses)

> Developing a patch for an ASIC, starting with Verilog and ending up with a physical chip, takes maybe six months and costs a million dollars.

It's not the cost of developing the patch, it's the fact that you cannot copy and deliver a "hardware upgrade" over the Internet because it is a physical object. That's the obvious and very clear line between hardware and software, hard to believe it needs to be explicited.

"Firmware" on the other hand has absolutely zero meaning, it's just software. Entire Android images including Linux kernel and userspace are routinely called "firmware images", go figure. I suspect firmware originally meant something like "software mostly hidden and invisible to humans" but that was very vague too.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 11:14 UTC (Wed) by excors (subscriber, #95769) [Link]

> I suspect firmware originally meant something like "software mostly hidden and invisible to humans" but that was very vague too.

Hmm, Wikipedia claims it was coined in 1967: https://archive.org/details/TNM_4th_generation_software_h...

> The guess is that fourth-generation hardware will use large-scale integration (LSI) logic specialized by microprogramming. Integrated logic chips will be needed to obtain the required fast circuitry and the microprogrammed specialization will be needed to obtain the required software and user interfaces.
>
> The elements of one of these computers might have these characteristics:
> circuitry (LSI) - 5 nanoseconds
> micromemory access - 10 nanoseconds (read time)
> main memory access - 400 nanoseconds
>
> In the most radical departure from present architecture, the computer would have no order set and no data structure. The computer would be specialized for the various roles it is to play by replaceable microprograms. Third-generation microprogrammed computers are delivered with a pre-designed, pre-installed microprogram of the read-only type. [...] In fourth-generation computers, many microprograms will be available from the manufacturer. Software and user specialists will also prepare and use their own. This should throw the whole field wide open. To better understand the nature of microprogramming a no-order-set/no-data-structure computer, I believe it worthwhile to introduce a new word into our vocabulary: firmware. I use this term to designate microprograms resident in the computer’s control memory, which specializes the logical design for a special purpose, e.g., the emulation of another computer. I project a tremendous expansion of firmware—obviously at the expense of hardware but also at the expense of software.

If I'm interpreting it right, the "microprogram" is what we'd now call microcode (except that probably *all* instructions used by software are microcoded, none of them are handled directly by the hardware). Typically the hardware would read an opcode from main memory, then call a microprogram function to handle it, with the microprogram being written in machine-specific assembly language. Because main memory was ~40x slower than micromemory, it was important that the FORTRAN/etc programs compiled into main memory were as small and dense as possible, using high-level instructions that expanded into many microprogram instructions. Or sometimes you'd write effectively a bytecode interpreter in the microprogram, which would fetch and decode instructions from main memory by itself, e.g. to emulate an incompatible ISA.

Previously the microprograms were in ROM and were seen as part of the hardware. The proposal was that fourth-generation computers would have a new type of "slow-write fast-read" micromemory instead of ROM, so users could install customised microprograms that provided new high-performance features to their applications. The microprogram would no longer be hardware, but was very different to the regular software in main memory (different programming language, different ISA, different performance characteristics, etc), so they invented the term "firmware" for that.

Of course it wouldn't do to let arbitrary users write their own microprograms, but they didn't have good cryptographic authentication algorithms, so they suggested a different form of access control:

> To visualize the preparation of firmware, consider a special keypunch not available to software and user programmers. This punches 67-column cards (4x8 inches) with triangular holes (try that on your keypunch and card reader!). A special card reader loads decks of these 67-column cards into the fourth-generation SW/FR memory. This card reader probably will be locked so that only firmware specialists have access to it (we hope)!

Verilog that's been "printed out"

Posted Apr 19, 2022 12:56 UTC (Tue) by mikebenden (guest, #74702) [Link]

See https://github.com/litex-hub/linux-on-litex-rocket for a working proof of concept -- it can even boot 64-bit linux, and run its own HDL toolchain (yosys, nextpnr -- sort-of, slowly) :)

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 15:27 UTC (Tue) by pizza (subscriber, #46) [Link] (2 responses)

> But the reality is that our laptop is indeed a network of hundreds of computers, each of them running their own software. And free software principales should be applied to all of them, not just to "the Big one".

The reality is that *every* component of the system, from the keyboard to the display, and everything inbetween (network/wifi, nvme/sata storage, cpu, embedded controller, usb controller, audio codec, card reader, etc etc) relies on non-free embedded software/firmware. Whether or not it's embedded in mask ROM, flash, or uploaded at runtime, it's still non-free.

In the beginning of the Free Software movement, it was acceptable to incrementally replace parts of a proprietary system with Free components as they became available, if ever. Things are no different today; it does not advance the interest of Free Software to make said software effectively impossible to use on readily-available equipment.

McIntyre: Firmware - what are we going to do about it?

Posted May 13, 2022 11:41 UTC (Fri) by immibis (subscriber, #105511) [Link] (1 responses)

A mask ROM is essentially hardware. Nobody can change it - you are stuck with it, including all bugs. Asking it to be free is like asking a logic gate to be free. You could also see it as a lookup table (microprograms in chips like the 6502 are very lookup-tabley) for an algorithm implemented in hardware.

On the other hand, replaceable firmware is something that the manufacturer can change but you can't, and that's pretty unfair. And when the firmware upload is mandatory, and especially if it's also signed, then they get to abuse copyright to make it effectively illegal to use their hardware at all without their permission.

McIntyre: Firmware - what are we going to do about it?

Posted May 14, 2022 2:47 UTC (Sat) by pabs (subscriber, #43278) [Link]

With firmware that is always uploaded on boot, usually you get a choice about which version you use, which is better than the case of firmware stored on the device, where often the upgrade mechanism is secret, or the upgrade mechanism prevents downgrades.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 15:19 UTC (Tue) by pizza (subscriber, #46) [Link]

> What this really is is a concerted effort to push the problem of firmware out of sight and out of mind.

It is analogous to liquor stores using brown paper bags for their wares; if what's in the bag can't be seen then it's somehow more acceptable.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 17:33 UTC (Tue) by dmoulding (subscriber, #95171) [Link] (3 responses)

> But perhaps even more absurd is the notion that it's bad to upload firmware to the device rather than load from onboard flash.

I couldn't agree more. It seems that many have lost sight of the situation that gave rise to the free software movement in the first place. Here's a relevant portion from the GNU Manifesto on the motivation for the whole undertaking:

> Why I Must Write GNU
> I consider that the Golden Rule requires that if I like a program I must share it with other people who like it. Software sellers want to divide the users and conquer them, making each user agree not to share with others. I refuse to break solidarity with other users in this way. I cannot in good conscience sign a nondisclosure agreement or a software license agreement.

Today's proprietary firmware doesn't match that old situation at all. First, nobody "likes" their firmware and feels an urge to share it with other users who might also like it. Consumers use firmware because it's necessary. Second, the manufacturers are not (generally) compelling users to sign license agreements or non-disclosure agreements with regard to the firmware. Probably most explicitly allow redistribution, right? (Otherwise the linux-firmware package couldn't exist in its current form).

Finally, the firmware isn't general purpose software designed to run on a general-purpose computing platform. In most cases, users simply could not arbitrarily change it to their liking because it generally must work with the hardware in very specific ways. Sure, it's possible in a lot of cases to tweak it here and there, but there are much stricter limits to what can be done with it due to the very nature of the hardware it runs on. And the vast, vast majority of users simply DO NOT CARE about how their firmware works, so long as it makes the hardware work the way it's supposed to. The motivation to "tweak" your firmware and "share" the derivative with other users who might like it simply does not exist like it does with general purpose software.

For these reasons, it fundamentally doesn't make sense to treat firmware the same as, say, a compiler, a shell, or a window system.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 18:17 UTC (Tue) by farnz (subscriber, #17727) [Link] (2 responses)

Note, though, that the defining incident was with printer firmware, not with general purpose software. RMS had access to a printer with imperfect firmware, and wanted it to send one more signal to the host so that it could notify all users of a problem; because the firmware was a binary blob, this was made hard for RMS to do, and he would not have been able to share the fixed version even if he reverse engineered the blob and created his own with the extra signal.

It does, however, illuminate a possible scale of "non-freeness". The worst case is locked down firmware where, as an end user, you cannot change the firmware even if you want to due to signatures and the like (Intel microcode updates). The least worst case is something where the firmware is written for a known architecture, is easy to disassemble, and there's no effort to stop you from changing the firmware freely.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 18:53 UTC (Tue) by dmoulding (subscriber, #95171) [Link] (1 responses)

I'm not sure that was actually firmware. It sounds more like it was a driver, a piece of general purpose software, that could run on the general purpose processor of the computer running the system's operating system. I believe RMS wanted to modify the driver's program so that it would send an alert via the operating system to users so they'd know the printer had jammed. That is the kind of change that people legitimately could want to make to a piece of software.

But the firmware in a modern printer (or even a printer back then) is not generally responsible for things like job control and error notification. It's more likely to be responsible for things like:
* what temperature should the fuser reach before feeding the paper
* how much should the laser's input voltage be modulated to increase/decrease the charge on the drum
* speed control of rollers appropriate for DPI output

These are generally highly technical parameters that most users should not change (nor would want to, or might risk damaging the device if done improperly). And, I'd argue, it does *less* harm to the user that they cannot change the firmware that controls all that, than it does harm the end user if after installing Debian their printer won't print at all.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 9:58 UTC (Wed) by farnz (subscriber, #17727) [Link]

The firmware in my printer, and in all the printers at my workplace, also has a full TCP/IP stack with IPv4 and IPv6 support, Ethernet, fax modem, LDAP, TLS, SMB, Apple Airprint, PDF rasterization, and many other bits beyond the low-level control you're describing. Sure, it does the low level control part as well, but from my point of view as a user of the device, that's just a component in a much larger firmware blob.

And the same applied to early Xerox laser printers, of the sort RMS is describing; the printer did not expose a low-level interface to the host computer, as cheap laser printers do today, and as dot matrix and inkjet printers have always done, but a high level interface where the host sends the printer a page description, and the printer reports when it's finished handling the page and is ready for the next page. All RMS wanted to do was to change that "finished this page" to "finished successfully", or "failed to print" so that the host software could send a message on "failed to print" to everyone.

Finally, your last paragraph applies to pretty much everything the kernel does - by swapping for a Linux kernel instead of the manufacturer-approved kernel (Windows, usually), you are risking damaging your device. For example, when I've been working on a (proprietary) system, I found by writing buggy code a SCSI command I could send to a drive that should have been rejected, but instead caused the drive to brick itself. Does this mean that users shouldn't be allowed to install a Linux kernel?

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 6:38 UTC (Wed) by eduperez (guest, #11232) [Link]

> There's a *strong* contingent in Free Software would prefer it one way vs the other: that uploading firmware onto a device is bad, but it's OK if the devices is running the firmware off of internal storage. Then it's "just hardware." Even better if it's a ROM because then it's especially "just hardware."

There is an explanation for that line of argument:

- I only use free software!

- What about firmware?

- I have created a distribution without blobs; I only use free software!

- What about firmware in ROM?

- Well, if the user cannot change it, then I will call it "hardware" instead of "software".

- What!?

- I only use free software!

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 7:20 UTC (Tue) by dankamongmen (subscriber, #35141) [Link]

his option #5 (a new non-free-firmware component) is exactly what i've hoped for and advocated for close to a decade. my only concern is that it possibly incentivizes moving code which perhaps logically belongs in a (open) driver into (closed) firmware, while retaining in-distro distribution. i don't think that's a serious enough concern to avoid taking this route, though.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 7:56 UTC (Tue) by pabs (subscriber, #43278) [Link]

HN discussion:

https://news.ycombinator.com/item?id=31079253

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 8:55 UTC (Tue) by civodul (guest, #58311) [Link] (43 responses)

Isn't it an exaggeration?

I have seen laptops without Ethernet (Apple hardware), but is it the majority? I don't remember seeing laptops that wouldn't get graphics or audio output without non-free firmware; I don't doubt that exists, but again, is it the case of all "modern laptops" as McIntyre implies?

For those who think firmware should be free, the question is that of finding the most efficient strategy towards that goal. To what extent does shipping non-free firmware help, compared to other strategies such as publishing lists of "known-good" laptops?

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 9:04 UTC (Tue) by LtWorf (subscriber, #124958) [Link] (1 responses)

My laptop 10 years ago had ethernet which required proprietary firmware to run.

Since then I've never bothered with the regular debian iso and just always go for the non-free one with the firmwares. The pure one is completely useless and just means I have to manually download that anyways.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 10:48 UTC (Tue) by amacater (subscriber, #790) [Link]

"Sort of" agreed: the fully free official .iso is ideal for virtual machines - where the hardware is already virtualised to "use" free drivers (a VM is as often as not a bunch of
YAML or similar over KVM).

It's tough when your laptop doesn't have free drivers - I hit this even on something that's
about ten years old (Intel Bay Trail laptop with 2G memory, 32 bit UEFI, Atom 64 bit capable processor) and no sound - Intel sound firmware required. Oh, and no Ethernet either.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 9:06 UTC (Tue) by kenmoffat (guest, #4807) [Link]

Most current laptops here in GB lack wired ethernet and need firmware for wifi and for bluetooth (even if you don't want bluetooth, wifi may need it). If you are using amdgpu you need firmware to get beyond an 80x25 tty. Not certain about sound.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 9:54 UTC (Tue) by bluca (subscriber, #118303) [Link] (3 responses)

Pretending firmware doesn't exist (it does) and is not needed (it is) is clearly not working as a strategy. If it did, it would have made a difference in the past 20 years or so, but as the article mentions there's way more user-visible non-free firmware today than there was back then. So the only thing we are achieving is making life difficult for our users, pushing them toward other distros, and in the worst case leaving users with gaping security holes (see: intel-microcode). It's high time we fixed this situation, and I'm very grateful Steve et al. put out this proposal.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 13:10 UTC (Tue) by ballombe (subscriber, #9523) [Link] (2 responses)

It is a working strategy: Debian still exists 20 years later, and hardware support is much better than 20 years ago.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 13:28 UTC (Tue) by seyman (subscriber, #1172) [Link]

> hardware support is much better than 20 years ago

I'm going to agree with McIntyre instead of you : the current setup is a horrible mess and its getting worse.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 14:38 UTC (Tue) by bluca (subscriber, #118303) [Link]

Those things have happened *despite* this strategy. Debian has been hemorrhaging users to more "friendly" (ie: they actually work) distros like Ubuntu/Mint. There is no more "open firmware" on my machine that there was 20 years ago. There is more proprietary firmware than there was 20 years ago. User experience is significantly worse, and security is significantly worse.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 10:09 UTC (Tue) by stevem (subscriber, #1512) [Link]

I'm not saying *all* modern laptops are unusable, but it's more and more common that major lumps of functionality won't work without adding non-free firmware. Personally, I'm mostly happy with my T470 Thinkpad. but if I want to use wifi I have to install the non-free firmware-iwlwifi package. Most of the time my laptop is connected via wired ethernet. However...

There's a security aspect to this - microcode updates are strongly recommended by both AMD and Intel.

Before the Debian 11 (Bullseye) release we spent a lot of time debugging and working on better support for newer AMD and Nvidia graphics cards that need firmware even to work with free drivers.

A lot of people will struggle to make their machines work without firmware. Rather than just scare those people away, I want to get a decision from all of Debian's developers on the best way forward.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 11:24 UTC (Tue) by tao (subscriber, #17563) [Link] (6 responses)

I think my last three laptops have lacked Ethernet, simply due to the fact that they're too thin to fit an Ethernet connector (ThinkPad Carbon series).

The WiFi definitely needs firmware (iwlwifi); to solve this I have a really old USB WiFi dongle and an equally old USB Ethernet dongle. I think both of them are USB2.

Luckily Intel's GFX until now can be used without the firmware (but Power Management, advanced video codecs, and etc. won't work).

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 11:31 UTC (Tue) by seyman (subscriber, #1172) [Link] (4 responses)

> I think my last three laptops have lacked Ethernet, simply due to the fact that they're too thin to fit an Ethernet connector (ThinkPad Carbon series).

I suspect the trend is becoming more common. Vendors want to release thinner and thinner laptops and are more and more willing to sacrifice ports to do so, reasoning that users who want those ports will just buy a USB-C dongle to get them.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 8:06 UTC (Wed) by marcH (subscriber, #57642) [Link] (3 responses)

After trying this for years, Apple finally listened to customers and backtracked.

https://www.theverge.com/22751921/apple-macbook-pro-14-16...

> The ports are wonderful, and I’m glad they’re back [..] escape from Dongletown.

Or read any other review of the same products.

There _is_ a market for super thin, Dongletown laptops but it's not for everyone.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 21, 2022 6:29 UTC (Thu) by danieldk (guest, #27876) [Link] (2 responses)

> After trying this for years, Apple finally listened to customers and backtracked.

Honestly, I would have cared 4-5 years ago. But now I just have a Thunderbolt Dock at the office and in my home office. I plug one cable and have power delivery, external 4k@60Hz display, Ethernet, and peripherals (like a webcam). The Thunderbolt over USB-C life is just to good to go back to always plugging N cables. Having more ports is sometimes handy on the go, but usually I only plug a power adapter anyway. It's only when I am presenting that it is mildly advantageous to have an HDMI or VGA port over a small dongle.

I am completely happy with the port situation of a laptop as long as there are 3 or 4 Thunderbolt 4 ports. I am more annoyed if a laptop doesn't have a Thunderbolt port (only USB over USB-C) or a mixture of USB-C ports that do not all have the same functionality.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 21, 2022 17:31 UTC (Thu) by marcH (subscriber, #57642) [Link] (1 responses)

Yes there is a market for it but it's not for everyone. For a start docks don't solve the problem of people traveling (the forgotten reason why laptops were invented). Then the Type-C port feels too small and fragile for professional use. Finally, the main reason to get rid of ports is to make absurdly thin laptops with... less battery, less ventilation and less TdP. All things that "creative" professionals (Apple's most loyal market) kept complaining about for years. Make laptops a bit thicker again and there's no strong reason to get rid of all ports again. Even better: now you have one-port-docking AND more ports, serving all markets and all use cases with a single product with only one exception: fashion victims who care about the 2 extra milliliters in their hand bag (and then there's still the MacBook Air. Maybe)

I'm not making any of this up, just watch any review of older generation MacBooks Pro. Windows users always had much more choice and in fact you can even find reports of some well known YouTubers "defecting"

McIntyre: Firmware - what are we going to do about it?

Posted Apr 21, 2022 17:40 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

One benefit of USB-C that I'll put forth: having a power port on either side of the laptop. No more "can't orient that way because the power cord is now being torqued" issues.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 14:09 UTC (Tue) by Thalience (subscriber, #4217) [Link]

> Luckily Intel's GFX until now can be used without the firmware (but Power Management, advanced video codecs, and etc. won't work).

Keeping in mind that even the basic GFX component only works because the motherboard firmware includes the GFX blob (alongside CPU microcode and a bunch of other blobs that are "out of sight, out of mind").

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 15:02 UTC (Tue) by ocrete (subscriber, #107180) [Link] (7 responses)

All of the thinner laptops no longer have ethernet ports. Just looking at the Linux supported ones, that includes the ThinkPad X1 Carbon as well as the Dell XPS 13 and 15. So yes, I think Ethernet ports are really confined to "desktop replacement" laptops now. And to be honest, I thought I'd miss it and I haven't yet had a case where I wanted wired ethernet elsewhere than the office (where I have a docking station).

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 16:01 UTC (Tue) by farnz (subscriber, #17727) [Link]

And even if you do have wired Ethernet, it's not guaranteed to work without firmware; I have systems with a Realtek 8168 Ethernet chipset that need firmware to work.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 20:05 UTC (Tue) by atai (subscriber, #10977) [Link] (2 responses)

Do these USB-to-ethernet adapters need special firmware? It seems not in my experience

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 20:33 UTC (Tue) by Kamilion (subscriber, #42576) [Link] (1 responses)

> Do these USB-to-ethernet adapters need special firmware? It seems not in my experience

OHhhhhhhhhhhh yes. And most of them are nightmarish little abominations of out-of-tree drivers, like ASIX.

There *are* a few gems out there, I recommend the Realtek rtl8153 gigabit adapter as one of the few iPXE will work with over USB. It supports USB CDC-ECM.
I've heard (but not verified) that Realtek rtl8156 also supports USB CDC-NCM.

Cable Matters or Plugable both offer units in both Type-A and Type-C with flash+eeprom onboard. Unfortunately, realtek does not offer any sort of tool to replace the flash contents. Some of them show up as Mass Storage Class and contain the windows .inf and signed driver, which I wanted to replace with iPXE. Nope.

It's worth noting that the official Linux 5.13 support from April 2021 that Realtek contributed for both the 8153 and 8156 was only a CDC-ECM driver.

Previous to this, to my knowledge there was no standardized USB Class driver for networking. RNDIS existed, but was microsoft-centric and I don't think it became a standard, but I could be mistaken. The Communication Device Classes came later with USB3 and have fallback support for USB2 (but are not officially part of the 2.0 spec...?)

USB Ethernet adapters

Posted Apr 20, 2022 9:12 UTC (Wed) by james (guest, #1325) [Link]

...most of them are nightmarish little abominations of out-of-tree drivers, like ASIX.
Things might be getting a little bit better there. I was looking for a cheap USB to Ethernet adapter earlier this year (for use in emergencies): most of them seemed to support recent Linux, and ASIX have been getting some drivers in-tree. Certainly the unbranded ASIX AX88179 I bought Just Worked in Fedora (no out-of-tree drivers needed).

Having said that, I only bought the dongle for use in emergencies, so I haven't done any more testing than quick browsing.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 21, 2022 20:35 UTC (Thu) by kpfleming (subscriber, #23250) [Link] (2 responses)

Thinkpad X1 Carbon, at least through Gen 6 (but probably still) does have an Ethernet port, it's just not an RJ-45 jack :-) It's a proprietary connector right next to the USB-C ports and requires a dongle from Lenovo to convert it into a usable form (or a Lenovo mechanical dock), but it's there and it works just fine in Linux. Same is true of the 'S' series of 'thin' business laptops like the T14s and some others, although my T480s had a full RJ-45 port which was very difficult to use.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 22, 2022 0:34 UTC (Fri) by Wol (subscriber, #4433) [Link] (1 responses)

My new laptop has a half-height RJ-45 :-) I think the connectors are normal, but the bottom half the catch on the plug clicks into, is a pop-out that enables the laptop to be slimmer when it's not in use ...

(I've not used it, so I can't speak for it. I probably will want it when I put gentoo on the laptop.)

Cheers,
Wol

McIntyre: Firmware - what are we going to do about it?

Posted Apr 22, 2022 10:57 UTC (Fri) by kpfleming (subscriber, #23250) [Link]

That's what the T480s has, and I hate it :-) Getting a plug into it is fine, getting the plug back out is quite difficult.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 17:59 UTC (Tue) by developer122 (guest, #152928) [Link] (19 responses)

Soon there will be no known-good laptops. Linux users are not a strong enough force in the market to change that.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 20:21 UTC (Tue) by johannbg (guest, #65743) [Link] (18 responses)

I'm not sure what you mean by "known-good laptops", what do you consider a "known good laptop"?.

My experience of using Lenovo's for decade is that it's support on Linux has been continuously improving over time ( and some areas it still needs improving upon ).

Recently they added another Linux distribution ( Fedora ) as an available OS when you buy one and they also have assigned liason between it's firmware team and the LVFS team ( which are actively working on improving that experience ).

As a vendor ( the largest vendor with quarter of the market share ) it's quite clear that it cares for it's Linux userbase regardless if it's "strong enough force" or not and I'm not seeing any indication that it's reversing that investment.

Do I care, expect or otherwise demand of Lenovo that the firmware it ships be free and open? Absolutly not, what I care about as an user is that the firmware on the devices I own is actively maintained ( which Lenovo most certainly is doing ) I receive update for it ( which I do via fwupd/lvfs ), the linux distro I use works on it out of the box ( which is Fedora and it does and I can even buy one directly from Lenovo with Fedora pre-installed ), there are parts available for my device from the vendor ( which are for Lenovo's ) and documentation how to repair it be provided by the vendor ( which Lenovo provides ) which gives me the ability to fix it myself or have some service center fix it for me ( authorised or not ).

To me the scenario above is a "known good vendor" that delivers me "known good devices" what is yours?

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 20:44 UTC (Tue) by fwyzard (subscriber, #90840) [Link] (15 responses)

What you are saying is "I want to be able to comfortably run a Linux distribution on my laptop", which is a perfectly valid point of view.
But it has very little to do with "I want to be able to run a fully Free Operating System on my laptop", which is what Debian is about.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 22:24 UTC (Tue) by johannbg (guest, #65743) [Link] (14 responses)

> But it has very little to do with "I want to be able to run a fully Free Operating System on my laptop", which is what Debian is about.

There is absolutly no obligation from vendors to partake in such endeavours which means that the community that upholds such an ideology will have to write the necessary code or engineer the required hw to achieve that by themselves ( or only support hw vendors that share/support that ideology which are what Chromebooks and System76 maybe few more ).

And since Debian is providing two set's of images one "offical" image which only contains free-software and another un-official image which contains non-free software which directly contradicts what Debian is supposingly supposed to be about, is that not a testament to what Debian claims to be about, being untrue? ( If Debian would stay true to it's ideology then those un-offical images would never have exist in the first place )

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 0:56 UTC (Wed) by anselm (subscriber, #2796) [Link] (13 responses)

And since Debian is providing two set's of images one "offical" image which only contains free-software and another un-official image which contains non-free software which directly contradicts what Debian is supposingly supposed to be about, is that not a testament to what Debian claims to be about, being untrue? ( If Debian would stay true to it's ideology then those un-offical images would never have exist in the first place )

Debian is considerably more pragmatic than the FSF when it comes to things like non-free firmware. According to Debian's social contract, “our priorities are our users and free software” (in that order). In fact, Debian has frequently got in trouble with the FSF because as far as the FSF is concerned the project is not sufficiently pure and zealous about free software.

In my experience, getting Debian to run on Lenovo laptops, especially the business-oriented ones, is generally not a huge problem (using the inofficial ISO images helps). People are more likely to have issues with getting Debian to run on whatever no-name el-cheapo laptop they have picked up somewhere, and then blame Debian.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 4:46 UTC (Wed) by pabs (subscriber, #43278) [Link] (4 responses)

> “our priorities are our users and free software” (in that order)

While the Debian SC does say that, there is no consistent prioritisation of one above the other, apart from the order they were written down affecting how people perceive the prioritisation to be.

https://lists.debian.org/debian-vote/2022/04/msg00039.html

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 5:29 UTC (Wed) by developer122 (guest, #152928) [Link] (3 responses)

If I were to break away from the dogma for a moment, tbh I don't give one flying hoot about the software. The software can rot in jail for all I care.

I care about the users, because that the THE WHOLE POINT. They are the ones that the four freedoms exist to protect. They are the ones that justify the existence of "free software." It was one single user long, long ago that couldn't get some printer signals mapped and he didn't embark on his mission for the sake of "the software." He did it because his printer didn't work.

Everyone in software needs to stop anthropomorphising the software. Right now.

(this is usually where I break into a whole story about the immense, tangible, practical user freedom that the minecraft modding community created, indeed *on the back of a definitely-proprietary game now owned by microsoft*)

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 7:21 UTC (Wed) by NYKevin (subscriber, #129325) [Link] (2 responses)

> (this is usually where I break into a whole story about the immense, tangible, practical user freedom that the minecraft modding community created, indeed *on the back of a definitely-proprietary game now owned by microsoft*)

See also Skyrim, where the massive company (Bethesda, which is now also owned by Microsoft) that owns the game has pretty much never been the actual source of problems (instead, it's always some variation on "yes, I know my mod is based on Bethesda's game, and available for free because I'm not allowed to charge for it, but I'm going to be as possessive as possible about it, because I can; nobody can change my mod or make other mods based on it, say that it's a bad mod, or otherwise annoy me, or else I'll yank it off the internet and accuse you of piracy if you rehost it anywhere - also the only place you're allowed to download it is this one website that has nine fives of availability and requires logins, so that I can ban individual people from downloading it if I don't like them").

McIntyre: Firmware - what are we going to do about it?

Posted Apr 23, 2022 22:24 UTC (Sat) by developer122 (guest, #152928) [Link]

Yes, many modding communities are toxic hellholes in and of themselves. That has no reflection on the general case.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 23, 2022 22:50 UTC (Sat) by bartoc (guest, #124262) [Link]

These modding communities tend to involve a fair amount of unpaid labor, volunteerism is great but it's really easy for "modding" to cross the line into labor exploitation, IMO.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 6:29 UTC (Wed) by johannbg (guest, #65743) [Link] (7 responses)

> Debian is considerably more pragmatic than the FSF when it comes to things like non-free firmware. According to Debian's social contract, “our priorities are our users and free software” (in that order). In fact, Debian has frequently got in trouble with the FSF because as far as the FSF is concerned the project is not sufficiently pure and zealous about free software.

Well if that was the case then obviously Debian has it's priorities mixed up.

If it's priorities was it's users then the official stamp of those images would be the otherway around since the inofficial image is what works for most users while it's religious one however does not.

If Debian's claims that it has two priorities, it's ideaology and it's users then both of those images would be presented on equal footing and the end users allowed to choose what works for *them* but in doing so it would go against it's religion.

It should be quite apparent that Debians have no faith in their ideology otherwise two images would not exist in the first place.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 15:56 UTC (Wed) by nix (subscriber, #2304) [Link] (6 responses)

> It should be quite apparent that Debians have no faith in their ideology otherwise two images would not exist in the first place.

Ten seconds of hanging out on any Debian mailing list would make it instantly clear that this is not true.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 22:12 UTC (Wed) by johannbg (guest, #65743) [Link] (5 responses)

Actions speak louder than some written words on a mailinglist, the existance of two images instantly make it true.

Can someone confirm or deny here on LWN that Debians own infrastructure is strictly being run on "free firmware" and is otherwise proprietary software free so atleast what is being pushed in the hands of the end users in the form of an "official" image and the ideology that is being preached is not just some wishful thinking and effectively is in use by the Debian project itself?

I would not even be surprised if it turned out that Debians own infrastructure was not being run on "free firmware" since it's infrastructure team probably cant and on top of that is probably smart enough to not make itself vulnerable for some ideology by refusing to update the existing non-free firmware, if updates are available.

If it turns out to be the case that the Debian project cant even follow it's own ideology and run off it, it can just as well stop riding that fat unicorn of wishful thinking, drop the religious shenanigans, stop confusing users with two images and deliver something that actually works in the hands of it's end users and itself actual is using in the form of an single official image.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 21, 2022 7:25 UTC (Thu) by pabs (subscriber, #43278) [Link] (4 responses)

Debian uses standard x86 servers for most of its infra, which requires using non-free firmware, both preinstalled and loaded at boot, including the CPU microcode packages that Debian distributes. In addition some non-free software packages are installed on some servers for managing hardware RAID and watching server health etc.

https://db.debian.org/machines.cgi
https://salsa.debian.org/dsa-team/mirror/dsa-puppet/

McIntyre: Firmware - what are we going to do about it?

Posted Apr 21, 2022 18:18 UTC (Thu) by nix (subscriber, #2304) [Link] (3 responses)

Quite. Isn't the whole point of this article that a rigid insistence on no non-free firmware is in fact *not* something Debian obliges its users to follow? If you want that, there are other distros that do it, right up to disabling pieces of the kernel that need non-free firmware and disabling warnings that your system is insecure because the microcode is old (they're mostly not terribly usable for exactly that reason).

McIntyre: Firmware - what are we going to do about it?

Posted Apr 22, 2022 4:10 UTC (Fri) by pabs (subscriber, #43278) [Link] (2 responses)

Right. Debian distributes proprietary firmware in the non-free component of the Debian archive. Debian distributes official installer images without proprietary firmware and separate non-free images that do contain proprietary firmware. Debian supports users who need non-free firmware (and software). Of course proprietary firmware/software isn't possible to support as well as libre firmware/software and Debian definitely aims to make that clear and allow users who want to avoid proprietary firmware/software to do so.

The proposal is to stop distributing the libre images and bless the non-free images as official.

I think that *neither* the status quo *nor* the non-free proposal balances the needs of *all* Debian users and our principles very well, both are biased in one direction or another.

I made a proposal that I think is better and solves most of the issues; The TLDR is a better download page; just make the download page direct each audience to the specific download that is most useful for them (Windows users get WSL, cloud users get cloud images, RPi users get the non-free RPi images (until the libre firmware improves), mobile users get Mobian, laptop/desktop/server users get non-free images, VM users get free images, RaptorCS users get free images etc). In each case where non-free is involved, both the download page and the downloaded image make it clear that there are downsides to non-free firmware/software.

https://lwn.net/Articles/892106/

McIntyre: Firmware - what are we going to do about it?

Posted Apr 22, 2022 5:46 UTC (Fri) by joib (subscriber, #8541) [Link]

IIUIC one of the motivations was to reduce the number of different images the installer team must provide? If so, reorganizing the download page doesn't help with that goal. Sure, by providing non-free firmware in the official images there's more disk space and BW usage for those users that don't want to use such things; OTOH there's also more BW usage with the current way of things where a user first downloads the official image, then notices it doesn't work on their HW, and then maybe finds and downloads the unofficial non-free image.

Personally I think the suggested option #5 is fine (but I'm not a DD so my opinion doesn't count). Perhaps with the interactive installer, upon detecting devices that require non-free FW, showing a screen "Your computer has devices X, Y, Z that require non-free firmware to work. These are provided by the device vendors and Debian has no way of inspecting or supporting them. Select Y to use non-free firmware or N to continue the installation without these devices". (If the Debian installer doesn't already do that, it's been quite a while since I've used the Debian interactive installer).

With such a thing people who don't want to use proprietary firmware will never load them, and AFAICS the purity of their computers isn't compromised.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 22, 2022 17:05 UTC (Fri) by salewski (subscriber, #121521) [Link]

> I think that *neither* the status quo *nor* the non-free proposal balances the needs of *all* Debian users and our principles very well, both are biased in one direction or another.
>
> I made a proposal that I think is better and solves most of the issues; The TLDR is a better download page; just make the download page direct each audience to the specific download that is most useful for them
[...]
> In each case where non-free is involved, both the download page and the downloaded image make it clear that there are downsides to non-free firmware/software.
>
> https://lwn.net/Articles/892106/

I think this proposal has much merit. Also, users are not necessarily in one camp or the other at all times -- some users may prefer an all-free system when they can (such as when spinning up VMs), but occasionally accept non-free bits to make a piece of actual hardware work (or similar).

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 4:57 UTC (Wed) by developer122 (guest, #152928) [Link] (1 responses)

See the comment I'm replying to. these are laptops which don't require firmware to be loaded in order to work. (or at least, give that appearance; nowadays intel's GPUs contain an entire 486 system just for power management and scheduling)

McIntyre: Firmware - what are we going to do about it?

Posted Apr 25, 2022 20:09 UTC (Mon) by flussence (guest, #85566) [Link]

I'm not sure why that's worded as criticism. Try describing competing GPU hardware in those terms and you'll quickly figure out why.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 16:51 UTC (Tue) by iteratedlateralus (guest, #102183) [Link] (1 responses)

I currently run a Debian build on a laptop that I bought about a year ago. I was forced to stomach the default windows installation and run everything through a VirtualBox instance before I could confidently install any sort of Linux on it.
Among the many things that didn't work, the wifi network card was a deal breaker as this netbook does not have an ethernet port.. It also wouldn't make much sense since this is a netbook: it's supposed to be portable.. I bought it so that I could choose a comfy place to sit and code.

I think support for hardware on Linux has come a LONG way in the past two decades. It would be nice to see manufacturers release a .deb in addition to whatever drivers are needed (and come with) the default windows installations. I say .deb because Ubuntu and Debian are HUGE in the Linux space.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 9:32 UTC (Wed) by eduperez (guest, #11232) [Link]

> It would be nice to see manufacturers release a .deb in addition to whatever drivers are needed (and come with) the default windows installations.

Yes, that would be nice, very nice indeed; but it will not happen, ever.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 17:06 UTC (Tue) by IanKelling (subscriber, #89418) [Link] (5 responses)

Based on my recollection, whenever this debate happens in debian, there are a bunch of debian developers, who say basically, lets all agree to make nonfree easier to install, oh and maybe we could do something really small to help people who want to avoid it. The problem is that in nonfree images, there is basically nothing which indicates to users why they would want to not install nonfree. A good approach is: first, make policy nonfree can only be made accessible in a way that concretely helps users understand they are compromising their freedom. https://www.gnu.org/philosophy/install-fest-devil.en.html makes a similar point.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 20:34 UTC (Tue) by NYKevin (subscriber, #129325) [Link]

This is the problem with the FSF - the use of charged emotional language and religious symbolism feels no different to propaganda like [1], and consequently I find it very difficult to take their core message seriously. (I'm sure lots of people in the 1940's laughed at that poster, too.)

[1]: https://energyhistory.yale.edu/library-item/when-you-ride...

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 1:32 UTC (Wed) by sjj (guest, #2020) [Link]

Install fests, seriously? What is this, the 90's?
My new idea is that the install fest could allow the devil to hang around, off in a corner of the hall, or the next room. (Actually, a human being wearing sign saying “The Devil,” and maybe a toy mask or horns.) The devil would offer to install nonfree drivers in the user's machine to make more parts of the computer function, explaining to the user that the cost of this is using a nonfree (unjust) program.
This is some cringe, as I've been told the kids say today.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 4:59 UTC (Wed) by developer122 (guest, #152928) [Link] (1 responses)

>make policy nonfree can only be made accessible in a way that concretely helps users understand they are compromising their freedom.

Yeah, except the only way I've ever seen that implemented was preachy and annoying. Often also raid-roaded the user into a more broken computer too.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 5:04 UTC (Wed) by developer122 (guest, #152928) [Link]

Well, that's not exactly true. PostmarketOS had the idea to offer a selection screen with various checkboxes: "sacrifice the functionality or include a particular blob in the install"

But they never made *any* attempt to cram "education" into it like the FSF does.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 21, 2022 1:02 UTC (Thu) by motk (subscriber, #51120) [Link]

This is so cringey I have turned inside-out from pure cringe.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 19, 2022 19:50 UTC (Tue) by developer122 (guest, #152928) [Link] (8 responses)

something that's often overlooked too: A lot of firmware is very poorly written. Not just in the sense that it's bad code, but that the actual act of writing is being done badly.

Oxide talked about it a while back, in reference to hard drive firmware: https://www.youtube.com/watch?v=qisoAIx8EE8

Most hard drives these days have firmware that isn't developed using version control, and continuous integration (even with software-hardware co-simulation being a thing for over 30 years now) isn't even something most companies are aware of.

And this is typical of virtually all firmware development. It's software that's written by hardware engineers with no real background or experience in programming or software development, and is a huge burden to maintain. They've gotten away with it for a long time now, but that's because the firmware has typically been relatively small and simple. Essentially on the scale of a CS course programming project. It's rapidly growing however, with more and more functionality offloaded to it, and bugs and security issues abound.

Even if we can't push for companies to release their code (or better yet, documentation) and let us do it, we should at least try to enlighten them as to how modern software is written. Version control, git, CI, and so on. Then at the very least the firmware we're forced to run might be a little less crap, and maybe down the road they might come around to community involvement for offloading/sharing some of the work too.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 8:08 UTC (Wed) by Vipketsh (guest, #134480) [Link] (2 responses)

In my experience it is very hard to find people who can do firmware well. One reason, I think, is that "it's just software" so you have people who were great at writing some GUI in java or some such and then decide to "do this other software that is all the same, right?". Managers end up hiring such people because they think so too. In reality firmware is a different bread of software and the mentality required is different too. Things could be a little better if there was a much bigger differentiation for firmware and engineers who are specialised to that field.

> It's software that's written by hardware engineers

Again, in my experience, if you want to see hardware engineers run away as fast as possible ask them to write any kind of code. So, no, I don't think this statement is true and that is part of the problem -- the hardware people should see a lot more of the software. When I was still only working with software (now I'm mostly on the hardware side) it wasn't once that after a few weeks of going crazy trying to find some issue and I managed to get the hardware folks to listen they would casually tell me something like "oh, you clearly need to respond to that interrupt in 15 cycles" (the only thing "clear" was that it wasn't documented anywhere). That's a long time for hardware but for software you are lucky to have dispatched an interrupt handler in that time. The end result was that the best I could do was to have the code was meet the deadline in about 90% of cases and have a whole slew of hacks in an attempt to detect the remaining 10% and try some restart. It definitely wasn't helping that the hardware guys liked to dismiss everything as "must be a software bug" and generally weren't happy to listen to anything.

The last problem that I see is that big (hardware) companies often have a very bureaucratic culture. So much so that every individual only knows a very small sliver of the overall problem and refuses to know anything outside of that because doing so would net them a crap ton of paper work and/or would feel more like a politician trying to win an election. I wouldn't be surprised if the people writing firmware without version control would like to do things differently but trying to get the appropriate people to agree to run svn or some such is so much effort to simply make it not worth it.

> even with software-hardware co-simulation being a thing for over 30 years now

You are fundamentally correct, but in practice it is so slow that it is impossible to do any kind of full verification that way.

In general, I do wish that software people would understand that working with hardware is its own unique specialty laden with its own practical problems and not simply "some other kind of software". Being mostly on the hardware side these days I can clearly tell that you and others in this thread talking about FPGAs above know a few things but don't have much, if any, experience working with hardware and end up with the wrong conclusions.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 13:26 UTC (Wed) by excors (subscriber, #95769) [Link]

> The last problem that I see is that big (hardware) companies often have a very bureaucratic culture. So much so that every individual only knows a very small sliver of the overall problem and refuses to know anything outside of that because doing so would net them a crap ton of paper work and/or would feel more like a politician trying to win an election. I wouldn't be surprised if the people writing firmware without version control would like to do things differently but trying to get the appropriate people to agree to run svn or some such is so much effort to simply make it not worth it.

This seems plausible to me - a lot of bad firmware isn't written by hardware engineers, it's written by dedicated firmware engineers who work for hardware-centric companies that don't put them in a position to succeed. The firmware teams are probably small in both absolute and relative terms, so the company doesn't think it's worth investing in a dedicated infrastructure team to maintain VCS and CI and code review tools etc. They don't have a security team to write policies and do architecture reviews and code audits. They haven't developed good hiring processes and career paths to find and promote the best software engineers. They don't have people transferring internally from other software teams to firmware teams, bringing experience of best practices from different fields.

I think it's much easier to develop good-quality firmware in a software-centric company, because all that infrastructure and those processes are already set up and well-tested, and the company recognises that they're critical to its core business. It's not that there's any less bureaucracy, it's just better aligned with the job requirements. Firmware development still needs specialist knowledge and experience that most software engineers don't have, so those companies might still produce bad firmware with the wrong people (though a different kind of "bad" - probably massively overengineered and inefficient and expensive), but if they get the right people they'll have the support needed to do a good job.

Of course you then risk the opposite problem where the hardware team is not well supported by the company, or the firmware and hardware teams might be in different companies entirely (and maybe different countries with different languages) so you get much worse communication problems. It's very hard to get the right balance, though companies like Apple and Google seem to indicate it is possible to do both well.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 14:01 UTC (Wed) by farnz (subscriber, #17727) [Link]

On top of everything you write, there's also the human issue of forming cliques; when I've worked on firmware in the past, I was told by my mentor (a competent firmware engineer) that I needed to learn a whole new way of reporting issues to the hardware guys to avoid the "it's a software bug" dismissiveness - instead, I needed to break out logic analyzers (including things like Xilinx Chipscope when the board I was working with used FPGAs) and/or oscilloscopes and put my issue reports in terms of "Software causes this sequence of pin changes, which should result in this happening on this measurement device here, but doesn't".

Other software people at the same company didn't bother learning how to speak "new grad level" hardware engineer, and didn't get on as well with their hardware counterparts as a result, forming a "software" and a "hardware" group within the same team. And breaking that (as I had to do when I moved team) was challenging; I succeeded only because I was able to avoid bringing in the hardware engineers on bugs until I had shown that the bug could be fixed by my lousy soldering (adding bodge wires), and by learning enough about schematics and VHDL (the HDL in use) that I could read their designs myself instead of asking them to interpret.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 8:08 UTC (Wed) by marcH (subscriber, #57642) [Link]

At last one comment with the only technically accurate definition of the word "firmware", thanks!

McIntyre: Firmware - what are we going to do about it?

Posted Apr 21, 2022 20:00 UTC (Thu) by marcH (subscriber, #57642) [Link] (3 responses)

> Most hard drives these days have firmware that isn't developed using version control, and continuous integration (even with software-hardware co-simulation being a thing for over 30 years now) isn't even something most companies are aware of.
> And this is typical of virtually all firmware development. It's software that's written by hardware engineers with no real background or experience in programming or software development, and is a huge burden to maintain.

This sounds like hardware development does not use version control and continuous integration.

Good quality hardware does, bad quality hardware does not. From the specific perspective of development practices hardware design is exactly like software design: its quality level is defined by its QA practices.

Now are these basic practices _less frequent_ in hardware design than in software? Maybe. That would be incredibly hard to measure and prove.

Also, " hardware engineer" is a bit vague: I'm guessing teams designing hard drives may not write much RTL code.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 23, 2022 22:41 UTC (Sat) by developer122 (guest, #152928) [Link] (2 responses)

They do not teach good software engineering practices to engineers who write RTL, design digital or analog circuits, do CMOS layout, do board layout, or handle really any other aspect of constructing hardware. Still, most of the time writing firmware is tacked onto these other roles. The reasons are largely historical: need for it has grown gradually, hiring new talent is an expensive investment, and the act requires a lot of hardware knowledge that would have to be conveyed. Turns out, when you try to get guys trained in another area to write software on the side, you get poorly-made software.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 24, 2022 3:27 UTC (Sun) by marcH (subscriber, #57642) [Link] (1 responses)

Maybe it changed recently but last I checked new _software_ graduates knew nothing about software engineering either. Computer science yes, engineering no.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 24, 2022 12:04 UTC (Sun) by mathstuf (subscriber, #69389) [Link]

When I went through, CS developers only saw VCS in:

- game development (IT-provided SVN server that was down every other week because someone managed to commit their `.svn` directory or do some other horrible no-no)
- software development and documentation where they *discussed* VCS, but never *taught* it
- open source practices course (covered the basics, but one class project was to contribute to an existing FOSS project)

Of these, SD&D was the only required course for CS students. I don't think things have changed much (though AFAIK the OSP course has become more popular).

Luckily, I had been involved with other FOSS activities since high school and had interacted with svn (via KDE) and git (fellow students) before graduating. I ended up hosting a git server on an old Pentium 2 box for my team in SD&D and the teams in GD that wanted to use Git (GitHub was just becoming a thing at the time).

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 8:12 UTC (Wed) by pabs (subscriber, #43278) [Link]

Here is a page about the libre firmware that we know about:

https://wiki.debian.org/Firmware/Open

Are there any other libre firmware projects?

McIntyre: Firmware - what are we going to do about it?

Posted Apr 20, 2022 11:22 UTC (Wed) by ledow (guest, #11753) [Link] (2 responses)

It's the old "idealism" vs "practicality" debate again. Any minute now someone will bring up Minix and microkernels.

The fact is that computers have to be available on a practical basis, on hardware that people are able to obtain, in order to be what they should be: useful tools.

It's like complaining that a hammer isn't made from 100% pure low-background-radiation steel. For 99.9999% of people, that really isn't going to make any difference and just makes it almost impossible to obtain such a tool or use it practically. I'm sure, for certain people, tasks, etc. that you need that perfect tool to be as sterile as possible and will absorb the expense and hassle of doing so.

But I can't imagine that the target audience of any Debian distro is that kind of person, as it would be incredibly niche. The target audience is just someone that has a machine and just wants to use it.

I understand the creep that occurs in allowing certain things, and moving away from the ideals, but the fact is that just about anyone using any Debian-based tool needs just such non-free software in order to make it work in any practical or reasonable manner. Even as someone who for years used to compile their own kernels, dedicating hours of computer time to apply tiny personal patches, it was inevitable that I had to "load blobs" for wireless, networking, even sound cards (sound fonts, etc.) to make things work, even 15, 20 years ago.

Debian's idealism means that there are a number of distros that are basically nothing more than Debian + nonfree, just to make it usable.

And, yes, in an ideal world, I'd love all that firmware to be open-sourced, but it's never going to happen (e.g. radio regulatory blobs for wireless cards which cannot legally be tampered with).

McIntyre: Firmware - what are we going to do about it?

Posted Apr 21, 2022 7:19 UTC (Thu) by pabs (subscriber, #43278) [Link]

There are several open firmware projects for radio devices (including GSM and WiFi) already:

https://wiki.debian.org/Firmware/Open

McIntyre: Firmware - what are we going to do about it?

Posted Apr 22, 2022 2:53 UTC (Fri) by dvdeug (subscriber, #10998) [Link]

In your analogy, Debian GNU/Linux would be designed by volunteers who have agreed that making equipment of low-background-radiation steel is important. Just dismissing that as a goal is pointless; that is part of the goal of the people making Debian GNU/Linux. It is not true that "The target audience is just someone that has a machine and just wants to use it"; there are other distributions for that.

McIntyre: Firmware - what are we going to do about it?

Posted Apr 21, 2022 7:33 UTC (Thu) by pabs (subscriber, #43278) [Link]

My preferred solution to this issue looks somewhat like what is written below.

I believe this solution does not require a vote, as it balances the needs of different segments of the Debian community and Debian's principles.

Keep providing images without proprietary firmware, disable the automatic download on the website download page, create a new download page containing separate side-by-side panels catering to different segments of the audience for installing Debian, ordered by the sophistication of the audience and the amount of people in the audience. Each of the panels should contain appropriate iconography and minimal text to avoid overwhelming people; longer explanations could be hidden behind links/dropdowns. Where the images containing proprietary firmware images are mentioned, include an item pointing out the non-free firmware and the limitations of Debian support for that firmware. Some examples of the panels that could be included:

* VMs; linking to the free images
* relatively libre hardware like RaptorCS's POWER9 computers; linking to the free images and explaining that we need help packaging the free firmware for those devices, since most of it isn't in Debian yet
* desktops/laptops; linking to the non-free images
* smartphones/tablets; linking to Mobian images
* clouds; linking to the cloud images
* servers; linking to the non-free images
* Windows; linking to the Debian WSL app and win32-loader
* SBCs; linking to some relevant info on the wiki
* Raspberry Pi; linking to the RPi non-free images

If the Debian CD/live teams don't want to test the free images, then those teams should feel free to stop testing them, and call for other people to do that work.

If the Debian CD/live teams don't want to build the free images, then that would pose a problem for people who don't want to download or redistribute non-free firmware. I think there are enough of these people that the opportunity to hand over building the free images to a separate team would be needed.

I feel that the above is a good balance of preventing the problems pointed out in the proposal mail (along with some other ones), while not completely fixing all of them due to the conflict with the needs of the segment of Debian users who want to not have non-free firmware and then mitigating that through the creation of a separate team for that.

We need to solve the issues around non-redistributable firmware too:

Some firmware is only available in the operating system preinstalled on the device and needs to be manually extracted before d-i is run, potentially even only from processes running on the preinstalled operating system in cases where the storage must be wiped (such as Android devices) before an alternative OS like Debian can be installed. IIRC there is some laptop WiFi firmware that is like this.

Some firmware is not redistributable and is only available in proprietary drivers on websites and is hard to extract from those drivers. IIRC some of the proprietary nvidia firmware for use by the libre nouveau GPU driver is like this, both signed firmware for very
new nvidia hardware and unsigned firmware for very old nvidia hardware, although the firmware for the old nvidia hardware has libre firmware in nouveau, but the libre firmware is/was buggier than the proprietary firmware. The tools for extracting the old firmware aren't in Debian.

We also need to do something to improve open/libre firmware:

We could package the libre/open firmware projects that are missing.

We could lobby hardware vendors to release their existing proprietary firmware and hardware management software under libre licenses.

We could lobby hardware vendors to stop requiring signatures for the existing libre/open firmware projects.

We could lobby hardware vendors to allow users to enrol their own keys for verifying firmware.

We could use some of our money to fund new work on the existing libre/open firmware projects.

We could initiate the creation of a new non-profit dedicated to the reverse engineering and reimplementation of proprietary firmware.

McIntyre: Firmware - what are we going to do about it?

Posted Jul 30, 2022 11:21 UTC (Sat) by amacater (subscriber, #790) [Link]

It may also be worth looking at the recent session at Debconf22 in Prizren where this was discussed in detail by Steve McIntyre.

https://meetings-archive.debian.net/pub/debian-meetings/2022/DebConf22/debconf22-199-fixing-the-firmware-mess.webm


Copyright © 2022, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds