Quotes of the week [LWN.net]

We need to start designing our code in a way that doesn't require extensive testing to validate it as correct. If the only way to validate new code is correct is via stochastic coverage via error injection, then that is a clear sign we've made poor design choices along the way.

— Dave Chinner

May all kernel developer's CVs be filled with CVEs, think of it as an early holiday present :)

— Greg Kroah-Hartman (some scrolling required)

to post comments

Quotes of the week

Posted Feb 15, 2024 18:47 UTC (Thu) by k3ninho (subscriber, #50375) [Link] (4 responses)

I normally groan at /design/ but the Quality Assurance movement has been "pushing left" to get easy-to-validate testing done with short cycle times and at the earliest stages of releasing working software.

I don't know if the general use-case of an operating system platform can have simple designs and obvious or not-onerous verification, but I like the idea and would take on the challenge.

K3n.

Quotes of the week

Posted Feb 18, 2024 22:43 UTC (Sun) by dgc (subscriber, #6611) [Link] (3 responses)

> the Quality Assurance movement has been "pushing left" to get easy-to-validate testing
> done with short cycle times and at the earliest stages of releasing working software.

Right. The problem is that CI testingmust be brief enough that it can run all the time, yet comprehensive enough that it replaces typical level 1 testing. i.e. that the software can be used by everyone and it won't catastrophically fail.

The way we currently do testing is that every developer has dedicated resources and no fixed timeframe to test their own code, and then we expect QE teams and CI systems to take that and use it all the time on every commit. What we end up with is either CI testing runs within timeframe but does not have enough coverage to find "obvious" bugs, or CI takes so long it is a barrier to progress.

When every subsystem assumes they have -all- the testing resources -all the time- and designs code and test infrastructure around this resource availability, we end up in a situation that nobody can perform level 1 integration tests in a sane amount of time or with a sane amount of resources.

IOWs, the current approach we have to testing our code simply doesn't scale down, up or out. Nothing will change if we don't start to think a little differently how we write and test code - the two are not separate problems, they need to be thought about when designing the code. i.e. asking "how do we test this code" whilst we are designing it, not just "how do we implement this".

(This is a common engineering problem - think about new cars and how difficult they are to maintain. e.g. having to remove the engine from the engine bay just to replace the rear spark plugs. That's because no engineer designing the engine thought about how to access the parts of the engine that need regular maintenance during their design phase. They thought about how cheap is it, how long it needs to last (i.e. warranty period), optimised for the factory install process and thought no further than that. It's pretty clear we are failing in similar ways when it comes to thinking about test and validation when designing software....)

Quotes of the week

Posted Feb 19, 2024 0:43 UTC (Mon) by pizza (subscriber, #46) [Link] (2 responses)

> This is a common engineering problem - think about new cars and how difficult they are to maintain. e.g. having to remove the engine from the engine bay just to replace the rear spark plugs.

It's more complex than that -- Modern vehicles are designed to require _far_ less maintenance than their predecessors, so engineers are able to optimize for something other than ease of routine maintenance. This is _especially_ true when it comes to spark plugs; due to the advancement of material science, spark plugs are no longer a routine maintenance item! Modern spark plugs (along with very-high-voltage oil-on-plug ignition systems) are designed to easily last 160,000km, which coincides with a typical engine's recommended timing belt and water pump service interval. Given that the latter typically requires (partially) removing the engine on most vehicles, changing the plugs as well will take about five minutes of extra labor.

Quotes of the week

Posted Feb 19, 2024 6:25 UTC (Mon) by dgc (subscriber, #6611) [Link] (1 responses)

Yes, I know that modern spark plugs might now last that long; that's part of the point I was trying to make. Spark plugs used to be a 10-40,000km replacement item, and so it was a major pain in cars that had poorly accessible spark plugs (and there are a lot of them!).

Your comment points out the fact that design changes can completely remove the need for something to occur in downstream processes. i.e. instead of having to replace spark plugs regularly and revise the engine bay layout to make that less expensive, the engineers leveraged advances in spark plug technology to avoid changing them at all between major lifetime service intervals.

That was the point I was actually trying to make: when you think about failure and maintenance at design time, solutions can look -very- different but don't have any of the downsides of the current ways of doing things. That's what we need to change - thinking about verification at design time, not as an implementation afterthought...

Quotes of the week

Posted Feb 19, 2024 10:35 UTC (Mon) by farnz (subscriber, #17727) [Link]

It's worth noting that this approach to design leads to conflict between manufacturers and hackers; if your engine is designed so that many parts are easily replaced (because you expect that they'll be replaced 10 to 30 times in the vehicle's lifetime), you've built an engine that's relatively easy to modify. If you design an engine knowing that those self-same parts might not need replacing at all in the vehicle's lifetime, and if they do need replacing, it'll be co-timed with another big job, you're more likely to design the engine such that it's hard to modify (but cheaper to manufacture, or more fuel-efficient, or otherwise better).

This then has hackers seeing a conspiracy against them; the 1997 vehicle was much easier to modify than the 2011 vehicle, so "clearly" the manufacturer has made those changes to crack down on people who modify their vehicles, when in fact it's about designing for maintenance. And I've seen this argument made about fixing kernel vulnerabilities; clearly, people fix local privilege escalation bugs because that makes it harder to jailbreak locked-down Android phones and the like.