LWN: Comments on "Security policies for GNU toolchain projects"

Security policies for GNU toolchain projects

fest3er — Thu, 08 Feb 2024 17:48:56 +0000

Let's not conflate security flaws with bugs.

IMO, it's quite acceptable for your program to simply crash when it tries to handle impossible input. But it is quite unacceptable for such a crash to cause you to be able to do things that you cannot do and ordinarily are not allowed to do.

Let's not allow chaotics, however well-intentioned they may be, to drive wedges between us all. Fuzzing is akin to panning for gold. You run a ton of dirt and sand through your pan to find an ounce of gold. Fuzzies need to learn that "all that glitters is not gold." They need to learn that, after their panning, they will have nearly ten pounds of iron pyrite (general bugs) in addition to the ounce of gold (security issues).

CNAs who behave like Soltan Gris ("Mission Earth")--who stamps everything that crosses his path with his idento-plate--should be censured or ostracised.

Security policies for GNU toolchain projects

PengZheng — Sat, 07 Oct 2023 04:41:03 +0000

> My point is that unless you've audited it line by line, you can't run that program outside of a sandbox safely.

I agree.

>So why is it a big deal to have to compile it inside the same sandbox you run it in?

Maybe just open an interesting project within my IDE to satisfy curiosity.
Some IDEs such as Eclipse CDT will try to build the project.
I just knew yesterday even if I checked the build system carefully, opening a small GitHub project may get my computer hacked. Please note that the workload of line-by-line review of the source code is much larger than that of reviewing the build system. For small projects, the difference could be several orders of magnitude.

Security policies for GNU toolchain projects

PengZheng — Sat, 07 Oct 2023 02:43:30 +0000

Too many judgements you are not qualified to make. I'll stop here.

Security policies for GNU toolchain projects

pizza — Sat, 07 Oct 2023 02:30:24 +0000

> Otherwise, just ignore my comment, since it's meant to give some helpful feedback to someone who is responsible for GCC.

Your "Helpful feedback" essentially boils down to "you should re-architect and re-implement most of GCC to make me happy"

So unless your "helpful feedback" coincides with great deal of funding, it's not worth the storage it takes up.

Security policies for GNU toolchain projects

PengZheng — Sat, 07 Oct 2023 02:18:52 +0000

> Sure, that's your right. Feel free to write a better toolchain or switch careers to something you find more acceptable.

I noticed you are debating with others about the meaning of "unacceptable". If you are maintaining GCC, then I apologize for noting using the most suitable word. Otherwise, just ignore my comment, since it's meant to give some helpful feedback to someone who is responsible for GCC.

Security policies for GNU toolchain projects

pizza — Fri, 06 Oct 2023 18:09:05 +0000

> So it's only a little bit insecure? A small crash? A tiny memory corruption?

*shrug*

Either it's okay for what you need it to do (and let's be honest, it's been considered okay for several decades) or it's not okay, therefore *you refuse to use it*. That's what "acceptable" means.

(...Since there is no meaningful alternative, "refuse to use it" means you're going to be looking for a different line of work!)

Most of my career I've been beholden to contractually-specified "acceptance criteria" -- if what I deliver doesn't meet that criteria, it's not accepted and I don't get paid. The specifics are negotiated based on their estimated cost, and once agreed upon, they can't be arbitrarily changed.

Security policies for GNU toolchain projects

mb — Fri, 06 Oct 2023 17:33:57 +0000

So it's only a little bit insecure? A small crash? A tiny memory corruption?

Security policies for GNU toolchain projects

pizza — Fri, 06 Oct 2023 17:02:01 +0000

> Cost is a filter, not a magic thing for making things acceptable.

"acceptable" == "good enough"

Security policies for GNU toolchain projects

mb — Fri, 06 Oct 2023 16:57:48 +0000

>And funnily enough, when the _cost_ is factored in, most "unacceptable" situations suddenly become "acceptable"

That's not how I see it. It makes the software be restricted to the acceptable use cases and leave the unacceptable use cases unacceptable. Cost is a filter, not a magic thing for making things acceptable.

Security policies for GNU toolchain projects

pizza — Fri, 06 Oct 2023 15:20:47 +0000

> But telling "don't do that" or "if you want that, do it yourself" is not a way of discussing things.

What _is_ important is that the _cost_ of rectifying things be considered. Otherwise it's just people demanding O_PONIES.

(And funnily enough, when the _cost_ is factored in, most "unacceptable" situations suddenly become "acceptable")

> Can we agree on that the current situation is as it is and there are workarounds available, but it would be better if programs would not be vulnerable?

Sure, I don't think anyone disagrees with this.

Security policies for GNU toolchain projects

mb — Fri, 06 Oct 2023 15:00:50 +0000

I might want to build test it and not run it. E.g. on a CI server.

Please stop making excuses. We do understand the current situation and we do understand why it is as it is.
But telling "don't do that" or "if you want that, do it yourself" is not a way of discussing things.

Can we agree on that the current situation is as it is and there are workarounds available, but it would be better if programs would not be vulnerable?

The same discussion comes up again and again. I remember filesystems and toolchains and other complex tools.
It's not acceptable for these to be vulnerable. But it's the best we currently have.
We all need to change our mindset into the direction that things shall not be vulnerable by default.
In a world where everything is networked and tools get thrown into network accessible setups that have never been designed for it, this becomes gradually more important to improve that mindset.

Security policies for GNU toolchain projects

dvdeug — Fri, 06 Oct 2023 14:22:03 +0000

My point is that unless you've audited it line by line, you can't run that program outside of a sandbox safely. So why is it a big deal to have to compile it inside the same sandbox you run it in?

Security policies for GNU toolchain projects

pizza — Fri, 06 Oct 2023 12:50:02 +0000

> I found this unacceptable.

Sure, that's your right. Feel free to write a better toolchain or switch careers to something you find more acceptable.

Security policies for GNU toolchain projects

PengZheng — Fri, 06 Oct 2023 02:02:17 +0000

The security policies imply that checking the build system is not enough if I want to build an interesting small GibHub project on my local machine. I should do it in VMs/containers/sandboxes, otherwise I need to check all source code line by line.

I found this unacceptable.

Security policies for GNU toolchain projects

mirabilos — Thu, 05 Oct 2023 22:44:53 +0000

WTF?

Especially a tool such as objdump must be robust and secure with ANY input.

Security policies for GNU toolchain projects

dvdeug — Thu, 05 Oct 2023 20:34:56 +0000

It gets hard, though; which of the twelve dozen standard Unix utilities can't be trusted to work with arbitrary input? I believe the numbers dropped a lot between OG Unix and the BSD/GNU versions, and then dropped some more, but what's the set that's left? If I'm poking at an unknown binary, file, strings, grep, then ldd and objdump?

Note that the ldd man page says "A safer alternative when dealing with untrusted executables is: $ objdump -p /path/to/program | grep NEEDED". I don't know how to know what could bite me on untrusted data, because obviously even the manpages don't consistently know.

Security policies for GNU toolchain projects

dvdeug — Thu, 05 Oct 2023 19:35:43 +0000

What do you plan to do with the results of gcc -Wall arbitrary.c -o hello? "gcc -Wall arbitrary.c -o /dev/null" could be a useful command to run with a secure C compiler, but "./hello" absolutely hands the keys to your computer to the authors of arbitrary.c and if you're going to run hello in a sandbox, you may as well have run gcc in a sandbox.

Security policies for GNU toolchain projects

PengZheng — Wed, 04 Oct 2023 14:39:02 +0000

I understand that `ldd` can not be used securely on an untrusted executable, which is mentioned by its manual page. I am too surprised to know objdump should be used in VM/sandbox.

> Anybody who expects to feed such input to a compiler and maintain the integrity of their system is naive. Compilation of untrusted code should be done in a sandbox.

If playing an arbitrary MPEG-TS file with VLC gets my machine hacked, this is definitely a security issue of the VLC media player. I really cannot see why GCC is allowed to break my computer by running a simple `gcc -Wall arbitrary.c -o hello`.

Whatever a security policy the GNU toolchain projects define, if `gcc -Wall arbitrary.c -o hello` may get a computer hacked, open source would die.

Security policies for GNU toolchain projects

farnz — Tue, 03 Oct 2023 19:33:53 +0000

The underlying incentive issue (that leads to the Goodhart's Law problem) is that "security researchers" are rewarded for identifying bugs to at least as high a degree as they're rewarded for fixing and/or mitigating them. This made a lot of sense when finding security-relevant bugs was done as manual labour, and where you'd not only identify a bug, you'd also spend time showing that this bug was security-relevant (e.g. with a proof-of-concept); but nowadays, you also get the "points" for finding a bug that fits a security-relevant category, even if you don't then go as far as the PoC nor confirming that it's still a relevant bug in currently supported software (I've even seen people try to argue that bugs in Windows XP should count, because it's still used, even though it's completely unsupported).

Hopefully, we'll see incentives realign once people realise that "CVE count" and "bugs filed" are gamed metrics, and we'll go back to a world where the valued metrics are those that indicate human intelligence behind the bug report.

Security policies for GNU toolchain projects

rgmoore — Tue, 03 Oct 2023 17:55:04 +0000

No, you block account of fuzz-diot who thinks that someone else should do all that work just to make him (or her) happy.

The big problem is you are effectively prevented from doing that. Instead of just filing bugs, the fuzzers are filing CVEs with third parties, so the developers don't have the power to shut them down. And the CVEs are taken very seriously in some sectors- hey, they're handled by a third party so you know the developers can't hide bugs!- so developers are often forced to deal with them rather than the problems they would rather be dealing with.

Security policies for GNU toolchain projects

rgmoore — Tue, 03 Oct 2023 17:51:42 +0000

It sounds like the real problem is a classic case of Goodhart's Law: when a measure becomes a target, it ceased to be a good measure. Because "security researchers" are rewarded for finding bugs and filing CVEs, they're just filing bugs and CVEs without bothering to perform any kind of quality control. Even worse, the CVE authorities aren't performing adequate quality control on the CVEs they issue, so the garbage bug reports turn into garbage CVEs. If we really want to solve the problem, we need to fix the incentives so people are no longer rewarded based on sheer volume of bug reports and CVEs.

Security policies for GNU toolchain projects

farnz — Tue, 03 Oct 2023 15:46:26 +0000

You said that the fuzzer log and finding the bug would be useful to you. Now you're going back on that, and saying that no, actually, you need a better report. Getting a better report from this user took me around 90 minutes of effort, before I finally got to the point of being able to be confident that the bug they were finding had been fixed ~3 year ago, and if they were testing supported versions (instead of a random ancient release that their professor had assigned them to fuzz with their student fuzzer), they'd not have found this bug, or any of the other bugs they were finding (which were all variations on the initial bug in the input parser - from correspondence with the professor, the whole reason they were being asked to test their fuzzers against this version was that the parser bug was relatively easy to find, because so many different inputs would find the bug, and they'd see many different failures for one bug in the supplied code).

And this is my entire point - the bug finding is of zero value, but the report that you give me having found a bug can be of value, as long as it's a good enough report. You keep telling me that a report on its own has value, and then saying, "ah, but not like that - it needs more to be valuable", and I simply disagree with that premise; a bug report has to meet a minimum standard to be useful.

Security policies for GNU toolchain projects

mb — Tue, 03 Oct 2023 15:38:51 +0000

>Seed 0xaa17a25a caught SIGSEGV in child. FAILED
>Since you say this is useful to you

It's just a bad fuzzer and a bad report.
Software is bad. Your software crashes and the fuzzer doesn't generate good reports. Both are cases of bad software.

I would still appreciate to receive such fuzzer reports. It enables me to deal with the problem in my software by getting back to the reporter, asking for further information, or by just ignoring it. Both ways are fine.

>can you explain how I'm meant to use this without going back to the sender and asking them for more detail?

I never made that restriction.

Security policies for GNU toolchain projects

farnz — Tue, 03 Oct 2023 15:16:59 +0000

But I did tell you what I found - I gave you as much detail as I'm getting in fuzzer reports from one university student (whose reports have finally stopped coming in now that I've set up rules to redirect them to one of the CS professors at that university).

This is the problem that the GNU toolchain policy is aiming to stave off - useless reports that boil down to "I found a bug in some version of a GNU toolchain project". You need to be able to tell them which version, how to reproduce that bug, and what the symptoms are for a fuzzer report to be useful.

But since you say the fuzzer log is a useful report to you, the following is the full fuzzer log I was getting until I put the redirect rules in place:

Seed 0xaa17a25a caught SIGSEGV in child. FAILED

Since you say this is useful to you, can you explain how I'm meant to use this without going back to the sender and asking them for more detail? I had to ask them what code they were testing, what the fuzzer they were using was, how to get the test case that the fuzzer was running (since once I'd found out what the fuzzer was, it turned out that seeds were not portable between builds of the fuzzer - it was one built as an academic exercise, not something like AFL or libFuzz).

Maybe bring this to a close?

corbet — Tue, 03 Oct 2023 14:57:17 +0000

I'm not sure that this conversation is converging on anything useful, and I would really rather not see terms like "fuzz-diots" used here. May I gently suggest that the time is coming to wind this discussion down?

Security policies for GNU toolchain projects

mb — Tue, 03 Oct 2023 14:57:08 +0000

You are right. Finding bugs and not telling anyone about what I found is not useful.
Surprise surprise.
I obviously wasn't talking about that nonsense case.

>And no, a straight fuzzer log is not a useful report

To me it's useful. So?

Security policies for GNU toolchain projects

farnz — Tue, 03 Oct 2023 14:45:24 +0000

You literally did say "That's completely wrong. Finding bugs, especially security bugs, enables the users to react and for example shut down or otherwise mitigate a security problem.". But when I pressured you to explain how finding bugs enables any of this, you asked me for a usable report of the bug I'd found, and not just proof that I'd found a bug.

And no, a straight fuzzer log is not a useful report; IME, it doesn't include enough information to reproduce the bug, since it omits the software version that it ran against, and often does not include enough of the test case for me to replicate the bug locally, since some fuzzer logs simply tell me the "seed" I need to make the fuzzer produce the same output, but don't tell me how to get a build of the fuzzer that'll reproduce the output given the seed value.

On top of that, too many bugs in the bug tracker means that bugs that affect a real user are hidden by the flood of bugs that have only (so far) been reproduced by a fuzzer. Especially with tools like the GNU toolchain, a bug that affects a developer using the tool is far more important than one that depends on providing the sort of input that a human would never supply. The time taken to analyse it, ignore it, or wont-fix it is not zero; if the bug is theoretical only (as many fuzzer bugs are), that's wasted time on the part of the triager, who may miss a more important bug that should be dealt with sooner as a result of triaging fuzzer bugs.

Security policies for GNU toolchain projects

khim — Tue, 03 Oct 2023 14:34:53 +0000

> Run a fuzzer and dump the logs into the bug tracker.
> That is useful. It is a useful report.

No. It's not. Most bug-reports filed by fuzz-diots don't include enough information to do anything with that information without conducting a lot of research. Many (most?) don't even bother describing how you can reproduce bug in question and even don't tell you what component is affected and what precise version was tested.

You have to [try to] glean that information from fuzz-diot, like you have tried to do with farnz “bugreport” but more prudent action would be to plug that disaster at the source.

> And you as the developer may choose how to handle it.
> You may analyze it. You may ignore it. You may wont-fix it. Or you may blame the reporter. etc..
> Up to you.

Since we are dealing with DoS attack the best action here is to find the host that sends them and make it stop. And that means “attacking the messenger”. Filtering out these useless “bugreports” is only the second best, but on current “woke culture” it's, often, the only option because best route is not accessible to you.

Security policies for GNU toolchain projects

mb — Tue, 03 Oct 2023 14:21:00 +0000

>Because you said that the bug finding part of the process is valuable in its own right,
>even if there's no useful report,

No. I didn't say that.
It's obvious and it's common sense that just finding something and not telling anybody about it has no value for anybody.

The "finding a bug" that I am discussing here is:
Run a fuzzer and dump the logs into the bug tracker.
That is useful. It is a useful report.

And you as the developer may choose how to handle it.
You may analyze it. You may ignore it. You may wont-fix it. Or you may blame the reporter. etc..
Up to you.

Security policies for GNU toolchain projects

farnz — Tue, 03 Oct 2023 12:55:32 +0000

Because you said that the bug finding part of the process is valuable in its own right, even if there's no useful report, and that's what I strongly disagree with. Bug finding is the valueless part of the process; it's the reporting and fixing that gives it value.

And you absolutely can drive your counter with invalid bug reports; this is part of the problem, since you face things like "here is a bug I have found in your project", which people count even if it's against an ancient version of the project that's now clearly unsupported. And they try to justify this with "well, I know that it's out of support, but if you're rebuilding the project for RHEL 7, then rather than starting with the current version, you'd start with the ancient version that I tested!".

But, to go back to the point I was making, the value is not in finding the bug; the value is in reporting the bug in a manner that other people can make use of. And your replies to me keep circling this - you want more information from me, because the fact that I found a bug has zero value to you - whereas if I gave you more information, we could get to a point where you could extract value from my report.

Note that the valuable thing is not that I found a bug - it's that I reported it with enough information for you to make a useful decision about what I'm reporting. Finding the bug is a necessary step in that process - without finding a bug, I can't write a bug report with any useful information in it - but it has no value on its own.

Security policies for GNU toolchain projects

mb — Tue, 03 Oct 2023 10:08:11 +0000

>given that all you know is that I've found a bug in a named project.

How is that relevant to the discussion about fuzzing?
The fuzzer has an output log.
If you don't give that to me, then the only thing I can do is ignore it.

But that is not what actually happens.
"There is a bug" is not a valid bug report and you can't drive your CVE counter with that.

Security policies for GNU toolchain projects

farnz — Tue, 03 Oct 2023 08:55:40 +0000

But you said that there were other responses you could do, given that I'd found a bug. I'd like you to talk me through how you'd do those other responses, given that all you know is that I've found a bug in a named project.

Or are you shooting me for being the messenger here, and wasting my time by ignoring my valuable bug finding activity?

Security policies for GNU toolchain projects

anselm — Mon, 02 Oct 2023 22:52:50 +0000

I don't know what you want. I already said that 1000 automated notifications from fuzz testing are a nuisance at best. Nobody wants those. That obviously applies to all kinds of software including, but not limited to, compilers.

Having said that, I wouldn't worry too much about compilers because few people put language compilers in places where they need to deal with arbitrary possibly-malicious input (other than perhaps JavaScript JIT compilers). In the usual case of somebody running a compiler on their development machine, getting, e.g., a meaningful privilege escalation out of source code that is fed to the compiler is usually a pretty long shot. IOW, compilers are not prime targets for crackers. It's probably easier and cheaper to issue a security policy saying that the compiler only works with input that resembles valid source code in the language it is supposed to compile than it is to make it deal meaningfully with a gzipped copy of War and Peace. Which is not to say that compilers shouldn't carefully check their input, but perhaps we don't need to be quite as hard on compiler authors as we would be on the authors of web servers or mail servers which are exposed to the Internet as a matter of course, and are expected to be bombarded with malicious stuff.

Of course, if you want to be known as the king of fuzzers by number of crashes detected (and CVE IDs issued), then fuzzing a compiler and associated tools is like shooting fish in a barrel. There are probably more worthwhile projects you could direct your energy towards, but those might not deliver the same quick wins and are therefore less fun.

Security policies for GNU toolchain projects

mb — Mon, 02 Oct 2023 20:51:51 +0000

Ignore it. As I said, that is a valid response.

Security policies for GNU toolchain projects

farnz — Mon, 02 Oct 2023 20:39:59 +0000

I have. My users are all well aware that I do not support Fedora 30 versions of the package. But someone out there has taken it upon themselves to fuzz the package as found in Fedora 30, and report bugs at an incredible rate - all of which are fixed in the current version. They refuse to test the current version, because that's not going to get them the points their professor has said they'll get for their course, since it's got many fewer bugs.

And you still haven't explained how reporting that I found a bug is useful, if that's the sum total of the report.

Security policies for GNU toolchain projects

farnz — Mon, 02 Oct 2023 20:34:26 +0000

Also note that you said in this comment that given that I have found a bug, you can do things other than just ignore it: "I can evaluate it. I can mitigate it. I can fix it."

All I want you to do is tell me how you'd do one of those things, given that I've found a bug - after all, you said that the act of finding a bug is valuable in its own right, even if the report is literally "there is a bug", and you said that you can do all of those 3 things off the back of me finding a bug.

Security policies for GNU toolchain projects

farnz — Mon, 02 Oct 2023 20:32:19 +0000

That's avoiding the question - you said that the act of finding a bug is valuable, because the fact that I have found a bug enables you to take one of several actions, or you can ignore the report as having no value.

You're now saying that because my bug report was insufficiently detailed, you're going to ignore the report. What happens if I respond by re-opening with a script every time you click close?

Security policies for GNU toolchain projects

mb — Mon, 02 Oct 2023 20:18:44 +0000

Still no link to the tracker that receives almost 7 invalid bug reports per minute?

>Fedora package I maintain

You should disclose that information to warn your users.

Security policies for GNU toolchain projects

farnz — Mon, 02 Oct 2023 20:13:15 +0000

I get a pile by e-mail, from people whose source addresses I've ended up blocking because it's a huge flood of unwanted crap, relating to a Fedora package I maintain. Simpler to just block the senders than to deal with them - and just ignore the bugs because the act of bug finding is valueless without some human intelligence applied to reporting the bug in a useful fashion.

Security policies for GNU toolchain projects

mb — Mon, 02 Oct 2023 20:11:10 +0000

>I found a bug, and reported that I had done so.
>You are now asking me to more than just the bug finding -
>I need to provide a log, a stack trace, or a fuzzer script.

If your bug report literally is "there's a bug", then I click on close.
Simple.
And has never happened.