A fuzzy issue of responsible disclosure [LWN.net]

A fuzzy issue of responsible disclosure

Posted Aug 12, 2022 22:11 UTC (Fri) by developer122 (guest, #152928)
Parent article: A fuzzy issue of responsible disclosure

It's probably a good idea to rate-limit the public reporting of bugs of all kinds to something developers can manage, otherwise to provide them through a secure channel. Asking people who find bugs to also triage them is not practical, however.

to post comments

A fuzzy issue of responsible disclosure

Posted Aug 13, 2022 1:26 UTC (Sat) by willy (subscriber, #9762) [Link] (29 responses)

Isn't it though? You're saying people have the resources to run fuzzers or write fuzzing tools, but not to investigate the results?

A fuzzy issue of responsible disclosure

Posted Aug 13, 2022 9:08 UTC (Sat) by dottedmag (subscriber, #18590) [Link] (28 responses)

These are different skillsets.

A fuzzy issue of responsible disclosure

Posted Aug 13, 2022 13:12 UTC (Sat) by willy (subscriber, #9762) [Link] (26 responses)

Most of the people I see running fuzzers are students. If their skill sets are already that fossilised, they're not going to make it.

A fuzzy issue of responsible disclosure

Posted Aug 13, 2022 15:24 UTC (Sat) by mcatanzaro (subscriber, #93033) [Link] (22 responses)

Investigating and fixing bugs requires significant time and expertise over and beyond writing the fuzzer. A security bug report is an almost universally-appreciated contribution in any other project, and it's where security researchers are generally expected to stop. Going above and beyond to fix the bug is even better, of course, but if we expected researchers to do that for us, we would be much less secure because there would be far fewer bug reports, probably 50-100x fewer.

Besides: "Attackers have fuzzers too and know how to run them." Attackers are going to find these vulnerabilities regardless. Keeping the results hidden provides only minor benefit.

A fuzzy issue of responsible disclosure

Posted Aug 13, 2022 18:46 UTC (Sat) by willy (subscriber, #9762) [Link] (21 responses)

Yes, investigating and fixing bugs is a lot harder than finding them. That's work that apparently Linux developers are expected to put in, on whatever schedule the fuzzer runners decide.

And these things don't just affect filesystems. I saw a bug the other day reported against the memory allocator. It took some investigating to find that it was produced (a) by a fuzzer, (b) by creating a corrupt ReiserFS image.

At that point I stopped caring. It wasn't a bug in the MM. If anybody cares about ReiserFS, they can fix it.

A fuzzy issue of responsible disclosure

Posted Aug 13, 2022 23:07 UTC (Sat) by NYKevin (subscriber, #129325) [Link] (20 responses)

> Yes, investigating and fixing bugs is a lot harder than finding them. That's work that apparently Linux developers are expected to put in, on whatever schedule the fuzzer runners decide.

Nobody is expecting Linux developers to do anything. The bugs exist, fuzzer users are informing LKML of it, it's up to LKML to decide what to do next.

The bugs will exist regardless of whether they are found or not, and they will be found regardless of whether LKML wants them to be found or not. The sole thing that LKML and its developers can control is whether fuzzer users feel comfortable disclosing bugs (that already exist, and already have been found) to LKML directly, or if they instead just quietly post them to Metasploit or something (or worse, sell them to somebody like NSO).

> At that point I stopped caring. It wasn't a bug in the MM. If anybody cares about ReiserFS, they can fix it.

Nobody is claiming that this is an incorrect approach. However, users benefit from transparency. If a lot of fuzzer bugs can be traced back to ReiserFS, and are subsequently not fixed, then users ought to know that so that they can take appropriate security measures of their own initiative.

A fuzzy issue of responsible disclosure

Posted Aug 15, 2022 9:32 UTC (Mon) by jan.kara (subscriber, #59161) [Link] (19 responses)

> Nobody is expecting Linux developers to do anything. The bugs exist, fuzzer users are informing LKML of it, it's up to LKML to decide what to do next.
>
> The bugs will exist regardless of whether they are found or not, and they will be found regardless of whether LKML wants them to be found or not. The sole > thing that LKML and its developers can control is whether fuzzer users feel comfortable disclosing bugs (that already exist, and already have been found) to > LKML directly, or if they instead just quietly post them to Metasploit or something (or worse, sell them to somebody like NSO).

Well, here comes the point someone already made in this discussion: What is the motivation of the people running these fuzzers? If they just want to get some credit or need to report as many bugs as possible as part of their job duties, then there's not much we can do and it's upto their own consciousness to decide whether it's justifiable what they are doing. If their motivation is to help the project, then it is fair to ask them to put some more effort into trying to analyze the problems they've found because currently we have more fuzzer reports than resources to analyze and fix them.

For ext4, good example is the work of Huawei guys (and luckily they are not the only ones, they just came to my mind as one good example ;). They do run fuzzers a lot, find bugs, analyze them and even come up with suggested fixes. Initially it required quite some helping to make patches useful but they got better and by now they contribute a lot of fixes for problems found by fuzzing of ext4 images. So this is an example where it worked out well.

A fuzzy issue of responsible disclosure

Posted Aug 15, 2022 11:56 UTC (Mon) by mcatanzaro (subscriber, #93033) [Link]

This entire discussion feels a little ridiculous. Why are we questioning the motivations of people who are reporting security bugs? Money, fame, an integer value in an academic paper... who cares? A report from a fuzzer is a gold standard bug report. If it's NOT reported, attackers are going to find it anyway... often by using the exact same fuzzers!

The quantity of bugs reported by researchers running fuzzers is proportional to the quantity of bugs in the code. Want fewer bug reports? Write better code. (Easier said than done, I know.)

Meanwhile, in somewhere userspace: WebKit does not have time to address all the fuzzer reports we receive, and our users are less safe for it. But we certainly do not complain that they're reported.

A fuzzy issue of responsible disclosure

Posted Aug 15, 2022 21:16 UTC (Mon) by NYKevin (subscriber, #129325) [Link] (17 responses)

It depends on the person:

* Some people want to help the project, but don't understand kernel development well enough to read and understand the offending code.
* Some people have been told that fuzzing "helps the project find bugs" and are simply following the fuzzer's documentation.
* Some people want to prove that their fuzzer works.
* Some people are trying to earn bug bounties from tech companies.
* Some people are trying to get an academic paper published all about how effective fuzzers are, how bad C's memory safety is, etc.
* Some people want to sell zero days on the black market. (Such people aren't reporting bugs to LKML, of course.)

But ultimately, this is a side show. What matters is that people are motivated to find bugs. You can accept their reports, or not, but the bugs will be found regardless (and the latter group isn't going to tell you about them anyway). It's up to you what you want to do with that information.

A fuzzy issue of responsible disclosure

Posted Aug 16, 2022 11:11 UTC (Tue) by Wol (subscriber, #4433) [Link] (16 responses)

> But ultimately, this is a side show. What matters is that people are motivated to find bugs. You can accept their reports, or not, but the bugs will be found regardless (and the latter group isn't going to tell you about them anyway). It's up to you what you want to do with that information.

What SHOULD matter is that people are motivated to GET BUGS FIXED. (If people find bugs, and the reports just disappear "into the ether" as resource for black-hats, then the overall level of goodness has gone DOWN.)

And a firehose of bugs from a fuzzer is likely to get the same reception as lkml - almost completely ignored. (I'm not saying lkml doesn't serve a purpose - it's an archive and it's there to be searched - but in the main it's a write-only resource.)

At the end of the day, it's all about COMMUNICATION. Which is SUPPOSED to be two-way. If I - as a lone developer - get a firehose of bug-reports from a fuzzer, it's going to one place only - /dev/null.

Cheers,
Wol

A fuzzy issue of responsible disclosure

Posted Aug 16, 2022 15:06 UTC (Tue) by NYKevin (subscriber, #129325) [Link] (15 responses)

You can say that people "should" do something all you like. But in my experience, people do what they want, not what you want. People *are* motivated to find bugs, not necessarily to fix bugs, and to some extent, we're just stuck with that. We can change our own individual behavior, but we cannot tell others what to do (unless we, y'know, pay them or something).

A fuzzy issue of responsible disclosure

Posted Aug 16, 2022 19:08 UTC (Tue) by Wol (subscriber, #4433) [Link] (14 responses)

> We can change our own individual behavior, but we cannot tell others what to do (unless we, y'know, pay them or something).

Or we ask the police to have a quiet word with them for harassment ...

This is the problem here, if there is *respectful* *communication*, there is no problem. If either side "does what the hell they like", and it has ANY impact on the other side, then it's harassment at best, if not worse.

We should be applying the same norms of decent behaviour to online interactions as offline. The problem, of course, is that people have different norms, greatly magnified by different cultural standards, legal systems, etc etc. And if I want to ignore your norms, there's precious little you can do, even if my actions are illegal by your jurisdiction.

Cheers,
Wol

A fuzzy issue of responsible disclosure

Posted Aug 16, 2022 19:24 UTC (Tue) by mcatanzaro (subscriber, #93033) [Link] (13 responses)

Reporting a legitimate security vulnerability is not harassment. Come on, seriously?

Bug reports from fuzzers are gold standard because they always contain a reproducer and almost always contain output from asan. And almost all fuzzer reports are security vulnerabilities. If it's not remote code execution, then it's denial of service. Crashes in parsers are not benign.

A fuzzy issue of responsible disclosure

Posted Aug 16, 2022 20:33 UTC (Tue) by Wol (subscriber, #4433) [Link] (12 responses)

Yes seriously.

If I'm a lone developer, and I'm being flooded with reports I don't want, then it's harassment.

Let's assume it's typically 4 hours, say, to fix a bug. I'm probably being optimistic? And the fuzzer is filing 6 bug reports a day. Is that conservative for a fuzzer? And nagging me because I'm apparently not doing anything to fix them ...

Do the maths. That's harassment.

That's the problem. What LOOKS reasonable at first glance, is totally UNACCEPTABLE when you dig a bit deeper ...

Cheers,
Wol

A fuzzy issue of responsible disclosure

Posted Aug 16, 2022 20:48 UTC (Tue) by mcatanzaro (subscriber, #93033) [Link] (4 responses)

The comments on this story might themselves be worthy of an LWN story. Claiming that reporting *legitimate* security bugs is akin to harassment is pretty wild. I don't think anybody expects you to fix all those bugs, but the public deserves to know about them, and it's hard to think of a better place than the upstream bug tracker. I suppose there are plenty of other ways researchers can report vulnerabilities if you don't want your bug tracker used that purpose, though.

Ideally, they would receive CVEs so they can be tracked, but we all know only a tiny minority of security vulnerabilities actually receive CVEs.

It's unusual to receive the quantity of fuzzer reports that you are receiving. It indicates a very serious safety problem. Even web engines, which are full of vulnerabilities and fuzzed regularly by security researchers, do not deal with anywhere near that many fuzzer reports.

A fuzzy issue of responsible disclosure

Posted Aug 16, 2022 22:41 UTC (Tue) by Wol (subscriber, #4433) [Link] (1 responses)

I'm talking hypothetically. But running a fuzzer on a code base that hasn't been fuzzed before could probably generate a stream like that without any problem at all.

I once took over a program. I think it took me six months to clean up all the problems revealed just by upping the compiler warning level. And the comments earlier in the article imply that people are being hit by exactly this - projects being flooded with fuzzer reports.

> Claiming that reporting *legitimate* security bugs is akin to harassment is pretty wild.

Not when it's done with no regard whatsoever for the *people* running that project. If you're talking about faceless corporations then of course it APPEARS to be impersonal. But that only makes it worse for the real individuals on the receiving end.

What's that definition of hell? Responsibility without authority? If you are being held accountable for something you have no control over, then that's hell. That's harassment. And legitimate or not, flooding A PERSON with bug reports - serious or not - beyond their ability to cope is not acceptable.

I was fine - that 6-month cleanup was something I opted in to - I felt the company's development practices were extremely lackadaisical and it was my choice to fix it, but if I'd had it dumped on me and been pressured to get it fixed yesterday, then ...

Cheers,
Wol

A fuzzy issue of responsible disclosure

Posted Aug 16, 2022 23:34 UTC (Tue) by rahulsundaram (subscriber, #21946) [Link]

> Not when it's done with no regard whatsoever for the *people* running that project

It may be slightly problematic behavior but certainly not something you can call harassment. That's reaching way too far.

A fuzzy issue of responsible disclosure

Posted Aug 25, 2022 2:10 UTC (Thu) by milesrout (guest, #126894) [Link] (1 responses)

If Wol making awful uninformed comments was worthy of an LWN story, then LWN would not have time to write about anything else.

A fuzzy issue of responsible disclosure

Posted Aug 25, 2022 4:19 UTC (Thu) by Wol (subscriber, #4433) [Link]

:-)

Cheers,
Wol

A fuzzy issue of responsible disclosure

Posted Aug 16, 2022 20:48 UTC (Tue) by NYKevin (subscriber, #129325) [Link]

If you want the fuzzer users to stop filing reports, that's up to you. You can just tell them to stop filing reports, or block them from your mailing list if it comes to that, no need for the police to get involved. But at least some of them will still issue reports to the general public, likely with no embargo (as you have declined to accept their reports in private), so that end users can take reasonable security precautions. What effect this has on your software's adoption is, frankly, your problem. You can't label such disclosures as "harassment."

A fuzzy issue of responsible disclosure

Posted Aug 17, 2022 13:10 UTC (Wed) by klindsay (guest, #7459) [Link]

I can see how it would being annoying for the bug reporter to nag the developer because the developer is "not doing anything to fix them". However, the scenario has become "totally unacceptable" (your words, not mine) not because you have dug deeper, it has become "totally unacceptable" because you have changed the scenario.

A fuzzy issue of responsible disclosure

Posted Aug 17, 2022 13:56 UTC (Wed) by anselm (subscriber, #2796) [Link] (4 responses)

Let's assume it's typically 4 hours, say, to fix a bug. I'm probably being optimistic? And the fuzzer is filing 6 bug reports a day. Is that conservative for a fuzzer? And nagging me because I'm apparently not doing anything to fix them ...

What bothers me is that those 6 bug reports a day are probably not due to 6 different previously-unnoticed bugs in the code, every day. I can sympathise with developers who don't enjoy being inundated with a steady stream of raw unfiltered bug reports from some fuzzer that may or may not be symptoms of a possibly much smaller set of issues and consequently being forced to deal with what is effectively annoying noise in the project's bug tracker. If anything, having to close bug reports X, Y, and Z as duplicates of yesterday's bug report B is a hassle, especially if you need to do it every single day.

If you are dealing with somebody whose primary goal is to show how great their fuzzer is at filing bug reports, there's probably little to be done. But if you are dealing with somebody whose goal is to help you improve your project, it should be possible to come to some arrangement where they learn enough about your code to discard at least obviously-duplicate bug reports, or work with someone from your project to pre-triage bugs so only new ones are actually filed and not all developers are confronted with the raw fuzzer output.

A fuzzy issue of responsible disclosure

Posted Aug 17, 2022 18:32 UTC (Wed) by NYKevin (subscriber, #129325) [Link] (2 responses)

It should probably also be emphasized that this is a Four Freedoms issue. Fuzzing a program is expressly protected under freedoms 0 and 1 (i.e. "run the program as you wish, for any purpose" and "study how the program works"). Communicating the results of a fuzzer run is *technically* not within the literal wording of freedom 3 ("distribute copies of your modified versions to others") but it's obviously within the spirit of the freedom.

In other words: If you don't want people to fuzz your software, then you should not make free software in the first place. You don't have to read their bug reports, and you can nicely ask them to pre-triage or to take other reasonable steps, but ultimately, the user has an absolute right to fuzz the software and tell anyone who will listen about the bugs they find.

A fuzzy issue of responsible disclosure

Posted Aug 17, 2022 19:41 UTC (Wed) by pebolle (guest, #35204) [Link]

> the user has an absolute right to fuzz the software and tell anyone who will listen about the bugs they find.

Exactly!

Why does this even needs to be stated? It wouldn't be Free Software if we're not allowed to use it for whatever reason we fancy. Like noticing it's prone to certain crashes.

I seem to remember the OpenBSD developers rejecting the notion of responsible disclosure. If I remember correctly, my sympathy for their position just increased a bit.

A fuzzy issue of responsible disclosure

Posted Aug 25, 2022 2:16 UTC (Thu) by milesrout (guest, #126894) [Link]

You've made a classic error: confusing the question of what people are *legally entitled* to do with free software and the question of what is appropriate and acceptable conduct in the free software community. Nobody makes this mistakes any more with forks: the ability to fork is one of the four freedoms *explicitly*, but it is also regarded by most as hostile---a last resort, when friendly communication has broken down. Why, then, knowing this, do people continue to confuse these two completely different things?

Nobody is saying anyone is *legally prohibited* from fuzzing free software. The discussion is not even about fuzzing, it is about *communication* of the *results* of fuzzing, and how it can be done in a way that does not cause burnout and frustration from developers, while recognising that fuzzers are reporting bugs, which is something that, at least in the abstract, ought to be encouraged.

A fuzzy issue of responsible disclosure

Posted Aug 17, 2022 20:38 UTC (Wed) by fenncruz (subscriber, #81417) [Link]

Sounds like there is an opening for a fuzzing tool that uploads data somewhere, which then de-duppicates the fuzz results. That way if people keep finding the same issue (or issues within some 'distance') then there only needs to be one bug report.

I know abrt does this for normal crashes, so it seems doable if the fuzzers can generate a backtrace.

A fuzzy issue of responsible disclosure

Posted Aug 13, 2022 16:35 UTC (Sat) by developer122 (guest, #152928) [Link] (2 responses)

Student's can't necessarily be expected to *already* posses the skillset required to debug a linux kernel bug.

Nobody is going to go out and learn the everything required to debug the kernel just to appease someone on a mailing list.

A fuzzy issue of responsible disclosure

Posted Aug 13, 2022 18:56 UTC (Sat) by willy (subscriber, #9762) [Link]

Me: This isn't working
You: It should work

Repeat ad nauseam. I think I'm done here.

A fuzzy issue of responsible disclosure

Posted Aug 15, 2022 11:16 UTC (Mon) by kleptog (subscriber, #1183) [Link]

> Student's can't necessarily be expected to *already* posses the skillset required to debug a linux kernel bug.

On the other hand, reading other people's code is a very important skill and while you're a student you're likely to have more time available to practice than you probably will have at any other time in your life.

Sure, there are some very complicated bits of code out there, but there is quite a lot that is reasonably accessible if you're willing to spend a day or two pouring over the code and trying to understand it. Sure, the first time may be frustrating, but you do get better at it.

A fuzzy issue of responsible disclosure

Posted Aug 13, 2022 13:18 UTC (Sat) by khim (subscriber, #9252) [Link]

To show you one example which really impressed me. When Dmitry Vykov just started fuzzing PCI-Express drivers someone had the bright idea to create Thunderbolt contraption with FPGA which was teached to apply “bad sequences” (found by fuzzers) to the live Thunderbolt port. And when they found few dozens of such sequences they tested them with Linux laptop and lo and behold, it successfully crashed it.

That was not surprising. The surprise came when Windows laptop was tested. Most “bad sequences” were ignored (bugs in the independently written code tend to be different), but some of them crashed Windows, too.

Now, think about it: how much chance would they have WRT successful “investigation of results” on Windows?

It's one thing to find out that “this or that violation of specs leads to buffer overrun”. Completely different skillset is to invent the “proper way” to fix these.

A fuzzy issue of responsible disclosure

Posted Aug 13, 2022 10:16 UTC (Sat) by edeloget (subscriber, #88392) [Link] (3 responses)

Unfortunately, rate-limiting the discovery of bugs is not going to fly well. This will only make the fuzzer frustrated and the upstream community that does that is likely to experience the birth of an outside fuzzer community that will wontinue to disclose bugs as they find them but will not, in any case, warn the maintainers of the discovery. To understand why I think that, you have to understand that fuzzers have incentives to report the largest number of bugs (for numerous reasons : to prove their products works, because their research project got a federal grant and they have to show results...).

And that's probably something nobody wants :)

There is no good solution to the issue outlined in this article. Security researcher ok'ed the principle of responsible disclosure because they typically don't find 10 bugs per week -- it's not their role and it's not how they work. They also have to prove that a bug is exploitable, otherwise their research is of less use. But fuzzers? They just have to prove that the bug exists. The "exploit" is written by a program and the testing is done by a program. The whole idea of fuzzing is based upon automation in order to quickly find new bugs. Asking for them to go the road of responsible disclosure is a bit contrary to the philosophy of fuzzing.

A fuzzy issue of responsible disclosure

Posted Aug 13, 2022 16:41 UTC (Sat) by developer122 (guest, #152928) [Link] (1 responses)

It's not actually that complicated. Make a list. Append bugs to the end of that list. Retrieve bugs from the beginning of the list as needed. There is not an infinite number of fuzzer-detectable bugs in any given software.

If you want, you could also provide all the bugs in a non-public forum (as mentioned above). You only need to apply the above rate-limiting when making them public.

This is not a new concept. The entire reason behind the "90 day grace period" in "industry standard responsible disclosure" (which everyone should have encountered at least once by now even if it's not universally implemented) is a recognition that it takes time and effort for a vendor to resolve a bug. It's effectively the same rate-limiting, with a predefined time window set based on an assumed rate of bugs.

A fuzzy issue of responsible disclosure

Posted Aug 17, 2022 9:22 UTC (Wed) by taladar (subscriber, #68407) [Link]

> There is not an infinite number of fuzzer-detectable bugs in any given software.

Maybe not, but there is a for all practical purposes infinite number of ways to reproduce even simple bugs. If the fuzzer reports are low effort enough and just report every reproduction instead of deduplicating the ones that cause issues in the same line with the same branches taken you might get lots and lots of effort-creating reports for every bug.

Lets say a bug is that an integer is used as a boolean and it crashes if that integer is neither 0 nor 1. Do you really want one bug report for 2,3,4,5,... each or do you want one that tells you it crashes for all values other than 0 or 1?

A fuzzy issue of responsible disclosure

Posted Aug 14, 2022 2:32 UTC (Sun) by pabs (subscriber, #43278) [Link]

Seems like the best way forward is to have the fuzzer community integrate their systems with the CI of upstream projects, since that is where all automated project testing/management is usually done.

A fuzzy issue of responsible disclosure

Posted Aug 13, 2022 18:18 UTC (Sat) by sfeam (subscriber, #2841) [Link]

My experience as a developer receiving a slew of fuzz-generated "crash-causing input" reports is that a huge fraction of the submitted cases tend to hit the same small number of actual bugs in the code. Even if 100% of the submitted bad inputs yield reproducible crashes of the unpatched code, finding and fixing the cause of a single representative submission may clear anywhere up to all of the remaining submission from the reproducible category when run on the now-patched code.

So a larger number of fuzz-generated bug reports does not necessarily translate to a proportionally larger amount of work. I think it is unreasonable to expect a reporter to identify in advance how much redundancy is in their set of fuzz bugs. That would amount to a stricture "only report bugs for which you have already found a fix".

A fuzzy issue of responsible disclosure

Posted Aug 17, 2022 12:34 UTC (Wed) by IanKelling (subscriber, #89418) [Link]

Interesting idea, and it reminds me that the argument about whether to post publicly seems to have ignored the fact that the kernel devs have control over whether posts to their mailing list become public. I'd expect there to be a general rate limit already, and it certainly could be made so that messages which look like fuzzing bug reports get diverted to a private area with a lower rate limit, at least until they are reviewed by a human.