Vetting the cargo [LWN.net]

Vetting the cargo

Posted Jun 10, 2022 19:21 UTC (Fri) by josh (subscriber, #17465)
In reply to: Vetting the cargo by edeloget
Parent article: Vetting the cargo

No, we expect the people submitting vetting audits to be good actors, who are reviewing the work of others, not their own. Audits would be submitted by developers of the project adding a dependency, not by the developers of the dependency; this just records whatever review work took place, by whatever process the project has in place.

to post comments

Vetting the cargo

Posted Jun 11, 2022 9:27 UTC (Sat) by edeloget (subscriber, #88392) [Link] (11 responses)

> No, we expect the people submitting vetting audits to be good actors, who are reviewing the work of others, not their own. Audits would be submitted by developers of the project adding a dependency, not by the developers of the dependency; this just records whatever review work took place, by whatever process the project has in place.

I understand that, but it's still not a good solution. Let me introduce all-padding, a crate that depends on both right-padding (developped by an associate of mine, yet you don't know that) and left-padding. Both dependencies are innocuous at the time you need them, so I audited them and told the world they are safe to deploy. Since padding function are "difficult" and your time is better spent implementing the REST API of the coffee machine, you add all-padding as a dependency. You audit my code, and it's ok, so all-padding is audited as safe to deploy as well.

A few update cycles down the road, eveybody forget about all-padding - it's a safe library. Good time for me and my associate to maliciously update right-padding. Since the dependency changes quite a bit, I "re-audit" it and mark is as safe to deploy again. And since you've worked with all-padding for ages, you trust me, so you don't reevaluate right-padding by yourself : I just told you it was ok.

I also understand that I'm kind of a pessimist :)

The main problem here is not even that some packages might be malicious. It's the fact that you might have hundred of dependencies in your project. This is not a rust-related problem (nodejs is a worse offender here, but python, php and a lot of other languages are also at risk) but such a large number of dependencies will always mean that you cannot develop reasonably secure programs with them - no amount of public audit will ever help. The only thing that will come from these audits is a very weak proof, something along the line of "I think that maybe there is a chance that this lib is not actively malicious", which is not very interesting in the long term (not to mention I'm pretty sure that most of them will be of the "I need this lib, I'd like my project to be accepted/pass our CI tests, so let's mark all my dependencies as fully audited").

To this main problem, "cargo vet" adds another layer: a false sense of security. You may end up thinking your project is safe, because all the dependencies you added have been "audited" by people that, you may assume, are better than you at finding security issues. Yet security is hard. Even obvious issues are not that easy to spot (think of the goto fail problem in the Apple SSL library: it tooks the whole world two years to find this very obvious issue in a widely used, public SSL library) so having a random individual tell you that the dependencies he uses are ok is kind of... well, it's kind of alarming.

Vetting the cargo

Posted Jun 11, 2022 15:35 UTC (Sat) by josh (subscriber, #17465) [Link] (5 responses)

> I understand that, but it's still not a good solution. Let me introduce all-padding, a crate that depends on both right-padding (developped by an associate of mine, yet you don't know that) and left-padding. Both dependencies are innocuous at the time you need them, so I audited them and told the world they are safe to deploy. Since padding function are "difficult" and your time is better spent implementing the REST API of the coffee machine, you add all-padding as a dependency. You audit my code, and it's ok, so all-padding is audited as safe to deploy as well.

And then cargo-vet tells you that you also need to audit left-padding and right-padding, because it intentionally does not have a transitive trust model.

> And since you've worked with all-padding for ages, you trust me, so you don't reevaluate right-padding by yourself : I just told you it was ok.

And that was a process mistake that no tool can prevent you from making.

While cargo-vet allows you to import audits from other projects, I have the impression that's intended for use cases like "I'm a small developer and don't want to vet all my dependencies, so I'm going to import the audits of a large project that has security reviewers I trust, such as Mozilla". Which will leave you better off than the audits you might otherwise have done yourself. If, instead of delegating your trust to developers of a large well-known project, you delegate your trust to the author of all-padding, you will get a corresponding level of non-diligence.

Vetting the cargo

Posted Jun 11, 2022 16:48 UTC (Sat) by edeloget (subscriber, #88392) [Link] (4 responses)

> And then cargo-vet tells you that you also need to audit left-padding and right-padding, because it intentionally does not have a transitive trust model.

In real work terms, it means that you'll just import the audits done by someone else. This is why the import function exists in the first place. It's not a transitive trust model, it's a delegate trust model. In the end, there is no difference as the author of all-padding will tell you that his project and its dependencies are trusted by project foo, bar and baz and will points you to the audit tile in order to ease the integration of his own module in your project. And most people will do it, because it's easier this way. Or the author will find a way to add himself in a global, trusted repository. Or he will tell you to trust another registry.

Developpers tend to pick easy solutions (otherwise the npm development model would not even exist), and as a module author, giving them clear, easy instructions to follow in order to avoid being turned down by this pesky CI tests is going to be valued.

You might argue that this is dumb and it should not exist, but, hey, large security appliances are often presenting stupid security holes (like this one: https://twitter.com/AnnaViolet20/status/15235646321405091... ; take some time to think about how is it even possible for a global leader in security to spawn a REST API which runs as root and does not even check for any credential apart from the presence of a specific header which is widely documented. The answer is along the line of "well... Oh! look at this wonderful sky!").

> And that was a process mistake that no tool can prevent you from making.

I would argue that it's definitevely worse, as I now have a tool that tells me it's definitely ok to do that.

> While cargo-vet allows you to import audits from other projects, I have the impression that's intended for use cases like "I'm a small developer and don't want to vet all my dependencies, so I'm going to import the audits of a large project that has security reviewers I trust, such as Mozilla". Which will leave you better off than the audits you might otherwise have done yourself.

I would not assume that this feature will only be used by small developpers. Large developpers shops are also likely to use it in order to make sure they hit their delivery time - because now that the tool exist, support for the tool will be added in CI systems, and suddenly, as a developper, you'll have to accept audits results.

> If, instead of delegating your trust to developers of a large well-known project, you delegate your trust to the author of all-padding, you will get a corresponding level of non-diligence.

Well, this is one of the reason for the development of cargo vet: you cannot assume the users (even large companies) will audit the code because this is not practical. So the level of non-diligence is already here. I would even say that refusing the audit would lead you directly to square zero.

I would also add that there is no actual reason to not trust the author of all-padding. On what ground should I do that? (and I'm OK that in real life, I will have a harder time to trust him than to trust Mozilla, so that's more of a devil's attorney question).

Don't misunderstand my position: I understand that having a tool to help code audit is a good thing. But given the choosen development model (as the developpers of cargo-vet calls it: "low-friction reuse of third-party components") this is just bandage on a wooden leg. The problem is that the development model in itself is dangerous. This has been proven times and times again. While it might reduce risks (and I'm not even sure of that) no amount of tooling might change that.

Vetting the cargo

Posted Jun 11, 2022 19:32 UTC (Sat) by josh (subscriber, #17465) [Link] (1 responses)

> In the end, there is no difference as the author of all-padding will tell you that his project and its dependencies are trusted by project foo, bar and baz and will points you to the audit tile in order to ease the integration of his own module in your project.

And that audit will cover only the versions that were audited, so if you trust that organization to do good audits, you'll be safer than you otherwise would be. If you make the mistake of trusting that because that organization has audited an *old* version the *new* version must be safe, that's your mistake, not that organization's mistake.

This is an improvement over the current state, where it's much easier to do malicious things and not get caught.

> I would argue that it's definitevely worse, as I now have a tool that tells me it's definitely ok to do that.

No, you have a tool that's making it possible to be selective about who you trust and tracking exactly what versions you trust, as opposed to implicitly trusting everyone. If you then choose to trust the wrong people, a tool can't help you with that.

> I would also add that there is no actual reason to not trust the author of all-padding. On what ground should I do that?

There's no reason *to* trust the author of all-padding to vet other software. You are repeatedly introducing the assumption that because a previous version appears safe the developer of that version should be trusted to audit other things, and then saying that if you do that, someone who earns trust with their previous code can break that trust. cargo-vet is designed to *not* make that assumption.

Potentially good criteria for delegating your audits to someone else would include "they're a part of the same company as you and this is their job", or "they're part of a trusted industry organization". Bad criteria for delegating your audits include "they wrote an apparently decent library once".

> But given the choosen development model (as the developpers of cargo-vet calls it: "low-friction reuse of third-party components") this is just bandage on a wooden leg. The problem is that the development model in itself is dangerous.

There are other solutions in the works as well, such as WebAssembly components that allow using a dependency without giving full trust to that dependency.

But in the meantime, this is an improvement over the status quo. And using third-party dependencies rather than reimplementing them is a net improvement.

Vetting the cargo

Posted Jun 12, 2022 3:01 UTC (Sun) by ms-tg (subscriber, #89231) [Link]

Thank you for your patience in expressing the reasoning here.

Vetting the cargo

Posted Jun 12, 2022 9:50 UTC (Sun) by ilammy (subscriber, #145312) [Link]

> I would also add that there is no actual reason to not trust the author of all-padding. On what ground should I do that?

This is the crux of the issue, I believe.

There is no reason to distrust a library when its maintainers are well-behaved: don’t introduce vulnerabilities themselves, don’t pull in libraries with known vulnerabilities, and depend on the libraries you trust -- note: “you”, *not* other people.

But that’s the thing: trust is good until it’s not, then it gets lost. You want to be notified of something worthy of your distrust early. When you see a CVE reported for some library, you immediately distrust that particular version. If that was a malicious change, you might want to distrust whoever did that change and whoever let it get in.

Now, how do you get CVEs reported in the first place? That’s where audits come into the picture. Someone supposedly reviews the code then either vouches that it’s good or reports these CVEs. You want to have all code from your dependencies reviews by someone -- either yourself, or someone you trust.

Tools like this formalize this “thousand eyeballs” expectations that come with depending on libraries. When you pull in dependency, you trust that someone has looked at the code and did not see any obvious malicious parts in it. Without these tools your trust in that is placed in absence of reported CVEs -- which is a bad metric: that’s akin to saying that bugs don’t exist until there is a ticket. With tools like this you’d know exactly who vetted which code and when, then you are free to establish whatever policy you want on top of that.

Naturally, the risk of getting malicious code in your build tree is still there. But there is only one way to *remove* it: never depend on any code that you did not write yourself (and even then, you can introduce vulnerabilities). All other ways are merely reducing the risk to an acceptable level.

Vetting the cargo

Posted Jun 20, 2022 7:12 UTC (Mon) by marcH (subscriber, #57642) [Link]

> Don't misunderstand my position: I understand that having a tool to help code audit is a good thing. But given the choosen development model (as the developpers of cargo-vet calls it: "low-friction reuse of third-party components") this is just bandage on a wooden leg.

In which other development model(s) do you understand that having a tool to help code audit is a good thing?

How incompatible with these other development models is cargo-vet?

Vetting the cargo

Posted Jun 13, 2022 8:45 UTC (Mon) by taladar (subscriber, #68407) [Link] (4 responses)

> but such a large number of dependencies will always mean that you cannot develop reasonably secure programs with them - no amount of public audit will ever help.

As if a small number of extremely large dependencies (think Qt or Boost) would be better here. If anything the large dependencies make it much easier to overlook a suspicious change.

Vetting the cargo

Posted Jun 13, 2022 11:51 UTC (Mon) by hkario (subscriber, #94864) [Link] (3 responses)

large dependencies also usually have large number of developers, and those developers are concentrated on that one project, causing much more eyes looking at the code every day

Vetting the cargo

Posted Jun 13, 2022 12:33 UTC (Mon) by farnz (subscriber, #17727) [Link]

Only if most developers range over the entire project; in my employer's codebase, there is a foundational library that is part of a big dependency where I know for a fact that there is only one person in the entire company who looks at the code regularly, and gets their code reviewed by a selection of people who trust them to get it right. If they were malicious, they have a very good chance of slipping bad code in without being caught despite code review.

At least with small dependencies, the effort to rewrite if upstream is malicious is low.

Vetting the cargo

Posted Jun 13, 2022 13:09 UTC (Mon) by colejohnson66 (subscriber, #134046) [Link] (1 responses)

The "many eyes" theory is complete bogus. OpenSSL had Heartbleed in it for literal years and no one noticed.

Vetting the cargo

Posted Jun 13, 2022 13:22 UTC (Mon) by Wol (subscriber, #4433) [Link]

"Many eyes makes bugs shallow".

Which is true. It's extremely easy to look straight through something you're not expecting to see, many eyes on their own are useless. Note also the "makes bugs shallow". It doesn't say it makes them easy to notice. It *does* mean that once you know there is a bug, it's not going to be able to hide for long.

Which has been borne out many times. What is the expected life-time of a bug once it's been spotted? Hours? Certainly not much more.

(But then, of course, in addition to the time at the start where no-one knew there was a bug, you also have the long tail where the bug has been fixed, but the fix has not been deployed.)

Cheers,
Wol