LWN: Comments on "The malicious "rustdecimal" crate"

The malicious "rustdecimal" crate

nybble41 — Sun, 12 Jun 2022 21:38:05 +0000

> In general, "cmdA | cmdB" is not safe without "set -o pipefail".

In the case of `curl | bash` the command isn't safe even *with* `set -o pipefail`. The problem isn't the exit status of the pipeline (which would typically be ignored in any case) but rather the fact that it runs the script *as it's being downloaded*, without ensuring that the entire script is available before the first command is executed, and even worse that the shell will attempt to execute the last line downloaded even it was truncated.

The malicious "rustdecimal" crate

marcH — Sun, 12 Jun 2022 06:54:13 +0000

In general, "cmdA | cmdB" is not safe without "set -o pipefail".

Safety discussions are a bit ridiculous the moment you use a language that ignores errors by default. We love high level discussions about memory safety and borrow checkers but in the real world the problems often start much more mundane: many C programs discard errors and even when they don't the error handling code has never been tested.

The malicious "rustdecimal" crate

marcH — Sun, 12 Jun 2022 06:45:12 +0000

> the reason it hasn't is that upstreams are generally not malicious.

"Generally" is good enough except for security or safety issues. My smartphone "generally" does not crash: good enough. My bank account is "generally" safe from hackers: not good enough.

But of course security bugs are also "just bugs". So it's funny to see some $BIGCORPS suddenly very obsessed about preventing security bugs and wondering "how did that happen?" when the answer is often just:
because crashing from time to time was considered good enough... until hackers look at our products.

The malicious "rustdecimal" crate

dilinger — Fri, 20 May 2022 02:21:41 +0000

And there's the fact that such a malicious bug would only affect a certain subset of debian users (those running unstable) rather than stable users. An upstream could plant something debian-specific that waited several years before triggering so that it happened on debian stable systems, but that's quite the long game. Or stable could be targeted by a fast-changing package in stable, like firefox/chromium.

The malicious "rustdecimal" crate

mjg59 — Sun, 15 May 2022 03:53:05 +0000

> without looking at what was done since in order to make sure this issue doesn't happen anymore

Nothing was done. There's no process in Debian to prevent something like this happening again - the reason it hasn't is that upstreams are generally not malicious.

The malicious "rustdecimal" crate

wtarreau — Sun, 15 May 2022 03:46:55 +0000

"nitpicking" :-)

Selecting an issue that happened more than 10 years before the ecosystem subject of this article was even imagined is hardly relevant to this discussion, I'm sorry. It's presenting a process that's supposed to be better than another one without looking at what was done since in order to make sure this issue doesn't happen anymore. Yet the current situation is presented as having that limitation while it may very well be what made this issue not happen anymore. That's precisely a swap of cause and consequence.

The malicious "rustdecimal" crate

khim — Sat, 14 May 2022 12:30:28 +0000

Not even Apple is arrogant enough to ask for the sources, only Linux distro guys ever tried to enforce that (and even they no longer try to do that).

Of course only binaries are sent.

The malicious "rustdecimal" crate

flussence — Sat, 14 May 2022 05:49:29 +0000

A webpage hit counter on its own can only ever tell you it probably wasn't malicious for a high percentile of those downloads - hopefully 100, but that's not a certainty. And notably, it doesn't do anything for the OpenOffice problem where something gets to an impressive-sounding number then becomes tautologically self-sustaining based on that alone.

Here's a pretty extreme counterexample: https://pecl.php.net/package-stats.php
Everyone knows what imagemagick and sdl_image are, they have an overlapping set of use cases, and from a quick glance they both appear perfectly legitimate packages by legitimate people. The download numbers though, if used as evidence in this way, tell a wildly different (and incorrect) story.

Statistics are a footgun.

The malicious "rustdecimal" crate

patrakov — Sat, 14 May 2022 05:36:17 +0000

I am not familiar with iOS development. Do authors send source code to Apple, or only the compiled binaries?

The malicious "rustdecimal" crate

pabs — Sat, 14 May 2022 04:11:27 +0000

Distros do very little review, so them taking responsibility for something doesn't really help the code quality/trust problem, you still need review even with a distro.

The malicious "rustdecimal" crate

jwilk — Fri, 13 May 2022 20:24:51 +0000

https://bugs.debian.org/842939

The malicious "rustdecimal" crate

Chousuke — Fri, 13 May 2022 19:53:56 +0000

What distros have that upstream authors don't are enforced policies and best practices for packaging software. If you get a package signed with a distribution's packaging key, you know there's at least some level of quality guaranteed (depending on distribution).

In Fedora, any packaged software is required to build offline (without any network access), and *only* from the sources provided; that means you at least have some hope of knowing what sources the binary was built from and rebuilding the software yourself.

As an example of the frankly incompetent crap vendors sometimes do that wouldn't fly in a serious distro, AWS's package for OpenDistro for Elasticsearch at one point used a pre-install script to download a binary from the internet upon installation and dump it in /usr/lib. This was done in a noarch package that's supposed to be architecture-independent! I remember it well because I had to spend time untangling a surprise broken upgrade in an ES cluster that had no internet access...

The malicious "rustdecimal" crate

zdzichu — Fri, 13 May 2022 13:50:05 +0000

The question was for ANY example of malicious code going into the distro. The example was provided. Nitpicking about the date is just moving the goalposts.

The malicious "rustdecimal" crate

mathstuf — Fri, 13 May 2022 13:16:26 +0000

> You can filter rarely-used-untested crates, but not the malicious ones.

There is `cargo-crev` which is aiming to help this problem. https://github.com/crev-dev/cargo-crev

The malicious "rustdecimal" crate

mathstuf — Fri, 13 May 2022 13:15:21 +0000

> one in five bugs are planted with malicious intent

This doesn't pass my smell test. The article says "vulnerabilities" without defining what they mean. I presume they actually mean "CVEs on GitHub-hosted projects" here, but it's really hard to tell. There are plenty of vulnerabilities that never get a CVE. There are also plenty not on GitHub. So without some metric of what they're talking about, I have no idea how useful such a statistic is. But "bugs" is almost certainly not what you mean here (at least as I understand the term).

The malicious "rustdecimal" crate

excors — Fri, 13 May 2022 11:43:58 +0000

I think they got the right numbers but the wrong context. The original report (https://octoverse.github.com/static/github-octoverse-2020...) says:

> Most software vulnerabilities are mistakes, not malicious attacks. Analysis on a random sample of 521 advisories from across our six ecosystems [Composer, Maven, npm, NuGet, PyPI, RubyGems] found that 17% of the advisories were related to explicitly malicious behavior such as backdoor attempts. These malicious vulnerabilities were generally in seldom-used packages, but triggered just 0.2% of alerts. While malicious attacks are more likely to get attention in security circles, most vulnerabilities are caused by mistakes.

and:

> Analysis on a random sample of 521 advisories from across our six ecosystems finds that 17% of the advisories are related to explicitly malicious behavior such as backdoor attempts. Of those 17%, the vast majority come from the npm ecosystem. While 17% of malicious attacks will steal the spotlight in security circles, vulnerabilities introduced by mistake can be just as disruptive and are much more likely to impact popular projects. Out of all the alerts GitHub sent developers notifying them of vulnerabilities in their dependencies, only 0.2% were related to explicitly malicious activity. That is, most vulnerabilities were simply those caused by mistakes.

To be honest, I can't actually understand what that's trying to say. But I don't believe it's saying that 17% (nor 0.2%) of vulnerabilities were maliciously inserted: it's about advisories not vulnerabilities (and most projects won't bother with advisories for regular boring vulnerabilities found through code review), and "advisories related to explicitly malicious behaviour/activity" sounds like it's talking about active exploitation of a possibly-accidental vulnerability (vs an unexploited vulnerability found through code review), probably? Without more detail on the methodology (which I haven't been able to find) the numbers seem largely meaningless.

The malicious "rustdecimal" crate

mkj — Fri, 13 May 2022 11:03:05 +0000

That zdnet article is off by a factor of 100. It has mistaken 0.2% for 20%.

The malicious "rustdecimal" crate

wtarreau — Fri, 13 May 2022 09:31:35 +0000

> A more immediate problem with `curl | bash` is ensuring that the script is safe with respect to truncation; there is no buffering, so if the connection is interrupted `curl` just reports EOF and `bash` will attempt to run whatever was received for the interrupted line as if it were a complete command.

Exactly! I'm pleased to read this because I feel like I'm constantly blowing in the wind when I say this, and even people do not believe my examples :-/

The malicious "rustdecimal" crate

wtarreau — Fri, 13 May 2022 09:29:14 +0000

Review is not so much important as having a 3rd party accept to take some responsibility for it (which is what happens in a distro). That does help, obviously, but asking someone you don't know to accept to maintain your crap over the long term is not an easy task, and doing that with purposely malicious software is even harder.

The malicious "rustdecimal" crate

wtarreau — Fri, 13 May 2022 09:27:15 +0000

> This a nearly 20 year-old example, and it was done by the upstream maintainer.

In fact it's the perfect counter-example: all that could be found dates 20 years because the distro model works very well!

The malicious "rustdecimal" crate

wtarreau — Fri, 13 May 2022 09:26:10 +0000

Distros have good virtues for packaging, they impose some deadlines and some stability. That prevents one from appearing in the middle of nowhere and becoming the new standard thing of the month then disappearing and leaving everyone in the dust. You also have to convince a package maintainer to adopt and maintain your package, so that usually comes with some compelling arguments about popularity and proofs of understanding of cycles of development and what long-term maintenance means.

I'd say that software that are distributed a bit too directly to users tend to evade such controls and by losing some of these constraints they also lose many of the guarantees they used to offer to their users. I do have absolutely zero trust in such software distributed on such channels that are nothing but a market place where fame replaces money, and where engagement can be zero without anyone noticing.

The malicious "rustdecimal" crate

pabs — Thu, 12 May 2022 23:40:23 +0000

The review problem is potentially solvable through distributed review and a web of "review trust". The CREV folks are working on something like that.

https://github.com/crev-dev/

The malicious "rustdecimal" crate

khim — Thu, 12 May 2022 22:57:20 +0000

iOS does have malware (although much smaller amount than Android) thus it's clear than labor intensive review of all changes doesn't work.

Debian approach works more-or-less fine if you accept the fact that majority of software wouldn't delivered through that mechanism. This leaves you with feel-good impression without solving any real problems.

This problem is not, actually, solveable: you are either popular (and then have some malware) and are not (and then people are using other things and receive their fix of malware from other sources).

The malicious "rustdecimal" crate

nybble41 — Thu, 12 May 2022 22:01:24 +0000

> There are instance where domain name renewal was allowed to lapse and the new owner replaced the software by a trojaned one.

In terms of installation of software from new sources (as opposed to updates to existing software or centralized repositories managed by distros) the validation instructions, e.g. a GPG public key or signature file hosted on the same domain, are just as vulnerable to this sort of attack as the script itself. If you don't want to be reliant on control of the domain name you need some out-of-band system to validate your sources, such as a PGP Web of Trust, but these systems have not really caught on as a practical general solution thus far. Identifying the source by its public key, e.g. using IPNS, or Tor .onion domains, would be one alternative; then the root of trust is pushed back to whoever directed you to that source.

A more immediate problem with `curl | bash` is ensuring that the script is safe with respect to truncation; there is no buffering, so if the connection is interrupted `curl` just reports EOF and `bash` will attempt to run whatever was received for the interrupted line as if it were a complete command.

The malicious "rustdecimal" crate

cesarb — Thu, 12 May 2022 17:16:04 +0000

> It's also possible it was a targetted attack. i.e. someone uploaded the malicious crate, then sent a patch or pull request to a specific victim rather than waiting for people to innocently stumble on it.

If that's the case, perhaps looking for all merge requests at projects which are on gitlab, depend on the original crate, and where the merge request changed that dependency, might find the victim (if it's a public project).

Depending on how the CI is configured, it might automatically run on all merge requests, and it might expose important tokens (for instance, tokens to publish a finished and signed release) to the CI process. So this might be just the first step, and later we might be seeing a legitimate project publishing a malicious release due to this compromise.

The malicious "rustdecimal" crate

ballombe — Thu, 12 May 2022 17:09:02 +0000

There are instance where domain name renewal was allowed to lapse and the new owner replaced the software by a trojaned one.
HTTPS and domain name ownership are insufficient.

The malicious "rustdecimal" crate

ballombe — Thu, 12 May 2022 16:49:49 +0000

This a nearly 20 year-old example, and it was done by the upstream maintainer.
If you have nothing better, that rather prove that Linux distributions are quite safe
against malicious code injection.

The malicious "rustdecimal" crate

zdzichu — Thu, 12 May 2022 14:50:06 +0000

mjg59 provided a link recently: https://lists.debian.org/debian-devel/2003/02/msg00771.html

Upstream included code targeting Debian specifically. Maintainer did not spot this. (I don't blame him. I wouldn't spot it either. Life is too short to read diffs between versions).

The malicious "rustdecimal" crate

tchernobog — Thu, 12 May 2022 14:13:34 +0000

Well, log4j has been packaged and distributed for quite a while despite allowing remote execution, has it not? Not to mention a long list of CVEs in packages more (e.g. openssl) or less critical.

Humans are imperfect, no human review process is 100% safe.

A survey conducted through GitHub claims that about one in five bugs are planted with malicious intent.

https://www.zdnet.com/article/open-source-software-how-ma...

Considering the thousands of CVEs that have been open against a distro like Debian over the years, there is a good and fair chance that several bugs we have seen (and many we haven't yet) were indeed intentionally planted.

The malicious "rustdecimal" crate

sionescu — Thu, 12 May 2022 13:26:09 +0000

> Unfortunately, history shows that malicious code has landed and will continue to land also in mainstream distributions.

Do you have examples ?

The malicious "rustdecimal" crate

amarao — Thu, 12 May 2022 13:03:40 +0000

> Next you'll be telling me that anyone can register a domain name and offer software for download, with no package management at all.

You miss the point of my argument. I'm saying that imitating some reasonable number of downloads for crate is simple. It's the simplest part of the trickery. That means, you can't use it to meaningfully defend yourself from malicious crates by looking on download counter (and all cousins, like number of forks and stars on GH). That doesn't mean you can't use 'little downloads' as a red flag, but you can't use it as a qualifier for 'safe to use'.

The malicious "rustdecimal" crate

wtarreau — Thu, 12 May 2022 13:01:14 +0000

Ah ah, the trusted source is almost always https://github.com/something and it always comes with a valid certificate without having to make any effort :-)

The malicious "rustdecimal" crate

amarao — Thu, 12 May 2022 13:00:12 +0000

> crates.io prominently shows not just the total download count, but a chart of downloads per version over the last 90 days.

Okay. I need to a broken crate with misleading description, put a systemd timer with a curl with some randomness in it ... 15 minutes of playing with execution times based on chart from other well-known crate?

Then I need to wait for 90 days, upload malicious create and send you a link.

Still, 'malicious crate' and 'misleading description' is the hardest part in my opinion.

That means, that looking on download counter as safe measure is useless. You can filter rarely-used-untested crates, but not the malicious ones.

The malicious "rustdecimal" crate

excors — Thu, 12 May 2022 12:11:40 +0000

> So, to install a malware on your machine I need to write a malicious crate with misleading description, send you a link and .. make 10000 downloads?

crates.io prominently shows not just the total download count, but a chart of downloads per version over the last 90 days. (See the bottom of https://crates.io/crates/rust_decimal). So if you want to make a malicious crate that doesn't look immediately suspicious and untrustworthy, you'll need to spend 90 days replicating that pattern of downloads.

I suspect you'd have more success by e.g. finding a popular but not-very-well-maintained crate, and sending a pull request that correctly implements a frequently-requested feature but includes a dependency on a new crate that you maintain yourself. Hopefully the maintainer will be glad to receive the patch and won't be concerned enough about the new dependency to reject it or rewrite it.

Developers who check the download history of the popular crate will have no reason to be concerned, and nobody has time to check the download history of every dependency of a crate before they use it. Then you can make a new release of your crate with some malicious code, and anyone starting a new project or running "cargo update" will receive your update.

(Old projects that don't explicitly update won't see the new version, because the old version number is stored in the project's Cargo.lock file, which is usually checked into VCS so every developer builds exactly the same code.)

For a conscientious application developer, I think one possible solution is to run "cargo vendor" to retrieve a local copy of all dependencies, and check that into your VCS. Use the net.offline config option to prevent Cargo accidentally using a non-local crate. Whenever you want to update dependencies: run "cargo update", "cargo vendor" again, then push it to your code review system and read the diffs of all the vendored dependencies. And whenever you want to add a new dependency to your project, look at the number and complexity of its indirect dependencies and consider the long-term costs of reviewing all that code and its updates forever, and balance that against the benefits it provides.

The malicious "rustdecimal" crate

tchernobog — Thu, 12 May 2022 12:10:05 +0000

In a perfect world where every package maintainer had infinite time and would be paid to read and audit every single package you would be right. Unfortunately, history shows that malicious code has landed and will continue to land also in mainstream distributions.

There is also a problem of freshness of packages, with patches taking time and effort in percolating from upstream to downstream. I've seen several packages with known vulnerabilities remaining outdated for far more time in distros rather than when following the upstream releases directly.

The complexity of the current software ecosystem on a typical installation necessarily puts the burden of proof on developers who are including certain dependencies in their projects. Rather than on distro maintainers, I would expect library users (developers) to have some form of sandboxing in place on their CI systems, and static analysis of their code and its dependencies.

A CI system should also not be allowed to connect to the Internet by default, rather going through a proxy such as Artifactory for downloading of packages. Hopefully, something like sigstore (https://www.sigstore.dev/) might take traction and help in ameliorating the issue.

Putting the blame on cargo, pypi, npm, etc. misses the mark in my opinion. Except that they should be enforcing verified author-signing of packages.

For a certain class of applications (e.g. GUIs) flatpak and its sandboxing approach offer a good improvement on security, provided it's properly set up. For the rest... I can assure you I've seen developers bundling a lot of outdated and vulnerable dependencies without thinking, no matter how easy or hard a certain tool makes it.

The malicious "rustdecimal" crate

cortana — Thu, 12 May 2022 09:21:43 +0000

No, the necessary level of assurance can only be achieved by using a https URL and checking the signature of the downloaded data throughout the supply chain.

The malicious "rustdecimal" crate

ddevault — Thu, 12 May 2022 08:09:32 +0000

This is a fundamentally unsolvable problem with the (bloody stupid) approach to package management endorsed by Cargo, PyPI, npm, etc. Vendors cannot be trusted to publish their own packages. The distribution model, with a dispassionate third-party reviewing, packaging, and signing the software, is much better for security.

The malicious "rustdecimal" crate

pabs — Thu, 12 May 2022 04:42:35 +0000

You also have to trust their hoster, while you don't have to trust the hoster with a traditional package manager.

The malicious "rustdecimal" crate

developer122 — Thu, 12 May 2022 04:04:25 +0000

Curl Sudo isn't the level of problem people make it out to be.

If HTTPS is enforced and you got your download URL from a trusted source, then it's no different from installing through a package manager. It's merely a question of whether you trust the person owning that website and providing the install script as much as you trust canonical/debian/etc.

It's a janky way to install things, but it isn't an insecure one.

The malicious "rustdecimal" crate

raven667 — Thu, 12 May 2022 03:52:17 +0000

I don't think it is possible to always prevent this kind of typosquatting or malicious libraries in public distribution systems, without a much higher bar for upload, like Debian, or manual, labor intensive review of all changes, like Apple, but it is possible to catch them after the fact and stop them. With a good audit trail you can see exactly when and by who the malicious code was introduced and, if you keep the download logs, who was affected, so attacks using this mechanism don't stick around for long, are noisy and relatively easy to clean up, making this mechanism a less attractive attack point for mass exploitation. Sometimes you can't make you can't make something perfectly secure but you can encourage the attacker to move along to a softer target elsewhere.