The malicious "rustdecimal" crate
The crate contained identical source code and functionality as the legit rust_decimal crate, except for the Decimal::new function.When the function was called, it checked whether the GITLAB_CI environment variable was set, and if so it downloaded a binary payload into /tmp/git-updater.bin and executed it. The binary payload supported both Linux and macOS, but not Windows.
Posted May 11, 2022 16:44 UTC (Wed)
by mss (subscriber, #138799)
[Link] (3 responses)
Posted May 11, 2022 17:36 UTC (Wed)
by qyliss (subscriber, #131684)
[Link] (2 responses)
That probably doesn't apply to very many developer machines.
And it had <500 downloads total, so it's plausible it didn't compromise anything at all, and was only ever even seen by crawlers.
Posted May 12, 2022 1:21 UTC (Thu)
by plugwash (subscriber, #29694)
[Link] (1 responses)
The report says the payload was gone by the time they analysed the issue which IMO supports this hypothosis.
Posted May 12, 2022 17:16 UTC (Thu)
by cesarb (subscriber, #6266)
[Link]
If that's the case, perhaps looking for all merge requests at projects which are on gitlab, depend on the original crate, and where the merge request changed that dependency, might find the victim (if it's a public project).
Depending on how the CI is configured, it might automatically run on all merge requests, and it might expose important tokens (for instance, tokens to publish a finished and signed release) to the CI process. So this might be just the first step, and later we might be seeing a legitimate project publishing a malicious release due to this compromise.
Posted May 11, 2022 18:59 UTC (Wed)
by ballombe (subscriber, #9523)
[Link] (22 responses)
Posted May 11, 2022 19:03 UTC (Wed)
by magfr (subscriber, #16052)
[Link]
An analysis of all the crates on crates.io was also performed, and no other crate with similar code patterns was found.
but it could stand out more.
Posted May 12, 2022 8:09 UTC (Thu)
by ddevault (subscriber, #99589)
[Link] (20 responses)
Posted May 12, 2022 12:10 UTC (Thu)
by tchernobog (guest, #73595)
[Link] (19 responses)
There is also a problem of freshness of packages, with patches taking time and effort in percolating from upstream to downstream. I've seen several packages with known vulnerabilities remaining outdated for far more time in distros rather than when following the upstream releases directly.
The complexity of the current software ecosystem on a typical installation necessarily puts the burden of proof on developers who are including certain dependencies in their projects. Rather than on distro maintainers, I would expect library users (developers) to have some form of sandboxing in place on their CI systems, and static analysis of their code and its dependencies.
A CI system should also not be allowed to connect to the Internet by default, rather going through a proxy such as Artifactory for downloading of packages. Hopefully, something like sigstore (https://www.sigstore.dev/) might take traction and help in ameliorating the issue.
Putting the blame on cargo, pypi, npm, etc. misses the mark in my opinion. Except that they should be enforcing verified author-signing of packages.
For a certain class of applications (e.g. GUIs) flatpak and its sandboxing approach offer a good improvement on security, provided it's properly set up. For the rest... I can assure you I've seen developers bundling a lot of outdated and vulnerable dependencies without thinking, no matter how easy or hard a certain tool makes it.
Posted May 12, 2022 13:26 UTC (Thu)
by sionescu (subscriber, #59410)
[Link] (13 responses)
Do you have examples ?
Posted May 12, 2022 14:13 UTC (Thu)
by tchernobog (guest, #73595)
[Link] (3 responses)
Humans are imperfect, no human review process is 100% safe.
A survey conducted through GitHub claims that about one in five bugs are planted with malicious intent.
https://www.zdnet.com/article/open-source-software-how-ma...
Considering the thousands of CVEs that have been open against a distro like Debian over the years, there is a good and fair chance that several bugs we have seen (and many we haven't yet) were indeed intentionally planted.
Posted May 13, 2022 11:03 UTC (Fri)
by mkj (subscriber, #85885)
[Link] (1 responses)
Posted May 13, 2022 11:43 UTC (Fri)
by excors (subscriber, #95769)
[Link]
> Most software vulnerabilities are mistakes, not malicious attacks. Analysis on a random sample of 521 advisories from across our six ecosystems [Composer, Maven, npm, NuGet, PyPI, RubyGems] found that 17% of the advisories were related to explicitly malicious behavior such as backdoor attempts. These malicious vulnerabilities were generally in seldom-used packages, but triggered just 0.2% of alerts. While malicious attacks are more likely to get attention in security circles, most vulnerabilities are caused by mistakes.
and:
> Analysis on a random sample of 521 advisories from across our six ecosystems finds that 17% of the advisories are related to explicitly malicious behavior such as backdoor attempts. Of those 17%, the vast majority come from the npm ecosystem. While 17% of malicious attacks will steal the spotlight in security circles, vulnerabilities introduced by mistake can be just as disruptive and are much more likely to impact popular projects. Out of all the alerts GitHub sent developers notifying them of vulnerabilities in their dependencies, only 0.2% were related to explicitly malicious activity. That is, most vulnerabilities were simply those caused by mistakes.
To be honest, I can't actually understand what that's trying to say. But I don't believe it's saying that 17% (nor 0.2%) of vulnerabilities were maliciously inserted: it's about advisories not vulnerabilities (and most projects won't bother with advisories for regular boring vulnerabilities found through code review), and "advisories related to explicitly malicious behaviour/activity" sounds like it's talking about active exploitation of a possibly-accidental vulnerability (vs an unexploited vulnerability found through code review), probably? Without more detail on the methodology (which I haven't been able to find) the numbers seem largely meaningless.
Posted May 13, 2022 13:15 UTC (Fri)
by mathstuf (subscriber, #69389)
[Link]
This doesn't pass my smell test. The article says "vulnerabilities" without defining what they mean. I presume they actually mean "CVEs on GitHub-hosted projects" here, but it's really hard to tell. There are plenty of vulnerabilities that never get a CVE. There are also plenty not on GitHub. So without some metric of what they're talking about, I have no idea how useful such a statistic is. But "bugs" is almost certainly not what you mean here (at least as I understand the term).
Posted May 12, 2022 14:50 UTC (Thu)
by zdzichu (guest, #17118)
[Link] (7 responses)
mjg59 provided a link recently: https://lists.debian.org/debian-devel/2003/02/msg00771.html
Posted May 12, 2022 16:49 UTC (Thu)
by ballombe (subscriber, #9523)
[Link] (6 responses)
Posted May 13, 2022 9:27 UTC (Fri)
by wtarreau (subscriber, #51152)
[Link] (5 responses)
In fact it's the perfect counter-example: all that could be found dates 20 years because the distro model works very well!
Posted May 13, 2022 13:50 UTC (Fri)
by zdzichu (guest, #17118)
[Link] (4 responses)
Posted May 15, 2022 3:46 UTC (Sun)
by wtarreau (subscriber, #51152)
[Link] (3 responses)
Selecting an issue that happened more than 10 years before the ecosystem subject of this article was even imagined is hardly relevant to this discussion, I'm sorry. It's presenting a process that's supposed to be better than another one without looking at what was done since in order to make sure this issue doesn't happen anymore. Yet the current situation is presented as having that limitation while it may very well be what made this issue not happen anymore. That's precisely a swap of cause and consequence.
Posted May 15, 2022 3:53 UTC (Sun)
by mjg59 (subscriber, #23239)
[Link] (2 responses)
Nothing was done. There's no process in Debian to prevent something like this happening again - the reason it hasn't is that upstreams are generally not malicious.
Posted May 20, 2022 2:21 UTC (Fri)
by dilinger (subscriber, #2867)
[Link]
Posted Jun 12, 2022 6:45 UTC (Sun)
by marcH (subscriber, #57642)
[Link]
"Generally" is good enough except for security or safety issues. My smartphone "generally" does not crash: good enough. My bank account is "generally" safe from hackers: not good enough.
But of course security bugs are also "just bugs". So it's funny to see some $BIGCORPS suddenly very obsessed about preventing security bugs and wondering "how did that happen?" when the answer is often just:
Posted May 13, 2022 20:24 UTC (Fri)
by jwilk (subscriber, #63328)
[Link]
Posted May 12, 2022 23:40 UTC (Thu)
by pabs (subscriber, #43278)
[Link] (2 responses)
Posted May 13, 2022 9:29 UTC (Fri)
by wtarreau (subscriber, #51152)
[Link] (1 responses)
Posted May 14, 2022 4:11 UTC (Sat)
by pabs (subscriber, #43278)
[Link]
Posted May 13, 2022 9:26 UTC (Fri)
by wtarreau (subscriber, #51152)
[Link]
I'd say that software that are distributed a bit too directly to users tend to evade such controls and by losing some of these constraints they also lose many of the guarantees they used to offer to their users. I do have absolutely zero trust in such software distributed on such channels that are nothing but a market place where fame replaces money, and where engagement can be zero without anyone noticing.
Posted May 13, 2022 19:53 UTC (Fri)
by Chousuke (subscriber, #54562)
[Link]
In Fedora, any packaged software is required to build offline (without any network access), and *only* from the sources provided; that means you at least have some hope of knowing what sources the binary was built from and rebuilding the software yourself.
As an example of the frankly incompetent crap vendors sometimes do that wouldn't fly in a serious distro, AWS's package for OpenDistro for Elasticsearch at one point used a pre-install script to download a binary from the internet upon installation and dump it in /usr/lib. This was done in a noarch package that's supposed to be architecture-independent! I remember it well because I had to spend time untangling a surprise broken upgrade in an ES cluster that had no internet access...
Posted May 11, 2022 19:41 UTC (Wed)
by epa (subscriber, #39769)
[Link] (12 responses)
That’s not to claim that Haskell is some promised land of strictly typed and fine-grained permissions: as far as I know you will tend to just label everything as ‘IO’ and it’s not common to declare that a library function may perform disk access but not network access, or whatever. But let’s dream for a moment.
What if you wrote a library as pure code with no side effects, enforced by the type system, and then when importing a crate you require this stricter type? Then adding the malicious code would require a type change and no longer build.
I’d like to apply a higher review standard to libraries that are permitted to do I/O. If I want to decompress some gzip data from one buffer in memory to another buffer, I am not that paranoid about using third party code to do it, if the language can guarantee memory safety (as Rust does) and that no side effects or I/O can be performed by the library code. (Even better if you can guarantee that it cannot read the state of the outside world, so its behaviour must only depend on the input buffer and arguments explicitly passed.) Of course there would be libraries that need full access to the environment, but I would pick those more carefully, just as code inside ‘unsafe’ blocks needs extra scrutiny.
(I am not claiming that there could not still be exploits from a malicious decompression routine, any more than ‘safe’ or managed code does not need reviewing. I just mean the risk is higher if full I/O is allowed.)
So when picking the libraries included in your build, you would see the ‘purity’ of each one — which could include an ultra-strict level where not even memory allocation is allowed. The more freedom the library code has, the more carefully you must review it. And if a newer version wants more abilities, then like configuring permissions for apps on your phone, that must be an explicit step.
Posted May 11, 2022 20:22 UTC (Wed)
by NYKevin (subscriber, #129325)
[Link]
Furthermore, in Haskell's data model, manually allocating memory (which is not the usual case, since Haskell is a high-level language) is considered a side-effect that normally lives within the IO monad, because it alters the state of the heap. This is why the comment linked above complains of "alias[ing] all your buffers" - that particular iteration of unsafePerformIO is subject to inlining, meaning that it will happily inline calls to malloc, which results in Haskell eliminating identical calls and aliasing the resulting buffers.
Posted May 11, 2022 20:28 UTC (Wed)
by wtarreau (subscriber, #51152)
[Link] (9 responses)
I should serve as an illustration about "what if the author was really badly intented". Because clearly it looks like an experiment that should serve as a warning against such bad practices. Similarly if I had time to waste I'd create a stupid project that randomly does nasty things to those who install it via the documented "curl | sudo" procedure.
Posted May 12, 2022 4:04 UTC (Thu)
by developer122 (guest, #152928)
[Link] (8 responses)
If HTTPS is enforced and you got your download URL from a trusted source, then it's no different from installing through a package manager. It's merely a question of whether you trust the person owning that website and providing the install script as much as you trust canonical/debian/etc.
It's a janky way to install things, but it isn't an insecure one.
Posted May 12, 2022 4:42 UTC (Thu)
by pabs (subscriber, #43278)
[Link]
Posted May 12, 2022 9:21 UTC (Thu)
by cortana (subscriber, #24596)
[Link]
Posted May 12, 2022 13:01 UTC (Thu)
by wtarreau (subscriber, #51152)
[Link]
Posted May 12, 2022 17:09 UTC (Thu)
by ballombe (subscriber, #9523)
[Link] (4 responses)
Posted May 12, 2022 22:01 UTC (Thu)
by nybble41 (subscriber, #55106)
[Link] (3 responses)
In terms of installation of software from new sources (as opposed to updates to existing software or centralized repositories managed by distros) the validation instructions, e.g. a GPG public key or signature file hosted on the same domain, are just as vulnerable to this sort of attack as the script itself. If you don't want to be reliant on control of the domain name you need some out-of-band system to validate your sources, such as a PGP Web of Trust, but these systems have not really caught on as a practical general solution thus far. Identifying the source by its public key, e.g. using IPNS, or Tor .onion domains, would be one alternative; then the root of trust is pushed back to whoever directed you to that source.
A more immediate problem with `curl | bash` is ensuring that the script is safe with respect to truncation; there is no buffering, so if the connection is interrupted `curl` just reports EOF and `bash` will attempt to run whatever was received for the interrupted line as if it were a complete command.
Posted May 13, 2022 9:31 UTC (Fri)
by wtarreau (subscriber, #51152)
[Link]
Exactly! I'm pleased to read this because I feel like I'm constantly blowing in the wind when I say this, and even people do not believe my examples :-/
Posted Jun 12, 2022 6:54 UTC (Sun)
by marcH (subscriber, #57642)
[Link] (1 responses)
Safety discussions are a bit ridiculous the moment you use a language that ignores errors by default. We love high level discussions about memory safety and borrow checkers but in the real world the problems often start much more mundane: many C programs discard errors and even when they don't the error handling code has never been tested.
Posted Jun 12, 2022 21:38 UTC (Sun)
by nybble41 (subscriber, #55106)
[Link]
In the case of `curl | bash` the command isn't safe even *with* `set -o pipefail`. The problem isn't the exit status of the pipeline (which would typically be ignored in any case) but rather the fact that it runs the script *as it's being downloaded*, without ensuring that the entire script is available before the first command is executed, and even worse that the shell will attempt to execute the last line downloaded even it was truncated.
Posted May 12, 2022 3:25 UTC (Thu)
by dtlin (subscriber, #36537)
[Link]
Posted May 11, 2022 20:30 UTC (Wed)
by flussence (guest, #85566)
[Link] (8 responses)
If I click the crates.io link to (the real) rust_decimal's info, it's presented in an information vacuum. I'd expect, at a bare minimum after this, a list in the sidebar of similar names ranked by seniority. Without some kind of hint like that it's impossible to find out there's even something amiss.
Posted May 11, 2022 20:52 UTC (Wed)
by Gaelan (guest, #145108)
[Link] (7 responses)
Posted May 11, 2022 21:56 UTC (Wed)
by amarao (guest, #87073)
[Link] (5 responses)
So, to install a malware on your machine I need to write a malicious crate with misleading description, send you a link and .. make 10000 downloads?
I'd say that the latest is the easiest...
Posted May 12, 2022 0:27 UTC (Thu)
by NYKevin (subscriber, #129325)
[Link] (1 responses)
Posted May 12, 2022 13:03 UTC (Thu)
by amarao (guest, #87073)
[Link]
You miss the point of my argument. I'm saying that imitating some reasonable number of downloads for crate is simple. It's the simplest part of the trickery. That means, you can't use it to meaningfully defend yourself from malicious crates by looking on download counter (and all cousins, like number of forks and stars on GH). That doesn't mean you can't use 'little downloads' as a red flag, but you can't use it as a qualifier for 'safe to use'.
Posted May 12, 2022 12:11 UTC (Thu)
by excors (subscriber, #95769)
[Link] (2 responses)
crates.io prominently shows not just the total download count, but a chart of downloads per version over the last 90 days. (See the bottom of https://crates.io/crates/rust_decimal). So if you want to make a malicious crate that doesn't look immediately suspicious and untrustworthy, you'll need to spend 90 days replicating that pattern of downloads.
I suspect you'd have more success by e.g. finding a popular but not-very-well-maintained crate, and sending a pull request that correctly implements a frequently-requested feature but includes a dependency on a new crate that you maintain yourself. Hopefully the maintainer will be glad to receive the patch and won't be concerned enough about the new dependency to reject it or rewrite it.
Developers who check the download history of the popular crate will have no reason to be concerned, and nobody has time to check the download history of every dependency of a crate before they use it. Then you can make a new release of your crate with some malicious code, and anyone starting a new project or running "cargo update" will receive your update.
(Old projects that don't explicitly update won't see the new version, because the old version number is stored in the project's Cargo.lock file, which is usually checked into VCS so every developer builds exactly the same code.)
For a conscientious application developer, I think one possible solution is to run "cargo vendor" to retrieve a local copy of all dependencies, and check that into your VCS. Use the net.offline config option to prevent Cargo accidentally using a non-local crate. Whenever you want to update dependencies: run "cargo update", "cargo vendor" again, then push it to your code review system and read the diffs of all the vendored dependencies. And whenever you want to add a new dependency to your project, look at the number and complexity of its indirect dependencies and consider the long-term costs of reviewing all that code and its updates forever, and balance that against the benefits it provides.
Posted May 12, 2022 13:00 UTC (Thu)
by amarao (guest, #87073)
[Link] (1 responses)
Okay. I need to a broken crate with misleading description, put a systemd timer with a curl with some randomness in it ... 15 minutes of playing with execution times based on chart from other well-known crate?
Then I need to wait for 90 days, upload malicious create and send you a link.
Still, 'malicious crate' and 'misleading description' is the hardest part in my opinion.
That means, that looking on download counter as safe measure is useless. You can filter rarely-used-untested crates, but not the malicious ones.
Posted May 13, 2022 13:16 UTC (Fri)
by mathstuf (subscriber, #69389)
[Link]
There is `cargo-crev` which is aiming to help this problem. https://github.com/crev-dev/cargo-crev
Posted May 14, 2022 5:49 UTC (Sat)
by flussence (guest, #85566)
[Link]
Here's a pretty extreme counterexample: https://pecl.php.net/package-stats.php
Statistics are a footgun.
Posted May 12, 2022 3:52 UTC (Thu)
by raven667 (subscriber, #5198)
[Link] (3 responses)
Posted May 12, 2022 22:57 UTC (Thu)
by khim (subscriber, #9252)
[Link] (2 responses)
iOS does have malware (although much smaller amount than Android) thus it's clear than labor intensive review of all changes doesn't work. Debian approach works more-or-less fine if you accept the fact that majority of software wouldn't delivered through that mechanism. This leaves you with feel-good impression without solving any real problems. This problem is not, actually, solveable: you are either popular (and then have some malware) and are not (and then people are using other things and receive their fix of malware from other sources).
Posted May 14, 2022 5:36 UTC (Sat)
by patrakov (subscriber, #97174)
[Link] (1 responses)
Posted May 14, 2022 12:30 UTC (Sat)
by khim (subscriber, #9252)
[Link]
Not even Apple is arrogant enough to ask for the sources, only Linux distro guys ever tried to enforce that (and even they no longer try to do that). Of course only binaries are sent.
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
When the function was called, it checked whether the
GITLAB_CI
environment variable was set, and if so it downloaded a binary payload into /tmp/git-updater.bin
and executed it. The binary payload supported both Linux and macOS, but not Windows.
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
If you have nothing better, that rather prove that Linux distributions are quite safe
against malicious code injection.
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
because crashing from time to time was considered good enough... until hackers look at our products.
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
HTTPS and domain name ownership are insufficient.
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
Everyone knows what imagemagick and sdl_image are, they have an overlapping set of use cases, and from a quick glance they both appear perfectly legitimate packages by legitimate people. The download numbers though, if used as evidence in this way, tell a wildly different (and incorrect) story.
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate
The malicious "rustdecimal" crate