event-stream, npm, and trust

Malware inserted into a popular npm package has put some users at risk of losing Bitcoin, which is certainly worrisome. More concerning, though, is the implications of how the malware got into the package—and how the package got distributed. This is not the first time we have seen package-distribution channels exploited, nor will it be the last, but the underlying problem requires more than a technical solution. It is, fundamentally, a social problem: trust.

Npm is a registry of JavaScript packages, most of which target the Node.js event-driven JavaScript framework. As with many package repositories, npm helps manage dependencies so that picking up a new version of a package will also pick up new versions of its dependencies. Unlike, say, distribution package repositories, however, npm is not curated—anyone can put a module into npm. Normally, a module that wasn't useful would not become popular and would not get included as a dependency of other npm modules. But once a module is popular, it provides a ready path to deliver malware if the maintainer, or someone they delegate to, wants to go that route.

That is just what happened with the event-stream package, as was recently discovered. The package allows creating streams that can be used both for I/O and for event handling. Its maintainer, Dominic Tarr, had stopped using the package some time ago, so his interest in maintaining it was low. As he noted in a comment on the bug report filed in the event-stream GitHub repository, someone volunteered to take it over:

As detailed in a blog post by Zach Schneider, who plucked various pieces out of the voluminous GitHub bug report thread, the attack that was inserted by the new maintainer, "right9ctrl", was clever. The commit log of changes right9ctrl made to event-stream was fairly innocuous; even the commit that added the malware was simply adding a new dependency on another npm module: flatmap-stream.

Had anyone looked, flatmap-stream might have seemed a bit of an odd dependency: it had one contributor and no downloads prior to its inclusion. Its contents might seem reasonable at first glance, but there is a tangled chain of malware contained there.

The flatmap-stream npm package had an extra file added into it that was not in the GitHub repository. It also had "minified" code that read the AES256-encrypted data stored in the file using the parent package's npm_package_description as the key. For all except one npm package, that decryption would fail (and be ignored) but, for the victim package, it resulted in JavaScript code that would be executed. That code does a decryption of a different chunk of the "extra" file that results in the payload code, which, naturally, gets executed.

As determined by brute-forcing the key from a list of all the npm package descriptions, the victim package was copay-dash, which is a "secure bitcoin wallet platform" from a company called Copay. Given the presence of the word "bitcoin", one can probably guess what the malware ultimately targeted. It would send account information to the attacker, who would, presumably, use it to abscond with the Bitcoin.

The dependency on flatmap-stream only lasted a little over ten days before it was replaced with a non-malware implementation of a "flat map" in event-stream itself. The npm blog post about the incident says that it was the Copay build process that was being subverted:

As Schneider noted, the JavaScript-development community is particularly vulnerable to this kind of problem:

He goes on to note JavaScript applications tend to be fast moving: "its users install a lot of packages and updates, and are thus vulnerable to malicious updates". On the other hand, problems can also occur from not updating frequently enough, he said, pointing to the Equifax breach. He suggested two ways to avoid this kind of thing in the future: locking the version number of dependencies to "known good" versions and paying attention to the dependencies a project is adding.

We have seen other related mayhem in the npm world before. Back in 2016, a developer deleting a simple left-pad npm module "broke the internet" because so much of the rest of the npm ecosystem relied on it to pad strings.

But the problem is not at all restricted to npm or JavaScript. Other languages have similar problems with their non-curated package repositories. Typosquatting is a related problem that has occurred with some frequency as well. Beyond that, it is not even just a problem for languages; as Dirk Hohndel pointed out in a talk back in May, today's containers are built up from many constituent parts gathered from all over the internet. Most of the container creators have no idea what is actually in them, what versions of code are being used, and so on. Docker and similar technologies are also part of the "move fast" school of development.

Certainly there have been some failures even in curated repositories—humans are not infallible. But curation and "move fast" tend not to play all that well together, which is why there is always such tension between the language-specific installation methods (e.g. npm, pip) and a distribution's package-management system. Users often just want the latest and greatest; they are not willing to wait for a distribution to get around to packaging it. That may be reasonable for a personal desktop or laptop—there are obvious risks (e.g. Bitcoin wallets) but they may be considered manageable—but the public release or deployment of a web application or component seems like it warrants a higher level of scrutiny.

Beyond more scrutiny, which is surely something that development teams should be doing no matter whether it slows things down, package maintenance is an area that clearly needs to be addressed. Tarr created a package that was useful to some, but apparently got no help in maintaining it. Once he had shared it, the left-pad fiasco shows there is no real way to "unshare" it, but he lost interest in maintaining it. In his statement about the event-stream malware, Tarr noted that the problem is widespread:

He continued by noting that sharing commit and publish rights was a longstanding npm-community practice. "Open source is driven by sharing! It's great! it worked really well before bitcoin got popular." He suggested that people should either be paying maintainers of the packages they use or step up to help maintain packages they depend on.

Once again, this is not in any way an "npm problem". The explosion of availability of open-source software has not really been met with a concomitant increase in the number of maintainers. There are, it seems, a lot of companies and others that are using open source without truly considering what that means. Even large projects like the Linux kernel suffer from a dearth of maintainers in some areas and events like Heartbleed exposed the maintenance problem for critical internet infrastructure like OpenSSL. Heartbleed led to the founding of the Core Infrastructure Initiative, but it is hard to see that kind of effort being extended down to the "leaves"—fixing it really requires users to step up.