The many failures leading to the LiteLLM compromise
A Trivyal pursuit
This story starts with a different project, Trivy, which is a widely used security scanner, distributed under the Apache-2.0 license. As is often the case with these scanners, Trivy releases normally include tests for newly discovered vulnerabilities, so projects that depend on Trivy to detect potential security problems in their code have every reason to want to rerun their scans when a new release is made. Many projects hosted on GitHub thus set up an action so that, when a new release tag shows up for Trivy, new scans are automatically run, just in case Trivy has any new problems to point out.
On March 20, Paul McCarty announced that Trivy had been compromised; Philipp Burckhardt wrote up a detailed report on what was done. Somebody had managed to obtain credentials giving write access to the Trivy repository. This attacker then placed commits with malware in that repository, but they did so without changing any existing branches, avoiding the notifications that would have normally gone out in response to such changes. Instead, a large number of release tags were force-pushed to point to the new commits, which was sufficient to cause other projects to perform automatic Trivy runs.
The Trivy malware was of the information-stealing variety; whenever it ran within a project's context, it would sweep up everything it could get its virtual hands on and send it back to the attackers. Trivy itself was never the primary target; it was just a stepping stone giving access to other projects of interest. Just how many projects this exploit was able to hit may not become clear for some time.
On to LiteLLM
One of the projects that fell into that trap was LiteLLM; specifically, the PyPI credentials of one of its developers (krrishdholakia) were harvested. Those credentials were then used to upload compromised versions of LiteLLM (1.82.7 and 1.82.8) to PyPI, where they would subsequently be downloaded by the large number of people who install LiteLLM each day. Specifically, according to futuresearch, there were 47,000 downloads of the compromised LiteLLM package in just 46 minutes.
The word went out quickly, but the distribution of information was hampered by a set of hostile bots that targeted the LiteLLM GitHub issues (#24512 and #24518) with hundreds of spam comments, creating vast amounts of noise. At one point, people were advised to follow the Hacker News discussion rather than the GitHub issues themselves. The malware on PyPI was taken down quickly, but a lot of damage had been done by then.
The second compromised version of LiteLLM, in particular, used an insidious method to run the hostile code on systems where it was installed. Normally, if one installs a compromised Python program, that program will not be able to do anything until somebody runs or imports it. But when the Python interpreter starts up, it looks in the site-packages directory for files with a .pth suffix, and automatically executes the contents of every one it finds. Normally these files are used to configure Python's import paths and such, but they contain ordinary Python code. The litellm_init.pth file added by the compromised LiteLLM package was thus run anytime a Python interpreter started in the install environment (which may be anywhere in the compromised system if LiteLLM was not installed in a virtual environment), regardless of whether anything from LiteLLM itself was run.
That code, once again, gathered up all of the information it could find, including environment variables, SSH keys, Git credentials, Kubernetes secrets, shell histories, crypto wallets, and more. The whole thing was encrypted, packaged into a tar file, and pushed to a system behind the attacker-controlled litellm.cloud domain. There is no way to know how many of these uploads were done or what was in them — until the attackers make use of what they have learned.
Anybody who may have installed the compromised packages clearly has a significant cleanup job to do, including changing any credentials that might have been exfiltrated. The compromised code, for both Trivy and LiteLLM, has been taken down and should not attack any others, but this episode is far from over.
There are going to be vast numbers of compromised systems out there. It seems unlikely that LiteLLM was the only project that was compromised by the Trivy exploit. We have not heard about the others (with the exception of the telnyx compromise reported just as this article was going out); perhaps the attackers have not yet acted on other credentials that they may have acquired. But the attackers are still out there, and they seem unlikely to sit on any such material for long; expect to hear news of other compromised projects. There is also the matter of all those systems that installed the bad LiteLLM packages; again, there is a wealth of information that will have been collected by the attackers, and they seem likely to want to make use of it at some point.
In the end, this episode represents a multi-level cascading failure. A
branch-change notification mechanism was bypassed. A software system meant
to improve security instead compromised it. Important credentials were
stored where automated exploits could read them, and two-factor
authentication was evidently not in use. The protections meant to
prevent the uploading of hostile packages failed. The ability of Python
packages to add configuration code allowed an exploit to reach beyond the
compromised package itself. Any of these failures on their own would have
been enough to cause a lot of damage; taken together, then enabled a
catastrophic security failure that may have only begun to play out.
| Index entries for this article | |
|---|---|
| Security | Python |
| Security | Supply chain |
| Python | Security |
