Study: Attacks on package managers

Bittorrent should definitely be an option, but it cannot be the only one.  Far too many
corporate sites, and even some educational ones, ban or heavily restrict bittorrent (and any
other peer-to-peer protocol).

>Indeed, it is strange to even have mirror sites in this day and age when we have Bittorrent.

The next local mirror here (4 hops away according to traceroute) is delivering files faster
than any BT does. Well, that's the purpose of mirrors in the first place...

> Indeed, it is strange to even have mirror sites in this day and age when we have Bittorrent.

Well, *some* of us live in North America, and are forced to use an ISP that throttles p2p
protocols.

Bittorrent works well for transfer of large files.. it does not work well with small packages.
I would think the torrent nightmare of having a seperate torrent for each updated package
would make pre-yum rpm/pre-apt .deb hell look like a picnic.

Considering the current topology of the Internet mirrors will stay making 
sense and be more efficient than peer-to-peer systems. Throw in the huge
differense between clients' upload and download bandwidth and mirrors make 
even more sense.

Better to experiment with multicast...

From the user's perspective, the a mirror is better than bittorrent.  Mirrors are known-good
hosts with lots of bandwidth.

The point of bittorrent is that you can distribute a file which otherwise you would not be
able to distribute.  People seem to forget that central element of bittorrent, and turn it
around to mean that users should expect extremely fast download rates.  That's not true.
Bittorrent allows you to download a file that otherwise would not be available.  That it is
sometimes fast is simple and artifact of the implementation.

Getting closer to that:

http://debtorrent.alioth.debian.org/

I can think of a few reasons. Bittorrent is more complex (which would make the install program
larger, and might not fit on a netboot ram image). It is slower. (If it's faster for you,
switch mirror.) It uses uplink capacity for the end users (which some may have to pay for).
Distribution of torrent metadata is a single point of failure.

So in short, it's not robust, it's slow, and it's complicated.

You make it sound like more modern equals better, which should be obvious it is not true.
Otherwise we would not use a web browser when we have 3D games. Different uses, different
protocols.

Bittorrent uses a ridiculous amount of bandwidth, which is not always Cheap.

There is an additional problem with BitTorrent or other P2P solutions that hasn't been
mentioned in the discussion here.   When you download the current version of a package, you
are commonly doing so because you are upgrading an old version.   So when downloading a
package from an untrusted party (like a mirror) you disclose that you are running outdated
software to that party.   This is obviously bad because they may be able to root you, etc.
Using something like BitTorrent increases the effect because now a much larger group of people
with a lower barrier to entry are aware of you requesting a package.

I don't think this is a good trade-off given the current status quo.

To put it more plainly:

The attacker cannot use a malicious mirror inject old content against security.debian.org,
since security.debian.org isn't mirrored by third parties.

Testing users could become vulnerable. Mitigating against this would be relatively easy to
implement, as the Signed "Release" file already has A "Date" field. - Just check that it isn't
older than X days. As a added bonus, users will start noticing if their mirror has problems
getting updates.

One option the attacker has is transparent proxies, but then again you are in big trouble
anyway (mmm.. cookies..) if a cracker manages to root your ISP's transparent proxy.

(Disclosure: I'm an author of the study)

I agree with you that using a security repository significantly reduces the vulnerability to
attack.   This doesn't protect against the "endless data" attack we describe that can be used
by a mirror to crash clients, but this is not as big of a threat as compromising the client.
(Does Debian need to contact so many mirrors by default?)

There are several other minor issues that remain and may impact Debian users that we didn't
see discussed here.   First, there is no authentication that you are talking with the security
repo, so a MITM attacker can still launch attacks by masquerading as the repo.   HTTPS with
correct certificate checking would prevent this, hence the HTTPS ("Bad research at its best")
suggestion.  :)

Second if the security repo fails or is not contactable from a client (non-transitive
connectivity, etc.) then the mirrors can attack clients by replaying content from the security
repo.

Third, clients who use netselect-apt, etc. should be aware that they are likely to remove the
security repo from their list of mirrors and thus become vulnerable.


In general we found that Debian's practice of using a security repository is effective in
protecting their users from replay / freeze attacks in the majority of cases for users with
default configurations.

There is another issue with Debian that we didn't bring up on the web pages because we felt
there was already too much loosely connected content.   We briefly looked at the developer
update process and if I understand correctly from reading the documentation any developer can
update any package (they are encouraged not to except in extreme situations but have the
ability to).   Furthermore, if I understand correctly there are thousands of keys in the
developer database so really this means that a compromise of any key allows an attacker to
upload any package.   Also there are keys as short as 768 bits and as old as 1993.   I'm not a
crypto expert so I don't really know how to quantify risk, but both of those numbers trigger a
mental alarm.   Anyways, I was hoping that you could also clarify / correct / confirm any of
this information as well.

Thanks,
Justin Cappos

There is a 'yum-fastestmirror' plugin, so if you know the target computer is using it, you can
set up a mirror a small number of hops away and have a good chance of being preselected.
Signing and dating the package list is still the safest solution.

That is a very nasty attack.  To defend against this, an organization should have a dedicated
"mirroring" computer that runs almost nothing.  This computer does all the downloads and then
serves the updated packages to other machines.

By decoupling the machine doing the downloading from the machine being updated, you can
mitigate against evil mirrors.  (You can't completely block the attack because the downloader
machine itself might happen to require a package that is found to have a vulnerability.
That's really hard to protect against other than by upgrading the downloader machine manually.)

Why is this a nasty attack? Compare to:

1. Portscan a lot of computers; save the results
2. When there's a security update, hit the computers running that program

It doesn't require you to have a mirror (and hence a large traceable presence) and hits all
targets, not just one distro. It's less targetted, but how often has that been a problem in
Internet attacks?

 ...
Alternative (3)
"Root" a whole bunch of NAT routers.

would be nice if yum would get a digest of the package list from the master (together with the
mirror list etc) which it then can use to reject anything other than the real package list...
if a mirror doesn't have the right one the client could them report back to MM (triggering a
rescan prematurely possibly) and get another mirror...

What kind of verification would you suggest for a voluntary mirror? If you add too much
overhead, good mirrors will just walk away and you will lose. That isn't the gateway where you
should be adding security. You should assume malicious mirrors are already present and work to
mitigate that within the distribution. 

That is the wrong approach.  You are suggesting there should be verification so that only
trustworthy people (by some measure) can set up a mirror site.  But it will always be possible
for bad guys to slip through the net.  Even the US nuclear weapons programme, with the
strictest possible vetting of participants, contained spies.

And even a well-meaning mirror site can be taken over by an attacker.

Better to make sure the update system is secure so that even with total control of one or more
mirrors an attacker cannot push out bad packages or cause a denial of service for more than a
few minutes.

All files downloaded from Gentoo have SHA1 and SHA256 sum associated with them. If a file does
not match the file the developer was using, the user will receive a digest error and the
package manger will not continue. Does any one see any loopholes with this scheme?

I've posted a few corrections on the original blog entry.   I'll post here as well.



Hello, I'm one of the authors of the study.   I wanted to first of all thank you for
commenting on our research.   One of the major benefits we hoped would come from making this
public is that the Linux community would become more aware and interested in fixing the
problems we point out.


I also wanted to respond to several of the issues you brought up in your blog post.   First, I
appreciate you pointing out that many distributions check to see if their mirrors are current
and try to remove mirrors that are not.   We under-emphasized this in the webpage and other
documents because we did not view this as a mechanism used to detect a malicious party (we
thought the intent was to detect negligent administrators and broken scripts).   As I'm sure
you and the savvy reader are aware, it is possible for a web server to serve different content
to different users.   We examined our web request logs from our CentOS mirror and I believe we
can identify the "checking bot" IP addresses.   If we were malicious, we could serve "good"
content to your checking bot and "malicious" content to other users.   I would be happy to
provide what I believe to be the IPs used to check if a mirror is current to you offline for
verification / rebuttal.   However, since you view this information as important to the
security of your users, I will not list the information here.

Additionally, I wanted to mention that we found significant security problems with Fedora's
MirrorManager (our FAQ talks about how it can be used to target attacks).   However, other
redirectors we looked at (like Download Redirector for OpenSUSE) do improve security in a
similar manner to what you describe.   I was wondering if we could talk more offline about how
your mirror list redirection works so we can discuss the potential for abuse? 


I also wondered if you might want to look in detail at the other attacks page of the web site
and the technical report which mentions detailed information about flaws in YUM.   We would be
happy to discuss the feasibility of attacks that target these issues with you.   However, I
will point out one attack that is extremely simple that I hope illustrates there is a real
danger to your users.   If I control a mirror and you attempt to retrieve a file from my
mirror, I can return an endless stream of data which (on YUM) will fill the disk and crash the
client system (stopping logging, corrupt databases, etc.).   This is obviously a real threat
to all of your users regardless of any mirror redirection strategies you perform.

Anyways, we thank you for taking a look at our research and hope to hear more rebuttal /
confirmation in the future.

Thanks,
Justin Cappos