LWN: Comments on "Counting vulnerabilities"

Counting vulnerabilities

John_Emerald — 2007-10-03T13:48:34+00:00

First, remember that what was counted was "security patches".
It is completely irrelevant to count the number of "security patches" for many different reasons, as already mentionned. To name a few : Security breach are not all took to public attention, the count of known security vulnerabilities has almost nothing to do with the count of existing vulnerabilities (which in any case would be a lot more numerous), open source has more eyes to bring more "known vulnerabilities", etc.

So at the objective point of view this analysis is totaly meaningless to help compare the two systems and can't prove anything as acknowledged even on Mr. Jones's blog. Jones said the point was to explore "common misperceptions".

The error is that "security patches" has been translated to "vulnerabilities"... Instead of "known vulnerabilities", should it need to be translated. If you do the correction, you can see that this could completely invert the meaning of the statistics and of a great part of the text, at a "common misperceptions" level of course. We could for example say that some companies were more concerned about security and released more patches.

Objective deductible facts #1) So lets recapitulate and put things strait :
This analysis is either on one side meaningless or on the other totally wrong, misleading and the opposite of reality.

Objective deductible facts #2) Lots of these articles are based on this translation "error". So either we have a security director who doesn't understand a bit of what he is talking about, or we have a marketing campaign disguised as personal blogs (plus, comments are moderated so the truth cannot be revealed to the public).
The two cases are showing objectively that Miscrosoft is a bad choice regarding security. The first case would mean their security director cannot understand security analysis basics. In fact it would show that he is not able to run or understand any analysis at all. The second case would mean that they are disguising their actions and lying to give misperceptions to the public (their consumers), which would mean we cannot trust them as a company. Honestly, can anyone see another possibility ?

I tried to mention the translation "error" in the comments on one of the Miscrosoft employees blog, but of course the comment has been "moderated".

Each time we don't address the simple and basic problem/error/lie directly and argue at their level, we are helping them to push and hide the real problem/error/lie further into the dark and intoxicate the public with wrong beliefs.

So... Please be cautious, don't help them on their campaign if you don't mean to and / or aren't paid to do it.
I'm asking everyone who reads this : Please, if you comment on these articles start by uncovering this basic and critical error, don't give them too much credits.

Why not built-in vulnerability checking???

jabby — 2007-07-05T11:45:42+00:00

A huge step forward would be for major project hosting sites (like SourceForge) to include vulnerability scanning. A quick search at freshmeat.net turns up a few FOSS source checkers that could be deployed against the code stored in these repositories (sparse, Audit-Perl). [And there's always the potential for these services to collaborate with the commercial scanning companies (Coverity) and/or DHS.]

The first objection will be that this will show all of those vulnerabilities to every black hat in the world. I shouldn't have to point out that this is something that they could have done themselves with a script. But, to appease this fear, the security scan results could be kept behind password access for project members/owners only.

Beyond that, the hosting service would just need to encourage its use. If scans were not done automatically after every update to the code (conceivably a burden on the hosting service), then there should at least be some sort of effort to force the code maintainers to scan their code before each release. If they elect *not* to scan, then a red flag could be inserted somewhere in the downloads page to notify potential users that this code has not be checked with the hosting service's vulnerability scanning tool.

It's not a perfect solution, but I'd consider it a way to avoid a raft of vulnerabilities now and into the future in one fell swoop.

Jason

Counting vulnerabilities

jengelh — 2007-07-01T14:21:29+00:00

Perhaps Jeff Jones should try a different distribution (in fact, more than one!). Otherwise I'd easily accuse him of specifically taking one of the more ~~broken~~ experimental distributions.

Counting vulnerabilities

giraffedata — 2007-06-29T20:25:37+00:00

The study compares products, so Wireshark on Windows doesn't count because it's not part of the Windows Vista product.

The article says the study doesn't count bugs in "packages" that are on RHEL but not Vista, which I assume means capabilities. And it doesn't say that the Wireshark bugs were counted against RHEL. (Though I could imagine they were if Vista comes with a similar tracing facility).

I believe a much more interesting figure would be number of bugs that were exploited during the period. That would discount bugs in unused code and bugs with no realistic way to exploit. It would be more applicable to the question, "should I do this job with Windows or with Linux"?

Counting vulnerabilities

amikins — 2007-06-28T16:45:45+00:00

I think the allegation here is more to the effect of patching installed versions without any notification that there is an issue or that the system is updating itself.
I haven't seen anything to that effect, but then I avoid learning anything about Vista when I can.. It just infuriates me.

Minimizing packages

jzbiciak — 2007-06-28T16:21:19+00:00

The point is that distro binaries very, very, very seldomly get rebuilt by the distro's users for most other distros. So, if one RHEL4 box is vulnerable in package XYZ due to how package XYZ was configured for RHEL4, then pretty much all RHEL4 users that have that package installed will have that vulnerability on their system.

In contrast, if the same package is available on Gentoo, and it's only vulnerable if you configure XYZ to make use of some other package PDQ, then only Gentoo users who have configured XYZ to use PDQ will be vulnerable. The same goes if you build a bunch of packages from source on any other distro, but only for those packages.

That's the distinction that's being drawn.

The flipside is that there might be some unforseen interaction between packages that a particular Gentoo configuration might expose and that other configurations won't. Since any given configuration will get less scrutiny, there's a higher chance of a given Gentoo box being vulnerable *on the basis of scrutiny alone*. The reality is that that's mostly theoretical, and in general removing features tends to (but does not always) remove vulnerabilities.

Minimizing packages

aglet — 2007-06-28T11:29:12+00:00

Given that dependency definition and management is one of (maybe the most!) important task in preparing a Linux distribution, I have to say I'm disappointed in RedHat. Trying to work out *why* things are installed on the system is always a pain, and the mix of "file", "library" and "package" dependencies makes it worse. It took me a while to figure out what was bloating it out, and how to keep it small, in the end I ended up with this in a kickstart file:

%packages --resolvedeps --nobase
@ core
openssh-server
up2date
# ... extra packages go here...

Including my app support bits & bobs, I wound up with about 180 RPMs or so -- this seems to be about the minimum to have a box that is actually useful.
I found rpmgraph helpful in visualising where things were coming from (I used poster to print the diagram out)
Perhaps it's better with yum (I'm still on RHEL 4), but I also loathe up2date with a passion. RHN is also annoying -- why no interface that I can use from the command line? I hate having to visit a web page to manipulate the subscriptions, I'm seriously considering wrapping it with Mechanize, then exposing it as a RESTful interface...

Minimizing packages

evgeny — 2007-06-28T05:51:22+00:00

> but it is only important if you are a minimalist,

I remember from Debian-3.0 days, when I wanted to install the php-odbc module on a server (no X stuff at all), it attempted to install Gtk+, Corba, etc, plus some of Gnome, only because the ODBC GUI setup utility was linked against all those libs. Call me minimalist...

Counting vulnerabilities

jimparis — 2007-06-28T01:40:47+00:00

Wait a sec. You said there were 52 vulnerabilities in Wireshark. Wireshark can also be installed on Windows. That means Windows had at least 57 vulnerabilities, not 5. Or did they not count it just because it wasn't installed by default? Well, installed or not, the vulnerabilities couldn't have been triggered unless you actually RAN Wireshark.

If you want a fair comparison, you'll have to perform the same tasks on both systems. In most cases, that will involve either:
(1) Not running some of the software on the Linux machine. Even if it's installed, if the program is never executed (and can't be started by an attacker), it doesn't matter from a security point of view.
(2) Or, you need to install the same or equivalent set of software on the Windows machine -- in which case you've just introduced more vulnerabilities.

Minimizing packages

ksmathers — 2007-06-27T21:35:54+00:00

Package dependencies arise from the configuration options selected on the machine where the developer built the system. Source based distributions do not have arbitrary dependencies. If you build an app without another library present, and if it can configure itself for the lack of library, then it does.

That can't happen effectively in a system of compiled packages, but it is only important if you are a minimalist, or if you tend to use packages from different sources (which might therefor have conflicting binary dependencies.)

Minimizing packages

hazelsct — 2007-06-27T20:43:33+00:00

Uh, how does that provide any advantage at all? You can remove .rpms or .debs to minimize vulnerabilities as easily as source-based packages (arguably more easily).

Minimizing packages

HenrikH — 2007-06-27T07:47:03+00:00

You are correct in that USE flags would yield a possibility that one runs programs with unknown holes in them, but then the attacker must also be aware of these unknown holes and also know that you compiled your packages with that very specific USE flags.

Not that it gives a warm and fuzzy feeling, but it would still be some uphill for a potential attacker. And more importantly is that thanks to the wide spread of USE flags a lot of previously unknown bugs will be reported (and hopefully fixed) due to the great variety of the users.

Minimizing packages

flewellyn — 2007-06-26T08:21:02+00:00

I can see how that might theoretically be a problem, but in all honesty, the set of USE flags for a
particular package doesn't have that much crazy variation.

In general, the USE flags tend to correspond to features that can be added or removed via
arguments to ./configure, so the number of variants for any particular package is no larger using
Gentoo than it would be using source-compiled packages that you did the ./configure; make;
make install dance on yourself.

Setting weird compiler optimization flags in the global make.conf, now, there I can see a source
of issues. Of course, so could the Gentoo developers, which is why the docs say not to do that.

Minimizing packages

njs — 2007-06-26T04:01:27+00:00

I know, but that's not very relevant to what I'm talking about. The issue I'm pointing out is that your particular set of programs each compiled with your particular quirky set of USE flags might have some novel security bug that no-one else noticed. Code that lots of people use tends to be well tested and highly scrutinized; weird and rarely used #ifdef'ed code tends to be just the opposite. This thread is advocating using more of the latter sort of code, and thus might actually increase security exposure. Running that same rarely used code across 10 boxes instead of 1 won't affect how much scrutiny it gets, you need lots of people in lots of different situations to get that.

It's hard to know whether this extra risk is important or just theoretical, though, hence my curiosity about quantifying it...

Minimizing packages

flewellyn — 2007-06-25T23:05:00+00:00

But the trade-off is that every box ends up running its own unique mix of code. Security-wise, this is good in that it increases diversity (so it's less likely someone will be able to pwn *all* Gentoo boxes), but reduces scrutiny on the actual code and interactions present on any particular box (so it's more likely that any particular Gentoo box *can* be pwned, which is probably what most sysadmins care about more).

Actually, Gentoo does support binary packages, stored on a repository server that can be used to update the other machines in the cluster. I know of several clusters that do exactly this: one machine builds the updates, and then serves the binaries to the other machines. Obviously this really only works if you have a cluster of identical machines, but that's not unusual in a decent-sized server farm. Or a cluster of workstations.

Minimizing packages

njs — 2007-06-25T22:03:58+00:00

>Gentoo and other source-based distros, to me, seem to have the best solution: build only with what you actually need, and disable features you don't need to use. Thus, excess packages get pruned from the dependency tree and you end up with less clutter (and, not incidentally, fewer possible vulnerabilities).

Hard to know how much extra risk this is in practice, though; that'd make a lovely study, if anyone can figure out how to measure it.

Minimizing packages

nix — 2007-06-25T14:38:49+00:00

It's not even font rasterizing. The thing's getting font metrics and that's all.

(Client-side fonts, guys?)

Counting vulnerabilities

nix — 2007-06-25T14:14:14+00:00

But everyone does that.

To be specific: everyone quietly fixes bugs which *might potentially* be considered security vulnerabilities, if just because they don't realise that they're vulnerabilities at the time they fix them.

You don't need to be nefarious to do that.

(Equally, known vulnerabilities in unreleased or released-as-development versions of free software are often fixed without formal vulnerability announcements, on the basis that anyone who was bitable by this bug is going to be upgrading often anyway, or why else would they be running a development release?)

Counting vulnerabilities

Randakar — 2007-06-25T11:55:43+00:00

Heh. That's not even the whole story; A few days ago I saw a site reporting that Microsoft is SILENTLY FIXING security vulnerabilities; As in, not reporting them at all - just fixing them without telling anyone.*

So any study based on "official vulnerabilities" falls down right there. If the vendor isn't even honest about it there is no way in hell the numbers will tell us anything about the actual security provided when you run their OS.

*) I don't remember where I saw it though - If somebody could post the link that'd be kind.

Minimizing packages

flewellyn — 2007-06-25T07:23:07+00:00

I remember this nonsense from the days when I ran Red Hat Linux, back before Fedora and RHEL.
I would try to install packages like KDE updates, and naturally they would require print
drivers...even though I had no printer. Or, when I wanted to use a GTK package, sometimes the
package in question would ask for GNOME...but I didn't USE GNOME at all! SuSE and other RPM-
based distros seem to have the same issue. I would have hoped they'd have it fixed by now.

DPKG-based distros seem better at allowing you to select "required" versus "recommended"
packages, but even there you have the issue of "Do I really need this, or is it just satisfying a
dependency for a feature I'll never use?"

Gentoo and other source-based distros, to me, seem to have the best solution: build only with
what you actually need, and disable features you don't need to use. Thus, excess packages get
pruned from the dependency tree and you end up with less clutter (and, not incidentally, fewer
possible vulnerabilities).

Everything installs

mattdm — 2007-06-24T12:40:43+00:00

That's kind of a big hammer for a little problem.

You can either do a quick awk command to feed those buildreqs to xargs yum install, or if you do this a lot, you could install mock to automatically ensure it's done right every time.

Everything installs

ronaldcole — 2007-06-23T23:41:40+00:00

You need "everything" installs when you're building and distributing the RPMs of a package that "configure"s itself differently depending on what packages are installed (when the alternative is to be forever satisfying the whims of the "BuildRequires:" gods; who apparently have a rather ravenous appetite for "obscure apps").

Minimizing packages

AJWM — 2007-06-23T18:36:00+00:00

> You need to install Oracle

Oracle is _horrible_ as far as that goes. One of our customers runs some Oracle reporting apps that _require_ X Windows to be running (and with xhost+ to the other servers that use the reporting app). Yeah, we can configure Xvfb to do the job, but it's a pain. All this so the Oracle app can do some font rasterizing or something.

Other "management" apps are just as bad, requiring a graphic or browser interface (hello, SANsurfer) to do anything useful. Try that when your server is in a private (10.x.x.x) LAN that requires a couple of intervening ssh servers to get to. Yes, it can be done by setting up the right ssh tunnelling, but it's a pain in the butt compared to command line (and scriptable!) access.

Even amongst the apps in a single distro that DO permit an alternate command line mode, it's inconsistent. up2date-nox vs printconf-tui, anyone?

Okay, rant off.

Minimizing packages

evgeny — 2007-06-23T12:26:44+00:00

Solution: Gentoo (or like, source-based distros).

Minimizing packages

mattdm — 2007-06-23T10:29:58+00:00

Sure, but that's a special case, and in that case, it's no surprise that something in your installed stack of apps requires sound libraries.

Minimizing packages

pcampe — 2007-06-23T09:00:48+00:00

>But why are you putting *any* desktop environment on your server?

You need to install Oracle (I haven't done this in the last 2 years, so maybe I'm wrong, but for sure there are server programs that you could install/use/monitor only with a graphical interface, and for security, licenses, performance reasons you can't work from a remote workstation, eve if "remote" means "in the same LAN").

Microsoft better at patching XP than Vista (security.itworld.com)

HalloBaum — 2007-06-23T08:44:24+00:00

http://security.itworld.com/4347/070622vistapatching/page...

Minimizing packages

chaneau — 2007-06-23T04:35:13+00:00

There are legitimate reasons to put a desktop environment on a server, here, for example, I'm in the process of replacing all the PC with thin clients.

Minimizing packages

mattdm — 2007-06-23T03:02:25+00:00

But why are you putting *any* desktop environment on your server?

Everything installs

arjan — 2007-06-23T01:16:08+00:00

This is why distros like Fedora want to get rid of the "everything" install options... if people are installing mostly only the pieces they need (at least in broader groups), the exposure of the user becomes a lot lower. From a distro pov, it reduces the risk of a hole in some of the more obscure apps to affect a large numbers of users.

well one of the reasons at least (the otherone probably is "what does everything mean in the merged Fedora case")

Minimizing packages

jmorris42 — 2007-06-23T01:10:58+00:00

> reminder that we should be aggressively weeding newly installed systems for unused software.

Always sound advice, but becoming harder to do. The minimum RPM transaction to bootstrap RHEL4's packageset is 75. That gets you rpm installed and not much else. Drop in the next 84 package interdependent bundle and you will have vim-minimal available. There are literally hundreds of packages on a typical system that aren't really being used but can't be removed because some other little used package has a tenuous relationship with it.

Examples:

Servers with no sound capability that must have alsa-lib because gnome-libs and kdebase ultimately depend on it.

Well a RHEL4 server certainly doesn't need ogg support, right? If you stick to the recommended GNOME desktop it doesn't, but kdebase does depend on it.

Don't have a Palm(tm) device? Well you need to keep it installed if you use Evolution.

Or check this dependency trail. On a system running only GNOME, certainly there shouldn't be a need to keep Qt hanging around right? Wrong. Qt is needed by arts, with is needed by gstreamer-plugins -> gnome-applets -> gnome-media -> nautilus-media -> rhythmbox -> gnome-volume-manager -> gnome-session-manager. So remove Qt and by the time the cascade settles gnome is toast.

Counting vulnerabilities

dmarti — 2007-06-22T22:41:53+00:00

"a reminder that we can be doing better" for developers and distribution maintainers, and for everyone else a reminder that we should be aggressively weeding newly installed systems for unused software. Your package manager's remove command, "ls /etc/init.d", and "nmap localhost" are your friends.

Counting vulnerabilities

pr1268 — 2007-06-22T22:30:42+00:00

> It would also be interesting to make a comparison of 180,360,720, and 1440 days.

Or even easier, compare Windows XP to RHEL 4 in the above time frames.

Counting vulnerabilities

smoogen — 2007-06-22T21:37:17+00:00

Another big issue that seems to be lost is that outside researchers can not see the Vista code in large enough numbers to really evaluate the code. A lot of the same bugs could be there but may only be known internally or by the usual BlackMarket crack organization.

It would be interesting to make a Vista comparible OS. Just equivalent packages what MS ships in XP and Vista... and then have everything as 'extra-packs'

It would also be interesting to make a comparison of 180,360,720, and 1440 days.