Weekly Edition
Return to the Security pageSponsored link
Serve your customers, not your servers, with VERIO Linux VPS. Full-access test-drive here.
| Weekly edition | Kernel | Security | Distributions | Search |
| Archives | Calendar | Subscribe | Write for LWN | LWN.net FAQ |
A look at the MS-SQL worm
| From: | "Karsten M. Self" <kmself@ix.netcom.com> | |
| To: | linux-elitists@zgp.org | |
| Subject: | Re: [linux-elitists] MS SQL WORM IS DESTROYING INTERNET BLOCK PORT 1434! | |
| Date: | Wed, 29 Jan 2003 23:17:57 +0000 |
on Sat, Jan 25, 2003 at 01:26:39PM -0800, Don Marti (dmarti@zgp.org) wrote:
> begin Michael Bacarella quotation of Sat, Jan 25, 2003 at 02:11:41AM -0500:
>
> > All admins with access to routers should block port 1434 (ms-sql-m)!
>
> Anybody who has _any_ relational database server directly connected
> to the Internet please save some of whatever you're smoking for me.
A few further points on this issue.
Looking over the BUGTRAQ and NANOG lists, a few trends start to emerge.
Apologies if this is fundamental knowledge -- if I'm duplicating
well-known summaries, please post links as followup as I'm unaware of
them.
- Attacks worldwide appear to start at 05:29:30 UCT, give or take a
few seconds. The launch of this attack *does* appear to be highly
coordinated. I've seen reports of up to several minutes later, but
nothing earlier.
- University of Dartmouth registers 10k independent sources within the
first 30 minutes of the attack, and a peak of 16k independent
sources, speaking for extremely rapid propagation. Early
propagation appears to be from many widely dispersed sites, though
large colo facilities (e.g.: Hurricane Electric) appear in several
reports. Other references speak of ~19k distinct sources. Whether
or not this represents the maximum scope of the attack isn't clear,
but let's presume that the total number of infected hosts were <
100k. Current estimates of total Internet nodes tend to range in
the 200m - 400m range, though I don't have good numbers on this.
I'd be interested in same if anyone has a reference.
- Another number I've been pulling out of /dev/ass (mostly because
nobody's provided anything more useful) is that there are 10m Win2K
systems in existence.
- This means that the infected hosts were on the order of 1% of all
potential hosts. That is, Microsoft users were attaining a 99%
patch and/or secure rate of systems publicly visible to the worm.
This is a pretty good compliance rate. It was also wholly
inadequate in preventing this attack.
- Several NANOG sources report prior scans of the 1434 port across
systems earlier in January, particularly on the 16th and 19th. This
may have been preparatory work for the sort of rapid-propagation
exploit attack that was hypothesized last summer.
- The MS SQL engine is incorporated into a large number of MSFT
products. While not absolving guilt, it does help to explain why
so many exposed systems existed. The overhead of knowing what
services exist on a given system, and of keeping these systems
patched, increases consequently.
http://www.microsoft.com/technet/security/MSDEapps.asp
- In balance, the level of infection for this attack was *small*, not
large. The effects were disproportionate to the number of directly
infected systems. Calling this the result of a widespread software
monoculture may not be appropriate (IMO it is, for complex reasons,
but that's a longer discussion). A similar vulnerability in a
widely deployed free software utility could produce similar results,
and the GNU/Linux & free software communities shouldn't enjoy
excessive schadenfreude over this incident.
I recall (but can't locate) a reference, possibly following the
Mindcraft Apache / IIS rigged shootout, in which it was observed that
raw webserving capacity was a poor performance metric, as a score or
so Sun workstations would be more than sufficient to flood major
Internet backbone links.
While it's fun (however unsporting) to blast away at Microsoft for its
security deficiencies, IMO the free software world should view the
Sapphire / Slammer worm as more a cautionary tale. This is the sort of
attack which _could_ potentially hit GNU/Linux or another 'Nix. I feel
that the likelihood is lower than that for legacy MS Windows, though
there are a large number of likely poorly maintained GNU/Linux and other
'Nix systems live on the Net.
Smugness kills.
Peace.
--
Karsten M. Self <kmself@ix.netcom.com> http://kmself.home.netcom.com/
What Part of "Gestalt" don't you understand?
The truth behind the H-1B indentured servant scam:
http://heather.cs.ucdavis.edu/itaa.real.html
http://www.zazona.com/ShameH1B/
_______________________________________________
linux-elitists
http://zgp.org/mailman/listinfo/linux-elitists
(Log in to post comments)
Don't ignore network level filtering
Posted Jan 30, 2003 0:14 UTC (Thu) by jneves (guest, #2859) [Link]
That is, Microsoft users were attaining a 99% patch and/or secure rate of systems publicly visible to the worm. This is a pretty good compliance rate.
This number is ignoring the way a lot of the machines were protected. For instance, in Portugal, I know for a fact that the second biggest ISP refused all traffic to port 1434. This allowed several thousands of companies and vulnerable machines to avoid the attack.
I believe this to be a direct result of Code Red, nimda and others. ISPs developed ways of reacting to worms and distributed attacks. I've seen how well it worked here (by "well" I mean that damage was averted) and I think that a lot more ISPs and some big companies did the same: filtered the attack at the network level.
This means that, if only 1% of the potential machines were affected, it has as much to do with Microsoft and its users as to how network administrators deal with distributed and/or worm attacks.
Don't ignore network level filtering
Posted Jan 30, 2003 10:05 UTC (Thu) by beejaybee (subscriber, #1581) [Link]
Yeah. Nessus has been identifying the expolit for ages; despite at least three rounds of warnings, there were still some systems at my employer's site which weren't patched. I would estimate around 50% of the (not very many) hosts running MS-SQL-S were patched.We had the foresight to filter incoming UDP on almost all ports at our site router and therefore were not directly hit by the outbreak.
Another point here - it's obvious that a high proportion of the sysadmins of the hosts running MS-SQL-S were not even aware that the service was running. Disabling services that aren't essential is as much a part of securing a system as keeping up to date with patches. This applies to _all_ operating systems; many out-of-the-box linux systems are also running services they don't need to; Solaris systems seem to be totally infested with a huge raft of RPC services, many of which are a complete mystery to almost everyone!
A look at the MS-SQL worm
Posted Jan 30, 2003 0:35 UTC (Thu) by ihutch (guest, #9352) [Link]
What a misuse of statistics! The fraction of *MS-SQL servers* patched is1% divided by the fraction of Win2k systems that runs a such server. That
fraction must be very small. So probably a large fraction of all MS-SQL
servers was infected.
Conclusion: just the opposite -- fractional patch rate of MS-SQl servers
was probably feeble.
A look at the MS-SQL worm
Posted Jan 30, 2003 17:18 UTC (Thu) by slamb (guest, #1070) [Link]
The fraction of *MS-SQL servers* patched is 1% divided by the fraction of Win2k systems that runs a such server. That fraction must be very small. So probably a large fraction of all MS-SQL servers was infected.... divided again by fraction that are open to the Internet (that is, not only connected at all but not firewalled). Pretty soon this is going to start looking like the Drake equation.
Not so fast
Posted Jan 30, 2003 0:56 UTC (Thu) by ncm (subscriber, #165) [Link]
Karsten wrote:... infected hosts were on the order of 1% of all potential hosts ... Microsoft users were attaining a 99% patch and/or secure rate ... of systems...Sorry, this is a fundamentally misleading number. It's distinctly abnormal for an SQL port to be open to the outside world; it's a back-office function. Therefore, that 1% figure identifies a level of gross incompetence beyond all analysis. The 99% were not patched or secured, they were just out of the line of fire.
It would be hard to imagine somebody competent enough to apply the patch, but still leave the port exposed, so that 1% figure must represent substantially all of the exposed hosts, unpatched. In other words, we can't conclude anything about the number that were patched because those were also the ones behind firewalls.
It would be hard to justify not firing someone who was responsible for any of the hosts involved.
Justify firings carefully
Posted Jan 30, 2003 14:47 UTC (Thu) by utoddl (subscriber, #1232) [Link]
It would be hard to justify not firing someone who was responsible for any of the hosts involved.Right. Let's just list the benefits to the org that such firings bring:
- We were short-handed before, now we're spread even thinner.
- The person with the most valuable, recent, first-hand, hard-earned experience -- i.e., the one least likely to make such a mistake again -- just left the building with his personal possessions in a box.
- The remaining staff lives in fear and dread, knowing that all people make mistakes, and their next one may cost them their jobs, or...
- their coworker's mistake may result in their workload suddenly increasing with likely no increase in pay.
Question: Suppose the unpatched service had been discovered a week before the worm. Would you fire the admin on the spot even though no damage had resulted? Do you fire everybody who makes a mistake, or just the ones whose mistakes become too visible?
Not so fast
Posted Jan 30, 2003 15:47 UTC (Thu) by sphealey (guest, #1028) [Link]
It would be hard to justify not firing someone who was responsible for any of the hosts involved.I see a lot of statements like this after every attack. I would suggest keeping in mind the old proverb that the winner of a game of chess is the person who makes the second-to-last mistake. Even Kasparov makes mistakes. Sooner or later you will too.
sPh
Perhaps - perhaps not
Posted Jan 30, 2003 17:17 UTC (Thu) by sphealey (guest, #1028) [Link]
I would suggest reading Russ Cooper's take on this situation over at NTBugTraq. Seems that even an ordinarily competent and watchful MSSQL admin might have been caught by this one.sPh
A look at the MS-SQL worm
Posted Jan 30, 2003 2:09 UTC (Thu) by JoeBuck (subscriber, #2330) [Link]
I think an argument over whether Karsten's numbers are right might be missing an important point: while he may not be right in his suggestion that in this case 99% of the people applied the patch and we still had major damage from the worm (I, for one, doubt this), I could easily conceive of future cases where this might be true.
Consider this scenario: a remote exploit is discovered in Apache. Patches are quickly produced, but only 99% of people running Apache apply the patch by the time that a worm is deployed. The worm is designed to generate so much traffic as to completely saturate a number of crucial backbone links, and it's smartly written (that is, it closes the security hole behind itself, so that the same hole can't be used to clear the infection; it stores itself on disk, so that a reboot doesn't cure it). Blocking the SQL port is one thing; blocking port 80 all over the net basically turns off the web, generating massive economic disruption.
I think that the only way out of this mess is to start to think about techniques that can yield formal proofs that certain flaws are not present. This is probably only feasible if applied to very small pieces of code, but perhaps software can be architected so that the volume of code that is ever exposed to untrusted data is kept very small, so that it can be proved to be free of buffer overflows, vulnerability to cross-site scripting and similar attacks, and race conditions. OpenBSD-style code auditing won't do the trick: to have hope of proving safety properties you pretty much have to design the code and the proof together.
Such techniques don't have to be incompatible with C (though C does make things harder). Some form of assume-guarantee approach can be used to determine preconditions and postconditions for each function, and given a meticulous effort it might be provable that a given daemon contains no buffer overflows, or that cross-site scripting is impossible, or that it's not possible to get to a shell. I would expect that it will take 100 times as long to write such code, but it could be argued that this most be done to ensure the continued operation of the net (so maybe government funding could be obtained).
A look at the MS-SQL worm
Posted Jan 30, 2003 9:21 UTC (Thu) by skellba (guest, #8043) [Link]
formal proof of correctness of programs is pratically impossible: the task is too big to be done. It would be better to not use programming languages like C or C++ which are inherently prone to buffer overflows and memory overwrites.What me really concerns is that there are so many database-servers obviously not behind a firewall: this is definitiv a problem of understanding network security and missing knowledge about Microsoft products.
A look at the MS-SQL worm
Posted Jan 30, 2003 10:29 UTC (Thu) by libra (guest, #2515) [Link]
-> "missing knowledge about Microsoft products"But who should care about that? It is not a problem of knowing Microsoft or Linux or Solaris or whatever. The problem is understanding what you do, this require knowledge about something more basic : TCP/IP, Filtering, 3-tier architecture etc...
I personnaly feel as confident with Microsoft or Linux not because I know one or the other extremly well, but because I can analyse and try to understand what I do. This is a lot more difficult on Windows because everything is hidden in metabases, registry, secret entries and behind frustrating GUIs but for the basics I can still manage what I need.
So the problem is not knowledge about Microsoft to my opinion, it is basic knowledge and intelligence. But of course how do you want people to devellop both when they lose their time with bugs and patches to correct them?
A look at the MS-SQL worm
Posted Jan 30, 2003 11:11 UTC (Thu) by NAR (subscriber, #1313) [Link]
It would be better to not use programming languages like C or C++ which are inherently prone to buffer overflows and memory overwrites.I heard once a guy, who proudly announced his 100 line long secure webserver written in perl, and asked folks to audit it. Of course, it is not that hard to audit 100 lines of code, even if it's written in perl :-) But don't forget, that the C code of the perl interpreter should be audited also, and that is not 100 lines long...
Language versus application security
Posted Jan 30, 2003 12:36 UTC (Thu) by copsewood (subscriber, #199) [Link]
I think the people who design and implement very high-level programming languages (e.g. Perl, Python, Java, SQL, - 'C' is too low-level) are somewhat more likely to be able to write interpreters secured against buffer overflows etc. than the people likely to be writing network-based applications which use these tools, due to the likely differences in programming knowledge and experience.Hence it is likely that an application designer needs less security knowledge to design a reasonably robust web application in Perl (which does the memory management housekeeping within the language) than in 'C' (which manages memory within the application). Not only that but the ISP hosting virtual domains is likely to want to patch insecurities discovered in the language runtimes much quicker than most of the domain owners will patch insecure web applications.
Language versus application security
Posted Jan 30, 2003 18:21 UTC (Thu) by JoeBuck (subscriber, #2330) [Link]
Perl doesn't have buffer overflows, but Perl-based web applications have had tons of security holes, mostly caused by not sufficiently checking user-supplied data (../../.. in paths, cross-site scripting, and the like).
Language versus application security
Posted Jan 31, 2003 0:54 UTC (Fri) by copsewood (subscriber, #199) [Link]
Yep. The approach of deciding what valid data should look like and excluding everything else is similar to the default-deny approach to firewall setups. This can reduce functionality slightly or slightly increases the effort of getting added functionality secure, but its probably much less hassle than trying specifically to exclude what is considered dangerous when you can only ever have limited knowledge of what might be exploitable in future.
A look at the MS-SQL worm
Posted Jan 30, 2003 12:11 UTC (Thu) by philips (guest, #937) [Link]
> formal proof of correctness of programs is pratically impossible:> the task is too big to be done. It would be better to not use
> programming languages like C or C++ which are inherently prone to
> buffer overflows and memory overwrites.
Take a look at:
http://downloads.securityfocus.com/library/ncsc-bblue.txt
B1 Mandatory Protection includes:
----- quote start -----
Design Documentation remains the same as C2, but also describes the
security policy model (either formally, i.e., mathematically, or
informally, i.e., in English) and how the TCB implements this model.
----- quote end -----
In other words, you have to open source at some degree your application
to conform to even to B1 class. You should specify how secure your
solution is and how did you achived this.
Another language buys you nothing - implementation & design flaws is
the point. Rember that 75% of errors are made at design time - even
before you started coding ;)
MS-SQL worm - open source not needed for B1 rating
Posted Jan 30, 2003 20:27 UTC (Thu) by wittenberg (guest, #4473) [Link]
You don't need to open source to get B1 certification. Ten years ago I worked on a product that was to have been certified A1. We had to show our code to NCSC, but only under a non-disclosure agreement. So you need other people to look at the code, but you don't let everyone look at it.
A look at the MS-SQL worm
Posted Jan 30, 2003 11:07 UTC (Thu) by NAR (subscriber, #1313) [Link]
I think that the only way out of this mess is to start to think about techniques that can yield formal proofs that certain flaws are not present.Don't forget, that formal proofs work on only formal programs. As soon as you start to type in a formally proved program, flaws can creep in, and no formal proof can protect from this. You still need testing and code audit. BTW the formal proving methods I studied at the university usually concentrated on what the program should do with the correct input. They didn't deal with incorrect inputs (i.e. it was enough to prove, that the program works on the specified inputs, noone cared, what happens with incorrect input, because it was out of scope), but most flaws are exploited only by incorrect input.
The other thing is that formal proving is an extraordinarily timeconsuming process, even on small, but parallel programs, and it needs strong mathematical tools (i.e. no high school kid will do it). I think that your "100 times" assumption is quite low, but imagine, if you have to wait 200 years for the next stable Linux kernel instead of 2 years.
A look at the MS-SQL worm
Posted Jan 30, 2003 15:11 UTC (Thu) by utoddl (subscriber, #1232) [Link]
BTW the formal proving methods I studied at the university usually concentrated on what the program should do with the correct input. They didn't deal with incorrect inputs (i.e. it was enough to prove, that the program works on the specified inputs, noone cared, what happens with incorrect input, because it was out of scope)[way off topic, but...] One of the most interesting assignments in my second programming class spelled out what constituted correct input and that the program should handle incorrect input. The instructor was going to run each of our programs against an unspecified input stream, and we would be graded in part on how well we handled bad input.
We were still using punch cards back then, and there were plenty of discarded punch cards lying around -- old data sets, discarded (literally!) lines from old programs, etc. The input the instructor chose was... handfulls of old punch cards he pulled from the trash in the key punch room. Snippets of FORTRAN, PL/I, raw data, arbitrary text, etc. made various programs produce, er, interesting results. It was a real eye opener to most of us; we had though "bad input" might mean something not in the right column, or a keyword slightly mispelled. Very few of the programs could handle garbage -- literal garbage -- as input.
Sorry for the Old Man tale. You kids go back to your discussion...
A look at the MS-SQL worm
Posted Jan 30, 2003 18:33 UTC (Thu) by JoeBuck (subscriber, #2330) [Link]
I come from the electronic design automation world; when I speak of formal proof, I'm talking about the real thing, and it most certainly is possible to do, though it requires considerable expertise and it will require more research to make it scale.
Of course I know that high school students won't do this; it will take Ph.D's. And available proofs will refer to particular library routines, at most; it might be possible to prove that a given JPEG-decoding routine contains no buffer overflows or memory access errors, given a correct compiler for the language it is written in (even if that language is C).
Of course, compilers can have bugs. But we'll still be better off than we are today. And in the meantime, semi-formal techniques like the Stanford Checker are turning up hundreds of bugs in existing code. The point is that, for security testing, debuggers are no good, as you can only run a debugger with an input that you have thought of. An attacker will construct an input you never thought of to try to break your code.
The software world is way behind the hardware world on this one: after the Pentium bug cost Intel half a billion dollars, they've thrown tons of money into formal verification. Register-transfer-level to gate equivalence checking by formal techniques is well established, for example, and it is standard practice to mathematically prove that cache coherency protocols or floating point units are designed correctly.
A look at the MS-SQL worm
Posted Feb 1, 2003 7:40 UTC (Sat) by goonie (subscriber, #4252) [Link]
The point is that, for security testing, debuggers are no good, as you can only run a debugger with an input that you have thought of. An attacker will construct an input you never thought of to try to break your code.
Two possible approaches:
- get other peoplpe to think up inputs - seperate the programmers and the testers, in other words. This is a very basic software engineering practice.
- Develop systematic methods for generating test cases to break your software (in the specific case of security testing, break into your software). The state of the art in this area is somewhat primitive.
On the number of Windows 2000 servers
Posted Jan 30, 2003 3:34 UTC (Thu) by ribbo (guest, #2400) [Link]
If there are 10m windows 2000 servers estimated to be on the internet, hence the estimate of 1 percent infection. I think its a better measure to ask how many of those 10m hosts were running SQL Server or MSDE, I would have thought probably only 10 percent. When in comes to IIS, which comes as is generally installed as default with a W2K system I think a much lower number (say 10%) would have a SQL Server derived product, hence the actual number of attacks is much higher (more like 10%).
Patching is bad!
Posted Jan 30, 2003 9:26 UTC (Thu) by libra (guest, #2515) [Link]
For my part I'm totally opposed to the idea of using patch. If I use a patch when installing a product that means that I have installed something that is bad a one time during the process. Certainly among the SQL-server that where infected some where installed less than 6 month ago, but from an incorrect cdrom which is anyway the reference cdrom for such an installation.On this topic the big difference here between proprietary programs and open-source programs is that during the first install with open-source you can immediatly install the right version (at the time of installation). With patches on proprietary programs you must first install something buggy, and then try to correct it, and that gives far less predictible results in the end, and a lot more work too.
Of course one can argue that when there is a bug in some open-source program there is also a requirement for applying some correction, put it is not a patch, with a rpm or whatever other method you can update : that is completely replace what has become wrong by something completely good. And the next time if you want to install the product you won't have to install the bad one before the good one if you want.
Really patches are hawfull things, what we need is software without bugs, and with proprietary software it is difficult because vendors don't want to offer "prepatched" (that is corrected) version of their products by fear some people would download them without licenses. I let you appreciate the absurdity of this situation.
For those having an MSDN at hand just count the number of "product cdrom" against "patches cdrom" in the server section. You will discover that Microsoft offers a lot more patches than real (usefull?) products. Finally you may understand that the commercial offer of Microsoft is bigger on the front of bugs than on the front of business. I hope you will enjoy this revelation.
Patching is bad!
Posted Jan 30, 2003 11:22 UTC (Thu) by NAR (subscriber, #1313) [Link]
I have installed something that is bad a one time during the process.I think, it is more a joke, than a real issue. First of all, a really really really paranoid administrator can first deconnect the machine from the network, install the original ("flawed") version, then install the patch, then reconnect the machine, so in this case, the flawed version is not exposed to malicious user at all. Secondly, (at least our) costumers do get upgrades, not patches, even though our products are proprietary. And least, but not least, who cares (except you:-), if you install a patch, or install an upgrade? The end result, the exposure time to attacks, etc. are the exactly same.
Patching is bad!
Posted Jan 30, 2003 15:48 UTC (Thu) by libra (guest, #2515) [Link]
Obviously you have never seen a system dll badly replaced during a patch because another process locks it, or it has been duplicated in the dllcache, or whatever other nasty trick.I've seen such things happen already. But I must say that it tends to happen more often with Microsoft products than other products develloped for Windows (except those of Microsoft partners). Maybe because they know Windows too well to use common sense when making a development.
Flashback
Posted Jan 30, 2003 15:29 UTC (Thu) by cjdewey (guest, #5128) [Link]
Did this article remind anyone else of _A Fire Upon the Deep_ by Vernor Vinge?
A look at the MS-SQL worm
Posted Jan 30, 2003 20:59 UTC (Thu) by dbhost (guest, #3461) [Link]
This brings to mind the Sysadmin's mantra, security, security, security. When a weakness is found, and a patch released that can be trusted, apply that patch. GNU/Linux and other nixes aren't immune to attack and we certainly cannot afford to be smug thinking we are above this insanity.
Which conflicts with...
Posted Jan 31, 2003 14:07 UTC (Fri) by sphealey (guest, #1028) [Link]
"Security security security" unfortunately conflicts with "test test test". The cannonical example being Windows NT 4 Service Pack 6, which (i) contained numerous critical security patches (b) broke Lotus Notes servers. Microsoft patches are notorious for breaking existing apps, but it happens with other vendors' software/patches as well. Now what does the diligent sysadmin do?sPh
Which conflicts with...
Posted Feb 3, 2003 23:30 UTC (Mon) by sphealey (guest, #1028) [Link]
Oops - Microsoft have just withdrawn a critical security patch because it "may cause Windows NT to fail". Hope you didn't install it. But wait: if you didn't install it, you were vulnerable... But if you did install it...sPh
A look at the MS-SQL worm
Posted Feb 2, 2003 3:31 UTC (Sun) by wolfrider (guest, #3105) [Link]
--Well said. "If you think you're standing firm, be careful that you don't fall."http://perso.wanadoo.fr/john.kathie/Micah%203.5-12.html
( http://bible.gospelcom.net/cgi-bin/bible?language=english&version=NIV&passage=1+Cor+10 )