LWN: Comments on "Vulnerability hoarding and Wcry"

Takeaways from a global malware disaster

raven667 — Mon, 29 May 2017 04:40:59 +0000

Not only can you put off being unhappy into the future, where it might be someone elses problem, but depending on the failure you have an opportunity to come in on your white horse and save the day, proving your worthiness to an organization that doesn't understand the failure was preventable. Those are powerful motivations for professional malpractice.

Vulnerability hoarding and Wcry

flussence — Sat, 27 May 2017 14:49:25 +0000

And that's why RS232 expansion cards for modern buses typically cost as much as a mid-range GPU with a billion transistors... taking advantage of desperate users.

Takeaways from a global malware disaster

ghane — Sat, 27 May 2017 03:49:34 +0000

> This attitude always betrays an organization that is exposed to a lot of risk to continuity of operations, if they can't take the downtime for patching, they are really going to be unhappy when their system is totally dead due to preventable predictable failures.

Yes, but if I _have_ to be unhappy, why not be unhappy at some point in the future? Why be unhappy now?

(And all reboots are dangerous, you never know what unsaved configs you are running)

It is not only CEO who have short-term objectives, it is Sys Adms too.

--
Sanjeev, who is a smoker. Not died even once yet. So there!

Vulnerability hoarding and Wcry

Wol — Thu, 25 May 2017 20:35:24 +0000

> It's completely unacceptable for some medical bit of kit costing millions that might be running for decades to be designed with no way to update its embedded OS or eventually upgrade it to something supportable. (WinXP in MRI scanners?)

Shades of the company that complained that, with all the PCs moving over to USB, it was getting harder and harder to get computers with RS232 ports to drive the peripherals. "Well, get new peripherals, then" was the response of the guy they were complaining to (it might have been Bill Gates, spec'ing that new PCs should have USB not serial).

Problem was the guy complaining had a LOT of said peripherals, at typically $250K or more each ...

Cheers,
Wol

Takeaways from a global malware disaster

Wol — Thu, 25 May 2017 20:19:51 +0000

And if it's for one of the (many) apps that you can't delete and don't use?

Part of the trouble is quite likely all of the crapware on the system that shouldn't be there!

Cheers,
Wol

microsoft where sitting on a patch for this since march

coolhandluke — Tue, 23 May 2017 21:20:24 +0000

Microsoft *did* release a patch for Windows 7. The problem was that folks didn't install it.

Takeaways from a global malware disaster

raven667 — Tue, 23 May 2017 03:47:03 +0000

This attitude always betrays an organization that is exposed to a lot of risk to continuity of operations, if they can't take the downtime for patching, they are really going to be unhappy when their system is totally dead due to preventable predictable failures.

Takeaways from a global malware disaster

ringerc — Tue, 23 May 2017 03:19:33 +0000

Often we just can't get them to schedule downtime, or provide the time and resources for a low- or zero-downtime upgrade/update.

Takeaways from a global malware disaster

bandrami — Mon, 22 May 2017 13:22:43 +0000

Even from a security standpoint it can be a question of picking your poison; Lenny users missed out on Heartbleed entirely, e.g.

Takeaways from a global malware disaster

flussence — Sun, 21 May 2017 16:45:37 +0000

I think the multi-month siege of increasingly hostile Windows 10 upgrade attempts has left most surviving Windows 7 PCs in a state of “updates forcibly disabled, to the maximum extent possible”. Although the initial danger passed most of those people wouldn't have bothered to re-enable updates, and that's probably the main reason this worm hit as hard as it did.

microsoft where sitting on a patch for this since march

johnjones — Sun, 21 May 2017 07:28:11 +0000

It specifically abuses a bug designated MS17-010 that Redmond patched in March for modern versions of Windows

so basically windows 7 did not get the update and we have this problem... the only organisation that is a problem is Microsoft
they had an amazing Press team that blamed others...

Takeaways from a global malware disaster

nix — Sat, 20 May 2017 21:59:50 +0000

I think technically aware users see security updates as an annoyance. The rest have no idea Windows Update exists at all, overlook the nag notifications (they don't pop up for very long and are easy to overlook, particularly if you're not at the machine when they appear) and think "why is my machine rebooting on its own? is it broken?". I know several people who took machines to the shop for "repair" (and were charged for it) when the only problem was Windows Update autoreboots.

Takeaways from a global malware disaster

gezza — Sat, 20 May 2017 17:09:48 +0000

> Part of folks' reluctance to do this is that these "updates" routinely come with intentional side effects -- An extreme example is Windows 10. Or how Netflix will now refuse to install on unlocked/rooted devices.

Android was mentioned in the article. For me, when an update comes with a demand for new
access rights, there is an immediate dilemma - do I apply it or not?

So yes, I should invest the time in Cyanogen-Mod, and fine tune the access anything has. Who really has the time for that, on every system they use?

Takeaways from a global malware disaster

biergaizi — Sat, 20 May 2017 13:35:46 +0000

Most pirated copies of Windows have their activation system cracked, running Windows Update is not a problem at all.

I believe most Windows users see security updates as annoyance, even if Windows Update itself is reliable. Patches pop up every several days and strongly pushes the users to update, and users who don't understand the value of security updates just hate it... Large organizations also disable updates to ensure the consistency of their system, and prevents updates to interrupt their workflow.

User does not own the vulnerable software

giraffedata — Sat, 20 May 2017 00:23:42 +0000

I have a question: the software does not belong to the user, it's licensed for use by Microsoft, right? So how come this is not Microsoft's problem?

Why would that make it Microsoft's problem?

And which problem are you talking about - the problem that people had to pay ransom and/or were deprived of their computers for a while, or the problem that people need to update their Windows?

Takeaways from a global malware disaster

giraffedata — Sat, 20 May 2017 00:11:23 +0000

Part of folks' reluctance to do this is that these "updates" routinely come with intentional side effects

And another part is the unintentional side effects - bugs.

I decided a while ago not to apply updates as they come out. I believe my risk of breaking something exceeds my risk of being hacked. I'd love to see a scientific study of that; my gut feeling is just based on the fact that I haven't been hacked yet and I've broken my system, sometimes very badly, dozens of times by applying updates.

The worst breakage-by-update that has happened to me so far is from the recent trend in browser publishers to discontinue the ability to use insecure communication protocols. Unfortunately for me, there are a bunch of servers I need to access that use these protocols. I was naive when I updated those browsers, not realizing backward compatibility is not as sacred as it used to be.

The only way to eliminate this update dilemma is to have finer grained updates through smaller software modules. If you didn't have to install thousands of kernel or browser updates to get one security fix, applying security fixes wouldn't be as risky.

Takeaways from a global malware disaster

excors — Fri, 19 May 2017 14:09:10 +0000

To some extent that's the same thing, since vulnerable computers outside the US can be turned into botnets and used to attack the US, or can be used by non-nice people to steal money (as with ransomware, Bitcoin mining malware, etc) that might be used to fund physical attacks on the US. In the same way that you might want to, say, sabotage Iran's nuclear centrifuges to slow down their development of weapons that could hurt the US's allies or fall into the hands of terrorists that want to hurt the US directly, you might equally want to secure the world's computers to reduce your enemies' abilities to hurt you. And it's the same way that foreign aid is important for national security, since a nation is more secure when it's part of a more stable world.

And for the US in particular, cyber warfare almost completely bypasses their conventional military advantage. A group of smart motivated hackers in North Korea with a few million dollars to buy zero-day vulnerabilities could cause as much damage to US computers as the US could to theirs. Better to eliminate that threat globally by improving security for everyone, so that warfare has to instead be done with missiles and trillion-dollar planes where the US has a big lead.

Takeaways from a global malware disaster

NAR — Fri, 19 May 2017 12:05:07 +0000

"to really make the world more secure, the NSA would need to search for vulnerabilities and _publish_ them."

I'm not sure it's their job to make the world secure. Making the US government computer network more secure is part of their job, but for example making random computers in the Brussels neighborhood of Molenbeek more secure interferes with their job.

Takeaways from a global malware disaster

gracinet — Fri, 19 May 2017 10:12:06 +0000

> Now our admins clone a machine, install all of the updates and test it, and then they can almost transparently shut down the old copy and replace it.

Yes, of course, but… lots of applications around there have circular logical dependencies, such as having their own URL encoded in the database, the potential to send thousands of emails (again, specified in the DB) once their scheduled tasks fire, data too big to swap that easily, etc. In short, they aren't designed for easing up moving back and forth staging and production (of course some are). While you and I would certainly call that a design fault, that kind of stuff is often not in the selling criteria, as it's too much of a technician's concern.

Anyway, in the cases I was referring to, with a classic customer/developer/sysadmin separation of concerns, the admins just don't know what the application does, how to test it and with whom to share the results. Unit and integration tests can help, obviously, but they are a developer thing.

Another anecdote: it's been a while now (6 years), but once I was in a datacenter, wearing the developer hat, with the functional guy and the sysadmin for a major upgrade of a web application (first in years) and it really pleased the admin to witness the functional guy actually testing the application. It was the first time he'd even seen it in a browser, and actually we had some prior work on the firewalls to make it even possible to access it from the datacenter network. The thing that drove that exceptional gathering was the perception from management that the upgrade was both necessary and very risky.

What we can hope for is that this is mostly a thing of the past, with the dev/ops, microservices, release and test often mantras taking slowly over, but we shouldn't underestimate the human communication gaps that lie behind all this, if we don't want to end up with the same problems just spelled differently.

More broadly, non technical people in the IT/web business don't trust us for managing priorities : they fear that we drown into our own, useless, generated pile of work that they don't understand at all. In their discharge, I won't swear that never happens. We have to understand their point of view and provide better, more understandable feedback. It's so easy to just have contempt on them when one is the only one around to understand what's at stake, and that's why I've been advocating for a while that developers should have project management experience and vice-versa.

As for customers outside the IT business, I've been trying to explain that a computer system is more akin to a living body that needs continuous care (some kind of virtual horse) than to an inert tool, with some, yet limited, results.

Takeaways from a global malware disaster

zlynx — Fri, 19 May 2017 03:23:01 +0000

> I don't know if that's your case, but with the kind of tailored software I'm producing (nothing technically fancy, just piles of business rules), the root cause is often that the admins just don't dare doing it, being too afraid to create breakage of applications they can't even test on their own. Instead, they rely too much on what they can actually maintain: the surrounding infrastructure, firewalls etc.

Virtual machines have been great for this. Years ago the company I worked for replaced all of our physical rack servers with a blade thing (Dell maybe?) running VMware ESX. Now our admins clone a machine, install all of the updates and test it, and then they can almost transparently shut down the old copy and replace it.

Takeaways from a global malware disaster

gracinet — Thu, 18 May 2017 22:31:01 +0000

>> People need to keep their systems up to date. I don't feel like enough education is happening in this field as to how important it is that your system is up to date and still receiving security patches.

>Indeed. I routinely gets questions from customers on how they should upgrade our particular software and then they tell me that they currently use a version that we released several years ago. My agony with this is not that they have not updated our particular software suite but that we put software on their machines via both DEB and RPM repositories so this means that they have not even run "apt upgrade" or "yum update" for all that time either.

I don't know if that's your case, but with the kind of tailored software I'm producing (nothing technically fancy, just piles of business rules), the root cause is often that the admins just don't dare doing it, being too afraid to create breakage of applications they can't even test on their own. Instead, they rely too much on what they can actually maintain: the surrounding infrastructure, firewalls etc.
YMMV, but it's also true that there are very few customers that can tolerate a breakage due to an update they didn't ask for (even if hundreds of previous ones happened silently and prevented lots of problems). All of this requires lots of prior explanations and mutual understanding - this is hard.

So, it's quite common in my experience to be called for some application-level bugfix, and to notice that the surrounding system never had a single upgrade for years. I often raise the issue, hoping that the testing windows can be mutualized, but that's a double-edged sword : usually people call you with a specific goal in mind (very urgent), and evaluate your action with respect to that goal only. It's also quite common for the application to be scheduled for complete replacement (which is always late) after some years of production, and in that case, of course, it's very hard to plead for any extra work. And it's true that after too many upgrades have been skipped, things can get a bit dangerous.

This is the part where I heartily thank Debian, for its stability : it makes applying upgrades automatically a reasonable trade-off.

A colleague of mine even once made the acknowledgement of a situation of that kind a prerequisite to proceeding further (à la: here's the list of outdated system packages with security issues, please notice that's almost all of them, that wasn't even in our mission, so we consider it's your problem to fix that, please acknowledge that we can't be accountable about consequences of that situation or let's push the price up a bit if you want us to fix that also).

This is indeed the kind of human organizational dysfunction that the dev-ops movement has been trying to solve, but I fear that dev-ops is better understood if it's done from within an organization (usually tech-savvy), not by outside contractors. And also, for some people, dev-ops doesn't mean much more than that they can deploy Docker containers without dependency hell ; it's easy to forget that these are meant to have upgrades, too, even if there are no changes in the app itself.

To be fair, I'm aware of an exception: hosting companies that, by law, have to abid to mandatory security regulations in specialized fields (happens, e.g., with health related personal data in France). Unfortunately, it's bureaucratic and very expensive. If it weren't mandatory for the client, too, it wouldn't happen in many cases.

> Or the client who recently asked me if I could do a build for CentOS 3...

Oh, that's a nice one! EOL'ed on 2010-10-01!

Takeaways from a global malware disaster

HenrikH — Thu, 18 May 2017 20:53:47 +0000

> People need to keep their systems up to date. I don't feel like enough education is happening in this field as to how important it is that your system is up to date and still receiving security patches.

Indeed. I routinely gets questions from customers on how they should upgrade our particular software and then they tell me that they currently use a version that we released several years ago. My agony with this is not that they have not updated our particular software suite but that we put software on their machines via both DEB and RPM repositories so this means that they have not even run "apt upgrade" or "yum update" for all that time either.

Or the client who recently asked me if I could do a build for CentOS 3...

Takeaways from a global malware disaster

HenrikH — Thu, 18 May 2017 20:48:30 +0000

If anything I would say that this actually shows that agencies like The NSA could both release the exploit to Microsoft and still use it to attack computers since many people don't patch their systems anyway.

Vulnerability hoarding and Wcry

jchaxby — Thu, 18 May 2017 20:28:35 +0000

Vulnerability hoarding is dangerous. Eventually the vulnerability and an exploit will become public. If you're lucky by the time that happens there'll be a patch.

So far so obvious.

We all know, now, that unpatched machines are going to get exploited.

What's unconscionable though is people deploying systems both from a distro revision that's years out of date and not even considering the processes by which it needs to be updated and, eventually upgraded or decommissioned.

It doesn't matter if it's home routers, MRI scanners or cloud data centres; they need to be *designed* so that their embedded software can be updated and upgraded for as long as the system still works.

You can probably get away with a $10 router not getting updates after two or three years but it needs to be sold under the premise that at the end of that time it _will_ get an update that will deactivate it and you can then recycle it or trade it in for an upgrade.

It's completely unacceptable for some medical bit of kit costing millions that might be running for decades to be designed with no way to update its embedded OS or eventually upgrade it to something supportable. (WinXP in MRI scanners?)

We know how to do this, we've all been designing distros that can be updated (and in some cases upgraded) for years now. It's about time that all that got put into practise.

Takeaways from a global malware disaster

drag — Thu, 18 May 2017 18:42:44 +0000

The way I deal with Windows updates is to boot linux and use the tools supplied by "WSUS Offline Downloader", get all the windows updates on a USB key, and then install them to Windows before I ever connect it to a network.

It's rare that I have to deal with this though. Usually only when people come to me with a jacked up PC and they want me to fix it.

Takeaways from a global malware disaster

habarnam — Thu, 18 May 2017 13:26:44 +0000

> The vulnerabilities exist whether the NSA finds them or whether someone else does.

Indeed, but in the case of the good guys finding them, responsible disclosure should be the first step, not storing them for darker days. I think this is, or should be, the basis of most of the NSA directed criticism.

Vulnerability hoarding and Wcry

smcv — Thu, 18 May 2017 12:59:14 +0000

The original Windows 8 release (what you might call Windows 8.0, although its official name was just Windows 8) is unsupported. For a while after 8.1 was released, Microsoft continued to produce security patches that were applicable to Windows 8 systems that had not already been upgraded to 8.1, but they do not normally do this any more (making an exception to that policy for this vuln). If I understand correctly, every Windows 8 license is also a valid Windows 8.1 license, so this is more like a service pack than a major-version upgrade.

This is analogous to how, for example, Ubuntu 16.04 LTS users are expected to upgrade to the 16.04.2 point release to get continued support, even if they do not want to upgrade to a newer major version like 16.10.

You might reasonably think "well, obviously, if you don't update then you don't get updates" but historically Microsoft has made some effort to make fixes individually applicable to older versions for a while, as a response to users' unwillingness to risk regressions by applying service packs and other large updates early (or in some cases at all).

If I understand correctly, the complexity required to support applying arbitrary combinations of patches (and for that matter detecting which ones are missing) is a large part of why Windows Update is so slow and horrible, particularly on fresh installs of old Windows releases where bringing the system up to date requires a huge number of patches. Linux distributions have tended to dodge this by having well-defined packages with incrementing version numbers, and refusing to support anything other than a linear sequence of upgrades per package: if foobar version 1.2.3-4 fixed CVE-2014-12345 (but introduced a regression) and foobar 1.2.3-5 fixed CVE-2014-54321, then you can't opt to install the fix for CVE-2014-54321 but remain vulnerable to -12345, except by rebuilding foobar yourself (at which point you are the OS vendor for a very small fork).

The other reasons we don't suffer from this in FOSS distributions to the extent that Microsoft does are that our major-version updates are free of charge, so some perverse financial incentives go away, leaving technical decisions (like how much regression risk to accept) as the only factor in how far to upgrade; and that if anyone feels sufficiently strongly that a particular distribution is doing it wrong (for example introducing too many regressions in their updates), forking the distribution is always an option.

Vulnerability hoarding and Wcry

triddell — Thu, 18 May 2017 12:31:54 +0000

"...and two other no-longer-supported versions (Windows 8 and Windows Server 2003)"

It seems updates for Windows 8 will be available for some time to come: https://support.microsoft.com/en-ca/help/13853/windows-li...

Did the author mean Vista perhaps?

Takeaways from a global malware disaster

jschrod — Thu, 18 May 2017 10:30:32 +0000

> Additionally, I feel like there's a disconnect between clients and contractors about purpose-built systems like this; clients believe that this is a one-time job and that there is no on-going maintenance involved, but contractors know better but choose to not correct the clients in their beliefs.

I'm the CEO of a consulting company, and my experience is different. Contractors know about the need for maintenance and tell the customers. After all, they want maintenance contracts, these are a very good way to earn money: You have low aquisation costs, and remain in contact with the client to check around what other needs he has that one can help to solve (and earn money...).

But customers often don't allocate a budget for on-going maintenance, they don't see the business need for it. Or, the IT sees the business need, but the C[EFO]O doesn't. (Actually, events like WannaCry are good, in this regard, it helps to illustrate the business case.)

Upfront charging for on-going maintenance is only possible for mass-market software, for bespoke software it would raise the price to a point where one is not competetive in the market any more.

I.e., the state of affair is even more complicated than you presented.

Takeaways from a global malware disaster

NAR — Thu, 18 May 2017 09:59:23 +0000

"no excuse for anyone running Windows 7 to get affected by this exploit, but people did."

Not much of an excuse, but pirated (non-activated) copies of Windows 7 might not be able to get updated.

User does not own the vulnerable software

cpanceac — Thu, 18 May 2017 09:51:38 +0000

I have a question: the software does not belong to the user, it's licensed for use by Microsoft, right? So how come this is not Microsoft's problem?

Takeaways from a global malware disaster

cpanceac — Thu, 18 May 2017 09:47:18 +0000

Right. The Windows update process is painfull and buggy and it takes a looong time to complete.

Takeaways from a global malware disaster

Seegras — Thu, 18 May 2017 09:00:25 +0000

> but I also think that the NSA not finding those vulnerabilities would not make the world any more secure.

This is beside the point. Demand for those exploits by secret services and law enforcement agencies has lead to a sprawling industry trading in zero-days. Here's and example what we're talking about:

https://www.zerodium.com/program.html

The NSA could start by not hoarding vulnerabilities for instance. But to really make the world more secure, the NSA would need to search for vulnerabilities and _publish_ them.

The cool thing about publishing a vulnerability is that it also denies the use of that vulnerability to your enemies. So if you want to increase security, your only option is to publish.

In fact, there is one thing that distinguishes the White Hats from the others. White Hats publish.

Right now, the NSA is as a malicious Black Hat as it gets.

Takeaways from a global malware disaster

eru — Thu, 18 May 2017 04:58:41 +0000

> There was no excuse for anyone running Windows 7 to get affected by this exploit, but people did.

I know a lot of people who turned off Windows 7 updates as a response to Microsoft's increasingly-underhanded attempts to force them to update to Win10 via Windows Update.

Another reason is the unreliability of Windows update software itself! I have seen it get somehow wedged for good on three different Windows versions on home laptops, so that it tries to update, wastes fifteen minutes of time and then gives up with a hex guru meditation. Last time saw this on the WannaCry weekend; I checked if the Windows 10 laptop was up to date, and noticed the effect again. The log was full of failed attempts. Fortunately the last succesful update was sometime in April, so it possibly is patched against the SMB issue in question. But the next might get it.

Takeaways from a global malware disaster

pabs — Thu, 18 May 2017 04:28:47 +0000

I think it is about time for a "Right to Repair" software.

Takeaways from a global malware disaster

simcop2387 — Wed, 17 May 2017 23:43:25 +0000

> But then I wonder, why do people not look for alternatives instead?

Because of the thought, "If it's not broken don't fix it". The problem is getting them to understand that it can still be broken behind the scenes where you can't see anything. Just like a leaky pipe under a building eventually causing foundation damage or a sinkhole.

Takeaways from a global malware disaster

pizza — Wed, 17 May 2017 20:01:56 +0000

> There was no excuse for anyone running Windows 7 to get affected by this exploit, but people did.

I know a lot of people who turned off Windows 7 updates as a response to Microsoft's increasingly-underhanded attempts to force them to update to Win10 via Windows Update.

Takeaways from a global malware disaster

fratti — Wed, 17 May 2017 19:55:35 +0000

I agree that this is a big issue. One example is µTorrent; people stay on old versions (2.2.x I believe?) because newer versions ship advertisements and bundled crapware. But then I wonder, why do people not look for alternatives instead?

In the case of strictly only security updates for still supported software, I don't think it's that though. There was no excuse for anyone running Windows 7 to get affected by this exploit, but people did.

Takeaways from a global malware disaster

NightMonkey — Wed, 17 May 2017 19:46:13 +0000

You can educate people until you turn blue, but if they are only held accountable for their budgets, "externalities" like software updates will never be addressed properly. Corporations are created expressly to avoid personal liability, and the people so absolved act accordingly.

There is a fundamental problem in the metaphors and abstractions we have made for computing resources and networks. The "use case" for open networks changed when the Internet was handed over from academic and defense research institutions to business, and we have suffered from that "scope creep" ever since.

On the topic of the NSA creating 'weapons' exploiting bugs and design flaws... Edward Snowden made plain the dangers that the NSA and other intelligence agencies present to ordinary people. And in making this plain, he showed that these organizations cannot be trusted with their digital assets. From the massive data vacuums they have created, to weaponized math (which is what software is), they are quite cavalier in how they secure these resources, and are creating dangerous threats where none existed before.

Takeaways from a global malware disaster

pizza — Wed, 17 May 2017 19:42:33 +0000

Part of folks' reluctance to do this is that these "updates" routinely come with intentional side effects -- An extreme example is Windows 10. Or how Netflix will now refuse to install on unlocked/rooted devices.

Vendors have a pretty poor track record, and they're getting worse, not better.