No Metrics [LWN.net]

No Metrics

Posted May 7, 2011 3:09 UTC (Sat) by mlawren (guest, #10136)
Parent article: Scale Fail (part 1)

Great article! Sadly everything you wrote rings very true. I've seen the No Metrics conversation go the other way as well:

"app: The network is broken."
"net: Hmmm... let me check. Nope, my measurements indicate everything is ok"
"app: Are you sure?"
"net: Yes, here is the data, see for yourself.
"app: But must be the network!!!"
"net: We haven't made any changes. Look at these historical graphs of usage. That spike from last week was the file-sharer who had that unfortunate accident in the carpark on Monday. Nothing difference since then."
"net: Have you checked your application?"
"app: Of course, it's running fine."
"net: How do you know?"
"app: I just know."
"net: When was the last time it ran fine?"
"app: Tuesday."
"net: What have you changed since then?"
"app: Only re-worked the dispatcher, and migrated the cache location to the other campus. But we checked, the code runs fine!"
"net: wtf??!?"
"net: Say, you drive that green Ford Focus don't you?"
"app: Yeah, why?."
"net: No reason. Your problem will be solved by tomorrow."

to post comments

No Metrics

Posted May 7, 2011 19:20 UTC (Sat) by mstefani (guest, #31644) [Link] (21 responses)

Hah! "Blame the network" is industry best practice...

No Metrics

Posted May 9, 2011 4:36 UTC (Mon) by gdt (subscriber, #6284) [Link] (19 responses)

Whilst I've seen far too many database administrators blame "the network", it is also true that a lot of the points made by this article apply as equally to networks as to databases.

Particularly the lack of instrumentation, especially of problematic middleboxes such as application (de)accelerators and firewalls. Even basic monitoring is poor, link with application-performance-killing high errors rates often creeping under the radar of monitoring tools like Nagios.

It's rare to see routing designed with good choices and configured correctly. There's a simple tell-tale test: type in an unassigned IP address in the corporate network, does it error immediately or time out?

The poor state of corporate networks isn't helped by networking equipment vendors, who often ship equipment with near-essential settings off for "backward compatibility".

Finally, many sysadmins and applications are their own worst enemy. Using IP addresses rather than DNS names (they're going to regret that, come IPv6). Disabling ethernet autonegotiation. Assuming link layer connectivity for high-availability schemes. Refusing to deal with authentication and authorisation issues within the application, but pushing that into VLANs and VPNs, thus turning the corporate network into a flat layer two network, with resulting poor behaviour under fault conditions.

No Metrics

Posted May 9, 2011 8:39 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (18 responses)

Yes. In my opinion, the lack of good IP-layer security is stunning. And treating local wired networks as 'trusted' is even more stunning.

I have seen many times internal services using HTTP with plain-text auth on the local networks - because administrators think it's 'secure'. Hell, probably everyone here is guilty of that.

Fortunately, situation is changing. With IPv6 it's possible to do end-to-end IPSec (which is not possible due to #(*$&(@$& NATs right now) and with DNSSEC it's possible to reliably store host certs in RDNS.

No Metrics

Posted May 9, 2011 12:43 UTC (Mon) by paulj (subscriber, #341) [Link] (17 responses)

You're assuming any coming IPv6 world will have near-universal end-to-end connectivity. Sadly, that's unlikely to be true.

No Metrics

Posted May 9, 2011 16:13 UTC (Mon) by raven667 (subscriber, #5198) [Link] (1 responses)

This may be off-topic but why do you say that? While it's true that we will still have just as many firewalls as before it should be easier to enable end-to-end connectivity and such connectivity is the default and harder to work around than not doing so.

Universal end-to-end nightmare

Posted May 19, 2011 18:39 UTC (Thu) by oelewapperke (guest, #74309) [Link]

This sadly is a good reason *not* to implement ipv6. Security is a problem, and NAT means universal firewalling.

Given how many security problems we have, and how quickly they get fixed ... this is sadly a good thing.

No Metrics

Posted May 9, 2011 18:03 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (14 responses)

Probably not. With IPv4 it's a struggle to keep things globally adressable. Hey, I've just got my IPv4 PI allocation and happily renumbered my networks - now I can reach every host from every host. It's a heaven. But getting it was certainly not trivial.

With IPv6 it's exactly backwards - it's a struggle NOT to make your computers globally addressable.

No Metrics

Posted May 9, 2011 21:48 UTC (Mon) by paulj (subscriber, #341) [Link] (13 responses)

Globally addressable != global end-end connectivity, sadly. Lots (most, even) of people will still have restrictive firewalls between their network and the internet, and some people even want NAT for IPv6..

No Metrics

Posted May 10, 2011 8:18 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (12 responses)

Firewalls can be disabled or reconfigured. NAT can't be disabled (there's not enough global IPv4 addresses) even in principle.

The changes will take years, so there'll be plenty of time for security to evolve. But we now have foundation for it.

No Metrics

Posted May 10, 2011 9:55 UTC (Tue) by paulj (subscriber, #341) [Link] (11 responses)

Many users will lack either the administrative or technical capability (or both) to reconfigure firewalls, sadly.

No Metrics

Posted May 10, 2011 11:50 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (10 responses)

That's the beauty of IPv6 it's open by default.

And corporate networks will benefit from end-to-end security most, so I expect that they'll migrate to IPsec even before home users.

No Metrics

Posted May 10, 2011 17:35 UTC (Tue) by dlang (guest, #313) [Link] (5 responses)

corporate users will suffer from open end-to-end connectivity most, including tunnels that make it impossible to see what's happening.

No Metrics

Posted May 10, 2011 17:41 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

Not really. It's possible to sniff encrypted tunnels if you know the private key of one host (and enterprise admins probably would know it), it's even supported in Wireshark.

Besides, it's not like I can't make an HTTPS tunnel which can pierce all but the most paranoid firewalls right now. Skype does this, for example.

No Metrics

Posted May 10, 2011 21:37 UTC (Tue) by Tobu (subscriber, #24111) [Link]

Nitpick: that depends on the key exchange. Sniffing after a Diffie Helman requires the cooperation of one of the parties, and I don't think wireshark has support for this at the moment.

No Metrics

Posted May 10, 2011 21:45 UTC (Tue) by raven667 (subscriber, #5198) [Link] (2 responses)

*sigh* that is one thing that is probably true, some network operators will break their networks in the name of security making life difficult for the people who use them and that won't really protect anything because so much traffic is tunneled over 80/443 which is almost universally allowed.

No Metrics

Posted May 19, 2011 18:40 UTC (Thu) by oelewapperke (guest, #74309) [Link] (1 responses)

That's because it requires the "end" party to take the initiative. That's the beauty of NAT. Put an ancient totally unsupported bug-riddled system that every grandmother knows how to exploit remotely behind a nat firewall ...

And it's perfectly secure.

No Metrics

Posted May 23, 2011 4:24 UTC (Mon) by RobertBrockway (guest, #48927) [Link]

No it isn't. If that was true then most successful attacks today wouldn't even occur. For some time now the bulk of attacks have occurred over connections initiated by the end user system.

No Metrics

Posted May 11, 2011 18:36 UTC (Wed) by Baylink (guest, #755) [Link] (3 responses)

> That's the beauty of IPv6 it's open by default.

The problem with utopias is that it only takes *one* Bad Guy to fuck things up for the rest of us.

"That's not a feature, that's a bug."

No Metrics

Posted May 12, 2011 13:42 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

And closing down everything like we already do now does not help at all.

No Metrics

Posted May 19, 2011 18:46 UTC (Thu) by oelewapperke (guest, #74309) [Link] (1 responses)

Actually, given the amount of security holes ... and how secure a company is by default from remote (ie. they only get infected by surfing porn sites or opening suspicious mail) ...

It kinda does solve a lot of problems.

I mean, I hate nat just like the next guy. But you won't get anywhere by declaring it doesn't solve anything. You'll be just like gaia idiots screaming before the capitol to get America off oil, not realizing they're basically asking America to cut it's economy by 95% or more. Not going to happen (and it's a *good* thing we don't honor such requests)

NAT is a beautifully simple solution. And it is possible to modify just about any protocol to work with nat. I fear nat and ipv4 may be here to stay.

Certainly converting RIPE, APNIC and AFRINIC over to ARIN rules would give us another 10 years easily. Saying "an IP will cost you $0.01 per year" will get us another 100 years.

No Metrics

Posted May 19, 2011 19:11 UTC (Thu) by nybble41 (subscriber, #55106) [Link]

A stateful firewall which simply blocks all incoming connections (i.e. a NAT setup minus the actual address and/or port translation) gets you all the security benefits of NAT without most of the hassle. As a bonus, if you want to run the same services on two or more servers they can each use their own addresses rather than competing for the standard port numbers.

Anyway, most home routers aren't much more secure with NAT, since they allow ports to be forwarded via UPnP requests. If you're running a server and opening forwarding ports with UPnP you might as well permit direct access; if not, blocking the connection at the server (because the port is closed) is just as effective as blocking it at the firewall. An effective firewall must be configured by the network administrator to accept or reject specific traffic, not simply permit incoming connections to any local server that asks politely while blocking the ones which would have been rejected anyway.

Blame the network

Posted May 9, 2011 5:35 UTC (Mon) by ringerc (subscriber, #3071) [Link]

"accounting-package-x doesn't perform terribly using a shared pseudo-ISAM file over a SMB network, it's great! If you're having problems, there's something wrong with your network..."

Suuuure, MYOB.