LWN.net Logo

The future of unencrypted web traffic

By Jake Edge
January 2, 2008

Hypertext transfer protocol (http) is the heart of the web, providing the means to retrieve content from remote servers. It is an unencrypted, text-based protocol which allows malicious intermediaries to snoop on and potentially modify the traffic. Unfortunately, internet service providers (ISPs) are getting increasingly bold in manipulating the traffic that they carry. This has lead some to call for the elimination of http, in favor of encrypted http (aka secure http or https).

An ISP is perfectly situated to gather an enormous amount of information about its users, their website preferences and habits (often called clickstream data). Some have reportedly been selling some of that data in a thinly-anonymized form to advertisers and others. As AOL's well-intentioned, but poorly implemented, release of search queries showed, it is rather easy to analyze this kind of data and pierce the anonymity, deriving the specific user.

Another recent ISP trick is to modify a retrieved web page to display other information – under the control of the ISP – which looks like it comes from the website itself. Canadian ISP Rogers Internet has been testing a system to add content to the Google homepage for their customers who are near their monthly bandwidth limits. There are also plans afoot for ISPs to use clickstream data to target advertising – though just where those ads would show up is far from clear.

This kind of manipulation is unlikely to be what internet users expect – to the extent they think about it all. The model folks tend to use is that of a phone company; we do not expect them to sell our call records to the highest bidder, nor do we give them license to modify our calls. Various telecommunications privacy laws protect that data, but those laws have not (yet) been applied to internet traffic. In addition, ISPs tend to have a monopoly or near-monopoly, which restricts alternative, less-intrusive ISPs from competing.

Fortunately, there are technical solutions possible in the internet realm that would be difficult or impossible to implement network-wide in the phone system. Encrypting website traffic will go a long way towards eliminating this kind of ISP abuse, though it is no panacea. As more of these kinds of privacy invasions occur, we should see more routine use of https by websites.

Currently, https is almost exclusively used for e-commerce transactions; typing in credit card numbers and the like. Authentication via username and password is another area that sees widespread encrypted pages. Sites may start to use https for their entire site to combat clickstream and page rewriting abuse – though there will still be some information leakage as the ISPs can still see what sites are being visited.

In order to make an https connection, the server must have a certificate with its public key. Typically those are signed by an authority recognized by browsers which allows the browser to authenticate that the certificate belongs to the host visited. Getting signed certificates is a bit cumbersome, costs some money, and they need to be renewed periodically – all of which adds up to a headache for a site, especially a small, non-commercial site, that wants to switch to using https. Self-signed certificates are an alternative, but because they are susceptible to man-in-the-middle attacks, browsers warn their users when they receive one.

Another problem with this approach is the extra processing required on the server to support encrypting each and every request. There is a non-trivial amount of extra work that must be done per request and cannot be cached. Sites that wish to avoid the problems that some ISPs are introducing will just have to bear that cost.

Pushing bits is not very glamorous, but that is really what one hires an ISP to do. Since they seem to be finding new and exciting ways to interfere with those bits – Comcast messing with BitTorrent traffic for example – internet users will have to find ways to thwart their schemes and encryption will be a big part of that effort. Using https site-wide is only one step, other services will also need to be protected from ISP abuse. What if an ISP started manipulating the results returned from DNS queries, perhaps routing some to a server they control?


(Log in to post comments)

Regulatory solution, not technical, is required

Posted Jan 3, 2008 3:48 UTC (Thu) by dskoll (subscriber, #1630) [Link]

Really, we need a regulatory solution. Something like: IF ISP X modifies data in transit or
modifies DNS results, THEN ISP X is no longer a "common carrier" and becomes responsible for
all the content it transmits.

I bet that'd put an end to these kinds of funny games much faster than any technical solution.

copyright?

Posted Jan 3, 2008 5:49 UTC (Thu) by pjm (subscriber, #2080) [Link]

An ISP that modifies web pages that pass through it may well be in breach of the copyright of
the page author.  They have an implicit license to redistribute the unmodified page, but I
believe do not have any license to redistribute adaptations/derived works of that page.

This presumably applies to all transcoding proxies and other non-transparent proxies too.  The
only differences I can think of is that adding advertizing is “for commercial advantage”, and
that copyright owners may be more likely to sue someone for adding/changing/removing
advertisements than other modifications.

I must note that IANAL, and in fact I'm rather less sure about the above than most things I
might say about copyright.

Surprisingly enough it's quite clear in U.S.

Posted Jan 3, 2008 13:49 UTC (Thu) by khim (subscriber, #9252) [Link]

You can read yourself:
In general terms, section 512(a) limits the liability of service providers in circumstances where the provider merely acts as a data conduit, transmitting digital information from one point on a network to another at someone else’s request. This limitation covers acts of transmission, routing, or providing connections for the information, as well as the intermediate and transient copies that are made automatically in the operation of a network.

In order to qualify for this limitation, the service provider’s activities must meet the following conditions:

  • The transmission must be initiated by a person other than the provider.
  • The transmission, routing, provision of connections, or copying must be carried out by an automatic technical process without selection of material by the service provider.
  • The service provider must not determine the recipients of the material.
  • Any intermediate copies must not ordinarily be accessible to anyone other than anticipated recipients, and must not be retained for longer than reasonably necessary.
  • The material must be transmitted with no modification to its content.

If you are selecting and changing content then you are losing this protection and need special permission from Google (and other web sites) to operate...

Regulatory solution, not technical, is required

Posted Jan 3, 2008 8:53 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]

the problem is that right now the common carrier status of ISPs is very nebulous.

if they should get common carrier status then they need to be neutral about the traffic
passing over their networks (solving the network neutrality problem) and they cannot tinker
with the traffic (solving this problem), but they also cannot be liable for what is passing
over their networks.

if they should not have common carrier status, then they gain the right to tinker with network
traffic, but should then become liable for what's on their network (including the need to
filter out content that is offensive for the people who it are offended)

I don't think any of the ISPs have really looked at the tradeoff, right now they are trying to
play both sides of the fence

Regulatory solution, not technical, is required

Posted Jan 3, 2008 12:09 UTC (Thu) by copsewood (subscriber, #199) [Link]

"the problem is that right now the common carrier status of ISPs is very nebulous."

Those who deliver snail mail have common carrier status but similar problems arise there. They
are not obliged to carry decomposing organic materials, live animals or hazardous substances.
They are not obliged to carry packages greater than a certain size or weight or combination.
They are allowed to sniff for explosives, narcotics and radioactive materials and xray suspect
packages. You also can't get an unlimited amount of mail into a post box designed for a
certain volume and have to make special arrangements for delivery or collection. The fact that
some but not all illegal packages can be detected by scanning does not make the snail mail
delivery companies responsible for illegal packages which can't be detected.

The nebulousness referred to is because sufficient and comprehensive case law does not yet
exist for much of what goes on in the relatively new ISP industry.

Regulatory solution, not technical, is required

Posted Jan 4, 2008 3:55 UTC (Fri) by dirtyepic (subscriber, #30178) [Link]

They also do not (can not?) stamp advertisements on your parcels or sell information about how
much mail you receive and where it came from to 3rd parties.

We need ISPs to be able to do a certain amount of filtering

Posted Jan 3, 2008 10:07 UTC (Thu) by james (subscriber, #1325) [Link]

Such regulation would need to be well-thought-out, or it might prevent (or place a legal cloud over):
  • ISPs filtering spam (and other unwanted messages);
  • ISPs providing optional parental filters;
  • ISPs filtering port 25 outbound by default;
  • ISPs monitoring for and disconnecting virus-infected computers;
  • ISPs redirecting HTTP traffic away from malicious websites.

That list isn't complete -- I'm sure there are other things we'd expect ISPs to be able to do for the general public. There'll be other things in future, too.

Do you trust Congress, Parliament, or whoever to come up with legislation which will prevent behaviour we don't like, now and in the future, will permit filtering that is in the interests of netizens, will be sufficiently flexible for future technologies and problems, and stand up to commercial interests?

We need ISPs to be able to do a certain amount of filtering

Posted Jan 3, 2008 11:58 UTC (Thu) by epa (subscriber, #39769) [Link]

Hmm, I'm not sure, it might be better if ISPs indeed did not do any of the above things.  Just
give each user a quota of packets to send and receive and let them get on with it.

Manifestly, all the measures ISPs are currently taking to block port 25 or cut off zombie PCs
are ineffective in stopping spam, so it would be no great loss if they went back to just
pushing bits.

We need ISPs to be able to do a certain amount of filtering

Posted Jan 3, 2008 19:44 UTC (Thu) by vonbrand (subscriber, #4458) [Link]

Just think about how bad it would become if they did nothing (Shudder!)

Several categories of service here

Posted Jan 3, 2008 16:28 UTC (Thu) by kirkengaard (subscriber, #15022) [Link]

The ISP, in its capacity as an email provider, should filter spam and other nuisances _within
email traffic_, but only to the extent to which these are contained within email traffic
directed to or provided by users of its email services.  This is a service provided through
that company's internet service provision, but separate from it.  This is filtering done by a
mail server or associated apparatus on mail arriving over the TCP/IP stream.  This should not
violate common-carrier status because they don't care what the bits are until they become an
endpoint.

Services provided to the user to filter traffic at the user end, such as optional parental
filtering, are not therefore filtering done at the discretion of the ISP, but end-user defined
filtering of end-user defined traffic.  This should not violate common-carrier status because
the ISP retains its neutrality as a carrier, and filtering is done per-endpoint/per-user at
user request.

Monitoring and disconnecting virus-infected computers is a grey area as far as caring what the
content of packets are, because it does require analysis without being an endpoint.  It is,
however, undeniably malicious content.  This intent to harm the user is the valid reason for
filtering, just as the parallel filtering of the mail systems for explosives, toxins and other
undeniably hazardous content.  Users that desire hazardous materials for legitimate purposes
need to go through separate carrier systems and much red tape from the recipient end to
request such materials.  This asserts the express non-malicious nature of that particular
'packet' of hazardous-to-the-user material.  The ISP has a customer safety burden which makes
virus filtering and quarantine a legitimate intrusion upon carrier neutrality.

We need ISPs to be able to do a certain amount of filtering

Posted Jan 4, 2008 2:03 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link]

I specificly picked an ISP that doesn't do any of those things (well, they do offer optional
spam filtering)

another drawback

Posted Jan 3, 2008 4:57 UTC (Thu) by mattdm (subscriber, #18) [Link]

Name-based virtual hosting doesn't work with https. This means there needs to be one IP
address per web site, which doesn't seem viable until we are all using IPv6.

another drawback

Posted Jan 3, 2008 8:15 UTC (Thu) by TRS-80 (subscriber, #1804) [Link]

Firefox 2, Opera 8 and IE7 (on Vista) all support TLS SNI (RFC 3546), so it's probably feasible to start deploying widely in a year or two (which is sooner than IPv6 will become mainstream). Try out your browser at https://carol.sni.velox.ch/.

another drawback

Posted Jan 3, 2008 11:48 UTC (Thu) by sitaram (subscriber, #5959) [Link]

it's RFC 4366 now

another drawback

Posted Jan 3, 2008 12:10 UTC (Thu) by redenisc (guest, #43086) [Link]

Browsers support for SNI is improving indeed. What about server support 
though. OpenSSL supports SNI as of 0.9.9 which is not released at the time 
of writing. Hence Apache mod_ssl does not support SNI, hence most deployed 
HTTP servers will not support SNI for the next few years.

another drawback

Posted Jan 3, 2008 13:31 UTC (Thu) by tialaramex (subscriber, #21167) [Link]

Bulk hosting is the main application though, and the people doing bulk hosting already have
some guy with a beard and a hand-modified version of Perl working for them so this isn't so
scary. A lot of them still run Linux 2.4, have their own CVS tree for Apache, that sort of
thing.

This is the right way round to deploy stuff anyway, you only need one server to provide the
service, but you need as many user agents as possible to support it, or it's useless. If in
2008 just one company, say Dreamhost, offer this as a service, but 95% of people with web
browsers have a new enough one that it supports SNI, then you've got something useful. The
opposite way around would be completely worthless.

another drawback

Posted Jan 3, 2008 13:55 UTC (Thu) by cortana (subscriber, #24596) [Link]

FYI, SNI was backported to OpenSSL 0.9.8g, and there is a patch in Apache's bug tracking
system which works fine in my informal tests (which included backporting OpenSSL 0.9.8g to the
current stable release of Debian, and rebuilding Apache on same--both tasks were nice and
easy). :)

workaround

Posted Jan 3, 2008 11:04 UTC (Thu) by DonDiego (subscriber, #24141) [Link]

A workaround that I use is to create a certificate for *.site.com, which can then be used for
all subdomains.

workaround

Posted Jan 3, 2008 12:06 UTC (Thu) by mattdm (subscriber, #18) [Link]

I don't think that helps, actually. There's still no way to make the virtual hosts work (short
of RFC 4366 mentioned above).

not exactly true

Posted Jan 3, 2008 17:13 UTC (Thu) by ccyoung (guest, #16340) [Link]

if sites on the same ip address share the certificate, then can use rewrite rules to
accomplish pretty much everything you want - or such has been true for me.

another drawback

Posted Jan 3, 2008 12:07 UTC (Thu) by jschrod (subscriber, #1646) [Link]

All recent browsers support the cert extension »Certificate Subject Alt Names«, with DNS names
as entries, and thus allow to use name-based virtual https hosts. We use them on several of
our sites and have had no complaints at all until now. As an example you can look at the cert
of https://lists.dante.de/, our site for the mailing lists of the German TeX Users Group.

Of course, if one still needs to support IE4 or such, one is out of luck.

That said, this is no solution for mass hosters, as one needs to recreate the certificate for
every virtual host that's added. In the end, that is not manageable, especially from a
security viewpoint. But for a company or an organization which has a few dozen hosts and a low
change rate, it's a working solution.

The future of unencrypted web traffic

Posted Jan 3, 2008 8:09 UTC (Thu) by V-Li (subscriber, #48321) [Link]

In Germany the situation is specal.  Providers are not liable for forbidden content
(pornography for minors, "special interest" pornography as sodomy, nazi propaganda), because
they only pass it through.  BUT if the manipulate pages they may turn into a content provider
which is responsible.

service- versus content-providers

Posted Jan 3, 2008 16:32 UTC (Thu) by kirkengaard (subscriber, #15022) [Link]

Internet Service Providers are not liable for the content over their service; Internet Content
Providers are liable for the content they provide.  Good distinction to keep in mind.

The future of unencrypted web traffic

Posted Jan 3, 2008 15:56 UTC (Thu) by drag (subscriber, #31333) [Link]

The real solution is to make it mandatory for ISPs to indicate if they are manipulating
content and what restrictions they are putting on their network connections. (like not giving
you a public IP address, port blocking, content filtering, transparent proxies, P2P
monitoring, etc etc.)

Then make it easier and cheaper to compete by eliminating regional monopolies for phone and
cable providing and releasing restrictions on the use of the radio spectrum for data
transmissions. In addition to releasing these restrictions encourage the growth of
fiber-to-the-home as a municipal service, like sewage or electrical connections. Have cities
hire companies to do the cabling and maintain it, but not provide data services over it. Then
let people pick what ISP they choose to use over the fiber.

All of this together will allow customers to choose between having content filtered and not
and will incourage a massive growth in competition, bandwidth, and drop in the costs
associated with internet access. This would free up resources for VoIP and other stuff
lowering the costs of that also.


All this speak of regulating what ISPs can and cannot do is just going to make problems worse.
This would increase the red tape ISPs go through and all that sort of BS and in addition will
restrict innovation by locking these people into a forced mode of operation. There are plenty
of people that would love to have ISPs filtering out malicious websites and such things, for
example. All that is needed is a way to make it easy for people to know if this sort of thing
is happenning.

The future of unencrypted web traffic

Posted Jan 4, 2008 18:28 UTC (Fri) by giraffedata (subscriber, #1954) [Link]

I completely agree with all of this -- we should retain our right to negotiate whatever service we want with our ISPs. If the ISP offers crappy service and a low price, I should be able to buy that. I also would rather see the government own the network outright than tell the putative owners how to operate it (i.e. commandeer it).

But, sadly, I'm sure this means that the kind of service people are asking for here would not exist. There just aren't enough of us who care about unmolested web pages, privacy, RFC compliance, transparent channels, and such to create a market. ISPs will offer an extremely limited service that fits the needs of 95% of their home customers and the other 5% won't be able to afford what they want.

What people crying for encryption and regulation want is to stifle their competition -- other consumers. When my neighbors aren't able to get their crappy restricted, low-price service, they will instead demand the higher service level I want and the volume will bring my price down.

The future of unencrypted web traffic

Posted Jan 4, 2008 11:27 UTC (Fri) by NRArnot (subscriber, #3033) [Link]

I hate to recommend anything to do with lawyers, but isn't this a perfect job for them? If an
ISP takes someone's web page and alters its content before serving it up, they are surely
creating a derived work for commercial gain, in flagrant violation of the owner's copyright.
We aren't talking a few lines of symbol definitions in a million lines of system. We're
talking about plagiarism of an entire work and them deliberate misrepresentation of the
authorship and origin of the bastardised version! 

So set up a web site, with copyrighted material. Make it necessary for the users to accept the
copyrights explicitly (sign up, read, all modification strictly verboten, click OK). Get a
tame user to browse and obtain evidence of the derived works. Then sue! Possibly even for
libel, rather than copyright violation. Adding adverts, possibly containing malware, could be
highly damaging to the copyright owner and/or the owner of the website serving the copyrighted
material without the adverts).

Note the adverts might inadvertently cause offense or outrage through mis-targetting. "First
picture of motorcyclist shot dead on motorway" alongside a motorcycle advert tagline "Nobody
ever wished they stayed in more". An advert for a mobile phone called a "Blast" alongside what
later proved to be an erroneous report of a man killed by an exploding mobile. Such is
unfortunate, but the ads were placed there by an organisation invited in by the copyright
owner. Imagine if not. Also the ads might contain malware. ISPs beware. You could get sued
from both sides (copyright owner and browser user) at once!

And finally, consider the ISP's customers. If my ISP starts doing this I'll immediately take
my custom elsewhere, and make a small-claims claim for the cost of being forced to do so. Some
customers might also be able to sue for larger damages, if for example they are already
post-processing web pages for themselves (loading share prices into databases and suchlike)
when such post-processing is broken by an ISP choosing to alter the HTML in transit. 

All in all, it's a really dumb idea for the ISPs. I can't imagine it'll catch on, they'll be
making themselves into large and ill-defended targets if they do. 

Man in the middle

Posted Jan 4, 2008 15:07 UTC (Fri) by rfunk (subscriber, #4054) [Link]

The paragraph about signed certificates misses an important point.  The 
only thing keeping an ISP from proxying https traffic and effectively 
mounting a man-in-the-middle attack is the browser's certificate check.  
And if the end-user just clicks through the browser warnings, using https 
instead of http does nothing to prevent the ISP from messing with content.

Man in the middle

Posted Jan 5, 2008 13:38 UTC (Sat) by copsewood (subscriber, #199) [Link]

An ISP doing this would be carrying out a misrepresentation. In some jurisdictions this would
be a forgery offence, in others plain and simple fraud. In the UK it would classify as
preparing/planning for unauthorised access and theft of data within the computer misuse and
data protection acts. Even if a private prosecution against such an ISP failed, the bad
publicity would cost it more than it might have to gain. The Sony rootkit is a similar example
and frankly I'm surprised that prosecutions did not take place over it. 

There seems to be an attitude here that large companies are above the law; that it does not
apply to them, but this is partly because the rudimentary laws which do cover the digital
domain are widely misunderstood and not yet adequately tested in court in such cases of
corporate abuse.

Technically the way to defeat this involves DNSSEC and a certificate forest with trees rooted
at the top-level domains established and maintained as part and parcel of DNS domain
registration and renewal.

Man in the middle

Posted Jan 5, 2008 16:29 UTC (Sat) by rfunk (subscriber, #4054) [Link]

I'm not qualified to say much about the legal aspects in any country, though the 
combination of big companies and technology often makes for a lack of reason in the 
judicial world.

But your DNSSEC solution does nothing to protect against the ISP doing a MIM attack.  
The scenario I was talking about doesn't depend on DNS forgery at all.  That's the 
advantage the ISP has that other attackers don't have.

Man in the middle

Posted Jan 7, 2008 1:19 UTC (Mon) by copsewood (subscriber, #199) [Link]

If DNSSEC secures the DNS and DNS domain registration includes provision of certificates this
makes having certificates as routine as registering a domain. 

Man in the middle

Posted Jan 7, 2008 2:10 UTC (Mon) by rfunk (subscriber, #4054) [Link]

Sorry, you're apparently still not understanding my point.  Or I'm not getting yours.  Or 
both.

The future of unencrypted web traffic

Posted Jan 12, 2008 11:59 UTC (Sat) by alynch (guest, #49925) [Link]

A simple solution to this particular problem is to for the web server to encrpyt simply a
checksum to the contents of the page. This would greatly cut down on the comuptational
overhead on the server but retain the benefit of preventing modification by the ISP.

Copyright © 2008, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds