Sharefest, WebRTC, and file distribution [LWN.net]

By Nathan Willis
June 26, 2013

The WebRTC standard is much hyped for its ability to facilitate browser-to-browser media connections—such as placing video calls without the need to install a separate client application. Relying solely on the browser is certainly one of WebRTC's strengths, but another is the fact that the media stream is delivered directly from one client to the other, without making a pit-stop at the server. But direct connections are available for arbitrary binary data streams, too, through WebRTC's RTCDataChannel. Now the first applications to take advantage of this feature are beginning to appear, such as the Sharefest peer-to-peer file transfer tool.

Changing the channel

WebRTC data channels are an extension to the original WebRTC specification, which focused first on enabling two-way audio/video content delivery in real-time. The general WebRTC framework remains the same; sessions are set up using the WebRTC session layer, with ICE and STUN used to connect the endpoints across NAT. Likewise, the RTCPeerConnection is used as a session control channel, taking care of signaling, negotiating options, and other such miscellany. The original justification for RTCDataChannel was that multimedia applications also needed a conduit to simultaneously exchange ancillary information, anything from text chat messages to real-time status updates in a multiplayer game.

But data and real-time media are not quite the same, so WebRTC defines a separate channel for the non-multimedia content. While audio/video streams are delivered over Real-time Transport Protocol (RTP), a RTCDataChannel is delivered over Stream Control Transmission Protocol (SCTP), tunneled over UDP using Datagram TLS (DTLS). Using SCTP enables a TCP-like reliability on top of UDP. After all, if one video chat frame in an RTP conversation arrives too late to be displayed, it is essentially worthless; arbitrary binary data, however, might require retransmission or error handling.

That said, there is no reason that a web application must implement a media stream in order to use an RTCDataChannel. The API is deliberately similar to Web Sockets, with the obvious difference being that the remote end of a RTCDataChannel is another browser, rather than a server. In addition, RTCDataChannels use DTLS to encrypt the channel by default, and SCTP provides a mechanism for congestion control.

Obviously there are other ways to send binary data from one browser to another, but the WebRTC API should be more efficient, since the clients can choose whatever (hopefully compact) format they wish, and use low-overhead UDP to deliver it. At a WebRTC meetup in March, developer Eric Zhang estimated [SlideShare] that encoding data in Base64 for exchange through JSON used about 37% more bandwidth than sending straight binary data over RTCDataChannels.

Browser support for RTCDataChannels is just now rolling out. Chrome and Chromium support the feature as of version 26 (released in late March); Firefox currently requires a nightly build.

Sharefest

Sharefest is a project started by a video streaming company called Peer5. A blog post introducing the tool says the impetus was two coworkers who needed to share a large file while in a coffee shop, but Dropbox took ages to transfer the file to the remote server then back again. With RTCDataChannels, however, the transfer would be considerably faster since it could take place entirely on the local wireless LAN. A few months later, Sharefest was unveiled on GitHub.

The Sharefest server presents an auto-generated short-URL for each file shared by the "uploading" (so to speak) party. When a client browser visits the URL, the server connects it to the uploader's browser and the uploader's browser transfers the file over an RTCDataChannel. If multiple clients connect from different locations, the server attempts to pair nearby clients to each other to maximize the overall throughput; thus for a persistent sharing session Sharefest does offer bandwidth savings for the sharer.

Currently the size limit for shared files is around 1GB, though there is an issue open to expand this limit. A demo server is running at sharefest.me (continuing the near-ubiquitous domination of the .me domain for web-app demos). The server side is written in Node.js, and the entire work is available under the Apache 2.0 license.

Naturally, only the recent versions of Chrome/Chromium and Firefox are supported, although the code detects the browser and supplies a friendly message illustrating the problem for unsupported browsers. This is particularly important because the RTCDataChannel API is not yet set in stone, and there are several pieces of WebRTC that are implemented differently in the two browsers—not differences distinct enough to break compatibility; mostly just varying default configuration settings.

Sharefest is certainly simple to use; it does require that the uploading browser keep the Sharefest page open, but it allows straightforward "ephemeral" file sharing in a (largely) one-to-many fashion. From the user's perspective, probably the closest analogy would be to Eduardo Lima's FileTea, which we covered in 2011. But FileTea is entirely one-to-one; Sharefest does its best to maximize efficiency by dividing each file into "slices" that the downloading peers exchange with each other.

This is useful, of course, but to many people the term "peer-to-peer" effectively means BitTorrent, which is a considerably more complex distribution model than Sharefest supports (supporting, for example, super-seeding and distributed hash trackers). One could implement the existing BitTorrent protocol on top of RTCDataChannels, perhaps, and pick up some nice features like DTLS encryption along the way, but so far no one seems to have taken that challenge and run with it.

But there are others pursuing RTCDataChannels as a file distribution framework. Chris Ball, for example, has written a serverless WebRTC file-sharing tool, although it still requires sharing the STUN-derived public IP address of the uploader's machine, presumably off-channel in an email or instant message. The PeerJS library is a project that may evolve in the longer term to implement more pieces of the file-sharing puzzle.

In the meantime, for those curious to test out the new API, Mozilla and Google have publicized several other RTCDataChannel examples involving games. But as Zhang noted in his presentation, direct client-to-client data transfer is likely to have more far-reaching effects than that: people are likely to write libraries for file I/O, database drivers, persistent memory, etc. The potential is there for many assumptions about the web's client-server nature to be broken for good.

Sharefest, WebRTC, and file distribution

Posted Jun 27, 2013 9:49 UTC (Thu) by niner (guest, #26151) [Link] (19 responses)

It is an incredible shame that in the year 2013, 43 years after the ARPANET went online, we still struggle to transfer a file from one client to another. WebRTC may be the usable workaround till the real solution (IPv6) finally arrives. All others seem to require some account with some proprietary service (like Skype or Dropbox), work less often than not (XMPP clients) or are too complicated for most users.

Sharefest, WebRTC, and file distribution

Posted Jun 27, 2013 14:28 UTC (Thu) by zlynx (guest, #2285) [Link]

Just think back to 1999. Microsoft made it TOO easy to share files. Everyone with a cable modem and a Windows file share was doing it. Usually by accident.

Sharefest, WebRTC, and file distribution

Posted Jun 27, 2013 15:20 UTC (Thu) by kjp (guest, #39639) [Link] (15 responses)

I hate to burst your bubble, but IPv6 ain't changing anything.

Sharefest, WebRTC, and file distribution

Posted Jun 27, 2013 15:23 UTC (Thu) by niner (guest, #26151) [Link] (13 responses)

And why would that be?

Sharefest, WebRTC, and file distribution

Posted Jun 27, 2013 15:54 UTC (Thu) by mpr22 (subscriber, #60784) [Link] (12 responses)

Because IPv6 is not going to magically make opaque boundaries, restrictive ingress and egress policies, and incompetent network administrators disappear.

Sharefest, WebRTC, and file distribution

Posted Jun 27, 2013 16:32 UTC (Thu) by niner (guest, #26151) [Link] (11 responses)

Probably not. But getting rid of NAT will allow someone with more than a decade of network administration experience to transfer a file from one client to another which is good enough for me.

Sharefest, WebRTC, and file distribution

Posted Jun 28, 2013 16:06 UTC (Fri) by kjp (guest, #39639) [Link] (4 responses)

So by getting rid of nat, you mean the kernel supports it? http://atoomnet.net/howto-ipv6-nat-in-centos-6/

Sharefest, WebRTC, and file distribution

Posted Jun 28, 2013 16:16 UTC (Fri) by niner (guest, #26151) [Link] (3 responses)

Yes, NAT is still supported. No, I will not use it at home. My desktop and laptop will have globally reachable addresses as soon as my ISP arrives in the 21st century.

Sharefest, WebRTC, and file distribution

Posted Jun 28, 2013 16:40 UTC (Fri) by mjg59 (subscriber, #23239) [Link] (2 responses)

Providing, of course, that your ISP doesn't NAT you.

Sharefest, WebRTC, and file distribution

Posted Jun 28, 2013 16:42 UTC (Fri) by niner (guest, #26151) [Link]

Which none of the many ISPs that already offer IPv6 to end users do. So the chances for that are rather low.

Sharefest, WebRTC, and file distribution

Posted Jul 4, 2013 15:10 UTC (Thu) by farnz (subscriber, #17727) [Link]

Note, however, that there is a big disincentive for ISPs to NAT you at all, which is only outweighed in IPv4 by the difficulty in obtaining enough IPv4 addresses for all your customers' needs. Specifically, the state required to NAT someone results in two costs to an ISP:

Some of your equipment costs more, because it has to store and act on both routing state and NAT state.
To seamlessly fail over requires you to ship NAT state around, not just routing state. The alternative is to convert brief blips in connectivity (where a router reboots and the VRRP spare kicks in and starts shifting packets) into lost connections (where a router reboots, and all NAT state in that router is lost).

This doesn't mean that an ISP won't NAT you in an IPv6-only world, only that there is good cause to be optimistic that ISPs won't spend the extra to provide a reduced service, when their competitors can spend less and have any IPv6-only NAT desired by customers provided at the customer-controlled side of the CPE.

Sharefest, WebRTC, and file distribution

Posted Jun 29, 2013 17:03 UTC (Sat) by dmag (guest, #17775) [Link] (5 responses)

If my cable company (ISP) suddenly switched to IPv6, I would still run NAT on my router because I don't control the OS-level firewalls on all the devices on my (home) network.

IPv6 is not going to fix file sharing.

Sharefest, WebRTC, and file distribution

Posted Jun 29, 2013 20:36 UTC (Sat) by apoelstra (subscriber, #75205) [Link]

> If my cable company (ISP) suddenly switched to IPv6, I would still run NAT on my router because I don't control the OS-level firewalls on all the devices on my (home) network.

What does NAT have to do with firewalls?

Sharefest, WebRTC, and file distribution

Posted Jun 29, 2013 21:09 UTC (Sat) by paulj (subscriber, #341) [Link]

Why not just run a firewall???

Sharefest, WebRTC, and file distribution

Posted Jun 30, 2013 0:07 UTC (Sun) by foom (subscriber, #14868) [Link] (2 responses)

I hope you wouldn't actually run *NAT*, but would instead use a stateful firewall.

NAT should be useless with IPv6, but stateful perimeter firewalls -- which do cause most of the same issues as NAT w.r.t. sending data between hosts -- will certainly still be around.

Sharefest, WebRTC, and file distribution

Posted Jun 30, 2013 13:27 UTC (Sun) by cesarb (subscriber, #6266) [Link] (1 responses)

> I hope you wouldn't actually run *NAT*, but would instead use a stateful firewall.

In fact, unlike with IPv4, where the commonly used form of NAT implies a stateful firewall, with IPv6 NAT does not need any firewall (since it will often be a direct 1:1 mapping).

Sharefest, WebRTC, and file distribution

Posted Jun 30, 2013 14:23 UTC (Sun) by raven667 (guest, #5198) [Link]

But even 1:1 NAT will still break any peer-to-peer protocol and require some third party host to play matchmaker and inform the endpoints what the real public routable IPs look like as the endpoints only know about their local IPs, which is suboptimal and should be unnecessary in the IPv6 world, especially for home users. For business users there is some case for 1:1 NAT so that the organizations IP space is portable between providers, without running a routing protocol and getting an assignment or re-IPing when switching. For home users IP allocation is generally automatic so re-IPing has a lower cost.

Sharefest, WebRTC, and file distribution

Posted Jun 28, 2013 3:37 UTC (Fri) by filteredperception (guest, #5692) [Link]

> I hate to burst your bubble, but IPv6 ain't changing anything.

I disagree. I may lose my battle, but today I'm feeling more optimistic than a month ago. I've used the fact that GoogleFiber was my first ISP choice involving IPv6 to press a new novel interpretation of NetworkNeutrality. It seems to be going somewhere. ComIntercept(FCC->Google):

"The enclosed informal complaint, dated September 1, 2012, has been filed with the Commission by Douglas McClendon against Google pursuant to section 1.41 of Comissions's Rules, 47 C.F.R. // 1.41. Also attached is Mr. McClendon's October 24, 2012 complaint forwarded to the FCC by the Kansas Office of the Attorney General. Mr. McClendon asserts that Google's policy prohibiting use of its fixed broadband internet service (Google Fiber connection) to host any type of server violates the Open Internet Order, FCC 10-201, and the Commission's rules at 47 C.F.R. // 8.1-11.

We are forwarding a copy of the informal complaint so that you may satisfy or answer the informal complaint based on a thorough review of all relevant records and other information. You should respond in writing specifically and comprehensively to all material allegations raised in the informal complaint, being sure not to include the specifics of any confidential settlement discussions. ...

Your written response to the informal complaint must be filed with the Commission contact listed below by U.S. mail and e-mail by July 29, 2013. On that same day, you must mail and e-mail your response to Douglas McClendon.

The parties shall retain all records that may be relevant to the informal complaint until final Commission disposition of the informal complaint or of any formal complaint that may arise from this matter. See 47 C.F.R. //1.812-17. (seriously, can't I and Google just depend on the NSA's backups of our records? :)

Failure of any person to answer any lawful Commission inquiry is considered a misdemeanor punishable by a fine... ... ...

((see my other comment on this article for the document links))

Sharefest, WebRTC, and file distribution

Posted Jun 28, 2013 3:29 UTC (Fri) by filteredperception (guest, #5692) [Link]

Here is what it's all about-

http://cloudsession.com/dawg/downloads/misc/mcclendon_not...
http://cloudsession.com/dawg/downloads/misc/mcclendon_oct...

This represents Google getting 'served' this week, my form 2000F 'informal' 53 page complaint that suggests that NetNeutrality provides protections against ISP blocking to my home servers as well as to Skype's. Google has been compelled by the government to respond to me on July 29th. GoogleFiber's 'evil' terms of service prohibit hosting any kind of server without prior written permission against your residential connection. And zero transparency for any alternate server-allowed plan rates, or what kinds of reasons they might use to disallow a requested written permission (which is laughable as the FCC 10-201 NetNeutrality document goes out of it's way to laud Tim Berner Lee's invention of the web atop tcp/ip, specifically, without having to have gotten any permission from any government or network provider)

I forwarded the documents to schneier@schneier.com and requested any insight he might have into the matter. I got an email response (theoretically perhaps spoofed) that read "Thanks.\n\nGood Luck."

In general this article reaks of some orwellian redefinitions of network terminology so that the lies can continue to make sense. I.e. client to client file sharing. As if there is any sane reason why two people at a coffeeshop shouldn't be able to use some sensible tool built atop sftp or the like. (yes, I see how ice/stun/turn etc work around legacy NAT issues, and try to wriggle a dependence on a central third party server into the picture. And I'm sure it will be valuable. But the persecution of server operators by residential ISPs needs to stop. And in a sane way that doesn't require calling a system that can sit and wait for input and then serve responses, a 'client'.

Sharefest, WebRTC, and file distribution

Posted Jun 28, 2013 4:58 UTC (Fri) by dlang (guest, #313) [Link]

in the example case where two people have machines on the same wifi network and are trying to share files, IPv6 would not make any difference.

There was no NAT between them.

If they had Linux systems, they could have trivially enabled ssh, fired up a webserver, or used any of several hundred possible options to share the file (including bittorrent)

running the file through SCTP on top of UDP (all through the browser, hardly the most efficient or secure program available) would be so far down the list as to be invisible.

Even the idea of sending it offsite to a remote system just to pull it back again would be quite a ways down the list.

This sounds like it's a problem only for users of proprietary operating systems who are not willing to install readily available free software on them.

I sure don't want features like this enabled on browsers I use (at least not without me going in to explicitly enable it for this particular file)

Sharefest, WebRTC, and file distribution

Posted Jun 27, 2013 15:22 UTC (Thu) by kjp (guest, #39639) [Link]

A nice article, but there was something conspicuously missing: any mention of security when connecting these channels. Sure, I understand things are encrypted, but how does the authentication work?

Sharefest, WebRTC, and file distribution

Posted Jun 27, 2013 23:25 UTC (Thu) by dowdle (subscriber, #659) [Link] (2 responses)

I'm running Fedora 19 (RC3 is Go!) and installed Fedora 22 from the updates testing repo... and now I'm WebRTC'ing.

So, I tried the sharefest.me site to send a test file between a co-worker in the same room as myself and it told me the size was limited to 250MB which is significantly lower than the 1GB mentioned in fine article. Perhaps they changed it recently? The co-worker who transferred the file from me was running Firefox 22 on Windows 7.

Supposedly there isn't any problem using WebRTC between Chrome and Firefox but I haven't tried that yet.

I'd like to try the video chat thing but so far I haven't found anyone who has a webcam/mic and a flashy new browser. I do.

Sharefest, WebRTC, and file distribution

Posted Jun 28, 2013 2:37 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (1 responses)

> I'm running Fedora 19 (RC3 is Go!) and installed Fedora 22 from the updates testing repo...

Huh. And here I thought Rawhide was going to be Fedora 20. ;)

Sharefest, WebRTC, and file distribution

Posted Jun 28, 2013 18:59 UTC (Fri) by dowdle (subscriber, #659) [Link]

Opps... I meant to say Firefox 22 not Fedora 22.