Development

Building HTTP/2 services with gRPC

Google recently released a new remote procedure call (RPC) framework that is designed to supplant traditional representational state transfer (REST) in the development of web applications. Called gRPC (which the FAQ defines as "gRPC Remote Procedure Calls"), the new framework takes advantage of several features in the recently approved HTTP/2 standard that the project claims will result in better performance and greater flexibility than REST APIs. Implementations are available—for both client- and server-side code—in a variety of programming languages.

The basic idea behind gRPC is quite similar to that used by the RESTful services that make up so many web applications today. The methods of the server application are accessible to clients through a well-defined set of HTTP requests. The server answers each of these calls with an HTTP response (or, if necessary, an error code). But gRPC dispenses with HTTP 1.1, relying strictly on HTTP/2—and thus, the project claims, gaining considerable performance.

gRPC was announced in a February 26 blog post. The post described gRPC as enabling "highly performant, scalable APIs and microservices" and said that Google is starting to expose gRPC endpoints to its own services. It also said that other companies have been involved in the creation of gRPC, naming mobile-payment processor Square as one example. Square posted an account of its own that explains some of the background. Both of the posts highlight a few key features that are enabled by building on top of HTTP/2—namely, bidirectional streaming, multiplexed connections, flow control, and HTTP header compression.

Those improvements, of course, would be available to any RPC framework using HTTP/2. As for gRPC itself, it is built on top of Google's protocol buffers, which was updated to version 3.0 (a.k.a. "proto3") in conjunction with the gRPC release. Proto3 is largely a stripped-down version of proto2: multiple fields have been removed, including several that were required in proto2 or that had required default values, and the extension mechanism has been replaced with a simple Any standard type that can hold. Proto3 also adds standard types for times and dates and for dynamic data, plus bindings for Ruby and JavaNano (an Android-centric Java library optimized for low-resource systems).

gRPC uses protocol buffers to serialize and deserialize its over-the-wire messages. Employing that technique leverages HTTP/2's binary wire format, allowing for more efficient RPC traffic than HTTP 1.1's ASCII encoding. But gRPC also uses proto3's interface definition language (IDL) to specify the structure and the content that a web service's RPC messages will take.

To get started, the application developer writes a schema for the new web service, defining each of the requests and the possible responses that the service will use. The protocol buffers compiler (protoc) can then be used to generate classes and stub code (for client and server) from the schema, in any of the supported languages. At launch time, the available languages were Python, C++, C#, Objective-C, PHP, Ruby, Go, Java (both in generic and Android flavors), and JavaScript tailored to use with Node.js.

Quick-start guides are available for most of the languages (C#, Objective-C, and PHP guides are absent at the moment). Each of these tutorials begins with the same basic service definition:

    syntax = "proto2";

    // The greeting service definition.
    service Greeter {
      // Sends a greeting
      rpc SayHello (HelloRequest) returns (HelloReply) {}
    }

    // The request message containing the user's name.
    message HelloRequest {
      string name = 1;
    }

    // The response message containing the greetings
    message HelloReply {
      string message = 1;
    }

One side point is worth noting: for reasons that are not explained, the project's Python tutorial does, indeed, specify proto2 syntax rather than proto3. For the simple "Hello World" example used, proto2 and proto3 syntax are identical (which is visible when comparing it with the Java and Go tutorials), but the generated code does leave some artifacts behind, in the _pb2 portion of the module name.

The output language of protoc compiler is specified with a command-line switch. Using --python_out, for example, generates a helloworld_pb2 module, a stub function called SayHello() for the client, and a Greeter class for the server that includes a stub function to respond to the SayHello() message as well as a simple serve() function that waits for incoming messages.

By using an abstract IDL, the argument goes, developers can quickly generate client code for a variety of different platforms, both desktop- and mobile-oriented. gRPC also makes it easy to include a version number in each message-format definition, so that clients and servers can support interoperability across versions.

The project has a detailed description of how gRPC's messages are sent in HTTP/2 frames. This is where some of HTTP/2's other advantages over HTTP 1.1 come into play.

For example, whenever an endpoint (client or server) sends a message, it can choose to keep the stream open, so that it can be used for bi-directional communication for the remainder of the session. gRPC uses HTTP/2's stream IDs as its internal identifiers for RPC calls, so the mapping between the RPC call and its associated stream comes for free. Multiplexing several open streams over one connection is another way that HTTP/2 enables more efficient use of the available bandwidth; it is up to the developer to determine how many active streams make sense for the service.

gRPC also piggybacks on HTTP/2 for its error codes. That allows the application to pick up on HTTP/2 error conditions (such as one endpoint of the stream throttling the connection) with little additional effort. Finally, gRPC is designed to support pluggable authentication methods; TLS 1.2 (or greater) and OAuth 2.0 are supported so far.

Where gRPC heads from here, naturally, remains to be seen. It is certainly a straightforward enough framework to warrant closer examination, and there will, no doubt, be many developers interested in seeing how they can take advantage of HTTP/2's promised improvements over HTTP 1.1.

That said, HTTP/2 is still in its infancy—one should expect to see a wide array of new frameworks announced in the next few years, all claiming to leverage the new protocol for a range of enhanced features. gRPC may be one of the first, but only time will tell how many developers outside of Google and its partners find it a good fit.

Comments (9 posted)

Quotes of the week

— Sriram Ramkrishna, on prioritizing cross-platform application development.

— Tom Callaway, in regard to GitHub's new Licenses API.

— Moxie Marlinspike, on the difficulties of working with GnuPG.

Comments (21 posted)

Samba 4.2.0 released

The Samba team has announced the first release in the new stable 4.2.x series. This release adds transparent file compression, access to "Snapper" snapshots via the Windows Explorer "previous versions" dialog, better clustering support, and much more. This release also marks the end of support for Samba 3.

Full Story (comments: 8)

Sphinx 1.3 released

Version 1.3 of the Sphinx documentation system has been released. Support for Python 3.4 has been added, while support for Python 2.5, 3.1, and 3.2 was dropped. Among the new features are several new themes, a builder for generating Apple Help output, and an extension for NumPy and Google-style docstrings.

Full Story (comments: none)

xf86-input-libinput 0.8.0 available

Version 0.8.0 of xf86-input-libinput has been released. The update fixes excess scroll-speed problems on touchpads as well as a driver crash, and adds a configuration option for choosing between different click methods. Notably, the update relies on version 0.11 of the libinput library.

Full Story (comments: none)

Mailpile: Beta Rejected!

The Mailpile project announced that, after processing the feedback that resulted from its first public beta release, the team has pulled back the release so that it can rethink things and continue development. Among the highlighted issues were IMAP and GnuPG support, but "there are a very large number of other smaller bugs and loose ends that need work. Almost all of these are in the "back end" of the app, the low level plumbing, gears and turbines and closures and APIs that work behind the scenes. The back-end has been playing catch up to the user interface for a while, it needs some focus and attention before we can ship a real product." We took a look at the beta release in September 2014.

Comments (none posted)

vdpauinfo 1.0 available

Version 1.0 of vdpauinfo has been released. Vdpauinfo is a command-line utility that can be used to query the hardware-acceleration capabilities of Video Decode and Presentation API for Unix (VDPAU) video devices. New in 1.0 is support for querying the latest set of H.265 profiles.

Full Story (comments: none)

Development newsletters from the past week

• What's cooking in git.git (March 5)
• What's cooking in git.git (March 6)
• LLVM Weekly (March 9)
• OCaml Weekly News (March 10)
• OpenStack Community Weekly Newsletter (March 6)
• Perl Weekly (March 9)
• PostgreSQL Weekly News (March 8)
• Python Weekly (March 5)
• Ruby Weekly (March 5)
• This Week in Rust (March 9)
• Tor Weekly News (March 11)
• Wikimedia Tech News (March 9)

Comments (none posted)

Edmundson: High DPI Progress

At his blog, David Edmundson writes about the state of high-DPI support in KDE. "For some applications supporting high DPI has been easy. It is a single one line in KWrite, and suddenly all icons look spot on with no regressions. For applications such as Dolphin which do a lot more graphical tasks, this has not been so trivial. There are a lot of images involved, and a lot of complicated code around caching these which conflicts with the high resolution support without some further work." He is personally tracking the progress of many applications, but notes that there are many unsolved issues. "There are still many applications without a frameworks release even in the upcoming 15.04 applications release. Even in the next applications release in 15.08 August we are still unlikely to see a released PIM stack. Is it a good idea to add an option into our UIs that improves some applications at the cost of consistency? It's not an easy answer." This update is Edmunsdon's second post on the subject; the first, from November 2014, is also quite informative.

Comments (41 posted)

Morevna: How do we deal with spoilers?

The Morevna project is developing an open-source animated series using a suite of open-source graphics applications. But working in the opening poses an interesting question: how does one cope with spoilers leaking out to fans of the project? At the Morevna blog, Konstantin Dmitriev explains the process chosen by the project. "The content that classified as containing spoilers is published on the website, but hidden (locked) from the public view. When the new episode is released, all hidden content related to it becomes publicly visible. On our Patreon page we have introduced a new category of supporters – Premium Patrons. Premium Patrons have a special privilege for an early access to the hidden content before the release." Morevna only decided to move away from a feature-length production toward an episodic framework in January 2015, so it will be interesting to see how the new plan plays out.

(Hat tip to Paul Wise.)

Comments (none posted)