ELC: Google learns to drive

Andrew Chatham came up the peninsula to San Francisco from Google to talk to the 2013 Embedded Linux Conference about the self-driving car project. Chatham has worked on the project since 2009 and seen it make great strides. It is by no means a finished product, but the project has done 400,000 miles of automated driving so far.

History

"Cars are a miracle", he said. The 45-mile drive he did to Mountain View yesterday would have taken our ancestors all day to do on a horse. But, cars are also problematic, with more than 30,000 annual deaths in the US due to car accidents. That number has "finally started dropping", likely due to more seat belt usage, but it is still too high. Even if there are no fatalities, accidents cost time, money, and more. We have done a pretty good job figuring out how to survive accidents, he said, but it is time to stop having them.

In the mid-2000s, the US Defense Advanced Research Projects Agency (DARPA) ran several challenges for self-driving cars on a 150-mile course in the Mojave Desert. The first year, the winning team's vehicle went only seven miles. But the next year, five teams actually completed the course, which was eye-opening progress. In 2007, DARPA moved the challenge to a simulated urban environment that featured a limited set of traffic interactions (four-way stops, but no traffic lights, for example). After that event, "DARPA declared victory" and moved on to other challenges, Chatham said.

In 2009, Google stepped in to solve the problem "for real". Chatham noted that people have asked why Google would get involved since driving cars doesn't involve searching the internet or serving ads. The company thinks it is an important problem that needs to be solved, he said. Google is qualified to attack the problem even though it has never made cars because it is mostly a software problem. Also, one major component of the problem involves maps, which is an area where Google does have some expertise.

Broadly, there are two categories for self-driving cars: one for cars with all the smarts in the car itself and one where the smarts are in the road. For cars that are self-contained, they need to be ready for anything and cannot make assumptions about the obstacles they will face. That tends to lead to cars that move slowly and drive cautiously, much differently than humans. Smart roads allow for dumb cars, but there are some serious obstacles to overcome. Infrastructure is expensive, so there is a chicken-and-egg problem: who will build expensive smart roads (or even lanes) for non-existent dumb cars that can use them?

The Google approach is something of a hybrid. There are no actual infrastructure changes, but the system creates a "virtual infrastructure". That virtual infrastructure, which is built up from sensor readings and map information, can be used by the car to make assumptions much like a human does about what to expect, and what to do.

Sensors and such

The car's most obvious sensor is the laser rangefinder that lives in a bubble on top of the car. It spins ten times per second and produces 100,000 3D points on each spin. Each of those points have 5cm accuracy. The laser can only see so far, though, and can be degraded in conditions that affect photons, such as rain.

The car also has radar, which is not as precise as the laser, but it can see further. It can also see behind cars and other solid objects. Using the doppler effect, speed information for other objects can be derived. There are also cameras on the car. The general "computer vision problem" is hard, and still unsolved, but it isn't needed for the car's usage of the camera. The camera is used for things that humans use as well, which means they are generally rather obvious and are of known shapes, sizes, and likely positions (e.g. traffic lights). Beyond that are the expected sensors like gyroscope, accelerometer, GPS, compass, and so on.

There are two main computers in the car. One is a very simple "drive by wire system" that has no operating system and is just in a tight loop controlling the brakes, steering, and accelerator. The second is a "workstation class system running FreeBSD", Chatham joked. In reality it is running a lightly customized Ubuntu 12.04 LTS. It is not running the realtime kernel, but uses SCHED_FIFO and control groups to provide "realtime-ish" response.

There are several classes of processes that run on the system, with the least critical being put into control groups with strict resource limits. If any of the critical processes miss their deadlines, it is a red flag event which gets logged and fixed. In the 400,000 miles the cars have traveled (so far always with a human on board), those kinds of problems have been largely eliminated. All of the data for those journeys has been stored, so it can be played back whenever the code is changed to try to find any regressions.

From the "blank slate" state of the car, GPS data is added so that it knows its location. That data is "awful really", with 5m accuracy "on a good day" and 30m accuracy at other times. The car's sensors will allow it to accurately know which way it is pointing and how fast it is going. From there, it adds a logical description of the roads in that location derived from the Google Maps data. It uses maps with 10cm resolution plus altitude data, on top of which the logical information, like road locations, is layered.

All of that information is used to build a model of the surroundings. The altitude data is used to recognize things like trees alongside the road, as well as to determine the acceleration profile when climbing hills. The goal is to stay close to the center of the lane in which the car is traveling, but not if something is blocking the lane (or part of it). You also don't want to hit the guy in front of you, so don't go faster than he does. Once the model is built, driving is largely a matter of following those two rules, he said.

Problems and solutions

In California (unlike many other places) it is legal for motorcycles to travel between the lanes, moving between the cars that are in each of those lanes. That was a difficult problem to solve because that situation could fool the sensors to some extent.

One of the tests they ran was a race track that was set up to see how the car did versus Google employees. It reliably beat them in that test, though Chatham believes the employees would have eventually beaten the car. It's "a Prius, not a sports car", so there were limits to the kinds of maneuvering that can be done, but the test really showed the "precision with which we can reliably drive the car", he said.

Lots about driving is "social", Chatham said. For example, there is a set of rules that are supposed to be followed at a four-way stop, but no one follows them. The car had to learn to start to edge out to see when the others would let it go through. Similarly, merging is social, and they have spent a lot of time getting that right. Unlike human drivers, the car can't make eye contact, so it is a matter of getting the speed and timing right for what is expected.

The "bug on the window problem" is another difficult one. For the car, anything that messes up its sensors needs to be handled gracefully. In those cases, handing control back to the human in a sensible fashion is the right thing to do.

Many people ask about how the car does in snow, but it hasn't been tried yet. Currently, Chatham thinks it wouldn't do all that well, but thinks it "could do OK eventually". One problem there is that snowbanks appear to be giant walls of water to the lasers.

"People do stupid things", he said. If you drive 400K miles, you are going to experience some of them. Normally the expectation is that other people value their lives; if you didn't believe that, you would never leave home. But there are exceptions, so a self-driving car, like a regularly driving human, needs to be prepared for some of that craziness.

The video of a blind man using the self-driving car is the kind of story that shows where this technology could lead, Chatham said. There are a lot of people who can't drive for one reason or another, so a self-driving car has the potential to change their lives. "I wish it were done now, but it's not", he said.

Chatham answered a few questions after the talk. They have done very little work on "evasive maneuvers", he said. Everyone overestimates their ability in that area and the advice from police and others is to just use the brakes. There are no plans as yet to release any of the source code, nor are there any plans for a product at this point. Three states have "legalized" self-driving cars, California, Nevada, and Florida. It is furthest along in California where the Department of Motor Vehicles is currently drafting rules to govern their use.

[ I would like to thank the Linux Foundation for travel assistance to attend ELC. ]