Linux Conference often has talks about interesting Linux-powered
should come as no surprise, but talks about devices that fly tend to
attract a larger audience. Gregoire Gentil's presentation on a
video "nano-copter" was no exception. While there was no free-flying demo,
there were several tethered demos that clearly showed some of the
possibilities of the device.
Gentil started his talk by showing a YouTube video of the
marketing-level pitch from his company, AlwaysInnovating (AI), for the MeCam
device. The MeCam is a small quad-copter that allows people to video their
every move as it follows
them around. The video will stream to their mobile phone, where it can be
uploaded to Facebook, Twitter, and the like.
The device itself, pictured at right, is a "flying Pandaboard" with "lots
of modifications", Gentil said. The copter has four propellers, a camera,
and can communicate via WiFi. He had one of the copters tethered to a
stand so that it could go up and down, but not "kill anybody if something
goes wrong", he said with a laugh.
The copter runs an OMAP4 with dual Cortex-A9 processors. It uses
pulse-width modulation (PWM) to control each of the four propeller motors. The
camera has 1080p resolution and uses CSI-2 for the data interface as USB is
not fast enough, he said. There are also a "bunch of sensors", including
sensors, altitude sensors, and wall-detection sensors.
There is a single battery powering both the CPU and the motors, which is
different from some radio-controlled (RC) copters, he said. The
motors are quite small, and run at 10,000-15,000rpm. That can
create a lot of noise in the electrical system, so many RC devices use two
separate batteries. Instead, AI has added a lot of filtering
on the power, so that it could avoid the extra weight of an additional
The MeCam is not an RC device, instead it has an auto-pilot that is voice
controlled and uses facial recognition to position itself. AI considered
three different possibilities for running the auto-pilot code. The first
was to run a standard Linux kernel, which would give them a "mature full
environment" in which to develop and run the code. The downside is the
latency. There were times when the motors would not
be serviced for 50ms, which was enough time to cause the MeCam to "crash
against the wall", Gentil said.
The second option was to use the realtime Linux kernel, which has "much
better latency", but is "less mature than the standard kernel". That is
the approach being used now, but AI is pursuing another approach as well:
writing a custom realtime operating system (RTOS) for the Cortex-M3 which
is present on the OMAP4. That will allow the system to have
"perfect latency", he said, but it "will be complex to develop and is not
For the demos, and until the Cortex-M3 RTOS version is working, the MeCam
is running a PREEMPT_RT kernel on the Cortex-A9s. The
auto-pilot process is given a priority of 90 using the SCHED_FIFO
The auto-pilot uses Euler angles (i.e. roll, pitch, and yaw) but due to the
gimbal lock effect,
that is not sufficient for navigating. The solution to that problem is to
which use four numbers in a complex number space. That requires a math
library and floating-point numbers, which is a problem for the Cortex-M3
version because it doesn't have any floating-point support. There are
plans to use a fixed-point library to work around that.
To control the movement once the desired direction has been calculated, the
MeCam uses a proportional-integral-derivative
(PID) controller. The PID controller uses a feedback loop that
produces movement that smoothly narrows in on the goal location without
overcompensating. In addition, its "implementation is very
straightforward", Gentil said. There are constants used in the PID
algorithm, which can either be derived experimentally or calculated
theoretically using a program like MATLAB. AI chose the experimental approach,
and he recommended the PID
without a PhD article for those interested.
There is an ultrasonic altitude sensor that uses frequencies above 40kHz to
determine how far the copter is above the ground so that it can maintain a
constant height. It uses the time for an echo return to determine its height. Someone asked about it getting
confused when flying past a cliff (or off the edge of a table), but Gentil
said there is a barometer
that is also used for more coarse altitude information and that it would
detect that particular problem.
The OMAP4 has a "bunch of video coprocessing stuff" that is used by the
MeCam. The camera data is routed to two different tasks, one for streaming
to the phone, the other for face detection. It uses Video4Linux2
(V4L2) media controls to control the camera and its output. He mentioned
the yavta (Yet Another
V4L2 Test Application) as an excellent tool for testing and debugging.
The camera sensor provides multiple outputs, which are routed to resizers
and then to the live streaming and face detection. With the OMAP4 and
V4L2, "you can definitely do crazy things on your system", he said. For
streaming, the MeCam uses Gstreamer to produce Real Time Streaming Protocol
Gentil had various demos, including the copter operating from a four-way
stand (tethered at each corner), from a two-way stand (tethered at opposite
corners) to show the PID algorithm recovering from his heavy-handed inputs,
as well as video streaming to attendees' laptops if their browser supported
RTSP. There is still plenty of work to do it would seem, but there is
quite a bit that is functioning already. Current battery life is around 15
but "I think we can do better", Gentil said. One can imagine plenty of
applications for such a device, well beyond the rather self-absorbed
examples shown in the marketing video.
[ I would like to thank the Linux Foundation for travel assistance to attend ELC. ]
Comments (10 posted)
On day two of the 2013 Embedded
Linux Conference, Robert Rose of SpaceX spoke about the "Lessons
Learned Developing Software for Space Vehicles". In his talk, he discussed
how SpaceX develops its Linux-based software for a wide variety of tasks
needed to put spacecraft into orbit—and eventually beyond. Linux
runs everywhere at SpaceX, he said, on everything from desktops to spacecraft.
Rose is the lead for the avionics flight software team at SpaceX. He is
a former video game programmer, and said that some lessons from that work
were valuable in his current job. He got his start with Linux in
1994 with Slackware.
SpaceX as a company strongly believes in making humans into a
multi-planetary species. A Mars colony is the goal, but in order to get
there, you need rockets and spaceships, he said. It is currently expensive
to launch space vehicles, so there is a need to "drive costs down" in order
to reach the goal.
The company follows a philosophy of reusability, which helps in driving
costs down, Rose
said. That has already been tried to some extent with the space shuttle
program, but SpaceX takes it further. Not only are hardware
components reused between different spacecraft, but the software is shared
as well. The company builds its rockets from the ground up at its
facility, rather than contracting out various pieces. That allows for
closer and more frequent hardware-software integration.
One thing that Rose found hard to get used to early on in his time at
SpaceX is the company's focus on the "end goal". When decisions are
people will often bring it up: "is this going to work for
the Mars mission?" That question is always considered when decisions are
being made; Mars doesn't always win, but that concern is always examined,
Some of the challenges faced by the company are extreme, because the
safety of people and property are involved. The spacecraft are dangerous
vehicles that could cause serious damage if their fuel were to explode, for
example. There is "no undo", no second chance to get things right; once
the rocket launches "it's just gonna go". Another problem that he didn't
encounter until he started working in the industry is the effects of
radiation in space, which can "randomly flip bits"—something that the
system design needs to take into account.
There are some less extreme challenges that SpaceX shares with other industries,
Rose said. Dealing with proprietary hardware and a target platform that is
not the same as the development platform are challenges shared with
embedded Linux, for example. In addition, the SpaceX team has had to face
the common problem that "no one outside of software understands software".
SpaceX started with the Falcon rocket and eventually
transitioned the avionics code to the Dragon spacecraft. The obvious
advantage of sharing code is
that bugs fixed on one platform are automatically fixed on the other. But
there are differences in the software requirements for the launch
vehicles and spacecraft, largely having to do with the different reaction
As long as a spacecraft is not within 250 meters of the International Space
Station (ISS), it can take some time to react to any problem. For a
rocket, that luxury is not available; it must react in short order.
False positives are one problem that needs to be taken into
account. Rose mentioned the heat shield indicator on the Mercury 6 mission
(the first US manned orbital flight) which showed that the heat shield had
separated. NASA tried to figure out a way to do a re-entry with no heat
shield, but "eventually just went for it". It turned out to be a false
positive. Once again, the amount of time available to react is different
for launch vehicles and spacecraft.
Quoting Fred Brooks (of The Mythical Man-Month fame), Rose said
"software is invisible". To make software more visible, you need to know what
it is doing, he said, which means creating "metrics on everything you can
think of". With a rocket, you can't just connect via JTAG and "fire up
gdb", so the software needs to keep track of what it is doing. Those
metrics should cover areas like performance, network utilization, CPU load,
and so on.
The metrics gathered, whether from testing or real-world use, should be
stored as it is "incredibly valuable" to be able to go back through them,
he said. For his systems, telemetry data is stored with the program
metrics, as is the version of all of the code running so that everything
can be reproduced if needed.
SpaceX has programs to parse the metrics data and raise an
alarm when "something goes bad". It is important to automate that, Rose
said, because forcing a human to do it "would suck". The same programs run on
the data whether it is generated from a developer's test, from a run on the
spacecraft, or from a mission. Any failures should be seen as an
opportunity to add new metrics. It takes a while to "get into the rhythm"
of doing so, but it is "very useful". He likes to "geek out on error
reporting", using tools like libSegFault and ftrace.
Automation is important, and continuous integration is "very valuable",
Rose said. He suggested building for every platform all of the time, even
for "things you don't use any more". SpaceX does that and has found
interesting problems when building unused code. Unit tests are run from
the continuous integration system any time the code changes. "Everyone
here has 100% unit test coverage", he joked, but running whatever tests are
available, and creating new ones is useful. When he worked on video games,
they had a test to just "warp" the character to random locations in a level
and had it look in the four directions, which regularly found problems.
"Automate process processes", he said. Things like coding standards,
static analysis, spaces vs. tabs, or detecting the use of Emacs should be
done automatically. SpaceX has a complicated process where changes cannot
be made without tickets, code review, signoffs, and so forth, but all of
that is checked automatically. If static analysis is part of the workflow,
make it such that the code will not build unless it passes that analysis
When the build fails, it should "fail loudly" with a "monitor that
starts flashing red" and email to everyone on the team. When that happens,
you should "respond immediately" to fix the problem. In his team, they
have a full-size Justin Bieber cutout that gets placed facing the team
member who broke the build. They found that "100% of software
engineers don't like Justin Bieber", and will work quickly to fix the build
In his transition to becoming a manager, Rose has had to learn to worry
about different things than he did before. He pointed to the "Make
the Invisible More Visible" essay from the 97
Things Every Programmer Should Know project as a source of
inspiration. For hardware, it's obvious what its integration state is
because you can look at it and see, but that's not true for software.
There is "no progress bar for software development". That has led his team
to experiment with different methods to try to do project planning.
Various "off the shelf" project management methodologies and ways to
estimate how long projects will take do not work for his team. It is
important to set something up that works for your people and
set of tasks, Rose said. They have tried various techniques for estimating
time requirements, from wideband delphi to
scheduling and found that no technique by itself works well for the
group. Since they are software engineers, "we wrote our own tool", he said
with a chuckle, that is a hybrid of several different techniques. There is
"no silver bullet" for scheduling, and it is "unlikely you could pick up
our method and apply it" to your domain. One hard lesson he learned is
that once you have some success using a particular scheduling method, you
"need to do a sales job" to show the engineers that it worked. That will
make it work even better the next time because there will be more buy-in.
Some technical details
Linux is used for everything at SpaceX. The Falcon, Dragon, and
Grasshopper vehicles use it for flight control, the ground stations run
Linux, as do the developers' desktops. SpaceX is "Linux, Linux, Linux", he
Rose went on to briefly describe the Dragon flight system, though he
said he couldn't give
too many details. It is a fault-tolerant system in order to satisfy NASA
requirements for when it gets close to the ISS. There are rules about how
many faults a craft needs to be able to tolerate and still be allowed to
approach the station. It uses triply redundant computers to achieve the
required level of fault tolerance. The Byzantine
generals' algorithm is used to handle situations where the computers do
not agree. That situation could come about because of a radiation event
changing memory or register values, for example.
For navigation, Dragon uses positional information that it receives from
the ISS, along with GPS data it calculates itself. As it approaches the
station, it uses imagery of the ISS and the relative size of the
station to compute the distance to the station. Because it might well be
in darkness, Dragon
uses thermal imaging as the station is slightly warmer than the background.
His team does not use "off-the-shelf distro kernels". Instead, they spend
a lot of time evaluating kernels for their needs. One of the areas they
focus on is scheduler performance. They do not have hard realtime
requirements, but do care about wakeup latencies, he said. There are tests
they use to quantify the performance of the scheduler under different
scenarios, such as while stressing the network. Once a kernel is chosen,
"we try not to change it".
The development tools they use are "embarrassingly non-sophisticated", Rose
said. They use GCC and gdb, while "everyone does their own thing" in terms
of editors and development environments. Development has always targeted
Linux, but it was not always the desktop used by developers, so they
have also developed a lot of their own POSIX-based tools. The main reason for
switching to Linux desktops was because of the development tools that "you
get out of the box", such as ftrace, gdb (which can be directly attached to
debug your target
platform), netfilter, and iptables.
Rose provided an interesting view inside the software development for a
large and complex embedded Linux environment. In addition, his talk was
more open than a previous SpaceX talk we
covered, which was nice to see. Many of the techniques used by the
will sound familiar to most programmers, which makes it clear that the
process of creating
code for spacecraft is not exactly rocket science.
[ I would like to thank the Linux Foundation for travel assistance to attend ELC. ]
Comments (5 posted)
Law, government regulations, and public policy rarely pique the
interests of free software developers, which is unfortunate
considering the impact these topics have on the success of free
software. At SCALE
11x in Los Angeles, Mario Obejas provided an interesting look into
the "dark arts" of policy-making, with an account
of his trip to Washington DC to speak at a public forum on open source
software in the Department of Defense (DoD). Obejas went to the forum
on his own initiative, not his employer's, and his session reported on
his presentation as well as those from big defense contractors,
lobbying groups—and some well-known open source software
Departments, sub-departments, and agendas
Obejas works in information systems and engineering for a large
defense contractor, although he declined to say which one because he
wished to emphasize that he was not appearing or speaking on the
company's behalf. The DoD forum in question was held in January 2012
by the Defense Acquisition Regulations System (DARS), a directorate of
the Defense Procurement and Acquisition Policy (DPAP). At the name
suggests, that office sets procurement policy for the DoD, which
includes software and IT contracts in addition to aircraft carriers
and similar high-visibility items. The forum was announced through the
DPAP web site in December 2011, along with three agenda items about
which the DoD solicited public input.
The first topic asked "What are the risks that open source
software may include proprietary or copyrighted material incorporated
into the open source software without the authorization of the actual
author," thus exposing the contractor and the
government to a copyright infringement liability (in this case, presumably, the
question was about material copyrighted by parties other than the contractor, of
course). The second topic asked whether contractors were
"facing performance and warranty deficiencies to the extent that
the open source software does not meet contract requirements, and the
open source software license leaves the contractors without
recourse." The third item was the question of whether the
Defense Federal Acquisition Regulation Supplement (DFARS) should be
revised to "specify clearly the rights the Government obtains
when a contractor acquires open source software for the
Obejas submitted a presentation proposal to the DoD
forum as an individual open source enthusiast, he said, and
was accepted. In January, he joined the other participants in a
decidedly non-fancy auditorium chamber. Looking around the room, he
said, the crowd included "lawyer, lawyer, lawyer, VP of engineering
for Boeing, lawyer, lawyer, lawyer," and so on. In fact, he estimated
that he was one of only three or four non-lawyers in the room, "but
this is how policy gets crafted."
The first presenter at the forum was Black Duck, a vendor of
license compliance and source code management software products. On
the copyright infringement topic, the Black Duck speaker commented that the reverse
situation was considerably more likely to happen—open source code
wandering into a proprietary product—or that open source code
with conflicting licenses would be merged into a combined work. On
the performance deficiency topic, Black Duck noted that contractors
have the option of contracting with third-party open source companies
for support if they encounter deficiencies with an open source
components. The company spokesperson did not offer an answer on the
DFARS agenda item, but did comment that the benefits of open source
are too great to ignore, and advised that the DoD ensure its
contractors are open source savvy to manage the risks involved.
Obejas spoke second. He briefly addressed the three agenda items,
beginning by reminding the attendees of the DoD's own FAQ asserting
that open source software is commercial and off-the-shelf, so
it should be treated like other commercial off-the-shelf (COTS)
products. Consequently, it follows that the risks asked about in
the first two topics should be addressed in the same way as they are
with proprietary software. On the third question about revising DFARS
for clarity, Obejas said "I must be too much of an engineer but I
don't see the downside of the DFARS being more specific."
But Obejas then introduced a major point of his own, arguing that open
source adoption in DoD projects is inhibited by fear stemming from the
misconception that the GPL forces public disclosure of source code.
In reality, of course, the GPL allows a vendor to provide the
corresponding source code of a binary to the customer when
the binary is delivered. The customer in this case would be the DoD,
who is not likely to redistribute the code to others. But the
misconception that the source must be made available to the public
persists, and contractors avoid GPL-licensed components as a
Obejas cited several factors contributing to this hangup, including
the fact that the GNU project does not provide black-and-white rules
on what constitutes a combined work under the GPL and what does not.
Instead it uses "mushy" language like "communicate at arms length" and
"partly a matter of substance and partly a matter of form." Managers
react to this nebulous language with fear, and open source adoption is
impeded. Better education and more specifics are the remedy, he
said. The DoD FAQ and other documents already advise contractors that
ignoring open source software makes them less competitive, but there
is a disconnect between this policy and reality on the ground.
Furthermore, the DoD has consulted with groups like the Software
Freedom Law Center and has written Memoranda
Of Understanding (MOU) documents, but they remain unpublished.
After Obejas, a representative from the OpenGeo
project spoke about the copyright infringement and performance
deficiency agenda topics, also advising that
open source software does not carry different risks of copyright
infringement or warranty liability than does proprietary software.
There was also a paper submitted to the forum by the Aerospace
Industries Association (AIA), a lobbying group. The paper was
wishy-washy at some times and dead wrong at least once, Obejas said.
On the copyright infringement topic, it listed examples of potential copyright
infringement risks, but did not provide any statistics of prevalence.
On the performance and warranty question, it said contractors are
"absolutely without recourse" under open source licenses—but
that that was also true of using proprietary software. It incorrectly
claimed that open sources licenses prohibit the use of the software in
"hazardous" activities, he said, and incorrectly said that the GPL was
at odds with US export law. It also requested more clarity on
incorporating GPL code, and cited the 2012
issue of the Apache License's indemnification clause as an example
where licensing can be tricky to understand.
The next speaker was Craig Bristol, an intellectual-property lawyer
at DoD contractor Raytheon. He mostly discussed the aforementioned
Apache license "debacle," and also stated publicly that he did not
believe that open source and proprietary software should be treated
the same. Obejas said that he respectfully disagreed with that
viewpoint, but that it did concern him that a major contractor's IP
lawyer took a public stance at odds with the DoD's written
The final speaker was from Red Hat, and did not address
either of the first two agenda topics. Red Hat's speaker did say it
was important to remove discrimination against open source, hailing
the 2009 DoD memo on open source usage
and the 2011 Office of Management and Budget (OMB) memo on
software acquisition. Red Hat was wary of changing the DFARS,
however, saying the DoD must make sure it addressed both open source
and proprietary software and that it does not build on legacy
The DoD forum included a question-and-answer session, during which
Obejas requested that the DoD publish its memos and accounts of its
conversations with the Free Software Foundation (FSF) and related
groups. Several others agreed that the complexity of current software
development regulations is making matters worse, and that an effort
should be made to reduce regulatory complexity.
The session ended with comments from Dan Risacher, who is the DoD's
official "Developer Advocate" and was the primary author of the 2009
memo. Risacher said that the government has done the legal research
and determined that open source software meets the definition of
commercial computer software as defined by the DFARS—even though
some "non believers" at the forum disagreed with that conclusion.
He then responded to two points from the AIA paper. First, he said
that when contractors encounter any warranty problems with open source
software components, those are the contractor's problem to deal with:
You have the source code so you can fix it , so
the idea that there’s not a warranty from the copyright holder .... is
kind of irrelevant and I don’t think there’s a need for a DFARS change
or any other sort of policy change, right? If your contract says
you’re responsible for warranting that software, you’re responsible
for ... open source, for proprietary, for all those components.
He also rejected the notion that the GPL requires public
disclosure, calling it "completely a misunderstanding of the
Risacher's comments marked the end of the DoD forum recap, but
Obejas also addressed two "weird edge cases" encountered during the
event. The first was the Apache indemnification clause debate. The
issue stemmed from clause 9 of the Apache License, which says that the
licensee may choose to offer indemnity to its clients, but if it does
so it must also extend that same indemnity to the upstream project. A
US Army lawyer interpreted that clause incorrectly, Obejas said, and
commented publicly that the Apache License loads unlimited legal liability
onto the government in the form of liability guarantees to multiple
(upstream) third parties. That would run afoul of regulations. The
Apache project responded that the indemnification clause is only
triggered if the licensee chooses to offer indemnity.
Eventually the Army came around and dropped its objection in March
2012, Obejas said, but considerable time and effort were consumed in
The second edge case he described was an unnamed government
contractor who built a software product based on classified
modifications to a GPL-licensed work. The contractor asked the
government to give it a waiver from copyright law compliance, so that
it would not be required to deliver the relevant source code. The
notion that a company would knowingly violate copyright law is
objectionable enough, Obejas observed, but the contractor's concern
was also based on the misconception that the GPL requires public
source code disclosure. In reality, the US government is the
contractor's customer, and would not distribute the binary (or source
code) to anyone itself.
In conclusion, Obejas said he can muster little sympathy for a
company that starts building a product with components it acquired
through a well-known open source license, then expects the government
to relieve it from its legal obligation. Contractors are responsible
for the components they deliver to the government, and should pick
them prudently—including complying with open source licenses.
The DoD already recognizes the value of open source software, he said.
He wants contractors (including the company he works for) to
recognize that value as well, and to utilize it.
Moving forward, Obejas noted several practical obstacles to
increasing open source adoption in DoD contract work. Contractors'
Information Systems offices do not always agree with the DoD's
interpretation of licensing issues, he said. Contractors have
different incentives than their government customers, and they may
decide that complying with open source licenses takes more work than
sticking with proprietary software. Inertia is a another real
problem; defense contractors frequently want to see precedents before
they adopt a new strategy. Consequently, very few contractors want
to be the first to try something new, such as using open source
That last point was where Obejas said that publishing more legal
case studies and clearer guidelines on license compliance—in
particular on how to combine proprietary and free software components
in a single product—would attract more contractors to open
source. On Sunday afternoon, Obejas raised the issue in Bradley
Kuhn's talk about the AGPL. Kuhn's reply was twofold. First, he said
that there were already sufficient public examples of how to combine
GPL components with non-free components, including compliance actions
and court cases.
But more importantly, providing clear guidelines about how to
combine free and non-free components without running afoul of the
license runs contrary to the FSF's (or other licensor's) goals. It
would amount to unilaterally "disarming," Kuhn said. A guide to how to
walk the line between compliance and non-compliance would essentially
be a guide to how to skirt the license when making derivative works.
In addition, any such published guidelines would be fodder for
precedent in future court cases—including cases between third
parties. Fundamentally, the goal of the copyleft movement is to
expand the scope of software freedom, he said; it is unreasonable to
expect copyleft proponents to weaken their position just to provide
"clarity" for people not interested in furthering the principle of
Of course, groups like the FSF and SFLC should be expected to take
a hardened position on combining free and non-free software; they
exist for the purpose of promoting free software. It is the defense
contractors whose decisions are motivated by other factors (such as
profitability), which will shape how they select components and which
subcontractors they work with. But defense contractors are in an
unusual position in one big respect: they have one large client (the
government), and they can be fairly certain that the client will not
give away their product to the public and cost them future revenue.
It is impressive how far open source has worked its way into the DoD
contractor world already—whichever direction it heads in the
days to come. Or, as Obejas put it, whatever else one thinks about
the government, it is pretty cool to stop and notice that the DoD even
has a position like the one Dan Risacher's occupies: developer advocate.
Comments (9 posted)
Page editor: Jonathan Corbet
Inside this week's LWN.net Weekly Edition
- Security: Oxford blocks Google Docs; New vulnerabilities in kernel, openafs, openjdk-6, sudo, ...
- Kernel: 3.9 merge window conclusion; big LITTLE MP; Simplifying RCU; Namespaces part 6.
- Distributions: Ubuntu unveils its next-generation shell and display server; Ubuntu, openSUSE, Puppy, ...
- Development: Static site generators; Upstart 1.7; Zero Install 2.0; LibreOffice moves to make; ...
- Announcements: SFC annual report, PyCon AU CfP, LPC Android microconference, ...