LWN.net Weekly Edition for March 7, 2013

ELC: Nano-copters!

The Embedded Linux Conference often has talks about interesting Linux-powered devices, which should come as no surprise, but talks about devices that fly tend to attract a larger audience. Gregoire Gentil's presentation on a video "nano-copter" was no exception. While there was no free-flying demo, there were several tethered demos that clearly showed some of the possibilities of the device.

Gentil started his talk by showing a YouTube video of the marketing-level pitch from his company, AlwaysInnovating (AI), for the MeCam device. The MeCam is a small quad-copter that allows people to video their every move as it follows them around. The video will stream to their mobile phone, where it can be uploaded to Facebook, Twitter, and the like.

The device itself, pictured at right, is a "flying Pandaboard" with "lots of modifications", Gentil said. The copter has four propellers, a camera, and can communicate via WiFi. He had one of the copters tethered to a stand so that it could go up and down, but not "kill anybody if something goes wrong", he said with a laugh.

The copter runs an OMAP4 with dual Cortex-A9 processors. It uses pulse-width modulation (PWM) to control each of the four propeller motors. The camera has 1080p resolution and uses CSI-2 for the data interface as USB is not fast enough, he said. There are also a "bunch of sensors", including Euler angle sensors, altitude sensors, and wall-detection sensors.

There is a single battery powering both the CPU and the motors, which is different from some radio-controlled (RC) copters, he said. The motors are quite small, and run at 10,000-15,000rpm. That can create a lot of noise in the electrical system, so many RC devices use two separate batteries. Instead, AI has added a lot of filtering on the power, so that it could avoid the extra weight of an additional battery.

The MeCam is not an RC device, instead it has an auto-pilot that is voice controlled and uses facial recognition to position itself. AI considered three different possibilities for running the auto-pilot code. The first was to run a standard Linux kernel, which would give them a "mature full environment" in which to develop and run the code. The downside is the latency. There were times when the motors would not be serviced for 50ms, which was enough time to cause the MeCam to "crash against the wall", Gentil said.

The second option was to use the realtime Linux kernel, which has "much better latency", but is "less mature than the standard kernel". That is the approach being used now, but AI is pursuing another approach as well: writing a custom realtime operating system (RTOS) for the Cortex-M3 which is present on the OMAP4. That will allow the system to have "perfect latency", he said, but it "will be complex to develop and is not mainstream".

For the demos, and until the Cortex-M3 RTOS version is working, the MeCam is running a PREEMPT_RT kernel on the Cortex-A9s. The auto-pilot process is given a priority of 90 using the SCHED_FIFO scheduling class.

The auto-pilot uses Euler angles (i.e. roll, pitch, and yaw) but due to the gimbal lock effect, that is not sufficient for navigating. The solution to that problem is to use quaternions, which use four numbers in a complex number space. That requires a math library and floating-point numbers, which is a problem for the Cortex-M3 version because it doesn't have any floating-point support. There are plans to use a fixed-point library to work around that.

To control the movement once the desired direction has been calculated, the MeCam uses a proportional-integral-derivative (PID) controller. The PID controller uses a feedback loop that produces movement that smoothly narrows in on the goal location without overcompensating. In addition, its "implementation is very straightforward", Gentil said. There are constants used in the PID algorithm, which can either be derived experimentally or calculated theoretically using a program like MATLAB. AI chose the experimental approach, and he recommended the PID without a PhD article for those interested.

There is an ultrasonic altitude sensor that uses frequencies above 40kHz to determine how far the copter is above the ground so that it can maintain a constant height. It uses the time for an echo return to determine its height. Someone asked about it getting confused when flying past a cliff (or off the edge of a table), but Gentil said there is a barometer that is also used for more coarse altitude information and that it would detect that particular problem.

The OMAP4 has a "bunch of video coprocessing stuff" that is used by the MeCam. The camera data is routed to two different tasks, one for streaming to the phone, the other for face detection. It uses Video4Linux2 (V4L2) media controls to control the camera and its output. He mentioned the yavta (Yet Another V4L2 Test Application) as an excellent tool for testing and debugging.

The camera sensor provides multiple outputs, which are routed to resizers and then to the live streaming and face detection. With the OMAP4 and V4L2, "you can definitely do crazy things on your system", he said. For streaming, the MeCam uses Gstreamer to produce Real Time Streaming Protocol (RTSP) data.

Gentil had various demos, including the copter operating from a four-way stand (tethered at each corner), from a two-way stand (tethered at opposite corners) to show the PID algorithm recovering from his heavy-handed inputs, as well as video streaming to attendees' laptops if their browser supported RTSP. There is still plenty of work to do it would seem, but there is quite a bit that is functioning already. Current battery life is around 15 minutes, but "I think we can do better", Gentil said. One can imagine plenty of applications for such a device, well beyond the rather self-absorbed examples shown in the marketing video.

[ I would like to thank the Linux Foundation for travel assistance to attend ELC. ]

Comments (10 posted)

ELC: SpaceX lessons learned

On day two of the 2013 Embedded Linux Conference, Robert Rose of SpaceX spoke about the "Lessons Learned Developing Software for Space Vehicles". In his talk, he discussed how SpaceX develops its Linux-based software for a wide variety of tasks needed to put spacecraft into orbit—and eventually beyond. Linux runs everywhere at SpaceX, he said, on everything from desktops to spacecraft.

Rose is the lead for the avionics flight software team at SpaceX. He is a former video game programmer, and said that some lessons from that work were valuable in his current job. He got his start with Linux in 1994 with Slackware.

SpaceX as a company strongly believes in making humans into a multi-planetary species. A Mars colony is the goal, but in order to get there, you need rockets and spaceships, he said. It is currently expensive to launch space vehicles, so there is a need to "drive costs down" in order to reach the goal.

The company follows a philosophy of reusability, which helps in driving costs down, Rose said. That has already been tried to some extent with the space shuttle program, but SpaceX takes it further. Not only are hardware components reused between different spacecraft, but the software is shared as well. The company builds its rockets from the ground up at its facility, rather than contracting out various pieces. That allows for closer and more frequent hardware-software integration.

One thing that Rose found hard to get used to early on in his time at SpaceX is the company's focus on the "end goal". When decisions are being made, people will often bring it up: "is this going to work for the Mars mission?" That question is always considered when decisions are being made; Mars doesn't always win, but that concern is always examined, he said.

Challenges

Some of the challenges faced by the company are extreme, because the safety of people and property are involved. The spacecraft are dangerous vehicles that could cause serious damage if their fuel were to explode, for example. There is "no undo", no second chance to get things right; once the rocket launches "it's just gonna go". Another problem that he didn't encounter until he started working in the industry is the effects of radiation in space, which can "randomly flip bits"—something that the system design needs to take into account.

There are some less extreme challenges that SpaceX shares with other industries, Rose said. Dealing with proprietary hardware and a target platform that is not the same as the development platform are challenges shared with embedded Linux, for example. In addition, the SpaceX team has had to face the common problem that "no one outside of software understands software".

SpaceX started with the Falcon rocket and eventually transitioned the avionics code to the Dragon spacecraft. The obvious advantage of sharing code is that bugs fixed on one platform are automatically fixed on the other. But there are differences in the software requirements for the launch vehicles and spacecraft, largely having to do with the different reaction times available. As long as a spacecraft is not within 250 meters of the International Space Station (ISS), it can take some time to react to any problem. For a rocket, that luxury is not available; it must react in short order.

False positives are one problem that needs to be taken into account. Rose mentioned the heat shield indicator on the Mercury 6 mission (the first US manned orbital flight) which showed that the heat shield had separated. NASA tried to figure out a way to do a re-entry with no heat shield, but "eventually just went for it". It turned out to be a false positive. Once again, the amount of time available to react is different for launch vehicles and spacecraft.

Gathering data

Quoting Fred Brooks (of The Mythical Man-Month fame), Rose said "software is invisible". To make software more visible, you need to know what it is doing, he said, which means creating "metrics on everything you can think of". With a rocket, you can't just connect via JTAG and "fire up gdb", so the software needs to keep track of what it is doing. Those metrics should cover areas like performance, network utilization, CPU load, and so on.

The metrics gathered, whether from testing or real-world use, should be stored as it is "incredibly valuable" to be able to go back through them, he said. For his systems, telemetry data is stored with the program metrics, as is the version of all of the code running so that everything can be reproduced if needed.

SpaceX has programs to parse the metrics data and raise an alarm when "something goes bad". It is important to automate that, Rose said, because forcing a human to do it "would suck". The same programs run on the data whether it is generated from a developer's test, from a run on the spacecraft, or from a mission. Any failures should be seen as an opportunity to add new metrics. It takes a while to "get into the rhythm" of doing so, but it is "very useful". He likes to "geek out on error reporting", using tools like libSegFault and ftrace.

Automation is important, and continuous integration is "very valuable", Rose said. He suggested building for every platform all of the time, even for "things you don't use any more". SpaceX does that and has found interesting problems when building unused code. Unit tests are run from the continuous integration system any time the code changes. "Everyone here has 100% unit test coverage", he joked, but running whatever tests are available, and creating new ones is useful. When he worked on video games, they had a test to just "warp" the character to random locations in a level and had it look in the four directions, which regularly found problems.

"Automate process processes", he said. Things like coding standards, static analysis, spaces vs. tabs, or detecting the use of Emacs should be done automatically. SpaceX has a complicated process where changes cannot be made without tickets, code review, signoffs, and so forth, but all of that is checked automatically. If static analysis is part of the workflow, make it such that the code will not build unless it passes that analysis step.

When the build fails, it should "fail loudly" with a "monitor that starts flashing red" and email to everyone on the team. When that happens, you should "respond immediately" to fix the problem. In his team, they have a full-size Justin Bieber cutout that gets placed facing the team member who broke the build. They found that "100% of software engineers don't like Justin Bieber", and will work quickly to fix the build problem.

Project management

In his transition to becoming a manager, Rose has had to learn to worry about different things than he did before. He pointed to the "Make the Invisible More Visible" essay from the 97 Things Every Programmer Should Know project as a source of inspiration. For hardware, it's obvious what its integration state is because you can look at it and see, but that's not true for software. There is "no progress bar for software development". That has led his team to experiment with different methods to try to do project planning.

Various "off the shelf" project management methodologies and ways to estimate how long projects will take do not work for his team. It is important to set something up that works for your people and set of tasks, Rose said. They have tried various techniques for estimating time requirements, from wideband delphi to evidence-based scheduling and found that no technique by itself works well for the group. Since they are software engineers, "we wrote our own tool", he said with a chuckle, that is a hybrid of several different techniques. There is "no silver bullet" for scheduling, and it is "unlikely you could pick up our method and apply it" to your domain. One hard lesson he learned is that once you have some success using a particular scheduling method, you "need to do a sales job" to show the engineers that it worked. That will make it work even better the next time because there will be more buy-in.

Some technical details

Linux is used for everything at SpaceX. The Falcon, Dragon, and Grasshopper vehicles use it for flight control, the ground stations run Linux, as do the developers' desktops. SpaceX is "Linux, Linux, Linux", he said.

Rose went on to briefly describe the Dragon flight system, though he said he couldn't give too many details. It is a fault-tolerant system in order to satisfy NASA requirements for when it gets close to the ISS. There are rules about how many faults a craft needs to be able to tolerate and still be allowed to approach the station. It uses triply redundant computers to achieve the required level of fault tolerance. The Byzantine generals' algorithm is used to handle situations where the computers do not agree. That situation could come about because of a radiation event changing memory or register values, for example.

For navigation, Dragon uses positional information that it receives from the ISS, along with GPS data it calculates itself. As it approaches the station, it uses imagery of the ISS and the relative size of the station to compute the distance to the station. Because it might well be in darkness, Dragon uses thermal imaging as the station is slightly warmer than the background.

His team does not use "off-the-shelf distro kernels". Instead, they spend a lot of time evaluating kernels for their needs. One of the areas they focus on is scheduler performance. They do not have hard realtime requirements, but do care about wakeup latencies, he said. There are tests they use to quantify the performance of the scheduler under different scenarios, such as while stressing the network. Once a kernel is chosen, "we try not to change it".

The development tools they use are "embarrassingly non-sophisticated", Rose said. They use GCC and gdb, while "everyone does their own thing" in terms of editors and development environments. Development has always targeted Linux, but it was not always the desktop used by developers, so they have also developed a lot of their own POSIX-based tools. The main reason for switching to Linux desktops was because of the development tools that "you get out of the box", such as ftrace, gdb (which can be directly attached to debug your target platform), netfilter, and iptables.

Rose provided an interesting view inside the software development for a large and complex embedded Linux environment. In addition, his talk was more open than a previous SpaceX talk we covered, which was nice to see. Many of the techniques used by the company will sound familiar to most programmers, which makes it clear that the process of creating code for spacecraft is not exactly rocket science.

[ I would like to thank the Linux Foundation for travel assistance to attend ELC. ]

Comments (5 posted)

SCALE: Advocating FOSS at the DoD

Law, government regulations, and public policy rarely pique the interests of free software developers, which is unfortunate considering the impact these topics have on the success of free software. At SCALE 11x in Los Angeles, Mario Obejas provided an interesting look into the "dark arts" of policy-making, with an account of his trip to Washington DC to speak at a public forum on open source software in the Department of Defense (DoD). Obejas went to the forum on his own initiative, not his employer's, and his session reported on his presentation as well as those from big defense contractors, lobbying groups—and some well-known open source software vendors.

Departments, sub-departments, and agendas

Obejas works in information systems and engineering for a large defense contractor, although he declined to say which one because he wished to emphasize that he was not appearing or speaking on the company's behalf. The DoD forum in question was held in January 2012 by the Defense Acquisition Regulations System (DARS), a directorate of the Defense Procurement and Acquisition Policy (DPAP). At the name suggests, that office sets procurement policy for the DoD, which includes software and IT contracts in addition to aircraft carriers and similar high-visibility items. The forum was announced through the DPAP web site in December 2011, along with three agenda items about which the DoD solicited public input.

The first topic asked "What are the risks that open source software may include proprietary or copyrighted material incorporated into the open source software without the authorization of the actual author," thus exposing the contractor and the government to a copyright infringement liability (in this case, presumably, the question was about material copyrighted by parties other than the contractor, of course). The second topic asked whether contractors were "facing performance and warranty deficiencies to the extent that the open source software does not meet contract requirements, and the open source software license leaves the contractors without recourse." The third item was the question of whether the Defense Federal Acquisition Regulation Supplement (DFARS) should be revised to "specify clearly the rights the Government obtains when a contractor acquires open source software for the Government"

Obejas submitted a presentation proposal to the DoD forum as an individual open source enthusiast, he said, and was accepted. In January, he joined the other participants in a decidedly non-fancy auditorium chamber. Looking around the room, he said, the crowd included "lawyer, lawyer, lawyer, VP of engineering for Boeing, lawyer, lawyer, lawyer," and so on. In fact, he estimated that he was one of only three or four non-lawyers in the room, "but this is how policy gets crafted."

Presentations

The first presenter at the forum was Black Duck, a vendor of license compliance and source code management software products. On the copyright infringement topic, the Black Duck speaker commented that the reverse situation was considerably more likely to happen—open source code wandering into a proprietary product—or that open source code with conflicting licenses would be merged into a combined work. On the performance deficiency topic, Black Duck noted that contractors have the option of contracting with third-party open source companies for support if they encounter deficiencies with an open source components. The company spokesperson did not offer an answer on the DFARS agenda item, but did comment that the benefits of open source are too great to ignore, and advised that the DoD ensure its contractors are open source savvy to manage the risks involved.

Obejas spoke second. He briefly addressed the three agenda items, beginning by reminding the attendees of the DoD's own FAQ asserting that open source software is commercial and off-the-shelf, so it should be treated like other commercial off-the-shelf (COTS) products. Consequently, it follows that the risks asked about in the first two topics should be addressed in the same way as they are with proprietary software. On the third question about revising DFARS for clarity, Obejas said "I must be too much of an engineer but I don't see the downside of the DFARS being more specific."

But Obejas then introduced a major point of his own, arguing that open source adoption in DoD projects is inhibited by fear stemming from the misconception that the GPL forces public disclosure of source code. In reality, of course, the GPL allows a vendor to provide the corresponding source code of a binary to the customer when the binary is delivered. The customer in this case would be the DoD, who is not likely to redistribute the code to others. But the misconception that the source must be made available to the public persists, and contractors avoid GPL-licensed components as a result.

Obejas cited several factors contributing to this hangup, including the fact that the GNU project does not provide black-and-white rules on what constitutes a combined work under the GPL and what does not. Instead it uses "mushy" language like "communicate at arms length" and "partly a matter of substance and partly a matter of form." Managers react to this nebulous language with fear, and open source adoption is impeded. Better education and more specifics are the remedy, he said. The DoD FAQ and other documents already advise contractors that ignoring open source software makes them less competitive, but there is a disconnect between this policy and reality on the ground. Furthermore, the DoD has consulted with groups like the Software Freedom Law Center and has written Memoranda Of Understanding (MOU) documents, but they remain unpublished.

After Obejas, a representative from the OpenGeo project spoke about the copyright infringement and performance deficiency agenda topics, also advising that open source software does not carry different risks of copyright infringement or warranty liability than does proprietary software. There was also a paper submitted to the forum by the Aerospace Industries Association (AIA), a lobbying group. The paper was wishy-washy at some times and dead wrong at least once, Obejas said. On the copyright infringement topic, it listed examples of potential copyright infringement risks, but did not provide any statistics of prevalence. On the performance and warranty question, it said contractors are "absolutely without recourse" under open source licenses—but that that was also true of using proprietary software. It incorrectly claimed that open sources licenses prohibit the use of the software in "hazardous" activities, he said, and incorrectly said that the GPL was at odds with US export law. It also requested more clarity on incorporating GPL code, and cited the 2012 issue of the Apache License's indemnification clause as an example where licensing can be tricky to understand.

The next speaker was Craig Bristol, an intellectual-property lawyer at DoD contractor Raytheon. He mostly discussed the aforementioned Apache license "debacle," and also stated publicly that he did not believe that open source and proprietary software should be treated the same. Obejas said that he respectfully disagreed with that viewpoint, but that it did concern him that a major contractor's IP lawyer took a public stance at odds with the DoD's written declarations.

The final speaker was from Red Hat, and did not address either of the first two agenda topics. Red Hat's speaker did say it was important to remove discrimination against open source, hailing the 2009 DoD memo on open source usage and the 2011 Office of Management and Budget (OMB) memo on software acquisition. Red Hat was wary of changing the DFARS, however, saying the DoD must make sure it addressed both open source and proprietary software and that it does not build on legacy misconceptions.

Discussion

The DoD forum included a question-and-answer session, during which Obejas requested that the DoD publish its memos and accounts of its conversations with the Free Software Foundation (FSF) and related groups. Several others agreed that the complexity of current software development regulations is making matters worse, and that an effort should be made to reduce regulatory complexity.

The session ended with comments from Dan Risacher, who is the DoD's official "Developer Advocate" and was the primary author of the 2009 memo. Risacher said that the government has done the legal research and determined that open source software meets the definition of commercial computer software as defined by the DFARS—even though some "non believers" at the forum disagreed with that conclusion.

He then responded to two points from the AIA paper. First, he said that when contractors encounter any warranty problems with open source software components, those are the contractor's problem to deal with:

He also rejected the notion that the GPL requires public disclosure, calling it "completely a misunderstanding of the license."

Risacher's comments marked the end of the DoD forum recap, but Obejas also addressed two "weird edge cases" encountered during the event. The first was the Apache indemnification clause debate. The issue stemmed from clause 9 of the Apache License, which says that the licensee may choose to offer indemnity to its clients, but if it does so it must also extend that same indemnity to the upstream project. A US Army lawyer interpreted that clause incorrectly, Obejas said, and commented publicly that the Apache License loads unlimited legal liability onto the government in the form of liability guarantees to multiple (upstream) third parties. That would run afoul of regulations. The Apache project responded that the indemnification clause is only triggered if the licensee chooses to offer indemnity. Eventually the Army came around and dropped its objection in March 2012, Obejas said, but considerable time and effort were consumed in the process.

The second edge case he described was an unnamed government contractor who built a software product based on classified modifications to a GPL-licensed work. The contractor asked the government to give it a waiver from copyright law compliance, so that it would not be required to deliver the relevant source code. The notion that a company would knowingly violate copyright law is objectionable enough, Obejas observed, but the contractor's concern was also based on the misconception that the GPL requires public source code disclosure. In reality, the US government is the contractor's customer, and would not distribute the binary (or source code) to anyone itself.

In conclusion, Obejas said he can muster little sympathy for a company that starts building a product with components it acquired through a well-known open source license, then expects the government to relieve it from its legal obligation. Contractors are responsible for the components they deliver to the government, and should pick them prudently—including complying with open source licenses. The DoD already recognizes the value of open source software, he said. He wants contractors (including the company he works for) to recognize that value as well, and to utilize it.

The future

Moving forward, Obejas noted several practical obstacles to increasing open source adoption in DoD contract work. Contractors' Information Systems offices do not always agree with the DoD's interpretation of licensing issues, he said. Contractors have different incentives than their government customers, and they may decide that complying with open source licenses takes more work than sticking with proprietary software. Inertia is a another real problem; defense contractors frequently want to see precedents before they adopt a new strategy. Consequently, very few contractors want to be the first to try something new, such as using open source software components.

That last point was where Obejas said that publishing more legal case studies and clearer guidelines on license compliance—in particular on how to combine proprietary and free software components in a single product—would attract more contractors to open source. On Sunday afternoon, Obejas raised the issue in Bradley Kuhn's talk about the AGPL. Kuhn's reply was twofold. First, he said that there were already sufficient public examples of how to combine GPL components with non-free components, including compliance actions and court cases.

But more importantly, providing clear guidelines about how to combine free and non-free components without running afoul of the license runs contrary to the FSF's (or other licensor's) goals. It would amount to unilaterally "disarming," Kuhn said. A guide to how to walk the line between compliance and non-compliance would essentially be a guide to how to skirt the license when making derivative works. In addition, any such published guidelines would be fodder for precedent in future court cases—including cases between third parties. Fundamentally, the goal of the copyleft movement is to expand the scope of software freedom, he said; it is unreasonable to expect copyleft proponents to weaken their position just to provide "clarity" for people not interested in furthering the principle of copyleft.

Of course, groups like the FSF and SFLC should be expected to take a hardened position on combining free and non-free software; they exist for the purpose of promoting free software. It is the defense contractors whose decisions are motivated by other factors (such as profitability), which will shape how they select components and which subcontractors they work with. But defense contractors are in an unusual position in one big respect: they have one large client (the government), and they can be fairly certain that the client will not give away their product to the public and cost them future revenue. It is impressive how far open source has worked its way into the DoD contractor world already—whichever direction it heads in the days to come. Or, as Obejas put it, whatever else one thinks about the government, it is pretty cool to stop and notice that the DoD even has a position like the one Dan Risacher's occupies: developer advocate.

Comments (9 posted)

Page editor: Jonathan Corbet

Inside this week's LWN.net Weekly Edition

• Security: Oxford blocks Google Docs; New vulnerabilities in kernel, openafs, openjdk-6, sudo, ...
• Kernel: 3.9 merge window conclusion; big LITTLE MP; Simplifying RCU; Namespaces part 6.
• Distributions: Ubuntu unveils its next-generation shell and display server; Ubuntu, openSUSE, Puppy, ...
• Development: Static site generators; Upstart 1.7; Zero Install 2.0; LibreOffice moves to make; ...
• Announcements: SFC annual report, PyCon AU CfP, LPC Android microconference, ...