Access to complex video devices with libcamera
In truth, if your editor may interject a brief comment, even the basic V4L API is not entirely straightforward for the uninitiated. There is a negotiation process that must happen so that the application can determine whether a given camera can deliver the sort of data stream that is needed. The number of parameters to tweak is large. It was not uncommon to find applications that worked with some camera devices, but not others. V4L makes many things possible, but even the simplest tasks may not be easy.
libcamera
Now (returning to the talk), consider the situation with contemporary
hardware, which is much more complex. A typical camera device has a
collection of processing blocks (handling tasks like image scaling, color
correction, color-space conversion, autofocus, etc.) that can be
interconnected in a variety of ways. The media
controller subsystem was introduced to expose this complexity to user
space, but it only helps so much. A special-purpose application that, for
example, understands a single video device found on a specific handset can
be managed, but writing an application that can handle a
wide variety of video hardware is challenging at best.
Developers at Nokia had, some years ago, envisioned a plugin-based mechanism for camera configuration but, before it could be implemented, Nokia canceled its smartphone project and development lapsed. Ten years later, libcamera was started with the intent of being "the Mesa of the camera stack"; its purpose is to make it easy for applications to interface with camera devices.
What is envisioned is a four-layer stack:
- libcamera is the lowest-level layer, interfacing directly with the kernel. It is implemented entirely in user space, with no changes to kernel APIs planned.
- A set of bindings will make libcamera available in a range of different programming languages.
- The "adaptation layer" provides a set of interfaces to libcamera for existing applications; they will include a V4L compatibility layer, an Android HAL interface, and a GStreamer interface. The intent is to make libcamera suitable for all Linux-based devices.
- The application layer exists already, in the form of GStreamer, native V4L applications, Android applications, etc. There will also be native libcamera applications in the future.
Application interface
The first thing a libcamera application has to do is to enumerate the available cameras. A "camera" in this context is what users might see as a camera device; much of the underlying complexity (sensor, DMA bridge, processing units, etc.) is hidden within each camera device. These cameras expose a set of capabilities, such as how many concurrent video streams they can support, what types of controls they have, and the resolutions they are capable of. "Profiles" exist as a way of pulling together the capabilities needed for given tasks; there can be profiles for applications like "point-and-shoot camera" or "video conferencing".
The handling of concurrent streams is a key feature of libcamera. For example, a point-and-shoot device might have one mid-resolution stream used to preview a scene on a handset's screen and a full-resolution stream for image capture.
Controls in libcamera can be set on a per-frame basis, hardware permitting. These controls can include exposure time, focus settings, white balance, etc. This is not a useful feature for applications like video conferencing, Pinchart said, but it's important for tasks like face recognition or machine vision, where the application needs to know the parameters associated with each frame.
A native libcamera application will, after enumeration of the available devices, reserve access to the device(s) needed. Access is exclusive in libcamera; if multiplexing is needed, a framework like GStreamer can provide it. There is a configuration stage where the camera creates a template configuration from a set of available roles; the application will then tweak the parameters as needed and validate the result to ensure that the camera can support it. There is, thus, still a negotiation process required, but the creation of an initial configuration should ease that process considerably. This configuration is done for every stream that the application needs.
Once that is done, the application will allocate a set of buffers for incoming video data. A "create request" operation will create a request to capture a single video frame with a given set of parameters and queue it to the camera. Most applications will queue multiple requests to keep the video pipeline flowing; buffers can be turned around and queued with new requests after their data is consumed.
Advanced algorithms
Naturally, there is full support for image-processing algorithms, ranging from automatic exposure, white-balance, and focus setting through to advanced noise reduction and more. There is a balancing act required here: these algorithms, it seems, are often provided by the manufacturer of the camera, and many of them are proprietary software. Libcamera will support them as separate, loadable modules; Pinchart said that we want all of this code to be open source, but that's not the case now and the first priority is to make it all work in a safe and reliable manner.
One important design decision here is that image-processing modules do not talk directly to the hardware; they go through the standard interfaces like everything else. A typical module will get statistics (or image data) from the hardware, compute the optimal image parameters, then use libcamera interfaces to configure the device accordingly. "There will be no secret ioctl() calls", he said. Modules will also be sandboxed to limit the damage they can do to the rest of the system.
The low-level camera device abstraction is designed with as much device-independent code as possible. There are a lot of independent low-level camera implementations now; for example, the Android and ChromeOS teams do not talk to each other and each create their own implementations, he said. The intent is to make it easy for vendors to add support for their devices directly to libcamera so that everybody can work from the same implementation.
The libcamera project is in a relatively early stage of development; no actual releases have been made yet. It worked well enough for Pinchart to do a quick video-conferencing demonstration with an obliging developer in Europe who stayed awake until the time came; the image quality showed that work remains to be done, but the basic pipeline works. If libcamera continues to progress and meets its goals, it seems likely to show up on systems in the not-too-distant future.
[Your editor thanks the Linux Foundation for supporting his travel to the
event.]
Index entries for this article | |
---|---|
Conference | Open Source Summit Japan/2019 |
Posted Jul 25, 2019 22:35 UTC (Thu)
by andy_shev (subscriber, #75870)
[Link] (5 responses)
Posted Jul 26, 2019 4:12 UTC (Fri)
by creemj (subscriber, #56061)
[Link] (4 responses)
Posted Jul 26, 2019 19:01 UTC (Fri)
by guus (subscriber, #41608)
[Link] (1 responses)
1. Cameras and/or framegrabbers requiring firmware to be loaded before they work.
UVC and DC1394 are very nice because they at least have a standard way of setting parameters that almost all cameras use. Gig-E has the GenICam standard, but it's not supported by all cameras.
For USB webcams, setting parameters is trivial nowadays, but actually getting the stream of images is getting more and more complex. The reason is that high end cameras are getting better resolution and framerates, yet are still bound by USB bandwidth limitations. So while early cameras sent simple YUV or JPEG-compressed images, now we have cameras that can actually send a H.264 stream. Videochat applications might want to get the H.264 stream because they can then send it out over the Internet without having to recompress anything.
Posted Jul 31, 2019 14:55 UTC (Wed)
by loose11 (guest, #114677)
[Link]
Some camera manufactures claiming that they implemented the standard. I had a issue with GenICam, where the standard was "implemented" but they made some hidden calls for specific parameters in there library. At the end, they were overriding my parameters.
Posted Jul 27, 2019 5:49 UTC (Sat)
by flussence (guest, #85566)
[Link] (1 responses)
Posted Aug 1, 2019 10:39 UTC (Thu)
by swilmet (subscriber, #98424)
[Link]
In the talk/article:
> libcamera was started with the intent of being "the Mesa of the camera stack"; its purpose is to make it easy for applications to interface with camera devices.
I would be interested to know more about this comparison to Mesa, the libcamera website doesn't mention Mesa. What do libcamera and Mesa have in common?
Posted Jul 29, 2019 12:55 UTC (Mon)
by Kamiccolo (subscriber, #95159)
[Link] (1 responses)
Posted Aug 1, 2019 10:41 UTC (Thu)
by swilmet (subscriber, #98424)
[Link]
Posted Aug 1, 2019 15:48 UTC (Thu)
by lamby (subscriber, #42621)
[Link]
Access to complex video devices with libcamera
Access to complex video devices with libcamera
Access to complex video devices with libcamera
2. Cameras using a proprietary protocol for configuring parameters like exposure time, frame rate and so on.
Access to complex video devices with libcamera
Access to complex video devices with libcamera
Access to complex video devices with libcamera
https://who-t.blogspot.com/2018/07/why-its-not-good-idea-...
Access to complex video devices with libcamera
> Other types of camera, including analog cameras, depth cameras, thermal cameras, external digital picture or movie cameras, are out of scope for this project.
Access to complex video devices with libcamera
Access to complex video devices with libcamera