Free software support for virtual and augmented reality

A talk at the recent X.Org Developers Conference in Montréal, Canada looked at support for "XR" in free software. XR is an umbrella term that includes both virtual reality (VR) and augmented reality (AR). In the talk, Joey Ferwerda and Christoph Haag from Collabora gave an overview of XR and the Monado project that provides support for those types of applications.

Ferwerda started by defining the term "HMD", which predates VR and AR. It is a head-mounted display, which basically means "taking a screen and some sensors and duct-taping it to your face". All of the devices that are being used for XR are HMDs. They typically include some kind of tracking system to determine the position and orientation of the HMD itself. Multiple different technologies, including inertial measurement units (IMUs), photodiodes, lasers, and cameras, are used to do the tracking depending on the device and its use case.

AR is intended to augment the real world with extra information; the user sees the real world around them, but various kinds of status and additional data is tagged to objects or locations in their view of the world. AR is a rather over-hyped technology these days, he said. The general idea is that users would wear glasses that would augment their view in some fashion, but, unfortunately, what most people think of as AR is Pokémon Go.

VR uses two screens, one for each eye, to create a 3D world that the user inhabits and can interact with in some fashion. Instead of seeing the real world, the user sees a completely separate world. There are two words that are often used to describe the feel of VR, he said: "presence" and "immersion". That means users are aware of themselves as being part of the VR environment.

XR encompasses both. Ferwerda said that he is not really sure what the "X" stands for; he has heard "cross reality" and "mixed reality" for XR. Haag said that "extended reality" was another definition that he had heard.

Monado and OpenXR

The project that they have been working on is Monado, which is a free-software OpenXR runtime for Linux. OpenXR is a standard from the Khronos Group, which is also the organization behind OpenGL, Vulkan, and various other graphics-related standards.

Ferwerda made way for Haag, who said that prior to OpenXR, each XR engine had to support each different XR runtime. The XR runtimes are generally device-specific. That creates something of a combinatorial explosion to support all of the devices from each engine, as can be seen on slide 16 in their slides [PDF]. With OpenXR, the device runtimes provide an application interface that allows any XR engine to work with any device runtime that supports the standard. Eventually, the OpenXR standard will add a device layer between the device runtimes and the hardware, he said.

The stack for XR handling consists of three layers. There is a program that does VR or AR sitting atop a software platform that does rendering and handles the input devices. That platform interfaces with the hardware that has the sensors to provide the input as well as the devices to display the rendered data.

Haag went through a brief introduction to the application API, which is similar in some ways to the Vulkan API for 3D graphics. More information can be found in the slides, in the YouTube video from XDC, or on the OpenXR site. The standard is still a work in progress, he said, so it doesn't support everything needed for AR yet, for example.

An application will start by creating an XR instance that is a handle to be used throughout. There are modes for handheld devices, such as smartphones, and for HMDs, as well as view modes for single or dual displays, which can be attached to the instance. Sessions are created for a particular rendering type; there are multiple types available, such as OpenGL, OpenGL ES, Vulkan, and several versions of Direct3D. There is no support for multiple sessions at this point, so you cannot overlay two (or more) application's output, though he thinks that will change in the future. VR desktops will need that ability, for example.

Rendering is handled by a cycle of three calls: waiting for the previous frame to finish rendering, beginning the next frame, and ending it. The position data for the next frame will be predicted by OpenXR so that the scene can be rendered for the expected orientation and position (i.e. "pose") when the frame will be displayed. Inputs from controllers are handled in the API as "actions", which are similar to those in the OpenVR API from Valve for its SteamVR platform.

The last year or two have been "really good" in terms of supporting XR graphics on Linux, Ferwerda said, taking back the microphone; there have been additions to the kernel and the display drivers to make things work better. Extended mode has been around for a while; in that mode, the VR display is treated as another screen where windows can be placed to display the content. That works, but there is somewhat painful manual setup for users and it introduces extra latency, he said.

Direct mode for VR graphics has been added more recently. It relies on Keith Packard's work to allow device "leases" for rendering using kernel modesetting (KMS) drivers without X or Wayland getting in the way. In order to get a "less nausea-inducing" display when the generated frame rate is lower than the rate supported by the device, reprojection is used to duplicate frames or make small changes to existing frames based on movement predictions to fill in, he said.

The VR hardware is pretty much all Vulkan-based, so that is what is used for rendering. If the application is OpenGL-based, it can use the Vulkan-OpenGL interoperability mode, but that only works for some hardware, Ferwerda said. Display correction is another piece of the puzzle; lenses are not perfect, they have distortion and chromatic aberrations that need to be corrected for.

Hardware support

For hardware, Monado supports "whatever we can find". That includes the Open Source Virtual Reality (OSVR) Hacker Developer Kit (HDK), which is no longer available but well supported. There has been a lot of reverse-engineering work done by the project for supporting other devices. In addition, the OpenHMD project has a C library that has at least some partial support for a wide range of hardware.

The project has also gotten the Razer Hydra game controller "working to some degree". That week, support for the PlayStation VR and PlayStation Move controller was merged, he said. This is the first full-featured support for a device that people already have or can fairly easily get their hands on (for less than €300), he said.

There are other notable free and open-source software projects, he said. One is libsurvive, which supports the lighthouse tracking used by the HTC Vive HMD. Another is maplab, which supports tracking using simultaneous mapping and tracking (SLAM). For SLAM tracking, the headset is actually mapping the room it is in; it is the "next big thing" in XR.

Even though XR is relatively new, Ferwerda said, there has been a community looking at the devices from a free and open standpoint for some time now. People have gotten the devices and been frustrated that there was no driver for them, so they started reverse engineering them. He has been part of that community since 2013, going to FOSDEM and other events, "getting an apartment and a lot of beer" with like-minded folks to work on making the hardware work on various operating systems. He noted that an OpenBSD developer spent a lot of time getting that system ready for VR, which put it ahead of Linux at the time.

Reverse engineering is a major part of getting the hardware to work, but there is hope that this will eventually change. There have been efforts to create more open platforms, but that is not a reality yet, he said. These devices share a fair amount of the same components, which makes it somewhat easier, but it still takes a lot of looking at Wireshark traces to see if you can find accelerometer and gyroscope information from the IMU, for example. Figuring out how to access the camera and how to interface with the display modes is also part of the fun.

Next up was a demo, of sorts, which can be seen in the photo (and in the video, of course). It was a recording of the classic glxgears running in a VR context. It is displayed by a custom compositor, displaying output rendered by Vulkan, and was shown in the orientation required by a particular VR headset. Each display is rotated and corrected for the needs of that device.

It has been a lot of fun working on Monado, Ferwerda said. The project was able to release an OpenXR implementation at the same time the specification was released. Only Monado and Microsoft were able to release a usable implementation at that time, "which was really cool". They hope to continue working on Monado to add more support for OpenXR and XR hardware; he asked for those interested to get in touch with the project to help make that a reality.

[I would like to thank the X.Org Foundation and LWN's travel sponsor, the Linux Foundation, for travel assistance to Montréal for XDC.]