Accessibility and the graphics stack

At the 2014 X.Org Developers Conference, Samuel Thibault led off three days of presentations with a look at accessibility and how it relates to the X and Wayland graphics stacks. Accessibility is the blanket term for making computers more usable by people with specific needs—not necessarily handicaps. Those needs range from vision problems to loss of motor control, which might be temporary (e.g. a broken arm) or more permanent (e.g. blindness, deafness, or Parkinson's disease)

Introduction

Thibault started the talk with a slide of gnuplot output and asked what accessibility problems it illustrated (slides [PDF]). Since the two functions plotted used red and green lines, the problem is for those with color blindness. He asked those who could not see one or more of the colors to raise their hands and got three from around fifty in the room. Roughly 8% of males and 0.5% of females are affected by color blindness.

Accessibility is helping people who have special needs when interacting with a computer. Blindness is an obvious candidate, but there are also people who have low vision (or color blindness). Deafness is not really an issue for X.Org, but it clearly requires changes to assist those who are affected by it. There are people who can only use one hand ("try to press ctrl-alt-backspace with one hand", he said) or with motor control issues. Elderly people often have several different needs all at once.

But, he asked, why make the graphical user interface (GUI) accessible when text mode is so much easier to handle? There are many things that are not available in text mode, however, JavaScript support for example. There are also business applications that are GUI-oriented. Beyond that, non-technical people need to be able to get help from others, but that is much harder if the two are not running the same applications.

One approach might be to make dedicated software for those with special needs, but that is generally a bad idea, Thibault said. Typically, those types of applications are oriented toward a single disability. His example was edbrowse, which is a combination editor and browser targeted at blind users. That means it is not helpful for those with other disabilities.

Developers would need to work on both the browser (e.g. JavaScript, Flash, and CSS) and on office suite features such as compatibility with Microsoft Office and OpenOffice. With limited developer resources, that becomes an unneeded duplication of effort. Also, when both disabled and non-disabled people are working together, it would be best to be using the same software.

So, one of the design principles for the accessibility work is that it uses the same software, but makes it accessible. That makes it easier to work together and to get help from others. Another principle is that it should support "synchronized work" for two people working together; it should just alternate input and output mechanisms as needed to support both users. Finally, accessible software should be pervasive; it should not require a special software installation or configuration, it should just be available to be enabled on "all platforms all the time".

The status of accessibility support in free software is a bit of a mixed bag. Text mode is generally accessible, he said, but is not good for beginners. GNOME is "quite accessible", though GNOME 3 was a restart from scratch on accessibility. But free software is late compared to Windows, which has a many-year lead on accessibility. Compared to Apple, though, free software support is "Stone Age". The accessibility in Apple's products is easily available and well integrated.

Pressing "a"

Next up was a look at input and output in X, with an eye toward where accessibility fits in. When you press an "a" on a keyboard, it doesn't matter what is printed on that key, it is the first key on the third row and gets converted to a scancode (0x1e) that represents that location. The keyboard driver in the kernel turns that into an internal kernel value (KEY_A, which is 30 or 0x1e) and passes it to the input driver and then to the evdev driver. The evdev driver creates an event that it hands to the input-evdev driver in the X server, which adds 8 (for reasons that were not explained in the talk) before pushing it out over the wire to the X client as a KeyPress.

It is important to recognize that it is still just a location on the keyboard (first key of third row) at that point. The client toolkit will get the event, recognize that it is a KeyPress, and hand it off to Xkb, which finally translates it into a KeySym: an "a". At that point, the KeySym is sent to the widget it was directed at, which may decide to do something with it.

Perhaps the widget wants to append the "a" to its text. These days, clients will typically use a library like Pango to render the text. That generates a PixMap, which is pushed to the driver and the video card to display. It is important to notice that the PixMap is generated early, in the client. But, there may not be a screen to direct the PixMap to. Blind people do not have screens and do not want them. There need to be alternatives on the output side so that blind people do not have to buy screens they will not use.

Input

Certain people can only use certain kinds of input devices. There are those who can only use keyboards; they need to use keyboard shortcuts and to move the cursor using it. Others can only use a joystick (as a mouse, essentially) or only a mouse—to type they have to use some kind of virtual keyboard. There are many more kinds of input devices, because "you never know what people may need".

If you can only use one hand and it is a permanent condition, you may want a faster way to input text. Using a standard keyboard requires a lot of hand motion. One way to deal with that is to have a key that "mirrors" the keyboard, effectively switching between the keys on each side without moving the hand. Thibault initially said that it might be difficult to do that in X, but an audience member said that Xkb can already do those kind of remappings.

AccessX adds a number of features to Xkb and X clients that can be used for accessibility. StickyKeys allows someone to press modifier keys (e.g. ctrl, alt, shift) multiple times to make them "stick" so they can input things like "ctrl-alt-backspace". MouseKeys turns the keyboard into a mouse. For those who have trouble moving their hands quickly or often initially miss the key they wanted, SlowKeys will wait for a bit before deciding a key has been pressed. BounceKeys is useful for those with Parkinson's or other motor control problems; it will see only one key press even if there are several of them in a short period of time. There are others as well.

A virtual keyboard is available with xvkdb. It injects physical positions as KeyPress events on the wire, directly to the client. Some braille devices have regular PC keyboards, so brltty just hands key symbols (e.g. KEY_A) to the kernel input driver.

Actual braille keyboards are a harder problem. These have eight keys that correspond to the eight dots in braille. That gives 256 possibilities, but the representation of some characters and their braille equivalents are not standard. "a" to "z" are standard, but the rest depend on both language and country—Belgian French, Canadian French, and French in France do not share the same symbol mappings. Because a key press is not really a physical position on a standard keyboard, it results in brltty creating a KeySym, rather than a KeyPress. Xkb is used to back-translate the "a" to a KeyPress then to send it into the client.

There are a number of other issues with braille, including users wanting to use certain keys on a regular keyboard to represent the dots from a braille keyboard, composing certain characters (e.g. "ô"), and handling braille abbreviations (which sometimes map to regular characters, with rules to disambiguate). Handling all of that is complex, but some of it will be moving to use ibus soon. Support for ibus was just completed in the most recent Google Summer of Code cycle.

He then asked the audience: what about Wayland? He wondered if it was passing keyboard codes or KeySyms. Ideally, he said, it would handle both which would help remove some of the existing complexity. Someone replied that Wayland was using keyboard codes currently, but it might be possible to change that.

Output

On the output side, there are already a fair number of features that are available. People with low vision can tweak the DPI setting which will result in larger icons and fonts. XRandR can be used for panning and to do some basic zooming; there is also support for gamma tuning and color inversion for assisting those with color blindness. In addition, GTK 3 has a method to do perfect zooming by rendering into a much larger PixMap.

All of that doesn't really help blind people, however. In order to provide information for them, the application must help. In the past, snooping the X protocol was used to help determine what text was being displayed and its context, but that no longer works as PixMaps are sent over the wire. So applications need to have an abstract representation of the information they are displaying that can be passed to a screen reader over a bus. Then the screen reader can "render" the information in a sensible fashion.

So the widget that receives the "a" and is displaying it will also send a signal via the Accessibility Toolkit (ATK) to the screen reader. But that information also needs some context, including the hierarchy of widgets, windows, and so on. Similar to CSS, the application needs to describe its windows, menus, labels, buttons, etc. in a way that the screen reader can query. It can then do the right thing for the user when it puts it on a braille device, speaks the data aloud, or outputs it in some other way.

Making applications accessible from the start is fairly easy, Thibault said. It is a matter of designing without the GUI in mind. If you use the standard widgets, they have already been made accessible. In addition, images should always have some alternate text that the screen reader can use.

There is a tool for testing an application for accessibility called Accerciser. It will show the application and the tree of widgets, so that you can see what a screen reader would get.

Thibault concluded by saying that accessibility has diverse requirements that causes it to have to be plugged in at various levels. When moving to Wayland, there should be no regressions in terms of accessibility, which means that developers need to be paying attention to that now. In the end, accessibility needs the semantics from a desktop application, not just the rendering, so it is important to separate form from content.

For those interested, there is a video [WebM] and, interestingly, a transcript of Thibault's talk.

[ I would like to thank the X.Org Foundation for travel assistance to Bordeaux for XDC. ]