Accessibility and the graphics stack

Posted Oct 23, 2014 8:23 UTC (Thu) by nim-nim (subscriber, #34454)
Parent article: Accessibility and the graphics stack

Actually, when inputing text, you have four (not two) different concepts:

1. the hardware key location (hardware or virtal hardware)
2. the symbol code this location is mapped to (unicode codes or application actions)
3. the language which is being inputed (language being actually a locale, the same language is not written the same way everywhere, the unicode/opentype rendering changes, the spellcheck dictionnary changes, you can not give safely symbols to apps without locale context)
4. the app localization (the locale used to render its UI) Actually this fourth is a fake, it should not matter a bit on the input side, but C locales confuse it with the rest so it participates in the problem today when it should not

To get accessibility and i18n working (and you need to do both simultaneously, every disabled person does not work in a single locale and every locale has both disabled and non disabled persons) your stack must work very hard to avoid confusing the four concepts.

So, while you may want to transmit both keyboard codes and keysyms to apps, it is a very dangerous thing to do. Because then app devs confuse both concepts and map keyboard locations to actions (many many games make this mistake, starting with the "console key" which is *not* a console key and is needed to input human languages in various parts of the world). They assume those locations are "free to use" because in *their* locale those locations do not map to symbols they care about. And then the app is not i18n-safe, switching to any i18n mapping that uses the same location for something important to this particular locale will break. A localized app must allow a different mapping for actions per locale, keys symbols available for action overriding are not the same in every locale.

So by all means allow virtual keyboards to emit key locations, but this should never ever be exposed to anything but the component responsible from mapping those locations to symbols. Otherwise different parts of the app stack will fight for the ownership of the key locations, depending of which part wins things will break and users will be unhappy.

Likewise X (and IIRC wayland too) makes the mistake of confusing the input mapping with the language being inputed. They are not one and the same, it is common for the same symbol to be mapped to different locations depending on the layout the user uses (typically, azerty vs qwerty, so a and z move), but that does not mean the user of an azerty (French) layout wants a and q to be switched when he starts writing English text. As the article states "Belgian French, Canadian French, and French in France do not share the same symbol mappings". They are all used to input French. A Belgian braille user do not want his mapping to change just because he is contributing to a document in Canadian French (the dictionnaries are not exactly the same, you use the one of the target locale).

But there is no way to indicate apps the target language you are writing, so most of them infer it from the your input mapping ! This is seriously broken compared to other OSes. And don't get me started on the common UI locale = text locale confusion (many people work in an English locale because they feel the UI localization sucks, but they are still writing text for non-English locale targets)

So while reusing x input concepts in wayland was a good idea to get started quickly, they really need a refresh unless wayland gets stuck with the same i18n and atk problems that plagued X for decades.

to post comments

Accessibility and the graphics stack

Posted Oct 23, 2014 9:28 UTC (Thu) by daniels (subscriber, #16193) [Link] (5 responses)

a) Neither X11 nor Wayland have any notion of a UI locale. All they provide is keyboards with a specified mapping; that's it.

b) Wayland does offer a keysym-based interface which is entirely usable by specialist input methods (e.g. Maalit has been ported for a couple of years), and at no point requires decomposition to keycodes. This is something which would be perfectly usable by a braille input method, today, without any changes. The only reason the keycode-based interface also exists is because you can't handle shortcuts otherwise.

c) Wayland and X11 share the XKB _dataset_ (and not protocol) for keymaps and layout handling. Everything else was done from scratch.

d) The reason for the offset-by-8 is that the minimum valid keycode in X11 is 8, whereas it's 0 with evdev. So we just blindly add 8 to the keycode mapping; something which survived into Wayland because the dataset already reflected that.

Accessibility and the graphics stack

Posted Oct 23, 2014 16:02 UTC (Thu) by nim-nim (subscriber, #34454) [Link] (4 responses)

We agree that a) is the only thing available in GUI. UI is locale is available from lower in the stack. However since it is *not* what apps need it is a problem.

What apps want is :
1. the locale of the text which is being written (to apply spellcheck and other locale-specific processing), ie the language toolbar MS deployed for office then generalized in the 90's
2. what hardware key combos are free to use for keyboard shortcuts (and how they are labelled/where they are located so they can show something in config settings)

Since the only thing they can get is a) they try to infer the rest from it which can not work reliably since a) is not designed for that.

Accessibility and the graphics stack

Posted Oct 23, 2014 16:32 UTC (Thu) by daniels (subscriber, #16193) [Link]

Again, shortcut handling belongs at a higher level than raw keyboard layout handling. This has to account for things like negotiation between global and local shortcuts (e.g. when the media keys such as Play are pressed, who handles it? if the foreground application, then use that, else try to find a backgrounded media player?).

The shortcut handling itself is also intensely difficult - for instance, when you have mixed layouts such as Russian and US, you want to base the shortcuts on the US layout, but then you can throw in mixed Dvorak/US for fun.

It's not that this isn't supported, but just that it's the wrong level to support it. The kernel doesn't deal with keymaps at all and just forwards on keycodes at all; the window system doesn't, at its lowest/fundamental level, deal with shortcuts or locales either.

Nothing in Wayland precludes this from being built on top, at all; in fact, it was designed with the explicit intent of being able to do so.

Accessibility and the graphics stack

Posted Oct 30, 2014 16:50 UTC (Thu) by arielb1@mail.tau.ac.il (guest, #94131) [Link] (2 responses)

Wayland is not in the business of handling locale settings (whether the dictionary should be French or French-Canadian). It does provide keymaps, which translate physical keyboard buttons to the symbols printed on them.

Accessibility and the graphics stack

Posted Oct 30, 2014 21:43 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (1 responses)

> the symbols printed on them

Guess I'm screwed[1] then, huh? :P

[1]http://cdn.shopify.com/s/files/1/0152/0433/products/Top_V...

Accessibility and the graphics stack

Posted Nov 2, 2014 15:41 UTC (Sun) by nix (subscriber, #2304) [Link]

No, no, you just get a whole bunch of XK_space out the other end!

Accessibility and the graphics stack

Posted Oct 24, 2014 20:27 UTC (Fri) by epa (subscriber, #39769) [Link] (1 responses)

Yes, I've never understood why switching input language should also switch the keyboard layout at the same time. If the key is marked Q or @ it should stay like that and not change to A or " just because I want a different spellchecker. I think all major platforms (Win and Mac too) make this mistake. Odd, given the number of users who speak more than one language and need to work in both (without plugging in a new keyboard each time...)

Accessibility and the graphics stack

Posted Oct 25, 2014 8:09 UTC (Sat) by nim-nim (subscriber, #34454) [Link]

Actually, they do not make this mistake, the windows language bar for example let the user map different locales to the same layout from day one (it certainly helped that it was originally written by the office team which understands text constrains; but it was never restricted to office unlike the various toys we get *nix-side)

You can see from dstone denials understanding still has not come to the X/wayland devs (so much the pity)