Display Shell

Posted Jun 20, 2025 15:29 UTC (Fri) by linuxrocks123 (subscriber, #34648)
Parent article: Enhancing screen-reader functionality in modern GNOME

I've discovered that AT-SPI2 is useful for programmatically interacting with GUI applications, so I've implemented it as an alternative backend for my dish tool:

https://github.com/linuxrocks123/dish

See dish-atk-backend.c in that repo.

I wrote dish as a way to script interactions with GUI applications -- start this web browser, log into this website, download this HTML, etc. -- and that's currently what I use it for. However, I also wrote it as a backend for accessibility software that allows control of the computer through the spoken word, for people whose hands don't like using the keyboard or mouse, or who don't have hands. sonic.cpp in that repo implements a very crude prototype of such software, taking as its input the output of voice2json transcribe-stream.

Unfortunately, the reason AT-SPI2 is an alternative backend to dish, rather than the primary backend, is because not every application supports ATK or AT-SPI2. I would like it very much if every application did support one of those interfaces. I am sure blind people would like that even more. But, since some applications do not register, the primary backend for dish is instead an AI OCR nightmare called paddleocr.

My suggestion for the accessibility community is to make the AT-SPI2 layer as useful for GUI scripting and UI testing as possible. Explicitly support and advocate for that use case in addition to the primary use case of providing data for screen readers and similar tools. Shell scripting the GUI is cool, but the real reason to promote it is that the number of Linux users who would like to shell script the GUI is probably about two orders of magnitude larger than the number of Linux users who are blind.

Right now, if you break something that blind people need to use Linux, you'll get a few people complaining, and maybe it'll get fixed next year. But, if you break Firefox's CI, that thing you broke is going to get fixed within the hour. Make the AT-SPI2 layer the foundation of every major Linux GUI application's CI, as well as the foundation for lots of hackers who want to write GUI shell scripts, and you'll get a much larger outcry when some project's AT-SPI2 support breaks or degrades, and a lot more people asking for AT-SPI2 support as a new feature for software that currently doesn't have it.

Strength in numbers.