Debugging tools for input devices
We're bad at marketingWe can admit it, marketing is not our strong suit. Our strength is writing the kind of articles that developers, administrators, and free-software supporters depend on to know what is going on in the Linux world. Please subscribe today to help us keep doing that, and so we don’t have to get good at marketing.
Most users don't know how to help diagnose a broken input device (e.g. mouse, touchpad, keyboard), Benjamin Tissoires said to start his X.Org Developers Conference talk (slides, YouTube video). Generally, users can't say much more than "my touchpad is broken", but he hopes to change that. There are tools that can be used to do better.
There are a number of stages in the input process. An input device communicates with the kernel over some transport. That information goes to libevdev, then to libinput, and on to the X.Org input driver. From there it goes to the X server or compositor and then to the toolkit (e.g. Qt or GTK+). Each of those layers might be broken, so there is a need to figure out which is the culprit when an input problem is discovered.
![[Benjamin Tissoires]](https://static.lwn.net/images/2015/xdc-tissoires-sm.jpg)
There are three main problem areas in input handling. The first is the user, but "we cannot fix it". The other two are the kernel and libinput (which was the subject of an earlier talk). In order to narrow the problem area down, the first step is to use evemu to record events and other information. It records the output of the kernel as seen by libinput and it allows others to replay the sequence of events to reproduce the problem. There is information on the evemu web page that will help users submit useful recordings, Tissoires said.
The evemu-record program has a straightforward interface that lists the input devices on the system and allows the user to choose which is of interest. It will then dump information about the kernel, the device and its capabilities, and the events generated by the device, in a form that can be used to replay them elsewhere. Based on the recording, if it looks like the kernel is doing the right thing, "talk to Peter [Hutterer]", otherwise the problem is probably something that Tissoires needs to look at.
There are other tools for diagnosing input problems, including mtdiag-qt, which he demonstrated. It intercepts events at the same level as evemu, but has a GUI to display the events graphically. He uses mtdiag-qt for kernel debugging, while Hutterer uses mtview for libinput. There are separate tools because developers of different pieces want to be able to examine different things.
If evemu, mtdiag-qt, mtview, or something else shows that the kernel is doing the right thing, libinput-debug-events can be used to show the events as they have been processed by libinput. It has an interface similar to evemu, with a list of devices to choose from to determine which to capture events for. It now ships with libinput. Once you have the output from libinput-debug-events, then it is time to talk to Hutterer (or to the wayland-devel mailing list, presumably).
On the other hand, if evemu and friends show that the problem lies in the kernel, the process gets more complex, because that means it could be any of the layers from the kernel down. There are multiple reasons that the kernel might "send garbage". It could be the device is broken in some fashion that needs to be worked around in the kernel. There is also room for interpretation in the protocol—Tissoires and Hutterer don't necessarily agree on some of the gray areas.
But driver bugs are usually the source of garbage from the kernel. For example, if the device maker changes the device and its protocol in some way, the driver will not reflect that change. Another recent example was a bug when placing three fingers on the touchpad one at a time, then lifting one (and leaving two). That caused a spurious event from the driver that was interpreted by libinput as if all the fingers had been removed, which led to several other bugs caused by that one underlying bug in the kernel driver.
In order to find where the problem exists, you "need to know what is going on down below", which includes both the physical layers (i.e. transports) and the protocols. For physical layers, there are USB, which is used widely, PS/2, which is mostly for keyboards, and I2C, which is quite new (for input devices) and mostly used on phones. Beyond that, there are SMBus, Bluetooth, serial, and so on. "Whatever you can think of, basically somebody made it", he said.
There are three classes of protocols for input devices. There are the public protocols, like the human interface device (HID) and psmouse protocols. There are the semi-public protocols, which are documented but only used by one manufacturer, such as RMI4 (Synaptics) or HID++ 2.0 (Logitech). Proprietary protocols, which have been reverse engineered or their documentation is only available under NDA, round out the types of protocols.
The physical layers and protocols can be combined in various creative ways by the device makers. For example, Synaptics has devices that use RMI4 over HID over I2C. Some versions of the Thinkpad trackstick use PS/2 over RMI4 over SMBus. Logitech also gets "quite creative" with devices that use HID++ in a multi-step combination that converts to and from HID twice. His final example was one that he would not explain: "WTF over GTFO over SNAFU".
In order to figure out which physical layer and protocol are being used by a device, kernel messages should be consulted:
$ dmesg | grep input
This will show the devices, how they are connected, and the protocol(s)
they are using.
If the device uses the HID protocol, the hid-replay tool can be used to show the raw events from the device before processing by the kernel. It works with any transport and can replay the events for debugging purposes. The hid-recorder tool is used to record the events. It has an interface much like the other tools, with a list of devices to choose from. If the kernel is doing the wrong thing with the raw events, it is time to blame him, Tissoires said.
For non-HID devices, the usbmon kernel facility can be used to capture the raw events from the USB transport. There is no provision for replaying events, however. For PS/2 devices, ps2emu (the subject of another XDC talk) can be used to record and play back raw events. Even though PS/2 is a rather old transport, 99% of laptops still use it internally, he said. Other transport layers will require specialized tools or hacks to the kernel to get raw event data.
One question that is often asked is whether there are regression tests for the input stack. The answer is "yes and no". Libinput does have regression tests. Testing X without libinput, though, relies on the X.Org integration test suite (XIT), which starts a new X server for each test, so it is not very efficient. That limits the amount of testing that is done. The kernel HID drivers have a basic wrapper for running regression tests, hid-test, but it is not run much any more, Tissoires said.
When reporting an input bug, there are a few guidelines that should be followed. First off, provide the full dmesg output. Second, provide a recording using evemu that shows the bug triggering (and, if possible, not much more). Lastly, "do not be afraid" if the input developers ask for additional information and recordings, as it is all part of the process for tracking down these kinds of bugs.
[I would like to thank the X.Org Foundation for travel assistance to
Toronto for XDC.]
| Index entries for this article | |
|---|---|
| Conference | X.Org Developers Conference/2015 |