I must admit that I love bugs. I also hate bugs which leads to a very
conflicted relationship with the blighters, but for now I just want to
focus on the positive - I love bugs.
The bugs I am talking about are, of course, the incorrect behavior of
software systems due to incorrect code and the reason that I love them
is that they are so educational. There are a number of reasons for
this, the significant one for the present being that they provide
motivation and focus to read and study new code.
One of the freedoms that free and open source software provides is the
freedom to study the source code. However having that freedom doesn't
mean it's easy to use it. When faced with a large body of code such as the
Linux kernel it can be hard to know where to start, or when to move
on. There is no story-line, no curriculum, no plot to guide your
study. This is where a bug comes in: it can provide a story line.
This may be a particular sequence of events or a particular
combination of features, or just a starting point to spiral out from.
But by providing a clear focus and a clear finishing point, it makes
study easier and more rewarding.
More than a phone
I have recently been spending some time trying to make the Linux
kernel run well on the
"Phoenux" replacement motherboard for the
Openmoko "Freerunner" and "Neo 1973" mobile phones. This is an
attempt to revitalize the Openmoko effort to provide an open (or at
least "as open as possible") platform for a mobile computing device.
It fits in the same case, uses the same display and speakers, but
provides more memory, faster processor, and faster networking plus a few
other extras. During this (ongoing) effort I have discovered a number
of bugs and missing features, most of which only came with smaller
learning experiences. One, though, stands out for leading me on a
somewhat longer path of discovery than most, and so I would like to
share that path and those discoveries so that others might benefit. I
should note first that the path is still open. I have not actually
fixed the bug, so others could follow the path themselves. Anyone keen
to do that should stop after the next paragraph, get a GTA04,
and start hunting themselves. Others can read on.
The symptom of the bug is easy to describe. Like most computer
systems the GTA04 has an RTC (real time clock) which has an alarm
function. When the time in the clock matches the time in the alarm,
it generates an interrupt. The problem is that, though this interrupt
works perfectly while the system is awake, it does not reliably wake the
device when it is asleep (i.e. in suspend-to-RAM state). Other
interrupts configured much the same way (for example data received on
the serial console) wake the system with 100% reliability. However
the RTC alarm almost always fails. Understanding this bug leads on a
path through the interrupt management code, the suspend management
code, and even the USB code. But it must start with a brief
description of the hardware.
In the GTA04 there are two particular integrated circuits (ICs) that
are relevant. One is the OMAP3 SoC (system-on-a-chip) from TI. The
other is the PMIC (power management IC) which is referred to as
"twl4030" in the various drivers - various because, like the OMAP3,
it is a multi-function chip combining diverse functions such as battery
charging, keyboard control, and the USB electrical interface. There are a
number of connections between these chips but only two are relevant
for now. These are a single interrupt line and an I2C bus.
The interrupt line signals the interrupt controller (INTC) in the
OMAP3 that something has happened. The I2C (inter-integrated-circuit)
bus is a simple 2-wire bus that allows the CPU to read and write
registers in the PMIC. In particular it can be used to find out which
of the several functions generated the interrupt, and why. It can
also be used to clear the source of the interrupt.
Having one interrupt line that indicates any of a number of possible
events like this is not uncommon. For example the UARTs (Universal
Asynchronous Receiver Transmitter - a serial port) in the OMAP3 each
have a single interrupt, but it can be generated by a character
arriving, a character transmit having completed, or an error
condition. As with most such devices there is a register that can be
inspected which has one bit for each possible interrupt source.
When all the interrupt sources are related to a single device and
handled by a single driver this is all quite easy to manage. However
when the different interrupt sources are related to logically separate
devices, having to manage all these behind a single interrupt could
So many interruptions
To address this situation, Linux has an abstraction called an
"irq_chip". An irq_chip represents a set of
interrupt sources each of which is assigned an interrupt number (as
listed in /proc/interrupts). It provides functions to enable or
disable each interrupt, to allow the interrupt to wake the device from
suspend, to set the trigger type (edge or level), and various other
functions. It also arranges that the interrupt handler for each
interrupt will be called when appropriate.
irq_chips with sources
Using this, the driver for the INTC in the OMAP3 defines an
irq_chip for all the interrupts that it knows about directly,
and the driver for the PMIC can define a separate irq_chip
which describes the various interrupt sources in the PMIC. Each
irq_chip will provide very different functions to configure
and control the interrupt sources, but will provide a uniform
interface to all device drivers. When the single PMIC interrupt
arrives at the INTC, it will call an interrupt handler which will
examine the PMIC registers and then call the appropriate handler
that was registered with the second irq_chip. This way the
drivers for the various parts of the PMIC don't need to know too much
about each other - the irq_chip mediates between them.
In our PMIC there is a "Primary Interrupt Status Register" which has
one bit for each module that can generate interrupts. Each of these
modules has their own "Secondary Interrupt Status Register" to report
what actually caused the interrupt, much like the UART described
earlier. There is normally no need for a tertiary irq_chip
to represent these registers as the one driver manages the whole
module and all of its interrupts.
There are two exceptions, only one of which interests us. There is a
"power interrupts" module in the PMIC which combines interrupts from a
diverse range of sources: press of the power button, insertion of USB
cable, over-temperature warning, and the RTC alarm. These are
sufficiently varied that a separate irq_chip makes sense here.
The twl4030-irq driver manages all this quite elegantly, providing a
secondary irq_chip for the device as a whole, and supporting
tertiary irq_chips, including for the Power Interrupts module.
So when the RTC alarm triggers, the primary interrupt is handled by
the INTC. This calls an interrupt handler which inspects the PMIC,
determines that a power interrupt is active and calls the relevant
handler. It in turn examines the PMIC again and finds that the RTC
alarm has fired, and so calls the handler that was registered for that
particular interrupt. This all works perfectly. But it doesn't seem
to wake the device from suspend. At least, not always. It seems that
something happens on the way to suspend that interferes with this.
A funny thing happened on the way to suspend
One of the many things that happens on the way to suspend is that
each individual interrupt gets disabled - unless it is flagged as
IRQF_NO_SUSPEND in which case it is left alone. However this doesn't
mean exactly what it sounds like it means. Being "disabled" just
means that the handler routine will not be run, it doesn't mean that
the interrupt will not be generated. We have a different word for
that, which is "masking". When an interrupt is masked the originating
source of the interrupt is told to never post that interrupt.
Linux uses a lazy scheme for disabling interrupts. When the disable
request is made, the fact is recorded in internal data structures, but
that is all. If the interrupt is subsequently delivered, only then
might the interrupt be masked. This can be a useful optimization as
masking an interrupt can take a lot longer than just setting a flag in
So, on the way to suspend, interrupts are disabled but not masked. If
the interrupt does actually arrive before we reach full suspend, the
fact is recorded. If it was an interrupt that should wake from
suspend, this is detected in
and suspend aborts.
If the interrupt doesn't arrive before full suspend, then it is still
unmasked and will successfully wake up the device, which will resume
and then handle the interrupt. This might all seem a bit complex, but
once it is fully understood it is actually quite neat and it works
well ... except for my RTC alarm.
Who turned out the lights?
As promised we will need to glimpse at the USB code as well and now is
the time to bring that in. Like most modern phones, the GTA04 can be
attached to a computer or a charger via a USB port. When the USB
interface detects 5 volts on the power line, it will route this to the
battery charger (BCI) which will use it to charge the battery and
power the device.
This is relevant because - when tracing is added to the twl4030-irq
driver - we see that during suspend we get an interrupt from the PMIC
which turns out to be due to the battery charger losing power and saying
"Hey, just thought you should know that I'm running on battery again
What is happening is that when the USB port is told to go to sleep it
turns off its various power supply regulators. One of these (USB3V1) also
powers the voltage comparator which determines if 5V is present.
As soon as that presence is no longer signaled, the current stops
being routed to the BCI, and the BCI justifiably complains.
This results in the interrupt line from the PMIC being asserted
which causes the interrupt management code to run. This notices that
the interrupt was disabled, and so masks it. When we get to
check_wakeup_irqs(), the PMIC IRQ is pending, but as it has not been
set for wakeup it is just ignored.
The net result here is that all interrupts from the PMIC are ignored
during suspend, so the RTC alarm doesn't work, the power button
doesn't wake the device up, and plugging or unplugging the USB cable
has no effect either.
So now that we understand the problem ...
So, what to do? We could just unplug the USB cable when testing the
RTC alarm. Then the BCI would not notice the current disappearing and
so wouldn't generate an interrupt. This certainly works - once. It
doesn't seem to work a second time, but we'll leave that issue for the
We could tell the USB controller not to turn off USB3V1 when the BCI
is active and power is available on the USB bus. This is certainly a good
idea and would mean that the battery can charge during suspend. It
also brings up the interesting issue of how these separate drivers
should communicate, but it doesn't really solve the general problem,
just this specific one.
It could be that an interrupt occurs that we really do want to ignore
during suspend, that we have even explicitly disabled (but due to
lazy-masking has not been masked). So we want to be able to mask
those interrupts without masking the main interrupt. A few moments
reflection on what we have learned so far suggests that setting
IRQF_NO_SUSPEND on the intermediate interrupts should help. That way
they will not be disabled during suspend so the interrupt management
will trace through the chain of irq_chips far enough to find
the originating interrupt source and will mask just that. The other
interrupt source will still be available.
This may sound convincing, but reality has a way of getting in the
way: testing shows that just setting IRQF_NO_SUSPEND is not enough. Diving
back into the code and the data sheets reveals an interesting and important
fact. The PMIC allows each individual interrupt source to be masked, but it
does not allow a whole module as a set of interrupt sources to be masked. So
when an interrupt arrives from the BCI (one of the modules) the IRQ manager
notices that interrupt is disabled and so the secondary irq_chip is
asked to mask it. But as the irq_chip cannot mask it (the hardware
doesn't support that), it simply ignores the request. You can imagine what
happens next - an unending stream of attempts to mask the BCI interrupt, none
of which are effective and the interrupt keeps re-firing.
The root problem seems to be that while the TWL4030 PMIC appears to
have an interrupt structure that lends itself to being represented as
a small tree of irq_chips, this isn't actually the case. The
details of the behavior of suspend makes it essential that a working
"mask" function be provided, and that cannot be provided for modules
in the twl4030, only for individual interrupts.
So it seems that a complete fix requires some change to the structure
of the irq_chips in the twl4030 driver. Possibly we could
make the structure simpler (no tree), or possibly we could make the
implementation more clever in some way. The important point is that
each interrupt that is visible to the Linux IRQ management code must
support being masked in the hardware in some way.
Another funny thing on the way to resume
As mentioned, after one successful wake-from-alarm, the RTC alarm in
the GTA04 doesn't work a second time. In fact it stops working
completely, even when not suspended. It seems that something has gone
wrong on the way to resume.
During resume all the interrupts that were disabled by suspend are re-enabled.
The three interrupts which relate to the RTC - the base PMIC interrupt, the
interrupt for the "power interrupts" module, and the actual RTC interrupt - are
enabled in that order with PMIC first. As this interrupt is still asserted (as
it is what triggered resume) the PMIC interrupt handler runs as soon as it
is enabled. It reads the interrupt status register to see which module
was responsible, and triggers the relevant interrupt for that module.
This secondary interrupt is still disabled so nothing happens. In
particular the interrupt source is not cleared, nor is it masked,
so we again get a repeated sequence of attempts to handle an interrupt
which cannot be handled. This one isn't unending as the loop that
enables all the interrupts is still running and it eventually gets to
the one for the target module (the power interrupts module). This is
triggered again and this time it actually does something. It reads
the final status register (thus clearing the interrupt) and calls the
interrupt handler for the RTC alarm. Unfortunately that is still
disabled so nothing else happens.
Eventually the RTC interrupt is enabled but by now it is too late.
In some cases an interrupt that fires while disabled gets the
IRQS_PENDING flag set so that when it is re-enabled, the interrupt is
resent. But that is not the case when the twl4030-irq interrupt
controller tries to handle a tertiary IRQ. Whether this is a bug or a
misunderstood feature is as yet unclear. Another journey of discovery
would be needed to fully understand that issue.
Meanwhile, because the interrupt is delivered but the interrupt service routine
isn't run, the handshaking between the driver and the device gets
confused and no other interrupt is ever seen from the RTC.
We can work around this problem by a simple expedient. When
re-enabling the interrupts during resume, do it in the reverse order.
This ensures that the RTC interrupt is enabled before it is called,
and everything works. This even feels like the "right" thing to do -
it is balanced for "enable" to happen in the reverse order of
"disable". But even if it is "right", it is not really a fix. There
would still be bugs lurking and ready to spring out again.
The point of this little tour is not to point out bugs in the code or
to shame the relevant developers. In fact the code is in general
quite elegant, well structured, largely very effective, and the
developers are only to be congratulated. Rather, the point is to
highlight the complexity of the task being tackled by developers
working the embedded space:
There are complex interactions between distinct hardware. Here the
USB interface and the battery charger have important interdependencies
that need to be reflected in the drivers. Some interdependencies already
are but there are subtleties that are easy to miss.
The requirements of an "irq_chip" are not really documented
anywhere, and given the current rate of development there is a good
chance that such documentation would be out of date. Extracting the
requirements from the code is far from trivial. In this case a seemingly
correct and obvious implementation was only found to be deeply wrong
by a fairly obscure test case.
The IRQF_NO_SUSPEND flag is clearly important, but not easy to
understand. It is fairly obvious what it does, but much less obvious
why you would want to do that. I imagine that as a developer my
approach would be to not set it until I found a bug which could be
fixed by setting it. Then I would set it and hope it didn't break
anything. This is not really a robust way to do development.
With complex hardware interactions and complex software dependencies
it is not surprising that we seem to see a new class of problems in
the embedded space. As we have observed, there are often ways to work
around the bug rather than fix it, and that is usually much simpler
and quicker. They may be tempting, but for this developer at least
they are nowhere near as satisfying.
Because if you follow the bug all the way to the root cause, and if
you understand all that is happening and find the right fix, then you
have not only helped other future developers, but have learned a whole
lot in the process. And that is why I love bugs.
to post comments)