Juggling software interrupts and realtime tasks

Hardware interrupts (or just "interrupts") are initiated when a physical component in the system wants the kernel's attention; they will usually cause an immediate trap into a special handler function. Since interrupts take the system away from whatever else it was doing, interrupt handlers have to do their work quickly; there is not time for any sort of extended processing. This is not a new problem; pre-Linux Unix systems often included the concept of a "bottom half" as a way of deferring work that could not be done in an interrupt handler.

The Linux kernel, too, has had to develop mechanisms to defer processing until a more convenient time. One of those mechanisms is software interrupts (or "softirqs"). It was first introduced in the 0.99 kernel under the familiar "bottom half" name; the term "softirq" doesn't appear until the 1.1.77 development release. The abbreviation "bh" ("bottom half") can still be found in the names of kernel functions related to software interrupts.

In the current implementation, there is a handful of software-interrupt "vectors" assigned to specific subsystems. When one of those subsystems has to defer some work, it sets a bit in a per-CPU mask that will cause its software-interrupt handler to be called at a later time. Often, the software interrupt will run immediately after the completion of hardware-interrupt handling, meaning that it runs at a high priority, before just about any other work in the kernel. The software-interrupt mechanism is reserved for a handful of core-kernel subsystems, including networking and the block layer, with one big exception: the kernel's tasklet mechanism, which is available to any subsystem, executes tasklets in a software-interrupt handler.

Software interrupts can create latency in an otherwise well-functioning system, so they have long drawn the attention of developers interested in response times. In kernels configured for realtime preemption, software interrupts are handled in a process like any other, so their priority can be set relative to other important tasks. Most of us do not run realtime preemption kernels, though, so software-interrupt handling will usually steal time from other tasks that would like to be running. Even on normal systems, though, a system that is generating large numbers of software interrupts will eventually defer them to the per-CPU ksoftirqd kernel thread.

Normally, one expects realtime tasks to be the highest-priority work on the machine, even one that is not running a realtime kernel. But software-interrupt handling trumps even realtime tasks and can prevent them from meeting their deadlines. The latency that software interrupts can add has, evidently, been seen to cause audio glitches on Android systems, inspiring the work being proposed by Stultz now.

There are two components to the suggested change, the first of which is a scheduler tweak to modify its placement of realtime tasks. Currently, when a realtime task becomes runnable, the scheduler will search for a CPU that appears to have sufficient available capacity to run that task; among other things, that check will prevent the placement of a resource-hungry task on a slow CPU. Stultz's patch adds a new check to see whether a candidate CPU is handling (or is about to handle) a "slow" software interrupt that is expected to take a fair amount of processing time; for the purposes of this decision, "slow" is defined as being software interrupts from the networking or block subsystems. If the CPU is indeed busy in that way, the scheduler will try to place the realtime task elsewhere so that, with luck, it will gain access to the CPU more quickly.

The second piece changes how the kernel chooses to run software-interrupt handlers. Normally, those handlers are run as soon as possible on the current CPU unless the ksoftirqd thread is already running, in which case they will be queued for handling there instead. The patch series causes the kernel to check, before running a software-interrupt handler, whether there is currently a realtime task running on the current CPU; if so, and if the software interrupt is of the "slow" variety, it will be deferred to ksoftirqd so that the realtime task can continue running. Faster software interrupts (such as those for timers) will still execute immediately on the current CPU.

To effect that split, a subtle change was required. In current kernels, if ksoftirqd is running on a CPU, all software interrupts will be deferred to it. If these changes are active (there is a new kernel configuration option to control that), "fast" software interrupts will be handled immediately whether ksoftirqd is running or not. This change was necessary because the deferral of slower software interrupt handlers, as added by this patch series, may cause ksoftirqd to be running at times when the overall software-interrupt load is not high. With the old test, faster interrupt handlers could be delayed by an unnecessary deferral to ksoftirqd. The new code, though, could have the potential to not defer software interrupts at times when they are loading the system.

The end result of this work is much better latency results with realtime tests like cyclictest. The patches have been running on Android kernels "for a number of years" and presumably do not cause problems, at least in that context. Even so, Peter Zijlstra is not entirely happy with this work; it would be better, he suggested, to simply stop software-interrupt processing when there is a high-priority task to run. He has a series of patches that make that kind of change that he had posted previously, but admitted that the work "fell on its face due to regressions" at that time.

Stultz gave Zijlstra's series a try and concluded that, while it improves the situation, it does not fully address the problem. Specifically, it doesn't help if a single software-interrupt handler runs for a long time, and that is indeed something that happens. Thus, he said: "I'm not sure if it will let us move away from the softirq-rt placement optimization patches". Hopefully, some sort of solution that is acceptable to all of the developers involved will eventually be worked out. Otherwise, the outcome may end up being that Android continues to carry this patch series for several more years.