++Cc Jens on his kernel.org email
On Wed, 1 Jul 2026 at 12:12, Sebastian Andrzej Siewior bigeasy@linutronix.de wrote:
I have been reviewing how OP‑TEE is implemented and how secure‑world invocations behave. The goal was to determine whether an OP‑TEE call can delay the Linux side and introduce latency depending on the time spent in the secure world.
Similar latency effects are already known for EFI runtime services, but this was not documented. To mitigate the impact, EFI runtime invocations can be restricted to specific CPUs so that real‑time workloads on other CPUs remain unaffected. This mechanism, however, is only described in the commit that introduced it.
This change adds a firmware section that documents these behaviours explicitly. It highlights cases where firmware can delay the kernel, information that may be unfamiliar to some users and surprising-or concerning-to others.
Assisted-by: Microsoft-Copilot Signed-off-by: Sebastian Andrzej Siewior bigeasy@linutronix.de
I would appreciate an ACK from the OP-TEE camp that I got my little research right.
Documentation/core-api/real-time/hardware.rst | 92 +++++++++++++++++++ 1 file changed, 92 insertions(+)
diff --git a/Documentation/core-api/real-time/hardware.rst b/Documentation/core-api/real-time/hardware.rst index 19f9bb3786e03..fb113848c2f70 100644 --- a/Documentation/core-api/real-time/hardware.rst +++ b/Documentation/core-api/real-time/hardware.rst @@ -130,3 +130,95 @@ https://github.com/Linutronix/RTC-Testbench. The goal of this project is to validate real-time network communication. It can be thought of as a "cyclictest" for networking and also serves as a starting point for application development.
+Firmware +--------
+The firmware often plays a significant role in system operation because it can +perform tasks that the kernel cannot directly access, and in some cases it can +even preempt or intercept the kernel.
+A common example of firmware assisting the kernel is when it provides a generic +interface to a resource. Instead of accessing an RTC chip through an I2C host +controller, the kernel may query the firmware for the current time, and the +firmware then accesses the RTC behind the scenes.
+Firmware can also intercept kernel execution by providing services that +temporarily take control of the system. One example is memory scrubbing, where +the firmware periodically pauses the kernel, reads back portions of system +memory, and then returns control. During this time, the kernel is effectively +interrupted. +In contrast, some systems provide hardware‑based memory scrubbing, which +operates independently of firmware or software. See +Documentation/edac/scrub.rst for details.
+If the kernel is intercepted for longer periods then these periods can be made +visible with the hardware latency detector. See +Documentation/trace/hwlat_detector.rst.
+The kernel can also be intercepted in response to specific events, such as +overheating. In this case, the firmware may throttle the CPU or shut it down +immediately to prevent hardware damage.
+Unless the firmware is well documented, it should be thoroughly tested to +uncover any unexpected behaviour.
+EFI +~~~~
+EFI provides runtime services that act as a communication interface between the +firmware and the operating system. One such service is reading and writing EFI +variables, which are used, for example, to determine the boot source.
+Invoking a runtime service may require the architecture to disable kernel +preemption or interrupts during the call. This means the duration of a service +invocation directly affects the system’s observable latency. There is also +nothing that prevents a service call from disabling interrupts internally while +it runs.
+For these reasons, EFI runtime services are disabled by default on a PREEMPT_RT +kernel. They can still be enabled at boot time or via a Kconfig option if +required. +The native EFI runtime service implementation (where both the EFI service and +the kernel are either 32‑bit or 64‑bit executables) uses a wrapper mechanism +that invokes the service through a dedicated workqueue. This workqueue is named +efi_runtime, and it can be restricted to a housekeeping CPU using the +``/sys/devices/virtual/workqueue/efi_runtime/cpumask`` sysfs file. Assigning it +to a housekeeping CPU ensures that potentially long service invocations do not +impact the real‑time workload which is restricted to other CPUs.
+It must also be verified that the runtime services behave as expected. Some +implementations on the x86 architecture pause all other CPUs while one CPU +performs the service call. In such cases, the interruption affects all CPUs, +and restricting the workqueue to a single CPU provides no benefit.
+OP-TEE (ARM) +~~~~~~~~~~~~
+OP‑TEE uses a global serialization mechanism (the "big lock"), ensuring that on +each core only one OP‑TEE thread executes secure‑world code at a time.
+Execution flows from the normal world (Linux) into the secure world (OP‑TEE) +through the secure monitor at EL3. Linux and OP‑TEE cannot disable or mask each +other’s interrupts because both run at EL1 in different security states.
+Architecturally, the secure monitor can mask or reroute normal‑world interrupts +before entering the secure world. In a correct OP‑TEE/ TF‑A implementation, it +does not do this for the duration of secure calls. Normal‑world interrupts +remain deliverable, and a normal‑world IRQ will preempt OP‑TEE via EL3 and +return control to Linux.
+Secure‑world interrupts (FIQs) are possible if the SoC routes a device's +interrupt as secure. Such a secure FIQ will preempt Linux immediately, trap +into EL3, and transfer control to OP‑TEE's secure interrupt handler. Linux +cannot mask or preempt this. Secure FIQ handlers must therefore be extremely +short to avoid introducing noticeable latency.
+The transition from normal world -> secure monitor -> OP‑TEE and back introduces +additional latency due to world switching and context save/ restore. This +overhead is typically a few microseconds and usually remains in the noise +floor.
+If the secure monitor masks normal‑world interrupts during OP‑TEE invocations, +or if OP‑TEE uses long‑running secure FIQ handlers, then OP‑TEE can introduce
+measurable latency spikes.
2.53.0