Wander Lairson Costa

What every kernel programmer should know about Jump Labels

• kernel

This tutorial explains Linux kernel “jump labels” — also called “static keys” — from first principles, using the actual source in this tree (upstream v7.2) as the reference. x86_64 is the architecture covered throughout.

It is meant to be readable even if you have never touched kernel text patching before. The early sections lay down the CPU and instruction-encoding background that later sections build on — skim them if that part is already familiar.

Core files referenced:

File Role
include/linux/jump_label.h Public API, macros, struct static_key
kernel/jump_label.c Architecture-independent core logic
arch/x86/include/asm/jump_label.h x86 inline asm that emits the branch
arch/x86/kernel/jump_label.c x86 code-patching backend
arch/x86/include/asm/text-patching.h JMP/INT3/CALL/RET encodings, text_gen_insn()
arch/x86/include/asm/nops.h x86_nops[] table, BYTES_NOPx raw byte defs
arch/x86/kernel/alternative.c text_poke(), smp_text_poke_*() — the SMP-safe patcher
include/linux/jump_label_ratelimit.h struct static_key_deferred
include/linux/tracepoint.h A real, heavily-used consumer
Documentation/staging/static-keys.rst Upstream overview (somewhat outdated on x86 sizes)
tools/objtool/ Compile-time rewrite of jmp→nop under HAVE_JUMP_LABEL_HACK

Naming: “jump label” is the low-level mechanism — a patchable site in .text plus an entry in __jump_table. “Static key” is the higher-level API programmers use (DEFINE_STATIC_KEY_*, static_branch_*). People often use the two names interchangeably, but this tutorial keeps them distinct: “static key” for the API, “jump label” for the patching machinery underneath it.

1 Hardware background (why this is hard)

Jump labels work by rewriting machine code while the kernel is running - self-modifying code, executed by CPUs that were never designed to expect it. Making sense of that requires a bit of hardware background first: how a CPU actually treats instructions and memory loads, the exact byte-level encodings involved, and why a live, multi-core kernel can’t just overwrite them with an ordinary write.

1.1 What a CPU actually does with instructions

A modern x86_64 core does not “read one instruction, execute it, repeat”. Roughly:

  1. Fetch bytes from the instruction cache (I-cache / L1i).
  2. Decode those bytes into microcodes (variable-length on x86 — an instruction can be 1–15 bytes).
  3. Execute out of order, with a branch predictor guessing which way conditional branches go so the pipeline stays full.
  4. Commit results in order.

Two consequences follow from that pipeline, and together they are the entire performance case for jump labels. The first is about what a naive if (feature_enabled) check actually costs. Out-of-order execution and branch prediction make the branch itself close to free: predict correctly often enough and there is no pipeline flush to pay for. But prediction only hides the cost of guessing which way a branch goes - it does nothing about the feature_enabled load that feeds the guess. That load still has to happen on every single hit of the path, whether or not the predictor gets the branch right, and it still claims a real data-cache access and an execution port each time. Under cache pressure, or if some other CPU ever writes that flag and bounces its cache line out from under you, the “cheap” branch stops being cheap at all.

The second consequence is the whole point of jump labels: an unconditional nop or an unconditional jmp sidesteps that entire problem, because there is no flag to load and no condition to evaluate. The “off” path can be literally empty work for the frontend - decode a nop, move on - with no cache line to bounce and nothing for the branch predictor to even weigh in on.

1.2 x86 instruction encoding: JMP and NOP

§1.1’s cost difference between a load and a nop/jmp has to actually be encoded in real bytes for a patch to swap between them, and matching sizes will matter as soon as §1.3 gets to atomicity - so here is the exact shape of both. x86 is a variable-length ISA; the encodings jump labels care about:

Instruction Opcode bytes Total size Reach
INT3 (breakpoint) CC 1 byte n/a
JMP rel8 (short) EB xx 2 bytes -128..+127 bytes from the end of the insn
JMP rel32 (near) E9 xx xx xx xx 5 bytes ±2 GiB
2-byte NOP 66 90 2 bytes —
5-byte NOP 0f 1f 44 00 00 (nopl 0x0(%rax,%rax,1)) 5 bytes —

Constants live in arch/x86/include/asm/text-patching.h (JMP8_INSN_*, JMP32_INSN_*, INT3_INSN_*) and arch/x86/include/asm/nops.h (BYTES_NOP5, etc.). The relative displacement is measured from the byte after the instruction:

disp = dest - (addr + insn_size)

That is exactly what text_gen_insn() / __text_gen_insn() compute.

Why matching size matters: if you replace a 5-byte nop with a 5-byte jmp, surrounding addresses do not move. Return addresses on stacks, other jump targets, exception tables, ORC unwind info — none of them need updating. Patching is an in-place byte swap of equal length.

1.3 Why you cannot just memcpy over live code on SMP

Patching kernel text is harder than patching an ordinary data structure because three things are true of it at once:

  1. It is mapped read-only after boot (CONFIG_STRICT_KERNEL_RWX), so a normal store to it would simply fault — whatever mechanism does the patching has to get around that on purpose, not by accident.
  2. It is being fetched by other CPUs concurrently — nothing pauses the rest of the machine while one CPU edits a function that every core can call at any moment.
  3. It may already be sitting half-decoded in the pipeline of another CPU, having been fetched moments ago but not yet executed.

Point 2 is the dangerous one, and it is worth walking through concretely. Picture two CPUs, A and B, where A is patching a 5-byte instruction that B keeps calling in a loop. The store A makes is not one atomic operation — the CPU issues it as however many bus-width writes it takes to cover 5 bytes, and each of those writes becomes visible to the rest of the system separately. The fetch B makes can land in the middle of that sequence, seeing some bytes from before the write and some from after:

                     time --->

 CPU A (patcher)   [ write bytes 2-4 ]   [ write bytes 0-1 ]
                                      ^
                                      |
 CPU B (fetcher)             [ fetch all 5 bytes, right here ]
                                      |
                                      v
                 byte-by-byte: 0=old  1=old  2=new  3=new  4=new
                 = torn mix: 2 old bytes + 3 new bytes
                 = neither the old instruction
                   nor the new one — garbage

If CPU A writes five bytes while CPU B is mid-fetch of that same instruction, B can observe this torn mix of old and new bytes — not a valid instruction, and not something the decoder in B can safely execute. x86 does not guarantee that a multi-byte store to a concurrently executing instruction is atomic from the point of view of instruction fetch.

A single-byte store, by contrast, is atomic for instruction fetch — no CPU can ever see it half-written, because there is no “half” of one byte. Both the Intel SDM and the approach the kernel takes build on exactly that fact, turning one unsafe multi-byte write into three safe single-step moves:

That protocol lives in smp_text_poke_batch_finish() (§10), and jump labels are only one of several clients that share it — ftrace, static calls, kprobes, and the alternatives-patching machinery all reuse this same three-step dance.

1.4 Writing read-only kernel text: text_poke()

Modern kernels no longer patch text by clearing the WP bit in %cr0, writing, and setting it back. That old trick worked, but it was a blunt instrument: between the clear and the restore, every write from every CPU could land on write-protected memory, not just the one instruction being patched. Anything else that happened to run during that window could corrupt memory it was never supposed to touch. __text_poke() replaces it with a narrower idea. Instead of unlocking the existing mapping of .text, it builds a second, private virtual mapping of the exact same physical page and writes through that instead.

The intuition is worth stating before the mechanism. Physical RAM does not know or care how it is mapped; the same page of memory can be reached through more than one virtual address at once, each with its own permissions. .text normally has exactly one mapping, visible to every CPU, always read-only and executable. That single mapping is what lets any core fetch and run it at any moment.

__text_poke() temporarily adds a second mapping to that same physical page: writable, not executable, and visible only to the CPU doing the patching. That mapping is torn down within a handful of instructions. It is a second door into the same room. The contents of the room — the instruction bytes — are the same no matter which door you walk through, but only one of the two doors is ever locked.

Reaching that second mapping takes several steps, and each one closes off a different way this could otherwise go wrong. All five below are pieces of one function, __text_poke(), working with these locals:

static void *__text_poke(text_poke_f func, void *addr, const void *src, size_t len)
{
        bool cross_page_boundary = offset_in_page(addr) + len > PAGE_SIZE;
        struct page *pages[2] = {NULL};
        struct mm_struct *prev_mm;
        unsigned long flags;
        pte_t pte, *ptep;
        spinlock_t *ptl;
        pgprot_t pgprot;
        ...

func is the actual copy routine: memcpy-like for a real text_poke() call, memset-like for the _set() variant. addr/src/len are simply the target and payload the caller passed in.

First, the function has to identify the physical page or pages backing the address being patched — usually one page, two if the write straddles a page boundary. It gets there via virt_to_page() for core kernel text, or vmalloc_to_page() for text living in a module:

if (!core_kernel_text((unsigned long)addr)) {
        pages[0] = vmalloc_to_page(addr);
        if (cross_page_boundary)
                pages[1] = vmalloc_to_page(addr + PAGE_SIZE);
} else {
        pages[0] = virt_to_page(addr);
        if (cross_page_boundary)
                pages[1] = virt_to_page(addr + PAGE_SIZE);
}

Second, it points a pre-allocated page-table entry at that physical page, inside a dedicated, otherwise-empty address space called text_poke_mm (allocated once, at boot, by poking_init()). That entry is marked writable, and deliberately not global: it carries no _PAGE_GLOBAL bit.

That one detail is what keeps the whole scheme cheap. A non-global mapping is only ever cached in the TLB of the current CPU, so tearing it down later is a plain, local flush_tlb_mm_range() — no IPI to other CPUs, because no other CPU ever loaded text_poke_mm in the first place:

pgprot = __pgprot(pgprot_val(PAGE_KERNEL) & ~_PAGE_GLOBAL);
ptep = get_locked_pte(text_poke_mm, text_poke_mm_addr, &ptl);

local_irq_save(flags);

pte = mk_pte(pages[0], pgprot);
set_pte_at(text_poke_mm, text_poke_mm_addr, ptep, pte);
if (cross_page_boundary) {
        pte = mk_pte(pages[1], pgprot);
        set_pte_at(text_poke_mm, text_poke_mm_addr + PAGE_SIZE, ptep + 1, pte);
}

Third, the current CPU actually switches onto that private address space via use_temporary_mm(text_poke_mm), saving whatever mm it had loaded so it can restore it afterward. Writing %cr3 changes what every virtual address on this CPU means. But the pipeline is deep and speculative: instructions ahead of that write may already have been fetched, decoded, or executed under the old mapping. Left alone, execution could keep running past the switch on stale translations, resolving a load or fetch as if the old address space were still current.

The x86 architecture closes that off by defining writes to control registers (%cr0/%cr3/%cr4/%cr8) as serializing: the CPU must retire everything prior, discard any speculative work in flight, and drop non-global TLB entries before it begins executing under the new value. That guarantee holds on every implementation — it is part of the ISA, not a performance accident. Loading %cr3 here gets it for free: the CPU is certain to see the page-table entry from the previous step before it can fetch anything through it, with no separate synchronization needed:

prev_mm = use_temporary_mm(text_poke_mm);

The switch also has to deal with a second, unrelated hazard: hardware watchpoints. The debug registers that hold watchpoint addresses (%dr0–%dr3) are global CPU state, not scoped to whichever address space happens to be loaded, so a watchpoint stays armed straight through a page-table switch regardless of serialization.

text_poke_mm_addr is deliberately placed in the low, user-range half of the address space (the hardware sidebar below explains why), which happens to be exactly where the watchpoints of a debugger live. If a userspace watchpoint aliased onto that address while this CPU was mid-write through it, the CPU would fire a debug exception in the middle of the very code-patching machinery the kernel itself relies on. That would misdeliver a signal that has nothing to do with whatever the debugger was actually watching for — or worse, interrupt the sensitive write itself.

So use_temporary_mm() explicitly calls hw_breakpoint_disable() right after the switch, and its counterpart, hw_breakpoint_restore(), restores them afterward. Breakpoints are disabled wholesale rather than only for the specific colliding address, so this even suppresses unrelated kernel breakpoints (e.g. ones set by perf) for that brief window. That’s accepted as a reasonable trade-off, since the window is so short.

Fourth, with the writable alias finally in place, the actual copy happens by calling func, at the address text_poke_mm_addr + offset_in_page(addr):

func((u8 *)text_poke_mm_addr + offset_in_page(addr), src, len);

For a real text_poke() call, func is text_poke_memcpy() — the small wrapper the caller handed in as the func argument (the _set() variant passes the memset-flavored text_poke_memset() instead):

static void text_poke_memcpy(void *dst, const void *src, size_t len)
{
        lass_stac();
        __inline_memcpy(dst, src, len);
        lass_clac();
}

That sequence raises two questions: why the write needs STAC/CLAC around it at all, and why the copy inside them has to be inline rather than a real call to memcpy(). The hardware sidebar below answers both together.

Fifth, the temporary mapping is dismantled in the reverse order it was built: pte_clear() removes the page-table entry, unuse_temporary_mm() switches %cr3 back to the saved mm (serializing again, for the same reason as the switch in step three), and flush_tlb_mm_range() drops the now-stale local TLB entry.

Finally, for a real text_poke() call (though not for the _set/_copy variants, which skip this) the function reads back what it just wrote and memcmps it against what was intended. Any mismatch is a BUG(): a mechanism whose entire premise is “the bytes we write are exactly the bytes we meant to write” cannot be allowed to fail silently — a loud crash is the only honest response:

pte_clear(text_poke_mm, text_poke_mm_addr, ptep);
if (cross_page_boundary)
        pte_clear(text_poke_mm, text_poke_mm_addr + PAGE_SIZE, ptep + 1);

unuse_temporary_mm(prev_mm);
flush_tlb_mm_range(text_poke_mm, text_poke_mm_addr, text_poke_mm_addr +
                   (cross_page_boundary ? 2 : 1) * PAGE_SIZE, PAGE_SHIFT, false);

if (func == text_poke_memcpy)
        BUG_ON(memcmp(addr, src, len));

local_irq_restore(flags);

Put together, this is one physical page reached through two different virtual addresses with two different permissions:

                          physical page (the actual RAM
                          holding the instruction bytes)
                                    ^        ^
                                    |        |
              normal kernel         |        |   text_poke_mm_addr
              mapping (all CPUs,    |        |   (this CPU only,
              always present)       |        |    exists briefly)
                     |               \      /            |
                     v                \    /             v
          .text  [ RO, executable ]    \  /    [ RW, not executable ]
          (every CPU's %cr3 maps        \/     (only this CPU's %cr3
           this address, forever)                maps this address,
                                                  only while patching)

Every CPU, all the time, can execute the left-hand mapping — that one never changes permission or address. Only the current CPU, and only during __text_poke(), can additionally reach the exact same bytes through the right-hand mapping, and only to write them. Once the teardown step above clears that PTE and flushes the local TLB, the right-hand mapping is gone again; the left-hand one is all that is left, now showing the new bytes.

So the permanent kernel mapping of .text stays read-only everywhere, all the time; only a throwaway, single-CPU-visible alias is ever writable, and it exists for only a handful of instructions. All of this is serialized by text_mutex, so two concurrent patchers never race to build competing temporary mappings.

Hardware sidebar — why STAC/CLAC, and why the copy must be inline. Some CPU features exist specifically to catch the kernel touching low, user-range addresses by accident. The classic bug — and attack surface — is a corrupted or attacker-influenced pointer that the kernel ends up dereferencing as if it pointed to trusted kernel memory. STAC/CLAC are how the kernel tells the CPU, in effect, “the access I’m about to make into that range is deliberate, stand down for a moment.” Two independent features watch for exactly this, and they don’t watch for the same thing:

  • SMAP (“Supervisor Mode Access Prevention”) faults if kernel code accesses a page whose page-table entry has _PAGE_USER set — that is, a page actually mapped user-accessible. It looks only at that one permission bit, never at the numeric address.
  • LASS (“Linear Address Space Separation”), newer than SMAP, faults on any kernel access to an address below the canonical-address midpoint. That means anywhere numerically in the user range, regardless of whether _PAGE_USER is set on that particular page.

text_poke_mm_addr falls right in the gap between those two rules. To be clear, it is not actually a userspace mapping: poking_init() sets it to TASK_UNMAPPED_BASE, plus a KASLR-style random offset — the same range where calls to mmap() made by an ordinary process would land. That’s simply because text_poke_mm is built with mm_alloc(), the ordinary allocator for a process address space, and a freshly allocated mm just happens to have empty space to carve one throwaway page out of down there. The page-table entry actually built at that address is an ordinary kernel-only mapping, with _PAGE_USER left clear, so nothing about it is reachable from user mode.

That distinction is exactly what splits the two checks apart. SMAP only ever looks at the _PAGE_USER bit, and that bit is clear here, so SMAP has nothing to object to. LASS blocks by address alone, regardless of the bit — and this address, purely by where it numerically sits, is exactly what LASS would fault on.

That is why the actual code calls lass_stac()/lass_clac() rather than plain STAC/CLAC. They emit the same underlying instructions, just gated on X86_FEATURE_LASS instead of X86_FEATURE_SMAP — so the call compiles down to a no-op on CPUs without LASS, and to a real access-check override on CPUs that have it. Either way it works, whether the CPU has SMAP, LASS, both, or neither. Conceptually this is the same AC-bit mechanism the kernel already uses whenever it deliberately touches real userspace memory (copy_from_user() and friends); text poking just happens to need it too, for a kernel-internal mapping that only looks like a userspace address.

Opening that window has one more consequence. objtool (§6.3 covers it in depth) enforces a build-time rule that no call instruction may appear between a STAC and the next CLAC. The reason is concrete: AC is ordinary CPU state, but unlike registers, it is not saved and restored across a context switch. A call is a black box — it might transitively reach schedule() and put the task to sleep, and if that happens while AC is set, the override can leak into whichever task runs next, or fail to be restored when this one resumes.

That rule is why the copy can never be a real call to memcpy(). On x86_64, memcpy() is hand-written assembly living in a separate object, reachable only through a genuine call instruction. __inline_memcpy()/__inline_memset() sidestep that problem: forced inline, they compile to the same rep movsb/rep stosb sequence the real functions would use, but leave no call for objtool to flag, because there is no separate function left to call.

2 The problem jump labels solve

Kernel code is full of rarely-taken checks that guard optional functionality: “is tracing enabled for this tracepoint?”, “is this security module active?”, “is this debug feature on?”. A naive implementation:

if (some_feature_enabled)
        do_something();

Even when some_feature_enabled is almost always false, the CPU still must:

  1. Load some_feature_enabled from memory (a cache line).
  2. Compare it against zero.
  3. Predict/branch on the result.

As §1.1 worked out in detail, modern CPUs predict step 3 well, but “well” is not “free”: prediction hides the misprediction penalty, not the guaranteed memory load in step 1. When the check sits in a hot path that runs millions of times a second (scheduler, networking, every trace_*() site), that unavoidable load adds up.

Jump labels remove the load and the compare for the common case by rewriting the machine code at runtime. When the feature is off, the hot path literally has no branch to the rare code — it is a nop (or an unconditional jmp over an out-of-line block, depending on polarity). When someone turns the feature on, the kernel walks every call site for that key and overwrites nop↔jmp in place.

Tradeoff in one sentence: toggling is expensive (machine-wide sync, text poke); running the hot path is nearly free.

3 The mental model, in one diagram

§2 described the tradeoff in words; here is the same idea as the literal code the compiler produces for one if, side by side in both of its two possible forms:

   SOURCE CODE                     FEATURE OFF (common)         FEATURE ON
   ------------                    --------------------         ----------
   if (static_branch_unlikely      nop  (2 or 5 bytes)          jmp .Lout_of_line
       (&my_key)) {                ...normal path...            ...normal path...
           rare_code();            .Lout_of_line:               .Lout_of_line:
   }                                   rare_code();                 rare_code();
                                       jmp back                     jmp back
                                   (unreachable without a jmp)

Both columns are compiled from the exact same source line — nothing about the C code changes. What changes is which of the two already-compiled forms happens to be sitting in memory at any given moment, decided entirely by whether the key is currently enabled. “FEATURE OFF” has no branch at all: rare_code() still exists in the binary, but with no jmp pointing at it, normal execution can never reach it — that is what “(unreachable without a jmp)” is calling out. The nop is exactly as wide as the jmp it might become (2 or 5 bytes on x86_64 — see §6.3), so turning one column into the other is an atomic in-place replacement of equal length, not a resize.

That jmp back at the end of the out-of-line block is not part of the jump-label machinery at all. rare_code() here is a real C label (§6 covers how asm goto reaches it), and in the C source, execution simply falls through from that label into whatever statement follows the if. Since the compiler physically moved the labeled block elsewhere in the function, it has to end that block with an ordinary unconditional jmp back to wherever it placed the following statement — the same technique a compiler uses to lay out any unlikely branch, nothing specific to jump labels. That returning jump is fixed at compile time and never patched by anything in this tutorial; only the nop/jmp at the top of the site ever changes.

Key state Instruction in the hot path Cost when not taking the rare path Cost when taking it
disabled (for an unlikely site) nop ~0 (no load, trivial decode) N/A — nothing patched in points at rare_code(), so this column cannot happen
enabled jmp <out-of-line> one unconditional jump jump + rare code + jump back

Compare this to the naive version from §2, which always pays the load + compare price.

4 How to use static keys (the cookbook)

One fact underlies almost every rule in this section: toggling a key is not a cheap flag flip. It is real cross-CPU code patching — the IPI-synchronized protocol walked through in §10.3 — so every API choice below is really a choice about how often that machinery gets invoked and who is allowed to trigger it.

4.1 Minimal example

Before this section works through individual API choices - polarity, refcounting, RO-after-init, rate limiting - it helps to see the entire lifecycle of one key end to end: how it is declared, how the hot path guards on it, and how something elsewhere flips it. This is a synthetic example, not lifted from a real kernel file the way §4.10 later is, but every line in it corresponds to a real mechanism this tutorial covers elsewhere:

#include <linux/jump_label.h>

DEFINE_STATIC_KEY_FALSE(foo_key);

void hot_path(void)
{
        /* Fast path: compiled as NOP while the key is false. */
        if (static_branch_unlikely(&foo_key))
                do_rare_thing();

        do_common_work();
}

void foo_enable(void)
{
        static_branch_enable(&foo_key);   /* slow path: patches text */
}

void foo_disable(void)
{
        static_branch_disable(&foo_key);  /* slow path: patches text */
}

Every piece of that example maps onto a term from earlier or later sections. foo_key compiles down to the struct static_key_false from §7 - a plain atomic_t wrapped in a type the compiler can tell apart from static_key_true at compile time.

The if in hot_path() is the patchable site itself: static_branch_unlikely() picks the nop-by-default helper here because a _FALSE key paired with unlikely is exactly the case where the hint agrees with the initial value of the key, which §5 and §6 work out in full - that agreement is why the comment can already say “compiled as NOP” before the key is ever toggled.

foo_enable()/foo_disable() are not part of the jump-label API itself; wrapping static_branch_enable()/static_branch_disable() in a subsystem-named function like this is a real pattern, not an invented one - __sched_core_enable() in the scheduler does exactly this around its own DEFINE_STATIC_KEY_FALSE. Underneath that one-line call is the entire boot-vs-runtime patching machinery covered in §9 and §10.

The two comments - “Fast path” and “slow path” - are the entire tutorial in miniature: everything before this example explains why the fast path can be free, and everything after it explains what the slow path actually has to do to make that true.

4.2 Choosing TRUE vs FALSE and likely vs unlikely

Getting this pair wrong is not a correctness bug — the code still runs — it is a silent performance foot-gun. Pick the polarity that disagrees with your steady state, and you get a real jmp sitting on the hot path where a nop belonged. That mistake is invisible in any correctness test, and it isn’t something you can fix at runtime: which of arch_static_branch or arch_static_branch_jump gets compiled in (§5) is baked in at build time, not decided later by toggling the key. So it’s worth asking two questions up front, rather than guessing:

  1. What is the default at boot? Most optional features start off → DEFINE_STATIC_KEY_FALSE. Features that are on unless turned off → DEFINE_STATIC_KEY_TRUE.
  2. Which way does this if lean in steady state? Use static_branch_unlikely() when the body is the rare path; static_branch_likely() when the body is the common path.

You can mix them. A _FALSE key works with both likely and unlikely; a _TRUE key does too. The kernel picks a compiled-in nop or jmp so the default case is the cheap one (§5).

Rule of thumb for most new code:

DEFINE_STATIC_KEY_FALSE(feature_key);

if (static_branch_unlikely(&feature_key))
        rare_enabled_path();

4.3 Boolean enable vs refcounted enable

This distinction exists because of a failure mode that shows up the moment a key has more than one owner. Say two independent subsystems both call static_branch_enable() on the same key, each wanting the feature on for its own reasons. Whichever one finishes first and calls static_branch_disable() turns the feature off for both — out from under the other subsystem, which has no way to know its dependency just vanished. Refcounting exists precisely to make “enabled” mean “at least one owner still wants this,” instead of “the opinion of the last caller wins”:

API Semantics When to use
static_branch_enable / disable Force enabled count to 1 or 0 Single owner; simple on/off
static_branch_inc / dec Refcount; patch only on 0↔1 Multiple independent users
static_branch_slow_dec_deferred Dec, but delay the 1→0 patch Userspace-driven toggles

inc/dec treat the key as “enabled iff count ≠ 0”. The first inc (0→1) patches code on; the last dec (1→0) patches off. Intermediate increments are cheap atomics with no text poke.

Do not mix enable/disable with inc/dec on the same key. Both APIs act on the same underlying key->enabled counter, but each assumes a different range of values is valid: the boolean API assumes it only ever sees 0 or 1, while inc/dec is happy to let it climb to any count of current owners.

If some other inc() caller has already pushed the count to 2 or higher, static_key_enable() and static_key_disable() won’t corrupt that count — but they won’t do what the caller expects, either. enable() treats “already above zero” as “already on” and returns immediately; disable() finds the count isn’t the 1 it expects for a clean shutdown and also returns without patching anything off. Both paths hit a WARN_ON_ONCE and silently no-op instead, leaving the feature exactly as it was — with only a kernel warning to show that anything went wrong.

4.4 Reading the state without taking the branch

Every example so far, including the hot_path() from §3.1, uses static_branch_likely/unlikely as the condition of an if - the whole point of those macros is to become the patched branch. Sometimes what a caller actually wants is the current boolean value of the key as an ordinary expression, not a branch to take. static_key_enabled() is the separate, deliberately unoptimized API for exactly that:

if (static_key_enabled(&foo_key))
        /* plain atomic read of the count — NOT the patched fast path */

Two real patterns from the tree show why this exists as a separate API, rather than just “the slow way to write an if”.

The first is reporting state, not branching on it. arch/x86/kernel/cpu/bugs.c logs which Spectre/IBPB mitigation got selected with pr_info(..., static_key_enabled(&switch_mm_always_ibpb) ? "always-on" : "conditional") — there is no hot-path branch here at all, just a boolean being formatted into a string once, at boot.

The second is a control-plane guard. drivers/md/dm-stats.c does if (!static_key_enabled(&stats_enabled.key)) static_branch_enable(&stats_enabled); before calling the actual (expensive, IPI-synchronized, §10.3) enable path, specifically to avoid re-triggering a full patch round when the feature is already on.

Both cases want the current boolean value as an ordinary expression — to print, compose, or make a one-off decision with — which is something static_branch_likely/unlikely aren’t really built for: their asm-goto trick expects to be the condition of an if, not to hand back a bool you can store or reuse.

On an actual hot path, though, still prefer the branch macros so you get the patched instruction. Using static_key_enabled() there instead means paying, on every single call, exactly the cache-line load this whole tutorial opened by trying to eliminate (§1.1).

4.5 Keys must be global / static storage

A static key cannot live on the stack or be kmalloc’d. The compiler embeds its address into __jump_table as a link-time constant (a relative offset, §7.3) — fixed once, at link time, for the life of the kernel image.

A stack-allocated key would work right up until its function returned: the “distance to my key” offset of the patchable site would then point at whatever now occupies that stack slot. A kmalloc’d key has the same problem the moment it is freed. Either way, the corruption is silent until something happens to patch or read that site again. Typical patterns:

DEFINE_STATIC_KEY_FALSE(global_key);           /* .data */
static DEFINE_STATIC_KEY_FALSE(file_local);    /* file scope */

/* header */
DECLARE_STATIC_KEY_FALSE(global_key);

Arrays:

DEFINE_STATIC_KEY_ARRAY_FALSE(keys, 4);
if (static_branch_unlikely(&keys[i]))
        ...

Conditional on Kconfig:

DEFINE_STATIC_KEY_MAYBE(CONFIG_FOO, foo_key);
/* expands to TRUE if CONFIG_FOO=y, else FALSE */

if (static_branch_maybe(CONFIG_FOO, &foo_key))
        ...

4.6 Read-only-after-init keys

§3.5 covered keys that live for the life of the running kernel and can be toggled at any point in it. Some keys never need that: a mitigation decided once at boot and never revisited benefits from a stronger guarantee than “nothing happens to toggle it” - a guarantee that nothing can, even a bug. DEFINE_STATIC_KEY_FALSE_RO (and its _TRUE_RO counterpart) provide exactly that:

DEFINE_STATIC_KEY_FALSE_RO(configured_once_at_boot);

Placed in __ro_after_init. You may still static_branch_enable/disable during __init (before mark_rodata_ro()). After that:

Use _RO for “decide once at boot, then freeze” features (many security / mitigation toggles). If you need to flip the key at runtime for the life of the system, use plain DEFINE_STATIC_KEY_*, not _RO.

That freeze is hardware-enforced. After mark_rodata_ro() runs, an attacker who has already won an arbitrary-write primitive elsewhere still cannot flip the key of a hardened mitigation, because enabled sits in genuinely read-only memory — the write faults at the hardware level, the same way any other write to .rodata would.

4.7 Rate-limited disable (userspace-facing knobs)

If userspace can flip a feature rapidly, naively patching on every toggle thrashes text and tanks performance — recall from §10.3 that a single toggle is a full three-phase IPI round to every online CPU, not a local write. A sysctl or socket option a user flips and unflips in a loop would otherwise turn into a machine-wide synchronization storm, one round per flip.

The API below is deliberately asymmetric, and that asymmetry is the whole design. static_branch_deferred_inc() is nothing more than the ordinary, immediate static_branch_inc() from §4.3: turning a feature on is never delayed. If this key is guarding something like counting or tracing, a delayed enable would mean silently missing whatever happened during the delay. Turning it off is the side that can safely wait, since a feature staying active a little longer than strictly necessary is harmless.

static_branch_slow_dec_deferred() reflects that asymmetry with two distinct branches, and knowing which one runs is the key to the whole mechanism:

So for the entire timeout window, nothing about the feature has changed: the count is still 1, still fully enabled, still fully patched. That is what makes the coalescing work. If a fresh inc() arrives during that window, the count moves from 1 to 2 with an ordinary atomic increment (the fast path from §4.3) — before the timer has fired. When the timer does eventually fire, it performs one ordinary decrement exactly as if the earlier dec() call had never been the final one. By now, it genuinely isn’t: the count drops from 2 to 1, not from 1 to 0, so the “is this the transition to zero” check inside the real dec path (the same cmpxchg pattern from §4.3/§9.2) never trips, and no patch happens. A rapid enable → disable → enable sequence that lands entirely inside one timeout window therefore costs zero IPI rounds, not one per toggle.

#include <linux/jump_label_ratelimit.h>

DEFINE_STATIC_KEY_DEFERRED_FALSE(sockopt_key, HZ);

/* enable immediately */
static_branch_deferred_inc(&sockopt_key);

/* disable — may wait up to `timeout` before actually patching off */
static_branch_slow_dec_deferred(&sockopt_key);

/* force pending delayed work to finish (e.g. module exit) */
static_key_deferred_flush(&sockopt_key);

That last call is not just tidiness. static_key_deferred_flush() blocks until any pending deferred disable has actually run. The enclosing struct, struct static_key_false_deferred, bundles the key itself, the timeout, and the delayed_work together. It usually lives in memory that is about to go away, e.g. a module being unloaded. Free that memory with the timer still armed, and when it eventually fires, jump_label_update_timeout() will run container_of() on a delayed_work that no longer exists — a use-after-free, not merely a stale toggle. Concretely:

struct static_key_false_deferred {
        struct static_key_false key;
        unsigned long timeout;
        struct delayed_work work;
};

4.8 CPU hotplug / deadlock rule

static_branch_enable/disable/inc/dec take cpus_read_lock() (and jump_label_mutex) so that no CPU can come online mid-patch (§9.2). That lock is not reentrant: a CPU hotplug notifier already runs with the write side of that same lock held, so calling one of these functions from inside a notifier would have the kernel try to take a lock it already holds — a straightforward self-deadlock, not a rare race.

If you are already inside a hotplug callback that holds the hotplug lock, use the *_cpuslocked variants, which skip re-acquiring it:

static_branch_enable_cpuslocked(&key);
static_branch_disable_cpuslocked(&key);
static_branch_inc_cpuslocked(&key);
static_branch_dec_cpuslocked(&key);

These are not general-purpose — only for that context.

4.9 What not to do

4.10 A real consumer: tracepoints

The example in §3.1 was synthetic, built to show the API shape end to end. Tracepoints are the real thing: a single DEFINE_TRACE() can back dozens or hundreds of trace_foo() call sites scattered across vmlinux and modules alike - the exact scale §9.6 worked through when discussing IPI-round math - which makes the polarity choice from §3.2 matter in practice, not just in theory:

/* include/linux/tracepoint.h — __DECLARE_TRACE */
static inline void trace_##name(proto)
{
        if (static_branch_unlikely(&__tracepoint_##name.key))
                __do_trace_##name(args);
        ...
}

Each DEFINE_TRACE() embeds a struct static_key_false in struct tracepoint, initially false. Combined with the initial false value of the key, _unlikely makes every dormant trace_foo() site a nop in the hot path. Registering a probe enables the key and live-patches every site (vmlinux + modules) to jmp.

5 Two polarities: key default × branch hint

At the moment the compiler emits a patchable site, jump_label_init() has not run yet — there is no live enabled count to read, only whatever initial value the key was declared with. So the compiler has to pick some concrete instruction to put there, and it does so by pretending the future has already happened. It computes the exact same “does the branch hint agree with the value of the key?” comparison that the runtime patcher (§9) will keep recomputing every time the key actually toggles — just using the declared initial value in place of the live one. That is why this section and the runtime patching logic in §9 end up sharing one formula instead of two: the compile-time default isn’t a special case, it is simply what the general rule produces on day one, before anything has ever been enabled or disabled. Two independent bits feed that formula:

  1. Key initial value — DEFINE_STATIC_KEY_TRUE vs _FALSE.
  2. Call-site hint — static_branch_likely() vs _unlikely().

Get the two out of sync — the mistake §4.2 warns about — and the default compiled-in instruction is the expensive jmp, not the free nop, for as long as the key stays at its initial value. The kernel arranges that the default/common case uses nop (fall through), and the uncommon case uses an out-of-line jmp. From the comment at include/linux/jump_label.h:

              likely()                   unlikely()
            ------------                ------------
 key=true   ...                         ...
            NOP                         JMP L
            <br-stmts>               1: ...
        L:  ...
                                     L: <br-stmts>
                                        jmp 1b
 ------------------------------------------------------------
 key=false  ...                         ...
            JMP L                       NOP
            <br-stmts>               1: ...
        L:  ...
                                     L: <br-stmts>
                                        jmp 1b
static key / branch macro static_branch_likely() static_branch_unlikely()
DEFINE_STATIC_KEY_TRUE nop → fall into body jmp out to body
DEFINE_STATIC_KEY_FALSE jmp out to body nop → skip body

Memorize: when the hint matches the initial truth of the key, you get nop; when they disagree, you get jmp. That “agree vs. disagree” phrasing is exactly what XOR computes on two booleans — 0 when its inputs match, 1 when they don’t. That’s why the eight-row table below collapses into one line of arithmetic, twice over: once for the live, run-time-toggling case, and once for the frozen, compile-time snapshot described above:

   enabled  type  branch    instruction
  --------------------------------------
     0       0      0     |   NOP
     0       0      1     |   JMP
     0       1      0     |   NOP
     0       1      1     |   JMP
     1       0      0     |   JMP
     1       0      1     |   NOP
     1       1      0     |   JMP
     1       1      1     |   NOP

    dynamic (at runtime): instruction = enabled ^ branch
    static  (at compile): instruction = type    ^ branch

(type = compile-time initial value; enabled = live state; branch = 1 for likely, 0 for unlikely.) Implemented as jump_label_type() / jump_label_init_type() in kernel/jump_label.c. Here is the dynamic half of the formula, verbatim — jump_label_init_type() has the same shape, just reading the compile-time type bit of key instead of enabled:

static enum jump_label_type jump_label_type(struct jump_entry *entry)
{
        struct static_key *key = jump_entry_key(entry);
        bool enabled = static_key_enabled(key);
        bool branch = jump_entry_is_branch(entry);

        return enabled ^ branch;
}

5.1 How the macros pick the asm

static_branch_likely() and static_branch_unlikely() — the two public macros from §4 that every call site actually uses — never touch a patchable site directly. Underneath them sit two lower-level, arch-specific helper functions:

Both helpers report their result the same unusual way: not by loading the state of the key from memory and computing a boolean, but by reporting which of the two hard-wired paths execution actually took through the patched instruction itself. Concretely: if the byte or bytes currently sitting at the patchable site are a nop, the CPU just falls through to the next instruction, and the call reports false; if they are a jmp, the CPU diverts to the out-of-line block containing the body of the branch, and the call reports true. (§6.1 shows the actual mechanism behind this — an asm goto that lands on a label called l_yes when the jump is taken.)

So the return value tells you which physical instruction is at the site right now, for either helper. It does not yet tell you whether that corresponds to “the if-body runs” for the macro that called it — that translation is what the ! fix-up in the next paragraph is for.

There are four (key type, branch hint) combinations — TRUE/FALSE key crossed with likely/unlikely — but, as the list above shows, only two distinct patchable-site shapes to choose from. So the entire job of static_branch_likely()/ _unlikely() is collapsing those four cases onto the two helpers without losing any information. That collapse happens in two independent steps. Which helper to call is decided by the TRUE/FALSE type of the key, matching each site to the shape it actually needs. Whether to negate the result is decided by likely vs. unlikely, for a reason the paragraph right after the code below walks through in detail:

/* CONFIG_JUMP_LABEL path in jump_label.h */
static_branch_likely(x):
  TRUE  key → !arch_static_branch(&(x)->key, true)       /* nop-default site */
  FALSE key → !arch_static_branch_jump(&(x)->key, true)  /* jmp-default site */

static_branch_unlikely(x):
  TRUE  key →  arch_static_branch_jump(&(x)->key, false) /* jmp-default site */
  FALSE key →  arch_static_branch(&(x)->key, false)      /* nop-default site */

That table is exactly the XOR formula from §5 written out one case at a time: arch_static_branch (nop-default) is the one actually called whenever type ^ branch == 0 — the branch hint agrees with the compiled-in value of the key, so the cheap, fall-through case is the one that needs no jump at all. arch_static_branch_jump (jmp-default) is called whenever type ^ branch == 1 — the hint disagrees with the compiled-in value of the key, so reaching the if-body requires an actual jump even before the key is ever toggled.

The lone ! in front of the two likely calls is not a mistake; it is a polarity fix-up. Its job is to make the meaning of the returned boolean — “does the if-body run?” — consistent regardless of which of the two helpers above happened to be chosen, since the two helpers report their own fall-through-vs-jump outcome, not “is the key enabled.”

Walking through one case makes this clear. For a TRUE key read with likely, the code table above calls the nop-default helper, arch_static_branch — and here, the compiled-in default really is a nop. Per the calling convention above (nop → falls through → false), arch_static_branch returns false in this common case, even though the if-body does run. The false is only reporting “no jump was taken,” not “the condition is false.” Negating that mismatch, !false == true, turns “no jump was taken” back into the answer the macro actually needs: “yes, the if-body runs” — correct, even though zero jumps were taken to get there.

The true/false argument passed alongside &(x)->key does more than feed that fix-up math, though. The call site doesn’t just use the branch hint once, at compile time, to help pick nop vs. jmp — it also writes that same hint permanently into the small piece of metadata the compiler emits for this patchable site (called a jump table entry; §6.2 shows what it looks like in the raw asm, and §7.2 covers its layout in detail). That matters because the type ^ branch formula from §5 is not a one-time, compile-time calculation: the runtime patcher recomputes it every time the live state of the key changes (§9), and it needs to read branch back from somewhere at that point — this stored hint is where it comes from.

__builtin_types_compatible_p distinguishes struct static_key_true vs struct static_key_false at compile time; anything else calls ____wrong_branch_error() (unresolved symbol → link error).

6 What the compiler emits (x86_64)

6.1 The two asm helpers

§5.1 treated arch_static_branch()/arch_static_branch_jump() as black boxes that return false or true; this is what is actually inside them, from arch/x86/include/asm/jump_label.h:

static __always_inline bool arch_static_branch(struct static_key * const key,
                                               const bool branch)
{
        asm goto(ARCH_STATIC_BRANCH_ASM("%c0 + %c1", "%l[l_yes]")
                : :  "i" (key), "i" (branch) : : l_yes);

        return false;
l_yes:
        return true;
}

static __always_inline bool arch_static_branch_jump(struct static_key * const key,
                                                    const bool branch)
{
        asm goto("1:"
                "jmp %l[l_yes]\n\t"
                JUMP_TABLE_ENTRY("%c0 + %c1", "%l[l_yes]")
                : :  "i" (key), "i" (branch) : : l_yes);

        return false;
l_yes:
        return true;
}

Notice these are two separate C functions, rather than one shared function that takes an extra argument telling it whether to build a nop-shaped site or a jmp-shaped one — something like an imagined arch_static_branch(key, branch, use_jmp). That design is not just unused, it is impossible here: the assembly text inside each function is fixed, literal text, embedded straight into the compiled function body at build time. A runtime argument (a value only known while the kernel is running) cannot make a single function body sometimes contain one instruction and sometimes another — the bytes are baked in for good the moment this file is compiled.

So instead, the two possible shapes get two separate functions, each with its own hard-coded assembly. The choice between them is made where the macros in §5.1 decide which one to call — a decision the compiler makes once, from the TRUE/FALSE type of the key, not something either function decides for itself while running.

The body of both functions is almost entirely raw GNU assembler (GAS) text, handed to the compiler through the asm goto extension GCC provides, instead of being written as ordinary C. If that syntax isn’t already familiar, here is every piece of notation used below and in §6.2, explained once:

With that vocabulary available, arch_static_branch_jump() is the more direct of the two functions to read, since its body is written out plainly rather than hidden behind another macro. Ignoring the surrounding C-string quoting, the assembly it hands to asm goto is:

1:
jmp %l[l_yes]
<jump table entry for this site, via JUMP_TABLE_ENTRY>

arch_static_branch() is built the same way, but its 1:-and-instruction line is not written out directly inside the function — it is delegated to the ARCH_STATIC_BRANCH_ASM macro:

#ifdef CONFIG_HAVE_JUMP_LABEL_HACK
#define ARCH_STATIC_BRANCH_ASM(key, label)      \
    "1: jmp " label " # `objtool` NOPs this \n\t"   \
    JUMP_TABLE_ENTRY(key " + 2", label)
#else /* !CONFIG_HAVE_JUMP_LABEL_HACK */
#define ARCH_STATIC_BRANCH_ASM(key, label)      \
    "1: .byte " __stringify(BYTES_NOP5) "\n\t"  \
    JUMP_TABLE_ENTRY(key, label)
#endif /* CONFIG_HAVE_JUMP_LABEL_HACK */

That #ifdef/#else is resolved once, by the plain C preprocessor, at build time — every kernel build takes exactly one of these two branches, never both, depending on HAVE_JUMP_LABEL_HACK (§6.3 explains what decides which). The two branches emit genuinely different bytes at 1::

The last line shared by both asm goto statements — : : "i" (key), "i" (branch) : : l_yes); — is what makes %c0, %c1, and %l[l_yes] meaningful in the first place. Reading it through the (empty) output section, the input section lists two operands: "i" (key) becomes operand %0, and "i" (branch) becomes operand %1. "i" is a constraint: it tells the compiler “this operand must end up as an immediate, compile-time-constant value” — as opposed to, say, "r" (put it in a register) or "m" (leave it addressable in memory).

That constraint is not a style choice: the jump-table entry from §6.2 needs the actual numeric values of key and branch baked directly into .long/.quad data at assembly time, and an immediate is the only kind of operand still guaranteed to have a fixed, known value after this __always_inline function has been inlined away and any register it might otherwise have used no longer exists. The empty section right after the inputs is the clobber list (nothing here touches any register or memory the compiler isn’t already tracking), and the final l_yes is the goto-label list mentioned above — the one part of this whole construct that has no equivalent in ordinary asm(...).

6.2 The jump table entry (sidecar metadata)

#define JUMP_TABLE_ENTRY(key, label)                   \
        ".pushsection __jump_table,  \"aw\" \n\t"      \
        _ASM_ALIGN "\n\t"                              \
        ANNOTATE_DATA_SPECIAL "\n"                     \
        ".long 1b - . \n\t"                            \
        ".long " label " - . \n\t"                     \
        _ASM_PTR " " key " - . \n\t"                   \
        ".popsection \n\t"

This emits no executable code. It appends one struct jump_entry into the __jump_table ELF section:

Field Asm Meaning
code .long 1b - . relative offset to the patchable insn
target .long label - . relative offset to the l_yes target
key _ASM_PTR key - . relative offset to the static_key, low bits = flags

Every line in the macro is an assembler directive, in the sense §6.1 introduced — none of it is a CPU instruction, and none of it ever executes. Reading it top to bottom with the vocabulary of that section in hand:

Stepping back from the line-by-line reading: code/target use .long (a fixed 32 bits) while key uses _ASM_PTR (native pointer width — 64 bits on x86_64). That is not an inconsistency: the patchable instruction and its own l_yes label are always emitted right next to each other, in the same function, so a 32-bit displacement can’t help but reach. The static_key itself carries no such guarantee. As the comment on struct jump_entry itself puts it, the key “may be far away from the core kernel under KASLR” — or it may simply live in a module loaded who-knows-where relative to this call site. That one field needs the full address range a 32-bit offset couldn’t promise.

Each of those three fields is a self-relative offset. The natural first guess for what that means is “distance from the start of the struct jump_entry” — i.e., you’d find the instruction by taking the starting address of the entry itself and adding code to it. That is not what actually happens. Instead, each field stores its distance to whatever it points at, measured from the address of that one field itself — not the address of the struct, not the address of the array, but the address of that specific 4-or-8-byte field. It sounds like a small difference, but it is the reason all three fields, despite sitting at different offsets inside the struct, can be read back by the exact same recipe. Working through the first field concretely:

   __jump_table[i].code is a field, and like any variable, it lives
   somewhere in memory. Call that address F.

   The patchable nop/jmp instruction this entry describes lives at
   some other address, C, out in .text.

   What .pushsection/.long actually wrote into the .code field, back
   at assembly time (§5.2), is the *distance* between those two
   addresses:

       value stored in entry->code  =  C - F

   To go the other way — given only the entry, find C — you need F
   again. But F is just "the address of this field", which C code can
   always get with the & operator:

       C  =  &entry->code  +  entry->code
              ^^^^^^^^^^^^    ^^^^^^^^^^^^
              this field's     the distance
              own address      stored in it

   That expression is, verbatim, the body of jump_entry_code():
   "my own address, plus whatever I'm holding."

.target (distance to l_yes) and .key (distance to the static_key, with 2 low bits reserved as flags — see §7.2) are computed the same way, each from its own field address rather than that of the struct: jump_entry_target() is &entry->target + entry->target, and jump_entry_key() is the same shape with the flag bits masked off first. Because each field carries its own address into the calculation, none of these three functions needs to know or care where inside the struct its field happens to sit. “Struct-start-relative” offsets would have needed a different fixed adjustment hard-coded per field to account for that. “Field-address-relative” offsets don’t, by construction.

This is also precisely what makes the table survive KASLR (§7.2) with zero patching at boot. Say the whole kernel image ends up loaded 0x1000 bytes higher than the linker originally assumed. Every address in it — including both F and C above — shifts by that same +0x1000, because the entire image moves as one block. The difference C - F, which is the only thing ever actually stored, does not change at all. There is nothing here for jump_label_init() to go fix up on this front. The encoding is already correct no matter where the kernel ends up in memory.

6.3 HAVE_JUMP_LABEL_HACK: why sites are 2 or 5 bytes

§6.1 showed the nop-default site of arch_static_branch written as a hand-coded .byte BYTES_NOP5 — a fixed 5-byte NOP — but that is not the size that actually lands in a compiled kernel. Depending on the call site, the NOP that ends up in memory can be either a compact 2-byte instruction or the full 5-byte one, and which of the two you get is settled long before the kernel ever boots, by a build-time trick this section walks through in full: get the compiler to emit a real jmp (so it can pick whichever encoding is actually shortest for that call site), then have a separate build step convert that jmp into a NOP of the same size, after compilation but before the kernel image is final.

That trick is gated by a single config option, HAVE_JUMP_LABEL_HACK. arch/x86/Kconfig turns it on for any build with objtool available (select HAVE_JUMP_LABEL_HACK if HAVE_OBJTOOL) — which, in practice, means every modern x86_64 build. Recall ARCH_STATIC_BRANCH_ASM from §6.1, which branches on this exact config option:

#ifdef CONFIG_HAVE_JUMP_LABEL_HACK
#define ARCH_STATIC_BRANCH_ASM(key, label)             \
        "1: jmp " label " # `objtool` NOPs this \n\t"    \
        JUMP_TABLE_ENTRY(key " + 2", label)
#else
#define ARCH_STATIC_BRANCH_ASM(key, label)             \
        "1: .byte " __stringify(BYTES_NOP5) "\n\t"     \
        JUMP_TABLE_ENTRY(key, label)
#endif

The hack exists to solve a problem the compiler cannot: a hand-written .byte BYTES_NOP5 always produces a 5-byte NOP, even for sites that will only ever need the shorter 2-byte form, wasting I-cache space on every nop-default site in the kernel. The fix is to have the compiler emit a real, correctly-sized jmp — the compiler already knows how to pick the shortest encoding that reaches its target — and then convert that jmp into a NOP of the same size after the fact, once, at build time. Concretely, with the hack enabled (the normal case on x86_64):

  1. The compiler emits a real jmp to l_yes, exactly as it would for an ordinary conditional branch. x86 has two different encodings for an unconditional jump, and the assembler is free to pick whichever one actually fits. JMP rel8 is a 1-byte opcode (0xEB) followed by a single signed byte saying “how many bytes forward or backward from the next instruction”: 2 bytes total, usable only if the target is between 128 bytes behind and 127 bytes ahead of that next instruction (the range of a signed byte, -128 to +127). JMP rel32 is a different 1-byte opcode (0xE9) followed by a 4-byte signed displacement using the same next-instruction-relative scheme: 5 bytes total, but able to reach anywhere in a 64-bit kernel image.1 (“rel8”/“rel32” here just names the size, in bits, of that displacement number — the same self-relative idea §6.1 and §6.2 used for jump-table fields, just encoded directly in an instruction instead of stored as separate metadata.) The assembler already knows the real distance to l_yes when it assembles this function, so it picks the shorter rel8 form whenever that distance allows it, and only falls back to rel32 when the target is too far. The size, in other words, is already whatever is optimal for that specific call site.
  2. The jump-table key expression becomes "%c0 + %c1 + 2", which simply sets bit 1 of the stored key value. That bit is not consumed by anything at runtime; it exists purely as a signal for the next step.
  3. objtool (tools/objtool/check.c:handle_jump_alt), which runs once over the compiled object files as part of the build, sees key_addend & 2 set, and rewrites that jmp in place into a same-sized NOP, clearing the relocation that would otherwise have pointed at it. This happens entirely at build time — by the time the kernel boots, the bytes are already NOPs, and objtool itself is long gone. Bit 1 having done its job as an objtool-only signal, jump_label_init() later repurposes that same bit position for something unrelated — the “__init text” flag, via jump_entry_set_init() — since nothing at runtime ever needs the original meaning again.

By the time the kernel image is built — objtool has already run, long before the kernel ever boots — every nop-default site (the ones written with arch_static_branch) contains a real NOP instruction, and it is already the smallest one that fits: 2 bytes where the jmp it replaced used the rel8 encoding, 5 bytes where it used rel32. Nothing shrinks or grows it later; boot time just inherits whatever objtool left behind.

The catch is that struct jump_entry (§6.2) never records which of the two sizes a given site ended up with. It could not: the size is only decided when objtool runs, which is after the on-disk layout of the struct has already been fixed by the compiler.

So later, whenever the kernel actually needs to patch a site — flipping a NOP to a jmp or back, in response to static_branch_enable()/static_branch_disable() — it cannot just trust a stored number. It has to look at the real bytes sitting in memory and work out for itself whether they encode a 2-byte or a 5-byte instruction, then build a same-sized replacement so the layout of the surrounding code does not shift. That decode-then-patch machinery (arch_jump_entry_size(), insn_decode_kernel(), __jump_label_patch()) is exactly what §8 walks through in full; the point to take away here is only that whatever size objtool committed to at build time is faithfully rediscovered and reproduced at every later patch, so a site never ends up needing more room than the code around it left for it.

Builds without the hack (CONFIG_HAVE_JUMP_LABEL_HACK unset — older toolchains without objtool support) never go through any of this. They use the other ARCH_STATIC_BRANCH_ASM branch shown at the top of this section: a hand-written .byte BYTES_NOP5 that always emits a fixed 5-byte NOP, no matter how close the target label actually is, and patching such a site later always installs a 5-byte JMP32 to match. The result is still correct — the branch still works exactly the same way — it just never gets the chance to use the shorter 2-byte encoding, so every nop-default site costs 3 extra bytes of I-cache footprint compared to a hack-enabled build.

That whole objtool rewrite (the numbered steps above) only applies to arch_static_branch, the nop-default helper from §6.1. arch_static_branch_jump, the jmp-default helper, builds its asm goto directly (also shown in §6.1) instead of going through ARCH_STATIC_BRANCH_ASM, and its JUMP_TABLE_ENTRY key expression is the plain "%c0 + %c1" — no + 2 added.

With no such tag on the entry, objtool has nothing telling it to touch that jmp, so it is left alone and reaches boot as a real jmp, exactly as a jmp-default site is supposed to (it stays a jump until something calls static_branch_disable() on it). That jmp is still free to come out as either the 2-byte rel8 form or the 5-byte rel32 form, by the same distance-based assembler choice described in step 1 above — the hack changes whether objtool converts the instruction afterward, not which of the two jmp encodings the assembler reaches for in the first place.

Bit 1 of that key field — the same low-order bits jump_entry_key() masks off before turning it into a struct static_key pointer (§6.2) — does double duty. During the build it is a private signal for objtool to convert a jmp into a NOP (step 2 above). From boot onward, that same bit is repurposed as what jump_entry_is_init() reads back as the __init-text flag (§9.1). Conflating the two is an easy way to misread this code cold:

  build time  --------------------------------->  boot time  ---> forever after

  bit 1 = "objtool: NOP this jmp"        jump_label_init() unconditionally
  (only meaningful to objtool;           OVERWRITES it to mean:
   consumed and discarded before                bit 1 = "jump_entry_is_init"
   the kernel ever boots)                (site is in __init text, unpatchable
                                          once init memory is freed — §8.1)

That boot-time flag is set by jump_entry_set_init() inside jump_label_init() (§9.1), and read back by jump_entry_is_init() — the same bit, doing unrelated jobs on either side of the build/boot line shown above.

6.4 Assembly-level picture

§6.1 gave the two C helpers, §6.2 gave the jump-table entry they emit, and §6.3 explained why the bytes at the patchable site itself aren’t fixed. Put together, for a nop-default site on a HAVE_JUMP_LABEL_HACK build (the normal case on x86_64), this is everything that lands in the object file:

.text:
        1:  0f 1f 44 00 00  ; 5-byte NOP if l_yes was far, OR 66 90 (2-byte) if
            ...             ; close (objtool rewrote a real `jmp` into this,
                            ; at build time — §5.3)

__jump_table:                   ; non-executable metadata, in __jump_table (§5.2)
        .long   1b - .          ; code:   self-relative offset to the NOP above
        .long   L - .           ; target: self-relative offset to l_yes
        .quad   key+branch+2 - .; key: static_key address, branch bit set (§5.2),
                                ;      plus the objtool-only "+2" signal (§5.3)

The two possible byte sequences shown for the 1: label in .text are genuinely different instructions, not interchangeable padding: 0f 1f 44 00 00 is the same 5-byte NOP named in §6.1 (nopl 0x0(%rax,%rax,1) — a multi-byte no-op instruction, dressed up with an unused addressing mode purely to reach 5 bytes). 66 90 is the 2-byte alternative. 0x66 is the operand-size override prefix of x86, normally used to shrink the operand of a following instruction from 32 to 16 bits, stacked in front of 0x90, the classic single-byte NOP (historically the opcode for XCHG AX, AX). Combining them doesn’t change what the CPU actually does; the prefix is there purely to pad the encoding out to exactly 2 bytes. Either sequence is a genuine no-op either way: the CPU decodes it, spends a cycle or so, and falls straight through to whatever comes next, leaving every register and flag untouched.

Two things in that second block are easy to misread if you haven’t just finished §6.2–§6.3, so it’s worth naming them explicitly:

One more x86-specific config bit is worth flagging before moving past “what the compiler emits” to “what the kernel does with it”: HAVE_JUMP_LABEL_BATCH is also defined on x86, which is what makes the batched, IPI-amortized patch path in §9–§10 available at all — without it, every jump-table entry would have to be patched (and synchronized across every CPU) one at a time.

7 Core data structures

7.1 struct static_key

Every key you DEFINE_STATIC_KEY_{TRUE,FALSE} in C boils down, at runtime, to this same underlying struct — regardless of which macro you used (the TRUE/FALSE wrapper types that keep them distinct at compile time show up later in this section):

struct static_key {
        atomic_t enabled;
#ifdef CONFIG_JUMP_LABEL
        union {
                unsigned long type;
                struct jump_entry *entries;
                struct static_key_mod *next;
        };
#endif
};

enabled is the live state §4 and §5 keep referring to: an atomic refcount, 0 meaning off and any positive value meaning on (this is what lets static_branch_inc()/_dec() in §4.3 stack multiple owners on one key). It can also transiently hold -1, which means “the first static_key_slow_inc() on this key is in progress right now, patching the instruction stream” (§9.2 covers that window in detail). While that is happening, any other reader calling static_key_count() still needs to see “enabled” rather than a confusing negative number, so that function maps -1 back to 1 before returning it.

The second field is where this struct gets unusual: it is one word that means two entirely different things, chosen by a tag bit hidden inside it. That is only possible because pointers to struct jump_entry and struct static_key_mod are both at least 4-byte aligned in practice, which means their real value always has its low 2 bits set to 0 — those 2 bits are free for the taking, so the union borrows them to store extra information instead of leaving them as always-zero padding:

Bit Macro Meaning
0 JUMP_TYPE_TRUE compile-time initial value of the key was true — this is the same type bit the type ^ branch formula from §5 uses
1 JUMP_TYPE_LINKED 1: the rest of the word is a next pointer (a linked list); 0: it is an entries pointer (a flat array)

(Do not confuse this bit-packed word with the low 2 bits of jump_entry::key itself from §6.2/§7.2 — same trick, applied twice, to two unrelated pointers in two unrelated structs. One tags the type of this key itself and which pointer kind it holds; the other tags the branch hint of a call site and its init-section status.)

Bit 1 exists because a single, contiguous array is not always enough to describe every call site for a key. For a key only ever used inside vmlinux itself, the linker sees every call site at link time and can sort them all into one contiguous run inside __jump_table (§7.3) — entries just points at the start of that run. But a key can also be used from inside a module, loaded long after boot, with its own private __jump_table section that was never linked against the main kernel image at all (§7.4). There is no way to splice the entries of a module into the already-built vmlinux array after the fact, and modules can be loaded and unloaded repeatedly over the lifetime of the kernel, so the set of “all call sites for this key” can grow and shrink at runtime. When that happens, the union switches meaning: next becomes the head of a linked list of

struct static_key_mod {
        struct static_key_mod *next;
        struct jump_entry *entries;
        struct module *mod;
};

nodes — one node per module currently contributing call sites for this key — rather than a direct pointer into one flat array. JUMP_TYPE_LINKED records, for a given key, which of these two representations is currently in effect.

None of the code outside kernel/jump_label.c has to know any of this: accessors like static_key_entries(), static_key_type(), static_key_linked(), and static_key_set_entries() mask these two bits off before handing the pointer to anyone else, so the rest of the kernel just sees “the entries for this key,” never the tag bits.

Finally, the two type wrappers from §5.1’s __builtin_types_compatible_p check are trivial by design — each is nothing but a struct static_key in a differently-named box:

struct static_key_true  { struct static_key key; };
struct static_key_false { struct static_key key; };

Their entire purpose is to exist as two distinct C types the compiler can tell apart at compile time, even though they carry identical data — exactly what §5.1’s static_branch_likely()/_unlikely() macros rely on to pick the right arch helper.

7.2 struct jump_entry (relative form)

This is the struct §6.2 has already been building up piece by piece — the one JUMP_TABLE_ENTRY writes one instance of, per call site, into __jump_table. x86 opts into a specific variant of it by selecting CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE in its Kconfig, which is what makes the struct look like this:

struct jump_entry {
        s32 code;
        s32 target;
        long key;       /* full width: module↔vmlinux may be far under KASLR */
};

code and target hold the self-relative distances §6.2 walked through in detail (&entry->code + entry->code recovers the real address of the patchable instruction, and likewise for target/l_yes); key, with its low 2 bits masked off, recovers the address of the owning static_key the same way. Architectures that do not select HAVE_ARCH_JUMP_LABEL_RELATIVE use a plainer struct instead, where all three fields simply are absolute addresses — which is visible directly in the accessors of that fallback itself (in jump_entry_code() there is just return entry->code;, no arithmetic at all). x86 pays the small extra cost of the “address of self plus stored offset” computation on every read in exchange for two things:

  1. Size. A kernel can easily contain tens of thousands of jump-table entries — one per call site. s32 (4 bytes) for code/target instead of a full pointer-width field (8 bytes on x86_64) roughly halves the size of two-thirds of every entry, multiplied across the whole table.
  2. KASLR immunity, which §6.2 already derived in full: since code and target are always within a couple of bytes of the single function they belong to, a 32-bit signed distance can always reach; recovering the address costs one addition instead of a boot-time fixup. key is kept at full pointer width (long, not s32) because it points at a static_key, which can live anywhere — including in a different module from the one containing this call site, or in vmlinux while the call site itself is in a module loaded who-knows-where in the address space (§7.1). The distance between two independently-placed pieces of memory like that can exceed what a signed 32-bit number can express, so this one field cannot be shrunk the way code/target were.

The low 2 bits stolen from key — the same trick §7.1 used on the pointer field of static_key itself, applied here to a different pointer for a different purpose — carry two more pieces of information:

Bit Meaning
0 jump_entry_is_branch: the branch hint from §5/§5.1 and the "%c0 + %c1" from §6.2 — 1 if the call site used likely(), 0 for unlikely()
1 jump_entry_is_init: 1 if the code for this site lives in __init text — freed after boot, and therefore unpatchable from that point on

Both accessors are exactly as small as a single bit-check suggests:

static inline bool jump_entry_is_branch(const struct jump_entry *entry)
{
        return (unsigned long)entry->key & 1UL;
}

static inline bool jump_entry_is_init(const struct jump_entry *entry)
{
        return (unsigned long)entry->key & 2UL;
}

Bit 1 is the same bit §6.3 followed through its “two lifetimes” story: at build time it briefly means “objtool, turn this jmp into a nop,” and only after jump_label_init() has consumed that meaning and moved on does it settle into this second, permanent meaning for the rest of the uptime of the kernel.

7.3 Relationship

§7.1 and §7.2 described the two structs in isolation; this is how they fit together in memory, for a static_key holding a plain entries pointer (the common, non-module-linked case from §7.1):

        struct static_key
        +------------------------+
        | enabled (atomic)       |
        | type / entries / next  |   tagged union
        +-----------+------------+
                    |
                    | first jump_entry for this key
                    v
__jump_table[]  (sorted by key, then by code address)

  [ entries for key A ... ][ entries for key B ... ] ...
        |                           |
        +--> code / target / key <--+

Notice static_key does not store how many call sites reference it, just a pointer to the first one. That is only enough information because of how the table is sorted: jump_label_sort_entries() orders the whole __jump_table array not by the raw bits stored in entry->key, but by what jump_entry_key() decodes those bits into — the actual, absolute static_key address of the entry.

That distinction matters because, like code and target (§6.2), key is self-relative: it stores a distance to the key, measured from the address of this entry itself in the table, not the address of the key directly. That’s exactly what the function does. It masks off the two flag bits (§7.2) to recover the offset, then adds its own field address back in:

static inline struct static_key *jump_entry_key(const struct jump_entry *entry)
{
    long offset = entry->key & ~3L;

    return (struct static_key *)((unsigned long)&entry->key + offset);
}

Two entries that both belong to the same static_key but sit at different slots in __jump_table measure that distance from two different starting points (&entry->key differs per slot), so their raw key bits will generally differ even though they mean “the same key”:

              addr of      raw key    decode: addr + raw key
              entry->key   field      (jump_entry_key())
              ----------   --------   -----------------------
slot 0 (A):   0x1000       +0x4000    0x1000 + 0x4000 = 0x5000  ─┐
   ...                                                           ├─ same
slot 5 (B):   0x2000       +0x3000    0x2000 + 0x3000 = 0x5000  ─┘  static_key!

                                           static_key @ 0x5000

0x4000 and 0x3000 look unrelated as raw bit patterns — a sort on those values would happily place A and B far apart. Only after factoring in the address of each entry itself do both resolve to the same 0x5000, which is what jump_entry_key() — and therefore the sort — actually compares.

With the array sorted that way, every entry for the same key ends up contiguous, so “find every call site for this key” is just “start at key->entries and keep reading forward until the decoded key changes” — a plain linear scan, no index or count required.

Within that same-key run, entries get a secondary sort by jump_entry_code() — the decoded address of the patchable instruction itself — purely so that addresses come out in ascending order. The batched patching machinery from §10 needs that ordering to do its work efficiently; without it, the entries for one key could point at instructions scattered arbitrarily across memory in no particular sequence.

Why sorting a relative-offset table needs special care. An ordinary in-place sort works by swapping raw bytes between two array slots — but here, swapping the bytes of two entries verbatim would silently corrupt both of them.

The fields of each entry are self-relative (§6.2): they encode “distance from my own address to the target.” Move the bytes of an entry to a different slot, and its own address changes, but a naive byte-for-byte copy would carry over distances computed for the old address, now pointing at the wrong place entirely. jump_label_swap() fixes this by computing delta, the fixed byte distance between the two slots being swapped, and adjusting every field by that same delta (adding it in one direction, subtracting it in the other) as part of the swap — so each field, now living at its new address, still decodes to exactly the same absolute target it did before:

static void jump_label_swap(void *a, void *b, int size)
{
        long delta = (unsigned long)a - (unsigned long)b;
        struct jump_entry *jea = a;
        struct jump_entry *jeb = b;
        struct jump_entry tmp = *jea;

        jea->code   = jeb->code - delta;
        jea->target = jeb->target - delta;
        jea->key    = jeb->key - delta;

        jeb->code   = tmp.code + delta;
        jeb->target = tmp.target + delta;
        jeb->key    = tmp.key + delta;
}

Architectures using the plain, absolute-address form of jump_entry (§7.2) need none of this: an absolute address doesn’t care which slot it happens to sit in, so a plain byte swap already works.

7.4 Linker section

Every translation unit that expands JUMP_TABLE_ENTRY (§6.2) contributes its own little .pushsection __jump_table … .popsection block, scattered across dozens of separately-compiled .o files. Something still has to gather all of those into the one contiguous __jump_table[] array §7.3 assumes exists. That something is the main linker script of the kernel, via include/asm-generic/vmlinux.lds.h:

BOUNDED_SECTION_BY(__jump_table, ___jump_table)

This macro expands to three linker-script directives:

__start___jump_table = .;
KEEP(*(__jump_table))
__stop___jump_table = .;

The middle line is the one doing the actual work: *(__jump_table) tells the linker “collect the __jump_table input section from every object file being linked, in whatever order they’re linked, and place them one after another, right here” — which is exactly how the individually-emitted entries of each translation unit end up concatenated into one array (the same “magic section” trick the kernel also uses for initcalls).

KEEP(...) matters because nothing in the C code of the kernel ever takes the address of this section or calls into it the way it would a normal function — as far as the dead-code elimination of the linker can tell, it looks unreferenced and safe to discard, so KEEP explicitly overrides that and forces it to stay.

The two assignments surrounding it, __start___jump_table and __stop___jump_table, are what let jump_label_init() find the bounds of the array at boot (§9.1) — they resolve to the addresses immediately before and after the concatenated data, with no explicit entry count needed anywhere, mirroring how the linear scan by jump_entry_key() in §7.3 needed no count either.

That linker-driven concatenation only covers code built directly into vmlinux. A module compiled and loaded later has no way to participate in a linker script that already finished running long before the module even existed, so it carries its own private __jump_table section inside its own .ko file instead. When the module loader maps that module in, it reads that section itself and records its bounds in the two fields struct module reserves for exactly this — jump_entries (a pointer to the start of the array owned by that module) and num_jump_entries (how many entries it holds, since there is no vmlinux-wide linker symbol to bound it by). Those are precisely the entries that end up wrapped in a struct static_key_mod node (§7.1) whenever a key is shared between a module and vmlinux, or between two modules.

8 Size of the patchable site on x86 (runtime)

§6.3 showed that a nop-default site can end up being either 2 or 5 bytes, decided by the assembler at build time based on the real distance to l_yes. Nothing in struct jump_entry (§7.2) records which one it ended up as — there is no size field anywhere in it. So months or years later, when this key is actually toggled on a running system, the code doing the patching has to rediscover that size from scratch, by looking at the live bytes currently sitting in .text:

/* arch/x86/kernel/jump_label.c */
int arch_jump_entry_size(struct jump_entry *entry)
{
        struct insn insn = {};

        insn_decode_kernel(&insn, (void *)jump_entry_code(entry));
        BUG_ON(insn.length != 2 && insn.length != 5);
        return insn.length;
}

insn_decode_kernel() is the general-purpose x86 instruction decoder of the kernel (the same kind of code that also has to understand arbitrary instructions for kprobes), pointed here at the address that jump_entry_code(entry) from §6.2 already knows how to recover. It does not just count bytes: it genuinely parses the instruction at that address (opcode, prefixes, displacement, all of it) and reports how many bytes it occupies, via insn.length. Whatever is sitting there right now — the original compiler-emitted nop or jmp from §6.3, or a previously-patched replacement from an earlier toggle — this call decodes it and returns its true size.

BUG_ON(insn.length != 2 && insn.length != 5) is a sanity check, not a normal error path: if this ever decodes to any width other than the two shapes §6.3 established, something has gone badly wrong (the table and the text it describes have desynchronized), and there is no safe way to keep patching.

This decode-on-demand step is specific to x86. Other architectures — arm64, for instance, which defines a fixed JUMP_LABEL_NOP_SIZE — never need it, because every patchable site on those architectures is always the same width, known in advance, with nothing to discover. x86 is the odd one out precisely because the size optimization from §6.3 means the width of a site is a fact about that specific call site, not a constant true of the whole kernel. The architecture-independent jump_entry_size() reflects that split: it returns the fixed JUMP_LABEL_NOP_SIZE where an architecture defines one, and only falls back to calling arch_jump_entry_size() — the decoder above — when no such constant exists:

static inline int jump_entry_size(struct jump_entry *entry)
{
#ifdef JUMP_LABEL_NOP_SIZE
    return JUMP_LABEL_NOP_SIZE;
#else
    return arch_jump_entry_size(entry);
#endif
}

Once the size is known, __jump_label_patch() builds the two byte sequences this site could possibly need — the jmp form and the nop form, regardless of which one is about to be installed — and picks between them:

size = arch_jump_entry_size(entry);
switch (size) {
case JMP8_INSN_SIZE:   /* 2 */
        code = text_gen_insn(JMP8_INSN_OPCODE, addr, dest);
        nop  = x86_nops[size];
        break;
case JMP32_INSN_SIZE:  /* 5 */
        code = text_gen_insn(JMP32_INSN_OPCODE, addr, dest);
        nop  = x86_nops[size];
        break;
}

JMP8_INSN_OPCODE (0xEB) and JMP32_INSN_OPCODE (0xE9) are the same two jump encodings named in the rel8/rel32 explanation from §6.3, just given their real opcode values here. text_gen_insn() builds the actual jmp bytes for whichever opcode it’s given, computing the displacement itself as dest - (addr + size) — precisely the “distance from the address of the next instruction” convention §6.3 described for rel8/rel32 encodings, just computed here at patch time instead of by the assembler at build time. x86_nops is a small lookup table of ready-made no-op byte sequences, one per length, so x86_nops[size] hands back a correctly-sized nop without having to construct one on the fly — the same 2-byte and 5-byte encodings already named in §6.4.

Building both code and nop up front, rather than only the one being installed, is what makes the next safety check possible. Before writing anything, __jump_label_patch() decides which of the two it expects to find already sitting at this address — and, importantly, that expectation is the opposite of what it is about to install: if type says “install a jmp,” the live bytes had better currently be the old nop (that is the only state a nop-default site should be transitioning from); if type says “install a nop,” the live bytes had better currently be the old jmp.

A memcmp() of the real bytes against that expectation runs before any write. If they don’t match, the jump-table metadata and the actual instruction stream have diverged somehow — corruption, a bug elsewhere in the kernel, or a race this code did not anticipate — and there is no safe way to proceed: it logs the mismatch via pr_crit and calls BUG(), deliberately crashing rather than patching blind and risking that the CPU ends up executing whatever garbage caused the mismatch in the first place.

9 Life of a static key: boot, enable, disable

Everything in this section is driven by one atomic_t: key->enabled. Its value tells you both “is the feature on” and “is a patch pass currently running”. The boolean on/off cycle looks like this:

        set to -1          patch, set 1          cmpxchg(1,0), off
+----+             +----+                +----+                     +----+
|  0 | ----------> | -1 | -------------> |  1 | ------------------> |  0 |
+----+             +----+                +----+                     +----+
 off               enable                  on                        off

Read left to right, the three values in that diagram mean:

There is no symmetric -1-like transient for disable: going 1 -> 0 reads as briefly stale “on” instead (see §9.3), which is harmless.

Above value 1, a second, independent ladder exists purely for refcounting — static_branch_inc()/dec() (§4.3) climb and descend it without ever touching text, because the key is already known to be enabled:

   1 --inc--> 2 --inc--> 3 --inc--> ...        (static_key_fast_inc_not_disabled:
   1 <--dec-- 2 <--dec-- 3 <--dec-- ...          pure atomic increment/decrement,
                                                  no jump_label_update() at all)

Only a dec that would land exactly on 1 -> 0 re-enters the boolean cycle above and triggers a real patch-off. Put differently: of all the edges across both diagrams, only the three that make up the boolean cycle itself (0 -> -1, -1 -> 1, and 1 -> 0) ever touch instruction bytes. Every step on the refcount ladder above 1 (1<->2, 2<->3, …) is a bare atomic increment or decrement, with no jump_label_update() call anywhere in it.

That single fact is the whole point of the refcounted API in §4.3: many callers can share a key without each one paying for a text-patch round-trip — the cost of patching is paid exactly once, by whichever caller happens to be the first to enable it or the last to disable it.

9.1 Boot: jump_label_init()

Before this function ever runs, the jump-label machinery is in a half-built state. The linker has already concatenated the slice of __jump_table from every translation unit into one array (§7.4), but that array is simply in link order — sites for the same key can be scattered anywhere in it, and key->entries still holds whatever its static initializer left there, which for a plain struct static_key key = STATIC_KEY_INIT_FALSE; (§7.1) is nothing useful. In other words: the raw table of call sites exists, but nothing yet knows which sites belong to which key, so static_branch_enable() or static_key_slow_inc() (§4.3, §9.2) would have nothing to walk if called this early.

arch/x86/xen/multicalls.c declares static struct static_key mc_debug __ro_after_init; and registers xen_mc_debug as an early_param() whose handler calls static_key_slow_inc(&mc_debug) directly, synchronously, while parse_early_param() is walking the command line. If that increment ran before jump_label_init() had sorted the table and pointed mc_debug at its entries, it would have nothing to patch and the key would silently stay un-patched despite the user asking for it on the command line.

The kernel guards against exactly this ordering mistake with one boolean, static_key_initialized — whose only job, per its own comment, is “to generate warnings if static_key manipulation functions are used before jump_label_init is called”; jump_label_init_ro() later even enforces it with a WARN_ON_ONCE(). That is why jump_label_init() is called very early from start_kernel() in init/main.c, strictly before parse_early_param() gets a chance to run any handler like xen_parse_mc_debug():

void __init jump_label_init(void)
{
        ...
        jump_label_sort_entries(iter_start, iter_stop);

        for (iter = iter_start; iter < iter_stop; iter++) {
                if (jump_label_type(iter) == JUMP_LABEL_NOP)
                        arch_jump_label_transform_static(iter, JUMP_LABEL_NOP);

                in_init = init_section_contains((void *)jump_entry_code(iter), 1);
                jump_entry_set_init(iter, in_init);

                iterk = jump_entry_key(iter);
                if (iterk == key)
                        continue;
                key = iterk;
                static_key_set_entries(key, iter);
        }
        static_key_initialized = true;
}

Before the loop even starts, jump_label_sort_entries() sorts the whole of __jump_table by key, then by code address — the precondition that lets the “walk all sites for this key as a linear scan” from §7.3 and the “batch must stay address-ordered” requirement from §10.2 both work later.

The loop that follows does three things per entry, in this order, trusting that the table is already sorted:

  1. Optionally run arch_jump_label_transform_static for NOP sites — a no-op on x86, since objtool and the compiler have already left the right bytes in place (§6.3); other architectures without that build-time trick do real work here.
  2. Mark __init sites, so nothing later tries to patch a call site living in memory that will be freed once init finishes.
  3. Point each key at the first entry of its contiguous run in the now-sorted table, so key->entries is ready to use the moment something calls static_branch_enable().

None of this touches instruction bytes for sites that are already correct: the compiled-in nop/jmp already matches the initial value of each key (§5). What this pass builds is bookkeeping — sort order, the __init flag, and the entries pointer — so that a later toggle knows exactly which sites to patch and in what order.

jump_label_init_ro() runs much later, from mark_readonly(). It walks __jump_table a second time, but this pass skips almost everything — it only acts on keys that is_kernel_ro_after_init() recognizes as living in __ro_after_init storage.2 For each matching key it calls:

static inline bool static_key_sealed(struct static_key *key)
{
        return (key->type & JUMP_TYPE_LINKED) && !(key->type & ~JUMP_TYPE_MASK);
}

static inline void static_key_seal(struct static_key *key)
{
        unsigned long type = key->type & JUMP_TYPE_TRUE;
        key->type = JUMP_TYPE_LINKED | type;
}

static_key_seal() keeps only the JUMP_TYPE_TRUE bit of the current key->type and throws the rest of the word away, replacing it with JUMP_TYPE_LINKED plus that one preserved bit. Recall from §7.1 that this word is normally a tagged pointer — the low 2 bits are a type tag, and everything above them is either the entries pointer the loop earlier in this section just installed, or a module-chain next pointer from §11. Sealing collapses it down to just the tag, with nothing left standing above bit 1:

before sealing (a live `entries` or `next` pointer, tagged):

    bit63                                        bit1  bit0
    [ real pointer value ..................... ] [ L ] [ T ]

after static_key_seal():

    bit63                                        bit1  bit0
    [ 0000000000000000000000000000000000000000 ] [ 1 ] [ T ]

T is the preserved JUMP_TYPE_TRUE bit; L is JUMP_TYPE_LINKED. static_key_sealed() is exactly the test for the “after” picture — JUMP_TYPE_LINKED set and nothing above bit 1 — so it can recognize a sealed key at a glance, regardless of which of the two pointer kinds that word used to hold.

Several call sites can share the same key, so this loop visits the same key more than once as it walks the table entry by entry. static_key_sealed() is checked before static_key_seal() runs, so the first visit seals the key and every later visit for that same key becomes a no-op.

This is safe only because __ro_after_init is a promise that the key is never toggled again after boot: once sealed, key->entries is gone, so nothing could walk it even if some later code mistakenly tried.

The payoff shows up in §11. When a module loaded afterward references a sealed key, jump_label_add_module() skips allocating and chaining a struct static_key_mod for it — the bookkeeping §11 otherwise needs so a future toggle can find and patch call sites living in other modules. Instead, it just patches the sites of that module once, immediately, to match the already-final value of the key.

The ordering against mark_rodata_ro() follows from the same fact: sealing is the last write anything makes to key->type, and that field sits inside the very __ro_after_init section mark_rodata_ro() is about to make actually read-only in the page tables. Running first just means the field has already settled into its final value before write access to it disappears.

9.2 Enabling: static_key_enable() / static_branch_enable()

jump_label_init() (§9.1) only builds bookkeeping — it never flips a key. Every site in the kernel image boots running whatever nop/jmp the compiler and objtool baked in (§5, §6), on or off, and stays that way until something calls static_branch_enable() or static_key_slow_inc() for the first time. This section is about that first flip, which can happen years into uptime, on a system where other CPUs may already be executing the very instructions about to be rewritten.

drivers/md/dm-stats.c shows a concrete case of this (§4.4): the first time a user asks the device-mapper stats ioctl to start recording per-region I/O counters, it flips stats_enabled, a key that has sat disabled since boot:

if (!static_key_enabled(&stats_enabled.key))
        static_branch_enable(&stats_enabled);

The guard matters as much as the call: it is what keeps a second, third, or hundredth request for the same device from re-triggering a full patch round once the key is already on (§4.4). But the first time through, this static_branch_enable(&stats_enabled) does run, and from that call onward every stats_enabled check in the I/O path — on every CPU, some of which may be running that exact code right now — has to observe the new state, and none of them may ever see a half-patched instruction. static_key_enable_cpuslocked() below is what makes that safe:

void static_key_enable_cpuslocked(struct static_key *key)
{
        ...
        jump_label_lock();
        if (atomic_read(&key->enabled) == 0) {
                atomic_set(&key->enabled, -1);      /* "enabling" */
                jump_label_update(key);             /* patch all sites */
                atomic_set_release(&key->enabled, 1);
        }
        jump_label_unlock();
}

Walking through what that function does, in order:

  1. It first checks whether the key is already enabled, and bails out if so.
  2. jump_label_mutex serializes all jump-label patching globally, so two callers enabling different keys at the same time still cannot have their text pokes race one another.
  3. It sets enabled = -1 before touching any code — this is the 0 → -1 transient from the state diagram above, and it is what makes jump_label_update() below safe to run concurrently with readers: static_key_count() and static_key_enabled() both treat -1 as enabled, so no concurrent reader ever sees a window of “disabled” while sites are only half-patched.
  4. jump_label_update() does the actual work: it patches every site for this key (§9.5).
  5. Finally, it stores 1.

static_key_enable() wraps this in cpus_read_lock() so CPUs cannot come online mid-patch.

The refcounted path, static_key_slow_inc_cpuslocked(), is not just this same function reused for inc/dec (§4.3). With multiple independent owners, more than one CPU can call it at once, and only one of them may actually be the one that flips 0 → 1:

bool static_key_slow_inc_cpuslocked(struct static_key *key)
{
        lockdep_assert_cpus_held();

        if (static_key_fast_inc_not_disabled(key))
                return true;

        guard(mutex)(&jump_label_mutex);
        if (!atomic_cmpxchg(&key->enabled, 0, -1)) {
                jump_label_update(key);
                atomic_set_release(&key->enabled, 1);
        } else {
                if (WARN_ON_ONCE(!static_key_fast_inc_not_disabled(key)))
                        return false;
        }
        return true;
}

It tries a lock-free fast path first (below), and only takes jump_label_mutex if that fails. Past that point it is the same 0 → 1 dance as static_key_enable_cpuslocked() above: exactly one caller actually performs the transition and runs jump_label_update(); anyone else who reaches the mutex finds the key already on and just falls back to the fast path to add their own count.

static_key_fast_inc_not_disabled() is what makes “further incs” in the table from §4.3 cost nothing more than an atomic — no lock, no text poke:

bool static_key_fast_inc_not_disabled(struct static_key *key)
{
        int v;

        STATIC_KEY_CHECK_USE(key);
        /*
         * Negative key->enabled has a special meaning: it sends
         * static_key_slow_inc/dec() down the slow path, and it is non-zero
         * so it counts as "enabled" in jump_label_update().
         *
         * The INT_MAX overflow condition is either used by the networking
         * code to reset or detected in the slow path of
         * static_key_slow_inc_cpuslocked().
         */
        v = atomic_read(&key->enabled);
        do {
                if (v <= 0 || v == INT_MAX)
                        return false;
        } while (!likely(atomic_try_cmpxchg(&key->enabled, &v, v + 1)));

        return true;
}

The retry loop only succeeds once it can prove, atomically, that the count is already an ordinary positive number. v <= 0 catches both a genuinely disabled key (0) and an enable already in progress (-1), sending both cases down to the slow path above instead of incrementing a value that doesn’t mean what it looks like yet. v == INT_MAX guards against overflows.

9.3 Disabling

§9.2 walked through what happens when a key crosses from off to on. Disabling is the same problem in reverse: patch every site back to its original instruction, without letting any reader ever observe a torn one. It runs on the very same rule as enabling — not every call to disable/dec needs to touch an instruction at all. jump_label_update() only has to run on the one transition that actually flips what is patched into the instruction stream: 1 → 0 for the boolean API, N → 0 for the refcounted one. Every other call just moves a plain integer and can be answered with a single atomic instruction — no lock, no text poke, because nothing about the patched code needs to change.

That is exactly the same shape as the static_key_fast_inc_not_disabled() from §9.2 on the enable side — a bare CAS loop, no lock, no patch, for every increment that doesn’t cross 0 → 1. That rule (every transition that isn’t on a patch-triggering boundary is a bare atomic op) is what both functions below are built around, on either side of the key.

void static_key_disable_cpuslocked(struct static_key *key)
{
        ...
        if (atomic_read(&key->enabled) != 1) {
                WARN_ON_ONCE(atomic_read(&key->enabled) != 0);
                return;
        }

        jump_label_lock();
        if (atomic_cmpxchg(&key->enabled, 1, 0) == 1)
                jump_label_update(key);
        jump_label_unlock();
}

static_key_disable_cpuslocked() is the mirror, in the boolean API, of the enable path from §9.2, but it gets to skip the -1 choreography of that path entirely — for a reason worth spelling out rather than just noting. It first checks that the key is currently exactly 1, bailing out (and warning if the value isn’t 0 either, which would mean the boolean and refcounted APIs got mixed on this key — §4.3) before doing anything else. The actual disable is then a single atomic_cmpxchg(&key->enabled, 1, 0): “if the value is currently 1, replace it with 0, and tell me whether you succeeded.” If some other CPU changed it first, the compare fails and this call does nothing further. Only on success does it call jump_label_update() to patch every site for this key back to its disabled instruction.

Compare that to enabling: there, enabled is deliberately set to -1 before any text is touched, specifically so no reader can mistake in-progress patching for “off” (point 3 of §9.2). Disabling has no matching problem to solve. During the window between the cmpxchg above and jump_label_update() finishing, enabled already reads 0 while some sites out there are still physically holding their “enabled” instruction — the opposite kind of staleness from enabling (stale “on” instead of stale “off”), but just as harmless. A reader who calls static_key_enabled() during that window is simply told “off” a few instructions before the code itself has caught up; §4.4 already covers why these state reads only ever need to be eventually correct, never instruction-exact.

The refcounted side runs through a different function, __static_key_slow_dec_cpuslocked(), which only reaches jump_label_update() on the one decrement that actually drives the count to 0 (atomic_dec_and_test() reports true exactly then, and only then). Every decrement that lands above 1 — 3 -> 2, 2 -> 1, and so on — is intercepted earlier, by static_key_dec_not_one(), which performs a plain atomic decrement and returns without ever taking the jump-label lock or looking at an instruction stream. That is the same “every transition that isn’t on a patch-triggering boundary is a bare atomic op” rule we discussed earlier, now seen from the decrement side:

static bool static_key_dec_not_one(struct static_key *key)
{
        int v;

        v = atomic_read(&key->enabled);
        do {
                WARN_ON_ONCE(v < 0);

                if (WARN_ON_ONCE(v == 0))
                        return true;
                if (v <= 1)
                        return false;
        } while (!likely(atomic_try_cmpxchg(&key->enabled, &v, v - 1)));

        return true;
}

The CAS loop is what makes this safe against concurrent decrements: if another CPU wins the race and changes key->enabled between the read of this CPU and its cmpxchg, v is refreshed and the loop just re-checks the same two conditions against the new value rather than clobbering it.

9.4 Deferred / rate-limited dec

This is the internal counterpart to the rate-limited disable in §4.7, seen here from the function-by-function angle of this section rather than the call-site angle of the cookbook. §4.7 already covers why a delayed decrement exists and how the coalescing works when toggles arrive faster than the timeout; this is only the piece that was left out there — where the state-machine logic actually lives.

__static_key_slow_dec_deferred() opens with the exact same static_key_dec_not_one() check §9.3 just introduced: if this decrement would not bring the count down to the 1 -> 0 boundary, it is a plain atomic decrement and the function returns immediately, no different from the non-deferred path.

The two paths only diverge on the one decrement that would disable the key. Where the __static_key_slow_dec_cpuslocked() from §9.3 reaches straight for jump_label_update() at that point, this function instead calls schedule_delayed_work() — a standard kernel workqueue primitive that runs a callback once a given delay has elapsed, rather than right away — and returns without touching a single instruction. The count is deliberately left at 1 rather than dropped to 0; only the plan to disable has been recorded, in the timer. In full:

void __static_key_slow_dec_deferred(struct static_key *key,
                    struct delayed_work *work,
                    unsigned long timeout)
{
        if (static_key_dec_not_one(key))
                return;

        schedule_delayed_work(work, timeout);
}

When that timer eventually fires, jump_label_update_timeout() runs the ordinary, undeferred decrement path from §9.3. If nothing else touched the key in the meantime, the count is still exactly 1, the decrement finally lands on the 1 -> 0 boundary, and jump_label_update() runs for real. If instead another caller incremented the key again while the timer was pending, the count is no longer 1 by the time the timer fires — so static_key_dec_not_one() intercepts that decrement too, as an ordinary atomic op, and jump_label_update() is never reached. Nothing needed patching back, because nothing was ever patched in the first place.

9.5 jump_label_update() → __jump_label_update()

Every path in §9.2 through §9.4 eventually funnels into this function — it is the one that actually walks the sites of a key and asks the architecture layer to patch each one. Reading the outer function first:

static void jump_label_update(struct static_key *key)
{
        ...
        if (static_key_linked(key)) {
                __jump_label_mod_update(key);   /* walk module list */
                return;
        }
        entry = static_key_entries(key);
        if (entry)
                __jump_label_update(key, entry, stop, init);
}

The first branch is the module case from §7.1: if bit 1 of key->type is set (LINKED), the sites of this key are not one contiguous run inside __jump_table, but scattered across a linked list of per-module entry tables (struct static_key_mod), so a separate helper has to walk that list instead of a flat array — §11 covers __jump_label_mod_update() in full. Otherwise, static_key_entries(key) recovers the pointer §9.1 stored during boot — the first entry of the contiguous run for this key inside the sorted, vmlinux-only table — and the real patching happens in __jump_label_update().

On x86, which defines HAVE_JUMP_LABEL_BATCH, that function looks like this:

for (; entry < stop && jump_entry_key(entry) == key; entry++) {
        if (!jump_label_can_update(entry, init))
                continue;
        if (!arch_jump_label_transform_queue(entry, jump_label_type(entry))) {
                arch_jump_label_transform_apply();
                BUG_ON(!arch_jump_label_transform_queue(...));
        }
}
arch_jump_label_transform_apply();

The loop condition — advance while entry < stop and jump_entry_key(entry) == key — only works because of the sort from §9.1: every site belonging to this key is guaranteed to sit in one unbroken run starting at entry, so the loop can walk forward blindly and stop the instant it reaches a site belonging to some other key, with no need to search the rest of the table.

For each site still in range, jump_label_type(entry) computes enabled ^ branch — the exact static/dynamic formula from the table in §5, but now evaluated against the current live state of the key rather than its compile-time initial value — to decide whether this specific site should end up holding a JUMP_LABEL_NOP or a JUMP_LABEL_JMP.

Before acting on that answer, jump_label_can_update() filters out two kinds of site that must not be touched at all: one still living in __init text after boot has finished (that memory may already have been freed, which is exactly what the jump_entry_is_init flag from §9.1 was recorded to detect), and one that kernel_text_address() does not even recognize as live kernel text — built-in code that is __exit-only and therefore can never run, so patching it would be pointless even though it is technically still present:

static bool jump_label_can_update(struct jump_entry *entry, bool init)
{
        if (!init && jump_entry_is_init(entry))
                return false;

        if (!kernel_text_address(jump_entry_code(entry))) {
                WARN_ONCE(!jump_entry_is_init(entry),
                          "can't patch jump_label at %pS",
                          (void *)jump_entry_code(entry));
                return false;
        }

        return true;
}

Rather than patching each surviving site immediately, one at a time, the loop splits the work into two separate jobs: arch_jump_label_transform_queue() builds up a batch, and arch_jump_label_transform_apply() executes it. Splitting them apart is what lets every site belonging to one key ride through a single INT3-synchronized patch round (§10.2) instead of paying for one round per site.

Queuing a site. Each call to arch_jump_label_transform_queue() computes the replacement bytes for that one site and hands (address, new bytes, length) to smp_text_poke_batch_add(). That function appends the request to a pending array; nothing is written to memory yet. The one exception is early boot: only one CPU is running, so there is no concurrent fetcher to synchronize against and nothing worth batching — the function calls the non-batching arch_jump_label_transform() directly instead.

Applying the batch. arch_jump_label_transform_apply() executes everything queued so far. It calls smp_text_poke_batch_finish() which runs the three-step INT3 dance once for the whole batch instead of once per site. __jump_label_update() calls it once, after its loop ends, to flush whatever is still pending.

10 x86 text patching: the gory details

This is where the “you cannot just memcpy over live code” argument from §1.3 turns into actual working code. Roadmap: §10.1 picks early-boot-single-CPU vs. later-multi-CPU; §10.2 covers the batching array that collects many sites before paying for synchronization; §10.3 is the INT3 protocol itself — the heart of the mechanism; §10.4–§10.5 fill in what “synchronize” and “write to RO memory” actually mean underneath; §10.6 strings it all into one timeline.

10.1 Early boot vs live SMP

Very early in boot, only the boot CPU is running and .text is still writable. __jump_label_transform(), the function every x86 patch eventually funnels through, checks for exactly that window and takes the cheap route whenever it still holds:

/* arch/x86/kernel/jump_label.c: __jump_label_transform() */
if (init || system_state == SYSTEM_BOOTING) {
        text_poke_early(...); /* IRQ-disabled + `memcpy()` + sync_core() */
        return;
}
smp_text_poke_single(...);      /* or batch_add during queueing (§9.2) */

system_state == SYSTEM_BOOTING is a proxy for one fact: only the boot CPU exists so far. That alone is reason enough to skip the whole IPI-synchronized protocol of §10.3 — with nobody else around to race the write, text_poke_early() can just do a plain IRQ-disabled + memcpy() + sync_core() and be done. .text also happens to still be writable at this point, so the alias trick from §1.4 isn’t needed either — but that is a bonus the check gets for free, not something it verifies directly: smp_init() wakes every other CPU well before mark_rodata_ro() ever runs, so there is a real stretch of boot where other CPUs are already up while .text is still writable. Jump labels take the full protocol for that entire stretch anyway, because the only thing this check ever verifies is whether this CPU is still provably alone.

The leading init in that condition is easy to misread as the per-site __init-text flag from §9.1 (jump_entry_is_init()) — it is not. It is a separate, whole-system flag threaded down from init = system_state < SYSTEM_RUNNING inside jump_label_update() itself (§9.5), true a little longer than SYSTEM_BOOTING alone. On the batching path of x86, though, that value never actually reaches here: arch_jump_label_transform_queue() only calls this function through its own system_state == SYSTEM_BOOTING fallback, passing a hardcoded 0 for init every time it does. So on x86 this condition is, in practice, exactly system_state == SYSTEM_BOOTING — the init || half of it only ever matters on architectures that call this function directly, without going through batching at all.

10.2 Batching API used by jump labels

§10.1 settled how a single site gets patched once the decision to patch it is made; this section is about when jump labels actually pull that trigger. A busy tracepoint can have thousands of call sites sharing one key, and paying the full IPI-synchronized protocol (§10.3) separately for each one would be needless — the sites can be collected first and the expensive part paid once for the whole group. Two arch-level hooks make that possible: one that queues the newly computed bytes for a site without touching hardware yet, and one that flushes everything queued so far in a single synchronized pass:

bool arch_jump_label_transform_queue(...)
{
        if (system_state == SYSTEM_BOOTING) {
                arch_jump_label_transform(entry, type);
                return true;
        }
        mutex_lock(&text_mutex);
        jlp = __jump_label_patch(entry, type);
        smp_text_poke_batch_add(addr, jlp.code, jlp.size, NULL);
        mutex_unlock(&text_mutex);
        return true;
}

void arch_jump_label_transform_apply(void)
{
        mutex_lock(&text_mutex);
        smp_text_poke_batch_finish();
        mutex_unlock(&text_mutex);
}

The queue collects many sites (a busy tracepoint may have thousands); one batch_finish() amortizes the IPI syncs. The queue must stay address-sorted; if a new address would break order, or the page-sized array is full, smp_text_poke_batch_add() flushes early (text_poke_addr_ordered() in alternative.c). That is why jump_label_cmp sorts by code address within each key.

How many fit in one batch? The pending patches live in a single statically-allocated page, struct smp_text_poke_loc:

struct smp_text_poke_loc {
        s32 rel_addr;   /* addr := _stext + rel_addr           */
        s32 disp;       /* branch displacement, for emulation  */
        u8  len;        /* 1, 2, 5, or 6 bytes                 */
        u8  opcode;     /* first opcode byte, for emulation    */
        u8  text[5];    /* the new instruction bytes           */
        u8  old;        /* byte that used to be there, for perf/PT tracing */
};                       /* 16 bytes, naturally aligned         */

#define TEXT_POKE_ARRAY_MAX (PAGE_SIZE / sizeof(struct smp_text_poke_loc))
/* 4096 / 16 = 256 entries per flush on a 4K-page x86_64 build */

rel_addr is relative to _stext3, not to the entry itself like jump_entry — cheap, because every patch site is, by definition, in kernel text. A tracepoint or jump-label key with more than 256 call sites needs more than one batch_finish() round (i.e. more than 3 IPI rounds, §10.3) to fully enable/disable.

__jump_label_update() (§9.5) does contain a generic “queue full → apply → retry” branch, and it looks like the natural place to expect this 256-site overflow to be handled.

On x86 the “queue full → apply → retry” branch never runs, though: it only fires when arch_jump_label_transform_queue() itself returns false, and the implementation of that function on this architecture never returns false. So the branch is dead code here.

for (; (entry < stop) && (jump_entry_key(entry) == key); entry++) {

        if (!jump_label_can_update(entry, init))
                continue;

        if (!arch_jump_label_transform_queue(entry, jump_label_type(entry))) {
                /*
                 * Queue is full: Apply the current queue and try again.
                 */
                arch_jump_label_transform_apply();
                BUG_ON(!arch_jump_label_transform_queue(entry, jump_label_type(entry)));
        }
}
arch_jump_label_transform_apply();

The overflow is actually caught one layer further down, entirely inside smp_text_poke_batch_add() — the entire guard is these three lines:

void smp_text_poke_batch_add(void *addr, const void *opcode, size_t len, const void *emulate)
{
        if (text_poke_array.nr_entries == TEXT_POKE_ARRAY_MAX || !text_poke_addr_ordered(addr))
                smp_text_poke_batch_finish();
        __smp_text_poke_batch_add(addr, opcode, len, emulate);
}

Every iteration of that for loop does one thing: it calls arch_jump_label_transform_queue(), which computes the new bytes for one site and appends one smp_text_poke_loc to the array — the cheap, per-site step this section has been describing. The if (!arch_jump_label_transform_queue(...)) branch is the “queue full → apply → retry” path discussed above; on x86 it never runs, since that function never returns false. Nothing expensive happens inside the loop.

arch_jump_label_transform_apply() sits outside the loop, called exactly once after it exits, for every site the loop just queued. That single call is what finally triggers batch_finish(), the three-phase IPI-synchronized protocol from §10.3.

So the entire set of call sites for a key — whether it has one or close to 256 — rides through on that one shared batch_finish(), and only a key with more than 256 sites forces a second round.

10.3 The INT3 SMP algorithm (smp_text_poke_batch_finish)

This is the payoff of everything §1.3 through §10.2 built toward. The key fact from §1.3 was that only a single-byte store is atomic with respect to instruction fetch — nothing wider is. The protocol below never trusts a multi-byte write to be safe on its own; instead it uses one atomic single-byte store to plant a trap on top of the site, uses that trap to absorb any CPU unlucky enough to fetch through mid-update, and only then fills in the rest. Three writes, three synchronizations, one site at a time across the whole batch. Documented at the top of the function in alternative.c:

For each site in the vector:
  (1) Write INT3 (0xCC) over the first byte
      → IPI sync all CPUs   (serialize pipelines / I-caches)

  (2) Write bytes 1..N-1 of the new instruction
      → IPI sync again      unnecessary, according to Intel,
                            but better safe than sorry 

  (3) Write byte 0 of the new instruction (replaces INT3)
      → IPI sync again

Here are the actual bytes of one site across the three phases, patching a 5-byte NOP (0f 1f 44 00 00) into a 5-byte JMP rel32 (e9 + 4-byte displacement, shown as ?? ?? ?? ??):

 start (before)      0f 1f 44 00 00      any fetch: executes the NOP

 phase 1 (INT3 in)  cc 1f 44 00 00      any fetch: #BP -> handler emulates
                    ^^                  the NEW instruction (jumps to l_yes)
                    trap byte

                    -------- IPI sync --------

 phase 2 (tail in)  cc ?? ?? ?? ??      same as phase 1: byte 0 is still
                    ^^                  INT3, so any fetch still traps and
                    still traps         gets emulated — the real tail bytes
                                        underneath are now correct, but
                                        nothing reads them yet

                    -------- IPI sync --------

 phase 3 (done)     e9 ?? ?? ?? ??      any fetch: executes the real JMP
                    ^^ real opcode      directly, no trap needed anymore

                    -------- IPI sync --------

The subtle point this diagram is here to make: the observable behavior of the site flips the instant the sync in phase 1 completes, not at phase 3. From phase 1 onward, any CPU landing on this address — whether by falling into it in a hot loop or by literally executing byte 0 — gets the effect of the new instruction, because the #BP handler always emulates the pending new instruction (it was computed and stashed in the queue back at smp_text_poke_batch_add() time, long before phase 1 starts). A CPU that hits the address mid-update and one that hits it after phase 3 land on the same outcome — the only difference is whether it got there by trapping into the handler or by executing the finished bytes directly. Phases 2 and 3 exist to make that direct path available, so steady-state execution stops paying the #BP tax.

The writing side. All of the above is driven by smp_text_poke_batch_finish() (it early-returns immediately if text_poke_array.nr_entries is 0 — nothing queued, nothing to do). Trimmed of the cond_resched() softlockup guard, the perf/Intel-PT tracing hook, and a 6-byte-opcode edge case, it opens by arming the refcount of every CPU — the release side of the release/acquire pairing the “Why INT3?” sidebar below explains — then runs the three phases in order:

for_each_possible_cpu(i)
        atomic_set_release(per_cpu_ptr(&text_poke_array_refs, i), 1);
smp_wmb();

Phase 1 writes INT3 over the first byte of every site, saving the byte it replaces (for the perf/PT tracing hook trimmed out above), then syncs once for the whole batch:

for (i = 0; i < text_poke_array.nr_entries; i++) {
        text_poke_array.vec[i].old = *(u8 *)text_poke_addr(&text_poke_array.vec[i]);
        text_poke(text_poke_addr(&text_poke_array.vec[i]), &int3, INT3_INSN_SIZE);
}
smp_text_poke_sync_each_cpu();

Phase 2 writes bytes 1..len-1 of the new instruction for every site — safe, since byte 0 is still INT3 — and syncs again, but only if some site actually has tail bytes to write (a single-byte patch has none):

for (do_sync = 0, i = 0; i < text_poke_array.nr_entries; i++) {
        int len = text_poke_array.vec[i].len;

        if (len - INT3_INSN_SIZE > 0) {
                text_poke(text_poke_addr(&text_poke_array.vec[i]) + INT3_INSN_SIZE,
                          text_poke_array.vec[i].text + INT3_INSN_SIZE,
                          len - INT3_INSN_SIZE);
                do_sync++;
        }
}
if (do_sync)
        smp_text_poke_sync_each_cpu();

Phase 3 writes byte 0, replacing the INT3 — skipped for the corner case (discussed next) where the new opcode is itself 0xCC — and syncs again if anything actually changed:

for (do_sync = 0, i = 0; i < text_poke_array.nr_entries; i++) {
        u8 byte = text_poke_array.vec[i].text[0];

        if (byte == INT3_INSN_OPCODE)
                continue;
        text_poke(text_poke_addr(&text_poke_array.vec[i]), &byte, INT3_INSN_SIZE);
        do_sync++;
}
if (do_sync)
        smp_text_poke_sync_each_cpu();

Draining after phase 3. After the last sync, the writer cannot yet assume every CPU has left the handler — a CPU could be inside smp_text_poke_int3_handler() right up until that sync completes (it entered before the sync, is still emulating). So smp_text_poke_batch_finish() ends with this drain and the final reset:

for_each_possible_cpu(i) {
        atomic_t *refs = per_cpu_ptr(&text_poke_array_refs, i);

        if (unlikely(!atomic_dec_and_test(refs)))
                atomic_cond_read_acquire(refs, !VAL);
}
text_poke_array.nr_entries = 0;

atomic_dec_and_test() decrements the refcount of every CPU; for any that don’t immediately hit zero, atomic_cond_read_acquire(refs, !VAL) spin-waits — i.e. for stragglers still mid-handler to finish and drop their own reference. Only then does the function reset text_poke_array.nr_entries = 0, making the buffer safe to reuse for the next batch.

In the common case — jump labels, static calls, ftrace, none of which ever replace a site with a literal 0xCC — phase 3 already wrote every final byte, and smp_text_poke_sync_each_cpu() already fenced out any in-flight handler. So this drain loop observes zero immediately, and the comment in the source calls it out explicitly: “unless the replacement instruction is INT3, this case goes unused.”

It exists for the corner case (other clients, not jump labels) where the new opcode of a site is 0xCC itself. Byte 0 is therefore left alone in phase 3 (writing 0xCC over an existing 0xCC would be a no-op anyway, so that per-entry sync is skipped), and the only thing standing between “batch done” and “safe to reuse the array” is the atomic_cond_read_acquire() spin-wait itself.

Why INT3? The whole point of writing INT3 first (phase 1 of the three-phase protocol above) is that it gives the kernel a way to intercept any CPU that would otherwise have executed half-written bytes, and make it run the finished instruction instead. While the site is mid-update, any CPU that hits it takes #BP (a fault, vector 3), and that fault always lands in smp_text_poke_int3_handler(), which is wired up as the #BP handler in traps.c. Its locals are just tpl (the matched struct smp_text_poke_loc *), ret, and ip; here is what it actually does, one check at a time.

1. Bail on user mode. This handler only ever concerns itself with traps hit while executing kernel text:

if (user_mode(regs))
        return 0;

2. Confirm a batch is actually in flight, via a per-CPU refcount, text_poke_array_refs:

smp_rmb();

if (!try_get_text_poke_array())
        return 0;

try_get_text_poke_array() is just an atomic increment-if-nonzero:

static __always_inline bool try_get_text_poke_array(void)
{
        atomic_t *refs = this_cpu_ptr(&text_poke_array_refs);
        return raw_atomic_inc_not_zero(refs);   /* 0 → fails: no batch active */
}

Before touching any site, smp_text_poke_batch_finish() arms the refcount of every CPU to 1 with atomic_set_release(), then smp_wmb()s, and only then writes the INT3 bytes. The smp_rmb() above is the mirror-image acquire. That release/acquire pairing is what guarantees: if the #BP of a CPU fires (meaning it must have fetched the INT3 the writer stored), that same CPU is also guaranteed to see a fully-populated, non-zero-refcount text_poke_array — never a half-written vector.

3. Find which site trapped, binary-searching text_poke_array.vec for this regs->ip - 1 (skipping straight to a direct compare when there is exactly one entry):

ip = (void *) regs->ip - INT3_INSN_SIZE;

if (unlikely(text_poke_array.nr_entries > 1)) {
        tpl = __inline_bsearch(ip, text_poke_array.vec, text_poke_array.nr_entries,
                              sizeof(struct smp_text_poke_loc),
                              patch_cmp);
        if (!tpl)
                goto out_put;
} else {
        tpl = text_poke_array.vec;
        if (text_poke_addr(tpl) != ip)
                goto out_put;
}

ip += tpl->len;

4. Emulate the new instruction by editing regs and returning — the return address of the #BP handler becomes the effect of the emulated instruction, so iret resumes execution as if the new bytes had actually executed:

switch (tpl->opcode) {
case INT3_INSN_OPCODE:
        goto out_put;             /* someone else's deliberate INT3 */

case RET_INSN_OPCODE:
        int3_emulate_ret(regs);
        break;

case CALL_INSN_OPCODE:
        int3_emulate_call(regs, (long)ip + tpl->disp);
        break;

case JMP32_INSN_OPCODE:
case JMP8_INSN_OPCODE:
        int3_emulate_jmp(regs, (long)ip + tpl->disp);
        break;

case 0x70 ... 0x7f: /* Jcc */
        int3_emulate_jcc(regs, tpl->opcode & 0xf, (long)ip, tpl->disp);
        break;

default:
        BUG();
}

ret = 1;
  • JMP rel8/rel32 (also a pending NOP, encoded as JMP with disp == 0, i.e. “jump to the next instruction”): int3_emulate_jmp() just overwrites regs->ip.
  • CALL/RET/Jcc: same idea (push a fake return address / pop one / conditionally add the displacement) — jump labels never generate these, but static calls and ftrace share this exact engine and do.
  • If the replacement opcode is itself 0xCC (some other text-poke client is intentionally installing a breakpoint, not passing through this emulator): the handler does not consume the trap — it falls through so that debugging infrastructure (kgdb, kprobes) gets its #BP.

Jump labels only ever exercise the JMP32/JMP8 case — the RET/CALL/Jcc arms exist because static calls and ftrace share this exact handler.

5. Release the refcount and report the trap as handled:

out_put:
        put_text_poke_array();
        return ret;

So no CPU ever executes a torn multi-byte instruction: control either sees the old bytes (before the sync in phase 1 completes everywhere), takes the INT3+emulation path (the entire window from phase 1 to phase 3), or sees the finished new bytes (after the sync in phase 3).

10.4 What “sync” means

The three-phase protocol from §10.3 names “IPI sync” as a step three times over, without ever saying what happens when that IPI lands. This section answers that — not what the patching CPU does (§10.3 already covered that), but what every other CPU is forced to do in response.

smp_text_poke_sync_each_cpu() is the function invoked at each of those three points, and its job is deceptively narrow: send an IPI to every CPU other than the one doing the patching, have each of them run sync_core(), and block until every last one has reported back. That blocking is not incidental — the whole structure of §10.3 depends on each phase being globally visible before the next one begins. If the patching CPU raced ahead and wrote the tail bytes of phase 2 while some other core was still mid-fetch on the phase-1 view of the site, the entire point of planting the INT3 trap first would be defeated.

What actually happens on the receiving end is where the CPU model from §1.1 finally pays for itself. A modern core does not execute an instruction the moment it sees its bytes: it fetches ahead of where it is currently retiring, decodes into microcodes, and may be holding several instructions of that pipeline in flight at once. A CPU that fetched the old bytes moments before the patch landed can still be sitting on a stale decode of them — and no ordinary memory write, however carefully sequenced, undoes that. Something has to reach into the pipeline itself and discard the stale work. That is exactly what a serializing instruction is architecturally defined to do: retire everything already in flight, drop any speculative or partially-decoded work, and guarantee that the very next fetch goes out fresh.

Two ways to get that guarantee are available, and which one runs depends on the CPU generation. X86_FEATURE_SERIALIZE, present on most CPUs manufactured since around 2020, provides a dedicated instruction whose only job is this flush — cheap and direct:

static __always_inline void serialize(void)
{
        /* Instruction opcode for SERIALIZE; supported in binutils >= 2.35. */
        asm volatile(".byte 0xf, 0x1, 0xe8" ::: "memory");
}

That comment is the real reason this is written as raw .byte values rather than a mnemonic the assembler recognizes: SERIALIZE was only added to binutils in 2.35, so a kernel built with an older assembler still needs to be able to emit the instruction’s three raw opcode bytes (0F 01 E8) by hand. The "memory" clobber tells the compiler this call is a full optimization barrier — it must not reorder ordinary memory accesses across it.

On older hardware, the kernel falls back to iret_to_self(). An interrupt frame is just the handful of words — return SS, RSP, RFLAGS, CS, and RIP — that the CPU itself pushes onto the stack whenever a real interrupt or exception fires, and which iret later pops to hand control back to whatever was running. Normally software never builds one by hand; the CPU builds it automatically at the moment of a real trap, the same way it did for the #BP frame the INT3 handler above edited before its own iret. iret_to_self() does the CPU’s half of that job itself: it pushes those same five words by hand, with no real interrupt behind them, sets the saved RIP field to the very next instruction after the iret, and then executes iret against that fabricated frame. As far as the CPU can tell, this is a completely ordinary return from an interrupt, so it does everything architecture requires of one — including the serializing flush this function exists to get. But because the fabricated return address is simply “keep going from here,” nothing about the actual control flow changes: execution lands right back where it would have anyway, one instruction later.

Here is the actual body, from arch/x86/include/asm/sync_core.h (the 64-bit variant — a 32-bit build takes a shorter path that skips the SS/RSP pushes, since a same-privilege 32-bit iret doesn’t pop them):

static __always_inline void iret_to_self(void)
{
        unsigned int tmp;

        asm volatile (
                "mov %%ss, %0\n\t"      /* SS is a segment register; it can't be   */
                                         /* pushed directly in this form, so copy   */
                                         /* it into a GPR first                     */
                "pushq %q0\n\t"          /* frame field 1 (bottom): return SS       */
                "pushq %%rsp\n\t"        /* frame field 2: return RSP — but this    */
                                         /* captures RSP *after* the SS push above  */
                                         /* already moved it down by 8              */
                "addq $8, (%%rsp)\n\t"   /* ...so correct the just-pushed copy back */
                                         /* up by 8, to the RSP value from before   */
                                         /* this function started pushing anything  */
                "pushfq\n\t"             /* frame field 3: return RFLAGS            */
                "mov %%cs, %0\n\t"       /* same GPR trick as SS, for CS this time  */
                "pushq %q0\n\t"          /* frame field 4: return CS                */
                "pushq $1f\n\t"          /* frame field 5 (top): return RIP — the   */
                                         /* address of local label "1:" below, i.e. */
                                         /* the instruction right after this one    */
                "iretq\n\t"              /* pop all five fields and "return" — the  */
                                         /* CPU treats this exactly like returning  */
                                         /* from a genuine interrupt                */
                "1:"                     /* execution resumes here, indistinguishable */
                                         /* from simply falling through to this point */
                : "=&r" (tmp), ASM_CALL_CONSTRAINT : : "cc", "memory");
}

The iret is architecturally required to be a serializing event on every CPU, which is precisely the guarantee this fallback needs — it works identically at any privilege level (so it survives under paravirtualization) and never exits to a hypervisor, both properties this code cannot give up. The price is that it measures a bit more than twice as slow as the dedicated instruction, and it carries one side effect worth knowing about even though it doesn’t matter for this call site: it unconditionally unmasks NMIs, something the fast-path SERIALIZE instruction simply does not do.

One more candidate is conspicuously missing. CPUID also serializes, and on paper looks like the most portable option of all. The kernel avoids it here for a practical reason, not a correctness one: under virtualization, CPUID commonly traps out to the hypervisor, and this is exactly the kind of hot, latency-sensitive path — run on every online CPU, on every single key toggle — that cannot tolerate an unpredictable VM exit in the middle of it.

Put together, the choice between the two real options is not a fallback chain padded with special cases — it is a single feature check, decided once per call:

static __always_inline void sync_core(void)
{
        if (static_cpu_has(X86_FEATURE_SERIALIZE)) {
                serialize();
                return;
        }
        iret_to_self();
}

10.5 Writing through RO mappings

Every store §10.3 walked through — the INT3, the tail bytes, the final first byte — is not a raw write to .text. Each one is a full call to text_poke(), meaning each one pays the entire §1.4 dance in full: build the temporary alias in text_poke_mm, switch %cr3 onto it, copy through STAC/CLAC, switch back, tear the alias down. Nothing about these writes being unusually small — as small as the single INT3 byte in phase 1 — or unusually frequent lets any of them skip a step.

All of that happens under text_mutex, and this is the piece §1.4 leaned on without yet saying where it comes from: text_mutex is what stops two unrelated patchers — jump labels, static calls, ftrace, kprobes, the alternatives machinery — from ever building two competing temporary mappings onto the same text_poke_mm address space at once. Jump labels layer a second lock, jump_label_mutex, on top of that, but the two are not protecting the same thing: text_mutex serializes individual pokes at the hardware level, while jump_label_mutex serializes the higher-level operation of enabling or disabling one whole key — the enabled counter update and the walk over the jump_entry run for that key, not just the bytes it eventually writes.

The two are also held for deliberately different spans. On the queueing path (§10.2), text_mutex is acquired and released once per call to arch_jump_label_transform_queue() — bracketing only the __jump_label_patch() computation for that one site and its single smp_text_poke_batch_add() append. It is free again in between sites, so some unrelated text_poke() caller elsewhere in the kernel is free to interleave its own single-site poke while jump labels are still accumulating theirs for this key. Only once the whole batch is ready does arch_jump_label_transform_apply() take text_mutex back and hold it continuously across the entire three-phase smp_text_poke_batch_finish() from §10.3 — that phase genuinely cannot tolerate a second patcher walking in mid-batch, since the correctness argument of the INT3 protocol assumes the batch array it iterates is exactly the one it built. jump_label_mutex, by contrast, stays held across all of that from the first line of jump_label_update() onward — there is no benefit to releasing it early, since doing so would only let a second, unrelated static_branch_enable() call start interleaving its own bookkeeping with that of this one, not let this one finish any faster.

10.6 End-to-end timeline for one enable

§10.1 through §10.5 examined the machinery one piece at a time — which patch path early boot takes, how sites get batched, the three INT3 phases, what “synchronize” actually does on the wire, and how each of those writes reaches memory that is nominally read-only. Laid end to end, a single static_branch_enable() call looks like this:

static_branch_enable(&key)
  cpus_read_lock()
  jump_label_mutex
  enabled = -1
  jump_label_update(key)
    for each jump_entry of key:
      __jump_label_patch()           # compute nop↔jmp bytes, sanity memcmp
      smp_text_poke_batch_add()      # append to vector (may flush if full)
    smp_text_poke_batch_finish()
      arm text_poke_array_refs = 1 on every CPU (smp_wmb)
      text_poke INT3 × N
      IPI sync                          # step 1
      text_poke tails (bytes 1..N-1) × N
      IPI sync                          # step 2 ("paranoid")
      text_poke first bytes × N
      IPI sync                          # step 3
      drain: wait for text_poke_array_refs == 0 on every CPU
      text_poke_array.nr_entries = 0    # buffer free for next batch
  enabled = 1 (release)
  unlock…

After this, every previously-nop site for that key is a jmp (or vice versa), and the hot path behavior has flipped — without any flag load.

Notice what does, and does not, scale with the number of call sites. A key can have one jump_entry or a few hundred, but the number of IPI rounds is fixed at three,4 because smp_text_poke_batch_finish() runs its three phases once over the entire batched vector, not once per site (§10.3). The only thing that grows that fixed cost is the §10.2 overflow case: past 256 queued sites, a second full batch_finish() round is required, so a tracepoint with, say, 300 call sites pays six IPI rounds total, not 300 x 3.

11 Modules: the trickiest part

Static keys often live in vmlinux (or module A) while call sites live in module B — a tracepoint defined in the core kernel, say, with trace_*() call sites scattered across several drivers loaded as modules. The picture from §7.1 of key->entries as one pointer into one contiguous run of jump_entry records only holds while every call site for a key lives in a single object.

Modules break that assumption in the least convenient way possible: they load and unload independently of vmlinux and of each other, in an order nothing can predict ahead of time, so the representation has to be able to grow and shrink at runtime instead of being settled once at boot the way §9.1 settles it for the vmlinux-only case. That is what makes this section “the trickiest part” — every step has to stay correct no matter how many objects currently contribute to a key, or in what order they arrived. Over its lifetime, that one word/pointer union from §7.1 can end up in exactly three states:

/*
               next                  next                  next
 key (LINKED) -----> static_key_mod -----> static_key_mod -----> NULL
                     (mod A / vmlinux)     (mod B)
                         |                    |
                      entries              entries
                         |                    |
                         v                    v
                    [jump_entry…]        [jump_entry…]
*/
struct static_key_mod {
        struct static_key_mod *next;
        struct jump_entry *entries;
        struct module *mod;
};

Concretely: say the static_key of a tracepoint is declared in vmlinux, but nothing in the core kernel itself calls it — only the driver module A does, loaded first. Module A becomes the sole contributor for the key, so key->entries points straight at the own run of module A, no list involved (the one-home-only case above, even though the struct of the key lives in vmlinux while its only call site lives in a module — those are two independent facts, and only the second one matters here). Module B now loads and also calls the same tracepoint. A second object just started contributing, so the key flips into linked mode: one static_key_mod node is built to wrap what key->entries already pointed at (the run of module A), a second node is built for module B, and key->next now heads that two-node list — exactly the diagram above.

Exactly two functions drive every transition between those three states: jump_label_add_module() when a module loads, and jump_label_del_module() when one unloads. Neither is called directly — both run from a module notifier5 (jump_label_module_notify(), registered with .priority = 1) that hooks MODULE_STATE_COMING/ MODULE_STATE_GOING — the two transitions the module loader fires while a module is being mapped in and while it is being torn down.

The priority value is not an arbitrary tie-breaker — it enforces a strict order. Notifier chains run higher-priority callbacks first, and while jump labels register at .priority = 1, tracepoints register their own, separate module notifier at .priority = 0 (lower) — so the jump-label notifier always runs first. That ordering matters because tracepoints are themselves built on static keys (§4.10). When a new module loads (MODULE_STATE_COMING), its notifier wants to start touching the static keys behind its own tracepoints — but those keys are only ready to be touched once jump_label_add_module() has registered (and patched where needed), the jump entries of that module. Running the jump-label notifier first guarantees that the setup is already done by the time the tracepoint notifier runs.

11.1 Loading: jump_label_add_module()

This is the function that actually produces every transition described above, once per key contributed by the newly-loaded module. Trimmed of its early-return-if-empty check, the __init-flag bookkeeping §9.1 already covered, and its -ENOMEM error paths, here it is, one piece at a time.

The signature, its local state, and the first thing it does:

static int jump_label_add_module(struct module *mod)
{
        struct jump_entry *iter_start = mod->jump_entries;
        struct jump_entry *iter_stop = iter_start + mod->num_jump_entries;
        struct jump_entry *iter;
        struct static_key *key = NULL;
        struct static_key_mod *jlm, *jlm2;

        jump_label_sort_entries(iter_start, iter_stop);

jump_label_sort_entries() sorts the jump table of this module — the same routine §9.1 used on the vmlinux-wide table, for the same reason. After sorting, all entries that reference the same static_key sit next to each other in the array.

That adjacency matters because the work this function does — allocating static_key_mod nodes, wiring them into the list, deciding whether to patch — is per-key, not per-call-site. A module that calls trace_sched_switch() at ten different places still needs just one list node for that tracepoint key, not ten. With entries sorted by key, the loop can handle each distinct key exactly once: run the per-key body when the first entry for a key appears, then skip all subsequent entries for the same key with a single pointer comparison.

That skip pattern is what the top of the loop implements:

        for (iter = iter_start; iter < iter_stop; iter++) {
                struct static_key *iterk = jump_entry_key(iter);

                if (iterk == key)
                        continue;
                key = iterk;

The for loop advances iter through every entry in the table, one at a time. On each iteration, jump_entry_key() reads which static_key that entry belongs to. If it is the same key the loop just finished processing, continue skips the entry — all per-key work was already done when the first entry of that group was reached. Only when iterk differs from key does execution fall through into the per-key body below.

Two consequences follow from that design. First, the per-key body runs once per distinct key in this module, not once per call site, and each of the three states from the introduction of §11 corresponds to exactly one path through it. Second, iter at the point where the body runs always points at the first entry of a key group in the sorted table — the start of a contiguous run. That matters at the end of the function, where iter is passed as a range start to __jump_label_update(), which walks forward from there to patch every call site for that key in this module.

Once inside the per-key body, the first check asks whether this module even needs to consider the linked-list machinery at all — the one-home-only case:

                if (within_module((unsigned long)key, mod)) {
                        static_key_set_entries(key, iter);   /* one home only */
                        continue;
                }

within_module() asks whether the static_key struct itself — not a call site, not a jump entry, but the struct that holds enabled and entries — lives inside memory owned by mod.

If it does, then mod is, by construction, the very first and only object that has ever contributed call sites for this key. The reason is physical: before this module loaded, the memory backing that static_key was not even mapped. No other module or vmlinux could have built a jump entry referencing an address that did not yet exist. The one-home-only direct-pointer form is therefore not just adequate here, it is the only form this key has ever needed — which is why this branch continues past all the linked-list machinery that follows.

A key that fails that check has call sites outside this module, so before building any list node it is worth asking whether one is even possible — the sealed case:

                if (static_key_sealed(key))
                        goto do_poke;   /* sealed: patch once, keep no link */

A sealed key has already forgotten its entries/next union for good (§9.1), so there is nothing left to link this module into. Instead the goto skips straight past all the list-building below, to a single comparison shared with the ordinary case — reached down in the text.

An ordinary, unsealed key that reaches this point is the linked case. This is where the list from the introduction of §11 actually gets built or grown, in two steps.

The first step handles a one-time transition. Up to this point the key might still be in direct-pointer form — one pointer, one contributing object, no list. Before anything can be prepended to it, that existing pointer has to be wrapped in a list node so there is a next field to link through:

                jlm = kzalloc(sizeof(struct static_key_mod), GFP_KERNEL);
                if (!static_key_linked(key)) {
                        jlm2 = kzalloc(sizeof(struct static_key_mod), GFP_KERNEL);
                        scoped_guard(rcu)
                                jlm2->mod = __module_address((unsigned long)key);
                        jlm2->entries = static_key_entries(key);
                        static_key_set_mod(key, jlm2);
                        static_key_set_linked(key);
                }

static_key_linked() returns false only when the key is still in direct-pointer form, meaning exactly one object has contributed to it so far. The code allocates jlm2 to retroactively wrap whatever key->entries already pointed at — the run belonging to that first object — discovers which module owns that object via __module_address(), and installs jlm2 as the head of a new one-node list. Once static_key_set_linked() flips the linked bit, this wrapping step never runs again for this key — every later module finds static_key_linked() already true and skips straight into the second step.

The second step prepends a node for the newly-arriving module onto the now-guaranteed-to-exist list:

                jlm->mod = mod;
                jlm->entries = iter;
                jlm->next = static_key_mod(key);
                static_key_set_mod(key, jlm);
                static_key_set_linked(key);

jlm->entries is set to iter — the first entry of this key group in the sorted table, the same pointer the one-home-only case would have stored directly into key->entries. static_key_set_mod() makes jlm the new head of the list, with the previous head linked behind it via jlm->next.

Both the sealed-key goto and the ordinary path above fall into the same final check, once per key:

do_poke:
                if (jump_label_type(iter) != jump_label_init_type(iter))
                        __jump_label_update(key, iter, iter_stop, true);
        }
        return 0;
}

Every call site a module ships with is compiled around one fixed assumption: the default value the key had at compile time — the same type ^ branch formula from §5, exposed here as jump_label_init_type() — which decided whether the assembler macro emitted this site as a nop or a jmp in the first place (§5.1), frozen into the module from then on. But the live jump label type value can have moved away from that compiled-in default already, if something else called static_branch_enable()/disable() on this key before the module ever loaded. do_poke catches exactly that mismatch and patches the new sites immediately, before the code of the module has a chance to run and observe them in the wrong state — whether it arrived there via the sealed-key goto above, or by falling through normally after linking the key into the list.

11.2 Toggling an already-linked key

When someone calls static_branch_enable()/disable()/inc()/dec() on a key whose call sites span more than one object, the flat array scan from §9.5 is not enough — the sites are scattered across separate per-object tables, each with its own bounds, and the patcher has to visit every one of them.

jump_label_update() detects that case: if static_key_linked() returns true, it hands off to __jump_label_mod_update() instead of doing the walk itself. That function walks the linked list, calling __jump_label_update() once per node:

static void __jump_label_mod_update(struct static_key *key)
{
        struct static_key_mod *mod;

        for (mod = static_key_mod(key); mod; mod = mod->next) {
                struct jump_entry *stop;
                struct module *m;

                if (!mod->entries)
                        continue;

                m = mod->mod;
                if (!m)
                        stop = __stop___jump_table;
                else
                        stop = m->jump_entries + m->num_jump_entries;
                __jump_label_update(key, mod->entries, stop,
                                    m && m->state == MODULE_STATE_COMING);
        }
}

The loop visits each static_key_mod node in the list and passes that entries pointer and matching stop bound to __jump_label_update(). Three details in that loop deserve explanation.

First, stop cannot be a single kernel-wide constant the way it is in the flat walk from §9.5. Each module has its own private __jump_table section (§7.4), bounded by the jump_entries/num_jump_entries fields of that module. Without the correct per-node bound, __jump_label_update() would walk forward past the end of a module table into unrelated memory. The loop has to look up that bound for each node individually.

Second, m — the mod field of the list node, not the function parameter — can be NULL. That is not an error: it is the node that represents vmlinux itself. The NULL originates in §11.1: the first-time linking step calls __module_address() to discover which module owns the key, and __module_address() returns NULL for addresses inside the core kernel. For that node the correct bound is the global __stop___jump_table, which marks the end of the vmlinux-wide table.

Third, mod->entries can genuinely be NULL. That happens when an object defines a key but contains no call sites for it — the linking code from §11.1 still creates a list node to represent that object (it is a contributor), but there is nothing there to patch. Concretely: a module exports a DEFINE_STATIC_KEY_FALSE, and call sites in other modules are the only consumers. Skipping the node is correct.

The final argument, m && m->state == MODULE_STATE_COMING, tells __jump_label_update() whether to also patch entries living in the __init section of the module. A module still in MODULE_STATE_COMING has not finished running its init function yet, so its init section is still mapped and its call sites there are reachable. A module already in MODULE_STATE_LIVE has discarded that section — patching into freed memory would be a use-after-free, not a harmless no-op.

11.3 Unloading: jump_label_del_module()

jump_label_del_module() is the mirror image of jump_label_add_module(), run on MODULE_STATE_GOING: for each distinct key this module contributes to, it finds and removes the static_key_mod node that §11.1 created. The text-patching machinery does not need to undo anything here — the .text section of the module is about to be unmapped entirely, so the patchable sites in it simply cease to exist. What does need cleaning up is the linked-list bookkeeping that still references them.

The function uses the same sorted-table, skip-by-key loop structure as loading. Three of the four skip cases mirror §11.1 directly:

        for (iter = iter_start; iter < iter_stop; iter++) {
                if (jump_entry_key(iter) == key)
                        continue;

                key = jump_entry_key(iter);

                if (within_module((unsigned long)key, mod))
                        continue;

                /* No @jlm allocated because key was sealed at init. */
                if (static_key_sealed(key))
                        continue;

                /* No memory during module load */
                if (WARN_ON(!static_key_linked(key)))
                        continue;

The first three are familiar from §11.1: skip duplicate entries for the same key (the sorted-table grouping from the sort step), skip keys whose struct lives inside this module (the struct vanishes with the module, so there is no list to update), and skip sealed keys (no node was ever created for them). The fourth is a defensive check: if the key is not in the linked state at this point, something went wrong during loading — likely an allocation failure that jump_label_add_module() could not recover from. The WARN_ON flags the inconsistency without crashing6, and the continue skips the key rather than dereferencing a pointer that was never set up.

For every key that passes all four checks, the function walks the linked list to find and splice out the node belonging to this module:

                prev = &key->next;
                jlm = static_key_mod(key);

                while (jlm && jlm->mod != mod) {
                        prev = &jlm->next;
                        jlm = jlm->next;
                }

                /* No memory during module load */
                if (WARN_ON(!jlm))
                        continue;

                if (prev == &key->next)
                        static_key_set_mod(key, jlm->next);
                else
                        *prev = jlm->next;

                kfree(jlm);

The while loop advances through the list until it finds the node whose mod field matches the departing module, keeping prev pointed at the next pointer of the preceding node so the splice has something to patch. If no matching node is found — again a sign that something went wrong during loading — a second WARN_ON fires and the key is skipped. Otherwise, the standard singly-linked-list splice removes the node: if it was the head of the list (prev == &key->next), the next pointer of the key itself is updated via static_key_set_mod(); if it was somewhere in the middle, the next pointer of the predecessor is patched directly.

After the splice, one more step checks whether the list can be eliminated entirely:

                jlm = static_key_mod(key);
                /* if only one etry is left, fold it back into the static_key */
                if (jlm->next == NULL) {
                        static_key_set_entries(key, jlm->entries);
                        static_key_clear_linked(key);
                        kfree(jlm);
                }

If exactly one node remains, the key no longer needs the list form — only one object still contributes to it. The code folds that last node’s entries pointer back into key->entries directly, clears the LINKED bit, and frees the node. This is the linked-state transition from §11.1 running in reverse: a key that needed the list form only because two objects happened to overlap returns to the one-home-only direct-pointer form the moment that overlap ends.

12 Fallback: CONFIG_JUMP_LABEL=n

CONFIG_JUMP_LABEL is optional. Most distro kernels end up with it on — arm64 selects it outright, and on x86 PREEMPT_DYNAMIC pulls it in — but a minimal config can legitimately leave it off. Everything from §5 onward assumed the option was enabled. This section covers what happens when it is not.

Without CONFIG_JUMP_LABEL, the struct shrinks to a bare counter. The entries/next/type union from §7 is compiled out entirely:

struct static_key {
        atomic_t enabled;
};

jump_label_init() sets static_key_initialized to true and returns. There is no jump table to sort, no entries to pre-patch.

The call-site macros turn into ordinary branch-hinted conditionals. static_branch_likely() and static_branch_unlikely() reduce to:

#define static_branch_likely(x)   likely_notrace(static_key_enabled(&(x)->key))
#define static_branch_unlikely(x) unlikely_notrace(static_key_enabled(&(x)->key))

No asm goto, no jump table, no patching — just a likely_notrace()/unlikely_notrace() hint around a read of enabled.

static_key_count() is a plain raw_atomic_read():

static __always_inline int static_key_count(struct static_key *key)
{
        return raw_atomic_read(&key->enabled);
}

Compare this with the CONFIG_JUMP_LABEL=y version in kernel/jump_label.c, which clamps negative values (n >= 0 ? n : 1). That clamp exists because static_key_enable() temporarily sets enabled to -1 while the patching pass runs (see §9.2). Without patching, enabled never goes negative, so the clamp is unnecessary.

static_key_enable() and static_key_disable() are simpler for the same reason. Each one checks whether enabled already holds the target value and returns early if so. If it holds something unexpected (neither 0 nor 1), a WARN_ON_ONCE fires. Otherwise, a plain atomic_set() writes the new value. No cmpxchg, no intermediate -1, no jump_label_update() call.

jump_label_lock()/jump_label_unlock() are empty stubs — there is no patch pass to serialize.

The net effect on every hot path is exactly the cost §2 opened with: a memory load of enabled, a compare, and a conditional branch. The likely()/unlikely() hint steers the branch predictor the same way a compiled-in NOP or JMP would, but it cannot eliminate the branch itself — that is the optimization that CONFIG_JUMP_LABEL=y adds.

Jump labels are an optimization, not a correctness feature: behavior matches; only the mechanism changes.

13 Worked micro-example (bytes on the wire)

Every mechanism described so far — the asm helpers, the jump-table entry, the objtool hack, the size-discovery check, the enabled state machine, and the INT3 protocol — touches one concrete call site at some point in its life. This section traces a single, minimal site through that entire life, from compile time through one enable and one disable, close enough to see the actual instruction bytes change rather than just the names of the steps that change them. Suppose:

DEFINE_STATIC_KEY_FALSE(k);

void f(void)
{
        if (static_branch_unlikely(&k))
                printk("on\n");
        something();
}

Two small assumptions turn this from a symbolic description into an actual trace, and neither changes anything about how the mechanism works, only which specific numbers show up:

  1. f() is called after boot, on a live, multi-CPU system — the interesting §10 INT3 path, not the single-CPU §10.1 boot shortcut (a boot-time toggle of this same key would use text_poke_early() instead, with no INT3 involved at all).
  2. The compiler places l_yes — the out-of-line block containing the printk() call — 80 bytes past the end of the patch site. 80 fits in a signed byte, so the build-time trick from §6.3 picks the 2-byte JMP rel8 encoding rather than the 5-byte rel32 form. A farther l_yes would just mean 5 bytes instead of 2 everywhere below (§1.2); nothing else about the trace would change.

From the source code above, the compiler, linker, and objtool produce the bytes that sit in vmlinux at the patch site. For orientation, here is where the three pieces end up — the patch site in .text, the out-of-line target, and the sidecar entry in __jump_table:

 .text (function f)                  __jump_table (one entry)
 ──────────────────────────         ──────────────────────────
       ...                           code:   delta to 1:
  1:   [  2 bytes  ]  patch site     target: delta to l_yes
       ...                           key:    delta to &k.key + 2
       call something                         bit 0 = 0 (branch)
       ret                                    bit 1 = 1 (objtool)
       ...
  l_yes:     80 bytes past 1:+2
       call printk
       jmp back ------> (after 1:+2)

The six steps below trace how those bytes arrive at their final state:

  1. k is a FALSE key read with static_branch_unlikely(). Per the table in §5.1, that combination calls the nop-default helper arch_static_branch(&k.key, false), not the jmp-default arch_static_branch_jump(). That is the type ^ branch formula from §5 at work: type = 0 (FALSE), branch = 0 (unlikely), so type ^ branch = 0 — the hint agrees with the default, and the nop-default path gets chosen.

  2. The asm goto inside arch_static_branch() emits 1: jmp l_yes at the patch site, plus one raw row in __jump_table via JUMP_TABLE_ENTRY() (§6.2): code is the self-relative distance to 1:, target is the self-relative distance to l_yes, and key is the self-relative distance to &k.key + 0 + 2.

    That + 2 puts a 1 in bit 1 of the stored key address. Bit 0 is branch — here 0, matching the unlikely() hint. Bit 1 is the build-time signal to objtool: “NOP this jmp” (§6.3); after boot, jump_entry_set_init() repurposes this same bit as the __init-text flag. At runtime, jump_entry_key() masks both bits off to recover the real address of k.

  3. The assembler picks the encoding by the real distance to l_yes — here, 80 bytes forward (assumption 2 above), well inside the -128..+127 reach of a signed byte. It emits the 2-byte JMP rel8 form: opcode EB, followed by disp = dest - (addr + insn_size) = 80 = 0x50 (the disp formula from §1.2). The two bytes actually sitting at 1: right after assembly, before objtool ever runs, are EB 50.

  4. During the build, objtool calls handle_jump_alt() (§6.3 step 3), which sees bit 1 set in the stored key operand and rewrites those exact two bytes, in place, from the jmp (EB 50) to the 2-byte NOP (66 90, the NOP encoding from §1.2) — same size, so nothing around the site shifts. By the time vmlinux is linked, the live bytes at 1: are already 66 90, and objtool itself is long gone.

  5. At boot, jump_label_init() sorts __jump_table by key, then loops over every entry (§9.1). For the one entry belonging to k, the loop body does the following:

     jump_label_init()
 ├── jump_label_sort_entries()                sort __jump_table by key
 └── for each entry:                          (k has exactly one)
     ├── jump_label_type() = NOP              enabled(0) ^ branch(0)
     │   └── arch_jump_label_transform_static()  no-op on x86
     ├── jump_entry_set_init(entry, false)    code not in __init
     └── static_key_set_entries(&k, entry)    wires k.entries → entry

    jump_label_type() computes enabled(0) ^ branch(0) = NOP — the expected state matches the live bytes, which are already 66 90. So arch_jump_label_transform_static() is a genuine no-op: x86 never overrides the generic fallback, whose entire body is one comment, /* nothing to do on most architectures */. No instruction bytes get rewritten at this site.

  6. The same loop iteration does two pieces of bookkeeping that wire k into the runtime data structures. First, jump_entry_set_init() checks whether the code at 1: lives in __init text — it does not, so bit 1 of the stored key address (the same bit objtool used in step 2) gets cleared to 0. Second, static_key_set_entries() points k.entries at this entry — the pointer that jump_label_update() will follow when static_branch_enable(&k) runs later. With that in place, the hot path in f() from the very first time it runs is: decode 66 90 (falls through, no load of k, §1.1), then call something.

In summary, the same two bytes at 1: passed through three stages before the kernel ever ran a line of f():

 Stage                Bytes at 1:    Why
 ─────────────────    ───────────    ──────────────────────────────────
 After assembly       EB 50 (JMP)   assembler picks rel8 for +80 distance
 After objtool        66 90 (NOP)   handle_jump_alt() sees bit 1 in key
 At boot              66 90 (NOP)   jump_label_init(): NOP expected, NOP found

13.2 Runtime static_branch_enable(&k)

§10.6 already lays out the full call chain a batch of sites goes through on enable; k has exactly one entry, so this is that same chain with the actual bytes for every step filled in. Both directions of the public API are one-line macros in include/linux/jump_label.h:

#define static_branch_enable(x) static_key_enable(&(x)->key)
#define static_branch_disable(x)    static_key_disable(&(x)->key)

The full call chain for this one enable, with the concrete byte values for k filled in at each level:

 static_branch_enable(&k)
 └─ static_key_enable(&k.key)
    ├── cpus_read_lock / jump_label_lock
    ├── enabled: 0 --> -1              callers already see "on"
    ├── jump_label_update(key)
    │   └─ __jump_label_update()
    │      ├── type = enabled(true) ^ branch(0) = JMP
    │      ├── arch_jump_label_transform_queue()
    │      │   └─ __jump_label_patch(entry, JMP)
    │      │      ├── size  = 2        (live-decode of 66 90)
    │      │      ├── code  = EB 50    (text_gen_insn)
    │      │      ├── nop   = 66 90    (x86_nops[2])
    │      │      ├── memcmp(addr, nop) -- pre-flight OK
    │      │      └── smp_text_poke_batch_add(addr, EB 50, 2)
    │      └── arch_jump_label_transform_apply()
    │          └─ smp_text_poke_batch_finish()
    │             └── INT3 three-phase: 66 90 --> EB 50
    ├── enabled: -1 --> 1              atomic_set_release
    └── jump_label_unlock / cpus_read_unlock

The numbered steps below walk through this chain in detail:

  1. static_branch_enable(&k) expands to static_key_enable(&k.key), which acquires jump_label_lock() and walks the 0 → -1 → (patch) → 1 state machine from §9.2. enabled starts at 0, gets set to -1 (“enabling in progress”), and static_key_count() already reports -1 as “on” — so no caller sees a false “off” window while patching runs. Then jump_label_update(&k.key) does the actual patching, and only after it returns does enabled get published as 1 with release ordering.

  2. jump_label_update() finds the one entry of k and computes jump_label_type(entry) = enabled ^ branch. enabled is the transient -1 from step 1, which static_key_enabled() reports as true; branch is the stored hint bit, 0. true ^ false = JMP — the live nop becomes a jmp.

  3. arch_jump_label_transform_queue() calls __jump_label_patch(), which re-derives the size by decoding the live bytes at 1: — arch_jump_entry_size() returns 2, matching what objtool left behind (§8). The function then builds both sequences: nop = 66 90 (from x86_nops[2]) and code = EB 50 (from text_gen_insn() — the same bytes computed in §13.1 step 3, since addr and dest have not moved). Because this is a nop→jmp transition, it calls memcmp() to verify the live bytes are currently 66 90; a mismatch would be a BUG(). They match, so the patch 66 90 → EB 50 gets queued via smp_text_poke_batch_add() (§10.2).

  4. The loop of jump_label_update() over the entries of k ends here — there is only the one — so arch_jump_label_transform_apply() immediately calls smp_text_poke_batch_finish(), which runs the three-phase protocol from §10.3 on this one queued site, now with the real two bytes instead of a placeholder:

    start (before)     66 90              any fetch: executes the 2-byte NOP

phase 1 (INT3 in)  cc 90              any fetch: #BP -> handler emulates
                   ^^                 the pending JMP (jumps to l_yes)
                   trap byte

                   -------- IPI sync --------

phase 2 (tail in)  cc 50              same as phase 1: byte 0 still
                   ^^                 traps and gets emulated — byte 1
                   still traps        is now its final value underneath

                   -------- IPI sync --------

phase 3 (done)     eb 50              any fetch: executes the real
                   ^^ real opcode     JMP rel8 directly, no trap needed

                   -------- IPI sync --------

  5. From the instant the sync in phase 1 completes, any CPU landing on this address already gets the effect of the jump via emulation (§10.3); phases 2-3 only retire the trap-and-emulate path in favor of the real bytes. Once phase 3 lands, every subsequent call to f() decodes EB 50, jumps 80 bytes forward into l_yes, runs printk("on\n"), then hits the compiler-emitted jmp back (§3) and falls into something().

13.3 static_branch_disable(&k): same protocol, asymmetric math

Disabling runs the mirror call, static_branch_disable(&k) → static_key_disable(&k.key) — a single atomic_cmpxchg(&key->enabled, 1, 0) instead of the enable state machine, because disable has no in-progress state to protect (§9.3: a reader who sees stale “on” for a few more instructions is exactly the harmless case, unlike stale “off”). If that cmpxchg succeeds, jump_label_update() runs again, this time computing jump_label_type() = false ^ false = NOP (enabled now 0, branch still 0).

__jump_label_patch() rebuilds the same two sequences as before — nop = 66 90, code = EB 50, both unchanged, since addr, dest, and size haven’t moved — but this time expects the live bytes to be the jmp and installs the nop. That is the one genuine asymmetry in this whole worked example: enabling always needs a fresh, target-specific displacement computed by text_gen_insn(); disabling never does, because the encoding of a NOP does not depend on where the branch would have gone — the exact same fixed bytes go back every time a site of this size is disabled, no matter what it was jumping to.

The same three-phase protocol (§10.3) runs again, in the opposite byte direction:

 start (before)     eb 50              executes the JMP rel8

 phase 1 (INT3 in)  cc 50              #BP -> handler now emulates the
                    ^^                 pending NOP (a JMP with disp == 0
                    trap byte          — not a special case)

                    -------- IPI sync --------

 phase 2 (tail in)  cc 90              byte 1 now its final value; byte 0
                    ^^                 still traps
                    still traps

                    -------- IPI sync --------

 phase 3 (done)     66 90              executes the real NOP directly
                    ^^ real opcode

                    -------- IPI sync --------

Put together, the build-time 66 90 of this one site, the EB 50 of the first enable, and the 66 90 of this disable again are the entire lifecycle that §1-§10 spent this whole tutorial describing in the abstract — the same two bytes, chosen and re-derived by a different mechanism at each stage, but never touched by anything other than the three sanctioned writers: the assembler once at compile time, objtool once at build time, and __jump_label_patch() through the protocol from §10.3 every time after that.

Stage Live bytes at 1: Who wrote them
After assembly, before objtool EB 50 Compiler/assembler (§6.1, §1.2)
After objtool, at boot 66 90 handle_jump_alt() (§6.3)
After static_branch_enable(&k) EB 50 __jump_label_patch() via INT3 protocol (§10.3)
After static_branch_disable(&k) 66 90 Same, reverse direction

14 Further reading in-tree

  1. A 4-byte signed field covers ±2 GiB, and x86_64 kernels are built with -mcmodel=kernel, which packs the entire kernel image into the top 2 GiB of the address space specifically so that a 32-bit relative displacement is always enough — no two locations inside vmlinux can ever be farther apart than a rel32 can reach. ↩

  2. __ro_after_init is a section attribute (include/linux/cache.h:60, __section(".data..ro_after_init")) for data that is written during boot but never again afterward — unlike const, which the compiler must be able to enforce at compile time, this is a promise the author makes about runtime behavior. The kernel makes that promise real at mark_rodata_ro() time, when the whole .data..ro_after_init section is remapped read-only in the page tables, so any later write attempt — a bug, or an author breaking their own promise — faults instead of silently corrupting state. ↩

  3. _stext is a linker-defined symbol, not a C variable — it marks the address where the .text section of the kernel begins, set directly in arch/x86/kernel/vmlinux.lds.S. Every function in core kernel text lives at some fixed offset from it, which is what lets smp_text_poke_loc.rel_addr (a plain s32) address any patch site with 4 bytes instead of the full 8-byte pointer jump_entry.key needs (§6.2) for its potentially-far-away static_key. ↩

  4. Strictly, phases 2 and 3 each sync only if at least one site in the batch actually needed a write in that phase, tracked by a do_sync counter inside smp_text_poke_batch_finish(). The phase 3 sync, for instance, is skipped if the final first byte of every site already happens to equal INT3. An ordinary nop-to-jmp toggle always writes something in both phases, so three rounds is what actually happens in practice; the fixed count just isn’t unconditional at the code level. ↩

  5. A module notifier is a callback registered against the kernel’s module-load notifier chain (module_notify_list) via register_module_notifier(). The module loader walks that chain with blocking_notifier_call_chain() at each state transition (MODULE_STATE_COMING, MODULE_STATE_GOING, etc.), invoking every registered struct notifier_block in priority order — this is the generic mechanism subsystems use to react to modules loading/unloading, not something jump labels invented. ↩

  6. Unless panic_on_warn = 1 ↩

comments powered by Disqus