The Verifier and Safety

Running arbitrary code in the kernel sounds terrifying, one bug could crash the whole machine. The eBPF verifier is what makes it safe, and understanding it explains both eBPF's power and its constraints.

The Concept Explained

You write forty lines that count packets per source address, run the loader, and get back a rejection log longer than the program, ending in a line about register 0 and an invalid memory access. Nothing about your code is wrong in the ordinary sense. It compiles, a reviewer would approve it, and the equivalent user-space version has been running for years. The kernel refuses it anyway.

Understand what the kernel is being asked to do and the refusal stops feeling arbitrary. Your program is about to be called from a softirq context with preemption disabled, on a CPU that will do nothing else until it returns, with a pointer to a live socket buffer. If it dereferences null, the machine panics and every pod on the node dies with it. If it loops forever, that CPU never comes back, the watchdog fires, and the node is gone in a different way. These are not hypothetical failure modes — they are the ordinary failure modes of kernel modules, and they are exactly what stopped anyone sane from shipping kernel code as part of a normal deploy.

The verifier's job is to make those outcomes impossible in advance rather than survivable afterward. It is not a linter and not a style checker. It is a static analyzer that simulates the program over every path it could possibly take, and it must come out the other side holding a proof of two properties: the program terminates, and every memory access it performs is inside a region it is entitled to touch. If it cannot construct that proof, the load fails. There is no override flag, no "I know what I am doing," no partial acceptance.

That design has a consequence people miss. Because everything is proven at load time, nothing is checked at run time. The JIT emits no bounds checks, no null checks, and no loop guards, which is why an eBPF program costs tens of nanoseconds and not microseconds. The verifier is not a tax on execution; it is the reason execution is free. And it is also why the rules are inflexible: a rule the verifier relaxes is a check nobody performs anywhere, ever.

The rejection you are staring at is the system working. Your program was fine on the path you had in mind, and the verifier found a path you did not — usually the one where a map lookup returned null, or where a packet was shorter than the header you read out of it.

How It Works

The gate is deliberately one-sided. The verifier is allowed to reject safe programs, and does so constantly; it is not allowed to accept unsafe ones. Every version of the analysis is an approximation that errs toward refusal, which means your real task is not writing correct code but writing code whose safety this particular analyzer, on this particular kernel, can follow.

Two rejections account for most of what you will hit. The first is the map lookup. bpf_map_lookup_elem returns a pointer that the verifier types as map value or null, and in that state it is unusable. Comparing it against null and branching converts it, on the branch where the comparison failed, into a plain map value pointer you may dereference. The check is not defensive programming — it is the operation that changes the type.

$ bpftool prog load count.o /sys/fs/bpf/count
libbpf: prog 'count_pkts': BPF program load failed: Permission denied
libbpf: prog 'count_pkts': -- BEGIN PROG LOAD LOG --
0: (b7) r1 = 0
1: (85) call bpf_map_lookup_elem#1
2: (79) r2 = *(u64 *)(r0 +0)
R0 invalid mem access 'map_value_or_null'
verification time 38 usec, processed 12 insns (limit 1000000)
-- END PROG LOAD LOG --

The second is packet access. A packet pointer is only valid up to data_end, and the verifier tracks that boundary as a range. Every read has to be preceded by a comparison that proves the bytes exist, and the comparison must dominate the read on every path.

__u64 *count = bpf_map_lookup_elem(&counts, &key);
if (!count)
    return XDP_PASS;
__sync_fetch_and_add(count, 1);

void *data = (void *)(long)ctx->data;
void *data_end = (void *)(long)ctx->data_end;
struct ethhdr *eth = data;
if ((void *)(eth + 1) > data_end)
    return XDP_DROP;

Why Your 200-Line Program Is "Too Large"

The instruction budget is not a limit on program size. It counts instructions processed during verification, summed across every path the analyzer explores, and paths multiply. Ten independent conditionals in sequence describe 1,024 distinct states, and each one gets walked with its own register ranges. State pruning cuts this down enormously by recognizing when a state is equivalent to one already proven safe, but pruning fails exactly where the state differs in ways that matter, which is where your bounds checks are.

processed 612847 insns (limit 1000000) max_states_per_insn 12
total_states 21883 peak_states 1024 mark_read 19

That line is the one to watch. A program at 600,000 processed instructions is not comfortable; it is one refactor away from failing to load, and the refactor that tips it over will look harmless. The counterintuitive fix is usually to add code rather than remove it — re-deriving a bounds check inside a branch, instead of relying on one performed earlier, narrows the state the verifier has to carry forward and often shrinks the total dramatically.

Practical Application

Read the log from the bottom. The last few lines are the answer: the failing instruction, the register that broke a rule, and the rule it broke. Everything above it is the path the verifier took to get there, useful only once you know which path you care about.

Raise the log level when the bottom is not enough. Level 2 prints register state at every instruction, which is how you find the branch where a range widened unexpectedly. The log buffer needs to be large enough or you get a truncated tail and no clue.

Confirm the program is actually JIT compiled. A missing jited figure means the interpreter, which is several times slower per event and usually the result of a hardening baseline rather than any decision your team made.

$ bpftool prog show id 247 | grep -E 'xlated|jited'
     xlated 4728B  jited 2896B  memlock 8192B  map_ids 33,41,58
$ sysctl net.core.bpf_jit_enable net.core.bpf_jit_harden
net.core.bpf_jit_enable = 1
net.core.bpf_jit_harden = 0

Treat processed-instruction counts as a budget you monitor. When you build datapath programs whose complexity scales with policy or service count, the verifier ceiling becomes a real capacity limit, and the failure arrives at the worst time: on the biggest cluster, during a rollout, as a program that will not load on nodes where the previous version did.

Tradeoffs and Decision Framework

The decision usually starts as "the verifier said no" and needs to become something more precise. Is the rejection about safety, where the fix is a check you genuinely forgot, or about complexity, where the code is safe and the analysis cannot afford to prove it? Does the work have a fixed small bound, which unrolling handles for free, or a data-dependent one, which wants bpf_loop? Does the program have distinct phases that could be separate programs joined by a tail call? And does this logic need to be in the kernel at all, or is it there because the rest of the agent is?

Default to keeping the loop bounded and small and letting the compiler unroll it, since that produces the fastest code and the simplest failure mode. Reach for bpf_loop when iteration count is genuinely dynamic and your kernel floor allows it, and reach for tail calls when a program has grown into several distinct stages rather than one hot path with a loop in it.

Common Mistakes

Dereferencing a map lookup without the null check. The comparison is what converts the pointer's type. Without it the value is unusable no matter how certain you are the key exists.

Assuming a bounds check survives a helper call. Any helper that can change the socket buffer invalidates packet pointers, so data and data_end must be reloaded and re-compared afterward or the verifier rejects every subsequent access.

Developing on a newer kernel than you run. Verifier capability is version-gated, so the rejection surfaces during a rollout on the oldest node pool rather than during development.

Reading "too large" as "too many lines." The limit counts processed instructions across all explored paths. Branch count drives it, not source length.

Skimming the verifier log. The answer is in the last three lines, and people who scroll past it spend hours guessing at a message that already told them the register and the rule.

Reading uninitialized stack. The 512 bytes of stack are untyped until written, and a read before a write is rejected, which is why zero-initializing key structs is not optional.

Believing verified means correct. The verifier proves memory safety and termination. It has no opinion about whether your program drops the right packets.

Disabling or hardening the JIT reflexively. Turning off bpf_jit_enable or raising bpf_jit_harden on nodes where only root loads programs buys nothing and costs measurable CPU on every event.