What eBPF Actually Is

Everyone says eBPF is 'programmable kernel' and 'the future of infrastructure,' but few engineers can explain what it actually is. Let's fix that: eBPF lets you run sandboxed programs inside the Linux kernel without changing kernel source or loading modules.

The Concept Explained

A pod in your cluster is opening connections to an address nobody recognizes, and you need to know which process, in which container, on which node. Every tool you reach for is wrong in the same way. tcpdump shows you packets with no PID attached, ss shows you sockets with no history, and auditing every syscall answers the question at a cost you will not pay on a node running two hundred pods. What you want is ten lines of code sitting exactly where the kernel establishes a TCP connection, reading the socket and the task that owns it, and handing user space one aggregated record every few seconds.

For most of Linux's life there were two ways to get those ten lines into the kernel, and both are absurd for the question you are asking. You could patch the kernel source, which means carrying an out-of-tree patch and rebuilding it on every kernel bump. Or you could write a loadable kernel module, which is the same ten lines with no seatbelt: a null dereference in module code panics the machine and takes every workload on it, and the module has to be compiled and re-tested against each kernel version in your fleet. Managed node pools generally will not load one at all.

The third option grew out of a 1992 idea. BPF, the Berkeley Packet Filter, existed because packet capture was expensive: the kernel copied every packet to user space so a process could decide it did not want most of them. BPF moved the decision to where the data already was — a deliberately feeble virtual machine, two 32-bit registers, no loops, no memory beyond a scratch buffer, interpreted in the kernel, returning keep or drop. It was too weak to be dangerous, and that was the point.

In 2014 that machine was rewritten into something general. Modern eBPF has eleven 64-bit registers that map cleanly onto real hardware registers, a JIT that turns bytecode into native instructions, maps that hold state between invocations and are readable from user space, a fixed set of helper functions that form its only kernel API, and a verifier that proves before load that a program terminates and touches only memory it is allowed to touch. The attach points moved far past sockets: packet paths, tracepoints, kernel and user function entry, syscalls, cgroups, security hooks. Packet filtering is now one use out of dozens, and the name is vestigial.

What changed is not that you can run code in the kernel — you always could. What changed is the cost of being wrong. A module bug panics a node; a bad eBPF program prints a rejection log and returns an error from a syscall. That single difference turns something you would never risk on five hundred production nodes into something a DaemonSet does during a normal rollout, which is why CNIs, runtime security, profilers, and service meshes all moved into the kernel at once.

How It Works

These are not three points on one axis. The real difference is who absorbs the risk of your mistake: with a module it is the node, with eBPF it is your deploy pipeline. eBPF pays for that by giving up generality — you cannot write a filesystem, a device driver, or anything that must block and wait. Those still belong in modules. eBPF takes the enormous middle ground of observe, decide, and transform, which is almost everything a platform team needs from the kernel.

Mechanically, an eBPF program is a function the kernel calls for you. It takes one argument, a context pointer whose type depends on where you attached it, and returns an integer whose meaning also depends on where you attached it. It has no thread and no main loop; it runs synchronously on the CPU that took the event, and the event waits until it returns. That is the entire cost model, and the next lesson takes it apart properly.

State lives in maps — typed key/value structures the kernel and user space share by file descriptor. Everything else the program needs it asks for through helper functions like bpf_map_lookup_elem, bpf_ktime_get_ns, or bpf_get_current_pid_tgid. That allow-list is the API boundary: a program cannot call an arbitrary kernel function, which is exactly why it does not break when that function is renamed.

struct {
    __uint(type, BPF_MAP_TYPE_LRU_HASH);
    __uint(max_entries, 65536);
    __type(key, struct conn_key);
    __type(value, __u64);
} conns SEC(".maps");

SEC("kprobe/tcp_connect")
int trace_connect(struct pt_regs *ctx)
{
    struct conn_key k = build_key(ctx);
    __u64 now = bpf_ktime_get_ns();
    bpf_map_update_elem(&conns, &k, &now, BPF_ANY);
    return 0;
}

That is a complete program: a few dozen instructions after compilation, attached to one kernel function, updating one map. Nothing about it looks like kernel code, and that is the shift.

The Sandbox Is Not a Runtime

The word "sandboxed" misleads people here. There is no supervisor watching an eBPF program while it runs, no memory protection around it, no timer that kills it if it takes too long. All of the safety is static, proven at load time, and once the JIT is done the program is native machine code executing in kernel context with kernel privileges. That is why it is fast, why the verifier is uncompromising, and why verifier bugs are serious CVEs rather than curiosities.

The privilege model follows from that. Loading historically required CAP_SYS_ADMIN; kernel 5.8 split out CAP_BPF, usually paired with CAP_NET_ADMIN for networking programs or CAP_PERFMON for tracing. Unprivileged loading is disabled by default on essentially every distribution kernel you will meet, and 5.16 added a permanent lockout value.

$ cat /proc/sys/kernel/unprivileged_bpf_disabled
2
$ bpftool feature probe kernel | grep -E 'bpf_probe_read_kernel|JIT'
JIT compiler is enabled
bpf_probe_read_kernel is available

Why It Survives a Kernel Upgrade

The other half of the module comparison used to be a fair fight, because early eBPF tooling compiled against the running kernel's headers on the target host, which meant shipping a compiler and headers to every node. BTF, the type information the kernel now exposes about its own structures, plus CO-RE relocations that the loader fixes up against the running kernel's layout, removed that entirely: one compiled object detects field offsets at load time and runs across a range of kernels. The lesson on tooling covers how libbpf does this; what matters here is the consequence: eBPF is the only way to extend the kernel that does not create a build artifact per kernel version in your fleet.

Practical Application

Start by finding out what is already loaded. On any node running a modern CNI, a runtime security agent, and a profiler, the answer is "more than you think," and those programs are stacked on shared hooks in a specific order.

$ bpftool prog show
247: sched_cls  name cil_from_container  tag 8f2b1a3c9d4e5f60
     loaded_at 2026-02-11T09:14:02+0000  uid 0
     xlated 4728B  jited 2896B  memlock 8192B  map_ids 33,41,58
312: kprobe     name trace_connect       tag 1c7d90ab44e3f2aa
     loaded_at 2026-02-11T09:14:07+0000  uid 0
     xlated 296B   jited 187B   memlock 4096B  map_ids 61

Read xlated and jited together. xlated is the post-verifier bytecode size, jited is the native code the CPU actually runs. A program showing no jited value is running in the interpreter, several times slower per event, which usually means net.core.bpf_jit_enable got set to zero by a hardening baseline.

Check the kernel before you promise a feature. eBPF is not one capability, it is dozens of independently version-gated ones. Bounded loops need 5.3, CAP_BPF needs 5.8, memory accounting moved to cgroups in 5.11, and most of Cilium's interesting datapath features want 5.10 or newer. bpftool feature probe answers per-feature; the kernel version alone does not, because distributions backport aggressively.

Budget for locked memory on older kernels. Before 5.11, map memory came out of RLIMIT_MEMLOCK, and the classic symptom is an agent that loads fine on a workstation and fails on a node with a stingy default limit. The error is Operation not permitted on map creation, which reads like a capability problem and is not.

Measure per-program cost instead of guessing at it. Enabling kernel.bpf_stats_enabled makes the kernel account time spent inside every loaded program.

$ sysctl -w kernel.bpf_stats_enabled=1
kernel.bpf_stats_enabled = 1
$ bpftool prog show id 247
247: sched_cls  name cil_from_container  tag 8f2b1a3c9d4e5f60
     run_time_ns 4128993201  run_cnt 61213847
     xlated 4728B  jited 2896B  memlock 8192B

That works out to roughly 67 nanoseconds per invocation, which is the only honest answer to "is the datapath expensive on this node."

Tradeoffs and Decision Framework

Four questions settle it. Does the data you need ever cross a kernel boundary, or does it exist only inside your application, where no kernel technology will ever see it? Does the event volume make a copy to user space untenable, which is exactly where in-kernel filtering earns its keep? Do you need to change behavior rather than observe it, since dropping or redirecting has to happen inline where the decision is made? And what is your oldest production kernel, because that number, not the newest, sets your feature floor.

Default to eBPF for anything that observes or steers kernel behavior at high event rates, and reach for a user-space agent when the event rate is low and the logic is complicated — code that calls an API, retries, and holds a lot of state does not belong in a verified 512-byte-stack environment even when it technically fits.

Common Mistakes

Treating "sandboxed" as runtime isolation. Nothing supervises a running eBPF program. The guarantee is a static proof at load time, and after JIT it is native code with kernel privileges.

Handing out CAP_BPF because it is not CAP_SYS_ADMIN. Combined with tracing capabilities it grants arbitrary kernel memory reads. It is a root-equivalent privilege for anything you care about keeping secret.

Emitting one event per call. The technology's entire advantage is deciding and aggregating in the kernel. Pushing every event to user space reintroduces the copy cost eBPF exists to remove, and you find out about it as ring buffer drops on your busiest node.

Assuming a kernel version implies a feature set. Distributions backport heavily and disable selectively. Probe for the specific capability rather than comparing version strings.

Expecting eBPF to replace application instrumentation. It sees syscalls, packets, and kernel functions. It does not see which customer the request belonged to or why your retry logic gave up.

Losing track of what is attached. Three agents that each install tc programs on the same interface interact in an order nobody chose deliberately. Inventory loaded programs the same way you inventory DaemonSets.

Reading eBPF as free. Every attached program adds work to a hot path. Measure it with per-program run time rather than assuming the overhead is noise.