The eBPF Ecosystem and Tooling
You don't usually write raw eBPF. A whole ecosystem sits on top of it, bcc, bpftrace, libbpf, CO-RE. Knowing the landscape helps you understand what's possible and what tools to reach for.
The Concept Explained
A node is burning eighteen percent of its CPU in system time and nobody knows why. You suspect something is hammering a file under /etc, but you cannot name the process, you cannot restart anything, and there are two hundred containers on the box. strace attaches to one PID at a time and stops the target on every syscall, which is unusable at this scale. Fleet-wide audit rules would catch it and would also cost you on every syscall from every process, and the answer arrives in a log you still have to ship somewhere.
What you actually want is a filter that runs in the kernel, on the one syscall you care about, aggregates in kernel memory, and prints a histogram when you press Ctrl-C. That is exactly the shape of an eBPF program, and it is also the shape nobody wants to hand-assemble as bytecode with a map declaration and a loader.
So the ecosystem exists in three layers, and the layers answer different questions rather than being levels of abstraction over the same thing. The bottom layer is who loads the program: libbpf in C, cilium/ebpf in Go, Aya in Rust, and the older bcc Python bindings all do the same job of compiling, relocating, verifying, and attaching. The middle layer is who writes it: bpftrace turns a domain-specific one-liner into a program on the fly, and the bcc tool collection ships a few hundred prewritten scripts. The top layer is who owns it forever: Cilium, Tetragon, Falco, and Pixie load programs at boot and keep them attached for the life of the node.
The dividing line that actually predicts operational pain runs sideways through all three. It is whether the tool compiles on your production node or ships a portable object. bcc embeds a full LLVM and Clang, generates the program text at runtime, and compiles it against the kernel headers present on that host. CO-RE builds do the compiling once on a build machine and ship an ELF object that relocates itself at load time.
The question that decides whether an eBPF tool is deployable at fleet scale is not which language it is written in — it is whether it compiles on the target node. A bcc-style tool needs matching kernel headers, a compiler toolchain, seconds of CPU and hundreds of megabytes of RAM on every node at startup, and it breaks the day someone slims the node image. A CO-RE binary compiles once against a BTF description of the kernel, ships as a single static object, and relocates struct field offsets against /sys/kernel/btf/vmlinux at load time in milliseconds. That difference is why eBPF moved from a debug-box technique to something you run as a DaemonSet on ten thousand nodes.
How It Works
The eBPF Ecosystem, Layer By Layer
Long-lived agents that load and pin programs at node startup and keep them attached. You configure policy, not probes, and you inherit their upgrade and hook-collision behavior for the life of the node.
Ad hoc tracing you run for ninety seconds and stop. bpftrace compiles a one-liner into a program and a map at invocation; the bcc tools are a few hundred prewritten scripts for the questions people ask most.
Libraries that open the ELF object, apply relocations, create maps, call bpf() to load and verify, and attach to hooks. Everything above this line is a client of one of these, and the language is your choice rather than a kernel constraint.
BTF is the type description the kernel publishes about itself at /sys/kernel/btf/vmlinux. CO-RE relocations let one prebuilt object find the right struct field offsets on any kernel that has BTF, which is what removes the compiler from the node.
Everything above ultimately produces bytecode that the verifier proves safe and the JIT turns into native instructions. No layer above can grant a capability the kernel does not already expose as a hook or a helper.
Hover to expand each layer
Reading that stack top to bottom is useful, but the relationship is not a dependency chain. bpftrace links libbpf and can run entirely CO-RE, so it is a peer of your Go agent rather than a thing built on it. And every layer converges on the same kernel objects, which is why bpftool is the ground truth when two tools disagree: a program loaded by a Helm-installed platform and one loaded by your one-liner appear side by side in the same listing, with the same ids and the same map references.
CO-RE And BTF: Why Your Tool No Longer Needs A Compiler
An eBPF program that reads task->mm->arg_start needs the byte offset of those fields in the running kernel's structs. Those offsets change between kernel versions and between distro configs. bcc's answer was to compile on the target, which is correct and expensive. CO-RE's answer is to compile once against a BTF type description, emit a relocation record for every struct access, and let libbpf rewrite the offsets at load time using the BTF the running kernel exposes.
The prerequisite is CONFIG_DEBUG_INFO_BTF=y, standard on modern distro kernels and absent on a few older and custom ones. Check it before you plan a rollout:
$ ls -l /sys/kernel/btf/vmlinux
-r--r--r-- 1 root root 5304428 Aug 3 09:14 /sys/kernel/btf/vmlinux
$ bpftool btf list | head -3
1: name [vmlinux] size 5304428B
9: name [cilium] size 41232B prog_ids 187,188 map_ids 44,45If that file is missing, CO-RE tools fall back to shipped BTF from an external archive or refuse to load, and this is the single most common reason an eBPF agent works on your laptop and not on a hardened node image.
Ad Hoc Or Always On
Everything in the ecosystem is one of two things, and conflating them causes most tooling mistakes. An ad hoc tool answers a question and exits, so it can afford a kprobe on an obscure function and a high per-event cost. An always-on platform runs for months across a fleet, so it needs stable attach points, bounded overhead, and a story for kernel upgrades.
The interview question at the end of this lesson is the ad hoc case, and it is a one-liner:
$ bpftrace -e 'tracepoint:syscalls:sys_enter_openat
/str(args->filename) == "/etc/resolv.conf"/
{ @opens[comm, pid] = count(); }'
Attaching 1 probe...
^C
@opens[coredns, 41522]: 18344
@opens[python3, 8801]: 212The aggregation happens in a kernel map. Nothing is copied to user space per event, no process is stopped, nothing is restarted, and the filter runs before the kernel does anything expensive. That is the whole argument for eBPF over strace and over audit rules in one command.
A kprobe on an internal kernel function is not a stable interface. Attaching to a static function that a later kernel inlines or renames does not degrade gracefully — the load fails with -ENOENT, or worse, the tool starts and reports nothing while looking perfectly healthy. Prefer tracepoints and fentry/fexit, which carry stability commitments and cost less per event, and treat a security or observability agent that pins itself to kprobes on internal symbols as something you must re-validate on every kernel bump. Silence from a tracing agent is never proof that nothing happened.
Practical Application
Reach for bpftrace before anything heavier. It is on most distro repos, it needs no build step, and the answer to "which process, how often, how long" is usually one line. Run it from a debug pod with hostPID: true and CAP_BPF plus CAP_PERFMON (or privileged on pre-5.8 kernels) rather than by shelling into the host.
Establish whether each agent is CO-RE before it goes fleet-wide. This is a procurement question, not a preference. Ask the vendor, then verify: a CO-RE agent has no clang in its image and no kernel-headers mount in its DaemonSet spec.
$ kubectl -n kube-system get ds falco -o jsonpath='{.spec.template.spec.volumes[*].hostPath.path}'
/proc /etc/os-releaseKnow which programs are already attached before you add another. Every platform and every one-liner lands in the same kernel, and bpftool is the only view that shows all of them together.
$ bpftool prog show | grep -E 'sched_cls|tracepoint'
187: sched_cls name cil_from_container tag 3f1c9a5d2b7e4801
loaded_at 2026-07-28T04:11:02+0000 uid 0
btf_id 9 memlock 4096B map_ids 44,45
412: tracepoint name tp_sys_enter_exec tag 88ba2d16fc0e33a1
loaded_at 2026-08-01T18:22:47+0000 uid 0Pick the platform by what it does at the moment of the event, not by feature lists. Falco is a detection engine: it evaluates rules and emits an alert. Tetragon evaluates in the kernel and can act there, killing the process before the syscall returns, which is a different guarantee from alerting on it a second later. Pixie captures application-level telemetry with no instrumentation and is an observability tool, not a control. Cilium owns the network datapath. The deep incident-response and forensics workflow on top of these belongs to the Kubernetes Security course; what matters here is that enforcement in the kernel and detection after the fact are not substitutes.
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: block-write-to-etc
spec:
kprobes:
- call: security_file_permission
syscall: false
selectors:
- matchArgs:
- index: 0
operator: Prefix
values: ["/etc/"]
matchActions:
- action: SigkillAfter a routine node-image bump, a fleet's continuous profiler started crash-looping, but only on the new nodes. The container log said fatal error: linux/types.h: No such file or directory — the agent was bcc-based and compiled its programs on the node at startup against kernel headers the slimmer image no longer shipped. The team's first fix was a privileged init container that installed the headers package, which worked and also meant every node now ran a compiler at boot, adding roughly a minute and several hundred megabytes of RSS to node readiness on a fleet that autoscaled hard. The real fix was the vendor's CO-RE build, which loads a prebuilt object against /sys/kernel/btf/vmlinux and starts in milliseconds. Before an eBPF agent goes fleet-wide, find out whether it compiles on the node, because that one fact decides whether an image change is a non-event or an outage.
Tradeoffs and Decision Framework
| Approach | Compiles where | Time to an answer | Fleet-wide viable | Reach for it when |
|---|---|---|---|---|
| bpftrace one-liner | On the host, at invocation | Seconds | No, ad hoc only | You have a question and a live node |
| bcc tool collection | On the host, needs headers | Seconds | Poorly, headers and RAM | A prewritten tool already asks your question |
| libbpf or Go/Rust with CO-RE | Once, on a build machine | Days of work | Yes | You need a custom always-on collector |
| Detection platform (Falco, Pixie) | Prebuilt, mostly CO-RE | Hours to install | Yes | You want rules and telemetry, not enforcement |
| Enforcing platform (Cilium, Tetragon) | Prebuilt, CO-RE | Days, it owns the datapath | Yes | The kernel must act, not just report |
Four questions sort almost every case. Are you answering a question once or watching forever, because that alone rules out half the table? Does the tool need to change what the kernel does, or only observe it? Do your nodes expose BTF, and if some do not, what happens to those nodes? And how many eBPF agents already run on that node, since each one carries its own maps, its own memory, and its own claim on hooks.
Default to bpftrace for investigation and a CO-RE platform for anything permanent, and do not write custom eBPF until you have confirmed that no existing tool answers the question. Leave that default when you need a metric nobody ships — a domain-specific latency histogram keyed by a field only your application knows — which is the honest case for a libbpf or Go program of your own.
Common Mistakes
Installing kernel headers on every node to make an agent start. It works, and it makes your node image a dependency of your observability vendor's build process forever. Push for a CO-RE build instead.
Assuming every kernel exposes BTF. Without CONFIG_DEBUG_INFO_BTF=y there is no /sys/kernel/btf/vmlinux, CO-RE tools cannot relocate, and the failure looks like a broken agent rather than a kernel config gap.
Leaving bpftrace running on a hot path. A probe on a per-packet or per-context-switch function fires millions of times a second, and the per-event cost that was invisible on openat becomes a measurable share of the node.
Attaching to internal kernel functions in permanent tooling. Kprobes on static symbols break silently across kernel upgrades. Tracepoints and fentry exist precisely so long-lived agents survive a bump.
Running two platforms that both want the same hook. The tc ingress filter and the cgroup connect hooks are shared resources, and a second agent attaching where your CNI already lives produces packet loss that looks like a network fault.
Treating detection and enforcement as the same product. An alert on a container writing to /etc and a kernel that kills the process before the write returns are different controls, and only one of them holds when the attacker is faster than your alert pipeline.
Forgetting the cost of every agent's maps. Each DaemonSet pins its own maps and charges its own memory, so a node running a CNI, a security agent, and a profiler is carrying three independent allocations that nobody budgeted together.
Leaving pinned programs behind after an investigation. A tool that pinned into /sys/fs/bpf and was killed rather than stopped keeps its programs attached and its memory charged until the node reboots.
You want to quickly trace which processes are opening a specific file across a busy server. What eBPF-based tool would you reach for, and why is eBPF better than traditional approaches here?