eBPF Maps: Sharing State
An eBPF program runs on an event and then it's gone. So how does it remember anything, or communicate with user space? eBPF maps are the shared data structures that make stateful eBPF possible.
The Concept Explained
A tc program on a pod's veth sees a SYN leave for a service IP. It rewrites the destination to a backend, returns TC_ACT_OK, and its 512-byte stack is reclaimed. Forty microseconds later the SYN-ACK comes back on a different CPU, and a second invocation of the same program has to decide whether this is the reply direction of a connection it already approved, and which backend to un-NAT it to. It has no registers left from last time, no globals, no heap, and no way to call into the agent that programmed it.
Everything eBPF does that looks like memory is a map. A map is a kernel object with a fixed type, a key size, a value size, and a max_entries ceiling, created by user space through the bpf() syscall and referred to by file descriptor. Kernel-side programs touch it only through helpers — bpf_map_lookup_elem, bpf_map_update_elem, bpf_map_delete_elem — and the verifier insists you null-check the pointer a lookup returns before dereferencing it, because a miss is a legal outcome.
Two consequences fall out. Maps are what make an event-triggered program stateful: connection tracking, NAT bindings, service and backend tables, LPM-matched CIDR policy, and the identity cache all live in maps rather than in program text. They are also the sanctioned bidirectional channel across the kernel boundary — the agent writes policy and backends down, the datapath pushes drops and flows up, and there is no netlink socket and no character device involved.
The second point is stronger than it sounds. The map is the API, not a cache in front of one. When a service scales from three backends to nine, Cilium recompiles and reloads nothing on the hot path; it writes six entries into the backends map and one updated entry into the services map. That is why an eBPF dataplane absorbs churn in constant time while an iptables dataplane rewrites and reinstalls chains.
It also makes capacity a decision you make once. max_entries is fixed at creation; growing it means a new map and a reload of every program that uses it.
The map, not the program, is the object you actually operate. Programs are close to static once loaded; the maps hold every piece of mutable state, carry every control-plane update, and impose a hard, pre-declared ceiling on how much your dataplane can remember. Sizing, memory accounting, and the behavior at that ceiling — silent LRU eviction versus a hard E2BIG and a dropped packet — are the operational surface of eBPF, and almost every capacity incident here is a map that hit a limit somebody set at install time and never revisited.
How It Works
How One Connection Tracking Entry Crosses The Kernel Boundary
Click each step to explore
What the sequence hides is that the two sides of that boundary have very different consistency guarantees. Kernel-side lookups are per-bucket and RCU-protected, so a program never blocks. User-space iteration is not a snapshot: BPF_MAP_GET_NEXT_KEY walks live buckets while the datapath inserts and evicts underneath you, so a dump of a busy conntrack table smears across a few hundred milliseconds in which you can see the same flow twice or miss one entirely.
The declaration and the access pattern are small enough to read at a glance:
struct {
__uint(type, BPF_MAP_TYPE_LRU_HASH);
__uint(max_entries, 524288);
__type(key, struct ipv4_ct_tuple);
__type(value, struct ct_entry);
} CT_MAP_TCP4 SEC(".maps");
struct ct_entry *entry = bpf_map_lookup_elem(&CT_MAP_TCP4, &tuple);
if (!entry) /* verifier rejects the load without this */
return ct_create(&CT_MAP_TCP4, &tuple);
entry->packets++; /* direct store into the map value */
entry->last_tx_report = bpf_ktime_get_ns();
return CT_ESTABLISHED;That counter increment is the detail worth internalizing. The helper handed back a pointer into the map's own memory, so updating a field is a store, not a second helper call and not a copy in and out. Per-packet state is cheap because of that, and it is also why the verifier is strict about the null check.
Per-CPU Maps Are Not Just A Faster Hash
BPF_MAP_TYPE_PERCPU_HASH and PERCPU_ARRAY allocate one value per possible CPU and hand the program only the local core's copy. No atomics, no cache line bouncing — which matters enormously for counters, because a shared __sync_fetch_and_add on a per-packet counter serializes every core onto one cache line and becomes a throughput ceiling long before the network does.
The cost moves to user space. A lookup returns an array of nr_cpus values you must sum yourself, and possible CPUs is not the same as online CPUs. Reading index zero and reporting it as the count is the most common per-CPU bug, and the metric is then wrong by roughly the core count with no error anywhere. Per-CPU values are also capped at 32KB and multiply memory by CPU count, so a million-entry per-CPU hash on a 96-core node is a budget conversation.
Events Out: Ring Buffer Or Perf Event Array
PERF_EVENT_ARRAY gives you one ring per CPU: ordering holds only within a CPU, memory multiplies by core count, and you size each ring blind. BPF_MAP_TYPE_RINGBUF, available from kernel 5.8, is one shared multi-producer buffer with global ordering, and bpf_ringbuf_reserve fails before you build the record, so the program counts the drop precisely instead of you inferring the loss later. Reach for ringbuf unless you must support pre-5.8 kernels.
A plain BPF_MAP_TYPE_HASH preallocates all of max_entries at creation unless you pass BPF_F_NO_PREALLOC, so a million-entry map with a 64-byte value charges hundreds of megabytes the instant the agent loads. Oversized settings therefore show up as nodes going OOM at agent startup, not under load. The mirror-image failure is worse: a full non-LRU map returns -E2BIG on insert, the datapath drops the packet, and nothing in the application or CNI logs says the word map. Conntrack and NAT tables are LRU precisely because forgetting an old entry is a better failure than refusing a new connection.
Practical Application
Start at map pressure, not at the packet drops. Cilium exports cilium_bpf_map_pressure per map — entries over max_entries. Alert at 0.85 and you get days of warning; wait for drops and you are debugging a network problem that is really a capacity problem.
$ kubectl -n kube-system exec ds/cilium -- cilium-dbg map list --verbose
Name Num entries Num errors Cache enabled
cilium_ct4_global 411238 0 true
cilium_lb4_services_v2 1902 0 true
cilium_ipcache 18344 0 trueRead the datapath's own tables when the higher-level tools go quiet. If Hubble shows nothing and the pod insists it sent the packet, the conntrack map says whether the datapath ever created state for that flow.
$ kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf ct list global | head -2
TCP OUT 10.244.1.7:52344 -> 10.96.14.22:80 expires=16783241 RxPackets=6 TxPackets=8 RevNAT=14
$ bpftool map show id 412
412: lru_hash name cilium_ct4_globa flags 0x0
key 38B value 56B max_entries 524288 memlock 42663936BSize maps at install, because you cannot resize them live. Few Helm values here carry a real decision, and mapDynamicSizeRatio is the one most clusters should use instead of fixed numbers: it scales the tables with node memory, so a 16GB node and a 256GB node do not get the same conntrack table.
bpf:
mapDynamicSizeRatio: 0.0025 # share of node RAM for CT/NAT maps
ctTcpMax: 1048576 # explicit values override the ratio
ctAnyMax: 524288
natMax: 1048576 # at or above ctTcpMax, or NAT fills first
policyMapMax: 16384 # per endpoint, not per nodeKnow what an agent restart does to that state. Cilium pins its maps under /sys/fs/bpf/tc/globals, so a normal restart re-opens them and keeps every established connection tracked. Changing max_entries or the value struct layout across an upgrade forces a recreate, and every in-flight connection loses its reverse-NAT binding at that instant. Long-lived connections through a service VIP are what break, and they break silently.
A cluster running a heavy outbound crawler began failing roughly one in two hundred new connections, but only on four nodes out of sixty and only during the 09:00 batch window. Hubble reported the drops without a policy verdict, which sent the team hunting for a network policy that did not exist. cilium-dbg map list showed cilium_ct4_global pegged at max_entries on those four nodes while the other fifty-six idled around thirty percent. The workload opened short-lived connections faster than the TCP timeouts reclaimed entries, so the LRU map was evicting entries for connections that still had packets in flight, and a reply with no conntrack match is treated as unsolicited and dropped. Raising ctTcpMax and shortening the closed-connection timeout cleared it in one rollout. An LRU map never reports an error; it degrades by forgetting, and the symptom surfaces somewhere that looks nothing like a full table.
Tradeoffs and Decision Framework
| Map type | Lookup cost | When full | Memory shape | Reach for it when |
|---|---|---|---|---|
HASH | Hash plus bucket walk | -E2BIG on insert | Preallocated unless NO_PREALLOC | Losing an entry silently is unacceptable |
LRU_HASH | Same, plus LRU bookkeeping | Evicts least recently used | Preallocated, per-CPU free lists | Conntrack, NAT, anything self-healing |
ARRAY | Direct index, no hashing | Fixed size, no inserts | Preallocated, dense | Config, lookup tables, tail-call maps |
PERCPU_* | Lock-free, local core only | As the base type | Multiplied by possible CPUs | Hot-path counters and scratch buffers |
LPM_TRIE | Longest-prefix walk, slower | Insert fails | Sized by prefix count | CIDR matching and identity lookups |
RINGBUF | Reserve and commit | Reserve fails, drop is countable | One shared buffer | Streaming events out on 5.8 and later |
SOCKMAP / SOCKHASH | Hash to a socket reference | Insert fails | One entry per socket | Socket redirect and sidecar-free L7 |
Four questions decide it. Is losing an entry acceptable, or does the datapath break undetectably when state disappears? Is the value written on a hot path by one CPU and aggregated later, or read by everyone? Is the key an exact tuple or a prefix, since LPM_TRIE is the only honest answer to a prefix and it is not free? And how many entries at the busiest minute of the busiest day, not at the median?
The default for most dataplane state is an LRU_HASH sized with mapDynamicSizeRatio and alerted on pressure, plus PERCPU_ARRAY for counters and RINGBUF for events. Leave that default when correctness beats availability: a policy or identity map where a missing entry means allowing or denying the wrong traffic should be a plain hash that fails loudly rather than an LRU that quietly forgets.
Common Mistakes
Reading index zero of a per-CPU value. The value is an array of per-CPU copies you have to sum. Take the first element and your counter is wrong by roughly the core count, consistently, with no error anywhere.
Preallocating a huge hash map and blaming the OOM on the workload. Memory is charged at creation, not on first insert, so the failure appears at agent startup and looks nothing like a capacity problem.
Assuming a full map produces an error you will see. LRU maps evict silently and hash maps return -E2BIG into a datapath that drops the packet. Map pressure is the only early signal.
Sizing the NAT map below the conntrack map. Every NATed connection needs an entry in both, so the smaller table is the real ceiling and the metric you watch is the wrong one.
Assuming an LRU map keeps the entries that matter. Eviction is by recency, not importance, so a long-lived idle connection goes before a burst of short-lived ones nobody will use again.
Treating a map dump as a consistent snapshot. Iteration walks a live table, so entries appear twice or not at all, and diffing two dumps invents connections.
Forgetting that pinning controls lifetime. Ad hoc tooling that pins under /sys/fs/bpf and exits without cleanup leaves the memory charged indefinitely, and an unpinned map takes its state with it when the last descriptor closes.
eBPF programs are event-triggered and stateless on their own. How do eBPF maps let them maintain state and communicate with user space? Give an example use.