Our Redis client was pushing a sustained 8% CPU on every API node, and pprof could not tell us why. The CPU profile blamed runtime.futex, which is not an answer; it is the shape of a goroutine blocked in the kernel. pprof samples user space only, so the entire kernel-side cost of 180k syscalls/second per node was invisible to us. The fix was to stop sampling user space and instrument the kernel directly. eBPF lets us attach programs to kernel events, read raw syscall and scheduler data, and stream it to user space — all without patching the kernel, reloading modules, or shipping a custom strace.

Pick the attach point before you write a line of C

The single biggest decision is tracepoint vs kprobe. Kprobes attach to arbitrary symbols like sys_clone and cover anything, but they are brittle: syscall names are architecture-dependent (__x64_sys_clone vs __arm64_sys_clone), they can break across kernel versions, and hooking a function call adds overhead to every invocation. Tracepoints are stable, versioned hooks compiled into the kernel, with documented argument layouts. Start with a tracepoint; reach for a kprobe only when no tracepoint exposes the symbol you need. In our case sched/sched_process_fork gives us every new goroutine-or-thread with zero guesswork about symbols.

The C program

The kernel side is a small C file, compiled with clang -target bpf into an ELF object. For modern kernels, compile against BTF — the kernel’s own type information — with CO-RE (compile once, run everywhere), so one object works across every kernel in the fleet:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

struct event { pid_t pid; __u64 ts; };

struct {
	__uint(type, BPF_MAP_TYPE_RINGBUF);
	__uint(max_entries, 1 << 24);
} events SEC(".maps");

SEC("tracepoint/sched/sched_process_fork")
int track_fork(struct trace_event_raw_sched_process_fork *ctx)
{
	struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
	if (!e)
		return 0;
	e->pid = ctx->parent_pid;
	e->ts = bpf_ktime_get_ns();
	bpf_ringbuf_submit(e, 0);
	return 0;
}

char LICENSE[] SEC("license") = "GPL";

Loading it from Go

Loading is Go’s job: parse the ELF, create the maps and programs, attach the tracepoint, then read events off the ring buffer. The Cilium cilium/ebpf library is the standard here, and its BPF-to-Go struct generation (bpf2go) keeps the userspace event layout in sync with the kernel program at build time:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
package main

import (
	"encoding/binary"
	"fmt"

	"github.com/cilium/ebpf"
	"github.com/cilium/ebpf/link"
	"github.com/cilium/ebpf/rlimit"
	"github.com/cilium/ebpf/ringbuf"
)

//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cc clang probe fork.bpf.c

func main() {
	if err := rlimit.RemoveMemlock(); err != nil {
		panic(err)
	}
	objs := probeObjects{}
	if err := loadProbeObjects(&objs, nil); err != nil {
		panic(err)
	}
	defer objs.Close()

	lp, err := link.Tracepoint("sched", "sched_process_fork", objs.TrackFork, nil)
	if err != nil {
		panic(err)
	}
	defer lp.Close()

	rd, err := ringbuf.NewReader(objs.Events)
	if err != nil {
		panic(err)
	}
	defer rd.Close()

	var e probeEvent
	for {
		rec, err := rd.Read()
		if err != nil {
			continue
		}
		if err := binary.Read(bytes.NewReader(rec.RawSample), binary.NativeEndian, &e); err != nil {
			continue
		}
		fmt.Printf("fork pid=%d at %dns\n", e.Pid, e.Ts)
	}
}

RemoveMemlock matters: on older kernels eBPF maps draw from the user’s RLIMIT_MEMLOCK, and a fresh 1 MiB×256 ring buffer will fail the default 8 MiB limit. On 5.11+ the charge moved to memcg, but we still call it defensively — our oldest nodes still enforce the old limit.

Performance numbers that justify the machinery

We measured an attached sched_process_fork tracepoint at roughly 300–500 ns of added latency per event, versus 1–3 µs for a kprobe on the same path, versus the 50 µs+ floor of perf_event sampling at any useful rate. Ring buffer reads in Go cost about 200 ns per event in bulk and batch well: we ingest 50k events/sec on one core with the reader thread at 15% CPU. The map memory is the real budget line — each max_entries slot is 24 bytes in our case, and a 16M-entry hashmap is ~400 MB of kernel memory. We size max_entries to the measured peak and cap the ring at 16 MiB, which at 50k events/sec gives us a ~20 second replay window if the user-space reader stalls.

Production lessons

• The reader is the drop point. If the Go reader falls behind, bpf_ringbuf_reserve fails and you silently lose events. Put the reader on a dedicated goroutine with a WorkerPool at least two deep, and treat ring buffer overruns as an alertable metric, not a log line.
• Never attach a kprobe by guess. Verify the symbol exists before shipping: grep __x64_sys /proc/kallsyms. A kprobe on a nonexistent symbol is a deployment that looks healthy and does nothing.
• Pin your maps if telemetry outlives the process. link.Tracepoint closes on process exit; a one-shot CLI is fine, but a long-lived agent must link.Pin or the kernel detaches your probe and you lose the tail of the trace exactly when you need it.
• CO-RE is a build-time contract. Ship the same .o everywhere, but test it against your oldest kernel in CI before you trust “compile once, run everywhere.”
• Cap collection, not analysis. We sample aggressively in the kernel and store everything downstream; the cost is in instrumentation, not storage.

eBPF did not replace pprof for us — it explained the parts pprof is structurally blind to. Once we could see the fork storm from the scheduler’s side, the fix was a 40-line change to the connection pool, and CPU dropped from 8% to 5.4%.