If you serve LLM inference at any real scale, your standard monitoring stack has a blind spot exactly where your tokens actually spend time: the kernel. Prometheus exporters and Datadog agents sample user-space counters every 15s and, worse, the sampling loop itself steals CPU from the workload you are trying to watch. When we chased a regression where ttft_p95 jumped from 400ms to 1.9s on a vLLM endpoint, none of our dashboards explained it — because the stall was in kernel space: a streaming GET blocked in tcp_sendmsg, and the GPU driver spending 300ms faulting pages in. That is the gap eBPF fills.

What user-space tracing misses

CUDA-aware metrics (DCGM, NVTX, CUPTI) give you GPU-internal timing, and the vLLM metrics endpoint gives you engine timing. Neither sees the boundaries: socket writes that backpressure on a slow client, NVMe page-in latency under memory pressure, nvidia module ioctl stalls, and the scheduler delays between a token being produced in the CUDA stream and it landing on the wire. Those boundaries are kernel code, so we profile them in kernel code. eBPF hooks into the kernel (and, via kprobes/uprobes, into driver and library paths) with sub-microsecond overhead — a couple of hundred nanoseconds per event versus the tens of microseconds strace or a user-space agent costs.

The probe set we actually run

For every serving container we attach three things:

• kprobe/kretprobe on tcp_sendmsg and tcp_sendpage, bucketed by bpf_get_current_cgroup_id() so each model deployment maps to its own histogram. This catches client-backpressure stalls that show up as p50 token-stream gaps.
• kretprobe on __do_page_fault and the compaction/swap paths, to surface memory-pressure spikes from KV-cache growth colliding with other tenants on the box.
• kprobe on the NVIDIA module’s ioctl entry (nvidia_ioctl in /proc/kallsyms, visible to a root tracer) so we can time driver calls per process. The proprietary driver strips most internal symbols, so treat eBPF on it as a boundary timer, not a profiler; pair it with DCGM for the SM-level picture.

The tricky part is attribution: the kernel sees a PID and a cgroup, not a request ID. We solve it with a side table — a user-space agent watches vLLM’s request logs, reads the PID a request is running on, and annotates the cgroup bucket with the model and phase (prefill vs decode). It is coarse but correct, and it is what turns a kernel histogram into an actionable per-model metric.

Here is the production ring-buffer probe we ship (BCC/BTF, kernel 6.x):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

struct sock_write_evt {
    u64 cgroup;
    u64 delta_ns;
    u32 pid;
};

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 65536);
    __type(key, u64);
    __type(value, u64);
} starts SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 1 << 20);
} events SEC(".maps");

SEC("kprobe/tcp_sendmsg")
int BPF_KPROBE(tcp_sendmsg_entry)
{
    u64 key = bpf_get_current_pid_tgid();
    u64 ts = bpf_ktime_get_ns();
    bpf_map_update_elem(&starts, &key, &ts, BPF_ANY);
    return 0;
}

SEC("kretprobe/tcp_sendmsg")
int BPF_KRETPROBE(tcp_sendmsg_ret)
{
    u64 key = bpf_get_current_pid_tgid();
    u64 *ts = bpf_map_lookup_elem(&starts, &key);
    if (!ts)
        return 0;
    bpf_map_delete_elem(&starts, &key);

    struct sock_write_evt *ev = bpf_ringbuf_reserve(&events,
        sizeof(*ev), 0);
    if (!ev)
        return 0;
    ev->cgroup = bpf_get_current_cgroup_id();
    ev->pid = bpf_get_current_pid_tgid() >> 32;
    ev->delta_ns = bpf_ktime_get_ns() - *ts;
    bpf_ringbuf_submit(ev, 0);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

A ring buffer (not the old perf buffer) drops nothing under burst and reports BPF_RINGBUF_BUSY backpressure in the userspace consumer, which we expose as a metric — because a probe that silently drops its own events is worse than no probe. At our peak of ~11k req/s across the fleet, roughly 44k events/s through a 1 MiB ring buffer costs about 1% of one core. That is the entire tax for a continuous trace of every socket write in the cluster.

What it caught

Two incidents this quarter. First: a “slow GPU” ticket that was actually a tcp_sendmsg p99 of 8.7s — a client fetching a streamed completion but reading the body too slowly, so TCP backpressure stalled the generator thread and inflated decode latency for all concurrent requests on that replica. One ss -ti plus the eBPF histogram pointed at the client, not the GPU. Second: KV-cache-driven page faults. Under peak concurrency the box’s working set exceeded RAM, and __do_page_fault p95 jumped to 4.2ms; the fix was capping per-tenant KV cache, not buying more machines.

Production notes

• Scope probes to the cgroup, not the host. Attaching globally floods you with noise from unrelated workloads; filter on bpf_get_current_cgroup_id() at the source.
• Keep maps tiny and bounded. A hash map keyed by pid_tgid leaks one entry on any crash — bound it and let the bpf_map_update_elem(..., BPF_NOEXIST) fail loudly rather than grow unbounded.
• Do not try to replace GPU telemetry with eBPF. Use eBPF for the host-kernel boundary and DCGM/CUPTI for the GPU interior; they answer different questions.
• Export drop counters. If the ring buffer or the userspace consumer falls behind, you are flying blind under exactly the load you care about most.

eBPF is not a replacement for your metrics stack — it is the tool that finally lets you see the kernel half of every token’s round trip, and it is cheap enough to run always-on.