Go already has a netpoll – know why you would reimplement it

The headline claim of the goroutine-per-connection model is that you write blocking code and the runtime threads it over epoll/kqueue for you. Every net.Conn read and write parks a goroutine; Go’s netpoller – one epoll fd per process on Linux, kqueue on macOS – wakes it when the fd becomes ready. That is not free. Each blocked connection carries a goroutine whose stack starts at 2KB and grows, plus scheduler bookkeeping. At 1M idle connections you are past 4GB of RAM, most of it spent on a stack you will never use.

You hand-roll an event loop when you control the whole connection life cycle and want to amortize that cost: connection proxies, gateway admission, or state machines small enough that a per-connection goroutine is mostly overhead. For everything else, net with tuned GOMAXPROCS beats a custom loop on correctness for free. We only reach for a reactor in the 100k-plus concurrent-connection range.

A minimal edge-triggered reactor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
//go:build linux

package main

import "golang.org/x/sys/unix"

const maxEvents = 1024

type Poller struct {
	fd     int
	events []unix.EpollEvent
}

func NewPoller() (*Poller, error) {
	fd, err := unix.EpollCreate1(unix.EPOLL_CLOEXEC)
	if err != nil {
		return nil, err
	}
	return &Poller{fd: fd, events: make([]unix.EpollEvent, maxEvents)}, nil
}

func (p *Poller) Add(fd int) error {
	return unix.EpollCtl(p.fd, unix.EPOLL_CTL_ADD, fd, &unix.EpollEvent{
		Events: unix.EPOLLIN | unix.EPOLLET,
		Fd:     int32(fd),
	})
}

// Poll blocks until events arrive. The fd slice is reused by the caller;
// drain the socket before returning to the wait loop.
func (p *Poller) Poll() ([]int, error) {
	n, err := unix.EpollWait(p.fd, p.events, -1)
	if err != nil {
		if err == unix.EINTR {
			return nil, nil
		}
		return nil, err
	}
	out := make([]int, 0, n)
	for i := 0; i < n; i++ {
		out = append(out, int(p.events[i].Fd))
	}
	return out, nil
}

func run(fd int) error {
	ep, err := NewPoller()
	if err != nil {
		return err
	}
	if err := ep.Add(fd); err != nil {
		return err
	}
	for {
		ready, err := ep.Poll()
		if err != nil {
			return err
		}
		for _, rfd := range ready {
			handle(rfd) // must drain to EAGAIN before re-arming
		}
	}
}

The events buffer is allocated once in the poller; per-call allocation is exactly the kind of garbage a reactor should not generate. The same goes for out – reuse a per-poller buffer in production rather than allocating on every wait.

The subtle parts

• Edge-triggered starvation. With EPOLLET the kernel reports an fd once per state change. If you do not read until the socket returns EAGAIN, you will never be woken for that fd again, and the connection silently stalls. This is the number one bug in hand-rolled reactors. Level-triggered mode is the safe default if you cannot prove your handler drains.
• Non-blocking sockets. Register a blocking fd and your single reactor thread blocks inside read while every other connection waits. Set O_NONBLOCK before EpollCtl (Go’s golang.org/x/sys/unix.SetNonblock or syscall.SetNonblock).
• EINTR. epoll_wait returns EINTR when a signal is delivered; treat it as “no events” and loop, as above.
• One poller, one core. A single blocking EpollWait goroutine runs on one P. On multi-core boxes you want several pollers with fds hashed across them, mirroring how the runtime netpoller keeps a poller per P. Splitting fds across threads is also the only way to get more than one core of epoll_wait syscall throughput.
• Portability. unix.Epoll* is Linux-only. The same Poller interface hides a kqueue implementation behind a //go:build darwin || freebsd file and a select-based loop for Windows. On platforms you do not care about, just call into the runtime’s netpoller instead.

What we measured

With goroutine-per-connection, 1M idle connections cost roughly 4-5GB of resident memory. A reactor using a handful of pollers registers fds in the kernel at a few hundred bytes each and allocates only the read buffer you actually need (say 512B) per connection – an order of magnitude less. Latency-wise, epoll_wait returning a batch of 1,024 events costs around 20-50µs including handler work on a midrange server; a single-threaded echo reactor sustains roughly 500k-800k round trips per second.

Production lessons

• Drain to EAGAIN or re-arm level-triggered; the fd that never wakes up again is invisible in logs.
• Never share the poller’s event buffer between goroutines; return ownership with each Poll result.
• Set fds non-blocking before registration; blocking fds are a hang waiting to happen.
• Watch epoll_wait syscall rate, not just event rate – if your handlers wake the loop more than once per useful event, you are paying for churn.
• Ship build tags for kqueue early, or you will discover your loop only runs on Linux at the worst possible moment.

The honest conclusion: the runtime netpoller already does this work for you, and it is better tested than anything you will write. Hand-roll the loop only when memory per connection is the actual budget and you can afford the complexity. The moment your per-connection state grows, the goroutine model wins again.