The memory math nobody does up front

The load test passed. The marketing benchmark said “1M connections on one box.” Then production filled up and the nodes started OOM-killing at 1 AM. The pattern is always the same: nobody budgets for what a million idle connections actually cost before the boxes are ordered.

A naive goroutine-per-connection WebSocket server in Go spends, per connection, roughly this:

• Goroutine stack — 2 KB minimum, but a real handler with a read buffer and call depth commits 4–8 KB.
• net.Conn and runtime bookkeeping — about 1 KB of internal state you cannot see.
• Read buffer — 4 KB if you preallocate one per connection.
• WebSocket frame state and upgrade context — another 1–2 KB.
• Write-side queue — unbounded channels on every fanout path are a ticking time bomb of its own.

That is 12–15 KB per connection before GC pressure, which on a shared heap doubles the effective footprint because the collector scans all of it. Ten gigabytes per million connections is not an exaggeration; it is the baseline. Our first attempt at 900k connections ran 9.1 GB per node and 70% CPU at idle, doing nothing except waiting for the next ping.

The levers that actually move the needle

Go’s runtime already gives you epoll on Linux and kqueue on macOS through the netpoller — your connection goroutines are not spinning, they are parked on the poller. So the “event loop vs. goroutine” debate is mostly a false one. What actually kills you is what you keep alive per connection: the goroutine stack, the buffers, and worst of all, a time.Ticker per connection so the server can detect dead peers.

The standard readPump/writePump pattern still holds up at scale, with three non-negotiable rules: one goroutine per connection is fine up to a few hundred thousand, never more than one writer at a time, and read buffers belong in a pool. On the read side, github.com/coder/websocket — the maintained successor to the archived gorilla/websocket — reuses its internal buffer between reads, so the frame data is only valid until the next call. Copy before handing off:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
type conn struct {
	ws   *websocket.Conn
	send chan []byte
}

const (
	writeWait = 10 * time.Second
	pongWait  = 60 * time.Second
	pingPeriod = pongWait * 9 / 10
	sendBuffer = 1024
)

func (c *conn) reader() {
	defer c.ws.CloseNow()
	c.ws.SetReadLimit(1 << 20)
	for {
		_, data, err := c.ws.Read(context.Background())
		if err != nil {
			return
		}
		handle(append([]byte(nil), data...))
	}
}

func (c *conn) writer() {
	ticker := time.NewTicker(pingPeriod)
	defer ticker.Stop()
	for {
		select {
		case data := <-c.send:
			ctx, cancel := context.WithTimeout(context.Background(), writeWait)
			err := c.ws.Write(ctx, websocket.MessageText, data)
			cancel()
			if err != nil {
				return
			}
		case <-ticker.C:
			ctx, cancel := context.WithTimeout(context.Background(), writeWait)
			err := c.ws.Ping(ctx)
			cancel()
			if err != nil {
				return
			}
		}
	}
}

The single writer goroutine matters more than any epoll trick: concurrent writers on a WebSocket are a protocol violation waiting to corrupt a frame. Everything that wants to send — broadcast loops, per-user triggers — pushes to the bounded send channel, and backpressure is a drop policy, never an unbounded queue.

The timer problem everyone discovers late

A time.Ticker per connection means a million tickers. Even after Go 1.23 moved timers to per-P heaps and made scheduling dramatically cheaper, a million active timers generate constant wakeups and GC churn, and they all want to fire at once. That thundering herd of simultaneous pings and pongs is how you end up with a CPU spike every 60 seconds.

We replaced per-connection tickers with a single hierarchical timing wheel: one 250 ms driver goroutine, a fixed ring of slots, and each connection registered in the slot for its next ping or hard deadline. Scanning a slot is amortized O(1), total timer goroutines drop from one million to one, and we jitter each connection’s phase so pongs spread out instead of stacking. This single change cut idle CPU from 70% to 11% at the same connection count.

Kernel, load balancer, and the settings that matter

The application is half the battle; the OS is the other half. On our fleet, on a systemd node:

1
2
3
4
5
6
7
8
9
# /etc/sysctl.d/90-websocket.conf
fs.file-max = 2000000
net.core.somaxconn = 65535
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_time = 120
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 5

Plus LimitNOFILE=1048576 in the unit file and the equivalent ulimit locally. Three production lessons we earned the hard way:

• Load balancers kill idle connections. Every proxy has an idle timeout, and a WebSocket that sits silent for 10 minutes is a WebSocket the LB will reap. Our ping period is 54 seconds specifically because the LB we ran behind dropped connections after 60. Tune the LB and the application together, or the cheapest fix is a heartbead at a third of the LB timeout.
• Disable permessage-deflate (RFC 7692) on the server for fanout-heavy workloads. Compression per message is CPU-bound and hurts throughput more than the bandwidth it saves once payloads are small. Turn it on only for big text frames.
• Detect dead connections fast. The read deadline backed by pongWait in the pattern above is what frees a goroutine and its buffers when a client vanishes; without it, half-dead connections leak until a ping times out. A reactor framework such as lxzan/gws or a hand-rolled loop on gobwas/ws can shave the per-connection footprint below 1.5 KB by skipping goroutine stacks entirely — that is how you get a million connections per box — but read-deadline discipline is mandatory in both worlds.

What production looks like after the rewrite

Our notification fanout service runs 1.2M connections per node across eight c5.4xlarge boxes: 1.6 GB per node at idle, 11% baseline CPU, and a 99th-percentile fanout latency of 18 ms at four million messages per second. Compared to the goroutine-per-connection version at 900k, that is an 82% memory reduction and a 6x drop in idle CPU. The rewrite took two engineers three weeks, and the pings — spread, wheeled, and jittered — are the reason the fleet stays flat.

And the honest footnote: if you are under ~300k connections, none of this complexity is worth it. A plain coder/websocket server with a single writer per connection and a pooled read buffer will carry you for months. Ignore the “1M connections!” benchmark posts — they are almost always measured at zero traffic with no heartbeats. Measure at your real message rate, budget the memory per connection up front, and reach for the timing wheel and the reactor loop only when the profile says the goroutine stacks and the timers are what is eating you.