A 2GB Kubernetes pod running a Go service with default GOGC=100 has a GC that will happily let the heap double before it starts collecting. That is by design: the goal is throughput, not a low footprint. But it is a terrible match for containers, where the ceiling is fixed and OOMKilled is final. I have stood in front of dashboards where heap went 700MB -> 1.3GB -> 1.6GB in seconds and the pod restarted, with the GC never once getting to run before the cgroup killed us. The fix was not “tune harder” but “set an actual budget,” and Go’s GOMEMLIMIT (Go 1.19+) is the tool for that.

How the GC budget works

The concurrent, tri-color mark-and-sweep collector is triggered by heap growth, not a clock. With the default GOGC=100, the target heap is live + GOGC% * live, so it doubles. After a cycle, the runtime only starts the next one when the heap reaches that goal again. For a service with a stable live set that is cheap: a few cycles per minute, each pause in the tens to hundreds of microseconds. The trouble starts with allocation spikes. A burst that multiplies live heap in one frame leaves the collector no room, the process breaches the container limit, and the kernel does what it does.

GOMEMLIMIT changes the calculus: the runtime treats it as a soft target for total heap and starts GC early as the limit approaches. Two properties matter:

• It is soft. Under extreme pressure the runtime may exceed it, because GC can only do so much; it caps GC work at roughly 50% of CPU time, beyond which it gives up on meeting the limit.
• It only counts Go-managed heap. OS thread stacks, cgo memory, syscall buffers, and page cache are not part of the limit, so you still need headroom.

The right configuration for spiky services

The pairing that works in production is a high GOGC with a hard GOMEMLIMIT. Set GOGC high (200-400) so steady state stays cheap – fewer cycles, less CPU spent marking – and set GOMEMLIMIT as the backstop so spikes get collected before the container dies. In the incident above, the service had a live heap of ~500MB, so the default budget was 1GB. A spike to 1.6GB OOM-killed it. With GOGC=400 and GOMEMLIMIT=1.6GB in a 2GB pod, steady state runs a GC every few minutes, and the spike triggers exactly one when it needs one.

Two mistakes I see constantly:

• Setting GOMEMLIMIT below the peak live heap. If reachable memory exceeds the limit, the runtime runs GC continuously trying to get under it, and you burn up to half your CPU in runtime.gcBgMarkWorker while still failing. The limit must sit above your worst-case reachable set.
• Chasing the limit too closely. Leave 10-15% of the container for non-GC memory. A 2GB pod with a 1.9GB GOMEMLIMIT has no room for thread stacks under load.

Watching it instead of guessing

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
package main

import (
	"fmt"
	"runtime/metrics"
	"time"
)

func main() {
	const goal = "/gc/heap/goal:bytes"
	const inuse = "/memory/classes/total:bytes"
	samples := []metrics.Sample{{Name: goal}, {Name: inuse}}

	for {
		metrics.Read(samples)
		goalMB := samples[0].Value.Uint64() >> 20
		inuseMB := samples[1].Value.Uint64() >> 20
		fmt.Printf("heap goal=%dMB inuse=%dMB headroom=%dMB\n",
			goalMB, inuseMB, goalMB-inuseMB)
		time.Sleep(10 * time.Second)
	}
}

/gc/heap/goal:bytes is what the runtime is aiming for; /memory/classes/total:bytes is what it has in play. The gap between them is your real headroom. Alert when the gap collapses for sustained seconds, or when the runtime is overrunning the limit. Use runtime/metrics rather than runtime.ReadMemStats on a timer: ReadMemStats forces a stop-the-world pause to build its snapshot, which is exactly the latency event you are trying to avoid.

Production lessons

• Set GOMEMLIMIT via the environment variable in your deployment, not in code. The value belongs next to the pod spec so SRE can change it without a rebuild.
• Pair it with a raised GOGC for spiky services; the two knobs cover different failure modes.
• Keep the limit 10-15% below the container ceiling.
• Never set the limit below peak reachable heap, or you trade OOMs for a permanent GC tax.
• Monitor the goal-vs-used gap with runtime/metrics and alert on its collapse.