Go’s concurrency story rests on one runtime component most engineers never look at: the scheduler. If you have ever seen a service with 300,000 goroutines and 12 cores stay responsive, that is the work-stealing scheduler doing its job. If you have ever seen a Go container with a 4-core quota burning hundreds of OS threads, that is the same scheduler doing what you configured it to do. The GMP model – goroutines, OS threads, and the per-thread processor contexts they run on – is not an academic detail; it determines your latency under load.

The GMP model

• G is a goroutine: a stack that grows and a runnable state, plus the CPU register state of its execution context.
• M is an OS thread. The scheduler can create and destroy Ms, and the runtime’s sysmon thread retires idle ones.
• P is a logical processor, an execution context holding a local run queue of Gs. GOMAXPROCS sets the number of Ps, which is the degree of parallelism.

At any instant, an M must hold a P to run Gs. The key trick is where runnable goroutines wait. Each P has a local run queue; when one goes empty, an idle M with a P steals work from another P’s queue – hence “work stealing.” A global run queue catches overflow. This design keeps most scheduling decisions off shared locks: local queues are per-P, and stealing happens rarely relative to push/pop. That is why spawning a goroutine costs around 200ns and the scheduler scales to millions of Gs.

Preemption keeps it honest

Before Go 1.14, a tight CPU-bound loop with no function call could starve the scheduler: no safe point, no preemption, and GC could not run. Since Go 1.14 the runtime sends an asynchronous preemption signal (SIGURG) to goroutines that run too long, letting the scheduler reclaim the M at the next safe point. The practical result: a 100ms CPU-bound loop in one goroutine no longer freezes the other 99. The known caveat is that preemption can still be delayed by C code holding the thread, and runtime.LockOSThread disables async preemption on that thread entirely – I have seen a CGo library do this and silently freeze a process for seconds.

The GOMAXPROCS-in-a-container trap

Here is the production landmine. GOMAXPROCS defaults to NumCPU(), and NumCPU() returns the host’s CPU count – the runtime does not read your cgroup quota. On a 64-core host with a 4-core quota container, your Go process runs 64 Ps on 4 allowed cores. Sixty-four threads are eligible to run, the OS constrains them, and under load you get thread churn, higher context-switch overhead, and GC that overshoots because it assumes it has 64 parallel markers. We hit this at 1am with a service showing 300 threads, 40% sys time, and latency that made no sense. The fix was forcing GOMAXPROCS=4 in the deployment (or running uber-go/automaxprocs, which reads the cgroup quota and sets it for you), and the sys time vanished.

The same mechanism bites in another way: long blocking syscalls. When a goroutine makes a blocking syscall, the runtime detaches the M from the P and hands the P to a new M so parallelism is not lost. If every request does a 50ms blocking call, the scheduler allocates new Ms to keep the Ps busy. That is correct behavior, but it means M count is a health signal: a sudden thread surge usually means goroutines stuck in blocking calls, not a scheduling bug. Monitor /sched/threads:threads from runtime/metrics and alert on regressions.

Bounding CPU-bound work

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
package main

import (
	"context"
	"runtime"
)

// RunBounded runs CPU-bound jobs on at most GOMAXPROCS concurrent workers.
func RunBounded(ctx context.Context, jobs []func()) error {
	slots := make(chan struct{}, runtime.GOMAXPROCS(0))
	for _, j := range jobs {
		select {
		case <-ctx.Done():
			return ctx.Err()
		case slots <- struct{}{}:
		}
		go func(j func()) {
			defer func() { <-slots }()
			j()
		}(j)
	}
	return nil
}

The pattern to internalize: the scheduler multiplexes logical parallelism over GOMAXPROCS physical slots, and oversubscription beyond that is where threads multiply. When you want to limit CPU-bound concurrency to the machine, gate with a semaphore sized to GOMAXPROCS(0) rather than spawning unbounded goroutines and trusting the scheduler to sort it out.

Production lessons

• Respect GOMAXPROCS in containers: set it to the cgroup quota or run automaxprocs. The runtime will not do it for you.
• Trust the work-stealing scheduler’s per-P queues: goroutine creation is cheap, so model with goroutines and only bound them when you have a real reason (CPU-bound work, external limits).
• Async preemption (Go 1.14+) covers pure-Go tight loops; it does not cover CGo frames or LockOSThread. Know which of those you run.
• Watch /sched/threads:threads and goroutine counts as health metrics; thread storms precede latency incidents, they do not follow them.
• Size semaphores and pools off GOMAXPROCS(0) – which at least respects the value you set – not runtime.NumCPU().