gRPC is built on HTTP/2, and HTTP/2 wants a few long-lived connections, not many short ones. Put a classic L4 or L7 load balancer in front and you have re-introduced the exact problem gRPC was designed to escape: the LB terminates the connection, every request pays for that hop, and — the subtle killer — because your client keeps one connection open, all your traffic rides a single TCP stream to a single backend. We measured it on three identical backends behind an NLB: CPU sat at 95% / 10% / 5%. Not a load problem. A load distribution problem.

The fix is client-side load balancing: the client discovers every backend itself and picks one per request. In grpc-go that means two pieces: a resolver that produces the backend address set, and a balancer policy that decides which connection a request uses. For our money, a custom resolver plus the built-in round_robin policy is the sweet spot.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
package main

import (
	"context"
	"sync"
	"time"

	"google.golang.org/grpc/resolver"
)

const scheme = "reg"

type regResolver struct {
	mu   sync.Mutex
	cc   resolver.ClientConn
	stop chan struct{}
}

func (r *regResolver) Start() {
	go r.watch()
}

func (r *regResolver) watch() {
	t := time.NewTicker(15 * time.Second)
	defer t.Stop()
	for {
		r.refresh()
		select {
		case <-r.stop:
			return
		case <-t.C:
		}
	}
}

func (r *regResolver) refresh() {
	ips, err := registryBackends(context.Background()) // Consul, etcd, or control plane
	if err != nil {
		return // keep serving the last good address set
	}
	addrs := make([]resolver.Address, 0, len(ips))
	for _, ip := range ips {
		addrs = append(addrs, resolver.Address{Addr: ip})
	}
	r.cc.UpdateState(resolver.State{Addresses: addrs})
}

func (r *regResolver) ResolveNow(resolver.ResolveNowOptions) {}
func (r *regResolver) Close()                                 { close(r.stop) }

type regBuilder struct{}

func (regBuilder) Scheme() string { return scheme }

func (regBuilder) Build(target resolver.Target, cc resolver.ClientConn, _ resolver.BuildOptions) (resolver.Resolver, error) {
	return &regResolver{cc: cc, stop: make(chan struct{})}, nil
}

func init() { resolver.Register(regBuilder{}) }

Then dial with the policy explicitly — the default is pick_first, which pins you to one connection and recreates the whole problem:

1
2
3
4
5
conn, err := grpc.Dial(
	scheme+":///backend-svc",
	grpc.WithInsecure(),
	grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`),
)

With round_robin the client maintains one HTTP/2 connection per backend and rotates RPCs across them. TLS handshakes amortize over thousands of calls, and no middlebox sits in your request path.

What changed in production:

• Backend CPU went 95/10/5 to 35/33/32, and p99 dropped from 180 ms to 60 ms — the hot node’s queue simply vanished.
• Connection count stayed flat: one connection per backend, reused. Connection setup (TLS, ~1–5 ms) is paid once, not per RPC.

Production lessons:

• pick_first is the default, and it is a trap here. If you don’t set the service config, a client with a stable address list reuses one connection forever. Opt into round_robin explicitly.
• Update state from one goroutine. ClientConn.UpdateState is not safe for concurrent callers; our watch loop owns it exclusively and returns early on registry errors.
• Fail silently, log loudly. When the registry is unreachable we keep the last good address set — the client doesn’t flap between snapshots — but the error must hit the alerting path. Liveness failover is the health checker’s job, not the resolver’s.
• Pair it with gRPC health checking. The balancer can pick a healthy subset only if your registry data says who’s alive. We added the grpc.health.v1 probe to each backend and excluded unhealthies from the address set.
• Register the scheme before any Dial. resolver.Register must run before the first connection that references the scheme, or grpc-go fails to find the builder. init() in the same binary is the safe home.
• Cap discovery frequency when clients are many. Removing the LB from the data path means every client now does discovery; with thousands of clients, a 15-second poll each is fine, but a shared cache or a watch API (Consul/etcd) beats polling at scale.

Client-side balancing trades a middlebox for client responsibility: you take over discovery, and you get per-backend connection reuse, no extra hop, and even distribution. For a fleet that can tolerate every client doing its own lookups, it is the simplest load-balancing design that actually respects what HTTP/2 was built for.