Why serialization keeps topping your pprof

Take any RPC-heavy Go service and run it under pprof -http=:8081. Nine times out of ten the same two functions dominate both the CPU and the allocation profile: the marshal path and the unmarshal path of encoding/json. Serialization is not an implementation detail. It is the single largest recurring cost in the request path, and every byte it allocates gets scanned by the GC on the next cycle. When we rewrote our internal event pipeline, we benchmarked three ways to move a struct across a wire — standard JSON, Protocol Buffers, and FlatBuffers — and the results reshaped how we pick formats. This is that write-up, mistakes included.

The three candidates work in fundamentally different ways:

• encoding/json walks the struct at runtime with reflection, boxing values into interface{} and writing into a growable buffer. Flexible, schema-optional, and allocation-happy.
• protobuf bakes the schema into generated code at compile time. Marshaling is a tight loop over field tags; the wire format is compact and evolves by field number rather than by name.
• FlatBuffers is a different animal. The builder writes fields directly into a byte buffer, and that buffer is the wire format. There is no parse step and no intermediate representation; readers index into the buffer and read fields in place.

A benchmark we can defend

We benchmarked on Go 1.25 on an M-series MacBook and validated on our c7g.4xlarge fleet. We replayed 100,000 real events through each format with -benchmem, and measured wire size with len(Marshal(...)). The message is deliberately realistic, not a toy: an Event with an id, a Unix-ms ts, a region, four readings, and three attrs:

The read column is the number that matters. Protobuf unmarshal is roughly 8.5x faster than json.Unmarshal and allocates a seventh as much, and the wire format is a quarter the size. FlatBuffers’ read path — field accessors plus a small loop over the readings vector — is sub-microsecond with zero allocations, because there is no decode to speak of. The build path is slower than protobuf, which surprises people; the builder’s scratch state and reverse-order vector writes cost more than protobuf’s generated fast path. FlatBuffers buys you nothing on the write side and everything on the read side, and only when reads vastly outnumber writes.

What idiomatic use actually looks like

The schema for the message above, in proto3:

1
2
3
4
5
6
7
8
9
10
syntax = "proto3";
package pipeline;

message Event {
  string id = 1;
  int64 ts = 2;
  string region = 3;
  repeated double readings = 4;
  map<string, string> attrs = 5;
}

The single biggest protobuf mistake we see in production is allocating a fresh output buffer per message. proto.MarshalOptions accepts an Out slice and appends to it, so the hot loop can reuse one buffer per worker:

1
2
3
4
5
6
7
8
9
func marshalEvent(ev *pipeline.Event, pool *sync.Pool) []byte {
	b, _ := pool.Get().([]byte)
	out, err := proto.MarshalOptions{Out: b[:0]}.Marshal(ev)
	if err != nil {
		panic(err)
	}
	pool.Put(out[:cap(out)])
	return out
}

Two gotchas that cost us days each:

• Maps are unordered. map[string]string fields serialize in nondeterministic order, so two structurally identical messages can produce different bytes. If you hash serialized output or compare messages byte-wise, use repeated message fields with explicit ordering instead.
• FlatBuffers prepends vectors in reverse. EventStartReadingsVector plus forward iteration silently produces a reversed vector; the convention is to walk the input backwards while prepending:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
func buildEvent(builder *flatbuffers.Builder, id, region string, ts int64, readings []float64) []byte {
	builder.Reset()
	idOff := builder.CreateString(id)
	regOff := builder.CreateString(region)

	pipeline.EventStartReadingsVector(builder, len(readings))
	for i := len(readings) - 1; i >= 0; i-- {
		builder.PrependFloat64(readings[i])
	}
	vec := builder.EndVector(len(readings))

	pipeline.EventStart(builder)
	pipeline.EventAddId(builder, idOff)
	pipeline.EventAddRegion(builder, regOff)
	pipeline.EventAddTs(builder, ts)
	pipeline.EventAddReadings(builder, vec)
	root := pipeline.EventEnd(builder)
	builder.Finish(root)
	return builder.FinishedBytes()
}

Note that the FlatBuffers Go accessors return value copies, not pointers into the buffer. That looks clumsy next to protobuf’s generated structs, but it is deliberate: you cannot accidentally retain a reference that outlives the buffer.

Where each one earns its keep

• JSON stays on the public edge. Payloads are small, traffic is modest, and curl-ability and schema drift tolerance are worth more than microseconds. Even the encoding/json v2 work in the stdlib pipeline does not change that calculation for us.
• Protobuf is our default for anything internal. The 8.5x read speedup and 4.5x size reduction compound across every hop in a service mesh. Keep protojson for debugging wire traffic; its output settled more than one cross-service field dispute.
• FlatBuffers is reserved for one pattern: read-heavy fanout where the decode step itself is the bottleneck. Our market-data fanout service pushes every update to hundreds of thousands of subscribers. Profiles showed 61% of CPU inside json.Unmarshal, and FlatBuffers removed the decode entirely — p99 went from 320 µs to 41 µs and allocs per message from 6 to 0. But we only adopted it after that profile proved the case. The DX cost is real: flatc codegen, a builder API that is the least pleasant in the Go ecosystem, and no built-in compression. It does not pay for itself on a request path with one writer and one reader.

The war story with the opposite sign: a teammate protobuf’d a low-traffic checkout API “for performance.” Payloads shrank, but every engineer lost curl-ability and gained a codegen step, for a service doing under 5k requests per second. We reverted it. Serialization is an allocation budget, not a technology contest.

The rule we apply

• Under ~10 KB payloads, need to debug it, or must tolerate schema drift: JSON, without apology.
• Internal RPC, long-lived schema, more than ~5k req/s: protobuf, always, with buffer reuse and gRPC.
• Read-heavy hot path where your pprof shows decode dominating: FlatBuffers, and only after the profile proves it.

Benchmark on your own messages with -benchmem, not just ns/op. Alloc count, not wall time, is what predicts your GC pause and your p99 under load. All three formats now coexist in our stack, and knowing exactly why each one is there is what keeps them all honest.