Kubernetes terminates pods the same way every time: it sends SIGTERM, waits terminationGracePeriodSeconds (30 by default), then sends SIGKILL. If your service treats SIGTERM as “start dying right now”, every deploy loses a slice of in-flight traffic — failed webhook deliveries, half-finished transactions, queue messages that were taken off the broker but never acked. At a few deploys a day, that’s a small but steady trickle of corruption.

The contract we actually want on shutdown:

1. Stop accepting new work first.
2. Drain in-flight work within a fixed budget.
3. Tear down outbound dependencies (DB pool, producers) only after inbound is quiet.
4. Exit non-zero if the budget is blown, so the orchestrator knows the drain failed.

The last point is the one people skip, and it matters: a silent timeout exit looks like success to the platform.

Here is the shape we use in every HTTP service:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
package main

import (
	"context"
	"errors"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	srv := &http.Server{
		Addr:              ":8080",
		ReadHeaderTimeout: 5 * time.Second,
	}

	errCh := make(chan error, 1)
	go func() { errCh <- srv.ListenAndServe() }()

	select {
	case err := <-errCh:
		if !errors.Is(err, http.ErrServerClosed) {
			log.Fatalf("server: %v", err)
		}
	case <-ctx.Done():
		log.Println("shutdown signal received")
	}

	drainCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
	defer cancel()

	if err := srv.Shutdown(drainCtx); err != nil {
		log.Printf("drain timed out: %v", err)
		os.Exit(1)
	}
	log.Println("HTTP drained, closing outbound dependencies")
	closeOutbound() // DB pool, message producer
}

signal.NotifyContext (Go 1.16+) collapses signal handling and cancellation into one mechanism, and http.Server.Shutdown stops the accept loop, waits for active handlers, then returns. Everything after that must happen in order: outbound dependencies only after inbound work is done.

The ordering lesson cost us a production incident. We used to close the DB pool before draining HTTP. Under normal load the gap is milliseconds and nobody notices; at 5k RPS with a ~300 ms average handler, the drain took seconds, handlers hit a closed pool, and every deploy produced a burst of 500s. Reordering — drain HTTP fully, then close the pool, then the producer — took the error rate from 0.4% to 0 during deploys, and unacked queue messages dropped from ~2k per release to zero.

A few things worth baking in:

• Background workers need the same treatment. Draining HTTP is not draining the service. If a worker goroutine is mid-item when you shut down, closing the jobs channel lets it finish the current item, then exit; joining workers with the server drain in one sync.WaitGroup (or errgroup) means the process never exits while work is genuinely in flight.
• Register one-off cleanup with srv.RegisterOnShutdown. It runs after connections are drained and is the right place for idempotent flushes. Anything registered there must be fast — it happens inside the drain budget.
• Do the timeout math against Kubernetes. With a 15 s drain, set terminationGracePeriodSeconds to 30, not 15: K8s adds its own bookkeeping, and the moment SIGKILL lands mid-write you have lost the game anyway. We also add a short preStop sleep (5 s) so the load balancer deregisters the pod before SIGTERM arrives; otherwise a draining pod can still receive brand-new traffic for a second or two.
• Handlers that spawn their own goroutines must propagate cancellation themselves. Shutdown waits on the handler, not on whatever the handler fired off; a fire-and-forget goroutine can outlive the process. Thread the request context through and make child work observe it.
• If the drain times out, log loudly and exit(1). The platform will SIGKILL you regardless; a non-zero exit at least makes it visible in your deploy tooling.

Graceful shutdown is a contract with your orchestrator, not a nice-to-have. Get the ordering right, size the budget honestly against your slowest legitimate request, and deploy becomes invisible to your users.