The platform bill grows linearly with a problem you did not choose: a few years of real traffic later, the Heroku/Render invoices looked like a third engineering salary, and Kubernetes was a fourth. Kamal (renamed from MRSK in December 2023) occupies the gap: Docker-based deploys to plain VMs over SSH, Traefik for routing and TLS, zero-downtime rolling deploys, and one kamal deploy to rule them all. What makes it production-safe for us is that the entire surface is a private Tailscale overlay — no public IPs, no exposed SSH, no world-visible admin ports.

The topology

Three Hetzner VMs, each running only Tailscale and Docker. config/deploy.yml addresses them by their 100.x Tailscale addresses:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# config/deploy.yml
service: checkout
image: sijin/checkout

servers:
  web:
    hosts:
      - 100.64.0.15   # tailscale app-1
      - 100.64.0.16   # tailscale app-2
    env:
      secret: true
      clear:
        RAILS_ENV: production

registry:
  server: registry.tailnet.ts.net
  username: deploy
  password:
    - KAMAL_REGISTRY_PASSWORD

traefik:
  host_port: 443
  options:
    publish:
      - "443:443"

healthcheck:
  path: /up
  interval: 2s
  timeout: 3s

hooks:
  before-docker:
    - bin/rails assets:precompile

The 100.x addresses are Tailscale’s CGNAT range. The VMs accept SSH only from admin machines and CI runners, and Tailscale ACLs define who may even reach the box — so kamal deploy from a laptop is indistinguishable from kamal deploy from GitHub Actions, and nothing is exposed to the internet.

Healthchecks that mean something

Kamal’s default healthcheck hits /up, which we point at a database-backed endpoint. “The container booted” is not “the app is ready”, and a deploy that mounts broken database state must not receive traffic:

1
2
3
4
5
6
7
8
9
10
11
12
# config/routes.rb
get "/up", to: "health#show"

# app/controllers/health_controller.rb
class HealthController < ApplicationController
  def show
    ActiveRecord::Base.connection.select_value("SELECT 1")
    head :ok
  rescue ActiveRecord::ConnectionNotEstablished
    head :service_unavailable
  end
end

We once deployed a revision with a broken migration on a Friday; the healthcheck caught it within the 2-second interval and Kamal rolled back to the previous container before the load balancer noticed. That is the entire value of a database-aware healthcheck.

Private images, fast deploys

Image pulls from Docker Hub on a cold VM cost 15–20 seconds each and rate-limit under bursts. Running a local registry on the tailnet — a small container only the VMs can reach — cut our push-to-live deploy from about 90 seconds to about 40, and a first pull becomes sub-second because the registry is one WireGuard hop away. Tailscale’s overhead is a few percent of CPU and a couple of milliseconds of added latency per hop; on a Rails app it is unmeasurable in p99.

The production checklist

• Registry on the tailnet. Keeps images private without a cloud registry bill and makes deploys fast.
• ACLs, not firewall hacks. Tailscale ACLs replace half a dozen ufw rules; SSH and Docker are reachable only from the admin group and CI.
• Log rotation. Docker containers spew JSON logs forever. Rotate /var/lib/docker/containers/**/*-json.log with logrotate; a month without it filled a 40 GB disk.
• Traefik TLS. Let’s Encrypt via Traefik means no certificate choreography; just keep the healthcheck path outside any auth so Kamal and the load balancer both get a clean /up.
• Disk headroom for rollbacks. Kamal keeps the previous image to roll back. Budget at least one app image per server in kamal prune policy — we keep --keep-last 2.

Deploys and secrets

A deploy is kamal deploy from CI: it builds the image on the runner, pushes to the private tailnet registry, then connects to each host over Tailscale and starts new containers behind Traefik. Kamal’s rolling update brings up the new version, waits for the /up healthcheck to pass, and only then reroutes traffic and prunes the old container — which is how “zero downtime” actually behaves on a 3-server fleet. Secrets live in config/env/with-secrets (gitignored) and are passed to the containers at run time, not baked into the image, so a registry leak does not leak credentials.

What we gave up

No autoscaling, no managed Postgres, no magic. When a VM dies, Docker restarts containers and Tailscale routes around it, but horizontal scaling is you running kamal deploy to a box you added to the hosts list. For a steady-state Rails workload that is the right trade: the platform bill dropped to about an eighth of what it was, deploys are repeatable, and the whole fleet is invisible to the public internet. That last part is the security posture most “cloud-native” setups never actually achieve.