Deploying ZippChat on k3s (Proxmox)¶

Reference

Rendered from Server/docs/deployment-k3s.md on main at build time. Kept current with the code it describes, but not a protocol specification.

The path from nothing to a running cluster: OpenTofu provisions a Proxmox VM and installs k3s, a bootstrap script adds the cluster-level components the manifests assume, and Drone deploys on every push to main.

This replaces the older compose-over-SSH deploy. See Why not compose if you are wondering what happened to it.

0. Before you start¶

You need:

Thing	Why
A Proxmox host with an API token	OpenTofu creates the VM
`tofu` (or `terraform`), `kubectl`, `helm`	Local tooling
A DNS name pointing at the node	cert-manager issues TLS over HTTP-01
Drone with access to this repo	CI/CD

The Proxmox API token is equivalent to root on the hypervisor. It lives in terraform.tfvars, which is gitignored, as is the state file — state stores the token in cleartext.

1. Provision the node¶

cd Server/infra/tofu
cp terraform.tfvars.example terraform.tfvars
# fill in endpoint, token, node name, SSH key, and a STATIC vm_ip_cidr
tofu init
tofu apply

Use a static address. Drone deploys to it and the kubeconfig embeds it, so a changing DHCP lease breaks both.

cloud-init installs k3s with Traefik and ServiceLB disabled — ingress-nginx is used instead, because the Ingress manifests are written against its annotations and Traefik would ignore them silently rather than failing.

Pull the kubeconfig:

tofu output -raw fetch_kubeconfig   # prints the command; run it
export KUBECONFIG=$PWD/kubeconfig.yaml
kubectl get nodes

1b. Edge relay (private cluster behind a public VPS)¶

What is actually deployed: nginx on the VPS terminating TLS and re-encrypting to ingress-nginx over ZeroTier, with proxy_ssl_verify off, plus nftables for LiveKit's UDP. The HAProxy layer-4 variant below is the alternative — it keeps TLS entirely in the cluster, at the cost of running a second proxy on a host that already has one.

The cluster config differs between the two: the HTTP-proxy setup needs EDGE_FORWARDED_HEADERS=true and no cert-manager, and the layer-4 setup needs EDGE_PROXY_PROTOCOL=true plus the Ingress tls: block restored. They are mutually exclusive.

The cluster node has no public address, so a small VPS holds the public IP and relays to it over WireGuard. Full detail in infra/edge/README.md; the short version:

WireGuard is dialled from home, so CGNAT and un-forwardable routers are fine — nothing at home needs to be reachable inbound.
HAProxy on the VPS forwards TCP :80/:443 at layer 4, without decrypting. Certificates and keys stay in the cluster and cert-manager's HTTP-01 challenge works unchanged, because :80 reaches ingress-nginx.
LiveKit media is forwarded by nftables (UDP 7882, plus TCP 7881 via HAProxy). LiveKit uses a single UDP port precisely so this is two rules.

Ports that must reach the cluster¶

Whatever holds the public IP — the VPS above, or a router doing plain port-forwarding — has to pass all of these through. They are not interchangeable, and three of the six are easy to miss.

Port	Proto	Carries	Miss it and
80	TCP	HTTP→HTTPS redirect, ACME HTTP-01	certificates stop renewing after 90 days
443	TCP	Everything HTTP: all three vhosts + the gateway WebSocket	nothing works at all
7881	TCP	LiveKit ICE-TCP fallback	voice fails on UDP-blocking networks only
7882	UDP	LiveKit media — the normal path for a call	every call falls back to relay, or fails
3478	TCP+UDP	TURN/STUN control — where a client asks for a relay	symmetric-NAT clients cannot join
49160–49200	UDP	TURN relay media	TURN negotiates, then carries nothing

The last row is the one that bites. Port 3478 only hands a client an allocation; coturn answers with a relay address inside the range, and the audio itself flows there. Forward 3478 alone and the failure is a call that appears to connect and is silent — which reads as a codec or a mute bug, not a firewall one. The range is set by --min-port/--max-port in k8s/base/services/voice/coturn.yaml; shrink it there if 41 forwarding rules is unreasonable for your edge, since one port is consumed per concurrent relayed stream and only clients that cannot connect directly ever use one.

5349/TCP (TURN over TLS) is deliberately not in the table. coturn listens on it, but nothing advertises it, so forwarding it achieves nothing until iceServers() in services/voice/internal/voice/service.go offers a turns: URL.

Nothing else should be exposed. Postgres, Valkey, NATS, Vault and Garage publish host ports in dev.docker-compose.yml only; in the cluster they are ClusterIP and must stay that way.

If the edge proxies UDP at layer 4 (nginx stream, or HAProxy), coturn sees every client as arriving from the proxy. Relaying still works, but STUN can no longer tell a client its real public address — so more clients fall back to relaying than would otherwise, which makes the relay range matter more, not less.

# on the VPS, after installing /etc/wireguard/wg0.conf
sudo ./infra/edge/install-edge.sh

Costs, stated plainly: all traffic including video transits the VPS and counts against its bandwidth twice; media takes a longer path (client → VPS → home → VPS → client); the VPS is a single point of failure. Client IPs survive for HTTP via PROXY protocol, and are lost for UDP media.

2. Bootstrap the cluster¶

cd ../..            # back to Server/
export ACME_EMAIL=[email protected]
export REGISTRY_USER=... REGISTRY_PASSWORD=...
export EDGE_PROXY_PROTOCOL=true          # REQUIRED when behind infra/edge
./k8s/bootstrap/bootstrap.sh

EDGE_PROXY_PROTOCOL must match the edge. HAProxy prepends a PROXY protocol header, and ingress-nginx has to expect it. Mismatched in either direction and nothing serves at all: header sent but not expected, nginx reads PROXY TCP4 ... as a malformed request line; expected but not sent, it rejects every real request for lacking one.

Installs ingress-nginx (DaemonSet + hostPort, since there is no cloud load balancer), cert-manager and a letsencrypt-prod ClusterIssuer, External Secrets Operator, and the registry pull secret.

Skipping this produces failures that look like application bugs: Ingresses that never get an address, ExternalSecrets stuck SecretSyncedError, Deployments stuck ImagePullBackOff.

3. Apply the manifests¶

kubectl apply -k k8s/overlays/single-node

Everything needing a secret will fail at this point. That is expected — Vault is up but uninitialised and sealed, so External Secrets has nothing to read yet.

Use overlays/single-node, not base, on a single node. The base has HPAs with minReplicas: 3 and a PDB of minAvailable: 2; on one node the HPA scales services straight back up and the PDB makes the api pod permanently undrainable, hanging any node upgrade.

4. Initialise Vault¶

kubectl -n zipp-chat port-forward svc/vault 8200:8200 &
VAULT_ADDR=http://127.0.0.1:8200 ./infra/vault/prod-init.sh

This generates real secrets, sets up the AppRole that External Secrets uses, and prints the kubectl create secret command for the AppRole credentials. Run it, then watch the secrets sync:

kubectl -n zipp-chat get externalsecrets -w
kubectl -n zipp-chat rollout restart deploy    # pods started before secrets existed

Read this part¶

prod-init.sh writes unseal keys and the root token to vault-init-keys.json. Move it somewhere safe and delete it. There is no recovery path.

Vault holds the channel KEKs. Lose them and every server_escrowed message and every media file ever stored becomes permanently undecryptable — and the API answers 200 with an empty channel rather than an error, so nobody notices until far too late.

The services share one periodic Vault token, issued here. It dies if it goes 30 days unrenewed — and then every escrow-key fetch and voice join answers 500. The vault-token-renew CronJob renews it daily; if kubectl -n zipp-chat get jobs shows its runs failing, fix that within the month. A token that has already expired cannot be renewed: issue a new one with the root token and write it to secret/zipp-chat/{api,media,keyserver}.

The script writes each secret only if it is absent, so re-running it never rotates a key that is already encrypting stored data. That check is per key, not per path — so re-running it after a new secret is added to the repo fills in the missing one and leaves the rest alone.

It was per path until 2026-09-08, and that silently broke TURN on any cluster provisioned before TURN_SECRET was added to secret/zipp-chat/voice. Re-running printed exists — leaving it alone for the whole path and never wrote the new key. If your cluster was provisioned before then, re-run prod-init.sh once and restart the two consumers — neither reads a Secret change without it:
kubectl -n zipp-chat rollout restart deploy/voice deploy/coturn
Confirm it took by minting a voice token: ice_servers must be non-empty. An empty list with a 200 is exactly what a missing TURN_SECRET looks like, because iceServers() returns nothing rather than failing, and coturn runs happily with an empty --static-auth-secret while rejecting every credential.

Instance admins¶

The admin console lives at https://<api host>/admin/. Nobody is an admin until you say so, and there is deliberately no API for it — granting admin is a database write by whoever runs the box:

kubectl -n zipp-chat exec -it postgres-0 -- sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "UPDATE users SET instance_admin = true WHERE lower(username) = lower('"'"'<username>'"'"') AND deleted_at IS NULL"'

false revokes it, effective on the admin's next request.

Web Push keys (optional, and deliberately manual)¶

Push notifications need a VAPID key pair. prod-init.sh does not generate one, for two reasons: a VAPID pair is a long-lived identity rather than a secret that can be rotated freely — every browser subscription is bound to the applicationServerKey it was created with, so replacing the pair silently unsubscribes every user — and deriving a correct P-256 pair in shell is error-prone in a way that fails intermittently. A first attempt with openssl ec -text produced a 31-byte scalar; a key that is wrong one time in 256 is worse than one that is absent.

Generate one with the tool in this repo and write it once.

On a cluster node there is no Go toolchain — go run there is "command not found", which is how this instruction was found to be wrong. The generator now ships in the worker image, so the machine with kubectl can just run it:

kubectl -n zipp-chat exec deploy/worker -- /vapidgen

From a checkout on any machine with Docker, without installing Go:

docker run --rm -v "$PWD:/src" -w /src/services/worker golang:1.24-alpine \
  go run ./cmd/vapidgen

Then write both halves. patch, not put — vault kv put replaces the whole secret, and on a path that already holds other keys that silently destroys them:

kubectl -n zipp-chat port-forward svc/vault 8200:8200 &
VAULT_ADDR=http://127.0.0.1:8200 vault kv patch secret/zipp-chat/worker \
  VAPID_PUBLIC_KEY=... VAPID_PRIVATE_KEY=...

Until you do, worker-secrets shows as not-ready in kubectl -n zipp-chat get externalsecrets. That is expected, not a fault. The worker's secretRef for it is optional: true, so the pod still starts and its NATS consumers and cleanup loop run normally; it logs that Web Push is disabled, GET /api/device-tokens/vapid-key answers 503, and the client shows "This server has not set up push notifications" rather than a broken toggle. Everything except push works.

Nothing else needs doing after writing it — External Secrets syncs within refreshInterval (1h), or immediately on kubectl -n zipp-chat rollout restart deploy/worker once the Secret exists.

Outgoing mail (verification and forgot-password links)¶

The api mails two things and nothing else: an address-verification link after sign-up, and a forgot-password link to verified addresses. Settings are in zipp-chat-config (SMTP_HOST, SMTP_PORT, SMTP_SECURITY, MAIL_FROM, PUBLIC_URL); the login is in Vault. The relay is the operator's Mailcow at mail.zigz.uk, submission on 587 with STARTTLS — the mailer refuses to continue if STARTTLS is not offered, rather than sending the password in the clear.

1. In Mailcow, add zippchat.app as a domain, create the mailbox [email protected] (its address and password are the SMTP login), and generate a 2048-bit DKIM key for the domain (Configuration → ARC/DKIM keys, selector dkim).

2. DNS on Cloudflare for zippchat.app (all DNS-only, there is nothing to proxy):

Type	Name	Content
MX	`zippchat.app`	`mail.zigz.uk`, priority 10 — so bounces and DMARC reports have somewhere to go
TXT	`zippchat.app`	`v=spf1 a:mail.zigz.uk -all`
TXT	`dkim._domainkey.zippchat.app`	the `v=DKIM1;k=rsa;p=…` value Mailcow shows for the domain
TXT	`_dmarc.zippchat.app`	`v=DMARC1; p=quarantine; adkim=s; aspf=s; rua=mailto:[email protected]`

and, because the reports go to another domain, on zigz.uk:

Type	Name	Content
TXT	`zippchat.app._report._dmarc.zigz.uk`	`v=DMARC1`

Nothing else sends as zippchat.app, which is why SPF ends in -all and DMARC can start at quarantine. Check with dig +short TXT dkim._domainkey.zippchat.app and by mailing a test address you own at mail-tester.com or similar before turning it on for users.

3. The login, into Vault — put is right here, this path holds nothing else:

kubectl -n zipp-chat port-forward svc/vault 8200:8200 &
VAULT_ADDR=http://127.0.0.1:8200 vault kv put secret/zipp-chat/smtp \
  SMTP_USERNAME=[email protected] SMTP_PASSWORD='<the mailbox password>'
kubectl -n zipp-chat annotate externalsecret smtp-secrets force-sync=$(date +%s) --overwrite
kubectl -n zipp-chat rollout restart deploy/api

Until then smtp-secrets is not-ready in kubectl get externalsecrets, as with worker-secrets; the api starts anyway (optional: true) and logs a relay error for each mail it tries. To turn mail off entirely, set SMTP_HOST: "" — sign-up is unaffected and clients hide the banner.

Verify on the cluster: kubectl -n zipp-chat logs deploy/api | grep "mail enabled", then sign up with an address you own. In dev every message lands in Mailpit at http://localhost:8025, and scripts/verify-email.sh runs the whole flow.

When a deploy has not happened¶

Check the running image before anything else.

kubectl -n zipp-chat get deploy -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.containers[0].image}{"\n"}{end}'

A missing feature on the cluster is far more often an image that never built than a configuration problem, and the two look identical from outside: the endpoint answers, it just answers as though the feature does not exist.

This is not hypothetical. On 2026-09-08 golang-migrate released v4.20.0 requiring Go >= 1.25.11, the Dockerfiles installed it with @latest, and every image build failed from that moment with no change to this repo. The cluster kept serving the last thing that built. Nothing was down, nothing alerted, and a week of merged work sat in main running nowhere. It surfaced as one person unable to join a voice call, and two rounds of diagnosis went into Vault and ConfigMaps for a feature that was not in the running binary at all.

The Dockerfiles are pinned now. The habit still matters: "pushed to main" is not "deployed", and one command tells you which you have.

Unsealing after a reboot¶

There is no auto-unseal. That needs a cloud KMS or a second Vault, and neither exists on a single self-hosted node. Vault therefore starts sealed after every restart:

kubectl -n zipp-chat exec -it vault-0 -- vault operator unseal   # twice, two different shares

Until then, everything needing a key fails. This is the deliberate trade for not storing the unseal keys next to the data they protect. If that becomes intolerable, the honest options are a hardware token, a second Vault with transit auto-unseal, or accepting the weaker posture of an unseal-on-boot sidecar — not going back to -dev mode.

5. CI/CD¶

Drone secrets required:

Secret	Value
`registry_user`, `registry_password`	Registry credentials
`kubeconfig`	The kubeconfig from step 1, server address rewritten to the node IP

The vps_host / vps_user / vps_ssh_key secrets from the compose deploy are no longer used and can be deleted.

On push to main: test → build and push five images → kustomize edit set image to the commit SHA → kubectl apply -k → wait for rollouts → roll back all services if any fails.

It applies the manifests, not just the images. An earlier version ran kubectl set image alone, which meant no ConfigMap, env var or Service change ever reached the cluster from CI. That failed the hard way: an api image requiring CORS_ALLOWED_ORIGINS was deployed against a ConfigMap without it and crash-looped, while every other service rolled out fine.

Images are pinned to the commit SHA, not latest. latest gives no way to tell what is running, no way to roll back, and since the tag does not change, Kubernetes may not restart the pod at all.

The web client deploys from Client-Flutter/.drone.yml, and the public website (site Deployment) from ProjectInfo/.drone.yml — not this pipeline. Neither image names a domain: the web client talks to the origin that served it.

The API certificate must cover every address you point kubectl at¶

k3s signs its API server certificate for the node's own IP, 10.43.0.1 and loopback — nothing else. A kubeconfig pointing at a ZeroTier/WireGuard address or a DNS name therefore fails with:

tls: failed to verify certificate: x509: certificate is valid for
10.20.30.40, 10.43.0.1, 127.0.0.1, ::1, not 10.24.25.28

This usually surfaces as a CI deploy that cannot reach the cluster, after every image has already built and pushed.

Add the addresses to /etc/rancher/k3s/config.yaml:

tls-san:
  - 10.24.25.28

The certificate is not regenerated on its own. Delete it and restart:

sudo rm -f /var/lib/rancher/k3s/server/tls/serving-kube-apiserver.{crt,key}
sudo systemctl restart k3s

Workload pods survive a k3s restart — but Vault does not stay unsealed if its pod restarts, so check it afterwards.

Any kubeconfig copied out before this change still works; the client verifies the certificate it is served, not the one it was issued alongside.

Configuration that will stop the api booting¶

CORS_ALLOWED_ORIGINS is required whenever ENVIRONMENT is not development. The api logs a fatal and exits rather than falling back to a permissive default.

That is deliberate. Allowing every origin would grant any site on the internet credentialed cross-origin access to the API, and nothing would visibly break — so it would never be noticed. Refusing to start is the only failure mode that gets seen.

It is written in for you from the public host (below): the web client is served from that origin under /app/. Add further origins, comma separated, only if some other site really must call the API.

The public host — one file¶

Everything is served on one host: the website at /, the web client at /app, the API at /api/ and /gateway, media at /media/. LiveKit signaling is on its own host. Both are set in exactly one place:

# k8s/base/public-host.env
PUBLIC_HOST=www.zippchat.app
PUBLIC_LIVEKIT_HOST=livekit.zippchat.app

k8s/base/kustomization.yaml copies them (kustomize replacements) into the Ingress's two host: fields, CORS_ALLOWED_ORIGINS and LIVEKIT_PUBLIC_URL. The manifests themselves say public-host.invalid, so an unreplaced value is obvious. The web image has no domain compiled in, and the Cloudflare tunnel config (k8s/tunnel/) forwards everything to ingress-nginx, so neither names a domain either. The edge (DNS, TLS) still has to serve the host and the LiveKit host, and redirect the bare apex to www. LiveKit is livekit.zippchat.app, not livekit.www.…, so that one *.zippchat.app certificate covers both names.

Known gaps¶

Voice needs the edge addresses set. Two separate values, both pointing at the VPS: LIVEKIT_PUBLIC_URL (the signaling WebSocket, returned to clients as livekit_url; derived as wss://<PUBLIC_LIVEKIT_HOST> from base/public-host.env) and LIVEKIT_NODE_IP (where LiveKit tells browsers to send media; in overlays/single-node/patches/single-node-config.yaml). LIVEKIT_HOST stays internal — it is the server-to-server address. Getting these wrong gives a call that connects and carries no audio, with every API call still returning 200. The LiveKit pod refuses to start if LIVEKIT_NODE_IP is unset, which is deliberate.
No backups. Postgres, Garage and Vault all sit on local-path PVCs on one disk. Nothing here makes a copy of anything.
Single node means single point of failure. Fine for a test box; the base overlay exists for when it stops being one.
Parts of this document are still unexercised. The manifests, the Vault init and the edge relay are now proven on a live cluster; the automation around them is not. Specifically:

Piece	State
`k8s/overlays/single-node`	Applied and running
`infra/vault/prod-init.sh`	Run for real
Edge relay (VPS → cluster)	Working, but built as nginx + ZeroTier by hand, not via `infra/edge/install-edge.sh`
`infra/tofu/`	Never applied — the first node was built by hand
`k8s/bootstrap/bootstrap.sh`	Never run — helm commands were issued individually

So the OpenTofu module and the two convenience scripts remain statically validated only. Treat them as drafts until someone rebuilds a node from scratch with them.

Why not compose¶

The previous deploy ran docker compose up over SSH against Server/docker-compose.yml. That file runs Vault in -dev mode:

VAULT_DEV_ROOT_TOKEN_ID: ${VAULT_TOKEN:-dev-root-token}

In-memory, re-seeded on every start. Its own comment says so. On any restart of that stack, every channel KEK is regenerated and all escrowed history becomes unreadable — silently.

The k8s path uses a real Vault with file storage on a persistent volume, which is why docker-compose.yml should be treated as a local convenience only, and never as a deploy target.