Shade/docs/PRODUCTION-CHECKLIST.md

# Shade Production Checklist

A flat punch-list for taking a Shade prekey server from "it boots" to
"production-ready". Every item below is a hard gate — if you can't tick it,
don't ship.

The deeper "why" behind each item lives in `THREAT-MODEL.md`,
`SECURITY.md`, and `docs/DEPLOYMENT.md`. This file is the operator's
checklist.

> Scope: a single Shade prekey container (`@shade/server`) plus any
> consumer apps that talk to it. For E2EE file transfer hardening
> (max-size, retention, quotas), see the **Hardening** and **Retention**
> sections of `docs/streams.md`.

---

## 1. TLS termination

- [ ] Public traffic is **TLS 1.2+ only** — Shade itself speaks plain HTTP
      and assumes a reverse proxy (Caddy, Traefik, nginx, Dokploy's
      built-in proxy) terminates TLS in front of it.
- [ ] HSTS is on (`Strict-Transport-Security: max-age=15552000`).
- [ ] The proxy is configured to pass the original `Host` header through
      so signed payloads bound to the canonical address don't trip the
      replay-window check on a mismatch.
- [ ] Internal traffic between consumer apps and the prekey container
      runs on a private network (Docker bridge / VPC); the prekey port
      is **not** exposed to the public internet without TLS in front.

> **Why:** identity signatures and observer bearer tokens travel in
> request bodies / headers. Without TLS, a network attacker can read
> the observer token and replay it for the full validity window, and
> can read the metadata (who registers, who fetches whose bundle).
> See `THREAT-MODEL.md § 1` (network attacker).

## 2. Backups

- [ ] **SQLite:** scheduled `sqlite3 /data/shade-prekeys.db ".backup ..."`
      at least daily. The `.db` file plus `-wal` and `-shm` together is
      the recovery unit; never copy the bare `.db` while the container
      is running without using the online backup API.
- [ ] **Postgres:** `pg_dump` (or your provider's snapshot) at least
      daily; verify a restore at least once per quarter.
- [ ] Backups are stored on different infrastructure than the primary
      volume (different host / region / provider).
- [ ] Backups are encrypted at rest (your storage provider's
      server-side encryption, age, or restic with a passphrase).
- [ ] **Restore drill:** at least once before going live, restore the
      backup into a fresh volume and confirm `/health` is green and a
      registered identity is still resolvable.

> **Why:** prekey records contain identity public keys and one-time
> prekeys. Losing them means new sessions can't be established to those
> identities until each user re-registers. Existing sessions keep
> ratcheting on the device-side state.

## 3. Observer token rotation

- [ ] `SHADE_OBSERVER_TOKEN` is set to **≥ 16 chars** of high-entropy
      random data (e.g. `openssl rand -hex 32`). The server logs a
      warning and disables the observer if the token is shorter.
- [ ] The token is held in your secret manager (Dokploy secret, GitHub
      Actions secret, Vault, 1Password CLI), **never** committed to a
      compose file or `.env` checked into git.
- [ ] The token is rotated on a schedule (recommended: every 90 days)
      and immediately if it has been shared with anyone who no longer
      needs access.
- [ ] If you expose the dashboard publicly, you also gate it behind
      basic-auth at the proxy layer — bearer tokens are not
      revocation-friendly on their own.

> **Why:** the observer dashboard exposes metadata about every active
> identity, registration timestamp, and recent activity. Anyone with
> the token can scrape the entire prekey directory.

## 4. SQLite vs PostgreSQL

Pick one and stick to it.

- [ ] **SQLite** is the default. Use it when **one** Shade container is
      enough, you can tolerate downtime during backup snapshots, and
      your write rate is below ~500 req/s. Path: `SHADE_PREKEY_DB_PATH`,
      default `/data/shade-prekeys.db`.
- [ ] **PostgreSQL** is for multi-replica deployments, shared
      infrastructure, or when you already operate a managed Postgres
      and want one fewer thing to back up. Path: `SHADE_PREKEY_PG_URL`.
      Tables are auto-created with `shade_server_*` prefix.
- [ ] Whichever you pick, the database lives behind TLS for the
      connection (`sslmode=require` for Postgres) and on storage that
      is itself encrypted (LUKS, EBS encryption, managed-DB encryption).
- [ ] You do **not** mix them in the same deployment. Setting
      `SHADE_PREKEY_PG_URL` overrides SQLite silently — pick one in
      `compose.yml` and document which.

> **Why:** Shade does **not** encrypt the database itself (V3.2 will).
> Disk-level / volume-level encryption is the operator's responsibility
> until at-rest encryption ships.

## 5. Log level and structured logs

- [ ] `SHADE_LOG_LEVEL` is set to `info` (production) or `warn`
      (high-traffic). Avoid `debug` in prod — it logs request bodies
      including signed payloads.
- [ ] Logs are shipped to a retention-bounded sink (Loki, CloudWatch,
      Datadog) with **redaction of `Authorization` headers and signed
      bodies** if your sink doesn't already strip them.
- [ ] You alert on `error`-level logs and on the absence of cleanup
      cycles (a stuck cleanup loop = unbounded DB growth).

> **Why:** at `debug` level the server logs signature material. While
> Ed25519 signatures are not secrets per se, leaking them widens the
> replay-window blast radius and reveals timing patterns.

## 6. Stale-identity cleanup parameters

- [ ] `SHADE_STALE_DAYS` is set deliberately for your product. The
      default (30 days) is right for "active chat app"; "occasional
      use" apps should bump to 90+ to avoid surprise re-registration.
- [ ] `SHADE_CLEANUP_INTERVAL_HOURS` is left at 24 unless you have a
      specific reason — running cleanup more often does not free more
      space, and running it less often risks one cycle missing a day.
- [ ] You watch the `shade_cleanup_purged_total` metric (Prometheus) and
      alert on a sudden 10× spike — that often signals a bug or a
      deployment that broke client-side activity timestamps.

> **Why:** stale cleanup is the only thing keeping the prekey directory
> from growing forever. A misconfigured `SHADE_STALE_DAYS = 0` would
> nuke every identity on every cycle. Bound the value at ≥ 1 in your
> deployment config.

## 7. Secret rotation

- [ ] Identity signing keys: each consumer rotates via the documented
      identity-rotation flow (7-day grace period for old sessions).
      Operators do **not** touch identity keys directly.
- [ ] Observer token: see § 3.
- [ ] Database credentials (Postgres only): rotate per your standard
      cadence, with the connection string supplied through the secret
      manager.
- [ ] No long-lived API keys or service tokens are stored in the
      container image or volume.

## 8. Rate-limit and body-size caps

- [ ] You have not lowered the built-in rate limits below the defaults
      (per-IP register/bundle and per-identity replenish/delete).
- [ ] You have not raised the 64 KiB POST body limit. Prekey bundles
      fit comfortably; raising the limit only enables abuse.
- [ ] Your reverse proxy enforces an additional connection / request-
      rate limit at the edge (Caddy `rate_limit`, Cloudflare, etc.)
      so a single noisy IP can't even reach Shade's per-route limits.

## 9. Health checks and metrics scrape

- [ ] Container has a Docker `HEALTHCHECK` (the official image already
      ships one against `/health`).
- [ ] `/metrics` is scraped by Prometheus / OpenTelemetry and
      retained ≥ 30 days.
- [ ] Alerts are wired for: `/health` failing for > 2 min, request
      latency p99 > 1 s, error rate > 1 %, cleanup cycles missing for
      > 25 h.

## 10. OpenAPI contract drift

- [ ] CI runs the OpenAPI lint (`bun test packages/shade-server/tests/openapi-lint.test.ts`)
      on every PR — the spec must remain valid OpenAPI 3.1 with no
      dangling `$ref`s.
- [ ] Generated clients (Python, Go, Kotlin) are regenerated from the
      shipped spec on each release; mismatches between server and
      client are caught at integration test time, not production.

---

## Pre-flight summary

If you can answer "yes" to every box above, ship it. If you can't,
write down which box and why before you do — that note belongs in your
runbook so the next operator inherits the gap, not the surprise.