release(v4.0.0): Shade GA — V3.x consolidation + audit prep

V3.1 → V3.12 consolidated and tagged for the first GA release. Wire format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers byte-for-byte. The version bump is semantic: audit-cycle complete, opt-in surface fully exposed, threat model refreshed for every new surface. Highlights: - All 24 @shade/* packages bumped to 4.0.0 in lockstep. - CHANGELOG 4.0.0 section is the canonical manifest of what landed. - THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12 Web-Worker boundary) + residual-risks table refreshed. - OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox, bridge, observer, /metrics, /healthz, /ready. - MIGRATION 0.3.x → 4.0 documented + smoke-tested against shade migrate-storage on a real SQLite DB. - docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer. - scripts/soak.ts harness for the GA-stable 2-week soak window. - All V*.md plans archived under docs/archive/ with Status: Done. - Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen non-realtime stack. Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green. Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports version 4.0.0 on /health. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
parent 8b055912b7
commit e6fdf31b49
298 changed files with 37909 additions and 256 deletions
--- a/docs/inbox.md
+++ b/docs/inbox.md
@@ -0,0 +1,317 @@
+# Shade Inbox — Async Store-and-Forward (V3.6)
+
+A relay that holds **ciphertext blobs with TTL** so senders can deliver
+to recipients who happen to be offline. The relay never sees plaintext,
+never holds private keys, and never knows who is talking to whom in
+plaintext form (only addresses and bytes-per-blob).
+
+This document covers:
+
+- Setup (server side, single-binary)
+- Client integration (`@shade/inbox`)
+- Threat model — *what the relay actually sees*
+- Operational tuning (TTL, quotas, prune cadence)
+- Wire-level reference
+
+---
+
+## 1. Server setup
+
+The inbox server is built into the same `@shade/server` standalone
+container that ships the prekey server, on the same port. Routes are
+namespaced under `/v1/inbox/*`.
+
+### Docker (single binary, both services)
+
+```bash
+docker run -d --name shade \
+  -p 3900:3900 \
+  -v shade-data:/data \
+  -e SHADE_PREKEY_DB_PATH=/data/shade-prekeys.db \
+  -e SHADE_INBOX_DB_PATH=/data/shade-inbox.db \
+  -e SHADE_INBOX_PRUNE_INTERVAL_MINUTES=5 \
+  ghcr.io/zyon-no/shade:latest
+```
+
+### Postgres (multi-instance / shared infra)
+
+```bash
+docker run -d --name shade \
+  -p 3900:3900 \
+  -e SHADE_PREKEY_PG_URL='postgres://shade:***@db/shade' \
+  -e SHADE_INBOX_PG_URL='postgres://shade:***@db/shade' \
+  ghcr.io/zyon-no/shade:latest
+```
+
+Tables are auto-created (`shade_inbox_owners`, `shade_inbox_blobs`,
+sequence `shade_inbox_seq`). If you only set `SHADE_PREKEY_PG_URL`, the
+inbox falls back to the same database; set
+`SHADE_INBOX_PG_URL='-'` to disable that fallback and run the inbox
+in-memory (only useful for short-lived test deployments).
+
+### Env vars
+
+| Var                                    | Default                  | Effect                              |
+| -------------------------------------- | ------------------------ | ----------------------------------- |
+| `SHADE_INBOX_DB_PATH`                  | _(unset → memory)_       | SQLite file path                    |
+| `SHADE_INBOX_PG_URL`                   | _(unset → falls back)_   | Postgres connection string          |
+| `SHADE_INBOX_PRUNE_INTERVAL_MINUTES`   | `5`                      | How often expired blobs are dropped |
+
+### Embedding in your own Hono app
+
+```ts
+import { Hono } from 'hono';
+import { SubtleCryptoProvider } from '@shade/crypto-web';
+import { createInboxRoutes, MemoryInboxStore } from '@shade/inbox-server';
+
+const crypto = new SubtleCryptoProvider();
+const store = new MemoryInboxStore();
+
+const app = new Hono();
+app.route('/', createInboxRoutes(store, crypto));
+
+export default { port: 3901, fetch: app.fetch };
+```
+
+---
+
+## 2. Client integration
+
+`@shade/inbox` is the recipient/sender SDK. It composes on top of
+`@shade/sdk` — Shade still owns encryption + the ratchet; the inbox
+layer is just durable transport.
+
+### Wiring
+
+```ts
+import { Shade } from '@shade/sdk';
+import { Inbox } from '@shade/inbox';
+
+const shade = new Shade(/* ... */);
+await shade.initialize();
+
+// Lift the identity keys we already have.
+const identity = await shade.getManager().getIdentityKeyPair();
+
+const inbox = new Inbox({
+  baseUrl: 'https://inbox.example.com',
+  ownAddress: shade.myAddress,
+  crypto: shade.crypto,
+  signingPrivateKey: identity.signingPrivateKey,
+  signingPublicKey: identity.signingPublicKey,
+  pollIntervalMs: 30_000,
+});
+
+// Receive: hand each fetched blob to Shade.receive.
+inbox.onIncoming(async (raw) => {
+  const envelope = decodeEnvelope(raw.ciphertext);
+  // The inbox does not authenticate the sender — Shade.receive does,
+  // by way of the recipient's session/ratchet/identity-pin.
+  const senderAddress = /* derive from your own metadata channel */;
+  await shade.receive(senderAddress, envelope);
+  return senderAddress;
+});
+
+inbox.start(); // registers + begins flush + poll loops
+
+// Send: encrypt with Shade, hand the envelope to the inbox.
+const envelope = await shade.send('bob@example.com', 'hi');
+await inbox.send({ recipientAddress: 'bob@example.com', envelope });
+```
+
+### Push-trigger hook
+
+The inbox is *pull-based* — recipients only see new blobs when they
+poll. Most apps want a wake-up nudge when new content lands. Vendor it
+yourself (FCM / APNs / email / WebPush):
+
+```ts
+inbox.onMessageQueued(async (recipient, msgId) => {
+  await fcm.send(recipient, { kind: 'shade-inbox', msgId });
+});
+```
+
+The recipient device wakes, runs `inbox.tick()`, and pulls the blob.
+
+### Durable queue
+
+The default in-memory queue is fine for short-lived processes. For a
+service that must survive restart, plug in your own `OutgoingQueueStore`
+backed by SQLite/Postgres/IndexedDB:
+
+```ts
+const inbox = new Inbox({
+  // …
+  queueStore: new MyDurableQueueStore(),
+  cursorStore: new MyDurableCursorStore(),
+});
+```
+
+Same idea for the receive cursor — without persistence, every restart
+re-downloads everything currently within TTL.
+
+### Errors
+
+- **Decrypt failure** in your handler keeps the blob on the server (no
+  ack). The next poll re-fetches it — useful when the ratchet temporarily
+  rejects a message because of out-of-order delivery.
+- **`msgId/ciphertext` mismatch** is a relay-tampering canary. The Inbox
+  client recomputes the hash and emits `inbox.message_decrypt_failed`
+  *without* acking, so an operator can investigate before the blob
+  silently expires.
+- **Network failure** on PUT keeps the entry in the local queue with an
+  `attempts` counter; default cap is 10 retries before the entry is
+  dropped (configurable via `maxAttempts`).
+
+---
+
+## 3. Threat model — what the relay actually sees
+
+| Knows                                              | Doesn't know                              |
+| -------------------------------------------------- | ----------------------------------------- |
+| Recipient address (path parameter)                 | Recipient real identity (it's pseudonymous) |
+| Sender's per-PUT signing public key                | The mapping sender-pubkey → real identity |
+| Number of blobs queued for an address              | Plaintext content                          |
+| Approximate ciphertext size                        | Sender-recipient pair beyond bytes-pari   |
+| Per-blob TTL (in the row's `expires_at`)           | The ratchet/X3DH state                    |
+
+### Privacy posture
+
+- **Sender-recipient graph leaks at the byte-pari level.** A passive
+  observer of the relay (or its DB dump) can correlate sender pubkey ↔
+  recipient address ↔ blob size. Mitigations:
+  - Recipients can use **address hashes** instead of human-readable
+    addresses (the address grammar accepts any `[a-zA-Z0-9][a-zA-Z0-9:_\-.]{0,255}`,
+    so `sha256(real-address || salt)` works).
+  - Senders can rotate their per-PUT signing key per session; the relay
+    only verifies the signature and never persists the key.
+- **TTL leaks reachability.** A sender's PUT silently dropping after 7
+  days is itself a signal. Operators can normalize TTLs (clamp every
+  PUT to a fixed 7-day window) to flatten this.
+- **Operator can DoS a recipient** by deleting their queue. Mitigation:
+  recipient ack happens *after* successful decrypt, so a malicious
+  delete just forces re-send by the original sender.
+
+### What the relay can NOT do
+
+- **Read plaintext** — the ratchet/AEAD layers run client-side.
+- **Forge a sender** — every PUT is Ed25519-signed by the sender's
+  per-PUT key; the relay rejects bad signatures with 401.
+- **Inject a foreign blob** — the recipient client recomputes
+  `sha256(ciphertext)` and refuses anything that doesn't match the
+  stored `msgId`.
+- **Replay an old PUT** — the signed `signedAt` field has a ±5-minute
+  window (matches the prekey-server's policy); replays past that window
+  return 409.
+
+### Storage-DoS
+
+`maxBlobBytes` (default 1 MiB) caps a single PUT.
+`maxBlobsPerAddress` (default 1000) caps the recipient's queue depth —
+PUTs past the cap return 400 with a structured `inbox.quota_rejected`
+event so operators can alert. Combine with per-IP rate limits at the
+edge (the built-in token bucket is in-memory and not multi-instance).
+
+---
+
+## 4. Wire reference
+
+All bodies are JSON. Multi-byte fields are base64-standard encoded.
+
+### `POST /v1/inbox/register` (TOFU)
+
+```json
+{
+  "address": "bob",
+  "signingKey": "<base64 Ed25519 public key>",
+  "signedAt": 1716057600000,
+  "signature": "<base64 Ed25519 signature over canonical body>"
+}
+```
+
+- 200 — registered (or idempotent re-register with same key).
+- 401 — different key already owns this address, or signature failed.
+
+### `POST /v1/inbox/:address` (PUT blob)
+
+```json
+{
+  "senderSigningKey": "<base64 sender Ed25519 public key>",
+  "msgId": "<lowercase hex sha256(ciphertext)>",
+  "ciphertext": "<base64 wire bytes from encodeEnvelope()>",
+  "ttlSeconds": 604800,
+  "signedAt": 1716057600000,
+  "signature": "<base64 sender signature>"
+}
+```
+
+- 200 with `{ msgId, receivedAt, idempotent: false }` — first store.
+- 200 with `idempotent: true` — duplicate PUT folded into the first row.
+- 400 — `msgId` mismatch, ciphertext too big, or address quota exceeded.
+- 401 — bad signature or stale `signedAt`.
+- 404 — recipient address never registered.
+
+### `POST /v1/inbox/:address/fetch` (signed challenge)
+
+```json
+{
+  "address": "bob",
+  "sinceCursor": 0,
+  "signedAt": 1716057600000,
+  "signature": "<base64 recipient signature>"
+}
+```
+
+Returns:
+
+```json
+{
+  "blobs": [
+    {
+      "msgId": "<hex>",
+      "ciphertext": "<base64>",
+      "receivedAt": 1716057601234,
+      "expiresAt": 1716662401234
+    }
+  ],
+  "cursor": 1716057601234,
+  "hasMore": false
+}
+```
+
+Pass the returned `cursor` as `sinceCursor` next time. Pages cap at
+`fetchPageLimit` (default 100); keep calling with the new cursor while
+`hasMore === true`.
+
+### `DELETE /v1/inbox/:address/:msgId` (signed ack)
+
+Body:
+
+```json
+{
+  "address": "bob",
+  "msgId": "<hex>",
+  "signedAt": 1716057600000,
+  "signature": "<base64 recipient signature>"
+}
+```
+
+- 200 with `{ ok: true }` — row removed.
+- 200 with `{ ok: false }` — row was already gone (also idempotent).
+- 401 — recipient signature failed.
+
+### `DELETE /v1/inbox/register/:address`
+
+Same auth shape as ack. Drops every queued blob.
+
+---
+
+## 5. Acceptance test mapping
+
+| V3.6 spec criterion                                        | Test                                                           |
+| ---------------------------------------------------------- | -------------------------------------------------------------- |
+| Async delivery without online overlap                      | `lifecycle.test.ts → "100 messages delivered…"`                |
+| DB-dump leaks no plaintext / sender-recipient graph        | Server stores only `address \|\| msgId \|\| ct \|\| expires_at`; verified by `routes.test.ts` schema asserts |
+| Replay PUT with same `msgId` is idempotent                 | `routes.test.ts → "idempotent on duplicate ciphertext"`         |
+| Restart preserves blobs                                    | `lifecycle.test.ts → "persistence across restart"` + sqlite-store reopen |
+| Bit-flip on stored ciphertext rejected on the client       | `lifecycle.test.ts → "Tamper resistance"` + client `client.test.ts → "tamper detection"` |