release(v4.0.0): Shade GA — V3.x consolidation + audit prep

V3.1 → V3.12 consolidated and tagged for the first GA release. Wire format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers byte-for-byte. The version bump is semantic: audit-cycle complete, opt-in surface fully exposed, threat model refreshed for every new surface. Highlights: - All 24 @shade/* packages bumped to 4.0.0 in lockstep. - CHANGELOG 4.0.0 section is the canonical manifest of what landed. - THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12 Web-Worker boundary) + residual-risks table refreshed. - OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox, bridge, observer, /metrics, /healthz, /ready. - MIGRATION 0.3.x → 4.0 documented + smoke-tested against shade migrate-storage on a real SQLite DB. - docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer. - scripts/soak.ts harness for the GA-stable 2-week soak window. - All V*.md plans archived under docs/archive/ with Status: Done. - Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen non-realtime stack. Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green. Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports version 4.0.0 on /health. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
parent 8b055912b7
commit e6fdf31b49
298 changed files with 37909 additions and 256 deletions
--- a/README.md
+++ b/README.md
@@ -2,10 +2,18 @@

 End-to-end encryption library implementing the Signal Protocol (X3DH + Double Ratchet) for TypeScript/Bun. Drop into any project — frontend, backend, mobile — to get forward secrecy, post-compromise recovery, and self-healing security.

-> **0.3.0 — wire format breaking change.** The wire VERSION was bumped from
-> `0x01` to `0x02` (length prefixes u16 → u32) to support inline file ops up
-> to 256 KiB. **0.3.x peers cannot interoperate with 0.2.x peers** — both
-> ends must upgrade. See [CHANGELOG.md](./CHANGELOG.md) for the full diff.
+> **4.0.0 — General Availability.** All V3.1 → V3.12 work is merged,
+> the cross-platform vector suite is green on TS + Kotlin, the threat
+> model has been refreshed for every new surface, and the core stack
+> (X3DH, ratchet, storage encryption, recovery, WebRTC P2P, Key
+> Transparency) has been packaged for external review. The wire
+> format is **unchanged from 0.4.x** — 4.0 peers interoperate with
+> 0.4.x peers byte-for-byte. See
+> [MIGRATION.md § 0.3.x → 4.0](./MIGRATION.md#migrating-from-03x-to-40-ga)
+> for the upgrade path and [CHANGELOG.md § 4.0.0](./CHANGELOG.md) for
+> the consolidated release notes. Voice / Video have been moved to
+> [V5.0](./docs/V5.0.md), to be built on top of the frozen 4.0
+> baseline.

 ## What you get

@@ -20,7 +28,12 @@ End-to-end encryption library implementing the Signal Protocol (X3DH + Double Ra
 - **Crash-safe** — sessions survive container restarts, power outages, SIGKILL
 - **Live observability** — bundled dashboard SPA + embeddable React widgets to see what's happening between every step
 - **E2EE file transfers** — multi-lane chunked uploads/downloads with resume, integrity checks, and HTTP/WS fallback (`@shade/streams` + `@shade/transfer`)
+- **WebRTC P2P transport (V3.11)** — opt-in `RTCDataChannel` upload path with public-STUN defaults, TURN-relay support, glare-safe peer pool, and automatic `MultiTransportFallback` back to HTTP when NAT traversal fails (`@shade/transport-webrtc`, [docs/webrtc.md](./docs/webrtc.md))
+- **Web Workers crypto** — AEAD, HKDF, HMAC, X25519, Ed25519 and per-lane stream state run in a dedicated worker. 100 MB+ uploads stay smooth without frame drops, lane keys never cross the thread boundary (`@shade/crypto-web/worker`, [docs/web-workers.md](./docs/web-workers.md))
 - **E2EE filesystem RPC** — typed `list/stat/mkdir/delete/move/read/write/getThumbnail` + custom ops between peers, with rate-limit, retention, and fingerprint-gate hooks (`@shade/files`)
+- **Async store-and-forward** — deliver to offline recipients via a relay that holds ciphertext-only blobs with TTL, idempotent PUT, signed fetch/ack, and an `onMessageQueued` push-trigger hook (`@shade/inbox` + `@shade/inbox-server`)
+- **Social key recovery** — Shamir-split your identity to `n` guardians; any threshold-many `k` together restore it on a new device. No centralized recovery agent; OOB-fingerprint gate on every guardian release; AES-GCM authenticates the reconstruction (`@shade/recovery` + `<RecoverySetup />` / `<RecoveryRequest />` / `<RecoveryApprove />` widgets, [docs/recovery.md](./docs/recovery.md))
+- **Key Transparency (V3.12)** — opt-in append-only Merkle log over the prekey server. Every `register` / `delete` becomes a signed leaf; every bundle-fetch carries an inclusion proof; an Ed25519-signed Tree Head ties roots to a fixed `log_id`. A `LightWitness` cross-checks STHs across clients so a malicious server that splits its view or rewrites history is caught (`@shade/key-transparency`, [docs/key-transparency.md](./docs/key-transparency.md))

 ## Quick start

@@ -104,42 +117,90 @@ const manager = new ShadeSessionManager(
 await manager.initialize();
 ```

-## Architecture
+## Architecture — keys vs. payloads
+
+Shade splits the network into two planes. The **prekey server** only
+sees public keys; **everything else** rides the encrypted Double
+Ratchet between peers. If you remember nothing else from this README,
+remember this picture:

 ```
-                     Shade Prekey Server (Hono)
-                              │
-              POST /v1/keys/register (signed)
-              GET  /v1/keys/bundle/:address
-              POST /v1/keys/replenish (signed)
-              DELETE /v1/keys/:address (signed)
-                              │
-        ┌─────────────────────┴─────────────────────┐
-        │                                           │
-    [Client A]                                  [Client B]
-    ShadeSessionManager                         ShadeSessionManager
-        │                                           │
-        ├──── X3DH ────────────────────────────────►│
-        │                                           │
-        │◄──── Double Ratchet messages ────────────►│
-        │                                           │
-    SQLiteStorage / PostgresStorage             SQLiteStorage / PostgresStorage
+              Shade Prekey Server (Hono, public keys only)
+                           │
+                  POST /v1/keys/register   (signed)
+                  GET  /v1/keys/bundle/:address
+                  POST /v1/keys/replenish  (signed)
+                  DELETE /v1/keys/:address (signed)
+                           │
+        ┌──────────────────┴──────────────────┐
+        │                                     │
+    [Client A]                            [Client B]
+    ShadeSessionManager                   ShadeSessionManager
+        │                                     │
+        ├── X3DH (handshake via prekey srv) ─►│
+        │                                     │
+        │◄── Double Ratchet messages ────────►│   ← end-to-end,
+        │   (ratchet 0x02 / chunks 0x11)      │     never on the
+        │                                     │     prekey server
+        │◄── @shade/transfer chunks ─────────►│
+        │   POST /v1/transfer/:id/chunk        │   ← peer-to-peer
+        │   GET  /v1/transfer/:id/state        │     HTTP, opaque
+        │                                     │     ciphertext
+    SQLiteStorage / PostgresStorage       SQLiteStorage / PostgresStorage
+        (private keys + sessions)             (private keys + sessions)
 ```

+### What goes via the prekey server
+
+- Identity public keys (Ed25519 + X25519)
+- Signed prekeys + one-time prekey bundles
+- Registration / replenish / delete writes, all Ed25519-signed
+- Operator-only metrics, `/health`, and the optional observer
+  dashboard
+
+### What does **not** go via the prekey server
+
+- **Message plaintext, ever.** Encrypted ratchet envelopes flow peer-
+  to-peer over whatever transport you choose (HTTP, WebSocket, your
+  own broker).
+- **File chunks.** `@shade/transfer` POSTs ciphertext directly to the
+  receiver's `/v1/transfer/:streamId/chunk` route — the prekey server
+  is not involved.
+- **Identity private keys.** They never leave the device's storage.
+- **Filesystem RPC.** `@shade/files` rides the Double Ratchet for
+  control + small payloads, then promotes to direct `@shade/transfer`
+  streams for larger blobs.
+- **Stream resume secrets.** Persisted only on the local device,
+  encrypted under a device-key derived from the identity signing key.
+
+The prekey server is metadata-bearing (see `THREAT-MODEL.md § 2`):
+it sees who registers, who fetches whose bundle, and when. It does
+**not** see message contents, transfer contents, or session state.
+
+For the full threat model and mitigations, read
+[THREAT-MODEL.md](./THREAT-MODEL.md). For deployment-time guarantees,
+read [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md).
+
 ## Packages

 | Package | Purpose |
 |---------|---------|
 | `@shade/core` | Protocol logic (X3DH, Double Ratchet, session manager, errors, events) |
-| `@shade/crypto-web` | SubtleCrypto + @noble/curves provider, in-memory storage |
+| `@shade/crypto-web` | SubtleCrypto + @noble/curves provider, in-memory storage. Includes the V3.8 Web Workers entrypoint (`@shade/crypto-web/worker`) — drop-in `WorkerCryptoProvider` plus `createEncryptStream` / `createDecryptStream` TransformStream factories |
 | `@shade/storage-sqlite` | Persistent SQLite storage (zero-config, bun:sqlite) |
 | `@shade/storage-postgres` | PostgreSQL storage with Drizzle for shared databases |
 | `@shade/server` | Prekey server (Hono routes, auth, rate limit, health, metrics) |
 | `@shade/transport` | HTTP + WebSocket transport wrappers with auto-encryption |
+| `@shade/transport-bridge` | WS → SSE → long-poll fallback chain (V3.7) — single `IncomingMessage` shape across transports for clients that can't keep a WebSocket open |
+| `@shade/transport-webrtc` | V3.11 P2P chunk transport via `RTCDataChannel`. Plugs into `@shade/transfer` as an `ITransferTransport`; signaling rides Shade's own ratchet. Memory factory + native (`globalThis.RTCPeerConnection`) factory included; `MultiTransportFallback([webrtc, http])` wired automatically when `shade.configureWebRTC()` is called. |
 | `@shade/proto` | Compact binary wire format (smaller than JSON) |
 | `@shade/streams` | Multi-lane chunk encryption — HKDF-derived per-lane keys, deterministic AES-GCM nonces, streaming SHA-256 |
 | `@shade/transfer` | Transfer engine on top of streams: parallel lanes, resume, HTTP + WS transport with auto-fallback, integrity verification |
 | `@shade/files` | Typed E2EE filesystem RPC — list/stat/mkdir/delete/move/read/write/getThumbnail + custom ops, auto inline/streams routing, production hooks (rate limit, retention, fingerprint gate, metrics), React hooks |
+| `@shade/recovery` | Social key recovery (V3.10) — Shamir-split identity to `n` guardians; threshold-many `k` reconstruct on a new device. AES-GCM-authenticated reconstruction; OOB-fingerprint gate per guardian release |
+| `@shade/key-transparency` | Key Transparency (V3.12) — RFC 6962-style append-only Merkle log, address-index commitment, signed tree heads, and a `LightWitness` for split-view detection. Opt-in on both server and client. See [docs/key-transparency.md](./docs/key-transparency.md) |
+| `@shade/inbox-server` | Async store-and-forward relay (V3.6) — Hono routes, signed PUT/FETCH/DELETE, per-recipient TTL + quota, idempotent on `(address, msgId)`. Bundles into the same standalone container as the prekey server |
+| `@shade/inbox` | Inbox client + durable outgoing queue + receive cursor + push-trigger hook (`onMessageQueued`); composes on top of `Shade.send`/`Shade.receive` for offline-recipient delivery |
 | `@shade/observer` | Live debugger backend (snapshot, SSE, dashboard) — see [README](./packages/shade-observer/README.md) |
 | `@shade/widgets` | Embeddable React widgets including transfer uploader/downloader — see [README](./packages/shade-widgets/README.md) |
 | `@shade/dashboard` | Standalone dashboard SPA bundled into the observer |
@@ -181,13 +242,22 @@ bun run publish:all
 | **Memory zeroization** | Key material is zeroed after use (best-effort in JS) |
 | **Identity verification** | Safety numbers (60 digits) for out-of-band comparison |
 | **Identity rotation** | 7-day grace period for old sessions during rotation |
+| **Key Transparency** *(V3.12, opt-in)* | Append-only Merkle log + signed tree heads + witness gossip — split-view and history-rewrite are detected by clients |

 ## Documentation

 - [docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md) — **Modular toolkit**: pick packages by scenario (messages, files, browser, ops)
+- [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md) — Pre-flight gates for going to production
 - [docs/files.md](./docs/files.md) — `@shade/files` API + design (filesystem RPC, custom ops, hooks, React)
- [docs/streams.md](./docs/streams.md) — `@shade/streams` + `@shade/transfer` deep dive
- [SECURITY.md](./SECURITY.md) — Reporting vulnerabilities, security policy
+- [docs/recovery.md](./docs/recovery.md) — `@shade/recovery` social key recovery (V3.10): Shamir setup, guardian-side gates, threshold tuning
+- [docs/streams.md](./docs/streams.md) — `@shade/streams` + `@shade/transfer` deep dive (incl. hardening + retention)
+- [docs/inbox.md](./docs/inbox.md) — `@shade/inbox` + `@shade/inbox-server` async store-and-forward relay (V3.6)
+- [docs/transport.md](./docs/transport.md) — `@shade/transport-bridge` SSE / long-poll / WS bridge layer (V3.7)
+- [docs/webrtc.md](./docs/webrtc.md) — `@shade/transport-webrtc` P2P transport (V3.11): NAT-traversal, TURN config, glare resolution, wire format, multi-fallback wiring
+- [docs/key-transparency.md](./docs/key-transparency.md) — `@shade/key-transparency` (V3.12): operator + client onboarding, witness role, recovery procedures
+- [docs/V3.12-DESIGN.md](./docs/V3.12-DESIGN.md) — V3.12 design notat (threat model, RFC 6962 vs CONIKS choice, freshness model, open-questions resolution)
+- [docs/web-workers.md](./docs/web-workers.md) — V3.8 Web Workers crypto: setup, bundler recipes (Vite/Webpack/Rollup), Safari notes, lifecycle, threat-model
+- [SECURITY.md](./SECURITY.md) — Reporting vulnerabilities, security policy, threat-/test-matrix
 - [THREAT-MODEL.md](./THREAT-MODEL.md) — Honest threat model and assumptions
 - [examples/](./examples/) — Runnable example applications, including
  [`07-streams-upload`](./examples/07-streams-upload) (multi-lane file transfer)