Shade/README.md

# Shade

End-to-end encryption library implementing the Signal Protocol (X3DH + Double Ratchet) for TypeScript/Bun. Drop into any project — frontend, backend, mobile — to get forward secrecy, post-compromise recovery, and self-healing security.

> **4.0.0 — General Availability.** All V3.1 → V3.12 work is merged,
> the cross-platform vector suite is green on TS + Kotlin, the threat
> model has been refreshed for every new surface, and the core stack
> (X3DH, ratchet, storage encryption, recovery, WebRTC P2P, Key
> Transparency) has been packaged for external review. The wire
> format is **unchanged from 0.4.x** — 4.0 peers interoperate with
> 0.4.x peers byte-for-byte. See
> [MIGRATION.md § 0.3.x → 4.0](./MIGRATION.md#migrating-from-03x-to-40-ga)
> for the upgrade path and [CHANGELOG.md § 4.0.0](./CHANGELOG.md) for
> the consolidated release notes. Voice / Video have been moved to
> [V5.0](./docs/V5.0.md), to be built on top of the frozen 4.0
> baseline.

## What you get

- **X3DH** initial key agreement (works asynchronously via prekey bundles)
- **Double Ratchet** for per-message forward secrecy and post-compromise security
- **Self-authenticated prekey server** (Hono, Docker-ready) with rate limiting, metrics, health checks
- **Persistent storage backends**: SQLite (zero-config) and PostgreSQL (Drizzle)
- **Identity rotation** with grace period for old sessions
- **Safety numbers** (Signal-style fingerprints) for out-of-band verification
- **Constant-time comparisons** and **memory zeroization** for hardened operation
- **Binary wire format** that's significantly smaller than JSON
- **Crash-safe** — sessions survive container restarts, power outages, SIGKILL
- **Live observability** — bundled dashboard SPA + embeddable React widgets to see what's happening between every step
- **E2EE file transfers** — multi-lane chunked uploads/downloads with resume, integrity checks, and HTTP/WS fallback (`@shade/streams` + `@shade/transfer`)
- **WebRTC P2P transport (V3.11)** — opt-in `RTCDataChannel` upload path with public-STUN defaults, TURN-relay support, glare-safe peer pool, and automatic `MultiTransportFallback` back to HTTP when NAT traversal fails (`@shade/transport-webrtc`, [docs/webrtc.md](./docs/webrtc.md))
- **Web Workers crypto** — AEAD, HKDF, HMAC, X25519, Ed25519 and per-lane stream state run in a dedicated worker. 100 MB+ uploads stay smooth without frame drops, lane keys never cross the thread boundary (`@shade/crypto-web/worker`, [docs/web-workers.md](./docs/web-workers.md))
- **E2EE filesystem RPC** — typed `list/stat/mkdir/delete/move/read/write/getThumbnail` + custom ops between peers, with rate-limit, retention, and fingerprint-gate hooks (`@shade/files`)
- **Async store-and-forward** — deliver to offline recipients via a relay that holds ciphertext-only blobs with TTL, idempotent PUT, signed fetch/ack, and an `onMessageQueued` push-trigger hook (`@shade/inbox` + `@shade/inbox-server`)
- **Social key recovery** — Shamir-split your identity to `n` guardians; any threshold-many `k` together restore it on a new device. No centralized recovery agent; OOB-fingerprint gate on every guardian release; AES-GCM authenticates the reconstruction (`@shade/recovery` + `<RecoverySetup />` / `<RecoveryRequest />` / `<RecoveryApprove />` widgets, [docs/recovery.md](./docs/recovery.md))
- **Key Transparency (V3.12)** — opt-in append-only Merkle log over the prekey server. Every `register` / `delete` becomes a signed leaf; every bundle-fetch carries an inclusion proof; an Ed25519-signed Tree Head ties roots to a fixed `log_id`. A `LightWitness` cross-checks STHs across clients so a malicious server that splits its view or rewrites history is caught (`@shade/key-transparency`, [docs/key-transparency.md](./docs/key-transparency.md))

## Quick start

Add the Gitea npm registry to your project's `.npmrc`:

```
@shade:registry=https://gt.zyon.no/api/packages/Stian/npm/
```

Then install the SDK (one-liner for most use cases):

```bash
bun add @shade/sdk
```

Or install specific packages if you need fine-grained control:

```bash
bun add @shade/core @shade/crypto-web @shade/storage-sqlite
```

Even faster — scaffold a new project with the CLI:

```bash
bun add -g @shade/cli
shade init my-app --template bun-server
cd my-app && bun install && bun run start
```

Magic one-liner with the SDK:

```ts
import { createShade } from '@shade/sdk';

const shade = await createShade({
  prekeyServer: 'https://shade.example.com',
  storage: 'sqlite:/data/shade.db',
  address: 'alice@example.com',
});

// Send (auto-establishes session if none exists)
const envelope = await shade.send('bob@example.com', 'Hello, encrypted world!');

// Receive
const plaintext = await shade.receive('alice@example.com', incomingEnvelope);

// Your safety number for out-of-band verification
console.log(await shade.fingerprint);
```

Need to ship a file or expose a filesystem to a peer? `Shade.files` is the high-level entrypoint:

```ts
// Server side — Bob exposes a virtual filesystem
const stop = await shade.files.serve({
  list:  async (ctx) => ({ entries: await readdirAt(ctx.path), hasMore: false }),
  read:  async (ctx) => readAt(ctx.path),    // returns inline ≤ 256 KiB or streams
  write: async (ctx) => writeAt(ctx.args),   // receives inline or streams
  // + stat, mkdir, delete, move, getThumbnail, plus typed custom ops
});

// Client side — Alice consumes Bob's filesystem
const fs = await shade.files.client('bob');
await fs.write('/photos/cover.png', new Uint8Array(...));   // auto inline/streams
const result = await fs.read('/photos/cover.png');
```

Files ≤ 256 KiB ride inline in the RPC envelope; larger files automatically promote to multi-lane `@shade/transfer` streams with sha256 integrity. See [`docs/files.md`](./docs/files.md) for the full API.

Or use the lower-level packages directly if you need full control:

```ts
import { ShadeSessionManager } from '@shade/core';
import { SubtleCryptoProvider } from '@shade/crypto-web';
import { SQLiteStorage } from '@shade/storage-sqlite';

const manager = new ShadeSessionManager(
  new SubtleCryptoProvider(),
  new SQLiteStorage('/data/shade.db'),
);
await manager.initialize();
```

## Architecture — keys vs. payloads

Shade splits the network into two planes. The **prekey server** only
sees public keys; **everything else** rides the encrypted Double
Ratchet between peers. If you remember nothing else from this README,
remember this picture:

```
              Shade Prekey Server (Hono, public keys only)
                           │
                  POST /v1/keys/register   (signed)
                  GET  /v1/keys/bundle/:address
                  POST /v1/keys/replenish  (signed)
                  DELETE /v1/keys/:address (signed)
                           │
        ┌──────────────────┴──────────────────┐
        │                                     │
    [Client A]                            [Client B]
    ShadeSessionManager                   ShadeSessionManager
        │                                     │
        ├── X3DH (handshake via prekey srv) ─►│
        │                                     │
        │◄── Double Ratchet messages ────────►│   ← end-to-end,
        │   (ratchet 0x02 / chunks 0x11)      │     never on the
        │                                     │     prekey server
        │◄── @shade/transfer chunks ─────────►│
        │   POST /v1/transfer/:id/chunk        │   ← peer-to-peer
        │   GET  /v1/transfer/:id/state        │     HTTP, opaque
        │                                     │     ciphertext
    SQLiteStorage / PostgresStorage       SQLiteStorage / PostgresStorage
        (private keys + sessions)             (private keys + sessions)
```

### What goes via the prekey server

- Identity public keys (Ed25519 + X25519)
- Signed prekeys + one-time prekey bundles
- Registration / replenish / delete writes, all Ed25519-signed
- Operator-only metrics, `/health`, and the optional observer
  dashboard

### What does **not** go via the prekey server

- **Message plaintext, ever.** Encrypted ratchet envelopes flow peer-
  to-peer over whatever transport you choose (HTTP, WebSocket, your
  own broker).
- **File chunks.** `@shade/transfer` POSTs ciphertext directly to the
  receiver's `/v1/transfer/:streamId/chunk` route — the prekey server
  is not involved.
- **Identity private keys.** They never leave the device's storage.
- **Filesystem RPC.** `@shade/files` rides the Double Ratchet for
  control + small payloads, then promotes to direct `@shade/transfer`
  streams for larger blobs.
- **Stream resume secrets.** Persisted only on the local device,
  encrypted under a device-key derived from the identity signing key.

The prekey server is metadata-bearing (see `THREAT-MODEL.md § 2`):
it sees who registers, who fetches whose bundle, and when. It does
**not** see message contents, transfer contents, or session state.

For the full threat model and mitigations, read
[THREAT-MODEL.md](./THREAT-MODEL.md). For deployment-time guarantees,
read [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md).

## Packages

| Package | Purpose |
|---------|---------|
| `@shade/core` | Protocol logic (X3DH, Double Ratchet, session manager, errors, events) |
| `@shade/crypto-web` | SubtleCrypto + @noble/curves provider, in-memory storage. Includes the V3.8 Web Workers entrypoint (`@shade/crypto-web/worker`) — drop-in `WorkerCryptoProvider` plus `createEncryptStream` / `createDecryptStream` TransformStream factories |
| `@shade/storage-sqlite` | Persistent SQLite storage (zero-config, bun:sqlite) |
| `@shade/storage-postgres` | PostgreSQL storage with Drizzle for shared databases |
| `@shade/server` | Prekey server (Hono routes, auth, rate limit, health, metrics) |
| `@shade/transport` | HTTP + WebSocket transport wrappers with auto-encryption |
| `@shade/transport-bridge` | WS → SSE → long-poll fallback chain (V3.7) — single `IncomingMessage` shape across transports for clients that can't keep a WebSocket open |
| `@shade/transport-webrtc` | V3.11 P2P chunk transport via `RTCDataChannel`. Plugs into `@shade/transfer` as an `ITransferTransport`; signaling rides Shade's own ratchet. Memory factory + native (`globalThis.RTCPeerConnection`) factory included; `MultiTransportFallback([webrtc, http])` wired automatically when `shade.configureWebRTC()` is called. |
| `@shade/proto` | Compact binary wire format (smaller than JSON) |
| `@shade/streams` | Multi-lane chunk encryption — HKDF-derived per-lane keys, deterministic AES-GCM nonces, streaming SHA-256 |
| `@shade/transfer` | Transfer engine on top of streams: parallel lanes, resume, HTTP + WS transport with auto-fallback, integrity verification |
| `@shade/files` | Typed E2EE filesystem RPC — list/stat/mkdir/delete/move/read/write/getThumbnail + custom ops, auto inline/streams routing, production hooks (rate limit, retention, fingerprint gate, metrics), React hooks |
| `@shade/recovery` | Social key recovery (V3.10) — Shamir-split identity to `n` guardians; threshold-many `k` reconstruct on a new device. AES-GCM-authenticated reconstruction; OOB-fingerprint gate per guardian release |
| `@shade/key-transparency` | Key Transparency (V3.12) — RFC 6962-style append-only Merkle log, address-index commitment, signed tree heads, and a `LightWitness` for split-view detection. Opt-in on both server and client. See [docs/key-transparency.md](./docs/key-transparency.md) |
| `@shade/inbox-server` | Async store-and-forward relay (V3.6) — Hono routes, signed PUT/FETCH/DELETE, per-recipient TTL + quota, idempotent on `(address, msgId)`. Bundles into the same standalone container as the prekey server |
| `@shade/inbox` | Inbox client + durable outgoing queue + receive cursor + push-trigger hook (`onMessageQueued`); composes on top of `Shade.send`/`Shade.receive` for offline-recipient delivery |
| `@shade/observer` | Live debugger backend (snapshot, SSE, dashboard) — see [README](./packages/shade-observer/README.md) |
| `@shade/widgets` | Embeddable React widgets including transfer uploader/downloader — see [README](./packages/shade-widgets/README.md) |
| `@shade/dashboard` | Standalone dashboard SPA bundled into the observer |
| `@shade/sdk` | High-level wrapper with `createShade()` one-liner, auto-publish, auto-establish, auto-replenish, `Shade.files` namespace |
| `@shade/cli` | `shade init` scaffolder + utilities (fingerprint, rotate, peer, dashboard, doctor) |

## Shade as a modular toolkit

Shade is split into packages so each project can depend on **only what it needs**—encrypted messaging, file transfer, prekey hosting, or lower-level building blocks. You do not need one giant stack for every use case.

For a **plain-language map** (which packages to add, what the prekey server does vs your own wiring, and where to start in code), see **[docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md)**.

## Publishing

All packages publish to a self-hosted Gitea npm registry on `gt.zyon.no`.

```bash
# Bump all packages in lockstep
bun run version 1.1.0

# Dry-run (pack all tarballs without publishing)
bun run publish:dry

# Real publish (requires GITEA_TOKEN env var)
bun run publish:all

# Or via CI: push a git tag v1.1.0 and .gitea/workflows/publish.yml runs
```

## Security properties

| Property | Description |
|----------|-------------|
| **Forward secrecy** | Compromising a key cannot decrypt past messages |
| **Post-compromise security** | Self-heals after key compromise on next DH ratchet |
| **Authentication** | Ed25519 identity signatures on prekey server writes |
| **Replay protection** | ±5 minute timestamp window on signed requests |
| **Constant-time comparisons** | Timing attacks on identity keys are blocked |
| **Memory zeroization** | Key material is zeroed after use (best-effort in JS) |
| **Identity verification** | Safety numbers (60 digits) for out-of-band comparison |
| **Identity rotation** | 7-day grace period for old sessions during rotation |
| **Key Transparency** *(V3.12, opt-in)* | Append-only Merkle log + signed tree heads + witness gossip — split-view and history-rewrite are detected by clients |

## Documentation

- [docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md) — **Modular toolkit**: pick packages by scenario (messages, files, browser, ops)
- [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md) — Pre-flight gates for going to production
- [docs/files.md](./docs/files.md) — `@shade/files` API + design (filesystem RPC, custom ops, hooks, React)
- [docs/recovery.md](./docs/recovery.md) — `@shade/recovery` social key recovery (V3.10): Shamir setup, guardian-side gates, threshold tuning
- [docs/streams.md](./docs/streams.md) — `@shade/streams` + `@shade/transfer` deep dive (incl. hardening + retention)
- [docs/inbox.md](./docs/inbox.md) — `@shade/inbox` + `@shade/inbox-server` async store-and-forward relay (V3.6)
- [docs/transport.md](./docs/transport.md) — `@shade/transport-bridge` SSE / long-poll / WS bridge layer (V3.7)
- [docs/webrtc.md](./docs/webrtc.md) — `@shade/transport-webrtc` P2P transport (V3.11): NAT-traversal, TURN config, glare resolution, wire format, multi-fallback wiring
- [docs/key-transparency.md](./docs/key-transparency.md) — `@shade/key-transparency` (V3.12): operator + client onboarding, witness role, recovery procedures
- [docs/V3.12-DESIGN.md](./docs/V3.12-DESIGN.md) — V3.12 design notat (threat model, RFC 6962 vs CONIKS choice, freshness model, open-questions resolution)
- [docs/web-workers.md](./docs/web-workers.md) — V3.8 Web Workers crypto: setup, bundler recipes (Vite/Webpack/Rollup), Safari notes, lifecycle, threat-model
- [SECURITY.md](./SECURITY.md) — Reporting vulnerabilities, security policy, threat-/test-matrix
- [THREAT-MODEL.md](./THREAT-MODEL.md) — Honest threat model and assumptions
- [examples/](./examples/) — Runnable example applications, including
  [`07-streams-upload`](./examples/07-streams-upload) (multi-lane file transfer)
  and [`08-files-browser`](./examples/08-files-browser) (filesystem RPC)
- [MIGRATION.md](./MIGRATION.md) — How to replace existing crypto with Shade

## Deployment — one container per project

Shade ships as a self-contained Docker image. Deploy one container per project, point your app at it, done. Any stack (Bun, Python, Go, Rust, Kotlin) can use it — the container exposes a plain HTTP API documented in OpenAPI.

```bash
docker run -d \
  --name my-project-shade \
  -v my-project-shade:/data \
  -p 3900:3900 \
  -e SHADE_OBSERVER_TOKEN=change-me-to-at-least-16-chars \
  gt.zyon.no/stian/shade-prekey:latest
```

The container includes:
- **Prekey server** — `/v1/keys/*` REST API
- **Observer dashboard** — `/shade-observer/dashboard/` (off unless token is set)
- **OpenAPI spec** — `/openapi.yaml` and interactive `/docs` viewer
- **Prometheus metrics** — `/metrics`
- **Health check** — `/health`
- **Stale cleanup** — purges inactive identities automatically

See [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md) for the full deployment guide, environment variables, PostgreSQL config, backup strategy, and Dokploy instructions.

## License

MIT