Shade/README.md

# Shade

End-to-end encryption library implementing the Signal Protocol (X3DH + Double Ratchet) for TypeScript/Bun. Drop into any project — frontend, backend, mobile — to get forward secrecy, post-compromise recovery, and self-healing security.

> **0.3.0 — wire format breaking change.** The wire VERSION was bumped from
> `0x01` to `0x02` (length prefixes u16 → u32) to support inline file ops up
> to 256 KiB. **0.3.x peers cannot interoperate with 0.2.x peers** — both
> ends must upgrade. See [CHANGELOG.md](./CHANGELOG.md) for the full diff.

## What you get

- **X3DH** initial key agreement (works asynchronously via prekey bundles)
- **Double Ratchet** for per-message forward secrecy and post-compromise security
- **Self-authenticated prekey server** (Hono, Docker-ready) with rate limiting, metrics, health checks
- **Persistent storage backends**: SQLite (zero-config) and PostgreSQL (Drizzle)
- **Identity rotation** with grace period for old sessions
- **Safety numbers** (Signal-style fingerprints) for out-of-band verification
- **Constant-time comparisons** and **memory zeroization** for hardened operation
- **Binary wire format** that's significantly smaller than JSON
- **Crash-safe** — sessions survive container restarts, power outages, SIGKILL
- **Live observability** — bundled dashboard SPA + embeddable React widgets to see what's happening between every step
- **E2EE file transfers** — multi-lane chunked uploads/downloads with resume, integrity checks, and HTTP/WS fallback (`@shade/streams` + `@shade/transfer`)
- **E2EE filesystem RPC** — typed `list/stat/mkdir/delete/move/read/write/getThumbnail` + custom ops between peers, with rate-limit, retention, and fingerprint-gate hooks (`@shade/files`)

## Quick start

Add the Gitea npm registry to your project's `.npmrc`:

```
@shade:registry=https://gt.zyon.no/api/packages/Stian/npm/
```

Then install the SDK (one-liner for most use cases):

```bash
bun add @shade/sdk
```

Or install specific packages if you need fine-grained control:

```bash
bun add @shade/core @shade/crypto-web @shade/storage-sqlite
```

Even faster — scaffold a new project with the CLI:

```bash
bun add -g @shade/cli
shade init my-app --template bun-server
cd my-app && bun install && bun run start
```

Magic one-liner with the SDK:

```ts
import { createShade } from '@shade/sdk';

const shade = await createShade({
  prekeyServer: 'https://shade.example.com',
  storage: 'sqlite:/data/shade.db',
  address: 'alice@example.com',
});

// Send (auto-establishes session if none exists)
const envelope = await shade.send('bob@example.com', 'Hello, encrypted world!');

// Receive
const plaintext = await shade.receive('alice@example.com', incomingEnvelope);

// Your safety number for out-of-band verification
console.log(await shade.fingerprint);
```

Need to ship a file or expose a filesystem to a peer? `Shade.files` is the high-level entrypoint:

```ts
// Server side — Bob exposes a virtual filesystem
const stop = await shade.files.serve({
  list:  async (ctx) => ({ entries: await readdirAt(ctx.path), hasMore: false }),
  read:  async (ctx) => readAt(ctx.path),    // returns inline ≤ 256 KiB or streams
  write: async (ctx) => writeAt(ctx.args),   // receives inline or streams
  // + stat, mkdir, delete, move, getThumbnail, plus typed custom ops
});

// Client side — Alice consumes Bob's filesystem
const fs = await shade.files.client('bob');
await fs.write('/photos/cover.png', new Uint8Array(...));   // auto inline/streams
const result = await fs.read('/photos/cover.png');
```

Files ≤ 256 KiB ride inline in the RPC envelope; larger files automatically promote to multi-lane `@shade/transfer` streams with sha256 integrity. See [`docs/files.md`](./docs/files.md) for the full API.

Or use the lower-level packages directly if you need full control:

```ts
import { ShadeSessionManager } from '@shade/core';
import { SubtleCryptoProvider } from '@shade/crypto-web';
import { SQLiteStorage } from '@shade/storage-sqlite';

const manager = new ShadeSessionManager(
  new SubtleCryptoProvider(),
  new SQLiteStorage('/data/shade.db'),
);
await manager.initialize();
```

## Architecture

```
                     Shade Prekey Server (Hono)
                              │
              POST /v1/keys/register (signed)
              GET  /v1/keys/bundle/:address
              POST /v1/keys/replenish (signed)
              DELETE /v1/keys/:address (signed)
                              │
        ┌─────────────────────┴─────────────────────┐
        │                                           │
    [Client A]                                  [Client B]
    ShadeSessionManager                         ShadeSessionManager
        │                                           │
        ├──── X3DH ────────────────────────────────►│
        │                                           │
        │◄──── Double Ratchet messages ────────────►│
        │                                           │
    SQLiteStorage / PostgresStorage             SQLiteStorage / PostgresStorage
```

## Packages

| Package | Purpose |
|---------|---------|
| `@shade/core` | Protocol logic (X3DH, Double Ratchet, session manager, errors, events) |
| `@shade/crypto-web` | SubtleCrypto + @noble/curves provider, in-memory storage |
| `@shade/storage-sqlite` | Persistent SQLite storage (zero-config, bun:sqlite) |
| `@shade/storage-postgres` | PostgreSQL storage with Drizzle for shared databases |
| `@shade/server` | Prekey server (Hono routes, auth, rate limit, health, metrics) |
| `@shade/transport` | HTTP + WebSocket transport wrappers with auto-encryption |
| `@shade/proto` | Compact binary wire format (smaller than JSON) |
| `@shade/streams` | Multi-lane chunk encryption — HKDF-derived per-lane keys, deterministic AES-GCM nonces, streaming SHA-256 |
| `@shade/transfer` | Transfer engine on top of streams: parallel lanes, resume, HTTP + WS transport with auto-fallback, integrity verification |
| `@shade/files` | Typed E2EE filesystem RPC — list/stat/mkdir/delete/move/read/write/getThumbnail + custom ops, auto inline/streams routing, production hooks (rate limit, retention, fingerprint gate, metrics), React hooks |
| `@shade/observer` | Live debugger backend (snapshot, SSE, dashboard) — see [README](./packages/shade-observer/README.md) |
| `@shade/widgets` | Embeddable React widgets including transfer uploader/downloader — see [README](./packages/shade-widgets/README.md) |
| `@shade/dashboard` | Standalone dashboard SPA bundled into the observer |
| `@shade/sdk` | High-level wrapper with `createShade()` one-liner, auto-publish, auto-establish, auto-replenish, `Shade.files` namespace |
| `@shade/cli` | `shade init` scaffolder + utilities (fingerprint, rotate, peer, dashboard, doctor) |

## Shade as a modular toolkit

Shade is split into packages so each project can depend on **only what it needs**—encrypted messaging, file transfer, prekey hosting, or lower-level building blocks. You do not need one giant stack for every use case.

For a **plain-language map** (which packages to add, what the prekey server does vs your own wiring, and where to start in code), see **[docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md)**.

## Publishing

All packages publish to a self-hosted Gitea npm registry on `gt.zyon.no`.

```bash
# Bump all packages in lockstep
bun run version 1.1.0

# Dry-run (pack all tarballs without publishing)
bun run publish:dry

# Real publish (requires GITEA_TOKEN env var)
bun run publish:all

# Or via CI: push a git tag v1.1.0 and .gitea/workflows/publish.yml runs
```

## Security properties

| Property | Description |
|----------|-------------|
| **Forward secrecy** | Compromising a key cannot decrypt past messages |
| **Post-compromise security** | Self-heals after key compromise on next DH ratchet |
| **Authentication** | Ed25519 identity signatures on prekey server writes |
| **Replay protection** | ±5 minute timestamp window on signed requests |
| **Constant-time comparisons** | Timing attacks on identity keys are blocked |
| **Memory zeroization** | Key material is zeroed after use (best-effort in JS) |
| **Identity verification** | Safety numbers (60 digits) for out-of-band comparison |
| **Identity rotation** | 7-day grace period for old sessions during rotation |

## Documentation

- [docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md) — **Modular toolkit**: pick packages by scenario (messages, files, browser, ops)
- [docs/files.md](./docs/files.md) — `@shade/files` API + design (filesystem RPC, custom ops, hooks, React)
- [docs/streams.md](./docs/streams.md) — `@shade/streams` + `@shade/transfer` deep dive
- [SECURITY.md](./SECURITY.md) — Reporting vulnerabilities, security policy
- [THREAT-MODEL.md](./THREAT-MODEL.md) — Honest threat model and assumptions
- [examples/](./examples/) — Runnable example applications, including
  [`07-streams-upload`](./examples/07-streams-upload) (multi-lane file transfer)
  and [`08-files-browser`](./examples/08-files-browser) (filesystem RPC)
- [MIGRATION.md](./MIGRATION.md) — How to replace existing crypto with Shade

## Deployment — one container per project

Shade ships as a self-contained Docker image. Deploy one container per project, point your app at it, done. Any stack (Bun, Python, Go, Rust, Kotlin) can use it — the container exposes a plain HTTP API documented in OpenAPI.

```bash
docker run -d \
  --name my-project-shade \
  -v my-project-shade:/data \
  -p 3900:3900 \
  -e SHADE_OBSERVER_TOKEN=change-me-to-at-least-16-chars \
  gt.zyon.no/stian/shade-prekey:latest
```

The container includes:
- **Prekey server** — `/v1/keys/*` REST API
- **Observer dashboard** — `/shade-observer/dashboard/` (off unless token is set)
- **OpenAPI spec** — `/openapi.yaml` and interactive `/docs` viewer
- **Prometheus metrics** — `/metrics`
- **Health check** — `/health`
- **Stale cleanup** — purges inactive identities automatically

See [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md) for the full deployment guide, environment variables, PostgreSQL config, backup strategy, and Dokploy instructions.

## License

MIT