Shade/README.md

Shade

End-to-end encryption library implementing the Signal Protocol (X3DH + Double Ratchet) for TypeScript/Bun. Drop into any project — frontend, backend, mobile — to get forward secrecy, post-compromise recovery, and self-healing security.

4.0.0 — General Availability. All V3.1 → V3.12 work is merged, the cross-platform vector suite is green on TS + Kotlin, the threat model has been refreshed for every new surface, and the core stack (X3DH, ratchet, storage encryption, recovery, WebRTC P2P, Key Transparency) has been packaged for external review. The wire format is unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers byte-for-byte. See MIGRATION.md § 0.3.x → 4.0 for the upgrade path and CHANGELOG.md § 4.0.0 for the consolidated release notes. Voice / Video have been moved to V5.0, to be built on top of the frozen 4.0 baseline.

What you get

• X3DH initial key agreement (works asynchronously via prekey bundles)
• Double Ratchet for per-message forward secrecy and post-compromise security
• Self-authenticated prekey server (Hono, Docker-ready) with rate limiting, metrics, health checks
• Persistent storage backends: SQLite (zero-config) and PostgreSQL (Drizzle)
• Identity rotation with grace period for old sessions
• Safety numbers (Signal-style fingerprints) for out-of-band verification
• Constant-time comparisons and memory zeroization for hardened operation
• Binary wire format that's significantly smaller than JSON
• Crash-safe — sessions survive container restarts, power outages, SIGKILL
• Live observability — bundled dashboard SPA + embeddable React widgets to see what's happening between every step
• E2EE file transfers — multi-lane chunked uploads/downloads with resume, integrity checks, and HTTP/WS fallback (@shade/streams + @shade/transfer)
• WebRTC P2P transport (V3.11) — opt-in RTCDataChannel upload path with public-STUN defaults, TURN-relay support, glare-safe peer pool, and automatic MultiTransportFallback back to HTTP when NAT traversal fails (@shade/transport-webrtc, docs/webrtc.md)
• Web Workers crypto — AEAD, HKDF, HMAC, X25519, Ed25519 and per-lane stream state run in a dedicated worker. 100 MB+ uploads stay smooth without frame drops, lane keys never cross the thread boundary (@shade/crypto-web/worker, docs/web-workers.md)
• E2EE filesystem RPC — typed list/stat/mkdir/delete/move/read/write/getThumbnail + custom ops between peers, with rate-limit, retention, and fingerprint-gate hooks (@shade/files)
• Async store-and-forward — deliver to offline recipients via a relay that holds ciphertext-only blobs with TTL, idempotent PUT, signed fetch/ack, and an onMessageQueued push-trigger hook (@shade/inbox + @shade/inbox-server)
• Social key recovery — Shamir-split your identity to n guardians; any threshold-many k together restore it on a new device. No centralized recovery agent; OOB-fingerprint gate on every guardian release; AES-GCM authenticates the reconstruction (@shade/recovery + <RecoverySetup /> / <RecoveryRequest /> / <RecoveryApprove /> widgets, docs/recovery.md)
• Key Transparency (V3.12) — opt-in append-only Merkle log over the prekey server. Every register / delete becomes a signed leaf; every bundle-fetch carries an inclusion proof; an Ed25519-signed Tree Head ties roots to a fixed log_id. A LightWitness cross-checks STHs across clients so a malicious server that splits its view or rewrites history is caught (@shade/key-transparency, docs/key-transparency.md)

Quick start

Add the Gitea npm registry to your project's .npmrc:

@shade:registry=https://gt.zyon.no/api/packages/Stian/npm/

Then install the SDK (one-liner for most use cases):

bun add @shade/sdk

Or install specific packages if you need fine-grained control:

bun add @shade/core @shade/crypto-web @shade/storage-sqlite

Even faster — scaffold a new project with the CLI:

bun add -g @shade/cli
shade init my-app --template bun-server
cd my-app && bun install && bun run start

Magic one-liner with the SDK:

import { createShade } from '@shade/sdk';

const shade = await createShade({
  prekeyServer: 'https://shade.example.com',
  storage: 'sqlite:/data/shade.db',
  address: 'alice@example.com',
});

// Send (auto-establishes session if none exists)
const envelope = await shade.send('bob@example.com', 'Hello, encrypted world!');

// Receive
const plaintext = await shade.receive('alice@example.com', incomingEnvelope);

// Your safety number for out-of-band verification
console.log(await shade.fingerprint);

Need to ship a file or expose a filesystem to a peer? Shade.files is the high-level entrypoint:

// Server side — Bob exposes a virtual filesystem
const stop = await shade.files.serve({
  list:  async (ctx) => ({ entries: await readdirAt(ctx.path), hasMore: false }),
  read:  async (ctx) => readAt(ctx.path),    // returns inline ≤ 256 KiB or streams
  write: async (ctx) => writeAt(ctx.args),   // receives inline or streams
  // + stat, mkdir, delete, move, getThumbnail, plus typed custom ops
});

// Client side — Alice consumes Bob's filesystem
const fs = await shade.files.client('bob');
await fs.write('/photos/cover.png', new Uint8Array(...));   // auto inline/streams
const result = await fs.read('/photos/cover.png');

Files ≤ 256 KiB ride inline in the RPC envelope; larger files automatically promote to multi-lane @shade/transfer streams with sha256 integrity. See docs/files.md for the full API.

Or use the lower-level packages directly if you need full control:

import { ShadeSessionManager } from '@shade/core';
import { SubtleCryptoProvider } from '@shade/crypto-web';
import { SQLiteStorage } from '@shade/storage-sqlite';

const manager = new ShadeSessionManager(
  new SubtleCryptoProvider(),
  new SQLiteStorage('/data/shade.db'),
);
await manager.initialize();

Architecture — keys vs. payloads

Shade splits the network into two planes. The prekey server only sees public keys; everything else rides the encrypted Double Ratchet between peers. If you remember nothing else from this README, remember this picture:

              Shade Prekey Server (Hono, public keys only)
                           │
                  POST /v1/keys/register   (signed)
                  GET  /v1/keys/bundle/:address
                  POST /v1/keys/replenish  (signed)
                  DELETE /v1/keys/:address (signed)
                           │
        ┌──────────────────┴──────────────────┐
        │                                     │
    [Client A]                            [Client B]
    ShadeSessionManager                   ShadeSessionManager
        │                                     │
        ├── X3DH (handshake via prekey srv) ─►│
        │                                     │
        │◄── Double Ratchet messages ────────►│   ← end-to-end,
        │   (ratchet 0x02 / chunks 0x11)      │     never on the
        │                                     │     prekey server
        │◄── @shade/transfer chunks ─────────►│
        │   POST /v1/transfer/:id/chunk        │   ← peer-to-peer
        │   GET  /v1/transfer/:id/state        │     HTTP, opaque
        │                                     │     ciphertext
    SQLiteStorage / PostgresStorage       SQLiteStorage / PostgresStorage
        (private keys + sessions)             (private keys + sessions)

What goes via the prekey server

• Identity public keys (Ed25519 + X25519)
• Signed prekeys + one-time prekey bundles
• Registration / replenish / delete writes, all Ed25519-signed
• Operator-only metrics, /health, and the optional observer dashboard

What does not go via the prekey server

• Message plaintext, ever. Encrypted ratchet envelopes flow peer- to-peer over whatever transport you choose (HTTP, WebSocket, your own broker).
• File chunks. @shade/transfer POSTs ciphertext directly to the receiver's /v1/transfer/:streamId/chunk route — the prekey server is not involved.
• Identity private keys. They never leave the device's storage.
• Filesystem RPC. @shade/files rides the Double Ratchet for control + small payloads, then promotes to direct @shade/transfer streams for larger blobs.
• Stream resume secrets. Persisted only on the local device, encrypted under a device-key derived from the identity signing key.

The prekey server is metadata-bearing (see THREAT-MODEL.md § 2): it sees who registers, who fetches whose bundle, and when. It does not see message contents, transfer contents, or session state.

For the full threat model and mitigations, read THREAT-MODEL.md. For deployment-time guarantees, read docs/PRODUCTION-CHECKLIST.md.

Packages

Package	Purpose
@shade/core	Protocol logic (X3DH, Double Ratchet, session manager, errors, events)
@shade/crypto-web	SubtleCrypto + @noble/curves provider, in-memory storage. Includes the V3.8 Web Workers entrypoint (@shade/crypto-web/worker) — drop-in WorkerCryptoProvider plus createEncryptStream / createDecryptStream TransformStream factories
@shade/storage-sqlite	Persistent SQLite storage (zero-config, bun:sqlite)
@shade/storage-postgres	PostgreSQL storage with Drizzle for shared databases
@shade/server	Prekey server (Hono routes, auth, rate limit, health, metrics)
@shade/transport	HTTP + WebSocket transport wrappers with auto-encryption
@shade/transport-bridge	WS → SSE → long-poll fallback chain (V3.7) — single IncomingMessage shape across transports for clients that can't keep a WebSocket open
@shade/transport-webrtc	V3.11 P2P chunk transport via RTCDataChannel. Plugs into @shade/transfer as an ITransferTransport; signaling rides Shade's own ratchet. Memory factory + native (globalThis.RTCPeerConnection) factory included; MultiTransportFallback([webrtc, http]) wired automatically when shade.configureWebRTC() is called.
@shade/proto	Compact binary wire format (smaller than JSON)
@shade/streams	Multi-lane chunk encryption — HKDF-derived per-lane keys, deterministic AES-GCM nonces, streaming SHA-256
@shade/transfer	Transfer engine on top of streams: parallel lanes, resume, HTTP + WS transport with auto-fallback, integrity verification
@shade/files	Typed E2EE filesystem RPC — list/stat/mkdir/delete/move/read/write/getThumbnail + custom ops, auto inline/streams routing, production hooks (rate limit, retention, fingerprint gate, metrics), React hooks
@shade/recovery	Social key recovery (V3.10) — Shamir-split identity to n guardians; threshold-many k reconstruct on a new device. AES-GCM-authenticated reconstruction; OOB-fingerprint gate per guardian release
@shade/key-transparency	Key Transparency (V3.12) — RFC 6962-style append-only Merkle log, address-index commitment, signed tree heads, and a LightWitness for split-view detection. Opt-in on both server and client. See docs/key-transparency.md
@shade/inbox-server	Async store-and-forward relay (V3.6) — Hono routes, signed PUT/FETCH/DELETE, per-recipient TTL + quota, idempotent on (address, msgId). Bundles into the same standalone container as the prekey server
@shade/inbox	Inbox client + durable outgoing queue + receive cursor + push-trigger hook (onMessageQueued); composes on top of Shade.send/Shade.receive for offline-recipient delivery
@shade/observer	Live debugger backend (snapshot, SSE, dashboard) — see README
@shade/widgets	Embeddable React widgets including transfer uploader/downloader — see README
@shade/dashboard	Standalone dashboard SPA bundled into the observer
@shade/sdk	High-level wrapper with createShade() one-liner, auto-publish, auto-establish, auto-replenish, Shade.files namespace
@shade/cli	shade init scaffolder + utilities (fingerprint, rotate, peer, dashboard, doctor)

Shade as a modular toolkit

Shade is split into packages so each project can depend on only what it needs—encrypted messaging, file transfer, prekey hosting, or lower-level building blocks. You do not need one giant stack for every use case.

For a plain-language map (which packages to add, what the prekey server does vs your own wiring, and where to start in code), see docs/SHADE-BY-SCENARIO.md.

Publishing

All packages publish to a self-hosted Gitea npm registry on gt.zyon.no.

# Bump all packages in lockstep
bun run version 1.1.0

# Dry-run (pack all tarballs without publishing)
bun run publish:dry

# Real publish (requires GITEA_TOKEN env var)
bun run publish:all

# Or via CI: push a git tag v1.1.0 and .gitea/workflows/publish.yml runs

Security properties

Property	Description
Forward secrecy	Compromising a key cannot decrypt past messages
Post-compromise security	Self-heals after key compromise on next DH ratchet
Authentication	Ed25519 identity signatures on prekey server writes
Replay protection	±5 minute timestamp window on signed requests
Constant-time comparisons	Timing attacks on identity keys are blocked
Memory zeroization	Key material is zeroed after use (best-effort in JS)
Identity verification	Safety numbers (60 digits) for out-of-band comparison
Identity rotation	7-day grace period for old sessions during rotation
Key Transparency (V3.12, opt-in)	Append-only Merkle log + signed tree heads + witness gossip — split-view and history-rewrite are detected by clients

Documentation

• docs/SHADE-BY-SCENARIO.md — Modular toolkit: pick packages by scenario (messages, files, browser, ops)
• docs/PRODUCTION-CHECKLIST.md — Pre-flight gates for going to production
• docs/files.md — @shade/files API + design (filesystem RPC, custom ops, hooks, React)
• docs/recovery.md — @shade/recovery social key recovery (V3.10): Shamir setup, guardian-side gates, threshold tuning
• docs/streams.md — @shade/streams + @shade/transfer deep dive (incl. hardening + retention)
• docs/inbox.md — @shade/inbox + @shade/inbox-server async store-and-forward relay (V3.6)
• docs/transport.md — @shade/transport-bridge SSE / long-poll / WS bridge layer (V3.7)
• docs/webrtc.md — @shade/transport-webrtc P2P transport (V3.11): NAT-traversal, TURN config, glare resolution, wire format, multi-fallback wiring
• docs/key-transparency.md — @shade/key-transparency (V3.12): operator + client onboarding, witness role, recovery procedures
• docs/V3.12-DESIGN.md — V3.12 design notat (threat model, RFC 6962 vs CONIKS choice, freshness model, open-questions resolution)
• docs/web-workers.md — V3.8 Web Workers crypto: setup, bundler recipes (Vite/Webpack/Rollup), Safari notes, lifecycle, threat-model
• SECURITY.md — Reporting vulnerabilities, security policy, threat-/test-matrix
• THREAT-MODEL.md — Honest threat model and assumptions
• examples/ — Runnable example applications, including 07-streams-upload (multi-lane file transfer) and 08-files-browser (filesystem RPC)
• MIGRATION.md — How to replace existing crypto with Shade

Deployment — one container per project

Shade ships as a self-contained Docker image. Deploy one container per project, point your app at it, done. Any stack (Bun, Python, Go, Rust, Kotlin) can use it — the container exposes a plain HTTP API documented in OpenAPI.

docker run -d \
  --name my-project-shade \
  -v my-project-shade:/data \
  -p 3900:3900 \
  -e SHADE_OBSERVER_TOKEN=change-me-to-at-least-16-chars \
  gt.zyon.no/stian/shade-prekey:latest

The container includes:

• Prekey server — /v1/keys/* REST API
• Observer dashboard — /shade-observer/dashboard/ (off unless token is set)
• OpenAPI spec — /openapi.yaml and interactive /docs viewer
• Prometheus metrics — /metrics
• Health check — /health
• Stale cleanup — purges inactive identities automatically

See docs/DEPLOYMENT.md for the full deployment guide, environment variables, PostgreSQL config, backup strategy, and Dokploy instructions.

License

MIT