Stian/Shade
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
Some checks failed
Test / test (push) Has been cancelled
Details

Browse Source 
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).

@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
  Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
  + per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
  (> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
  / shadeFilesReadStreamId correlation. Server-side TransformStream
  bridges accept inbound transfers immediately (engine rejects chunks
  that arrive before accept) and park the readable for the matching
  RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
  uploadDirectory()/downloadDirectory() with bounded concurrency pool
  (default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
  byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
  policy (traversal + percent-decode hardening), fingerprint gate
  (required/optional/reject), pluggable Ed25519 sig verification with
  ±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
  useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
  entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.

Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
  u16 to u32. The previous u16 silently truncated payloads above
  64 KiB — a hard correctness ceiling that blocked inline file ops
  up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
  only. Cross-platform Kotlin port (android/shade-android) updated to
  match; test-vectors/wire-format.json regenerated.

Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
  Concurrent decryptions of the same peer raced ratchet state
  (manifested as sporadic "Failed to decrypt — wrong key or tampered
  data" under load — surfaced once concurrent uploadDirectory pumped
  many writes in flight). Encrypt was already serialized via
  Shade.send's encryptChains; decrypt is now serialized at the
  manager layer too.

@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
  application-level key/value pairs that round-trip verbatim through
  stream-init plaintext. Used by @shade/files for write/read
  correlation; available to any consumer.

@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
  BackgroundTasks.setHook(name, fn) for runtime hook registration.

Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
  shade-sdk streams-bridge + shade-widgets transfer hooks were
  uncommitted prior to this session. Including them keeps the
  workspace consistent at 0.3.0 since @shade/files depends on them.

Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
  3× stable). Coverage spans unit (inline-threshold + concurrency),
  integration (read-write inline + streams up to 1 MiB, walk +
  upload/download directory, custom-op, metrics, SDK namespace
  end-to-end), and security (tampered-envelope sig verification,
  replay window, fingerprint gate, rate-limit + quota).

Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
  topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
  server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Sterister
2026-05-02 14:00:01 +02:00
parent 7e0f7320a9
commit fa770d3063
198 changed files with 20412 additions and 256 deletions
Download Patch File Download Diff File Expand all files Collapse all files

63
docs/SHADE-BY-SCENARIO.md Normal file
View File

@@ -0,0 +1,63 @@
# Shade by scenario — modular E2EE toolkit
This page is for **builders**, not cryptography specialists. Shade is packaged so you can **drop in small pieces** per project instead of importing a heavy “everything” stack.
---
## Plain-language mental model
1. **Identity & session (the hard crypto):** Shade establishes a **secret channel** between two parties using the same kind of cryptographic core as Signal (initial setup + ongoing “ratchet” updates). **You mostly call high-level APIs** (`send`, `receive`, fingerprints) rather than assembling primitives by hand.
2. **Who is “the server”?**
The **prekey server** only helps with **public key material** (so Alice can fetch Bob’s public bundle before the first message). It is **not** your general-purpose message relay unless **you** build that separately. Normal **message payloads and file chunks** typically flow over **your** transport (your HTTP routes, websocket, bridge, queue, etc.).
3. **Small vs large payloads:**
Short messages ride the usual ratchet envelopes. Very large payloads use **Streams + Transfer**: secrets are negotiated over the ratchet; ** ciphertext chunks** ship over optimized HTTP/WebSocket transports with parallelism and resume.
4. **Trust:** Strong encryption does **not** replace **verifying who you are talking to**. For high-stakes use, compare **safety numbers** out-of-band (see [THREAT-MODEL.md](../THREAT-MODEL.md)).
---
## Scenario → minimum packages
Pull in **one row** that matches your project; add optional columns only when needed.
| Scenario | What you need | Minimum packages / surface | Where to start |
|----------|----------------|----------------------------|----------------|
| **Backend or Bun service** — encrypted messages between users | Session storage + crypto provider + prekey URL | `@shade/sdk` + `sqlite:` or `@shade/storage-postgres` | `createShade()` → `send` / `receive` |
| **Browser / frontend** — same, in the client | Web crypto + durable or memory storage | `@shade/sdk` or `@shade/core` + `@shade/crypto-web` (+ storage you provide) | Same APIs; ensure `prekeyServer` is reachable from the browser (CORS, etc.) |
| **Large files** — resumable E2EE upload/download | Above + stream protocol + HTTP (or WS) transport | `@shade/sdk` (re-exports transfer) + mount transfer routes on **your** HTTP server | `shade.upload` / `onIncomingTransfer` — see [streams.md](./streams.md) |
| **React UI** — upload/download widgets | Runtime from SDK + widgets | `@shade/sdk` + `@shade/widgets` | `ShadeRuntimeProvider`, `useShadeUpload` / `useShadeDownload` |
| **Prekey hosting only** — one container per product | No app crypto in the container | Docker image / `@shade/server` | Deploy prekey image; point `prekeyServer` at it from apps |
| **Maximum control** — custom wire, custom transport | Wire + session manager | `@shade/core` + `@shade/proto` (+ your storage + crypto provider) | `ShadeSessionManager`, encode/decode envelopes yourself |
| **HTTP or WebSocket convenience** | Auto-wrap application bytes | `@shade/transport` on top of your stack | Use when you want transport helpers, not a new protocol |
| **Android** | Byte-compatible with TS (roadmap) | `shade-android` module | See [android/shade-android/README.md](../android/shade-android/README.md) — parity work in progress |
You can **mix rows**: e.g. backend with `@shade/sdk` + SQLite for sessions, separate service mounting `transfer` routes, browser clients using `@shade/widgets`.
---
## New project checklist (lightweight)
1. Run a **prekey server** (Docker or embedded `@shade/server`) for your environment.
2. Pick **storage** (`sqlite:…`, Postgres, or project-specific adapter implementing the core storage interfaces).
3. Choose **surface**: usually `@shade/sdk` unless you truly need `@shade/core` only.
4. For files: enable **transfer routes** and authenticate chunk uploads using the patterns in the SDK (see streams doc).
5. Run **`shade doctor`** when something fails in production-ish setups (install the CLI as in repository [Quick start](../README.md#quick-start)); coverage is evolving — roadmap in [V2.2](./V2.2.md).
---
## Related docs & roadmap
| Topic | Doc |
|--------|-----|
| File transfer architecture | [streams.md](./streams.md) |
| Deployment & operations | [DEPLOYMENT.md](./DEPLOYMENT.md) |
| Threat model | [THREAT-MODEL.md](../THREAT-MODEL.md) |
| Planned improvements | [V2.1](./V2.1.md), feature backlog [V2.2](./V2.2.md), trust/ops [V2.3](./V2.3.md) |
---
## Version note
This file describes how Shade is **intended** to be composed. Package names and re-exports may gain small aliases over time; the **scenario table** should remain the source of truth for “what to install for what job.” Update this page when adding major surfaces (new transport bridges, richer `shade init` templates, etc.).

151
docs/V2.1.md Normal file
View File

@@ -0,0 +1,151 @@
# Shade V2.1 — Improvements (infrastructure, storage, operations, security)
This document describes **improvements** agreed for next-generation work on Shade: clearer product story, stronger storage, mobile parity, operational hardening, transfer abuse, and a formal security narrative.
**Audience:** **Maintainers and contributors** implementing the changes. Add status fields as items land in code/docs.
---
## 1. Clear “who is the server?” and data flow
**Problem:** New users may think the prekey server is a message hub or that all E2EE traffic goes through the Shade container.
**Goal:** One consistent explanation across the root README, package READMEs, and optional onboarding: **the prekey server distributes public keys and bundles**; **actual messages and (typically) file chunks go through your app’s own channel** (your transport, your backend, your URLs).
**Deliverables (proposal):**
- Diagram + short “keys vs payloads” text in the root README and in `@shade/server` README.
- Link to `THREAT-MODEL.md` from the same section (MITM on first contact ↔ safety numbers).
- Optionally one “concept page” (or extend `MIGRATION.md`) with typical architecture: *A ↔ B via app; both talk to the prekey host for X3DH material*.
**Acceptance criteria:** A new developer without domain background understands in one reading *what* goes to the Shade server and *what* does not.
---
## 2. Optional encryption of storage (at-rest)
**Problem:** `THREAT-MODEL.md` states that a stolen DB + filesystem can expose private keys because Shade does not encrypt the storage layer by default.
**Goal:** **Opt-in** protection for sensitive state (identity, session, optional stream resume secrets) with keys that **do not** live in plaintext in the DB — e.g. OS keychain/Keystore, passphrase + KDF, or an explicit device key injected by the app.
**Design principles:**
- Default developer experience (dev, simple demos) stays unchanged or includes a clear “insecure mode” warning in docs.
- APIs implementable per platform (Bun/SQLite, Postgres, web/IndexedDB, Android).
- Document limitations: what remains uncovered (e.g. active memory compromise).
**Acceptance criteria:** Threat model updated for “when encrypted storage is enabled”; at least one reference implementation + migration note.
---
## 3. Android parity and a published roadmap
**Problem:** `shade-android` is under development; drift from the TS SDK undermines the “byte-compatible” promise.
**Goal:** A **published roadmap** (milestones + what counts as parity vs TS-only) and **CI running shared test vectors** as a merge gate before release.
**Deliverables:**
- Roadmap section in `android/shade-android/README.md` or dedicated `ROADMAP-ANDROID.md` with explicit cross-checkpoints: wire format, fingerprints, rotations, streams (`0x11`) where applicable, resume semantics.
- CI job that fails on Kotlin vs TS vector mismatch.
**Acceptance criteria:** Parity coverage is visible and enforceable; the first critical cross-surface (e.g. core ratchet + proto) is green before a “production” label.
---
## 4. Operational hardening — prekey container and production
**Problem:** Many teams deploy the Docker image quickly; mistakes around TLS, backups, and secrets add avoidable risk.
**Goal:** A **production checklist**: TLS termination, volume backup (`/data`), rotation of `SHADE_OBSERVER_TOKEN`, use of `SHADE_PREKEY_PG_URL` vs SQLite, observability hooks, logging levels, meaning of stale cleanup parameters.
**Deliverables:**
- Extend `docs/DEPLOYMENT.md` or add short `docs/PRODUCTION-CHECKLIST.md` with bullet defaults.
- Link from the main README under “Deployment”.
**Acceptance criteria:** A checklist operators can follow without reading the whole codebase first.
---
## 5. Abuse and resource limits on the transfer plane
**Problem:** Parallel lanes and large uploads can be abused for resource or storage if consumer mounts of `createTransferRoutes()` share no coherent policy.
**Goal:** Documented **limits and patterns**: authentication (already an active SDK topic), max stream size, TTL for temporary chunk storage, quotas per identity or IP where sensible.
**Deliverables:**
- Guidelines in `docs/streams.md` or a dedicated “Transfer hardening” section.
- Optional helpers or middleware examples in `@shade/transfer` / server routes for common limits (without forcing every deployment into one DB model).
**Acceptance criteria:** A clear “recommended minimum” for production that teams can copy.
---
## 6. Security review and formal test / narrative
**Problem:** Enterprises and security-conscious users often ask for independent review and a traceable test matrix.
**Goal:** Plan for **independent crypto review** (timing, scope, deliverables) and a **published test / threat matrix** linking `THREAT-MODEL.md` to concrete automated tests (replay, tamper, out-of-order, resume, etc.).
**Deliverables:**
- Internal checklist “preparing for external review” (which files, assumptions, known limits).
- Short section in `SECURITY.md` on review status and how to report findings.
**Acceptance criteria:** One authoritative source for “what is tested automatically” vs “what needs manual/MITM/out-of-band process”.
---
## Dependencies and order (proposal)
| Priority | Topic | Note |
|---------|-------|------|
| Low friction | §1, §4 | Documentation and checklists only |
| Medium | §5 | Design + docs + possibly small API helpers |
| High effort | §2, §3 | Platform work and long-term maintenance |
| Coordinated | §6 | Parallel with maturity in §2–§5 |
---
## Document versioning
- **V2.1** — first consolidated plan for the six improvement areas above. Update with dates and short per-section status lines as work starts and completes.
Gitt der du er nå (post-streams 0.2.0), ville jeg prioritert:
1. Sosial nøkkel-recovery — løser "hva skjer hvis jeg mister telefonen?" som er det store hullet i alle
E2EE-systemer
2. P2P WebRTC transport — naturlig kompletterende til streams; gir massiv throughput-gevinst gratis
3. E2EE Pub/Sub — lavt scope, høy nytte, og passer perfekt med din "magic drop-in"-filosofi
Disse tre har alle samme egenskap: de er infrastruktur-features som hver app du bygger fremover vil dra nytte av,
ikke bare Dispatch. Helt i tråd med "Shade som komponent som dras inn i hvilket som helst prosjekt".
/btw Hva med Shade videostreaming, VOIP streaming, og lignende?
Magic drop-in-tilnærming (din filosofi)
Tenk deg at i en Dispatch-app kan du bare gjøre:
<ShadeVideoCall to="device:server-admin" />
<ShadeVoiceButton to={peerAddress} />
<ShadeBroadcaster streamKey="game-stream-1" />
<ShadeViewer streamKey="game-stream-1" />
…og det bare fungerer, E2EE, P2P når mulig, fallback til relay automatisk. Det er det samme mønsteret som streams
0.2.0, bare for sanntid.
Realisme-sjekk
Video/VOIP er det vanskeligste i hele E2EE-verdenen. Signal brukte år på å få det riktig. Du bør:
1. Ferdigstille streams 0.2.0 først (verifiserer crypto-fundamentet)
2. Bygge P2P WebRTC-transport som separat milestone
3. Da har du alle byggeklossene og Voice 0.4.0 blir 70% gjenbruk
Men ja — dette hører absolutt hjemme i Shade. Shade som "alt-i-ett E2EE-platform" er en mye sterkere posisjon enn
"bare messaging + filer". Du kan bli til E2EE hva Twilio er til vanlig kommunikasjon.

126
docs/V2.2.md Normal file
View File

@@ -0,0 +1,126 @@
# Shade V2.2 — Feature plan: product, platform, and developer experience
This document gathers **planned features** that extend Shade beyond today’s core (X3DH + Double Ratchet + Streams/transfer): groups, asynchronous delivery, richer file UX, web workers, CLI, API docs, and scaffolding.
Add optional per-feature status (Idea / Design / IMP / Done).
---
## 1. Groups / multiple participants
**Vision:** Beyond strict 1:1 — multiple identities in the same “conversation” or shared context (messages and possibly shared file/stream policy).
**Challenges:**
- Today’s Signal-like core is naturally 1:1; groups need either **pairwise sessions per member**, **sender keys / fan-out**, or a **dedicated group protocol** (larger architectural step).
- Lifecycle: invites, member removal, compromised device, history, and group PCS.
**Goals for early milestones (proposal):**
1. **Document** a recommended pattern for “group-lite” (e.g. coordinator relays ciphertext without decrypting + clients encrypt per-recipient).
2. **Optional** minimal API making fan-out easier in the SDK (without promising full MLS).
**Acceptance criteria (MVP definition):** Explicit scope in docs + one reference architecture; no ambiguous “group is done” without an updated threat model.
---
## 2. Async store-and-forward messaging
**Vision:** Recipient need not be online; **ciphertext** is stored temporarily and fetched when the recipient returns — the server never sees plaintext.
**Distinction from the prekey server:**
- Prekey stays **public keys only** (or extended only under strict policy).
- A **dedicated relay/inbox service** (or app-owned backend) holds **encrypted blobs only** with TTL, idempotency, and authorization (who may list/fetch).
**Deliverables (proposal):**
- Protocol sketch: address registration, `PUT` blob, `GET`/`DELETE` or lease, replay protection at the application layer.
- SDK helpers: outgoing queue, poll/pull, or push-notification hook (without dictating mobile platform).
**Acceptance criteria:** Threat-model section “what the relay sees” + reference implementation or example app.
---
## 3. File metadata and preview
**Vision:** Richer UX **without** leaking sensitive content to the server: filename, MIME type, length where known; **optional** client-generated thumbnails or previews encrypted as separate blocks or small payloads on the control init path.
**Technical considerations:**
- Anything sent must be **E2EE** or omitted; plaintext metadata on the server must be deliberate and minimal.
- Thumbnails should use **format hardening** on the client (size limits, sandboxing in UI).
**Acceptance criteria:** Extended `stream-init` (or sidecar envelope) with optional fields + widget support for “preview when available”.
---
## 4. Web: worker-based crypto and streaming
**Vision:** Large files in the browser without blocking the main thread or blowing RAM — **Web Crypto / noble** inside a **Worker**, **ReadableStream/WritableStream** end-to-end chunk pipeline aligned with `@shade/streams` / transfer.
**Deliverables:**
- `@shade/crypto-web` (or companion) patterns: transferable buffers, lifecycle, errors surfaced to the UI.
- Documented constraints (Safari, chunk sizing, Service Worker vs dedicated worker).
**Acceptance criteria:** E2E demo or test that sends multi‑MiB through a worker without a blocking UI.
---
## 5. CLI: `shade doctor`
**Vision:** One command that **diagnoses the environment** before production debugging.
**Typical checks (proposal):**
- Reachability of `prekeyServer` (`/health`, optional OpenAPI).
- Local config: storage path, rotation headers, clock skew (relevant for signed requests).
- **Streams:** transfer routes mounted, auth matches expected key, `GET .../state` behaves as expected in test mode.
- CLI / Node/Bun runtime versions and `@shade/*` packages where readable from `package.json`.
**Acceptance criteria:** `shade doctor` with exit codes suitable for CI (warn vs fail).
---
## 6. OpenAPI / docs
**Vision:** All HTTP contracts teams are expected to implement (prekey **and** transfer) appear in **one** OpenAPI story or clearly linked specs — not only README examples.
**Deliverables:**
- Consolidate or cross-reference `openapi.yaml` with transfer endpoints (`/v1/transfer/*`) and security schemes for chunk upload.
- `/docs` (Redoc or similar) or published static artifacts for versioned specs.
**Acceptance criteria:** Generated client (e.g. Python/Go) from spec without manual fixes for the happy path.
---
## 7. `shade init`
**Vision:** Scaffolding from **empty repo** to a **minimal runnable** app with Shade and optional streams.
**Extensions:**
- New or extended template: **minimal Hono/Fastify** app with `Shade.transferRoute()` mounted, auth example matching the SDK authenticator, `.env` template.
- Optional: demo `shade doctor` after init.
**Acceptance criteria:** `shade init …` produces a project that `bun install && bun run start` runs with documented env vars.
---
## Dependencies between items
```text
shade init ─────► doctor (same conventions for URLs and secrets)
openapi/docs ◄── transfer + prekey (single source)
web workers ───► streams UX (large file in browser)
groups ◄──────── store-and-forward (often related socially/technically)
metadata/preview► widgets + proto/control plane
```
---
## Document versioning
- **V2.2** — feature backlog as described. Split into issues/ADR per feature when implementation starts.

102
docs/V2.3.md Normal file
View File

@@ -0,0 +1,102 @@
# Shade V2.3 — Tillit, retention, integrasjon og observability
Dette dokumentet beskriver **høyere ambisjonsnivå** og **plattformkryssende** arbeid: brukertilfeller der tillit må være **eksplisitt**, data må **utkrympes**/ryddes automatisk, apper må **enkelt koble seg på** kjente transportmønstre, og drift må **observere** uten å lekke innhold.
---
## 1. Key transparency / bundle-attestasjon **eller** kritiske fingerprint-øyeblikk i UI
Dette kan sees som **to spor** samme mål: redusere tillit til én korrumperbar prekey-server uten brukerens merknad.
### Spor A — Key transparency / bundle-attestasjon (ambisiøst)
**Idé:** Logging av **hvilket bundle som ble utlevert når**, kryptografisk forankret og **verifiserbar av klienter** eller tredjeparts-audit (inspirert av KT-litteratur, tilpasset Shade sin trusselmodell).
**Leveranser (målpris høyt):**
- Trusselmodell-oppdatering: *hva* CT/attest løser vs *fortsatt MITM før første verifisering*.
- Designnotat: datastruktur, friskhetsbevis, klient-verifikasjonssteg, operatørkost.
**Risiko:** Kompleks drift og kryptodesign — bør være **valgfritt** lag for motiverte deploys.
### Spor B — Fingerprints i app-UI på kritiske hendelser (pragmatisk)
**Idé:** Gjør **safety numbers** synlige og **handlingspålagte** i presiserte flyt:
- **Før første stor fil** (eller før første stream over terskel i bytes).
- **Før «trust this device»** / backup-importer / ny enhet som gjenbruker identitet.
**Leveranser:**
- `@shade/widgets` + SDK-hooks: modal/sheet med fingerprint, kopier-OOB-tekst, «jeg har verifisert».
- Dokumenterte UX-retningslinjer (unngå «alert fatigue» kun på lave risiko-events).
### Felles akseptansekriterier
Enten spor A eller B må ha **eksplisitt testcoverage** på «blokkerer/handhever verifisering» der det er lovet, og trusselmodellen må nevne kombinasjonen av OOB + tekniske grep.
---
## 2. Retention policies
**Vision:** Standardiserte **TTL- og oppryddingsregler** for data Shade-økosystemet etterlater på server eller i klient-lagring — spesielt:
- **Stream-state** og midlertidige chunk-referanser etter fullførte/avbrutte transfers.
- **Eventuell** inbox/relay-ciphertext (`V2.2`).
- Prekey-server: kobling til eksisterende `SHADE_STALE_DAYS` / cleanup, plus **policy-dokumentasjon** for operatører.
**Leveranser:**
- Default-anbefalinger (f.eks. «ferdige streams: prune etter N dager») i `@shade/storage-*` helpers eller server-factory.
- Konfigurerbare hooks: `maxAge`, `maxBytesPerIdentity`, cron vs on-access prune.
**Akseptansekriterier:** Ingen «uendelig vekst» som default i nye templates; dokumentert adferd i `streams.md` / deployment.
---
## 3. Ferdig «bridge» (transport)
**Vision:** Apper som ikke kan eller vil bruke WebSocket, får **ferdig mønster** for mottak av små meldinger eller kontroll-signaler.
**Eksempler:**
- **Server-Sent Events (SSE)** som **ren ciphertext-pipe** eller som signal «hent fra inbox» uten plaintext på server (kombinasjon med `V2.2`).
- **Lang-poll fallback** dokumentert ved siden av WS i `@shade/transport`.
**Leveranser:**
- Modulær `bridge`-pakke eller tydelig undermodul med få eksponerte typer og én felles `IncomingMessage`-modell på klient.
**Akseptansekriterier:** Ett fungende eksempel + test som viser dekryptering likt eksisterende transport.
---
## 4. Observability
**Vision:** Produksjonsteam får **målepunkter og spor** uten **innholdslekkasjer** eller kryptomatrise i logger.
**Forslag til innhold:**
- **Metrics (Prometheus-stil eller vendor-nøytralt):** opplastings-/nedlastings-varighet, lane-telling, retry counts, abort vs complete rates, HTTP/WS-feilkoder (aggregert).
- **OpenTelemetry:** spans rundt `TransferEngine` og prekey-endepunkter (uten payload-lengde i klartekst som PII — bruk binære størrelser eller binning).
- **Sampling og PII-policy** dokumentert (ikke logg adresser i full hvis compliance krever maskering).
**Akseptansekriterier:** Opt-in flags (default av i lib, på i server-container der det gir mening), og `docs/` avsnitt om hva som **aldri** skal logges.
---
## Prioritering mot V2.1 / V2.2
| Dette dokument (V2.3) | Naturlig forutsetning |
|----------------------|------------------------|
| Retention | Streams/transfer i bruk (`V2.1` §5, `V2.2` metadata) |
| Bridge | `V2.2` store-and-forward eller egen meldingskanal |
| UI fingerprints | Widgets/SDK allerede i bruk |
| KT / attest | Moden trusselmodell + juridisk/operativ vilje |
| Observability | Stabil nok intern API for hooks |
---
## Versjonering
- **V2.3** — første samlet plan for tillit, retention, bridge og observability. Splitt i ADR-er når konkret design er valgt.

201
docs/files.md Normal file
View File

@@ -0,0 +1,201 @@
# `@shade/files` — E2EE filesystem RPC
`@shade/files` exposes a typed, end-to-end-encrypted RPC surface for
filesystem-style operations between two Shade peers. Both sides ride a
single Double Ratchet session for control envelopes; large-file content
(`> 256 KiB`) flows through `@shade/transfer` lanes, correlated back to
the RPC by an opaque `userMetadata.shadeFilesWriteId` /
`shadeFilesReadStreamId`.
The package is a **primitive**, not a UI: it ships hooks and clients, not
file managers. Consumers (e.g. Dispatch, Mail, Drive-style apps) keep
their own UI and plug `@shade/files` underneath.
## Quick start
### Server (Bob exposes a filesystem)
```ts
import { createShade } from '@shade/sdk';
const bob = await createShade({ prekeyServer, address: 'bob' });
bob.configureTransfers({ resolveBaseUrl: ... });
Bun.serve({ port: 8080, fetch: (await bob.transferRoute()).fetch });
const stop = await bob.files.serve({
list: async (ctx) => ({ entries: await readdirAt(ctx.path), hasMore: false }),
stat: async (ctx) => statAt(ctx.path),
mkdir: async (ctx) => mkdirAt(ctx.path, ctx.args.recursive),
delete: async (ctx) => deleteAt(ctx.path, ctx.args.recursive),
move: async (ctx) => moveAt(ctx.args.src, ctx.args.dst, ctx.args.overwrite),
read: async (ctx) => readAt(ctx.path), // returns inline | streams
write: async (ctx) => writeAt(ctx.args), // receives inline | streams
getThumbnail: async (ctx) => thumbnailAt(ctx.path, ctx.args.size, ctx.args.format),
});
// Later: stop the handler.
await stop();
```
### Client (Alice consumes Bob's filesystem)
```ts
const alice = await createShade({ prekeyServer, address: 'alice' });
alice.configureTransfers({ resolveBaseUrl: ... });
Bun.serve({ port: 8081, fetch: (await alice.transferRoute()).fetch });
const fs = await alice.files.client('bob');
const page = await fs.list('/photos');
await fs.write('/photos/cover.png', new Uint8Array(...)); // auto inline/streams
const result = await fs.read('/photos/cover.png');
if (result.kind === 'inline') console.log(result.bytes.byteLength);
else for await (const _chunk of /* result.stream */) { /* ... */ }
```
## Op surface
| Op | Args | Result |
|----------------|------------------------------------------|-----------------------------------------|
| `list` | `path`, `cursor?`, `pageSize?`, `filter?`| `{ entries, hasMore, nextCursor? }` |
| `stat` | `path` | `FileEntry` |
| `mkdir` | `path`, `recursive?` | `{ entry: FileEntry }` |
| `delete` | `path`, `recursive?` | `{ deletedCount }` |
| `move` | `src`, `dst`, `overwrite?` | `{ entry: FileEntry }` |
| `read` | `path`, `range?`, `preferInline?` | inline `{ bytes, sha256, size }` or streams `{ stream, size, sha256 }` |
| `write` | `path`, `Uint8Array \| Blob \| Stream` | `{ entry: FileEntry }` |
| `getThumbnail` | `path`, `size: 64\|128\|256\|512`, `format?` | `{ bytes, format, width, height, sha256 }` |
| `custom<K>` | typed via `CustomOpsMap[K]` | typed via `CustomOpsMap[K]` |
`MUTATION_OPS = { mkdir, delete, move, write, custom }` — these auto-generate
an idempotency key per logical call so transparent retries under the SDK
don't produce duplicates.
## Inline vs streams
The threshold is `INLINE_THRESHOLD = 256 * 1024` bytes (plaintext). The
client's `decideInline()` runs at `write` time:
* `Uint8Array` / `Blob` with known size → direct comparison.
* Bare `ReadableStream` → peek the first chunks; promote to streams as
soon as the buffered prefix exceeds the threshold.
Streams paths kick `shade.upload(...)` with `userMetadata.shadeFilesWriteId`
in parallel with the RPC envelope. The server's streams-bridge accepts the
inbound transfer immediately into a `TransformStream` and parks the
readable side until the matching `write` RPC arrives.
## Custom ops
Augment `CustomOpsMap` once globally for type-safe consumer-defined ops:
```ts
declare module '@shade/files' {
interface CustomOpsMap {
'app.deploy': CustomOpDef<{ jarPath: string }, { deploymentId: string }>;
}
}
// Server registers a Zod-backed handler:
await shade.files.serve({
custom: {
'app.deploy': {
args: z.object({ jarPath: z.string() }),
response: z.object({ deploymentId: z.string() }),
handler: async (args, ctx) => ({ deploymentId: deploy(args.jarPath) }),
},
},
});
// Client gets typed I/O:
const result = await fs.custom('app.deploy', { jarPath: '/mods/foo.jar' });
// ^? { deploymentId: string }
```
## Production hooks
All hooks are callbacks the consumer plugs in — `@shade/files` enforces
the *mechanism*; the app owns the *policy*.
```ts
await shade.files.serve({
pathPolicy: { rootScope: '/srv/shade-root', forbidTraversal: true },
rateLimits: {
maxOpsPerMinutePerSender: 600,
maxBytesPerHourPerSender: 10 * 1024 ** 3,
opCost: { read: 1, write: 5, delete: 3, default: 1 },
},
idempotency: { ttlMs: 60 * 60 * 1000, maxEntriesPerSender: 10_000 },
requireFingerprintVerifiedFor: (ctx) =>
['delete', 'write', 'mkdir'].includes(String(ctx.op)) ? 'required' : 'optional',
isFingerprintVerified: (sender) => verifiedSet.has(sender),
verifySender: async (sender, canonical, sig) => {
return ed25519.verify(base64ToBytes(sig), canonical, getPubKey(sender));
},
onMetric: (name, value, tags) => prometheus.observe(name, value, tags),
onError: (err, ctx) => logger.error({ op: ctx.op, sender: ctx.sender }, err),
});
```
## React
```tsx
import { ShadeFilesProvider, useFileList } from '@shade/files/react';
function FileBrowser({ shade, peer }: { shade: Shade; peer: string }) {
return (
<ShadeFilesProvider shade={shade}>
<Listing peer={peer} path="/" />
</ShadeFilesProvider>
);
}
function Listing({ peer, path }) {
const { entries, isLoading, hasMore, loadMore, refresh } = useFileList(peer, path);
if (isLoading) return <Spinner />;
return (
<ul>
{entries.map((e) => <li key={e.name}>{e.name} ({e.kind})</li>)}
{hasMore && <button onClick={loadMore}>More</button>}
</ul>
);
}
```
`useFileTransfer` exposes a generic state machine for `BulkTransferHandle`s
returned by `uploadDirectory()` / `downloadDirectory()`:
```tsx
const { start, abort, state, filesDone, filesTotal, bytesDone } = useFileUpload();
const handle = uploadDirectory(fs, localDir, '/uploaded');
useEffect(() => { start(handle); return () => void abort(); }, []);
```
## Path safety
The dispatcher applies `validatePath()` before invoking the user handler:
1. Length check on raw input.
2. Forbidden-bytes check (NUL/CR/LF/DEL/backslash).
3. Percent-decode (defends against `%2e%2e` smuggling).
4. POSIX normalization.
5. `..` traversal rejection.
6. Root-scope boundary check.
7. Consumer-supplied `extra` predicate.
The user handler receives the *normalized* path; using `args.path` raw is a
TOCTOU bug.
## Wire compatibility
`@shade/files` 0.3.0 requires `@shade/proto` 0.3.0+. The proto layer's wire
VERSION was bumped from `0x01` to `0x02` to lift the 64 KiB length-prefix
ceiling that previously capped ratchet payloads. **Sessions are
incompatible across the bump**; both peers must run 0.3.0+.
## Related modules
* `@shade/streams` — chunk encryption, lane key derivation. Indirect dep.
* `@shade/transfer` — multi-lane transport with HTTP / WS fallback.
* `@shade/sdk` — `Shade.files` getter; `BackgroundHooks.onPruneFiles` for
retention.

128
docs/shade-overview.html
View File

@@ -1,9 +1,9 @@
<!DOCTYPE html>
<html lang="no">
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Shade — ende-til-ende kryptering som modul</title>
<title>Shade — end-to-end encryption as a module</title>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link href="https://fonts.googleapis.com/css2?family=Fraunces:ital,opsz,wght@0,9..144,400;0,9..144,600;0,9..144,700;1,9..144,400&family=Source+Serif+4:ital,opsz,wght@0,8..60,400;0,8..60,600;1,8..60,400&display=swap" rel="stylesheet" />
@@ -364,32 +364,32 @@
<header>
<h1>Shade</h1>
<p class="lede">
En gjenbrukbar modul for <strong style="color: var(--text); font-weight: 600;">ende-til-ende-kryptert</strong> kommunikasjon i egne apper — med samme type protokoll som brukes i Signal.
A reusable module for <strong style="color: var(--text); font-weight: 600;">end-to-end encrypted</strong> communication in your own apps — using the same kind of protocol as Signal.
</p>
<div class="badge-row">
<span class="badge">X3DH</span>
<span class="badge">Double Ratchet</span>
<span class="badge">TypeScript</span>
<span class="badge">Plattformagnostisk crypto</span>
<span class="badge">Platform-agnostic crypto</span>
</div>
</header>
<section id="hva">
<h2>Hva gjør prosjektet?</h2>
<section id="what">
<h2>What does the project do?</h2>
<p>
<strong>Shade</strong> er et monorepo som implementerer sikker meldingskryptering mellom to parter (for eksempel nettleser og server, eller to klienter). Meldingene er kryptert slik at transportlaget (HTTP, WebSocket, e.l.) bare ser uleselige bytes — ikke innholdet.
<strong>Shade</strong> is a monorepo that implements secure messaging encryption between two parties (for example browser and server, or two clients). Messages are encrypted so the transport layer (HTTP, WebSocket, etc.) only sees opaque bytes — not the content.
</p>
<div class="callout">
<strong>Kjerneideen:</strong> Du bygger inn <code>ShadeSessionManager</code> (fra <code>@shade/core</code>) sammen med en <code>CryptoProvider</code> (f.eks. Web Crypto i nettleser/Bun) og lagring. Deretter kan du kalle <code>encrypt</code> / <code>decrypt</code> per motpart, akkurat som i demo-koden <code>demo.ts</code>.
<strong>Core idea:</strong> Embed <code>ShadeSessionManager</code> (from <code>@shade/core</code>) together with a <code>CryptoProvider</code> (e.g. Web Crypto in the browser/Bun) and storage. Then call <code>encrypt</code> / <code>decrypt</code> per peer, just like in the demo code <code>demo.ts</code>.
</div>
<p>
Første melding til noen ny inneholder nøkkelavtale (X3DH). Etterpå bruker hver melding <em>Double Ratchet</em>: nye meldingsnøkler og periodiske DH-steg gir <strong>forward secrecy</strong> (gamle meldinger overlever ikke nøkkellekkasje) og <strong>post-compromise security</strong> (systemet «helbreder» seg over tid etter kompromittering).
The first message to someone new performs key agreement (X3DH). After that each message uses the <em>Double Ratchet</em>: fresh message keys and periodic DH steps provide <strong>forward secrecy</strong> (past messages do not survive key compromise) and <strong>post-compromise security</strong> (the system “recovers” over time after a compromise).
</p>
</section>
<section id="pakker">
<h2>Pakkene (hvordan det henger sammen)</h2>
<div class="tabs" role="tablist" aria-label="Pakkeoversikt">
<section id="packages">
<h2>Packages (how they fit)</h2>
<div class="tabs" role="tablist" aria-label="Package overview">
<button type="button" class="tab-btn" role="tab" id="tab-core" aria-selected="true" aria-controls="panel-core">shade-core</button>
<button type="button" class="tab-btn" role="tab" id="tab-crypto" aria-selected="false" aria-controls="panel-crypto">shade-crypto-web</button>
<button type="button" class="tab-btn" role="tab" id="tab-proto" aria-selected="false" aria-controls="panel-proto">shade-proto</button>
@@ -397,117 +397,117 @@
<button type="button" class="tab-btn" role="tab" id="tab-server" aria-selected="false" aria-controls="panel-server">shade-server</button>
</div>
<div id="panel-core" class="tab-panel active" role="tabpanel" aria-labelledby="tab-core">
<p style="margin-top:0"><strong>Protokollen.</strong> X3DH, Double Ratchet, sesjonstyper, feiltyper. Ingen plattformkrypto her — bare grensesnittet <code>CryptoProvider</code>.</p>
<p style="margin-top:0"><strong>The protocol.</strong> X3DH, Double Ratchet, session shapes, errors. No platform crypto here — only the <code>CryptoProvider</code> interface.</p>
<ul>
<li><code>ShadeSessionManager</code> — høynivå-API: <code>initialize</code>, <code>createPreKeyBundle</code>, <code>initSessionFromBundle</code>, <code>encrypt</code>, <code>decrypt</code></li>
<li>Symmetrisk kryptering: <strong>AES-256-GCM</strong> med AAD fra ratchet-header</li>
<li><code>ShadeSessionManager</code> — high-level API: <code>initialize</code>, <code>createPreKeyBundle</code>, <code>initSessionFromBundle</code>, <code>encrypt</code>, <code>decrypt</code></li>
<li>Symmetric encryption: <strong>AES-256-GCM</strong> with AAD from the ratchet header</li>
</ul>
</div>
<div id="panel-crypto" class="tab-panel" role="tabpanel" aria-labelledby="tab-crypto" hidden>
<p style="margin-top:0"><strong>Implementasjon av crypto for web/Bun/Node</strong> via SubtleCrypto — X25519, Ed25519, HKDF, HMAC, tilfeldige bytes.</p>
<p style="margin-top:0"><strong>Crypto implementation for web/Bun/Node</strong> via SubtleCrypto — X25519, Ed25519, HKDF, HMAC, random bytes.</p>
<ul>
<li>Gjør det mulig å bruke <code>shade-core</code> i nettleser og i servere som støtter Web Crypto</li>
<li>Kommentarer i koden peker på fremtidig Android (f.eks. Tink) som egen provider</li>
<li>Lets you use <code>shade-core</code> in the browser and on servers that support Web Crypto</li>
<li>Comments in source point toward future Android (e.g. Tink) as a separate provider</li>
</ul>
</div>
<div id="panel-proto" class="tab-panel" role="tabpanel" aria-labelledby="tab-proto" hidden>
<p style="margin-top:0"><strong>Binært wire-format</strong> for meldinger: versjon + type + lengdeprefiksede felt (big-endian).</p>
<p style="margin-top:0"><strong>Binary wire format</strong> for messages: version + type + length-prefixed fields (big-endian).</p>
<ul>
<li>Type <code>0x01</code> = PreKeyMessage, <code>0x02</code> = RatchetMessage</li>
<li>Brukes når du vil serialisere <code>ShadeEnvelope</code> effektivt over nettet</li>
<li>Use when you want to serialize <code>ShadeEnvelope</code> efficiently on the wire</li>
</ul>
</div>
<div id="panel-transport" class="tab-panel" role="tabpanel" aria-labelledby="tab-transport" hidden>
<p style="margin-top:0"><strong>Transportadaptere</strong> — ikke selve krypteringen, men hvordan du sender bytes (f.eks. fetch eller WebSocket).</p>
<p style="margin-top:0"><strong>Transport adapters</strong> — not encryption itself, but how you move bytes (e.g. fetch or WebSocket).</p>
<ul>
<li>Kobler applikasjonen din til den kanalen du allerede bruker</li>
<li>Hooks your application to the channel you already use</li>
</ul>
</div>
<div id="panel-server" class="tab-panel" role="tabpanel" aria-labelledby="tab-server" hidden>
<p style="margin-top:0"><strong>Prekey-server (Hono)</strong> — lagrer offentlige nøkler slik at Alice kan starte samtale mens Bob er «offline».</p>
<p style="margin-top:0"><strong>Prekey server (Hono)</strong> — stores public keys so Alice can start a conversation while Bob is “offline”.</p>
<ul>
<li><code>POST /v1/keys/register</code> — registrer identitet + bundle</li>
<li><code>GET /v1/keys/bundle/:address</code> — hent bundle (forbruker én engangsnøkkel om tilgjengelig)</li>
<li><code>POST /v1/keys/replenish</code> — etterfyll engangsnøkler</li>
<li><code>POST /v1/keys/register</code> — register identity + bundle</li>
<li><code>GET /v1/keys/bundle/:address</code> — fetch bundle (consumes one-time prekey when available)</li>
<li><code>POST /v1/keys/replenish</code> — replenish one-time prekeys</li>
</ul>
</div>
</section>
<section id="nokler">
<h2>Nøkler i korthet</h2>
<section id="keys-in-brief">
<h2>Keys at a glance</h2>
<div class="accordion" id="key-acc">
<div class="acc-item">
<button type="button" class="acc-trigger" aria-expanded="true" aria-controls="acc-identity" id="btn-identity">
Identitetsnøkkel (langvarig)
Identity key (long-term)
</button>
<div class="acc-panel" id="acc-identity" role="region" aria-labelledby="btn-identity">
<strong>Ed25519</strong> brukes til å signere den «signerte prekeyen». <strong>X25519</strong> brukes i Diffie-Hellman i X3DH og i ratchet. Én identitet per enhet/bruker i typisk oppsett.
<strong>Ed25519</strong> signs the “signed prekey”. <strong>X25519</strong> is used in Diffie–Hellman in X3DH and in the ratchet. One identity per device/user is typical.
</div>
</div>
<div class="acc-item">
<button type="button" class="acc-trigger" aria-expanded="false" aria-controls="acc-spk" id="btn-spk">
Signert prekey (mediumvarig, roteres)
Signed prekey (medium-lived, rotated)
</button>
<div class="acc-panel" id="acc-spk" role="region" aria-labelledby="btn-spk" hidden>
X25519-nøkkel som publiseres og signeres med identiteten. Mottaker verifiserer signaturen før DH. I koden anbefales rotasjon omtrent hver 1–7 dag.
An X25519 key that is published and signed by the identity. The recipient verifies the signature before DH. The codebase recommends rotation on the order of 1–7 days.
</div>
</div>
<div class="acc-item">
<button type="button" class="acc-trigger" aria-expanded="false" aria-controls="acc-otpk" id="btn-otpk">
Engangsnøkler (one-time prekeys)
One-time prekeys
</button>
<div class="acc-panel" id="acc-otpk" role="region" aria-labelledby="btn-otpk" hidden>
Valgfrie, men viktige for ekstra sikkerhet: hver hentet bundle kan inkludere én engangsnøkkel som slettes etter bruk (<code>processPreKeyMessage</code> fjerner den fra lager). Gir bedre beskyttelse mot visse angrep når mange klienter kobler til samme mottaker.
Optional but important for stronger security: each fetched bundle can include one one-time key that is removed after use (<code>processPreKeyMessage</code> clears it from storage). Improves protection against certain attacks when many clients connect to the same recipient.
</div>
</div>
</div>
</section>
<section id="flyt">
<h2>Interaktiv flyt: fra null til kryptert melding</h2>
<section id="flow-demo">
<h2>Interactive flow: zero to encrypted message</h2>
<p>
Klikk «Neste» for å gå gjennom rekkefølgen slik Shade er bygget. Dette speiler <code>ShadeSessionManager</code> og demoen i repoet.
Click “Next step” to walk through the sequence as Shade builds it. This mirrors <code>ShadeSessionManager</code> and the demo in the repo.
</p>
<div class="flow">
<h3>Sesjon og meldinger</h3>
<h3>Sessions and messages</h3>
<div class="flow-steps" id="flow-steps"></div>
<div class="flow-actions">
<button type="button" class="btn" id="flow-next">Neste steg</button>
<button type="button" class="btn btn-secondary" id="flow-reset">Start på nytt</button>
<button type="button" class="btn" id="flow-next">Next step</button>
<button type="button" class="btn btn-secondary" id="flow-reset">Start over</button>
</div>
</div>
</section>
<section id="x3dh-ratchet">
<h2>X3DH og Double Ratchet (kort forklart)</h2>
<h2>X3DH and Double Ratchet (brief)</h2>
<p>
<strong>X3DH</strong> løser problemet «jeg vil snakke med Bob nå, men Bob svarer ikke før senere». Bob legger ut en <em>prekey bundle</em> på serveren. Alice henter den, gjør 3 eller 4 DH-operasjoner (avhengig av om engangsnøkkel brukes), og deriverer en felles rot-nøkkel som begge kan rekonstruere uten at serveren kjenner hemmeligheten.
<strong>X3DH</strong> solves “I want to talk to Bob now, but Bob may not reply until later”. Bob publishes a <em>prekey bundle</em> on the server. Alice fetches it, runs 3 or 4 DH operations (depending on whether a one-time key is used), and derives a shared root key both parties can reconstruct — without the server learning the secret.
</p>
<p>
<strong>Double Ratchet</strong> bruker den roten som startpunkt. For hver melding (eller ved nye DH-nøkler) avledes nye nøkler; meldinger på ledningen er AES-GCM med autentisering (AAD binder kryptoteksten til ratchet-header). Protokollen håndterer også meldinger i feil rekkefølge innenfor grenser (<code>MAX_SKIP</code>).
<strong>Double Ratchet</strong> uses that root as a starting point. For each message (or when new DH keys spin), keys are derived; on the wire payloads are AES-GCM with authentication (AAD binds ciphertext to the ratchet header). The protocol also tolerates messages arriving out of order within limits (<code>MAX_SKIP</code>).
</p>
<p>
Spesifikasjoner fra Signal (engelsk): <a href="https://signal.org/docs/specifications/x3dh/" target="_blank" rel="noopener">X3DH</a> · <a href="https://signal.org/docs/specifications/doubleratchet/" target="_blank" rel="noopener">Double Ratchet</a>.
Signal specifications (English): <a href="https://signal.org/docs/specifications/x3dh/" target="_blank" rel="noopener">X3DH</a> · <a href="https://signal.org/docs/specifications/doubleratchet/" target="_blank" rel="noopener">Double Ratchet</a>.
</p>
</section>
<section id="gjenbruk">
<h2>Bruke Shade i flere prosjekter</h2>
<section id="reuse">
<h2>Using Shade across projects</h2>
<p>
Tenk på Shade som tre lag du kan kombinere etter behov:
Treat Shade as three layers you combine as needed:
</p>
<ol>
<li><strong>Core + crypto-provider + storage</strong> — selve E2EE-motoren (kan kjøre i klient eller serverprosess som skal dekryptere).</li>
<li><strong>Proto</strong> — når du vil ha kompakt binær serialisering.</li>
<li><strong>Transport + prekey-server</strong> — når du vil standardisere nøkkelutveksling og kanaler.</li>
<li><strong>Core + crypto provider + storage</strong> — the E2EE engine itself (runs in a client or a server process that must decrypt).</li>
<li><strong>Proto</strong> — when you want compact binary serialization.</li>
<li><strong>Transport + prekey server</strong> — when you want standardized key discovery and channels.</li>
</ol>
<p>
Referansekjøring: <code class="mono">bun demo.ts</code> i rotmappen viser frontend/backend-flyt med minnelager og ekte kryptoprimitiver.
Reference path: <code class="mono">bun demo.ts</code> from the repo root shows a frontend/backend flow with memory storage and real crypto primitives.
</p>
</section>
<footer>
<p>Shade — oversikt generert som statisk HTML i <code class="mono">docs/shade-overview.html</code>. Åpne filen direkte i nettleseren eller server den statisk.</p>
<p>Shade — overview written as static HTML in <code class="mono">docs/shade-overview.html</code>. Open the file directly in the browser or serve it as static assets.</p>
</footer>
</div>
@@ -561,28 +561,28 @@
// Flow steps
var steps = [
{
title: "Initialiser klient",
body: "Kall initialize(): last eller generer identitetsnøkkel (Ed25519 + X25519), registrationId og signert prekey.",
title: "Initialize client",
body: "Call initialize(): load or generate the identity keys (Ed25519 + X25519), registrationId, and signed prekey.",
},
{
title: "Publiser prekey bundle",
body: "Bygg bundle med createPreKeyBundle() / generateOneTimePreKeys() og registrer på prekey-server (eller del ut av band for demo).",
title: "Publish prekey bundle",
body: "Build a bundle with createPreKeyBundle() / generateOneTimePreKeys() and register it on the prekey server (or share out-of-band for a demo).",
},
{
title: "Start sesjon mot peer",
body: "Hent motpartens bundle, kjør initSessionFromBundle(address, bundle). X3DH kjører og ratchet-sesjon lagres i StorageProvider.",
title: "Start session with peer",
body: "Fetch the peer bundle, run initSessionFromBundle(address, bundle). X3DH runs and the ratchet session is stored in StorageProvider.",
},
{
title: "Første encrypt",
body: "encrypt() returnerer ShadeEnvelope type 'prekey': inneholder ephemeral nøkkel, prekey-ID-er og første RatchetMessage (AES-GCM).",
title: "First encrypt",
body: "encrypt() returns a ShadeEnvelope of type 'prekey': includes ephemeral keys, prekey IDs, and the first RatchetMessage (AES-GCM).",
},
{
title: "Motpart decrypt",
body: "decrypt() på PreKeyMessage: gjenskaper samme root key, initReceiverSession, ratchetDecrypt — plaintext ut.",
title: "Peer decrypt",
body: "decrypt() on PreKeyMessage: restores the same root key, initReceiverSession, ratchetDecrypt — plaintext out.",
},
{
title: "Videre meldinger",
body: "Neste kall til encrypt() gir type 'ratchet'. DH-ratchet steg gir nye kjeder og forbedret sikkerhet over tid.",
title: "Further messages",
body: "The next encrypt() calls yield type 'ratchet'. DH ratchet steps rotate chains and improve security over time.",
},
];

117
docs/streams.md Normal file
View File

@@ -0,0 +1,117 @@
# Shade Streams 0.2.0
E2EE chunked upload/download for Shade. Drop into any Shade-using app:
```ts
const handle = await shade.upload({ to: 'bob', input: file });
const result = await handle.done(); // { sha256, bytesSent, durationMs }
```
…and on the receiver:
```ts
shade.onIncomingTransfer(async (incoming) => {
const handle = await incoming.accept({ output: { kind: 'file', path: '/uploads/x' } });
await handle.done();
});
```
Or in React:
```tsx
<ShadeRuntimeProvider runtime={shade}>
<ShadeUploader to="bob" onComplete={(r) => console.log(r.sha256)} />
</ShadeRuntimeProvider>
```
## How it works
A transfer has two planes:
- **Control plane** — `stream-init`, `stream-finish`, `stream-abort`, and
`stream-resume-*` messages, encoded as JSON plaintext and shipped through
the existing Double Ratchet (envelope type `0x02`). One ratchet step
establishes a stream; the rest is per-chunk AEAD.
- **Data plane** — `stream-chunk` envelopes (envelope type `0x11`),
AES-256-GCM-encrypted under a per-lane key, shipped over HTTP POST (or
WebSocket if opted-in). Lanes run in parallel for throughput.
```
Sender Receiver
────── ────────
streamSecret = randomBytes(32)
streamId = randomBytes(16)
streamKey = HKDF(streamSecret, streamId, "shade-stream/v1\0master")
laneKey[i] = HKDF(streamKey, streamId, "...\0lane\0" || u32(i))
[stream-init JSON over Double Ratchet] ─▶
parses streamSecret, derives same keys
spawns L per-lane receivers
[chunk 0x11 over HTTP] ─▶
AES-GCM(laneKey[i], plaintext, nonce=laneId||seq, aad=streamId||laneId||seq||isLast)
decrypts, verifies, writes to sink
(× 4 lanes in parallel)
[stream-finish JSON over Double Ratchet] ─▶
verifies per-lane sha256 + overall sha256
throws TransferIntegrityError on mismatch
```
## Partition strategies
- **Range** (default for known-size inputs) — lane `i` owns bytes
`[i·N/L, (i+1)·N/L)`. Receiver reconstructs by concatenating lane outputs
in laneId order.
- **Round-robin** (default for unknown-size streams) — chunk `i` goes to
lane `i mod L`. Receiver reorders via a per-stream chunk-index buffer.
## Resume
Persistence is opt-in via a `ResumeStore` (memory, SQLite, Postgres,
IndexedDB-ready). State persisted on init; sender's resume queries the
receiver's `lastSeqAcked` per lane via `GET /v1/transfer/:streamId/state`,
then continues from there. The streamSecret is encrypted at rest under a
device-key derived from the local identity's signing private key — a
stolen DB without the identity key cannot resume.
```ts
const handle = await shade.resumeUpload(streamId, sameInputAsBefore);
await handle.done();
```
Resume across **identity rotation** is not supported (rotation invalidates
the device key — by design, to prevent a stolen pre-rotation DB from
deriving keys for any post-rotation transfer). Restart the transfer
manually after rotation.
## Throughput
- Default 4 lanes × 1 MiB chunks × 4 in-flight chunks per lane =
16 MiB peak in-flight per direction.
- Memory-bounded: receivers stream chunks to the configured sink without
buffering the full payload. 1 GB transfer = O(chunkSize) RSS, not O(file).
- AES-GCM is hardware-accelerated via `SubtleCrypto`; SHA-256 streaming via
`@noble/hashes`.
## Security properties
| ID | Property |
|---|---|
| S1 | streamSecret never on the wire in plaintext (Double Ratchet only) |
| S2 | Unique per-(streamKey, laneId, seq) AEAD nonce — no nonce reuse |
| S3 | Tampered chunk header / ciphertext / tag → AEAD reject |
| S4 | Per-lane sha256 + overall sha256 verified at finish |
| S5 | streamKey/laneKey zeroized on abort/finish (`destroy()`) |
| S6 | Concurrent streams have independent lane keys |
| S7 | seq overflow practical-impossible (u64 max) |
| S8 | At-rest streamSecret encrypted under device-key |
## API surface
See package READMEs:
- `packages/shade-streams/README.md` — crypto + state machines
- `packages/shade-transfer/README.md` — orchestration, transports, persistence
- `packages/shade-sdk/README.md` — magic drop-in
- `packages/shade-widgets/README.md` — React UI