Stian/Shade
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

release(v4.0.0): Shade GA — V3.x consolidation + audit prep

Browse Source 
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.

Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
  Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
  bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
  shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
  non-realtime stack.

Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
  version 4.0.0 on /health.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Sterister
2026-05-03 18:35:35 +02:00
parent 8b055912b7
commit 95b0a1c530
298 changed files with 37909 additions and 256 deletions
Download Patch File Download Diff File Expand all files Collapse all files

90
.gitea/workflows/cross-vectors.yml Normal file
View File

@@ -0,0 +1,90 @@
name: Cross-platform vectors
# V3.5 §CI parity gate. Both runners load test-vectors/*.json and verify their
# native implementation produces byte-identical output to the recorded vectors.
# Any divergence — KDF labels, AAD encoding, wire format — fails CI immediately
# so cross-platform messaging breakage cannot land on main.
on:
push:
branches: [main]
paths:
- 'test-vectors/**'
- 'packages/shade-core/tests/cross-platform-vectors.test.ts'
- 'packages/shade-core/src/**'
- 'packages/shade-crypto-web/src/**'
- 'packages/shade-proto/src/**'
- 'android/**'
- 'scripts/generate-vectors.ts'
- '.gitea/workflows/cross-vectors.yml'
pull_request:
branches: [main]
paths:
- 'test-vectors/**'
- 'packages/shade-core/tests/cross-platform-vectors.test.ts'
- 'packages/shade-core/src/**'
- 'packages/shade-crypto-web/src/**'
- 'packages/shade-proto/src/**'
- 'android/**'
- 'scripts/generate-vectors.ts'
- '.gitea/workflows/cross-vectors.yml'
jobs:
ts-vectors:
name: TypeScript vectors (bun)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Bun
run: curl -fsSL https://bun.sh/install | bash
- name: Install dependencies
run: ~/.bun/bin/bun install --frozen-lockfile
- name: Run TS vector tests
run: ~/.bun/bin/bun run test:vectors
- name: Verify vectors are up-to-date
# Regenerate vectors and fail if they would change. Forces vector
# commits to come from `bun run vectors:gen`, never hand-edited.
run: |
~/.bun/bin/bun run vectors:gen
if ! git diff --quiet test-vectors/; then
echo "::error::test-vectors/ is out of date. Run 'bun run vectors:gen' and commit."
git diff test-vectors/
exit 1
fi
kotlin-vectors:
name: Kotlin vectors (gradle)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up JDK 17
uses: actions/setup-java@v4
with:
distribution: temurin
java-version: '17'
- name: Cache Gradle
uses: actions/cache@v4
with:
path: |
~/.gradle/caches
~/.gradle/wrapper
key: gradle-${{ hashFiles('android/**/*.gradle.kts', 'android/gradle/wrapper/gradle-wrapper.properties') }}
restore-keys: gradle-
- name: Run Kotlin vector tests
working-directory: android
run: ./gradlew :shade-android:test --no-daemon --info
- name: Upload Gradle test report
if: always()
uses: actions/upload-artifact@v4
with:
name: kotlin-test-report
path: android/shade-android/build/reports/tests/test/
if-no-files-found: ignore

640
CHANGELOG.md
View File

@@ -5,6 +5,646 @@ All notable changes to Shade are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [4.0.0] — 2026-05-03 — General Availability
Shade 4.0 is the first GA-marked release: every plan from V3.1 through
V3.12 is merged, the cross-platform vector suite is green on TS + Kotlin,
the threat model has been updated to reflect every new surface, and the
core stack (X3DH, Double Ratchet, storage encryption, recovery, WebRTC
P2P, Key Transparency) has been packaged for external review. Voice and
video — the only big-ticket V2.x ask — have been moved to V5.0 so the
4.0 audit can focus on a frozen non-realtime core.
The wire format is **unchanged from 0.4.x**: 4.0 peers interoperate with
0.4.x peers byte-for-byte. The version bump is semantic (audit-cycle
complete, opt-in surface fully exposed), not breaking. Apps that have
been running 0.4.x in production move forward by `bun add @shade/sdk@^4.0.0`
and (optionally) wiring any of the new opt-in surfaces.
### Highlights
- **External crypto-review-ready.** A "review-bundle" (`docs/audit/`)
ships with this release: links to every protocol spec, the threat
model, the cross-platform test corpus, the build instructions, and
scope guidance for the auditor.
- **Migration guide locked in.** `MIGRATION.md` documents the exact
0.3.x → 4.0 path, including the optional opt-ins, the schema
superset, and the `shade migrate-storage` workflow.
- **Cross-platform parity gated in CI.** `.gitea/workflows/cross-vectors.yml`
runs the same vector corpus on TS (bun) and Kotlin (gradle). A
divergent KDF label, AAD layout, or wire byte fails the build.
- **All V*.md plans archived.** `docs/V3.1.md` through `docs/V3.12.md`
and the original V2.1/V2.2/V2.3 backlog now live under
`docs/archive/` with `Status: Done`. Active planning continues in
`docs/V5.0.md` (Voice & Video).
- **Operator-facing OpenAPI is complete.** `packages/shade-server/openapi.yaml`
now covers prekey, transfer, KT, inbox, bridge (SSE / long-poll / WS),
observer, and the `/metrics`, `/healthz`, `/ready` operations
endpoints — every HTTP surface a 4.0 client can talk to.
- **Threat-model refresh.** Sections 10 (V3.3 fingerprint gates), 11
(V3.11 WebRTC), 12 (V3.8 Web-Worker boundary) are new; the residual-
risk table updates the §1 / §2 / §6 entries with the
4.0 mitigations now landed.
### What's already in 4.0 (consolidated from 0.4.x)
The detailed CHANGELOG entries below list everything that landed in
the 0.4.x series and is now part of the GA baseline:
- V3.2 — At-Rest Storage Encryption (`@shade/storage-encrypted`,
`@shade/keychain`, `shade migrate-storage`).
- V3.3 — Fingerprint Gates & Trust UX (`Shade.beforeFirstLargeFile` /
`beforeBackupImport` / `beforeNewDeviceTrust`,
`<FingerprintCompare />`, `<FingerprintGate />`).
- V3.4 — Observability v2 (OpenTelemetry-shaped events,
`@shade/observability`).
- V3.5 — Android parity + cross-platform CI gate.
- V3.6 — Async Store-and-Forward (`@shade/inbox`,
`@shade/inbox-server`, `InboxPruneTask`).
- V3.7 — Transport Bridge (`@shade/transport-bridge`, SSE +
long-poll + WS adapters).
- V3.8 — Web Workers Crypto (`@shade/crypto-web/worker`).
- V3.9 — Rich File Metadata + thumbnails (in `@shade/files`).
- V3.10 — Social Key Recovery (`@shade/recovery`,
`<RecoverySetup />`, `<RecoveryRequest />`,
`<RecoveryApprove />`).
- V3.11 — WebRTC P2P Transport (`@shade/transport-webrtc`,
`MultiTransportFallback`).
- V3.12 — Key Transparency (`@shade/key-transparency`,
`createPrekeyServerWithKT(...)`, `LightWitness`).
### Acceptance criteria
- [x] V3.1 → V3.12 merged into `main`.
- [x] No open critical / high-severity security issues at the time of
tagging.
- [x] Cross-platform test vectors green: TS (1000 / 1000) and
Kotlin (11 / 11).
- [x] Production-checklist (`docs/PRODUCTION-CHECKLIST.md`) is the
canonical operator gate.
- [x] OpenAPI covers every HTTP surface (`/v1/keys/*`,
`/v1/transfer/*`, `/v1/kt/*`, `/v1/inbox/*`, `/v1/bridge/*`,
`/metrics`, `/healthz`, `/ready`).
- [x] Threat model reflects every new V3.x surface.
- [x] `0.3.x → 4.0` migration documented in `MIGRATION.md` and
validated against the `shade migrate-storage` CLI on a real
SQLite DB.
- [ ] **Pending external review.** A `docs/audit/REVIEW-BUNDLE.md`
pointer is shipped; the actual external review window opens
after tag.
### Migration
See [MIGRATION.md § Migrating from 0.3.x to 4.0 (GA)](./MIGRATION.md#migrating-from-03x-to-40-ga).
The short version: bump every `@shade/*` to `^4.0.0`, run
`bun install`, restart, opt in to the V3.x surfaces you actually need.
No on-disk schema is destructive; no peer wire format changes.
## [Unreleased] — Key Transparency (V3.12) + WebRTC (V3.11)
### V3.12 — Key Transparency
Verifiable prekey distribution. The prekey server can now run in
**Key-Transparency mode**: every register / delete event is committed
to an append-only Merkle log (RFC 6962-style), every bundle-fetch
includes an inclusion proof, and every Signed Tree Head (STH) is
signed with an operator-controlled Ed25519 key that clients pin
out-of-band.
A malicious server that swaps a bundle, splits its view between two
clients, or rewrites history is detected by the client's KT verifier
or by an independent witness. KT is **opt-in** on both server and
client — existing deployments work unchanged until upgraded.
See `docs/V3.12-DESIGN.md` for the design notat (threat model,
data-structure choices, freshness model, recovery procedures) and
`docs/key-transparency.md` for operator + client onboarding.
### Added
#### `@shade/key-transparency` (new package)
- `MerkleLog` — RFC 6962 append-only hash tree over pre-hashed leaves.
In-memory mirror with O(N) leaf storage and O(log N) audit-path /
consistency-proof generation.
- `auditPath`, `recomputeRootFromAuditPath`, `consistencyProof`,
`verifyConsistencyProof` — standalone primitives matching RFC 6962
§2.1.1 and §2.1.2.
- `AddressIndex` + `verifyInclusionProof` / `verifyAbsenceProof` —
lexicographically sorted address commitment with both inclusion and
neighbor-pair absence proofs. The index commitment becomes part of
every STH so `address → bundle_hash` is auditable, not just the
raw event log.
- `SignedTreeHead` + `signSth` / `verifySthSignature` /
`canonicalSthBytes` / `computeLogId` — Ed25519-signed commitment to
the tree state. `log_id = SHA-256(public_key)` so a forged STH that
claims a different log key is rejected.
- `KTLogManager` — server-side orchestration that wires `MerkleLog`,
`AddressIndex`, persistent `KTLogStore`, and STH signing under one
serial-mutation API (`recordRegister`, `recordReplenish`,
`recordDelete`, `publishSTH`, `buildBundleInclusionProof`,
`buildBundleAbsenceProof`, `buildConsistencyProof`).
- `KTLogStore` interface + `MemoryKTLogStore` reference impl. The
interface is append-only by contract (no `update()` or `delete()`
on historical leaves).
- `LightWitness` — passive observer that polls a server's `/v1/kt/sth`
endpoint, verifies signature + freshness + consistency, stores
observed STHs, and exposes `compare(otherSth)` for split-view
detection. Used by both witness CLIs and (transparently) by the SDK.
- Bundle-proof verifiers: `verifyBundleInclusion`,
`verifyBundleAbsence`, `verifyBundleTombstone`. Each re-derives the
bundle hash, checks the audit path against the STH root, verifies
the index commitment, and confirms freshness.
- Errors: `KTError`, `KTVerificationError`, `KTSplitViewError`,
`KTStaleSTHError`, `KTLogIdMismatchError`. Mapped to
`SHADE_KT_*` codes.
- Wire-format helpers: `ktProofToWire` / `ktProofFromWire` /
`sthToWire` / `sthFromWire` for JSON-safe transport.
#### `@shade/server`
- `createPrekeyServerWithKT(...)` — convenience that builds the KT
service and wires it into the prekey routes in one call.
- `KeyTransparencyService` — single-writer wrapper around
`KTLogManager` with mutex-serialized mutations, cached latest STH,
and configurable heartbeat interval (default 10 min).
- New routes mounted under `/v1/kt/`:
- `GET /v1/kt/log_id` — operator's signing public key + log_id.
- `GET /v1/kt/sth` — latest signed tree head.
- `GET /v1/kt/sth/:treeSize` — historical STH lookup.
- `GET /v1/kt/consistency?from=N1&to=N2` — RFC 6962 consistency proof.
- `POST /v1/keys/register` and `DELETE /v1/keys/:address` now commit
to the KT log (when enabled). `GET /v1/keys/bundle/:address`
returns a `ktProof` field on success and on 404 (absence/tombstone).
- KT is fully opt-in. Existing deployments are byte-compatible until
`keyTransparency` is configured.
#### `@shade/storage-postgres`
- `PostgresKTLogStore` — durable KTLogStore on Postgres. Uses three
tables (`shade_kt_leaves`, `shade_kt_index`, `shade_kt_sths`) with
an `BEFORE UPDATE/DELETE/TRUNCATE` trigger on `shade_kt_leaves`
that blocks any mutation — defense-in-depth against operator error.
- `ensureKTLogTables(sql)` exported for embedding.
#### `@shade/transport`
- `ShadeFetchTransport` accepts `keyTransparency: KTVerifierOptions`.
Modes: `'observe'` verifies when proof present, `'observe-strict'`
requires proof on every response.
- `fetchBundleVerified(address)` returns `{ bundle, ktSth? }` so
callers can route the verified STH into a `LightWitness`.
- 404 responses are also verified (absence or tombstone proof) under
strict mode.
#### `@shade/sdk`
- `ShadeConfig.keyTransparency` — opt-in client config:
```ts
createShade({
prekeyServer: 'https://shade.example.com',
keyTransparency: { mode: 'observe-strict', logPublicKey: KEY_BYTES_32 },
});
```
- `Shade.getKTWitness()` returns the auto-wired `LightWitness` so app
code can introspect observed STHs or run manual gossip checks.
- The SDK transparently feeds every fetched STH into the witness so
split-view detection runs by default whenever KT is on.
### Tests
- 76 new tests across the KT stack: hash primitives, Merkle audit
paths, consistency proofs, address-index inclusion/absence proofs,
STH signing, manager orchestration, witness ingest, server-side
HTTP routes, transport-side verification, and an end-to-end
acceptance test that simulates two divergent server views and
asserts a `KTSplitViewError` is raised.
### V3.11 — WebRTC P2P Transport
Direct peer-to-peer chunk delivery for `@shade/transfer` (and therefore
`@shade/files`) via `RTCDataChannel`. Signaling — SDP offer / answer +
trickle ICE — rides on top of `Shade.send` / `Shade.onMessage` so the
same Double Ratchet that authenticates regular messages authenticates
WebRTC negotiation. Throughput-heavy uploads (multi-MB / multi-GB) skip
the HTTP relay entirely when NAT allows; when traversal fails, the new
`MultiTransportFallback([webrtc, http])` demotes back to HTTP within
the configured connect-timeout window without losing any chunks already
in flight. See `docs/webrtc.md` and `docs/V3.11.md`.
### Added
#### `@shade/transport-webrtc` (new package)
- `WebRtcConnection` — per-peer wrapper around an `IPeerConnection`
plus the single bidirectional `RTCDataChannel` (label
`shade-transfer/v1`). Drives offer/answer/ICE through a
`WebRtcSignalingChannel`; handles the receiver-side dispatch loop
for chunk-ack / resume-state / ping-pong / error frames; exposes
per-request reqId-correlated `request()` for the transport layer.
- `WebRtcConnectionManager` — per-peer pool with deterministic glare
resolution (lexicographic address compare). `getOrCreate(peer)`
returns the live connection or initiates a fresh one; following
through a glare-yield is automatic so the user-facing promise
resolves to whichever role survives.
- `WebRtcSignalingChannel` — multiplexes the four signaling kinds
(`shade.webrtc-offer/v1`, `shade.webrtc-answer/v1`,
`shade.webrtc-ice/v1`, `shade.webrtc-bye/v1`) over any `ShadeBridge`
(real `Shade.send`/`onMessage`, or `MemoryShadeBridge` for tests).
Non-signaling plaintext is forwarded to a configurable `passthrough`
hook so consumer `onMessage` handlers stay untouched.
- `WebRtcTransferTransport` — implements
`@shade/transfer`'s `ITransferTransport` over the managed
DataChannel. Encodes chunks into the package's binary wire format,
awaits chunk-ack frames matched by 16-byte requestId tokens, and
enforces SCTP-friendly backpressure by polling `bufferedAmount`
(default threshold 4 MiB).
- `IRtcFactory` interface + `nativeRtcFactory()` adapter wrapping
`globalThis.RTCPeerConnection` for browsers / Deno / Cloudflare
Workers. `MemoryRtcFactory` ships an in-process WebRTC simulator
used by the package's own tests and by `@shade/sdk` integration
tests.
- `createShadeBridgeFromShade(shade)` — turns any `Shade`-shaped
object into a `ShadeBridge`. Calls `shade.send(plaintext)` to
ratchet-encrypt the JSON, then `shade.deliverControlEnvelope(...)`
(when present) to ship the envelope over HTTP — same path the
existing control-plane already uses.
- Wire-format constants (`WIRE_CHUNK`, `WIRE_CHUNK_ACK`, etc.) +
`encode*Frame` / `decodeFrame` helpers exported for adapters that
want to interoperate with `ShadeTransferWsTransport` (the wire
matches frame-for-frame).
- Errors: `WebRtcConnectError`, `WebRtcDataChannelError`,
`WebRtcSignalingError`, `WebRtcTimeoutError` — all extend
`TransferTransportError` so `MultiTransportFallback` automatically
demotes on failure.
#### `@shade/transfer`
- `MultiTransportFallback` — N-ary generalisation of the existing
two-arg `FallbackTransferTransport`. Constructor takes
`[{ name: 'webrtc', transport }, { name: 'ws', transport }, ...]`;
layers are tried in order and demote sticky on
`TransferTransportError`. Exposes `activeName`, `hasFallenBack`,
`failures` (diagnostic log), and `onSwitch((from, to) => ...)` for
observability hooks.
#### `@shade/sdk`
- `Shade.configureWebRTC({ factory, iceServers?, iceTransportPolicy?,
bundlePolicy?, connectTimeoutMs?, requestTimeoutMs?,
backpressureThresholdBytes? })` — opt-in entrypoint. MUST be called
before the engine is built (i.e. before the first `upload()`,
`onIncomingTransfer()`, or `transferRoute()` call). When
configured, the engine is wired with
`MultiTransportFallback([webrtc, http])` and the WebRTC manager
receives receiver-hooks pointing at `engine.receiveChunk` /
`engine.getResumeState`.
- `Shade.getWebRtcRuntime(): ShadeWebRtcRuntime | null` — diagnostic
accessor returning the live signaling channel, manager, transport,
and `MultiTransportFallback` after `engine()` builds.
- `@shade/transport-webrtc` is a (optional) peer-dep — projects that
don't call `configureWebRTC()` don't pay the install or runtime
cost.
### Tests
- `packages/shade-transport-webrtc/tests/` — wire-format roundtrips,
signaling routing, full memory-factory caller/callee handshake,
receiver-hook dispatch (chunk + resume-query), glare convergence,
TURN-only configuration plumbing, native-adapter availability
smoke test.
- `packages/shade-transfer/tests/multi-fallback.test.ts` — N-ary
demotion, sticky-after-failure, non-transport-error preservation,
empty-list rejection.
- `packages/shade-sdk/tests/webrtc-integration.test.ts` — two real
Shade instances upload via WebRTC primary; verifies the engine
picks `webrtc` and never demotes during the run.
- `packages/shade-sdk/tests/webrtc-failover.test.ts` — broken-RTC
factory provokes connect timeout; SDK demotes to HTTP within the
V3.11 5-second SLO without losing chunks.
- `packages/shade-sdk/tests/webrtc-throughput.test.ts` — 4 MiB / 4
lanes loopback over WebRTC vs HTTP; integrity match across both
transports + diagnostic speedup ratio.
### Documentation
- `docs/webrtc.md` — full V3.11 guide (NAT-traversal table, TURN
config matrix, connection flow, glare resolution, backpressure,
multi-fallback wiring, diagnostics, wire format, limits, migration).
- `packages/shade-transport-webrtc/README.md` — package quickstart.
- README + CHANGELOG + ROADMAP marked V3.11 as Done.
## [Earlier Unreleased] — Social Key Recovery (V3.10)
The biggest UX hole in any E2EE system — "what happens if I lose my
phone?" — closed without a centralized recovery agent. Pick `n`
guardians from your peers, set a threshold `k`; any `k` of them
together can rebuild your identity onto a new device, but `k-1` or
fewer cannot. Shamir Secret Sharing over GF(2^8) gates the recovery
key; AES-GCM authentication on the backup blob detects forged
shares; an OOB-confirmed fingerprint gate on the guardian side
blocks social-engineering. See `docs/recovery.md` and
`docs/V3.10.md`.
### Added
#### `@shade/recovery` (new package)
- `setupRecovery({ shade, guardians, threshold, deliver })` —
primary-device flow. Generates a 32-byte `recoveryKey`,
encrypts an identity backup under the recoveryKey-derived
passphrase via `Shade.exportBackup`, Shamir-splits the key into
`n` shares, and ships one `share-deposit` envelope per guardian
over the existing 1:1 Shade session. Returns a per-guardian
delivery report so partial-distribution is recoverable.
- `attachGuardian({ shade, store, approve, deliver })` —
guardian-side receiver. Wires a `Shade.onMessage` handler that
persists incoming deposits in a caller-supplied `RecoveryStore`
and gates `recovery-request` envelopes behind a user-driven
`approve` callback. Auto-declines requests for unknown
`(originalAddress, setupId)` pairs.
- `requestRecovery({ shade, originalAddress, setupId, threshold,
guardians, deliver })` — new-device flow. Sends one
`recovery-request` per guardian, collects `share-grant` /
`share-decline` replies, Shamir-combines the threshold-many
grants, and atomically swaps in the restored identity via
`Shade.importBackup`. Forged shares are detected by the
AES-GCM tag on the backup blob; the loop tries every
threshold-sized subset of grants before giving up.
- Pure-TS Shamir Secret Sharing primitives (`splitSecret`,
`combineShares`, `encodeShare`, `decodeShare`) over GF(2^8)
with constant-time table lookups. Exported for advanced
callers and hardware-token integrations.
- `MemoryRecoveryStore` for tests + a `RecoveryStore` interface
apps implement against IndexedDB / SQLite / AsyncStorage / etc.
- Errors: `RecoveryError`, `RecoveryDeclinedError`,
`RecoveryTimeoutError`, `RecoveryReconstructionError`,
`RecoveryProtocolError`, `RecoveryGuardianRejectedError`.
- Wire protocol: `share-deposit`, `recovery-request`,
`share-grant`, `share-decline` JSON envelopes carried over
Double-Ratchet plaintext.
#### `@shade/widgets`
- `<RecoverySetup />` — primary-device guardian-picker + threshold
slider, drives `setupRecovery` and exposes `formatRecoveryCard`
for the user's offline copy.
- `<RecoveryRequest />` — new-device widget that displays the
temporary fingerprint prominently, drives `requestRecovery`,
and reports per-guardian progress live.
- `<RecoveryApprove />` — guardian-side widget. Renders the
pending request with original-vs-new fingerprint side-by-side
and enforces a two-checkbox gate ("matches" + "OOB-verified")
before the release button is clickable.
- `createApprovalQueue()` — turns the `attachGuardian.approve`
callback into a deferred queue the widget can consume.
#### `@shade/core`
- **Bug fix.** `initReceiverSession` now copies the
`localDHKeyPair` into the session so the eventual zeroize on
DH ratchet step touches a scratch buffer, not the persisted
signed prekey. Pre-V3.10 this corrupted the receiver's signed
prekey after the first incoming X3DH from any sender — a bug
surfaced by V3.10's multi-sender recovery flow but harmful to
any user receiving messages from more than one peer.
Regression test in `packages/shade-core/tests/ratchet.test.ts`.
### Acceptance criteria (V3.10)
- [x] 3-of-5 recovery works end-to-end on two separate Shade
instances. (`packages/shade-recovery/tests/integration.test.ts`)
- [x] No coalition of `(k-1)` guardians can reconstruct the
`recoveryKey` (verified with `fast-check` property tests).
(`packages/shade-recovery/tests/shamir.test.ts`,
`tests/adversarial.test.ts`)
- [x] Guardian-side widget requires fingerprint-confirmation
before sending a share. Two-checkbox enforcement +
symmetric tests of both honest-OOB-confirm and
hostile-fingerprint-mismatch paths.
## [Unreleased] — Web Workers Crypto (V3.8)
Big in-browser uploads stay smooth: AES-GCM, HKDF, HMAC, X25519, Ed25519
and full per-lane stream state now run in a dedicated Web Worker. The
main thread only buffers and forwards plaintext slices over zero-copy
`postMessage`; lane keys never cross the thread boundary. Opt-in via
`shade.configureWorkerCrypto({ workerUrl })`. See `docs/web-workers.md`
and `docs/archive/V3.8.md`.
### Added
#### `@shade/crypto-web`
- `WorkerCryptoProvider` — drop-in `CryptoProvider` proxy that forwards
every async op to a dedicated Web Worker via the `worker-protocol`.
Sync helpers (`randomBytes`, `randomUint32`, `constantTimeEqual`,
`zeroize`) execute on the calling thread — no useless round-trips.
- `createWorkerCryptoProvider({ workerUrl, idleTimeoutMs?, spawn? })`
factory. Spawns lazily, completes a protocol-version handshake, and
self-terminates after 30 s (configurable) of inactivity. Idempotent
re-spawn on next call.
- `WorkerStreamSender` / `WorkerStreamReceiver` — main-thread handles on
`StreamSender` / `StreamReceiver` instances that live entirely inside
the worker. Plaintext is shipped via transferable `ArrayBuffer`s; lane
keys + running sha256 stay worker-side.
- `createEncryptStream` / `createDecryptStream` — TransformStream
factories. `pipeThrough(encryptStream)` consumes plaintext and emits
one wire-encoded `stream-chunk` envelope per write. Both expose a
`laneSha256` promise that resolves once the stream finishes.
- New subpath export: `@shade/crypto-web/worker` is the dedicated
module-worker entrypoint. Bundle with the standard
`new URL('@shade/crypto-web/worker', import.meta.url)` idiom.
- `rotate()` and `destroy()` lifecycle controls — call after identity
rotation to bound the worst-case duration any lane key sits in worker
memory.
#### `@shade/sdk`
- `shade.configureWorkerCrypto({ workerUrl, idleTimeoutMs? })` —
opt-in setup. Without it, `encryptStream` / `decryptStream` throw a
clear error pointing to the docs.
- `shade.encryptStream({ streamId, streamSecret, laneId?, chunkSize? })`
→ `{ stream, laneSha256 }` — TransformStream with an end-of-stream
sha256 promise for end-to-end integrity proofs.
- `shade.decryptStream(...)` — inverse. Strict in-order seq, AAD-bound
AEAD, replay-rejecting.
- `shade.getWorkerCrypto()` — direct access to the worker-backed
`CryptoProvider` for one-off heavy ops.
- `shade.shutdown()` now also `destroy()`s the worker provider.
### Acceptance criteria (V3.8)
- [x] 100 MB upload in Chrome without blocking the main thread
> 16 ms in P99 (verification recipe in
`docs/web-workers.md#verifying-main-thread-budget`).
- [x] Safari works at default chunk-size — every `postMessage` carries
≤ 256 KiB + AEAD overhead, far below Safari's transferable cap.
- [x] Worker terminates within 30 s of last use (default
`idleTimeoutMs`), and re-spawns transparently on the next call.
---
## [Unreleased] — Transport Bridge (V3.7)
A canonical fallback chain for clients that cannot or will not run a
WebSocket: SSE primary, long-poll secondary, plus a thin WS adapter for
the happy path. All three transports surface the same `IncomingMessage`
shape so application code stays portable across browser-extension,
edge-runtime, and proxy-locked environments. See `docs/transport.md`
and `docs/archive/V3.7.md`.
### Added
#### `@shade/transport-bridge` (new)
- `IncomingMessage` — `{ from, bytes, receivedAt, msgId? }` — single
shape across every transport.
- `BridgeTransport` — `connect({ onMessage }) → disconnect()` contract.
- `WsBridge`, `SseBridge`, `LongPollBridge` — three concrete transports
consuming the matching `/v1/bridge/{ws,stream,poll}` endpoints.
- `FallbackBridgeTransport` — sticky-after-first-success priority chain.
Exposes `activeKind` and `attempts` for observability.
- `signBridgeQuery` — Ed25519-signed query-string builder (the only
carrier that survives `EventSource`'s no-headers restriction).
- Auto-reconnect with exponential backoff for WS + SSE; `Last-Event-ID`
cursor resume for SSE; bounded one-outstanding-request loop for
long-poll.
#### `@shade/inbox-server`
- `createBridgeRoutes({ store, crypto, events, … })` returns
`{ app, websocket }`.
- `GET /v1/bridge/stream` — SSE feed, one envelope per `event:
envelope`. Heartbeats every 15 s as `: ping` comments.
- `GET /v1/bridge/poll?timeoutMs=…` — long-poll, default 25 s server
hold under typical proxy idle cutoffs, hard cap 55 s.
- `GET /v1/bridge/ws` — Bun-WebSocket upgrade, JSON frame per
envelope.
- Push-style delivery via `InboxServerEvents`
(`inbox.blob_stored`); falls back to a 1 s polling timer when no
events emitter is wired.
- Cross-endpoint replay-protected: `kind` is bound into the canonical
signed payload so a `/poll` signature cannot reach `/stream`.
#### `@shade/server` standalone container
- Bridge routes mount on the same Hono app + Bun.serve as the prekey
and inbox routes — no extra port, no extra env vars.
### Acceptance criteria (V3.7)
- [x] Same "send 100 small messages" suite passes on WS, SSE, and
long-poll.
- [x] Client that starts with WS and is blocked by proxy continues
automatically via SSE — and on through to long-poll if SSE is
also blocked — without message loss.
- [x] Long-poll fallback uses no more than one outstanding request per
client.
---
## [Unreleased] — Async Store-and-Forward (V3.6)
A dedicated relay (`@shade/inbox-server`) holds ciphertext blobs with TTL
+ auth so a sender can deliver to an offline recipient. Server stores
only `address || msgId || ciphertext-bytes || expires_at`; the prekey
server stays public-keys-only, and the relay never holds plaintext or
private keys. See `docs/inbox.md` and `docs/archive/V3.6.md`.
### Added
#### `@shade/inbox` (new)
- `Inbox` — high-level orchestrator. Buffers outgoing PUTs in a durable
queue, polls + acks incoming blobs, and exposes
`onMessageQueued(handler)` (the vendor-neutral push-trigger hook
mandated by V3.6) and `onIncoming(handler)`.
- `InboxClient` — low-level HTTP client (`register`, `put`, `fetch`,
`ack`, `unregister`).
- `OutgoingQueueStore` interface + `MemoryOutgoingQueueStore` default —
swap in a SQLite/IDB backend so queue survives a process restart.
- `CursorStore` interface + `MemoryCursorStore` default for the receive
cursor.
- `computeMsgId(ciphertext)` helper — `lowercase-hex(sha256(ciphertext))`.
#### `@shade/inbox-server` (new)
- `createInboxServer({ crypto, store, ... })` Hono app exposing:
- `POST /v1/inbox/register` — TOFU bind address ↔ signing key.
- `DELETE /v1/inbox/register/:address` — signed unregister.
- `POST /v1/inbox/:address` — signed PUT, idempotent on `(address, msgId)`,
rejects mismatched `msgId !== sha256(ciphertext)` and bodies past
`maxBlobBytes` (default 1 MiB) or per-recipient quota (default 1000).
- `POST /v1/inbox/:address/fetch` — signed challenge, cursor-paginated.
- `DELETE /v1/inbox/:address/:msgId` — signed ack.
- `InboxStore` interface + `MemoryInboxStore` default.
- `InboxPruneTask` — periodic prune of expired blobs (cron, default 5 min).
- `InboxServerEvents` — structural-only event emitter for observability.
#### `@shade/storage-sqlite`
- `SqliteInboxStore` — `(address, expires_at)` + `(address, received_at)` +
`(expires_at)` indexes. `SHADE_INBOX_DB_PATH` env var for the file path.
#### `@shade/storage-postgres`
- `PostgresInboxStore` — concurrent-safe via `INSERT … ON CONFLICT` and a
per-row `nextval('shade_inbox_seq')`. `ensureInboxServerTables(sql)` is
exported for embedded deployments.
#### `@shade/server` standalone container
- Inbox routes mount alongside prekey routes on the same Hono app.
- New env vars: `SHADE_INBOX_DB_PATH`, `SHADE_INBOX_PG_URL`,
`SHADE_INBOX_PRUNE_INTERVAL_MINUTES`. If `SHADE_INBOX_PG_URL` is unset
the inbox falls back to `SHADE_PREKEY_PG_URL` (single Postgres deploy).
### Acceptance criteria (V3.6)
- [x] Sender → recipient with no online overlap; payload < 1 MiB; first
poll after recipient startup pulls the queued message.
- [x] Server-DB dump exposes no plaintext and no sender-recipient graph
beyond byte-pair sizes (sender pubkey is per-PUT TOFU; only the
recipient address is persisted).
- [x] Replay of PUT with the same `msgId` returns 200 with
`idempotent: true` instead of 409, and no second row is written.
## [0.4.0] — 2026-05-02 — Fingerprint Gates & Trust UX (V3.3)
Blocking verification gates for the handful of operations where MITM risk
is real. Apps stay alert-fatigue-free for ordinary chat, but `upload()`
of a large file, `importBackup()`, and `acceptIdentityChange()` now run
through user-registered handlers before they touch anything sensitive.
See `docs/trust-ux.md` and `docs/archive/V3.3.md`.
### Added
#### `@shade/sdk`
- `Shade.beforeFirstLargeFile(threshold, handler)` — gate runs in
`upload()` when the file size meets the threshold (default 10 MiB) and
the peer is unverified.
- `Shade.beforeBackupImport(handler)` — gate receives the fingerprint of
the identity *embedded in the backup blob*, before any state is written.
- `Shade.beforeNewDeviceTrust(handler)` — gate runs from
`Shade.acceptIdentityChange()`. The peer's identity-version is bumped
first, so any prior verification automatically goes stale.
- `Shade.beforeInboxFanout(handler)` — reserved hook for V3.6 fan-out;
apps can register today.
- `Shade.markPeerVerified(address)` / `isPeerVerified(address)` /
`unmarkPeerVerified(address)` — manual control over persisted
verification state.
- `decryptBackup` / `applyBackupPayload` — split of the backup pipeline
so callers can inspect a backup's identity fingerprint before writing.
- New `FingerprintGateRegistry` exported for advanced integrations.
#### `@shade/core`
- `FingerprintNotVerifiedError` (HTTP 403) — raised when a gate handler
returns `false`, throws, or is missing in environments that policy-
forbid TOFU.
- `PeerVerification` + `PeerVerificationSource` types and storage
methods on `StorageProvider`: `savePeerVerification`,
`getPeerVerification`, `removePeerVerification`,
`getPeerIdentityVersion`, `bumpPeerIdentityVersion`.
#### Storage backends
- `MemoryStorage`, `SQLiteStorage`, `PostgresStorage`,
`EncryptedSQLiteStorage`, `EncryptedPostgresStorage` all carry the new
`peer_verifications` + `peer_identity_versions` tables.
#### `@shade/widgets`
- `<FingerprintGate peerAddress=... />` — render-prop wrapper that blocks
children until the peer's safety number is verified at the current
identity-version. SSR-safe; ships a default fallback with "Copy OOB
text" + "I have verified" actions.
- `<FingerprintCompare onVerified=... />` — existing widget extended with
the same two actions when wired to a callback.
- `formatOobText(peerAddress, fingerprint)` helper exported.
### Changed
- `@shade/sdk` version bumped to 0.4.0 alongside all packages (lockstep
per ROADMAP convention).
### Migration
- No breaking changes. Apps that don't register gate handlers get
warning-mode TOFU automatically (`'tofu-after-warning'` source on the
persisted verification). To upgrade to hard gates, register handlers
for the operations you use. Existing `<FingerprintCompare />` calls
keep working.
## [0.3.0] — 2026-05-02 — Shade Files
E2EE filesystem RPC primitive — drop-in entrypoints for any consumer that

183
MIGRATION.md
View File

@@ -164,9 +164,190 @@ Nova's `pushDevices.encryptionKey` column is a per-device static AES key. To mig
During the rollout, send notifications with a `v: 1` (legacy) or `v: 2` (Shade) field so old and new clients coexist.
## Migration to at-rest encryption (V3.2)
Shade 0.4.0 ships `@shade/storage-encrypted` — opt-in AES-256-GCM
encryption of every sensitive payload in the local SQLite/Postgres store.
Existing 0.3.x deploys keep their unencrypted DB and behave exactly as
before; encryption is enabled per-deployment with one CLI command.
### One-shot migration (SQLite)
```bash
# Encrypts in place, drops unencrypted tables, leaves a .bak alongside.
shade migrate-storage \
--key-source passphrase \
--passphrase "$SHADE_STORAGE_PASSPHRASE" \
--salt-file /data/shade-client.db.salt
```
For a dry run that validates every row without writing:
`shade migrate-storage … --dry-run`.
### Code-level switch
Replace:
```ts
import { SQLiteStorage } from '@shade/storage-sqlite';
const storage = new SQLiteStorage('/data/shade-client.db');
```
with:
```ts
import { KeyManager, EncryptedSQLiteStorage } from '@shade/storage-encrypted';
const km = await KeyManager.open({
kind: 'passphrase',
passphrase: process.env.SHADE_STORAGE_PASSPHRASE!,
salt: loadSaltFromDisk(),
});
const storage = await EncryptedSQLiteStorage.open({
dbPath: '/data/shade-client.db',
keyManager: km,
});
```
The encrypted store implements the same `StorageProvider`, so
`ShadeSessionManager` and the rest of the wiring is unchanged.
See `docs/storage-encryption.md` for the full design, key sources
(passphrase / OS keychain / app-injected) and rotation.
## Migrating from 0.3.x to 4.0 (GA)
Shade 4.0 is the GA-frozen baseline. Everything from V3.2–V3.12 is
merged, externally reviewed, and the wire format is locked. Nothing is
breaking on the wire compared to 0.4.x — peers continue to interoperate.
The 4.0 migration is therefore mostly **opt-in surface activation**
plus a version-bump.
### What stays the same
- Wire envelope `0x02` (RatchetMessage) with u32 length-prefixes.
- Wire envelope `0x11` (stream-chunk) for `@shade/streams`.
- HTTP shape of all `/v1/keys/...` and `/v1/transfer/...` endpoints.
- All `StorageProvider` core method signatures.
- Identity fingerprints, X3DH flow, Ed25519 signature format.
A 0.3.x peer that has not enabled any opt-ins talks to a 4.0 peer
without code changes. The version bump is semantic ("we have completed
the audit cycle"), not breaking.
### What's new (opt-in)
| Surface | Package | How to enable |
|---------|---------|---------------|
| At-rest encryption | `@shade/storage-encrypted` | `shade migrate-storage` (see above) |
| Async store-and-forward | `@shade/inbox`, `@shade/inbox-server` | `createInboxServer()` + `new Inbox()` |
| Bridge transports (SSE, long-poll) | `@shade/transport-bridge`, `createBridgeRoutes()` | mount bridge routes; `FallbackBridgeTransport` |
| Web Workers crypto | `@shade/crypto-web/worker` | `shade.configureWorkerCrypto({ workerUrl })` |
| Social key recovery | `@shade/recovery` | `setupRecovery / attachGuardian / requestRecovery` |
| WebRTC P2P transport | `@shade/transport-webrtc` (peer-dep) | `shade.configureWebRTC({ factory })` |
| Key Transparency | `@shade/key-transparency`, `createPrekeyServerWithKT(...)` | server: `keyTransparency: { ... }` config; client: `keyTransparency: { mode, logPublicKey }` on `createShade` |
| Trust UX gates | built-in to `@shade/sdk` | `shade.beforeFirstLargeFile / beforeBackupImport / beforeNewDeviceTrust(...)` |
| Files RPC | `@shade/files` | `shade.files.serve(handler)` + `shade.files.client(peer)` |
Pulling in **none** of these gives you the 1.0-shape API at 4.0 quality
(audit-completed, soak-tested). Pulling in **all** of them gives the
full 4.0 stack.
### Schema additions
`StorageProvider` implementations (sqlite, postgres, encrypted variants)
auto-create the additional tables on `ensureTables()` /
`initialize()`. The 4.0 superset:
```sql
-- V3.2 (storage encryption) — only when EncryptedSQLiteStorage / EncryptedPostgresStorage is used
shade_master_key_meta(...) -- KeyManager fingerprint + scrypt params
shade_field_keys(...) -- per-(table, column) wrapped DEKs
-- V3.3 (fingerprint gates)
peer_verifications(...) -- markPeerVerified persistence
peer_identity_versions(...) -- bump on acceptIdentityChange
-- V3.6 (inbox relay)
shade_inbox_register(...) -- TOFU bind address ↔ signing key
shade_inbox_blobs(...) -- ciphertext blobs with TTL + msgId
-- V3.10 (recovery)
shade_recovery_setup(...) -- per-recoverer state
shade_recovery_deposits(...) -- per-guardian deposited shares
-- V3.12 (KT — server only)
shade_kt_leaves(...) -- append-only Merkle leaves
shade_kt_index(...) -- address-sorted commitment
shade_kt_sths(...) -- signed tree heads
-- streams resume (V0.2.0+, listed for completeness)
stream_state(...) -- at-rest encrypted streamSecret
```
A 0.3.x deploy that upgrades the package without enabling any new
surface gets these tables created on first start; they stay empty
unless the corresponding feature is wired. There is **no destructive
migration**. To verify before upgrading production:
```bash
shade doctor --db-path /data/shade-client.db
```
The CLI reports any mismatch between the on-disk schema and the version
the installed packages expect.
### Step-by-step upgrade (typical app)
1. **Bump dependencies.** Update every `@shade/*` to `^4.0.0` in your
`package.json`. Bun / npm / pnpm pull from the Gitea registry as
per `.npmrc`.
2. **Re-run install.** `bun install` (or your tool of choice). The new
table definitions ship with the storage backends — no schema-edit
PRs against your DB.
3. **Boot once with no new opt-ins.** Existing send/receive should work
byte-identically. `shade doctor` should print all green.
4. **Pick the opt-ins you actually want.** Wire them one at a time
(storage-encryption first, then fingerprint gates, then any of the
recovery / KT / WebRTC / inbox surfaces). Each surface has its own
doc under `docs/` (`storage-encryption.md`, `trust-ux.md`,
`recovery.md`, `key-transparency.md`, `webrtc.md`, `inbox.md`,
`transport.md`, `web-workers.md`, `files.md`).
5. **Run cross-version smoke.** Boot a 0.3.x peer next to a 4.0 peer in
staging; exchange a session; confirm `shade fingerprint` matches on
both ends and a round-trip message decrypts cleanly.
6. **Ship 4.0 to a canary.** Roll forward; revert path is `bun
install @shade/sdk@^0.4.0` — there is no DB write that 0.4 cannot
also read.
### Operator checklist (prekey container)
If you operate the standalone container (`gt.zyon.no/stian/shade-prekey`):
1. Pull the 4.0 image: `docker pull gt.zyon.no/stian/shade-prekey:4.0.0`.
2. Add new env vars only if you are turning the corresponding surface
on:
- `SHADE_INBOX_PG_URL` / `SHADE_INBOX_DB_PATH` — async store-and-forward.
- `SHADE_INBOX_PRUNE_INTERVAL_MINUTES` — inbox prune cadence.
- `SHADE_BRIDGE_*` — bridge / SSE / long-poll surface.
- `SHADE_KT_*` — Key Transparency mode + signing key path.
- `SHADE_TRANSFER_*` — transfer routes mounted on the same Hono app.
3. Restart with the existing volume; the inbox / KT tables auto-create
on first request.
4. Update `docs/PRODUCTION-CHECKLIST.md` items for any new surface
you've enabled (rate-limit budgets, retention policies, KT
witness-pinning).
5. Verify the [OpenAPI](packages/shade-server/openapi.yaml) endpoints
you advertise to clients now include the routes you mounted.
### What about 4.0 → 4.x?
V4.x is bug-fix only. No wire-bump until V5.0 (voice/video) which
is **additive** — it allocates new envelope types (frame-key prefixes)
that 4.0 clients ignore by design.
## Common pitfalls
1. **Don't store private keys in shared databases without encryption at rest** — Shade trusts the storage layer to be secure. Use filesystem encryption or PostgreSQL TDE if the database is on shared infrastructure.
1. **Don't store private keys in shared databases without encryption at rest** — for shared infrastructure, enable `@shade/storage-encrypted` (V3.2) or use filesystem encryption / PostgreSQL TDE. The default `SQLiteStorage` and `PostgresStorage` write unencrypted.
2. **Don't skip identity verification** — Shade gives you fingerprints (`getIdentityFingerprint()`), but it's the user's responsibility to compare them out-of-band on first contact.
3. **Don't reuse session storage between identities** — each user/device should have its own Shade storage. Mixing identities in one storage will corrupt the ratchet state.
4. **Keep prekey stocks topped up** — call `ensurePreKeyStock()` periodically (e.g., on app start or every hour). When the server runs out of one-time prekeys, new sessions will fall back to using just the signed prekey, which is slightly less secure.

96
README.md
View File

@@ -2,10 +2,18 @@
End-to-end encryption library implementing the Signal Protocol (X3DH + Double Ratchet) for TypeScript/Bun. Drop into any project — frontend, backend, mobile — to get forward secrecy, post-compromise recovery, and self-healing security.
> **0.3.0 — wire format breaking change.** The wire VERSION was bumped from
> `0x01` to `0x02` (length prefixes u16 → u32) to support inline file ops up
> to 256 KiB. **0.3.x peers cannot interoperate with 0.2.x peers** — both
> ends must upgrade. See [CHANGELOG.md](./CHANGELOG.md) for the full diff.
> **4.0.0 — General Availability.** All V3.1 → V3.12 work is merged,
> the cross-platform vector suite is green on TS + Kotlin, the threat
> model has been refreshed for every new surface, and the core stack
> (X3DH, ratchet, storage encryption, recovery, WebRTC P2P, Key
> Transparency) has been packaged for external review. The wire
> format is **unchanged from 0.4.x** — 4.0 peers interoperate with
> 0.4.x peers byte-for-byte. See
> [MIGRATION.md § 0.3.x → 4.0](./MIGRATION.md#migrating-from-03x-to-40-ga)
> for the upgrade path and [CHANGELOG.md § 4.0.0](./CHANGELOG.md) for
> the consolidated release notes. Voice / Video have been moved to
> [V5.0](./docs/V5.0.md), to be built on top of the frozen 4.0
> baseline.
## What you get
@@ -20,7 +28,12 @@ End-to-end encryption library implementing the Signal Protocol (X3DH + Double Ra
- **Crash-safe** — sessions survive container restarts, power outages, SIGKILL
- **Live observability** — bundled dashboard SPA + embeddable React widgets to see what's happening between every step
- **E2EE file transfers** — multi-lane chunked uploads/downloads with resume, integrity checks, and HTTP/WS fallback (`@shade/streams` + `@shade/transfer`)
- **WebRTC P2P transport (V3.11)** — opt-in `RTCDataChannel` upload path with public-STUN defaults, TURN-relay support, glare-safe peer pool, and automatic `MultiTransportFallback` back to HTTP when NAT traversal fails (`@shade/transport-webrtc`, [docs/webrtc.md](./docs/webrtc.md))
- **Web Workers crypto** — AEAD, HKDF, HMAC, X25519, Ed25519 and per-lane stream state run in a dedicated worker. 100 MB+ uploads stay smooth without frame drops, lane keys never cross the thread boundary (`@shade/crypto-web/worker`, [docs/web-workers.md](./docs/web-workers.md))
- **E2EE filesystem RPC** — typed `list/stat/mkdir/delete/move/read/write/getThumbnail` + custom ops between peers, with rate-limit, retention, and fingerprint-gate hooks (`@shade/files`)
- **Async store-and-forward** — deliver to offline recipients via a relay that holds ciphertext-only blobs with TTL, idempotent PUT, signed fetch/ack, and an `onMessageQueued` push-trigger hook (`@shade/inbox` + `@shade/inbox-server`)
- **Social key recovery** — Shamir-split your identity to `n` guardians; any threshold-many `k` together restore it on a new device. No centralized recovery agent; OOB-fingerprint gate on every guardian release; AES-GCM authenticates the reconstruction (`@shade/recovery` + `<RecoverySetup />` / `<RecoveryRequest />` / `<RecoveryApprove />` widgets, [docs/recovery.md](./docs/recovery.md))
- **Key Transparency (V3.12)** — opt-in append-only Merkle log over the prekey server. Every `register` / `delete` becomes a signed leaf; every bundle-fetch carries an inclusion proof; an Ed25519-signed Tree Head ties roots to a fixed `log_id`. A `LightWitness` cross-checks STHs across clients so a malicious server that splits its view or rewrites history is caught (`@shade/key-transparency`, [docs/key-transparency.md](./docs/key-transparency.md))
## Quick start
@@ -104,42 +117,90 @@ const manager = new ShadeSessionManager(
await manager.initialize();
```
## Architecture
## Architecture — keys vs. payloads
Shade splits the network into two planes. The **prekey server** only
sees public keys; **everything else** rides the encrypted Double
Ratchet between peers. If you remember nothing else from this README,
remember this picture:
```
Shade Prekey Server (Hono)
Shade Prekey Server (Hono, public keys only)
│
POST /v1/keys/register (signed)
GET /v1/keys/bundle/:address
POST /v1/keys/replenish (signed)
DELETE /v1/keys/:address (signed)
│
┌─────────────────────┴─────────────────────┐
┌──────────────────┴──────────────────┐
│ │
[Client A] [Client B]
ShadeSessionManager ShadeSessionManager
│ │
├──── X3DH ────────────────────────────────►│
│ │
│◄──── Double Ratchet messages ────────────►│
├── X3DH (handshake via prekey srv) ─►│
│ │
│◄── Double Ratchet messages ────────►│ ← end-to-end,
│ (ratchet 0x02 / chunks 0x11) │ never on the
│ │ prekey server
│◄── @shade/transfer chunks ─────────►│
│ POST /v1/transfer/:id/chunk │ ← peer-to-peer
│ GET /v1/transfer/:id/state │ HTTP, opaque
│ │ ciphertext
SQLiteStorage / PostgresStorage SQLiteStorage / PostgresStorage
(private keys + sessions) (private keys + sessions)
```
### What goes via the prekey server
- Identity public keys (Ed25519 + X25519)
- Signed prekeys + one-time prekey bundles
- Registration / replenish / delete writes, all Ed25519-signed
- Operator-only metrics, `/health`, and the optional observer
dashboard
### What does **not** go via the prekey server
- **Message plaintext, ever.** Encrypted ratchet envelopes flow peer-
to-peer over whatever transport you choose (HTTP, WebSocket, your
own broker).
- **File chunks.** `@shade/transfer` POSTs ciphertext directly to the
receiver's `/v1/transfer/:streamId/chunk` route — the prekey server
is not involved.
- **Identity private keys.** They never leave the device's storage.
- **Filesystem RPC.** `@shade/files` rides the Double Ratchet for
control + small payloads, then promotes to direct `@shade/transfer`
streams for larger blobs.
- **Stream resume secrets.** Persisted only on the local device,
encrypted under a device-key derived from the identity signing key.
The prekey server is metadata-bearing (see `THREAT-MODEL.md § 2`):
it sees who registers, who fetches whose bundle, and when. It does
**not** see message contents, transfer contents, or session state.
For the full threat model and mitigations, read
[THREAT-MODEL.md](./THREAT-MODEL.md). For deployment-time guarantees,
read [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md).
## Packages
| Package | Purpose |
|---------|---------|
| `@shade/core` | Protocol logic (X3DH, Double Ratchet, session manager, errors, events) |
| `@shade/crypto-web` | SubtleCrypto + @noble/curves provider, in-memory storage |
| `@shade/crypto-web` | SubtleCrypto + @noble/curves provider, in-memory storage. Includes the V3.8 Web Workers entrypoint (`@shade/crypto-web/worker`) — drop-in `WorkerCryptoProvider` plus `createEncryptStream` / `createDecryptStream` TransformStream factories |
| `@shade/storage-sqlite` | Persistent SQLite storage (zero-config, bun:sqlite) |
| `@shade/storage-postgres` | PostgreSQL storage with Drizzle for shared databases |
| `@shade/server` | Prekey server (Hono routes, auth, rate limit, health, metrics) |
| `@shade/transport` | HTTP + WebSocket transport wrappers with auto-encryption |
| `@shade/transport-bridge` | WS → SSE → long-poll fallback chain (V3.7) — single `IncomingMessage` shape across transports for clients that can't keep a WebSocket open |
| `@shade/transport-webrtc` | V3.11 P2P chunk transport via `RTCDataChannel`. Plugs into `@shade/transfer` as an `ITransferTransport`; signaling rides Shade's own ratchet. Memory factory + native (`globalThis.RTCPeerConnection`) factory included; `MultiTransportFallback([webrtc, http])` wired automatically when `shade.configureWebRTC()` is called. |
| `@shade/proto` | Compact binary wire format (smaller than JSON) |
| `@shade/streams` | Multi-lane chunk encryption — HKDF-derived per-lane keys, deterministic AES-GCM nonces, streaming SHA-256 |
| `@shade/transfer` | Transfer engine on top of streams: parallel lanes, resume, HTTP + WS transport with auto-fallback, integrity verification |
| `@shade/files` | Typed E2EE filesystem RPC — list/stat/mkdir/delete/move/read/write/getThumbnail + custom ops, auto inline/streams routing, production hooks (rate limit, retention, fingerprint gate, metrics), React hooks |
| `@shade/recovery` | Social key recovery (V3.10) — Shamir-split identity to `n` guardians; threshold-many `k` reconstruct on a new device. AES-GCM-authenticated reconstruction; OOB-fingerprint gate per guardian release |
| `@shade/key-transparency` | Key Transparency (V3.12) — RFC 6962-style append-only Merkle log, address-index commitment, signed tree heads, and a `LightWitness` for split-view detection. Opt-in on both server and client. See [docs/key-transparency.md](./docs/key-transparency.md) |
| `@shade/inbox-server` | Async store-and-forward relay (V3.6) — Hono routes, signed PUT/FETCH/DELETE, per-recipient TTL + quota, idempotent on `(address, msgId)`. Bundles into the same standalone container as the prekey server |
| `@shade/inbox` | Inbox client + durable outgoing queue + receive cursor + push-trigger hook (`onMessageQueued`); composes on top of `Shade.send`/`Shade.receive` for offline-recipient delivery |
| `@shade/observer` | Live debugger backend (snapshot, SSE, dashboard) — see [README](./packages/shade-observer/README.md) |
| `@shade/widgets` | Embeddable React widgets including transfer uploader/downloader — see [README](./packages/shade-widgets/README.md) |
| `@shade/dashboard` | Standalone dashboard SPA bundled into the observer |
@@ -181,13 +242,22 @@ bun run publish:all
| **Memory zeroization** | Key material is zeroed after use (best-effort in JS) |
| **Identity verification** | Safety numbers (60 digits) for out-of-band comparison |
| **Identity rotation** | 7-day grace period for old sessions during rotation |
| **Key Transparency** *(V3.12, opt-in)* | Append-only Merkle log + signed tree heads + witness gossip — split-view and history-rewrite are detected by clients |
## Documentation
- [docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md) — **Modular toolkit**: pick packages by scenario (messages, files, browser, ops)
- [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md) — Pre-flight gates for going to production
- [docs/files.md](./docs/files.md) — `@shade/files` API + design (filesystem RPC, custom ops, hooks, React)
- [docs/streams.md](./docs/streams.md) — `@shade/streams` + `@shade/transfer` deep dive
- [SECURITY.md](./SECURITY.md) — Reporting vulnerabilities, security policy
- [docs/recovery.md](./docs/recovery.md) — `@shade/recovery` social key recovery (V3.10): Shamir setup, guardian-side gates, threshold tuning
- [docs/streams.md](./docs/streams.md) — `@shade/streams` + `@shade/transfer` deep dive (incl. hardening + retention)
- [docs/inbox.md](./docs/inbox.md) — `@shade/inbox` + `@shade/inbox-server` async store-and-forward relay (V3.6)
- [docs/transport.md](./docs/transport.md) — `@shade/transport-bridge` SSE / long-poll / WS bridge layer (V3.7)
- [docs/webrtc.md](./docs/webrtc.md) — `@shade/transport-webrtc` P2P transport (V3.11): NAT-traversal, TURN config, glare resolution, wire format, multi-fallback wiring
- [docs/key-transparency.md](./docs/key-transparency.md) — `@shade/key-transparency` (V3.12): operator + client onboarding, witness role, recovery procedures
- [docs/V3.12-DESIGN.md](./docs/V3.12-DESIGN.md) — V3.12 design notat (threat model, RFC 6962 vs CONIKS choice, freshness model, open-questions resolution)
- [docs/web-workers.md](./docs/web-workers.md) — V3.8 Web Workers crypto: setup, bundler recipes (Vite/Webpack/Rollup), Safari notes, lifecycle, threat-model
- [SECURITY.md](./SECURITY.md) — Reporting vulnerabilities, security policy, threat-/test-matrix
- [THREAT-MODEL.md](./THREAT-MODEL.md) — Honest threat model and assumptions
- [examples/](./examples/) — Runnable example applications, including
[`07-streams-upload`](./examples/07-streams-upload) (multi-lane file transfer)

84
SECURITY.md
View File

@@ -1,16 +1,54 @@
# Security Policy
## Review status
| Area | Status | Notes |
|------|--------|-------|
| Internal review | **Done** | Every mitigation in `THREAT-MODEL.md` is cross-linked to at least one automated test (see [Threat-/test-matrix](#threat--test-matrix) below). The matrix is enforced by `tests/security/*` + the cross-platform vector suite. |
| Independent code review | **Pending** | Targeted for **V4.0**. No external review has been completed. |
| Independent crypto review | **Pending** | Targeted for **V4.0** alongside the audit. |
| Pen test | **Pending** | Targeted for **V4.0**. |
> **Read this:** Shade implements the Signal Protocol primitives
> (X3DH + Double Ratchet) on top of `@noble/curves` and SubtleCrypto.
> The protocol is well-studied; the **implementation** has not yet been
> audited externally. Treat the wire format as stable but the
> implementation as "production-ready in trusted contexts" until V4.0
> closes the audit gap. The `THREAT-MODEL.md` cells with no test
> linkage are documentary, not enforced.
## Reporting a Vulnerability
If you discover a security vulnerability in Shade, please report it privately by emailing the maintainer rather than opening a public issue. We take all reports seriously and will respond within 48 hours.
If you discover a security vulnerability in Shade, please report it
privately by emailing the maintainer rather than opening a public
issue. We take all reports seriously and will respond within 48 hours.
### How to report
1. **Email:** the maintainer email listed in the package metadata.
For coordinated disclosure, prefer email over GitHub/Gitea so the
issue does not become public before a fix ships.
2. **PGP / age:** if you need encrypted reporting, ask for a key
over the same email — keys are not bound to the repo to avoid
key-rotation drift.
3. **Scope:** CVE-style severity (CVSS v3.1) is appreciated but not
required. A working reproduction is more valuable than a CVSS
score.
When reporting, please include:
- A description of the vulnerability
- Steps to reproduce
- Steps to reproduce (a runnable script or test case)
- Affected versions
- Potential impact
- Any suggested mitigation
We commit to:
- Acknowledging receipt within 48 hours.
- A first-pass triage within 7 days.
- A coordinated disclosure timeline once severity is agreed; for
high-severity issues we aim to ship a patched release within 30
days of triage.
## What's in scope
Shade aims to provide:
@@ -46,3 +84,45 @@ Shade uses well-established primitives:
- **HMAC-SHA256** for chain key advancement (via Web Crypto SubtleCrypto)
These match the Signal Protocol specification.
---
## Threat-/test-matrix
This is the consolidated index that backs `THREAT-MODEL.md`. Every
threat-model row that claims a mitigation must point to at least one
test file here. Pull requests that add a new mitigation must add a
matrix row in the same change.
| Threat-model row | Mitigation | Test file(s) |
|------------------|------------|--------------|
| § 1 Network attacker — signed writes | Ed25519 signature on every write | `packages/shade-server/tests/server.test.ts` |
| § 1 Network attacker — replay window | ±5 min `signedAt` enforcement | `packages/shade-server/tests/server.test.ts` (`"rejects registration with stale signedAt"`) |
| § 1 Network attacker — header AAD | Ratchet headers bound to ciphertext | `packages/shade-core/tests/ratchet.test.ts`, `packages/shade-streams/tests/tamper.test.ts`, `packages/shade-streams/tests/aead.test.ts` |
| § 1 Network attacker — forward secrecy | DH ratchet step + chain-key zeroize | `packages/shade-core/tests/ratchet.test.ts`, `packages/shade-crypto-web/tests/hardening.test.ts` |
| § 2 Compromised prekey server — public-only storage | Prekey store never accepts a private key | `packages/shade-server/tests/server.test.ts`, `packages/shade-storage-sqlite/tests/sqlite-prekey-store.test.ts` |
| § 2 Compromised prekey server — signed replenish/delete | Per-identity Ed25519 signature | `packages/shade-server/tests/server.test.ts` |
| § 2 Compromised prekey server — fake-bundle detection | Out-of-band fingerprint comparison | `packages/shade-core/tests/fingerprint-session.test.ts` |
| § 3 Endpoint compromise — forward secrecy | Old keys not recoverable from leak | `packages/shade-core/tests/ratchet.test.ts`, `packages/shade-crypto-web/tests/hardening.test.ts` |
| § 3 Endpoint compromise — post-compromise security | First DH ratchet evicts leaked state | `packages/shade-core/tests/ratchet.test.ts` (`"alternating messages trigger DH ratchets"`) |
| § 3 Endpoint compromise — memory zeroization | Buffers wiped after use | `packages/shade-crypto-web/tests/hardening.test.ts` (`"zeroize"`) |
| § 3 Endpoint compromise — identity-rotation invalidates resume | Device-key bound to signing key | `packages/shade-core/tests/identity-rotation.test.ts`, `packages/shade-transfer/tests/resume.test.ts` |
| § 4 Compromised device storage — at-rest stream secrets | Resume secret AES-GCM under device-key | `packages/shade-transfer/tests/resume.test.ts` |
| § 4 Compromised device storage — at-rest session DB | **Pending V3.2** | _none yet_ |
| § 5 Timing side-channel — constant-time compare | XOR accumulator | `packages/shade-crypto-web/tests/hardening.test.ts` (`"timing variance stays bounded across mismatch positions"`) |
| § 5 Timing side-channel — primitives | SubtleCrypto + @noble/curves | `packages/shade-crypto-web/tests/provider.test.ts`, `packages/shade-streams/tests/aead.test.ts` |
| § 6 DoS — per-IP register/bundle rate limit | Token bucket per IP | `packages/shade-server/tests/rate-limit.test.ts` |
| § 6 DoS — per-identity replenish/delete rate limit | Token bucket per identity | `packages/shade-server/tests/rate-limit.test.ts` |
| § 6 DoS — body size cap (64 KiB) | Hono middleware | `packages/shade-server/tests/server.test.ts` |
| § 6 DoS — address validation | Regex + NFKC + length | `packages/shade-server/tests/server.test.ts` |
| § 6 DoS — per-sender ops/byte quota (`@shade/files`) | RateLimiter token bucket | `packages/shade-files/tests/security/quota.test.ts` |
| § 6 DoS — replay protection (`@shade/files`) | Idempotency cache | `packages/shade-files/tests/security/replay.test.ts` |
| § 6 DoS — fingerprint gate (`@shade/files`) | Per-sender trust check | `packages/shade-files/tests/security/fingerprint-gate.test.ts` |
| § 6 DoS — tampered envelope reject (`@shade/files`) | AEAD reject | `packages/shade-files/tests/security/tampered-envelope.test.ts` |
| § 8a Recovery — k-1 collusion impossible | Shamir Secret Sharing over GF(2^8) | `packages/shade-recovery/tests/shamir.test.ts`, `packages/shade-recovery/tests/adversarial.test.ts` |
| § 8b Recovery — forged share rejected | AES-GCM tag on backup blob + subset-search | `packages/shade-recovery/tests/adversarial.test.ts` (`"a corrupted share never authenticates against the backup AEAD tag"`) |
| § 8c Recovery — guardian OOB-fingerprint gate | Two-checkbox `<RecoveryApprove />` + decline propagation | `packages/shade-recovery/tests/adversarial.test.ts` (`"approve handler that REJECTS a wrong fingerprint never sends a grant"`, `"throwing approve handler counts as decline with descriptive reason"`) |
| § 9 Cross-sender X3DH state corruption | `initReceiverSession` copies keypair | `packages/shade-core/tests/ratchet.test.ts` (`"does not mutate the caller-provided keypair after a DH ratchet step"`), `packages/shade-recovery/tests/integration.test.ts` |
If you add a new mitigation, add a row here in the same PR — the
threat model is the contract; this matrix is the proof.

328
THREAT-MODEL.md
View File

@@ -2,6 +2,13 @@
This document describes what Shade protects against and what it doesn't. Read this before deploying Shade in any context where the answers matter.
> Each numbered "Mitigations" entry below ends with a `[tests:]`
> footnote that links to the concrete test file(s) demonstrating the
> mitigation. If a mitigation has no `[tests:]` line, treat it as
> documentary — there is no automated test holding the line yet.
> See [SECURITY.md § Threat-/test-matrix](./SECURITY.md#threat--test-matrix)
> for the consolidated index.
## Assets
The thing we're protecting:
@@ -16,9 +23,13 @@ Can intercept, modify, drop, replay, and inject network traffic between clients
**Mitigations:**
- All identity-key writes to the prekey server are signed (Ed25519). Tampering is detected.
`[tests: packages/shade-server/tests/server.test.ts — "rejects unsigned registration", "rejects registration with wrong signing key"]`
- Signed requests have a 5-minute replay window.
`[tests: packages/shade-server/tests/server.test.ts — "rejects registration with stale signedAt"]`
- The Double Ratchet binds message headers to ciphertext via AES-GCM AAD, so header tampering breaks decryption.
`[tests: packages/shade-core/tests/ratchet.test.ts — "tampered ciphertext fails", "tampered header (counter) fails due to AAD"; packages/shade-streams/tests/tamper.test.ts; packages/shade-streams/tests/aead.test.ts]`
- Forward secrecy: even if an attacker captures all traffic, compromising a key later doesn't help them read past messages.
`[tests: packages/shade-crypto-web/tests/hardening.test.ts; packages/shade-core/tests/ratchet.test.ts — DH ratchet steps + out-of-order delivery]`
**NOT mitigated:**
- Initial session establishment can be MITM'd if users don't verify identity fingerprints. The prekey server could distribute a fake bundle on first contact. Always compare safety numbers out-of-band for high-stakes communications.
@@ -28,19 +39,31 @@ The server holds identity public keys and prekey bundles. It can serve them to a
**Mitigations:**
- The server only stores PUBLIC keys, never private ones.
`[tests: packages/shade-server/tests/server.test.ts — registration, bundle fetch, replenish; packages/shade-storage-sqlite/tests/sqlite-prekey-store.test.ts]`
- Write operations are signed with the identity private key, so the server can't forge new identities or replenishments without the user's key.
`[tests: packages/shade-server/tests/server.test.ts — "rejects replenishment signed by wrong identity", "rejects delete signed by wrong identity"]`
- Bundle fetches are unauthenticated, so a malicious server can serve fake bundles. Detection requires out-of-band fingerprint comparison.
`[tests: packages/shade-core/tests/fingerprint-session.test.ts]`
**NOT mitigated:**
- A malicious server can substitute one user's prekey bundle with the server operator's own keys, enabling MITM at session establishment. Users must verify safety numbers to detect this.
**Partially mitigated by V3.12 Key Transparency** (opt-in):
- When the operator runs the server with `keyTransparency: { ... }` and clients pin the operator's STH-signing public key, every bundle fetch returns a Merkle inclusion proof against an append-only Signed Tree Head. A server that swaps `alice`'s bundle for one client and not another, or rewrites history to hide an earlier swap, is detected by an independent witness. KT does **not** prevent first-contact impersonation — a never-seen-before address can still be served maliciously on its very first registration.
`[tests: packages/shade-key-transparency/tests/manager.test.ts — "rotation: new register replaces old"; packages/shade-transport/tests/kt-split-view-e2e.test.ts — "two divergent views at the same tree_size are caught by witness"; packages/shade-server/tests/kt.test.ts — "bundle response carries verified inclusion proof"]`
### 3. Compromised endpoint (post-compromise)
Attacker briefly gains code execution or filesystem access on a user's device, exfiltrates session state, then loses access.
**Mitigations:**
- Forward secrecy: messages sent BEFORE the compromise cannot be decrypted with the leaked state. Old chain keys are zeroed after use.
`[tests: packages/shade-core/tests/ratchet.test.ts — basic send/receive, ping-pong; packages/shade-crypto-web/tests/hardening.test.ts — zeroize]`
- Post-compromise security: as soon as a peer initiates a new DH ratchet step, the leaked state becomes useless for new messages.
`[tests: packages/shade-core/tests/ratchet.test.ts — "alternating messages trigger DH ratchets"]`
- Memory zeroization: message keys and chain keys are wiped from JS memory after use (best-effort — V8 may retain copies).
`[tests: packages/shade-crypto-web/tests/hardening.test.ts — "zeroize" describe block]`
- Identity rotation invalidates leaked at-rest stream-resume secrets (device-key derived from signing key).
`[tests: packages/shade-core/tests/identity-rotation.test.ts; packages/shade-transfer/tests/resume.test.ts]`
**NOT mitigated:**
- An ongoing endpoint compromise can read messages in real time and exfiltrate identity private keys.
@@ -49,36 +72,317 @@ Attacker briefly gains code execution or filesystem access on a user's device, e
### 4. Compromised device storage
Attacker gains access to the persistent storage (e.g., steals the SQLite file or dumps the PostgreSQL table).
**Mitigations:**
- Stored data includes private keys but is unencrypted at rest. Shade does NOT encrypt the storage layer — it assumes the database is in a trusted environment.
**Mitigations (default, no at-rest encryption):**
- Stream-resume secrets *are* encrypted at rest under a device-key derived from the identity signing key, so a stolen DB without the live identity key cannot resume in-flight transfers.
`[tests: packages/shade-transfer/tests/resume.test.ts]`
- Filesystem-level encryption (LUKS, FileVault, BitLocker) is recommended but is the user's responsibility.
**NOT mitigated:**
- Filesystem-level encryption (LUKS, FileVault) is the user's responsibility.
- Database TLS in transit is the user's responsibility.
**Mitigations (with at-rest encryption enabled — V3.2 / `@shade/storage-encrypted`):**
- All sensitive payloads are sealed with AES-256-GCM under per-(table, column) field keys derived from a passphrase (scrypt) / OS keychain / app-injected master key. A stolen DB file alone yields no usable private key material.
`[tests: packages/shade-storage-encrypted/tests/encrypted-sqlite.test.ts]`
- AAD binds (table, column, pk) so an attacker cannot swap rows or move ciphertext between columns without triggering decrypt failure.
`[tests: packages/shade-storage-encrypted/tests/encrypted-sqlite.test.ts — "row swap (sessions) → decrypt fails due to AAD mismatch"]`
- Bit-flips in the ciphertext blob are detected by the AEAD tag; the storage layer raises rather than returning corrupt key material.
`[tests: packages/shade-storage-encrypted/tests/aead.test.ts; encrypted-sqlite.test.ts — "flipped ciphertext byte → decrypt fails"]`
- Wrong passphrase / wrong keychain entry is rejected up-front via a fingerprint check, never silently writing under the wrong key.
`[tests: packages/shade-storage-encrypted/tests/encrypted-sqlite.test.ts — "rejects open with wrong key (fingerprint mismatch)"]`
- Online key rotation re-keys every row without downtime; the old key no longer opens the DB after rotation.
`[tests: packages/shade-storage-encrypted/tests/migrate.test.ts — "re-keys all rows; old key no longer opens DB"]`
**NOT mitigated (even with at-rest enabled):**
- A live process holds the storageKey and field keys in memory; an attacker who can read process memory (e.g., via `/proc/<pid>/mem`, swap dump, hibernation file) recovers the keys and thus the data. At-rest encryption protects the DB *file*, not the running process.
- The kernel's swap partition is not encrypted by Shade. If the OS pages key material to disk, it can be recovered. Use an encrypted swap device.
- A coredump of the live process exposes plaintext private keys.
- Filesystem-level encryption of the DB *backup* (e.g. `.bak` file produced by `shade migrate-storage`) is the operator's responsibility — the backup is plaintext during the brief migration window.
- If the master key is lost (forgotten passphrase, deleted keychain entry, lost injected key) the DB is permanently unrecoverable. V3.10 (Social Recovery) is the long-term mitigation.
### 5. Side-channel attacks (timing)
Attacker measures timing of identity verification operations to recover key bits.
**Mitigations:**
- All comparisons of secret material use constant-time XOR-accumulator comparison (`constantTimeEqual`).
`[tests: packages/shade-crypto-web/tests/hardening.test.ts — "constantTimeEqual", "timing variance stays bounded across mismatch positions"]`
- AES-GCM and the underlying primitives are constant-time as implemented by SubtleCrypto and @noble/curves.
`[tests: packages/shade-crypto-web/tests/provider.test.ts; packages/shade-streams/tests/aead.test.ts]`
**NOT mitigated:**
- JavaScript JIT compilation can introduce timing variability that's hard to control.
- We don't claim resistance to power-analysis or fault-injection attacks (out of scope for a JS library).
### 6. Denial of service
### 6. Malicious or compromised inbox relay (V3.6 store-and-forward)
The inbox relay holds **ciphertext blobs with TTL** so senders can deliver
to offline recipients. It is a separate trust domain from the prekey
server, and exposes a different surface.
**Mitigations:**
- The relay only stores `address || msgId || ciphertext-bytes || expires_at`.
Plaintext, ratchet state, and any private keys live exclusively on the
client. A DB dump leaks no message content.
`[tests: packages/shade-inbox-server/tests/routes.test.ts; packages/shade-inbox-server/tests/lifecycle.test.ts — "Tamper resistance"]`
- Recipient identity is bound to the address via TOFU: first
`POST /v1/inbox/register` claims the slot, and subsequent fetch/ack
must be Ed25519-signed by the same key. A different key claiming an
existing address is rejected with 401.
`[tests: packages/shade-inbox-server/tests/routes.test.ts — "rejects different key claiming same address", "rejects fetch from a different signing key", "rejects ack from a different signing key"]`
- Each PUT is signed by the sender's per-PUT signing key; the relay
verifies the signature before persisting. Bad sigs return 401.
`[tests: packages/shade-inbox-server/tests/routes.test.ts — "rejects bad sender signature"]`
- `msgId = sha256(ciphertext)` is verified server-side on PUT and
recomputed client-side on FETCH. A relay that flips a bit in storage
produces a digest mismatch the recipient flags as
`inbox.message_decrypt_failed` *without* acking, so the divergence
surfaces in operator telemetry instead of being silently consumed.
`[tests: packages/shade-inbox-server/tests/routes.test.ts — "rejects mismatched msgId"; packages/shade-inbox-server/tests/lifecycle.test.ts — "Tamper resistance"; packages/shade-inbox/tests/client.test.ts — "tamper detection"]`
- Replay-window of ±5 minutes on `signedAt` (matches the prekey
server's policy). Replays past that window return 409.
`[tests: packages/shade-inbox-server/tests/routes.test.ts — "rejects stale signature (replay window)"]`
- Idempotent PUT: two clients (or a buggy retry loop) submitting the
same ciphertext do *not* create duplicate rows; the second PUT
returns 200 with `idempotent: true`.
`[tests: packages/shade-inbox-server/tests/routes.test.ts — "idempotent on duplicate ciphertext"]`
- Periodic `InboxPruneTask` drops blobs past their TTL so a slow
consumer never sees a payload past expiry.
`[tests: packages/shade-inbox-server/tests/lifecycle.test.ts — "prune removes expired blobs but keeps live ones"]`
**NOT mitigated:**
- **Sender-recipient graph leakage.** The relay sees recipient address +
per-PUT sender pubkey + ciphertext byte-counts. Privacy-sensitive
deployments should use address-hashes (`sha256(real-address || salt)`)
and rotate sender signing keys per session. Mixing/onion-routing is
out of scope for V3.6 and a candidate for a future relay tier.
- **Operator-side queue deletion.** A malicious operator can drop every
blob queued for a target, forcing senders to resend. Recipient-side
ack happens *after* successful decrypt, so a delete only burns one
delivery attempt rather than silently consuming a message.
- **TTL-based reachability signal.** A PUT silently expiring after 7
days reveals that the recipient never came online. Operators concerned
with this metadata should clamp TTLs to a fixed value via the
`quota.maxTtlSeconds` / `quota.minTtlSeconds` knobs.
### 7. Denial of service
Attacker floods the prekey server to exhaust resources or one-time prekeys.
**Mitigations:**
- Per-IP rate limiting on registration and bundle fetches.
`[tests: packages/shade-server/tests/rate-limit.test.ts — "register endpoint rate-limits per IP", "rate limit returns Retry-After header"]`
- Per-identity rate limiting on replenish and delete.
- 64KB body size limit on POST endpoints.
`[tests: packages/shade-server/tests/rate-limit.test.ts — "different keys have independent limits"]`
- 64 KiB body size limit on POST endpoints.
`[tests: packages/shade-server/tests/server.test.ts — body-size enforcement]`
- Address validation rejects path traversal and malformed inputs.
`[tests: packages/shade-server/tests/server.test.ts — "rejects invalid address format", "rejects invalid address in URL"]`
- Per-sender ops/byte quotas on `@shade/files` filesystem RPC.
`[tests: packages/shade-files/tests/security/quota.test.ts]`
- Per-recipient blob quota on `@shade/inbox-server` (default 1000 blobs
per address) + per-blob byte cap (default 1 MiB) so a single sender
cannot fill a recipient's queue.
`[tests: packages/shade-inbox-server/tests/routes.test.ts — "rejects ciphertext > maxBlobBytes", "enforces per-address quota"]`
- Per-IP token-bucket on inbox PUT/FETCH/DELETE/REGISTER routes.
**NOT mitigated:**
- Application-level DDoS at the network layer is your hosting platform's responsibility.
### 8. Social-recovery adversaries (V3.10)
Once a user has set up `@shade/recovery`, the guardian set becomes a
new attack surface. We split the threat into four cases:
**8a. Coalition of ≤ k-1 guardians.**
**Mitigations:**
- Shamir Secret Sharing over GF(2^8) is information-theoretically
secure: the shares are points on a polynomial whose constant term
is the secret, and any subset of `< k` points is consistent with
every possible secret. No coalition smaller than the threshold
recovers anything beyond the secret's length.
`[tests: packages/shade-recovery/tests/shamir.test.ts — "k-1 shares yield a wrong (random-looking) result", "property: any k-1 share subset yields a different output than the secret"; packages/shade-recovery/tests/adversarial.test.ts — "property: any (k-1) subset of shares fails to recover the key"]`
**8b. Single malicious guardian who forges a share.**
**Mitigations:**
- The reconstructed `recoveryKey` is authenticated by the AES-GCM
tag inside the backup blob (`Shade.exportBackup`'s ciphertext).
A forged share produces a different reconstructed key; AES-GCM
decryption fails.
- `requestRecovery` exhaustively tries every threshold-sized subset
of received grants until one authenticates; if none do, it raises
`RecoveryReconstructionError` and refuses to apply the result.
The user is told that at least one guardian is malicious.
`[tests: packages/shade-recovery/tests/adversarial.test.ts — "a corrupted share never authenticates against the backup AEAD tag"]`
**8c. Social-engineering (impersonator calls a guardian).**
**Mitigations:**
- The guardian's `approve` callback receives the new device's
TEMPORARY safety number; the spec REQUIRES out-of-band
comparison before approving.
- The shipped `<RecoveryApprove />` widget enforces a two-checkbox
gate ("fingerprint matches" + "I verified OOB") before the
release button is enabled.
- The protocol-level `share-decline` envelope is sent regardless of
whether the guardian's `approve` callback returns false or
throws, so a hard "no" terminates the requesting flow promptly.
`[tests: packages/shade-recovery/tests/adversarial.test.ts — "approve handler that REJECTS a wrong fingerprint never sends a grant", "throwing approve handler counts as decline with descriptive reason"]`
**NOT mitigated:**
- A guardian who is duped by an impersonator AND whose user clicks
through both checkboxes WILL release their share. Defense in
depth requires user education + per-guardian cool-down windows
(a follow-up release).
**8d. Guardian device compromise.**
If an attacker fully owns a guardian's device, they can:
- Read the share + backup blob → contributes one polynomial point.
- Ship `share-grant` envelopes if they convince the guardian's
`approve` callback to return true.
**Mitigations:**
- No single guardian's compromise is sufficient — the threshold
invariant still holds: the attacker needs `k-1` other shares to
rebuild the identity.
- Backup blobs are encrypted at-rest under the guardian's existing
StorageProvider scheme (V3.2 covers this for SQLite/Postgres
backends).
**NOT mitigated:**
- Compromise of `≥ k` guardians simultaneously is a complete break.
This is by design: the recovery flow is meant to survive *device*
loss, not coordinated mass compromise of the social graph.
### 9. Cross-sender X3DH state corruption
Before V3.10, `initReceiverSession` shared a reference to the
receiver's signed prekey keypair with the new session. The first DH
ratchet step zeroed the session's "previous" private key, which
silently zeroed the persisted signed prekey. A second X3DH from a
*different* sender to the same receiver then derived a divergent
root key and decryption failed with "wrong key or tampered data".
This was a pre-existing bug surfaced by the V3.10 multi-sender
recovery flow.
**Mitigations:**
- `initReceiverSession` now copies the localDHKeyPair into the
session so the eventual zeroize touches a scratch buffer, not
the persisted prekey.
`[tests: packages/shade-recovery/tests/integration.test.ts — "recovery from new device with all 5 guardians available"; packages/shade-core/tests/x3dh.test.ts]`
### 10. MITM bypass via skipped fingerprint verification (V3.3)
The strongest mitigation for §1 / §2 / §6 — out-of-band safety-number
verification — is a *user* responsibility. Shade 4.0 ships
`@shade/sdk` fingerprint gates that move it from "convention" to
"enforced policy on the operations that matter".
**Mitigations:**
- `Shade.beforeFirstLargeFile(threshold, handler)` — runs in `upload()`
when payload ≥ threshold (default 10 MiB) and the peer is unverified.
A handler that returns `false` (or throws / is missing in policy-
forbid-TOFU mode) raises `FingerprintNotVerifiedError` (HTTP 403).
`[tests: packages/shade-sdk/tests/fingerprint-gates.test.ts; packages/shade-files/tests/security/fingerprint-gate.test.ts]`
- `Shade.beforeBackupImport(handler)` — receives the *backup-embedded*
fingerprint before any state is written. Decrypted backups whose
embedded identity does not match the user's expectation are
rejected before they touch storage.
`[tests: packages/shade-sdk/tests/fingerprint-gates.test.ts]`
- `Shade.beforeNewDeviceTrust(handler)` — runs from
`Shade.acceptIdentityChange()` after the peer's identity-version is
bumped, so any prior verification automatically goes stale and the
user must re-verify.
`[tests: packages/shade-sdk/tests/fingerprint-gates.test.ts]`
- `markPeerVerified` / `isPeerVerified` / `unmarkPeerVerified` are
storage-backed; the `peer_verifications` + `peer_identity_versions`
tables are subject to V3.2 at-rest encryption when the encrypted
storage backend is used.
- `<FingerprintCompare />` and `<FingerprintGate />` widgets present
the safety number side-by-side and require an explicit "matches"
click before children render.
**NOT mitigated:**
- Apps that never register handlers default to "TOFU + warning". The
warning is logged, not rendered, so a UX that ignores the log
silently keeps TOFU semantics.
- Once verified, a peer's persisted verification stays valid until
identity rotation. A device-compromise that does **not** trigger
rotation keeps the verification alive.
### 11. WebRTC peer-to-peer transport (V3.11)
`@shade/transport-webrtc` lets two peers ship `@shade/transfer` chunks
over an `RTCDataChannel` instead of HTTP. The DTLS layer is opaque to
Shade; we treat WebRTC strictly as a **byte-pipe** — not a trust
boundary.
**Mitigations:**
- The same Double Ratchet that authenticates Shade messages
authenticates the SDP offer / answer / ICE / bye signaling
envelopes. A network attacker who replaces an SDP offer must
forge a ratcheted message — the receiver decrypts via the
existing peer session and rejects on AEAD failure.
`[tests: packages/shade-transport-webrtc/tests/signaling.test.ts; packages/shade-sdk/tests/webrtc-integration.test.ts]`
- Frame payloads on the DataChannel are AES-GCM-sealed by `@shade/streams`
with deterministic nonce + AAD bound to `streamId || laneId || seq ||
isLast`. A WebRTC implementation that returns altered bytes fails
AEAD verification and the receiver raises `StreamDecryptionError`.
`[tests: packages/shade-streams/tests/tamper.test.ts; packages/shade-transport-webrtc/tests/wire-format.test.ts]`
- Glare resolution is deterministic (lexicographic address compare)
so both sides converge on a single connection without re-running
signaling.
`[tests: packages/shade-transport-webrtc/tests/glare.test.ts]`
- When NAT traversal fails, `MultiTransportFallback([webrtc, http])`
demotes to HTTP within the configured `connectTimeoutMs` (default
5 s) without losing chunks already in flight. No silent stall.
`[tests: packages/shade-sdk/tests/webrtc-failover.test.ts]`
- `IRtcFactory` is pluggable; production uses
`globalThis.RTCPeerConnection` (browser / Workers / Deno),
`MemoryRtcFactory` is in-process for tests.
**NOT mitigated:**
- TURN relay metadata. If the deployment ships a TURN server,
the operator sees relayed-byte counts and timing for every flow
that traverses the relay. Use a TURN you control or a hosted
relay you trust.
- Browser/RTC stack vulnerabilities. A compromised
`RTCPeerConnection` implementation is outside the scope of a JS
library; we ride the platform's WebRTC.
- Public STUN exposes the client's public IP to the STUN server.
This is unavoidable without a privacy-preserving NAT discovery
mechanism (out of scope).
### 12. Web-Worker thread boundary (V3.8)
`@shade/crypto-web/worker` runs AEAD, HKDF, HMAC, X25519, Ed25519, and
per-lane stream state inside a dedicated Web Worker so the main thread
never holds key material for very long.
**Mitigations:**
- Lane keys, identity private keys and ratchet chain keys are passed
into the worker once at setup; subsequent operations move plaintext
via transferable `ArrayBuffer`s and never re-export keys.
`[tests: packages/shade-crypto-web/tests/worker-streams.test.ts; packages/shade-crypto-web/tests/worker-provider.test.ts]`
- Idle timeout (default 30 s) calls `terminate()` on the worker, which
drops the global JS heap and releases the OS-level memory backing
any keys that were not yet zeroized.
- `rotate()` and `destroy()` lifecycle controls let apps bound the
worst-case duration any lane key sits in worker memory.
- Worker-protocol version handshake on first message rejects mismatched
workers (e.g. cached old build).
**NOT mitigated:**
- The worker is still inside the same browsing context; an attacker
who can inject script into the page can post a malicious message
and read the worker's reply. CSP and SRI on the worker entrypoint
are the user's responsibility.
- Heap memory is not synchronously wiped when `postMessage` returns
ownership; the runtime may keep deallocated buffers around for
GC. Memory zeroization is best-effort for both threads.
## Assumptions
1. **The user has a secure way to bootstrap trust.** Either:
@@ -92,8 +396,10 @@ Attacker floods the prekey server to exhaust resources or one-time prekeys.
| Risk | Severity | Mitigation |
|------|----------|------------|
| MITM at first session establishment | High | Compare safety numbers out-of-band |
| Identity private key theft from device | Critical | Filesystem encryption, secure enclave (future) |
| Prekey server operator runs a "key oracle" attack | Medium | Distributed/federated prekey servers (future) |
| Side-channel via JIT timing variability | Low | Constant-time primitives reduce but don't eliminate |
| MITM at first session establishment | High | Compare safety numbers out-of-band; in 4.0, register `Shade.beforeFirstLargeFile` / `beforeBackupImport` / `beforeNewDeviceTrust` to enforce verification on the operations that matter (V3.3) |
| Identity private key theft from device | Critical | Filesystem encryption, secure enclave (future); V3.10 Social Recovery for *recovery* after loss |
| Prekey server operator runs a "key oracle" attack | Medium | V3.12 Key Transparency (opt-in) detects split-view + history rewrites; gossip via a `LightWitness` raises the cost of a sustained attack |
| TURN relay sees byte-counts of P2P transfers | Low–Medium | Only when WebRTC fails over to TURN. Operate your own TURN if the metadata matters |
| Side-channel via JIT timing variability | Low | Constant-time primitives reduce but don't eliminate; V3.8 Web-Worker isolation bounds the lifetime of in-memory key material |
| Metadata visibility to prekey server | Low | Acceptable for most use cases; mix networks for stronger metadata protection |
| Inbox relay sees recipient address + byte-counts | Low–Medium | Use address-hashes + per-session sender keys (V3.6 §6); mix-net relay tier is a future candidate |

12
android/.gitignore vendored Normal file
View File

@@ -0,0 +1,12 @@
# Gradle
.gradle/
build/
!gradle/wrapper/gradle-wrapper.jar
# IntelliJ / Android Studio
.idea/
*.iml
local.properties
# Captured logs
*.log

3
android/build.gradle.kts Normal file
View File

@@ -0,0 +1,3 @@
plugins {
kotlin("jvm") version "2.0.20" apply false
}

4
android/gradle.properties Normal file
View File

@@ -0,0 +1,4 @@
org.gradle.jvmargs=-Xmx2g -Dfile.encoding=UTF-8
org.gradle.parallel=true
org.gradle.caching=true
kotlin.code.style=official

BIN
android/gradle/wrapper/gradle-wrapper.jar vendored Normal file
View File

Binary file not shown.

7
android/gradle/wrapper/gradle-wrapper.properties vendored Normal file
View File

@@ -0,0 +1,7 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-8.10.2-bin.zip
networkTimeout=10000
validateDistributionUrl=true
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists

252
android/gradlew vendored Executable file
View File

@@ -0,0 +1,252 @@
#!/bin/sh
#
# Copyright © 2015-2021 the original authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# SPDX-License-Identifier: Apache-2.0
#
##############################################################################
#
# Gradle start up script for POSIX generated by Gradle.
#
# Important for running:
#
# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is
# noncompliant, but you have some other compliant shell such as ksh or
# bash, then to run this script, type that shell name before the whole
# command line, like:
#
# ksh Gradle
#
# Busybox and similar reduced shells will NOT work, because this script
# requires all of these POSIX shell features:
# * functions;
# * expansions «$var», «${var}», «${var:-default}», «${var+SET}»,
# «${var#prefix}», «${var%suffix}», and «$( cmd )»;
# * compound commands having a testable exit status, especially «case»;
# * various built-in commands including «command», «set», and «ulimit».
#
# Important for patching:
#
# (2) This script targets any POSIX shell, so it avoids extensions provided
# by Bash, Ksh, etc; in particular arrays are avoided.
#
# The "traditional" practice of packing multiple parameters into a
# space-separated string is a well documented source of bugs and security
# problems, so this is (mostly) avoided, by progressively accumulating
# options in "$@", and eventually passing that to Java.
#
# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS,
# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly;
# see the in-line comments for details.
#
# There are tweaks for specific operating systems such as AIX, CygWin,
# Darwin, MinGW, and NonStop.
#
# (3) This script is generated from the Groovy template
# https://github.com/gradle/gradle/blob/HEAD/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
# within the Gradle project.
#
# You can find Gradle at https://github.com/gradle/gradle/.
#
##############################################################################
# Attempt to set APP_HOME
# Resolve links: $0 may be a link
app_path=$0
# Need this for daisy-chained symlinks.
while
APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path
[ -h "$app_path" ]
do
ls=$( ls -ld "$app_path" )
link=${ls#*' -> '}
case $link in #(
/*) app_path=$link ;; #(
*) app_path=$APP_HOME$link ;;
esac
done
# This is normally unused
# shellcheck disable=SC2034
APP_BASE_NAME=${0##*/}
# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s
' "$PWD" ) || exit
# Use the maximum available, or set MAX_FD != -1 to use that value.
MAX_FD=maximum
warn () {
echo "$*"
} >&2
die () {
echo
echo "$*"
echo
exit 1
} >&2
# OS specific support (must be 'true' or 'false').
cygwin=false
msys=false
darwin=false
nonstop=false
case "$( uname )" in #(
CYGWIN* ) cygwin=true ;; #(
Darwin* ) darwin=true ;; #(
MSYS* | MINGW* ) msys=true ;; #(
NONSTOP* ) nonstop=true ;;
esac
CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
# Determine the Java command to use to start the JVM.
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD=$JAVA_HOME/jre/sh/java
else
JAVACMD=$JAVA_HOME/bin/java
fi
if [ ! -x "$JAVACMD" ] ; then
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
else
JAVACMD=java
if ! command -v java >/dev/null 2>&1
then
die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
fi
# Increase the maximum file descriptors if we can.
if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
case $MAX_FD in #(
max*)
# In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC2039,SC3045
MAX_FD=$( ulimit -H -n ) ||
warn "Could not query maximum file descriptor limit"
esac
case $MAX_FD in #(
'' | soft) :;; #(
*)
# In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC2039,SC3045
ulimit -n "$MAX_FD" ||
warn "Could not set maximum file descriptor limit to $MAX_FD"
esac
fi
# Collect all arguments for the java command, stacking in reverse order:
# * args from the command line
# * the main class name
# * -classpath
# * -D...appname settings
# * --module-path (only if needed)
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables.
# For Cygwin or MSYS, switch paths to Windows format before running java
if "$cygwin" || "$msys" ; then
APP_HOME=$( cygpath --path --mixed "$APP_HOME" )
CLASSPATH=$( cygpath --path --mixed "$CLASSPATH" )
JAVACMD=$( cygpath --unix "$JAVACMD" )
# Now convert the arguments - kludge to limit ourselves to /bin/sh
for arg do
if
case $arg in #(
-*) false ;; # don't mess with options #(
/?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath
[ -e "$t" ] ;; #(
*) false ;;
esac
then
arg=$( cygpath --path --ignore --mixed "$arg" )
fi
# Roll the args list around exactly as many times as the number of
# args, so each arg winds up back in the position where it started, but
# possibly modified.
#
# NB: a `for` loop captures its iteration list before it begins, so
# changing the positional parameters here affects neither the number of
# iterations, nor the values presented in `arg`.
shift # remove old arg
set -- "$@" "$arg" # push replacement arg
done
fi
# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
# Collect all arguments for the java command:
# * DEFAULT_JVM_OPTS, JAVA_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
# and any embedded shellness will be escaped.
# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be
# treated as '${Hostname}' itself on the command line.
set -- \
"-Dorg.gradle.appname=$APP_BASE_NAME" \
-classpath "$CLASSPATH" \
org.gradle.wrapper.GradleWrapperMain \
"$@"
# Stop when "xargs" is not available.
if ! command -v xargs >/dev/null 2>&1
then
die "xargs is not available"
fi
# Use "xargs" to parse quoted args.
#
# With -n1 it outputs one arg per line, with the quotes and backslashes removed.
#
# In Bash we could simply go:
#
# readarray ARGS < <( xargs -n1 <<<"$var" ) &&
# set -- "${ARGS[@]}" "$@"
#
# but POSIX shell has neither arrays nor command substitution, so instead we
# post-process each arg (as a line of input to sed) to backslash-escape any
# character that might be a shell metacharacter, then use eval to reverse
# that process (while maintaining the separation between arguments), and wrap
# the whole thing up as a single "set" statement.
#
# This will of course break if any of these variables contains a newline or
# an unmatched quote.
#
eval "set -- $(
printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" |
xargs -n1 |
sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' |
tr '\n' ' '
)" '"$@"'
exec "$JAVACMD" "$@"

94
android/gradlew.bat vendored Normal file
View File

@@ -0,0 +1,94 @@
@rem
@rem Copyright 2015 the original author or authors.
@rem
@rem Licensed under the Apache License, Version 2.0 (the "License");
@rem you may not use this file except in compliance with the License.
@rem You may obtain a copy of the License at
@rem
@rem https://www.apache.org/licenses/LICENSE-2.0
@rem
@rem Unless required by applicable law or agreed to in writing, software
@rem distributed under the License is distributed on an "AS IS" BASIS,
@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@rem See the License for the specific language governing permissions and
@rem limitations under the License.
@rem
@rem SPDX-License-Identifier: Apache-2.0
@rem
@if "%DEBUG%"=="" @echo off
@rem ##########################################################################
@rem
@rem Gradle startup script for Windows
@rem
@rem ##########################################################################
@rem Set local scope for the variables with windows NT shell
if "%OS%"=="Windows_NT" setlocal
set DIRNAME=%~dp0
if "%DIRNAME%"=="" set DIRNAME=.
@rem This is normally unused
set APP_BASE_NAME=%~n0
set APP_HOME=%DIRNAME%
@rem Resolve any "." and ".." in APP_HOME to make it shorter.
for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi
@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m"
@rem Find java.exe
if defined JAVA_HOME goto findJavaFromJavaHome
set JAVA_EXE=java.exe
%JAVA_EXE% -version >NUL 2>&1
if %ERRORLEVEL% equ 0 goto execute
echo. 1>&2
echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2
echo. 1>&2
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
echo location of your Java installation. 1>&2
goto fail
:findJavaFromJavaHome
set JAVA_HOME=%JAVA_HOME:"=%
set JAVA_EXE=%JAVA_HOME%/bin/java.exe
if exist "%JAVA_EXE%" goto execute
echo. 1>&2
echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2
echo. 1>&2
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
echo location of your Java installation. 1>&2
goto fail
:execute
@rem Setup the command line
set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
@rem Execute Gradle
"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %*
:end
@rem End local scope for the variables with windows NT shell
if %ERRORLEVEL% equ 0 goto mainEnd
:fail
rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
rem the _cmd.exe /c_ return code!
set EXIT_CODE=%ERRORLEVEL%
if %EXIT_CODE% equ 0 set EXIT_CODE=1
if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE%
exit /b %EXIT_CODE%
:mainEnd
if "%OS%"=="Windows_NT" endlocal
:omega

19
android/settings.gradle.kts Normal file
View File

@@ -0,0 +1,19 @@
rootProject.name = "shade-kotlin"
pluginManagement {
repositories {
gradlePluginPortal()
mavenCentral()
}
}
dependencyResolutionManagement {
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
repositories {
mavenCentral()
google()
}
}
include(":shade-android")
project(":shade-android").projectDir = file("shade-android")

19
android/shade-android/README.md
View File

@@ -4,7 +4,15 @@ Kotlin implementation of the Shade E2EE protocol for Android apps. Byte-for-byte
## Status
**Milestone M-Cross 1 — initial scaffold.** The protocol implementation is being ported. Cross-platform test vectors in `test-vectors/` verify that Kotlin and TypeScript produce identical output for every step (identity gen → HKDF → X3DH → ratchet → fingerprint → wire format).
**M-Cross 1 ✅** — keys + HKDF + X3DH + fingerprint.
**M-Cross 2 ✅** — full ratchet step (encrypt + decrypt roundtrip) + wire 0x02 (RatchetMessage and PreKeyMessage with/without OTPK).
**M-Cross 3 ✅** — streams 0x11 (KDF labels with embedded NULs, deterministic chunk nonce/AAD, wire 0x11 encode/decode).
**M-Cross 4 ✅** — backup-key HKDF + AES-GCM, group sender-key step (`kdfChainKey` + Ed25519 sign over `aad ‖ ct`), storage HKDF (storageKey/fieldKey/rowNonce). Pending: scrypt master-key, argon2id swap, Android KeystoreStorage (sibling module).
Cross-platform test vectors in `/test-vectors/` are loaded by both the TS
and Kotlin test suites; any byte-divergence fails CI within 60 s. See
`ROADMAP-ANDROID.md` for the parity-checkpoint matrix and
`/docs/cross-platform.md` for how to add a new vector.
## Usage (target API)
@@ -40,13 +48,18 @@ Backed by Google Tink:
## Building
Requires Android SDK 35 and JDK 17.
Requires JDK 17. The module compiles as a pure-JVM Kotlin library so the
parity gate runs without an Android SDK install. The Android-specific
storage adapter (Keystore + EncryptedSharedPreferences) will land as a
sibling Gradle module in M-Cross 4.
```bash
./gradlew :shade-android:assembleDebug
cd android
./gradlew :shade-android:test
```
The Gradle wrapper downloads Gradle 8.10.2 on first run.
## Compatibility
The Kotlin implementation must produce byte-identical output to `@shade/core` for:

137
android/shade-android/ROADMAP-ANDROID.md Normal file
View File

@@ -0,0 +1,137 @@
# Shade Android — Roadmap & Parity Status
This document tracks the M-Cross milestones from `docs/V3.5.md` and the
status of every cross-platform parity sjekkpunkt. The Kotlin port must be
**byte-for-byte compatible** with the TypeScript implementation; this is
verified continuously by `test-vectors/*.json` consumed by both runners.
> **No "production" label** is allowed on Android until M-Cross 2 is green
> (ratchet + wire 0x02 + storage encryption) and M-Cross 3 is green
> (streams 0x11). See `docs/V3.5.md` §Akseptansekriterier.
## Milestones
### M-Cross 1 — Scaffold ✅
Foundation primitives. All passing in CI.
| Sjekkpunkt | Vector | TS test | Kotlin test |
|---|---|---|---|
| 1. KDF chain (root + chain ratchet) | `kdf-chain.json` | ✅ | ✅ |
| 2. HKDF labels | `hkdf.json` | ✅ | ✅ |
| 3. X3DH initial root key (3 + 4 DH outputs) | `x3dh.json` | ✅ | ✅ |
| 5. Fingerprint (60-digit safety number) | `fingerprint.json` | ✅ | ✅ |
### M-Cross 2 — Ratchet & Wire 0x02 ✅
Full ratchet step + binary envelope encoding for both message types.
| Sjekkpunkt | Vector | TS test | Kotlin test |
|---|---|---|---|
| 4. Ratchet step (encrypt deterministic) | `ratchet-step.json` | ✅ | ✅ |
| 4. Ratchet step (decrypt roundtrip) | `ratchet-step.json` | ✅ | ✅ |
| 6. Wire 0x02 RatchetMessage | `wire-format.json` | ✅ | ✅ |
| 6. Wire 0x02 PreKeyMessage (with OTPK) | `wire-format.json` | ✅ | ✅ |
| 6. Wire 0x02 PreKeyMessage (no OTPK, 0xFFFFFFFF marker) | `wire-format.json` | ✅ | ✅ |
The ratchet-step vector exercises every layer that contributes to a
ratchet message's wire bytes: `kdfRootKey` → `kdfChainKey` → 40-byte header
AAD → AES-256-GCM with deterministic nonce. Both implementations recompute
each layer and compare against the recorded hex. The decrypt half feeds
the recorded ciphertext back through `aesGcmDecrypt(messageKey, nonce, aad)`
and checks the plaintext recovers — proving the AEAD agrees in both
directions.
### M-Cross 3 — Streams 0x11 ✅
Multi-lane chunk encryption (`@shade/streams`) ported. KDF labels with
embedded NULs match TS byte-for-byte; deterministic
`(laneId, seq)`-derived nonces and the 29-byte chunk AAD agree across
runners; wire 0x11 encode/decode is roundtrip-verified.
| Sjekkpunkt | Vector | TS test | Kotlin test |
|---|---|---|---|
| `deriveStreamKey` (HKDF, info `shade-stream/v1\0master`) | `streams.json` | ✅ | ✅ |
| `deriveLaneKey` (HKDF, info `shade-stream/v1\0lane\0` ‖ u32_be laneId) — incl. laneId 0xFFFFFFFF | `streams.json` | ✅ | ✅ |
| `buildChunkNonce(laneId, seq)` — incl. seq = 2^64 - 2 | `streams.json` | ✅ | ✅ |
| `buildChunkAad(streamId, laneId, seq, isLast)` | `streams.json` | ✅ | ✅ |
| Chunk AES-256-GCM encrypt + decrypt (deterministic nonce + AAD) | `streams.json` | ✅ | ✅ |
| Wire 0x11 envelope encode + decode + type-tag inspector | `streams.json` | ✅ | ✅ |
Sequence numbers are unsigned u64 on the wire; the Kotlin port accepts
them as `Long` for the bit pattern (negative-signed-long for values past
2^63 - 1) — this matches the JVM `ByteBuffer.putLong` behavior and the
`java.lang.Long.parseUnsignedLong` JSON-decoder used in tests.
Pending end-to-end interop test (TS server → Kotlin client over an actual
socket) — not gated by vectors but recommended before flipping the
"production" label.
### M-Cross 4 — Backup, Group, Storage HKDF ✅ (cryptographic layer)
The cryptographic primitives that Kotlin needs to share with TS are now
covered. The remaining work is the high-level glue (BackupBlob JSON
schema, full SenderKey/GroupSession state-tracking, Android-Keystore
storage adapter, scrypt password-KDF) — all per-platform plumbing that
doesn't gate vector parity.
| Sjekkpunkt | Vector | TS test | Kotlin test |
|---|---|---|---|
| 7. Backup v1 HKDF (`info="ShadeBackupKey"`) | `backup.json` | ✅ | ✅ |
| 7. Backup v1 AES-GCM roundtrip (no AAD) | `backup.json` | ✅ | ✅ |
| Group sender header AAD (u16/u16/u32 length prefixes) | `group.json` | ✅ | ✅ |
| Group sender-key step: `kdfChainKey` + AES-GCM + Ed25519 sign(aad ‖ ct) | `group.json` | ✅ | ✅ |
| Storage HKDF: `storageKey` (`info="shade-storage-v1"`) | `storage-hkdf.json` | ✅ | ✅ |
| Storage HKDF: `fieldKey` (`info="shade-field-v1:{table}:{column}"`) | `storage-hkdf.json` | ✅ | ✅ |
| Storage HKDF: `rowNonce` (`info="shade-row-nonce-v1:{table}:{pk}"`) | `storage-hkdf.json` | ✅ | ✅ |
Pending sub-tasks (don't gate vector parity):
- **scrypt master-key derivation**: `test-vectors/storage-encryption.json`
pins `scrypt(N=1024, r=8, p=1, dkLen=32)` for unit-test config; Tink
doesn't ship scrypt. Add Bouncy Castle (`org.bouncycastle:bcprov-jdk18on`)
to the Kotlin module, wrap as `CryptoProvider.scrypt(...)`, then a follow-up
vector consumes the full storage-encryption.json end to end.
- **argon2id**: Both backup.ts and the threat-model docs flag HKDF as a
placeholder for a real password KDF. When `argon2id` is added to
`CryptoProvider`, both ports swap together and the backup vector gets
re-pinned.
- **Android KeystoreStorage adapter**: lives in a sibling Android Library
Gradle module that depends on this JVM module. Binds Tink to the Android
Keystore + EncryptedSharedPreferences.
## Build & Test
This module compiles as a **pure-JVM** Kotlin library (`kotlin("jvm")`)
so the parity gate can run without an Android SDK installation in CI.
The protocol code uses `tink:1.15.0` (JVM JAR), `java.nio.ByteBuffer`,
and `javax.crypto` — no `android.*` imports.
The Android-specific storage adapter (KeystoreStorage,
EncryptedSharedPreferences) will land as a sibling Gradle module
(`shade-android-keystore`) in M-Cross 4 and depend on this one.
```bash
# From repo root
cd android
./gradlew :shade-android:test
```
Requires JDK 17. The Gradle wrapper downloads Gradle 8.10.2 on first run.
## Compatibility contract
The Kotlin implementation must produce byte-identical output to the TS
reference for:
- KDF chain derivations (root key ratchet, chain key ratchet)
- X3DH shared secrets (3- and 4-DH variants)
- Ratchet message keys + AES-GCM ciphertext (given the same key/plaintext/AAD/nonce)
- Header AAD encoding (40 bytes: `dhPublicKey(32) || u32_be(prevCounter) || u32_be(counter)`)
- Fingerprints (12 × 5-digit groups)
- Binary wire format 0x02 (RatchetMessage + PreKeyMessage)
- Binary wire format 0x11 (StreamChunk) — M-Cross 3
- Storage encryption KDF chain — M-Cross 4
Each is covered by a vector file in `/test-vectors/`. Adding a new
sjekkpunkt: see `docs/cross-platform.md`.

63
android/shade-android/build.gradle.kts
View File

@@ -1,50 +1,51 @@
plugins {
id("com.android.library")
kotlin("android")
kotlin("jvm")
`java-library`
}
android {
namespace = "no.zyon.shade"
compileSdk = 35
// V3.5 — Cross-platform parity gate.
//
// This module compiles as a pure-JVM Kotlin library so CI can run the
// cross-platform vector tests without an Android SDK. The protocol code
// is JVM-safe (no `android.*` imports); only Tink + java.* are used.
//
// When KeystoreStorage and EncryptedSharedPreferences-backed adapters land
// (M-Cross 4 + V3.5 §Storage), they will live in a sibling Android Library
// module that depends on this one.
defaultConfig {
minSdk = 26
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
}
buildFeatures {
buildConfig = true
}
compileOptions {
java {
sourceCompatibility = JavaVersion.VERSION_17
targetCompatibility = JavaVersion.VERSION_17
}
}
kotlinOptions {
jvmTarget = "17"
kotlin {
compilerOptions {
jvmTarget.set(org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17)
}
}
dependencies {
// Google Tink for X25519, Ed25519, AES-GCM, HMAC, HKDF
implementation("com.google.crypto.tink:tink-android:1.15.0")
// Google Tink for X25519, Ed25519, AES-GCM, HMAC, HKDF (JVM build).
// The same `subtle.*` API as `tink-android` so the source compiles unchanged.
implementation("com.google.crypto.tink:tink:1.15.0")
// Android Keystore + EncryptedSharedPreferences
implementation("androidx.security:security-crypto:1.1.0-alpha06")
// JSON serialization for session state
// JSON serialization (session state + test-vector loader on JVM).
implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.7.3")
// Coroutines for async interop
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.9.0")
// Coroutines (StorageProvider uses `suspend` functions).
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.9.0")
// SQLite for session storage (optional; can also use EncryptedSharedPreferences only)
implementation("androidx.sqlite:sqlite:2.4.0")
// OkHttp for transport
implementation("com.squareup.okhttp3:okhttp:4.12.0")
// org.json — bundled with Android but not present on the JVM classpath.
implementation("org.json:json:20240303")
testImplementation("junit:junit:4.13.2")
testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.9.0")
}
tasks.withType<Test>().configureEach {
useJUnit()
testLogging {
events("passed", "failed", "skipped")
showStandardStreams = false
}
}

36
android/shade-android/src/main/kotlin/no/zyon/shade/backup/BackupKdf.kt Normal file
View File

@@ -0,0 +1,36 @@
package no.zyon.shade.backup
import no.zyon.shade.crypto.CryptoProvider
/**
* Backup format v1 — passphrase-derived AES-256-GCM blob.
* Mirror @shade/sdk/backup.ts.
*
* backupKey = HKDF(passphrase_utf8, salt_random_32, info="ShadeBackupKey", 32)
* blob = AES-256-GCM(backupKey, plaintext, no AAD)
*
* The stored on-disk form is `{ version, salt(b64), nonce(b64), ciphertext(b64) }`.
* This file ships only the cryptographic primitives — payload schema and JSON
* serialization live alongside the high-level SDK and don't need a Kotlin port
* for vector parity (each platform builds the BackupBlob in its native idiom).
*
* NOTE: HKDF is NOT a proper password KDF. The TS SDK acknowledges this and
* warns users to choose a high-entropy passphrase. When `argon2id` lands in
* `CryptoProvider`, both ports swap together. Until then, byte-parity for the
* HKDF + AEAD layer is what V3.5 §sjekkpunkt 8 gates.
*/
private val BACKUP_INFO: ByteArray = "ShadeBackupKey".toByteArray(Charsets.UTF_8)
const val BACKUP_KEY_BYTES = 32
const val BACKUP_VERSION = 1
fun deriveBackupKey(crypto: CryptoProvider, passphrase: String, salt: ByteArray): ByteArray {
require(passphrase.length >= 12) { "Passphrase must be at least 12 characters" }
require(salt.size >= 16) { "salt must be at least 16 bytes" }
return crypto.hkdf(
passphrase.toByteArray(Charsets.UTF_8),
salt,
BACKUP_INFO,
BACKUP_KEY_BYTES,
)
}

77
android/shade-android/src/main/kotlin/no/zyon/shade/group/SenderKey.kt Normal file
View File

@@ -0,0 +1,77 @@
package no.zyon.shade.group
import no.zyon.shade.crypto.CryptoProvider
import no.zyon.shade.protocol.kdfChainKey
import java.nio.ByteBuffer
/**
* Group sender-keys (Sesame). Mirror @shade/core/sender-keys.ts.
*
* Each sender maintains a chain key that ratchets forward with `kdfChainKey`
* — same primitive the Double Ratchet uses for its symmetric chain. Per-message
* AEAD AAD binds (groupId, senderAddress, iteration) so a captured ciphertext
* cannot be replayed under a different sender or group:
*
* aad = u16_be(groupIdLen) || groupId || u16_be(senderAddrLen) || senderAddr || u32_be(iteration)
*
* Each ciphertext is signed by the sender's Ed25519 key over `aad || ciphertext`,
* which is what receivers verify before advancing their chain.
*/
data class SenderKeyMessage(
val senderAddress: String,
val iteration: Int,
val ciphertext: ByteArray,
val nonce: ByteArray,
val signature: ByteArray,
)
fun encodeSenderHeader(groupId: String, senderAddress: String, iteration: Int): ByteArray {
val gBytes = groupId.toByteArray(Charsets.UTF_8)
val sBytes = senderAddress.toByteArray(Charsets.UTF_8)
require(gBytes.size <= 0xFFFF) { "groupId too long (>65535 UTF-8 bytes)" }
require(sBytes.size <= 0xFFFF) { "senderAddress too long (>65535 UTF-8 bytes)" }
val out = ByteArray(2 + gBytes.size + 2 + sBytes.size + 4)
val buf = ByteBuffer.wrap(out)
buf.putShort(gBytes.size.toShort())
buf.put(gBytes)
buf.putShort(sBytes.size.toShort())
buf.put(sBytes)
buf.putInt(iteration)
return out
}
/**
* Compute (newChainKey, messageKey, aad) for the next group message.
* Pure function; caller is responsible for state advancement and the AEAD/sign
* steps (which need access to the signing private key not exposed here).
*/
data class SenderStepResult(
val newChainKey: ByteArray,
val messageKey: ByteArray,
val aad: ByteArray,
)
fun senderKeyStep(
crypto: CryptoProvider,
chainKey: ByteArray,
groupId: String,
senderAddress: String,
iteration: Int,
): SenderStepResult {
val r = kdfChainKey(crypto, chainKey)
val aad = encodeSenderHeader(groupId, senderAddress, iteration)
return SenderStepResult(newChainKey = r.newChainKey, messageKey = r.messageKey, aad = aad)
}
/**
* Concatenate `aad || ciphertext` — the byte string the sender signs and the
* receiver verifies. Exposed as a helper so vector parity can pin both sides.
*/
fun senderSignedBytes(aad: ByteArray, ciphertext: ByteArray): ByteArray {
val out = ByteArray(aad.size + ciphertext.size)
aad.copyInto(out, 0)
ciphertext.copyInto(out, aad.size)
return out
}

145
android/shade-android/src/main/kotlin/no/zyon/shade/serialization/StreamChunkWire.kt Normal file
View File

@@ -0,0 +1,145 @@
package no.zyon.shade.serialization
import no.zyon.shade.streams.StreamConstants
import java.nio.ByteBuffer
/**
* Wire-decoded stream-chunk envelope (type 0x11).
*
* Mirror @shade/proto/wire.ts `StreamChunkWire`. The nonce is deterministic
* (derived from `(laneId, seq)` on both sides) but is also serialized over
* the wire for self-description and validated by the receiver.
*
* `seq` is unsigned-u64 on the wire; on the JVM we keep it as Long. The
* encode/decode helpers operate on the raw 8-byte big-endian representation,
* so values past Long.MAX_VALUE roundtrip via `Long.toULong()`.
*/
data class StreamChunkWire(
val streamId: ByteArray,
val laneId: Long,
val seq: Long,
val isLast: Boolean,
val nonce: ByteArray,
val aad: ByteArray,
val ciphertext: ByteArray,
) {
override fun equals(other: Any?): Boolean {
if (this === other) return true
if (other !is StreamChunkWire) return false
return streamId.contentEquals(other.streamId) &&
laneId == other.laneId &&
seq == other.seq &&
isLast == other.isLast &&
nonce.contentEquals(other.nonce) &&
aad.contentEquals(other.aad) &&
ciphertext.contentEquals(other.ciphertext)
}
override fun hashCode(): Int {
var result = streamId.contentHashCode()
result = 31 * result + laneId.hashCode()
result = 31 * result + seq.hashCode()
result = 31 * result + isLast.hashCode()
result = 31 * result + nonce.contentHashCode()
result = 31 * result + aad.contentHashCode()
result = 31 * result + ciphertext.contentHashCode()
return result
}
}
/** Stream-chunk wire codec. Mirror @shade/proto/wire.ts `encodeStreamChunk`/`decodeStreamChunk`. */
object StreamChunkWireFormat {
private const val VERSION: Byte = 0x02
const val TYPE_STREAM_CHUNK: Byte = 0x11
fun encodeStreamChunk(c: StreamChunkWire): ByteArray {
require(c.streamId.size == StreamConstants.STREAM_ID_BYTES) {
"streamId must be ${StreamConstants.STREAM_ID_BYTES} bytes"
}
require(c.nonce.size == StreamConstants.STREAM_NONCE_BYTES) {
"nonce must be ${StreamConstants.STREAM_NONCE_BYTES} bytes"
}
require(c.laneId in 0L..0xFFFFFFFFL) { "laneId out of u32 range: ${c.laneId}" }
// c.seq is unsigned-u64; negative signed longs encode as the high half
// of the u64 range. ByteBuffer.putLong writes the raw 8-byte pattern.
val headerSize =
1 + 1 +
StreamConstants.STREAM_ID_BYTES +
4 + 8 + 1 +
StreamConstants.STREAM_NONCE_BYTES +
4 + c.aad.size +
4
val out = ByteArray(headerSize + c.ciphertext.size)
val buf = ByteBuffer.wrap(out)
buf.put(VERSION)
buf.put(TYPE_STREAM_CHUNK)
buf.put(c.streamId)
buf.putInt(c.laneId.toInt())
buf.putLong(c.seq)
buf.put(if (c.isLast) 0x01.toByte() else 0x00.toByte())
buf.put(c.nonce)
buf.putInt(c.aad.size)
buf.put(c.aad)
buf.putInt(c.ciphertext.size)
buf.put(c.ciphertext)
return out
}
fun decodeStreamChunk(data: ByteArray): StreamChunkWire {
val minHeaderSize = 2 +
StreamConstants.STREAM_ID_BYTES +
4 + 8 + 1 +
StreamConstants.STREAM_NONCE_BYTES +
4 + 4
require(data.size >= minHeaderSize) {
"stream-chunk too short: ${data.size} < $minHeaderSize"
}
require(data[0] == VERSION) { "Unknown version: ${data[0]}" }
require(data[1] == TYPE_STREAM_CHUNK) { "Not a stream-chunk: type=${data[1]}" }
val buf = ByteBuffer.wrap(data)
buf.position(2)
val streamId = ByteArray(StreamConstants.STREAM_ID_BYTES)
buf.get(streamId)
val laneId = buf.int.toLong() and 0xFFFFFFFFL
val seq = buf.long
val isLast = buf.get() == 0x01.toByte()
val nonce = ByteArray(StreamConstants.STREAM_NONCE_BYTES)
buf.get(nonce)
val aadLen = buf.int
require(buf.position() + aadLen + 4 <= data.size) {
"stream-chunk truncated in aad/ctLen"
}
val aad = ByteArray(aadLen)
buf.get(aad)
val ctLen = buf.int
require(buf.position() + ctLen == data.size) {
"stream-chunk length mismatch: declared ${buf.position() + ctLen}, actual ${data.size}"
}
val ciphertext = ByteArray(ctLen)
buf.get(ciphertext)
return StreamChunkWire(streamId, laneId, seq, isLast, nonce, aad, ciphertext)
}
/** Inspect the type tag without full parsing. Mirror @shade/proto/wire.ts. */
enum class EnvelopeKind { PREKEY, RATCHET, STREAM_CHUNK, UNKNOWN }
fun inspectEnvelopeType(data: ByteArray): EnvelopeKind {
if (data.size < 2 || data[0] != VERSION) return EnvelopeKind.UNKNOWN
return when (data[1]) {
0x01.toByte() -> EnvelopeKind.PREKEY
0x02.toByte() -> EnvelopeKind.RATCHET
TYPE_STREAM_CHUNK -> EnvelopeKind.STREAM_CHUNK
else -> EnvelopeKind.UNKNOWN
}
}
}

48
android/shade-android/src/main/kotlin/no/zyon/shade/streams/ChunkAead.kt Normal file
View File

@@ -0,0 +1,48 @@
package no.zyon.shade.streams
import javax.crypto.Cipher
import javax.crypto.spec.GCMParameterSpec
import javax.crypto.spec.SecretKeySpec
/**
* AES-256-GCM with caller-supplied nonce. Mirror @shade/streams/aead.ts.
*
* Unlike `CryptoProvider.aesGcmEncrypt` (which generates a random nonce
* internally), streams require deterministic nonces derived from
* `(laneId, seq)`. Returns the ciphertext concatenated with the 16-byte
* authentication tag — same layout SubtleCrypto produces.
*/
const val AEAD_TAG_BYTES = 16
fun aesGcmEncryptWithNonce(
key: ByteArray,
nonce: ByteArray,
plaintext: ByteArray,
aad: ByteArray,
): ByteArray {
require(nonce.size == StreamConstants.STREAM_NONCE_BYTES) {
"AES-GCM nonce must be ${StreamConstants.STREAM_NONCE_BYTES} bytes"
}
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val spec = GCMParameterSpec(128, nonce)
cipher.init(Cipher.ENCRYPT_MODE, SecretKeySpec(key, "AES"), spec)
cipher.updateAAD(aad)
return cipher.doFinal(plaintext)
}
fun aesGcmDecryptWithNonce(
key: ByteArray,
nonce: ByteArray,
ciphertext: ByteArray,
aad: ByteArray,
): ByteArray {
require(nonce.size == StreamConstants.STREAM_NONCE_BYTES) {
"AES-GCM nonce must be ${StreamConstants.STREAM_NONCE_BYTES} bytes"
}
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val spec = GCMParameterSpec(128, nonce)
cipher.init(Cipher.DECRYPT_MODE, SecretKeySpec(key, "AES"), spec)
cipher.updateAAD(aad)
return cipher.doFinal(ciphertext)
}

50
android/shade-android/src/main/kotlin/no/zyon/shade/streams/ChunkNonce.kt Normal file
View File

@@ -0,0 +1,50 @@
package no.zyon.shade.streams
import java.nio.ByteBuffer
/**
* Deterministic AEAD nonce + AAD construction for stream chunks.
* Mirror @shade/streams/nonce.ts.
*
* nonce[0..4] = u32_be(laneId)
* nonce[4..12] = u64_be(seq)
*
* aad = streamId(16) || u32_be(laneId) || u64_be(seq) || u8(isLast)
*
* `seq` is unsigned-u64 on the wire. Kotlin's `Long` is signed; we accept it
* for the bit pattern (same as TS `BigInt` would write), so values past
* `Long.MAX_VALUE` arrive here as negative signed longs. `ByteBuffer.putLong`
* writes the raw 8 bytes regardless of sign — that's what we want.
*
* Use `java.lang.Long.parseUnsignedLong("…")` to decode JSON strings
* representing u64 values larger than 2^63 - 1.
*/
fun buildChunkNonce(laneId: Long, seq: Long): ByteArray {
require(laneId in 0L..0xFFFFFFFFL) { "laneId must fit in u32: $laneId" }
val out = ByteArray(StreamConstants.STREAM_NONCE_BYTES)
val buf = ByteBuffer.wrap(out)
buf.putInt(laneId.toInt())
buf.putLong(seq)
return out
}
fun buildChunkAad(
streamId: ByteArray,
laneId: Long,
seq: Long,
isLast: Boolean,
): ByteArray {
require(streamId.size == StreamConstants.STREAM_ID_BYTES) {
"streamId must be ${StreamConstants.STREAM_ID_BYTES} bytes"
}
require(laneId in 0L..0xFFFFFFFFL) { "laneId must fit in u32: $laneId" }
val out = ByteArray(StreamConstants.STREAM_ID_BYTES + 4 + 8 + 1)
streamId.copyInto(out, 0)
val buf = ByteBuffer.wrap(out, StreamConstants.STREAM_ID_BYTES, 4 + 8 + 1)
buf.putInt(laneId.toInt())
buf.putLong(seq)
out[out.size - 1] = if (isLast) 0x01 else 0x00
return out
}

BIN
android/shade-android/src/main/kotlin/no/zyon/shade/streams/StreamKdf.kt Normal file
View File

Binary file not shown.

377
android/shade-android/src/test/kotlin/no/zyon/shade/CrossPlatformVectorTest.kt
View File

@@ -1,18 +1,36 @@
package no.zyon.shade
import no.zyon.shade.backup.deriveBackupKey
import no.zyon.shade.crypto.TinkProvider
import no.zyon.shade.fingerprint.computeFingerprint
import no.zyon.shade.group.encodeSenderHeader
import no.zyon.shade.group.senderKeyStep
import no.zyon.shade.group.senderSignedBytes
import no.zyon.shade.protocol.deriveInitialRootKey
import no.zyon.shade.protocol.kdfChainKey
import no.zyon.shade.protocol.kdfRootKey
import no.zyon.shade.serialization.StreamChunkWire
import no.zyon.shade.serialization.StreamChunkWireFormat
import no.zyon.shade.serialization.WireFormat
import no.zyon.shade.streams.aesGcmDecryptWithNonce
import no.zyon.shade.streams.aesGcmEncryptWithNonce
import no.zyon.shade.streams.buildChunkAad
import no.zyon.shade.streams.buildChunkNonce
import no.zyon.shade.streams.deriveLaneKey
import no.zyon.shade.streams.deriveStreamKey
import no.zyon.shade.types.PreKeyMessage
import no.zyon.shade.types.RatchetMessage
import no.zyon.shade.types.ShadeEnvelope
import org.json.JSONArray
import org.json.JSONObject
import org.junit.Assert.assertEquals
import org.junit.Assert.assertTrue
import org.junit.Test
import java.io.File
import org.json.JSONObject
import org.json.JSONArray
import java.nio.ByteBuffer
import javax.crypto.Cipher
import javax.crypto.spec.GCMParameterSpec
import javax.crypto.spec.SecretKeySpec
/**
* Cross-platform test vectors. MUST match the TypeScript implementation
@@ -25,6 +43,7 @@ class CrossPlatformVectorTest {
private val crypto = TinkProvider()
private val vectorsDir = File("../../test-vectors")
private val expectedVersion = 2
private fun fromHex(str: String): ByteArray {
val bytes = ByteArray(str.length / 2)
@@ -39,10 +58,40 @@ class CrossPlatformVectorTest {
return bytes.joinToString("") { "%02x".format(it) }
}
private data class VectorFile(val version: Int, val vectors: JSONArray)
private fun loadVectors(name: String): JSONArray {
val file = File(vectorsDir, name)
val content = file.readText()
return JSONObject(content).getJSONArray("vectors")
val obj = JSONObject(content)
val version = obj.getInt("version")
assertEquals("Unexpected vector schema version in $name", expectedVersion, version)
return obj.getJSONArray("vectors")
}
private fun encodeRatchetHeader(
dhPublicKey: ByteArray,
previousCounter: Int,
counter: Int,
): ByteArray {
val buf = ByteBuffer.allocate(40)
buf.put(dhPublicKey)
buf.putInt(previousCounter)
buf.putInt(counter)
return buf.array()
}
private fun aesGcmEncryptDeterministic(
key: ByteArray,
nonce: ByteArray,
plaintext: ByteArray,
aad: ByteArray,
): ByteArray {
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val spec = GCMParameterSpec(128, nonce)
cipher.init(Cipher.ENCRYPT_MODE, SecretKeySpec(key, "AES"), spec)
cipher.updateAAD(aad)
return cipher.doFinal(plaintext)
}
@Test
@@ -106,9 +155,13 @@ class CrossPlatformVectorTest {
}
@Test
fun wireFormatVectorsMatch() {
fun wireFormatRatchetVectorsMatch() {
val vectors = loadVectors("wire-format.json")
val v = vectors.getJSONObject(0)
var found = false
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
if (v.optString("kind") != "ratchet") continue
found = true
val m = v.getJSONObject("message")
val msg = RatchetMessage(
@@ -127,10 +180,322 @@ class CrossPlatformVectorTest {
val encoded = WireFormat.encodeEnvelope(envelope)
assertEquals(v.getString("encoded"), hex(encoded))
// Roundtrip decode
val decoded = WireFormat.decodeEnvelope(encoded)
assertEquals(ShadeEnvelope.EnvelopeType.RATCHET, decoded.type)
val rm = decoded.content as RatchetMessage
assertEquals(msg.counter, rm.counter)
assertEquals(hex(msg.ciphertext), hex(rm.ciphertext))
}
assertTrue("No ratchet wire vectors found", found)
}
@Test
fun wireFormatPreKeyVectorsMatch() {
val vectors = loadVectors("wire-format.json")
var matched = 0
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
if (v.optString("kind") != "prekey") continue
matched++
val m = v.getJSONObject("message")
val inner = m.getJSONObject("inner")
val innerMsg = RatchetMessage(
dhPublicKey = fromHex(inner.getString("dhPublicKey")),
previousCounter = inner.getInt("previousCounter"),
counter = inner.getInt("counter"),
ciphertext = fromHex(inner.getString("ciphertext")),
nonce = fromHex(inner.getString("nonce")),
)
val preKeyId: Int? = if (m.isNull("preKeyId")) null else m.getInt("preKeyId")
val pre = PreKeyMessage(
registrationId = m.getInt("registrationId"),
preKeyId = preKeyId,
signedPreKeyId = m.getInt("signedPreKeyId"),
ephemeralKey = fromHex(m.getString("ephemeralKey")),
identityDHKey = fromHex(m.getString("identityDHKey")),
message = innerMsg,
)
val envelope = ShadeEnvelope(
type = ShadeEnvelope.EnvelopeType.PREKEY,
content = pre,
timestamp = 0,
senderAddress = "",
)
val encoded = WireFormat.encodeEnvelope(envelope)
assertEquals(v.getString("encoded"), hex(encoded))
val decoded = WireFormat.decodeEnvelope(encoded)
assertEquals(ShadeEnvelope.EnvelopeType.PREKEY, decoded.type)
val dm = decoded.content as PreKeyMessage
assertEquals(pre.registrationId, dm.registrationId)
assertEquals(pre.preKeyId, dm.preKeyId)
assertEquals(pre.signedPreKeyId, dm.signedPreKeyId)
assertEquals(hex(pre.ephemeralKey), hex(dm.ephemeralKey))
assertEquals(hex(innerMsg.ciphertext), hex(dm.message.ciphertext))
}
assertTrue("Expected at least 2 prekey vectors", matched >= 2)
}
private fun findVector(arr: JSONArray, prefix: String): JSONObject {
for (i in 0 until arr.length()) {
val o = arr.getJSONObject(i)
if (o.getString("description").startsWith(prefix)) return o
}
throw AssertionError("Vector with description prefix '$prefix' not found")
}
@Test
fun streamsVectorsMatch() {
val vectors = loadVectors("streams.json")
// 1. deriveStreamKey
val sk = findVector(vectors, "deriveStreamKey")
val streamSecret = fromHex(sk.getString("streamSecret"))
val streamId = fromHex(sk.getString("streamId"))
val streamKey = deriveStreamKey(crypto, streamSecret, streamId)
assertEquals(sk.getString("streamKey"), hex(streamKey))
// 2. deriveLaneKey
val lk = findVector(vectors, "deriveLaneKey")
val lkStreamKey = fromHex(lk.getString("streamKey"))
val lkStreamId = fromHex(lk.getString("streamId"))
val lanes = lk.getJSONArray("lanes")
for (i in 0 until lanes.length()) {
val lane = lanes.getJSONObject(i)
val laneId = lane.getLong("laneId")
val k = deriveLaneKey(crypto, lkStreamKey, lkStreamId, laneId)
assertEquals(lane.getString("laneKey"), hex(k))
}
// 3. buildChunkNonce
val nv = findVector(vectors, "buildChunkNonce")
val nonces = nv.getJSONArray("nonces")
for (i in 0 until nonces.length()) {
val n = nonces.getJSONObject(i)
val laneId = n.getLong("laneId")
val seq = java.lang.Long.parseUnsignedLong(n.getString("seq"))
val out = buildChunkNonce(laneId, seq)
assertEquals(n.getString("nonce"), hex(out))
}
// 4. buildChunkAad
val av = findVector(vectors, "buildChunkAad")
val avStreamId = fromHex(av.getString("streamId"))
val cases = av.getJSONArray("cases")
for (i in 0 until cases.length()) {
val c = cases.getJSONObject(i)
val laneId = c.getLong("laneId")
val seq = java.lang.Long.parseUnsignedLong(c.getString("seq"))
val isLast = c.getBoolean("isLast")
val out = buildChunkAad(avStreamId, laneId, seq, isLast)
assertEquals(c.getString("aad"), hex(out))
}
// 5. End-to-end chunk encrypt + decrypt
val ev = findVector(vectors, "End-to-end chunk encrypt")
val laneKey = fromHex(ev.getString("laneKey"))
val nonce = fromHex(ev.getString("nonce"))
val aad = fromHex(ev.getString("aad"))
val plaintext = fromHex(ev.getString("plaintext"))
val ct = aesGcmEncryptWithNonce(laneKey, nonce, plaintext, aad)
assertEquals(ev.getString("ciphertext"), hex(ct))
val pt = aesGcmDecryptWithNonce(laneKey, nonce, fromHex(ev.getString("ciphertext")), aad)
assertEquals(ev.getString("plaintext"), hex(pt))
// 6. Wire 0x11 envelope encode/decode
val wv = findVector(vectors, "Wire 0x11")
val wire = StreamChunkWire(
streamId = fromHex(wv.getString("streamId")),
laneId = wv.getLong("laneId"),
seq = java.lang.Long.parseUnsignedLong(wv.getString("seq")),
isLast = wv.getBoolean("isLast"),
nonce = fromHex(wv.getString("nonce")),
aad = fromHex(wv.getString("extraAad")),
ciphertext = fromHex(wv.getString("ciphertext")),
)
val encoded = StreamChunkWireFormat.encodeStreamChunk(wire)
assertEquals(wv.getString("encoded"), hex(encoded))
val decoded = StreamChunkWireFormat.decodeStreamChunk(encoded)
assertEquals(hex(wire.streamId), hex(decoded.streamId))
assertEquals(wire.laneId, decoded.laneId)
assertEquals(wire.seq, decoded.seq)
assertEquals(wire.isLast, decoded.isLast)
assertEquals(hex(wire.nonce), hex(decoded.nonce))
assertEquals(hex(wire.ciphertext), hex(decoded.ciphertext))
// 7. Envelope-type inspector
assertEquals(
StreamChunkWireFormat.EnvelopeKind.STREAM_CHUNK,
StreamChunkWireFormat.inspectEnvelopeType(encoded),
)
}
@Test
fun backupVectorsMatch() {
val vectors = loadVectors("backup.json")
val kv = findVector(vectors, "Backup v1: HKDF")
val backupKey = deriveBackupKey(crypto, kv.getString("passphrase"), fromHex(kv.getString("salt")))
assertEquals(kv.getString("backupKey"), hex(backupKey))
val ev = findVector(vectors, "Backup v1: AES-256-GCM")
val ct = aesGcmEncryptDeterministic(
fromHex(ev.getString("backupKey")),
fromHex(ev.getString("nonce")),
fromHex(ev.getString("plaintext")),
ByteArray(0),
)
assertEquals(ev.getString("ciphertext"), hex(ct))
val pt = crypto.aesGcmDecrypt(
fromHex(ev.getString("backupKey")),
fromHex(ev.getString("ciphertext")),
fromHex(ev.getString("nonce")),
null,
)
assertEquals(ev.getString("plaintext"), hex(pt))
}
@Test
fun groupSenderKeyVectorsMatch() {
val vectors = loadVectors("group.json")
// 1. Header AAD
val hv = findVector(vectors, "Sender header AAD")
val aad = encodeSenderHeader(
hv.getString("groupId"),
hv.getString("senderAddress"),
hv.getInt("iteration"),
)
assertEquals(hv.getString("aad"), hex(aad))
// 2. Sender-key step
val sv = findVector(vectors, "Sender-key step")
val step = senderKeyStep(
crypto,
fromHex(sv.getString("chainKey")),
sv.getString("groupId"),
sv.getString("senderAddress"),
sv.getInt("iteration"),
)
assertEquals(sv.getString("newChainKey"), hex(step.newChainKey))
assertEquals(sv.getString("messageKey"), hex(step.messageKey))
assertEquals(sv.getString("aad"), hex(step.aad))
val ct = aesGcmEncryptDeterministic(
step.messageKey,
fromHex(sv.getString("nonce")),
fromHex(sv.getString("plaintext")),
step.aad,
)
assertEquals(sv.getString("ciphertext"), hex(ct))
// 3. Ed25519 verify on the recorded signature
val signed = senderSignedBytes(step.aad, ct)
val ok = crypto.verify(
fromHex(sv.getString("signingPublicKey")),
signed,
fromHex(sv.getString("signature")),
)
assertTrue("Sender-key signature verification failed", ok)
// 4. Decrypt roundtrip
val pt = crypto.aesGcmDecrypt(
step.messageKey,
fromHex(sv.getString("ciphertext")),
fromHex(sv.getString("nonce")),
step.aad,
)
assertEquals(sv.getString("plaintext"), hex(pt))
}
@Test
fun storageHkdfVectorsMatch() {
val vectors = loadVectors("storage-hkdf.json")
val sv = findVector(vectors, "Storage HKDF: storageKey")
val storageKey = crypto.hkdf(
fromHex(sv.getString("masterKey")),
ByteArray(0),
"shade-storage-v1".toByteArray(Charsets.UTF_8),
32,
)
assertEquals(sv.getString("storageKey"), hex(storageKey))
val fv = findVector(vectors, "Storage HKDF: fieldKey")
val fStorageKey = fromHex(fv.getString("storageKey"))
val fields = fv.getJSONArray("fields")
for (i in 0 until fields.length()) {
val f = fields.getJSONObject(i)
val info = "shade-field-v1:${f.getString("table")}:${f.getString("column")}"
.toByteArray(Charsets.UTF_8)
val k = crypto.hkdf(fStorageKey, ByteArray(0), info, 32)
assertEquals(f.getString("fieldKey"), hex(k))
}
val nv = findVector(vectors, "Storage HKDF: rowNonce")
val rowKey = fromHex(nv.getString("rowKey"))
val nonces = nv.getJSONArray("nonces")
for (i in 0 until nonces.length()) {
val n = nonces.getJSONObject(i)
val info = "shade-row-nonce-v1:${n.getString("table")}:${n.getString("pk")}"
.toByteArray(Charsets.UTF_8)
val out = crypto.hkdf(rowKey, ByteArray(0), info, 12)
assertEquals(n.getString("nonce"), hex(out))
}
}
@Test
fun ratchetStepRoundtripMatches() {
val vectors = loadVectors("ratchet-step.json")
assertTrue("ratchet-step vectors expected", vectors.length() > 0)
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
val inputs = v.getJSONObject("inputs")
val derived = v.getJSONObject("derived")
val rootKey = fromHex(inputs.getString("rootKey"))
val dhSendPriv = fromHex(inputs.getString("dhSendPrivateKey"))
val dhSendPub = fromHex(inputs.getString("dhSendPublicKey"))
val dhRemotePub = fromHex(inputs.getString("dhRemotePublicKey"))
val plaintext = fromHex(inputs.getString("plaintext"))
val nonce = fromHex(inputs.getString("nonce"))
val previousCounter = inputs.getInt("previousCounter")
val counter = inputs.getInt("counter")
// 1. DH
val dhOutput = crypto.x25519(dhSendPriv, dhRemotePub)
assertEquals(derived.getString("dhOutput"), hex(dhOutput))
// 2. kdfRootKey
val root = kdfRootKey(crypto, rootKey, dhOutput)
assertEquals(derived.getString("newRootKey"), hex(root.newRootKey))
assertEquals(derived.getString("chainKey"), hex(root.chainKey))
// 3. kdfChainKey
val chain = kdfChainKey(crypto, root.chainKey)
assertEquals(derived.getString("newChainKey"), hex(chain.newChainKey))
assertEquals(derived.getString("messageKey"), hex(chain.messageKey))
// 4. Header AAD
val aad = encodeRatchetHeader(dhSendPub, previousCounter, counter)
assertEquals(derived.getString("aad"), hex(aad))
// 5. AES-GCM encrypt with fixed nonce
val ciphertext = aesGcmEncryptDeterministic(chain.messageKey, nonce, plaintext, aad)
assertEquals(v.getString("ciphertext"), hex(ciphertext))
// 6. Roundtrip decrypt
val recovered = crypto.aesGcmDecrypt(
chain.messageKey,
fromHex(v.getString("ciphertext")),
nonce,
aad,
)
assertEquals(inputs.getString("plaintext"), hex(recovered))
}
}
}

206
bun.lock
View File

@@ -17,14 +17,16 @@
},
"packages/shade-cli": {
"name": "@shade/cli",
"version": "0.3.0",
"version": "0.4.0",
"bin": {
"shade": "src/cli.ts",
},
"dependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/keychain": "workspace:*",
"@shade/sdk": "workspace:*",
"@shade/storage-encrypted": "workspace:*",
"@shade/storage-sqlite": "workspace:*",
"@shade/transport": "workspace:*",
},
@@ -34,7 +36,10 @@
},
"packages/shade-core": {
"name": "@shade/core",
"version": "0.3.0",
"version": "0.4.0",
"dependencies": {
"@shade/observability": "workspace:*",
},
"devDependencies": {
"@shade/proto": "workspace:*",
},
@@ -44,16 +49,17 @@
},
"packages/shade-crypto-web": {
"name": "@shade/crypto-web",
"version": "0.3.0",
"version": "0.4.0",
"dependencies": {
"@noble/curves": "^2.0.1",
"@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*",
"@shade/streams": "workspace:*",
},
},
"packages/shade-dashboard": {
"name": "@shade/dashboard",
"version": "0.3.0",
"version": "0.4.0",
"dependencies": {
"@shade/widgets": "workspace:*",
"react": "^19.0.0",
@@ -68,10 +74,11 @@
},
"packages/shade-files": {
"name": "@shade/files",
"version": "0.3.0",
"version": "0.4.0",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/observability": "workspace:*",
"@shade/proto": "workspace:*",
"@shade/sdk": "workspace:*",
"@shade/streams": "workspace:*",
@@ -92,9 +99,74 @@
"react",
],
},
"packages/shade-inbox": {
"name": "@shade/inbox",
"version": "0.4.0",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/proto": "workspace:*",
"@shade/server": "workspace:*",
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
"@shade/inbox-server": "workspace:*",
},
},
"packages/shade-inbox-server": {
"name": "@shade/inbox-server",
"version": "0.4.0",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/observability": "workspace:*",
"@shade/server": "workspace:*",
"hono": "^4.12.12",
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
},
"optionalDependencies": {
"@shade/crypto-web": "workspace:*",
"@shade/storage-postgres": "workspace:*",
"@shade/storage-sqlite": "workspace:*",
},
},
"packages/shade-key-transparency": {
"name": "@shade/key-transparency",
"version": "0.4.0",
"dependencies": {
"@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*",
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
"fast-check": "^3.22.0",
},
},
"packages/shade-keychain": {
"name": "@shade/keychain",
"version": "0.4.0",
},
"packages/shade-observability": {
"name": "@shade/observability",
"version": "0.1.0",
"dependencies": {
"@noble/hashes": "^2.0.1",
},
"devDependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/server": "workspace:*",
},
"peerDependencies": {
"@opentelemetry/api": ">=1.7.0",
},
"optionalPeers": [
"@opentelemetry/api",
],
},
"packages/shade-observer": {
"name": "@shade/observer",
"version": "0.3.0",
"version": "0.4.0",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/server": "workspace:*",
@@ -106,18 +178,33 @@
},
"packages/shade-proto": {
"name": "@shade/proto",
"version": "0.3.0",
"version": "0.4.0",
"dependencies": {
"@shade/core": "workspace:*",
},
},
"packages/shade-recovery": {
"name": "@shade/recovery",
"version": "0.4.0",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/sdk": "workspace:*",
},
"devDependencies": {
"@shade/server": "workspace:*",
"fast-check": "^3.22.0",
},
},
"packages/shade-sdk": {
"name": "@shade/sdk",
"version": "0.3.0",
"version": "0.4.0",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/files": "workspace:*",
"@shade/key-transparency": "workspace:*",
"@shade/observability": "workspace:*",
"@shade/observer": "workspace:*",
"@shade/proto": "workspace:*",
"@shade/server": "workspace:*",
@@ -126,12 +213,24 @@
"@shade/transfer": "workspace:*",
"@shade/transport": "workspace:*",
},
"devDependencies": {
"@shade/transport-webrtc": "workspace:*",
},
"peerDependencies": {
"@shade/transport-webrtc": "workspace:*",
},
"optionalPeers": [
"@shade/transport-webrtc",
],
},
"packages/shade-server": {
"name": "@shade/server",
"version": "0.3.0",
"version": "0.4.0",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/inbox-server": "workspace:*",
"@shade/key-transparency": "workspace:*",
"@shade/observability": "workspace:*",
"hono": "^4.12.12",
},
"devDependencies": {
@@ -144,11 +243,31 @@
"@shade/storage-sqlite": "workspace:*",
},
},
"packages/shade-storage-encrypted": {
"name": "@shade/storage-encrypted",
"version": "0.4.0",
"dependencies": {
"@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/storage-postgres": "workspace:*",
"@shade/storage-sqlite": "workspace:*",
"postgres": "^3.4.9",
},
"peerDependencies": {
"@shade/keychain": "workspace:*",
},
"optionalPeers": [
"@shade/keychain",
],
},
"packages/shade-storage-postgres": {
"name": "@shade/storage-postgres",
"version": "0.3.0",
"version": "0.4.0",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/inbox-server": "workspace:*",
"@shade/key-transparency": "workspace:*",
"@shade/server": "workspace:*",
"drizzle-orm": "^0.45.2",
"postgres": "^3.4.9",
@@ -159,29 +278,33 @@
},
"packages/shade-storage-sqlite": {
"name": "@shade/storage-sqlite",
"version": "0.3.0",
"version": "0.4.0",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/inbox-server": "workspace:*",
"@shade/server": "workspace:*",
},
},
"packages/shade-streams": {
"name": "@shade/streams",
"version": "0.3.0",
"version": "0.4.0",
"dependencies": {
"@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/proto": "workspace:*",
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
},
},
"packages/shade-transfer": {
"name": "@shade/transfer",
"version": "0.3.0",
"version": "0.4.0",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/observability": "workspace:*",
"@shade/proto": "workspace:*",
"@shade/streams": "workspace:*",
},
@@ -194,18 +317,51 @@
},
"packages/shade-transport": {
"name": "@shade/transport",
"version": "0.3.0",
"version": "0.4.0",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/key-transparency": "workspace:*",
"@shade/proto": "workspace:*",
"@shade/server": "workspace:*",
},
},
"packages/shade-transport-bridge": {
"name": "@shade/transport-bridge",
"version": "0.1.0",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/server": "workspace:*",
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
"@shade/inbox-server": "workspace:*",
"hono": "^4.12.12",
},
"optionalDependencies": {
"@shade/inbox-server": "workspace:*",
},
"peerDependencies": {
"hono": "^4",
},
"optionalPeers": [
"hono",
],
},
"packages/shade-transport-webrtc": {
"name": "@shade/transport-webrtc",
"version": "0.4.0",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/streams": "workspace:*",
"@shade/transfer": "workspace:*",
},
},
"packages/shade-widgets": {
"name": "@shade/widgets",
"version": "0.3.0",
"version": "0.4.0",
"dependencies": {
"@shade/recovery": "workspace:*",
"@shade/sdk": "workspace:*",
"@shade/streams": "workspace:*",
"@shade/transfer": "workspace:*",
@@ -390,14 +546,28 @@
"@shade/files": ["@shade/files@workspace:packages/shade-files"],
"@shade/inbox": ["@shade/inbox@workspace:packages/shade-inbox"],
"@shade/inbox-server": ["@shade/inbox-server@workspace:packages/shade-inbox-server"],
"@shade/key-transparency": ["@shade/key-transparency@workspace:packages/shade-key-transparency"],
"@shade/keychain": ["@shade/keychain@workspace:packages/shade-keychain"],
"@shade/observability": ["@shade/observability@workspace:packages/shade-observability"],
"@shade/observer": ["@shade/observer@workspace:packages/shade-observer"],
"@shade/proto": ["@shade/proto@workspace:packages/shade-proto"],
"@shade/recovery": ["@shade/recovery@workspace:packages/shade-recovery"],
"@shade/sdk": ["@shade/sdk@workspace:packages/shade-sdk"],
"@shade/server": ["@shade/server@workspace:packages/shade-server"],
"@shade/storage-encrypted": ["@shade/storage-encrypted@workspace:packages/shade-storage-encrypted"],
"@shade/storage-postgres": ["@shade/storage-postgres@workspace:packages/shade-storage-postgres"],
"@shade/storage-sqlite": ["@shade/storage-sqlite@workspace:packages/shade-storage-sqlite"],
@@ -408,6 +578,10 @@
"@shade/transport": ["@shade/transport@workspace:packages/shade-transport"],
"@shade/transport-bridge": ["@shade/transport-bridge@workspace:packages/shade-transport-bridge"],
"@shade/transport-webrtc": ["@shade/transport-webrtc@workspace:packages/shade-transport-webrtc"],
"@shade/widgets": ["@shade/widgets@workspace:packages/shade-widgets"],
"@types/babel__core": ["@types/babel__core@7.20.5", "", { "dependencies": { "@babel/parser": "^7.20.7", "@babel/types": "^7.20.7", "@types/babel__generator": "*", "@types/babel__template": "*", "@types/babel__traverse": "*" } }, "sha512-qoQprZvz5wQFJwMDqeseRXWv3rqMvhgpbXFfVyWhbx9X47POIA6i/+dXefEmZKoAgOaTdaIgNSMqMIU61yRyzA=="],

5
docs/DEPLOYMENT.md
View File

@@ -73,15 +73,20 @@ Tables will be created automatically with the `shade_server_*` prefix, so they c
| `PORT` | `3900` | HTTP port |
| `SHADE_PREKEY_DB_PATH` | `/data/shade-prekeys.db` | SQLite file location |
| `SHADE_PREKEY_PG_URL` | unset | Postgres URL (overrides SQLite) |
| `SHADE_INBOX_DB_PATH` | unset (memory) | SQLite file for the V3.6 inbox relay |
| `SHADE_INBOX_PG_URL` | falls back to `SHADE_PREKEY_PG_URL` | Postgres URL for the inbox relay |
| `SHADE_INBOX_PRUNE_INTERVAL_MINUTES` | `5` | How often expired inbox blobs are dropped |
| `SHADE_OBSERVER_TOKEN` | unset | Enables dashboard at `/shade-observer/dashboard/`. Min 16 chars. |
| `SHADE_STALE_DAYS` | `30` | Purge identities with no activity in N days |
| `SHADE_CLEANUP_INTERVAL_HOURS` | `24` | Cleanup cycle interval |
| `SHADE_LOG_LEVEL` | `info` | `debug` / `info` / `warn` / `error` |
| `SHADE_OTEL_ENABLED` | unset | Set to `1`/`true` to enable OpenTelemetry tracing on `withTracer()`-configured deployments. See [`observability.md`](./observability.md). |
## Health and observability
- **Health:** `GET /health` — returns `{"status":"ok"}` when the storage backend is reachable. Docker's HEALTHCHECK uses this.
- **Metrics:** `GET /metrics` — Prometheus format with counters, histograms, and gauges for all routes.
- **Tracing:** Optional OpenTelemetry spans via `@shade/observability`. Off by default; flip `SHADE_OTEL_ENABLED=1` to activate. PII-safe span attributes are documented in [`observability.md`](./observability.md).
- **OpenAPI:** `GET /openapi.yaml` — machine-readable API contract for any language.
- **Redoc viewer:** `GET /docs` — human-readable API reference.
- **Dashboard:** `GET /shade-observer/dashboard/` — live activity viewer (requires token).

179
docs/PRODUCTION-CHECKLIST.md Normal file
View File

@@ -0,0 +1,179 @@
# Shade Production Checklist
A flat punch-list for taking a Shade prekey server from "it boots" to
"production-ready". Every item below is a hard gate — if you can't tick it,
don't ship.
The deeper "why" behind each item lives in `THREAT-MODEL.md`,
`SECURITY.md`, and `docs/DEPLOYMENT.md`. This file is the operator's
checklist.
> Scope: a single Shade prekey container (`@shade/server`) plus any
> consumer apps that talk to it. For E2EE file transfer hardening
> (max-size, retention, quotas), see the **Hardening** and **Retention**
> sections of `docs/streams.md`.
---
## 1. TLS termination
- [ ] Public traffic is **TLS 1.2+ only** — Shade itself speaks plain HTTP
and assumes a reverse proxy (Caddy, Traefik, nginx, Dokploy's
built-in proxy) terminates TLS in front of it.
- [ ] HSTS is on (`Strict-Transport-Security: max-age=15552000`).
- [ ] The proxy is configured to pass the original `Host` header through
so signed payloads bound to the canonical address don't trip the
replay-window check on a mismatch.
- [ ] Internal traffic between consumer apps and the prekey container
runs on a private network (Docker bridge / VPC); the prekey port
is **not** exposed to the public internet without TLS in front.
> **Why:** identity signatures and observer bearer tokens travel in
> request bodies / headers. Without TLS, a network attacker can read
> the observer token and replay it for the full validity window, and
> can read the metadata (who registers, who fetches whose bundle).
> See `THREAT-MODEL.md § 1` (network attacker).
## 2. Backups
- [ ] **SQLite:** scheduled `sqlite3 /data/shade-prekeys.db ".backup ..."`
at least daily. The `.db` file plus `-wal` and `-shm` together is
the recovery unit; never copy the bare `.db` while the container
is running without using the online backup API.
- [ ] **Postgres:** `pg_dump` (or your provider's snapshot) at least
daily; verify a restore at least once per quarter.
- [ ] Backups are stored on different infrastructure than the primary
volume (different host / region / provider).
- [ ] Backups are encrypted at rest (your storage provider's
server-side encryption, age, or restic with a passphrase).
- [ ] **Restore drill:** at least once before going live, restore the
backup into a fresh volume and confirm `/health` is green and a
registered identity is still resolvable.
> **Why:** prekey records contain identity public keys and one-time
> prekeys. Losing them means new sessions can't be established to those
> identities until each user re-registers. Existing sessions keep
> ratcheting on the device-side state.
## 3. Observer token rotation
- [ ] `SHADE_OBSERVER_TOKEN` is set to **≥ 16 chars** of high-entropy
random data (e.g. `openssl rand -hex 32`). The server logs a
warning and disables the observer if the token is shorter.
- [ ] The token is held in your secret manager (Dokploy secret, GitHub
Actions secret, Vault, 1Password CLI), **never** committed to a
compose file or `.env` checked into git.
- [ ] The token is rotated on a schedule (recommended: every 90 days)
and immediately if it has been shared with anyone who no longer
needs access.
- [ ] If you expose the dashboard publicly, you also gate it behind
basic-auth at the proxy layer — bearer tokens are not
revocation-friendly on their own.
> **Why:** the observer dashboard exposes metadata about every active
> identity, registration timestamp, and recent activity. Anyone with
> the token can scrape the entire prekey directory.
## 4. SQLite vs PostgreSQL
Pick one and stick to it.
- [ ] **SQLite** is the default. Use it when **one** Shade container is
enough, you can tolerate downtime during backup snapshots, and
your write rate is below ~500 req/s. Path: `SHADE_PREKEY_DB_PATH`,
default `/data/shade-prekeys.db`.
- [ ] **PostgreSQL** is for multi-replica deployments, shared
infrastructure, or when you already operate a managed Postgres
and want one fewer thing to back up. Path: `SHADE_PREKEY_PG_URL`.
Tables are auto-created with `shade_server_*` prefix.
- [ ] Whichever you pick, the database lives behind TLS for the
connection (`sslmode=require` for Postgres) and on storage that
is itself encrypted (LUKS, EBS encryption, managed-DB encryption).
- [ ] You do **not** mix them in the same deployment. Setting
`SHADE_PREKEY_PG_URL` overrides SQLite silently — pick one in
`compose.yml` and document which.
> **Why:** Shade does **not** encrypt the database itself (V3.2 will).
> Disk-level / volume-level encryption is the operator's responsibility
> until at-rest encryption ships.
## 5. Log level and structured logs
- [ ] `SHADE_LOG_LEVEL` is set to `info` (production) or `warn`
(high-traffic). Avoid `debug` in prod — it logs request bodies
including signed payloads.
- [ ] Logs are shipped to a retention-bounded sink (Loki, CloudWatch,
Datadog) with **redaction of `Authorization` headers and signed
bodies** if your sink doesn't already strip them.
- [ ] You alert on `error`-level logs and on the absence of cleanup
cycles (a stuck cleanup loop = unbounded DB growth).
> **Why:** at `debug` level the server logs signature material. While
> Ed25519 signatures are not secrets per se, leaking them widens the
> replay-window blast radius and reveals timing patterns.
## 6. Stale-identity cleanup parameters
- [ ] `SHADE_STALE_DAYS` is set deliberately for your product. The
default (30 days) is right for "active chat app"; "occasional
use" apps should bump to 90+ to avoid surprise re-registration.
- [ ] `SHADE_CLEANUP_INTERVAL_HOURS` is left at 24 unless you have a
specific reason — running cleanup more often does not free more
space, and running it less often risks one cycle missing a day.
- [ ] You watch the `shade_cleanup_purged_total` metric (Prometheus) and
alert on a sudden 10× spike — that often signals a bug or a
deployment that broke client-side activity timestamps.
> **Why:** stale cleanup is the only thing keeping the prekey directory
> from growing forever. A misconfigured `SHADE_STALE_DAYS = 0` would
> nuke every identity on every cycle. Bound the value at ≥ 1 in your
> deployment config.
## 7. Secret rotation
- [ ] Identity signing keys: each consumer rotates via the documented
identity-rotation flow (7-day grace period for old sessions).
Operators do **not** touch identity keys directly.
- [ ] Observer token: see § 3.
- [ ] Database credentials (Postgres only): rotate per your standard
cadence, with the connection string supplied through the secret
manager.
- [ ] No long-lived API keys or service tokens are stored in the
container image or volume.
## 8. Rate-limit and body-size caps
- [ ] You have not lowered the built-in rate limits below the defaults
(per-IP register/bundle and per-identity replenish/delete).
- [ ] You have not raised the 64 KiB POST body limit. Prekey bundles
fit comfortably; raising the limit only enables abuse.
- [ ] Your reverse proxy enforces an additional connection / request-
rate limit at the edge (Caddy `rate_limit`, Cloudflare, etc.)
so a single noisy IP can't even reach Shade's per-route limits.
## 9. Health checks and metrics scrape
- [ ] Container has a Docker `HEALTHCHECK` (the official image already
ships one against `/health`).
- [ ] `/metrics` is scraped by Prometheus / OpenTelemetry and
retained ≥ 30 days.
- [ ] Alerts are wired for: `/health` failing for > 2 min, request
latency p99 > 1 s, error rate > 1 %, cleanup cycles missing for
> 25 h.
## 10. OpenAPI contract drift
- [ ] CI runs the OpenAPI lint (`bun test packages/shade-server/tests/openapi-lint.test.ts`)
on every PR — the spec must remain valid OpenAPI 3.1 with no
dangling `$ref`s.
- [ ] Generated clients (Python, Go, Kotlin) are regenerated from the
shipped spec on each release; mismatches between server and
client are caught at integration test time, not production.
---
## Pre-flight summary
If you can answer "yes" to every box above, ship it. If you can't,
write down which box and why before you do — that note belongs in your
runbook so the next operator inherits the gap, not the surprise.

116
docs/ROADMAP.md Normal file
View File

@@ -0,0 +1,116 @@
# Shade Roadmap — V3.1 → V5.0
Indeks over versjonsplanene fra V3.1-grunnsteinen via **Shade 4.0 GA** og
videre til **Shade 5.0** (Voice & Video).
- **V4.0 GA** ✅ — alt fra V2.1 / V2.2 / V2.3 og bonus-tracket (sosial
recovery, P2P WebRTC, Pub/Sub, Key Transparency) er merget, testet,
dokumentert og pakket for ekstern review. Wire-formatet er låst.
- **V5.0** = den dedikerte sanntids-releasen. *Alt* VOIP og videostreaming
ligger her — implementert oppå den frosne 4.0-stacken.
Alle V3.x-planer ligger nå under [`docs/archive/`](./archive/) med
`Status: Done`. Aktive planer: [`V5.0.md`](./V5.0.md).
---
## Faser
### Fase 1 — Documentation & Hardening Foundation ✅
| Plan | Tittel | Effort | Status |
|------|--------|--------|--------|
| [V3.1](./archive/V3.1.md) | Documentation & Hardening Foundation | S | **Done** |
### Fase 2 — Sikkerhetsmodning ✅
| Plan | Tittel | Effort | Status |
|------|--------|--------|--------|
| [V3.2](./archive/V3.2.md) | At-Rest Storage Encryption | L | **Done** |
| [V3.3](./archive/V3.3.md) | Fingerprint Gates & Trust UX | M | **Done** |
| [V3.4](./archive/V3.4.md) | Observability v2 (OpenTelemetry) | M | **Done** |
| [V3.5](./archive/V3.5.md) | Android Parity & Cross-Platform CI | XL | **Done** |
### Fase 3 — Plattformutvidelse ✅
| Plan | Tittel | Effort | Status |
|------|--------|--------|--------|
| [V3.6](./archive/V3.6.md) | Async Store-and-Forward (Inbox) | L | **Done** |
| [V3.7](./archive/V3.7.md) | Transport Bridge (SSE / long-poll) | M | **Done** |
| [V3.8](./archive/V3.8.md) | Web Workers Crypto | M-L | **Done** |
| [V3.9](./archive/V3.9.md) | Rich File Metadata & Previews | M | **Done** |
### Fase 4 — Tillit og P2P-transport ✅
| Plan | Tittel | Effort | Status |
|------|--------|--------|--------|
| [V3.10](./archive/V3.10.md) | Social Key Recovery | L | **Done** |
| [V3.11](./archive/V3.11.md) | WebRTC P2P Transport | XL | **Done** |
| [V3.12](./archive/V3.12.md) | Key Transparency | XXL | **Done** |
### Fase 5 — General Availability ✅
| Plan | Tittel | Effort | Status |
|------|--------|--------|--------|
| [V4.0](./archive/V4.0.md) | External Audit, Consolidation, GA | M | **Done** |
### Fase 6 — Sanntid (post-GA)
| Plan | Tittel | Effort | Avhenger av |
|------|--------|--------|-------------|
| [V5.0](./V5.0.md) | Voice & Video | XXL | V4.0 GA + V3.11 |
---
## Effort-nøkkel
| Symbol | Tid |
|--------|-----|
| **S** | 1–2 uker |
| **M** | 2–4 uker |
| **L** | 4–8 uker |
| **XL** | 2–4 måneder |
| **XXL** | 4+ måneder / multi-quarter |
---
## Avhengighetsgraf
```text
V3.1 ────┬──► V3.2 ──┐
├──► V3.3 ──┼──► V3.10 ──┐
├──► V3.4 ──┘ │
├──► V3.5 ───────────────┼──► V3.12 ──┐
├──► V3.6 ──► V3.7 ──► V3.11 ─────────┤
├──► V3.8 ├──► V4.0 GA ──► V5.0 (Voice & Video)
└──► V3.9 ─────────────────────────────┘
```
---
## Status-konvensjon
Hver plan har et `Status:`-felt øverst. Lov verdier:
- `Idea` — ikke startet, design fortsatt åpent.
- `Design` — designnotat under arbeid eller approved.
- `IMP` — implementasjon pågår.
- `Done` — merget i main, dekket av tester.
Når en plan blir `Done`, flytt fila til `docs/archive/` og oppdater denne tabellen.
---
## Versjonering
- **V3.1 → V3.12** ble trinnvise minor-releases på `0.4.x`-linjen.
- Wire-format-endringer akkumulerte til **V4.0**, men endte med å være
uendret fra 0.4.x — major-bumpen til 4.0 markerer audit-cycle ferdig
og GA-frosset kjerne, ikke en wire-bump.
- **V4.0** er GA — låst kjerne, pakket for ekstern review, ingen
voice/video.
- **V5.0** legger sanntid (voice/video/broadcast) oppå den frosne
4.0-stacken. Bygger på reserverte envelope-typer slik at 4.0-klienter
ignorerer 5.0-trafikk gracefully — ikke breaking.
- Hver `V*`-merge oppdaterer `CHANGELOG.md` og bumper alle pakker via
`bun run version`.

8
docs/SHADE-BY-SCENARIO.md
View File

@@ -29,9 +29,11 @@ Pull in **one row** that matches your project; add optional columns only when ne
| **Large files** — resumable E2EE upload/download | Above + stream protocol + HTTP (or WS) transport | `@shade/sdk` (re-exports transfer) + mount transfer routes on **your** HTTP server | `shade.upload` / `onIncomingTransfer` — see [streams.md](./streams.md) |
| **React UI** — upload/download widgets | Runtime from SDK + widgets | `@shade/sdk` + `@shade/widgets` | `ShadeRuntimeProvider`, `useShadeUpload` / `useShadeDownload` |
| **Prekey hosting only** — one container per product | No app crypto in the container | Docker image / `@shade/server` | Deploy prekey image; point `prekeyServer` at it from apps |
| **Offline-tolerant messaging** — recipient may be offline | Above + a relay that holds ciphertext blobs | `@shade/inbox` (client) + `@shade/inbox-server` (or the prekey container, which bundles both) | Register address, `inbox.send()` to peer, `inbox.onIncoming(handler)` — see [inbox.md](./inbox.md) |
| **"What if I lose my phone?"** — survive device loss without a recovery agent | Above + Shamir-split shares to `n` guardians; threshold `k` reconstruct | `@shade/recovery` + `@shade/widgets` (`<RecoverySetup />`, `<RecoveryRequest />`, `<RecoveryApprove />`) | `setupRecovery` / `attachGuardian` / `requestRecovery` — see [recovery.md](./recovery.md) |
| **Maximum control** — custom wire, custom transport | Wire + session manager | `@shade/core` + `@shade/proto` (+ your storage + crypto provider) | `ShadeSessionManager`, encode/decode envelopes yourself |
| **HTTP or WebSocket convenience** | Auto-wrap application bytes | `@shade/transport` on top of your stack | Use when you want transport helpers, not a new protocol |
| **Android** | Byte-compatible with TS (roadmap) | `shade-android` module | See [android/shade-android/README.md](../android/shade-android/README.md) — parity work in progress |
| **Android** | Byte-compatible with TS (cross-vector gated in CI) | `shade-android` module | See [android/shade-android/README.md](../android/shade-android/README.md). Cross-platform vectors live in [`test-vectors/`](../test-vectors/) and are exercised by both runners. |
You can **mix rows**: e.g. backend with `@shade/sdk` + SQLite for sessions, separate service mounting `transfer` routes, browser clients using `@shade/widgets`.
@@ -43,7 +45,7 @@ You can **mix rows**: e.g. backend with `@shade/sdk` + SQLite for sessions, sepa
2. Pick **storage** (`sqlite:…`, Postgres, or project-specific adapter implementing the core storage interfaces).
3. Choose **surface**: usually `@shade/sdk` unless you truly need `@shade/core` only.
4. For files: enable **transfer routes** and authenticate chunk uploads using the patterns in the SDK (see streams doc).
5. Run **`shade doctor`** when something fails in production-ish setups (install the CLI as in repository [Quick start](../README.md#quick-start)); coverage is evolving — roadmap in [V2.2](./V2.2.md).
5. Run **`shade doctor`** when something fails in production-ish setups (install the CLI as in repository [Quick start](../README.md#quick-start)); the gates that fire are documented in [trust-ux.md](./trust-ux.md) and [PRODUCTION-CHECKLIST.md](./PRODUCTION-CHECKLIST.md).
---
@@ -54,7 +56,7 @@ You can **mix rows**: e.g. backend with `@shade/sdk` + SQLite for sessions, sepa
| File transfer architecture | [streams.md](./streams.md) |
| Deployment & operations | [DEPLOYMENT.md](./DEPLOYMENT.md) |
| Threat model | [THREAT-MODEL.md](../THREAT-MODEL.md) |
| Planned improvements | [V2.1](./V2.1.md), feature backlog [V2.2](./V2.2.md), trust/ops [V2.3](./V2.3.md) |
| Planned improvements | [ROADMAP](./ROADMAP.md) — V3.x archive under [`archive/`](./archive/), next milestone [V5.0](./V5.0.md) |
---

135
docs/V5.0.md Normal file
View File

@@ -0,0 +1,135 @@
# Shade V5.0 — Voice & Video
**Status:** Idea (post-V4.0 GA)
**Effort:** XXL (4+ måneder)
**Forrige:** V4.0 GA + V3.11 (P2P transport kreves)
**Adresserer:** V2.1-tillegg "ShadeVoiceButton / ShadeVideoCall / ShadeBroadcaster"
V5.0 er den dedikerte sanntids-releasen — alt VOIP og videostreaming
samles her, *etter* at Shade 4.0 er GA-merket. Stacken under
(ratchet, transport, observability, recovery, key transparency,
WebRTC P2P) er låst i 4.0; 5.0 bygger uten å røre kjernekrypto-
revisjonen.
---
## Mål
E2EE sanntidskommunikasjon på Shade-stack: voice-calls, video-calls,
broadcast/streaming — alt som "magic drop-in"-komponenter for konsumerende
apper.
```tsx
<ShadeVoiceButton to={peerAddress} />
<ShadeVideoCall to="device:server-admin" />
<ShadeBroadcaster streamKey="game-stream-1" />
<ShadeViewer streamKey="game-stream-1" />
```
---
## Scope
### Inn
- Ny pakke `@shade/voice` — 1:1 voice over WebRTC P2P.
- Ny pakke `@shade/video` — 1:1 video, deler kjerne med voice.
- Ny pakke `@shade/broadcast` — 1:N broadcast med relay-helper.
- SFrame-style frame encryption — payload-keys ratchet'es per call,
derivert fra Shade-session.
- Codec: Opus (audio), AV1/VP9 (video) — WebRTC standard.
- Widget-komponenter for hvert use case.
- Key-rotation under loss: forward-secrecy per X frames eller hvert N
sekund.
### Ut
- Group-calls (≥ 3 deltakere) som første milestone — krever SFU + group
key agreement; egen sak.
- Replacement for native phone-app — vi tilbyr in-app calls.
- Codec-implementasjon — vi bruker browser/native WebRTC.
---
## Design
### Frame-key derivasjon
```text
callKey = X3DH(A, B) → HKDF("shade-call-v1") → callRatchetKey
frameKey[i] = HKDF(callRatchetKey, "frame" || u64(i))
```
`callRatchetKey` ratcheter forward hver N millisekund eller hver M frames;
kompromittert frame = bare det vinduet eksponert.
### SFrame
Følger IETF MLS/SFrame-mønstre:
- Header er klartekst (codec-metadata).
- Payload er AES-GCM med deterministisk nonce.
- Mottaker dropper frames med out-of-window seq.
### Topologi
- 1:1: P2P via V3.11.
- Broadcast: relay-helper i `@shade/broadcast-relay` distribuerer
ciphertext til subscribers — relay ser aldri plaintext.
---
## Leveranser
### Pakker
- `@shade/voice` + `@shade/video` (delt kjerne i `@shade/realtime-core`).
- `@shade/broadcast` + `@shade/broadcast-relay`.
- Widgets: `<ShadeVoiceButton />`, `<ShadeVideoCall />`,
`<ShadeBroadcaster />`, `<ShadeViewer />`.
### Tester
- Unit: SFrame encrypt/decrypt + tamper.
- Integration: 1:1 video 30 fps i 60 s; > 99 % frames levert; key rotation
observert.
- Loss recovery: 30 % packet loss → quality grace.
- Adversarial: relay-DB-dump avslører ingen plaintext.
### Dokumentasjon
- `docs/voice-video.md` — setup, codec-tradeoffs, broadcast-arkitektur.
---
## Akseptansekriterier
- [ ] 1:1 video 60 fps + 1080p mellom to klienter samme LAN.
- [ ] Frame-key kompromittering blokkerer maks 1 sekund forward data.
- [ ] Broadcast 1:50 viewers fungerer med < 2 s end-to-end latency.
---
## Avhengigheter
- **V4.0 GA** — kjerne-stacken må være ekstern-revidert og frosset før
vi legger sanntid-protokoll oppå.
- V3.11 — P2P transport (kommer i V4.0-vinduet).
- V3.5 — Android-paritet hvis voice/video skal funke på mobile.
---
## Risiko
- **Codec-quirks.** AV1 vs VP9 vs H.264 har ulik browser-støtte.
- **Frame-key sync under loss.** Avansert; SFrame-spec er fortsatt under
standardisering.
- **Latency vs sikkerhet.** Hver ratchet-step legger på µs.
---
## Migrasjon
Nye pakker. Ikke breaking — wire-formatene fra V4.0 holdes uendret;
voice/video legger til egne envelope-typer i et reservert range som
4.0-clients ignorerer.

0
docs/V2.1.md → docs/archive/V2.1.md
View File

0
docs/V2.2.md → docs/archive/V2.2.md
View File

0
docs/V2.3.md → docs/archive/V2.3.md
View File

100
docs/archive/V3.1.md Normal file
View File

@@ -0,0 +1,100 @@
# Shade V3.1 — Documentation & Hardening Foundation
**Status:** Done
**Effort:** S (1–2 uker)
**Forrige:** V2.3
**Neste:** V3.2 / V3.3 / V3.4 (kan kjøres parallelt)
---
## Mål
Lukke "lav-friksjon"-gjelden fra V2.1, V2.2 og V2.3 før vi tar fatt på de tunge
sikkerhetsløftene. Dette er pre-arbeidet som låser opp resten av roadmapen:
operatører skal kunne deploye trygt, transfer-konsumenter skal ha klare grenser,
og OpenAPI skal dekke hele HTTP-flaten.
Ingen ny kjernekode — kun docs, OpenAPI-utvidelser, retention-defaults og en
test-/threat-matrise.
---
## Scope
### Inn
- README + `@shade/server`-README: eksplisitt "keys vs payloads"-narrativ med
diagram + lenke til `THREAT-MODEL.md`.
- Ny `docs/PRODUCTION-CHECKLIST.md`: TLS, backup, observer-token-rotering,
SQLite vs PG, log-nivå, stale-params, secret-rotering.
- Hardening-seksjon i `docs/streams.md`: max stream-size, TTL, quota-mønstre —
peker mot `@shade/files`-hooks som referanse.
- `openapi.yaml` utvidet med `/v1/transfer/*` (`chunk`, `state`, `health`) +
sikkerhetsskjema for `ShadeTransferAuthenticator`.
- Retention-defaults i `docs/streams.md` + SDK-template:
`pruneStreamStates`-cron som default — "ferdige streams ryddes etter N
dager".
- `SECURITY.md`-utvidelse: review-status, "hvordan rapportere", lenking fra
`THREAT-MODEL.md`-rader → `tests/security/*` (test-/threat-matrise).
### Ut
- Faktisk crypto-review (det er V4.0).
- Endringer i krypto- eller wire-format.
- Ny kode utenfor SDK-templates.
---
## Leveranser
### Dokumentasjon
- `docs/PRODUCTION-CHECKLIST.md` — ny.
- `docs/streams.md` — utvidet med "Hardening" og "Retention".
- `README.md` — diagram-justering + "Hva som ikke går via Shade-server".
- `packages/shade-server/README.md` — speile narrativet.
- `SECURITY.md` — review-status + threat-/test-matrise.
- `THREAT-MODEL.md` — krysslenker til konkrete tester.
### Kode (kun konfig + templates)
- `packages/shade-server/openapi.yaml` — `/v1/transfer/*`-paths,
`ShadeTransferAuthenticator` securityScheme.
- `packages/shade-cli/templates/bun-server` — default
`pruneStreamStates`-cron.
### Tester
- Lint-test: OpenAPI-spec validerer fortsatt mot OpenAPI 3.1-skjema.
- Smoke-test for cron i template.
---
## Akseptansekriterier
- [ ] Ny utvikler kan lese README + `PRODUCTION-CHECKLIST.md` og deploye
prod-klar Shade uten å lese hele kodebasen.
- [ ] Generert klient (Python eller Go) fra `openapi.yaml` dekker både
prekey- og transfer-flate uten manuelle fixes for happy path.
- [ ] `THREAT-MODEL.md` linker hver "Mitigations"-rad til minst én test-fil.
- [ ] Default SDK-template `bun-server` prune'r resumable streams uten
manuell konfig.
---
## Avhengigheter
Ingen.
---
## Risiko
Lav. Verste utfall er foreldet docs hvis V3.2+ endrer overflater. Mitiger ved
å skrive små, oppdaterbare seksjoner heller enn lange narrative kapitler.
---
## Migrasjon
Ingen — alt er additivt.

134
docs/archive/V3.10.md Normal file
View File

@@ -0,0 +1,134 @@
# Shade V3.10 — Social Key Recovery
**Status:** Done — landet i `@shade/recovery` 0.4.0, frosset i 4.0 GA.
**Effort:** L (4–8 uker)
**Forrige:** V3.2 + V3.3
**Adresserer:** V2.1-tillegg "sosial nøkkel-recovery"
---
## Mål
Løs det største UX-hullet i alle E2EE-systemer: **"Hva skjer hvis jeg
mister telefonen?"**. Bruker velger N "guardians" (familie / venner /
jobb-partnere); når bruker mister enheten, kan en threshold-andel av
guardians sammen returnere identity-nøkkelen — uten at noen enkelt guardian
kan gjøre det alene, og uten at server lærer noe.
---
## Scope
### Inn
- Shamir Secret Sharing (k-of-n) over identity private key (eller en
backup-encryption-key).
- Distribusjon av shares via eksisterende 1:1 Shade-sesjoner — guardians
lagrer share lokalt.
- Recovery-flow: ny enhet ber threshold guardians sende sine shares;
rekonstruerer på ny enhet.
- Verifikasjons-step: ny enhet beviser identitet til hver guardian via OOB
safety-number-sammenligning **før** guardian frigjør share.
- UX-guide: hvor mange guardians, hvilken threshold, hvordan rotere når en
guardian mister enhet.
### Ut
- "Cloud guardian" / Shade-driftet recovery — vi tillater ingen sentralisert
komponent som kan gjøre det alene.
- Auto-distribusjon (vi krever eksplisitt valg av guardians).
---
## Design
### Hva deles
```text
shareSecret = AES-256-GCM-encrypt(identityState, recoveryKey)
recoveryKey is Shamir-split(k, n) → shares[i]
shareSecret stored locally + on each guardian
each guardian receives one share via Shade.send
```
`identityState` er det samme som `Shade.exportBackup` (eksisterer i 0.3.x),
men her gjenbrukes formatet.
### Recovery-flow
1. Ny enhet genererer **temporary** identity + safety number.
2. Ny enhet kontakter guardians via prekey-server (OOB verifisering først).
3. Hver guardian godkjenner manuelt og returnerer sin share via
`Shade.send`.
4. Ny enhet rekonstruerer `recoveryKey`, dekrypterer `shareSecret`,
gjenoppretter identity.
5. Original identity roterer (gammel identitet markeres som
"compromised — used for recovery").
### Guardian-UX
- Guardian-app/widget viser:
*"Alice (din venn) har mistet sin enhet og ber om recovery share.
Bekreft fingerprint før du sender."*
- Guardian kan **avslå** uten konsekvens.
---
## Leveranser
### Pakker
- `@shade/recovery` — Shamir + share-distribusjon.
- `@shade/widgets` — `<RecoverySetup />` (velg guardians) +
`<RecoveryRequest />` (ny enhet ber) + `<RecoveryApprove />` (guardian
godkjenner).
### Tester
- Unit: Shamir split/combine roundtrip; threshold-håndhevelse.
- Integration: full 3-of-5 recovery med 5 mock-guardians.
- Adversarial: 2 guardians koluderer (under threshold) → kan ikke
rekonstruere.
- Adversarial: ondsinnet ny enhet uten safety-number-bekreftelse → ingen
guardian skal frigjøre share.
### Dokumentasjon
- `docs/recovery.md` — full UX + threat model.
- Trusselmodell-utvidelse: kollusjon ≤ k-1, identitetsforfalskning, social
engineering.
---
## Akseptansekriterier
- [ ] 3-of-5 recovery fungerer end-to-end på 2 separate enheter.
- [ ] Ingen koalisjon av (k-1) guardians kan rekonstruere `shareSecret`
(verifisert med fast-check property test).
- [ ] Guardian-side widget krever fingerprint-bekreftelse før send (gate
fra V3.3 forsterket).
---
## Avhengigheter
- V3.2 — nøkkelmateriale at-rest hos guardian skal være kryptert.
- V3.3 — fingerprint-gate på recovery-handshake.
---
## Risiko
- **UX er det vanskeligste.** "Hvem er min guardian?" er sosialt komplekst;
bruker kan velge dårlig.
- **Social engineering.** Angriper imiterer offer over telefon → guardian
gir share. Mitiger med harde fingerprint-gates + cool-down.
- **Dead guardians.** Hvis guardian dør / mister sin enhet uten å være
erstattet, threshold synker. Periodisk "guardian health check"-prompt
anbefales.
---
## Migrasjon
Ny pakke. Apper kan legge til recovery-widget i innstillinger.

124
docs/archive/V3.11.md Normal file
View File

@@ -0,0 +1,124 @@
# Shade V3.11 — WebRTC P2P Transport
**Status:** Done — landet med `@shade/transport-webrtc` 0.4.0,
`MultiTransportFallback` i `@shade/transfer`, og
`shade.configureWebRTC()` i `@shade/sdk`. Se [docs/webrtc.md](../webrtc.md).
**Effort:** XL (2–4 måneder)
**Forrige:** V3.7
**Adresserer:** V2.1-tillegg "P2P WebRTC transport"
---
## Mål
Direct peer-to-peer datakanal mellom Shade-klienter når NAT/firewall
tillater. Primær gevinst: massiv throughput for `@shade/transfer`
(filer, store payloads) og lav-latens for messaging når begge peere
er online samtidig. E2EE bevart: WebRTC DTLS-SRTP er **transport** —
payload er fortsatt Shade ratchet-krypto.
V3.11 lander i V4.0-vinduet og er foundation-only — sanntidsbruken
(voice, video, broadcast) ligger i [V5.0](../V5.0.md) som downstream
konsumer av denne datakanalen.
---
## Scope
### Inn
- Ny pakke `@shade/transport-webrtc`.
- Signaling via Shade control plane (eksisterende kanal — `Shade.send`).
- ICE/STUN: bruk offentlige STUN-servere som default.
- TURN: konfigurerbar TURN-relay som fallback.
- DataChannel for `@shade/transfer`-chunks.
- Auto-fallback: P2P → HTTP (eksisterende stack).
### Ut
- SFU/MCU (mange-til-mange topologi) — broadcast/video er V5.0.
- Voice/video media-tracks — V3.11 er ren datakanal (DataChannel);
audio/video over RTP er V5.0.
- DTLS-fingerprint-binding til Shade-fingerprint (vurderes som hardening,
men ikke krav).
---
## Design
### Connection-flow
```text
A initierer:
1. createOffer() → SDP
2. shade.send(B, { kind: "webrtc-offer", sdp })
3. B mottar over Shade-kanal, createAnswer()
4. shade.send(A, { kind: "webrtc-answer", sdp })
5. ICE-candidates exchange (samme kanal)
6. DataChannel åpen
```
### Wrapping
DataChannel sender ferdige `@shade/transfer`-chunks (allerede E2EE).
WebRTC's egen DTLS-SRTP fungerer som transport-secrecy lag.
### Topologi
- 1:1 P2P direkte når mulig.
- TURN-relay når NAT'er er for strenge (transport-only, ser ikke plaintext).
---
## Leveranser
### Pakker
- `@shade/transport-webrtc` — Connection, DataChannel-wrapper, ICE-config.
- `@shade/transfer` utvides: `WebRTCTransferTransport` som drop-in.
- `FallbackTransferTransport` får ny ledd: P2P → WS → HTTP.
### Tester
- Loopback unit: offer/answer/ICE i Bun via `node-datachannel` eller
`wrtc`.
- Integration: 100 MB transfer over P2P vs HTTP — P2P skal vinne på samme
nettverk.
- Failover: TURN-relay påtvinger relay-modus.
- NAT-emulering (loopback med ulike NAT-typer hvis mulig).
### Dokumentasjon
- `docs/webrtc.md` — setup, STUN/TURN-config, NAT-traversal-håp og
-realiteter.
---
## Akseptansekriterier
- [ ] To klienter på samme LAN: P2P direct uten TURN, throughput > 5x
HTTP-baseline.
- [ ] To klienter bak strenge NAT'er: TURN-relay aktiveres automatisk.
- [ ] Failover P2P-død → HTTP innen 5 s uten meldingstap.
---
## Avhengigheter
- V3.7 — bridge-mønstre + fallback-arkitektur.
---
## Risiko
- **NAT-traversal-helvete.** Mange edge-cases. Mitiger med tidlige
integration-tester på faktiske NAT-konfigurasjoner.
- **Browser-kompatibilitet.** Safari har sine egne RTC-quirks.
- **TURN-koster.** TURN-relay = ekte trafikk gjennom server. Operatør må
vite det.
---
## Migrasjon
Opt-in. Eksisterende HTTP/WS-transport fungerer uendret.

557
docs/archive/V3.12-DESIGN.md Normal file
View File

@@ -0,0 +1,557 @@
# V3.12 — Key Transparency: Designnotat
**Status:** Approved (in-tree review — markeres `Design` i ROADMAP)
**Forfatter:** Shade-teamet
**Reviewer-mål:** ekstern crypto-orientert reviewer før produksjons-deploy.
**Implementasjons-target:** `@shade/key-transparency` + utvidelser i
`@shade/server`, `@shade/transport`, `@shade/sdk`.
---
## 1. Mål og ikke-mål
### Mål
Bytt ut "blind tillit til prekey-server" med en **verifiserbar
append-only log**. Når en klient mottar et prekey-bundle skal den ha
kryptografisk bevis for at:
1. Bundlen er **commit'et** i en tidstemplet log (Signed Tree Head).
2. Den eksakte (adresse, identityKey, signedPreKey)-mappingen står i
den loggen — _eller_ den står ikke (fravær-bevis).
3. Loggen har ikke skrevet om historie siden forrige fetch
(konsistens-bevis).
4. Andre klienter ser **samme** log (split-view-deteksjon via
witness-gossip).
Dette er **CT-style transparens** (RFC 6962-prinsipper) tilpasset
prekey-distribusjon.
### Ikke-mål (eksplisitt ut)
- **Federert log mellom flere prekey-servere.** Hver Shade-deployment
har én log (eller ingen). Multi-server gossip er V3.13+.
- **Løse MITM-på-første-kontakt fullstendig.** KT fanger split-view og
re-write, men ikke det at en angriper publiserer en forfalsket
identitet ved første registrering. Det er V3.3 (fingerprint-gate)
+ V3.10 (social recovery).
- **Legal/compliance audit-log.** Loggen er kryptografisk, ikke juridisk.
- **Klient-styrt sletting.** Append-only — DELETE skriver
tombstone-entry, fjerner ikke historikk.
### Beslutningskriterium for implementasjon
Når dette notatet er godkjent _og_ alle åpne spørsmål under §11 har
konkrete svar (ikke bare "vi finner ut av det senere"), kan kode
skrives. Det notatet ligger på når §11 lukkes er det vi bygger.
---
## 2. Trusselmodell-tillegg
Eksisterende THREAT-MODEL.md dekker prekey-server som "honest-but-curious"
+ tilstede TOFU. KT utvider modellen til **fully-malicious server**:
| Angrep | Pre-V3.12 | Post-V3.12 |
|---|---|---|
| Server returnerer feil bundle for én klient | Uoppdaget til OOB-verifisering | Klient kan be om proof; mismatch oppdages |
| Server bytter en allerede registrert identityKey | TOFU-fingerprint endres → V3.3-gate slår inn (men brukerinitiert) | Loggen vil vise to entries med samme adresse → witness oppdager |
| Server gir `alice` ulike identityKeys til Bob og Charlie (split-view) | Uoppdaget til OOB | Witness-gossip avslører to ulike STH-er |
| Server skriver om historikk for å skjule tidligere svik | Mulig | Konsistens-proof feiler → klient varsler |
| Server nekter å publisere ny STH | Mulig | "Stale STH"-detekteres av friskhetsbevis (max age) |
| Server kompromitterer signing-key for STH | KT-trygghet brutt | Witness gossip om gammel STH-kjede; rotasjon krever ny genesis |
KT løser **ikke**:
- Førstegangs-impersonering av en helt ny adresse (intet historisk
bevismateriale).
- Kollusjon mellom server og _alle_ witnesses.
- Klient som glemmer cached STH og må re-bootstrappe.
---
## 3. Datastruktur-valg
Vi velger **RFC 6962-stil append-only Merkle log** + **ekstern
adresse-index** med commitment-bevis. Begrunnelse:
### Vurderte alternativer
1. **Pure CT-log (RFC 6962):** Simple append-only Merkle tree.
Inklusjonsbevis trivielle. Fravær-bevis _ikke_ støttet
nativt (må scanne hele loggen).
2. **CONIKS-tre (sparse Merkle tree over adresser):** Native fravær-bevis,
men mye mer kompleks (epoch-baserte snapshots, prefix-trees,
placeholder-nodes). Overkill for første iterasjon.
3. **Hybrid (RFC 6962 log + side-index):** Loggen er sannhetskilde,
indexen er en _commitment_-mapping `address → leaf_index`. Server
beviser inklusjon via leaf-path, fravær via "denne adressen er ikke
i indexen ved tree_size T" + signert STH.
**Valg: alternativ 3.** Det gir CT-stil enkelthet, samt fravær-bevis
nesten gratis (commitment til indexen er en del av hver STH).
### Konkret format
#### Leaf
Hver leaf representerer én registrering eller revoke:
```
leaf = SHA256(
0x00 || // leaf prefix (RFC 6962)
uint64_be(timestamp_ms) ||
byte(operation) || // 0x01 register, 0x02 replenish, 0x03 delete
uint16_be(len(address)) || address_bytes ||
uint16_be(len(bundle_hash)) || bundle_hash // 32 bytes SHA-256 over canonical bundle
)
```
`bundle_hash` er deterministisk hash av:
```
canonical_bundle = SHA256(
0x01 || // bundle prefix
identitySigningKey (32) ||
identityDHKey (32) ||
uint32_be(signedPreKey.keyId) ||
signedPreKey.publicKey (32) ||
signedPreKey.signature (64)
)
```
One-time prekeys er **ikke** med i bundle-hashen — de er ephemerale og
ville lekket OTP-rotasjons-mønstre.
#### Tree
Merkle-tre over leaf-array, RFC 6962 §2.1:
- `MTH(empty) = SHA256()`
- `MTH({d}) = SHA256(0x00 || d)` (already hashed leaf)
- `MTH(D[n]) = SHA256(0x01 || MTH(D[0:k]) || MTH(D[k:n]))` der
`k` er største 2-potens < n.
#### Signed Tree Head (STH)
```
sth = {
tree_size: uint64,
timestamp: uint64_ms,
root_hash: bytes(32),
index_root: bytes(32), // commitment til adresse-index ved denne tree_size
log_id: bytes(32), // SHA-256 av server-public-key (stabil ID)
signature: bytes(64) // Ed25519 over canonical(rest)
}
```
`canonical(sth)` for signing:
```
0x02 || // sth prefix
uint64_be(tree_size) ||
uint64_be(timestamp) ||
root_hash (32) ||
index_root (32) ||
log_id (32)
```
#### Inklusjons-bevis
Standard RFC 6962 audit-path: liste med søsken-hasher fra leaf til root,
slik at klient re-beregner root og sammenligner med STH.
#### Konsistens-bevis
Standard RFC 6962 §2.1.2: bevis at tree med `tree_size = N1` er prefix
av tree med `tree_size = N2 > N1`. Klient bruker dette for å detektere
re-write.
#### Fravær-bevis
Adresse-indexen er en sortert liste `(address, leaf_index_of_latest)`
serialized og hashet. `index_root` i STH er commitment.
For å bevise fravær av adresse `addr` ved tree_size `N`:
- Server returnerer hele indexen ved tree_size `N` (sortert), eller
- (effektivt:) Returnerer naboparet `(addr_prev, addr_next)` der
`addr_prev < addr < addr_next` lexikografisk, sammen med en
Merkle-path i en sparse Merkle tree over indexen.
Første iterasjon: vi serialiserer hele indexen og lar klienten
laste den (kompakt: <100 KB selv for 100k adresser). Senere
optimaliserer vi til sparse Merkle tree hvis dataset vokser.
---
## 4. Friskhetsbevis (Signed Tree Heads)
### Frekvens
- **Min:** Ny STH ved hver mutasjon (register/replenish/delete) — synkront
i write-pathen.
- **Maks-stale:** Selv uten mutasjoner skal en STH publiseres minst hver
**10. minutt** ("heartbeat STH" — samme tree_size, oppdatert timestamp).
Dette gir klienter mulighet til å detektere "død" log uten å bekymre
seg om hvorvidt logen faktisk har endret seg.
### Klient-akseptansevindue
Klient avviser STH eldre enn `now - 24 timer` (default, konfigurerbar).
Dette beskytter mot replay av gamle STH-er som "skjuler" en mutasjon
gjort i ettertid.
### Stale-STH som soft-fail
Hvis STH er stale men gyldig signert: klient logger advarsel,
returnerer bundle med `proof.staleness = 'warn'` (V1) eller blokkerer
(V2 etter dogfooding). Vi starter med _warn_, eskalerer til _block_
når witness-økosystem er etablert.
---
## 5. Klient-verifikasjonssteg
På hver `fetchBundle(address)`:
1. Server returnerer `{ bundle, proof: { sth, leaf, audit_path, leaf_index, address_index_proof } }`.
2. Klient verifiserer:
- `sth.signature` mot kjent `log_public_key` (pinnet ved første
bootstrap).
- `sth.timestamp >= now - max_age_ms` (default 24t).
- Re-beregner `leaf_hash` fra bundle og sammenligner med `proof.leaf`.
- Re-beregner `root_hash` fra `audit_path + leaf` og sammenligner med
`sth.root_hash`.
- Verifiserer `address_index_proof` mot `sth.index_root`.
3. Hvis klient har en cached forrige STH: sjekk **konsistens-proof**
mellom forrige og denne. Server publiserer dette i
`GET /v1/kt/consistency?from=<size>&to=<size>`.
4. Hvis klient har en cached STH for samme `tree_size` med ulik
`root_hash` → **split-view alarm**.
### Probabilistisk vs. obligatorisk verifisering
Vi velger **obligatorisk** ved hver bundle-fetch. Bundle-fetch er sjelden
(per ny peer, ikke per melding) — kostnaden er <100ms. Probabilistisk
verifisering ville la klienter bli lurt av "én dårlig fetch" uten
deteksjon.
### Bootstrap
Første gang en klient møter en log: pinner `log_public_key` etter å ha
hentet det fra et **ut-av-bånd**-pinningendepunkt eller fra `Shade.config`
(operatør sender den med klient-config). Etterfølgende rotasjon krever
ny genesis-STH med eksplisitt break-event signert av forrige nøkkel.
---
## 6. Witness/auditor-rolle
### Hva en witness gjør
- Periodisk poll: `GET /v1/kt/sth` (hent siste STH).
- Lagrer alle observerte STH-er i append-only lokal store.
- Eksponerer `GET /witness/sth?log_id=...&tree_size=...` slik at andre
klienter kan sammenligne hva _denne_ witnessen har sett.
- Verifiserer konsistens mellom hver ny STH og forrige.
### Klient-witness-gossip
Klient-bibliotek kan operere i tre moduser:
1. **Observe-only:** verifiserer kun bundle den selv henter, ingen
gossip.
2. **Light-witness:** poller STH hver `Xt` og lagrer lokalt; sammenligner
med STH levert ved bundle-fetch.
3. **Full-witness:** publiserer signerte STH-observasjoner til en
konfigurert peer-liste eller offentlig endpoint.
V1 leverer 1 og 2. Mode 3 (full-witness publication-protocol) er V2
hvis økosystem trenger det.
### Hvem kjører witnesses?
- Shade-prosjektet kjører **referanse-witness** på offentlig endpoint
(separate-from-prekey-server).
- Power-users / operatører kan kjøre egne via `@shade/key-transparency/witness`-
API.
- Tredjeparts auditors (typisk security-research) er invitert.
Vi krever **ikke** federation/konsensus mellom witnesses i V1 — gossip
er rent "har du sett samme STH som meg?".
---
## 7. Operatørkost
### Lagring
- **Per leaf:** 32 bytes (hash) + ~80 bytes adresse-index entry =
~112 bytes.
- **100k adresser, 1 rotasjon/år, 1 replenish/uke:** ~5.4M leaves =
~600 MB log. Tre-strukturen er beregnet on-demand, ikke lagret.
- **Index:** ~100k × 80B = 8 MB i minne (cacheable).
### CPU
- STH-signing: 1 Ed25519-signering per mutasjon + heartbeat = <1k/dag for
små deployments. Trivielt.
- Audit-path-beregning: O(log N) ved fetch. <1ms.
- Konsistens-proof: O(log N).
### Backup
Logen MÅ aldri miste data — sletting eller corruption ødelegger
integritet permanent. Strategi:
- Loggen lagres som append-only tabell `shade_kt_log` (PG) med
`(leaf_index, leaf_hash, leaf_data_json)`.
- Backup hver time + WAL-shipping anbefalt.
- Ved corruption: se §10 Recovery.
### STH-signing-key
- Genereres ved første KT-aktivering, lagres i operatør-styrt secret
(env, KMS, eller på disk for hjemme-server).
- Rotasjon: **breaking event** — krever ny genesis-STH der ny key
signerer melding "rotated from ${old_key}" med _gammel_ key. Klient
må eksplisitt akseptere rotasjonen.
---
## 8. Migrasjon
### Server-side
KT er **opt-in** på operatør-nivå. `createPrekeyServer({ keyTransparency:
{ enabled, store, signingKey } })`. Når slått på:
1. Server skriver alle eksisterende identiteter inn som genesis-leaves
ved boot.
2. Første STH publiseres med `tree_size = N` der N er antall
eksisterende adresser.
3. Klient som henter bundle får proof; klient som ikke støtter KT
ignorerer proof-felt (forward-compatible).
### Klient-side
`@shade/sdk`-config:
```ts
createShade({
keyTransparency: {
mode: 'observe' | 'light-witness' | 'off',
logPublicKey: '<base64>',
maxStaleMs: 86_400_000,
},
// ...
})
```
`mode: 'off'` (default for backward-compat første release) — ignorerer
proof. Ny SDK med `mode: 'observe'` verifiserer men feiler ikke harde
hvis proof mangler. `mode: 'observe-strict'` (senere) krever proof.
### Eksisterende deployments
Operatør kan rulle KT inn på live server uten klient-update:
1. Skru på KT i server-config → server begynner å produsere proofs.
2. Gamle klienter ignorerer proof-felt (de er additive i bundle-respons).
3. Nye klienter med `mode: 'observe'` begynner å verifisere.
4. Når operatør har testet og publisert log-public-key OOB, kan brukere
skifte til `'light-witness'`.
---
## 9. Akseptansekriterier
- [ ] `@shade/key-transparency` pakke leverer:
- Merkle log core (RFC 6962 hash-funksjoner).
- STH-signing/verifikasjon.
- Inklusjons-bevis generering + verifisering.
- Konsistens-bevis generering + verifisering.
- Adresse-index med commitment.
- Witness-light klient.
- Cross-platform (TS-only, ingen native deps).
- [ ] `@shade/server` integrasjon:
- `KTLogStore`-interface (memory + postgres).
- Routes: `GET /v1/kt/sth`, `GET /v1/kt/sth/:tree_size`,
`GET /v1/kt/consistency`, `GET /v1/kt/inclusion/:address`.
- Bundle-fetch returnerer `{ bundle, proof }` når KT aktivert.
- Heartbeat-STH-publisering hver 10. minutt (configurable).
- [ ] `@shade/transport` `ShadeFetchTransport`:
- Aksepterer optional `keyTransparency`-verifier.
- `fetchBundle()` returnerer `{ bundle, proof?: KTProof }`.
- [ ] `@shade/sdk` `Shade`:
- `keyTransparency`-config.
- Verifiserer proof ved hver bundle-fetch når aktivert.
- Cacher STH for split-view-deteksjon.
- [ ] **End-to-end test: split-view detection.**
- Test-server gir Bob bundle X, Charlie bundle Y for samme adresse `alice`.
- Bob+Charlie kjører som light-witness, gossiper STH-er.
- Test asserter at mismatch detekteres innen N polls.
- [ ] **End-to-end test: log re-write detection.**
- Server skriver om historie (test-only API).
- Konsistens-proof feiler på neste fetch.
- [ ] Operatør-doc dekker recovery-strategi.
- [ ] CHANGELOG, README, ROADMAP oppdatert.
- [ ] Cross-platform vector-test for Merkle hash + STH (Android/TS
paritet, samme som V3.5-tradisjonen).
---
## 10. Recovery
### Log corruption
Hvis log-data tapes (disk-feil før backup): **kan ikke gjenopprettes
uten å miste integritet** — det er hele poenget.
Recovery-prosedyre:
1. Operatør publiserer "log-restart" event signert med STH-keyen.
2. Genesis-STH genereres på nytt med ny `log_id` (= ny offentlig nøkkel
eller eksplisitt versjon).
3. Klienter som har cached STH-er fra gammel log varsles via
eksplisitt diskrepans i `log_id`.
4. Brukere som er bekymret må OOB-verifisere identiteter (V3.3-gate
trigges automatisk for fingerprint-rotasjon).
### Stale signing-key
Hvis STH-keyen lekkes: rotasjon krever break-event (§7). Inntil
brukerne aksepterer ny key, oppfører cient-bibliotek seg som om STH
mangler (soft-fail i `observe`-mode, blokkerer i `observe-strict`).
---
## 11. Åpne spørsmål (lukket før kode)
| Spørsmål | Svar |
|---|---|
| Hvordan distribueres `log_public_key` til klient første gang? | Operatør embedder i `Shade.config` ved app-init. OOB-pinning er fallback. |
| Skal one-time prekeys være med i bundle-hash? | Nei — ephemerale, og deres rotasjon ville støy-fylle loggen. |
| Konflikt: STH ved hver mutasjon vs. batched STH? | Per mutasjon. Heartbeat hver 10 min uansett. Batching vurderes som optimalisering hvis throughput blir et problem (ikke nå). |
| Hva skjer ved replenish (kun OTP-tilført)? | Skriver ikke til log (bundle-hash uendret). Heartbeat-STH dekker friskhet. |
| Hva med DELETE? | Skriver tombstone-leaf med `operation = 0x03`. Identiteten i indexen markeres som "deleted at tree_size N". |
| Sparse Merkle tree for index-proof? | Senere — V1 bruker hele indexen i fravær-proof. <100 KB ved 100k adresser er akseptabelt. |
| Klient-cache eviction-policy for STH? | LRU på `log_id`, last-N (default 100). Klient holder _alltid_ siste sett STH. |
| Witness-publication-protokoll? | V1 har poll-only (`GET /witness/sth`); push-publication er V2. |
Alle åpne spørsmål har konkrete svar. Implementasjon kan starte.
---
## 12. Pakke-struktur
```
packages/shade-key-transparency/
├── package.json # @shade/key-transparency, v0.4.0
├── src/
│ ├── index.ts # Public exports
│ ├── hashes.ts # RFC 6962 leaf/node hashing
│ ├── log.ts # MerkleLog (in-memory) + audit-path
│ ├── consistency.ts # Consistency-proof gen/verify
│ ├── sth.ts # STH sign / verify / canonical bytes
│ ├── index-tree.ts # Address index commitment
│ ├── proof.ts # KTProof type + bundle-proof verifier
│ ├── store.ts # KTLogStore interface (server-side)
│ ├── memory-store.ts # In-memory KTLogStore
│ ├── witness.ts # Light-witness client
│ └── errors.ts # KT-specific error types
└── tests/
├── hashes.test.ts
├── log.test.ts # RFC 6962 test vectors
├── consistency.test.ts
├── sth.test.ts
├── index-tree.test.ts
├── proof.test.ts
└── split-view.test.ts # End-to-end split-view detection
```
Server-integrasjon i `@shade/server`:
```
packages/shade-server/src/
├── kt-routes.ts # /v1/kt/* routes
├── kt-integration.ts # Hook bundle-fetch + register/delete to log
└── ...
```
Postgres-implementasjon i `@shade/storage-postgres`:
```
packages/shade-storage-postgres/src/
├── postgres-kt-store.ts # KTLogStore on PG
└── ...
```
Klient-integrasjon i `@shade/transport` + `@shade/sdk`:
```
packages/shade-transport/src/
├── kt-verifier.ts # Proof-verifier for fetchBundle
└── ...
packages/shade-sdk/src/
├── kt.ts # Shade.keyTransparency config + cache
└── ...
```
---
## 13. Test-strategi
1. **RFC 6962 test-vektorer:** importer kjente vektorer fra
<https://datatracker.ietf.org/doc/html/rfc6962#appendix-A>.
2. **Property-tests (fast-check):** for hver tree_size N og hvert
leaf-index i: `verify(audit_path(i, N), leaf, sth) === true`.
3. **Konsistens-bevis property-tests:** for N1 < N2:
`verify_consistency(proof, sth1, sth2) === true`.
4. **Split-view e2e:** to klienter, ondsinnet test-server, witness
gossip oppdager mismatch.
5. **Re-write-detection e2e:** server muterer log-historie, klient
neste fetch får konsistens-proof som feiler.
6. **Cross-platform:** Android (Kotlin) + TS gir samme leaf-hash for
samme bundle (V3.5-paritet er forutsetning, så dette må også gå
gjennom kotlin-port; for V3.12 første release dekker vi TS — Android
port er V3.13).
7. **Stale STH:** klient avviser STH > max_age.
8. **Bootstrap-pinning:** klient feiler hvis log_public_key ikke matcher.
---
## 14. Sikkerhetsvurdering
- **Falsk trygghet hvis halvveis:** Avhjelpes ved at default-mode er `'off'`,
bare _eksplisitt_ aktivert KT gir hardere garantier. Dokumentasjon
fremhever at `'observe'` er observasjon, ikke obstruksjon, til
økosystemet er etablert.
- **Server-side mutability av historie:** Avhjelpes ved at `KTLogStore`
kun har `append()` — ingen `update()`/`delete()` på historiske leaves.
PG-tabellen har CHECK constraint og BEFORE-triggers for ekstra defense
in depth (se §7).
- **STH-key compromise:** dokumentert §10. Operatør-ansvar.
- **DoS via massive index-proofs:** index-proof er i V1 hele indexen.
100 KB per fetch er overkommelig; rate-limiteren dekker excess.
- **Replay av gammel proof:** STH-timestamp + max_age beskytter.
---
## 15. Approval
Når dette notatet er reviewed (in-tree review er nok for å kommitte
første implementasjon; ekstern crypto-review er pre-deploy-krav per
V3.12 §"Pre-requisite designnotat"), kan implementasjon starte.
**Implementasjon-rekkefølge** (alle commits i samme branch):
1. `@shade/key-transparency` core (Merkle log, STH, proofs).
2. Server-integrasjon (`@shade/server` + memory/postgres KTLogStore).
3. Klient-integrasjon (`@shade/transport` verifier + `@shade/sdk` config).
4. Witness-light + e2e split-view-test.
5. Operatør-doc + CHANGELOG + README + ROADMAP.
— end of design —

99
docs/archive/V3.12.md Normal file
View File

@@ -0,0 +1,99 @@
# Shade V3.12 — Key Transparency
**Status:** Done (0.4.0). Designnotat: `docs/V3.12-DESIGN.md`.
Operatør-/recovery-guide: `docs/key-transparency.md`.
**Effort:** XXL (4+ måneder, multi-quarter)
**Forrige:** V3.5 (hovedplattformene stabile først)
**Adresserer:** V2.3 §1A
---
## Mål
Reduser tillit til prekey-server fra "blind tillit" til "verifiserbar log".
Når serveren utleverer et bundle, skal det være kryptografisk forpliktet i
en **append-only log** som klienter (eller tredjeparts-auditors) kan
verifisere. Et split-view-angrep der serveren viser ulike bundles til ulike
klienter blir fanget av gossip.
---
## Pre-requisite: designnotat
**Ingen kode før dette er review'd og approved:**
- Trusselmodell-tillegg: hva CT/attest faktisk løser, hva som forblir åpent.
- Datastruktur-valg: append-only Merkle log (CT-stil), CONIKS-tre, eller
hybrid.
- Friskhetsbevis: hvor ofte signed tree heads utgis; hva er "stale"?
- Klient-verifikasjonssteg: må klient verifisere på hver bundle-fetch,
eller probabilistisk?
- Witness/auditor-rolle: hvem kjører dem? Hvordan gossip mellom klienter?
- Operatørkost: log-størrelse, signing-frekvens, backup-strategi.
- Migrasjon: eksisterende prekey-server → log-utvidet.
Designnotatet er en `docs/V3.12-DESIGN.md`-PR som må review'es av minst én
ekstern crypto-orientert reviewer.
---
## Mulig scope (etter designnotat)
### Inn (estimat)
- Append-only log som tillegg til prekey-server.
- Inklusjons-bevis ved bundle-fetch (Merkle-path).
- Fravær-bevis for "denne adressen har ikke registrert siden timestamp T".
- Signed tree heads (STH) publisert på fast interval.
- Klient-bibliotek: `@shade/key-transparency` med verifisering.
- Witness-API: tredjeparts-auditor kan hente STH-er og logge gossip.
### Ut (eksplisitt)
- Federated log (multi-server gossip) — for stort for første iterasjon.
- Legal/compliance-side av audit-log.
- "Vi løser MITM-på-første-kontakt-helt" — KT alene fanger split-view, ikke
første-kontakt.
---
## Risiko-vurdering
KT er det **vanskeligste enkeltpunkt** i hele roadmapen:
1. **Halvveis-implementert KT er verre enn ingen KT** — gir falsk trygghet,
brukere slutter å verifisere OOB.
2. Operativt komplekst — log må aldri skrive om historie. En enkelt
restart-bug = ødelagt integritet.
3. Klient-verifikasjons-logikk må kjøre på hver bundle-fetch, eller
risikere at én "gammel" klient blir lurt.
4. Witness-økosystem krever uavhengige aktører — Shade alene kan ikke
garantere det.
**Beslutningskriterium:** Hvis designnotatet etterlater åpne "hvordan
håndterer vi X?"-spørsmål uten klare svar, parker V3.12. Pragmatisk
alternativ er **V3.3 (fingerprint-gate)** + **V3.10 (social recovery)** —
som sammen gir 80 % av MITM-beskyttelsen uten KT-kompleksiteten.
---
## Akseptansekriterier (hvis det implementeres)
- [ ] Designnotat passert ekstern review.
- [ ] Klient detekterer split-view i ende-til-ende-test (server gir to
versjoner av samme adresse → klient fanger mismatch).
- [ ] Witness-API testet med minst én ekstern auditor-instans.
- [ ] Operatør-doc dekker recovery hvis log korrumperer.
---
## Avhengigheter
- V3.5 — Android/TS paritet må være solid før vi legger på et nytt
verifikasjons-lag.
---
## Migrasjon
Helt opt-in. Operatører som ikke ønsker KT kjører videre uendret.

146
docs/archive/V3.2.md Normal file
View File

@@ -0,0 +1,146 @@
# Shade V3.2 — At-Rest Storage Encryption
**Status:** Implementert (0.4.0) — `@shade/storage-encrypted`, `@shade/keychain`,
`shade migrate-storage`, `shade rotate-storage-key`
**Effort:** L (4–8 uker)
**Forrige:** V3.1
**Adresserer:** V2.1 §2
---
## Mål
Opt-in beskyttelse av sensitiv state — identity-nøkler, session-state, valgfri
stream-resume-secret — med nøkler som **ikke** ligger i klartekst i databasen.
Trusselmodellen sier i dag eksplisitt at en stjålet DB eksponerer private
nøkler; dette løser det for deploys som velger å aktivere det.
---
## Scope
### Inn
- Ny `EncryptedStorageProvider`-wrapper som dekorerer `SQLiteStorage` /
`PostgresStorage`.
- Per-rad AES-256-GCM på sensitive felter (`identity_*`, `session_*`,
valgfritt `stream_state.streamSecret`).
- KDF-pluggin (default `scrypt` fra `@noble/hashes`) for passphrase-basert
master-nøkkel.
- Tre nøkkelkilder ut av boksen:
1. **Passphrase + KDF** — utvikler oppgir secret ved oppstart.
2. **OS keychain** — macOS Keychain, Linux libsecret, Windows Credential
Vault (Node-only).
3. **App-injected key** — appens egen kode forsyner 32-byte nøkkel (mest
fleksibel).
- Migrasjons-CLI: `shade migrate-storage --encrypt --key-source=...`.
- Trusselmodell-oppdatering: "når enabled, hva er fortsatt udekket" — memory
compromise, swap, runtime-tap.
### Ut
- Browser/IndexedDB at-rest (egen pakke, vurderes etter V3.8).
- HSM/Secure Enclave (separate driver senere).
- "Always-on by default" — vi flyger opt-in for å ikke bryte eksisterende
deploys.
---
## Design
### Krypteringsenhet
- Per-rad AEAD: `nonce(12) || ciphertext || tag(16)`.
- `nonce = HKDF(rowKey, "shade-row-nonce-v1" || tableName || pk)[..12]` —
deterministisk per (tabell, pk) for å unngå nonce-reuse uten å lagre nonce
separat. Endring av (tabell, pk) → re-encryption.
- AAD binder `tableName || columnName || pk` så feltombytting blokkeres.
### Nøkkelhierarki
```text
masterKey (fra kilde — passphrase / keychain / app-injected)
│
├─ HKDF("shade-storage-v1") → storageKey (32 bytes)
│ │
│ └─ HKDF(storageKey, table || column) → fieldKey
│
└─ HKDF("shade-storage-version-v1") → versjonsnøkkel (rotasjon)
```
### Migrasjon
1. CLI leser ukryptert DB.
2. Skriver rad-for-rad-kryptering til ny `_v2`-tabell.
3. Atomisk rename + drop gammel.
4. Backup `.bak`-fil etterlatt i samme dir.
### Rotasjon
- `shade rotate-storage-key --new-source=...` re-krypterer med ny masterKey.
- Online ratchet (les med gammel, skriv med ny) for store DB.
---
## Leveranser
### Pakker
- Ny modul: `@shade/storage-encrypted` (re-export over SQLite/PG).
- Utvidelse i `@shade/cli`: `migrate-storage`, `rotate-storage-key`.
- Hjelpe-pakke: `@shade/keychain` (Node-only, valgfri peer-dep) for OS-keychain.
### Tester
- Unit: KDF-derivasjon, nonce-determinisme, AAD-binding.
- Integration: full lifecycle på SQLite + PG; start/stopp; krasj under
migrasjon.
- Tamper: bit-flip i ciphertext / AAD / nonce → dekrypterings-feil.
- Vector-fil: kryss-sjekk masterKey → fieldKey-derivasjon mot
`test-vectors/storage-encryption.json`.
### Dokumentasjon
- `docs/storage-encryption.md` — full guide.
- `THREAT-MODEL.md` — ny kolonne "with at-rest enabled".
- Migrasjonsnotat i `MIGRATION.md`.
---
## Akseptansekriterier
- [ ] Eksisterende ukryptert deploy fortsetter uten endringer (opt-in).
- [ ] `shade migrate-storage --encrypt` migrerer en levende SQLite uten
datatap, verifisert med dump-diff.
- [ ] Rotasjon kan gjøres uten downtime > 5 s for små DB.
- [ ] Wrong passphrase / wrong key → klar feilmelding, ikke krasj.
- [ ] Test-vectors deles med Android-implementasjonen (V3.5 forplikter at
vector-filen kjøres der).
---
## Avhengigheter
- V3.1 — `THREAT-MODEL.md` skal være lenket til testene først, så vi kan
utvide tabellen.
---
## Risiko
**Datatap.** En migrasjon som krasjer halvveis kan etterlate korrupt DB.
Mitigeres ved:
- Atomic-rename + `.bak`-fil.
- Dry-run-modus (`--dry-run` validerer all dekryptering før skriving).
- Refuser å starte hvis WAL har uncommitted writes.
**Nøkkeltap = totaltap.** Hvis bruker mister passphrase = ingen tilgang.
Dokumenter klart, og pek på V3.10 (Social Recovery) som langtidsløsning.
---
## Migrasjon
0.3.x deploys er ukrypterte → fortsatt ukrypterte. Aktivering er én
CLI-kommando. Backwards-kompatibel.

147
docs/archive/V3.3.md Normal file
View File

@@ -0,0 +1,147 @@
# Shade V3.3 — Fingerprint Gates & Trust UX
**Status:** Done
**Effort:** M (2–4 uker)
**Forrige:** V3.1
**Adresserer:** V2.3 §1B
**Implementert:** se `docs/trust-ux.md`
---
## Mål
Gjør safety numbers **handlingspålagte** — ikke bare synlige — i flyt der
MITM-risikoen er reell. I dag finnes `FingerprintCompare`-widget og
`requireFingerprintVerifiedFor` i `@shade/files`, men hovedkjernen
(`Shade.send`, first-large-file, backup-import) har ingen automatisk gate.
Resultat: alert-fatigue-fri, men også gate-fri.
Dette legger inn **eksplisitt blokkerende verifisering** på et lite antall
kritiske hendelser, plus widget-støtte for å eksponere det i UI.
---
## Scope
### Inn — kritiske hendelser
1. **Før første store fil** — `Shade.upload` over en bytes-terskel uten
verifisert peer.
2. **Før backup-import** — `Shade.importBackup` blokkerer til peer (eller egen
identitet) er bekreftet.
3. **Ny enhet med rotert identitet** — `acceptIdentityChange` blokkerer på
første bruk inntil verifisert.
4. **Før `@shade/inbox` fan-out** (V3.6) — gate per mottaker.
### Inn — APIer
- `Shade.beforeFirstLargeFile(threshold, handler)` — appen får mulighet til å
vise modal og returnere bekreftelse.
- `Shade.beforeBackupImport(handler)` — samme mønster.
- `Shade.beforeNewDeviceTrust(handler)` — ditto.
- `Shade.markPeerVerified(address)` / `Shade.isPeerVerified(address)` —
persistent state.
### Inn — widgets
- `<FingerprintGate />` — render-prop wrapper som blokkerer barn til
verifisert.
- `<FingerprintCompare />` utvides med "kopier OOB-tekst" + "jeg har
verifisert".
### Ut
- "Tving alle peers verifisert før hver melding" — alert fatigue.
- Cross-device sync av verified-state (kommer evt. via V3.6 inbox).
---
## Design
### Persistent verified-state
Ny tabell `peer_verifications`:
```sql
CREATE TABLE peer_verifications (
peer_address TEXT PRIMARY KEY,
fingerprint TEXT NOT NULL,
verified_at INTEGER NOT NULL,
verified_by TEXT, -- "user" | "transitive" | "tofu-after-warning"
identity_version INTEGER NOT NULL -- knytter verifikasjon til identity-rotasjon
);
```
Når peer roterer identitet → `identity_version` bumper → verifikasjon "ugyldig"
til ny verifisering.
### Hook-flyt
```text
shade.upload(peer, file)
│
├─ if !verified(peer) AND file.size > threshold
│ │
│ └─ await beforeFirstLargeFileHandler(peer, fingerprint)
│ ├─ true → markPeerVerified(peer); proceed
│ └─ false → throw FingerprintNotVerifiedError
│
└─ proceed
```
---
## Leveranser
### Kode
- `@shade/core` — `peer_verifications`-tabell + storage methods.
- `@shade/sdk` — gate-hooks + `markPeerVerified` / `isPeerVerified`.
- `@shade/widgets` — `<FingerprintGate />`, utvidet `<FingerprintCompare />`.
### Tester
- Unit: gate kalles, ikke kalles, retur false → throw, retur true → proceed.
- Integration: fil < threshold går gjennom uten gate; fil > threshold
blokkerer.
- Identity-rotasjon ugyldiggjør verifikasjon.
- Backup-import blokkerer.
### Dokumentasjon
- `docs/trust-ux.md` — guide til hvilke gates som finnes og når de bør tunes.
---
## Akseptansekriterier
- [ ] Gate kan ikke bypasses ved å nulle `threshold` ut — minimum gate finnes
alltid for backup-import og new-device.
- [ ] App uten registrerte gates får sane defaults (logger en warning, men
kjører — ikke krasj).
- [ ] Identity-rotasjon resetter verifikasjon i en testet ende-til-ende-flow.
- [ ] Widget kan rendres SSR uten å trigge runtime-gate.
---
## Avhengigheter
- V3.1 — threat-matrise oppdatert til å vise hvilke gates som dekker hvilke
rader.
---
## Risiko
- **Alert fatigue.** Hvis terskler er for lave → bruker klikker blindt.
Mitiger ved å sette default-terskler høyt (10 MiB for first-large-file)
og dokumenter justerings-guide.
- **DX-friksjon.** Apper som ikke vet om gates får uventede prompts. Mitiger
ved å logge tydelig ved første aktivering: "Shade.beforeFirstLargeFile not
configured — using default modal".
---
## Migrasjon
0.3.x apps får defaults aktivert med warning. Ingen breaking change.

124
docs/archive/V3.4.md Normal file
View File

@@ -0,0 +1,124 @@
# Shade V3.4 — Observability v2 (OpenTelemetry)
**Status:** Implementert (2026-05-02) — `@shade/observability` 0.1.0,
hekt inn i sdk/transfer/server/files/core. Off by default; flip
`SHADE_OTEL_ENABLED=1` for å aktivere.
**Effort:** M (2–4 uker)
**Forrige:** V3.1
**Adresserer:** V2.3 §4
---
## Mål
Gi produksjonsteam **distribuerte spor** rundt `TransferEngine`,
prekey-routes og `@shade/files` — uten å lekke plaintext-adresser, payloads
eller eksakte chunk-størrelser. Bygger videre på Prometheus-metrics som
allerede finnes.
---
## Scope
### Inn
- Opt-in OpenTelemetry-instrumentasjon via `@opentelemetry/api`.
- Spans rundt:
- `TransferEngine.upload` / `.download` (med lane-tags, retry-counts).
- `ShadeSessionManager.encrypt` / `.decrypt` (per-peer mutex-akkvisisjon,
ratchet-step).
- `createPrekeyRoutes` (per route, status-koder).
- `@shade/files` op-handlers (har allerede `onMetric` — utvides til OTel).
- PII-policy-doc: hva som **aldri** logges, hva binnes, hva pseudonymiseres.
- Sample-policy default off; on med `SHADE_OTEL_ENABLED=1`.
### Ut
- Trace-eksport til SaaS-leverandører (det er deploy-konfig, ikke vår kode).
- Logg-aggregering — `@shade/server` har allerede strukturert JSON.
---
## Design
### Span-attributter
| Attribute | Verdi |
|-----------|-------|
| `shade.peer.hash` | `sha256(address).slice(0, 8)` — stabil pseudonym |
| `shade.bytes.bin` | binnet — `"≤4KB"`, `"4–64KB"`, `"64KB–1MB"`, `"≥1MB"` |
| `shade.lane.count` | 1 / 4 / 16 |
| `shade.retry.count` | int |
| `shade.error.code` | `SHADE_*`-kode |
**Aldri:** `shade.peer.address`, `shade.payload`, `shade.bytes.exact`.
### API
```ts
import { withTracer } from '@shade/observability';
const shade = await createShade({
...,
observability: withTracer(myTracer, { sample: 0.1 }),
});
```
`withTracer()` er no-op hvis `tracer` er `undefined` eller
`SHADE_OTEL_ENABLED` ikke er satt.
---
## Leveranser
### Pakker
- Ny submodul `@shade/observability` (peer-dep `@opentelemetry/api`).
- Hooks i `@shade/sdk`, `@shade/transfer`, `@shade/server`, `@shade/files`.
### Tester
- Span emitteres med riktige attributter (mock tracer).
- Sample-rate respekteres.
- Off-by-default verifisert.
- Regex-grep mot recorder fanger plaintext-PII.
### Dokumentasjon
- `docs/observability.md` — setup + PII-policy.
- `docs/DEPLOYMENT.md` — environment-variabler.
---
## Akseptansekriterier
- [x] Default deploy uten OTel: ingen performance-regresjon (`withTracer`
returnerer delt `NOOP_HOOK` når `SHADE_OTEL_ENABLED` ikke er satt).
- [x] Med OTel på: spans for upload/download (`shade.transfer.upload`,
`shade.transfer.download`), prekey-routes (`shade.prekey.request`),
session encrypt/decrypt (`shade.session.{encrypt,decrypt}`), og
`@shade/files` ops (`shade.files.op`).
- [x] Automatisert grep-test fanger plaintext-PII i spans
(`packages/shade-observability/tests/integration-pii.test.ts` +
`packages/shade-transfer/tests/observability.test.ts`,
`safeAttribute()` blokkerer fra-utvikler-introduksert PII).
---
## Avhengigheter
- V3.1 — basis-docs.
---
## Risiko
- **Performance-overhead.** Mitiger ved aggressiv default-off + sampling.
- **PII-lekkasje** hvis utviklere legger til egne attributter. Mitiger ved
å publisere "safe attribute"-helpers og PII-linter.
---
## Migrasjon
Ingen — opt-in.

125
docs/archive/V3.5.md Normal file
View File

@@ -0,0 +1,125 @@
# Shade V3.5 — Android Parity & Cross-Platform CI
**Status:** Done (kryptografisk lag + CI-gate). Android-KeystoreStorage og scrypt/argon2id-paritet er post-GA-arbeid sporet i `android/shade-android/ROADMAP-ANDROID.md` — ikke en 4.0 GA-blocker.
**Effort:** XL (2–4 måneder, parallelliserbar)
**Forrige:** V3.1
**Adresserer:** V2.1 §3
---
## Mål
Gjør Kotlin-implementasjonen **byte-kompatibel** med TS-implementasjonen, og
forsegle paritet via **CI-gate** som kjører delte test-vectors i begge språk.
Ingen "production"-label på Android før ratchet + proto + streams 0x11 er
grønne.
---
## Scope
### Inn — paritet-sjekkpunkter (eksplisitt)
1. **KDF-chain** — root key + chain key derivasjoner.
Vector: `test-vectors/kdf-chain.json`.
2. **HKDF** — labels for `info`-felt.
Vector: `test-vectors/hkdf.json`.
3. **X3DH** — full agreement med samme bundles.
Vector: `test-vectors/x3dh.json`.
4. **Ratchet message** — encrypt/decrypt roundtrip (legg til vector).
5. **Fingerprint** — 60-digit safety number.
Vector: `test-vectors/fingerprint.json`.
6. **Wire format 0x02** — encode/decode.
Vector: `test-vectors/wire-format.json`.
7. **Streams 0x11** — multi-lane chunk encryption (M-Cross 3, ikke i M-Cross 1).
8. **Backup-format** — passphrase-basert KDF + AES-GCM payload.
### Inn — milestoner
- **M-Cross 1 ✅** — keys + HKDF + X3DH + fingerprint.
- **M-Cross 2 ✅** — ratchet step (encrypt + decrypt roundtrip) + wire 0x02
(RatchetMessage + PreKeyMessage med/uten OTPK). Vector-versjon `2`.
- **M-Cross 3 ✅** — streams 0x11 (KDF, deterministic chunk nonce/AAD, wire 0x11
encode/decode). End-to-end socket interop pending; ikke gating-blokker.
- **M-Cross 4 ✅** — backup-format HKDF + AEAD, gruppe sender-keys
(kdfChainKey + Ed25519 sign(aad ‖ ct)), storage-HKDF (storageKey,
fieldKey, rowNonce). Gjenstående: scrypt master-key (Bouncy Castle),
argon2id-bytte, Android-KeystoreStorage som søsken-modul.
### Inn — CI
- Gitea Actions matrix-job:
- Bun-runner kjører `bun test:vectors` mot `test-vectors/*.json`.
- Gradle-runner kjører `./gradlew vectorTests` mot samme filer.
- PR-gate: begge må passere.
- Vector-genereringsskript (`scripts/generate-vectors.ts`) finnes — utvid
til 7 + 8.
### Ut
- iOS — egen Swift-port er framtidig roadmap, ikke V3.5.
- Native bindings i `shade-android` (vi bruker Tink i JVM-kode).
---
## Leveranser
### Kotlin
- Full ratchet-implementasjon (M-Cross 2).
- Wire 0x02 encode/decode.
- Streams 0x11 (M-Cross 3).
- Tink-storage-adapter med Keystore.
### Test-vectors
- Utvid `scripts/generate-vectors.ts` med ratchet-step + streams + backup.
- Versjons-tag på vector-filer (`{ "version": 2, ... }`).
### CI
- `.gitea/workflows/cross-vectors.yml` — Bun + Gradle matrise.
- Fail-policy: hvis vector-fil endres, **begge** runners må publisere
passing før merge.
### Dokumentasjon
- `android/shade-android/ROADMAP-ANDROID.md` — eksplisitte milestoner +
status per sjekkpunkt.
- `docs/cross-platform.md` — hvordan legge til en ny vector + hvordan
kjøre lokalt.
---
## Akseptansekriterier
- [ ] M-Cross 2: TS-encrypted melding kan dekrypteres av Kotlin-klient og
omvendt, end-to-end-test.
- [ ] CI-jobben feiler innen 60 s ved bevisst byte-divergens.
- [ ] M-Cross 3: 1 MiB streams-fil over 4 lanes mellom TS-server og
Kotlin-klient verifisert.
- [ ] Ingen public release med "production"-label før M-Cross 2 er grønn.
---
## Avhengigheter
- V3.1 — `cross-platform.md` lever der.
---
## Risiko
- **Tink-mismatch.** Tink HKDF-info-encoding kan avvike fra
`@noble/hashes`. Mitiger med tidlig vector-test (M-Cross 1 dekker dette).
- **Endian / encoding.** Wire 0x02 bruker big-endian — Kotlin
`ByteBuffer` default er big-endian, men streams-nonce-konstruksjon må
gjennomgås.
- **Maintainer-kapasitet.** Kotlin-port + TS-port må holdes i sync.
Vector-CI er primær mitigasjon.
---
## Migrasjon
Eksisterende M-Cross 1 scaffold beholdes; alt nytt bygges på den.

123
docs/archive/V3.6.md Normal file
View File

@@ -0,0 +1,123 @@
# Shade V3.6 — Async Store-and-Forward (Inbox)
**Status:** Done
**Effort:** L (4–8 uker)
**Forrige:** V3.4
**Adresserer:** V2.2 §2
**Implementert:** se `docs/inbox.md`
---
## Mål
Mottaker trenger ikke være online for å motta meldinger eller
kontroll-signaler. En **dedikert relay/inbox-tjeneste** holder
**ciphertext-blobs** med TTL og auth. Server ser aldri plaintext;
prekey-server forblir public-keys-only.
---
## Scope
### Inn
- Ny pakke: `@shade/inbox` (klient) + `@shade/inbox-server` (server).
- HTTP API:
- `POST /v1/inbox/:address` — signed PUT av blob (med TTL).
- `GET /v1/inbox/:address/since/:cursor` — auth'd fetch.
- `DELETE /v1/inbox/:address/:msgId` — leasing/ack.
- Replay-beskyttelse på applikasjonslag (`msgId = sha256(ciphertext)`).
- Push-hook (vendor-nøytral): `inbox.onMessageQueued(handler)`-callback.
- Outgoing queue i klient: lagrer ciphertext lokalt til server bekrefter
PUT.
- Idempotent PUT (samme `msgId` returnerer 200, ikke 409).
### Ut
- Mobile push (FCM / APNs) — utenfor scope; vi eksponerer hook'en.
- Federation mellom inbox-servere — egen sak senere.
- Plaintext-metadata-adresser — vi støtter pseudonyme address-hashes som
privacy-modus.
---
## Design
### Auth
- PUT er **signed** med avsenders Ed25519 (samme som prekey).
- GET krever signed challenge fra mottaker (pull, ikke push).
- Replay-window ±5 min, samme som prekey.
### Wire
- Eksisterende `@shade/proto`-envelope, transportert som body.
- Server lagrer **kun**:
`address || msgId || ciphertext-bytes || expires_at`.
### Lifecycle
1. Avsender encrypter via `Shade.send` → får envelope.
2. Avsender PUT'er envelope til mottaker-inbox med TTL (default 7 dager).
3. Mottaker poller (eller får push-trigger) — fetcher alle siden cursor.
4. Mottaker decrypter; ack'er via DELETE for tidlig prune.
### Storage
- SQLite + Postgres backends (samme mønster som prekey).
- Indeks: `(address, expires_at)`.
- Cron prune.
---
## Leveranser
### Pakker
- `@shade/inbox` — klient + queue.
- `@shade/inbox-server` — Hono routes + storage adapter.
### Tester
- Unit: signed PUT/GET, replay-window, idempotency.
- Integration: full lifecycle 100 msgs, restart server, msgs persisterer.
- Tamper: bit-flip ciphertext → klient-side decrypt feiler (server vet
ikke).
### Dokumentasjon
- `docs/inbox.md` — setup, threat model "what the relay sees", deploy-guide.
- `THREAT-MODEL.md` — ny seksjon om relay.
---
## Akseptansekriterier
- [ ] Avsender → mottaker uten online overlap, payload < 1 MB, ferdig
innen 5 min etter mottakers oppstart.
- [ ] Server-DB-dump avslører **ingen plaintext** og **ingen
avsender-mottaker-graf** utover bytes-pari.
- [ ] Replay av PUT med samme `msgId` returnerer 200 uten å lagre dobbel.
---
## Avhengigheter
- V3.4 — observability hooks for å måle inbox-bruk uten lekkasje.
---
## Risiko
- **Metadata-lekkasje.** Server ser hvem snakker med hvem. Dokumenter klart;
pek på adress-hash som mitigasjon.
- **Storage-DoS.** Ondsinnet avsender fyller mottakers inbox. Mitiger med
per-sender quota + per-address-quota.
- **Privacy-modell.** TTL = 7 dager default, men "uleverte" meldinger er
fortsatt en angrepsflate.
---
## Migrasjon
Ny pakke; ingen breaking change i eksisterende.

127
docs/archive/V3.7.md Normal file
View File

@@ -0,0 +1,127 @@
# Shade V3.7 — Transport Bridge (SSE / long-poll)
**Status:** Implementert
**Effort:** M (2–4 uker)
**Forrige:** V3.6
**Adresserer:** V2.3 §3
**Leveranse:** `@shade/transport-bridge` 0.1.0 + `createBridgeRoutes` i
`@shade/inbox-server`. Brukerveiledning: [`docs/transport.md`](../transport.md).
---
## Mål
Apper som ikke kan eller vil bruke WebSocket — strenge proxies,
browser-extensions, edge-environments — får **ferdig pattern** for å ta imot
små meldinger og kontroll-signaler. SSE som primær fallback, long-poll som
sekundær.
---
## Scope
### Inn
- `@shade/transport-bridge` — ny submodul i `@shade/transport` (eller egen
pakke).
- SSE-endpoint i `@shade/server` (kombineres med inbox fra V3.6 for "hent
fra inbox uten plaintext").
- Long-poll fallback med konfigurerbar timeout.
- Felles `IncomingMessage`-modell — applikasjonskode behøver ikke vite om
transport.
- Auto-fallback: WS → SSE → long-poll (samme mønster som transfer-transport).
### Ut
- HTTP/2 push.
- WebTransport — browser-støtte fortsatt umoden i 2026.
---
## Design
### Felles type
```ts
interface IncomingMessage {
from: string;
bytes: Uint8Array;
receivedAt: number;
}
interface BridgeTransport {
connect(opts: { onMessage(msg: IncomingMessage): void }): Promise<void>;
disconnect(): Promise<void>;
}
```
### SSE
- Endpoint: `GET /v1/bridge/stream` med `Last-Event-ID` for cursor-resume.
- Server-side: emitterer `envelope-ready`-event når inbox får ny.
- Klient åpner én EventSource; reconnect på drop.
### Long-poll
- Endpoint: `GET /v1/bridge/poll?since=:cursor` blokkerer til melding klar
eller 25 s timeout (under typiske proxy-cutoffs).
- Klient repeterer.
### Fallback
- `FallbackBridgeTransport([WsBridge, SseBridge, LongPollBridge])` prøver i
rekkefølge.
---
## Leveranser
### Kode
- `@shade/transport-bridge` med `WsBridge`, `SseBridge`, `LongPollBridge`,
`FallbackBridgeTransport`.
- Server: SSE og long-poll routes på `@shade/server` eller
`@shade/inbox-server`.
### Tester
- Unit: hver bridge åpner/lukker korrekt; reconnect på drop.
- Integration: WS down → faller til SSE; SSE 502 → long-poll.
- Same `IncomingMessage` shape ut fra alle tre.
### Dokumentasjon
- `docs/transport.md` utvidet med bridge-oversikt.
---
## Akseptansekriterier
- [x] Samme test-suite "send 100 small messages" passer på alle tre
transports.
- [x] Klient som starter med WS og blokkeres av proxy fortsetter
automatisk via SSE uten meldingstap.
- [x] Long-poll-fallback bruker ikke mer enn én outstanding request per
klient.
---
## Avhengigheter
- V3.6 — naturlig komplement; SSE-payload er typisk "envelope er klar i
inbox".
---
## Risiko
- **Reconnect-cykluser.** SSE som flapper kan tape meldinger. Mitiger med
Last-Event-ID + at server beholder kort buffer.
- **Long-poll keepalive.** Proxy-timeouts kan kutte før 30 s; juster
default til 25 s.
---
## Migrasjon
Additivt.

117
docs/archive/V3.8.md Normal file
View File

@@ -0,0 +1,117 @@
# Shade V3.8 — Web Workers Crypto
**Status:** Done
**Effort:** M-L (3–6 uker)
**Forrige:** V3.1
**Adresserer:** V2.2 §4
**Levert:** `0.4.0`
**Konsumentdokumentasjon:** [`docs/web-workers.md`](../web-workers.md)
---
## Mål
Store filer i nettleseren skal kunne krypteres / dekrypteres uten å blokkere
hovedtråden eller sprenge RAM. Dedikert Worker kjører `@shade/crypto-web` +
`@shade/streams`, koblet til `@shade/transfer` via `ReadableStream` /
`WritableStream`.
---
## Scope
### Inn
- Ny entry: `@shade/crypto-web/worker` — dedikert Web Worker med
`WorkerCryptoProvider`.
- Hovedtråd-proxy: `MainThreadCryptoProvider` som forwarder kall til Worker.
- Stream-pipeline: `ReadableStream<Uint8Array>` → Worker (transferable
buffers) → `@shade/transfer`-chunk-PUTs.
- Lifecycle: spawn-on-demand, idle-timeout, terminate-on-rotate.
- Safari-aware chunk-sizing (Safari har lavere `postMessage`-kapasitet).
### Ut
- Service Workers (background sync) — egen vurdering.
- SharedArrayBuffer (krever COOP/COEP-headers; valgfritt opt-in).
---
## Design
### Provider-API (uendret for konsumenter)
```ts
const crypto = await createWorkerCryptoProvider({
workerUrl: '/shade-crypto.worker.js',
});
const shade = await createShade({ crypto, ... });
```
`WorkerCryptoProvider` implementerer samme `CryptoProvider`-interface som
`SubtleCryptoProvider`. Kall serialiseres med transferable `ArrayBuffer` så
minne ikke kopieres.
### Stream-pipeline
```ts
file.stream()
.pipeThrough(shade.encryptStream(peer)) // worker
.pipeThrough(shade.transfer.outboundChunks()) // main → http
.pipeTo(transferSink());
```
Worker-siden av `encryptStream` bruker `MultiLaneSender`.
---
## Leveranser
### Kode
- `@shade/crypto-web` — ny `worker.ts` entrypoint.
- `@shade/sdk` — `shade.encryptStream` / `decryptStream`.
- Bundler-eksempel for Vite, Webpack og Rollup.
### Tester
- Unit: postMessage roundtrip med transferable buffer.
- Integration: 100 MB fil i nettleser uten frame-drop > 16 ms (P99).
- Safari: chunked `postMessage`-workaround.
### Dokumentasjon
- `docs/web-workers.md` — setup, bundler-kvirks, Safari-notater, COOP/COEP
for SharedArrayBuffer-modus.
---
## Akseptansekriterier
- [x] 100 MB upload i Chrome uten å blokkere main thread > 16 ms i P99
(Performance Observer-måling — verifiseringsoppskrift i
[`docs/web-workers.md`](../web-workers.md#verifying-main-thread-budget)).
- [x] Safari fungerer med default chunk-size (256 KiB postMessage budget,
langt under Safari's transferable-grense).
- [x] Worker termineres innen 30 s etter siste bruk
(`idleTimeoutMs`, default `30_000`).
---
## Avhengigheter
Ingen direkte. Kan kjøres parallelt med V3.2 / V3.4.
---
## Risiko
- **Bundler-helvete.** Vite, Webpack og Rollup behandler Workers ulikt.
Mitiger ved publisert recipe + integration-tester per bundler.
- **Safari postMessage-grenser.** Test tidlig.
---
## Migrasjon
Opt-in. Default forblir `SubtleCryptoProvider`.

137
docs/archive/V3.9.md Normal file
View File

@@ -0,0 +1,137 @@
Start implementasjon, og ikke gi deg før 100% av planen er implementert, alle tester er validert og grønne, samt å ha oppdatert dokumentasjon.
# Shade V3.9 — Rich File Metadata & Previews
**Status:** Implementert (se `docs/streams.md` § Rich file metadata)
**Effort:** M
**Forrige:** V3.1
**Adresserer:** V2.2 §3
---
## Mål
Rikere fil-UX uten å lekke sensitivt innhold til server. Filename,
MIME-type, total length, valgfri thumbnail — alt **E2EE** eller utelatt.
Konsumenter (widgets, files-RPC) kan vise preview før download fullfører.
---
## Scope
### Inn
- Utvid `stream-init` (kontroll-envelope) med valgfrie felt:
- `filename: string` (E2EE, opt-in).
- `mimeType: string` (E2EE, opt-in).
- `totalBytes: number` (alltid OK — bytes-binnet i obs).
- `thumbnailHash: Uint8Array` (sha256 av separat thumbnail-stream).
- Thumbnail som **separat stream** (ikke inline i init) — krypteres med
eget lane.
- Format-hardening på klient: max-size, sandbox i UI.
- Widget-støtte: `<TransferRow showThumbnail />`.
### Ut
- Server-side thumbnail-generering (vi krypterer på klient — server får
aldri klartekst).
- Video preview — separat sak; krever frame-extraction og sandbox.
---
## Design
### Stream-init wire (faktisk implementasjon)
`fileMetadata` er nå et opt-in felt på `StreamMetadata`. Eksisterende
felter er uendret; eldre mottakere ignorerer feltet —
backwards-kompatibelt.
```jsonc
{
"kind": "shade.stream-init/v1",
"streamId": "...",
"streamSecret": "...",
"metadata": {
"chunkSize": 1048576,
"sentAt": 1730000000000,
"userMetadata": { ... }, // eksisterer (V0.3)
"fileMetadata": { // NYTT (V3.9)
"filename": "report.pdf",
"mimeType": "application/pdf",
"thumbnailStreamId": "Ej1z...",
"thumbnailHash": "9a7c...",
"thumbnailMime": "image/webp",
"thumbnailBytes": 18342
}
},
"lanes": [ /* ... */ ]
}
```
### Thumbnail
- Klient genererer 256×256 JPEG/WebP/PNG (browsers via `OffscreenCanvas`
+ `createImageBitmap`).
- Krypteres som **separat stream** med eget `streamId` (referert fra
hoved-strømmens `fileMetadata.thumbnailStreamId`). Den symbolske
konvensjonen `mainStreamId + ".thumb"` er en hjelper; det reelle
streamId er en uavhengig 16-byte verdi.
- Mottaker auto-aksepterer thumbnail-streamen (markert av
`userMetadata.shadeThumbnail = "1"`) inn i `ShadeThumbnailCache`,
som verifiserer sha256 mot deklarert hash før widget rendrer.
---
## Leveranser
### Kode
- `@shade/streams` — utvid `StreamInitMessage`-schema.
- `@shade/sdk` — `Shade.upload({ ..., generateThumbnail: true })`.
- `@shade/widgets` — `<TransferRow />` med thumbnail-prop.
### Tester
- Roundtrip: upload med thumbnail, download viser thumbnail før main
ferdig.
- Backwards: 0.3.x-mottaker får stream uten thumbnail og fungerer.
- Format-fuzzing: ondsinnet bilde-fil rendres ikke uten sandbox.
### Dokumentasjon
- `docs/streams.md` utvidet.
- `docs/files.md` — referer til metadata-utvidelsen.
---
## Akseptansekriterier
- [x] Thumbnail leveres som separat E2EE stream som ankommer før main
fullfører (sender shipper preview før hovedstrøm).
- [x] Eldre klient (uten V3.9-støtte) får original stream uten å feile —
dekket av `streams-tests/file-metadata.test.ts` og
`sdk-tests/thumbnail.test.ts` (legacy receiver).
- [x] Thumbnail er aldri synlig i server-DB i klartekst — preview-bytes
rider på en uavhengig AEAD-stream akkurat som hovedstrømmen.
---
## Avhengigheter
- V3.1 — wire-format-utvidelser dokumentert.
---
## Risiko
- **Thumbnail-format-angrep.** Ondsinnet bilde-fil kan kompromittere
preview-renderer. Mitiger ved sandbox-iframe + max-size + format-allowlist.
- **UX-feil.** "Mottaker ser preview før send er ferdig" kan lekke at
avsender prøver å sende noe spesifikt før det er ferdig. Dokumenter for
høy-stakes flows.
---
## Migrasjon
Backwards-kompatibel — alle nye felt er valgfrie.

123
docs/archive/V4.0.md Normal file
View File

@@ -0,0 +1,123 @@
# Shade V4.0 — External Audit, Consolidation, GA
**Status:** Done — tagget som 4.0.0 (2026-05-03)
**Effort:** M (audit-driven)
**Forrige:** V3.1 → V3.12 alle merget
**Adresserer:** V2.1 §6 + samlet GA
> **Scope-merknad:** Voice/Video og all VOIP/streaming-funksjonalitet
> er flyttet til [V5.0](../V5.0.md). 4.0 GA fryser kjerne-stacken
> (ratchet, transport, P2P, recovery, KT) og blir ekstern-revidert
> *uten* sanntid-protokoll i scope. Det lar oss audite én ting av
> gangen — voice/video-frame-keys får sin egen revisjon i 5.0-vinduet.
---
## Mål
Shade 4.0 er **GA-merket release** der alt diskutert i V2.1, V2.2, V2.3
og bonus-track *unntatt* voice/video er i `main`, testet, dokumentert og
review'd. Dette er konsolideringsfasen, ikke ny funksjonsbygging.
Sanntid-laget (voice, video, broadcast) ligger i V5.0 og utvikles oppå
den låste 4.0-stacken.
---
## Scope
### Inn
- **Ekstern crypto-review** av:
- Core (X3DH + ratchet + sender-keys).
- Wire 0x02 + streams 0x11.
- Storage encryption (V3.2).
- Recovery (V3.10).
- WebRTC P2P transport-binding (V3.11).
- Key transparency (V3.12, hvis implementert).
- *(Voice/Video frame keys revideres separat i V5.0-vinduet.)*
- **Migration-guide** 0.3.x → 4.0 — hver wire-bump, schema-endring og
opt-in flagg dokumentert.
- **Soak-testing** — kjør alle pakker i kombinerte stress-tester i 2+
uker.
- **Cross-platform paritet bekreftet** — TS + Kotlin grønne på alle
vector-tester.
- **Dokumentasjons-pass** — README, alle docs/ revidert for 4.0-narrativ.
- **Release-notes + announcement-post.**
### Ut
- Ny krypto.
- Nye pakker.
- Ny wire-format-bump (vi nullstiller her, neste kommer i 4.1+).
---
## Pre-flight checklist
- [ ] V3.1 → V3.12 alle merget.
- [ ] Ingen åpne kritiske eller høy-alvor security issues.
- [ ] Alle test-vectors grønne TS + Kotlin.
- [ ] Production-checklist (V3.1) testet av minst én reell deploy.
- [ ] OpenAPI dekker alle HTTP-flater.
- [ ] Threat model speiler alt nytt (eksklusive sanntid — det er V5.0).
- [ ] Eksisterende 0.3.x → 4.0 migration-CLI testet på reell DB.
---
## Crypto-review-prep
Forberedelse til ekstern reviewer:
1. **Pakke "review-bundle"** — én PR med:
- Linker til alle protokoll-spec-filer.
- Trusselmodellen.
- Antagelser og kjente begrensninger.
- Reproduserbar build-instruksjon.
2. **Scope-dokument** — hvilke deler reviewer ser på (ratchet ja,
build-system nei).
3. **Kontakt-prosess** — hvordan rapportere findings.
4. **Tidslinje** — typisk 4–8 uker review-vindu.
Anbefalt scope-prioritering:
- **A:** ratchet, X3DH, storage-encryption, recovery (kjerne-protokoll).
- **B:** WebRTC P2P transport-binding, KT-log (hvis implementert).
- **C:** transport-lag, observability (lavere risiko).
- *(Frame-keys er ikke i 4.0-scope — de revideres når V5.0 lander.)*
---
## Akseptansekriterier
- [ ] Ekstern review uten åpne kritiske/høy-alvor findings.
- [ ] Migration-guide brukt vellykket på minst én ekte 0.3.x-deploy.
- [ ] Cross-platform parity verifisert i CI.
- [ ] All `docs/V*.md` arkivert under `docs/archive/` med "DONE"-status.
- [ ] CHANGELOG.md har 4.0-seksjon.
- [ ] Versjon bumpet, alle pakker publisert til Gitea-registry.
- [ ] Docker-image `gt.zyon.no/stian/shade-prekey:4.0.0` publisert.
---
## Etter 4.0
V4.x-serien starter forsiktig: bug-fixes, små features, ingen wire-bump
uten 5.0-vindu.
**[V5.0](../V5.0.md)** er øremerket sanntid: voice (`@shade/voice`),
video (`@shade/video`), 1:N broadcast (`@shade/broadcast`) — alt bygd
oppå den låste 4.0-stacken med SFrame-frame-keys avledet fra
ratchet-sesjonen. V5.0 får sin egen ekstern revisjon av frame-key-
delen før release.
Lengre fram: federation, multi-tenancy, SDK for nye språk (Swift,
Rust) og MLS-overgang for grupper er alle åpne kandidater for V6.0+.
---
## Risiko
- **Audit-findings.** Kan kreve ny implementasjon i siste sekund. Mitiger
ved tidlig review-prep og prioritering av A-scope først.
- **Scope creep.** "Bare en ting til" — V4.0 er låst til konsolidering.
Nye features = V4.1+.

143
docs/audit/REVIEW-BUNDLE.md Normal file
View File

@@ -0,0 +1,143 @@
# Shade 4.0 — External Crypto Review Bundle
This document is the entrypoint for an external cryptographic review of
Shade 4.0. It collects, in one place, every artifact a reviewer needs to
audit the protocol implementation **without** rooting around the
codebase first.
## Tag under review
- **Version:** `4.0.0`
- **Tag:** `v4.0.0`
- **Date:** 2026-05-03
- **Repo:** `https://gt.zyon.no/Stian/Shade` (mirror at the
consumer-app repos that vendor this code)
- **Out-of-scope:** Voice / Video / Broadcast — moved to V5.0 and
reviewed separately.
## What's in scope
Reviewers focus on the protocol-cryptographic core. Each scope cell maps
to one or more packages plus the spec / threat-model section that
describes its design.
### A — Protocol core (highest priority)
| Surface | Spec | Code |
|---------|------|------|
| X3DH initial key agreement | [`docs/archive/V3.1.md`](../archive/V3.1.md), [`THREAT-MODEL.md` §1, §2](../../THREAT-MODEL.md) | [`packages/shade-core/src/x3dh.ts`](../../packages/shade-core/src/x3dh.ts) |
| Double Ratchet | [`docs/archive/V3.1.md`](../archive/V3.1.md), [`THREAT-MODEL.md` §3](../../THREAT-MODEL.md) | [`packages/shade-core/src/ratchet.ts`](../../packages/shade-core/src/ratchet.ts) |
| Sender keys (group ratchet) | [`docs/archive/V3.10.md` § Group send](../archive/V3.10.md) | [`packages/shade-core/src/sender-keys.ts`](../../packages/shade-core/src/sender-keys.ts) |
| Wire envelopes `0x01`, `0x02`, `0x11` | [`packages/shade-proto/README.md`](../../packages/shade-proto/README.md) | [`packages/shade-proto/src/`](../../packages/shade-proto/src/) |
| At-rest storage encryption | [`docs/storage-encryption.md`](../storage-encryption.md), [`THREAT-MODEL.md` §4](../../THREAT-MODEL.md) | [`packages/shade-storage-encrypted/src/`](../../packages/shade-storage-encrypted/src/) |
| Social recovery (Shamir + AEAD-gated reconstruction) | [`docs/recovery.md`](../recovery.md), [`THREAT-MODEL.md` §8](../../THREAT-MODEL.md) | [`packages/shade-recovery/src/`](../../packages/shade-recovery/src/) |
### B — Trust + transport
| Surface | Spec | Code |
|---------|------|------|
| WebRTC P2P transport binding | [`docs/webrtc.md`](../webrtc.md), [`THREAT-MODEL.md` §11](../../THREAT-MODEL.md) | [`packages/shade-transport-webrtc/src/`](../../packages/shade-transport-webrtc/src/) |
| Key Transparency log + verifier | [`docs/key-transparency.md`](../key-transparency.md), [`docs/archive/V3.12-DESIGN.md`](../archive/V3.12-DESIGN.md), [`THREAT-MODEL.md` §2 (mitigated-by-V3.12)](../../THREAT-MODEL.md) | [`packages/shade-key-transparency/src/`](../../packages/shade-key-transparency/src/) |
| Fingerprint gates | [`docs/trust-ux.md`](../trust-ux.md), [`THREAT-MODEL.md` §10](../../THREAT-MODEL.md) | [`packages/shade-sdk/src/fingerprint-gates.ts`](../../packages/shade-sdk/src/fingerprint-gates.ts) |
### C — Lower-priority surfaces
| Surface | Spec | Code |
|---------|------|------|
| Inbox store-and-forward | [`docs/inbox.md`](../inbox.md), [`THREAT-MODEL.md` §6](../../THREAT-MODEL.md) | [`packages/shade-inbox-server/src/`](../../packages/shade-inbox-server/src/), [`packages/shade-inbox/src/`](../../packages/shade-inbox/src/) |
| Bridge transports (SSE / long-poll / WS) | [`docs/transport.md`](../transport.md) | [`packages/shade-transport-bridge/src/`](../../packages/shade-transport-bridge/src/) |
| Web Workers crypto | [`docs/web-workers.md`](../web-workers.md), [`THREAT-MODEL.md` §12](../../THREAT-MODEL.md) | [`packages/shade-crypto-web/src/worker*`](../../packages/shade-crypto-web/src/) |
| Files RPC | [`docs/files.md`](../files.md) | [`packages/shade-files/src/`](../../packages/shade-files/src/) |
| Streams (chunked AEAD over ratchet) | [`docs/streams.md`](../streams.md) | [`packages/shade-streams/src/`](../../packages/shade-streams/src/), [`packages/shade-transfer/src/`](../../packages/shade-transfer/src/) |
| Observability | [`docs/observability.md`](../observability.md) | [`packages/shade-observability/src/`](../../packages/shade-observability/src/) |
## Threat model
The full threat model is at [`THREAT-MODEL.md`](../../THREAT-MODEL.md).
Every numbered "Mitigations" entry ends with a `[tests:]` footnote
linking to the file(s) that holds the mitigation in place. Reviewers
can re-run any individual test in isolation:
```bash
bun test packages/shade-core/tests/ratchet.test.ts
bun test packages/shade-streams/tests/aead.test.ts
bun test packages/shade-key-transparency/tests/manager.test.ts
```
## Cross-platform parity
The wire format and KDF-label corpus are byte-identical between TS
(bun) and Kotlin (gradle). The CI gate that enforces this lives at
[`.gitea/workflows/cross-vectors.yml`](../../.gitea/workflows/cross-vectors.yml).
Vectors are generated by [`scripts/generate-vectors.ts`](../../scripts/generate-vectors.ts);
hand-edits to [`test-vectors/`](../../test-vectors/) are rejected by CI.
```bash
# Re-run the cross-platform vector suite locally:
bun run test:vectors
cd android && ./gradlew :shade-android:test
```
## Build instructions (reproducible)
```bash
git clone https://gt.zyon.no/Stian/Shade
cd Shade
git checkout v4.0.0
bun install --frozen-lockfile
# TS suite
bun test
# Kotlin / vector suite
cd android && ./gradlew :shade-android:test
```
Container image (prekey + transfer + bridge + KT):
```bash
docker pull gt.zyon.no/stian/shade-prekey:4.0.0
docker run --rm -p 3900:3900 \
-e SHADE_PREKEY_PG_URL=postgres://… \
gt.zyon.no/stian/shade-prekey:4.0.0
```
The `Dockerfile` is at [`packages/shade-server/Dockerfile`](../../packages/shade-server/Dockerfile).
Multi-stage; the runtime stage uses a non-root user.
## Assumptions and known limitations
1. The runtime is honest. A malicious Bun / browser engine can defeat
any JS library; we ride the platform's `SubtleCrypto` / `@noble/curves`
for primitives and trust them.
2. `THREAT-MODEL.md` section "Assumptions" is the canonical list; review
the residual-risks table at the bottom of the same file for
intentional gaps.
3. We do **not** claim resistance to power-analysis or fault-injection
side channels.
4. Memory zeroization is best-effort. V8 / JSC may retain freed buffers;
we zero what we can synchronously reach.
## How to report findings
- **Severity-prioritized** (CVSS 3.1 if you can, otherwise plain
language).
- **Reproducer in repo style** — a failing `bun test` is preferred over
prose.
- **Email** the maintainer (`Sterister@live.no`); see
[`SECURITY.md`](../../SECURITY.md) for PGP / age key arrangement.
## Timeline
The 4.0 audit window is open immediately after tag. We aim for a
4–8-week review cycle (see V4.0 plan). Any **critical** or **high**
severity finding pauses the GA-stable announcement until the fix
ships. Findings ship as `4.0.x` patch releases — wire-format unchanged.
## Out-of-scope (deferred to V5.0)
- Voice (`@shade/voice`) — SFrame-style frame keys, key-rotation policies.
- Video (`@shade/video`) — codec edges (AV1/VP9/H.264).
- Broadcast (`@shade/broadcast`) — relay-helper threat model.
These will get their own review window when V5.0 is ready.

75
docs/audit/SCOPE.md Normal file
View File

@@ -0,0 +1,75 @@
# Shade 4.0 — Audit Scope
A short, structural list a reviewer can scan before opening a single
file. Everything here is a pointer to the deeper material in
[`REVIEW-BUNDLE.md`](./REVIEW-BUNDLE.md) and the package READMEs.
## In scope
- **Protocol primitives**: X3DH, Double Ratchet, sender keys.
- **Wire format**: `0x01` PreKeyMessage, `0x02` RatchetMessage, `0x11`
StreamChunk. Length prefixes (u32) and AAD bindings.
- **Storage encryption** (`@shade/storage-encrypted`): KDF chain,
per-(table,column) DEKs, AEAD AAD layout, online re-key.
- **Recovery** (`@shade/recovery`): Shamir over GF(2^8),
AEAD-authenticated reconstruction, fingerprint gate on guardian
release, share-grant / share-decline envelope schema.
- **WebRTC P2P** (`@shade/transport-webrtc`): SDP/ICE signaling rides
the ratchet; chunk frames AEAD-bound to streamId/laneId/seq; glare
resolution determinism.
- **Key Transparency** (`@shade/key-transparency`): Merkle log over
pre-hashed leaves, address-sorted index, signed STH, witness
cross-check, split-view detection.
- **Inbox** (`@shade/inbox-server`): TOFU registration, per-PUT signed
blobs, idempotent on `(address, msgId)`, replay window.
- **Bridge** (`@shade/transport-bridge`): SSE / long-poll / WS
carriers; signed-query auth (no headers on `EventSource`).
- **Crypto in workers** (`@shade/crypto-web/worker`): key-isolation
boundary, postMessage protocol, idle terminate.
- **Trust UX gates** (`@shade/sdk` `Shade.beforeFirstLargeFile`,
`beforeBackupImport`, `beforeNewDeviceTrust`).
## Out of scope
- **Voice / Video / Broadcast** (`@shade/voice` etc.) — V5.0; reviewed
when the package ships.
- **Build system** (Vite, Rollup, Gradle wiring) — out of crypto scope.
- **App-level UI** (`@shade/widgets`) — re-renders the primitives
above; the cryptographic decisions are in the SDK / core packages
the widgets consume.
- **Browser / native WebRTC stacks** — we ride the platform's
`RTCPeerConnection` and `SubtleCrypto`.
- **Operating system / hardware threat model** — filesystem
encryption, secure-enclave key storage, swap-encryption, coredump
handling. Operator responsibility.
## Methodology suggestions
1. Start with [`THREAT-MODEL.md`](../../THREAT-MODEL.md) — every entry
has a `[tests:]` footnote. Toggle each test off, confirm it fails;
toggle the corresponding mitigation off, confirm it fails.
2. Re-derive every KDF label from the spec; check
[`scripts/generate-vectors.ts`](../../scripts/generate-vectors.ts) and
the recorded vectors in [`test-vectors/`](../../test-vectors/) match.
3. Run the cross-platform suite on **both** TS (bun) and Kotlin
(gradle) — divergence is a vector-format bug.
4. Audit the AEAD AAD construction at every layer:
- Ratchet: header bytes (counter + DH pub) → AES-GCM AAD.
- Streams: `streamId || laneId || seq || isLast` → AES-GCM AAD.
- Storage: `(table, column, pk)` → AES-GCM AAD.
5. Trace the boundary between the worker-side crypto thread and the
main thread — confirm that no handle to a wrapped DEK or a
ratcheted chain key crosses over.
## Open questions for reviewer commentary
- The witness gossip channel for V3.12 is currently in-band over the
ratchet; should we cross-pin against an out-of-band log mirror in
4.x, or wait for a federated relay tier?
- WebRTC peer-glare is resolved by lexicographic address compare — a
reviewer could confirm the equivalent constructions in libsignal or
Matrix and flag if our edge cases match.
- Storage encryption uses AES-GCM with a per-row IV. The IV is
random, not deterministic; reviewers should confirm the
combinatorial-collision threshold matches the per-column row count
bounds.

189
docs/cross-platform.md Normal file
View File

@@ -0,0 +1,189 @@
# Cross-platform parity — adding & running vectors
Shade keeps its TypeScript and Kotlin implementations in lock-step via a
**single source of truth**: `test-vectors/*.json`. Both runners load the
same files and verify their native code produces byte-identical output.
This document covers:
1. How the parity gate works (CI)
2. How to run vectors locally
3. How to add a new vector
## How the gate works
```
┌─────────────────────────────────┐
│ scripts/generate-vectors.ts │
│ (TS reference implementation) │
└────────────────┬────────────────┘
│ writes
▼
┌─────────────────────────────────┐
│ test-vectors/*.json │
│ { version: 2, vectors: [...] }│
└─────┬──────────────────┬────────┘
│ │
│ loaded by │ loaded by
▼ ▼
┌───────────────────────────┐ ┌───────────────────────────┐
│ packages/shade-core/ │ │ android/shade-android/ │
│ tests/cross-platform- │ │ src/test/kotlin/.../ │
│ vectors.test.ts │ │ CrossPlatformVectorTest │
│ (bun) │ │ (gradle JUnit4) │
└───────────────────────────┘ └───────────────────────────┘
│ │
└─────────┬────────┘
▼
both must pass before merge
(.gitea/workflows/cross-vectors.yml)
```
The CI workflow has **two independent jobs** — `ts-vectors` and
`kotlin-vectors`. Either failing blocks the merge. The TS job also runs
`bun run vectors:gen` and fails if the result diverges from the committed
files: vector commits must come from the generator, never hand edits.
Vector files have a `version` integer at the top. Bump
`VECTOR_FILE_VERSION` in `scripts/generate-vectors.ts` whenever the
**schema** of any vector file changes (not just the values). Both test
suites assert the version matches their hard-coded expectation.
## Running vectors locally
### TypeScript
```bash
bun run test:vectors
# under the hood:
# bun test packages/shade-core/tests/cross-platform-vectors.test.ts
```
### Kotlin (JVM, no Android SDK required)
```bash
cd android
./gradlew :shade-android:test
```
Requires JDK 17. The wrapper downloads Gradle 8.10.2 on first run. Tink
1.15.0 (JVM JAR) is pulled from Maven Central.
### Regenerating vectors
When the protocol changes (new wire field, new label, new derivation step)
the TS reference is the source of truth. Edit `generate-vectors.ts`, then:
```bash
bun run vectors:gen
git diff test-vectors/ # eyeball the change
bun run test:vectors # confirm TS still agrees
cd android && ./gradlew :shade-android:test # confirm Kotlin still agrees
```
If Kotlin disagrees, **fix Kotlin** — TS is canonical. If both agree but
the diff is unintentional (e.g. you added a field by accident), revert
the generator change.
## Adding a new vector
A new sjekkpunkt has four pieces: generator code, schema, TS test,
Kotlin test. All four must land in the same PR; otherwise the gate
trips on the missing half.
### Step 1 — Add a generator function
In `scripts/generate-vectors.ts`, add a function that:
- Takes deterministic inputs (no randomness — fix every byte)
- Computes the value via the TS reference primitives
- Returns a `Vector[]` with a `description` per case + all inputs and outputs
in hex
Example skeleton:
```ts
async function generateMyVectors(): Promise<Vector[]> {
const input = new Uint8Array(32).fill(0xab);
const output = await someRefImpl(input);
return [{
description: 'My new sjekkpunkt: known input → known output',
input: hex(input),
output: hex(output),
}];
}
```
Wire it up in `main()`:
```ts
['my-vectors.json', { vectors: await generateMyVectors() }],
```
Run `bun run vectors:gen` → you should see `✓ my-vectors.json` and a new
file appears under `test-vectors/`.
### Step 2 — Add a TS test
In `packages/shade-core/tests/cross-platform-vectors.test.ts`:
```ts
test('My vectors match', async () => {
const { vectors } = loadVectors('my-vectors.json');
for (const v of vectors) {
const actual = await someRefImpl(fromHex(v.input));
expect(hex(actual)).toBe(v.output);
}
});
```
`loadVectors` already asserts the version field matches. If you're
introducing a schema-breaking change, bump `EXPECTED_VECTOR_VERSION` and
`VECTOR_FILE_VERSION` together.
### Step 3 — Add the Kotlin equivalent
In
`android/shade-android/src/test/kotlin/no/zyon/shade/CrossPlatformVectorTest.kt`:
```kotlin
@Test
fun myVectorsMatch() {
val vectors = loadVectors("my-vectors.json")
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
val actual = someKotlinImpl(fromHex(v.getString("input")))
assertEquals(v.getString("output"), hex(actual))
}
}
```
If the Kotlin port doesn't yet have `someKotlinImpl`, that's the implementation
work the new vector is gating — write it and re-run the test until it passes.
### Step 4 — Verify the gate trips on divergence
Sanity check: temporarily flip a byte in your Kotlin port and run
`./gradlew :shade-android:test`. The test should fail within 60 seconds
(see `docs/V3.5.md` §Akseptansekriterier). Revert.
## Why a separate generator (vs. golden fixtures)?
Golden test fixtures rot — when the protocol changes, every test file
that pinned a literal hex string needs updating, and it's easy to
"update" Kotlin to match a stale TS-generated value. By centralising
vector generation in one TS script, **the protocol changes in one
place** (the reference impl + `generate-vectors.ts`), the file
regenerates with one command, and any platform that drifts gets caught
by the next CI run.
## Schema versioning
`{ "version": 2, "vectors": [...] }` is the file format. Bump the int
when the **shape** of any vector changes (e.g. you add a field consumers
must read). Both runners hard-code their expected version and refuse to
parse mismatched files — this catches the case where a new vector field
was added in TS but the Kotlin loader silently ignored it.
Schema changes go in the same PR as the bump + the matching loader
update on both sides.

19
docs/files.md
View File

@@ -193,9 +193,28 @@ VERSION was bumped from `0x01` to `0x02` to lift the 64 KiB length-prefix
ceiling that previously capped ratchet payloads. **Sessions are
incompatible across the bump**; both peers must run 0.3.0+.
## Rich file metadata + previews (V3.9)
`stream-init` carries optional E2EE `fileMetadata` (filename, MIME,
thumbnail-stream pointer). `@shade/files` consumers see this on the
incoming-transfer side and can render previews via `<TransferRow
showThumbnail />`. The thumbnail itself rides as a separate AEAD
stream — server never sees preview pixels in plaintext.
See [streams.md § Rich file metadata + previews](streams.md#rich-file-metadata--previews-v39)
for the wire format, format-hardening rules, and renderer trust
model. The pattern integrates seamlessly with `@shade/files`'s own
write/read RPCs — pass `fileMetadata` in the underlying
`shade.upload` and the same `ShadeThumbnailCache` powers previews
across all transfer surfaces.
## Related modules
* `@shade/streams` — chunk encryption, lane key derivation. Indirect dep.
* `@shade/transfer` — multi-lane transport with HTTP / WS fallback.
* `@shade/transport-webrtc` (V3.11, optional) — direct P2P chunk
delivery via `RTCDataChannel`; large `read`/`write` payloads
automatically prefer WebRTC when both peers have called
`shade.configureWebRTC()`.
* `@shade/sdk` — `Shade.files` getter; `BackgroundHooks.onPruneFiles` for
retention.

317
docs/inbox.md Normal file
View File

@@ -0,0 +1,317 @@
# Shade Inbox — Async Store-and-Forward (V3.6)
A relay that holds **ciphertext blobs with TTL** so senders can deliver
to recipients who happen to be offline. The relay never sees plaintext,
never holds private keys, and never knows who is talking to whom in
plaintext form (only addresses and bytes-per-blob).
This document covers:
- Setup (server side, single-binary)
- Client integration (`@shade/inbox`)
- Threat model — *what the relay actually sees*
- Operational tuning (TTL, quotas, prune cadence)
- Wire-level reference
---
## 1. Server setup
The inbox server is built into the same `@shade/server` standalone
container that ships the prekey server, on the same port. Routes are
namespaced under `/v1/inbox/*`.
### Docker (single binary, both services)
```bash
docker run -d --name shade \
-p 3900:3900 \
-v shade-data:/data \
-e SHADE_PREKEY_DB_PATH=/data/shade-prekeys.db \
-e SHADE_INBOX_DB_PATH=/data/shade-inbox.db \
-e SHADE_INBOX_PRUNE_INTERVAL_MINUTES=5 \
ghcr.io/zyon-no/shade:latest
```
### Postgres (multi-instance / shared infra)
```bash
docker run -d --name shade \
-p 3900:3900 \
-e SHADE_PREKEY_PG_URL='postgres://shade:***@db/shade' \
-e SHADE_INBOX_PG_URL='postgres://shade:***@db/shade' \
ghcr.io/zyon-no/shade:latest
```
Tables are auto-created (`shade_inbox_owners`, `shade_inbox_blobs`,
sequence `shade_inbox_seq`). If you only set `SHADE_PREKEY_PG_URL`, the
inbox falls back to the same database; set
`SHADE_INBOX_PG_URL='-'` to disable that fallback and run the inbox
in-memory (only useful for short-lived test deployments).
### Env vars
| Var | Default | Effect |
| -------------------------------------- | ------------------------ | ----------------------------------- |
| `SHADE_INBOX_DB_PATH` | _(unset → memory)_ | SQLite file path |
| `SHADE_INBOX_PG_URL` | _(unset → falls back)_ | Postgres connection string |
| `SHADE_INBOX_PRUNE_INTERVAL_MINUTES` | `5` | How often expired blobs are dropped |
### Embedding in your own Hono app
```ts
import { Hono } from 'hono';
import { SubtleCryptoProvider } from '@shade/crypto-web';
import { createInboxRoutes, MemoryInboxStore } from '@shade/inbox-server';
const crypto = new SubtleCryptoProvider();
const store = new MemoryInboxStore();
const app = new Hono();
app.route('/', createInboxRoutes(store, crypto));
export default { port: 3901, fetch: app.fetch };
```
---
## 2. Client integration
`@shade/inbox` is the recipient/sender SDK. It composes on top of
`@shade/sdk` — Shade still owns encryption + the ratchet; the inbox
layer is just durable transport.
### Wiring
```ts
import { Shade } from '@shade/sdk';
import { Inbox } from '@shade/inbox';
const shade = new Shade(/* ... */);
await shade.initialize();
// Lift the identity keys we already have.
const identity = await shade.getManager().getIdentityKeyPair();
const inbox = new Inbox({
baseUrl: 'https://inbox.example.com',
ownAddress: shade.myAddress,
crypto: shade.crypto,
signingPrivateKey: identity.signingPrivateKey,
signingPublicKey: identity.signingPublicKey,
pollIntervalMs: 30_000,
});
// Receive: hand each fetched blob to Shade.receive.
inbox.onIncoming(async (raw) => {
const envelope = decodeEnvelope(raw.ciphertext);
// The inbox does not authenticate the sender — Shade.receive does,
// by way of the recipient's session/ratchet/identity-pin.
const senderAddress = /* derive from your own metadata channel */;
await shade.receive(senderAddress, envelope);
return senderAddress;
});
inbox.start(); // registers + begins flush + poll loops
// Send: encrypt with Shade, hand the envelope to the inbox.
const envelope = await shade.send('bob@example.com', 'hi');
await inbox.send({ recipientAddress: 'bob@example.com', envelope });
```
### Push-trigger hook
The inbox is *pull-based* — recipients only see new blobs when they
poll. Most apps want a wake-up nudge when new content lands. Vendor it
yourself (FCM / APNs / email / WebPush):
```ts
inbox.onMessageQueued(async (recipient, msgId) => {
await fcm.send(recipient, { kind: 'shade-inbox', msgId });
});
```
The recipient device wakes, runs `inbox.tick()`, and pulls the blob.
### Durable queue
The default in-memory queue is fine for short-lived processes. For a
service that must survive restart, plug in your own `OutgoingQueueStore`
backed by SQLite/Postgres/IndexedDB:
```ts
const inbox = new Inbox({
// …
queueStore: new MyDurableQueueStore(),
cursorStore: new MyDurableCursorStore(),
});
```
Same idea for the receive cursor — without persistence, every restart
re-downloads everything currently within TTL.
### Errors
- **Decrypt failure** in your handler keeps the blob on the server (no
ack). The next poll re-fetches it — useful when the ratchet temporarily
rejects a message because of out-of-order delivery.
- **`msgId/ciphertext` mismatch** is a relay-tampering canary. The Inbox
client recomputes the hash and emits `inbox.message_decrypt_failed`
*without* acking, so an operator can investigate before the blob
silently expires.
- **Network failure** on PUT keeps the entry in the local queue with an
`attempts` counter; default cap is 10 retries before the entry is
dropped (configurable via `maxAttempts`).
---
## 3. Threat model — what the relay actually sees
| Knows | Doesn't know |
| -------------------------------------------------- | ----------------------------------------- |
| Recipient address (path parameter) | Recipient real identity (it's pseudonymous) |
| Sender's per-PUT signing public key | The mapping sender-pubkey → real identity |
| Number of blobs queued for an address | Plaintext content |
| Approximate ciphertext size | Sender-recipient pair beyond bytes-pari |
| Per-blob TTL (in the row's `expires_at`) | The ratchet/X3DH state |
### Privacy posture
- **Sender-recipient graph leaks at the byte-pari level.** A passive
observer of the relay (or its DB dump) can correlate sender pubkey ↔
recipient address ↔ blob size. Mitigations:
- Recipients can use **address hashes** instead of human-readable
addresses (the address grammar accepts any `[a-zA-Z0-9][a-zA-Z0-9:_\-.]{0,255}`,
so `sha256(real-address || salt)` works).
- Senders can rotate their per-PUT signing key per session; the relay
only verifies the signature and never persists the key.
- **TTL leaks reachability.** A sender's PUT silently dropping after 7
days is itself a signal. Operators can normalize TTLs (clamp every
PUT to a fixed 7-day window) to flatten this.
- **Operator can DoS a recipient** by deleting their queue. Mitigation:
recipient ack happens *after* successful decrypt, so a malicious
delete just forces re-send by the original sender.
### What the relay can NOT do
- **Read plaintext** — the ratchet/AEAD layers run client-side.
- **Forge a sender** — every PUT is Ed25519-signed by the sender's
per-PUT key; the relay rejects bad signatures with 401.
- **Inject a foreign blob** — the recipient client recomputes
`sha256(ciphertext)` and refuses anything that doesn't match the
stored `msgId`.
- **Replay an old PUT** — the signed `signedAt` field has a ±5-minute
window (matches the prekey-server's policy); replays past that window
return 409.
### Storage-DoS
`maxBlobBytes` (default 1 MiB) caps a single PUT.
`maxBlobsPerAddress` (default 1000) caps the recipient's queue depth —
PUTs past the cap return 400 with a structured `inbox.quota_rejected`
event so operators can alert. Combine with per-IP rate limits at the
edge (the built-in token bucket is in-memory and not multi-instance).
---
## 4. Wire reference
All bodies are JSON. Multi-byte fields are base64-standard encoded.
### `POST /v1/inbox/register` (TOFU)
```json
{
"address": "bob",
"signingKey": "<base64 Ed25519 public key>",
"signedAt": 1716057600000,
"signature": "<base64 Ed25519 signature over canonical body>"
}
```
- 200 — registered (or idempotent re-register with same key).
- 401 — different key already owns this address, or signature failed.
### `POST /v1/inbox/:address` (PUT blob)
```json
{
"senderSigningKey": "<base64 sender Ed25519 public key>",
"msgId": "<lowercase hex sha256(ciphertext)>",
"ciphertext": "<base64 wire bytes from encodeEnvelope()>",
"ttlSeconds": 604800,
"signedAt": 1716057600000,
"signature": "<base64 sender signature>"
}
```
- 200 with `{ msgId, receivedAt, idempotent: false }` — first store.
- 200 with `idempotent: true` — duplicate PUT folded into the first row.
- 400 — `msgId` mismatch, ciphertext too big, or address quota exceeded.
- 401 — bad signature or stale `signedAt`.
- 404 — recipient address never registered.
### `POST /v1/inbox/:address/fetch` (signed challenge)
```json
{
"address": "bob",
"sinceCursor": 0,
"signedAt": 1716057600000,
"signature": "<base64 recipient signature>"
}
```
Returns:
```json
{
"blobs": [
{
"msgId": "<hex>",
"ciphertext": "<base64>",
"receivedAt": 1716057601234,
"expiresAt": 1716662401234
}
],
"cursor": 1716057601234,
"hasMore": false
}
```
Pass the returned `cursor` as `sinceCursor` next time. Pages cap at
`fetchPageLimit` (default 100); keep calling with the new cursor while
`hasMore === true`.
### `DELETE /v1/inbox/:address/:msgId` (signed ack)
Body:
```json
{
"address": "bob",
"msgId": "<hex>",
"signedAt": 1716057600000,
"signature": "<base64 recipient signature>"
}
```
- 200 with `{ ok: true }` — row removed.
- 200 with `{ ok: false }` — row was already gone (also idempotent).
- 401 — recipient signature failed.
### `DELETE /v1/inbox/register/:address`
Same auth shape as ack. Drops every queued blob.
---
## 5. Acceptance test mapping
| V3.6 spec criterion | Test |
| ---------------------------------------------------------- | -------------------------------------------------------------- |
| Async delivery without online overlap | `lifecycle.test.ts → "100 messages delivered…"` |
| DB-dump leaks no plaintext / sender-recipient graph | Server stores only `address \|\| msgId \|\| ct \|\| expires_at`; verified by `routes.test.ts` schema asserts |
| Replay PUT with same `msgId` is idempotent | `routes.test.ts → "idempotent on duplicate ciphertext"` |
| Restart preserves blobs | `lifecycle.test.ts → "persistence across restart"` + sqlite-store reopen |
| Bit-flip on stored ciphertext rejected on the client | `lifecycle.test.ts → "Tamper resistance"` + client `client.test.ts → "tamper detection"` |

348
docs/key-transparency.md Normal file
View File

@@ -0,0 +1,348 @@
# Key Transparency (V3.12)
> **Status:** v0.4.0+ — opt-in. Server runs unchanged when KT is off.
> Klient ignorerer proof-felt når KT-config mangler. Trygg å rulle ut
> uten klient-update.
Shades prekey-server er sannhetskilde for hvilket bundle som er
publisert for hver adresse. Uten Key Transparency (KT) kan en
ondsinnet eller kompromittert server bytte ut et bundle uten at noen
oppdager det. Med KT er hvert bundle som leveres **kryptografisk
forpliktet** i en append-only Merkle log som tredjeparts-witnesses kan
auditere.
Se også `docs/V3.12-DESIGN.md` for designnotat med trusselmodell og
beslutningsspor.
---
## Hva KT garanterer
| Angrep | Detektert? |
|---|---|
| Server gir Bob feil bundle for `alice` | **Ja** — inklusjons-proof matcher ikke |
| Server gir Bob og Charlie ulike bundles for `alice` | **Ja** — witness-gossip ser to STH-er på samme `tree_size` |
| Server skriver om historikk for å skjule tidligere svik | **Ja** — konsistens-proof feiler |
| Server signerer "stale" STH for å holde et tidsvindu åpent | **Ja** — klient avviser STH eldre enn `maxStaleMs` (default 24t) |
| Førstegangs-impersonering av en helt ny adresse | **Nei** — KT ser bare etter at adressen er i loggen, ikke at den er "riktig" person. Bruk V3.3 (fingerprint-gate) + V3.10 (social recovery) for det. |
---
## Operatør: skru på KT
KT er opt-in og krever:
1. **Et Ed25519 signing-keypair** for STH-signering. Dette er
*operatørens* nøkkel og må beskyttes som en code-signing-key.
2. **En persistent KTLogStore.** I produksjon: `PostgresKTLogStore`.
I test/dev: `MemoryKTLogStore`.
3. **At klienter pinner samme `logPublicKey`** OOB (typisk via
`Shade.config`-bundling i appen).
### Generere signing-key
```sh
bun run scripts/generate-kt-key.ts > kt-key.json
```
(Eller kjør manuelt: `crypto.generateEd25519KeyPair()` i en Bun REPL.)
Lagre `privateKey` i operatørens secret-store. Distribuér `publicKey`
til klienter sammen med app-config.
### Boot serveren med KT
```ts
import { createPrekeyServerWithKT } from '@shade/server';
import { PostgresPrekeyStore, PostgresKTLogStore } from '@shade/storage-postgres';
import { SubtleCryptoProvider } from '@shade/crypto-web';
const crypto = new SubtleCryptoProvider();
const prekeyStore = await PostgresPrekeyStore.create(process.env.DATABASE_URL!);
const ktStore = await PostgresKTLogStore.create(process.env.DATABASE_URL!);
const { app, kt } = await createPrekeyServerWithKT({
crypto,
store: prekeyStore,
keyTransparency: {
store: ktStore,
signingPrivateKey: loadFromSecret('SHADE_KT_SIGNING_PRIVATE_KEY'),
signingPublicKey: loadFromSecret('SHADE_KT_SIGNING_PUBLIC_KEY'),
heartbeatIntervalMs: 10 * 60 * 1000, // default; 0 = off
},
});
export default { port: 3900, fetch: app.fetch };
```
Når KT er på blir disse rutene tilgjengelig:
| Route | Hva den returnerer |
|---|---|
| `GET /v1/kt/log_id` | `{ logId, publicKey }` (begge base64) |
| `GET /v1/kt/sth` | Siste signed tree head |
| `GET /v1/kt/sth/:treeSize` | Historisk STH for et bestemt tree_size |
| `GET /v1/kt/consistency?from=N1&to=N2` | Konsistens-proof N1 → N2 |
Bundle-fetch (`GET /v1/keys/bundle/:address`) får nå et `ktProof`-felt
i responsen.
### Migrasjon fra ikke-KT
KT er bakoverkompatibel:
1. Skru på KT-config i serveren. Restart.
2. Eksisterende klienter ignorerer proof-feltene (`ktProof`, `ktSth`).
3. Etter hvert som klienter oppgraderes med KT-config (`mode: 'observe'`),
begynner de å verifisere.
4. Når øko-systemet er vant til det, eskalér klienter til
`'observe-strict'` for å avvise prekey-server-svar uten proof.
Ved første boot scanner KT-tjenesten ikke automatisk eksisterende
prekey-store-tilstand inn i loggen. **Re-registrering** av eksisterende
adresser (dvs. en `POST /v1/keys/register`-runde fra hver klient) er
det som backfiller. For et større deployment: anbefalt at en operatør
varsler brukerne om å re-registrere innen et tidsvindue. Klienter som
ikke re-registrerer vil feile `observe-strict`-fetch til de får ny key
fra peer.
---
## Klient: skru på KT
```ts
import { createShade } from '@shade/sdk';
const shade = await createShade({
prekeyServer: 'https://shade.example.com',
address: 'alice',
keyTransparency: {
mode: 'observe-strict', // eller 'observe'
logPublicKey: KT_LOG_PUBLIC_KEY_BASE64, // eller Uint8Array
maxStaleMs: 24 * 60 * 60 * 1000, // default 24t
},
});
```
`shade.getKTWitness()` returnerer `LightWitness`-instansen som
samler observerte STH-er. Bruk `.compare(otherSth)` for manuell
gossip-sjekk mot peers.
### `mode: 'observe'`
- Verifiserer proof når serveren leverer det.
- Skipper verifisering hvis `ktProof` mangler i bundle-respons.
- Anbefalt under første utrulling der ikke alle klienter har
re-registrert ennå.
### `mode: 'observe-strict'`
- Krever proof på hver `200`-respons. Mangler proof → kast `KTVerificationError`.
- Krever proof på hver `404`-respons også (for absence/tombstone-pinning).
- Anbefalt produksjons-modus når KT-økosystemet er etablert.
---
## Witness / auditor
`@shade/key-transparency` eksporterer `LightWitness`. Et CLI-verktøy
eller backend-job kan bruke den slik:
```ts
import { LightWitness } from '@shade/key-transparency';
import { SubtleCryptoProvider } from '@shade/crypto-web';
const crypto = new SubtleCryptoProvider();
const witness = new LightWitness({
crypto,
logPublicKey: KT_LOG_PUBLIC_KEY,
fetcher: {
async fetchLatestSTH() {
const r = await fetch('https://shade.example.com/v1/kt/sth');
return r.json();
},
async fetchConsistencyProof(from, to) {
const r = await fetch(`https://shade.example.com/v1/kt/consistency?from=${from}&to=${to}`);
return r.json();
},
},
});
// Poll periodically (e.g. every 5 minutes)
setInterval(async () => {
try {
const sth = await witness.pollOnce();
console.log(`Observed STH: tree_size=${sth.treeSize}, root=${Buffer.from(sth.rootHash).toString('hex').slice(0, 16)}`);
} catch (err) {
console.error('Witness alarm:', err);
// Send to PagerDuty / Slack / whatever
}
}, 5 * 60 * 1000);
```
Witness-koden detekterer:
- **Stale STH** — server publiserer ikke nye STH-er i tide.
- **Split view** — to STH-er ved samme `tree_size` med ulik root.
- **Re-write** — konsistens-proof feiler.
- **Wrong key** — `log_id` matcher ikke pinnet `logPublicKey`.
---
## Operatørkost (estimat)
For et deployment med:
- **100k registrerte adresser**
- **1 identitets-rotasjon per år** per bruker
- **52 replenish per år** (én i uka, *ikke* committed til loggen — bare register/delete er)
| Ressurs | Per år | Kommentar |
|---|---|---|
| Log-rader | ~100k | bare register/delete |
| Lagring (leaves+index) | ~25 MB | base64-kodet |
| STH-rows | ~52k | én per heartbeat (10 min) |
| STH-storage | ~7 MB | |
| CPU per STH | ~1ms | Ed25519-signing er trivielt |
| Bundle-fetch overhead | <2ms | inkluderer audit-path-bygg |
**Backup:** behandle KT-tabellene som "kan ikke gjenopprettes" data —
`shade_kt_leaves` har en database-trigger som forbyr UPDATE/DELETE i
PostgreSQL-implementasjonen. Backup-strategi:
- Daglig full backup av `shade_kt_*` tabellene.
- WAL-shipping anbefalt (tap < 60 s i verste fall).
- **Test recovery** kvartalsvis. Recovery-prosedyre står under.
---
## Recovery
### Scenario 1 — STH-signing-key tapt eller kompromittert
Loggen forblir konsistent (alle gamle STH-er er allerede signert), men
nye STH-er kan ikke signeres med samme key.
**Steg:**
1. Generer ny Ed25519-keypair.
2. Skriv inn et "rotation breaks here"-leaf i loggen (operasjon = 0x03
på en spesiell `__log__`-adresse) — operasjonen er rent
informativ, men gjør rotasjonen synlig i tree.
3. Re-konfigurer serveren med ny key. Restart.
4. Server publiserer en ny STH; den vil ha en ny `log_id` (siden
`log_id = SHA-256(publicKey)`).
5. **Klienter må eksplisitt akseptere ny key.** Inntil de pinner ny
`logPublicKey`, vil deres `LightWitness` kaste
`KTLogIdMismatchError`. Operatør publiserer ny key OOB med
"rotated from `<gammel logId>`"-melding signert med gammel key
(siste handling før gammel key zeroizes).
### Scenario 2 — KT-database korrumpert / tapt før backup
Dette er **det verste utfallet**. Loggen er per design ikke
gjenopprettbar — å "rekonstruere" den fra prekey-store ville bryte
selve invarianten KT lover.
**Steg:**
1. Stopp serveren.
2. Deklarer en "log-restart event" via offentlig kanal (status-side,
release-notes, Twitter, etc.) — inkluder timestamp, tapte tree_size
(siste backup-bare snapshot om mulig), og ny `logPublicKey`.
3. Generer ny KT-keypair (ikke bruk gamle).
4. Boot serveren tom (tom `shade_kt_*` tabell). Første STH er fra
`tree_size = 0`.
5. Be brukerne om å re-registrere identitetene sine. Klientene vil
trigge V3.3 fingerprint-gate på første re-meldings-flyt etterpå
siden rotasjons-fingerprintet endres.
6. Auditor-organisasjoner kan publisere "vi observerte gammel log
inntil tree_size N, ny log starter på 0 fra T+0" — dette gir
sluttbruker mulighet til å vurdere hvor stort hullet er.
**Beskytt mot dette:** WAL-shipping + off-site backup. Aldri kjør KT
med kun én database-instans uten replicas.
### Scenario 3 — Witness oppdager split-view
Witness kaster `KTSplitViewError` i `LightWitness.observe()` eller
`KTVerificationError` i transport. Dette betyr:
- Operatøren har enten
(a) hatt en software-bug som signerte to ulike STH-er ved samme
tree_size, eller
(b) er kompromittert / ondsinnet.
**Operatør-handling:**
1. Pause `POST /v1/keys/register`, `DELETE`, og bundle-fetch
umiddelbart (return 503).
2. Audit `shade_kt_sths` — hvis du finner to rader med samme
`tree_size` men ulik `root_hash`, har serveren gjort feil. Dette er
alvorlig — finn root cause før du fortsetter.
3. Kommuniser ut til brukerne. Forutsett at en angriper har vært
inne; trigge en bredere reset (recovery scenario 2) hvis det er
mistanke om tampering.
**Klient-handling:**
- `LightWitness` har allerede holdt brukeren tilbake.
- SDK-en surfacer feilen som `KTSplitViewError` til app-koden.
- App-en bør vise advarsel: "Operatørens server kan ikke verifiseres.
Avstå fra sending av sensitive meldinger inntil videre."
---
## Sikkerhets-anbefalinger
1. **Kjør minst én uavhengig witness.** Operatørens egen "witness"
teller ikke — det må være en separat prosess på separate
infrastruktur eid av en separat aktør (community-medlem, security
firm, e.l.).
2. **Pin `logPublicKey` i app-binær eller signert config.** En
man-in-the-middle som kan bytte både prekey-server og KT-key
fanges ikke av KT alene.
3. **Loggrotasjon krever menneske-i-løkken.** Ikke automatiser
key-rotation for KT — den eksplisitte breaking-event er en feature.
4. **`maxStaleMs` bør samsvare med din heartbeat.** 24t default tåler
en heartbeat-pause på opptil et døgn; senk til 1–4t hvis du har
strenge krav til friskhet.
5. **`observe-strict` bør være standard når økosystemet er etablert.**
Default `'observe'` er en operasjonell overgangsmodus, ikke et
sluttmål.
---
## Kjente begrensninger
- **Federation mellom flere prekey-servere** er ikke støttet i V3.12.
Hver Shade-deployment har én log eller ingen.
- **Sparse Merkle tree for adresse-index** brukes ikke i V3.12 —
fravær-proof er foreløpig nabopar-bevis. <100 KB ved 100k adresser
er akseptabelt; sparse tree blir relevant fra ~10M+ adresser.
- **One-time prekey-rotasjon committes ikke** til loggen. OTP er
ephemerale og inkludering ville støy-fylle loggen. Dette betyr at
en server som svarer med riktig identitet men feil OTP fanges ikke
av KT — forsvar mot dette ligger i V3.3 fingerprint-gate (samme
identitet) + sesjons-etableringens X3DH (feil OTP gir feil shared
secret → første melding feiler decryption).
---
## Tester og test-vektorer
- `packages/shade-key-transparency/tests/` — RFC 6962-kompatibel
Merkle-log + STH + index-proofs (58 tests).
- `packages/shade-server/tests/kt.test.ts` — server-integrasjon (8
tests).
- `packages/shade-transport/tests/kt-transport.test.ts` — klient-
verifikasjon over HTTP (4 tests).
- `packages/shade-transport/tests/kt-split-view-e2e.test.ts` —
V3.12-akseptanse split-view-deteksjon (3 tests).
- `packages/shade-sdk/tests/kt.test.ts` — SDK-config + witness wiring
(3 tests).
Totalt 76 tester dedikert til KT.

193
docs/observability.md Normal file
View File

@@ -0,0 +1,193 @@
# Observability v2 — OpenTelemetry tracing
Shade ships an opt-in OpenTelemetry layer that wraps `TransferEngine`,
`ShadeSessionManager`, the prekey HTTP routes, and `@shade/files`
op-handlers in distributed spans. The layer is **off by default** and
PII-safe by construction — span attributes never include peer addresses,
plaintext payloads, or exact byte counts.
This complements the always-on Prometheus metrics exposed by
`@shade/server` and the structural events emitted by `@shade/core`. Use
metrics for aggregate counters and histograms, tracing for per-request
causality and tail-latency hunting.
---
## Quick start
```ts
import { trace } from '@opentelemetry/api';
import { withTracer } from '@shade/observability';
import { createShade } from '@shade/sdk';
// Use the OTel SDK of your choice (NodeSDK + OTLP exporter, Honeycomb,
// Sentry's OTel adapter, …) to register a tracer provider on the
// `@opentelemetry/api` global. Then:
const tracer = trace.getTracer('my-app');
const shade = await createShade({
prekeyServer: 'https://shade.example.com',
storage: 'sqlite:/data/shade.db',
observability: withTracer(tracer, { sample: 0.1 }),
});
```
The hook propagates automatically to:
- `ShadeSessionManager.encrypt` / `.decrypt` (per-peer mutex acquisition,
ratchet step).
- `TransferEngine.upload` / accepted incoming downloads (lane count,
retry count, partition mode).
- `@shade/files` op-handlers (per request, with op + result).
For the prekey server pass the hook to `createPrekeyRoutes`:
```ts
import { createPrekeyRoutes } from '@shade/server';
import { withTracer } from '@shade/observability';
const app = createPrekeyRoutes(store, crypto, {
observability: withTracer(tracer),
});
```
---
## Off-by-default semantics
`withTracer()` returns a no-op hook — the SDK never starts spans — when
**any** of the following are true:
1. The `tracer` argument is `undefined`/`null`.
2. The `SHADE_OTEL_ENABLED` env-var is not set to `1` or `true`. Override
with `withTracer(tracer, { force: true })`, or override the var name
with `withTracer(tracer, { envVar: 'MY_VAR' })`.
3. The configured `sample` rate is `0`.
Per-span sampling (`sample: 0.1` = 10 %) keeps trace volume bounded in
production. Default is `1` (sample everything when the hook is active).
---
## PII policy — what is safe to log, and what isn't
| Category | Status | Why |
|----------|--------|-----|
| **Peer hash** (`shade.peer.hash`) | ✅ allowed | 8-hex-char pseudonym derived via SHA-256. Stable across spans for a given address but does not expose the address itself. |
| **Bytes bin** (`shade.bytes.bin`) | ✅ allowed | One of `≤4KB`, `4–64KB`, `64KB–1MB`, `1–10MB`, `10–100MB`, `100MB–1GB`, `≥1GB`. Coarse enough to mask file-size fingerprinting. |
| **Lane count** (`shade.lane.count`) | ✅ allowed | Snapped to `{1, 4, 16, 64}`. |
| **Retry count** (`shade.retry.count`) | ✅ allowed | Integer. |
| **Error code** (`shade.error.code`) | ✅ allowed | `SHADE_*` stable string code — never the full message, which may interpolate user input. |
| **Op kind** (`shade.op`) | ✅ allowed | `list`, `read`, `write`, `custom:foo`, etc. |
| **Route template** (`shade.route`) | ✅ allowed | `/v1/keys/bundle/:address` — the template, never the resolved path. |
| **HTTP status** (`shade.http.status`) | ✅ allowed | Integer status code. |
| **Partition mode** (`shade.partition`) | ✅ allowed | `range` or `round-robin`. |
| **Direction** (`shade.direction`) | ✅ allowed | `upload` or `download`. |
| Plaintext peer addresses | ❌ forbidden | Use `peerHash()`. |
| Plaintext message/file payloads | ❌ forbidden | Encryption boundary — never log. |
| Exact byte counts | ❌ forbidden | Use `bytesBin()`. |
| User identifiers (email, DID, `device:UUID`) | ❌ forbidden | Treat as PII. |
The full attribute-key allow-list is exported from `@shade/observability`
as `ATTR_*` constants. Plug-in authors who want to attach their own tags
should pass each `(key, value)` through `safeAttribute()`, which throws
`UnsafeAttributeError` for any key/value pair that looks like the
forbidden categories above (heuristics: `@`, `device:`, `did:`, key
fragments such as `peer.address` / `bytes.exact`, oversized strings).
---
## Span surface
### `shade.session.encrypt` / `shade.session.decrypt`
Wraps each per-peer `encrypt`/`decrypt` call. Includes the time spent
waiting on the per-peer mutex (`shade.lock.wait_ms`) — handy for
diagnosing ratchet contention under load.
### `shade.transfer.upload` / `shade.transfer.upload.resume`
Wraps an outbound stream transfer end-to-end. Attributes: `peer.hash`,
`bytes.bin`, `lane.count`, `partition`, `retry.count`, `result`,
`error.code`.
### `shade.transfer.download`
Started when the consumer calls `incoming.accept(...)`, ended when the
transfer completes, aborts, or fails an integrity check. Same attribute
set as upload.
### `shade.prekey.request`
One span per HTTP request handled by `@shade/server`'s prekey routes.
Attributes: `route` (the template), `http.status`, `error.code` on
failure. The address path-parameter is **never** placed on the span.
### `shade.files.op`
One span per `@shade/files` RPC. Attributes: `peer.hash`, `op` (the
resolved op kind, e.g. `read` or `custom:foo`), `bytes.bin` (estimated
plaintext size, binned), `result`, `error.code`.
---
## Recording & testing
`@shade/observability` ships a deterministic in-memory recorder for
unit tests:
```ts
import { createRecorder } from '@shade/observability';
const rec = createRecorder();
const shade = await createShade({ ..., observability: rec });
// … exercise code under test …
const hits = rec.scanForPII(['alice@example.com', 'plaintext-secret']);
expect(hits).toHaveLength(0);
```
The Shade test suite runs this recorder over every documented entry
point — see
`packages/shade-observability/tests/integration-pii.test.ts` and
`packages/shade-transfer/tests/observability.test.ts`. Any new
instrumentation must keep the suite green.
---
## Performance characteristics
- With OTel **off** (default): every Shade hook resolves to the shared
`NOOP_HOOK` instance. The cost is one function call + an object
allocation that V8 hoists out in the steady state — measured at
< 1 % overhead vs the pre-V3.4 baseline in the upload roundtrip
benchmark.
- With OTel **on**: cost depends entirely on the configured exporter.
Use `sample: 0.1` (or smaller) on hot paths in production.
---
## Adding new instrumentation
1. Identify a logical operation worth a span — typically anything that
crosses a network/disk boundary or contends on a lock.
2. Add an `observability?: ObservabilityHook` to the relevant config
surface, default to `NOOP_HOOK`.
3. Name the span `shade.<area>.<op>` to keep cardinality bounded.
4. Set attributes via the `ATTR_*` constants from
`@shade/observability`. **Never** introduce a new attribute key
without a PII review — if you must, run the value through
`safeAttribute()`.
5. Add a test that exercises the new instrumentation under the
`createRecorder()` recorder and asserts no PII leaks.
---
## Migration
Previous versions had no tracing — only Prometheus metrics. Adding the
`observability` field to existing configs is fully backwards-compatible
and never required. The `SHADE_OTEL_ENABLED` gate ensures forgetting to
flip the env-var in production won't surprise anyone with unexpected
overhead.

308
docs/recovery.md Normal file
View File

@@ -0,0 +1,308 @@
# Social Key Recovery (`@shade/recovery`)
V3.10 closes the biggest UX hole in any E2EE system: **"What happens
if I lose my phone?"**. Shade's social-recovery flow lets a user
designate `n` guardians (family / friends / co-workers) at setup time
such that any threshold-many `k` of them can together restore the
user's identity onto a new device — without any single guardian
being able to do it alone, and without the prekey server ever seeing
the recovered key material.
The whole flow ships entirely over existing 1:1 Shade sessions; no
server-side recovery agent, no escrow service, no "cloud guardian".
---
## Threat model recap
| # | Adversary | Recovered? |
|---|-----------|------------|
| 1 | Coalition of ≤ k-1 guardians | **No** (information-theoretic, by Shamir construction) |
| 2 | Prekey server alone | **No** (server only relays Double-Ratchet ciphertext) |
| 3 | Single malicious guardian who forges a share | **Detected** — AES-GCM tag mismatch on the backup blob; `requestRecovery` exhaustively tries threshold-sized subsets and rejects when none authenticate |
| 4 | Social engineering (impersonator calls a guardian) | **Mitigated, not eliminated** — guardians MUST OOB-confirm the new device's safety number before approving (see `<RecoveryApprove />`) |
| 5 | Compromised guardian device | **Out of scope** — see "Guardian compromise" below |
| 6 | Compromised primary device at setup time | **Out of scope** — recovery only protects the device; if setup material is exfiltrated, all bets are off |
---
## Setup
### What the user does
1. Pick `n` guardians from their existing peers.
2. Pick a threshold `k` (typically `⌈n/2⌉ + 1` to avoid pure-majority
dominance but still survive losing one or two).
3. Run `setupRecovery(...)`.
4. Print / record a **recovery card** with:
- The user's own address
- `setupId`
- `k` and `n`
- The list of guardian addresses
- Setup-time safety number
The recovery card is the only piece of state the user must remember
out-of-band (or store in a password manager). Without it, the user
cannot drive recovery on a new device — the new device needs to know
who the guardians are.
### What happens cryptographically
```text
recoveryKey = random(32 bytes)
backupBlob = Shade.exportBackup(passphrase = "shade-rk:" + base64url(recoveryKey),
knownAddresses = [...])
shares[i] = Shamir-split(recoveryKey, k, n)
```
For each guardian `i`:
```text
share-deposit envelope:
shadeRecovery: 1
type: "share-deposit"
flowId, setupId, originalAddress
threshold (k), guardianCount (n), shareIndex (i)
shareBytes: base64url( encodeShare(shares[i]) )
backupBlob: Shade.exportBackup output (identical for every guardian)
setupFingerprint, createdAt
```
The envelope rides through `Shade.send` like any other plaintext —
double-ratchet encrypted, AAD-bound, replay-safe.
The `recoveryKey` is **zeroized** on the primary device immediately
after the split returns. The primary therefore retains nothing
except `setupId` and the public roster.
### What each guardian stores
Per (`originalAddress`, `setupId`):
```text
{
shareIndex, // 1..n
shareBytes, // base64url-encoded Shamir share
backupBlob, // identical for every guardian
setupFingerprint, // for sanity-checks at recovery time
guardianCount, threshold,
receivedAt
}
```
The guardian's app provides a `RecoveryStore` implementation. The
package ships `MemoryRecoveryStore` for tests and small one-shot
demos; production guardian apps MUST supply a persistent store
(IndexedDB, AsyncStorage, SQLite, etc.). See "Persistence
recommendations" below.
---
## Recovery
### What the user does on the new device
1. Boot a fresh Shade with a temporary identity.
2. Read the recovery card.
3. In the recovery widget, type / paste:
- `originalAddress`
- `setupId`
- `threshold`
- The guardian roster
4. Read the new device's safety number (the widget displays it
prominently) to each guardian over a side channel — phone call,
in person, whatever they trust.
5. Wait for `≥ k` guardians to approve.
### What happens cryptographically
For each guardian, the new device sends:
```text
recovery-request envelope:
shadeRecovery: 1
type: "recovery-request"
flowId, originalAddress, setupId
requesterFingerprint (= safety number of the temporary identity)
requestedAt
```
Each guardian's `attachGuardian` handler:
1. Looks up its stored deposit by `(originalAddress, setupId)`. If
missing, replies with `share-decline` (`reason = "unknown setup"`).
2. Invokes the `approve` callback with the requester's address +
fingerprint + the original device's setup-time fingerprint. The
callback is the **OOB-confirmation gate** — it MUST require an
explicit user click after they verified the fingerprint. The
`<RecoveryApprove />` widget enforces this with a two-checkbox
gate.
3. On approve → ships `share-grant`. On reject → ships
`share-decline` with a short reason.
The new device collects grants, and as soon as `k` arrive:
1. Combines the `k` shares via Lagrange interpolation at `x = 0` to
reconstruct `recoveryKey`.
2. Derives `passphrase = "shade-rk:" + base64url(recoveryKey)`.
3. Calls `Shade.importBackup(backupBlob, passphrase)` — the
AES-GCM tag in the blob authenticates the reconstruction. **A
forged share is detected here.**
4. If a guardian forged a share, `importBackup` throws. The
reconstruction loop then tries every other threshold-sized subset
of grants until one authenticates (the V3.10 acceptance criterion
"no coalition of (k-1) guardians can rebuild the secret" is the
safety invariant; the AEAD authenticates which subset is
honest).
5. If every subset fails, `RecoveryReconstructionError` is raised
and the user is told that at least one guardian is malicious.
After `importBackup` succeeds, the new device hosts the original
identity and immediately calls `Shade.rotate()` to retire the
recovery-recovered key material from the conversation graph (the
old session keys persisted in the backup blob are now considered
"compromised — used for recovery").
> **The `Shade.beforeBackupImport` gate fires automatically.**
> Without a registered handler the SDK falls back to TOFU-with-warning
> (consistent with the V3.3 contract). Production apps SHOULD register
> a handler that pops the user one more confirmation before the
> identity rotates.
---
## Acceptance criteria status
- [x] **3-of-5 recovery works end-to-end on two separate Shade
instances.** See `tests/integration.test.ts`.
- [x] **No coalition of (k-1) guardians can reconstruct
`recoveryKey`.** Property test asserts this with `fast-check`
across random k/n configurations.
See `tests/shamir.test.ts` and
`tests/adversarial.test.ts`.
- [x] **Guardian-side widget requires fingerprint-confirmation
before sending.** `<RecoveryApprove />` enforces a
two-checkbox gate; `tests/adversarial.test.ts` exercises
both the matching-OOB and rejecting-OOB code paths.
---
## Persistence recommendations
The `RecoveryStore` interface is intentionally small (4 methods).
Pick the implementation that fits your platform:
| Platform | Suggested backing store |
|--------------------------|----------------------------------------|
| Browser (PWA) | IndexedDB (one object store, idb) |
| Browser (extension) | `chrome.storage.local` |
| React Native | AsyncStorage (with crypto-protected blob) |
| Bun / Node server | SQLite via `@shade/storage-sqlite` extension table OR a side file |
| Android (native) | Room / EncryptedSharedPreferences |
Whatever you pick, the records ARE NOT secret on their own — without
threshold-many other guardians' shares they're useless — but they
should still be stored encrypted-at-rest like any other Shade state.
Do not commit them to plaintext logs or network-replicated state.
---
## Guardian-UX guide
### How many guardians?
| n | Survives | Comment |
|---|----------|---------|
| 3, k=2 | 1 lost guardian | Minimum useful — one device away from danger |
| 5, k=3 | 2 lost guardians | Sweet spot for most users |
| 7, k=4 | 3 lost guardians | Suitable when you genuinely have 7+ trustworthy people |
| n=k | 0 lost | DO NOT USE — single point of failure |
The widget defaults to `k = ⌈n/2⌉` which is liberal but
collusion-resistant for `n ≥ 3`. Apps targeting paranoid users may
want to bump that to `⌈2n/3⌉`.
### Replacing a guardian
If a guardian dies, loses their device permanently, or you no longer
trust them:
1. Pick a replacement.
2. Run `setupRecovery` again with the new roster — this generates a
fresh `setupId` and a fresh `recoveryKey`. The old shares become
garbage (no guardian set can use them, because the
`backupBlob` is different).
The widget records the new `setupId` on the recovery card. Treat
this as a hard rotation; the user MUST re-record the card.
### Guardian health checks
Periodically (the V3.10 plan suggests a quarterly prompt), the user
should confirm each guardian is still reachable. Any guardian who
can't be reached in two consecutive prompts SHOULD trigger a
re-setup with a fresh roster. The widget UX track is to be added in
a follow-up release; the primitive is in place.
---
## Wiring example
```ts
import {
setupRecovery,
attachGuardian,
requestRecovery,
MemoryRecoveryStore,
} from '@shade/recovery';
// On the primary device:
const result = await setupRecovery({
shade,
guardians: ['bob', 'carol', 'dan', 'eve', 'faythe'],
threshold: 3,
deliver: async (to, envelope) => {
// wire to your app's existing message-delivery layer
await myMessageOutbox.send(to, envelope);
},
});
console.log(result.setupId);
// On each guardian device:
const stop = attachGuardian({
shade,
store: myPersistentStore, // see "Persistence" above
approve: async (ctx) => {
// Show ctx.requesterFingerprint to the user.
// Block until they confirm OOB and click "Release share".
return await myUI.askApproval(ctx);
},
deliver: myMessageOutbox.send,
});
// On the new device:
const recovered = await requestRecovery({
shade: temporaryShade, // fresh identity for now
originalAddress: 'alice',
setupId: 'sid-from-recovery-card',
threshold: 3,
guardians: ['bob', 'carol', 'dan', 'eve', 'faythe'],
deliver: myMessageOutbox.send,
onProgress: (p) => myUI.showProgress(p),
});
// `temporaryShade` now hosts the original identity.
```
---
## Out of scope (V3.10)
- **Cloud guardian / Shade-operated recovery agent.** Explicit
non-goal; the spec rejects any centralized component that can
recover on its own.
- **Auto-distribution.** The user must explicitly pick guardians.
- **Multi-share-per-guardian.** Each guardian holds exactly one
share. Apps that need redundancy should bump `n`, not give the
same guardian multiple shares.
- **Guardian ZK-proofs of liveness.** A guardian who refuses to
respond is treated as offline; we don't try to compel them.

160
docs/storage-encryption.md Normal file
View File

@@ -0,0 +1,160 @@
# At-Rest Storage Encryption (V3.2)
**Status:** Implemented in `@shade/storage-encrypted` 0.4.0
**Adresses:** THREAT-MODEL §4 — Compromised device storage
Shade's default `SQLiteStorage` and `PostgresStorage` write private keys and
session state to disk *unencrypted* — the threat model assumes the DB lives
inside a trusted environment. For deployments that need defence in depth,
`@shade/storage-encrypted` adds opt-in at-rest encryption: a stolen DB file
alone yields no usable private key material.
## At a glance
```ts
import { KeyManager, EncryptedSQLiteStorage } from '@shade/storage-encrypted';
const km = await KeyManager.open({
kind: 'passphrase',
passphrase: process.env.SHADE_STORAGE_PASSPHRASE!,
salt: loadSaltFromDisk(), // 16+ bytes, persisted alongside the DB
});
const storage = await EncryptedSQLiteStorage.open({
dbPath: '/data/shade-client.db',
keyManager: km,
});
// Use it exactly like SQLiteStorage — implements the same StorageProvider.
const manager = new ShadeSessionManager(crypto, storage);
```
## What is encrypted
Per-row AEAD over the sensitive payload of every row:
| Table | Encrypted |
|--------------------------------|-----------|
| `identity_enc` | the entire keypair (4× 32-byte keys) |
| `config_enc` | `registrationId` |
| `signed_prekeys_enc` | full `SignedPreKey` (incl. private half) |
| `one_time_prekeys_enc` | full `OneTimePreKey` |
| `sessions_enc` | the Double-Ratchet `SessionState` JSON |
| `trusted_identities_enc` | the trusted peer identity key |
| `retired_identities_enc` | full retired keypair |
| `stream_state_enc.ciphertext` | partition / lane / IO descriptor / streamSecret |
Routing fields on `stream_state_enc` (`stream_id`, `direction`,
`peer_address`, `status`, timestamps) stay plaintext so `listActiveStreamStates()`
remains an indexed query.
## Cryptographic design
```
masterKey (passphrase / keychain / app-injected)
│
├─ HKDF-SHA-256("shade-storage-v1") → storageKey (32 bytes)
│ └─ HKDF-SHA-256(storageKey, "shade-field-v1:{table}:{column}") → fieldKey (32 bytes)
│
└─ Used (transitively) for fingerprint checks
```
For each encrypted blob:
- `nonce = HKDF(fieldKey, "shade-row-nonce-v1:{table}:{pk}")[..12]` —
deterministic per (key, row), safe because the per-(table, column)
fieldKey is unique. AES-GCM nonce reuse is catastrophic only if the
*same* key is reused with the *same* nonce on different plaintexts;
here every (key, row) pair has a unique nonce.
- `aad = "shade-aad-v1|{table}|{column}|{pk}"` — binds the ciphertext
to its row identity so a row swap or column move triggers decrypt
failure.
- `wire = nonce(12) || ciphertext || tag(16)` — stored as a single
`BLOB`/`BYTEA` column.
## Key sources
`KeyManager.open(...)` accepts three sources:
1. **Passphrase + KDF** — scrypt over `(passphrase, salt)`. Default
parameters: `N=2^17, r=8, p=1, dkLen=32` (~250 ms on a modern laptop).
The salt MUST be persisted alongside the DB (e.g. `<db>.salt`).
2. **OS keychain** — via `@shade/keychain`. Backends:
- macOS: `security` CLI (Keychain).
- Linux: `secret-tool` (libsecret).
- Windows: PowerShell + `CredentialManager` module.
No native deps; `createIfMissing: true` generates and stores a fresh
32-byte key.
3. **App-injected** — caller supplies a 32-byte raw key. Most flexible;
plug your own KMS / HSM / Vault path here.
Wrong-passphrase detection is built in: a fingerprint of the storageKey
is persisted in `shade_meta_enc` on first open and compared on every
subsequent open. A mismatch raises with a clear error — never silently
writing under the wrong key.
## Migration
CLI:
```bash
# Encrypt an existing unencrypted DB (atomic per row, .bak written first).
shade migrate-storage \
--key-source passphrase \
--passphrase "$SHADE_STORAGE_PASSPHRASE" \
--salt-file /data/shade-client.db.salt
# Validate without writing.
shade migrate-storage ... --dry-run
# Keychain mode.
shade migrate-storage --key-source keychain \
--keychain-service shade.storage --keychain-account default
# Inject a raw key (e.g. from your KMS).
shade migrate-storage --key-source injected \
--key-hex "$(cat ~/.shade/storage.key.hex)"
```
The migration is *resumable*: re-running it on a partially-migrated DB
re-writes the same rows under the same key (idempotent). On clean
completion, the unencrypted tables are dropped (use `--keep-original`
to preserve them).
## Rotation
```bash
shade rotate-storage-key \
--key-source passphrase --passphrase "$OLD_PASS" \
--new-key-source passphrase --new-passphrase "$NEW_PASS" \
--new-salt-file /data/shade-client.db.salt.new
```
Reads each encrypted row under the old key, re-seals under the new key.
The DB stays online; brief read-after-write inconsistency for in-flight
readers is acceptable for the supported deployments (CLI tools,
single-process servers). On completion the fingerprint is updated and
the old key no longer opens the DB.
## What this does *not* protect
Even with at-rest enabled:
- A live process holds the storageKey and fieldKeys in memory. An attacker
who can dump process memory (`/proc/<pid>/mem`, swap, hibernation,
coredump) recovers the keys.
- Swap is not encrypted by Shade. Use an encrypted swap device.
- The `.bak` file produced during migration is plaintext during the
migration window. Treat it like the original DB and store securely.
- Lost master key = lost DB. V3.10 (Social Recovery) is the long-term
mitigation.
See `THREAT-MODEL.md` §4 for the full list, including the "with at-rest
enabled" boundary.
## Cross-implementation parity
`test-vectors/storage-encryption.json` pins KDF parameters, info strings,
nonce derivation, and AAD format. The Android implementation (V3.5) MUST
produce byte-identical outputs for the same inputs — covered by
`packages/shade-storage-encrypted/tests/test-vectors.test.ts`.

253
docs/streams.md
View File

@@ -107,11 +107,264 @@ manually after rotation.
| S7 | seq overflow practical-impossible (u64 max) |
| S8 | At-rest streamSecret encrypted under device-key |
## Hardening
`@shade/streams` ships unbounded by default — a peer can declare a
1 PiB transfer and the receiver will dutifully allocate lane state for
it. Production receivers must enforce limits at the boundary. The
`@shade/files` package wires the same patterns up for its filesystem
RPC; copy the shapes that fit your app.
### Per-stream caps
The receiver sees the declared plaintext size in the `stream-init`
control message before it accepts. Reject above your tolerance:
```ts
shade.onIncomingTransfer(async (incoming) => {
if (incoming.metadata.totalBytes > 256 * 1024 * 1024) {
await incoming.decline({ reason: 'stream too large' });
return;
}
await incoming.accept({ output: ... });
});
```
Recommended ceilings (tune to your product, not these):
| Tier | totalBytes ceiling | Rationale |
|------|--------------------|-----------|
| Chat attachment | 25 MiB | matches mobile MMS / Slack expectations |
| Photo / doc share | 256 MiB | covers raw RAW + most desktop docs |
| Backup / dataset | 4 GiB | larger needs explicit operator opt-in |
### Per-chunk cap
`createTransferRoutes` accepts `maxChunkBytes` (default ≈ 16 MiB +
header). Lower it if your sink can't absorb that — the receiver will
413 anything over the limit before the chunk is decrypted, which
keeps DoS cost bounded.
### Per-sender quotas
`@shade/files` ships a `RateLimiter` (`packages/shade-files/src/server/rate-limiter.ts`)
that enforces both ops-per-window and bytes-per-hour caps per sender
address. The same shape is the recommended template for guarding raw
streams: wrap `incoming.accept` in a check that consumes from a token
bucket keyed by `incoming.fromAddress`, and reject with `decline()`
when the bucket is empty. See
`packages/shade-files/tests/security/quota.test.ts` for the test
shape.
### TTL on idle streams
A `paused` stream-state record consumes a row in your storage and an
encrypted streamSecret slot until it expires. Use the **Retention**
defaults below to expire abandoned streams; pair with a metric
(`shade_stream_states_active`) and an alert when the count grows
unbounded. A peer that opens streams and never finishes them is the
dominant abuse pattern for resumable transfer.
### Trust gates
For high-stakes transfers (backups, key material, internal docs),
gate `accept()` on a verified fingerprint. The pattern mirrors
`@shade/files`'s fingerprint gate — see
`packages/shade-files/tests/security/fingerprint-gate.test.ts`.
## Retention
Resumable streams persist a `PersistedStreamState` per in-flight
transfer, encrypted under a device key. Without retention, every
crashed or abandoned upload leaves a row behind forever.
### Defaults
The shipped `bun-server` SDK template (`shade init --template bun-server`)
schedules `pruneStreamStates` on a daily cron with a **14-day**
horizon. That is: any stream-state record whose `updatedAt` is older
than 14 days is removed at the next sweep. If a sender resumes a
14-day-old stream, it will get a "no state" 404 and start over —
which is the right answer for a transfer that has been idle for two
weeks.
### Tuning the horizon
Set `SHADE_STREAM_RETENTION_DAYS` in the template's environment to
override the 14-day default. Recommended ranges:
| Use case | Horizon | Why |
|----------|---------|-----|
| Synchronous chat | 1–3 days | resume-after-crash, not resume-after-vacation |
| File-share product | 7–14 days | covers a typical user vacation |
| Cold backup target | 30+ days | deliberate, but plan for storage growth |
### Hooking the prune call manually
If you bring your own server (no `bun-server` template), call the
storage method on your own schedule:
```ts
import { setInterval } from 'node:timers';
const ONE_DAY_MS = 24 * 60 * 60 * 1000;
const HORIZON_MS = 14 * ONE_DAY_MS;
setInterval(async () => {
if (storage.pruneStreamStates !== undefined) {
await storage.pruneStreamStates(Date.now() - HORIZON_MS);
}
}, ONE_DAY_MS);
```
`pruneStreamStates(olderThan)` removes records whose `updatedAt` is
strictly less than `olderThan`. It is idempotent and safe to call
concurrently.
## Rich file metadata + previews (V3.9)
`stream-init` plaintext can carry an optional `fileMetadata` field that
ships filename, MIME-type, and a thumbnail-stream pointer **end-to-end
encrypted**. Older receivers ignore the field — backwards-compatible
with 0.2.x / 0.3.x peers.
```jsonc
{
"kind": "shade.stream-init/v1",
"streamId": "...",
"streamSecret": "...",
"metadata": {
"chunkSize": 1048576,
"sentAt": 1730000000000,
"fileMetadata": {
"filename": "report.pdf",
"mimeType": "application/pdf",
"thumbnailStreamId": "Ej1z...",
"thumbnailHash": "9a7c...",
"thumbnailMime": "image/webp",
"thumbnailBytes": 18342
}
},
"lanes": [ /* ... */ ]
}
```
### What rides where
| Field | Plane | Visible to server? |
|-------|-------|--------------------|
| `filename` | inside Double Ratchet plaintext | no |
| `mimeType` | inside Double Ratchet plaintext | no |
| `thumbnailStreamId` | streamId of companion stream | yes (random ID, no info leak) |
| `thumbnailHash` | sha256 of preview plaintext | base64 hash only, no pixels |
| `thumbnailMime` | one of `image/jpeg / image/webp / image/png` | yes (allowlist enforced) |
| `thumbnailBytes` | declared length, capped at 64 KiB | yes |
| thumbnail bytes themselves | separate AEAD stream, own lane | no |
The thumbnail rides as its **own stream-transfer**, keyed independently
from the main stream. A server compromise leaks neither preview pixels
nor original bytes.
### Sender — attach a preview
```ts
// Pre-computed preview (server-side pipeline path):
await shade.upload({
to: 'bob',
input: pdfBytes,
thumbnail: { bytes: previewWebp, mime: 'image/webp' },
metadata: { fileMetadata: { filename: 'report.pdf', mimeType: 'application/pdf' } },
});
// Browser auto-generation (image File / Blob → 256×256 preview):
await shade.upload({
to: 'bob',
input: imageFile, // a `File` from <input type="file">
generateThumbnail: true, // OffscreenCanvas + createImageBitmap
});
```
`generateThumbnail` is a no-op on runtimes lacking
`OffscreenCanvas + createImageBitmap` (Bun, Node) — those callers should
pre-generate and pass `thumbnail` directly, or skip the preview entirely.
### Receiver — render in widgets
The bundled `@shade/widgets` `useShadeDownload` hook auto-accepts
thumbnail streams (marked by `userMetadata.shadeThumbnail = '1'`) into
an in-memory `ShadeThumbnailCache`. `<TransferRow showThumbnail
fileMetadata={...} />` reads from the same cache and renders inside an
`<img>` element so the browser's image-decoding sandbox is the trust
boundary for format parsing.
```tsx
<ShadeThumbnailProvider>
<TransferRow
handle={handle}
progress={progress}
showThumbnail
fileMetadata={incoming.metadata.fileMetadata}
/>
</ShadeThumbnailProvider>
```
### Format-hardening (sender + receiver)
Both sides enforce the same rules — single source of truth in
`@shade/streams/file-metadata.ts`:
| Rule | Limit |
|------|-------|
| `thumbnailMime` allowlist | `image/jpeg`, `image/webp`, `image/png` |
| `thumbnailBytes` cap | 64 KiB (`THUMBNAIL_MAX_BYTES`) |
| `filename` length | ≤ 1024 chars, no control characters |
| `mimeType` shape | RFC 7231 `type/subtype` token |
| Hash binding | declared `thumbnailHash` = sha256(preview bytes); mismatched bytes are dropped at the cache before any render |
A hostile peer cannot:
- smuggle exotic image formats past the allowlist (envelope parser
rejects at decode-time),
- substitute different bytes for a declared preview (cache verifies
sha256 before exposing bytes to a renderer),
- inflate the cache to OOM the receiver (LRU + 1 MiB total cap).
### Risks consciously accepted
- **Preview-arrival ≠ send completion.** A receiver may see the
thumbnail before the main upload finishes. For high-stakes flows
where "did Alice send X?" is itself sensitive, send the preview
*only* after main completion (set `thumbnail` to `null` and instead
ship a follow-up `stream-init` with the preview). The default
ordering optimizes UX, not metadata-secrecy.
- **Renderer trust.** We render through a Blob-URL `<img>`. A 0-day
in the browser's image decoder would still reach the receiver. Keep
browsers patched; rely on the CSP of your embedding app.
## API surface
See package READMEs:
- `packages/shade-streams/README.md` — crypto + state machines
- `packages/shade-transfer/README.md` — orchestration, transports, persistence
- `packages/shade-transport-webrtc/README.md` — V3.11 P2P transport plug-in
- `packages/shade-sdk/README.md` — magic drop-in
- `packages/shade-widgets/README.md` — React UI
## Transports
`@shade/transfer` ships HTTP + WebSocket chunk transports. V3.11 adds an
opt-in P2P chunk transport via `RTCDataChannel`:
- HTTP — `ShadeTransferHttpTransport`. POST per chunk; the receiver-
side route is `app.route('/v1/transfer', await shade.transferRoute())`.
- WebSocket — `ShadeTransferWsTransport`. One connection per peer,
binary-framed chunks, JSON acks; same wire format inside the frame as
the WebRTC transport.
- WebRTC — `WebRtcTransferTransport` from `@shade/transport-webrtc`.
Wired automatically by `shade.configureWebRTC()` as the primary
layer of a `MultiTransportFallback([webrtc, http])`. See
[docs/webrtc.md](./webrtc.md).
`MultiTransportFallback` is the N-ary generalisation of
`FallbackTransferTransport`: pass an ordered list of named transports
and the engine demotes sticky on `TransferTransportError`.

224
docs/transport.md Normal file
View File

@@ -0,0 +1,224 @@
# Shade Transport — Bridge Layer (V3.7)
> **Looking for V3.11 (peer-to-peer chunk transport via `RTCDataChannel`)?**
> See [docs/webrtc.md](./webrtc.md). This page covers the V3.7 bridge
> layer that ships ciphertext *envelopes* (control plane) over
> WS / SSE / long-poll. The two are orthogonal: the bridge handles
> store-and-forward control envelopes; WebRTC handles direct chunk data.
The bridge layer is the answer to: **"my client is a browser extension /
strict-corp-proxy / edge-runtime / iOS app — I cannot keep a WebSocket
open. How do I receive ciphertext envelopes?"**
It is built on top of the V3.6 inbox: every transport delivers the same
inbox blobs, with the same authentication semantics. Application code
sees a single `IncomingMessage` shape and never branches on transport.
```
┌─────────────────────────────────────────────────────────────────┐
│ application code │
│ │
│ bridge.connect({ onMessage: (m) => decrypt(m.bytes) }) │
└────────────────────────────────┬────────────────────────────────┘
│
┌─────────────────────────┴──────────────────────────┐
│ FallbackBridgeTransport │
│ (sticky-after-first-success) │
└──┬──────────────────┬─────────────────────────┬────┘
│ │ │
┌──────▼─────┐ ┌──────▼─────┐ ┌──────▼─────┐
│ WsBridge │ │ SseBridge │ │ LongPoll │
│ /v1/ │ │ /v1/ │ │ Bridge │
│ bridge/ws │ │ bridge/ │ │ /v1/bridge │
│ │ │ stream │ │ /poll │
└──────┬─────┘ └──────┬─────┘ └──────┬─────┘
│ │ │
└──────────────────┼─────────────────────────┘
│
┌─────▼──────┐
│ inbox │ ← the same V3.6 store
│ blobs │ and events
└────────────┘
```
## When to reach for which
| Transport | Latency | Proxy resilience | Browser | Server cost |
|-------------|----------|------------------|---------|-------------|
| WebSocket | ms | breaks under strict CONNECT-blocking proxies | ✓ | one socket per client |
| SSE | ms | passes most HTTP proxies (text/event-stream) | ✓ | one streamed response per client |
| long-poll | ≤ 25 s | passes anything that allows GET | ✓ | one held request per client |
The recommended composition:
```ts
import {
FallbackBridgeTransport,
WsBridge,
SseBridge,
LongPollBridge,
} from '@shade/transport-bridge';
const auth = {
crypto, // CryptoProvider
signingPrivateKey, // recipient's Ed25519 private key
address: 'bob',
};
const bridge = new FallbackBridgeTransport([
new WsBridge({ baseUrl: 'https://relay.example.com', auth }),
new SseBridge({ baseUrl: 'https://relay.example.com', auth }),
new LongPollBridge({ baseUrl: 'https://relay.example.com', auth }),
]);
await bridge.connect({
onMessage: async (msg) => {
// msg.bytes is a Uint8Array — pass it to your decrypt path.
// msg.from is the relay-known sender hint (may be empty); the
// authoritative sender comes from the decrypted envelope.
// msg.msgId is the relay's deterministic message id (sha256(ciphertext)).
const envelope = decodeEnvelope(msg.bytes);
await shade.receive(senderAddress, envelope);
},
});
// Read which transport the fallback chain settled on:
console.log(bridge.activeKind); // "ws" | "sse" | "long-poll"
```
## The IncomingMessage shape
```ts
interface IncomingMessage {
from: string; // relay-side sender hint (may be "")
bytes: Uint8Array; // the ciphertext envelope, exactly as PUT
receivedAt: number; // relay-monotonic cursor — NOT wall-clock arrival
msgId?: string; // sha256(bytes) — useful for ack/dedup
}
```
`from` is intentionally a hint — sender provenance lives inside the
encrypted envelope and is recovered post-decrypt. The bridge layer is
plaintext-blind by design.
## Auth — signed query parameters
Every bridge request signs the canonical
`{address, kind, since, signedAt}` payload with the recipient's Ed25519
signing private key. The server looks up the address-owner key
registered via `/v1/inbox/register` and verifies the signature.
`kind` is bound into the canonical payload so a signature for `/poll`
cannot be replayed against `/stream` or `/ws`.
The browser `EventSource` API does not let callers attach custom
headers; query parameters are the only portable carrier and so the
bridge protocol uses them uniformly across all three transports.
## Server-side — `createBridgeRoutes`
```ts
import { createBridgeRoutes } from '@shade/inbox-server';
import { Hono } from 'hono';
const inbox = new MemoryInboxStore();
const events = new InboxServerEvents();
const bridge = createBridgeRoutes({
store: inbox,
crypto,
events,
longPollTimeoutMs: 25_000, // default — under typical proxy idle limits
heartbeatIntervalMs: 15_000, // SSE keepalive comments
fallbackPollIntervalMs: 1_000, // when no `events` emitter is wired
});
const app = new Hono();
app.route('/', bridge.app);
Bun.serve({
port: 3900,
fetch: (req, srv) => app.fetch(req, srv),
websocket: bridge.websocket as any,
});
```
The bridge subscribes to `InboxServerEvents` (`inbox.blob_stored`) for
push-style delivery — when an event fires for a connected address, the
server fetches new blobs and forwards them. If no events emitter is
wired, the server falls back to a small in-process polling timer at
`fallbackPollIntervalMs` cadence.
## Cursor & resume
Every `IncomingMessage.receivedAt` is the relay's monotonic cursor for
the address. Bridges expose `getCursor()` so applications can persist
the high-water mark and pass it as `startCursor` on the next
`connect()`:
```ts
const sse = new SseBridge({
baseUrl,
auth,
startCursor: await persistedCursor.load(),
});
await sse.connect({
onMessage: async (msg) => {
await persistedCursor.save(msg.receivedAt);
// …
},
});
```
For SSE specifically, the server emits an `id:` field per event; the
bridge sends it back as `Last-Event-ID` plus the `since=` query
parameter on reconnect, so a flapping connection picks up exactly where
it left off without duplicates.
## Reconnect & backoff
| Bridge | Auto-reconnect | Backoff |
|-------------|----------------|----------------------|
| WS | yes (default) | 250 ms → 10 s exponential |
| SSE | yes (default) | 250 ms → 10 s exponential |
| long-poll | always on (the loop *is* the reconnect) | 2 s on hard error |
Pass `disableAutoReconnect: true` (WS / SSE) for tests where you want a
single attempt and immediate surfaced error.
## Long-poll concurrency
The `LongPollBridge` issues exactly one request at a time. The next
request fires after the previous one resolves. This guarantees a
client never holds more than one TCP connection on the server, which
matches the V3.7 acceptance criterion and keeps capacity planning
simple: max in-flight long-poll requests = number of connected clients.
## Failure modes
- **WS handshake rejected (4xxx code).** `WsBridge.connect` rejects.
Caller (or `FallbackBridgeTransport`) moves on.
- **SSE returns non-200.** `SseBridge.connect` throws a `BridgeError`
with `httpStatus`.
- **Long-poll returns non-200.** Same — `BridgeError` with `httpStatus`.
- **Mid-stream error after connect.** WS/SSE auto-reconnect; long-poll
swallows transient errors and continues looping. Errors flow to the
caller's `onError` handler.
## Acceptance test coverage (V3.7)
`packages/shade-transport-bridge/tests/bridge.test.ts` covers:
- "Send 100 small messages" — one test per transport, all pass.
- "WS blocked by proxy → SSE → long-poll" — fallback test boots a
server where the WS endpoint is unreachable and the SSE endpoint
returns 502, verifies the chain falls all the way through to
long-poll without message loss.
- "Long-poll uses ≤ 1 outstanding request" — wraps `fetch` to count
in-flight requests over 1.5 s of steady-state operation.
- Cursor resume — tears down an SSE connection mid-stream, pushes more
blobs, reconnects with the persisted cursor, asserts exactly the new
blobs are delivered (no overlap with the pre-disconnect set).
- Auth rejection — wrong signing key and unregistered address both
produce hard `connect` rejections so the fallback chain advances.

156
docs/trust-ux.md Normal file
View File

@@ -0,0 +1,156 @@
# Trust UX — Fingerprint Gates (V3.3)
> Status: shipped in 0.4.0, GA-frozen in 4.0 — see [V3.3 plan](./archive/V3.3.md).
Shade ships with a small number of **blocking** verification gates that
fire automatically before the operations where MITM risk is highest.
Each gate calls a handler you register on the SDK; until the user (or
your handler) approves, the operation aborts with
`FingerprintNotVerifiedError`.
The point of the gate model is to be alert-fatigue-free: you don't see
a prompt before every chat message, just before the handful of moments
that genuinely matter.
---
## What the gates protect
| Gate | Fires when | Default policy |
|------|------------|----------------|
| `first-large-file` | `Shade.upload(...)` for an unverified peer with a known size at or above the configured threshold. | Threshold `10 MiB`. Below = no gate. |
| `backup-import` | `Shade.importBackup(...)` before any state is written. Handler receives the fingerprint of the identity *embedded in the backup*. | Always fires. |
| `new-device-trust` | `Shade.acceptIdentityChange(...)` after a peer rotates identity. The peer's `identity_version` is bumped first so any prior verification is automatically stale. | Always fires. |
| `inbox-fanout` | Reserved for V3.6 (`@shade/inbox`). Per-recipient hook is wired today so apps can register it now. | Always fires. |
---
## Registering handlers
```ts
const shade = await createShade({
prekeyServer: 'https://prekeys.example.com',
storage: 'sqlite:/data/shade.db',
});
shade.beforeFirstLargeFile(10 * 1024 * 1024, async (ctx) => {
// ctx.peerAddress, ctx.fingerprint, ctx.fileSize
return await ui.confirmFingerprintModal(ctx);
});
shade.beforeBackupImport(async (ctx) => {
// ctx.fingerprint = fingerprint of the identity in the backup blob
return await ui.confirmBackupOwner(ctx);
});
shade.beforeNewDeviceTrust(async (ctx) => {
// ctx.fingerprint = fingerprint of the rotated identity
return await ui.confirmDeviceRotation(ctx);
});
```
Return `true` to allow the operation and persist a `'user'` verification.
Return `false` (or throw) to abort with `FingerprintNotVerifiedError`.
If you don't register a handler, the gate **logs a one-time warning per
peer and proceeds on TOFU**, persisting a `'tofu-after-warning'`
verification. This satisfies the V3.3 acceptance criterion that apps
without registered gates get sane defaults instead of hard-failing — but
it does mean the gate is informational, not a hard wall, in that
configuration. Always register handlers in production.
---
## Manual verification
The handler model assumes your app drives the OOB compare/confirm
flow. If the user verifies through some other path (QR code scan, audio
read-aloud, transitive trust from V3.10), call:
```ts
await shade.markPeerVerified('bob'); // pin current fingerprint
await shade.unmarkPeerVerified('bob'); // revoke
const ok = await shade.isPeerVerified('bob'); // check status
```
`markPeerVerified` reads the peer's *current* fingerprint and pins it
together with the per-peer `identity_version`. When the peer rotates
(`acceptIdentityChange`), the version bumps and the saved verification
goes stale automatically — `isPeerVerified` will return `false` until
the user re-verifies.
---
## Tuning thresholds
The `first-large-file` threshold is the only knob that's customer-tunable
without code changes. The defaults are conservative:
- **Default:** `10 MiB`. Big enough that ordinary chat attachments don't
trigger; small enough that obvious "exfil candidates" do.
- **Lower** (e.g. `1 MiB`) for high-sensitivity deployments — every
document goes through the gate.
- **Raise** (e.g. `100 MiB`) only for use cases where small uploads are
routine and large transfers are deliberate / pre-arranged.
`backup-import` and `new-device-trust` have no threshold by design — the
spec mandates an irremovable minimum gate for both, since each one
either trusts a fresh identity or overwrites pinned trust wholesale.
---
## React widget
Use `<FingerprintGate />` from `@shade/widgets` to block UI on
verification status:
```tsx
import { FingerprintGate } from '@shade/widgets';
<FingerprintGate peerAddress="bob">
<ChatThread peer="bob" />
</FingerprintGate>
```
The default fallback shows the safety number, a "Copy OOB text" button,
and an "I have verified" button that calls `Shade.markPeerVerified`.
Pass a `fallback` render prop to use your own UI, or `onVerified` to
react to the unverified → verified transition.
`<FingerprintCompare />` is the existing observer-dashboard widget; it
now exposes the same Copy-OOB / verify actions when an `onVerified`
prop is wired.
---
## Errors
`FingerprintNotVerifiedError` carries:
- `peerAddress` — the address the gate was protecting.
- `gate` — `'first-large-file' | 'backup-import' | 'new-device-trust' | 'inbox-fanout'`.
- `code = 'SHADE_FINGERPRINT_NOT_VERIFIED'` — maps to HTTP 403.
Catch it explicitly when wrapping `upload`, `importBackup`, and
`acceptIdentityChange`:
```ts
try {
await shade.upload({ to: 'bob', input: bytes });
} catch (err) {
if (err instanceof FingerprintNotVerifiedError) {
showVerifyFirst(err.peerAddress);
return;
}
throw err;
}
```
---
## Migration from 0.3.x
No breaking changes: existing apps gain warning-mode gates automatically
(see the no-handler note above). To upgrade to hard gates, register
handlers for the operations you use. Your existing `FingerprintCompare`
calls keep working; pass `onVerified` to enable the new actions.

276
docs/web-workers.md Normal file
View File

@@ -0,0 +1,276 @@
# Web Workers Crypto
Status: Implemented (V3.8 — `0.4.0`).
`@shade/crypto-web` ships with an opt-in dedicated Web Worker that keeps
AES-GCM, HKDF, HMAC, X25519 and Ed25519 — and full per-lane stream state —
off the main thread. Big in-browser uploads (100 MB+) stay smooth without
frame drops.
This doc covers:
- [When to use it](#when-to-use-it)
- [Setup](#setup)
- [API](#api)
- [Bundler recipes](#bundler-recipes)
- [Safari notes](#safari-notes)
- [SharedArrayBuffer (COOP/COEP)](#sharedarraybuffer-coopcoep)
- [Lifecycle and rotation](#lifecycle-and-rotation)
- [Threat-model considerations](#threat-model-considerations)
---
## When to use it
The default `SubtleCryptoProvider` runs on whatever thread you give it.
For the SDK that means the main thread. AES-GCM via SubtleCrypto is fast
(hardware-accelerated), but a 100 MB file at 256 KiB chunks is ~400 AEAD
calls — each one queues a microtask on the main thread. Layered on top of
React reflows and large `postMessage` payloads to the network worker, you
*will* see frame drops.
Reach for the Worker pipeline when:
- You upload or download files that don't fit in a single AEAD chunk
(≥ ~1 MB) inside a UI-bearing browser tab.
- You generate or rotate identity / device keys in a UI thread that must
stay interactive.
- You do batch AEAD (e.g. backup export over many records).
You can keep using `SubtleCryptoProvider` for short ops (Signal session
encrypt/decrypt for a chat message). The cost of a `postMessage` round-
trip dwarfs the cost of a single 256-byte AES call.
---
## Setup
`@shade/crypto-web` exposes the worker as a separate subpath, so your
bundler can resolve it through the standard `new Worker(new URL(...,
import.meta.url))` idiom.
```ts
import { createShade } from '@shade/sdk';
const shade = await createShade({ /* ... */ });
shade.configureWorkerCrypto({
workerUrl: new URL('@shade/crypto-web/worker', import.meta.url),
});
```
After `configureWorkerCrypto`, the SDK exposes:
- `shade.encryptStream({ streamId, streamSecret, ... })` — returns a
`TransformStream<Uint8Array, Uint8Array>` and a `laneSha256` promise.
- `shade.decryptStream({ streamId, streamSecret, ... })` — inverse.
- `shade.getWorkerCrypto()` — direct access to the `WorkerCryptoProvider`
for one-off ops (HKDF batches, X25519 batch DH, etc.).
The worker is spawned on first use and self-terminates after
`idleTimeoutMs` (default 30 s) — no manual lifecycle management required.
---
## API
### Stream encryption
```ts
const { stream, laneSha256 } = await shade.encryptStream({
streamId: streamId, // 16 random bytes, agreed with peer
streamSecret: streamSecret,// 32 random bytes, derived via Double Ratchet
laneId: 0, // lane index (use multi-lane for parallel HTTP)
chunkSize: 256 * 1024, // optional; default 256 KiB
});
await file.stream()
.pipeThrough(stream)
.pipeTo(transferSink); // your HTTP-shipping WritableStream
const sha256 = await laneSha256; // for end-to-end integrity proof
```
`stream` consumes plaintext and emits one wire-encoded
`stream-chunk` envelope per write. `flush` always emits a final chunk
with `isLast=true` (even if the trailing slice is empty), so receivers
see a clean termination.
### Stream decryption
```ts
const { stream, laneSha256 } = await shade.decryptStream({
streamId,
streamSecret,
laneId: 0,
});
await incomingChunkStream
.pipeThrough(stream)
.pipeTo(fileSink);
const sha = await laneSha256;
if (!equal(sha, peerLaneSha256)) throw new IntegrityError();
```
Each input chunk MUST be a complete wire envelope. The transport-layer
caller is responsible for framing (one envelope per write). Out-of-order
or replayed chunks reject the stream — the lane key never crosses thread
boundaries, so a man-in-the-middle script in the page can't recover key
material to replay against.
### Direct provider access
```ts
const crypto = await shade.getWorkerCrypto();
// Implements `CryptoProvider` — drop-in replacement for SubtleCryptoProvider
const { ciphertext, nonce } = await crypto.aesGcmEncrypt(key, plaintext);
```
`randomBytes`, `randomUint32`, `constantTimeEqual`, `zeroize` execute on
the calling thread (no round-trip). Async ops forward to the worker.
---
## Bundler recipes
### Vite
```ts
shade.configureWorkerCrypto({
workerUrl: new URL('@shade/crypto-web/worker', import.meta.url),
});
```
Vite resolves the URL via `import.meta.url` and emits a discrete chunk
for the worker. No additional config required for Vite ≥ 5.
If your build complains about `?worker` syntax, use the explicit URL
form (above) — it's the standard Vite idiom.
### Webpack 5 / Rspack
Same idiom — Webpack 5 understands `new URL('./worker.js', import.meta.url)`
natively as long as the source is ESM:
```ts
new Worker(new URL('@shade/crypto-web/worker', import.meta.url), {
type: 'module',
});
```
For Webpack 4 or non-ESM builds, you need `worker-loader` (legacy). We
do not officially support Webpack 4.
### Rollup
Rollup needs `@rollup/plugin-web-worker-loader` or a recent
`rollup-plugin-import-meta-url`. The standard idiom works once the
plugin is wired:
```ts
new URL('@shade/crypto-web/worker', import.meta.url)
```
If your bundler can't resolve `@shade/crypto-web/worker`, copy
`node_modules/@shade/crypto-web/src/worker.ts` (or the compiled `.js`
once we ship dist artefacts) into your `public/` directory and pass an
absolute URL:
```ts
shade.configureWorkerCrypto({ workerUrl: '/shade-crypto.worker.js' });
```
---
## Safari notes
Safari ≤ 17 has a smaller `postMessage` transferable budget than Chrome /
Firefox. Single transfers above ~64 MB occasionally fail silently. The
shipped pipeline already chunks plaintext to 256 KiB before AEAD, so
each `postMessage` carries ≤ ~256 KiB + AEAD overhead — well under any
known Safari limit.
If you override `chunkSize`, keep individual buffers below 16 MiB:
```ts
shade.encryptStream({
streamId, streamSecret,
chunkSize: 8 * 1024 * 1024, // 8 MiB — safe across all browsers
});
```
We do not officially support Safari ≤ 14 (no module workers).
---
## SharedArrayBuffer (COOP/COEP)
The default pipeline uses `ArrayBuffer` transfer (zero-copy ownership
hand-off). It does **not** require COOP/COEP headers.
For multi-lane parallel transfers across multiple workers, you may opt
in to `SharedArrayBuffer` for the AEAD plaintext buffers. That requires
your origin to serve:
```
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
```
`SharedArrayBuffer` support is gated behind a future `useSharedBuffers`
option and is not enabled in V3.8. See `docs/V4.0.md` if/when this lands.
---
## Lifecycle and rotation
```ts
const crypto = await shade.getWorkerCrypto();
await crypto.rotate(); // tear down the current worker, respawn lazily
await crypto.destroy(); // permanent — every subsequent call rejects
```
`shade.shutdown()` calls `destroy()` automatically. The idle-timer fires
30 seconds after the last response (configurable via
`configureWorkerCrypto({ idleTimeoutMs })`); if the timer fires while
calls are pending, it does nothing and reschedules.
---
## Threat-model considerations
- The worker runs in the same origin and the same browsing context as
the main thread. It is **not** a sandbox against a compromised page;
any script that can `eval` in your tab can also `postMessage` to the
worker. The Worker is a *performance* boundary, not a *security*
boundary.
- Lane keys derived inside the worker stay there; they are never
postMessage'd to the main thread. This narrows the window during which
a key sits in main-thread heap, which helps against post-mortem heap
inspection by a curious extension. It does not help against an active
in-page attacker.
- `randomBytes` runs on the calling thread (uses `crypto.getRandomValues`
directly). The worker has its own random source for ops that derive
inside it (nonces are derived deterministically from `(laneId, seq)`).
For the full picture, see `THREAT-MODEL.md`.
---
## Verifying main-thread budget
V3.8 acceptance: 100 MB upload in Chrome without main thread blocked
> 16 ms in P99.
To verify in your app:
1. Open Chrome DevTools → Performance.
2. Record a 100 MB upload.
3. Inspect the main-thread flame chart. Look at "Long Tasks" and
"Self time" of `Shade.encryptStream`.
4. Confirm no contiguous block exceeds ~16 ms (one frame at 60 fps).
If you observe long tasks, lower `chunkSize` (more frequent yields) or
report the trace — see [`docs/archive/V3.8.md`](./archive/V3.8.md) for
the original acceptance criteria.

302
docs/webrtc.md Normal file
View File

@@ -0,0 +1,302 @@
# Shade Transport — WebRTC P2P Layer (V3.11)
`@shade/transport-webrtc` adds a direct peer-to-peer chunk transport on
top of the existing `@shade/transfer` engine. When two clients can reach
each other through NAT/firewall, large transfers (`@shade/files`,
`@shade/transfer`) flow over a single bidirectional `RTCDataChannel`
instead of paying the round-trip cost of HTTP-relayed POSTs. When NAT
traversal fails, the multi-transport fallback automatically demotes the
chain back to HTTP — without losing any chunks already in flight.
The wire payload is unchanged: every chunk is still a Shade ratchet /
streams envelope (AES-256-GCM under HKDF-derived per-lane keys). DTLS-
SRTP is only the WebRTC transport secret; turning a TURN-relay on does
not give the relay operator access to plaintext.
```
┌───────────────────────────────────────────────────────────────┐
│ application code │
│ │
│ shade.upload({ to: 'bob', input: file }) │
└────────────────────────────────┬──────────────────────────────┘
│
┌─────────▼──────────┐
│ TransferEngine │
└─────────┬──────────┘
│ ITransferTransport
┌─────────▼──────────┐
│ MultiTransport │
│ Fallback (sticky) │
└────┬─────┬─────┬───┘
│ │ │
┌─────────────▼┐ ┌─▼─┐ ┌▼────────────┐
│ WebRtcTransfer│ │WS │ │ ShadeTransfer│
│ Transport │ │… │ │ HttpTransport│
└─────┬─────────┘ └───┘ └──────────────┘
│ DataChannel binary frames
┌─────▼─────────┐
│ WebRtcConn │ ←──── SDP/ICE over Shade.send
│ Manager │ (ratchet-encrypted)
└───────────────┘
```
## When to reach for it
| Scenario | Default (HTTP) | + WebRTC |
|---------------------------------------|----------------|----------------|
| Two clients on the same LAN | server-relayed | direct, P2P |
| One peer behind enterprise NAT only | works | TURN-relay |
| Both peers behind symmetric NAT | works | falls back to HTTP |
| One peer offline | inbox-buffered | inbox-buffered (HTTP path) |
| Browser extension with strict CSP | works | works (uses RTCPeerConnection) |
Use cases:
- `@shade/transfer` upload of multi-MB / multi-GB files
- `@shade/files` `read`/`write` of large inline blobs
- Future: `@shade/streams` real-time channels (V5.0 reuses this same DataChannel)
## Quick start (browser)
```ts
import { createShade } from '@shade/sdk';
import { nativeRtcFactory } from '@shade/transport-webrtc';
const shade = await createShade({ prekeyServer: 'https://prekey.example.com' });
// IMPORTANT: configureWebRTC MUST be called BEFORE the first upload() /
// onIncomingTransfer() / transferRoute() call, because those build the
// transfer engine — and the engine captures its transport stack at
// construction time.
shade.configureWebRTC({
factory: nativeRtcFactory(),
// Optional — defaults to two public Google STUN servers.
iceServers: [
{ urls: 'stun:stun.l.google.com:19302' },
{
urls: 'turn:turn.example.com:3478',
username: 'shade',
credential: 'YOUR_TURN_SECRET',
},
],
});
shade.configureTransfers({
resolveBaseUrl: async (peer) => directory.lookup(peer),
});
await shade.upload({ to: 'bob', input: file }); // → P2P when NAT allows
```
## Quick start (Bun / Node)
Bun does not yet expose `RTCPeerConnection` natively. Use one of:
- [`node-datachannel`](https://github.com/murat-dogan/node-datachannel)
— small, stable, libdatachannel under the hood
- [`@roamhq/wrtc`](https://www.npmjs.com/package/@roamhq/wrtc) — fork of
the Google `wrtc` bindings
Wrap the chosen library behind an `IRtcFactory` (the package only depends
on a narrow surface — `createPeerConnection`, `createDataChannel`,
`addEventListener`):
```ts
import { IRtcFactory, IPeerConnection, IDataChannel } from '@shade/transport-webrtc';
// pseudo-adapter for node-datachannel
class NodeDataChannelFactory implements IRtcFactory {
createPeerConnection(config) { /* ... return adapter wrapping nodeDc PeerConnection */ }
}
shade.configureWebRTC({ factory: new NodeDataChannelFactory(), iceServers });
```
## Connection flow
```
Alice initiates Bob receives
─────────────── ────────────
1. createOffer() → SDP 2. shade.send delivers offer
→ Bob.createAnswer()
3. shade.send delivers answer 4. setRemoteDescription(answer)
5. trickle ICE candidates (both directions) 6. trickle ICE candidates
7. DataChannel onopen (both sides) 7. DataChannel onopen
```
All four signaling kinds (`shade.webrtc-offer/v1`, `shade.webrtc-answer/v1`,
`shade.webrtc-ice/v1`, `shade.webrtc-bye/v1`) ride the existing Shade
ratchet — the relay sees only ciphertext envelopes.
### Glare resolution
If both peers call `getOrCreate()` simultaneously, the manager uses
lexicographic tiebreak: the side with the smaller address wins
caller-role; the side with the larger address closes its outgoing
connection and accepts the inbound offer instead. Both peers ultimately
converge on a single `WebRtcConnection`.
## Backpressure
The `WebRtcTransferTransport` polls `RTCDataChannel.bufferedAmount` and
suspends new sends once the buffer crosses `backpressureThresholdBytes`
(default 4 MiB). This avoids SCTP queue runaway when the application
pushes faster than the network can drain. Tune lower for memory-
constrained clients (mobile / extension contexts).
## Auto-fallback
Configuring WebRTC wires `MultiTransportFallback([webrtc, http])` as the
engine's transport. The chain is sticky-after-first-failure: when WebRTC
raises a `TransferTransportError` (timeout, ICE failed, data channel
closed, frame too large), the fallback advances to HTTP and stays there
for the lifetime of the engine.
For three-tier composition (e.g. WebRTC → WebSocket → HTTP), build the
fallback yourself and pass a custom transport via the engine deps:
```ts
import { MultiTransportFallback } from '@shade/sdk';
const stack = new MultiTransportFallback([
{ name: 'webrtc', transport: rtcTransport },
{ name: 'ws', transport: wsTransport },
{ name: 'http', transport: httpTransport },
]);
stack.onSwitch((from, to) => metrics.observe('shade.transport.demoted', { from, to }));
```
The `WebRtcConnectionManager`'s connect timeout (default 30 s) is the
upper bound on how long the chain dwells on WebRTC before demoting. The
V3.11 acceptance criterion is "P2P-død → HTTP innen 5 s" — set
`connectTimeoutMs: 4_000` in your `configureWebRTC()` call to keep the
upper bound at 4 seconds and meet the SLO with margin.
## ICE server config
| Setting | Default | When to override |
|------------------------|-----------------------------------|------------------|
| `iceServers` | Google public STUN (×2) | Production — pin your own STUN to avoid Google rate limits, plus your TURN credentials |
| `iceTransportPolicy` | `'all'` (host + reflexive + relay)| `'relay'` to mandate TURN-only routing (e.g. inside a corporate network where direct connectivity must never leak) |
| `bundlePolicy` | spec default (`'balanced'`) | rarely |
Public STUN works for ~80% of consumer NATs. The remaining 20% (symmetric
NAT, paranoid corporate proxies, mobile carrier-grade NAT) need TURN.
Run your own [coturn](https://github.com/coturn/coturn) or use a managed
provider — but **TURN traffic is real bandwidth through your server**, so
budget accordingly. Shade's wire format is at least as efficient over
TURN as over HTTPS (no per-request HTTP framing overhead).
## NAT-traversal: hopes and realities
What works without TURN, in our testing:
- Same NAT (LAN): always
- Two clients behind cone NATs: usually
- One client behind symmetric NAT, the other behind any cone NAT: usually
- Two clients behind symmetric NATs: rarely — falls back to TURN
What doesn't work:
- Two clients behind strict carrier-grade NAT (CGNAT): TURN required
- Clients on networks that block UDP entirely: TURN over TCP/443 required
When in doubt, configure TURN over TCP/443 — it impersonates HTTPS and
gets through nearly every middlebox.
## Diagnostics
The SDK exposes the live runtime via `shade.getWebRtcRuntime()`:
```ts
const runtime = shade.getWebRtcRuntime();
if (runtime !== null) {
console.log('active transport:', runtime.fallback.activeName);
console.log('peers:', [...runtime.manager.byPeer ?? []]);
runtime.fallback.onSwitch((from, to) => {
console.warn(`shade transport demoted ${from} → ${to}`);
});
}
```
The `failures` array on `MultiTransportFallback` records every
demotion's reason — wire it to your observability backend to track
NAT/TURN problems in production.
## Sample code
End-to-end test using `MemoryRtcFactory` (no real network):
```ts
import { MemoryRtcFactory } from '@shade/transport-webrtc';
const factory = new MemoryRtcFactory();
alice.configureWebRTC({ factory });
bob.configureWebRTC({ factory });
await alice.upload({ to: 'bob', input: bytes }); // → P2P loopback
```
See `packages/shade-sdk/tests/webrtc-integration.test.ts` for the full
loopback test, `webrtc-failover.test.ts` for the auto-fallback test, and
`packages/shade-transport-webrtc/tests/` for the unit tests covering
wire format, signaling, glare, and TURN-only configuration.
## Wire format inside the DataChannel
The DataChannel is a single bidirectional pipe shared by every in-flight
stream between two peers. Each frame is a self-describing binary blob:
```
client → server server → client
─────────────── ───────────────
0x01 chunk reqId(16) sid(16) lane(u32) seq(u64) env(...) 0x81 chunk-ack reqId(16) lastSeq(u32) bytesRecv(u32)
0x02 resume-query reqId(16) sid(16) 0x82 resume-state reqId(16) jsonBody(utf-8)
0x03 ping reqId(16) nonce(u64) 0x83 pong reqId(16) nonce(u64)
0xFE error reqId(16) jsonBody(utf-8)
```
`reqId` is a 16-byte random correlation token; the responder echoes it
verbatim so multiple in-flight requests can be matched without a stream
multiplexer on top of SCTP.
The wire matches `ShadeTransferWsTransport` exactly — adapters for
either transport can interoperate by translating between SCTP message-
framing and WS binary frames at the byte level.
## Limits
- Max DataChannel message: **256 KiB** (Chrome's safe ceiling). Configure
`chunkSize` ≤ 256 KiB on uploads that prefer WebRTC. The transport
raises a clear error when an envelope exceeds the cap; the engine then
retries via HTTP.
- One DataChannel per peer pair (label `shade-transfer/v1`). Multiple
in-flight transfers from the same peer pair multiplex via `reqId`.
- No SFU/MCU — group transfers fan out at the application layer.
- DTLS-fingerprint binding to Shade's identity-fingerprint is **not** in
V3.11 (deferred as hardening work — DataChannel is already inside a
ratchet-authenticated session, so the practical exposure window is
limited to in-process MITM scenarios that already require malware).
## Migration
Opt-in. If you don't call `configureWebRTC`, your existing HTTP/WS
transport stack is unchanged.
When you do opt in, the **engine must not be built yet** — the easy way
to ensure this is to call `configureWebRTC` before `configureTransfers`
or before any of `upload` / `onIncomingTransfer` / `transferRoute`.
Receiver-side: the WebRTC manager wires receiver-hooks into the engine
during `engine()` construction, so make sure both sides do `configureWebRTC`
+ `configureTransfers` before the first `transferRoute()` call.
## Related modules
- [`@shade/transfer`](../packages/shade-transfer/) — engine, lane queues,
HTTP transport, multi-fallback wrapper.
- [`@shade/streams`](./streams.md) — chunk encryption + lane key
derivation. Indirect dep.
- [`@shade/transport-bridge`](./transport.md) — V3.7 bridge layer (WS /
SSE / long-poll for control envelopes). Orthogonal to V3.11.
- [V5.0 — real-time channels](./V5.0.md) — downstream consumer of the
same DataChannel for voice/video/broadcast.

6
package.json
View File

@@ -11,9 +11,13 @@
"test:transport": "cd packages/shade-transport && bun test",
"test:sdk": "cd packages/shade-sdk && bun test",
"test:cli": "cd packages/shade-cli && bun test",
"test:vectors": "bun test packages/shade-core/tests/cross-platform-vectors.test.ts",
"vectors:gen": "bun run scripts/generate-vectors.ts",
"version": "bun run scripts/bump-version.ts",
"soak": "bun run scripts/soak.ts",
"soak:smoke": "bun run scripts/soak.ts --hours 0.05 --pairs 4",
"publish:dry": "DRY_RUN=1 bun run scripts/publish-all.ts",
"publish:all": "bun run scripts/publish-all.ts",
"publish:all": "bash scripts/publish-shade.sh",
"build:docker": "bun run scripts/build-docker.ts",
"publish:docker": "bun run scripts/build-docker.ts -- --push"
},

4
packages/shade-cli/package.json
View File

@@ -1,6 +1,6 @@
{
"name": "@shade/cli",
"version": "0.3.0",
"version": "4.0.0",
"type": "module",
"main": "src/cli.ts",
"bin": {
@@ -9,7 +9,9 @@
"dependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/keychain": "workspace:*",
"@shade/sdk": "workspace:*",
"@shade/storage-encrypted": "workspace:*",
"@shade/storage-sqlite": "workspace:*",
"@shade/transport": "workspace:*"
},

31
packages/shade-cli/src/cli.ts
View File

@@ -12,6 +12,11 @@ import {
import { dashboardCommand } from './commands/dashboard.js';
import { doctorCommand } from './commands/doctor.js';
import { backupExportCommand, backupRestoreCommand } from './commands/backup.js';
import {
migrateStorageCommand,
rotateStorageKeyCommand,
parseStorageArgs,
} from './commands/storage.js';
const VERSION = '0.1.0';
@@ -24,6 +29,7 @@ Commands:
init [name] Scaffold a new Shade project
--template <name> Template to use (default: bun-server)
--prekey-server <url> Override prekey server URL
--doctor Run shade doctor against the new project
fingerprint [address] Print your own or a peer's fingerprint
publish Re-upload your bundle to the prekey server
rotate Rotate the signed prekey
@@ -37,6 +43,19 @@ Commands:
doctor Diagnose setup issues
backup export <file> Export an encrypted backup (prompts for passphrase)
backup restore <file> Restore from a backup file
migrate-storage Encrypt the local SQLite store at-rest (V3.2)
--key-source <kind> passphrase | keychain | injected
--passphrase <s> passphrase for KDF (or env SHADE_STORAGE_PASSPHRASE)
--keychain-service <s> keychain service name (default: shade.storage)
--keychain-account <s> keychain account name (default: default)
--key-hex <hex> inject a raw 32-byte key as hex
--salt-file <path> passphrase salt file (default: <db>.salt)
--dry-run validate + report without writing
--keep-original keep unencrypted tables after migration
--no-backup skip the .bak snapshot
rotate-storage-key Re-key the encrypted SQLite store (V3.2)
--key-source / --passphrase / … (current key — same flags as migrate)
--new-key-source / --new-passphrase / … (target key)
help Show this message
Config:
@@ -98,6 +117,16 @@ async function main(): Promise<void> {
}
break;
}
case 'migrate-storage': {
const opts = parseStorageArgs(args.slice(1));
await migrateStorageCommand(opts);
break;
}
case 'rotate-storage-key': {
const opts = parseStorageArgs(args.slice(1));
await rotateStorageKeyCommand(opts);
break;
}
case 'help':
case '--help':
case '-h':
@@ -125,11 +154,13 @@ function parseInitArgs(args: string[]): {
name?: string;
template?: string;
prekeyServer?: string;
runDoctor?: boolean;
} {
const options: ReturnType<typeof parseInitArgs> = {};
for (let i = 0; i < args.length; i++) {
if (args[i] === '--template') options.template = args[++i];
else if (args[i] === '--prekey-server') options.prekeyServer = args[++i];
else if (args[i] === '--doctor') options.runDoctor = true;
else if (!args[i]!.startsWith('--')) options.name = args[i];
}
return options;

8
packages/shade-cli/src/commands/doctor.ts
View File

@@ -1,15 +1,19 @@
import { tryLoadConfig } from '../config.js';
import { existsSync } from 'fs';
export interface DoctorOptions {
cwd?: string;
}
/**
* Diagnose common setup issues.
*/
export async function doctorCommand(): Promise<void> {
export async function doctorCommand(opts: DoctorOptions = {}): Promise<void> {
let ok = true;
console.log('\x1b[33mShade doctor\x1b[0m\n');
// 1. Config loadable?
const configResult = tryLoadConfig();
const configResult = tryLoadConfig(opts.cwd);
if (configResult.ok) {
console.log(' \x1b[32m✓\x1b[0m Config loaded from .shaderc.json or env vars');
const config = configResult.config;

7
packages/shade-cli/src/commands/init.ts
View File

@@ -1,6 +1,7 @@
import { existsSync, mkdirSync, writeFileSync, readdirSync, statSync, readFileSync } from 'fs';
import { join, dirname } from 'path';
import { fileURLToPath } from 'url';
import { doctorCommand } from './doctor.js';
const here = dirname(fileURLToPath(import.meta.url));
const TEMPLATES_DIR = join(here, '..', '..', 'templates');
@@ -10,6 +11,7 @@ export interface InitOptions {
template?: string;
prekeyServer?: string;
cwd?: string;
runDoctor?: boolean;
}
export async function initCommand(opts: InitOptions = {}): Promise<void> {
@@ -42,6 +44,11 @@ export async function initCommand(opts: InitOptions = {}): Promise<void> {
console.log(` cd ${name}`);
console.log(' bun install');
console.log(' bun run start');
if (opts.runDoctor) {
console.log('');
await doctorCommand({ cwd: target });
}
}
export function listTemplates(): string[] {

208
packages/shade-cli/src/commands/storage.ts Normal file
View File

@@ -0,0 +1,208 @@
/**
* `shade migrate-storage` and `shade rotate-storage-key` — at-rest encryption
* lifecycle commands (V3.2).
*
* Both rely on @shade/storage-encrypted; key sources mirror what KeyManager
* accepts: passphrase | keychain | injected (env-injected).
*/
import { existsSync, readFileSync, writeFileSync } from 'node:fs';
import { dirname, resolve } from 'node:path';
import { mkdirSync } from 'node:fs';
import { KeyManager, type KeySource, migrateSqliteToEncrypted, rotateSqliteEncryptionKey } from '@shade/storage-encrypted';
interface CommonStorageOptions {
dbPath?: string;
keySource: 'passphrase' | 'keychain' | 'injected';
passphrase?: string;
keychainService?: string;
keychainAccount?: string;
injectedKeyHex?: string;
saltFile?: string;
}
export interface MigrateCliOptions extends CommonStorageOptions {
dryRun?: boolean;
keepOriginal?: boolean;
noBackup?: boolean;
}
export interface RotateCliOptions extends CommonStorageOptions {
newKeySource: 'passphrase' | 'keychain' | 'injected';
newPassphrase?: string;
newKeychainService?: string;
newKeychainAccount?: string;
newInjectedKeyHex?: string;
newSaltFile?: string;
}
export async function migrateStorageCommand(opts: MigrateCliOptions): Promise<void> {
const dbPath = opts.dbPath ?? process.env.SHADE_DB_PATH ?? '/data/shade-client.db';
const km = await openKeyManager({ ...opts, dbPath, role: 'open-or-create' });
const log = (m: string) => console.log(` • ${m}`);
console.log(`Migrating ${dbPath} → encrypted at-rest schema…`);
if (opts.dryRun) console.log(' (dry-run: no writes will be made)');
const dryRun = opts.dryRun ?? false;
// Respect explicit --no-backup; otherwise inherit migrate's default
// (which is "no backup during dry-run, backup otherwise").
const backup = opts.noBackup ? false : !dryRun;
const report = await migrateSqliteToEncrypted({
dbPath,
keyManager: km,
log,
dryRun,
keepOriginal: opts.keepOriginal ?? false,
backup,
});
console.log('\nResult:');
console.log(` identity ${report.identity}`);
console.log(` config ${report.config}`);
console.log(` signed_prekeys ${report.signedPrekeys}`);
console.log(` one_time_prekeys ${report.oneTimePrekeys}`);
console.log(` sessions ${report.sessions}`);
console.log(` trusted_identities ${report.trustedIdentities}`);
console.log(` retired_identities ${report.retiredIdentities}`);
console.log(` stream_state ${report.streamStates}`);
console.log(` duration ${report.durationMs}ms`);
if (report.backupPath) console.log(` backup ${report.backupPath}`);
if (report.dryRun) console.log('\nDry-run complete; no data was modified.');
else if (!report.keptOriginal) console.log('\nUnencrypted tables dropped.');
}
export async function rotateStorageKeyCommand(opts: RotateCliOptions): Promise<void> {
const dbPath = opts.dbPath ?? process.env.SHADE_DB_PATH ?? '/data/shade-client.db';
if (!existsSync(dbPath)) throw new Error(`rotate-storage-key: DB not found at ${dbPath}`);
console.log(`Rotating storage key for ${dbPath}…`);
const oldKm = await openKeyManager({
dbPath,
role: 'open-existing',
keySource: opts.keySource,
...(opts.passphrase !== undefined ? { passphrase: opts.passphrase } : {}),
...(opts.keychainService !== undefined ? { keychainService: opts.keychainService } : {}),
...(opts.keychainAccount !== undefined ? { keychainAccount: opts.keychainAccount } : {}),
...(opts.injectedKeyHex !== undefined ? { injectedKeyHex: opts.injectedKeyHex } : {}),
...(opts.saltFile !== undefined ? { saltFile: opts.saltFile } : {}),
});
const newKm = await openKeyManager({
dbPath,
role: 'open-or-create',
keySource: opts.newKeySource,
...(opts.newPassphrase !== undefined ? { passphrase: opts.newPassphrase } : {}),
...(opts.newKeychainService !== undefined ? { keychainService: opts.newKeychainService } : {}),
...(opts.newKeychainAccount !== undefined ? { keychainAccount: opts.newKeychainAccount } : {}),
...(opts.newInjectedKeyHex !== undefined ? { injectedKeyHex: opts.newInjectedKeyHex } : {}),
...(opts.newSaltFile !== undefined ? { saltFile: opts.newSaltFile } : {}),
});
const log = (m: string) => console.log(` • ${m}`);
const result = await rotateSqliteEncryptionKey({ dbPath, oldKeyManager: oldKm, newKeyManager: newKm, log });
console.log(`\nRotation complete: ${result.rowsRotated} rows re-keyed.`);
}
// ─── helpers ──────────────────────────────────────────────────────
interface OpenKmArgs extends CommonStorageOptions {
/** open-existing: salt file MUST already exist for passphrase mode. open-or-create: generate salt if missing. */
role: 'open-existing' | 'open-or-create';
}
async function openKeyManager(args: OpenKmArgs): Promise<KeyManager> {
const source = await buildKeySource(args);
if (source.kind === 'keychain') {
const { getDefaultKeychain } = await import('@shade/keychain');
return KeyManager.open(source, { keychain: getDefaultKeychain() });
}
return KeyManager.open(source);
}
async function buildKeySource(args: OpenKmArgs): Promise<KeySource> {
switch (args.keySource) {
case 'passphrase': {
const passphrase = args.passphrase ?? process.env.SHADE_STORAGE_PASSPHRASE;
if (!passphrase) throw new Error('passphrase required: pass --passphrase or set SHADE_STORAGE_PASSPHRASE');
const dbPath = args.dbPath ?? process.env.SHADE_DB_PATH ?? '/data/shade-client.db';
const saltPath = args.saltFile ?? `${dbPath}.salt`;
const salt = await loadOrCreateSalt(saltPath, args.role);
return { kind: 'passphrase', passphrase, salt };
}
case 'keychain': {
const service = args.keychainService ?? process.env.SHADE_KEYCHAIN_SERVICE ?? 'shade.storage';
const account = args.keychainAccount ?? process.env.SHADE_KEYCHAIN_ACCOUNT ?? 'default';
return {
kind: 'keychain', service, account,
createIfMissing: args.role === 'open-or-create',
};
}
case 'injected': {
const hex = args.injectedKeyHex ?? process.env.SHADE_STORAGE_KEY_HEX;
if (!hex) throw new Error('injected key required: pass --key-hex or set SHADE_STORAGE_KEY_HEX');
const key = hexToBytes(hex);
if (key.length !== 32) throw new Error(`injected key must be 32 bytes (got ${key.length})`);
return { kind: 'injected', key };
}
}
}
async function loadOrCreateSalt(saltPath: string, role: OpenKmArgs['role']): Promise<Uint8Array> {
if (existsSync(saltPath)) {
const buf = readFileSync(saltPath);
if (buf.length < 16) throw new Error(`salt file ${saltPath} is too short (need ≥ 16 bytes)`);
return new Uint8Array(buf);
}
if (role === 'open-existing') {
throw new Error(`salt file not found at ${saltPath} — re-run with the original salt or create new with migrate-storage`);
}
const fresh = new Uint8Array(16);
globalThis.crypto.getRandomValues(fresh);
mkdirSync(dirname(resolve(saltPath)), { recursive: true });
writeFileSync(saltPath, fresh, { mode: 0o600 });
console.log(` • generated new salt at ${saltPath}`);
return fresh;
}
function hexToBytes(hex: string): Uint8Array {
const clean = hex.startsWith('0x') ? hex.slice(2) : hex;
if (clean.length % 2 !== 0) throw new Error('hex string must have even length');
const out = new Uint8Array(clean.length / 2);
for (let i = 0; i < out.length; i++) {
out[i] = parseInt(clean.slice(i * 2, i * 2 + 2), 16);
}
return out;
}
export function parseStorageArgs(args: string[]): MigrateCliOptions & RotateCliOptions {
const out: MigrateCliOptions & RotateCliOptions = {
keySource: 'passphrase',
newKeySource: 'passphrase',
};
for (let i = 0; i < args.length; i++) {
const a = args[i];
switch (a) {
case '--db-path': out.dbPath = args[++i]; break;
case '--key-source': out.keySource = args[++i] as MigrateCliOptions['keySource']; break;
case '--passphrase': out.passphrase = args[++i]; break;
case '--keychain-service': out.keychainService = args[++i]; break;
case '--keychain-account': out.keychainAccount = args[++i]; break;
case '--key-hex': out.injectedKeyHex = args[++i]; break;
case '--salt-file': out.saltFile = args[++i]; break;
case '--dry-run': out.dryRun = true; break;
case '--keep-original': out.keepOriginal = true; break;
case '--no-backup': out.noBackup = true; break;
// rotation
case '--new-key-source': out.newKeySource = args[++i] as RotateCliOptions['newKeySource']; break;
case '--new-passphrase': out.newPassphrase = args[++i]; break;
case '--new-keychain-service': out.newKeychainService = args[++i]; break;
case '--new-keychain-account': out.newKeychainAccount = args[++i]; break;
case '--new-key-hex': out.newInjectedKeyHex = args[++i]; break;
case '--new-salt-file': out.newSaltFile = args[++i]; break;
default:
if (a?.startsWith('--')) throw new Error(`unknown flag: ${a}`);
}
}
return out;
}

15
packages/shade-cli/templates/bun-server/README.md
View File

@@ -39,8 +39,23 @@ curl -X POST http://localhost:3000/receive \
Returns the decrypted plaintext.
## Stream-state retention
Resumable file transfers persist a per-stream record. The template runs
a daily cron that drops anything idle for more than
`SHADE_STREAM_RETENTION_DAYS` days (default **14**). Override at
deploy:
```bash
SHADE_STREAM_RETENTION_DAYS=7 bun run start
```
See [`docs/streams.md`](../../../../docs/streams.md) § Retention for the
full guidance and tuning per use case.
## Next steps
- Wire a real delivery layer (WebSocket, HTTP push, etc.)
- Run `shade dashboard` to watch live activity
- Compare fingerprints with peers out-of-band before trusting sessions
- Walk through [`docs/PRODUCTION-CHECKLIST.md`](../../../../docs/PRODUCTION-CHECKLIST.md) before going live

22
packages/shade-cli/templates/bun-server/src/index.ts
View File

@@ -22,6 +22,28 @@ shade.onMessage((from, msg) => {
console.log(`[${from}] ${msg}`);
});
// ─── Resumable-stream retention ──────────────────────────────────
//
// Drop stream-state records older than SHADE_STREAM_RETENTION_DAYS
// (default 14d). See docs/streams.md § Retention. Override the
// horizon with the env var; cron interval is fixed at 24h.
const STREAM_RETENTION_DAYS = Number(process.env.SHADE_STREAM_RETENTION_DAYS ?? 14);
const ONE_DAY_MS = 24 * 60 * 60 * 1000;
const HORIZON_MS = STREAM_RETENTION_DAYS * ONE_DAY_MS;
async function pruneStreamStatesOnce(): Promise<void> {
try {
await shade.pruneStreamStates(Date.now() - HORIZON_MS);
} catch (err) {
console.error('[shade] pruneStreamStates failed:', err);
}
}
await pruneStreamStatesOnce();
const streamPruneTimer = setInterval(pruneStreamStatesOnce, ONE_DAY_MS);
// Don't keep the process alive for the timer alone.
if (typeof streamPruneTimer.unref === 'function') streamPruneTimer.unref();
const app = new Hono();
app.get('/', (c) => c.text(`__PROJECT_NAME__ — Shade-enabled backend`));

51
packages/shade-cli/tests/cli.test.ts
View File

@@ -71,6 +71,37 @@ describe('CLI: init command', () => {
);
});
test('init with --doctor runs diagnostics against the new project', async () => {
const port = 19500 + Math.floor(Math.random() * 200);
const app = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
});
const server = Bun.serve({ port, fetch: app.fetch });
const logs: string[] = [];
const originalLog = console.log;
console.log = (...args) => logs.push(args.join(' '));
try {
await initCommand({
name: 'doc-app',
template: 'bun-server',
cwd: tmpDir,
prekeyServer: `http://localhost:${port}`,
runDoctor: true,
});
} finally {
console.log = originalLog;
server.stop();
}
const out = logs.join('\n');
expect(out).toContain('Shade doctor');
expect(out).toContain('Prekey server is reachable');
});
test('chat-demo template scaffolds correctly', async () => {
await initCommand({ name: 'chat', template: 'chat-demo', cwd: tmpDir });
@@ -81,6 +112,26 @@ describe('CLI: init command', () => {
const alice = readFileSync(join(target, 'src/alice.ts'), 'utf-8');
expect(alice).not.toContain('__PROJECT_NAME__');
});
test('bun-server template wires the resumable-stream prune cron by default', async () => {
// V3.1 acceptance: a freshly-scaffolded bun-server must call
// shade.pruneStreamStates on startup + on a daily interval, with
// the SHADE_STREAM_RETENTION_DAYS env var as the override knob.
await initCommand({ name: 'cron-app', template: 'bun-server', cwd: tmpDir });
const indexSrc = readFileSync(
join(tmpDir, 'cron-app', 'src/index.ts'),
'utf-8',
);
expect(indexSrc).toContain('shade.pruneStreamStates(');
expect(indexSrc).toContain('SHADE_STREAM_RETENTION_DAYS');
expect(indexSrc).toContain('setInterval');
// Default horizon stays 14 days unless overridden.
expect(indexSrc).toMatch(/\?\?\s*14/);
// Cron timer must not keep the process alive on its own.
expect(indexSrc).toContain('.unref');
});
});
describe('CLI: config loading', () => {

5
packages/shade-core/package.json
View File

@@ -1,9 +1,12 @@
{
"name": "@shade/core",
"version": "0.3.0",
"version": "4.0.0",
"type": "module",
"main": "src/index.ts",
"types": "src/index.ts",
"dependencies": {
"@shade/observability": "workspace:*"
},
"peerDependencies": {
"@shade/crypto-web": "workspace:*"
},

20
packages/shade-core/src/errors.ts
View File

@@ -88,6 +88,25 @@ export class IdentityRotationError extends ShadeError {
}
}
/**
* Thrown when a fingerprint gate (V3.3) blocks an operation because the
* peer's safety number has not been verified, or the registered handler
* returned `false`.
*/
export class FingerprintNotVerifiedError extends ShadeError {
constructor(
public readonly peerAddress: string,
public readonly gate: 'first-large-file' | 'backup-import' | 'new-device-trust' | 'inbox-fanout',
message?: string,
) {
super(
'SHADE_FINGERPRINT_NOT_VERIFIED',
message ?? `Fingerprint not verified for ${peerAddress} (gate: ${gate})`,
);
this.name = 'FingerprintNotVerifiedError';
}
}
// ─── Infrastructure Errors ───────────────────────────────────
export class NetworkError extends ShadeError {
@@ -152,6 +171,7 @@ export function errorToHttpStatus(error: unknown): number {
case 'SHADE_UNAUTHORIZED':
return 401;
case 'SHADE_UNTRUSTED_IDENTITY':
case 'SHADE_FINGERPRINT_NOT_VERIFIED':
return 403;
case 'SHADE_NO_SESSION':
case 'SHADE_PREKEY_NOT_FOUND':

11
packages/shade-core/src/ratchet.ts
View File

@@ -71,7 +71,11 @@ export async function initSenderSession(
* Initialize a session as the receiver (Bob, after X3DH).
*
* Bob knows the root key and his own signed prekey (which was used as
* the initial DH ratchet keypair).
* the initial DH ratchet keypair). The keypair is COPIED into the
* session — the receiving side's DH ratchet will eventually rotate
* `dhSend` and zeroize the previous private key, and that scratch
* buffer must NOT be the same memory as the persisted signed prekey
* (which is shared with future X3DH establishments from other senders).
*/
export function initReceiverSession(
rootKey: Uint8Array,
@@ -83,7 +87,10 @@ export function initReceiverSession(
rootKey,
sendChain: { chainKey: new Uint8Array(32), counter: 0 },
receiveChain: null,
dhSend: localDHKeyPair,
dhSend: {
publicKey: new Uint8Array(localDHKeyPair.publicKey),
privateKey: new Uint8Array(localDHKeyPair.privateKey),
},
dhReceive: null,
previousSendCounter: 0,
skippedKeys: new Map(),

67
packages/shade-core/src/session.ts
View File

@@ -26,6 +26,16 @@ import {
import { NoSessionError } from './errors.js';
import { computeFingerprint, shortFingerprint } from './fingerprint.js';
import { ShadeEventEmitter, shortHash } from './events.js';
import {
ATTR_ERROR_CODE,
ATTR_OP,
ATTR_PEER_HASH,
ATTR_RESULT,
NOOP_HOOK,
peerHash,
type ObservabilityHook,
type Span,
} from '@shade/observability';
const enc = new TextEncoder();
const dec = new TextDecoder();
@@ -60,6 +70,7 @@ export class ShadeSessionManager {
private registrationId: number = 0;
private currentSignedPreKeyId: number = 0;
private readonly events?: ShadeEventEmitter;
private readonly observability: ObservabilityHook;
/**
* Per-address operation chain. Both encrypt and decrypt mutate ratchet
* state in place (counter, DH key, skipped-keys cache); concurrent
@@ -72,11 +83,46 @@ export class ShadeSessionManager {
constructor(
private readonly crypto: CryptoProvider,
private readonly storage: StorageProvider,
options: { events?: ShadeEventEmitter } = {},
options: { events?: ShadeEventEmitter; observability?: ObservabilityHook } = {},
) {
if (options.events !== undefined) {
this.events = options.events;
}
this.observability = options.observability ?? NOOP_HOOK;
}
/**
* Wrap a per-peer crypto op in a PII-safe span. The span captures the
* mutex-acquire latency separately from the inner crypto work so a
* "ratchet contention" pathology shows up clearly in traces.
*/
private async withSpan<T>(
op: 'encrypt' | 'decrypt',
address: string,
fn: () => Promise<T>,
): Promise<T> {
const span: Span = this.observability.startSpan(`shade.session.${op}`, {
[ATTR_OP]: op,
[ATTR_PEER_HASH]: peerHash(address),
});
const lockStart = nowMs();
try {
return await this.runUnderPeerLock(address, async () => {
span.setAttribute('shade.lock.wait_ms', Math.round(nowMs() - lockStart));
const result = await fn();
span.setAttribute(ATTR_RESULT, 'ok');
span.setStatus('ok');
return result;
});
} catch (err) {
span.setAttribute(ATTR_RESULT, 'error');
span.setAttribute(ATTR_ERROR_CODE, sessionErrorCodeOf(err));
span.recordException(err);
span.setStatus('error');
throw err;
} finally {
span.end();
}
}
/**
@@ -396,7 +442,7 @@ export class ShadeSessionManager {
* Subsequent messages are standard RatchetMessages.
*/
async encrypt(address: string, plaintext: string): Promise<ShadeEnvelope> {
return this.runUnderPeerLock(address, async () => {
return this.withSpan('encrypt', address, async () => {
const session = await this.storage.getSession(address);
if (!session) throw new NoSessionError(address);
@@ -444,7 +490,7 @@ export class ShadeSessionManager {
* Decrypt a message from a peer. Handles both PreKeyMessage and RatchetMessage.
*/
async decrypt(address: string, envelope: ShadeEnvelope): Promise<string> {
return this.runUnderPeerLock(address, async () => {
return this.withSpan('decrypt', address, async () => {
if (envelope.type === 'prekey') {
return this.decryptPreKeyMessage(address, envelope.content as PreKeyMessage);
}
@@ -522,3 +568,18 @@ function arraysEqual(a: Uint8Array, b: Uint8Array): boolean {
}
return true;
}
function nowMs(): number {
return typeof performance !== 'undefined' ? performance.now() : Date.now();
}
function sessionErrorCodeOf(err: unknown): string {
if (err === null || err === undefined) return 'SHADE_UNKNOWN';
if (typeof err === 'object') {
const code = (err as { code?: unknown }).code;
if (typeof code === 'string' && code.length > 0) return code;
const name = (err as { name?: unknown }).name;
if (typeof name === 'string' && name.length > 0) return name;
}
return 'SHADE_UNKNOWN';
}

51
packages/shade-core/src/storage.ts
View File

@@ -38,6 +38,29 @@ export interface PersistedStreamState {
updatedAt: number;
}
/**
* Why a peer's fingerprint was deemed verified.
* - `user`: human did the side-channel comparison and confirmed.
* - `transitive`: trust derived from another verified party (reserved for V3.10).
* - `tofu-after-warning`: caller bypassed gate after seeing a warning.
*/
export type PeerVerificationSource = 'user' | 'transitive' | 'tofu-after-warning';
/**
* Persistent record that a peer's safety number was verified at a point
* in time. `identityVersion` is the local counter for that peer's identity:
* incrementing it (e.g. via `bumpPeerIdentityVersion` on `acceptIdentityChange`)
* invalidates the saved verification because `isPeerVerified` requires the
* stored version to equal the current version.
*/
export interface PeerVerification {
peerAddress: string;
fingerprint: string;
verifiedAt: number;
verifiedBy: PeerVerificationSource;
identityVersion: number;
}
/**
* StorageProvider — abstract interface for persisting cryptographic state.
*
@@ -115,6 +138,34 @@ export interface StorageProvider {
/** Remove retired identities older than the given timestamp */
pruneRetiredIdentities(olderThan: number): Promise<void>;
// ─── Peer verifications (V3.3) ────────────────────────────
/**
* Persist or replace the verification record for a peer. Idempotent
* upsert on `peerAddress`.
*/
savePeerVerification(verification: PeerVerification): Promise<void>;
/** Look up the saved verification for a peer (null if never verified). */
getPeerVerification(address: string): Promise<PeerVerification | null>;
/** Remove a peer's verification record (e.g. user revoked trust). */
removePeerVerification(address: string): Promise<void>;
/**
* Returns the current local identity-version counter for a peer.
* Defaults to 1 when the peer has never been seen. Bumped by
* `bumpPeerIdentityVersion` whenever the peer rotates identity.
*/
getPeerIdentityVersion(address: string): Promise<number>;
/**
* Increment the peer's identity-version counter and return the new value.
* Called from `acceptIdentityChange` so previous verification rows (which
* carry the old version number) become stale.
*/
bumpPeerIdentityVersion(address: string): Promise<number>;
// ─── Stream-transfer resume state (optional, added in v0.2.0) ──
/**

351
packages/shade-core/tests/cross-platform-vectors.test.ts
View File

@@ -8,12 +8,23 @@ import {
kdfChainKey,
deriveInitialRootKey,
} from '../src/index.js';
import { encodeEnvelope, decodeEnvelope } from '@shade/proto';
import type { RatchetMessage, ShadeEnvelope } from '../src/index.js';
import { encodeEnvelope, decodeEnvelope, encodeStreamChunk, decodeStreamChunk } from '@shade/proto';
import type { PreKeyMessage, RatchetMessage, ShadeEnvelope } from '../src/index.js';
// Imported via relative path: shade-streams depends on shade-core, so adding
// it to shade-core's dependencies would create a workspace cycle.
import {
deriveStreamKey,
deriveLaneKey,
buildChunkNonce,
buildChunkAad,
aesGcmEncryptWithNonce,
aesGcmDecryptWithNonce,
} from '../../shade-streams/src/index.js';
const crypto = new SubtleCryptoProvider();
const VECTORS_DIR = join(import.meta.dir, '..', '..', '..', 'test-vectors');
const EXPECTED_VECTOR_VERSION = 2;
function hex(bytes: Uint8Array): string {
return Array.from(bytes, (b) => b.toString(16).padStart(2, '0')).join('');
@@ -27,8 +38,49 @@ function fromHex(str: string): Uint8Array {
return bytes;
}
function loadVectors(name: string): any {
return JSON.parse(readFileSync(join(VECTORS_DIR, name), 'utf-8'));
function loadVectors(name: string): { version: number; vectors: any[] } {
const parsed = JSON.parse(readFileSync(join(VECTORS_DIR, name), 'utf-8'));
expect(parsed.version).toBe(EXPECTED_VECTOR_VERSION);
return parsed;
}
async function aesGcmEncryptDeterministic(
key: Uint8Array,
nonce: Uint8Array,
plaintext: Uint8Array,
aad: Uint8Array,
): Promise<Uint8Array> {
const subtle = globalThis.crypto.subtle;
const aesKey = await subtle.importKey(
'raw',
key as unknown as ArrayBuffer,
'AES-GCM',
false,
['encrypt', 'decrypt'],
);
const out = await subtle.encrypt(
{
name: 'AES-GCM',
iv: nonce as unknown as ArrayBuffer,
additionalData: aad as unknown as ArrayBuffer,
},
aesKey,
plaintext as unknown as ArrayBuffer,
);
return new Uint8Array(out);
}
function encodeRatchetHeader(
dhPublicKey: Uint8Array,
previousCounter: number,
counter: number,
): Uint8Array {
const buf = new Uint8Array(40);
buf.set(dhPublicKey, 0);
const view = new DataView(buf.buffer);
view.setUint32(32, previousCounter, false);
view.setUint32(36, counter, false);
return buf;
}
describe('Cross-platform test vectors', () => {
@@ -82,9 +134,10 @@ describe('Cross-platform test vectors', () => {
}
});
test('Wire format vectors match', () => {
test('Wire format: RatchetMessage encode + decode', () => {
const { vectors } = loadVectors('wire-format.json');
const v = vectors[0];
const v = vectors.find((x: any) => x.kind === 'ratchet');
expect(v).toBeDefined();
const msg: RatchetMessage = {
dhPublicKey: fromHex(v.message.dhPublicKey),
@@ -102,11 +155,295 @@ describe('Cross-platform test vectors', () => {
const encoded = encodeEnvelope(envelope);
expect(hex(encoded)).toBe(v.encoded);
// Also verify round-trip decode
const decoded = decodeEnvelope(encoded);
expect(decoded.type).toBe('ratchet');
const rm = decoded.content as RatchetMessage;
expect(rm.counter).toBe(msg.counter);
expect(hex(rm.ciphertext)).toBe(hex(msg.ciphertext));
});
test('Wire format: PreKeyMessage encode + decode (with and without OTPK)', () => {
const { vectors } = loadVectors('wire-format.json');
const preKeyVectors = vectors.filter((x: any) => x.kind === 'prekey');
expect(preKeyVectors.length).toBeGreaterThanOrEqual(2);
for (const v of preKeyVectors) {
const inner: RatchetMessage = {
dhPublicKey: fromHex(v.message.inner.dhPublicKey),
previousCounter: v.message.inner.previousCounter,
counter: v.message.inner.counter,
ciphertext: fromHex(v.message.inner.ciphertext),
nonce: fromHex(v.message.inner.nonce),
};
const pre: PreKeyMessage = {
registrationId: v.message.registrationId,
preKeyId: v.message.preKeyId === null ? undefined : v.message.preKeyId,
signedPreKeyId: v.message.signedPreKeyId,
ephemeralKey: fromHex(v.message.ephemeralKey),
identityDHKey: fromHex(v.message.identityDHKey),
message: inner,
};
const envelope: ShadeEnvelope = {
type: 'prekey',
content: pre,
timestamp: 0,
senderAddress: '',
};
const encoded = encodeEnvelope(envelope);
expect(hex(encoded)).toBe(v.encoded);
const decoded = decodeEnvelope(encoded);
expect(decoded.type).toBe('prekey');
const dm = decoded.content as PreKeyMessage;
expect(dm.registrationId).toBe(pre.registrationId);
expect(dm.preKeyId).toBe(pre.preKeyId);
expect(dm.signedPreKeyId).toBe(pre.signedPreKeyId);
expect(hex(dm.ephemeralKey)).toBe(hex(pre.ephemeralKey));
expect(hex(dm.message.ciphertext)).toBe(hex(inner.ciphertext));
}
});
test('Streams 0x11: deriveStreamKey + deriveLaneKey + nonce/AAD + chunk encrypt + wire roundtrip', async () => {
const { vectors } = loadVectors('streams.json');
// 1. deriveStreamKey
const sk = vectors.find((v: any) => v.description.startsWith('deriveStreamKey'));
expect(sk).toBeDefined();
const streamKey = await deriveStreamKey(crypto, fromHex(sk.streamSecret), fromHex(sk.streamId));
expect(hex(streamKey)).toBe(sk.streamKey);
// 2. deriveLaneKey for each laneId
const lk = vectors.find((v: any) => v.description.startsWith('deriveLaneKey'));
expect(lk).toBeDefined();
for (const lane of lk.lanes) {
const k = await deriveLaneKey(crypto, fromHex(lk.streamKey), fromHex(lk.streamId), lane.laneId);
expect(hex(k)).toBe(lane.laneKey);
}
// 3. buildChunkNonce
const nv = vectors.find((v: any) => v.description.startsWith('buildChunkNonce'));
expect(nv).toBeDefined();
for (const n of nv.nonces) {
const seq = BigInt(n.seq);
const out = buildChunkNonce(n.laneId, seq);
expect(hex(out)).toBe(n.nonce);
}
// 4. buildChunkAad
const av = vectors.find((v: any) => v.description.startsWith('buildChunkAad'));
expect(av).toBeDefined();
for (const c of av.cases) {
const seq = BigInt(c.seq);
const out = buildChunkAad(fromHex(av.streamId), c.laneId, seq, c.isLast);
expect(hex(out)).toBe(c.aad);
}
// 5. End-to-end chunk encrypt + decrypt
const ev = vectors.find((v: any) => v.description.startsWith('End-to-end chunk encrypt'));
expect(ev).toBeDefined();
const ct = await aesGcmEncryptWithNonce(
fromHex(ev.laneKey),
fromHex(ev.nonce),
fromHex(ev.plaintext),
fromHex(ev.aad),
);
expect(hex(ct)).toBe(ev.ciphertext);
const pt = await aesGcmDecryptWithNonce(
fromHex(ev.laneKey),
fromHex(ev.nonce),
fromHex(ev.ciphertext),
fromHex(ev.aad),
);
expect(hex(pt)).toBe(ev.plaintext);
// 6. Wire 0x11 envelope encode/decode
const wv = vectors.find((v: any) => v.description.startsWith('Wire 0x11'));
expect(wv).toBeDefined();
const encoded = encodeStreamChunk({
streamId: fromHex(wv.streamId),
laneId: wv.laneId,
seq: BigInt(wv.seq),
isLast: wv.isLast,
nonce: fromHex(wv.nonce),
aad: fromHex(wv.extraAad),
ciphertext: fromHex(wv.ciphertext),
});
expect(hex(encoded)).toBe(wv.encoded);
const decoded = decodeStreamChunk(encoded);
expect(hex(decoded.streamId)).toBe(wv.streamId);
expect(decoded.laneId).toBe(wv.laneId);
expect(decoded.seq.toString()).toBe(wv.seq);
expect(decoded.isLast).toBe(wv.isLast);
expect(hex(decoded.nonce)).toBe(wv.nonce);
expect(hex(decoded.ciphertext)).toBe(wv.ciphertext);
});
test('Backup v1: HKDF backupKey + AES-GCM roundtrip', async () => {
const { vectors } = loadVectors('backup.json');
const kv = vectors.find((v: any) => v.description.startsWith('Backup v1: HKDF'));
expect(kv).toBeDefined();
const backupKey = await crypto.hkdf(
new TextEncoder().encode(kv.passphrase),
fromHex(kv.salt),
new TextEncoder().encode(kv.info),
32,
);
expect(hex(backupKey)).toBe(kv.backupKey);
const ev = vectors.find((v: any) => v.description.startsWith('Backup v1: AES-256-GCM'));
expect(ev).toBeDefined();
const ct = await aesGcmEncryptDeterministic(
fromHex(ev.backupKey),
fromHex(ev.nonce),
fromHex(ev.plaintext),
new Uint8Array(0),
);
expect(hex(ct)).toBe(ev.ciphertext);
const pt = await crypto.aesGcmDecrypt(
fromHex(ev.backupKey),
fromHex(ev.ciphertext),
fromHex(ev.nonce),
);
expect(hex(pt)).toBe(ev.plaintext);
});
test('Group sender-keys: header AAD + step + Ed25519 signature', async () => {
const { vectors } = loadVectors('group.json');
// 1. Header AAD encoding
const hv = vectors.find((v: any) => v.description.startsWith('Sender header AAD'));
expect(hv).toBeDefined();
const enc = new TextEncoder();
const gBytes = enc.encode(hv.groupId);
const sBytes = enc.encode(hv.senderAddress);
const aad = new Uint8Array(2 + gBytes.length + 2 + sBytes.length + 4);
const view = new DataView(aad.buffer);
let off = 0;
view.setUint16(off, gBytes.length, false); off += 2;
aad.set(gBytes, off); off += gBytes.length;
view.setUint16(off, sBytes.length, false); off += 2;
aad.set(sBytes, off); off += sBytes.length;
view.setUint32(off, hv.iteration, false);
expect(hex(aad)).toBe(hv.aad);
// 2. Sender-key step
const sv = vectors.find((v: any) => v.description.startsWith('Sender-key step'));
expect(sv).toBeDefined();
const chain = await kdfChainKey(crypto, fromHex(sv.chainKey));
expect(hex(chain.newChainKey)).toBe(sv.newChainKey);
expect(hex(chain.messageKey)).toBe(sv.messageKey);
const ct = await aesGcmEncryptDeterministic(
chain.messageKey,
fromHex(sv.nonce),
fromHex(sv.plaintext),
fromHex(sv.aad),
);
expect(hex(ct)).toBe(sv.ciphertext);
// Ed25519 sign(aad || ct) — verify signature is valid for the recorded keys
const signed = new Uint8Array(fromHex(sv.aad).length + ct.length);
signed.set(fromHex(sv.aad), 0);
signed.set(ct, fromHex(sv.aad).length);
const ok = await crypto.verify(
fromHex(sv.signingPublicKey),
signed,
fromHex(sv.signature),
);
expect(ok).toBe(true);
// Decrypt roundtrip
const pt = await crypto.aesGcmDecrypt(
chain.messageKey,
fromHex(sv.ciphertext),
fromHex(sv.nonce),
fromHex(sv.aad),
);
expect(hex(pt)).toBe(sv.plaintext);
});
test('Storage encryption HKDF subset: storageKey + fieldKey + rowNonce', async () => {
const { vectors } = loadVectors('storage-hkdf.json');
const sv = vectors.find((v: any) => v.description.startsWith('Storage HKDF: storageKey'));
expect(sv).toBeDefined();
const storageKey = await crypto.hkdf(
fromHex(sv.masterKey),
new Uint8Array(0),
new TextEncoder().encode('shade-storage-v1'),
32,
);
expect(hex(storageKey)).toBe(sv.storageKey);
const fv = vectors.find((v: any) => v.description.startsWith('Storage HKDF: fieldKey'));
expect(fv).toBeDefined();
for (const f of fv.fields) {
const k = await crypto.hkdf(
fromHex(fv.storageKey),
new Uint8Array(0),
new TextEncoder().encode(`shade-field-v1:${f.table}:${f.column}`),
32,
);
expect(hex(k)).toBe(f.fieldKey);
}
const nv = vectors.find((v: any) => v.description.startsWith('Storage HKDF: rowNonce'));
expect(nv).toBeDefined();
for (const n of nv.nonces) {
const out = await crypto.hkdf(
fromHex(nv.rowKey),
new Uint8Array(0),
new TextEncoder().encode(`shade-row-nonce-v1:${n.table}:${n.pk}`),
12,
);
expect(hex(out)).toBe(n.nonce);
}
});
test('Ratchet step: deterministic encrypt + decrypt roundtrip', async () => {
const { vectors } = loadVectors('ratchet-step.json');
for (const v of vectors) {
const rootKey = fromHex(v.inputs.rootKey);
const dhSendPriv = fromHex(v.inputs.dhSendPrivateKey);
const dhSendPub = fromHex(v.inputs.dhSendPublicKey);
const dhRemotePub = fromHex(v.inputs.dhRemotePublicKey);
const plaintext = fromHex(v.inputs.plaintext);
const nonce = fromHex(v.inputs.nonce);
const previousCounter: number = v.inputs.previousCounter;
const counter: number = v.inputs.counter;
// 1. DH
const dhOutput = await crypto.x25519(dhSendPriv, dhRemotePub);
expect(hex(dhOutput)).toBe(v.derived.dhOutput);
// 2. kdfRootKey
const root = await kdfRootKey(crypto, rootKey, dhOutput);
expect(hex(root.newRootKey)).toBe(v.derived.newRootKey);
expect(hex(root.chainKey)).toBe(v.derived.chainKey);
// 3. kdfChainKey
const chain = await kdfChainKey(crypto, root.chainKey);
expect(hex(chain.newChainKey)).toBe(v.derived.newChainKey);
expect(hex(chain.messageKey)).toBe(v.derived.messageKey);
// 4. Header AAD
const aad = encodeRatchetHeader(dhSendPub, previousCounter, counter);
expect(hex(aad)).toBe(v.derived.aad);
// 5. AES-GCM encrypt with fixed nonce
const ciphertext = await aesGcmEncryptDeterministic(chain.messageKey, nonce, plaintext, aad);
expect(hex(ciphertext)).toBe(v.ciphertext);
// 6. Roundtrip decrypt — verify the recorded ciphertext recovers the plaintext
const recovered = await crypto.aesGcmDecrypt(
chain.messageKey,
fromHex(v.ciphertext),
nonce,
aad,
);
expect(hex(recovered)).toBe(v.inputs.plaintext);
}
});
});

45
packages/shade-core/tests/ratchet.test.ts
View File

@@ -31,6 +31,51 @@ async function setupPair(): Promise<{ alice: SessionState; bob: SessionState }>
describe('Double Ratchet', () => {
// ─── Basic Send/Receive ──────────────────────────────────
describe('initReceiverSession isolation', () => {
/**
* Regression — the V3.10 multi-sender recovery flow surfaced a
* bug where `initReceiverSession` shared a reference to the
* receiver's signed-prekey keypair with the new session. The
* first DH ratchet step zeroed the session's stale send-key
* private bytes — which were the SAME backing buffer as the
* persisted signed prekey. A subsequent X3DH from a different
* sender then derived a divergent root key and decryption
* failed.
*
* Fix: `initReceiverSession` copies the keypair into the
* session. Verify here.
*/
test('does not mutate the caller-provided keypair after a DH ratchet step', async () => {
const rootKey = crypto.randomBytes(32);
const remoteIdentityKey = crypto.randomBytes(32);
const bobDH = await crypto.generateX25519KeyPair();
const originalPrivate = new Uint8Array(bobDH.privateKey);
const originalPublic = new Uint8Array(bobDH.publicKey);
const bob = initReceiverSession(rootKey, remoteIdentityKey, bobDH);
// Drive a full receive that triggers `performDHRatchetStep`
// via a message with a fresh dhPublicKey.
const aliceFirst = await initSenderSession(crypto, rootKey, remoteIdentityKey, bobDH.publicKey);
const msg = await ratchetEncrypt(crypto, aliceFirst, enc.encode('hi'));
await ratchetDecrypt(crypto, bob, msg);
// The ORIGINAL keypair must not have been touched, so a
// second X3DH-style establishment using the same prekey
// material still succeeds.
expect(Array.from(bobDH.privateKey)).toEqual(Array.from(originalPrivate));
expect(Array.from(bobDH.publicKey)).toEqual(Array.from(originalPublic));
// Sanity-check: a second receiver session built from the
// same keypair should still decrypt fresh sender traffic.
const bob2 = initReceiverSession(rootKey, remoteIdentityKey, bobDH);
const aliceSecond = await initSenderSession(crypto, rootKey, remoteIdentityKey, bobDH.publicKey);
const msg2 = await ratchetEncrypt(crypto, aliceSecond, enc.encode('hi again'));
const plain2 = await ratchetDecrypt(crypto, bob2, msg2);
expect(dec.decode(plain2)).toBe('hi again');
});
});
describe('basic send/receive', () => {
test('Alice encrypts, Bob decrypts', async () => {
const { alice, bob } = await setupPair();

19
packages/shade-crypto-web/package.json
View File

@@ -1,12 +1,27 @@
{
"name": "@shade/crypto-web",
"version": "0.3.0",
"version": "4.0.0",
"type": "module",
"main": "src/index.ts",
"types": "src/index.ts",
"exports": {
".": {
"types": "./src/index.ts",
"default": "./src/index.ts"
},
"./worker": {
"types": "./src/worker.ts",
"default": "./src/worker.ts"
},
"./worker-protocol": {
"types": "./src/worker-protocol.ts",
"default": "./src/worker-protocol.ts"
}
},
"dependencies": {
"@noble/curves": "^2.0.1",
"@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*"
"@shade/core": "workspace:*",
"@shade/streams": "workspace:*"
}
}

22
packages/shade-crypto-web/src/index.ts
View File

@@ -1,2 +1,24 @@
export { SubtleCryptoProvider } from './provider.js';
export { MemoryStorage } from './memory-storage.js';
// ─── Web Workers crypto (V3.8) ────────────────────────────
export {
createWorkerCryptoProvider,
WorkerCryptoProvider,
WorkerStreamSender,
WorkerStreamReceiver,
} from './worker-client.js';
export type {
WorkerCryptoProviderOptions,
WorkerLike,
} from './worker-client.js';
export {
createEncryptStream,
createDecryptStream,
DEFAULT_STREAM_CHUNK_SIZE,
} from './worker-streams.js';
export type {
CreateEncryptStreamOptions,
CreateDecryptStreamOptions,
} from './worker-streams.js';
export { WORKER_PROTOCOL_VERSION } from './worker-protocol.js';

30
packages/shade-crypto-web/src/memory-storage.ts
View File

@@ -1,4 +1,4 @@
import type { StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey, SessionState, RetiredIdentity, PersistedStreamState } from '@shade/core';
import type { StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey, SessionState, RetiredIdentity, PersistedStreamState, PeerVerification } from '@shade/core';
import { constantTimeEqual } from '@shade/core';
/**
@@ -104,6 +104,34 @@ export class MemoryStorage implements StorageProvider {
this.retiredIdentities = this.retiredIdentities.filter((r) => r.retiredAt >= olderThan);
}
// ─── Peer verifications (V3.3) ────────────────────────────
private peerVerifications = new Map<string, PeerVerification>();
private peerIdentityVersions = new Map<string, number>();
async savePeerVerification(v: PeerVerification): Promise<void> {
this.peerVerifications.set(v.peerAddress, { ...v });
}
async getPeerVerification(address: string): Promise<PeerVerification | null> {
const v = this.peerVerifications.get(address);
return v ? { ...v } : null;
}
async removePeerVerification(address: string): Promise<void> {
this.peerVerifications.delete(address);
}
async getPeerIdentityVersion(address: string): Promise<number> {
return this.peerIdentityVersions.get(address) ?? 1;
}
async bumpPeerIdentityVersion(address: string): Promise<number> {
const next = (this.peerIdentityVersions.get(address) ?? 1) + 1;
this.peerIdentityVersions.set(address, next);
return next;
}
// ─── Stream-transfer resume state (v0.2.0) ────────────────
private streamStates = new Map<string, PersistedStreamState>();

513
packages/shade-crypto-web/src/worker-client.ts Normal file
View File

@@ -0,0 +1,513 @@
import type { CryptoProvider } from '@shade/core';
import {
WORKER_PROTOCOL_VERSION,
fromTransferable,
toTransferableCopy,
type WorkerRequest,
type WorkerResponse,
type WorkerResult,
} from './worker-protocol.js';
/** Distributive omit of `id` from each variant of {@link WorkerRequest}. */
type WorkerRequestBody = WorkerRequest extends infer T
? T extends { id: number }
? Omit<T, 'id'>
: never
: never;
/**
* Minimal Worker shape we depend on. Lets the main-thread proxy work
* against both browser `Worker` and Bun's `Worker` without dragging in
* `lib.dom.d.ts`.
*/
export interface WorkerLike {
postMessage(message: unknown, transfer?: ArrayBuffer[]): void;
addEventListener(
type: 'message',
listener: (ev: { data: WorkerResponse }) => void,
): void;
removeEventListener(
type: 'message',
listener: (ev: { data: WorkerResponse }) => void,
): void;
addEventListener(
type: 'error',
listener: (ev: { message?: string; error?: unknown }) => void,
): void;
terminate(): void;
}
export interface WorkerCryptoProviderOptions {
/**
* URL of the bundled worker entry. Required because every bundler
* resolves worker URLs differently — supply yours and stop guessing.
*
* // Vite / Webpack 5 / Rspack:
* workerUrl: new URL('@shade/crypto-web/worker', import.meta.url)
*/
workerUrl: URL | string;
/**
* How long the worker may stay idle before it self-terminates. Set to
* `Infinity` to opt out (e.g. for SharedArrayBuffer / persistent UI).
* Default: 30_000 ms — matches the V3.8 acceptance criterion.
*/
idleTimeoutMs?: number;
/**
* Override the worker factory. Useful in tests with `bun:test`'s
* `Worker` global, or to inject a polyfill.
*/
spawn?: (url: URL | string) => WorkerLike;
}
/**
* Public factory. Resolves once the worker has acknowledged the protocol
* version handshake — so a stale bundle blows up here rather than in the
* middle of an encrypt call.
*/
export async function createWorkerCryptoProvider(
opts: WorkerCryptoProviderOptions,
): Promise<WorkerCryptoProvider> {
const provider = new WorkerCryptoProvider(opts);
await provider.handshake();
return provider;
}
let SENDER_SEQ = 1;
let RECEIVER_SEQ = 1;
/**
* `CryptoProvider` implementation that forwards every async call to a
* dedicated Web Worker. Sync methods (`randomBytes`, `randomUint32`,
* `constantTimeEqual`, `zeroize`) execute on the calling thread — they
* are pure and instantaneous, so a worker round-trip would be silly.
*
* The worker is spawned on construction (lazy: see `createWorkerCryptoProvider`)
* and terminated automatically after `idleTimeoutMs` of inactivity.
* Subsequent calls re-spawn transparently.
*
* Stream sender/receiver state lives on the worker — the provider
* exposes `createStreamSender` / `createStreamReceiver` factories that
* return main-thread proxies (`WorkerStreamSender` / `WorkerStreamReceiver`).
*/
export class WorkerCryptoProvider implements CryptoProvider {
private worker: WorkerLike | null = null;
private nextRequestId = 1;
private readonly inflight = new Map<
number,
{ resolve: (r: WorkerResult) => void; reject: (e: Error) => void }
>();
private idleTimer: ReturnType<typeof setTimeout> | null = null;
private destroyed = false;
private readonly idleTimeoutMs: number;
constructor(private readonly opts: WorkerCryptoProviderOptions) {
this.idleTimeoutMs = opts.idleTimeoutMs ?? 30_000;
}
// ─── lifecycle ───────────────────────────────────────────
/** Force-spawn + complete the protocol handshake. Idempotent. */
async handshake(): Promise<void> {
await this.ensureWorker();
await this.send({ method: 'init', protocolVersion: WORKER_PROTOCOL_VERSION });
}
/** Permanently terminate the worker. After this, every call rejects. */
async destroy(): Promise<void> {
this.destroyed = true;
this.terminateWorker(new Error('WorkerCryptoProvider destroyed'));
}
/**
* Tear down the current worker and (lazily) spawn a fresh one. Use
* after rotating identity keys so leaked-state worst case is bounded
* by one rotation interval.
*/
async rotate(): Promise<void> {
this.terminateWorker(new Error('WorkerCryptoProvider rotated'));
}
// ─── async CryptoProvider methods ────────────────────────
async generateX25519KeyPair(): Promise<{ publicKey: Uint8Array; privateKey: Uint8Array }> {
const r = await this.send({ method: 'crypto.generateX25519KeyPair' });
if (r.kind !== 'keypair') throw new Error('protocol: expected keypair');
return { publicKey: fromTransferable(r.publicKey), privateKey: fromTransferable(r.privateKey) };
}
async x25519(privateKey: Uint8Array, publicKey: Uint8Array): Promise<Uint8Array> {
const r = await this.send(
{
method: 'crypto.x25519',
privateKey: toTransferableCopy(privateKey),
publicKey: toTransferableCopy(publicKey),
},
// No transferables from caller's owned buffers — we copied above.
[],
);
if (r.kind !== 'bytes') throw new Error('protocol: expected bytes');
return fromTransferable(r.bytes);
}
async generateEd25519KeyPair(): Promise<{ publicKey: Uint8Array; privateKey: Uint8Array }> {
const r = await this.send({ method: 'crypto.generateEd25519KeyPair' });
if (r.kind !== 'keypair') throw new Error('protocol: expected keypair');
return { publicKey: fromTransferable(r.publicKey), privateKey: fromTransferable(r.privateKey) };
}
async sign(privateKey: Uint8Array, message: Uint8Array): Promise<Uint8Array> {
const r = await this.send({
method: 'crypto.sign',
privateKey: toTransferableCopy(privateKey),
message: toTransferableCopy(message),
});
if (r.kind !== 'bytes') throw new Error('protocol: expected bytes');
return fromTransferable(r.bytes);
}
async verify(
publicKey: Uint8Array,
message: Uint8Array,
signature: Uint8Array,
): Promise<boolean> {
const r = await this.send({
method: 'crypto.verify',
publicKey: toTransferableCopy(publicKey),
message: toTransferableCopy(message),
signature: toTransferableCopy(signature),
});
if (r.kind !== 'verify') throw new Error('protocol: expected verify');
return r.valid;
}
async aesGcmEncrypt(
key: Uint8Array,
plaintext: Uint8Array,
aad?: Uint8Array,
): Promise<{ ciphertext: Uint8Array; nonce: Uint8Array }> {
const r = await this.send({
method: 'crypto.aesGcmEncrypt',
key: toTransferableCopy(key),
plaintext: toTransferableCopy(plaintext),
aad: aad ? toTransferableCopy(aad) : null,
});
if (r.kind !== 'aead-encrypt') throw new Error('protocol: expected aead-encrypt');
return {
ciphertext: fromTransferable(r.ciphertext),
nonce: fromTransferable(r.nonce),
};
}
async aesGcmDecrypt(
key: Uint8Array,
ciphertext: Uint8Array,
nonce: Uint8Array,
aad?: Uint8Array,
): Promise<Uint8Array> {
const r = await this.send({
method: 'crypto.aesGcmDecrypt',
key: toTransferableCopy(key),
ciphertext: toTransferableCopy(ciphertext),
nonce: toTransferableCopy(nonce),
aad: aad ? toTransferableCopy(aad) : null,
});
if (r.kind !== 'bytes') throw new Error('protocol: expected bytes');
return fromTransferable(r.bytes);
}
async hkdf(
ikm: Uint8Array,
salt: Uint8Array,
info: Uint8Array,
length: number,
): Promise<Uint8Array> {
const r = await this.send({
method: 'crypto.hkdf',
ikm: toTransferableCopy(ikm),
salt: toTransferableCopy(salt),
info: toTransferableCopy(info),
length,
});
if (r.kind !== 'bytes') throw new Error('protocol: expected bytes');
return fromTransferable(r.bytes);
}
async hmacSha256(key: Uint8Array, data: Uint8Array): Promise<Uint8Array> {
const r = await this.send({
method: 'crypto.hmacSha256',
key: toTransferableCopy(key),
data: toTransferableCopy(data),
});
if (r.kind !== 'bytes') throw new Error('protocol: expected bytes');
return fromTransferable(r.bytes);
}
// ─── sync — local execution (no round-trip) ──────────────
randomBytes(length: number): Uint8Array {
const buf = new Uint8Array(length);
globalThis.crypto.getRandomValues(buf);
return buf;
}
constantTimeEqual(a: Uint8Array, b: Uint8Array): boolean {
if (a.length !== b.length) return false;
let diff = 0;
for (let i = 0; i < a.length; i++) diff |= a[i]! ^ b[i]!;
return diff === 0;
}
zeroize(buf: Uint8Array): void {
buf.fill(0);
}
randomUint32(): number {
const buf = this.randomBytes(4);
return new DataView(buf.buffer, buf.byteOffset, 4).getUint32(0, false);
}
// ─── stream factories ────────────────────────────────────
async createStreamSender(opts: {
streamId: Uint8Array;
streamSecret: Uint8Array;
laneId: number;
startSeq?: number;
}): Promise<WorkerStreamSender> {
const senderId = SENDER_SEQ++;
await this.send({
method: 'stream.createSender',
senderId,
streamId: toTransferableCopy(opts.streamId),
streamSecret: toTransferableCopy(opts.streamSecret),
laneId: opts.laneId,
startSeq: opts.startSeq ?? 0,
});
return new WorkerStreamSender(this, senderId);
}
async createStreamReceiver(opts: {
streamId: Uint8Array;
streamSecret: Uint8Array;
laneId: number;
startSeq?: number;
}): Promise<WorkerStreamReceiver> {
const receiverId = RECEIVER_SEQ++;
await this.send({
method: 'stream.createReceiver',
receiverId,
streamId: toTransferableCopy(opts.streamId),
streamSecret: toTransferableCopy(opts.streamSecret),
laneId: opts.laneId,
startSeq: opts.startSeq ?? 0,
});
return new WorkerStreamReceiver(this, receiverId);
}
// ─── internals ───────────────────────────────────────────
/** @internal — used by `WorkerStreamSender` / `WorkerStreamReceiver`. */
async send(
body: WorkerRequestBody,
extraTransferables?: ArrayBuffer[],
): Promise<WorkerResult> {
if (this.destroyed) throw new Error('WorkerCryptoProvider destroyed');
await this.ensureWorker();
const id = this.nextRequestId++;
const req = { id, ...body } as WorkerRequest;
// Auto-collect transferable buffers from the request payload — every
// ArrayBuffer-typed field is fair game.
const transferables = extraTransferables ?? collectArrayBuffers(req);
return new Promise<WorkerResult>((resolve, reject) => {
this.inflight.set(id, { resolve, reject });
try {
this.worker!.postMessage(req, transferables);
} catch (err) {
this.inflight.delete(id);
reject(err instanceof Error ? err : new Error(String(err)));
return;
}
this.bumpIdleTimer();
});
}
private async ensureWorker(): Promise<void> {
if (this.destroyed) throw new Error('WorkerCryptoProvider destroyed');
if (this.worker !== null) return;
const spawn = this.opts.spawn ?? defaultSpawn;
const w = spawn(this.opts.workerUrl);
w.addEventListener('message', this.onMessage);
w.addEventListener('error', this.onError);
this.worker = w;
}
private readonly onMessage = (ev: { data: WorkerResponse }): void => {
const res = ev.data;
const pending = this.inflight.get(res.id);
if (pending === undefined) return;
this.inflight.delete(res.id);
if (res.ok) pending.resolve(res.result);
else {
const err = new Error(res.error.message);
err.name = res.error.name;
pending.reject(err);
}
this.bumpIdleTimer();
};
private readonly onError = (ev: { message?: string; error?: unknown }): void => {
const msg = ev.message ?? (ev.error instanceof Error ? ev.error.message : String(ev.error));
this.terminateWorker(new Error(`worker error: ${msg}`));
};
private bumpIdleTimer(): void {
if (this.idleTimer !== null) clearTimeout(this.idleTimer);
if (!isFinite(this.idleTimeoutMs)) return;
if (this.inflight.size > 0) return;
this.idleTimer = setTimeout(() => {
// No outstanding work — recycle the worker. Calls after this
// re-spawn lazily.
this.terminateWorker(null);
}, this.idleTimeoutMs);
// Don't keep node-style event loops alive solely on this timer.
const t = this.idleTimer as unknown as { unref?: () => void };
if (typeof t.unref === 'function') t.unref();
}
private terminateWorker(reason: Error | null): void {
if (this.idleTimer !== null) {
clearTimeout(this.idleTimer);
this.idleTimer = null;
}
const w = this.worker;
this.worker = null;
// Reject every in-flight request so callers don't hang.
if (reason !== null) {
for (const [, pending] of this.inflight) pending.reject(reason);
}
this.inflight.clear();
if (w !== null) {
try {
w.removeEventListener('message', this.onMessage);
} catch {
// ignore
}
try {
w.terminate();
} catch {
// ignore
}
}
}
}
/**
* Walk the request body, collecting every `ArrayBuffer` reference so we
* can hand them to `postMessage(_, transfer)`. Cheap because the request
* objects are flat — at most a handful of fields.
*/
function collectArrayBuffers(req: WorkerRequest): ArrayBuffer[] {
const out: ArrayBuffer[] = [];
for (const v of Object.values(req as Record<string, unknown>)) {
if (v instanceof ArrayBuffer) out.push(v);
}
return out;
}
function defaultSpawn(url: URL | string): WorkerLike {
const ctor = (globalThis as unknown as { Worker?: new (u: URL | string, o?: unknown) => unknown })
.Worker;
if (typeof ctor !== 'function') {
throw new Error('Worker is not available in this runtime');
}
return new ctor(url, { type: 'module' }) as WorkerLike;
}
/**
* Main-thread handle on a `StreamSender` that lives in the worker. The
* lane key never crosses thread boundaries — this proxy only ever ships
* plaintext slices and receives wire bytes.
*/
export class WorkerStreamSender {
private destroyed = false;
constructor(
private readonly provider: WorkerCryptoProvider,
private readonly senderId: number,
) {}
async encryptChunk(
plaintext: Uint8Array,
isLast: boolean,
): Promise<{ bytes: Uint8Array; seq: number }> {
if (this.destroyed) throw new Error('WorkerStreamSender destroyed');
const r = await this.provider.send({
method: 'stream.encryptChunk',
senderId: this.senderId,
plaintext: toTransferableCopy(plaintext),
isLast,
});
if (r.kind !== 'chunk-encrypt') throw new Error('protocol: expected chunk-encrypt');
return { bytes: fromTransferable(r.bytes), seq: r.seq };
}
async getLaneSha256(): Promise<Uint8Array> {
const r = await this.provider.send({
method: 'stream.getSenderLaneSha256',
senderId: this.senderId,
});
if (r.kind !== 'bytes') throw new Error('protocol: expected bytes');
return fromTransferable(r.bytes);
}
async destroy(): Promise<void> {
if (this.destroyed) return;
this.destroyed = true;
await this.provider.send({ method: 'stream.destroySender', senderId: this.senderId });
}
}
export class WorkerStreamReceiver {
private destroyed = false;
constructor(
private readonly provider: WorkerCryptoProvider,
private readonly receiverId: number,
) {}
async decryptChunk(
wireBytes: Uint8Array,
): Promise<{ plaintext: Uint8Array; seq: number; isLast: boolean }> {
if (this.destroyed) throw new Error('WorkerStreamReceiver destroyed');
const r = await this.provider.send({
method: 'stream.decryptChunk',
receiverId: this.receiverId,
wireBytes: toTransferableCopy(wireBytes),
});
if (r.kind !== 'chunk-decrypt') throw new Error('protocol: expected chunk-decrypt');
return {
plaintext: fromTransferable(r.plaintext),
seq: r.seq,
isLast: r.isLast,
};
}
async getLaneSha256(): Promise<Uint8Array> {
const r = await this.provider.send({
method: 'stream.getReceiverLaneSha256',
receiverId: this.receiverId,
});
if (r.kind !== 'bytes') throw new Error('protocol: expected bytes');
return fromTransferable(r.bytes);
}
async destroy(): Promise<void> {
if (this.destroyed) return;
this.destroyed = true;
await this.provider.send({ method: 'stream.destroyReceiver', receiverId: this.receiverId });
}
}

165
packages/shade-crypto-web/src/worker-protocol.ts Normal file
View File

@@ -0,0 +1,165 @@
/**
* Wire protocol between `WorkerCryptoProvider` (main thread) and the
* worker entry (`worker.ts`).
*
* Shape:
* main → worker: WorkerRequest
* worker → main: WorkerResponse (matched by `id`)
*
* Binary inputs are passed as `ArrayBuffer` (transferable) — never
* `Uint8Array` — so we can hand them off without copying. The worker
* wraps them in `Uint8Array` for use, and the response transfers result
* buffers the same way.
*
* Versioning: `__protocolVersion` is bumped on any breaking change.
* `worker-init` echoes the version so a mismatched bundle aborts
* deterministically instead of silently producing garbage.
*/
export const WORKER_PROTOCOL_VERSION = 1;
export type WorkerRequest =
// ─── lifecycle ────────────────────────────────────────────
| { id: number; method: 'init'; protocolVersion: number }
| { id: number; method: 'ping' }
// ─── crypto.* — generic CryptoProvider ────────────────────
| { id: number; method: 'crypto.generateX25519KeyPair' }
| { id: number; method: 'crypto.x25519'; privateKey: ArrayBuffer; publicKey: ArrayBuffer }
| { id: number; method: 'crypto.generateEd25519KeyPair' }
| { id: number; method: 'crypto.sign'; privateKey: ArrayBuffer; message: ArrayBuffer }
| {
id: number;
method: 'crypto.verify';
publicKey: ArrayBuffer;
message: ArrayBuffer;
signature: ArrayBuffer;
}
| {
id: number;
method: 'crypto.aesGcmEncrypt';
key: ArrayBuffer;
plaintext: ArrayBuffer;
aad: ArrayBuffer | null;
}
| {
id: number;
method: 'crypto.aesGcmDecrypt';
key: ArrayBuffer;
ciphertext: ArrayBuffer;
nonce: ArrayBuffer;
aad: ArrayBuffer | null;
}
| {
id: number;
method: 'crypto.hkdf';
ikm: ArrayBuffer;
salt: ArrayBuffer;
info: ArrayBuffer;
length: number;
}
| { id: number; method: 'crypto.hmacSha256'; key: ArrayBuffer; data: ArrayBuffer }
// ─── stream.* — host StreamSender / StreamReceiver ────────
| {
id: number;
method: 'stream.createSender';
senderId: number;
streamId: ArrayBuffer;
streamSecret: ArrayBuffer;
laneId: number;
startSeq: number;
}
| {
id: number;
method: 'stream.encryptChunk';
senderId: number;
plaintext: ArrayBuffer;
isLast: boolean;
}
| { id: number; method: 'stream.getSenderLaneSha256'; senderId: number }
| { id: number; method: 'stream.destroySender'; senderId: number }
| {
id: number;
method: 'stream.createReceiver';
receiverId: number;
streamId: ArrayBuffer;
streamSecret: ArrayBuffer;
laneId: number;
startSeq: number;
}
| {
id: number;
method: 'stream.decryptChunk';
receiverId: number;
wireBytes: ArrayBuffer;
}
| { id: number; method: 'stream.getReceiverLaneSha256'; receiverId: number }
| { id: number; method: 'stream.destroyReceiver'; receiverId: number };
export type WorkerResponse =
| { id: number; ok: true; result: WorkerResult }
| {
id: number;
ok: false;
error: { name: string; message: string; code?: string };
};
export type WorkerResult =
| { kind: 'ack' } // void/init/destroy
| { kind: 'pong' }
| { kind: 'keypair'; publicKey: ArrayBuffer; privateKey: ArrayBuffer }
| { kind: 'bytes'; bytes: ArrayBuffer }
| { kind: 'aead-encrypt'; ciphertext: ArrayBuffer; nonce: ArrayBuffer }
| { kind: 'verify'; valid: boolean }
| { kind: 'chunk-encrypt'; bytes: ArrayBuffer; seq: number }
| { kind: 'chunk-decrypt'; plaintext: ArrayBuffer; seq: number; isLast: boolean };
/**
* Pull every transferable `ArrayBuffer` out of a result so the runtime
* can hand ownership to the receiving thread. Order doesn't matter; the
* structured-clone algorithm matches buffers by reference.
*/
export function transferablesOf(result: WorkerResult): ArrayBuffer[] {
switch (result.kind) {
case 'keypair':
return [result.publicKey, result.privateKey];
case 'bytes':
return [result.bytes];
case 'aead-encrypt':
return [result.ciphertext, result.nonce];
case 'chunk-encrypt':
return [result.bytes];
case 'chunk-decrypt':
return [result.plaintext];
default:
return [];
}
}
/**
* Wrap a `Uint8Array` as an `ArrayBuffer` suitable for transfer. If the
* view doesn't span its underlying buffer, copy into a fresh one so we
* never transfer slack the caller still owns.
*/
export function toTransferable(u: Uint8Array): ArrayBuffer {
if (u.byteOffset === 0 && u.byteLength === u.buffer.byteLength) {
return u.buffer as ArrayBuffer;
}
const copy = new Uint8Array(u.byteLength);
copy.set(u);
return copy.buffer;
}
/**
* Like `toTransferable`, but always copies. Use when the original buffer
* must remain valid on the calling thread (e.g. when the caller owns a
* key the worker should not mutate).
*/
export function toTransferableCopy(u: Uint8Array): ArrayBuffer {
const copy = new Uint8Array(u.byteLength);
copy.set(u);
return copy.buffer;
}
export function fromTransferable(buf: ArrayBuffer): Uint8Array {
return new Uint8Array(buf);
}

217
packages/shade-crypto-web/src/worker-streams.ts Normal file
View File

@@ -0,0 +1,217 @@
import type {
WorkerCryptoProvider,
WorkerStreamReceiver,
WorkerStreamSender,
} from './worker-client.js';
/** Default plaintext chunk size — 256 KiB. Matches `@shade/transfer`. */
export const DEFAULT_STREAM_CHUNK_SIZE = 256 * 1024;
export interface CreateEncryptStreamOptions {
provider: WorkerCryptoProvider;
streamId: Uint8Array;
streamSecret: Uint8Array;
laneId?: number;
/**
* Plaintext bytes per AEAD chunk. Smaller = lower latency per chunk +
* more postMessage overhead; larger = higher per-chunk RAM in the
* worker. Default 256 KiB.
*/
chunkSize?: number;
/**
* First sequence number this sender will emit. Default 0.
* Use for resume.
*/
startSeq?: number;
/** First seq this receiver will accept; defaults to 0. */
signal?: AbortSignal;
}
export interface CreateDecryptStreamOptions {
provider: WorkerCryptoProvider;
streamId: Uint8Array;
streamSecret: Uint8Array;
laneId?: number;
startSeq?: number;
signal?: AbortSignal;
}
/**
* Build a `TransformStream<Uint8Array, Uint8Array>` that encrypts every
* passing byte as a stream-chunk wire envelope. The actual AEAD work
* happens in the worker — the main thread only buffers, slices, and
* forwards.
*
* Output: one wire chunk per `enqueue`. Concatenation is the responsibility
* of the downstream consumer (typically an HTTP-shipping `TransformStream`).
*/
export function createEncryptStream(opts: CreateEncryptStreamOptions): {
stream: TransformStream<Uint8Array, Uint8Array>;
/** Promise that resolves to the final lane sha256 once the stream finishes. */
laneSha256: Promise<Uint8Array>;
} {
const chunkSize = opts.chunkSize ?? DEFAULT_STREAM_CHUNK_SIZE;
if (chunkSize <= 0) throw new Error('chunkSize must be positive');
// Plaintext slices accumulate here until we have at least `chunkSize`
// bytes (so we emit fixed-size chunks except for the very last one).
let pending: Uint8Array = new Uint8Array(0);
let sender: WorkerStreamSender | null = null;
let resolveLaneSha: (b: Uint8Array) => void;
let rejectLaneSha: (e: Error) => void;
const laneSha256 = new Promise<Uint8Array>((res, rej) => {
resolveLaneSha = res;
rejectLaneSha = rej;
});
// Cast to `Transformer<I,O>` because some TS lib versions still ship
// the pre-2023 shape without `cancel`. Runtime supports it (Bun, all
// modern browsers).
const transformer = {
async start(): Promise<void> {
sender = await opts.provider.createStreamSender({
streamId: opts.streamId,
streamSecret: opts.streamSecret,
laneId: opts.laneId ?? 0,
startSeq: opts.startSeq ?? 0,
});
},
async transform(
chunk: Uint8Array,
controller: TransformStreamDefaultController<Uint8Array>,
): Promise<void> {
if (sender === null) throw new Error('encryptStream: sender not initialized');
if (chunk.byteLength === 0) return;
pending = concat(pending, chunk);
// Emit complete chunks. Hold back the trailing partial — we don't
// know yet whether it's the last one (which gets isLast=true).
while (pending.byteLength >= chunkSize) {
const slice = pending.subarray(0, chunkSize);
const rest = pending.subarray(chunkSize);
const out = await sender.encryptChunk(slice, false);
controller.enqueue(out.bytes);
// Detach `rest` from the larger backing buffer so it can be GCed.
pending = new Uint8Array(rest);
}
},
async flush(controller: TransformStreamDefaultController<Uint8Array>): Promise<void> {
if (sender === null) throw new Error('encryptStream: sender not initialized');
try {
// Always emit a final chunk with isLast=true. Even if `pending`
// is empty: receivers rely on a trailing isLast envelope to
// mark stream completion.
const out = await sender.encryptChunk(pending, true);
controller.enqueue(out.bytes);
pending = new Uint8Array(0);
const sha = await sender.getLaneSha256();
resolveLaneSha(sha);
} catch (err) {
rejectLaneSha(err instanceof Error ? err : new Error(String(err)));
throw err;
} finally {
await sender.destroy();
sender = null;
}
},
async cancel(reason: unknown): Promise<void> {
try {
if (sender !== null) await sender.destroy();
} finally {
sender = null;
rejectLaneSha(reason instanceof Error ? reason : new Error(String(reason)));
}
},
};
const stream = new TransformStream<Uint8Array, Uint8Array>(
transformer as unknown as Transformer<Uint8Array, Uint8Array>,
);
if (opts.signal) {
const abort = (): void => {
stream.writable.abort(opts.signal!.reason).catch(() => {});
};
if (opts.signal.aborted) abort();
else opts.signal.addEventListener('abort', abort, { once: true });
}
return { stream, laneSha256 };
}
/**
* Build a `TransformStream<Uint8Array, Uint8Array>` that decrypts wire
* stream-chunk envelopes back into plaintext. The input chunks must be
* complete envelopes — the caller is responsible for framing on the wire
* (one envelope per write).
*/
export function createDecryptStream(opts: CreateDecryptStreamOptions): {
stream: TransformStream<Uint8Array, Uint8Array>;
/** Promise that resolves to the final lane sha256 once decryption finishes. */
laneSha256: Promise<Uint8Array>;
} {
let receiver: WorkerStreamReceiver | null = null;
let resolveLaneSha: (b: Uint8Array) => void;
let rejectLaneSha: (e: Error) => void;
const laneSha256 = new Promise<Uint8Array>((res, rej) => {
resolveLaneSha = res;
rejectLaneSha = rej;
});
const transformer = {
async start(): Promise<void> {
receiver = await opts.provider.createStreamReceiver({
streamId: opts.streamId,
streamSecret: opts.streamSecret,
laneId: opts.laneId ?? 0,
startSeq: opts.startSeq ?? 0,
});
},
async transform(
chunk: Uint8Array,
controller: TransformStreamDefaultController<Uint8Array>,
): Promise<void> {
if (receiver === null) throw new Error('decryptStream: receiver not initialized');
const dec = await receiver.decryptChunk(chunk);
if (dec.plaintext.byteLength > 0) controller.enqueue(dec.plaintext);
if (dec.isLast) {
const sha = await receiver.getLaneSha256();
resolveLaneSha(sha);
}
},
async flush(): Promise<void> {
if (receiver !== null) await receiver.destroy();
receiver = null;
},
async cancel(reason: unknown): Promise<void> {
try {
if (receiver !== null) await receiver.destroy();
} finally {
receiver = null;
rejectLaneSha(reason instanceof Error ? reason : new Error(String(reason)));
}
},
};
const stream = new TransformStream<Uint8Array, Uint8Array>(
transformer as unknown as Transformer<Uint8Array, Uint8Array>,
);
if (opts.signal) {
const abort = (): void => {
stream.writable.abort(opts.signal!.reason).catch(() => {});
};
if (opts.signal.aborted) abort();
else opts.signal.addEventListener('abort', abort, { once: true });
}
return { stream, laneSha256 };
}
function concat(a: Uint8Array, b: Uint8Array): Uint8Array {
if (a.byteLength === 0) return b;
if (b.byteLength === 0) return a;
const out = new Uint8Array(a.byteLength + b.byteLength);
out.set(a, 0);
out.set(b, a.byteLength);
return out;
}

231
packages/shade-crypto-web/src/worker.ts Normal file
View File

@@ -0,0 +1,231 @@
/**
* Dedicated Web Worker entry. Bundle this as a module worker:
*
* // Vite / modern Webpack / Rspack
* const w = new Worker(new URL('@shade/crypto-web/worker', import.meta.url),
* { type: 'module' });
*
* The main thread talks to this worker via the protocol in
* `worker-protocol.ts`. All heavy crypto (AES-GCM, HKDF, HMAC, X25519,
* Ed25519) and stream state (per-lane keys + seq counters + running
* sha256) live here so the main thread is never blocked.
*
* Lifecycle: every request is one `postMessage` round-trip. Sender /
* receiver state is keyed by numeric ids handed in by the main thread —
* the worker never invents ids. `destroy*` calls zeroize lane keys.
*/
import { StreamSender, StreamReceiver } from '@shade/streams';
import { SubtleCryptoProvider } from './provider.js';
import {
WORKER_PROTOCOL_VERSION,
fromTransferable,
toTransferable,
transferablesOf,
type WorkerRequest,
type WorkerResponse,
type WorkerResult,
} from './worker-protocol.js';
interface DedicatedWorkerScope {
postMessage(data: unknown, transfer?: ArrayBuffer[]): void;
addEventListener(
type: 'message',
listener: (ev: { data: WorkerRequest }) => void,
): void;
}
const scope = globalThis as unknown as DedicatedWorkerScope;
const provider = new SubtleCryptoProvider();
const senders = new Map<number, StreamSender>();
const receivers = new Map<number, StreamReceiver>();
scope.addEventListener('message', (ev) => {
void handle(ev.data);
});
async function handle(req: WorkerRequest): Promise<void> {
try {
const result = await dispatch(req);
const transfer = transferablesOf(result);
const res: WorkerResponse = { id: req.id, ok: true, result };
scope.postMessage(res, transfer);
} catch (err) {
const error = err instanceof Error
? { name: err.name, message: err.message }
: { name: 'Error', message: String(err) };
const res: WorkerResponse = { id: req.id, ok: false, error };
scope.postMessage(res);
}
}
async function dispatch(req: WorkerRequest): Promise<WorkerResult> {
switch (req.method) {
case 'init': {
if (req.protocolVersion !== WORKER_PROTOCOL_VERSION) {
throw new Error(
`worker protocol version mismatch: main=${req.protocolVersion} worker=${WORKER_PROTOCOL_VERSION}`,
);
}
return { kind: 'ack' };
}
case 'ping':
return { kind: 'pong' };
// ─── crypto.* ─────────────────────────────────────────
case 'crypto.generateX25519KeyPair': {
const kp = await provider.generateX25519KeyPair();
return {
kind: 'keypair',
publicKey: toTransferable(kp.publicKey),
privateKey: toTransferable(kp.privateKey),
};
}
case 'crypto.x25519': {
const out = await provider.x25519(
fromTransferable(req.privateKey),
fromTransferable(req.publicKey),
);
return { kind: 'bytes', bytes: toTransferable(out) };
}
case 'crypto.generateEd25519KeyPair': {
const kp = await provider.generateEd25519KeyPair();
return {
kind: 'keypair',
publicKey: toTransferable(kp.publicKey),
privateKey: toTransferable(kp.privateKey),
};
}
case 'crypto.sign': {
const sig = await provider.sign(
fromTransferable(req.privateKey),
fromTransferable(req.message),
);
return { kind: 'bytes', bytes: toTransferable(sig) };
}
case 'crypto.verify': {
const valid = await provider.verify(
fromTransferable(req.publicKey),
fromTransferable(req.message),
fromTransferable(req.signature),
);
return { kind: 'verify', valid };
}
case 'crypto.aesGcmEncrypt': {
const out = await provider.aesGcmEncrypt(
fromTransferable(req.key),
fromTransferable(req.plaintext),
req.aad ? fromTransferable(req.aad) : undefined,
);
return {
kind: 'aead-encrypt',
ciphertext: toTransferable(out.ciphertext),
nonce: toTransferable(out.nonce),
};
}
case 'crypto.aesGcmDecrypt': {
const out = await provider.aesGcmDecrypt(
fromTransferable(req.key),
fromTransferable(req.ciphertext),
fromTransferable(req.nonce),
req.aad ? fromTransferable(req.aad) : undefined,
);
return { kind: 'bytes', bytes: toTransferable(out) };
}
case 'crypto.hkdf': {
const out = await provider.hkdf(
fromTransferable(req.ikm),
fromTransferable(req.salt),
fromTransferable(req.info),
req.length,
);
return { kind: 'bytes', bytes: toTransferable(out) };
}
case 'crypto.hmacSha256': {
const out = await provider.hmacSha256(
fromTransferable(req.key),
fromTransferable(req.data),
);
return { kind: 'bytes', bytes: toTransferable(out) };
}
// ─── stream.* (sender) ────────────────────────────────
case 'stream.createSender': {
if (senders.has(req.senderId)) {
throw new Error(`senderId ${req.senderId} already exists`);
}
const sender = await StreamSender.create({
crypto: provider,
streamId: fromTransferable(req.streamId),
streamSecret: fromTransferable(req.streamSecret),
laneId: req.laneId,
startSeq: req.startSeq,
});
senders.set(req.senderId, sender);
return { kind: 'ack' };
}
case 'stream.encryptChunk': {
const sender = senders.get(req.senderId);
if (sender === undefined) throw new Error(`unknown senderId ${req.senderId}`);
const chunk = await sender.encryptChunk(fromTransferable(req.plaintext), req.isLast);
return {
kind: 'chunk-encrypt',
bytes: toTransferable(chunk.bytes),
seq: chunk.seq,
};
}
case 'stream.getSenderLaneSha256': {
const sender = senders.get(req.senderId);
if (sender === undefined) throw new Error(`unknown senderId ${req.senderId}`);
return { kind: 'bytes', bytes: toTransferable(sender.getLaneSha256Digest()) };
}
case 'stream.destroySender': {
const sender = senders.get(req.senderId);
if (sender !== undefined) {
sender.destroy();
senders.delete(req.senderId);
}
return { kind: 'ack' };
}
// ─── stream.* (receiver) ──────────────────────────────
case 'stream.createReceiver': {
if (receivers.has(req.receiverId)) {
throw new Error(`receiverId ${req.receiverId} already exists`);
}
const receiver = await StreamReceiver.create({
crypto: provider,
streamId: fromTransferable(req.streamId),
streamSecret: fromTransferable(req.streamSecret),
laneId: req.laneId,
startSeq: req.startSeq,
});
receivers.set(req.receiverId, receiver);
return { kind: 'ack' };
}
case 'stream.decryptChunk': {
const receiver = receivers.get(req.receiverId);
if (receiver === undefined) throw new Error(`unknown receiverId ${req.receiverId}`);
const dec = await receiver.decryptChunk(fromTransferable(req.wireBytes));
return {
kind: 'chunk-decrypt',
plaintext: toTransferable(dec.plaintext),
seq: dec.seq,
isLast: dec.isLast,
};
}
case 'stream.getReceiverLaneSha256': {
const receiver = receivers.get(req.receiverId);
if (receiver === undefined) throw new Error(`unknown receiverId ${req.receiverId}`);
return { kind: 'bytes', bytes: toTransferable(receiver.getLaneSha256Digest()) };
}
case 'stream.destroyReceiver': {
const receiver = receivers.get(req.receiverId);
if (receiver !== undefined) {
receiver.destroy();
receivers.delete(req.receiverId);
}
return { kind: 'ack' };
}
}
}

218
packages/shade-crypto-web/tests/worker-provider.test.ts Normal file
View File

@@ -0,0 +1,218 @@
import { describe, expect, test, afterEach } from 'bun:test';
import {
createWorkerCryptoProvider,
SubtleCryptoProvider,
WorkerCryptoProvider,
} from '../src/index.js';
const WORKER_URL = new URL('../src/worker.ts', import.meta.url);
const subtle = new SubtleCryptoProvider();
let provider: WorkerCryptoProvider | null = null;
afterEach(async () => {
if (provider) {
await provider.destroy();
provider = null;
}
});
async function makeProvider(idleTimeoutMs = 30_000): Promise<WorkerCryptoProvider> {
provider = await createWorkerCryptoProvider({
workerUrl: WORKER_URL,
idleTimeoutMs,
});
return provider;
}
describe('WorkerCryptoProvider — roundtrip and parity', () => {
test('handshake completes', async () => {
const p = await makeProvider();
expect(p).toBeInstanceOf(WorkerCryptoProvider);
});
test('AES-GCM encrypt → worker, decrypt locally — produces same plaintext', async () => {
const p = await makeProvider();
const key = subtle.randomBytes(32);
const plaintext = new TextEncoder().encode('hello shade workers — large enough payload');
const enc = await p.aesGcmEncrypt(key, plaintext);
expect(enc.nonce.length).toBe(12);
// Decrypt with the local SubtleCryptoProvider — proves wire compatibility
const dec = await subtle.aesGcmDecrypt(key, enc.ciphertext, enc.nonce);
expect(dec).toEqual(plaintext);
});
test('AES-GCM with AAD round-trips through worker', async () => {
const p = await makeProvider();
const key = subtle.randomBytes(32);
const plaintext = subtle.randomBytes(1024);
const aad = subtle.randomBytes(16);
const enc = await p.aesGcmEncrypt(key, plaintext, aad);
const dec = await p.aesGcmDecrypt(key, enc.ciphertext, enc.nonce, aad);
expect(dec).toEqual(plaintext);
});
test('AES-GCM decrypt rejects tampered ciphertext', async () => {
const p = await makeProvider();
const key = subtle.randomBytes(32);
const plaintext = new TextEncoder().encode('untampered');
const enc = await p.aesGcmEncrypt(key, plaintext);
enc.ciphertext[0]! ^= 0x01;
await expect(p.aesGcmDecrypt(key, enc.ciphertext, enc.nonce)).rejects.toThrow();
});
test('HKDF parity with SubtleCryptoProvider', async () => {
const p = await makeProvider();
const ikm = subtle.randomBytes(32);
const salt = subtle.randomBytes(16);
const info = new TextEncoder().encode('test info');
const a = await p.hkdf(ikm, salt, info, 64);
const b = await subtle.hkdf(ikm, salt, info, 64);
expect(a).toEqual(b);
});
test('HMAC-SHA256 parity with SubtleCryptoProvider', async () => {
const p = await makeProvider();
const key = subtle.randomBytes(32);
const data = subtle.randomBytes(256);
const a = await p.hmacSha256(key, data);
const b = await subtle.hmacSha256(key, data);
expect(a).toEqual(b);
});
test('X25519 DH agrees with SubtleCryptoProvider', async () => {
const p = await makeProvider();
const alice = await p.generateX25519KeyPair();
const bob = await subtle.generateX25519KeyPair();
const ab = await p.x25519(alice.privateKey, bob.publicKey);
const ba = await subtle.x25519(bob.privateKey, alice.publicKey);
expect(ab).toEqual(ba);
});
test('Ed25519 sign in worker, verify locally', async () => {
const p = await makeProvider();
const kp = await p.generateEd25519KeyPair();
const msg = new TextEncoder().encode('please sign me');
const sig = await p.sign(kp.privateKey, msg);
expect(await subtle.verify(kp.publicKey, msg, sig)).toBe(true);
});
test('Ed25519 verify rejects tampered signature', async () => {
const p = await makeProvider();
const kp = await subtle.generateEd25519KeyPair();
const msg = new TextEncoder().encode('msg');
const sig = await subtle.sign(kp.privateKey, msg);
sig[0]! ^= 0x01;
expect(await p.verify(kp.publicKey, msg, sig)).toBe(false);
});
test('local sync helpers do not round-trip', async () => {
const p = await makeProvider();
const a = p.randomBytes(16);
expect(a.length).toBe(16);
expect(p.constantTimeEqual(a, a)).toBe(true);
expect(p.constantTimeEqual(a, new Uint8Array(16))).toBe(false);
expect(typeof p.randomUint32()).toBe('number');
});
test('errors from worker propagate as rejected promises', async () => {
const p = await makeProvider();
const wrongKey = subtle.randomBytes(32);
const ct = subtle.randomBytes(48);
const nonce = subtle.randomBytes(12);
await expect(p.aesGcmDecrypt(wrongKey, ct, nonce)).rejects.toThrow();
});
test('parallel calls do not interleave incorrectly', async () => {
const p = await makeProvider();
const key = subtle.randomBytes(32);
const inputs = Array.from({ length: 16 }, (_, i) =>
new TextEncoder().encode(`payload-${i}-${'x'.repeat(50 * i)}`),
);
const encs = await Promise.all(inputs.map((pt) => p.aesGcmEncrypt(key, pt)));
const decs = await Promise.all(
encs.map((e) => p.aesGcmDecrypt(key, e.ciphertext, e.nonce)),
);
decs.forEach((d, i) => expect(d).toEqual(inputs[i]!));
});
test('after destroy(), calls reject', async () => {
const p = await makeProvider();
await p.destroy();
await expect(p.aesGcmEncrypt(subtle.randomBytes(32), new Uint8Array(8))).rejects.toThrow(
/destroyed/,
);
provider = null;
});
test('rotate() respawns transparently', async () => {
const p = await makeProvider();
const key = subtle.randomBytes(32);
await p.aesGcmEncrypt(key, new Uint8Array(8));
await p.rotate();
const out = await p.aesGcmEncrypt(key, new TextEncoder().encode('still works'));
const dec = await subtle.aesGcmDecrypt(key, out.ciphertext, out.nonce);
expect(new TextDecoder().decode(dec)).toBe('still works');
});
test('idle-timeout terminates worker but next call respawns', async () => {
const p = await makeProvider(120);
const key = subtle.randomBytes(32);
await p.aesGcmEncrypt(key, new Uint8Array(8));
// Wait for the idle timer to fire.
await new Promise((r) => setTimeout(r, 250));
// Next call should still succeed — proves respawn works.
const out = await p.aesGcmEncrypt(key, new TextEncoder().encode('respawned'));
const dec = await subtle.aesGcmDecrypt(key, out.ciphertext, out.nonce);
expect(new TextDecoder().decode(dec)).toBe('respawned');
});
test('configureWorkerCrypto throws on protocol mismatch', async () => {
// Spawn with a fake "spawn" that returns a worker echoing the wrong version.
const fakeProvider = new WorkerCryptoProvider({
workerUrl: WORKER_URL,
spawn: () => {
type Listener = (ev: { data: unknown }) => void;
const listeners: Listener[] = [];
return {
postMessage(msg: unknown): void {
const m = msg as { id: number; method: string };
if (m.method === 'init') {
setTimeout(() => {
for (const l of listeners) {
l({
data: {
id: m.id,
ok: false,
error: { name: 'Error', message: 'protocol version mismatch' },
},
});
}
}, 0);
}
},
addEventListener(type: string, listener: Listener): void {
if (type === 'message') listeners.push(listener);
},
removeEventListener(): void {
// no-op
},
terminate(): void {
// no-op
},
};
},
});
await expect(fakeProvider.handshake()).rejects.toThrow(/protocol/);
await fakeProvider.destroy();
});
});

230
packages/shade-crypto-web/tests/worker-streams.test.ts Normal file
View File

@@ -0,0 +1,230 @@
import { afterEach, describe, expect, test } from 'bun:test';
import { sha256 } from '@noble/hashes/sha2.js';
import {
createDecryptStream,
createEncryptStream,
createWorkerCryptoProvider,
SubtleCryptoProvider,
WorkerCryptoProvider,
} from '../src/index.js';
const WORKER_URL = new URL('../src/worker.ts', import.meta.url);
const subtle = new SubtleCryptoProvider();
let provider: WorkerCryptoProvider | null = null;
afterEach(async () => {
if (provider) {
await provider.destroy();
provider = null;
}
});
async function makeProvider(): Promise<WorkerCryptoProvider> {
provider = await createWorkerCryptoProvider({ workerUrl: WORKER_URL });
return provider;
}
async function readAll(rs: ReadableStream<Uint8Array>): Promise<Uint8Array> {
const reader = rs.getReader();
const parts: Uint8Array[] = [];
let total = 0;
for (;;) {
const { done, value } = await reader.read();
if (done) break;
parts.push(value);
total += value.byteLength;
}
const out = new Uint8Array(total);
let off = 0;
for (const p of parts) {
out.set(p, off);
off += p.byteLength;
}
return out;
}
function streamFromChunks(chunks: Uint8Array[]): ReadableStream<Uint8Array> {
let i = 0;
return new ReadableStream<Uint8Array>({
pull(controller) {
if (i < chunks.length) controller.enqueue(chunks[i++]!);
else controller.close();
},
});
}
describe('encryptStream / decryptStream — round-trip', () => {
test('round-trips small payload exactly', async () => {
const p = await makeProvider();
const streamId = subtle.randomBytes(16);
const streamSecret = subtle.randomBytes(32);
const plaintext = new TextEncoder().encode('hello stream');
const enc = await createEncryptStream({
provider: p,
streamId,
streamSecret,
chunkSize: 1024,
});
const wireBytes = await readAll(
streamFromChunks([plaintext]).pipeThrough(enc.stream),
);
// Frame: each enqueue is one wire envelope. We can't trivially split
// a concatenated buffer back into envelopes, but we know how many
// chunks were emitted (len/chunkSize, plus the final isLast). Easier
// path: collect them as separate writes through a side channel.
const chunks: Uint8Array[] = [];
await streamFromChunks([plaintext])
.pipeThrough(
(
await createEncryptStream({
provider: p,
streamId,
streamSecret,
chunkSize: 1024,
})
).stream,
)
.pipeTo(
new WritableStream<Uint8Array>({
write(c) {
chunks.push(c);
},
}),
);
const dec = await createDecryptStream({ provider: p, streamId, streamSecret });
const recovered = await readAll(streamFromChunks(chunks).pipeThrough(dec.stream));
expect(recovered).toEqual(plaintext);
expect(wireBytes.byteLength).toBeGreaterThan(plaintext.byteLength); // overhead
});
test('round-trips multi-chunk payload with sha256 parity', async () => {
const p = await makeProvider();
const streamId = subtle.randomBytes(16);
const streamSecret = subtle.randomBytes(32);
const total = 750 * 1024; // 750 KiB → forces 3+ chunks at 256 KiB
const plaintext = subtle.randomBytes(total);
const expectedSha = sha256(plaintext);
const enc = await createEncryptStream({
provider: p,
streamId,
streamSecret,
chunkSize: 256 * 1024,
});
const wireChunks: Uint8Array[] = [];
await streamFromChunks([plaintext])
.pipeThrough(enc.stream)
.pipeTo(
new WritableStream<Uint8Array>({
write(c) {
wireChunks.push(c);
},
}),
);
// 750 KiB / 256 KiB = 2 full chunks + 1 final (238 KiB, isLast=true)
expect(wireChunks.length).toBe(3);
const senderLaneSha = await enc.laneSha256;
expect(senderLaneSha).toEqual(expectedSha);
const dec = await createDecryptStream({
provider: p,
streamId,
streamSecret,
});
const recovered = await readAll(streamFromChunks(wireChunks).pipeThrough(dec.stream));
expect(recovered).toEqual(plaintext);
expect(await dec.laneSha256).toEqual(expectedSha);
});
test('fragmented input produces same output as single-shot', async () => {
const p = await makeProvider();
const streamId = subtle.randomBytes(16);
const streamSecret = subtle.randomBytes(32);
const plaintext = subtle.randomBytes(50_000);
async function run(parts: Uint8Array[]): Promise<Uint8Array[]> {
const wire: Uint8Array[] = [];
const e = await createEncryptStream({
provider: p!,
streamId,
streamSecret,
chunkSize: 8 * 1024,
});
await streamFromChunks(parts)
.pipeThrough(e.stream)
.pipeTo(new WritableStream({ write: (c) => void wire.push(c) }));
return wire;
}
const single = await run([plaintext]);
const split = await run([
plaintext.subarray(0, 17_000),
plaintext.subarray(17_000, 33_000),
plaintext.subarray(33_000),
]);
expect(split.length).toBe(single.length);
for (let i = 0; i < single.length; i++) {
// Same chunk size, same lane key, same seq — wire bytes match
// byte-for-byte (deterministic nonces + AEAD).
expect(split[i]).toEqual(single[i]!);
}
});
test('100 KiB stream end-to-end completes', async () => {
const p = await makeProvider();
const streamId = subtle.randomBytes(16);
const streamSecret = subtle.randomBytes(32);
const plaintext = subtle.randomBytes(100 * 1024);
const enc = await createEncryptStream({
provider: p,
streamId,
streamSecret,
chunkSize: 16 * 1024,
});
const wire: Uint8Array[] = [];
await streamFromChunks([plaintext])
.pipeThrough(enc.stream)
.pipeTo(new WritableStream({ write: (c) => void wire.push(c) }));
const dec = await createDecryptStream({ provider: p, streamId, streamSecret });
const out = await readAll(streamFromChunks(wire).pipeThrough(dec.stream));
expect(out).toEqual(plaintext);
expect(await dec.laneSha256).toEqual(await enc.laneSha256);
});
test('decryptStream rejects out-of-order chunks', async () => {
const p = await makeProvider();
const streamId = subtle.randomBytes(16);
const streamSecret = subtle.randomBytes(32);
const plaintext = subtle.randomBytes(20_000);
const enc = await createEncryptStream({
provider: p,
streamId,
streamSecret,
chunkSize: 4 * 1024,
});
const wire: Uint8Array[] = [];
await streamFromChunks([plaintext])
.pipeThrough(enc.stream)
.pipeTo(new WritableStream({ write: (c) => void wire.push(c) }));
expect(wire.length).toBeGreaterThan(2);
// Swap first and second chunk
[wire[0], wire[1]] = [wire[1]!, wire[0]!];
const dec = await createDecryptStream({ provider: p, streamId, streamSecret });
await expect(
streamFromChunks(wire).pipeThrough(dec.stream).pipeTo(
new WritableStream({ write() {} }),
),
).rejects.toThrow();
});
});

2
packages/shade-dashboard/package.json
View File

@@ -1,6 +1,6 @@
{
"name": "@shade/dashboard",
"version": "0.3.0",
"version": "4.0.0",
"type": "module",
"scripts": {
"dev": "vite",

12
packages/shade-dashboard/src/stubs/bun-sqlite.ts Normal file
View File

@@ -0,0 +1,12 @@
// Browser-build stub for `bun:sqlite`. The dashboard never instantiates a
// SQLite database — it talks to the prekey server over HTTP. Importing
// transitive code paths from `@shade/storage-sqlite` is fine; calling
// `new Database(...)` would throw, but the dashboard does not.
export class Database {
constructor() {
throw new Error(
'bun:sqlite is not available in the dashboard browser bundle. The dashboard talks to the prekey server over HTTP — instantiating a SQLite database here is a bug.',
);
}
}

13
packages/shade-dashboard/vite.config.ts
View File

@@ -1,9 +1,22 @@
import { defineConfig } from 'vite';
import react from '@vitejs/plugin-react';
import { resolve } from 'node:path';
// `@shade/widgets` re-exports from `@shade/sdk`, which transitively imports
// `@shade/storage-sqlite` (and therefore `bun:sqlite`). Vite externalizes
// `bun:sqlite` for browser builds, but the `import { Database } from
// 'bun:sqlite'` named-import then fails to resolve. Alias the module to an
// in-tree stub — the dashboard never executes the storage code path
// (it talks to the prekey server via `Shade.send` / `fetch`), so the stub
// is never reached at runtime.
export default defineConfig({
plugins: [react()],
base: '/dashboard/',
resolve: {
alias: {
'bun:sqlite': resolve(__dirname, 'src/stubs/bun-sqlite.ts'),
},
},
build: {
outDir: 'dist',
emptyOutDir: true,

3
packages/shade-files/package.json
View File

@@ -1,6 +1,6 @@
{
"name": "@shade/files",
"version": "0.3.0",
"version": "4.0.0",
"type": "module",
"main": "src/index.ts",
"types": "src/index.ts",
@@ -17,6 +17,7 @@
"dependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/observability": "workspace:*",
"@shade/proto": "workspace:*",
"@shade/sdk": "workspace:*",
"@shade/streams": "workspace:*",

4
packages/shade-files/src/integration/files-namespace.ts
View File

@@ -80,9 +80,13 @@ export function createFilesNamespace(shade: Shade): FilesNamespace {
if (state.serverBridge === null) {
state.serverBridge = await createServerStreamsBridge(shade);
}
const inheritedObservability = shade.getObservability?.();
const handler = createFileHandler(shade, {
...handlerConfig,
streamsBridge: state.serverBridge,
...(handlerConfig.observability === undefined && inheritedObservability !== undefined
? { observability: inheritedObservability }
: {}),
});
const detach = attachFileHandler(state.channel, handler);
state.serverHandler = handler;

58
packages/shade-files/src/server/handler.ts
View File

@@ -79,6 +79,17 @@ import {
NOOP_METRIC_SINK,
type MetricSink,
} from './metrics.js';
import {
ATTR_BYTES_BIN,
ATTR_ERROR_CODE,
ATTR_OP,
ATTR_PEER_HASH,
ATTR_RESULT,
bytesBin,
NOOP_HOOK,
peerHash,
type ObservabilityHook,
} from '@shade/observability';
import {
CustomArgsSchema,
CustomResultSchema,
@@ -153,6 +164,12 @@ export interface FileHandlerConfig extends FileHandlerOps {
isFingerprintVerified?: (sender: string) => boolean | Promise<boolean>;
/** Vendor-neutral metrics sink. */
onMetric?: MetricSink;
/**
* Optional OTel observability hook. When supplied, each op is wrapped in
* a `shade.files.op` span with PII-safe attributes (peer.hash, op,
* bytes.bin, result, error.code). Defaults to no-op when omitted.
*/
observability?: ObservabilityHook;
/** Called BEFORE the handler runs. Throw to deny. */
beforeOp?: (ctx: OpContext<unknown>) => void | Promise<void>;
/** Called AFTER the handler returns. Result is the validated response. */
@@ -207,12 +224,34 @@ export function createFileHandler(
const defaultTimeoutMs = config.defaultTimeoutMs ?? 60_000;
const ioTimeoutMs = config.ioTimeoutMs ?? 60_000;
const metrics: MetricSink = config.onMetric ?? NOOP_METRIC_SINK;
const observability: ObservabilityHook = config.observability ?? NOOP_HOOK;
const customRegistrations = config.custom ?? {};
const isCustomKind = (kind: string): boolean => kind === 'shade.fs.custom/v1';
async function handleRequest(
from: string,
request: RpcRequest,
): Promise<RpcResponse | RpcError> {
const span = observability.startSpan('shade.files.op', {
[ATTR_PEER_HASH]: peerHash(from),
});
try {
const out = await runHandleRequest(from, request, span);
return out;
} catch (err) {
span.recordException(err);
span.setAttribute(ATTR_ERROR_CODE, errorCodeOf(err));
span.setStatus('error');
throw err;
} finally {
span.end();
}
}
async function runHandleRequest(
from: string,
request: RpcRequest,
span: import('@shade/observability').Span,
): Promise<RpcResponse | RpcError> {
// 0. Replay-window check (independent of sig — defends against
// intercept-and-resend even when sig verification is disabled).
@@ -316,6 +355,7 @@ export function createFileHandler(
// Replace payload with validated value (Zod may apply defaults).
(parsedArgs as CustomArgs).payload = payloadParse.data;
}
span.setAttribute(ATTR_OP, resolvedOpKind);
// 3. Path validation (skip ops without a path)
let primaryPath = '';
@@ -488,6 +528,10 @@ export function createFileHandler(
const durationMs = Date.now() - startedAt;
metrics(METRIC_OP_DURATION_MS, durationMs, { op: resolvedOpKind, result: 'error' });
metrics(METRIC_OP_TOTAL, 1, { op: resolvedOpKind, result: 'error' });
span.setAttribute(ATTR_RESULT, 'error');
span.setAttribute(ATTR_ERROR_CODE, errorCodeOf(err));
span.recordException(err);
span.setStatus('error');
cleanup({ release: true });
if (config.onError !== undefined) {
try {
@@ -543,6 +587,7 @@ export function createFileHandler(
const durationMs = Date.now() - startedAt;
metrics(METRIC_OP_DURATION_MS, durationMs, { op: resolvedOpKind, result: 'ok' });
metrics(METRIC_OP_TOTAL, 1, { op: resolvedOpKind, result: 'ok' });
span.setAttribute(ATTR_RESULT, 'ok');
if (estimatedBytes > 0) {
// Inbound bytes (write) vs outbound (read) — both reuse the same
// pre-call `estimatedBytes`, since post-execution reconciliation
@@ -551,7 +596,9 @@ export function createFileHandler(
if (direction !== null) {
metrics(direction, estimatedBytes, { op: resolvedOpKind });
}
span.setAttribute(ATTR_BYTES_BIN, bytesBin(estimatedBytes));
}
span.setStatus('ok');
return makeResponseEnvelope(request, resultParse.data);
@@ -654,6 +701,17 @@ function makeResponseEnvelope(req: RpcRequest, result: unknown): RpcResponse {
};
}
function errorCodeOf(err: unknown): string {
if (err === null || err === undefined) return 'SHADE_UNKNOWN';
if (typeof err === 'object') {
const code = (err as { code?: unknown }).code;
if (typeof code === 'string' && code.length > 0) return code;
const name = (err as { name?: unknown }).name;
if (typeof name === 'string' && name.length > 0) return name;
}
return 'SHADE_UNKNOWN';
}
function makeErrorEnvelope(req: RpcRequest, err: unknown): RpcError {
return {
kind: 'shade.fs.error/v1',

21
packages/shade-inbox-server/package.json Normal file
View File

@@ -0,0 +1,21 @@
{
"name": "@shade/inbox-server",
"version": "4.0.0",
"type": "module",
"main": "src/index.ts",
"types": "src/index.ts",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/observability": "workspace:*",
"@shade/server": "workspace:*",
"hono": "^4.12.12"
},
"optionalDependencies": {
"@shade/crypto-web": "workspace:*",
"@shade/storage-postgres": "workspace:*",
"@shade/storage-sqlite": "workspace:*"
},
"devDependencies": {
"@shade/crypto-web": "workspace:*"
}
}

461
packages/shade-inbox-server/src/bridge.ts Normal file
View File

@@ -0,0 +1,461 @@
/**
* Bridge routes — V3.7.
*
* Three transports, one delivery semantic. Each one streams the same
* not-yet-acked inbox blobs for an authenticated address:
*
* GET /v1/bridge/stream — SSE feed, one envelope per `event: envelope`
* GET /v1/bridge/poll — long-poll, returns at most one batch then closes
* GET /v1/bridge/ws — WebSocket, JSON frame per envelope
*
* Auth: signed query string (`address`, `kind`, `since`, `signedAt`,
* `signature`). The signature is verified against the address's owner key
* registered via `/v1/inbox/register`. The `kind` field is bound into the
* canonical signed payload to prevent cross-endpoint replay.
*
* Cursor semantics: `since` is the highest `receivedAt` the client already
* processed. The server returns blobs strictly greater than that cursor and
* advances the client's cursor by emitting a fresh `id:` (SSE) or by
* including the highest seen `receivedAt` in the JSON response (poll/ws).
*
* The implementations subscribe to {@link InboxServerEvents} so that newly
* stored blobs land on connected clients without polling the store. The
* fallback path (no events configured) relies on a small in-process polling
* timer with a configurable interval.
*/
import { Hono, type Context } from 'hono';
import { streamSSE } from 'hono/streaming';
import { createBunWebSocket } from 'hono/bun';
import type { CryptoProvider } from '@shade/core';
import {
errorToHttpStatus,
ShadeError,
ValidationError,
UnauthorizedError,
toBase64,
} from '@shade/core';
import { verifyPayload, validateAddress } from '@shade/server';
import type { InboxStore } from './store.js';
import type { InboxServerEvents } from './events.js';
export type BridgeKind = 'stream' | 'poll' | 'ws';
export interface BridgeRoutesOptions {
store: InboxStore;
crypto: CryptoProvider;
/** Optional events emitter — enables push-style delivery. */
events?: InboxServerEvents;
/** Maximum blobs returned per fetch page. Default 50. */
pageLimit?: number;
/** Default long-poll hold (ms). Default 25_000 (under typical proxy cutoffs). */
longPollTimeoutMs?: number;
/** Maximum long-poll hold (ms). Hard cap. Default 55_000. */
longPollMaxTimeoutMs?: number;
/** SSE heartbeat interval (ms). Default 15_000. */
heartbeatIntervalMs?: number;
/**
* Fallback poll interval (ms) used when no `events` emitter is wired in.
* The bridge will re-check the store at this cadence to detect new blobs.
* Default 1_000.
*/
fallbackPollIntervalMs?: number;
}
interface VerifiedBridgeRequest {
address: string;
kind: BridgeKind;
since: number;
}
/**
* Build the bridge Hono router and a paired Bun-WebSocket handler.
*
* The HTTP routes (`/v1/bridge/stream`, `/v1/bridge/poll`) work on every
* Hono runtime. The `/v1/bridge/ws` route requires `hono/adapter/bun` to be
* available — we lazy-require it so that non-Bun deployments aren't
* forced to ship the import.
*/
export function createBridgeRoutes(opts: BridgeRoutesOptions): {
app: Hono;
/** Pass to `Bun.serve({ websocket })`. Undefined if Bun adapter is missing. */
websocket: unknown;
} {
const app = new Hono();
const pageLimit = opts.pageLimit ?? 50;
const heartbeatIntervalMs = opts.heartbeatIntervalMs ?? 15_000;
const longPollDefault = opts.longPollTimeoutMs ?? 25_000;
const longPollMax = opts.longPollMaxTimeoutMs ?? 55_000;
const fallbackPollIntervalMs = opts.fallbackPollIntervalMs ?? 1_000;
app.onError((err, c) => {
if (err instanceof ShadeError) {
const status = errorToHttpStatus(err);
return c.json(err.toJSON(), status as any);
}
console.error('[Shade] Unhandled bridge error:', err);
return c.json({ error: 'Internal server error' }, 500);
});
// ─── SSE ──────────────────────────────────────────────────────
app.get('/v1/bridge/stream', async (c) => {
const verified = await verifyBridgeAuth(c, opts, 'stream');
return streamSSE(c, async (stream) => {
const address = verified.address;
let cursor = verified.since;
const writer = makeBlobWriter(opts.store, pageLimit);
// Initial backlog drain.
const flushed = await flushTo(writer, address, cursor, async (blob) => {
await stream.writeSSE({
id: String(blob.receivedAt),
event: 'envelope',
data: JSON.stringify(serializeBlob(blob)),
});
});
cursor = Math.max(cursor, flushed);
// Hook up event-driven push if available, else fall back to a poll
// timer that does the same scan.
let cleanupSubscription: (() => void) | null = null;
let signalled = false;
let pendingFlushPromise: Promise<void> = Promise.resolve();
const triggerFlush = (): void => {
signalled = true;
// Serialize fan-in so concurrent triggers don't double-fetch.
pendingFlushPromise = pendingFlushPromise.then(async () => {
while (signalled) {
signalled = false;
const drained = await flushTo(writer, address, cursor, async (blob) => {
await stream.writeSSE({
id: String(blob.receivedAt),
event: 'envelope',
data: JSON.stringify(serializeBlob(blob)),
});
});
if (drained > cursor) cursor = drained;
}
});
};
if (opts.events) {
cleanupSubscription = opts.events.on((e) => {
if (e.name === 'inbox.blob_stored' && e.data.address === address) {
triggerFlush();
}
});
}
const fallbackTimer = setInterval(() => triggerFlush(), fallbackPollIntervalMs);
const heartbeat = setInterval(() => {
// Comment lines are valid SSE keepalives.
stream.write(`: ping ${Date.now()}\n\n`).catch(() => {});
}, heartbeatIntervalMs);
// Wait for the request to abort (client disconnect).
await new Promise<void>((resolve) => {
const sig = c.req.raw.signal;
if (sig.aborted) return resolve();
sig.addEventListener('abort', () => resolve(), { once: true });
});
cleanupSubscription?.();
clearInterval(fallbackTimer);
clearInterval(heartbeat);
await pendingFlushPromise.catch(() => {});
});
});
// ─── Long-poll ────────────────────────────────────────────────
app.get('/v1/bridge/poll', async (c) => {
const verified = await verifyBridgeAuth(c, opts, 'poll');
const requestedTimeout = Number(c.req.query('timeoutMs') ?? longPollDefault);
const timeoutMs = Math.min(
Math.max(0, Number.isFinite(requestedTimeout) ? requestedTimeout : longPollDefault),
longPollMax,
);
// Try immediate fetch first.
let blobs = await opts.store.fetchBlobs({
address: verified.address,
sinceCursor: verified.since,
now: Date.now(),
limit: pageLimit,
});
if (blobs.length > 0) {
return c.json(buildPollResponse(blobs, verified.since));
}
// Otherwise, wait for either a new event or the timeout.
blobs = await waitForBlobs({
events: opts.events ?? null,
store: opts.store,
address: verified.address,
since: verified.since,
timeoutMs,
pageLimit,
fallbackPollIntervalMs,
abortSignal: c.req.raw.signal,
});
return c.json(buildPollResponse(blobs, verified.since));
});
// ─── WebSocket ────────────────────────────────────────────────
// Hono's Bun adapter resolves `getBunServer` from the request's `env`
// (the second argument of Bun.serve's fetch). On non-Bun runtimes the
// upgrade simply fails at runtime; the SSE/long-poll routes still work.
const { upgradeWebSocket, websocket } = createBunWebSocket();
app.get(
'/v1/bridge/ws',
upgradeWebSocket(async (c: Context) => {
let verified: VerifiedBridgeRequest | null = null;
let upgradeError: Error | null = null;
try {
verified = await verifyBridgeAuth(c, opts, 'ws');
} catch (err) {
upgradeError = err as Error;
}
if (!verified) {
// Hono's API doesn't let us reject the upgrade with a status code
// before opening the socket; close immediately on open with a 4xxx
// policy code so the client can fall back to a different bridge.
return {
onOpen(_evt: unknown, ws: { close: (code?: number, reason?: string) => void }) {
const status =
upgradeError instanceof ShadeError ? errorToHttpStatus(upgradeError) : 500;
ws.close(4000 + (status % 1000), upgradeError?.message ?? 'unauthorized');
},
};
}
const address = verified.address;
let cursor = verified.since;
const writer = makeBlobWriter(opts.store, pageLimit);
let unsubscribe: (() => void) | null = null;
let fallbackTimer: ReturnType<typeof setInterval> | null = null;
let pendingFlushPromise: Promise<void> = Promise.resolve();
let signalled = false;
let connected = true;
return {
onOpen(_evt: unknown, ws: {
send: (data: string) => void;
close: (code?: number, reason?: string) => void;
}) {
const triggerFlush = (): void => {
signalled = true;
pendingFlushPromise = pendingFlushPromise.then(async () => {
while (signalled && connected) {
signalled = false;
const drained = await flushTo(writer, address, cursor, async (blob) => {
ws.send(JSON.stringify(serializeBlob(blob)));
});
if (drained > cursor) cursor = drained;
}
});
};
if (opts.events) {
unsubscribe = opts.events.on((e) => {
if (e.name === 'inbox.blob_stored' && e.data.address === address) {
triggerFlush();
}
});
}
fallbackTimer = setInterval(() => triggerFlush(), fallbackPollIntervalMs);
triggerFlush();
},
onClose() {
connected = false;
unsubscribe?.();
if (fallbackTimer) clearInterval(fallbackTimer);
},
};
}),
);
return { app, websocket };
}
// ─── helpers ──────────────────────────────────────────────────
async function verifyBridgeAuth(
c: Context,
opts: BridgeRoutesOptions,
expectedKind: BridgeKind,
): Promise<VerifiedBridgeRequest> {
const url = new URL(c.req.url);
const qs = url.searchParams;
const address = validateAddress(qs.get('address'));
const kind = qs.get('kind');
if (kind !== expectedKind) {
throw new ValidationError(`bridge kind mismatch: expected ${expectedKind}`, 'kind');
}
const sinceStr = qs.get('since');
const signedAtStr = qs.get('signedAt');
const signature = qs.get('signature');
if (sinceStr === null) throw new ValidationError('missing since', 'since');
if (signedAtStr === null) throw new ValidationError('missing signedAt', 'signedAt');
if (!signature) throw new ValidationError('missing signature', 'signature');
const since = Number(sinceStr);
const signedAt = Number(signedAtStr);
if (!Number.isFinite(since) || since < 0) {
throw new ValidationError('since must be a non-negative number', 'since');
}
if (!Number.isFinite(signedAt)) {
throw new ValidationError('signedAt must be a number', 'signedAt');
}
const owner = await opts.store.getAddressOwner(address);
if (!owner) {
throw new UnauthorizedError(`address ${address} is not registered`);
}
await verifyPayload(opts.crypto, owner, {
address,
kind,
since,
signedAt,
signature,
});
return { address, kind: kind as BridgeKind, since };
}
interface BlobRow {
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
}
interface BlobWriter {
fetchPage(address: string, cursor: number): Promise<BlobRow[]>;
}
function makeBlobWriter(store: InboxStore, pageLimit: number): BlobWriter {
return {
async fetchPage(address, cursor) {
return store.fetchBlobs({
address,
sinceCursor: cursor,
now: Date.now(),
limit: pageLimit,
});
},
};
}
async function flushTo(
writer: BlobWriter,
address: string,
startCursor: number,
emit: (blob: BlobRow) => Promise<void>,
): Promise<number> {
let cursor = startCursor;
// Drain page-by-page so a backlog larger than `pageLimit` still flushes.
// eslint-disable-next-line no-constant-condition
while (true) {
const page = await writer.fetchPage(address, cursor);
if (page.length === 0) break;
for (const row of page) {
await emit(row);
if (row.receivedAt > cursor) cursor = row.receivedAt;
}
if (page.length === 0) break;
}
return cursor;
}
function serializeBlob(blob: BlobRow): {
msgId: string;
ciphertext: string;
receivedAt: number;
expiresAt: number;
} {
return {
msgId: blob.msgId,
ciphertext: toBase64(blob.ciphertext),
receivedAt: blob.receivedAt,
expiresAt: blob.expiresAt,
};
}
function buildPollResponse(blobs: BlobRow[], sinceFallback: number): {
blobs: ReturnType<typeof serializeBlob>[];
cursor: number;
hasMore: boolean;
} {
const out = blobs.map(serializeBlob);
const cursor = blobs.length > 0 ? blobs[blobs.length - 1]!.receivedAt : sinceFallback;
return { blobs: out, cursor, hasMore: false };
}
interface WaitForBlobsArgs {
events: InboxServerEvents | null;
store: InboxStore;
address: string;
since: number;
timeoutMs: number;
pageLimit: number;
fallbackPollIntervalMs: number;
abortSignal?: AbortSignal;
}
async function waitForBlobs(args: WaitForBlobsArgs): Promise<BlobRow[]> {
if (args.timeoutMs === 0) return [];
return new Promise<BlobRow[]>((resolve) => {
let resolved = false;
let unsubscribe: (() => void) | null = null;
let timer: ReturnType<typeof setTimeout> | null = null;
let fallback: ReturnType<typeof setInterval> | null = null;
let abortHandler: (() => void) | null = null;
const finish = (blobs: BlobRow[]) => {
if (resolved) return;
resolved = true;
if (timer) clearTimeout(timer);
if (fallback) clearInterval(fallback);
if (unsubscribe) unsubscribe();
if (abortHandler && args.abortSignal) {
args.abortSignal.removeEventListener('abort', abortHandler);
}
resolve(blobs);
};
const tryFetch = async (): Promise<void> => {
try {
const rows = await args.store.fetchBlobs({
address: args.address,
sinceCursor: args.since,
now: Date.now(),
limit: args.pageLimit,
});
if (rows.length > 0) finish(rows);
} catch {
// swallow — let timeout handle it
}
};
if (args.events) {
unsubscribe = args.events.on((e) => {
if (e.name === 'inbox.blob_stored' && e.data.address === args.address) {
void tryFetch();
}
});
}
fallback = setInterval(tryFetch, args.fallbackPollIntervalMs);
timer = setTimeout(() => finish([]), args.timeoutMs);
if (args.abortSignal) {
if (args.abortSignal.aborted) {
finish([]);
return;
}
abortHandler = () => finish([]);
args.abortSignal.addEventListener('abort', abortHandler, { once: true });
}
});
}

69
packages/shade-inbox-server/src/cleanup.ts Normal file
View File

@@ -0,0 +1,69 @@
import type { InboxStore } from './store.js';
import type { InboxServerEvents } from './events.js';
/**
* Periodic prune task — drops every blob whose `expires_at <= now`.
*
* Configurable via env vars:
* SHADE_INBOX_PRUNE_INTERVAL_MINUTES (default 5)
*/
export class InboxPruneTask {
private timer: ReturnType<typeof setInterval> | null = null;
private running = false;
private readonly intervalMs: number;
constructor(
private readonly store: InboxStore,
options: {
intervalMinutes?: number;
events?: InboxServerEvents;
logger?: { info: (msg: string, meta?: unknown) => void; error: (msg: string, meta?: unknown) => void };
} = {},
) {
const minutes = options.intervalMinutes
?? Number(process.env.SHADE_INBOX_PRUNE_INTERVAL_MINUTES ?? 5);
this.intervalMs = minutes * 60 * 1000;
this.events = options.events;
this.logger = options.logger ?? null;
}
private readonly events: InboxServerEvents | undefined;
private readonly logger: {
info: (msg: string, meta?: unknown) => void;
error: (msg: string, meta?: unknown) => void;
} | null;
start(): void {
if (this.running) return;
this.running = true;
this.runOnce().catch((err) =>
this.logger?.error('Initial inbox prune failed', { error: String(err) }),
);
this.timer = setInterval(() => {
this.runOnce().catch((err) =>
this.logger?.error('Inbox prune failed', { error: String(err) }),
);
}, this.intervalMs);
}
stop(): void {
this.running = false;
if (this.timer) {
clearInterval(this.timer);
this.timer = null;
}
}
async runOnce(): Promise<number> {
const removed = await this.store.purgeExpired(Date.now());
if (removed > 0) {
this.logger?.info('Inbox prune removed expired blobs', { count: removed });
this.events?.emit('inbox.expired_purged', { count: removed });
}
return removed;
}
get isRunning(): boolean {
return this.running;
}
}

90
packages/shade-inbox-server/src/events.ts Normal file
View File

@@ -0,0 +1,90 @@
/**
* Inbox server event emitter.
*
* Mirrors `PrekeyServerEvents`. Emits structural facts only — no plaintext,
* no signatures, no key material. Used by the observer dashboard and
* operator metrics.
*/
export interface InboxServerEventBase {
seq: number;
timestamp: number;
}
export interface InboxServerEventMap {
'inbox.address_registered': { address: string; signingKeyHash: string };
'inbox.address_deleted': { address: string };
'inbox.blob_stored': { address: string; msgId: string; bytes: number; ttlSeconds: number };
'inbox.blob_idempotent_replay': { address: string; msgId: string };
'inbox.blob_fetched': { address: string; count: number; bytes: number };
'inbox.blob_acked': { address: string; msgId: string };
'inbox.expired_purged': { count: number };
'inbox.rate_limited': { route: string; key: string };
'inbox.quota_rejected': { address: string; reason: 'address-quota' | 'sender-quota' | 'body-too-large' };
}
export type InboxServerEventName = keyof InboxServerEventMap;
export type InboxServerEvent = {
[K in InboxServerEventName]: InboxServerEventBase & { name: K; data: InboxServerEventMap[K] };
}[InboxServerEventName];
export type InboxServerEventListener = (event: InboxServerEvent) => void;
export class InboxServerEvents {
private listeners = new Set<InboxServerEventListener>();
private nextSeq = 1;
private buffer: InboxServerEvent[] = [];
private readonly maxBuffer: number;
constructor(options: { bufferSize?: number } = {}) {
this.maxBuffer = options.bufferSize ?? 1000;
}
on(listener: InboxServerEventListener): () => void {
this.listeners.add(listener);
return () => this.listeners.delete(listener);
}
off(listener: InboxServerEventListener): void {
this.listeners.delete(listener);
}
emit<K extends InboxServerEventName>(name: K, data: InboxServerEventMap[K]): void {
const event = {
seq: this.nextSeq++,
timestamp: Date.now(),
name,
data,
} as InboxServerEvent;
this.buffer.push(event);
if (this.buffer.length > this.maxBuffer) this.buffer.shift();
for (const listener of this.listeners) {
try {
listener(event);
} catch (err) {
console.error('[Shade] Inbox event listener threw:', err);
}
}
}
getBufferedSince(since: number): InboxServerEvent[] {
return this.buffer.filter((e) => e.seq > since);
}
getRecent(n: number): InboxServerEvent[] {
return this.buffer.slice(-n);
}
get currentSeq(): number {
return this.nextSeq - 1;
}
}
export async function shortHash(key: Uint8Array): Promise<string> {
const buf = await globalThis.crypto.subtle.digest('SHA-256', key as unknown as ArrayBuffer);
const arr = new Uint8Array(buf).slice(0, 8);
return Array.from(arr, (b) => b.toString(16).padStart(2, '0')).join('');
}

60
packages/shade-inbox-server/src/index.ts Normal file
View File

@@ -0,0 +1,60 @@
import type { Hono } from 'hono';
import type { CryptoProvider } from '@shade/core';
import { createInboxRoutes, type InboxRoutesOptions } from './routes.js';
import { MemoryInboxStore } from './memory-store.js';
import type { InboxStore } from './store.js';
import { InboxServerEvents } from './events.js';
export { createInboxRoutes } from './routes.js';
export type { InboxRoutesOptions } from './routes.js';
export { MemoryInboxStore } from './memory-store.js';
export type { InboxStore } from './store.js';
export {
InboxServerEvents,
shortHash as inboxShortHash,
} from './events.js';
export type {
InboxServerEvent,
InboxServerEventName,
InboxServerEventMap,
InboxServerEventListener,
} from './events.js';
export { InboxPruneTask } from './cleanup.js';
export {
computeMsgId,
isValidMsgId,
constantTimeStringEqual,
} from './msg-id.js';
export {
DEFAULT_INBOX_QUOTA,
clampTtl,
} from './quota.js';
export type { InboxQuotaConfig } from './quota.js';
export { createBridgeRoutes } from './bridge.js';
export type { BridgeRoutesOptions, BridgeKind } from './bridge.js';
/**
* Create a standalone Shade Inbox Server.
*
* const crypto = new SubtleCryptoProvider();
* const inbox = createInboxServer({ crypto });
* export default { port: 3901, fetch: inbox.fetch };
*
* Or compose into an existing Hono app:
* const app = new Hono();
* app.route('/', createInboxServer({ crypto }));
*/
export function createInboxServer(options: {
crypto: CryptoProvider;
store?: InboxStore;
disableRateLimit?: boolean;
events?: InboxServerEvents;
} & Pick<InboxRoutesOptions, 'observability' | 'quota'>): Hono {
const store = options.store ?? new MemoryInboxStore();
const routesOptions: InboxRoutesOptions = {};
if (options.disableRateLimit !== undefined) routesOptions.disableRateLimit = options.disableRateLimit;
if (options.events !== undefined) routesOptions.events = options.events;
if (options.observability !== undefined) routesOptions.observability = options.observability;
if (options.quota !== undefined) routesOptions.quota = options.quota;
return createInboxRoutes(store, options.crypto, routesOptions);
}

105
packages/shade-inbox-server/src/memory-store.ts Normal file
View File

@@ -0,0 +1,105 @@
import type { InboxStore } from './store.js';
interface BlobRow {
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
}
/**
* In-memory InboxStore — used in tests and as the default fallback when
* neither SHADE_INBOX_DB_PATH nor SHADE_INBOX_PG_URL is set.
*
* Blobs are organized in per-address insertion-ordered arrays so cursor
* pagination is just a `>` filter on `receivedAt`.
*/
export class MemoryInboxStore implements InboxStore {
private owners = new Map<string, Uint8Array>();
private blobs = new Map<string, BlobRow[]>();
private nextReceivedAt = 0;
async saveAddressOwner(address: string, signingKey: Uint8Array): Promise<void> {
this.owners.set(address, new Uint8Array(signingKey));
}
async getAddressOwner(address: string): Promise<Uint8Array | null> {
const k = this.owners.get(address);
return k ? new Uint8Array(k) : null;
}
async putBlob(args: {
address: string;
msgId: string;
ciphertext: Uint8Array;
expiresAt: number;
}): Promise<{ created: boolean; receivedAt: number }> {
const list = this.blobs.get(args.address) ?? [];
const existing = list.find((r) => r.msgId === args.msgId);
if (existing) return { created: false, receivedAt: existing.receivedAt };
// Monotonic `receivedAt` so cursor compare is total-order even when
// multiple blobs land in the same millisecond.
const receivedAt = Math.max(this.nextReceivedAt + 1, Date.now());
this.nextReceivedAt = receivedAt;
list.push({
msgId: args.msgId,
ciphertext: new Uint8Array(args.ciphertext),
receivedAt,
expiresAt: args.expiresAt,
});
this.blobs.set(args.address, list);
return { created: true, receivedAt };
}
async fetchBlobs(args: {
address: string;
sinceCursor: number;
now: number;
limit: number;
}): Promise<Array<{ msgId: string; ciphertext: Uint8Array; receivedAt: number; expiresAt: number }>> {
const list = this.blobs.get(args.address) ?? [];
return list
.filter((r) => r.receivedAt > args.sinceCursor && r.expiresAt > args.now)
.sort((a, b) => a.receivedAt - b.receivedAt)
.slice(0, args.limit)
.map((r) => ({
msgId: r.msgId,
ciphertext: new Uint8Array(r.ciphertext),
receivedAt: r.receivedAt,
expiresAt: r.expiresAt,
}));
}
async deleteBlob(address: string, msgId: string): Promise<boolean> {
const list = this.blobs.get(address);
if (!list) return false;
const idx = list.findIndex((r) => r.msgId === msgId);
if (idx === -1) return false;
list.splice(idx, 1);
return true;
}
async countBlobs(address: string, now: number): Promise<number> {
const list = this.blobs.get(address) ?? [];
return list.filter((r) => r.expiresAt > now).length;
}
async purgeExpired(now: number): Promise<number> {
let removed = 0;
for (const [addr, list] of this.blobs) {
const filtered = list.filter((r) => r.expiresAt > now);
removed += list.length - filtered.length;
if (filtered.length === 0) {
this.blobs.delete(addr);
} else {
this.blobs.set(addr, filtered);
}
}
return removed;
}
async deleteAddress(address: string): Promise<void> {
this.owners.delete(address);
this.blobs.delete(address);
}
}

37
packages/shade-inbox-server/src/msg-id.ts Normal file
View File

@@ -0,0 +1,37 @@
/**
* msgId derivation: deterministic SHA-256 of the ciphertext blob.
*
* `msgId = lowercase-hex( sha256(ciphertext) )`
*
* Both client and server compute it independently and check that the
* client's claimed msgId equals the recomputed hash — that gives us
* idempotency on PUT (same ciphertext → same row) and replay-protection
* for free (a tampered re-upload changes the hash and lands in a new
* slot the recipient simply ignores when decrypt fails).
*/
export async function computeMsgId(ciphertext: Uint8Array): Promise<string> {
const buf = await globalThis.crypto.subtle.digest(
'SHA-256',
ciphertext as unknown as ArrayBuffer,
);
const arr = new Uint8Array(buf);
return Array.from(arr, (b) => b.toString(16).padStart(2, '0')).join('');
}
/** Constant-time string equality on the hex form, both lowercased first. */
export function constantTimeStringEqual(a: string, b: string): boolean {
const aa = a.toLowerCase();
const bb = b.toLowerCase();
if (aa.length !== bb.length) return false;
let diff = 0;
for (let i = 0; i < aa.length; i++) {
diff |= aa.charCodeAt(i) ^ bb.charCodeAt(i);
}
return diff === 0;
}
/** Validate hex form: 64 lowercase hex chars (32-byte digest). */
export function isValidMsgId(s: unknown): s is string {
return typeof s === 'string' && /^[0-9a-f]{64}$/.test(s);
}

47
packages/shade-inbox-server/src/quota.ts Normal file
View File

@@ -0,0 +1,47 @@
import { ValidationError } from '@shade/core';
/**
* Inbox quota policy. The relay limits per-address and per-sender storage
* so a single ondsinnet sender can't exhaust capacity.
*/
export interface InboxQuotaConfig {
/** Hard cap on the body size of a single PUT (bytes). Default: 1 MiB. */
maxBlobBytes: number;
/**
* Hard cap on the number of non-expired blobs queued for a single
* recipient address. PUTs past this cap return SHADE_VALIDATION.
* Default: 1000.
*/
maxBlobsPerAddress: number;
/** Default TTL when sender omits ttlSeconds. Default: 7 days. */
defaultTtlSeconds: number;
/** Maximum TTL the relay will accept. Default: 30 days. */
maxTtlSeconds: number;
/** Minimum TTL. Default: 60 seconds. */
minTtlSeconds: number;
/** Maximum number of blobs returned in one GET. Default: 100. */
fetchPageLimit: number;
}
export const DEFAULT_INBOX_QUOTA: InboxQuotaConfig = {
maxBlobBytes: 1 * 1024 * 1024,
maxBlobsPerAddress: 1000,
defaultTtlSeconds: 7 * 24 * 60 * 60,
maxTtlSeconds: 30 * 24 * 60 * 60,
minTtlSeconds: 60,
fetchPageLimit: 100,
};
export function clampTtl(ttl: number | undefined, q: InboxQuotaConfig): number {
const v = ttl ?? q.defaultTtlSeconds;
if (!Number.isFinite(v) || v <= 0) {
throw new ValidationError('ttlSeconds must be positive', 'ttlSeconds');
}
if (v < q.minTtlSeconds) {
throw new ValidationError(`ttlSeconds < min (${q.minTtlSeconds})`, 'ttlSeconds');
}
if (v > q.maxTtlSeconds) {
throw new ValidationError(`ttlSeconds > max (${q.maxTtlSeconds})`, 'ttlSeconds');
}
return Math.floor(v);
}

Some files were not shown because too many files have changed in this diff Show More