33 Commits

Author SHA1 Message Date
037f994572 release(v4.11.0): streaming Double-Ratchet sub-sessions (ShadeStream)
Some checks failed
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
Answers Vyvern FR shade-ws-streaming-ratchet.md with a first-class
streaming-session API rather than the documented-contract fallback.
The Double-Ratchet crypto was already safe for high-frequency
one-directional use; the send/receive wrapper was not (per-frame
saveSession keystore write; shared per-peer mutex + single stored
session row coupling reuse to the HTTP path).

- @shade/core: stream.ts — identity-bound 3-DH seeding (X3DH-minus-
  prekeys, no prekey-server round trip, mutually authenticated against
  the parent session's pinned identities), bootstrapStreamSession
  reusing init{Sender,Receiver}Session verbatim, in-memory-only
  StreamRatchet (own op-mutex, never persisted, zeroized on close).
  beginStream/acceptStream on ShadeSessionManager; Stream{Closed,
  Handshake}Error; stream.opened/closed events.
- @shade/proto: STREAM_OPEN/OPEN_ACK/FRAME wire (0x31/0x32/0x33),
  additive; inspectEnvelopeType extended.
- @shade/sdk: Shade.openStream/acceptStream → ShadeStream
  (handshakeFrame/handleHandshake/seal/open/close), transport-
  agnostic, independent of encrypt/decrypt queues + parent session,
  identical server (sqlite:) and browser (IndexedDB) — touches no
  storage.
- Tests: 5000-frame one-directional burst (bounded skipped keys + FS
  zeroize), parent-session independence, replay/rewind rejection,
  mutual-auth, proto wire round-trips. Full suite green (1159 pass).
- docs/streaming-sessions.md (R1–R7 contract); SECURITY.md matrix rows.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 11:29:09 +02:00
188c3db56a android: V4.9 + V4.10 Kotlin ports + KeystoreStorage adapter
Some checks failed
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Test / test (push) Has been cancelled
Pure-JVM additions to shade-android (no Android SDK needed):
- V4.9 blob primitives: BlobKdf (HKDF deriveBlobSlotId/Key/SigningSeed),
  BlobAead (nonce||ct||tag with shade-profile-aad-v1:<slot> AAD),
  BlobClient (java.net.http with hand-written canonical JSON signing
  matching TS signPayload output), Profile high-level namespace.
- V4.10 approval helpers: CanonicalProfileBlob schema with denormalized
  trustedApproverFingerprints, build/sign/verify proxy approvals via
  length-prefixed u16 BE UTF-8 canonical signing payload.
- Password KDFs: scrypt + argon2id via Bouncy Castle, NFKC-normalized.
- SessionStateJson at-rest serializer for persistence layer.

Cross-platform vectors (test-vectors/blob.json, approval.json) gate
byte-identical output between TS and Kotlin, including a TS-signed
Ed25519 signature the Kotlin port verifies and reproduces (Ed25519 is
deterministic).

New shade-android-keystore sibling Gradle module (Android-specific):
- KeystoreMasterKey: hardware-backed AES-256-GCM with BIOMETRIC_STRONG
  gating, StrongBox-backed when available, invalidated on enrollment.
- BiometricUnlock: coroutine wrapper around BiometricPrompt with
  tagged cancellation/failure exceptions.
- KeystoreStorage: StorageProvider over biometric-gated AES-encrypted
  SharedPreferences with AAD-bound row keys.

All 25 SDK packages typecheck clean; 104 SDK tests + 24 new Kotlin
tests + 11 cross-platform vector tests all green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 17:38:15 +02:00
1bd7037a6d release(v4.10.0): cross-host approval routing primitives in @shade/sdk
Some checks failed
Test / test (push) Has been cancelled
Builds on V4.9's encrypted profile blob: ships the canonical
profile-blob schema (hosts/clients/trustedApproverFingerprints) and
the build/sign/verify trio for proxy-approval frames. Headless servers
can now route a `linkRequest` to a trusted-approver phone, verify the
phone's Ed25519 signature against the fresh profile blob, and complete
pairing without a GUI host being available.

Length-prefixed binary signing payload so any platform (Kotlin, Swift,
Go) can produce byte-identical signing input from test vectors. No
relay or transport changes — entirely SDK-level.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 17:09:59 +02:00
80c410f518 release(v4.9.0): relay-side encrypted blob primitive + SDK Profile namespace
Some checks failed
Test / test (push) Has been cancelled
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Ships the Prism FR (encrypted-profile-storage-v4.9.md) as a generic
relay-side encrypted blob primitive: deterministically-located,
AEAD-sealed blobs keyed by a 32-byte slotId derived client-side via
HKDF from the user's master key. Unlocks credential-only bootstrap
of new devices into existing E2EE state — no QR, no physical access.

Server: BlobStore interface + Memory/Sqlite/Postgres impls,
createBlobRoutes for GET/PUT/DELETE /v1/blob/:slotId with TOFU pubkey
auth and If-Match CAS (409/412 semantics). Mounted on the same Hono
app as the inbox; SHADE_BLOB_PG_URL / SHADE_BLOB_DB_PATH /
SHADE_DISABLE_BLOB env-var plumbing in standalone.

SDK: createProfileNamespace high-level wrapper (HKDF derivation,
random-nonce AEAD seal, slotId-bound AAD) + low-level BlobClient.
Cross-platform test vectors in test-vectors/blob-storage.json.

New errors: ConflictError (409), PreconditionFailedError (412).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 02:44:42 +02:00
3c0db14904 release(v4.8.5): kill flushOnce 15s success-backoff + per-recipient parallel drain
Some checks failed
Test / test (push) Has been cancelled
Prism filed a per-recipient-flush-concurrency FR pointing at
serial-per-flush. Investigation surfaced the actual culprit:
`scheduleFlush` was using a 15 s backoff on **both** the success and
failure paths, so envelopes enqueued *during* an in-flight flush
sat ~15 s behind the next drain — visible as "10 s of silence then
25-frame burst" on the receiving side under sustained sender output.

Two fixes:

1. `scheduleFlush` now uses 0 ms delay when `flushOnce` delivered
   ≥1 envelope and more is queued (network healthy → drain
   remainder immediately). 15 s reserved for the actual failure
   case where every attempt this round failed. `flushOnce` returns
   `{ delivered, remaining } | null` so concurrent-flush early
   returns don't double-schedule.

2. `flushOnce` groups the outgoing queue by `recipientAddress` and
   drains buckets via `Promise.all`. Per-peer order preserved
   (sequential within a bucket); a slow POST to recipient A no
   longer head-of-line-blocks frames bound for B.

`Inbox.tick` public shape unchanged. `OutgoingQueueStore`
implementations see the same per-entry list/remove/bumpAttempts/
size contract; only cross-recipient interleaving changes.

Tests cover (1) 25-envelope burst behind a 100 ms slow PUT drains
within 1 s, and (2) carol's PUT lands within 150 ms even when bob's
PUT stalls 200 ms.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 22:56:27 +02:00
a98ea8a1bd release(v4.8.4): server-side cross-channel dedup via BridgeDeliveryLog
Some checks failed
Test / test (push) Has been cancelled
V4.8.3 shipped client-side cross-channel dedup hook
(`Inbox.acceptBridgeFrame`), but recipients that didn't migrate to
the new wiring still observed the same envelope twice — once via
WS bridge push, again ~30 s later via inbox-poll. Prism re-verified
the FR after 4.8.3 and asked for a relay-side enforcement so app
code doesn't have to ack-via-DELETE on every bridge frame.

V4.8.4 adds an in-memory `BridgeDeliveryLog` (default 60 s grace,
8192-per-address cap) that records every successful WS / SSE /
long-poll push of `(address, msgId)`. The `/v1/inbox/:addr/fetch`
route filters out blobs in the log's grace window so a recipient
running both a bridge and the 30 s poll cadence sees exactly one
delivery. Cursor advances over the full fetched window so a poll
that straddles a suppressed blob doesn't stall.

The standalone server auto-wires the log between
`createBridgeRoutes` and `createInboxRoutes`. Custom mounts thread
the same instance through `bridgeDeliveryLog` on both factories.

Tests cover WS-then-poll, SSE-then-poll, and a negative control
(non-bridge-pushed blob still comes through inbox-fetch).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 16:31:42 +02:00
d47774ef1c release(v4.8.3): cross-channel msgId dedup + Shade.aliasSession
Some checks failed
Test / test (push) Has been cancelled
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Two follow-ups to the V4.8.2 duplicate-fan-out fixes Prism filed.

1. `Inbox.acceptBridgeFrame(blob)` + shared 4096-entry msgId LRU.
   The relay durably stores blobs and pushes them to every active
   delivery channel; without a cross-channel ack the bridge frame
   ran first and the next inbox-poll re-dispatched the same blob
   ~30 s later, tripping on consumed prekeys. Bridge consumers now
   plumb pushed frames through `acceptBridgeFrame`, which shares
   the dedup gate + ack path with `pollOnce`. Whichever channel
   delivers first wins; the other acks-and-skips. Inbox records
   the msgId before the ack so a parallel poll can't observe an
   in-flight ack window.

2. `Shade.aliasSession(oldLabel, newLabel)`. First-contact forces
   the receiver to label the new session by the relay's sender
   fingerprint hint (`fp:<senderfp>`); the post-decrypt plaintext
   typically announces the peer's real address. Aliasing moves
   session, trusted identity, peer-verification, and identity-
   version under the canonical label. Holds the per-peer mutex on
   both labels (lexicographic order) so concurrent crypto ops can't
   observe a half-moved state. Refuses to overwrite an existing
   session at the new label.

Wire change: `IncomingMessage.expiresAt?` now surfaces the relay's
expiry so receivers can pass bridge frames straight to
`acceptBridgeFrame` without inventing a TTL.

Tests cover bridge-then-poll, poll-then-bridge, aliasSession happy
path, refuse-to-overwrite, and same-label no-op.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 15:49:36 +02:00
8c606ad498 release(v4.8.2): per-from receive serialization + per-connection bridge dedup
Some checks failed
Test / test (push) Has been cancelled
Two interlocking robustness fixes for the duplicate-fan-out / first-contact
class of failures Prism reported.

1. `Shade.receive(from, env)` now queues its `manager.decrypt` step
   per `from` so concurrent dispatches can't race the SessionManager
   ratchet or the StorageProvider (sqlite "database is locked", IDB
   transaction conflicts). User message handlers run *outside* the
   queue so streams + file-RPC's nested `shade.receive` calls don't
   self-deadlock.

2. Bridge WS + SSE handlers now run a per-connection bounded msgId
   LRU as defense-in-depth against any flushTo re-entry (event-storm,
   future refactor). Pending-flush chains are wrapped in `.catch(() =>
   {})` so a transient `ws.send` rejection no longer poisons the
   connection's flush loop.

Tests: storming `inbox.blob_stored` 10× per PUT yields exactly one WS/
SSE frame; 8 concurrent `bob.receive('alice', envelope)` calls keep
the ratchet intact and never surface "database is locked".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 12:13:46 +02:00
680d6386f3 release(v4.8.1): SHADE_DISABLE_RATE_LIMIT env var for single-tenant deploys
Plumbing fix only — both createPrekeyRoutes and createInboxRoutes
already accepted disableRateLimit; standalone.ts just didn't read
the env. Now SHADE_DISABLE_RATE_LIMIT=1 turns off IP rate-limits on
every prekey + inbox route, with a WARN log on startup so operators
see it.

Single-tenant deployments only — multi-tenant relays must leave it
unset. Documented in docs/DEPLOYMENT.md.

Reported by Prism: ~6 pair attempts/hour from a single dev IP +
the sidecar's register call tripped the 5/hour REGISTER_LIMIT every
dev iteration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 00:55:57 +02:00
1fb59a7076 release(v4.8.0): sender-fingerprint attribution + Inbox.start race fix
Two unblocking changes for first-contact flows.

Sender attribution: relay captures shortHash(senderSigningKey) at
PUT time (after signature verification, no new trust surface) and
surfaces it on bridge push (IncomingMessage.from) + inbox-fetch
(FetchedBlob.from) + DecryptHandler raw arg. Apps receiving a prekey
envelope from a never-before-seen peer can now bootstrap X3DH via
shade.receive('fp:<hex>', env) — pre-4.8 the wire envelope didn't
authenticate the sender and there was no out-of-band hint to use.
Idempotent ALTER TABLE migrations for SQLite + Postgres add a
sender_fp TEXT column; legacy rows surface as from=undefined
(inter-version compat).

Inbox.start() race: pre-4.8 start() called register() fire-and-forget
AND schedulePoll(0) synchronously, so the first poll on a fresh
address often beat the register HTTP RTT and got SHADE_NOT_FOUND.
start() now defers; register() success kicks schedulePoll(0). Manual
tick() is unaffected (deliberate user action, no gating).

Both reported by Prism. Tests cover all five acceptance criteria
from the sender-attribution request (PUT capture, bridge surface,
fetch surface, inter-version compat, end-to-end pair smoke) plus
the three from the race-fix request.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 00:11:59 +02:00
594992a183 release(v4.7.0): peer-presence events for instant BroadcastChannel revoke
Adds the bridge-connection-lifecycle signal that closes Prism's
~45s revoke window down to one server→client round-trip (~50ms).

Server (`@shade/inbox-server`):
- `inbox.peer_connected` / `inbox.peer_disconnected` events on the
  0↔1 boundary across WS + SSE bridges. Long-poll deliberately not
  tracked (every poll boundary would flap; push transports are also
  the only ones where instant revoke matters).
- `PresenceTracker` collapses two parallel bridges (e.g. WS + SSE
  during fallback handover) into one connect/disconnect pair.
- `GET /v1/bridge/presence` SSE endpoint: signed query with
  `kind: 'presence'`, `watched: string[]`; on open streams a
  per-address snapshot, then change frames filtered server-side.
  MAX_WATCHED_ADDRESSES = 64. Subscribing does not itself count as
  a peer-bridge connection.
- `createBridgeRoutes` now returns `{ app, websocket, presence }`.

Client (`@shade/transport-bridge`):
- `PresenceBridge.subscribe({ watch, onPresenceChange })` →
  `{ addPeer, removePeer, watching, unsubscribe }`. addPeer/removePeer
  mutate via reconnect with a fresh signed query.
- `signPresenceQuery` helper for non-PresenceBridge consumers.

Tests cover all four acceptance criteria from the Prism request:
server-event smoke, online→offline subscription, address scoping
(carol invisible to a [alice]-only sub), reconnect, plus an
addPeer/removePeer regression.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 23:16:35 +02:00
8746571d2a release(v4.6.1): bind globalThis.fetch in browser-receiver-sensitive call sites
Browsers' Window.fetch is a WebIDL bound operation; storing it as
this.fetchImpl / this.fetchFn and calling via the instance receiver
threw "Illegal invocation" on the first request. Bind once at
construction in InboxClient, LongPollBridge, and SseBridge. Reported
by Prism (multi-device E2EE terminal), blocking every browser
consumer of the v4.6 transport stack on inbox.start() / bridge.connect().

WsBridge unaffected (uses WebSocket). Node/Bun fetch tolerates a free
receiver, so the bug never surfaced server-side — added regression
tests that install a strict-receiver globalThis.fetch to catch the
issue without an actual browser harness.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 23:00:58 +02:00
2c400d7094 release(v4.6.0): broadcast channels — Signal sender-keys for one-to-many fan-out
Some checks failed
Test / test (push) Has been cancelled
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
Lands the broadcast-channel primitive Prism asked for in
Docs/shade-feature-request-sender-keys.md. The crypto in
@shade/core/sender-keys.ts was already in place; this release wires
it up as a first-class app-facing API, adds the persistence schema
across all six storage backends (memory, sqlite, indexeddb +
encrypted variants), introduces wire type 0x21 in @shade/proto,
and ships Prism's three acceptance tests verbatim.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 15:55:34 +02:00
2b1b4d6630 release(v4.5.0): browser-side encrypted storage + multi-factor unlock
Adds the foundations Prism's web client (and any future browser-based
Shade app) needs: at-rest-encrypted IndexedDB storage that mirrors the
SQLite backend byte-for-byte at the AAD/nonce level, browser-safe
subpath imports so Vite/webpack/esbuild stop hitting bun:sqlite, and
KeyManager support for argon2id and N-factor composite unlock.

@shade/storage-encrypted
- EncryptedIndexedDBStorage (subpath: /idb) — full StorageProvider
  using one object store per _enc table; reuses aeadSeal/aeadOpen +
  row-codec sealers so a row sealed under the SQLite or Postgres
  backend decrypts under IDB given the same KeyManager.
  bumpPeerIdentityVersion is atomic under one IDB transaction.
- KeyManager argon2id source — memory-hard KDF for low-entropy
  secrets (PINs). Backed by @noble/hashes/argon2 (already a transitive
  dep). DEFAULT_ARGON2ID exported (m=64 MiB, t=3, p=1).
- KeyManager composite source — HKDF-combine N sub-sources into one
  master. Every source mandatory; order significant by design;
  composite-of-composite rejected; optional info string for app-level
  domain separation.
- Subpath exports (/crypto, /sqlite, /postgres, /idb) plus a `browser`
  condition on the default import that resolves to a barrel
  excluding the Bun- and Postgres-specific entries. Browser bundles
  no longer pull bun:sqlite transitively.

Tests
- 73 tests in shade-storage-encrypted (was 31). New coverage:
  argon2id determinism + reject paths, composite same-factors → same
  master, wrong-PIN/passphrase/order-swap → different master, info
  domain separation, all 28 StorageProvider methods on
  EncryptedIndexedDBStorage, fingerprint-mismatch rejection, and
  cross-impl roundtrip with EncryptedSQLiteStorage proving the AAD/
  nonce derivation is implementation-agnostic.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 10:58:49 +02:00
dbb3a090d8 release(v4.4.0): public accessor for device identity public key
Expose the local device's 32-byte Ed25519 identity public key on Shade
so apps can hand it to their own backend at enrollment time for
signature verification, key pinning or per-device safety-number
computation. Closes the gap that forced consumers to ship placeholder
random bytes their backend could store but never verify against.

- @shade/sdk Shade.identityPublicKey: Promise<Uint8Array> — getter
  mirrors the existing fingerprint accessor. Throws pre-init,
  reflects the current key after rotate(), retired key preserved in
  retired-identities storage per existing grace-period contract.
  Private key remains unreachable.
- Test in shade-sdk/tests/sdk.test.ts: round-trip match against the
  underlying storage's signingPublicKey, plus value updates after
  rotate().
- Lockstep version bump 4.3.0 → 4.4.0 across all 25 packages.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 17:58:45 +02:00
f5f42fe557 release(v4.3.0): browser persistence via @shade/storage-indexeddb
Ship an official IndexedDB-backed StorageProvider so browser-based Shade
consumers persist identity, prekeys, sessions, retired identities,
peer-verification state and stream-resume rows across tab refresh and
browser restart. Closes the gap that forced browser apps onto
storage:"memory" (regenerated identity each load, orphaned device
records server-side).

- New package @shade/storage-indexeddb (4.3.0): full StorageProvider
  conformance, schema v1, idb-backed; bumpPeerIdentityVersion is wrapped
  in a single readwrite IDB transaction (atomic, vs SQLite's
  read-then-upsert race).
- @shade/sdk resolveStorage() accepts { type: 'indexeddb', dbName? } via
  dynamic import (lazy, optional dep — same pattern as
  @shade/storage-postgres). Named StorageSpec type now reused by
  ResolvedConfig.
- Tests: 16 new tests in shade-storage-indexeddb (StorageProvider
  surface + peer-verifications + full E2EE conversation surviving a
  simulated tab reload). Run on fake-indexeddb.
- Lockstep version bump 4.2.1 → 4.3.0 across all 25 packages.
- Publish scripts updated to include the new package.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 17:35:02 +02:00
b77b7e771c release(v4.2.1): fix concurrent-ratchet desync via OutboundQueue waiter cursor
Some checks failed
Publish / publish (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Pull-mode httpClient + drainer + parallel RPCs against the same peer
deteriorated after ~10s with `DecryptionError`. Two bugs combined:

- `OutboundQueue.enqueue` woke `drain` waiters with a `since=0`
  snapshot, replaying already-processed events into
  `Shade.acceptTransferEnvelope` → `manager.decrypt` twice. The
  duplicate consumed an already-used skipped key and corrupted the
  Double Ratchet receive chain.

- `ratchetDecrypt` then propagated the corruption: a same-DH
  message behind the chain with no cached skipped key fell through
  to `kdfChainKey` on the ahead state and rewound `chain.counter`,
  permanently desyncing the chain.

Fix `OutboundQueue` to honor each waiter's `since`, and harden
`ratchetDecrypt` so any future duplicate fails cleanly without
mutating state. Adds regression coverage at all three layers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 22:58:26 +02:00
7520b11b25 release(v4.2.0): pull-mode streams for browser @shade/files
Some checks failed
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
4.1.0's HTTP RPC for browsers capped at inline payloads (≤ 256 KiB).
4.2.0 unlocks streams: server queues outbound chunks + control
envelopes per peer, browser long-polls the queue. Browser-to-server
writes ride the existing /v1/transfer/<id>/chunk POST routes
unchanged.

For Dispatch this unlocks mod-jar uploads (50 MB) and world-backup
downloads (100+ MB) — the actual reason browser-side @shade/files
matters.

### New API

@shade/sdk:
- shade.transferQueueRoute(opts?) — Hono app with /queue +
  /v1/transfer/* routes. Auto-configures the queue transport.
- shade.configureTransfers extended: transport + envelopeTransport
  override slots; resolveBaseUrl optional when both supplied.

@shade/transfer:
- OutboundQueue — per-peer monotonic event log with long-poll
  semantics, idle-eviction GC, ring-buffered to maxEventsPerPeer.
- QueueTransferTransport — enqueues instead of POSTing.

@shade/files:
- httpClient({ outboundQueueUrl, transferBaseUrl }) — when set,
  starts a long-poll drainer + builds a streams-bridge. fs.read /
  fs.write of >256 KiB work end-to-end.
- startQueueDrainer(shade, opts) — exported helper for advanced
  consumers driving their own drainer.

### Implementation notes

- ClientStreamsBridge's TransformStream had HWM=0 by default which
  stalled the drainer's await chain at chunk 4 (writer.write pended
  before the consumer's reader was attached). Bumped to HWM=64 so
  the receive loop can buffer ahead of the consumer.

### Tests

3 new integration tests in tests/integration/http-rpc-streams.test.ts:
4 MiB streamed read round-trip, inline-only error path, idle-timeout
long-poll behaviour.

Wire-compatible. Source-compatible. Lockstep bump to 4.2.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 23:27:06 +02:00
da93b97cce release(v4.1.0): browser-friendly HTTP RPC for @shade/files
Some checks failed
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
Default shade.files.client(peer) requires both peers to be mutually
addressable over HTTP — the response round-trips through
Shade.deliverControlEnvelope (POST to peer's /v1/transfer/control).
Browser tabs can't host an HTTP server, so they couldn't consume
@shade/files at all. Dispatch's filutforsker (admin-panel browser UI)
is the canonical use-case.

This release adds a parallel request-response transport: one POST per
RPC, encrypted envelope in the body, encrypted response in the same
HTTP response. No inbound channel needed on the client.

### New API

- shade.files.rpcRoute(opts?) — Hono app exposing POST /rpc.
- shade.files.httpClient(peer, opts) — request-response FileClient.
- FilesNamespace.serve(handler, { inlineOnly: true }) — skip streams-
  bridge (and its configureTransfers pre-condition); also skip
  channel-based dispatch so requests aren't double-dispatched.

### Limitations (v1)

Inline only (≤ 256 KiB). Streamed reads/writes throw clear errors
directing to shade.files.client(peer) on a server-to-server deploy.

### Tests

7 integration tests in tests/integration/http-rpc.test.ts covering
round-trip + negative cases (sender header, empty/garbage body,
maxBodyBytes, rpcRoute-without-serve).

### Symmetry

Mirrors @shade/server's shade-auth-middleware: encrypted envelope in
request body, decrypted via existing ratchet, response in same HTTP
roundtrip. No WebSocket, no SSE, no outbound from server.

Wire-compatible. Source-compatible. Lockstep bump to 4.1.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 22:08:14 +02:00
0bdf9e859c release(v4.0.2): consumer-strict reader-shape fixes
Some checks failed
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
4.0.1's typecheck gate compiled each package internally against
lib: ["ES2022"]. That doesn't catch types that only fail when
*consumer* code (lib: ["DOM"] + exactOptionalPropertyTypes) tries to
assign a native browser type into one of our locally-defined narrower
types. Dispatch hit one such case in @shade/files inline-threshold.ts.

This release adds a tests/consumer-strict/ smoke project to the
pre-publish gate. It compiles a tiny "as if I were a downstream app"
TS file against:

  lib: ["ES2022", "DOM", "DOM.Iterable"]
  types: ["bun-types"]
  exactOptionalPropertyTypes: true
  strict: true
  paths → packages/*/src/index.ts

scripts/typecheck-all.ts now runs the smoke after per-package checks.
Both must pass before publish:dry / publish:all proceeds.

### Fixed
- @shade/files inline-threshold.ts: MinimalReader<T> rewritten as the
  explicit disjoint union { done:false, value:T } | { done:true,
  value?: T | undefined } that's assignable from every native reader
  shape (bun, DOM, node:stream/web). Fixes the
  "ReadableStreamReadResult is not assignable" Dispatch reported.
- @shade/files streams-bridge (client + server): stash setTimeout
  return in a local before .unref?.() via { unref?: () => void } cast.
  Fluent .unref?.() failed under lib: ["DOM"] (setTimeout returns
  number there).
- @shade/sdk background.ts: same setInterval .unref?.() fix.

Wire-compatible. No API shape changed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 19:51:46 +02:00
70e319fef8 release(v4.0.1): strict-TS publishability fixes
Some checks failed
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
4.0.0 shipped TypeScript source as published main/types, but several
files only compiled inside the monorepo. Consumer projects (Dispatch,
etc.) running their own strict tsc against our published source hit:

- @shade/key-transparency: 4 noUnusedLocals violations
  (IndexAbsenceProof, IndexInclusionProof, IndexProofWire, nodeHash)
- @shade/sdk: KT verifier callbacks returned Promise<unknown> instead
  of Promise<STHWire> / Promise<{ proof: string[] }>
- @shade/sdk: thumbnail.ts globalThis cast collided with consumer's
  lib.dom-supplied createImageBitmap signature
- @shade/files: cycle with @shade/sdk produced "this is not assignable
  to type 'Shade'" because hoisted node_modules layouts duplicated the
  Shade class. Broken by replacing `import type { Shade }` with a
  local structural ShadeBridge interface.
- @shade/storage-encrypted: KeyUsage (lib.dom) used under
  lib: ["ES2022"]
- @shade/transport-bridge: ReadableStreamDefaultReader<any> ↔
  <Uint8Array> mismatch
- @shade/keychain / @shade/dashboard / @shade/storage-encrypted
  tsconfig rootDir / include hygiene

Tooling: scripts/typecheck-all.ts runs `bunx tsc --noEmit` against
every workspace package's tsconfig and fails on any error. Wired into
publish:dry / publish:all and publish-shade.sh as a hard gate so this
class of bug cannot recur.

All 24 packages bumped to 4.0.1 in lockstep.

Migration: <ShadeFilesProvider> now requires an explicit `files` prop
(pass `shade.files`). Wire format unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 19:36:47 +02:00
f301b391a5 docs(archive): close out Status fields on V2.x backlog + V3.12 design notat
Some checks failed
Test / test (push) Has been cancelled
V4.0 acceptance §"All docs/V*.md arkivert med DONE-status" requires
every archived plan to carry an explicit Status field. V2.1 / V2.2 /
V2.3 inherited their pre-status format; V3.12-DESIGN was still
"Approved". Mark all four as Done with a one-line pointer to where
the work actually landed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 19:04:47 +02:00
40766c60f4 docs(readme): full 4.0 GA refresh — status table, all V3.x surfaces, audit pointers
Some checks failed
Test / test (push) Has been cancelled
- Status table (V3.2 → V3.12 all green; audit + soak harness ready)
- "What you get" expanded: storage encryption, fingerprint gates,
  inbox, bridge, web workers, recovery, KT, observability, soak
- Quick start: opt-in surface examples (gates, workers, WebRTC, KT)
  + at-rest encryption snippet + migrate-storage CLI invocation
- Architecture diagram refreshed: container now exposes /v1/inbox/*,
  /v1/bridge/*, /v1/kt/*, ops endpoints; WebRTC P2P pipe documented
- Packages table: all 24 entries at 4.0.0 (added observability,
  keychain, storage-encrypted, transport-bridge, etc.)
- Publishing section: 4.0.x examples; soak harness section added
- Security properties: V3.2 / V3.3 / V3.6 / V3.8 / V3.10 / V3.11
  rows added alongside the existing forward-secrecy / PCS list
- Documentation: grouped into Operator+integrator, Per-surface,
  Threat-model+audit, Migration+history, Examples
- Container includes: full V4.0 surface (inbox, bridge, KT, ops
  probes); image pinned at gt.zyon.no/stian/shade-prekey:4.0.0

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:58:38 +02:00
de25b19033 fix(publish): break recursion in publish-shade.sh → publish-all.ts
Some checks failed
Test / test (push) Has been cancelled
publish-shade.sh used to call `bun run publish:all`, which in turn was
wired to call publish-shade.sh (after the V4.0 cleanup). Point it
directly at scripts/publish-all.ts so the interactive flow runs the
TS publisher without re-entering itself.

Verified: dry-run from publish-shade.sh now packs all 24 @shade/*@4.0.0
packages cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:53:16 +02:00
e6fdf31b49 release(v4.0.0): Shade GA — V3.x consolidation + audit prep
Some checks failed
Test / test (push) Has been cancelled
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.

Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
  Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
  bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
  shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
  non-realtime stack.

Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
  version 4.0.0 on /health.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
8b055912b7 docs(readme): reflect @shade/files + @shade/streams + @shade/transfer + 0.3.0 wire bump
Some checks failed
Test / test (push) Has been cancelled
- Top-of-file callout for the 0.2.x → 0.3.0 wire incompatibility.
- "What you get" picks up E2EE file transfers + filesystem RPC bullets.
- Quick start gains a `Shade.files.serve()` / `.client()` snippet.
- Packages table adds @shade/streams, @shade/transfer, @shade/files;
  @shade/sdk and @shade/widgets descriptions extended for the new APIs.
- Documentation index links docs/files.md, docs/streams.md, examples/07
  and 08.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:11:42 +02:00
ebe3a50389 chore(publish): track publish-shade.sh + add streams/transfer/files to PACKAGES
Some checks failed
Test / test (push) Has been cancelled
publish-shade.sh's PACKAGES list was missing shade-streams, shade-transfer,
and shade-files — so the conflict-check + auto-bump loop silently skipped
those three packages. The final `bun run publish:all` step still picked
them up (via scripts/publish-all.ts), but a re-publish on a version that
already existed in the registry would not have bumped the missing
packages, leaving a workspace-wide version mismatch.

Also un-gitignore both publish scripts. publish-all.ts was already tracked
(git overrides ignore for tracked files), and publish-shade.sh contains
no secrets at rest — GITEA_TOKEN is read interactively and stored in
~/.bashrc by the script itself.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:03:57 +02:00
fa770d3063 feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
Some checks failed
Test / test (push) Has been cancelled
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).

@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
  Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
  + per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
  (> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
  / shadeFilesReadStreamId correlation. Server-side TransformStream
  bridges accept inbound transfers immediately (engine rejects chunks
  that arrive before accept) and park the readable for the matching
  RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
  uploadDirectory()/downloadDirectory() with bounded concurrency pool
  (default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
  byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
  policy (traversal + percent-decode hardening), fingerprint gate
  (required/optional/reject), pluggable Ed25519 sig verification with
  ±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
  useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
  entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.

Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
  u16 to u32. The previous u16 silently truncated payloads above
  64 KiB — a hard correctness ceiling that blocked inline file ops
  up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
  only. Cross-platform Kotlin port (android/shade-android) updated to
  match; test-vectors/wire-format.json regenerated.

Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
  Concurrent decryptions of the same peer raced ratchet state
  (manifested as sporadic "Failed to decrypt — wrong key or tampered
  data" under load — surfaced once concurrent uploadDirectory pumped
  many writes in flight). Encrypt was already serialized via
  Shade.send's encryptChains; decrypt is now serialized at the
  manager layer too.

@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
  application-level key/value pairs that round-trip verbatim through
  stream-init plaintext. Used by @shade/files for write/read
  correlation; available to any consumer.

@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
  BackgroundTasks.setHook(name, fn) for runtime hook registration.

Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
  shade-sdk streams-bridge + shade-widgets transfer hooks were
  uncommitted prior to this session. Including them keeps the
  workspace consistent at 0.3.0 since @shade/files depends on them.

Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
  3× stable). Coverage spans unit (inline-threshold + concurrency),
  integration (read-write inline + streams up to 1 MiB, walk +
  upload/download directory, custom-op, metrics, SDK namespace
  end-to-end), and security (tampered-envelope sig verification,
  replay window, fingerprint gate, rate-limit + quota).

Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
  topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
  server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
7e0f7320a9 feat(container): M-Box 1-8 — stack-agnostic standalone Docker container
Some checks failed
Test / test (push) Has been cancelled
Shade now ships as a self-contained Docker image. Deploy one container
per project, any stack (Bun, Python, Go, Rust, Kotlin) can talk to it via
plain HTTP. Zero coupling to consumer codebases.

M-Box 1: Stale identity cleanup API
- touchIdentity + purgeStaleIdentities on PrekeyStore interface
- Implemented for Memory, SQLite, and Postgres backends
- SQLite adds last_activity_at column with migration ALTER for existing DBs
- Postgres adds the same via raw SQL with IF NOT EXISTS guards
- Routes call touchIdentity on register, bundle fetch, replenish
- 4 new tests for the cleanup API

M-Box 2: Stale cleanup background task
- StaleCleanupTask runs purge on startup + every 24h (configurable)
- Reads SHADE_STALE_DAYS (default 30) and SHADE_CLEANUP_INTERVAL_HOURS
- Wired into standalone.ts, stopped on graceful shutdown
- 5 new tests for the task

M-Box 3: Observer baked into the container
- standalone.ts conditionally mounts @shade/observer at /shade-observer
  when SHADE_OBSERVER_TOKEN is set (and >= 16 chars)
- Shared PrekeyServerEvents emitter feeds both routes and observer
- @shade/observer added as optional dependency of @shade/server

M-Box 4: Dockerfile with dashboard build
- Multi-stage build: oven/bun:1 builder → oven/bun:1-alpine runtime
- COPY packages/ wholesale so workspace lockfile resolves cleanly
- RUN bun run build inside shade-dashboard → dist/ → observer/dist/
- Non-root shade user, /data volume, healthcheck, env defaults
- Final image: 260 MB

M-Box 5: OpenAPI spec for stack-agnostic clients
- packages/shade-server/openapi.yaml documents all 9 endpoints with
  request/response schemas, security (Ed25519 signatures + bearer token)
- createOpenApiRoutes serves /openapi.yaml and /docs (Redoc viewer)
- Any language can generate a client with openapi-generator

M-Box 6: Docker CI pipeline
- .gitea/workflows/docker.yml builds + pushes on git tag v*
- scripts/build-docker.ts for local builds, supports --push with GITEA_TOKEN
- Root package.json: build:docker, publish:docker scripts

M-Box 7: Deployment documentation
- packages/shade-server/README rewritten: 5-line quickstart with the image
- docs/DEPLOYMENT.md: full reference, env vars, backup, Dokploy, PG setup
- examples/05-dokploy-deployment/docker-compose.yml updated to pull
  published image (gt.zyon.no/stian/shade-prekey:latest)
- Root README deployment section rewritten

M-Box 8: End-to-end verification
- Image builds locally (bun run build:docker)
- /health, /openapi.yaml, /docs, /metrics, /shade-observer all respond
- 401 without observer token, 200 with
- Real SDK client round-trip: Alice → container → Bob → reply → Alice
- Persistence: identity + prekeys survive container restart (count 20→18
  as expected from two bundle fetches)

285 tests passing, 0 failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 14:29:00 +02:00
467dd5b065 feat(advanced): M-Adv 1-3 — multi-device, backup/restore, group messaging
Some checks failed
Test / test (push) Has been cancelled
Phase D complete. Shade is now at parity with Signal libsignal's core
feature set.

M-Adv 1: Multi-device support (simplified Sesame)
- DeviceListManager tracks per-user device lists ("user:deviceId" addresses)
- fanOutEncrypt() sends one message to all known devices via independent
  1:1 Double Ratchet sessions
- observeIncoming() auto-registers new devices from received messages
- JSON serialization for persistence
- userOfDevice/deviceIdOf address parsers

M-Adv 2: Backup and restore
- @shade/sdk exports BackupBlob format: version + salt + nonce + ciphertext
- Passphrase-derived key via HKDF (note: upgrade path to Argon2id documented)
- exportBackup()/importBackup() handle identity, prekeys, sessions, trust
- backupToString/backupFromString for single-string transport (copy/paste, QR)
- shade.exportBackup()/importBackup() convenience methods on SDK
- CLI: shade backup export <file> / shade backup restore <file>
- Rebuilds manager + transport after restore so ratchet state is consistent

M-Adv 3: Group messaging (Sender Keys)
- Per-sender chain key + Ed25519 signing key per group
- createSenderKey / buildDistribution / installDistribution for key distribution
- senderKeyEncrypt advances chain and signs ciphertext+header
- senderKeyDecrypt verifies signature then advances the sender's chain
- Out-of-order handling with bounded skip
- O(1) per message (once distributions are installed)
- Defensive ByteArray copies in distribution to prevent zeroize-across-refs

276 tests passing, 0 failures. All 13 SDK/tooling/platform/advanced
milestones complete. Shade is feature-complete for v2.0.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:51:34 +02:00
4bf9307548 feat(android): M-Cross 1-3 — Kotlin module + cross-platform test vectors
Some checks failed
Test / test (push) Has been cancelled
Phase C complete: Shade now has a Kotlin implementation with byte-for-byte
compatibility to the TypeScript core, verified by shared test vectors.

M-Cross 1: shade-android Kotlin module
- build.gradle.kts with Tink, EncryptedSharedPreferences, kotlinx.serialization
- Types (IdentityKeyPair, SessionState, RatchetMessage, PreKeyBundle, etc.)
- CryptoProvider interface
- TinkProvider implementation (X25519, Ed25519, AES-GCM, HKDF, HMAC)
- KDF chain functions (kdfRootKey, kdfChainKey, deriveInitialRootKey)
  with the same info strings and salts as @shade/core
- Fingerprint (safety number) computation matching TS exactly
- X3DH protocol: identity gen, signed prekey gen, OTPK gen, bundle processing
- Double Ratchet: initSenderSession, initReceiverSession, ratchetEncrypt,
  ratchetDecrypt, DH ratchet step, skipped key cache
- Wire format matching @shade/proto byte-for-byte
- StorageProvider interface + MemoryStorage impl
- High-level ShadeSessionManager mirroring @shade/core's API

M-Cross 2: Cross-platform test vectors
- scripts/generate-vectors.ts emits JSON fixtures from the TS implementation
- Vectors cover: HKDF, KDF chain (root + chain), X3DH root key,
  fingerprint computation, wire format encoding
- packages/shade-core/tests/cross-platform-vectors.test.ts verifies TS
  produces the same output as the committed vectors
- android/shade-android/src/test/kotlin/.../CrossPlatformVectorTest.kt
  loads the SAME JSON and verifies Kotlin produces identical bytes

M-Cross 3: Nova Android migration plan
- android/shade-android/MIGRATION-NOVA.md — concrete steps to replace
  Nova's static PushKeyStore AES with Shade sessions
- Phase 1 (dual-write) / Phase 2 (switch reads) / Phase 3 (deprecate)
- Smoke test recipe for end-to-end TS → Kotlin push flow

251 tests passing on the TS side. Kotlin tests run via Gradle when
the Android SDK is available; the vectors guarantee they'll pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:45:38 +02:00
518dc68c4f feat(cli): M-Tool 1-3 — CLI, templates, Gitea publishing pipeline
Some checks failed
Test / test (push) Has been cancelled
Phase B complete: Shade now has a full developer tooling story.

@shade/cli
- shade init with project scaffolding from templates
- shade fingerprint (own or peer)
- shade publish (re-upload bundle)
- shade rotate (--identity for full rotation, otherwise signed prekey)
- shade peer add/list/verify/remove
- shade dashboard (opens observer in browser)
- shade doctor (diagnose config, storage, prekey server reachability)
- Config from .shaderc.json or SHADE_* env vars

Templates (in packages/shade-cli/templates/)
- bun-server — Bun + Hono backend with /send + /receive endpoints
- chat-demo — Two-process Alice/Bob chat over HTTP

Publishing pipeline (Gitea npm registry)
- .gitea/workflows/test.yml — CI on push/PR with PostgreSQL service
- .gitea/workflows/publish.yml — publish on git tag v*
- scripts/publish-all.ts — local publish helper with DRY_RUN support
- scripts/bump-version.ts — lockstep version bump across all packages
- Root package.json scripts: version, publish:dry, publish:all

Also: /health endpoint now lives in createPrekeyRoutes so doctor can
probe it without needing the full standalone setup.

Dry-run verified: all 11 packages pack cleanly.
246 tests passing, 0 failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:38:00 +02:00
c95824f95f feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.

M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
  (memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
  on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
  signed operations work after rotation
- onMessage() handler API for incoming plaintext

API:
  const shade = await createShade({ prekeyServer, storage });
  await shade.send('bob', 'hello');
  await shade.receive('alice', envelope);
  shade.onMessage((from, msg) => ...);
  await shade.rotate();
  await shade.shutdown();

13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.

233 tests passing, 0 failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
567 changed files with 84036 additions and 314 deletions

View File

@@ -0,0 +1,90 @@
name: Cross-platform vectors
# V3.5 §CI parity gate. Both runners load test-vectors/*.json and verify their
# native implementation produces byte-identical output to the recorded vectors.
# Any divergence — KDF labels, AAD encoding, wire format — fails CI immediately
# so cross-platform messaging breakage cannot land on main.
on:
push:
branches: [main]
paths:
- 'test-vectors/**'
- 'packages/shade-core/tests/cross-platform-vectors.test.ts'
- 'packages/shade-core/src/**'
- 'packages/shade-crypto-web/src/**'
- 'packages/shade-proto/src/**'
- 'android/**'
- 'scripts/generate-vectors.ts'
- '.gitea/workflows/cross-vectors.yml'
pull_request:
branches: [main]
paths:
- 'test-vectors/**'
- 'packages/shade-core/tests/cross-platform-vectors.test.ts'
- 'packages/shade-core/src/**'
- 'packages/shade-crypto-web/src/**'
- 'packages/shade-proto/src/**'
- 'android/**'
- 'scripts/generate-vectors.ts'
- '.gitea/workflows/cross-vectors.yml'
jobs:
ts-vectors:
name: TypeScript vectors (bun)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Bun
run: curl -fsSL https://bun.sh/install | bash
- name: Install dependencies
run: ~/.bun/bin/bun install --frozen-lockfile
- name: Run TS vector tests
run: ~/.bun/bin/bun run test:vectors
- name: Verify vectors are up-to-date
# Regenerate vectors and fail if they would change. Forces vector
# commits to come from `bun run vectors:gen`, never hand-edited.
run: |
~/.bun/bin/bun run vectors:gen
if ! git diff --quiet test-vectors/; then
echo "::error::test-vectors/ is out of date. Run 'bun run vectors:gen' and commit."
git diff test-vectors/
exit 1
fi
kotlin-vectors:
name: Kotlin vectors (gradle)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up JDK 17
uses: actions/setup-java@v4
with:
distribution: temurin
java-version: '17'
- name: Cache Gradle
uses: actions/cache@v4
with:
path: |
~/.gradle/caches
~/.gradle/wrapper
key: gradle-${{ hashFiles('android/**/*.gradle.kts', 'android/gradle/wrapper/gradle-wrapper.properties') }}
restore-keys: gradle-
- name: Run Kotlin vector tests
working-directory: android
run: ./gradlew :shade-android:test --no-daemon --info
- name: Upload Gradle test report
if: always()
uses: actions/upload-artifact@v4
with:
name: kotlin-test-report
path: android/shade-android/build/reports/tests/test/
if-no-files-found: ignore

View File

@@ -0,0 +1,52 @@
name: Docker build and publish
on:
push:
tags:
- 'v*'
workflow_dispatch:
jobs:
docker:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Bun
run: curl -fsSL https://bun.sh/install | bash
- name: Install dependencies
run: ~/.bun/bin/bun install --frozen-lockfile
- name: Run tests (gate)
run: ~/.bun/bin/bun test --recursive
- name: Extract version from tag
id: version
run: |
VERSION="${GITHUB_REF_NAME#v}"
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to Gitea container registry
uses: docker/login-action@v3
with:
registry: gt.zyon.no
username: Stian
password: ${{ secrets.GITEA_PUBLISH_TOKEN }}
- name: Build and push
uses: docker/build-push-action@v5
with:
context: .
file: packages/shade-server/Dockerfile
push: true
tags: |
gt.zyon.no/stian/shade-prekey:${{ steps.version.outputs.version }}
gt.zyon.no/stian/shade-prekey:latest
labels: |
org.opencontainers.image.version=${{ steps.version.outputs.version }}
org.opencontainers.image.source=https://gt.zyon.no/Stian/Shade
org.opencontainers.image.revision=${{ github.sha }}

View File

@@ -0,0 +1,32 @@
name: Publish
on:
push:
tags:
- 'v*'
jobs:
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Bun
run: curl -fsSL https://bun.sh/install | bash
- name: Install dependencies
run: ~/.bun/bin/bun install --frozen-lockfile
- name: Run tests
run: ~/.bun/bin/bun test --recursive
- name: Build dashboard
run: |
cd packages/shade-dashboard
~/.bun/bin/bun run build
- name: Publish all packages to Gitea registry
env:
GITEA_TOKEN: ${{ secrets.GITEA_PUBLISH_TOKEN }}
GITEA_USER: Stian
run: ~/.bun/bin/bun run scripts/publish-all.ts

39
.gitea/workflows/test.yml Normal file
View File

@@ -0,0 +1,39 @@
name: Test
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_PASSWORD: test
POSTGRES_DB: postgres
ports:
- 5432:5432
steps:
- uses: actions/checkout@v4
- name: Install Bun
run: curl -fsSL https://bun.sh/install | bash
- name: Install dependencies
run: ~/.bun/bin/bun install --frozen-lockfile
- name: Run tests
env:
SHADE_TEST_PG_URL: postgres://postgres:test@localhost:5432/postgres
run: ~/.bun/bin/bun test --recursive
- name: Run examples
run: |
~/.bun/bin/bun run examples/01-basic-conversation/main.ts
~/.bun/bin/bun run examples/04-identity-verification/main.ts

3
.gitignore vendored
View File

@@ -2,3 +2,6 @@ node_modules/
dist/ dist/
*.tsbuildinfo *.tsbuildinfo
.DS_Store .DS_Store
**/.tmp-*.db
**/.tmp-*.db-shm
**/.tmp-*.db-wal

File diff suppressed because it is too large Load Diff

View File

@@ -164,9 +164,190 @@ Nova's `pushDevices.encryptionKey` column is a per-device static AES key. To mig
During the rollout, send notifications with a `v: 1` (legacy) or `v: 2` (Shade) field so old and new clients coexist. During the rollout, send notifications with a `v: 1` (legacy) or `v: 2` (Shade) field so old and new clients coexist.
## Migration to at-rest encryption (V3.2)
Shade 0.4.0 ships `@shade/storage-encrypted` — opt-in AES-256-GCM
encryption of every sensitive payload in the local SQLite/Postgres store.
Existing 0.3.x deploys keep their unencrypted DB and behave exactly as
before; encryption is enabled per-deployment with one CLI command.
### One-shot migration (SQLite)
```bash
# Encrypts in place, drops unencrypted tables, leaves a .bak alongside.
shade migrate-storage \
--key-source passphrase \
--passphrase "$SHADE_STORAGE_PASSPHRASE" \
--salt-file /data/shade-client.db.salt
```
For a dry run that validates every row without writing:
`shade migrate-storage … --dry-run`.
### Code-level switch
Replace:
```ts
import { SQLiteStorage } from '@shade/storage-sqlite';
const storage = new SQLiteStorage('/data/shade-client.db');
```
with:
```ts
import { KeyManager, EncryptedSQLiteStorage } from '@shade/storage-encrypted';
const km = await KeyManager.open({
kind: 'passphrase',
passphrase: process.env.SHADE_STORAGE_PASSPHRASE!,
salt: loadSaltFromDisk(),
});
const storage = await EncryptedSQLiteStorage.open({
dbPath: '/data/shade-client.db',
keyManager: km,
});
```
The encrypted store implements the same `StorageProvider`, so
`ShadeSessionManager` and the rest of the wiring is unchanged.
See `docs/storage-encryption.md` for the full design, key sources
(passphrase / OS keychain / app-injected) and rotation.
## Migrating from 0.3.x to 4.0 (GA)
Shade 4.0 is the GA-frozen baseline. Everything from V3.2V3.12 is
merged, externally reviewed, and the wire format is locked. Nothing is
breaking on the wire compared to 0.4.x — peers continue to interoperate.
The 4.0 migration is therefore mostly **opt-in surface activation**
plus a version-bump.
### What stays the same
- Wire envelope `0x02` (RatchetMessage) with u32 length-prefixes.
- Wire envelope `0x11` (stream-chunk) for `@shade/streams`.
- HTTP shape of all `/v1/keys/...` and `/v1/transfer/...` endpoints.
- All `StorageProvider` core method signatures.
- Identity fingerprints, X3DH flow, Ed25519 signature format.
A 0.3.x peer that has not enabled any opt-ins talks to a 4.0 peer
without code changes. The version bump is semantic ("we have completed
the audit cycle"), not breaking.
### What's new (opt-in)
| Surface | Package | How to enable |
|---------|---------|---------------|
| At-rest encryption | `@shade/storage-encrypted` | `shade migrate-storage` (see above) |
| Async store-and-forward | `@shade/inbox`, `@shade/inbox-server` | `createInboxServer()` + `new Inbox()` |
| Bridge transports (SSE, long-poll) | `@shade/transport-bridge`, `createBridgeRoutes()` | mount bridge routes; `FallbackBridgeTransport` |
| Web Workers crypto | `@shade/crypto-web/worker` | `shade.configureWorkerCrypto({ workerUrl })` |
| Social key recovery | `@shade/recovery` | `setupRecovery / attachGuardian / requestRecovery` |
| WebRTC P2P transport | `@shade/transport-webrtc` (peer-dep) | `shade.configureWebRTC({ factory })` |
| Key Transparency | `@shade/key-transparency`, `createPrekeyServerWithKT(...)` | server: `keyTransparency: { ... }` config; client: `keyTransparency: { mode, logPublicKey }` on `createShade` |
| Trust UX gates | built-in to `@shade/sdk` | `shade.beforeFirstLargeFile / beforeBackupImport / beforeNewDeviceTrust(...)` |
| Files RPC | `@shade/files` | `shade.files.serve(handler)` + `shade.files.client(peer)` |
Pulling in **none** of these gives you the 1.0-shape API at 4.0 quality
(audit-completed, soak-tested). Pulling in **all** of them gives the
full 4.0 stack.
### Schema additions
`StorageProvider` implementations (sqlite, postgres, encrypted variants)
auto-create the additional tables on `ensureTables()` /
`initialize()`. The 4.0 superset:
```sql
-- V3.2 (storage encryption) — only when EncryptedSQLiteStorage / EncryptedPostgresStorage is used
shade_master_key_meta(...) -- KeyManager fingerprint + scrypt params
shade_field_keys(...) -- per-(table, column) wrapped DEKs
-- V3.3 (fingerprint gates)
peer_verifications(...) -- markPeerVerified persistence
peer_identity_versions(...) -- bump on acceptIdentityChange
-- V3.6 (inbox relay)
shade_inbox_register(...) -- TOFU bind address ↔ signing key
shade_inbox_blobs(...) -- ciphertext blobs with TTL + msgId
-- V3.10 (recovery)
shade_recovery_setup(...) -- per-recoverer state
shade_recovery_deposits(...) -- per-guardian deposited shares
-- V3.12 (KT — server only)
shade_kt_leaves(...) -- append-only Merkle leaves
shade_kt_index(...) -- address-sorted commitment
shade_kt_sths(...) -- signed tree heads
-- streams resume (V0.2.0+, listed for completeness)
stream_state(...) -- at-rest encrypted streamSecret
```
A 0.3.x deploy that upgrades the package without enabling any new
surface gets these tables created on first start; they stay empty
unless the corresponding feature is wired. There is **no destructive
migration**. To verify before upgrading production:
```bash
shade doctor --db-path /data/shade-client.db
```
The CLI reports any mismatch between the on-disk schema and the version
the installed packages expect.
### Step-by-step upgrade (typical app)
1. **Bump dependencies.** Update every `@shade/*` to `^4.0.0` in your
`package.json`. Bun / npm / pnpm pull from the Gitea registry as
per `.npmrc`.
2. **Re-run install.** `bun install` (or your tool of choice). The new
table definitions ship with the storage backends — no schema-edit
PRs against your DB.
3. **Boot once with no new opt-ins.** Existing send/receive should work
byte-identically. `shade doctor` should print all green.
4. **Pick the opt-ins you actually want.** Wire them one at a time
(storage-encryption first, then fingerprint gates, then any of the
recovery / KT / WebRTC / inbox surfaces). Each surface has its own
doc under `docs/` (`storage-encryption.md`, `trust-ux.md`,
`recovery.md`, `key-transparency.md`, `webrtc.md`, `inbox.md`,
`transport.md`, `web-workers.md`, `files.md`).
5. **Run cross-version smoke.** Boot a 0.3.x peer next to a 4.0 peer in
staging; exchange a session; confirm `shade fingerprint` matches on
both ends and a round-trip message decrypts cleanly.
6. **Ship 4.0 to a canary.** Roll forward; revert path is `bun
install @shade/sdk@^0.4.0` — there is no DB write that 0.4 cannot
also read.
### Operator checklist (prekey container)
If you operate the standalone container (`gt.zyon.no/stian/shade-prekey`):
1. Pull the 4.0 image: `docker pull gt.zyon.no/stian/shade-prekey:4.0.0`.
2. Add new env vars only if you are turning the corresponding surface
on:
- `SHADE_INBOX_PG_URL` / `SHADE_INBOX_DB_PATH` — async store-and-forward.
- `SHADE_INBOX_PRUNE_INTERVAL_MINUTES` — inbox prune cadence.
- `SHADE_BRIDGE_*` — bridge / SSE / long-poll surface.
- `SHADE_KT_*` — Key Transparency mode + signing key path.
- `SHADE_TRANSFER_*` — transfer routes mounted on the same Hono app.
3. Restart with the existing volume; the inbox / KT tables auto-create
on first request.
4. Update `docs/PRODUCTION-CHECKLIST.md` items for any new surface
you've enabled (rate-limit budgets, retention policies, KT
witness-pinning).
5. Verify the [OpenAPI](packages/shade-server/openapi.yaml) endpoints
you advertise to clients now include the routes you mounted.
### What about 4.0 → 4.x?
V4.x is bug-fix only. No wire-bump until V5.0 (voice/video) which
is **additive** — it allocates new envelope types (frame-key prefixes)
that 4.0 clients ignore by design.
## Common pitfalls ## Common pitfalls
1. **Don't store private keys in shared databases without encryption at rest**Shade trusts the storage layer to be secure. Use filesystem encryption or PostgreSQL TDE if the database is on shared infrastructure. 1. **Don't store private keys in shared databases without encryption at rest** — for shared infrastructure, enable `@shade/storage-encrypted` (V3.2) or use filesystem encryption / PostgreSQL TDE. The default `SQLiteStorage` and `PostgresStorage` write unencrypted.
2. **Don't skip identity verification** — Shade gives you fingerprints (`getIdentityFingerprint()`), but it's the user's responsibility to compare them out-of-band on first contact. 2. **Don't skip identity verification** — Shade gives you fingerprints (`getIdentityFingerprint()`), but it's the user's responsibility to compare them out-of-band on first contact.
3. **Don't reuse session storage between identities** — each user/device should have its own Shade storage. Mixing identities in one storage will corrupt the ratchet state. 3. **Don't reuse session storage between identities** — each user/device should have its own Shade storage. Mixing identities in one storage will corrupt the ratchet state.
4. **Keep prekey stocks topped up** — call `ensurePreKeyStock()` periodically (e.g., on app start or every hour). When the server runs out of one-time prekeys, new sessions will fall back to using just the signed prekey, which is slightly less secure. 4. **Keep prekey stocks topped up** — call `ensurePreKeyStock()` periodically (e.g., on app start or every hour). When the server runs out of one-time prekeys, new sessions will fall back to using just the signed prekey, which is slightly less secure.

469
README.md
View File

@@ -2,83 +2,380 @@
End-to-end encryption library implementing the Signal Protocol (X3DH + Double Ratchet) for TypeScript/Bun. Drop into any project — frontend, backend, mobile — to get forward secrecy, post-compromise recovery, and self-healing security. End-to-end encryption library implementing the Signal Protocol (X3DH + Double Ratchet) for TypeScript/Bun. Drop into any project — frontend, backend, mobile — to get forward secrecy, post-compromise recovery, and self-healing security.
> **4.0.0 — General Availability.** All V3.1 → V3.12 work is merged,
> the cross-platform vector suite is green on TS + Kotlin (1000 / 1000
> + 11 / 11), the threat model has been refreshed for every new
> surface, and the core stack (X3DH, ratchet, storage encryption,
> recovery, WebRTC P2P, Key Transparency) has been packaged for
> external review. The wire format is **unchanged from 0.4.x** — 4.0
> peers interoperate with 0.4.x peers byte-for-byte. See
> [MIGRATION.md § 0.3.x → 4.0](./MIGRATION.md#migrating-from-03x-to-40-ga)
> for the upgrade path and [CHANGELOG.md § 4.0.0](./CHANGELOG.md) for
> the consolidated release notes. Voice / Video have been moved to
> [V5.0](./docs/V5.0.md), to be built on top of the frozen 4.0
> baseline.
## Status
| Area | 4.0 status | Pointers |
|------|-----------|----------|
| Protocol core (X3DH + ratchet + sender keys) | ✅ Done — frozen | [`packages/shade-core`](./packages/shade-core) |
| Storage encryption (V3.2) | ✅ Done — opt-in `EncryptedSQLiteStorage` / `EncryptedPostgresStorage`, key sources: passphrase / OS keychain / app-injected | [`docs/storage-encryption.md`](./docs/storage-encryption.md) |
| Fingerprint gates & trust UX (V3.3) | ✅ Done — `Shade.beforeFirstLargeFile / beforeBackupImport / beforeNewDeviceTrust` | [`docs/trust-ux.md`](./docs/trust-ux.md) |
| Observability v2 (V3.4) | ✅ Done — OpenTelemetry-shaped events, `/metrics`, observer dashboard | [`docs/observability.md`](./docs/observability.md) |
| Android parity & cross-platform CI (V3.5) | ✅ Done — TS + Kotlin vector-gate live; Android `KeystoreStorage` is post-GA | [`android/shade-android/README.md`](./android/shade-android/README.md), [`docs/cross-platform.md`](./docs/cross-platform.md) |
| Async store-and-forward (V3.6) | ✅ Done — `@shade/inbox` + `@shade/inbox-server` | [`docs/inbox.md`](./docs/inbox.md) |
| Transport bridge (V3.7) | ✅ Done — SSE / long-poll / WS adapters | [`docs/transport.md`](./docs/transport.md) |
| Web Workers crypto (V3.8) | ✅ Done — lane keys never cross the thread boundary | [`docs/web-workers.md`](./docs/web-workers.md) |
| Rich file metadata + thumbnails (V3.9) | ✅ Done — in `@shade/files` | [`docs/files.md`](./docs/files.md) |
| Social key recovery (V3.10) | ✅ Done — Shamir + AEAD-gated reconstruction + guardian widgets | [`docs/recovery.md`](./docs/recovery.md) |
| WebRTC P2P transport (V3.11) | ✅ Done — `RTCDataChannel` with `MultiTransportFallback([webrtc, http])` | [`docs/webrtc.md`](./docs/webrtc.md) |
| Key Transparency (V3.12) | ✅ Done — opt-in Merkle log, signed STH, witness gossip | [`docs/key-transparency.md`](./docs/key-transparency.md) |
| External crypto review | 🟡 Bundle ready — review window open after tag | [`docs/audit/REVIEW-BUNDLE.md`](./docs/audit/REVIEW-BUNDLE.md) |
| Soak (≥ 2 weeks under load) | 🟡 Harness shipped — operator runs it | [`scripts/soak.ts`](./scripts/soak.ts) (`bun run soak --hours 336`) |
| Voice / Video / Broadcast | 🔜 V5.0 — built on top of frozen 4.0 stack | [`docs/V5.0.md`](./docs/V5.0.md) |
## What you get ## What you get
**Protocol core**
- **X3DH** initial key agreement (works asynchronously via prekey bundles) - **X3DH** initial key agreement (works asynchronously via prekey bundles)
- **Double Ratchet** for per-message forward secrecy and post-compromise security - **Double Ratchet** for per-message forward secrecy and post-compromise security
- **Self-authenticated prekey server** (Hono, Docker-ready) with rate limiting, metrics, health checks - **Sender keys** for group ratchet (1:N broadcast key derivation)
- **Persistent storage backends**: SQLite (zero-config) and PostgreSQL (Drizzle)
- **Identity rotation** with grace period for old sessions - **Identity rotation** with grace period for old sessions
- **Safety numbers** (Signal-style fingerprints) for out-of-band verification - **Safety numbers** (Signal-style fingerprints) for out-of-band verification
- **Constant-time comparisons** and **memory zeroization** for hardened operation - **Constant-time comparisons** and **memory zeroization** for hardened operation
- **Binary wire format** that's significantly smaller than JSON - **Binary wire format** (`@shade/proto`) — significantly smaller than JSON
**Storage**
- **Persistent backends**: SQLite (zero-config, `bun:sqlite`) and PostgreSQL (Drizzle, FOR UPDATE SKIP LOCKED)
- **Crash-safe** — sessions survive container restarts, power outages, SIGKILL - **Crash-safe** — sessions survive container restarts, power outages, SIGKILL
- **Live observability** — bundled dashboard SPA + embeddable React widgets to see what's happening between every step - **At-rest encryption (V3.2, opt-in)** — AES-256-GCM under per-(table,column) DEKs; key sources: passphrase (scrypt), OS keychain (`@shade/keychain`), or app-injected. Online re-key, no downtime.
**Servers**
- **Self-authenticated prekey server** (`@shade/server`, Hono, Docker-ready) with rate limiting, metrics, health checks
- **Async store-and-forward relay** (`@shade/inbox-server`) — TTL-bound ciphertext blobs, signed PUT/FETCH/ACK, idempotent on `(address, msgId)`, per-recipient quota
- **Bridge transports** (`@shade/transport-bridge`) — WS → SSE → long-poll fallback chain for clients that can't keep a WebSocket open. Same `IncomingMessage` shape across all three.
- **Standalone container** — one image bundles prekey + inbox + bridge + transfer + KT + observer
**Trust UX**
- **Fingerprint gates (V3.3)** — `Shade.beforeFirstLargeFile(threshold, handler)`, `beforeBackupImport`, `beforeNewDeviceTrust`. Gates raise `FingerprintNotVerifiedError` on the operations that matter, default-warn TOFU otherwise.
- **`<FingerprintCompare />` / `<FingerprintGate />`** widgets for the matching UI side.
**File transfer & filesystem**
- **E2EE file transfers** (`@shade/streams` + `@shade/transfer`) — multi-lane chunked uploads/downloads with resume, integrity checks (per-lane sha256 + overall sha256), HTTP/WS fallback, `MultiTransportFallback` for N-ary demotion
- **WebRTC P2P transport (V3.11, opt-in)** — `RTCDataChannel` chunk path with public-STUN defaults, TURN-relay support, glare-safe peer pool, automatic fallback to HTTP when NAT traversal fails (`@shade/transport-webrtc`)
- **Web Workers crypto (V3.8, opt-in)** — AEAD, HKDF, HMAC, X25519, Ed25519 and per-lane stream state run in a dedicated worker. 100 MB+ uploads stay smooth without frame drops; lane keys never cross the thread boundary (`@shade/crypto-web/worker`)
- **E2EE filesystem RPC** (`@shade/files`) — typed `list/stat/mkdir/delete/move/read/write/getThumbnail` + custom ops, with rate-limit, retention, fingerprint-gate, and metrics hooks. React hooks under `@shade/files/react`.
**Recovery**
- **Social key recovery (V3.10)** — Shamir-split your identity to `n` guardians; any threshold-many `k` together restore it on a new device. No centralized recovery agent; OOB-fingerprint gate per guardian release; AES-GCM-authenticated reconstruction (`@shade/recovery` + `<RecoverySetup />` / `<RecoveryRequest />` / `<RecoveryApprove />`)
**Verifiable distribution**
- **Key Transparency (V3.12, opt-in)** — append-only Merkle log over the prekey server. Every `register` / `delete` becomes a signed leaf; every bundle-fetch carries an inclusion proof; an Ed25519-signed Tree Head ties roots to a fixed `log_id`. A `LightWitness` cross-checks STHs across clients so a malicious server that splits its view or rewrites history is caught (`@shade/key-transparency`).
**Observability**
- **Live observability** — OpenTelemetry-shaped events, bundled dashboard SPA + embeddable React widgets to see what's happening between every step (`@shade/observability` + `@shade/observer` + `@shade/dashboard`)
**Tooling**
- **CLI** — `shade init` scaffolder, `shade migrate-storage` (V3.2), `shade rotate-storage-key`, `shade fingerprint`, `shade rotate`, `shade peer`, `shade dashboard`, `shade doctor`, `shade backup` (`@shade/cli`)
- **Soak harness** — `bun run soak --hours 336` for the 2-week GA-stable window (`scripts/soak.ts`)
## Quick start ## Quick start
Add the Gitea npm registry to your project's `.npmrc`:
```
@shade:registry=https://gt.zyon.no/api/packages/Stian/npm/
```
Then install the SDK (one-liner for most use cases):
```bash
bun add @shade/sdk
```
Or install specific packages if you need fine-grained control:
```bash ```bash
# In your project
bun add @shade/core @shade/crypto-web @shade/storage-sqlite bun add @shade/core @shade/crypto-web @shade/storage-sqlite
``` ```
Even faster — scaffold a new project with the CLI:
```bash
bun add -g @shade/cli
shade init my-app --template bun-server
cd my-app && bun install && bun run start
```
Magic one-liner with the SDK:
```ts
import { createShade } from '@shade/sdk';
const shade = await createShade({
prekeyServer: 'https://shade.example.com',
storage: 'sqlite:/data/shade.db',
address: 'alice@example.com',
});
// Send (auto-establishes session if none exists)
const envelope = await shade.send('bob@example.com', 'Hello, encrypted world!');
// Receive
const plaintext = await shade.receive('alice@example.com', incomingEnvelope);
// Your safety number for out-of-band verification
console.log(await shade.fingerprint);
```
### Opt-in surfaces (V3.x → 4.0 GA)
All of these are off by default. Wire them only where you need them.
```ts
// V3.3 — Fingerprint gates: enforce verification on the operations that matter
shade.beforeFirstLargeFile(10 * 1024 * 1024, async ({ peer, fingerprint }) => {
return await ui.confirmSafetyNumberMatches(peer, fingerprint);
});
shade.beforeBackupImport(async ({ embeddedFingerprint }) => { /* ... */ });
shade.beforeNewDeviceTrust(async ({ peer, oldFp, newFp }) => { /* ... */ });
// V3.8 — Web Workers crypto: opt-in, lane keys stay off the main thread
shade.configureWorkerCrypto({
workerUrl: new URL('@shade/crypto-web/worker', import.meta.url),
});
// V3.11 — WebRTC P2P transport: file transfers ride RTCDataChannel where NAT allows
import { nativeRtcFactory } from '@shade/transport-webrtc';
shade.configureWebRTC({
factory: nativeRtcFactory(),
iceServers: [{ urls: 'stun:stun.l.google.com:19302' }],
});
// V3.12 — Key Transparency: detect server-side bundle swaps
const shade = await createShade({
prekeyServer: 'https://shade.example.com',
keyTransparency: { mode: 'observe-strict', logPublicKey: PINNED_KEY_BYTES_32 },
});
```
### Files RPC (`@shade/files`)
```ts
// Server side — Bob exposes a virtual filesystem
const stop = await shade.files.serve({
list: async (ctx) => ({ entries: await readdirAt(ctx.path), hasMore: false }),
read: async (ctx) => readAt(ctx.path), // returns inline ≤ 256 KiB or streams
write: async (ctx) => writeAt(ctx.args), // receives inline or streams
// + stat, mkdir, delete, move, getThumbnail, plus typed custom ops
});
// Client side — Alice consumes Bob's filesystem
const fs = await shade.files.client('bob');
await fs.write('/photos/cover.png', new Uint8Array([/* ... */])); // auto inline/streams
const result = await fs.read('/photos/cover.png');
```
Files ≤ 256 KiB ride inline in the RPC envelope; larger files automatically promote to multi-lane `@shade/transfer` streams with sha256 integrity. See [`docs/files.md`](./docs/files.md) for the full API.
### Lower-level access
```ts ```ts
import { ShadeSessionManager } from '@shade/core'; import { ShadeSessionManager } from '@shade/core';
import { SubtleCryptoProvider } from '@shade/crypto-web'; import { SubtleCryptoProvider } from '@shade/crypto-web';
import { SQLiteStorage } from '@shade/storage-sqlite'; import { SQLiteStorage } from '@shade/storage-sqlite';
const crypto = new SubtleCryptoProvider(); const manager = new ShadeSessionManager(
const storage = new SQLiteStorage('/data/shade-client.db'); new SubtleCryptoProvider(),
new SQLiteStorage('/data/shade.db'),
const manager = new ShadeSessionManager(crypto, storage); );
await manager.initialize(); await manager.initialize();
// Establish a session with a peer (after fetching their bundle)
await manager.initSessionFromBundle('bob', bobBundle);
// Encrypt
const envelope = await manager.encrypt('bob', 'Hello, encrypted world!');
// Decrypt
const plaintext = await manager.decrypt('alice', incomingEnvelope);
``` ```
## Architecture ### At-rest encryption (V3.2)
```ts
import { KeyManager, EncryptedSQLiteStorage } from '@shade/storage-encrypted';
const km = await KeyManager.open({
kind: 'passphrase',
passphrase: process.env.SHADE_STORAGE_PASSPHRASE!,
salt: loadSaltFromDisk(),
});
const storage = await EncryptedSQLiteStorage.open({
dbPath: '/data/shade-client.db',
keyManager: km,
});
```
To migrate an existing 0.3.x SQLite DB in place:
```bash
shade migrate-storage \
--key-source passphrase \
--passphrase "$SHADE_STORAGE_PASSPHRASE" \
--salt-file /data/shade-client.db.salt
```
## Architecture — keys vs. payloads
Shade splits the network into a **public-key plane** (the prekey
server) and an **encrypted plane** (everything else). The prekey
server only sees public key material. If you remember nothing else
from this README, remember this picture:
``` ```
Shade Prekey Server (Hono) Shade Prekey Container (Hono — public keys only)
POST /v1/keys/register (signed) /v1/keys/* /v1/inbox/* /v1/bridge/* /v1/transfer/*
GET /v1/keys/bundle/:address /v1/kt/* /metrics /healthz /ready
POST /v1/keys/replenish (signed)
DELETE /v1/keys/:address (signed)
┌─────────────────────┴─────────────────────┐ ┌────────────────────────────────────┐
│ │ │ │
[Client A] [Client B] [Client A] [Client B]
ShadeSessionManager ShadeSessionManager ShadeSessionManager ShadeSessionManager
│ │ │ │
├──── X3DH ────────────────────────────────►│ ├── X3DH (handshake via prekey srv) ─►│
│ │
│◄──── Double Ratchet messages ────────────►│
│ │ │ │
│◄── Double Ratchet messages ────────►│ ← end-to-end,
│ (ratchet 0x02 / chunks 0x11) │ never on the
│ │ prekey server
│◄── @shade/transfer chunks ─────────►│ ← peer-to-peer
│ POST /v1/transfer/:id/chunk │ HTTP, opaque
│ GET /v1/transfer/:id/state │ ciphertext
│◄── @shade/inbox blobs (offline) ───►│ ← TTL-bound
│ POST /v1/inbox/:address │ ciphertext-only
│ POST /v1/inbox/:address/fetch │ relay
│◄── @shade/transport-webrtc ────────►│ ← optional P2P
│ RTCDataChannel `shade-transfer/v1` │ `MultiTransportFallback`
│ │ auto-demotes to HTTP
SQLiteStorage / PostgresStorage SQLiteStorage / PostgresStorage SQLiteStorage / PostgresStorage SQLiteStorage / PostgresStorage
+ EncryptedSQLiteStorage (V3.2) + EncryptedSQLiteStorage (V3.2)
(private keys + sessions) (private keys + sessions)
``` ```
### What goes via the prekey server
- Identity public keys (Ed25519 + X25519)
- Signed prekeys + one-time prekey bundles
- Registration / replenish / delete writes, all Ed25519-signed
- (V3.6) Inbox ciphertext blobs with TTL — same container, separate
routes; the relay only sees `address || msgId || ciphertext-bytes`
- (V3.7) Bridge transports (SSE / long-poll / WS) — also delivered by
the same Hono app for clients that can't hold a WebSocket
- (V3.12, opt-in) KT inclusion proofs + signed tree heads on
`/v1/kt/*` — verifiable distribution
- Operator-only metrics and the optional observer dashboard
### What does **not** go via the prekey server
- **Message plaintext, ever.** Encrypted ratchet envelopes flow peer-
to-peer over whatever transport you choose (HTTP, WebSocket, your
own broker, or the inbox relay above — which carries ciphertext only).
- **File chunks.** `@shade/transfer` POSTs ciphertext directly to the
receiver's `/v1/transfer/:streamId/chunk` route — the prekey server
is not involved. With V3.11 + `configureWebRTC()`, chunks ride
`RTCDataChannel` peer-to-peer; the relay is bypassed entirely.
- **Identity private keys.** They never leave the device's storage.
- **Filesystem RPC.** `@shade/files` rides the Double Ratchet for
control + small payloads, then promotes to direct `@shade/transfer`
streams for larger blobs.
- **Stream resume secrets.** Persisted only on the local device,
encrypted under a device-key derived from the identity signing key.
The prekey server is metadata-bearing (see `THREAT-MODEL.md § 2`):
it sees who registers, who fetches whose bundle, and when. It does
**not** see message contents, transfer contents, or session state.
**V3.12 Key Transparency** (opt-in) makes its bundle distribution
*verifiable* so a malicious server that swaps a bundle is caught.
For the full threat model and mitigations, read
[THREAT-MODEL.md](./THREAT-MODEL.md). For deployment-time guarantees,
read [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md).
## Packages ## Packages
All packages publish in lockstep at `4.0.0`.
| Package | Purpose | | Package | Purpose |
|---------|---------| |---------|---------|
| `@shade/core` | Protocol logic (X3DH, Double Ratchet, session manager, errors, events) | | `@shade/core` | Protocol logic (X3DH, Double Ratchet, sender keys, session manager, errors, events) |
| `@shade/crypto-web` | SubtleCrypto + @noble/curves provider, in-memory storage | | `@shade/proto` | Compact binary wire format (`0x01` PreKeyMessage, `0x02` RatchetMessage, `0x11` StreamChunk) |
| `@shade/storage-sqlite` | Persistent SQLite storage (zero-config, bun:sqlite) | | `@shade/crypto-web` | SubtleCrypto + @noble/curves provider, in-memory storage. Includes the V3.8 Web Workers entrypoint (`@shade/crypto-web/worker`) — drop-in `WorkerCryptoProvider` plus `createEncryptStream` / `createDecryptStream` TransformStream factories |
| `@shade/storage-postgres` | PostgreSQL storage with Drizzle for shared databases | | `@shade/observability` | OpenTelemetry-shaped event bus consumed by `@shade/observer`, server hooks, and the dashboard |
| `@shade/server` | Prekey server (Hono routes, auth, rate limit, health, metrics) | | `@shade/keychain` | OS keychain bindings (libsecret / Keychain / Credential Manager) used by `@shade/storage-encrypted` and the CLI |
| `@shade/transport` | HTTP + WebSocket transport wrappers with auto-encryption | | `@shade/key-transparency` | Key Transparency (V3.12) — RFC 6962-style append-only Merkle log, address-index commitment, signed tree heads, and a `LightWitness` for split-view detection. Opt-in on both server and client. |
| `@shade/proto` | Compact binary wire format (smaller than JSON) | | `@shade/storage-sqlite` | Persistent SQLite storage (zero-config, `bun:sqlite`); also ships `SqliteInboxStore` |
| `@shade/storage-postgres` | PostgreSQL storage with Drizzle for shared databases; also ships `PostgresInboxStore` + `PostgresKTLogStore` |
| `@shade/storage-encrypted` | At-rest encryption (V3.2) — `EncryptedSQLiteStorage` / `EncryptedPostgresStorage`, `KeyManager`, online re-key |
| `@shade/streams` | Multi-lane chunk encryption — HKDF-derived per-lane keys, deterministic AES-GCM nonces, streaming SHA-256, file metadata + thumbnails (V3.9) |
| `@shade/transport` | HTTP + WebSocket transport wrappers with auto-encryption; KT-verifying `fetchBundleVerified` |
| `@shade/transport-bridge` | WS → SSE → long-poll fallback chain (V3.7) — single `IncomingMessage` shape across transports for clients that can't keep a WebSocket open |
| `@shade/transport-webrtc` | V3.11 P2P chunk transport via `RTCDataChannel`. Plugs into `@shade/transfer` as an `ITransferTransport`; signaling rides Shade's own ratchet. Memory factory + native (`globalThis.RTCPeerConnection`) factory included; `MultiTransportFallback([webrtc, http])` wired automatically when `shade.configureWebRTC()` is called. |
| `@shade/server` | Prekey server (Hono routes, auth, rate limit, health, metrics). `createPrekeyServerWithKT(...)` opts into V3.12 KT mode |
| `@shade/inbox-server` | Async store-and-forward relay (V3.6) — Hono routes, signed PUT/FETCH/DELETE, per-recipient TTL + quota, idempotent on `(address, msgId)`. Bundles into the same standalone container as the prekey server |
| `@shade/inbox` | Inbox client + durable outgoing queue + receive cursor + push-trigger hook (`onMessageQueued`); composes on top of `Shade.send`/`Shade.receive` for offline-recipient delivery |
| `@shade/transfer` | Transfer engine on top of streams: parallel lanes, resume, HTTP + WS transport with auto-fallback, `MultiTransportFallback` (N-ary demotion), integrity verification |
| `@shade/files` | Typed E2EE filesystem RPC — list/stat/mkdir/delete/move/read/write/getThumbnail + custom ops, auto inline/streams routing, production hooks (rate limit, retention, fingerprint gate, metrics), React hooks under `@shade/files/react` |
| `@shade/recovery` | Social key recovery (V3.10) — Shamir-split identity to `n` guardians; threshold-many `k` reconstruct on a new device. AES-GCM-authenticated reconstruction; OOB-fingerprint gate per guardian release |
| `@shade/observer` | Live debugger backend (snapshot, SSE, dashboard) — see [README](./packages/shade-observer/README.md) | | `@shade/observer` | Live debugger backend (snapshot, SSE, dashboard) — see [README](./packages/shade-observer/README.md) |
| `@shade/widgets` | Embeddable React widgets — see [README](./packages/shade-widgets/README.md) |
| `@shade/dashboard` | Standalone dashboard SPA bundled into the observer | | `@shade/dashboard` | Standalone dashboard SPA bundled into the observer |
| `@shade/sdk` | High-level wrapper with `createShade()` one-liner, auto-publish, auto-establish, auto-replenish, `Shade.files` namespace, fingerprint gates, KT integration, WebRTC opt-in |
| `@shade/widgets` | Embeddable React widgets — fingerprint compare/gate, recovery setup/request/approve, transfer uploader/downloader, observer panels |
| `@shade/cli` | `shade init` scaffolder + utilities (fingerprint, rotate, peer, dashboard, doctor, backup, migrate-storage, rotate-storage-key) |
## Shade as a modular toolkit
Shade is split into packages so each project can depend on **only what it needs**—encrypted messaging, file transfer, prekey hosting, social recovery, KT verification, or lower-level building blocks. You do not need one giant stack for every use case.
For a **plain-language map** (which packages to add, what the prekey server does vs your own wiring, and where to start in code), see **[docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md)**.
## Publishing
All packages publish to a self-hosted Gitea npm registry on `gt.zyon.no`. The Docker image of the standalone container ships at `gt.zyon.no/stian/shade-prekey:<tag>`.
```bash
# Bump all packages in lockstep
bun run version 4.0.1
# Dry-run (pack all tarballs without publishing) — no token required
bun run publish:dry
# Real publish — interactive (prompts for GITEA_TOKEN, checks
# registry for conflicts, publishes via scripts/publish-all.ts)
bun run publish:all
# Build + push the standalone Docker image
bun run scripts/build-docker.ts -- --tag 4.0.1 --push
```
The interactive `scripts/publish-shade.sh` is the human entrypoint;
`scripts/publish-all.ts` is the headless variant used by CI and
`publish:dry`. They share a single `PACKAGES` list (24 entries at
4.0.0) so the two flows can never drift.
## Soak / GA-stable
Before tagging `4.0.0` as `latest` and recommending production
upgrades, run the combined soak harness for ≥ 2 weeks:
```bash
# Full GA window (V4.0 §Soak): 14 days × 24 hours
bun run soak --hours 336
# Smoke (~3 minutes — ratchet ping-pong, integrity check)
bun run soak:smoke
```
The harness fans out N concurrent ratchet pairs, ping-pongs at
~400 ops/sec/pair, and reports cumulative counters every minute.
Any exception in any pair is captured and re-raised at shutdown so
silent failures cannot hide. Wrap it in `systemd-run --user`,
`nohup`, or a Gitea scheduled job for the actual 2-week window.
## Security properties ## Security properties
@@ -92,33 +389,95 @@ const plaintext = await manager.decrypt('alice', incomingEnvelope);
| **Memory zeroization** | Key material is zeroed after use (best-effort in JS) | | **Memory zeroization** | Key material is zeroed after use (best-effort in JS) |
| **Identity verification** | Safety numbers (60 digits) for out-of-band comparison | | **Identity verification** | Safety numbers (60 digits) for out-of-band comparison |
| **Identity rotation** | 7-day grace period for old sessions during rotation | | **Identity rotation** | 7-day grace period for old sessions during rotation |
| **At-rest encryption** *(V3.2, opt-in)* | AES-256-GCM under per-(table, column) DEKs; AAD binds `(table, column, pk)`; passphrase / OS keychain / app-injected master key; online re-key |
| **Fingerprint gates** *(V3.3)* | `Shade.beforeFirstLargeFile / beforeBackupImport / beforeNewDeviceTrust` raise `FingerprintNotVerifiedError` on the operations that matter; defaults to TOFU + warning when no gate is registered |
| **Async store-and-forward** *(V3.6)* | Relay only sees `address || msgId || ciphertext`; idempotent PUT; signed FETCH/ACK; TTL-bounded |
| **Web-Worker key isolation** *(V3.8)* | Lane keys, identity keys, and ratchet chain keys live inside a dedicated worker; main thread only ferries plaintext via transferable buffers; idle terminate releases worker memory |
| **Social key recovery** *(V3.10)* | Shamir over GF(2^8); AEAD-authenticated reconstruction (forged shares fail); guardian-side fingerprint gate before share release |
| **WebRTC P2P transport** *(V3.11)* | Same Double Ratchet authenticates SDP/ICE signaling; chunk frames AEAD-bound to `streamId/laneId/seq`; deterministic glare resolution; `MultiTransportFallback` auto-demotes to HTTP |
| **Key Transparency** *(V3.12, opt-in)* | Append-only Merkle log + signed tree heads + witness gossip — split-view and history-rewrite are detected by clients |
## Documentation ## Documentation
- [SECURITY.md](./SECURITY.md) — Reporting vulnerabilities, security policy **Operator + integrator**
- [THREAT-MODEL.md](./THREAT-MODEL.md) — Honest threat model and assumptions
- [examples/](./examples/) — Runnable example applications
- [MIGRATION.md](./MIGRATION.md) — How to replace existing crypto with Shade
## Deployment - [docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md) — **Modular toolkit**: pick packages by scenario (messages, files, browser, ops)
- [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md) — Pre-flight gates for going to production
- [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md) — Full deployment guide (Docker, env vars, PostgreSQL, backup, Dokploy)
- [docs/ROADMAP.md](./docs/ROADMAP.md) — V3.x → 4.0 GA → V5.0 trajectory
For containerized deployment (Docker/Dokploy): **Per-surface deep-dives**
```yaml - [docs/files.md](./docs/files.md) — `@shade/files` API + design (filesystem RPC, custom ops, hooks, React)
services: - [docs/streams.md](./docs/streams.md) — `@shade/streams` + `@shade/transfer` deep dive (incl. hardening + retention)
shade-prekey: - [docs/inbox.md](./docs/inbox.md) — `@shade/inbox` + `@shade/inbox-server` async store-and-forward relay (V3.6)
image: shade-prekey-server:latest - [docs/transport.md](./docs/transport.md) — `@shade/transport-bridge` SSE / long-poll / WS bridge layer (V3.7)
ports: - [docs/web-workers.md](./docs/web-workers.md) — V3.8 Web Workers crypto: setup, bundler recipes (Vite/Webpack/Rollup), Safari notes, lifecycle, threat-model
- "3900:3900" - [docs/recovery.md](./docs/recovery.md) — `@shade/recovery` social key recovery (V3.10): Shamir setup, guardian-side gates, threshold tuning
volumes: - [docs/webrtc.md](./docs/webrtc.md) — `@shade/transport-webrtc` P2P transport (V3.11): NAT-traversal, TURN config, glare resolution, wire format, multi-fallback wiring
- shade-data:/data - [docs/key-transparency.md](./docs/key-transparency.md) — `@shade/key-transparency` (V3.12): operator + client onboarding, witness role, recovery procedures
environment: - [docs/storage-encryption.md](./docs/storage-encryption.md) — V3.2 at-rest encryption: design, key sources, rotation
- SHADE_PREKEY_DB_PATH=/data/shade-prekeys.db - [docs/trust-ux.md](./docs/trust-ux.md) — V3.3 fingerprint gates: when each fires, handler patterns, widget integration
volumes: - [docs/observability.md](./docs/observability.md) — V3.4 event bus + dashboard
shade-data: - [docs/cross-platform.md](./docs/cross-platform.md) — V3.5 Android parity + cross-platform vector regime
**Threat model + audit**
- [SECURITY.md](./SECURITY.md) — Reporting vulnerabilities, security policy, threat-/test-matrix
- [THREAT-MODEL.md](./THREAT-MODEL.md) — Honest threat model and assumptions (12 numbered sections + residual-risks table)
- [docs/audit/REVIEW-BUNDLE.md](./docs/audit/REVIEW-BUNDLE.md) — External crypto-review entrypoint (scope, build instructions, reporting)
- [docs/audit/SCOPE.md](./docs/audit/SCOPE.md) — One-page audit-scope summary
**Migration + history**
- [MIGRATION.md](./MIGRATION.md) — How to replace existing crypto with Shade + the [0.3.x → 4.0 upgrade path](./MIGRATION.md#migrating-from-03x-to-40-ga)
- [CHANGELOG.md](./CHANGELOG.md) — `4.0.0` GA section + every prior release
- [docs/archive/](./docs/archive/) — V2.1 / V2.2 / V2.3 backlog and V3.1 → V3.12 implementation plans (all `Status: Done`)
- [docs/V5.0.md](./docs/V5.0.md) — Voice / Video / Broadcast (post-GA, built on the frozen 4.0 stack)
**Examples**
- [examples/](./examples/) — Runnable example applications, including
[`07-streams-upload`](./examples/07-streams-upload) (multi-lane file transfer)
and [`08-files-browser`](./examples/08-files-browser) (filesystem RPC)
## Deployment — one container per project
Shade ships as a self-contained Docker image. Deploy one container per project, point your app at it, done. Any stack (Bun, Python, Go, Rust, Kotlin) can use it — the container exposes a plain HTTP API documented in OpenAPI.
```bash
docker run -d \
--name my-project-shade \
-v my-project-shade:/data \
-p 3900:3900 \
-e SHADE_OBSERVER_TOKEN=change-me-to-at-least-16-chars \
gt.zyon.no/stian/shade-prekey:4.0.0
``` ```
The SQLite database persists to a Docker volume so all keys and prekey bundles survive restarts. The container includes:
- **Prekey server** — `/v1/keys/*` REST API
- **Inbox relay (V3.6)** — `/v1/inbox/*` async store-and-forward; enable
with `SHADE_INBOX_DB_PATH=/data/inbox.db` (or `SHADE_INBOX_PG_URL`).
`SHADE_INBOX_PRUNE_INTERVAL_MINUTES` controls TTL prune cadence.
- **Bridge transports (V3.7)** — `/v1/bridge/{stream,poll,ws}` SSE +
long-poll + WS adapters for clients that can't keep a WebSocket open.
- **Transfer routes** — `/v1/transfer/*` chunk + state + control routes
for `@shade/transfer`.
- **Key Transparency (V3.12)** — `/v1/kt/*` exposes `log_id`, latest +
historical STH, and consistency proofs. Enable with
`SHADE_KT_*` env vars; off by default.
- **Observer dashboard** — `/shade-observer/dashboard/` (off unless
`SHADE_OBSERVER_TOKEN` is set)
- **OpenAPI spec** — `/openapi.yaml` and interactive `/docs` viewer
(covers all 27 routes — prekey, inbox, bridge, transfer, KT,
observer, `/metrics`, `/healthz`, `/ready`)
- **Prometheus metrics** — `/metrics`
- **Health probes** — `/health` (full), `/healthz` (liveness),
`/ready` (readiness)
- **Stale cleanup** — purges inactive identities automatically
See [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md) for the full deployment guide, environment variables, PostgreSQL config, backup strategy, and Dokploy instructions.
## License ## License

View File

@@ -1,16 +1,54 @@
# Security Policy # Security Policy
## Review status
| Area | Status | Notes |
|------|--------|-------|
| Internal review | **Done** | Every mitigation in `THREAT-MODEL.md` is cross-linked to at least one automated test (see [Threat-/test-matrix](#threat--test-matrix) below). The matrix is enforced by `tests/security/*` + the cross-platform vector suite. |
| Independent code review | **Pending** | Targeted for **V4.0**. No external review has been completed. |
| Independent crypto review | **Pending** | Targeted for **V4.0** alongside the audit. |
| Pen test | **Pending** | Targeted for **V4.0**. |
> **Read this:** Shade implements the Signal Protocol primitives
> (X3DH + Double Ratchet) on top of `@noble/curves` and SubtleCrypto.
> The protocol is well-studied; the **implementation** has not yet been
> audited externally. Treat the wire format as stable but the
> implementation as "production-ready in trusted contexts" until V4.0
> closes the audit gap. The `THREAT-MODEL.md` cells with no test
> linkage are documentary, not enforced.
## Reporting a Vulnerability ## Reporting a Vulnerability
If you discover a security vulnerability in Shade, please report it privately by emailing the maintainer rather than opening a public issue. We take all reports seriously and will respond within 48 hours. If you discover a security vulnerability in Shade, please report it
privately by emailing the maintainer rather than opening a public
issue. We take all reports seriously and will respond within 48 hours.
### How to report
1. **Email:** the maintainer email listed in the package metadata.
For coordinated disclosure, prefer email over GitHub/Gitea so the
issue does not become public before a fix ships.
2. **PGP / age:** if you need encrypted reporting, ask for a key
over the same email — keys are not bound to the repo to avoid
key-rotation drift.
3. **Scope:** CVE-style severity (CVSS v3.1) is appreciated but not
required. A working reproduction is more valuable than a CVSS
score.
When reporting, please include: When reporting, please include:
- A description of the vulnerability - A description of the vulnerability
- Steps to reproduce - Steps to reproduce (a runnable script or test case)
- Affected versions - Affected versions
- Potential impact - Potential impact
- Any suggested mitigation - Any suggested mitigation
We commit to:
- Acknowledging receipt within 48 hours.
- A first-pass triage within 7 days.
- A coordinated disclosure timeline once severity is agreed; for
high-severity issues we aim to ship a patched release within 30
days of triage.
## What's in scope ## What's in scope
Shade aims to provide: Shade aims to provide:
@@ -46,3 +84,48 @@ Shade uses well-established primitives:
- **HMAC-SHA256** for chain key advancement (via Web Crypto SubtleCrypto) - **HMAC-SHA256** for chain key advancement (via Web Crypto SubtleCrypto)
These match the Signal Protocol specification. These match the Signal Protocol specification.
---
## Threat-/test-matrix
This is the consolidated index that backs `THREAT-MODEL.md`. Every
threat-model row that claims a mitigation must point to at least one
test file here. Pull requests that add a new mitigation must add a
matrix row in the same change.
| Threat-model row | Mitigation | Test file(s) |
|------------------|------------|--------------|
| § 1 Network attacker — signed writes | Ed25519 signature on every write | `packages/shade-server/tests/server.test.ts` |
| § 1 Network attacker — replay window | ±5 min `signedAt` enforcement | `packages/shade-server/tests/server.test.ts` (`"rejects registration with stale signedAt"`) |
| § 1 Network attacker — header AAD | Ratchet headers bound to ciphertext | `packages/shade-core/tests/ratchet.test.ts`, `packages/shade-streams/tests/tamper.test.ts`, `packages/shade-streams/tests/aead.test.ts` |
| § 1 Network attacker — forward secrecy | DH ratchet step + chain-key zeroize | `packages/shade-core/tests/ratchet.test.ts`, `packages/shade-crypto-web/tests/hardening.test.ts` |
| § 1 Network attacker — streaming sub-session FS/replay (V4.11) | Per-frame Double-Ratchet `seal`/`open`; counter-rewind & replay rejected; in-memory-only (never persisted) | `packages/shade-core/tests/stream.test.ts` (`"R1: replayed / rewound frame is rejected"`, `"R2/R3: long one-directional burst stays correct and memory-bounded"`) |
| § 1 Network attacker — streaming handshake auth (V4.11) | Identity-bound 3-DH against parent-session-pinned identities | `packages/shade-core/tests/stream.test.ts` (`"handshake is mutually authenticated against pinned identities"`) |
| § 3 Endpoint compromise — streaming sub-session isolation (V4.11) | Stream ratchet derived without touching the stored parent session; zeroized on close | `packages/shade-core/tests/stream.test.ts` (`"R5: opening/using/closing a stream never touches the parent session"`, `"close() zeroizes and blocks further use; idempotent"`) |
| § 2 Compromised prekey server — public-only storage | Prekey store never accepts a private key | `packages/shade-server/tests/server.test.ts`, `packages/shade-storage-sqlite/tests/sqlite-prekey-store.test.ts` |
| § 2 Compromised prekey server — signed replenish/delete | Per-identity Ed25519 signature | `packages/shade-server/tests/server.test.ts` |
| § 2 Compromised prekey server — fake-bundle detection | Out-of-band fingerprint comparison | `packages/shade-core/tests/fingerprint-session.test.ts` |
| § 3 Endpoint compromise — forward secrecy | Old keys not recoverable from leak | `packages/shade-core/tests/ratchet.test.ts`, `packages/shade-crypto-web/tests/hardening.test.ts` |
| § 3 Endpoint compromise — post-compromise security | First DH ratchet evicts leaked state | `packages/shade-core/tests/ratchet.test.ts` (`"alternating messages trigger DH ratchets"`) |
| § 3 Endpoint compromise — memory zeroization | Buffers wiped after use | `packages/shade-crypto-web/tests/hardening.test.ts` (`"zeroize"`) |
| § 3 Endpoint compromise — identity-rotation invalidates resume | Device-key bound to signing key | `packages/shade-core/tests/identity-rotation.test.ts`, `packages/shade-transfer/tests/resume.test.ts` |
| § 4 Compromised device storage — at-rest stream secrets | Resume secret AES-GCM under device-key | `packages/shade-transfer/tests/resume.test.ts` |
| § 4 Compromised device storage — at-rest session DB | **Pending V3.2** | _none yet_ |
| § 5 Timing side-channel — constant-time compare | XOR accumulator | `packages/shade-crypto-web/tests/hardening.test.ts` (`"timing variance stays bounded across mismatch positions"`) |
| § 5 Timing side-channel — primitives | SubtleCrypto + @noble/curves | `packages/shade-crypto-web/tests/provider.test.ts`, `packages/shade-streams/tests/aead.test.ts` |
| § 6 DoS — per-IP register/bundle rate limit | Token bucket per IP | `packages/shade-server/tests/rate-limit.test.ts` |
| § 6 DoS — per-identity replenish/delete rate limit | Token bucket per identity | `packages/shade-server/tests/rate-limit.test.ts` |
| § 6 DoS — body size cap (64 KiB) | Hono middleware | `packages/shade-server/tests/server.test.ts` |
| § 6 DoS — address validation | Regex + NFKC + length | `packages/shade-server/tests/server.test.ts` |
| § 6 DoS — per-sender ops/byte quota (`@shade/files`) | RateLimiter token bucket | `packages/shade-files/tests/security/quota.test.ts` |
| § 6 DoS — replay protection (`@shade/files`) | Idempotency cache | `packages/shade-files/tests/security/replay.test.ts` |
| § 6 DoS — fingerprint gate (`@shade/files`) | Per-sender trust check | `packages/shade-files/tests/security/fingerprint-gate.test.ts` |
| § 6 DoS — tampered envelope reject (`@shade/files`) | AEAD reject | `packages/shade-files/tests/security/tampered-envelope.test.ts` |
| § 8a Recovery — k-1 collusion impossible | Shamir Secret Sharing over GF(2^8) | `packages/shade-recovery/tests/shamir.test.ts`, `packages/shade-recovery/tests/adversarial.test.ts` |
| § 8b Recovery — forged share rejected | AES-GCM tag on backup blob + subset-search | `packages/shade-recovery/tests/adversarial.test.ts` (`"a corrupted share never authenticates against the backup AEAD tag"`) |
| § 8c Recovery — guardian OOB-fingerprint gate | Two-checkbox `<RecoveryApprove />` + decline propagation | `packages/shade-recovery/tests/adversarial.test.ts` (`"approve handler that REJECTS a wrong fingerprint never sends a grant"`, `"throwing approve handler counts as decline with descriptive reason"`) |
| § 9 Cross-sender X3DH state corruption | `initReceiverSession` copies keypair | `packages/shade-core/tests/ratchet.test.ts` (`"does not mutate the caller-provided keypair after a DH ratchet step"`), `packages/shade-recovery/tests/integration.test.ts` |
If you add a new mitigation, add a row here in the same PR — the
threat model is the contract; this matrix is the proof.

View File

@@ -2,6 +2,13 @@
This document describes what Shade protects against and what it doesn't. Read this before deploying Shade in any context where the answers matter. This document describes what Shade protects against and what it doesn't. Read this before deploying Shade in any context where the answers matter.
> Each numbered "Mitigations" entry below ends with a `[tests:]`
> footnote that links to the concrete test file(s) demonstrating the
> mitigation. If a mitigation has no `[tests:]` line, treat it as
> documentary — there is no automated test holding the line yet.
> See [SECURITY.md § Threat-/test-matrix](./SECURITY.md#threat--test-matrix)
> for the consolidated index.
## Assets ## Assets
The thing we're protecting: The thing we're protecting:
@@ -16,9 +23,13 @@ Can intercept, modify, drop, replay, and inject network traffic between clients
**Mitigations:** **Mitigations:**
- All identity-key writes to the prekey server are signed (Ed25519). Tampering is detected. - All identity-key writes to the prekey server are signed (Ed25519). Tampering is detected.
`[tests: packages/shade-server/tests/server.test.ts — "rejects unsigned registration", "rejects registration with wrong signing key"]`
- Signed requests have a 5-minute replay window. - Signed requests have a 5-minute replay window.
`[tests: packages/shade-server/tests/server.test.ts — "rejects registration with stale signedAt"]`
- The Double Ratchet binds message headers to ciphertext via AES-GCM AAD, so header tampering breaks decryption. - The Double Ratchet binds message headers to ciphertext via AES-GCM AAD, so header tampering breaks decryption.
`[tests: packages/shade-core/tests/ratchet.test.ts — "tampered ciphertext fails", "tampered header (counter) fails due to AAD"; packages/shade-streams/tests/tamper.test.ts; packages/shade-streams/tests/aead.test.ts]`
- Forward secrecy: even if an attacker captures all traffic, compromising a key later doesn't help them read past messages. - Forward secrecy: even if an attacker captures all traffic, compromising a key later doesn't help them read past messages.
`[tests: packages/shade-crypto-web/tests/hardening.test.ts; packages/shade-core/tests/ratchet.test.ts — DH ratchet steps + out-of-order delivery]`
**NOT mitigated:** **NOT mitigated:**
- Initial session establishment can be MITM'd if users don't verify identity fingerprints. The prekey server could distribute a fake bundle on first contact. Always compare safety numbers out-of-band for high-stakes communications. - Initial session establishment can be MITM'd if users don't verify identity fingerprints. The prekey server could distribute a fake bundle on first contact. Always compare safety numbers out-of-band for high-stakes communications.
@@ -28,19 +39,31 @@ The server holds identity public keys and prekey bundles. It can serve them to a
**Mitigations:** **Mitigations:**
- The server only stores PUBLIC keys, never private ones. - The server only stores PUBLIC keys, never private ones.
`[tests: packages/shade-server/tests/server.test.ts — registration, bundle fetch, replenish; packages/shade-storage-sqlite/tests/sqlite-prekey-store.test.ts]`
- Write operations are signed with the identity private key, so the server can't forge new identities or replenishments without the user's key. - Write operations are signed with the identity private key, so the server can't forge new identities or replenishments without the user's key.
`[tests: packages/shade-server/tests/server.test.ts — "rejects replenishment signed by wrong identity", "rejects delete signed by wrong identity"]`
- Bundle fetches are unauthenticated, so a malicious server can serve fake bundles. Detection requires out-of-band fingerprint comparison. - Bundle fetches are unauthenticated, so a malicious server can serve fake bundles. Detection requires out-of-band fingerprint comparison.
`[tests: packages/shade-core/tests/fingerprint-session.test.ts]`
**NOT mitigated:** **NOT mitigated:**
- A malicious server can substitute one user's prekey bundle with the server operator's own keys, enabling MITM at session establishment. Users must verify safety numbers to detect this. - A malicious server can substitute one user's prekey bundle with the server operator's own keys, enabling MITM at session establishment. Users must verify safety numbers to detect this.
**Partially mitigated by V3.12 Key Transparency** (opt-in):
- When the operator runs the server with `keyTransparency: { ... }` and clients pin the operator's STH-signing public key, every bundle fetch returns a Merkle inclusion proof against an append-only Signed Tree Head. A server that swaps `alice`'s bundle for one client and not another, or rewrites history to hide an earlier swap, is detected by an independent witness. KT does **not** prevent first-contact impersonation — a never-seen-before address can still be served maliciously on its very first registration.
`[tests: packages/shade-key-transparency/tests/manager.test.ts — "rotation: new register replaces old"; packages/shade-transport/tests/kt-split-view-e2e.test.ts — "two divergent views at the same tree_size are caught by witness"; packages/shade-server/tests/kt.test.ts — "bundle response carries verified inclusion proof"]`
### 3. Compromised endpoint (post-compromise) ### 3. Compromised endpoint (post-compromise)
Attacker briefly gains code execution or filesystem access on a user's device, exfiltrates session state, then loses access. Attacker briefly gains code execution or filesystem access on a user's device, exfiltrates session state, then loses access.
**Mitigations:** **Mitigations:**
- Forward secrecy: messages sent BEFORE the compromise cannot be decrypted with the leaked state. Old chain keys are zeroed after use. - Forward secrecy: messages sent BEFORE the compromise cannot be decrypted with the leaked state. Old chain keys are zeroed after use.
`[tests: packages/shade-core/tests/ratchet.test.ts — basic send/receive, ping-pong; packages/shade-crypto-web/tests/hardening.test.ts — zeroize]`
- Post-compromise security: as soon as a peer initiates a new DH ratchet step, the leaked state becomes useless for new messages. - Post-compromise security: as soon as a peer initiates a new DH ratchet step, the leaked state becomes useless for new messages.
`[tests: packages/shade-core/tests/ratchet.test.ts — "alternating messages trigger DH ratchets"]`
- Memory zeroization: message keys and chain keys are wiped from JS memory after use (best-effort — V8 may retain copies). - Memory zeroization: message keys and chain keys are wiped from JS memory after use (best-effort — V8 may retain copies).
`[tests: packages/shade-crypto-web/tests/hardening.test.ts — "zeroize" describe block]`
- Identity rotation invalidates leaked at-rest stream-resume secrets (device-key derived from signing key).
`[tests: packages/shade-core/tests/identity-rotation.test.ts; packages/shade-transfer/tests/resume.test.ts]`
**NOT mitigated:** **NOT mitigated:**
- An ongoing endpoint compromise can read messages in real time and exfiltrate identity private keys. - An ongoing endpoint compromise can read messages in real time and exfiltrate identity private keys.
@@ -49,36 +72,317 @@ Attacker briefly gains code execution or filesystem access on a user's device, e
### 4. Compromised device storage ### 4. Compromised device storage
Attacker gains access to the persistent storage (e.g., steals the SQLite file or dumps the PostgreSQL table). Attacker gains access to the persistent storage (e.g., steals the SQLite file or dumps the PostgreSQL table).
**Mitigations:** **Mitigations (default, no at-rest encryption):**
- Stored data includes private keys but is unencrypted at rest. Shade does NOT encrypt the storage layer — it assumes the database is in a trusted environment. - Stream-resume secrets *are* encrypted at rest under a device-key derived from the identity signing key, so a stolen DB without the live identity key cannot resume in-flight transfers.
`[tests: packages/shade-transfer/tests/resume.test.ts]`
- Filesystem-level encryption (LUKS, FileVault, BitLocker) is recommended but is the user's responsibility.
**NOT mitigated:** **Mitigations (with at-rest encryption enabled — V3.2 / `@shade/storage-encrypted`):**
- Filesystem-level encryption (LUKS, FileVault) is the user's responsibility. - All sensitive payloads are sealed with AES-256-GCM under per-(table, column) field keys derived from a passphrase (scrypt) / OS keychain / app-injected master key. A stolen DB file alone yields no usable private key material.
- Database TLS in transit is the user's responsibility. `[tests: packages/shade-storage-encrypted/tests/encrypted-sqlite.test.ts]`
- AAD binds (table, column, pk) so an attacker cannot swap rows or move ciphertext between columns without triggering decrypt failure.
`[tests: packages/shade-storage-encrypted/tests/encrypted-sqlite.test.ts — "row swap (sessions) → decrypt fails due to AAD mismatch"]`
- Bit-flips in the ciphertext blob are detected by the AEAD tag; the storage layer raises rather than returning corrupt key material.
`[tests: packages/shade-storage-encrypted/tests/aead.test.ts; encrypted-sqlite.test.ts — "flipped ciphertext byte → decrypt fails"]`
- Wrong passphrase / wrong keychain entry is rejected up-front via a fingerprint check, never silently writing under the wrong key.
`[tests: packages/shade-storage-encrypted/tests/encrypted-sqlite.test.ts — "rejects open with wrong key (fingerprint mismatch)"]`
- Online key rotation re-keys every row without downtime; the old key no longer opens the DB after rotation.
`[tests: packages/shade-storage-encrypted/tests/migrate.test.ts — "re-keys all rows; old key no longer opens DB"]`
**NOT mitigated (even with at-rest enabled):**
- A live process holds the storageKey and field keys in memory; an attacker who can read process memory (e.g., via `/proc/<pid>/mem`, swap dump, hibernation file) recovers the keys and thus the data. At-rest encryption protects the DB *file*, not the running process.
- The kernel's swap partition is not encrypted by Shade. If the OS pages key material to disk, it can be recovered. Use an encrypted swap device.
- A coredump of the live process exposes plaintext private keys.
- Filesystem-level encryption of the DB *backup* (e.g. `.bak` file produced by `shade migrate-storage`) is the operator's responsibility — the backup is plaintext during the brief migration window.
- If the master key is lost (forgotten passphrase, deleted keychain entry, lost injected key) the DB is permanently unrecoverable. V3.10 (Social Recovery) is the long-term mitigation.
### 5. Side-channel attacks (timing) ### 5. Side-channel attacks (timing)
Attacker measures timing of identity verification operations to recover key bits. Attacker measures timing of identity verification operations to recover key bits.
**Mitigations:** **Mitigations:**
- All comparisons of secret material use constant-time XOR-accumulator comparison (`constantTimeEqual`). - All comparisons of secret material use constant-time XOR-accumulator comparison (`constantTimeEqual`).
`[tests: packages/shade-crypto-web/tests/hardening.test.ts — "constantTimeEqual", "timing variance stays bounded across mismatch positions"]`
- AES-GCM and the underlying primitives are constant-time as implemented by SubtleCrypto and @noble/curves. - AES-GCM and the underlying primitives are constant-time as implemented by SubtleCrypto and @noble/curves.
`[tests: packages/shade-crypto-web/tests/provider.test.ts; packages/shade-streams/tests/aead.test.ts]`
**NOT mitigated:** **NOT mitigated:**
- JavaScript JIT compilation can introduce timing variability that's hard to control. - JavaScript JIT compilation can introduce timing variability that's hard to control.
- We don't claim resistance to power-analysis or fault-injection attacks (out of scope for a JS library). - We don't claim resistance to power-analysis or fault-injection attacks (out of scope for a JS library).
### 6. Denial of service ### 6. Malicious or compromised inbox relay (V3.6 store-and-forward)
The inbox relay holds **ciphertext blobs with TTL** so senders can deliver
to offline recipients. It is a separate trust domain from the prekey
server, and exposes a different surface.
**Mitigations:**
- The relay only stores `address || msgId || ciphertext-bytes || expires_at`.
Plaintext, ratchet state, and any private keys live exclusively on the
client. A DB dump leaks no message content.
`[tests: packages/shade-inbox-server/tests/routes.test.ts; packages/shade-inbox-server/tests/lifecycle.test.ts — "Tamper resistance"]`
- Recipient identity is bound to the address via TOFU: first
`POST /v1/inbox/register` claims the slot, and subsequent fetch/ack
must be Ed25519-signed by the same key. A different key claiming an
existing address is rejected with 401.
`[tests: packages/shade-inbox-server/tests/routes.test.ts — "rejects different key claiming same address", "rejects fetch from a different signing key", "rejects ack from a different signing key"]`
- Each PUT is signed by the sender's per-PUT signing key; the relay
verifies the signature before persisting. Bad sigs return 401.
`[tests: packages/shade-inbox-server/tests/routes.test.ts — "rejects bad sender signature"]`
- `msgId = sha256(ciphertext)` is verified server-side on PUT and
recomputed client-side on FETCH. A relay that flips a bit in storage
produces a digest mismatch the recipient flags as
`inbox.message_decrypt_failed` *without* acking, so the divergence
surfaces in operator telemetry instead of being silently consumed.
`[tests: packages/shade-inbox-server/tests/routes.test.ts — "rejects mismatched msgId"; packages/shade-inbox-server/tests/lifecycle.test.ts — "Tamper resistance"; packages/shade-inbox/tests/client.test.ts — "tamper detection"]`
- Replay-window of ±5 minutes on `signedAt` (matches the prekey
server's policy). Replays past that window return 409.
`[tests: packages/shade-inbox-server/tests/routes.test.ts — "rejects stale signature (replay window)"]`
- Idempotent PUT: two clients (or a buggy retry loop) submitting the
same ciphertext do *not* create duplicate rows; the second PUT
returns 200 with `idempotent: true`.
`[tests: packages/shade-inbox-server/tests/routes.test.ts — "idempotent on duplicate ciphertext"]`
- Periodic `InboxPruneTask` drops blobs past their TTL so a slow
consumer never sees a payload past expiry.
`[tests: packages/shade-inbox-server/tests/lifecycle.test.ts — "prune removes expired blobs but keeps live ones"]`
**NOT mitigated:**
- **Sender-recipient graph leakage.** The relay sees recipient address +
per-PUT sender pubkey + ciphertext byte-counts. Privacy-sensitive
deployments should use address-hashes (`sha256(real-address || salt)`)
and rotate sender signing keys per session. Mixing/onion-routing is
out of scope for V3.6 and a candidate for a future relay tier.
- **Operator-side queue deletion.** A malicious operator can drop every
blob queued for a target, forcing senders to resend. Recipient-side
ack happens *after* successful decrypt, so a delete only burns one
delivery attempt rather than silently consuming a message.
- **TTL-based reachability signal.** A PUT silently expiring after 7
days reveals that the recipient never came online. Operators concerned
with this metadata should clamp TTLs to a fixed value via the
`quota.maxTtlSeconds` / `quota.minTtlSeconds` knobs.
### 7. Denial of service
Attacker floods the prekey server to exhaust resources or one-time prekeys. Attacker floods the prekey server to exhaust resources or one-time prekeys.
**Mitigations:** **Mitigations:**
- Per-IP rate limiting on registration and bundle fetches. - Per-IP rate limiting on registration and bundle fetches.
`[tests: packages/shade-server/tests/rate-limit.test.ts — "register endpoint rate-limits per IP", "rate limit returns Retry-After header"]`
- Per-identity rate limiting on replenish and delete. - Per-identity rate limiting on replenish and delete.
- 64KB body size limit on POST endpoints. `[tests: packages/shade-server/tests/rate-limit.test.ts — "different keys have independent limits"]`
- 64 KiB body size limit on POST endpoints.
`[tests: packages/shade-server/tests/server.test.ts — body-size enforcement]`
- Address validation rejects path traversal and malformed inputs. - Address validation rejects path traversal and malformed inputs.
`[tests: packages/shade-server/tests/server.test.ts — "rejects invalid address format", "rejects invalid address in URL"]`
- Per-sender ops/byte quotas on `@shade/files` filesystem RPC.
`[tests: packages/shade-files/tests/security/quota.test.ts]`
- Per-recipient blob quota on `@shade/inbox-server` (default 1000 blobs
per address) + per-blob byte cap (default 1 MiB) so a single sender
cannot fill a recipient's queue.
`[tests: packages/shade-inbox-server/tests/routes.test.ts — "rejects ciphertext > maxBlobBytes", "enforces per-address quota"]`
- Per-IP token-bucket on inbox PUT/FETCH/DELETE/REGISTER routes.
**NOT mitigated:** **NOT mitigated:**
- Application-level DDoS at the network layer is your hosting platform's responsibility. - Application-level DDoS at the network layer is your hosting platform's responsibility.
### 8. Social-recovery adversaries (V3.10)
Once a user has set up `@shade/recovery`, the guardian set becomes a
new attack surface. We split the threat into four cases:
**8a. Coalition of ≤ k-1 guardians.**
**Mitigations:**
- Shamir Secret Sharing over GF(2^8) is information-theoretically
secure: the shares are points on a polynomial whose constant term
is the secret, and any subset of `< k` points is consistent with
every possible secret. No coalition smaller than the threshold
recovers anything beyond the secret's length.
`[tests: packages/shade-recovery/tests/shamir.test.ts — "k-1 shares yield a wrong (random-looking) result", "property: any k-1 share subset yields a different output than the secret"; packages/shade-recovery/tests/adversarial.test.ts — "property: any (k-1) subset of shares fails to recover the key"]`
**8b. Single malicious guardian who forges a share.**
**Mitigations:**
- The reconstructed `recoveryKey` is authenticated by the AES-GCM
tag inside the backup blob (`Shade.exportBackup`'s ciphertext).
A forged share produces a different reconstructed key; AES-GCM
decryption fails.
- `requestRecovery` exhaustively tries every threshold-sized subset
of received grants until one authenticates; if none do, it raises
`RecoveryReconstructionError` and refuses to apply the result.
The user is told that at least one guardian is malicious.
`[tests: packages/shade-recovery/tests/adversarial.test.ts — "a corrupted share never authenticates against the backup AEAD tag"]`
**8c. Social-engineering (impersonator calls a guardian).**
**Mitigations:**
- The guardian's `approve` callback receives the new device's
TEMPORARY safety number; the spec REQUIRES out-of-band
comparison before approving.
- The shipped `<RecoveryApprove />` widget enforces a two-checkbox
gate ("fingerprint matches" + "I verified OOB") before the
release button is enabled.
- The protocol-level `share-decline` envelope is sent regardless of
whether the guardian's `approve` callback returns false or
throws, so a hard "no" terminates the requesting flow promptly.
`[tests: packages/shade-recovery/tests/adversarial.test.ts — "approve handler that REJECTS a wrong fingerprint never sends a grant", "throwing approve handler counts as decline with descriptive reason"]`
**NOT mitigated:**
- A guardian who is duped by an impersonator AND whose user clicks
through both checkboxes WILL release their share. Defense in
depth requires user education + per-guardian cool-down windows
(a follow-up release).
**8d. Guardian device compromise.**
If an attacker fully owns a guardian's device, they can:
- Read the share + backup blob → contributes one polynomial point.
- Ship `share-grant` envelopes if they convince the guardian's
`approve` callback to return true.
**Mitigations:**
- No single guardian's compromise is sufficient — the threshold
invariant still holds: the attacker needs `k-1` other shares to
rebuild the identity.
- Backup blobs are encrypted at-rest under the guardian's existing
StorageProvider scheme (V3.2 covers this for SQLite/Postgres
backends).
**NOT mitigated:**
- Compromise of `≥ k` guardians simultaneously is a complete break.
This is by design: the recovery flow is meant to survive *device*
loss, not coordinated mass compromise of the social graph.
### 9. Cross-sender X3DH state corruption
Before V3.10, `initReceiverSession` shared a reference to the
receiver's signed prekey keypair with the new session. The first DH
ratchet step zeroed the session's "previous" private key, which
silently zeroed the persisted signed prekey. A second X3DH from a
*different* sender to the same receiver then derived a divergent
root key and decryption failed with "wrong key or tampered data".
This was a pre-existing bug surfaced by the V3.10 multi-sender
recovery flow.
**Mitigations:**
- `initReceiverSession` now copies the localDHKeyPair into the
session so the eventual zeroize touches a scratch buffer, not
the persisted prekey.
`[tests: packages/shade-recovery/tests/integration.test.ts — "recovery from new device with all 5 guardians available"; packages/shade-core/tests/x3dh.test.ts]`
### 10. MITM bypass via skipped fingerprint verification (V3.3)
The strongest mitigation for §1 / §2 / §6 — out-of-band safety-number
verification — is a *user* responsibility. Shade 4.0 ships
`@shade/sdk` fingerprint gates that move it from "convention" to
"enforced policy on the operations that matter".
**Mitigations:**
- `Shade.beforeFirstLargeFile(threshold, handler)` — runs in `upload()`
when payload ≥ threshold (default 10 MiB) and the peer is unverified.
A handler that returns `false` (or throws / is missing in policy-
forbid-TOFU mode) raises `FingerprintNotVerifiedError` (HTTP 403).
`[tests: packages/shade-sdk/tests/fingerprint-gates.test.ts; packages/shade-files/tests/security/fingerprint-gate.test.ts]`
- `Shade.beforeBackupImport(handler)` — receives the *backup-embedded*
fingerprint before any state is written. Decrypted backups whose
embedded identity does not match the user's expectation are
rejected before they touch storage.
`[tests: packages/shade-sdk/tests/fingerprint-gates.test.ts]`
- `Shade.beforeNewDeviceTrust(handler)` — runs from
`Shade.acceptIdentityChange()` after the peer's identity-version is
bumped, so any prior verification automatically goes stale and the
user must re-verify.
`[tests: packages/shade-sdk/tests/fingerprint-gates.test.ts]`
- `markPeerVerified` / `isPeerVerified` / `unmarkPeerVerified` are
storage-backed; the `peer_verifications` + `peer_identity_versions`
tables are subject to V3.2 at-rest encryption when the encrypted
storage backend is used.
- `<FingerprintCompare />` and `<FingerprintGate />` widgets present
the safety number side-by-side and require an explicit "matches"
click before children render.
**NOT mitigated:**
- Apps that never register handlers default to "TOFU + warning". The
warning is logged, not rendered, so a UX that ignores the log
silently keeps TOFU semantics.
- Once verified, a peer's persisted verification stays valid until
identity rotation. A device-compromise that does **not** trigger
rotation keeps the verification alive.
### 11. WebRTC peer-to-peer transport (V3.11)
`@shade/transport-webrtc` lets two peers ship `@shade/transfer` chunks
over an `RTCDataChannel` instead of HTTP. The DTLS layer is opaque to
Shade; we treat WebRTC strictly as a **byte-pipe** — not a trust
boundary.
**Mitigations:**
- The same Double Ratchet that authenticates Shade messages
authenticates the SDP offer / answer / ICE / bye signaling
envelopes. A network attacker who replaces an SDP offer must
forge a ratcheted message — the receiver decrypts via the
existing peer session and rejects on AEAD failure.
`[tests: packages/shade-transport-webrtc/tests/signaling.test.ts; packages/shade-sdk/tests/webrtc-integration.test.ts]`
- Frame payloads on the DataChannel are AES-GCM-sealed by `@shade/streams`
with deterministic nonce + AAD bound to `streamId || laneId || seq ||
isLast`. A WebRTC implementation that returns altered bytes fails
AEAD verification and the receiver raises `StreamDecryptionError`.
`[tests: packages/shade-streams/tests/tamper.test.ts; packages/shade-transport-webrtc/tests/wire-format.test.ts]`
- Glare resolution is deterministic (lexicographic address compare)
so both sides converge on a single connection without re-running
signaling.
`[tests: packages/shade-transport-webrtc/tests/glare.test.ts]`
- When NAT traversal fails, `MultiTransportFallback([webrtc, http])`
demotes to HTTP within the configured `connectTimeoutMs` (default
5 s) without losing chunks already in flight. No silent stall.
`[tests: packages/shade-sdk/tests/webrtc-failover.test.ts]`
- `IRtcFactory` is pluggable; production uses
`globalThis.RTCPeerConnection` (browser / Workers / Deno),
`MemoryRtcFactory` is in-process for tests.
**NOT mitigated:**
- TURN relay metadata. If the deployment ships a TURN server,
the operator sees relayed-byte counts and timing for every flow
that traverses the relay. Use a TURN you control or a hosted
relay you trust.
- Browser/RTC stack vulnerabilities. A compromised
`RTCPeerConnection` implementation is outside the scope of a JS
library; we ride the platform's WebRTC.
- Public STUN exposes the client's public IP to the STUN server.
This is unavoidable without a privacy-preserving NAT discovery
mechanism (out of scope).
### 12. Web-Worker thread boundary (V3.8)
`@shade/crypto-web/worker` runs AEAD, HKDF, HMAC, X25519, Ed25519, and
per-lane stream state inside a dedicated Web Worker so the main thread
never holds key material for very long.
**Mitigations:**
- Lane keys, identity private keys and ratchet chain keys are passed
into the worker once at setup; subsequent operations move plaintext
via transferable `ArrayBuffer`s and never re-export keys.
`[tests: packages/shade-crypto-web/tests/worker-streams.test.ts; packages/shade-crypto-web/tests/worker-provider.test.ts]`
- Idle timeout (default 30 s) calls `terminate()` on the worker, which
drops the global JS heap and releases the OS-level memory backing
any keys that were not yet zeroized.
- `rotate()` and `destroy()` lifecycle controls let apps bound the
worst-case duration any lane key sits in worker memory.
- Worker-protocol version handshake on first message rejects mismatched
workers (e.g. cached old build).
**NOT mitigated:**
- The worker is still inside the same browsing context; an attacker
who can inject script into the page can post a malicious message
and read the worker's reply. CSP and SRI on the worker entrypoint
are the user's responsibility.
- Heap memory is not synchronously wiped when `postMessage` returns
ownership; the runtime may keep deallocated buffers around for
GC. Memory zeroization is best-effort for both threads.
## Assumptions ## Assumptions
1. **The user has a secure way to bootstrap trust.** Either: 1. **The user has a secure way to bootstrap trust.** Either:
@@ -92,8 +396,10 @@ Attacker floods the prekey server to exhaust resources or one-time prekeys.
| Risk | Severity | Mitigation | | Risk | Severity | Mitigation |
|------|----------|------------| |------|----------|------------|
| MITM at first session establishment | High | Compare safety numbers out-of-band | | MITM at first session establishment | High | Compare safety numbers out-of-band; in 4.0, register `Shade.beforeFirstLargeFile` / `beforeBackupImport` / `beforeNewDeviceTrust` to enforce verification on the operations that matter (V3.3) |
| Identity private key theft from device | Critical | Filesystem encryption, secure enclave (future) | | Identity private key theft from device | Critical | Filesystem encryption, secure enclave (future); V3.10 Social Recovery for *recovery* after loss |
| Prekey server operator runs a "key oracle" attack | Medium | Distributed/federated prekey servers (future) | | Prekey server operator runs a "key oracle" attack | Medium | V3.12 Key Transparency (opt-in) detects split-view + history rewrites; gossip via a `LightWitness` raises the cost of a sustained attack |
| Side-channel via JIT timing variability | Low | Constant-time primitives reduce but don't eliminate | | TURN relay sees byte-counts of P2P transfers | LowMedium | Only when WebRTC fails over to TURN. Operate your own TURN if the metadata matters |
| Side-channel via JIT timing variability | Low | Constant-time primitives reduce but don't eliminate; V3.8 Web-Worker isolation bounds the lifetime of in-memory key material |
| Metadata visibility to prekey server | Low | Acceptable for most use cases; mix networks for stronger metadata protection | | Metadata visibility to prekey server | Low | Acceptable for most use cases; mix networks for stronger metadata protection |
| Inbox relay sees recipient address + byte-counts | LowMedium | Use address-hashes + per-session sender keys (V3.6 §6); mix-net relay tier is a future candidate |

12
android/.gitignore vendored Normal file
View File

@@ -0,0 +1,12 @@
# Gradle
.gradle/
build/
!gradle/wrapper/gradle-wrapper.jar
# IntelliJ / Android Studio
.idea/
*.iml
local.properties
# Captured logs
*.log

5
android/build.gradle.kts Normal file
View File

@@ -0,0 +1,5 @@
plugins {
kotlin("jvm") version "2.0.20" apply false
kotlin("android") version "2.0.20" apply false
id("com.android.library") version "8.7.3" apply false
}

View File

@@ -0,0 +1,6 @@
org.gradle.jvmargs=-Xmx2g -Dfile.encoding=UTF-8
org.gradle.parallel=true
org.gradle.caching=true
kotlin.code.style=official
android.useAndroidX=true
android.nonTransitiveRClass=true

Binary file not shown.

View File

@@ -0,0 +1,7 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-8.10.2-bin.zip
networkTimeout=10000
validateDistributionUrl=true
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists

252
android/gradlew vendored Executable file
View File

@@ -0,0 +1,252 @@
#!/bin/sh
#
# Copyright © 2015-2021 the original authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# SPDX-License-Identifier: Apache-2.0
#
##############################################################################
#
# Gradle start up script for POSIX generated by Gradle.
#
# Important for running:
#
# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is
# noncompliant, but you have some other compliant shell such as ksh or
# bash, then to run this script, type that shell name before the whole
# command line, like:
#
# ksh Gradle
#
# Busybox and similar reduced shells will NOT work, because this script
# requires all of these POSIX shell features:
# * functions;
# * expansions «$var», «${var}», «${var:-default}», «${var+SET}»,
# «${var#prefix}», «${var%suffix}», and «$( cmd )»;
# * compound commands having a testable exit status, especially «case»;
# * various built-in commands including «command», «set», and «ulimit».
#
# Important for patching:
#
# (2) This script targets any POSIX shell, so it avoids extensions provided
# by Bash, Ksh, etc; in particular arrays are avoided.
#
# The "traditional" practice of packing multiple parameters into a
# space-separated string is a well documented source of bugs and security
# problems, so this is (mostly) avoided, by progressively accumulating
# options in "$@", and eventually passing that to Java.
#
# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS,
# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly;
# see the in-line comments for details.
#
# There are tweaks for specific operating systems such as AIX, CygWin,
# Darwin, MinGW, and NonStop.
#
# (3) This script is generated from the Groovy template
# https://github.com/gradle/gradle/blob/HEAD/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
# within the Gradle project.
#
# You can find Gradle at https://github.com/gradle/gradle/.
#
##############################################################################
# Attempt to set APP_HOME
# Resolve links: $0 may be a link
app_path=$0
# Need this for daisy-chained symlinks.
while
APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path
[ -h "$app_path" ]
do
ls=$( ls -ld "$app_path" )
link=${ls#*' -> '}
case $link in #(
/*) app_path=$link ;; #(
*) app_path=$APP_HOME$link ;;
esac
done
# This is normally unused
# shellcheck disable=SC2034
APP_BASE_NAME=${0##*/}
# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s
' "$PWD" ) || exit
# Use the maximum available, or set MAX_FD != -1 to use that value.
MAX_FD=maximum
warn () {
echo "$*"
} >&2
die () {
echo
echo "$*"
echo
exit 1
} >&2
# OS specific support (must be 'true' or 'false').
cygwin=false
msys=false
darwin=false
nonstop=false
case "$( uname )" in #(
CYGWIN* ) cygwin=true ;; #(
Darwin* ) darwin=true ;; #(
MSYS* | MINGW* ) msys=true ;; #(
NONSTOP* ) nonstop=true ;;
esac
CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
# Determine the Java command to use to start the JVM.
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD=$JAVA_HOME/jre/sh/java
else
JAVACMD=$JAVA_HOME/bin/java
fi
if [ ! -x "$JAVACMD" ] ; then
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
else
JAVACMD=java
if ! command -v java >/dev/null 2>&1
then
die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
fi
# Increase the maximum file descriptors if we can.
if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
case $MAX_FD in #(
max*)
# In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC2039,SC3045
MAX_FD=$( ulimit -H -n ) ||
warn "Could not query maximum file descriptor limit"
esac
case $MAX_FD in #(
'' | soft) :;; #(
*)
# In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC2039,SC3045
ulimit -n "$MAX_FD" ||
warn "Could not set maximum file descriptor limit to $MAX_FD"
esac
fi
# Collect all arguments for the java command, stacking in reverse order:
# * args from the command line
# * the main class name
# * -classpath
# * -D...appname settings
# * --module-path (only if needed)
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables.
# For Cygwin or MSYS, switch paths to Windows format before running java
if "$cygwin" || "$msys" ; then
APP_HOME=$( cygpath --path --mixed "$APP_HOME" )
CLASSPATH=$( cygpath --path --mixed "$CLASSPATH" )
JAVACMD=$( cygpath --unix "$JAVACMD" )
# Now convert the arguments - kludge to limit ourselves to /bin/sh
for arg do
if
case $arg in #(
-*) false ;; # don't mess with options #(
/?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath
[ -e "$t" ] ;; #(
*) false ;;
esac
then
arg=$( cygpath --path --ignore --mixed "$arg" )
fi
# Roll the args list around exactly as many times as the number of
# args, so each arg winds up back in the position where it started, but
# possibly modified.
#
# NB: a `for` loop captures its iteration list before it begins, so
# changing the positional parameters here affects neither the number of
# iterations, nor the values presented in `arg`.
shift # remove old arg
set -- "$@" "$arg" # push replacement arg
done
fi
# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
# Collect all arguments for the java command:
# * DEFAULT_JVM_OPTS, JAVA_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
# and any embedded shellness will be escaped.
# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be
# treated as '${Hostname}' itself on the command line.
set -- \
"-Dorg.gradle.appname=$APP_BASE_NAME" \
-classpath "$CLASSPATH" \
org.gradle.wrapper.GradleWrapperMain \
"$@"
# Stop when "xargs" is not available.
if ! command -v xargs >/dev/null 2>&1
then
die "xargs is not available"
fi
# Use "xargs" to parse quoted args.
#
# With -n1 it outputs one arg per line, with the quotes and backslashes removed.
#
# In Bash we could simply go:
#
# readarray ARGS < <( xargs -n1 <<<"$var" ) &&
# set -- "${ARGS[@]}" "$@"
#
# but POSIX shell has neither arrays nor command substitution, so instead we
# post-process each arg (as a line of input to sed) to backslash-escape any
# character that might be a shell metacharacter, then use eval to reverse
# that process (while maintaining the separation between arguments), and wrap
# the whole thing up as a single "set" statement.
#
# This will of course break if any of these variables contains a newline or
# an unmatched quote.
#
eval "set -- $(
printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" |
xargs -n1 |
sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' |
tr '\n' ' '
)" '"$@"'
exec "$JAVACMD" "$@"

94
android/gradlew.bat vendored Normal file
View File

@@ -0,0 +1,94 @@
@rem
@rem Copyright 2015 the original author or authors.
@rem
@rem Licensed under the Apache License, Version 2.0 (the "License");
@rem you may not use this file except in compliance with the License.
@rem You may obtain a copy of the License at
@rem
@rem https://www.apache.org/licenses/LICENSE-2.0
@rem
@rem Unless required by applicable law or agreed to in writing, software
@rem distributed under the License is distributed on an "AS IS" BASIS,
@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@rem See the License for the specific language governing permissions and
@rem limitations under the License.
@rem
@rem SPDX-License-Identifier: Apache-2.0
@rem
@if "%DEBUG%"=="" @echo off
@rem ##########################################################################
@rem
@rem Gradle startup script for Windows
@rem
@rem ##########################################################################
@rem Set local scope for the variables with windows NT shell
if "%OS%"=="Windows_NT" setlocal
set DIRNAME=%~dp0
if "%DIRNAME%"=="" set DIRNAME=.
@rem This is normally unused
set APP_BASE_NAME=%~n0
set APP_HOME=%DIRNAME%
@rem Resolve any "." and ".." in APP_HOME to make it shorter.
for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi
@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m"
@rem Find java.exe
if defined JAVA_HOME goto findJavaFromJavaHome
set JAVA_EXE=java.exe
%JAVA_EXE% -version >NUL 2>&1
if %ERRORLEVEL% equ 0 goto execute
echo. 1>&2
echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2
echo. 1>&2
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
echo location of your Java installation. 1>&2
goto fail
:findJavaFromJavaHome
set JAVA_HOME=%JAVA_HOME:"=%
set JAVA_EXE=%JAVA_HOME%/bin/java.exe
if exist "%JAVA_EXE%" goto execute
echo. 1>&2
echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2
echo. 1>&2
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
echo location of your Java installation. 1>&2
goto fail
:execute
@rem Setup the command line
set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
@rem Execute Gradle
"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %*
:end
@rem End local scope for the variables with windows NT shell
if %ERRORLEVEL% equ 0 goto mainEnd
:fail
rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
rem the _cmd.exe /c_ return code!
set EXIT_CODE=%ERRORLEVEL%
if %EXIT_CODE% equ 0 set EXIT_CODE=1
if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE%
exit /b %EXIT_CODE%
:mainEnd
if "%OS%"=="Windows_NT" endlocal
:omega

View File

@@ -0,0 +1,23 @@
rootProject.name = "shade-kotlin"
pluginManagement {
repositories {
gradlePluginPortal()
mavenCentral()
google()
}
}
dependencyResolutionManagement {
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
repositories {
mavenCentral()
google()
}
}
include(":shade-android")
project(":shade-android").projectDir = file("shade-android")
include(":shade-android-keystore")
project(":shade-android-keystore").projectDir = file("shade-android-keystore")

View File

@@ -0,0 +1,68 @@
# shade-android-keystore
Android-specific bindings for `shade-android`. Lives as a sibling Gradle module so the JVM-only protocol code can keep running in CI without an Android SDK install.
Provides:
- **`KeystoreMasterKey`** — hardware-backed AES-256-GCM master key in the Android Keystore. Optionally biometric-gated (BIOMETRIC_STRONG only — Class 3 assurance), StrongBox-backed when available, invalidated on new biometric enrollment.
- **`BiometricUnlock`** — coroutine wrapper around `BiometricPrompt` for unlocking a `Cipher` instance bound to the keystore key. Throws `BiometricCancelledException` / `BiometricFailedException` so callers can handle the auth flow without writing custom callbacks.
- **`KeystoreStorage`** — `StorageProvider` implementation that persists session/identity/prekey state to `SharedPreferences`, each row encrypted under the keystore key with the row's preference key bound as AAD.
## Usage
```kotlin
import androidx.fragment.app.FragmentActivity
import no.zyon.shade.ShadeSessionManager
import no.zyon.shade.crypto.TinkProvider
import no.zyon.shade.keystore.BiometricUnlock
import no.zyon.shade.keystore.KeystoreStorage
class MyActivity : FragmentActivity() {
private val crypto = TinkProvider()
private lateinit var storage: KeystoreStorage
private lateinit var manager: ShadeSessionManager
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
storage = KeystoreStorage(this, crypto)
lifecycleScope.launch {
val unlock = BiometricUnlock(
activity = this@MyActivity,
title = "Unlock Shade",
subtitle = "Tap your fingerprint to access your messages",
)
try {
storage.unlock(unlock)
} catch (e: BiometricCancelledException) {
// user backed out — show a "tap to retry" UI
return@launch
}
manager = ShadeSessionManager(crypto, storage)
manager.initialize()
// ... use manager normally
}
}
}
```
For credential-driven bootstrap (V4.9 profile + V4.10 approval), pair this with `no.zyon.shade.blob.createProfileNamespace` and `no.zyon.shade.approval.signProxyApproval` — both pure-JVM (in `:shade-android`).
## Threat model
- **Compromised app process**: cannot read the AES key (it's in the secure environment). Can attempt to use the cipher only after the user has authenticated; biometric re-prompts are required after each biometric event.
- **Stolen device with known PIN**: cannot unlock — `setUserAuthenticationParameters(0, BIOMETRIC_STRONG)` excludes `DEVICE_CREDENTIAL`.
- **Attacker enrolls own biometric**: `setInvalidatedByBiometricEnrollment(true)` invalidates the key on enrollment, forcing a credential rebootstrap (which would need username + password + PIN).
- **Catastrophic recovery**: `forgetEverything()` deletes the master key and clears the SharedPreferences. Pair with `Profile.delete()` for full account erasure.
## Build
Requires an Android SDK. The Gradle build uses Android Gradle Plugin 8.7+, AGP minSdk 28 (Pie+ for BiometricPrompt baseline), targetSdk 35.
```bash
JAVA_HOME=/path/to/jdk-21 ./gradlew :shade-android-keystore:assembleDebug
```
Unit tests: none yet — `KeystoreStorage` requires Android runtime. Robolectric or instrumented tests against an emulator are tracked as a follow-up. The pure-JVM `SessionStateJson` round-trip serializer is tested in `:shade-android` (`SessionStateJsonTest`).

View File

@@ -0,0 +1,59 @@
plugins {
id("com.android.library")
kotlin("android")
}
// V4.10 — Android-specific KeystoreStorage adapter.
//
// Lives as a sibling module to `:shade-android` so the JVM-only
// protocol code can keep running in CI without an Android SDK.
// This module pulls in `:shade-android` for `StorageProvider`,
// `IdentityKeyPair`, etc., and binds those types to a hardware-
// backed Android Keystore master key with biometric gating.
android {
namespace = "no.zyon.shade.keystore"
compileSdk = 35
defaultConfig {
minSdk = 28 // BiometricPrompt + StrongBox baseline
}
compileOptions {
sourceCompatibility = JavaVersion.VERSION_17
targetCompatibility = JavaVersion.VERSION_17
}
kotlinOptions {
jvmTarget = "17"
}
buildTypes {
release {
isMinifyEnabled = false
}
}
testOptions {
unitTests.isReturnDefaultValues = true
}
}
dependencies {
// Sibling: protocol types + StorageProvider interface.
api(project(":shade-android"))
// androidx.biometric — fragment-safe BiometricPrompt wrapper.
// 1.2.0-alpha05 is the latest with stable BiometricPrompt API.
implementation("androidx.biometric:biometric:1.2.0-alpha05")
// androidx.fragment — BiometricPrompt requires FragmentActivity.
implementation("androidx.fragment:fragment-ktx:1.8.5")
// Coroutines for the suspend-function StorageProvider implementation.
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.9.0")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.9.0")
testImplementation("junit:junit:4.13.2")
testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.9.0")
}

View File

@@ -0,0 +1,2 @@
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android" />

View File

@@ -0,0 +1,112 @@
package no.zyon.shade.keystore
import androidx.biometric.BiometricManager
import androidx.biometric.BiometricPrompt
import androidx.fragment.app.FragmentActivity
import javax.crypto.Cipher
import kotlin.coroutines.resume
import kotlin.coroutines.resumeWithException
import kotlinx.coroutines.suspendCancellableCoroutine
/**
* Biometric unlock for a `KeystoreMasterKey`-bound `Cipher`.
*
* The Android keystore enforces that any operation on a
* user-authentication-required key must happen via a
* `BiometricPrompt.CryptoObject`-wrapped `Cipher`. The user sees a
* system biometric prompt; on success the same `Cipher` instance is
* usable for one operation (or one streaming session) before
* needing to re-prompt.
*
* This is a thin coroutine wrapper around `BiometricPrompt` that
* resolves to the authenticated cipher or throws on user
* cancellation. Callers typically run it once at app start to
* unlock the master key for the lifetime of the foreground session.
*/
class BiometricUnlock(
private val activity: FragmentActivity,
private val title: String,
private val subtitle: String? = null,
private val negativeButton: String = "Cancel",
) {
/**
* True if BIOMETRIC_STRONG is currently usable on this device.
* False means the user has no enrolled fingerprint/face that
* meets the class-3 assurance level — fall back to a credential
* recovery flow rather than crashing.
*/
fun canAuthenticate(): Boolean {
val mgr = BiometricManager.from(activity)
return mgr.canAuthenticate(BiometricManager.Authenticators.BIOMETRIC_STRONG) ==
BiometricManager.BIOMETRIC_SUCCESS
}
/**
* Show the biometric prompt and return the authenticated cipher.
*
* Cancellation paths:
* - User taps the negative button → throws `BiometricCancelledException`.
* - System errors out (e.g. too many failures) → throws
* `BiometricFailedException` with the system error code.
*/
suspend fun unlock(cipher: Cipher): Cipher = suspendCancellableCoroutine { cont ->
val callback = object : BiometricPrompt.AuthenticationCallback() {
override fun onAuthenticationSucceeded(result: BiometricPrompt.AuthenticationResult) {
val authedCipher = result.cryptoObject?.cipher
if (authedCipher == null) {
cont.resumeWithException(
BiometricFailedException(-1, "BiometricPrompt returned no cipher"),
)
} else {
cont.resume(authedCipher)
}
}
override fun onAuthenticationError(errorCode: Int, errString: CharSequence) {
if (errorCode == BiometricPrompt.ERROR_USER_CANCELED ||
errorCode == BiometricPrompt.ERROR_NEGATIVE_BUTTON ||
errorCode == BiometricPrompt.ERROR_CANCELED
) {
cont.resumeWithException(BiometricCancelledException(errString.toString()))
} else {
cont.resumeWithException(
BiometricFailedException(errorCode, errString.toString()),
)
}
}
override fun onAuthenticationFailed() {
// A single failed attempt — the prompt stays open and
// gives the user another try. Don't resume the
// continuation; let the system flow continue.
}
}
val prompt = BiometricPrompt(
activity,
activity.mainExecutor,
callback,
)
val info = BiometricPrompt.PromptInfo.Builder()
.setTitle(title)
.apply { if (subtitle != null) setSubtitle(subtitle) }
.setNegativeButtonText(negativeButton)
.setAllowedAuthenticators(BiometricManager.Authenticators.BIOMETRIC_STRONG)
.build()
cont.invokeOnCancellation { prompt.cancelAuthentication() }
prompt.authenticate(info, BiometricPrompt.CryptoObject(cipher))
}
}
/** User cancelled the biometric prompt. */
class BiometricCancelledException(message: String) : RuntimeException(message)
/**
* BiometricPrompt returned a non-cancellation error (lockout, hardware
* unavailable, no enrolled biometrics, etc.). Inspect `errorCode`
* against `BiometricPrompt.ERROR_*` constants to decide UX response.
*/
class BiometricFailedException(val errorCode: Int, message: String) :
RuntimeException("[$errorCode] $message")

View File

@@ -0,0 +1,172 @@
package no.zyon.shade.keystore
import android.os.Build
import android.security.keystore.KeyGenParameterSpec
import android.security.keystore.KeyProperties
import javax.crypto.Cipher
import javax.crypto.KeyGenerator
import javax.crypto.SecretKey
import javax.crypto.spec.GCMParameterSpec
import java.security.KeyStore
/**
* Hardware-backed AES-256-GCM master key in the Android Keystore.
*
* The key never leaves the secure environment — Android's keystore
* implementation enforces that all encrypt/decrypt operations
* happen inside the TEE (or StrongBox if present), and the raw
* key bytes are never returned to userspace.
*
* The key is created on first use with these properties:
*
* - AES-256-GCM, no padding
* - User authentication required: opt-in via the `requireBiometric`
* flag. When true, every encrypt/decrypt operation must be wrapped
* in a `BiometricPrompt.authenticate(CryptoObject(cipher))` call
* that succeeds within the same `Cipher` instance.
* - StrongBox-backed if available (Pixel 3+, most Samsung flagships).
* Falls back to TEE on devices without StrongBox.
* - InvalidatedByBiometricEnrollment(true): a newly enrolled
* fingerprint/face invalidates the key, forcing the user to
* re-bootstrap from credentials. Defends against a thief who
* enrolls their own biometric.
*
* Mirrors the role `KeyManager` plays in `@shade/storage-encrypted`'s
* V4.5 KDF chain: this is the *encryption-at-rest* master key, not
* the X3DH identity key. The Shade protocol's identity keys are
* stored encrypted under THIS key.
*/
class KeystoreMasterKey(
private val alias: String,
private val requireBiometric: Boolean = true,
) {
init {
require(alias.isNotEmpty()) { "alias must be non-empty" }
}
/**
* Build a `Cipher` initialized for encryption with the master key.
*
* If the key requires user auth, the returned cipher is *not yet
* usable* — the caller MUST wrap it in a
* `BiometricPrompt.authenticate(CryptoObject(cipher))` and use
* the cipher exposed by the auth-success callback. Calling
* `cipher.doFinal(...)` before authentication throws
* `UserNotAuthenticatedException`.
*/
fun cipherForEncrypt(): Cipher {
val key = getOrCreateKey()
val cipher = Cipher.getInstance(TRANSFORMATION)
cipher.init(Cipher.ENCRYPT_MODE, key)
return cipher
}
/**
* Build a `Cipher` initialized for decryption with the master key
* and a previously-stored 12-byte nonce. Same authentication
* requirement as `cipherForEncrypt`.
*/
fun cipherForDecrypt(nonce: ByteArray): Cipher {
require(nonce.size == 12) { "GCM nonce must be 12 bytes" }
val key = getOrCreateKey()
val cipher = Cipher.getInstance(TRANSFORMATION)
cipher.init(Cipher.DECRYPT_MODE, key, GCMParameterSpec(128, nonce))
return cipher
}
/** True if the key already exists in the Android Keystore. */
fun exists(): Boolean {
val ks = KeyStore.getInstance(ANDROID_KEYSTORE).apply { load(null) }
return ks.containsAlias(alias)
}
/**
* Delete the master key. Catastrophic — all data encrypted under
* it becomes unrecoverable. Used by the "forget everything" flow
* (paired with `Profile.delete()` in the V4.9 namespace).
*/
fun deleteKey() {
val ks = KeyStore.getInstance(ANDROID_KEYSTORE).apply { load(null) }
if (ks.containsAlias(alias)) ks.deleteEntry(alias)
}
private fun getOrCreateKey(): SecretKey {
val ks = KeyStore.getInstance(ANDROID_KEYSTORE).apply { load(null) }
ks.getEntry(alias, null)?.let { entry ->
return (entry as KeyStore.SecretKeyEntry).secretKey
}
return generateKey()
}
private fun generateKey(): SecretKey {
val builder = KeyGenParameterSpec.Builder(
alias,
KeyProperties.PURPOSE_ENCRYPT or KeyProperties.PURPOSE_DECRYPT,
)
.setBlockModes(KeyProperties.BLOCK_MODE_GCM)
.setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_NONE)
.setKeySize(256)
// Each encrypt operation generates a fresh IV in the secure
// env; we read it back via `cipher.iv` after init.
.setRandomizedEncryptionRequired(true)
if (requireBiometric) {
builder.setUserAuthenticationRequired(true)
// BIOMETRIC_STRONG only — class 3, the highest assurance
// level (Class 3 = false-accept rate < 1/50 000 per BiometricPrompt).
// DEVICE_CREDENTIAL is intentionally NOT included: a stolen
// device with a known PIN should not unlock Shade.
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.R) {
builder.setUserAuthenticationParameters(
/* timeout = */ 0,
KeyProperties.AUTH_BIOMETRIC_STRONG,
)
} else {
@Suppress("DEPRECATION")
builder.setUserAuthenticationValidityDurationSeconds(-1)
}
builder.setInvalidatedByBiometricEnrollment(true)
}
// StrongBox if available — bumps key storage to a dedicated
// tamper-resistant chip on Pixel 3+ / most Samsung flagships.
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) {
builder.setIsStrongBoxBacked(true)
}
val gen = KeyGenerator.getInstance(KeyProperties.KEY_ALGORITHM_AES, ANDROID_KEYSTORE)
return try {
gen.init(builder.build())
gen.generateKey()
} catch (_: Exception) {
// StrongBox not present or full → retry without StrongBox.
// Same for older devices that don't honor
// setUserAuthenticationParameters.
val fallback = KeyGenParameterSpec.Builder(
alias,
KeyProperties.PURPOSE_ENCRYPT or KeyProperties.PURPOSE_DECRYPT,
)
.setBlockModes(KeyProperties.BLOCK_MODE_GCM)
.setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_NONE)
.setKeySize(256)
.setRandomizedEncryptionRequired(true)
.apply {
if (requireBiometric) {
setUserAuthenticationRequired(true)
@Suppress("DEPRECATION")
setUserAuthenticationValidityDurationSeconds(-1)
setInvalidatedByBiometricEnrollment(true)
}
}
.build()
gen.init(fallback)
gen.generateKey()
}
}
companion object {
private const val ANDROID_KEYSTORE = "AndroidKeyStore"
private const val TRANSFORMATION = "AES/GCM/NoPadding"
}
}

View File

@@ -0,0 +1,229 @@
package no.zyon.shade.keystore
import android.content.Context
import android.content.SharedPreferences
import android.util.Base64
import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.withLock
import no.zyon.shade.crypto.CryptoProvider
import no.zyon.shade.serialization.SessionStateJson
import no.zyon.shade.storage.StorageProvider
import no.zyon.shade.types.IdentityKeyPair
import no.zyon.shade.types.OneTimePreKey
import no.zyon.shade.types.SessionState
import no.zyon.shade.types.SignedPreKey
/**
* `StorageProvider` implementation that gates all reads/writes through
* a biometric-locked `KeystoreMasterKey`. Mirrors `MemoryStorage` for
* the API surface but persists state to `SharedPreferences` with each
* row encrypted under the keystore key.
*
* Lifecycle:
*
* 1. App start → construct `KeystoreStorage(context, alias)`.
* 2. `unlock(BiometricUnlock)` runs the system biometric prompt.
* The Android keystore caches the auth state under the key's
* `setUserAuthenticationParameters(0, BIOMETRIC_STRONG)` policy
* until the next biometric event (re-enrollment, etc.).
* 3. While unlocked, `getSession`/`saveSession` etc. work normally.
* 4. `lock()` clears the in-memory unlocked flag so a future
* operation triggers another biometric prompt.
*
* Wire layout per row:
* `<base64(nonce(12))>:<base64(ct||tag)>`
*
* Stored as `String` SharedPreferences entries. AAD = the row's
* preference key (`session:<address>`, `signedPreKey:<id>`, etc.) so
* a substituted-prefs swap fails to open.
*/
class KeystoreStorage(
context: Context,
private val crypto: CryptoProvider,
keyAlias: String = DEFAULT_KEY_ALIAS,
prefsName: String = DEFAULT_PREFS_NAME,
requireBiometric: Boolean = true,
) : StorageProvider {
private val prefs: SharedPreferences =
context.applicationContext.getSharedPreferences(prefsName, Context.MODE_PRIVATE)
private val masterKey = KeystoreMasterKey(keyAlias, requireBiometric = requireBiometric)
private val writeMutex = Mutex()
@Volatile
private var unlocked: Boolean = false
/**
* Unlock the keystore via biometric prompt. Idempotent — calling
* twice without a `lock()` between is a no-op.
*/
suspend fun unlock(unlock: BiometricUnlock) {
if (unlocked) return
// The biometric flow returns an authenticated *encrypt*
// cipher; we discard it after a one-shot probe to confirm
// the master key is reachable. The actual encrypt/decrypt
// ciphers in the I/O path use the authentication state
// established here (Android Keystore caches the auth for
// the user-authentication-required key under the
// `setUserAuthenticationParameters(0, BIOMETRIC_STRONG)`
// policy until the next biometric event).
val probe = masterKey.cipherForEncrypt()
unlock.unlock(probe)
unlocked = true
}
/** Unlock without biometric — only valid for keys constructed with `requireBiometric=false`. */
fun unlockNoBiometric() {
unlocked = true
}
/** Wipe in-memory unlock state. The key itself stays in the keystore. */
fun lock() {
unlocked = false
}
/**
* Catastrophic reset: deletes the master key + all encrypted
* preferences. Used by the "forget everything" / 3-strikes-wipe
* path. The next bootstrap rebuilds from credentials.
*/
fun forgetEverything() {
masterKey.deleteKey()
prefs.edit().clear().apply()
unlocked = false
}
// ─── Identity ──────────────────────────────────────────────
override suspend fun getIdentityKeyPair(): IdentityKeyPair? {
val json = readDecrypted(KEY_IDENTITY) ?: return null
return SessionStateJson.deserializeIdentityKeyPair(json)
}
override suspend fun saveIdentityKeyPair(keyPair: IdentityKeyPair) {
writeEncrypted(KEY_IDENTITY, SessionStateJson.serializeIdentityKeyPair(keyPair))
}
override suspend fun getLocalRegistrationId(): Int {
return readDecrypted(KEY_REGISTRATION_ID)?.toIntOrNull() ?: 0
}
override suspend fun saveLocalRegistrationId(id: Int) {
writeEncrypted(KEY_REGISTRATION_ID, id.toString())
}
// ─── Signed prekeys ────────────────────────────────────────
override suspend fun getSignedPreKey(keyId: Int): SignedPreKey? {
val json = readDecrypted("$KEY_SIGNED_PREKEY:$keyId") ?: return null
return SessionStateJson.deserializeSignedPreKey(json)
}
override suspend fun saveSignedPreKey(key: SignedPreKey) {
writeEncrypted(
"$KEY_SIGNED_PREKEY:${key.keyId}",
SessionStateJson.serializeSignedPreKey(key),
)
}
override suspend fun removeSignedPreKey(keyId: Int) {
writeMutex.withLock { prefs.edit().remove("$KEY_SIGNED_PREKEY:$keyId").apply() }
}
// ─── One-time prekeys ──────────────────────────────────────
override suspend fun getOneTimePreKey(keyId: Int): OneTimePreKey? {
val json = readDecrypted("$KEY_ONETIME_PREKEY:$keyId") ?: return null
return SessionStateJson.deserializeOneTimePreKey(json)
}
override suspend fun saveOneTimePreKey(key: OneTimePreKey) {
writeEncrypted(
"$KEY_ONETIME_PREKEY:${key.keyId}",
SessionStateJson.serializeOneTimePreKey(key),
)
}
override suspend fun removeOneTimePreKey(keyId: Int) {
writeMutex.withLock { prefs.edit().remove("$KEY_ONETIME_PREKEY:$keyId").apply() }
}
override suspend fun getOneTimePreKeyCount(): Int {
return prefs.all.keys.count { it.startsWith("$KEY_ONETIME_PREKEY:") }
}
// ─── Sessions ──────────────────────────────────────────────
override suspend fun getSession(address: String): SessionState? {
val json = readDecrypted("$KEY_SESSION:$address") ?: return null
return SessionStateJson.deserialize(json)
}
override suspend fun saveSession(address: String, state: SessionState) {
writeEncrypted("$KEY_SESSION:$address", SessionStateJson.serialize(state))
}
override suspend fun removeSession(address: String) {
writeMutex.withLock { prefs.edit().remove("$KEY_SESSION:$address").apply() }
}
// ─── Trust ─────────────────────────────────────────────────
override suspend fun isTrustedIdentity(address: String, identityKey: ByteArray): Boolean {
val stored = readDecrypted("$KEY_TRUSTED:$address") ?: return true // TOFU
val storedBytes = Base64.decode(stored, Base64.NO_WRAP)
return crypto.constantTimeEqual(storedBytes, identityKey)
}
override suspend fun saveTrustedIdentity(address: String, identityKey: ByteArray) {
writeEncrypted(
"$KEY_TRUSTED:$address",
Base64.encodeToString(identityKey, Base64.NO_WRAP),
)
}
// ─── Encrypted-row plumbing ────────────────────────────────
private fun ensureUnlocked() {
check(unlocked) {
"KeystoreStorage is locked — call unlock(BiometricUnlock) first"
}
}
private suspend fun readDecrypted(prefKey: String): String? {
ensureUnlocked()
val raw = prefs.getString(prefKey, null) ?: return null
val parts = raw.split(":", limit = 2)
require(parts.size == 2) { "malformed encrypted row at $prefKey" }
val nonce = Base64.decode(parts[0], Base64.NO_WRAP)
val ct = Base64.decode(parts[1], Base64.NO_WRAP)
val cipher = masterKey.cipherForDecrypt(nonce)
cipher.updateAAD(prefKey.toByteArray(Charsets.UTF_8))
return cipher.doFinal(ct).toString(Charsets.UTF_8)
}
private suspend fun writeEncrypted(prefKey: String, plaintext: String) {
ensureUnlocked()
writeMutex.withLock {
val cipher = masterKey.cipherForEncrypt()
cipher.updateAAD(prefKey.toByteArray(Charsets.UTF_8))
val ct = cipher.doFinal(plaintext.toByteArray(Charsets.UTF_8))
val nonce = cipher.iv
val nonceB64 = Base64.encodeToString(nonce, Base64.NO_WRAP)
val ctB64 = Base64.encodeToString(ct, Base64.NO_WRAP)
prefs.edit().putString(prefKey, "$nonceB64:$ctB64").apply()
}
}
companion object {
const val DEFAULT_KEY_ALIAS = "shade-master-v1"
const val DEFAULT_PREFS_NAME = "shade-keystore-storage-v1"
private const val KEY_IDENTITY = "identity"
private const val KEY_REGISTRATION_ID = "registrationId"
private const val KEY_SIGNED_PREKEY = "signedPreKey"
private const val KEY_ONETIME_PREKEY = "oneTimePreKey"
private const val KEY_SESSION = "session"
private const val KEY_TRUSTED = "trusted"
}
}

View File

@@ -0,0 +1,85 @@
# Migrating Nova Android to Shade
This document describes the concrete steps to replace Nova's static AES push
notification encryption with Shade's Signal Protocol ratcheting.
## Current state
**Nova server** (`nova/src/server/services/notifications.ts`):
- Uses a per-device static AES-256-GCM key stored in `pushDevices.encryptionKey`
- Calls `encryptPayload(notificationJson, key)` directly
- Sends via FCM `data: { enc, v: '1' }`
**Nova Android** (`Android/nova-app/.../data/PushKeyStore.kt`):
- Generates the device's AES key once and stores it via EncryptedSharedPreferences
- Decrypts FCM data payload in `NovaFirebaseMessagingService`
- Uses `javax.crypto.Cipher` directly
**Problem:** A single compromised key exposes all past and future notifications.
No forward secrecy, no post-compromise recovery.
## Target state
**Nova server:**
- Uses `@shade/sdk` with `createShade({ prekeyServer, address: 'nova-server' })`
- Per-device Shade sessions stored in PostgreSQL via `@shade/storage-postgres`
- To notify a device: `await shade.send('device:${id}', notificationJson)`
- The envelope is base64-encoded and sent via FCM `data: { enc, v: '2' }`
**Nova Android:**
- Uses `shade-android` (Kotlin) with `ShadeSessionManager`
- Session state stored via `KeystoreStorage` (EncryptedSharedPreferences)
- On FCM receive: decode envelope → `manager.decrypt('nova-server', envelope)`
- First time registration: generate identity, upload prekey bundle to the Shade
prekey server, and tell the Nova backend the device address
## Migration steps
### Phase 1: Dual-write (both work simultaneously)
Add a `v` field to the FCM data payload. Android decrypts v=1 with legacy
`PushKeyStore` and v=2 with Shade. Server can send either. Old devices keep
working while new devices get Shade.
### Phase 2: Switch reads
When 95% of devices have a Shade session established, flip the server to
send v=2 by default. Fall back to v=1 only if the device has no Shade
session.
### Phase 3: Deprecate
Remove v=1 code paths, drop the `pushDevices.encryptionKey` column.
## Smoke test (prove it works end-to-end)
1. TS side creates a Shade instance for `nova-server` (using `@shade/sdk`)
2. TS side calls `shade.send('device:test', '{"title":"Hello"}')`
3. Encode the envelope as base64 → FCM `data.enc`
4. Kotlin side decodes base64 → `WireFormat.decodeEnvelope(bytes)`
5. Kotlin side calls `manager.decrypt('nova-server', envelope)`
6. Assert plaintext matches
This is verified by the cross-platform vector tests + a manual smoke run
described in `examples/07-nova-integration/` (to be added).
## Files to modify
Nova server:
- `nova/src/server/services/notifications.ts` — replace `encryptPayload` with `shade.send`
- `nova/src/server/services/push-devices.ts` — track Shade address per device
- Add `@shade/sdk` to `nova/package.json`
Nova Android:
- `Android/nova-app/app/src/main/java/no/zyon/nova/data/PushKeyStore.kt` — delegate
to `ShadeSessionManager`
- `Android/nova-app/app/src/main/java/no/zyon/nova/NovaFirebaseMessagingService.kt`
call `WireFormat.decodeEnvelope` and `manager.decrypt`
- Add `shade-android` as a Gradle dependency
## Not done in M-Cross 3
Running a full Android Gradle build + instrumented tests is out of scope for
this milestone. The cross-platform vector tests prove byte-for-byte
compatibility; the actual Nova integration happens when the user explicitly
wires up the Android module in their Nova project.

View File

@@ -0,0 +1,76 @@
# shade-android
Kotlin implementation of the Shade E2EE protocol for Android apps. Byte-for-byte compatible with `@shade/core` (TypeScript), so messages encrypted on a TS backend can be decrypted on Android and vice versa.
## Status
**M-Cross 1 ✅** — keys + HKDF + X3DH + fingerprint.
**M-Cross 2 ✅** — full ratchet step (encrypt + decrypt roundtrip) + wire 0x02 (RatchetMessage and PreKeyMessage with/without OTPK).
**M-Cross 3 ✅** — streams 0x11 (KDF labels with embedded NULs, deterministic chunk nonce/AAD, wire 0x11 encode/decode).
**M-Cross 4 ✅** — backup-key HKDF + AES-GCM, group sender-key step (`kdfChainKey` + Ed25519 sign over `aad ‖ ct`), storage HKDF (storageKey/fieldKey/rowNonce).
**M-Cross 5 ✅** — V4.9 blob KDF + AEAD (`deriveBlobSlotId / deriveBlobKey / deriveBlobSigningSeed`, AAD-bound seal/open), `BlobClient` HTTP, `Profile` namespace. Cross-platform vectors in `blob.json`.
**M-Cross 6 ✅** — V4.10 cross-host approval routing: canonical profile-blob schema (`hosts[]` / `clients[]` / `trustedApproverFingerprints[]`), build/sign/verify proxy approvals via `canonicalApprovalSigningBytes` (length-prefixed u16 BE UTF-8). Cross-platform vectors in `approval.json`, including a TS-signed Ed25519 signature that the Kotlin port verifies.
**M-Cross 7 ✅** — scrypt + argon2id password-KDF wrappers (Bouncy Castle), NFKC-normalized inputs.
**M-Cross 8 ✅**`:shade-android-keystore` sibling module: `KeystoreMasterKey` (StrongBox-backed AES-256-GCM, BIOMETRIC_STRONG-gated, invalidated on biometric enrollment), `BiometricUnlock`, `KeystoreStorage` (`StorageProvider` over biometric-gated AES-encrypted SharedPreferences).
Cross-platform test vectors in `/test-vectors/` are loaded by both the TS
and Kotlin test suites; any byte-divergence fails CI within 60 s. See
`ROADMAP-ANDROID.md` for the parity-checkpoint matrix and
`/docs/cross-platform.md` for how to add a new vector.
## Usage (target API)
```kotlin
import no.zyon.shade.ShadeSessionManager
import no.zyon.shade.crypto.TinkProvider
import no.zyon.shade.storage.KeystoreStorage
val crypto = TinkProvider()
val storage = KeystoreStorage(context)
val manager = ShadeSessionManager(crypto, storage)
manager.initialize()
// Establish a session with a peer
val bundle = fetchBundleFromServer("bob@example.com")
manager.initSessionFromBundle("bob@example.com", bundle)
// Encrypt
val envelope = manager.encrypt("bob@example.com", "hello")
// Decrypt
val plaintext = manager.decrypt("alice@example.com", incomingEnvelope)
```
## Crypto primitives
Backed by Google Tink:
- X25519 for Diffie-Hellman (via `X25519.generatePrivateKey()` / `computeSharedSecret`)
- Ed25519 for signing (via `Ed25519Sign` / `Ed25519Verify`)
- AES-256-GCM (via `AesGcmJce`)
- HKDF-SHA256 (via `Hkdf.computeHkdf`)
- HMAC-SHA256 (via `MacFactory`)
## Building
Requires JDK 17. The module compiles as a pure-JVM Kotlin library so the
parity gate runs without an Android SDK install. The Android-specific
storage adapter (Keystore + EncryptedSharedPreferences) will land as a
sibling Gradle module in M-Cross 4.
```bash
cd android
./gradlew :shade-android:test
```
The Gradle wrapper downloads Gradle 8.10.2 on first run.
## Compatibility
The Kotlin implementation must produce byte-identical output to `@shade/core` for:
- KDF chain derivations (root key ratchet, chain key ratchet)
- X3DH shared secrets
- Ratchet message keys and ciphertext (given the same keys)
- Fingerprints (safety numbers)
- Binary wire format (`@shade/proto`)
Shared test vectors in `test-vectors/` are loaded by both the TS and Kotlin test suites. Any divergence fails the CI immediately.

View File

@@ -0,0 +1,137 @@
# Shade Android — Roadmap & Parity Status
This document tracks the M-Cross milestones from `docs/V3.5.md` and the
status of every cross-platform parity sjekkpunkt. The Kotlin port must be
**byte-for-byte compatible** with the TypeScript implementation; this is
verified continuously by `test-vectors/*.json` consumed by both runners.
> **No "production" label** is allowed on Android until M-Cross 2 is green
> (ratchet + wire 0x02 + storage encryption) and M-Cross 3 is green
> (streams 0x11). See `docs/V3.5.md` §Akseptansekriterier.
## Milestones
### M-Cross 1 — Scaffold ✅
Foundation primitives. All passing in CI.
| Sjekkpunkt | Vector | TS test | Kotlin test |
|---|---|---|---|
| 1. KDF chain (root + chain ratchet) | `kdf-chain.json` | ✅ | ✅ |
| 2. HKDF labels | `hkdf.json` | ✅ | ✅ |
| 3. X3DH initial root key (3 + 4 DH outputs) | `x3dh.json` | ✅ | ✅ |
| 5. Fingerprint (60-digit safety number) | `fingerprint.json` | ✅ | ✅ |
### M-Cross 2 — Ratchet & Wire 0x02 ✅
Full ratchet step + binary envelope encoding for both message types.
| Sjekkpunkt | Vector | TS test | Kotlin test |
|---|---|---|---|
| 4. Ratchet step (encrypt deterministic) | `ratchet-step.json` | ✅ | ✅ |
| 4. Ratchet step (decrypt roundtrip) | `ratchet-step.json` | ✅ | ✅ |
| 6. Wire 0x02 RatchetMessage | `wire-format.json` | ✅ | ✅ |
| 6. Wire 0x02 PreKeyMessage (with OTPK) | `wire-format.json` | ✅ | ✅ |
| 6. Wire 0x02 PreKeyMessage (no OTPK, 0xFFFFFFFF marker) | `wire-format.json` | ✅ | ✅ |
The ratchet-step vector exercises every layer that contributes to a
ratchet message's wire bytes: `kdfRootKey``kdfChainKey` → 40-byte header
AAD → AES-256-GCM with deterministic nonce. Both implementations recompute
each layer and compare against the recorded hex. The decrypt half feeds
the recorded ciphertext back through `aesGcmDecrypt(messageKey, nonce, aad)`
and checks the plaintext recovers — proving the AEAD agrees in both
directions.
### M-Cross 3 — Streams 0x11 ✅
Multi-lane chunk encryption (`@shade/streams`) ported. KDF labels with
embedded NULs match TS byte-for-byte; deterministic
`(laneId, seq)`-derived nonces and the 29-byte chunk AAD agree across
runners; wire 0x11 encode/decode is roundtrip-verified.
| Sjekkpunkt | Vector | TS test | Kotlin test |
|---|---|---|---|
| `deriveStreamKey` (HKDF, info `shade-stream/v1\0master`) | `streams.json` | ✅ | ✅ |
| `deriveLaneKey` (HKDF, info `shade-stream/v1\0lane\0` ‖ u32_be laneId) — incl. laneId 0xFFFFFFFF | `streams.json` | ✅ | ✅ |
| `buildChunkNonce(laneId, seq)` — incl. seq = 2^64 - 2 | `streams.json` | ✅ | ✅ |
| `buildChunkAad(streamId, laneId, seq, isLast)` | `streams.json` | ✅ | ✅ |
| Chunk AES-256-GCM encrypt + decrypt (deterministic nonce + AAD) | `streams.json` | ✅ | ✅ |
| Wire 0x11 envelope encode + decode + type-tag inspector | `streams.json` | ✅ | ✅ |
Sequence numbers are unsigned u64 on the wire; the Kotlin port accepts
them as `Long` for the bit pattern (negative-signed-long for values past
2^63 - 1) — this matches the JVM `ByteBuffer.putLong` behavior and the
`java.lang.Long.parseUnsignedLong` JSON-decoder used in tests.
Pending end-to-end interop test (TS server → Kotlin client over an actual
socket) — not gated by vectors but recommended before flipping the
"production" label.
### M-Cross 4 — Backup, Group, Storage HKDF ✅ (cryptographic layer)
The cryptographic primitives that Kotlin needs to share with TS are now
covered. The remaining work is the high-level glue (BackupBlob JSON
schema, full SenderKey/GroupSession state-tracking, Android-Keystore
storage adapter, scrypt password-KDF) — all per-platform plumbing that
doesn't gate vector parity.
| Sjekkpunkt | Vector | TS test | Kotlin test |
|---|---|---|---|
| 7. Backup v1 HKDF (`info="ShadeBackupKey"`) | `backup.json` | ✅ | ✅ |
| 7. Backup v1 AES-GCM roundtrip (no AAD) | `backup.json` | ✅ | ✅ |
| Group sender header AAD (u16/u16/u32 length prefixes) | `group.json` | ✅ | ✅ |
| Group sender-key step: `kdfChainKey` + AES-GCM + Ed25519 sign(aad ‖ ct) | `group.json` | ✅ | ✅ |
| Storage HKDF: `storageKey` (`info="shade-storage-v1"`) | `storage-hkdf.json` | ✅ | ✅ |
| Storage HKDF: `fieldKey` (`info="shade-field-v1:{table}:{column}"`) | `storage-hkdf.json` | ✅ | ✅ |
| Storage HKDF: `rowNonce` (`info="shade-row-nonce-v1:{table}:{pk}"`) | `storage-hkdf.json` | ✅ | ✅ |
Pending sub-tasks (don't gate vector parity):
- **scrypt master-key derivation**: `test-vectors/storage-encryption.json`
pins `scrypt(N=1024, r=8, p=1, dkLen=32)` for unit-test config; Tink
doesn't ship scrypt. Add Bouncy Castle (`org.bouncycastle:bcprov-jdk18on`)
to the Kotlin module, wrap as `CryptoProvider.scrypt(...)`, then a follow-up
vector consumes the full storage-encryption.json end to end.
- **argon2id**: Both backup.ts and the threat-model docs flag HKDF as a
placeholder for a real password KDF. When `argon2id` is added to
`CryptoProvider`, both ports swap together and the backup vector gets
re-pinned.
- **Android KeystoreStorage adapter**: lives in a sibling Android Library
Gradle module that depends on this JVM module. Binds Tink to the Android
Keystore + EncryptedSharedPreferences.
## Build & Test
This module compiles as a **pure-JVM** Kotlin library (`kotlin("jvm")`)
so the parity gate can run without an Android SDK installation in CI.
The protocol code uses `tink:1.15.0` (JVM JAR), `java.nio.ByteBuffer`,
and `javax.crypto` — no `android.*` imports.
The Android-specific storage adapter (KeystoreStorage,
EncryptedSharedPreferences) will land as a sibling Gradle module
(`shade-android-keystore`) in M-Cross 4 and depend on this one.
```bash
# From repo root
cd android
./gradlew :shade-android:test
```
Requires JDK 17. The Gradle wrapper downloads Gradle 8.10.2 on first run.
## Compatibility contract
The Kotlin implementation must produce byte-identical output to the TS
reference for:
- KDF chain derivations (root key ratchet, chain key ratchet)
- X3DH shared secrets (3- and 4-DH variants)
- Ratchet message keys + AES-GCM ciphertext (given the same key/plaintext/AAD/nonce)
- Header AAD encoding (40 bytes: `dhPublicKey(32) || u32_be(prevCounter) || u32_be(counter)`)
- Fingerprints (12 × 5-digit groups)
- Binary wire format 0x02 (RatchetMessage + PreKeyMessage)
- Binary wire format 0x11 (StreamChunk) — M-Cross 3
- Storage encryption KDF chain — M-Cross 4
Each is covered by a vector file in `/test-vectors/`. Adding a new
sjekkpunkt: see `docs/cross-platform.md`.

View File

@@ -0,0 +1,56 @@
plugins {
kotlin("jvm")
`java-library`
}
// V3.5 — Cross-platform parity gate.
//
// This module compiles as a pure-JVM Kotlin library so CI can run the
// cross-platform vector tests without an Android SDK. The protocol code
// is JVM-safe (no `android.*` imports); only Tink + java.* are used.
//
// When KeystoreStorage and EncryptedSharedPreferences-backed adapters land
// (M-Cross 4 + V3.5 §Storage), they will live in a sibling Android Library
// module that depends on this one.
java {
sourceCompatibility = JavaVersion.VERSION_17
targetCompatibility = JavaVersion.VERSION_17
}
kotlin {
compilerOptions {
jvmTarget.set(org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17)
}
}
dependencies {
// Google Tink for X25519, Ed25519, AES-GCM, HMAC, HKDF (JVM build).
// The same `subtle.*` API as `tink-android` so the source compiles unchanged.
implementation("com.google.crypto.tink:tink:1.15.0")
// Bouncy Castle for scrypt + argon2id. Tink doesn't ship password
// KDFs; @shade/storage-encrypted uses @noble/hashes for both. We
// pin to the JDK18-on artifact so it works on JVM 17 + Android.
implementation("org.bouncycastle:bcprov-jdk18on:1.78.1")
// JSON serialization (session state + test-vector loader on JVM).
implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.7.3")
// Coroutines (StorageProvider uses `suspend` functions).
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.9.0")
// org.json — bundled with Android but not present on the JVM classpath.
implementation("org.json:json:20240303")
testImplementation("junit:junit:4.13.2")
testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.9.0")
}
tasks.withType<Test>().configureEach {
useJUnit()
testLogging {
events("passed", "failed", "skipped")
showStandardStreams = false
}
}

View File

@@ -0,0 +1,205 @@
package no.zyon.shade
import no.zyon.shade.crypto.CryptoProvider
import no.zyon.shade.fingerprint.computeFingerprint
import no.zyon.shade.protocol.createPreKeyBundle
import no.zyon.shade.protocol.generateIdentityKeyPair
import no.zyon.shade.protocol.generateOneTimePreKeys
import no.zyon.shade.protocol.generateSignedPreKey
import no.zyon.shade.protocol.initReceiverSession
import no.zyon.shade.protocol.initSenderSession
import no.zyon.shade.protocol.processPreKeyBundle
import no.zyon.shade.protocol.processPreKeyMessage
import no.zyon.shade.protocol.ratchetDecrypt
import no.zyon.shade.protocol.ratchetEncrypt
import no.zyon.shade.storage.StorageProvider
import no.zyon.shade.types.OneTimePreKey
import no.zyon.shade.types.PreKeyBundle
import no.zyon.shade.types.PreKeyMessage
import no.zyon.shade.types.RatchetMessage
import no.zyon.shade.types.ShadeEnvelope
import no.zyon.shade.types.SignedPreKey
/**
* High-level API mirroring @shade/core's ShadeSessionManager.
*
* Handles X3DH + Double Ratchet, persists state via StorageProvider.
*/
class ShadeSessionManager(
private val crypto: CryptoProvider,
private val storage: StorageProvider,
) {
private var identity: no.zyon.shade.types.IdentityKeyPair? = null
private var registrationId: Int = 0
private var currentSignedPreKeyId: Int = 0
// X3DH pending metadata (used for first message after bundle processing)
private val pendingX3DH = mutableMapOf<String, PendingX3DH>()
private data class PendingX3DH(
val ephemeralPublicKey: ByteArray,
val signedPreKeyId: Int,
val preKeyId: Int?,
val identityDHKey: ByteArray,
val registrationId: Int,
)
suspend fun initialize() {
identity = storage.getIdentityKeyPair() ?: run {
val fresh = generateIdentityKeyPair(crypto)
storage.saveIdentityKeyPair(fresh)
fresh
}
registrationId = storage.getLocalRegistrationId()
if (registrationId == 0) {
var id = crypto.randomUint32()
if (id == 0) id = 1
registrationId = id
storage.saveLocalRegistrationId(id)
}
val spk = storage.getSignedPreKey(1)
if (spk == null) {
val fresh = generateSignedPreKey(crypto, identity!!, 1)
storage.saveSignedPreKey(fresh)
currentSignedPreKeyId = 1
} else {
currentSignedPreKeyId = spk.keyId
}
}
fun getPublicIdentity(): Pair<ByteArray, ByteArray> {
val id = identity ?: throw IllegalStateException("Not initialized")
return id.signingPublicKey to id.dhPublicKey
}
suspend fun getIdentityFingerprint(): String {
val id = identity ?: throw IllegalStateException("Not initialized")
return computeFingerprint(crypto, id.signingPublicKey, id.dhPublicKey)
}
suspend fun createPreKeyBundle(): PreKeyBundle {
val id = identity ?: throw IllegalStateException("Not initialized")
val spk = storage.getSignedPreKey(currentSignedPreKeyId)
?: throw IllegalStateException("No signed prekey")
return createPreKeyBundle(registrationId, id, spk)
}
suspend fun generateOneTimePreKeys(count: Int): List<OneTimePreKey> {
val existing = storage.getOneTimePreKeyCount()
val startId = existing + 1
val keys = generateOneTimePreKeys(crypto, startId, count)
for (k in keys) storage.saveOneTimePreKey(k)
return keys
}
suspend fun rotateSignedPreKey(): SignedPreKey {
val id = identity ?: throw IllegalStateException("Not initialized")
val newId = currentSignedPreKeyId + 1
val spk = generateSignedPreKey(crypto, id, newId)
storage.saveSignedPreKey(spk)
currentSignedPreKeyId = newId
return spk
}
suspend fun initSessionFromBundle(address: String, bundle: PreKeyBundle) {
val id = identity ?: throw IllegalStateException("Not initialized")
val x3dhResult = processPreKeyBundle(crypto, id, bundle)
val session = initSenderSession(
crypto,
x3dhResult.rootKey,
x3dhResult.remoteIdentityKey,
x3dhResult.remoteSignedPreKey,
)
storage.saveSession(address, session)
storage.saveTrustedIdentity(address, x3dhResult.remoteIdentityKey)
pendingX3DH[address] = PendingX3DH(
ephemeralPublicKey = x3dhResult.ephemeralPublicKey,
signedPreKeyId = x3dhResult.signedPreKeyId,
preKeyId = x3dhResult.preKeyId,
identityDHKey = id.dhPublicKey,
registrationId = registrationId,
)
}
suspend fun encrypt(address: String, plaintext: ByteArray): ShadeEnvelope {
val session = storage.getSession(address)
?: throw IllegalStateException("No session for $address")
val ratchetMsg = ratchetEncrypt(crypto, session, plaintext)
val pending = pendingX3DH.remove(address)
if (pending != null) {
storage.saveSession(address, session)
val preKeyMsg = PreKeyMessage(
registrationId = pending.registrationId,
preKeyId = pending.preKeyId,
signedPreKeyId = pending.signedPreKeyId,
ephemeralKey = pending.ephemeralPublicKey,
identityDHKey = pending.identityDHKey,
message = ratchetMsg,
)
return ShadeEnvelope(
type = ShadeEnvelope.EnvelopeType.PREKEY,
content = preKeyMsg,
timestamp = System.currentTimeMillis(),
senderAddress = address,
)
}
storage.saveSession(address, session)
return ShadeEnvelope(
type = ShadeEnvelope.EnvelopeType.RATCHET,
content = ratchetMsg,
timestamp = System.currentTimeMillis(),
senderAddress = address,
)
}
suspend fun decrypt(address: String, envelope: ShadeEnvelope): ByteArray {
return when (envelope.type) {
ShadeEnvelope.EnvelopeType.PREKEY -> decryptPreKeyMessage(address, envelope.content as PreKeyMessage)
ShadeEnvelope.EnvelopeType.RATCHET -> decryptRatchetMessage(address, envelope.content as RatchetMessage)
}
}
private suspend fun decryptPreKeyMessage(address: String, message: PreKeyMessage): ByteArray {
val id = identity ?: throw IllegalStateException("Not initialized")
val spk = storage.getSignedPreKey(message.signedPreKeyId)
?: throw IllegalStateException("Signed prekey ${message.signedPreKeyId} not found")
val oneTimePrivate: ByteArray? = message.preKeyId?.let { keyId ->
val otpk = storage.getOneTimePreKey(keyId)
?: throw IllegalStateException("One-time prekey $keyId not found")
storage.removeOneTimePreKey(keyId)
otpk.keyPair.privateKey
}
val x3dhResult = processPreKeyMessage(
crypto,
id,
spk.keyPair.privateKey,
oneTimePrivate,
message,
)
val session = initReceiverSession(
rootKey = x3dhResult.rootKey,
remoteIdentityKey = x3dhResult.remoteIdentityKey,
localDHKeyPair = spk.keyPair,
)
val plaintext = ratchetDecrypt(crypto, session, message.message)
storage.saveSession(address, session)
storage.saveTrustedIdentity(address, x3dhResult.remoteIdentityKey)
return plaintext
}
private suspend fun decryptRatchetMessage(address: String, message: RatchetMessage): ByteArray {
val session = storage.getSession(address)
?: throw IllegalStateException("No session for $address")
val plaintext = ratchetDecrypt(crypto, session, message)
storage.saveSession(address, session)
return plaintext
}
}

View File

@@ -0,0 +1,273 @@
package no.zyon.shade.approval
import no.zyon.shade.crypto.CryptoProvider
import java.nio.ByteBuffer
import java.nio.ByteOrder
/**
* V4.10 — cross-host approval routing helpers. Mirror
* `@shade/sdk/approval.ts` byte-for-byte.
*
* The frames themselves (`approvalNeeded` / `linkApproveByProxy`) are
* app-defined payloads sent over the existing Shade bilateral E2EE
* channel. This file ships the canonical signing-payload layout, the
* Ed25519 sign step a phone runs after biometric unlock, and the
* verify step a host runs against the freshest profile blob.
*
* The signing payload is length-prefixed binary (u16 BE) so any
* platform — Kotlin, Swift, Go — can produce byte-identical input
* without needing a JSON canonicalizer. Cross-platform parity is
* gated by `test-vectors/blob-storage.json` (signing payload
* fixtures) plus a Kotlin↔TS round-trip in `CrossPlatformVectorTest`.
*/
/** Default domain separator. Apps with their own canonical name (e.g. Prism) override. */
const val DEFAULT_APPROVAL_DOMAIN = "shade-link-approve-v1"
/** Default expiry: 5 minutes after the host issues the request. */
const val DEFAULT_APPROVAL_EXPIRES_IN_MS = 5L * 60 * 1000
/** Information about the device the host received a `linkRequest` from. */
data class ApprovalRequestingDevice(
val fingerprint: String,
val deviceName: String? = null,
val userAgent: String? = null,
val ipHint: String? = null,
val receivedAt: Long,
)
data class ApprovalRequestFrame(
val kind: String = "approvalNeeded",
/** 128-bit hex (32 chars) random idempotency key. */
val requestId: String,
val hostAddress: String,
val hostFingerprint: String,
val requestingDevice: ApprovalRequestingDevice,
val expiresAt: Long,
val domain: String,
)
data class ProxyApprovalFrame(
val kind: String = "linkApproveByProxy",
val requestId: String,
val decision: String,
val approverFingerprint: String,
/** 64-byte Ed25519 signature, lowercase hex (128 chars). */
val signature: String,
val domain: String,
)
/**
* Build a fresh `approvalNeeded` frame with a 128-bit random
* `requestId`. Hosts SHOULD persist the requestId in a pending-set
* keyed by `expiresAt` so a returning `linkApproveByProxy` can be
* matched up — that's app state, the SDK doesn't track it.
*/
fun buildApprovalRequest(
crypto: CryptoProvider,
hostAddress: String,
hostFingerprint: String,
requestingDeviceFingerprint: String,
deviceName: String? = null,
userAgent: String? = null,
ipHint: String? = null,
expiresInMs: Long = DEFAULT_APPROVAL_EXPIRES_IN_MS,
domain: String = DEFAULT_APPROVAL_DOMAIN,
now: Long = System.currentTimeMillis(),
): ApprovalRequestFrame {
val requestId = crypto.randomBytes(16).joinToString("") { "%02x".format(it) }
return ApprovalRequestFrame(
requestId = requestId,
hostAddress = hostAddress,
hostFingerprint = hostFingerprint,
requestingDevice = ApprovalRequestingDevice(
fingerprint = requestingDeviceFingerprint,
deviceName = deviceName,
userAgent = userAgent,
ipHint = ipHint,
receivedAt = now,
),
expiresAt = now + expiresInMs,
domain = domain,
)
}
/**
* Sign a `linkApproveByProxy` frame with the approver's long-term
* Ed25519 identity key. The seed is the 32-byte Ed25519 private key
* (Tink's `Ed25519Sign(seed)` consumes it directly).
*/
fun signProxyApproval(
crypto: CryptoProvider,
request: ApprovalRequestFrame,
decision: String,
approverFingerprint: String,
approverSigningKey: ByteArray,
): ProxyApprovalFrame {
require(decision == "approve" || decision == "reject") {
"decision must be 'approve' or 'reject'"
}
require(approverSigningKey.size == 32) {
"approverSigningKey must be 32 bytes (Ed25519 seed)"
}
val payload = canonicalApprovalSigningBytes(
domain = request.domain,
requestId = request.requestId,
hostFingerprint = request.hostFingerprint,
requestingDeviceFingerprint = request.requestingDevice.fingerprint,
decision = decision,
)
val sig = crypto.sign(approverSigningKey, payload)
return ProxyApprovalFrame(
requestId = request.requestId,
decision = decision,
approverFingerprint = approverFingerprint,
signature = sig.joinToString("") { "%02x".format(it) },
domain = request.domain,
)
}
/** Tagged result of `verifyProxyApproval`. */
sealed class VerifyProxyApprovalResult {
data class Ok(val approver: ProfileClientEntry) : VerifyProxyApprovalResult()
data class Failed(val reason: Reason) : VerifyProxyApprovalResult()
enum class Reason {
REQUEST_ID_MISMATCH,
DOMAIN_MISMATCH,
UNKNOWN_APPROVER,
NOT_TRUSTED,
BAD_SIGNATURE,
EXPIRED,
}
}
/**
* Verify a `linkApproveByProxy` against the originating
* `approvalNeeded` and the host's freshest profile blob. Returns a
* tagged result rather than throwing — callers usually want to log
* the reason before deciding what to surface to the user.
*
* Order of checks:
*
* 1. requestId match (replay defense)
* 2. domain match (cross-app confusion defense)
* 3. approver resolves to a `clients[]` entry
* 4. approver is in `trustedApproverFingerprints[]` AND has the
* `trustedApprover` flag (cross-checked via `isTrustedApprover`)
* 5. expiresAt in the future
* 6. Ed25519 signature verifies against `clients[].identityPublicKey`
*
* Hosts MUST refetch the profile blob fresh before calling this — see
* the FR §5 "approver-revocation propagation" rationale.
*/
fun verifyProxyApproval(
crypto: CryptoProvider,
request: ApprovalRequestFrame,
approval: ProxyApprovalFrame,
profile: CanonicalProfileBlob,
now: Long = System.currentTimeMillis(),
): VerifyProxyApprovalResult {
if (approval.requestId != request.requestId) {
return VerifyProxyApprovalResult.Failed(
VerifyProxyApprovalResult.Reason.REQUEST_ID_MISMATCH,
)
}
if (approval.domain != request.domain) {
return VerifyProxyApprovalResult.Failed(
VerifyProxyApprovalResult.Reason.DOMAIN_MISMATCH,
)
}
val approver = findClientByFingerprint(profile, approval.approverFingerprint)
?: return VerifyProxyApprovalResult.Failed(
VerifyProxyApprovalResult.Reason.UNKNOWN_APPROVER,
)
if (!isTrustedApprover(profile, approval.approverFingerprint)) {
return VerifyProxyApprovalResult.Failed(
VerifyProxyApprovalResult.Reason.NOT_TRUSTED,
)
}
if (now > request.expiresAt) {
return VerifyProxyApprovalResult.Failed(VerifyProxyApprovalResult.Reason.EXPIRED)
}
val pubkey = try {
hexToBytes(approver.identityPublicKey)
} catch (_: Exception) {
return VerifyProxyApprovalResult.Failed(VerifyProxyApprovalResult.Reason.BAD_SIGNATURE)
}
val sig = try {
hexToBytes(approval.signature)
} catch (_: Exception) {
return VerifyProxyApprovalResult.Failed(VerifyProxyApprovalResult.Reason.BAD_SIGNATURE)
}
if (pubkey.size != 32 || sig.size != 64) {
return VerifyProxyApprovalResult.Failed(VerifyProxyApprovalResult.Reason.BAD_SIGNATURE)
}
val payload = canonicalApprovalSigningBytes(
domain = approval.domain,
requestId = approval.requestId,
hostFingerprint = request.hostFingerprint,
requestingDeviceFingerprint = request.requestingDevice.fingerprint,
decision = approval.decision,
)
val ok = crypto.verify(pubkey, payload, sig)
return if (ok) VerifyProxyApprovalResult.Ok(approver)
else VerifyProxyApprovalResult.Failed(VerifyProxyApprovalResult.Reason.BAD_SIGNATURE)
}
/**
* Build the canonical signing payload bytes for a proxy approval.
*
* Format (length-prefixed UTF-8, big-endian u16 lengths):
*
* u16(len(domain)) || domain
* u16(len(requestId)) || requestId
* u16(len(hostFp)) || hostFingerprint
* u16(len(requestFp)) || requestingDeviceFingerprint
* u16(len(decision)) || decision
*
* This is the EXACT byte layout `@shade/sdk`'s
* `canonicalApprovalSigningBytes` produces, ensuring an Android-signed
* approval verifies on a TS host and vice versa.
*/
fun canonicalApprovalSigningBytes(
domain: String,
requestId: String,
hostFingerprint: String,
requestingDeviceFingerprint: String,
decision: String,
): ByteArray {
val fields = listOf(
domain.toByteArray(Charsets.UTF_8),
requestId.toByteArray(Charsets.UTF_8),
hostFingerprint.toByteArray(Charsets.UTF_8),
requestingDeviceFingerprint.toByteArray(Charsets.UTF_8),
decision.toByteArray(Charsets.UTF_8),
)
for (f in fields) {
require(f.size <= 0xFFFF) { "signing field too long: ${f.size} bytes (max 65535)" }
}
val total = fields.sumOf { 2 + it.size }
val buf = ByteBuffer.allocate(total).order(ByteOrder.BIG_ENDIAN)
for (f in fields) {
buf.putShort(f.size.toShort())
buf.put(f)
}
return buf.array()
}
private fun hexToBytes(hex: String): ByteArray {
require(hex.length % 2 == 0) { "hex length must be even" }
require(hex.all { it.isDigit() || it in 'a'..'f' }) { "hex must be lowercase 0-9a-f" }
val out = ByteArray(hex.length / 2)
for (i in out.indices) {
out[i] = ((Character.digit(hex[i * 2], 16) shl 4) +
Character.digit(hex[i * 2 + 1], 16)).toByte()
}
return out
}

View File

@@ -0,0 +1,307 @@
package no.zyon.shade.approval
import org.json.JSONArray
import org.json.JSONObject
/**
* V4.10 — canonical profile-blob schema. Mirror
* `@shade/sdk/approval.ts` byte-for-byte: same field names, same
* JSON shape, same denormalization invariants.
*
* The blob is the AEAD plaintext stored in the V4.9 profile slot. It
* holds the user's list of paired hosts + clients; cross-host
* approval routing reads `clients[]` to find trusted approvers when
* a headless host needs to dispatch a `linkRequest` to a phone.
*
* Mutators (`upsertHost`, `setTrustedApprover`, ...) are immutable —
* they return a new blob and never modify the input. The denormalized
* `trustedApproverFingerprints[]` is rederived on every mutation so it
* can never drift from the per-client `trustedApprover` flag.
*/
/** A host: a device that receives `linkRequest` frames and runs pairing. */
data class ProfileHostEntry(
val address: String,
val name: String,
/** Open enum: `"desktop" | "server" | "laptop" | ...`. */
val kind: String,
val addedAt: Long,
)
/**
* A client: a device that initiates link/approval flows and may
* proxy-approve when `trustedApprover == true`. Stores both the
* 32-byte Ed25519 identity public key (hex) and the safety-number
* fingerprint — the public key is what `verifyProxyApproval` checks
* signatures against; the fingerprint is what UIs display.
*/
data class ProfileClientEntry(
val address: String,
/** 32-byte Ed25519 long-term identity public key, lowercase hex (64 chars). */
val identityPublicKey: String,
/** Safety-number fingerprint of the identity key (computeFingerprint output). */
val identityFingerprint: String,
val name: String,
/** Open enum: `"mobile" | "tablet" | "browser" | ...`. */
val kind: String,
val addedAt: Long,
val trustedApprover: Boolean = false,
)
/**
* Canonical profile blob. `version=1` is the only currently-supported
* shape; bump when an incompatible field is added. Unknown top-level
* fields are dropped on parse — additive changes need a coordinated
* schema bump on both platforms.
*/
data class CanonicalProfileBlob(
val version: Int = 1,
val hosts: List<ProfileHostEntry> = emptyList(),
val clients: List<ProfileClientEntry> = emptyList(),
/** Denormalized list of trusted-approver fingerprints. Rederived on mutate. */
val trustedApproverFingerprints: List<String> = emptyList(),
val updatedAt: Long = 0,
/** Optional hex-encoded pubkey of the writer; informational only. */
val signedBy: String? = null,
)
/** Build a fresh empty profile blob with `updatedAt = now ?? System.currentTimeMillis()`. */
fun emptyCanonicalProfile(now: Long? = null): CanonicalProfileBlob =
CanonicalProfileBlob(updatedAt = now ?: System.currentTimeMillis())
/**
* Decode a profile-blob plaintext (the AEAD-opened bytes) into the
* canonical shape. Throws `IllegalArgumentException` on malformed JSON
* or wrong shape.
*/
fun parseCanonicalProfile(plaintext: ByteArray): CanonicalProfileBlob =
parseCanonicalProfile(plaintext.toString(Charsets.UTF_8))
fun parseCanonicalProfile(plaintext: String): CanonicalProfileBlob {
val obj = try {
JSONObject(plaintext)
} catch (e: Exception) {
throw IllegalArgumentException("profile blob is not valid JSON: ${e.message}")
}
val version = obj.optInt("version", -1)
require(version == 1) { "unsupported profile blob version: $version" }
val hosts = parseArray(obj.optJSONArray("hosts"), "hosts", ::parseHostEntry)
val clients = parseArray(obj.optJSONArray("clients"), "clients", ::parseClientEntry)
val trustedApproverFingerprints = parseStringArray(
obj.optJSONArray("trustedApproverFingerprints"),
"trustedApproverFingerprints",
)
val updatedAt = if (obj.has("updatedAt") && !obj.isNull("updatedAt"))
obj.getLong("updatedAt") else 0L
val signedBy = obj.optString("signedBy", "").takeIf { it.isNotEmpty() }
return CanonicalProfileBlob(
version = 1,
hosts = hosts,
clients = clients,
trustedApproverFingerprints = trustedApproverFingerprints,
updatedAt = updatedAt,
signedBy = signedBy,
)
}
/** Serialize a profile blob to UTF-8 JSON ready for `Profile.put`. */
fun serializeCanonicalProfile(blob: CanonicalProfileBlob): ByteArray {
val json = JSONObject()
json.put("version", blob.version)
json.put("hosts", JSONArray().apply {
blob.hosts.forEach { put(hostEntryToJson(it)) }
})
json.put("clients", JSONArray().apply {
blob.clients.forEach { put(clientEntryToJson(it)) }
})
json.put("trustedApproverFingerprints", JSONArray(blob.trustedApproverFingerprints))
json.put("updatedAt", blob.updatedAt)
if (blob.signedBy != null) json.put("signedBy", blob.signedBy)
return json.toString().toByteArray(Charsets.UTF_8)
}
private fun parseHostEntry(o: JSONObject): ProfileHostEntry =
ProfileHostEntry(
address = o.requireString("address", "hosts"),
name = o.requireString("name", "hosts"),
kind = o.requireString("kind", "hosts"),
addedAt = o.requireLong("addedAt", "hosts"),
)
private fun parseClientEntry(o: JSONObject): ProfileClientEntry {
val identityPublicKey = o.requireString("identityPublicKey", "clients")
require(identityPublicKey.matches(Regex("^[0-9a-f]{64}$"))) {
"clients[].identityPublicKey must be 64 lowercase hex chars"
}
return ProfileClientEntry(
address = o.requireString("address", "clients"),
identityPublicKey = identityPublicKey,
identityFingerprint = o.requireString("identityFingerprint", "clients"),
name = o.requireString("name", "clients"),
kind = o.requireString("kind", "clients"),
addedAt = o.requireLong("addedAt", "clients"),
trustedApprover = o.optBoolean("trustedApprover", false),
)
}
private fun hostEntryToJson(e: ProfileHostEntry): JSONObject = JSONObject().apply {
put("address", e.address)
put("name", e.name)
put("kind", e.kind)
put("addedAt", e.addedAt)
}
private fun clientEntryToJson(e: ProfileClientEntry): JSONObject = JSONObject().apply {
put("address", e.address)
put("identityPublicKey", e.identityPublicKey)
put("identityFingerprint", e.identityFingerprint)
put("name", e.name)
put("kind", e.kind)
put("addedAt", e.addedAt)
if (e.trustedApprover) put("trustedApprover", true)
}
private fun <T> parseArray(
arr: JSONArray?,
field: String,
parse: (JSONObject) -> T,
): List<T> {
if (arr == null) return emptyList()
return (0 until arr.length()).map { i ->
val item = arr.opt(i)
require(item is JSONObject) { "$field[$i] must be an object" }
parse(item)
}
}
private fun parseStringArray(arr: JSONArray?, field: String): List<String> {
if (arr == null) return emptyList()
return (0 until arr.length()).map { i ->
val item = arr.opt(i)
require(item is String) { "$field[$i] must be a string" }
item
}
}
private fun JSONObject.requireString(key: String, ctx: String): String {
require(has(key) && !isNull(key)) { "$ctx[].$key is required" }
val v = get(key)
require(v is String) { "$ctx[].$key must be a string" }
return v
}
private fun JSONObject.requireLong(key: String, ctx: String): Long {
require(has(key) && !isNull(key)) { "$ctx[].$key is required" }
return when (val v = get(key)) {
is Number -> v.toLong()
else -> throw IllegalArgumentException("$ctx[].$key must be a number")
}
}
// ─── Mutators (immutable; return new blob, never mutate input) ──
/** Insert or replace a host entry by address. Bumps `updatedAt`. */
fun upsertHost(
blob: CanonicalProfileBlob,
host: ProfileHostEntry,
now: Long? = null,
): CanonicalProfileBlob {
val hosts = blob.hosts.filter { it.address != host.address } + host
return blob.copy(hosts = hosts, updatedAt = now ?: System.currentTimeMillis())
}
/** Remove the host with the given address, if any. */
fun removeHost(
blob: CanonicalProfileBlob,
address: String,
now: Long? = null,
): CanonicalProfileBlob {
val hosts = blob.hosts.filter { it.address != address }
if (hosts.size == blob.hosts.size) return blob
return blob.copy(hosts = hosts, updatedAt = now ?: System.currentTimeMillis())
}
/**
* Insert or replace a client entry by `identityFingerprint`. Re-derives
* `trustedApproverFingerprints` from the resulting `clients[]` so the
* denormalized list never drifts.
*/
fun upsertClient(
blob: CanonicalProfileBlob,
client: ProfileClientEntry,
now: Long? = null,
): CanonicalProfileBlob {
val clients = blob.clients
.filter { it.identityFingerprint != client.identityFingerprint } + client
return blob.copy(
clients = clients,
trustedApproverFingerprints = deriveTrustedApprovers(clients),
updatedAt = now ?: System.currentTimeMillis(),
)
}
/** Remove the client with the given identityFingerprint, if any. */
fun removeClient(
blob: CanonicalProfileBlob,
identityFingerprint: String,
now: Long? = null,
): CanonicalProfileBlob {
val clients = blob.clients.filter { it.identityFingerprint != identityFingerprint }
if (clients.size == blob.clients.size) return blob
return blob.copy(
clients = clients,
trustedApproverFingerprints = deriveTrustedApprovers(clients),
updatedAt = now ?: System.currentTimeMillis(),
)
}
/**
* Toggle the `trustedApprover` flag on a client by fingerprint.
* Returns the input unchanged if fingerprint isn't found OR the
* desired state already matches (no spurious updatedAt bump).
*/
fun setTrustedApprover(
blob: CanonicalProfileBlob,
identityFingerprint: String,
trusted: Boolean,
now: Long? = null,
): CanonicalProfileBlob {
var touched = false
val clients = blob.clients.map { c ->
if (c.identityFingerprint != identityFingerprint) c
else if (c.trustedApprover == trusted) c
else {
touched = true
c.copy(trustedApprover = trusted)
}
}
if (!touched) return blob
return blob.copy(
clients = clients,
trustedApproverFingerprints = deriveTrustedApprovers(clients),
updatedAt = now ?: System.currentTimeMillis(),
)
}
/**
* True iff the given fingerprint resolves to a client with both
* `trustedApprover == true` AND an entry in `trustedApproverFingerprints[]`.
*/
fun isTrustedApprover(blob: CanonicalProfileBlob, identityFingerprint: String): Boolean {
if (!blob.trustedApproverFingerprints.contains(identityFingerprint)) return false
val c = findClientByFingerprint(blob, identityFingerprint) ?: return false
return c.trustedApprover
}
fun findClientByFingerprint(
blob: CanonicalProfileBlob,
identityFingerprint: String,
): ProfileClientEntry? = blob.clients.firstOrNull { it.identityFingerprint == identityFingerprint }
fun findClientByAddress(blob: CanonicalProfileBlob, address: String): ProfileClientEntry? =
blob.clients.firstOrNull { it.address == address }
private fun deriveTrustedApprovers(clients: List<ProfileClientEntry>): List<String> =
clients.filter { it.trustedApprover }.map { it.identityFingerprint }

View File

@@ -0,0 +1,36 @@
package no.zyon.shade.backup
import no.zyon.shade.crypto.CryptoProvider
/**
* Backup format v1 — passphrase-derived AES-256-GCM blob.
* Mirror @shade/sdk/backup.ts.
*
* backupKey = HKDF(passphrase_utf8, salt_random_32, info="ShadeBackupKey", 32)
* blob = AES-256-GCM(backupKey, plaintext, no AAD)
*
* The stored on-disk form is `{ version, salt(b64), nonce(b64), ciphertext(b64) }`.
* This file ships only the cryptographic primitives — payload schema and JSON
* serialization live alongside the high-level SDK and don't need a Kotlin port
* for vector parity (each platform builds the BackupBlob in its native idiom).
*
* NOTE: HKDF is NOT a proper password KDF. The TS SDK acknowledges this and
* warns users to choose a high-entropy passphrase. When `argon2id` lands in
* `CryptoProvider`, both ports swap together. Until then, byte-parity for the
* HKDF + AEAD layer is what V3.5 §sjekkpunkt 8 gates.
*/
private val BACKUP_INFO: ByteArray = "ShadeBackupKey".toByteArray(Charsets.UTF_8)
const val BACKUP_KEY_BYTES = 32
const val BACKUP_VERSION = 1
fun deriveBackupKey(crypto: CryptoProvider, passphrase: String, salt: ByteArray): ByteArray {
require(passphrase.length >= 12) { "Passphrase must be at least 12 characters" }
require(salt.size >= 16) { "salt must be at least 16 bytes" }
return crypto.hkdf(
passphrase.toByteArray(Charsets.UTF_8),
salt,
BACKUP_INFO,
BACKUP_KEY_BYTES,
)
}

View File

@@ -0,0 +1,94 @@
package no.zyon.shade.blob
import javax.crypto.Cipher
import javax.crypto.spec.GCMParameterSpec
import javax.crypto.spec.SecretKeySpec
/**
* AEAD wrapper for the V4.9 profile blob.
*
* Wire format for one ciphertext blob:
* `nonce(12) || ciphertext(N) || tag(16)`
*
* Mirror `@shade/storage-encrypted/crypto/aead.ts` byte-for-byte. The
* relay stores this as a single opaque BLOB column; AAD is reconstructed
* at read-time as `"shade-profile-aad-v1:" + slotIdHex` and is NOT
* stored on the relay.
*/
const val BLOB_AEAD_NONCE_LEN = 12
const val BLOB_AEAD_TAG_LEN = 16
private const val MIN_CIPHERTEXT_LEN = BLOB_AEAD_NONCE_LEN + BLOB_AEAD_TAG_LEN
/**
* Seal a plaintext blob. Returns `nonce || ct||tag` ready for direct
* blob storage. The caller supplies the nonce so this function is
* deterministic — the high-level Profile namespace generates a fresh
* 12-byte random nonce per write to keep (key, nonce, plaintext)
* unique across re-uploads.
*/
fun aeadSeal(
key: ByteArray,
nonce: ByteArray,
plaintext: ByteArray,
aad: ByteArray,
): ByteArray {
require(key.size == 32) { "AES-256-GCM key must be 32 bytes" }
require(nonce.size == BLOB_AEAD_NONCE_LEN) {
"nonce must be $BLOB_AEAD_NONCE_LEN bytes"
}
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val spec = GCMParameterSpec(128, nonce)
cipher.init(Cipher.ENCRYPT_MODE, SecretKeySpec(key, "AES"), spec)
cipher.updateAAD(aad)
val ctTag = cipher.doFinal(plaintext)
val out = ByteArray(BLOB_AEAD_NONCE_LEN + ctTag.size)
System.arraycopy(nonce, 0, out, 0, BLOB_AEAD_NONCE_LEN)
System.arraycopy(ctTag, 0, out, BLOB_AEAD_NONCE_LEN, ctTag.size)
return out
}
/**
* Open a `nonce || ct||tag` blob and return the plaintext. Throws on
* tamper (AEAD tag mismatch) or short input. The caller may pass an
* `expectedNonce` to enforce a deterministic nonce — mismatch throws
* before the AEAD even runs (defense-in-depth against a relay returning
* the wrong slot's blob).
*/
fun aeadOpen(
key: ByteArray,
blob: ByteArray,
aad: ByteArray,
expectedNonce: ByteArray? = null,
): ByteArray {
require(key.size == 32) { "AES-256-GCM key must be 32 bytes" }
require(blob.size >= MIN_CIPHERTEXT_LEN) { "ciphertext blob too short" }
val nonce = blob.copyOfRange(0, BLOB_AEAD_NONCE_LEN)
if (expectedNonce != null && !ctEqual(nonce, expectedNonce)) {
throw IllegalArgumentException(
"nonce mismatch — ciphertext blob has been tampered or row identity changed",
)
}
val ctTag = blob.copyOfRange(BLOB_AEAD_NONCE_LEN, blob.size)
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val spec = GCMParameterSpec(128, nonce)
cipher.init(Cipher.DECRYPT_MODE, SecretKeySpec(key, "AES"), spec)
cipher.updateAAD(aad)
return cipher.doFinal(ctTag)
}
private fun ctEqual(a: ByteArray, b: ByteArray): Boolean {
if (a.size != b.size) return false
var diff = 0
for (i in a.indices) {
diff = diff or (a[i].toInt() xor b[i].toInt())
}
return diff == 0
}
/** Build the AAD for a given slotId hex string. */
fun blobAadForSlot(slotIdHex: String): ByteArray {
require(slotIdHex.length == 64) { "slotIdHex must be 64 hex chars" }
return "shade-profile-aad-v1:$slotIdHex".toByteArray(Charsets.UTF_8)
}

View File

@@ -0,0 +1,250 @@
package no.zyon.shade.blob
import no.zyon.shade.crypto.CryptoProvider
import org.json.JSONObject
import java.net.URI
import java.net.http.HttpClient
import java.net.http.HttpRequest
import java.net.http.HttpResponse
import java.time.Duration
import java.util.Base64
/**
* Low-level HTTP client for the V4.9 encrypted-blob primitive
* (`/v1/blob/<slotId>`). Mirror `@shade/inbox`'s `BlobClient` —
* stateless, reusable, and protocol-compatible with the TypeScript
* relay endpoints.
*
* The client doesn't care what the blob bytes mean — it just
* transports them. Higher-level wrappers (e.g. `Profile`) compose
* this client with AEAD-sealing of the actual payload.
*
* Auth model: every PUT/DELETE carries a detached Ed25519 signature
* (base64) over a canonical-JSON form of the request body. The
* canonicalization is deterministic — sorted keys, compact JSON, no
* trailing whitespace — so signatures generated on Kotlin verify on
* the TS server.
*/
class BlobClient(
private val baseUrl: String,
private val crypto: CryptoProvider,
private val httpClient: HttpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(15))
.build(),
) {
data class GetResult(
val blob: ByteArray,
val etag: String,
val updatedAt: Long,
)
data class PutResult(
val created: Boolean,
val etag: String,
val updatedAt: Long,
)
/**
* Read a slot. Returns null if no blob has ever been written there
* (or if it was DELETE'd). GET is unauthenticated by design — the
* slotId is itself a 256-bit secret derived from the master key.
*/
fun get(slotIdHex: String): GetResult? {
validateSlotIdHex(slotIdHex)
val req = HttpRequest.newBuilder(URI.create(joinUrl(baseUrl, "/v1/blob/$slotIdHex")))
.GET()
.build()
val res = httpClient.send(req, HttpResponse.BodyHandlers.ofString())
if (res.statusCode() == 404) return null
val json = parseJson(res, "GET")
val blob = Base64.getDecoder().decode(json.getString("blob"))
return GetResult(
blob = blob,
etag = json.getString("etag"),
updatedAt = json.getLong("updatedAt"),
)
}
/**
* Create or update a slot.
*
* `ifMatch` semantics:
* - `null`: create-only. Slot must be empty (else 409).
* - `<etag-string>`: compare-and-swap. Must match (else 412).
* - `"*"`: unconditional overwrite. Slot must already exist (else 412).
*/
fun put(
slotIdHex: String,
blob: ByteArray,
signingSeed: ByteArray,
ownerPubkey: ByteArray,
ifMatch: String? = null,
): PutResult {
validateSlotIdHex(slotIdHex)
require(blob.isNotEmpty()) { "Empty blob" }
require(ownerPubkey.size == 32) { "ownerPubkey must be 32 bytes (Ed25519)" }
require(signingSeed.size == 32) { "signingSeed must be 32 bytes (Ed25519)" }
// Canonical form for signing: sorted keys, slotId included,
// signature field absent. The wire body strips slotId (it's
// in the URL) but the signature is computed over the
// slotId-bearing form.
val signedAt = System.currentTimeMillis()
val canonical = sortedMapOf<String, Any>().apply {
put("blob", Base64.getEncoder().encodeToString(blob))
if (ifMatch != null) put("ifMatch", ifMatch)
put("ownerPubkey", Base64.getEncoder().encodeToString(ownerPubkey))
put("signedAt", signedAt)
put("slotId", slotIdHex)
}
val canonicalBytes = canonicalJson(canonical).toByteArray(Charsets.UTF_8)
val sig = crypto.sign(signingSeed, canonicalBytes)
// Wire body: same as canonical minus slotId, plus signature.
val wire = JSONObject()
wire.put("blob", Base64.getEncoder().encodeToString(blob))
if (ifMatch != null) wire.put("ifMatch", ifMatch)
wire.put("ownerPubkey", Base64.getEncoder().encodeToString(ownerPubkey))
wire.put("signedAt", signedAt)
wire.put("signature", Base64.getEncoder().encodeToString(sig))
val req = HttpRequest.newBuilder(URI.create(joinUrl(baseUrl, "/v1/blob/$slotIdHex")))
.header("content-type", "application/json")
.PUT(HttpRequest.BodyPublishers.ofString(wire.toString()))
.build()
val res = httpClient.send(req, HttpResponse.BodyHandlers.ofString())
val json = parseJson(res, "PUT")
return PutResult(
created = json.optBoolean("created"),
etag = json.getString("etag"),
updatedAt = json.getLong("updatedAt"),
)
}
/**
* Delete a slot. The next PUT TOFU-claims it again, possibly under
* a fresh signing key (e.g. after rotation). Used by "forget
* everything" flows.
*/
fun delete(slotIdHex: String, signingSeed: ByteArray): Boolean {
validateSlotIdHex(slotIdHex)
require(signingSeed.size == 32) { "signingSeed must be 32 bytes (Ed25519)" }
val signedAt = System.currentTimeMillis()
val canonical = sortedMapOf<String, Any>().apply {
put("signedAt", signedAt)
put("slotId", slotIdHex)
}
val canonicalBytes = canonicalJson(canonical).toByteArray(Charsets.UTF_8)
val sig = crypto.sign(signingSeed, canonicalBytes)
val wire = JSONObject()
wire.put("signedAt", signedAt)
wire.put("signature", Base64.getEncoder().encodeToString(sig))
val req = HttpRequest.newBuilder(URI.create(joinUrl(baseUrl, "/v1/blob/$slotIdHex")))
.header("content-type", "application/json")
.method("DELETE", HttpRequest.BodyPublishers.ofString(wire.toString()))
.build()
val res = httpClient.send(req, HttpResponse.BodyHandlers.ofString())
val json = parseJson(res, "DELETE")
return json.optBoolean("ok", false)
}
private fun parseJson(res: HttpResponse<String>, op: String): JSONObject {
val text = res.body() ?: ""
val json = if (text.isEmpty()) JSONObject() else try {
JSONObject(text)
} catch (e: Exception) {
throw BlobClientException(
code = "SHADE_NETWORK",
statusCode = res.statusCode(),
message = "Blob $op response not JSON: ${text.take(200)}",
)
}
if (res.statusCode() !in 200..299) {
throw BlobClientException(
code = json.optString("code", "SHADE_NETWORK"),
statusCode = res.statusCode(),
message = json.optString("message", text),
)
}
return json
}
private fun validateSlotIdHex(s: String) {
require(s.matches(Regex("^[0-9a-f]{64}$"))) {
"slotIdHex must be 64 lowercase hex chars (32 bytes)"
}
}
private fun joinUrl(base: String, path: String): String =
when {
base.endsWith("/") && path.startsWith("/") -> base + path.substring(1)
!base.endsWith("/") && !path.startsWith("/") -> "$base/$path"
else -> base + path
}
}
/**
* Mirror of TS `signPayload`'s canonicalization: sorted keys, compact
* JSON, signature field absent. Only handles the subset of types we
* need (strings + longs + base64 strings) — keeping the implementation
* narrow so it can't accidentally diverge from `JSON.stringify` on
* structurally-different inputs.
*/
internal fun canonicalJson(map: Map<String, Any>): String {
val sb = StringBuilder()
sb.append('{')
var first = true
for ((key, value) in map.toSortedMap()) {
if (!first) sb.append(',')
first = false
appendJsonString(sb, key)
sb.append(':')
appendJsonValue(sb, value)
}
sb.append('}')
return sb.toString()
}
private fun appendJsonValue(sb: StringBuilder, value: Any) {
when (value) {
is String -> appendJsonString(sb, value)
is Long, is Int -> sb.append(value.toString())
is Boolean -> sb.append(value.toString())
else -> throw IllegalArgumentException(
"canonicalJson: unsupported value type ${value::class.java}",
)
}
}
private fun appendJsonString(sb: StringBuilder, s: String) {
sb.append('"')
for (c in s) {
when (c) {
'\\' -> sb.append("\\\\")
'"' -> sb.append("\\\"")
'\b' -> sb.append("\\b")
'\u000C' -> sb.append("\\f")
'\n' -> sb.append("\\n")
'\r' -> sb.append("\\r")
'\t' -> sb.append("\\t")
else -> {
if (c.code < 0x20) {
sb.append("\\u").append("%04x".format(c.code))
} else {
sb.append(c)
}
}
}
}
sb.append('"')
}
class BlobClientException(
val code: String,
val statusCode: Int,
message: String,
) : RuntimeException("[$code @ $statusCode] $message")

View File

@@ -0,0 +1,88 @@
package no.zyon.shade.blob
import com.google.crypto.tink.subtle.Ed25519Sign
import no.zyon.shade.crypto.CryptoProvider
/**
* V4.9 — relay-side encrypted blob primitive: deterministic
* derivations from a 32-byte master key + per-app namespace string.
*
* Mirror `@shade/storage-encrypted/crypto/kdf.ts` byte-for-byte.
* Reference vectors: `test-vectors/blob-storage.json`.
*
* Three independent 32-byte derivations:
*
* slotId = HKDF(masterKey, info=`shade-blob-slot-v1:<app>`) // relay-visible opaque ID
* blobKey = HKDF(masterKey, info=`shade-blob-key-v1:<app>`) // AEAD key for the blob
* sigSeed = HKDF(masterKey, info=`shade-blob-sig-v1:<app>`) // Ed25519 owner signing seed
*
* `app` is a caller-supplied namespace string — distinct apps under
* the same master MUST pass different values (e.g. `prism-profile-v1`)
* so they don't collide on the same slot.
*
* The signing seed is an Ed25519 *seed* in the @noble/curves convention:
* `pubkey = Ed25519.publicFromSeed(seed)` is what the relay TOFU-stores
* on the first PUT and verifies subsequent writes against.
*/
private const val SLOT_INFO_PREFIX = "shade-blob-slot-v1:"
private const val BLOB_KEY_INFO_PREFIX = "shade-blob-key-v1:"
private const val SIG_SEED_INFO_PREFIX = "shade-blob-sig-v1:"
private const val DERIVED_LEN = 32
/** Lower-hex 64-char slotId derived deterministically from the master key. */
fun deriveBlobSlotId(crypto: CryptoProvider, masterKey: ByteArray, app: String): ByteArray {
require(masterKey.size == 32) { "masterKey must be 32 bytes" }
require(app.isNotEmpty()) { "app namespace must be non-empty" }
return crypto.hkdf(
masterKey,
ByteArray(0),
(SLOT_INFO_PREFIX + app).toByteArray(Charsets.UTF_8),
DERIVED_LEN,
)
}
/** AEAD key for sealing/opening the blob. The slotId hex is bound as AAD. */
fun deriveBlobKey(crypto: CryptoProvider, masterKey: ByteArray, app: String): ByteArray {
require(masterKey.size == 32) { "masterKey must be 32 bytes" }
require(app.isNotEmpty()) { "app namespace must be non-empty" }
return crypto.hkdf(
masterKey,
ByteArray(0),
(BLOB_KEY_INFO_PREFIX + app).toByteArray(Charsets.UTF_8),
DERIVED_LEN,
)
}
/**
* 32-byte Ed25519 signing seed. The pubkey, derived deterministically
* from the seed, is what the relay TOFU-stores on the first PUT.
*/
fun deriveBlobSigningSeed(crypto: CryptoProvider, masterKey: ByteArray, app: String): ByteArray {
require(masterKey.size == 32) { "masterKey must be 32 bytes" }
require(app.isNotEmpty()) { "app namespace must be non-empty" }
return crypto.hkdf(
masterKey,
ByteArray(0),
(SIG_SEED_INFO_PREFIX + app).toByteArray(Charsets.UTF_8),
DERIVED_LEN,
)
}
/**
* Recover the Ed25519 public key from a 32-byte seed. Mirrors
* `@shade/crypto-web`'s `ed25519PublicKeyFromSeed`. Tink's
* `Ed25519Sign.KeyPair.newKeyPairFromSeed(seed)` exposes both halves;
* we discard the private half here.
*/
fun ed25519PublicKeyFromSeed(seed: ByteArray): ByteArray {
require(seed.size == 32) { "Ed25519 seed must be 32 bytes" }
return Ed25519Sign.KeyPair.newKeyPairFromSeed(seed).publicKey
}
/** Convert a 32-byte slotId into the lowercase-hex wire form (64 chars). */
fun slotIdToHex(slotId: ByteArray): String {
require(slotId.size == 32) { "slotId must be 32 bytes" }
return slotId.joinToString("") { "%02x".format(it) }
}

View File

@@ -0,0 +1,118 @@
package no.zyon.shade.blob
import no.zyon.shade.crypto.CryptoProvider
/**
* V4.9 — high-level profile namespace. Mirror
* `@shade/sdk`'s `createProfileNamespace`. The relay never sees
* plaintext; AAD binds the slotId so a relay returning the wrong
* slot's blob fails to open.
*
* Usage:
*
* val crypto = TinkProvider()
* val masterKey = deriveMasterKey("password", salt) // V4.5 KDF chain
* val profile = createProfileNamespace(
* baseUrl = "https://shade.example/",
* crypto = crypto,
* masterKey = masterKey,
* app = "prism-profile-v1",
* )
*
* val current = profile.get() // null if no blob yet
* profile.put(serializeCanonicalProfile(...), ifMatch = current?.etag)
* profile.delete()
*
* Apps with the same master key + app namespace converge on the same
* slot — that's the whole point: a brand new device with the right
* credentials can locate, decrypt, and update the blob.
*/
class ProfileNamespace internal constructor(
/** Lower-hex 64-char slotId. Stable per (master, app). */
val slotIdHex: String,
private val blobKey: ByteArray,
private val signingSeed: ByteArray,
private val ownerPubkey: ByteArray,
private val aad: ByteArray,
private val client: BlobClient,
private val crypto: CryptoProvider,
) {
data class GetResult(
val plaintext: ByteArray,
val etag: String,
val updatedAt: Long,
)
data class PutResult(
val created: Boolean,
val etag: String,
val updatedAt: Long,
)
/** Returns null when the slot has never been written (or was deleted). */
fun get(): GetResult? {
val raw = client.get(slotIdHex) ?: return null
val plaintext = aeadOpen(blobKey, raw.blob, aad)
return GetResult(plaintext = plaintext, etag = raw.etag, updatedAt = raw.updatedAt)
}
/**
* Create or update. `ifMatch`:
* - null: create-only (fails with 409 if slot populated).
* - "<etag>": CAS (fails with 412 on stale).
* - "*": unconditional overwrite.
*/
fun put(plaintext: ByteArray, ifMatch: String? = null): PutResult {
// Fresh random nonce per write — see `BlobAead`. Re-uploading
// the same plaintext after a transient error reuses neither
// (key, nonce, plaintext) nor (key, nonce).
val nonce = crypto.randomBytes(BLOB_AEAD_NONCE_LEN)
val sealed = aeadSeal(blobKey, nonce, plaintext, aad)
val r = client.put(
slotIdHex = slotIdHex,
blob = sealed,
signingSeed = signingSeed,
ownerPubkey = ownerPubkey,
ifMatch = ifMatch,
)
return PutResult(created = r.created, etag = r.etag, updatedAt = r.updatedAt)
}
/** "Forget everything" path — the next PUT TOFU-claims it again. */
fun delete(): Boolean = client.delete(slotIdHex, signingSeed)
}
/**
* Build a Profile namespace bound to a (master key, app) pair. The
* derivations are deterministic: any device with the same master
* key + app namespace produces the same slot, so a fresh device
* after credential entry can locate the existing profile blob.
*/
fun createProfileNamespace(
baseUrl: String,
crypto: CryptoProvider,
masterKey: ByteArray,
app: String,
): ProfileNamespace {
require(masterKey.size == 32) { "masterKey must be 32 bytes" }
require(app.isNotEmpty()) { "app namespace must be non-empty" }
val slotIdBytes = deriveBlobSlotId(crypto, masterKey, app)
val slotIdHex = slotIdToHex(slotIdBytes)
val blobKey = deriveBlobKey(crypto, masterKey, app)
val signingSeed = deriveBlobSigningSeed(crypto, masterKey, app)
val ownerPubkey = ed25519PublicKeyFromSeed(signingSeed)
val aad = blobAadForSlot(slotIdHex)
val client = BlobClient(baseUrl = baseUrl, crypto = crypto)
return ProfileNamespace(
slotIdHex = slotIdHex,
blobKey = blobKey,
signingSeed = signingSeed,
ownerPubkey = ownerPubkey,
aad = aad,
client = client,
crypto = crypto,
)
}

View File

@@ -0,0 +1,67 @@
package no.zyon.shade.crypto
/**
* Platform-agnostic crypto primitives. Mirror @shade/core/crypto.ts.
*
* All implementations must produce byte-identical output to the
* TypeScript version for the same inputs.
*/
interface CryptoProvider {
// ─── X25519 ────────────────────────────────────────────────
/** Generate an X25519 keypair (32-byte public + 32-byte private) */
fun generateX25519KeyPair(): Pair<ByteArray, ByteArray> // (public, private)
/** X25519 Diffie-Hellman: returns 32-byte shared secret */
fun x25519(privateKey: ByteArray, publicKey: ByteArray): ByteArray
// ─── Ed25519 ───────────────────────────────────────────────
/** Generate an Ed25519 keypair */
fun generateEd25519KeyPair(): Pair<ByteArray, ByteArray>
/** Sign message with Ed25519 — returns 64-byte signature */
fun sign(privateKey: ByteArray, message: ByteArray): ByteArray
/** Verify Ed25519 signature — returns true if valid */
fun verify(publicKey: ByteArray, message: ByteArray, signature: ByteArray): Boolean
// ─── AES-256-GCM ──────────────────────────────────────────
/** Encrypt with AES-256-GCM. Generates random 12-byte nonce. */
fun aesGcmEncrypt(
key: ByteArray,
plaintext: ByteArray,
aad: ByteArray? = null,
): Pair<ByteArray, ByteArray> // (ciphertext, nonce)
/** Decrypt AES-256-GCM. Throws on authentication failure. */
fun aesGcmDecrypt(
key: ByteArray,
ciphertext: ByteArray,
nonce: ByteArray,
aad: ByteArray? = null,
): ByteArray
// ─── Key Derivation ────────────────────────────────────────
/** HKDF-SHA256: derive `length` bytes */
fun hkdf(ikm: ByteArray, salt: ByteArray, info: ByteArray, length: Int): ByteArray
/** HMAC-SHA256: 32-byte MAC */
fun hmacSha256(key: ByteArray, data: ByteArray): ByteArray
// ─── Random ────────────────────────────────────────────────
fun randomBytes(length: Int): ByteArray
fun randomUint32(): Int
// ─── Hardening ─────────────────────────────────────────────
/** Constant-time byte array comparison */
fun constantTimeEqual(a: ByteArray, b: ByteArray): Boolean
/** Overwrite a buffer with zeros */
fun zeroize(buf: ByteArray)
}

View File

@@ -0,0 +1,124 @@
package no.zyon.shade.crypto
import com.google.crypto.tink.subtle.Ed25519Sign
import com.google.crypto.tink.subtle.Ed25519Verify
import com.google.crypto.tink.subtle.Hkdf
import com.google.crypto.tink.subtle.X25519
import java.nio.ByteBuffer
import java.security.SecureRandom
import javax.crypto.Cipher
import javax.crypto.Mac
import javax.crypto.spec.GCMParameterSpec
import javax.crypto.spec.SecretKeySpec
/**
* CryptoProvider backed by Google Tink + javax.crypto.
*
* Must produce byte-identical output to @shade/crypto-web for the same
* inputs, otherwise cross-platform messaging breaks.
*/
class TinkProvider : CryptoProvider {
private val random = SecureRandom()
// ─── X25519 ────────────────────────────────────────────────
override fun generateX25519KeyPair(): Pair<ByteArray, ByteArray> {
val privateKey = X25519.generatePrivateKey()
val publicKey = X25519.publicFromPrivate(privateKey)
return publicKey to privateKey
}
override fun x25519(privateKey: ByteArray, publicKey: ByteArray): ByteArray {
return X25519.computeSharedSecret(privateKey, publicKey)
}
// ─── Ed25519 ───────────────────────────────────────────────
override fun generateEd25519KeyPair(): Pair<ByteArray, ByteArray> {
val keyPair = Ed25519Sign.KeyPair.newKeyPair()
return keyPair.publicKey to keyPair.privateKey
}
override fun sign(privateKey: ByteArray, message: ByteArray): ByteArray {
val signer = Ed25519Sign(privateKey)
return signer.sign(message)
}
override fun verify(publicKey: ByteArray, message: ByteArray, signature: ByteArray): Boolean {
return try {
Ed25519Verify(publicKey).verify(signature, message)
true
} catch (_: Exception) {
false
}
}
// ─── AES-256-GCM ──────────────────────────────────────────
override fun aesGcmEncrypt(
key: ByteArray,
plaintext: ByteArray,
aad: ByteArray?,
): Pair<ByteArray, ByteArray> {
val nonce = randomBytes(12)
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val spec = GCMParameterSpec(128, nonce)
cipher.init(Cipher.ENCRYPT_MODE, SecretKeySpec(key, "AES"), spec)
if (aad != null) cipher.updateAAD(aad)
val ciphertext = cipher.doFinal(plaintext)
return ciphertext to nonce
}
override fun aesGcmDecrypt(
key: ByteArray,
ciphertext: ByteArray,
nonce: ByteArray,
aad: ByteArray?,
): ByteArray {
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val spec = GCMParameterSpec(128, nonce)
cipher.init(Cipher.DECRYPT_MODE, SecretKeySpec(key, "AES"), spec)
if (aad != null) cipher.updateAAD(aad)
return cipher.doFinal(ciphertext)
}
// ─── Key Derivation ────────────────────────────────────────
override fun hkdf(ikm: ByteArray, salt: ByteArray, info: ByteArray, length: Int): ByteArray {
return Hkdf.computeHkdf("HMACSHA256", ikm, salt, info, length)
}
override fun hmacSha256(key: ByteArray, data: ByteArray): ByteArray {
val mac = Mac.getInstance("HmacSHA256")
mac.init(SecretKeySpec(key, "HmacSHA256"))
return mac.doFinal(data)
}
// ─── Random ────────────────────────────────────────────────
override fun randomBytes(length: Int): ByteArray {
val buf = ByteArray(length)
random.nextBytes(buf)
return buf
}
override fun randomUint32(): Int {
val buf = randomBytes(4)
return ByteBuffer.wrap(buf).int
}
// ─── Hardening ─────────────────────────────────────────────
override fun constantTimeEqual(a: ByteArray, b: ByteArray): Boolean {
if (a.size != b.size) return false
var diff = 0
for (i in a.indices) {
diff = diff or (a[i].toInt() xor b[i].toInt())
}
return diff == 0
}
override fun zeroize(buf: ByteArray) {
buf.fill(0)
}
}

View File

@@ -0,0 +1,40 @@
package no.zyon.shade.fingerprint
import no.zyon.shade.crypto.CryptoProvider
/**
* Safety number computation. Must produce byte-identical output
* to @shade/core/fingerprint.ts.
*
* Format: 12 groups of 5 decimal digits.
* Derived from: HKDF-SHA256(signingKey||dhKey, salt=32 zeros, info="ShadeFingerprint", 30)
* then interpret each 2-byte pair as a 16-bit unsigned int mod 10^5.
*
* Note: the TS version uses only the first 24 bytes (2 bytes × 12 groups),
* not all 30. We mirror that here.
*/
fun computeFingerprint(
crypto: CryptoProvider,
signingPublicKey: ByteArray,
dhPublicKey: ByteArray,
): String {
val combined = ByteArray(signingPublicKey.size + dhPublicKey.size)
signingPublicKey.copyInto(combined, 0)
dhPublicKey.copyInto(combined, signingPublicKey.size)
val salt = ByteArray(32)
val info = "ShadeFingerprint".toByteArray(Charsets.UTF_8)
val hash = crypto.hkdf(combined, salt, info, 30)
val groups = mutableListOf<String>()
for (i in 0 until 12) {
val offset = i * 2
val value = ((hash[offset].toInt() and 0xff) shl 8) or (hash[offset + 1].toInt() and 0xff)
groups.add(value.toString().padStart(5, '0'))
}
return groups.joinToString(" ")
}
fun shortFingerprint(full: String): String {
return full.split(" ").take(4).joinToString(" ")
}

View File

@@ -0,0 +1,77 @@
package no.zyon.shade.group
import no.zyon.shade.crypto.CryptoProvider
import no.zyon.shade.protocol.kdfChainKey
import java.nio.ByteBuffer
/**
* Group sender-keys (Sesame). Mirror @shade/core/sender-keys.ts.
*
* Each sender maintains a chain key that ratchets forward with `kdfChainKey`
* — same primitive the Double Ratchet uses for its symmetric chain. Per-message
* AEAD AAD binds (groupId, senderAddress, iteration) so a captured ciphertext
* cannot be replayed under a different sender or group:
*
* aad = u16_be(groupIdLen) || groupId || u16_be(senderAddrLen) || senderAddr || u32_be(iteration)
*
* Each ciphertext is signed by the sender's Ed25519 key over `aad || ciphertext`,
* which is what receivers verify before advancing their chain.
*/
data class SenderKeyMessage(
val senderAddress: String,
val iteration: Int,
val ciphertext: ByteArray,
val nonce: ByteArray,
val signature: ByteArray,
)
fun encodeSenderHeader(groupId: String, senderAddress: String, iteration: Int): ByteArray {
val gBytes = groupId.toByteArray(Charsets.UTF_8)
val sBytes = senderAddress.toByteArray(Charsets.UTF_8)
require(gBytes.size <= 0xFFFF) { "groupId too long (>65535 UTF-8 bytes)" }
require(sBytes.size <= 0xFFFF) { "senderAddress too long (>65535 UTF-8 bytes)" }
val out = ByteArray(2 + gBytes.size + 2 + sBytes.size + 4)
val buf = ByteBuffer.wrap(out)
buf.putShort(gBytes.size.toShort())
buf.put(gBytes)
buf.putShort(sBytes.size.toShort())
buf.put(sBytes)
buf.putInt(iteration)
return out
}
/**
* Compute (newChainKey, messageKey, aad) for the next group message.
* Pure function; caller is responsible for state advancement and the AEAD/sign
* steps (which need access to the signing private key not exposed here).
*/
data class SenderStepResult(
val newChainKey: ByteArray,
val messageKey: ByteArray,
val aad: ByteArray,
)
fun senderKeyStep(
crypto: CryptoProvider,
chainKey: ByteArray,
groupId: String,
senderAddress: String,
iteration: Int,
): SenderStepResult {
val r = kdfChainKey(crypto, chainKey)
val aad = encodeSenderHeader(groupId, senderAddress, iteration)
return SenderStepResult(newChainKey = r.newChainKey, messageKey = r.messageKey, aad = aad)
}
/**
* Concatenate `aad || ciphertext` — the byte string the sender signs and the
* receiver verifies. Exposed as a helper so vector parity can pin both sides.
*/
fun senderSignedBytes(aad: ByteArray, ciphertext: ByteArray): ByteArray {
val out = ByteArray(aad.size + ciphertext.size)
aad.copyInto(out, 0)
ciphertext.copyInto(out, aad.size)
return out
}

View File

@@ -0,0 +1,237 @@
package no.zyon.shade.protocol
import no.zyon.shade.crypto.CryptoProvider
import no.zyon.shade.types.ChainState
import no.zyon.shade.types.Constants
import no.zyon.shade.types.KeyPair
import no.zyon.shade.types.RatchetMessage
import no.zyon.shade.types.SessionState
import java.nio.ByteBuffer
/**
* Double Ratchet implementation. Mirrors @shade/core/ratchet.ts.
*
* Must produce byte-identical ciphertext to the TypeScript version
* for the same inputs.
*/
// ─── Session initialization ─────────────────────────────────
fun initSenderSession(
crypto: CryptoProvider,
rootKey: ByteArray,
remoteIdentityKey: ByteArray,
remoteDHPublicKey: ByteArray,
): SessionState {
val (dhSendPub, dhSendPriv) = crypto.generateX25519KeyPair()
val dhOutput = crypto.x25519(dhSendPriv, remoteDHPublicKey)
val (newRootKey, chainKey) = kdfRootKey(crypto, rootKey, dhOutput).let {
it.newRootKey to it.chainKey
}
return SessionState(
remoteIdentityKey = remoteIdentityKey,
rootKey = newRootKey,
sendChain = ChainState(chainKey = chainKey, counter = 0),
receiveChain = null,
dhSend = KeyPair(publicKey = dhSendPub, privateKey = dhSendPriv),
dhReceive = remoteDHPublicKey,
previousSendCounter = 0,
skippedKeys = mutableMapOf(),
)
}
fun initReceiverSession(
rootKey: ByteArray,
remoteIdentityKey: ByteArray,
localDHKeyPair: KeyPair,
): SessionState {
return SessionState(
remoteIdentityKey = remoteIdentityKey,
rootKey = rootKey,
sendChain = ChainState(chainKey = ByteArray(32), counter = 0),
receiveChain = null,
dhSend = localDHKeyPair,
dhReceive = null,
previousSendCounter = 0,
skippedKeys = mutableMapOf(),
)
}
// ─── Header encoding (for AES-GCM AAD) ──────────────────────
private fun encodeHeader(
dhPublicKey: ByteArray,
previousCounter: Int,
counter: Int,
): ByteArray {
val buf = ByteBuffer.allocate(40)
buf.put(dhPublicKey)
buf.putInt(previousCounter) // big-endian by default in ByteBuffer
buf.putInt(counter)
return buf.array()
}
// ─── Encrypt ─────────────────────────────────────────────────
fun ratchetEncrypt(
crypto: CryptoProvider,
session: SessionState,
plaintext: ByteArray,
): RatchetMessage {
val oldChainKey = session.sendChain.chainKey
val (newChainKey, messageKey) = kdfChainKey(crypto, oldChainKey).let {
it.newChainKey to it.messageKey
}
crypto.zeroize(oldChainKey)
val counter = session.sendChain.counter
val header = encodeHeader(session.dhSend.publicKey, session.previousSendCounter, counter)
val (ciphertext, nonce) = crypto.aesGcmEncrypt(messageKey, plaintext, header)
crypto.zeroize(messageKey)
session.sendChain.chainKey = newChainKey
session.sendChain.counter = counter + 1
return RatchetMessage(
dhPublicKey = session.dhSend.publicKey,
previousCounter = session.previousSendCounter,
counter = counter,
ciphertext = ciphertext,
nonce = nonce,
)
}
// ─── Decrypt ─────────────────────────────────────────────────
private fun skippedKeyId(dhPublicKey: ByteArray, counter: Int): String {
return dhPublicKey.joinToString("") { "%02x".format(it) } + ":" + counter
}
fun ratchetDecrypt(
crypto: CryptoProvider,
session: SessionState,
message: RatchetMessage,
): ByteArray {
// Case 1: skipped key
val skipId = skippedKeyId(message.dhPublicKey, message.counter)
val skippedKey = session.skippedKeys[skipId]
if (skippedKey != null) {
session.skippedKeys.remove(skipId)
try {
return decryptWithKey(crypto, skippedKey, message)
} finally {
crypto.zeroize(skippedKey)
}
}
// Case 2 or 3: DH ratchet check
val isNewRatchet = session.dhReceive == null ||
!message.dhPublicKey.contentEquals(session.dhReceive!!)
if (isNewRatchet) {
if (session.receiveChain != null && session.dhReceive != null) {
skipMessageKeys(
crypto,
session,
session.dhReceive!!,
session.receiveChain!!,
message.previousCounter,
)
}
performDHRatchetStep(crypto, session, message.dhPublicKey)
}
val receiveChain = session.receiveChain
?: throw IllegalStateException("No receiving chain available")
skipMessageKeys(crypto, session, message.dhPublicKey, receiveChain, message.counter)
val oldChainKey = receiveChain.chainKey
val (newChainKey, messageKey) = kdfChainKey(crypto, oldChainKey).let {
it.newChainKey to it.messageKey
}
crypto.zeroize(oldChainKey)
receiveChain.chainKey = newChainKey
receiveChain.counter = message.counter + 1
try {
return decryptWithKey(crypto, messageKey, message)
} finally {
crypto.zeroize(messageKey)
}
}
private fun performDHRatchetStep(
crypto: CryptoProvider,
session: SessionState,
remoteDHKey: ByteArray,
) {
session.previousSendCounter = session.sendChain.counter
session.dhReceive = remoteDHKey
// DH with current send key → new receiving chain
val dh1 = crypto.x25519(session.dhSend.privateKey, remoteDHKey)
val oldRootKey1 = session.rootKey
val recv = kdfRootKey(crypto, oldRootKey1, dh1)
crypto.zeroize(oldRootKey1)
crypto.zeroize(dh1)
session.rootKey = recv.newRootKey
session.receiveChain = ChainState(chainKey = recv.chainKey, counter = 0)
// Generate new DH keypair, zero old private
val oldDhPrivate = session.dhSend.privateKey
val (newDhPub, newDhPriv) = crypto.generateX25519KeyPair()
session.dhSend = KeyPair(publicKey = newDhPub, privateKey = newDhPriv)
crypto.zeroize(oldDhPrivate)
// DH with new send key → new sending chain
val dh2 = crypto.x25519(newDhPriv, remoteDHKey)
val oldRootKey2 = session.rootKey
val send = kdfRootKey(crypto, oldRootKey2, dh2)
crypto.zeroize(oldRootKey2)
crypto.zeroize(dh2)
session.rootKey = send.newRootKey
if (session.sendChain.chainKey.isNotEmpty()) {
crypto.zeroize(session.sendChain.chainKey)
}
session.sendChain = ChainState(chainKey = send.chainKey, counter = 0)
}
private fun skipMessageKeys(
crypto: CryptoProvider,
session: SessionState,
dhPublicKey: ByteArray,
chain: ChainState,
untilCounter: Int,
) {
val toSkip = untilCounter - chain.counter
if (toSkip < 0) return
if (toSkip > Constants.MAX_SKIP) {
throw IllegalStateException("Cannot skip $toSkip messages (max: ${Constants.MAX_SKIP})")
}
for (i in chain.counter until untilCounter) {
val (newChainKey, messageKey) = kdfChainKey(crypto, chain.chainKey).let {
it.newChainKey to it.messageKey
}
val id = skippedKeyId(dhPublicKey, i)
session.skippedKeys[id] = messageKey
chain.chainKey = newChainKey
chain.counter = i + 1
while (session.skippedKeys.size > Constants.MAX_CACHED_SKIPPED_KEYS) {
val firstKey = session.skippedKeys.keys.first()
session.skippedKeys.remove(firstKey)
}
}
}
private fun decryptWithKey(
crypto: CryptoProvider,
messageKey: ByteArray,
message: RatchetMessage,
): ByteArray {
val aad = encodeHeader(message.dhPublicKey, message.previousCounter, message.counter)
return crypto.aesGcmDecrypt(messageKey, message.ciphertext, message.nonce, aad)
}

View File

@@ -0,0 +1,60 @@
package no.zyon.shade.protocol
import no.zyon.shade.crypto.CryptoProvider
/**
* KDF chain functions for the Signal Protocol ratchet.
*
* MUST produce byte-identical output to @shade/core/keys.ts.
* Info strings and salts are fixed constants and must not change.
*/
// Must match the TypeScript version EXACTLY
private val ROOT_KDF_INFO = "ShadeRootRatchet".toByteArray(Charsets.UTF_8)
private val CHAIN_KEY_CONSTANT = byteArrayOf(0x01)
private val MESSAGE_KEY_CONSTANT = byteArrayOf(0x02)
private val X3DH_INFO = "ShadeX3DH".toByteArray(Charsets.UTF_8)
private val X3DH_SALT = ByteArray(32) // 32 zero bytes
data class RootKdfResult(val newRootKey: ByteArray, val chainKey: ByteArray)
data class ChainKdfResult(val newChainKey: ByteArray, val messageKey: ByteArray)
/**
* Root key ratchet step.
* HKDF(ikm=dhOutput, salt=rootKey, info="ShadeRootRatchet", length=64)
* → first 32 bytes = new root key, last 32 bytes = chain key
*/
fun kdfRootKey(crypto: CryptoProvider, rootKey: ByteArray, dhOutput: ByteArray): RootKdfResult {
val derived = crypto.hkdf(dhOutput, rootKey, ROOT_KDF_INFO, 64)
return RootKdfResult(
newRootKey = derived.copyOfRange(0, 32),
chainKey = derived.copyOfRange(32, 64),
)
}
/**
* Chain key ratchet step.
* HMAC(chainKey, 0x01) = new chain key
* HMAC(chainKey, 0x02) = message key (used once)
*/
fun kdfChainKey(crypto: CryptoProvider, chainKey: ByteArray): ChainKdfResult {
val newChainKey = crypto.hmacSha256(chainKey, CHAIN_KEY_CONSTANT)
val messageKey = crypto.hmacSha256(chainKey, MESSAGE_KEY_CONSTANT)
return ChainKdfResult(newChainKey, messageKey)
}
/**
* Derive the initial root key from concatenated X3DH DH outputs.
* HKDF(ikm=DH1||DH2||DH3[||DH4], salt=32 zeros, info="ShadeX3DH", length=32)
*/
fun deriveInitialRootKey(crypto: CryptoProvider, sharedSecrets: List<ByteArray>): ByteArray {
val total = sharedSecrets.sumOf { it.size }
val ikm = ByteArray(total)
var offset = 0
for (secret in sharedSecrets) {
secret.copyInto(ikm, offset)
offset += secret.size
}
return crypto.hkdf(ikm, X3DH_SALT, X3DH_INFO, 32)
}

View File

@@ -0,0 +1,181 @@
package no.zyon.shade.protocol
import no.zyon.shade.crypto.CryptoProvider
import no.zyon.shade.types.IdentityKeyPair
import no.zyon.shade.types.KeyPair
import no.zyon.shade.types.OneTimePreKey
import no.zyon.shade.types.PreKeyBundle
import no.zyon.shade.types.PreKeyMessage
import no.zyon.shade.types.SignedPreKey
/**
* X3DH key agreement. Mirrors @shade/core/x3dh.ts.
*
* Identity keys: separate Ed25519 (signing) + X25519 (DH) keypairs stored together.
*/
/** Generate a new identity keypair (Ed25519 + X25519) */
fun generateIdentityKeyPair(crypto: CryptoProvider): IdentityKeyPair {
val (signPub, signPriv) = crypto.generateEd25519KeyPair()
val (dhPub, dhPriv) = crypto.generateX25519KeyPair()
return IdentityKeyPair(
signingPublicKey = signPub,
signingPrivateKey = signPriv,
dhPublicKey = dhPub,
dhPrivateKey = dhPriv,
)
}
/** Generate a signed prekey (X25519 keypair + Ed25519 signature over public key) */
fun generateSignedPreKey(
crypto: CryptoProvider,
identity: IdentityKeyPair,
keyId: Int,
): SignedPreKey {
val (pub, priv) = crypto.generateX25519KeyPair()
val signature = crypto.sign(identity.signingPrivateKey, pub)
return SignedPreKey(
keyId = keyId,
keyPair = KeyPair(publicKey = pub, privateKey = priv),
signature = signature,
timestamp = System.currentTimeMillis(),
)
}
/** Generate a batch of one-time prekeys */
fun generateOneTimePreKeys(
crypto: CryptoProvider,
startId: Int,
count: Int,
): List<OneTimePreKey> {
val keys = mutableListOf<OneTimePreKey>()
for (i in 0 until count) {
val (pub, priv) = crypto.generateX25519KeyPair()
keys.add(OneTimePreKey(keyId = startId + i, keyPair = KeyPair(pub, priv)))
}
return keys
}
fun createPreKeyBundle(
registrationId: Int,
identity: IdentityKeyPair,
signedPreKey: SignedPreKey,
oneTimePreKey: OneTimePreKey? = null,
): PreKeyBundle {
return PreKeyBundle(
registrationId = registrationId,
identitySigningKey = identity.signingPublicKey,
identityDHKey = identity.dhPublicKey,
signedPreKey = PreKeyBundle.BundleSignedPreKey(
keyId = signedPreKey.keyId,
publicKey = signedPreKey.keyPair.publicKey,
signature = signedPreKey.signature,
),
oneTimePreKey = oneTimePreKey?.let {
PreKeyBundle.BundleOneTimePreKey(it.keyId, it.keyPair.publicKey)
},
)
}
/** Result of processing a prekey bundle (Alice's side) */
data class X3DHInitResult(
val rootKey: ByteArray,
val ephemeralPublicKey: ByteArray,
val signedPreKeyId: Int,
val preKeyId: Int?,
val remoteIdentityKey: ByteArray,
val remoteSignedPreKey: ByteArray,
)
/**
* Alice processes Bob's prekey bundle to establish a session.
*
* Steps:
* 1. Verify the signed prekey signature
* 2. Generate an ephemeral X25519 keypair
* 3. Compute DH1 = DH(Alice identity DH, Bob signed prekey)
* 4. Compute DH2 = DH(Alice ephemeral, Bob identity DH)
* 5. Compute DH3 = DH(Alice ephemeral, Bob signed prekey)
* 6. Compute DH4 = DH(Alice ephemeral, Bob one-time prekey) if available
* 7. Derive initial root key from concatenated DH outputs
*/
fun processPreKeyBundle(
crypto: CryptoProvider,
identity: IdentityKeyPair,
bundle: PreKeyBundle,
): X3DHInitResult {
// 1. Verify signed prekey signature
val valid = crypto.verify(
bundle.identitySigningKey,
bundle.signedPreKey.publicKey,
bundle.signedPreKey.signature,
)
if (!valid) throw SecurityException("Signed prekey signature is invalid")
// 2. Ephemeral keypair
val (ephPub, ephPriv) = crypto.generateX25519KeyPair()
// 3-6. DH computations
val dh1 = crypto.x25519(identity.dhPrivateKey, bundle.signedPreKey.publicKey)
val dh2 = crypto.x25519(ephPriv, bundle.identityDHKey)
val dh3 = crypto.x25519(ephPriv, bundle.signedPreKey.publicKey)
val secrets = mutableListOf(dh1, dh2, dh3)
var preKeyId: Int? = null
if (bundle.oneTimePreKey != null) {
val dh4 = crypto.x25519(ephPriv, bundle.oneTimePreKey.publicKey)
secrets.add(dh4)
preKeyId = bundle.oneTimePreKey.keyId
}
// 7. Derive root key
val rootKey = deriveInitialRootKey(crypto, secrets)
return X3DHInitResult(
rootKey = rootKey,
ephemeralPublicKey = ephPub,
signedPreKeyId = bundle.signedPreKey.keyId,
preKeyId = preKeyId,
remoteIdentityKey = bundle.identityDHKey,
remoteSignedPreKey = bundle.signedPreKey.publicKey,
)
}
/** Result of processing an incoming PreKeyMessage (Bob's side) */
data class X3DHResponseResult(
val rootKey: ByteArray,
val remoteIdentityKey: ByteArray,
val remoteEphemeralKey: ByteArray,
)
/**
* Bob processes an incoming PreKeyMessage to establish a session.
* Mirrors Alice's DH computations from Bob's perspective.
*
* Caller is responsible for looking up the signed prekey and (if present)
* the one-time prekey from storage.
*/
fun processPreKeyMessage(
crypto: CryptoProvider,
identity: IdentityKeyPair,
signedPreKeyPrivate: ByteArray,
oneTimePreKeyPrivate: ByteArray?,
message: PreKeyMessage,
): X3DHResponseResult {
val dh1 = crypto.x25519(signedPreKeyPrivate, message.identityDHKey)
val dh2 = crypto.x25519(identity.dhPrivateKey, message.ephemeralKey)
val dh3 = crypto.x25519(signedPreKeyPrivate, message.ephemeralKey)
val secrets = mutableListOf(dh1, dh2, dh3)
if (oneTimePreKeyPrivate != null) {
val dh4 = crypto.x25519(oneTimePreKeyPrivate, message.ephemeralKey)
secrets.add(dh4)
}
val rootKey = deriveInitialRootKey(crypto, secrets)
return X3DHResponseResult(
rootKey = rootKey,
remoteIdentityKey = message.identityDHKey,
remoteEphemeralKey = message.ephemeralKey,
)
}

View File

@@ -0,0 +1,133 @@
package no.zyon.shade.serialization
import no.zyon.shade.types.ChainState
import no.zyon.shade.types.IdentityKeyPair
import no.zyon.shade.types.KeyPair
import no.zyon.shade.types.OneTimePreKey
import no.zyon.shade.types.SessionState
import no.zyon.shade.types.SignedPreKey
import org.json.JSONObject
import java.util.Base64 as JdkBase64
/**
* Plain-JSON serialization for the persisted protocol state types
* (`IdentityKeyPair`, `SignedPreKey`, `OneTimePreKey`, `SessionState`).
*
* The on-disk shape is for at-rest storage only — it does NOT need
* to round-trip across platforms (TS uses its own JSON shape via
* `@shade/core/serialization`). What matters is that the Kotlin
* round-trip (`serialize` then `deserialize`) preserves every byte.
*
* Both Android-targeted (`shade-android-keystore`) and pure-JVM
* (`shade-android` tests) callers use this — the function works
* without any `android.*` imports so it compiles in both.
*/
object SessionStateJson {
fun serialize(state: SessionState): String {
val o = JSONObject()
o.put("remoteIdentityKey", b64(state.remoteIdentityKey))
o.put("rootKey", b64(state.rootKey))
o.put("sendChain", chainToJson(state.sendChain))
if (state.receiveChain != null) o.put("receiveChain", chainToJson(state.receiveChain!!))
o.put("dhSend", keyPairToJson(state.dhSend))
if (state.dhReceive != null) o.put("dhReceive", b64(state.dhReceive!!))
o.put("previousSendCounter", state.previousSendCounter)
val skipped = JSONObject()
for ((k, v) in state.skippedKeys) skipped.put(k, b64(v))
o.put("skippedKeys", skipped)
return o.toString()
}
fun deserialize(s: String): SessionState {
val o = JSONObject(s)
val skipped = mutableMapOf<String, ByteArray>()
val skJson = o.optJSONObject("skippedKeys")
if (skJson != null) {
val it = skJson.keys()
while (it.hasNext()) {
val k = it.next()
skipped[k] = fb64(skJson.getString(k))
}
}
return SessionState(
remoteIdentityKey = fb64(o.getString("remoteIdentityKey")),
rootKey = fb64(o.getString("rootKey")),
sendChain = chainFromJson(o.getJSONObject("sendChain")),
receiveChain = if (o.has("receiveChain"))
chainFromJson(o.getJSONObject("receiveChain")) else null,
dhSend = keyPairFromJson(o.getJSONObject("dhSend")),
dhReceive = if (o.has("dhReceive")) fb64(o.getString("dhReceive")) else null,
previousSendCounter = o.getInt("previousSendCounter"),
skippedKeys = skipped,
)
}
fun serializeIdentityKeyPair(k: IdentityKeyPair): String = JSONObject().apply {
put("signingPublicKey", b64(k.signingPublicKey))
put("signingPrivateKey", b64(k.signingPrivateKey))
put("dhPublicKey", b64(k.dhPublicKey))
put("dhPrivateKey", b64(k.dhPrivateKey))
}.toString()
fun deserializeIdentityKeyPair(s: String): IdentityKeyPair = JSONObject(s).run {
IdentityKeyPair(
signingPublicKey = fb64(getString("signingPublicKey")),
signingPrivateKey = fb64(getString("signingPrivateKey")),
dhPublicKey = fb64(getString("dhPublicKey")),
dhPrivateKey = fb64(getString("dhPrivateKey")),
)
}
fun serializeSignedPreKey(k: SignedPreKey): String = JSONObject().apply {
put("keyId", k.keyId)
put("keyPair", keyPairToJson(k.keyPair))
put("signature", b64(k.signature))
put("timestamp", k.timestamp)
}.toString()
fun deserializeSignedPreKey(s: String): SignedPreKey = JSONObject(s).run {
SignedPreKey(
keyId = getInt("keyId"),
keyPair = keyPairFromJson(getJSONObject("keyPair")),
signature = fb64(getString("signature")),
timestamp = getLong("timestamp"),
)
}
fun serializeOneTimePreKey(k: OneTimePreKey): String = JSONObject().apply {
put("keyId", k.keyId)
put("keyPair", keyPairToJson(k.keyPair))
}.toString()
fun deserializeOneTimePreKey(s: String): OneTimePreKey = JSONObject(s).run {
OneTimePreKey(
keyId = getInt("keyId"),
keyPair = keyPairFromJson(getJSONObject("keyPair")),
)
}
private fun chainToJson(c: ChainState): JSONObject = JSONObject().apply {
put("chainKey", b64(c.chainKey))
put("counter", c.counter)
}
private fun chainFromJson(o: JSONObject): ChainState =
ChainState(chainKey = fb64(o.getString("chainKey")), counter = o.getInt("counter"))
private fun keyPairToJson(k: KeyPair): JSONObject = JSONObject().apply {
put("publicKey", b64(k.publicKey))
put("privateKey", b64(k.privateKey))
}
private fun keyPairFromJson(o: JSONObject): KeyPair = KeyPair(
publicKey = fb64(o.getString("publicKey")),
privateKey = fb64(o.getString("privateKey")),
)
// android.util.Base64 isn't on the JVM classpath; java.util.Base64
// is available on both modern JVM and Android API 26+. Use JDK
// Base64 throughout — it's present on both targets.
private fun b64(b: ByteArray): String = JdkBase64.getEncoder().encodeToString(b)
private fun fb64(s: String): ByteArray = JdkBase64.getDecoder().decode(s)
}

View File

@@ -0,0 +1,145 @@
package no.zyon.shade.serialization
import no.zyon.shade.streams.StreamConstants
import java.nio.ByteBuffer
/**
* Wire-decoded stream-chunk envelope (type 0x11).
*
* Mirror @shade/proto/wire.ts `StreamChunkWire`. The nonce is deterministic
* (derived from `(laneId, seq)` on both sides) but is also serialized over
* the wire for self-description and validated by the receiver.
*
* `seq` is unsigned-u64 on the wire; on the JVM we keep it as Long. The
* encode/decode helpers operate on the raw 8-byte big-endian representation,
* so values past Long.MAX_VALUE roundtrip via `Long.toULong()`.
*/
data class StreamChunkWire(
val streamId: ByteArray,
val laneId: Long,
val seq: Long,
val isLast: Boolean,
val nonce: ByteArray,
val aad: ByteArray,
val ciphertext: ByteArray,
) {
override fun equals(other: Any?): Boolean {
if (this === other) return true
if (other !is StreamChunkWire) return false
return streamId.contentEquals(other.streamId) &&
laneId == other.laneId &&
seq == other.seq &&
isLast == other.isLast &&
nonce.contentEquals(other.nonce) &&
aad.contentEquals(other.aad) &&
ciphertext.contentEquals(other.ciphertext)
}
override fun hashCode(): Int {
var result = streamId.contentHashCode()
result = 31 * result + laneId.hashCode()
result = 31 * result + seq.hashCode()
result = 31 * result + isLast.hashCode()
result = 31 * result + nonce.contentHashCode()
result = 31 * result + aad.contentHashCode()
result = 31 * result + ciphertext.contentHashCode()
return result
}
}
/** Stream-chunk wire codec. Mirror @shade/proto/wire.ts `encodeStreamChunk`/`decodeStreamChunk`. */
object StreamChunkWireFormat {
private const val VERSION: Byte = 0x02
const val TYPE_STREAM_CHUNK: Byte = 0x11
fun encodeStreamChunk(c: StreamChunkWire): ByteArray {
require(c.streamId.size == StreamConstants.STREAM_ID_BYTES) {
"streamId must be ${StreamConstants.STREAM_ID_BYTES} bytes"
}
require(c.nonce.size == StreamConstants.STREAM_NONCE_BYTES) {
"nonce must be ${StreamConstants.STREAM_NONCE_BYTES} bytes"
}
require(c.laneId in 0L..0xFFFFFFFFL) { "laneId out of u32 range: ${c.laneId}" }
// c.seq is unsigned-u64; negative signed longs encode as the high half
// of the u64 range. ByteBuffer.putLong writes the raw 8-byte pattern.
val headerSize =
1 + 1 +
StreamConstants.STREAM_ID_BYTES +
4 + 8 + 1 +
StreamConstants.STREAM_NONCE_BYTES +
4 + c.aad.size +
4
val out = ByteArray(headerSize + c.ciphertext.size)
val buf = ByteBuffer.wrap(out)
buf.put(VERSION)
buf.put(TYPE_STREAM_CHUNK)
buf.put(c.streamId)
buf.putInt(c.laneId.toInt())
buf.putLong(c.seq)
buf.put(if (c.isLast) 0x01.toByte() else 0x00.toByte())
buf.put(c.nonce)
buf.putInt(c.aad.size)
buf.put(c.aad)
buf.putInt(c.ciphertext.size)
buf.put(c.ciphertext)
return out
}
fun decodeStreamChunk(data: ByteArray): StreamChunkWire {
val minHeaderSize = 2 +
StreamConstants.STREAM_ID_BYTES +
4 + 8 + 1 +
StreamConstants.STREAM_NONCE_BYTES +
4 + 4
require(data.size >= minHeaderSize) {
"stream-chunk too short: ${data.size} < $minHeaderSize"
}
require(data[0] == VERSION) { "Unknown version: ${data[0]}" }
require(data[1] == TYPE_STREAM_CHUNK) { "Not a stream-chunk: type=${data[1]}" }
val buf = ByteBuffer.wrap(data)
buf.position(2)
val streamId = ByteArray(StreamConstants.STREAM_ID_BYTES)
buf.get(streamId)
val laneId = buf.int.toLong() and 0xFFFFFFFFL
val seq = buf.long
val isLast = buf.get() == 0x01.toByte()
val nonce = ByteArray(StreamConstants.STREAM_NONCE_BYTES)
buf.get(nonce)
val aadLen = buf.int
require(buf.position() + aadLen + 4 <= data.size) {
"stream-chunk truncated in aad/ctLen"
}
val aad = ByteArray(aadLen)
buf.get(aad)
val ctLen = buf.int
require(buf.position() + ctLen == data.size) {
"stream-chunk length mismatch: declared ${buf.position() + ctLen}, actual ${data.size}"
}
val ciphertext = ByteArray(ctLen)
buf.get(ciphertext)
return StreamChunkWire(streamId, laneId, seq, isLast, nonce, aad, ciphertext)
}
/** Inspect the type tag without full parsing. Mirror @shade/proto/wire.ts. */
enum class EnvelopeKind { PREKEY, RATCHET, STREAM_CHUNK, UNKNOWN }
fun inspectEnvelopeType(data: ByteArray): EnvelopeKind {
if (data.size < 2 || data[0] != VERSION) return EnvelopeKind.UNKNOWN
return when (data[1]) {
0x01.toByte() -> EnvelopeKind.PREKEY
0x02.toByte() -> EnvelopeKind.RATCHET
TYPE_STREAM_CHUNK -> EnvelopeKind.STREAM_CHUNK
else -> EnvelopeKind.UNKNOWN
}
}
}

View File

@@ -0,0 +1,172 @@
package no.zyon.shade.serialization
import no.zyon.shade.types.PreKeyMessage
import no.zyon.shade.types.RatchetMessage
import no.zyon.shade.types.ShadeEnvelope
import java.nio.ByteBuffer
/**
* Compact binary wire format. MUST match @shade/proto/wire.ts byte-for-byte.
*
* Format: [version:1][type:1][payload...]
* Types: 0x01 = PreKeyMessage, 0x02 = RatchetMessage
* Integers: big-endian
* Byte arrays: 4-byte (u32) length prefix + data (since wire VERSION 0x02).
*
* VERSION 0x01 used a 2-byte length prefix and was capped at 64 KiB
* payloads — incompatible with inline file ops up to 256 KiB.
*/
object WireFormat {
private const val VERSION: Byte = 0x02
private const val TYPE_PREKEY: Byte = 0x01
private const val TYPE_RATCHET: Byte = 0x02
private const val PREKEY_NONE: Long = 0xFFFFFFFFL
// ─── Encode ──────────────────────────────────────────────
fun encodeEnvelope(envelope: ShadeEnvelope): ByteArray {
return when (envelope.type) {
ShadeEnvelope.EnvelopeType.PREKEY ->
encodePreKeyMessage(envelope.content as PreKeyMessage)
ShadeEnvelope.EnvelopeType.RATCHET ->
encodeRatchetMessage(envelope.content as RatchetMessage)
}
}
fun encodePreKeyMessage(msg: PreKeyMessage): ByteArray {
val ratchetBytes = encodeRatchetInner(msg.message)
val parts = mutableListOf<ByteArray>()
parts.add(byteArrayOf(VERSION, TYPE_PREKEY))
parts.add(uint32(msg.registrationId.toLong()))
parts.add(uint32(msg.preKeyId?.toLong() ?: PREKEY_NONE))
parts.add(uint32(msg.signedPreKeyId.toLong()))
parts.add(lpBytes(msg.ephemeralKey))
parts.add(lpBytes(msg.identityDHKey))
parts.add(lpBytes(ratchetBytes))
return concat(parts)
}
fun encodeRatchetMessage(msg: RatchetMessage): ByteArray {
val parts = mutableListOf<ByteArray>()
parts.add(byteArrayOf(VERSION, TYPE_RATCHET))
parts.add(encodeRatchetInner(msg))
return concat(parts)
}
private fun encodeRatchetInner(msg: RatchetMessage): ByteArray {
val parts = mutableListOf<ByteArray>()
parts.add(lpBytes(msg.dhPublicKey))
parts.add(uint32(msg.previousCounter.toLong()))
parts.add(uint32(msg.counter.toLong()))
parts.add(lpBytes(msg.ciphertext))
parts.add(lpBytes(msg.nonce))
return concat(parts)
}
// ─── Decode ──────────────────────────────────────────────
fun decodeEnvelope(data: ByteArray): ShadeEnvelope {
if (data.size < 2) throw IllegalArgumentException("Too short")
val version = data[0]
if (version != VERSION) throw IllegalArgumentException("Unknown version: $version")
val type = data[1]
val payload = data.copyOfRange(2, data.size)
return when (type) {
TYPE_PREKEY -> ShadeEnvelope(
type = ShadeEnvelope.EnvelopeType.PREKEY,
content = decodePreKeyMessageInner(payload),
timestamp = 0,
senderAddress = "",
)
TYPE_RATCHET -> {
val (msg, _) = decodeRatchetInner(payload, 0)
ShadeEnvelope(
type = ShadeEnvelope.EnvelopeType.RATCHET,
content = msg,
timestamp = 0,
senderAddress = "",
)
}
else -> throw IllegalArgumentException("Unknown type: $type")
}
}
private fun decodePreKeyMessageInner(data: ByteArray): PreKeyMessage {
var offset = 0
val registrationId = readUint32(data, offset).toInt(); offset += 4
val preKeyIdRaw = readUint32(data, offset); offset += 4
val preKeyId = if (preKeyIdRaw == PREKEY_NONE) null else preKeyIdRaw.toInt()
val signedPreKeyId = readUint32(data, offset).toInt(); offset += 4
val ephemeral = readLP(data, offset); offset = ephemeral.second
val identityDH = readLP(data, offset); offset = identityDH.second
val ratchetData = readLP(data, offset); offset = ratchetData.second
val (ratchet, _) = decodeRatchetInner(ratchetData.first, 0)
return PreKeyMessage(
registrationId = registrationId,
preKeyId = preKeyId,
signedPreKeyId = signedPreKeyId,
ephemeralKey = ephemeral.first,
identityDHKey = identityDH.first,
message = ratchet,
)
}
private fun decodeRatchetInner(data: ByteArray, startOffset: Int): Pair<RatchetMessage, Int> {
var offset = startOffset
val dhPub = readLP(data, offset); offset = dhPub.second
val prevCounter = readUint32(data, offset).toInt(); offset += 4
val counter = readUint32(data, offset).toInt(); offset += 4
val ciphertext = readLP(data, offset); offset = ciphertext.second
val nonce = readLP(data, offset); offset = nonce.second
return RatchetMessage(
dhPublicKey = dhPub.first,
previousCounter = prevCounter,
counter = counter,
ciphertext = ciphertext.first,
nonce = nonce.first,
) to offset
}
// ─── Helpers ─────────────────────────────────────────────
private fun uint32(n: Long): ByteArray {
val buf = ByteBuffer.allocate(4)
buf.putInt(n.toInt())
return buf.array()
}
private fun lpBytes(data: ByteArray): ByteArray {
val len = ByteBuffer.allocate(4)
len.putInt(data.size)
return concat(listOf(len.array(), data))
}
private fun readUint32(data: ByteArray, offset: Int): Long {
return ((data[offset].toLong() and 0xff) shl 24) or
((data[offset + 1].toLong() and 0xff) shl 16) or
((data[offset + 2].toLong() and 0xff) shl 8) or
(data[offset + 3].toLong() and 0xff)
}
private fun readLP(data: ByteArray, offset: Int): Pair<ByteArray, Int> {
val len = readUint32(data, offset).toInt()
val value = data.copyOfRange(offset + 4, offset + 4 + len)
return value to (offset + 4 + len)
}
private fun concat(parts: List<ByteArray>): ByteArray {
val total = parts.sumOf { it.size }
val result = ByteArray(total)
var offset = 0
for (p in parts) {
p.copyInto(result, offset)
offset += p.size
}
return result
}
}

View File

@@ -0,0 +1,47 @@
package no.zyon.shade.storage
import no.zyon.shade.crypto.CryptoProvider
import no.zyon.shade.types.IdentityKeyPair
import no.zyon.shade.types.OneTimePreKey
import no.zyon.shade.types.SessionState
import no.zyon.shade.types.SignedPreKey
/**
* In-memory storage for tests and embedded use.
* Mirrors MemoryStorage in @shade/crypto-web.
*/
class MemoryStorage(private val crypto: CryptoProvider) : StorageProvider {
private var identity: IdentityKeyPair? = null
private var registrationId: Int = 0
private val signedPreKeys = mutableMapOf<Int, SignedPreKey>()
private val oneTimePreKeys = mutableMapOf<Int, OneTimePreKey>()
private val sessions = mutableMapOf<String, SessionState>()
private val trustedIdentities = mutableMapOf<String, ByteArray>()
override suspend fun getIdentityKeyPair(): IdentityKeyPair? = identity
override suspend fun saveIdentityKeyPair(keyPair: IdentityKeyPair) { identity = keyPair }
override suspend fun getLocalRegistrationId(): Int = registrationId
override suspend fun saveLocalRegistrationId(id: Int) { registrationId = id }
override suspend fun getSignedPreKey(keyId: Int): SignedPreKey? = signedPreKeys[keyId]
override suspend fun saveSignedPreKey(key: SignedPreKey) { signedPreKeys[key.keyId] = key }
override suspend fun removeSignedPreKey(keyId: Int) { signedPreKeys.remove(keyId) }
override suspend fun getOneTimePreKey(keyId: Int): OneTimePreKey? = oneTimePreKeys[keyId]
override suspend fun saveOneTimePreKey(key: OneTimePreKey) { oneTimePreKeys[key.keyId] = key }
override suspend fun removeOneTimePreKey(keyId: Int) { oneTimePreKeys.remove(keyId) }
override suspend fun getOneTimePreKeyCount(): Int = oneTimePreKeys.size
override suspend fun getSession(address: String): SessionState? = sessions[address]
override suspend fun saveSession(address: String, state: SessionState) { sessions[address] = state }
override suspend fun removeSession(address: String) { sessions.remove(address) }
override suspend fun isTrustedIdentity(address: String, identityKey: ByteArray): Boolean {
val stored = trustedIdentities[address] ?: return true // TOFU
return crypto.constantTimeEqual(stored, identityKey)
}
override suspend fun saveTrustedIdentity(address: String, identityKey: ByteArray) {
trustedIdentities[address] = identityKey
}
}

View File

@@ -0,0 +1,107 @@
package no.zyon.shade.storage
import org.bouncycastle.crypto.generators.Argon2BytesGenerator
import org.bouncycastle.crypto.generators.SCrypt
import org.bouncycastle.crypto.params.Argon2Parameters
import java.text.Normalizer
/**
* Password / PIN key-derivation primitives. Mirror
* `@shade/storage-encrypted/crypto/kdf` (`deriveMasterKey` /
* `deriveMasterKeyArgon2id`) byte-for-byte — Tink doesn't ship password
* KDFs so we wrap Bouncy Castle.
*
* Both functions normalize string passphrases to NFKC before hashing,
* matching the TS implementation's `passphrase.normalize('NFKC')`.
* This ensures the same password typed on different OSes/keyboards
* produces the same master key regardless of which compatibility-form
* the input arrived in.
*
* The reference test-vector lives in `test-vectors/storage-encryption.json`
* and `test-vectors/blob-storage.json`. Cross-platform parity is gated
* by `CrossPlatformVectorTest`.
*/
/** scrypt parameters. Defaults match `DEFAULT_SCRYPT` in TS. */
data class ScryptParams(
/** CPU/memory cost. Must be a power of 2. */
val n: Int = 1 shl 17,
/** Block size. */
val r: Int = 8,
/** Parallelization. */
val p: Int = 1,
/** Output length in bytes. */
val dkLen: Int = 32,
)
/** Argon2id parameters. Defaults match `DEFAULT_ARGON2ID` in TS. */
data class Argon2idParams(
/** Memory cost in KiB. Default 64 MiB. */
val m: Int = 64 * 1024,
/** Time cost (iterations). Default 3. */
val t: Int = 3,
/** Parallelism. Default 1. */
val p: Int = 1,
/** Output length in bytes. Default 32. */
val dkLen: Int = 32,
)
/**
* Derive a 32-byte master key from a passphrase + salt via scrypt.
* Salt MUST be at least 16 bytes and persisted alongside the
* encrypted database. Throws on empty passphrase.
*/
fun deriveMasterKey(
passphrase: String,
salt: ByteArray,
params: ScryptParams = ScryptParams(),
): ByteArray {
require(passphrase.isNotEmpty()) { "passphrase must be non-empty" }
require(salt.size >= 16) { "salt must be at least 16 bytes" }
val nfkc = Normalizer.normalize(passphrase, Normalizer.Form.NFKC)
val pwBytes = nfkc.toByteArray(Charsets.UTF_8)
return SCrypt.generate(pwBytes, salt, params.n, params.r, params.p, params.dkLen)
}
/**
* Derive a 32-byte master key from a low-entropy secret (PIN) + salt
* via argon2id. Salt MUST be at least 16 bytes. The Bouncy Castle
* `Argon2BytesGenerator` parameters mirror RFC 9106's argon2id mode
* with version 1.3 (`Argon2Parameters.ARGON2_VERSION_13`), which is
* what `@noble/hashes/argon2` produces — keeping cross-platform parity.
*/
fun deriveMasterKeyArgon2id(
secret: String,
salt: ByteArray,
params: Argon2idParams = Argon2idParams(),
): ByteArray {
require(secret.isNotEmpty()) { "argon2id secret must be non-empty" }
require(salt.size >= 16) { "salt must be at least 16 bytes" }
val nfkc = Normalizer.normalize(secret, Normalizer.Form.NFKC)
return deriveMasterKeyArgon2id(nfkc.toByteArray(Charsets.UTF_8), salt, params)
}
/**
* Byte-array overload — useful when the secret is already binary
* (e.g. derived from a hardware token rather than typed) and
* shouldn't be NFKC-normalized as text.
*/
fun deriveMasterKeyArgon2id(
secret: ByteArray,
salt: ByteArray,
params: Argon2idParams = Argon2idParams(),
): ByteArray {
require(secret.isNotEmpty()) { "argon2id secret must be non-empty" }
require(salt.size >= 16) { "salt must be at least 16 bytes" }
val builder = Argon2Parameters.Builder(Argon2Parameters.ARGON2_id)
.withVersion(Argon2Parameters.ARGON2_VERSION_13)
.withIterations(params.t)
.withMemoryAsKB(params.m)
.withParallelism(params.p)
.withSalt(salt)
val gen = Argon2BytesGenerator()
gen.init(builder.build())
val out = ByteArray(params.dkLen)
gen.generateBytes(secret, out)
return out
}

View File

@@ -0,0 +1,42 @@
package no.zyon.shade.storage
import no.zyon.shade.types.IdentityKeyPair
import no.zyon.shade.types.OneTimePreKey
import no.zyon.shade.types.SessionState
import no.zyon.shade.types.SignedPreKey
/**
* StorageProvider interface. Mirror @shade/core/storage.ts.
*
* Implementations:
* - MemoryStorage (for tests)
* - KeystoreStorage (EncryptedSharedPreferences + Android Keystore)
* - RoomStorage (SQLite via Room, for larger datasets)
*/
interface StorageProvider {
// Identity
suspend fun getIdentityKeyPair(): IdentityKeyPair?
suspend fun saveIdentityKeyPair(keyPair: IdentityKeyPair)
suspend fun getLocalRegistrationId(): Int
suspend fun saveLocalRegistrationId(id: Int)
// Signed prekeys
suspend fun getSignedPreKey(keyId: Int): SignedPreKey?
suspend fun saveSignedPreKey(key: SignedPreKey)
suspend fun removeSignedPreKey(keyId: Int)
// One-time prekeys
suspend fun getOneTimePreKey(keyId: Int): OneTimePreKey?
suspend fun saveOneTimePreKey(key: OneTimePreKey)
suspend fun removeOneTimePreKey(keyId: Int)
suspend fun getOneTimePreKeyCount(): Int
// Sessions
suspend fun getSession(address: String): SessionState?
suspend fun saveSession(address: String, state: SessionState)
suspend fun removeSession(address: String)
// Trust
suspend fun isTrustedIdentity(address: String, identityKey: ByteArray): Boolean
suspend fun saveTrustedIdentity(address: String, identityKey: ByteArray)
}

View File

@@ -0,0 +1,48 @@
package no.zyon.shade.streams
import javax.crypto.Cipher
import javax.crypto.spec.GCMParameterSpec
import javax.crypto.spec.SecretKeySpec
/**
* AES-256-GCM with caller-supplied nonce. Mirror @shade/streams/aead.ts.
*
* Unlike `CryptoProvider.aesGcmEncrypt` (which generates a random nonce
* internally), streams require deterministic nonces derived from
* `(laneId, seq)`. Returns the ciphertext concatenated with the 16-byte
* authentication tag — same layout SubtleCrypto produces.
*/
const val AEAD_TAG_BYTES = 16
fun aesGcmEncryptWithNonce(
key: ByteArray,
nonce: ByteArray,
plaintext: ByteArray,
aad: ByteArray,
): ByteArray {
require(nonce.size == StreamConstants.STREAM_NONCE_BYTES) {
"AES-GCM nonce must be ${StreamConstants.STREAM_NONCE_BYTES} bytes"
}
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val spec = GCMParameterSpec(128, nonce)
cipher.init(Cipher.ENCRYPT_MODE, SecretKeySpec(key, "AES"), spec)
cipher.updateAAD(aad)
return cipher.doFinal(plaintext)
}
fun aesGcmDecryptWithNonce(
key: ByteArray,
nonce: ByteArray,
ciphertext: ByteArray,
aad: ByteArray,
): ByteArray {
require(nonce.size == StreamConstants.STREAM_NONCE_BYTES) {
"AES-GCM nonce must be ${StreamConstants.STREAM_NONCE_BYTES} bytes"
}
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val spec = GCMParameterSpec(128, nonce)
cipher.init(Cipher.DECRYPT_MODE, SecretKeySpec(key, "AES"), spec)
cipher.updateAAD(aad)
return cipher.doFinal(ciphertext)
}

View File

@@ -0,0 +1,50 @@
package no.zyon.shade.streams
import java.nio.ByteBuffer
/**
* Deterministic AEAD nonce + AAD construction for stream chunks.
* Mirror @shade/streams/nonce.ts.
*
* nonce[0..4] = u32_be(laneId)
* nonce[4..12] = u64_be(seq)
*
* aad = streamId(16) || u32_be(laneId) || u64_be(seq) || u8(isLast)
*
* `seq` is unsigned-u64 on the wire. Kotlin's `Long` is signed; we accept it
* for the bit pattern (same as TS `BigInt` would write), so values past
* `Long.MAX_VALUE` arrive here as negative signed longs. `ByteBuffer.putLong`
* writes the raw 8 bytes regardless of sign — that's what we want.
*
* Use `java.lang.Long.parseUnsignedLong("…")` to decode JSON strings
* representing u64 values larger than 2^63 - 1.
*/
fun buildChunkNonce(laneId: Long, seq: Long): ByteArray {
require(laneId in 0L..0xFFFFFFFFL) { "laneId must fit in u32: $laneId" }
val out = ByteArray(StreamConstants.STREAM_NONCE_BYTES)
val buf = ByteBuffer.wrap(out)
buf.putInt(laneId.toInt())
buf.putLong(seq)
return out
}
fun buildChunkAad(
streamId: ByteArray,
laneId: Long,
seq: Long,
isLast: Boolean,
): ByteArray {
require(streamId.size == StreamConstants.STREAM_ID_BYTES) {
"streamId must be ${StreamConstants.STREAM_ID_BYTES} bytes"
}
require(laneId in 0L..0xFFFFFFFFL) { "laneId must fit in u32: $laneId" }
val out = ByteArray(StreamConstants.STREAM_ID_BYTES + 4 + 8 + 1)
streamId.copyInto(out, 0)
val buf = ByteBuffer.wrap(out, StreamConstants.STREAM_ID_BYTES, 4 + 8 + 1)
buf.putInt(laneId.toInt())
buf.putLong(seq)
out[out.size - 1] = if (isLast) 0x01 else 0x00
return out
}

View File

@@ -0,0 +1,140 @@
package no.zyon.shade.types
/**
* Core Shade protocol types. Mirror @shade/core/types.ts.
*
* IMPORTANT: byte-for-byte compatibility with the TypeScript version
* is a hard requirement — the wire format, serialization, and KDF
* inputs must be identical.
*/
/** Long-term identity: Ed25519 for signing + X25519 for DH */
data class IdentityKeyPair(
val signingPublicKey: ByteArray,
val signingPrivateKey: ByteArray,
val dhPublicKey: ByteArray,
val dhPrivateKey: ByteArray,
) {
override fun equals(other: Any?): Boolean {
if (this === other) return true
if (other !is IdentityKeyPair) return false
return signingPublicKey.contentEquals(other.signingPublicKey) &&
signingPrivateKey.contentEquals(other.signingPrivateKey) &&
dhPublicKey.contentEquals(other.dhPublicKey) &&
dhPrivateKey.contentEquals(other.dhPrivateKey)
}
override fun hashCode(): Int {
var result = signingPublicKey.contentHashCode()
result = 31 * result + signingPrivateKey.contentHashCode()
result = 31 * result + dhPublicKey.contentHashCode()
result = 31 * result + dhPrivateKey.contentHashCode()
return result
}
}
/** Generic asymmetric keypair */
data class KeyPair(
val publicKey: ByteArray,
val privateKey: ByteArray,
) {
override fun equals(other: Any?): Boolean {
if (this === other) return true
if (other !is KeyPair) return false
return publicKey.contentEquals(other.publicKey) &&
privateKey.contentEquals(other.privateKey)
}
override fun hashCode(): Int {
var result = publicKey.contentHashCode()
result = 31 * result + privateKey.contentHashCode()
return result
}
}
/** Medium-term signed prekey, rotated periodically */
data class SignedPreKey(
val keyId: Int,
val keyPair: KeyPair,
val signature: ByteArray,
val timestamp: Long,
)
/** Single-use one-time prekey */
data class OneTimePreKey(
val keyId: Int,
val keyPair: KeyPair,
)
/** Prekey bundle fetched from the server to initiate a session */
data class PreKeyBundle(
val registrationId: Int,
val identitySigningKey: ByteArray,
val identityDHKey: ByteArray,
val signedPreKey: BundleSignedPreKey,
val oneTimePreKey: BundleOneTimePreKey? = null,
) {
data class BundleSignedPreKey(
val keyId: Int,
val publicKey: ByteArray,
val signature: ByteArray,
)
data class BundleOneTimePreKey(
val keyId: Int,
val publicKey: ByteArray,
)
}
/** Chain state (root key ratchet or chain key ratchet) */
data class ChainState(
var chainKey: ByteArray,
var counter: Int,
)
/** Full Double Ratchet session state */
data class SessionState(
var remoteIdentityKey: ByteArray,
var rootKey: ByteArray,
var sendChain: ChainState,
var receiveChain: ChainState?,
var dhSend: KeyPair,
var dhReceive: ByteArray?,
var previousSendCounter: Int,
val skippedKeys: MutableMap<String, ByteArray>,
)
/** A ratchet-encrypted message */
data class RatchetMessage(
val dhPublicKey: ByteArray,
val previousCounter: Int,
val counter: Int,
val ciphertext: ByteArray,
val nonce: ByteArray,
)
/** First message to a new peer (embeds X3DH + RatchetMessage) */
data class PreKeyMessage(
val registrationId: Int,
val preKeyId: Int?,
val signedPreKeyId: Int,
val ephemeralKey: ByteArray,
val identityDHKey: ByteArray,
val message: RatchetMessage,
)
/** Envelope wrapping a wire message */
data class ShadeEnvelope(
val type: EnvelopeType,
val content: Any, // PreKeyMessage or RatchetMessage
val timestamp: Long,
val senderAddress: String,
) {
enum class EnvelopeType { PREKEY, RATCHET }
}
/** Max skip constants — must match @shade/core */
object Constants {
const val MAX_SKIP = 1000
const val MAX_CACHED_SKIPPED_KEYS = 2000
}

View File

@@ -0,0 +1,593 @@
package no.zyon.shade
import no.zyon.shade.approval.ApprovalRequestFrame
import no.zyon.shade.approval.ApprovalRequestingDevice
import no.zyon.shade.approval.CanonicalProfileBlob
import no.zyon.shade.approval.DEFAULT_APPROVAL_DOMAIN
import no.zyon.shade.approval.ProfileClientEntry
import no.zyon.shade.approval.ProfileHostEntry
import no.zyon.shade.approval.ProxyApprovalFrame
import no.zyon.shade.approval.VerifyProxyApprovalResult
import no.zyon.shade.approval.buildApprovalRequest
import no.zyon.shade.approval.canonicalApprovalSigningBytes
import no.zyon.shade.approval.emptyCanonicalProfile
import no.zyon.shade.approval.findClientByAddress
import no.zyon.shade.approval.findClientByFingerprint
import no.zyon.shade.approval.isTrustedApprover
import no.zyon.shade.approval.parseCanonicalProfile
import no.zyon.shade.approval.removeClient
import no.zyon.shade.approval.serializeCanonicalProfile
import no.zyon.shade.approval.setTrustedApprover
import no.zyon.shade.approval.signProxyApproval
import no.zyon.shade.approval.upsertClient
import no.zyon.shade.approval.upsertHost
import no.zyon.shade.approval.verifyProxyApproval
import no.zyon.shade.blob.aeadOpen
import no.zyon.shade.blob.aeadSeal
import no.zyon.shade.blob.blobAadForSlot
import no.zyon.shade.blob.deriveBlobKey
import no.zyon.shade.blob.deriveBlobSigningSeed
import no.zyon.shade.blob.deriveBlobSlotId
import no.zyon.shade.blob.ed25519PublicKeyFromSeed
import no.zyon.shade.blob.slotIdToHex
import no.zyon.shade.crypto.TinkProvider
import no.zyon.shade.storage.Argon2idParams
import no.zyon.shade.storage.ScryptParams
import no.zyon.shade.storage.deriveMasterKey
import no.zyon.shade.storage.deriveMasterKeyArgon2id
import org.junit.Assert.assertArrayEquals
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertNotNull
import org.junit.Assert.assertNull
import org.junit.Assert.assertNotEquals
import org.junit.Assert.assertTrue
import org.junit.Test
/**
* Unit tests for the V4.9 blob primitive ports + V4.10 approval
* helpers + scrypt/argon2id wrappers. Cross-platform vector parity
* lives in `CrossPlatformVectorTest`; this file tests Kotlin-side
* round-trip behavior independent of the TS reference.
*/
class BlobAndApprovalTest {
private val crypto = TinkProvider()
private fun hex(bytes: ByteArray): String =
bytes.joinToString("") { "%02x".format(it) }
// ─── V4.9 blob KDF ─────────────────────────────────────────
@Test
fun deriveBlobSlotIdIsDeterministicPerMasterAndApp() {
val km = ByteArray(32) { it.toByte() }
val a1 = deriveBlobSlotId(crypto, km, "foo")
val a2 = deriveBlobSlotId(crypto, km, "foo")
assertArrayEquals(a1, a2)
val b = deriveBlobSlotId(crypto, km, "bar")
assertFalse(a1.contentEquals(b))
val km2 = ByteArray(32) { (it + 1).toByte() }
val c = deriveBlobSlotId(crypto, km2, "foo")
assertFalse(a1.contentEquals(c))
}
@Test
fun blobKdfHelpersAreIndependent() {
val km = ByteArray(32) { it.toByte() }
val slot = deriveBlobSlotId(crypto, km, "x")
val key = deriveBlobKey(crypto, km, "x")
val seed = deriveBlobSigningSeed(crypto, km, "x")
assertFalse(slot.contentEquals(key))
assertFalse(slot.contentEquals(seed))
assertFalse(key.contentEquals(seed))
}
@Test
fun ed25519PublicKeyFromSeedIsDeterministic() {
val seed = ByteArray(32) { it.toByte() }
val pk1 = ed25519PublicKeyFromSeed(seed)
val pk2 = ed25519PublicKeyFromSeed(seed)
assertArrayEquals(pk1, pk2)
assertEquals(32, pk1.size)
}
@Test
fun slotIdToHexProducesLowercase64Chars() {
val s = ByteArray(32) { 0xab.toByte() }
val hex = slotIdToHex(s)
assertEquals(64, hex.length)
assertEquals("a".repeat(0) + "ab".repeat(32), hex)
}
// ─── V4.9 AEAD round-trip ──────────────────────────────────
@Test
fun aeadSealOpenRoundTrip() {
val key = crypto.randomBytes(32)
val nonce = crypto.randomBytes(12)
val pt = "hello".toByteArray()
val aad = blobAadForSlot("00".repeat(32))
val sealed = aeadSeal(key, nonce, pt, aad)
val opened = aeadOpen(key, sealed, aad)
assertArrayEquals(pt, opened)
}
@Test
fun aeadOpenWithWrongAadFails() {
val key = crypto.randomBytes(32)
val nonce = crypto.randomBytes(12)
val pt = "hello".toByteArray()
val aad1 = blobAadForSlot("00".repeat(32))
val aad2 = blobAadForSlot("ff".repeat(32))
val sealed = aeadSeal(key, nonce, pt, aad1)
try {
aeadOpen(key, sealed, aad2)
org.junit.Assert.fail("expected AEAD to reject wrong AAD")
} catch (_: Exception) {
// expected
}
}
@Test
fun aeadOpenWithExpectedNonceMismatchFails() {
val key = crypto.randomBytes(32)
val nonce = crypto.randomBytes(12)
val wrongNonce = crypto.randomBytes(12)
val pt = "hello".toByteArray()
val aad = blobAadForSlot("00".repeat(32))
val sealed = aeadSeal(key, nonce, pt, aad)
try {
aeadOpen(key, sealed, aad, expectedNonce = wrongNonce)
org.junit.Assert.fail("expected expectedNonce check to reject")
} catch (_: IllegalArgumentException) {
// expected
}
}
// ─── Password KDFs ─────────────────────────────────────────
@Test
fun scryptDerivesDeterministically() {
val pw = "correct-horse-battery-staple"
val salt = ByteArray(16) { 0x42.toByte() }
val k1 = deriveMasterKey(pw, salt, ScryptParams(n = 1024))
val k2 = deriveMasterKey(pw, salt, ScryptParams(n = 1024))
assertArrayEquals(k1, k2)
assertEquals(32, k1.size)
}
@Test
fun argon2idDerivesDeterministically() {
val pw = "1234"
val salt = ByteArray(16) { 0x55.toByte() }
val k1 = deriveMasterKeyArgon2id(pw, salt, Argon2idParams(m = 256, t = 2))
val k2 = deriveMasterKeyArgon2id(pw, salt, Argon2idParams(m = 256, t = 2))
assertArrayEquals(k1, k2)
assertEquals(32, k1.size)
}
@Test
fun nfkcNormalizationMakesEquivalentInputsConverge() {
// "café" can be encoded either as 'c','a','f','é' (NFC) or
// 'c','a','f','e','́' (NFD). NFKC normalization on both
// should converge to the same bytes.
val nfc = "café"
val nfd = "café"
assertNotEquals(nfc, nfd)
val salt = ByteArray(16) { 1.toByte() }
val k1 = deriveMasterKey(nfc, salt, ScryptParams(n = 1024))
val k2 = deriveMasterKey(nfd, salt, ScryptParams(n = 1024))
assertArrayEquals(k1, k2)
}
// ─── Canonical profile schema ──────────────────────────────
private fun makeClient(name: String, trusted: Boolean = false): Pair<ProfileClientEntry, ByteArray> {
val seed = crypto.randomBytes(32)
val pubkey = ed25519PublicKeyFromSeed(seed)
val fp = "fp-$name-${hex(pubkey).take(8)}"
return ProfileClientEntry(
address = "device:$name",
identityPublicKey = hex(pubkey),
identityFingerprint = fp,
name = name,
kind = "mobile",
addedAt = 1_700_000_000_000L,
trustedApprover = trusted,
) to seed
}
private fun makeHost(): ProfileHostEntry = ProfileHostEntry(
address = "device:host-server",
name = "Server",
kind = "server",
addedAt = 1_700_000_000_000L,
)
@Test
fun emptyCanonicalProfileRoundTrips() {
val blob = emptyCanonicalProfile(now = 123L)
val bytes = serializeCanonicalProfile(blob)
val parsed = parseCanonicalProfile(bytes)
assertEquals(1, parsed.version)
assertTrue(parsed.hosts.isEmpty())
assertTrue(parsed.clients.isEmpty())
assertTrue(parsed.trustedApproverFingerprints.isEmpty())
assertEquals(123L, parsed.updatedAt)
}
@Test
fun upsertClientDenormalizesTrustedApprovers() {
var blob = emptyCanonicalProfile(0)
val (a, _) = makeClient("phone-a", trusted = true)
val (b, _) = makeClient("phone-b", trusted = false)
blob = upsertClient(blob, a)
blob = upsertClient(blob, b)
assertEquals(2, blob.clients.size)
assertEquals(listOf(a.identityFingerprint), blob.trustedApproverFingerprints)
assertTrue(isTrustedApprover(blob, a.identityFingerprint))
assertFalse(isTrustedApprover(blob, b.identityFingerprint))
}
@Test
fun setTrustedApproverIsIdempotentNoOpReturnsSameInstance() {
var blob = emptyCanonicalProfile(0)
val (c, _) = makeClient("phone", trusted = false)
blob = upsertClient(blob, c)
val before = blob
blob = setTrustedApprover(blob, c.identityFingerprint, false, now = 999L)
assertTrue(blob === before)
}
@Test
fun setTrustedApproverFlipsFlagAndDenormalizedList() {
var blob = emptyCanonicalProfile(0)
val (c, _) = makeClient("phone", trusted = false)
blob = upsertClient(blob, c)
blob = setTrustedApprover(blob, c.identityFingerprint, true, now = 100L)
assertEquals(listOf(c.identityFingerprint), blob.trustedApproverFingerprints)
assertEquals(true, blob.clients[0].trustedApprover)
blob = setTrustedApprover(blob, c.identityFingerprint, false, now = 200L)
assertTrue(blob.trustedApproverFingerprints.isEmpty())
assertEquals(false, blob.clients[0].trustedApprover)
}
@Test
fun removeClientCleansUpDenormalizedList() {
var blob = emptyCanonicalProfile(0)
val (c, _) = makeClient("phone", trusted = true)
blob = upsertClient(blob, c)
blob = removeClient(blob, c.identityFingerprint)
assertTrue(blob.clients.isEmpty())
assertTrue(blob.trustedApproverFingerprints.isEmpty())
}
@Test
fun findClientByFingerprintAndAddress() {
var blob = emptyCanonicalProfile(0)
val (c, _) = makeClient("phone")
blob = upsertClient(blob, c)
assertEquals(c.address, findClientByFingerprint(blob, c.identityFingerprint)?.address)
assertEquals(
c.identityFingerprint,
findClientByAddress(blob, c.address)?.identityFingerprint,
)
assertNull(findClientByFingerprint(blob, "unknown"))
assertNull(findClientByAddress(blob, "unknown"))
}
@Test
fun parseRejectsMalformedProfile() {
try {
parseCanonicalProfile("not json")
org.junit.Assert.fail("expected throw")
} catch (_: IllegalArgumentException) {}
try {
parseCanonicalProfile("""{"version":2}""")
org.junit.Assert.fail("expected throw")
} catch (_: IllegalArgumentException) {}
try {
parseCanonicalProfile(
"""{"version":1,"clients":[{"address":"x","name":"x","kind":"m","addedAt":0}]}""",
)
org.junit.Assert.fail("expected throw — missing identityPublicKey")
} catch (_: IllegalArgumentException) {}
try {
parseCanonicalProfile(
"""{"version":1,"clients":[{"address":"x","identityPublicKey":"NOTHEX","identityFingerprint":"x","name":"x","kind":"m","addedAt":0}]}""",
)
org.junit.Assert.fail("expected throw — bad pubkey hex")
} catch (_: IllegalArgumentException) {}
}
@Test
fun fullProfileSerializeParsePreservesAllFields() {
var blob = emptyCanonicalProfile(1L)
blob = upsertHost(blob, makeHost(), now = 2L)
val (c, _) = makeClient("phone", trusted = true)
blob = upsertClient(blob, c, now = 3L)
blob = blob.copy(signedBy = "aabbccdd")
val bytes = serializeCanonicalProfile(blob)
val parsed = parseCanonicalProfile(bytes)
assertEquals(blob, parsed)
}
// ─── Approval signing payload ──────────────────────────────
@Test
fun canonicalApprovalSigningBytesIsDeterministic() {
val a = canonicalApprovalSigningBytes(
domain = DEFAULT_APPROVAL_DOMAIN,
requestId = "aabbccddeeff00112233445566778899",
hostFingerprint = "11111 22222 33333 44444",
requestingDeviceFingerprint = "55555 66666 77777 88888",
decision = "approve",
)
val b = canonicalApprovalSigningBytes(
domain = DEFAULT_APPROVAL_DOMAIN,
requestId = "aabbccddeeff00112233445566778899",
hostFingerprint = "11111 22222 33333 44444",
requestingDeviceFingerprint = "55555 66666 77777 88888",
decision = "approve",
)
assertArrayEquals(a, b)
}
@Test
fun differentDecisionProducesDifferentSigningBytes() {
val approve = canonicalApprovalSigningBytes(
DEFAULT_APPROVAL_DOMAIN, "r", "h", "d", "approve",
)
val reject = canonicalApprovalSigningBytes(
DEFAULT_APPROVAL_DOMAIN, "r", "h", "d", "reject",
)
assertFalse(approve.contentEquals(reject))
}
@Test
fun differentDomainProducesDifferentSigningBytes() {
val a = canonicalApprovalSigningBytes("shade-link-approve-v1", "r", "h", "d", "approve")
val b = canonicalApprovalSigningBytes("prism-link-approve-v1", "r", "h", "d", "approve")
assertFalse(a.contentEquals(b))
}
// ─── Build / sign / verify ─────────────────────────────────
private data class Scenario(
val phone: ProfileClientEntry,
val phoneSeed: ByteArray,
val profile: CanonicalProfileBlob,
val request: ApprovalRequestFrame,
)
private fun buildScenario(): Scenario {
val (phone, seed) = makeClient("phone", trusted = true)
var profile = emptyCanonicalProfile(0)
profile = upsertHost(profile, makeHost())
profile = upsertClient(profile, phone)
val request = buildApprovalRequest(
crypto = crypto,
hostAddress = "device:host-server",
hostFingerprint = "host-fp-12345",
requestingDeviceFingerprint = "cafe-laptop-fp-67890",
deviceName = "cafe-laptop",
userAgent = "Mozilla/5.0",
ipHint = "203.0.113.7",
)
return Scenario(phone, seed, profile, request)
}
@Test
fun happyPathApproveVerifies() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request,
decision = "approve",
approverFingerprint = s.phone.identityFingerprint,
approverSigningKey = s.phoneSeed,
)
assertEquals("linkApproveByProxy", approval.kind)
assertEquals(s.request.requestId, approval.requestId)
assertEquals(128, approval.signature.length)
val r = verifyProxyApproval(crypto, s.request, approval, s.profile)
assertTrue(r is VerifyProxyApprovalResult.Ok)
assertEquals(s.phone.address, (r as VerifyProxyApprovalResult.Ok).approver.address)
}
@Test
fun happyPathRejectVerifies() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "reject",
s.phone.identityFingerprint, s.phoneSeed,
)
val r = verifyProxyApproval(crypto, s.request, approval, s.profile)
assertTrue(r is VerifyProxyApprovalResult.Ok)
}
@Test
fun replayAgainstDifferentRequestFails() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
val other = s.request.copy(requestId = "f".repeat(32))
val r = verifyProxyApproval(crypto, other, approval, s.profile)
assertEquals(
VerifyProxyApprovalResult.Reason.REQUEST_ID_MISMATCH,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun decisionTamperingFails() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
val tampered = approval.copy(decision = "reject")
val r = verifyProxyApproval(crypto, s.request, tampered, s.profile)
assertEquals(
VerifyProxyApprovalResult.Reason.BAD_SIGNATURE,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun hostFingerprintSwapFails() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
val swapped = s.request.copy(hostFingerprint = "evil-host-fp")
val r = verifyProxyApproval(crypto, swapped, approval, s.profile)
assertEquals(
VerifyProxyApprovalResult.Reason.BAD_SIGNATURE,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun domainMismatchIsRejectedBeforeSignature() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
val r = verifyProxyApproval(
crypto, s.request, approval.copy(domain = "prism-link-approve-v1"), s.profile,
)
assertEquals(
VerifyProxyApprovalResult.Reason.DOMAIN_MISMATCH,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun unknownApproverFails() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
val lying = approval.copy(approverFingerprint = "no-such-fingerprint")
val r = verifyProxyApproval(crypto, s.request, lying, s.profile)
assertEquals(
VerifyProxyApprovalResult.Reason.UNKNOWN_APPROVER,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun revokedApproverFailsWithNotTrusted() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
val revoked = setTrustedApprover(s.profile, s.phone.identityFingerprint, false)
val r = verifyProxyApproval(crypto, s.request, approval, revoked)
assertEquals(
VerifyProxyApprovalResult.Reason.NOT_TRUSTED,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun expiredRequestIsRejected() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
val r = verifyProxyApproval(crypto, s.request, approval, s.profile, now = s.request.expiresAt + 1)
assertEquals(
VerifyProxyApprovalResult.Reason.EXPIRED,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun signatureWithWrongKeyFails() {
val s = buildScenario()
val wrongSeed = crypto.randomBytes(32)
val approval = signProxyApproval(
crypto, s.request, "approve",
approverFingerprint = s.phone.identityFingerprint, // claim phone
approverSigningKey = wrongSeed, // sign with different key
)
val r = verifyProxyApproval(crypto, s.request, approval, s.profile)
assertEquals(
VerifyProxyApprovalResult.Reason.BAD_SIGNATURE,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun customDomainSurvivesRoundTrip() {
val s = buildScenario()
val request = s.request.copy(domain = "prism-link-approve-v1")
val approval = signProxyApproval(
crypto, request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
assertEquals("prism-link-approve-v1", approval.domain)
val r = verifyProxyApproval(crypto, request, approval, s.profile)
assertTrue(r is VerifyProxyApprovalResult.Ok)
}
@Test
fun requestIdIs32LowercaseHexChars() {
val r = buildApprovalRequest(
crypto, "device:h", "h", "r",
)
assertTrue(r.requestId.matches(Regex("^[0-9a-f]{32}$")))
}
@Test
fun consecutiveBuildsProduceDistinctRequestIds() {
val a = buildApprovalRequest(crypto, "device:h", "h", "r")
val b = buildApprovalRequest(crypto, "device:h", "h", "r")
assertNotEquals(a.requestId, b.requestId)
}
// ─── Sanity glue: TS-side reference frame parses on Kotlin ──
@Test
fun tsStyleProxyApprovalFrameParsesAndStructurallyMatches() {
// Construct a frame the way @shade/sdk would emit it via JSON,
// and check our Kotlin types accept the same field names.
val expected = ProxyApprovalFrame(
requestId = "00112233445566778899aabbccddeeff",
decision = "approve",
approverFingerprint = "fp",
signature = "ab".repeat(64),
domain = DEFAULT_APPROVAL_DOMAIN,
)
assertEquals("linkApproveByProxy", expected.kind)
assertEquals("approve", expected.decision)
val req = ApprovalRequestFrame(
requestId = "00112233445566778899aabbccddeeff",
hostAddress = "device:h",
hostFingerprint = "host-fp",
requestingDevice = ApprovalRequestingDevice(
fingerprint = "req-fp",
receivedAt = 1L,
),
expiresAt = 2L,
domain = DEFAULT_APPROVAL_DOMAIN,
)
assertNotNull(req)
}
}

View File

@@ -0,0 +1,594 @@
package no.zyon.shade
import no.zyon.shade.approval.canonicalApprovalSigningBytes
import no.zyon.shade.backup.deriveBackupKey
import no.zyon.shade.blob.aeadOpen
import no.zyon.shade.blob.blobAadForSlot
import no.zyon.shade.blob.deriveBlobKey
import no.zyon.shade.blob.deriveBlobSigningSeed
import no.zyon.shade.blob.deriveBlobSlotId
import no.zyon.shade.blob.ed25519PublicKeyFromSeed
import no.zyon.shade.crypto.TinkProvider
import no.zyon.shade.fingerprint.computeFingerprint
import no.zyon.shade.group.encodeSenderHeader
import no.zyon.shade.group.senderKeyStep
import no.zyon.shade.group.senderSignedBytes
import no.zyon.shade.protocol.deriveInitialRootKey
import no.zyon.shade.protocol.kdfChainKey
import no.zyon.shade.protocol.kdfRootKey
import no.zyon.shade.serialization.StreamChunkWire
import no.zyon.shade.serialization.StreamChunkWireFormat
import no.zyon.shade.serialization.WireFormat
import no.zyon.shade.streams.aesGcmDecryptWithNonce
import no.zyon.shade.streams.aesGcmEncryptWithNonce
import no.zyon.shade.streams.buildChunkAad
import no.zyon.shade.streams.buildChunkNonce
import no.zyon.shade.streams.deriveLaneKey
import no.zyon.shade.streams.deriveStreamKey
import no.zyon.shade.types.PreKeyMessage
import no.zyon.shade.types.RatchetMessage
import no.zyon.shade.types.ShadeEnvelope
import org.json.JSONArray
import org.json.JSONObject
import org.junit.Assert.assertEquals
import org.junit.Assert.assertTrue
import org.junit.Test
import java.io.File
import java.nio.ByteBuffer
import javax.crypto.Cipher
import javax.crypto.spec.GCMParameterSpec
import javax.crypto.spec.SecretKeySpec
/**
* Cross-platform test vectors. MUST match the TypeScript implementation
* byte-for-byte, otherwise cross-platform messaging breaks.
*
* The test-vectors/ directory is at the root of the Shade monorepo.
* Generated by scripts/generate-vectors.ts from the TypeScript implementation.
*/
class CrossPlatformVectorTest {
private val crypto = TinkProvider()
private val vectorsDir = File("../../test-vectors")
private val expectedVersion = 2
private fun fromHex(str: String): ByteArray {
val bytes = ByteArray(str.length / 2)
for (i in bytes.indices) {
bytes[i] = ((Character.digit(str[i * 2], 16) shl 4) +
Character.digit(str[i * 2 + 1], 16)).toByte()
}
return bytes
}
private fun hex(bytes: ByteArray): String {
return bytes.joinToString("") { "%02x".format(it) }
}
private data class VectorFile(val version: Int, val vectors: JSONArray)
private fun loadVectors(name: String): JSONArray {
val file = File(vectorsDir, name)
val content = file.readText()
val obj = JSONObject(content)
val version = obj.getInt("version")
assertEquals("Unexpected vector schema version in $name", expectedVersion, version)
return obj.getJSONArray("vectors")
}
private fun encodeRatchetHeader(
dhPublicKey: ByteArray,
previousCounter: Int,
counter: Int,
): ByteArray {
val buf = ByteBuffer.allocate(40)
buf.put(dhPublicKey)
buf.putInt(previousCounter)
buf.putInt(counter)
return buf.array()
}
private fun aesGcmEncryptDeterministic(
key: ByteArray,
nonce: ByteArray,
plaintext: ByteArray,
aad: ByteArray,
): ByteArray {
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val spec = GCMParameterSpec(128, nonce)
cipher.init(Cipher.ENCRYPT_MODE, SecretKeySpec(key, "AES"), spec)
cipher.updateAAD(aad)
return cipher.doFinal(plaintext)
}
@Test
fun hkdfVectorsMatch() {
val vectors = loadVectors("hkdf.json")
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
val out = crypto.hkdf(
fromHex(v.getString("ikm")),
fromHex(v.getString("salt")),
v.getString("info").toByteArray(Charsets.UTF_8),
v.getInt("length"),
)
assertEquals(v.getString("output"), hex(out))
}
}
@Test
fun kdfChainVectorsMatch() {
val vectors = loadVectors("kdf-chain.json")
val rootVec = vectors.getJSONObject(0)
val rootResult = kdfRootKey(
crypto,
fromHex(rootVec.getString("rootKey")),
fromHex(rootVec.getString("dhOutput")),
)
assertEquals(rootVec.getString("newRootKey"), hex(rootResult.newRootKey))
assertEquals(rootVec.getString("chainKey"), hex(rootResult.chainKey))
val chainVec = vectors.getJSONObject(1)
val chainResult = kdfChainKey(crypto, fromHex(chainVec.getString("chainKey")))
assertEquals(chainVec.getString("newChainKey"), hex(chainResult.newChainKey))
assertEquals(chainVec.getString("messageKey"), hex(chainResult.messageKey))
}
@Test
fun x3dhVectorsMatch() {
val vectors = loadVectors("x3dh.json")
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
val secretsArray = v.getJSONArray("secrets")
val secrets = (0 until secretsArray.length()).map { fromHex(secretsArray.getString(it)) }
val rootKey = deriveInitialRootKey(crypto, secrets)
assertEquals(v.getString("rootKey"), hex(rootKey))
}
}
@Test
fun fingerprintVectorsMatch() {
val vectors = loadVectors("fingerprint.json")
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
val fp = computeFingerprint(
crypto,
fromHex(v.getString("signingKey")),
fromHex(v.getString("dhKey")),
)
assertEquals(v.getString("fingerprint"), fp)
}
}
@Test
fun wireFormatRatchetVectorsMatch() {
val vectors = loadVectors("wire-format.json")
var found = false
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
if (v.optString("kind") != "ratchet") continue
found = true
val m = v.getJSONObject("message")
val msg = RatchetMessage(
dhPublicKey = fromHex(m.getString("dhPublicKey")),
previousCounter = m.getInt("previousCounter"),
counter = m.getInt("counter"),
ciphertext = fromHex(m.getString("ciphertext")),
nonce = fromHex(m.getString("nonce")),
)
val envelope = ShadeEnvelope(
type = ShadeEnvelope.EnvelopeType.RATCHET,
content = msg,
timestamp = 0,
senderAddress = "",
)
val encoded = WireFormat.encodeEnvelope(envelope)
assertEquals(v.getString("encoded"), hex(encoded))
val decoded = WireFormat.decodeEnvelope(encoded)
assertEquals(ShadeEnvelope.EnvelopeType.RATCHET, decoded.type)
val rm = decoded.content as RatchetMessage
assertEquals(msg.counter, rm.counter)
assertEquals(hex(msg.ciphertext), hex(rm.ciphertext))
}
assertTrue("No ratchet wire vectors found", found)
}
@Test
fun wireFormatPreKeyVectorsMatch() {
val vectors = loadVectors("wire-format.json")
var matched = 0
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
if (v.optString("kind") != "prekey") continue
matched++
val m = v.getJSONObject("message")
val inner = m.getJSONObject("inner")
val innerMsg = RatchetMessage(
dhPublicKey = fromHex(inner.getString("dhPublicKey")),
previousCounter = inner.getInt("previousCounter"),
counter = inner.getInt("counter"),
ciphertext = fromHex(inner.getString("ciphertext")),
nonce = fromHex(inner.getString("nonce")),
)
val preKeyId: Int? = if (m.isNull("preKeyId")) null else m.getInt("preKeyId")
val pre = PreKeyMessage(
registrationId = m.getInt("registrationId"),
preKeyId = preKeyId,
signedPreKeyId = m.getInt("signedPreKeyId"),
ephemeralKey = fromHex(m.getString("ephemeralKey")),
identityDHKey = fromHex(m.getString("identityDHKey")),
message = innerMsg,
)
val envelope = ShadeEnvelope(
type = ShadeEnvelope.EnvelopeType.PREKEY,
content = pre,
timestamp = 0,
senderAddress = "",
)
val encoded = WireFormat.encodeEnvelope(envelope)
assertEquals(v.getString("encoded"), hex(encoded))
val decoded = WireFormat.decodeEnvelope(encoded)
assertEquals(ShadeEnvelope.EnvelopeType.PREKEY, decoded.type)
val dm = decoded.content as PreKeyMessage
assertEquals(pre.registrationId, dm.registrationId)
assertEquals(pre.preKeyId, dm.preKeyId)
assertEquals(pre.signedPreKeyId, dm.signedPreKeyId)
assertEquals(hex(pre.ephemeralKey), hex(dm.ephemeralKey))
assertEquals(hex(innerMsg.ciphertext), hex(dm.message.ciphertext))
}
assertTrue("Expected at least 2 prekey vectors", matched >= 2)
}
private fun findVector(arr: JSONArray, prefix: String): JSONObject {
for (i in 0 until arr.length()) {
val o = arr.getJSONObject(i)
if (o.getString("description").startsWith(prefix)) return o
}
throw AssertionError("Vector with description prefix '$prefix' not found")
}
@Test
fun streamsVectorsMatch() {
val vectors = loadVectors("streams.json")
// 1. deriveStreamKey
val sk = findVector(vectors, "deriveStreamKey")
val streamSecret = fromHex(sk.getString("streamSecret"))
val streamId = fromHex(sk.getString("streamId"))
val streamKey = deriveStreamKey(crypto, streamSecret, streamId)
assertEquals(sk.getString("streamKey"), hex(streamKey))
// 2. deriveLaneKey
val lk = findVector(vectors, "deriveLaneKey")
val lkStreamKey = fromHex(lk.getString("streamKey"))
val lkStreamId = fromHex(lk.getString("streamId"))
val lanes = lk.getJSONArray("lanes")
for (i in 0 until lanes.length()) {
val lane = lanes.getJSONObject(i)
val laneId = lane.getLong("laneId")
val k = deriveLaneKey(crypto, lkStreamKey, lkStreamId, laneId)
assertEquals(lane.getString("laneKey"), hex(k))
}
// 3. buildChunkNonce
val nv = findVector(vectors, "buildChunkNonce")
val nonces = nv.getJSONArray("nonces")
for (i in 0 until nonces.length()) {
val n = nonces.getJSONObject(i)
val laneId = n.getLong("laneId")
val seq = java.lang.Long.parseUnsignedLong(n.getString("seq"))
val out = buildChunkNonce(laneId, seq)
assertEquals(n.getString("nonce"), hex(out))
}
// 4. buildChunkAad
val av = findVector(vectors, "buildChunkAad")
val avStreamId = fromHex(av.getString("streamId"))
val cases = av.getJSONArray("cases")
for (i in 0 until cases.length()) {
val c = cases.getJSONObject(i)
val laneId = c.getLong("laneId")
val seq = java.lang.Long.parseUnsignedLong(c.getString("seq"))
val isLast = c.getBoolean("isLast")
val out = buildChunkAad(avStreamId, laneId, seq, isLast)
assertEquals(c.getString("aad"), hex(out))
}
// 5. End-to-end chunk encrypt + decrypt
val ev = findVector(vectors, "End-to-end chunk encrypt")
val laneKey = fromHex(ev.getString("laneKey"))
val nonce = fromHex(ev.getString("nonce"))
val aad = fromHex(ev.getString("aad"))
val plaintext = fromHex(ev.getString("plaintext"))
val ct = aesGcmEncryptWithNonce(laneKey, nonce, plaintext, aad)
assertEquals(ev.getString("ciphertext"), hex(ct))
val pt = aesGcmDecryptWithNonce(laneKey, nonce, fromHex(ev.getString("ciphertext")), aad)
assertEquals(ev.getString("plaintext"), hex(pt))
// 6. Wire 0x11 envelope encode/decode
val wv = findVector(vectors, "Wire 0x11")
val wire = StreamChunkWire(
streamId = fromHex(wv.getString("streamId")),
laneId = wv.getLong("laneId"),
seq = java.lang.Long.parseUnsignedLong(wv.getString("seq")),
isLast = wv.getBoolean("isLast"),
nonce = fromHex(wv.getString("nonce")),
aad = fromHex(wv.getString("extraAad")),
ciphertext = fromHex(wv.getString("ciphertext")),
)
val encoded = StreamChunkWireFormat.encodeStreamChunk(wire)
assertEquals(wv.getString("encoded"), hex(encoded))
val decoded = StreamChunkWireFormat.decodeStreamChunk(encoded)
assertEquals(hex(wire.streamId), hex(decoded.streamId))
assertEquals(wire.laneId, decoded.laneId)
assertEquals(wire.seq, decoded.seq)
assertEquals(wire.isLast, decoded.isLast)
assertEquals(hex(wire.nonce), hex(decoded.nonce))
assertEquals(hex(wire.ciphertext), hex(decoded.ciphertext))
// 7. Envelope-type inspector
assertEquals(
StreamChunkWireFormat.EnvelopeKind.STREAM_CHUNK,
StreamChunkWireFormat.inspectEnvelopeType(encoded),
)
}
@Test
fun backupVectorsMatch() {
val vectors = loadVectors("backup.json")
val kv = findVector(vectors, "Backup v1: HKDF")
val backupKey = deriveBackupKey(crypto, kv.getString("passphrase"), fromHex(kv.getString("salt")))
assertEquals(kv.getString("backupKey"), hex(backupKey))
val ev = findVector(vectors, "Backup v1: AES-256-GCM")
val ct = aesGcmEncryptDeterministic(
fromHex(ev.getString("backupKey")),
fromHex(ev.getString("nonce")),
fromHex(ev.getString("plaintext")),
ByteArray(0),
)
assertEquals(ev.getString("ciphertext"), hex(ct))
val pt = crypto.aesGcmDecrypt(
fromHex(ev.getString("backupKey")),
fromHex(ev.getString("ciphertext")),
fromHex(ev.getString("nonce")),
null,
)
assertEquals(ev.getString("plaintext"), hex(pt))
}
@Test
fun groupSenderKeyVectorsMatch() {
val vectors = loadVectors("group.json")
// 1. Header AAD
val hv = findVector(vectors, "Sender header AAD")
val aad = encodeSenderHeader(
hv.getString("groupId"),
hv.getString("senderAddress"),
hv.getInt("iteration"),
)
assertEquals(hv.getString("aad"), hex(aad))
// 2. Sender-key step
val sv = findVector(vectors, "Sender-key step")
val step = senderKeyStep(
crypto,
fromHex(sv.getString("chainKey")),
sv.getString("groupId"),
sv.getString("senderAddress"),
sv.getInt("iteration"),
)
assertEquals(sv.getString("newChainKey"), hex(step.newChainKey))
assertEquals(sv.getString("messageKey"), hex(step.messageKey))
assertEquals(sv.getString("aad"), hex(step.aad))
val ct = aesGcmEncryptDeterministic(
step.messageKey,
fromHex(sv.getString("nonce")),
fromHex(sv.getString("plaintext")),
step.aad,
)
assertEquals(sv.getString("ciphertext"), hex(ct))
// 3. Ed25519 verify on the recorded signature
val signed = senderSignedBytes(step.aad, ct)
val ok = crypto.verify(
fromHex(sv.getString("signingPublicKey")),
signed,
fromHex(sv.getString("signature")),
)
assertTrue("Sender-key signature verification failed", ok)
// 4. Decrypt roundtrip
val pt = crypto.aesGcmDecrypt(
step.messageKey,
fromHex(sv.getString("ciphertext")),
fromHex(sv.getString("nonce")),
step.aad,
)
assertEquals(sv.getString("plaintext"), hex(pt))
}
@Test
fun storageHkdfVectorsMatch() {
val vectors = loadVectors("storage-hkdf.json")
val sv = findVector(vectors, "Storage HKDF: storageKey")
val storageKey = crypto.hkdf(
fromHex(sv.getString("masterKey")),
ByteArray(0),
"shade-storage-v1".toByteArray(Charsets.UTF_8),
32,
)
assertEquals(sv.getString("storageKey"), hex(storageKey))
val fv = findVector(vectors, "Storage HKDF: fieldKey")
val fStorageKey = fromHex(fv.getString("storageKey"))
val fields = fv.getJSONArray("fields")
for (i in 0 until fields.length()) {
val f = fields.getJSONObject(i)
val info = "shade-field-v1:${f.getString("table")}:${f.getString("column")}"
.toByteArray(Charsets.UTF_8)
val k = crypto.hkdf(fStorageKey, ByteArray(0), info, 32)
assertEquals(f.getString("fieldKey"), hex(k))
}
val nv = findVector(vectors, "Storage HKDF: rowNonce")
val rowKey = fromHex(nv.getString("rowKey"))
val nonces = nv.getJSONArray("nonces")
for (i in 0 until nonces.length()) {
val n = nonces.getJSONObject(i)
val info = "shade-row-nonce-v1:${n.getString("table")}:${n.getString("pk")}"
.toByteArray(Charsets.UTF_8)
val out = crypto.hkdf(rowKey, ByteArray(0), info, 12)
assertEquals(n.getString("nonce"), hex(out))
}
}
@Test
fun blobKdfAndAeadVectorsMatch() {
val vectors = loadVectors("blob.json")
var kdfMatched = 0
var aeadMatched = 0
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
val desc = v.getString("description")
if (desc.startsWith("V4.9 blob KDF")) {
kdfMatched++
val masterKey = fromHex(v.getString("masterKey"))
val app = v.getString("app")
val slotId = deriveBlobSlotId(crypto, masterKey, app)
assertEquals(v.getString("slotId"), hex(slotId))
assertEquals(v.getString("blobKey"), hex(deriveBlobKey(crypto, masterKey, app)))
val seed = deriveBlobSigningSeed(crypto, masterKey, app)
assertEquals(v.getString("signingSeed"), hex(seed))
assertEquals(v.getString("ownerPubkey"), hex(ed25519PublicKeyFromSeed(seed)))
} else if (desc.startsWith("V4.9 blob AEAD")) {
aeadMatched++
val key = fromHex(v.getString("key"))
val slotIdHex = v.getString("slotIdHex")
val expectedPlaintext = fromHex(v.getString("plaintext"))
val wire = fromHex(v.getString("wire"))
val aad = blobAadForSlot(slotIdHex)
val opened = aeadOpen(key, wire, aad)
assertEquals(hex(expectedPlaintext), hex(opened))
}
}
assertTrue("KDF vectors expected", kdfMatched >= 3)
assertTrue("AEAD vectors expected", aeadMatched >= 2)
}
@Test
fun approvalSigningPayloadVectorsMatch() {
val vectors = loadVectors("approval.json")
var payloadMatched = 0
var e2eMatched = 0
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
val desc = v.getString("description")
if (desc.startsWith("V4.10 approval signing payload")) {
payloadMatched++
val out = canonicalApprovalSigningBytes(
domain = v.getString("domain"),
requestId = v.getString("requestId"),
hostFingerprint = v.getString("hostFingerprint"),
requestingDeviceFingerprint = v.getString("requestingDeviceFingerprint"),
decision = v.getString("decision"),
)
assertEquals(v.getString("signingPayload"), hex(out))
} else if (desc.startsWith("V4.10 approval Ed25519 sign/verify")) {
e2eMatched++
val seed = fromHex(v.getString("seed"))
val pubkey = fromHex(v.getString("publicKey"))
assertEquals(hex(pubkey), hex(ed25519PublicKeyFromSeed(seed)))
val req = v.getJSONObject("request")
val payload = canonicalApprovalSigningBytes(
domain = req.getString("domain"),
requestId = req.getString("requestId"),
hostFingerprint = req.getString("hostFingerprint"),
requestingDeviceFingerprint = req.getString("requestingDeviceFingerprint"),
decision = req.getString("decision"),
)
assertEquals(v.getString("signingPayload"), hex(payload))
// Verify the TS-generated signature against our pubkey + payload.
// This is the load-bearing parity check: a Kotlin-implemented
// verifyProxyApproval running against a TS-signed approval
// succeeds.
val sig = fromHex(v.getString("signature"))
val ok = crypto.verify(pubkey, payload, sig)
assertTrue("Ed25519 verify of TS-signed approval failed", ok)
// And: Kotlin signs the same payload with the same seed and
// produces a sig the TS pubkey verifies. Ed25519 is
// deterministic, so the sig bytes also match exactly.
val mySig = crypto.sign(seed, payload)
assertEquals(v.getString("signature"), hex(mySig))
}
}
assertTrue("payload vectors expected", payloadMatched >= 3)
assertTrue("e2e sign/verify vector expected", e2eMatched >= 1)
}
@Test
fun ratchetStepRoundtripMatches() {
val vectors = loadVectors("ratchet-step.json")
assertTrue("ratchet-step vectors expected", vectors.length() > 0)
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
val inputs = v.getJSONObject("inputs")
val derived = v.getJSONObject("derived")
val rootKey = fromHex(inputs.getString("rootKey"))
val dhSendPriv = fromHex(inputs.getString("dhSendPrivateKey"))
val dhSendPub = fromHex(inputs.getString("dhSendPublicKey"))
val dhRemotePub = fromHex(inputs.getString("dhRemotePublicKey"))
val plaintext = fromHex(inputs.getString("plaintext"))
val nonce = fromHex(inputs.getString("nonce"))
val previousCounter = inputs.getInt("previousCounter")
val counter = inputs.getInt("counter")
// 1. DH
val dhOutput = crypto.x25519(dhSendPriv, dhRemotePub)
assertEquals(derived.getString("dhOutput"), hex(dhOutput))
// 2. kdfRootKey
val root = kdfRootKey(crypto, rootKey, dhOutput)
assertEquals(derived.getString("newRootKey"), hex(root.newRootKey))
assertEquals(derived.getString("chainKey"), hex(root.chainKey))
// 3. kdfChainKey
val chain = kdfChainKey(crypto, root.chainKey)
assertEquals(derived.getString("newChainKey"), hex(chain.newChainKey))
assertEquals(derived.getString("messageKey"), hex(chain.messageKey))
// 4. Header AAD
val aad = encodeRatchetHeader(dhSendPub, previousCounter, counter)
assertEquals(derived.getString("aad"), hex(aad))
// 5. AES-GCM encrypt with fixed nonce
val ciphertext = aesGcmEncryptDeterministic(chain.messageKey, nonce, plaintext, aad)
assertEquals(v.getString("ciphertext"), hex(ciphertext))
// 6. Roundtrip decrypt
val recovered = crypto.aesGcmDecrypt(
chain.messageKey,
fromHex(v.getString("ciphertext")),
nonce,
aad,
)
assertEquals(inputs.getString("plaintext"), hex(recovered))
}
}
}

View File

@@ -0,0 +1,123 @@
package no.zyon.shade
import no.zyon.shade.serialization.SessionStateJson
import no.zyon.shade.types.ChainState
import no.zyon.shade.types.IdentityKeyPair
import no.zyon.shade.types.KeyPair
import no.zyon.shade.types.OneTimePreKey
import no.zyon.shade.types.SessionState
import no.zyon.shade.types.SignedPreKey
import org.junit.Assert.assertArrayEquals
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNotNull
import org.junit.Assert.assertNull
import org.junit.Test
/**
* Round-trip tests for the at-rest JSON serialization used by
* `KeystoreStorage`. The format isn't cross-platform (TS uses its
* own shape) — what matters is `serialize → deserialize` preserves
* every byte of every key.
*/
class SessionStateJsonTest {
private fun bytes(n: Int, fill: Byte): ByteArray = ByteArray(n) { fill }
@Test
fun identityKeyPairRoundTrip() {
val k = IdentityKeyPair(
signingPublicKey = bytes(32, 0x11),
signingPrivateKey = bytes(32, 0x22),
dhPublicKey = bytes(32, 0x33),
dhPrivateKey = bytes(32, 0x44),
)
val s = SessionStateJson.serializeIdentityKeyPair(k)
val d = SessionStateJson.deserializeIdentityKeyPair(s)
assertArrayEquals(k.signingPublicKey, d.signingPublicKey)
assertArrayEquals(k.signingPrivateKey, d.signingPrivateKey)
assertArrayEquals(k.dhPublicKey, d.dhPublicKey)
assertArrayEquals(k.dhPrivateKey, d.dhPrivateKey)
}
@Test
fun signedPreKeyRoundTrip() {
val k = SignedPreKey(
keyId = 42,
keyPair = KeyPair(publicKey = bytes(32, 0x55), privateKey = bytes(32, 0x66)),
signature = bytes(64, 0x77),
timestamp = 1_700_000_000_000L,
)
val s = SessionStateJson.serializeSignedPreKey(k)
val d = SessionStateJson.deserializeSignedPreKey(s)
assertEquals(k.keyId, d.keyId)
assertArrayEquals(k.keyPair.publicKey, d.keyPair.publicKey)
assertArrayEquals(k.keyPair.privateKey, d.keyPair.privateKey)
assertArrayEquals(k.signature, d.signature)
assertEquals(k.timestamp, d.timestamp)
}
@Test
fun oneTimePreKeyRoundTrip() {
val k = OneTimePreKey(
keyId = 7,
keyPair = KeyPair(publicKey = bytes(32, 0x88.toByte()), privateKey = bytes(32, 0x99.toByte())),
)
val s = SessionStateJson.serializeOneTimePreKey(k)
val d = SessionStateJson.deserializeOneTimePreKey(s)
assertEquals(k.keyId, d.keyId)
assertArrayEquals(k.keyPair.publicKey, d.keyPair.publicKey)
assertArrayEquals(k.keyPair.privateKey, d.keyPair.privateKey)
}
@Test
fun sessionStateRoundTripFullPopulated() {
val state = SessionState(
remoteIdentityKey = bytes(32, 0x01),
rootKey = bytes(32, 0x02),
sendChain = ChainState(chainKey = bytes(32, 0x03), counter = 5),
receiveChain = ChainState(chainKey = bytes(32, 0x04), counter = 3),
dhSend = KeyPair(publicKey = bytes(32, 0x05), privateKey = bytes(32, 0x06)),
dhReceive = bytes(32, 0x07),
previousSendCounter = 9,
skippedKeys = mutableMapOf(
"remote:1" to bytes(32, 0x0A),
"remote:2" to bytes(32, 0x0B),
),
)
val s = SessionStateJson.serialize(state)
val d = SessionStateJson.deserialize(s)
assertArrayEquals(state.remoteIdentityKey, d.remoteIdentityKey)
assertArrayEquals(state.rootKey, d.rootKey)
assertArrayEquals(state.sendChain.chainKey, d.sendChain.chainKey)
assertEquals(state.sendChain.counter, d.sendChain.counter)
assertNotNull(d.receiveChain)
assertArrayEquals(state.receiveChain!!.chainKey, d.receiveChain!!.chainKey)
assertArrayEquals(state.dhSend.publicKey, d.dhSend.publicKey)
assertArrayEquals(state.dhSend.privateKey, d.dhSend.privateKey)
assertArrayEquals(state.dhReceive, d.dhReceive)
assertEquals(state.previousSendCounter, d.previousSendCounter)
assertEquals(state.skippedKeys.size, d.skippedKeys.size)
for ((k, v) in state.skippedKeys) {
assertArrayEquals(v, d.skippedKeys[k])
}
}
@Test
fun sessionStateRoundTripWithNullableFields() {
val state = SessionState(
remoteIdentityKey = bytes(32, 0x01),
rootKey = bytes(32, 0x02),
sendChain = ChainState(chainKey = bytes(32, 0x03), counter = 0),
receiveChain = null,
dhSend = KeyPair(publicKey = bytes(32, 0x05), privateKey = bytes(32, 0x06)),
dhReceive = null,
previousSendCounter = 0,
skippedKeys = mutableMapOf(),
)
val s = SessionStateJson.serialize(state)
val d = SessionStateJson.deserialize(s)
assertNull(d.receiveChain)
assertNull(d.dhReceive)
assertEquals(0, d.skippedKeys.size)
}
}

349
bun.lock
View File

@@ -8,30 +8,58 @@
"@noble/curves": "^2.0.1", "@noble/curves": "^2.0.1",
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
"hono": "^4.12.12", "hono": "^4.12.12",
"zod": "^3.23.8",
}, },
"devDependencies": { "devDependencies": {
"bun-types": "^1.3.11", "bun-types": "^1.3.11",
"fast-check": "^3.22.0",
},
},
"packages/shade-cli": {
"name": "@shade/cli",
"version": "4.8.5",
"bin": {
"shade": "src/cli.ts",
},
"dependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/keychain": "workspace:*",
"@shade/sdk": "workspace:*",
"@shade/storage-encrypted": "workspace:*",
"@shade/storage-sqlite": "workspace:*",
"@shade/transport": "workspace:*",
},
"devDependencies": {
"@shade/server": "workspace:*",
}, },
}, },
"packages/shade-core": { "packages/shade-core": {
"name": "@shade/core", "name": "@shade/core",
"version": "0.1.0", "version": "4.8.5",
"dependencies": {
"@shade/observability": "workspace:*",
},
"devDependencies": {
"@shade/proto": "workspace:*",
},
"peerDependencies": { "peerDependencies": {
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
}, },
}, },
"packages/shade-crypto-web": { "packages/shade-crypto-web": {
"name": "@shade/crypto-web", "name": "@shade/crypto-web",
"version": "0.1.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@noble/curves": "^2.0.1", "@noble/curves": "^2.0.1",
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/streams": "workspace:*",
}, },
}, },
"packages/shade-dashboard": { "packages/shade-dashboard": {
"name": "@shade/dashboard", "name": "@shade/dashboard",
"version": "0.1.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/widgets": "workspace:*", "@shade/widgets": "workspace:*",
"react": "^19.0.0", "react": "^19.0.0",
@@ -44,9 +72,101 @@
"vite": "^6.0.0", "vite": "^6.0.0",
}, },
}, },
"packages/shade-files": {
"name": "@shade/files",
"version": "4.8.5",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/observability": "workspace:*",
"@shade/proto": "workspace:*",
"@shade/sdk": "workspace:*",
"@shade/streams": "workspace:*",
"@shade/transfer": "workspace:*",
"zod": "^3.23.8",
},
"devDependencies": {
"@shade/server": "workspace:*",
"@types/react": "^19.2.14",
"fast-check": "^3.22.0",
"happy-dom": "^15.11.7",
"react": "^19.2.5",
},
"peerDependencies": {
"react": "^18.0.0 || ^19.0.0",
},
"optionalPeers": [
"react",
],
},
"packages/shade-inbox": {
"name": "@shade/inbox",
"version": "4.8.5",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/proto": "workspace:*",
"@shade/server": "workspace:*",
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
"@shade/inbox-server": "workspace:*",
},
},
"packages/shade-inbox-server": {
"name": "@shade/inbox-server",
"version": "4.8.5",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/observability": "workspace:*",
"@shade/server": "workspace:*",
"hono": "^4.12.12",
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
},
"optionalDependencies": {
"@shade/crypto-web": "workspace:*",
"@shade/storage-postgres": "workspace:*",
"@shade/storage-sqlite": "workspace:*",
},
},
"packages/shade-key-transparency": {
"name": "@shade/key-transparency",
"version": "4.8.5",
"dependencies": {
"@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*",
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
"fast-check": "^3.22.0",
},
},
"packages/shade-keychain": {
"name": "@shade/keychain",
"version": "4.8.5",
},
"packages/shade-observability": {
"name": "@shade/observability",
"version": "4.8.5",
"dependencies": {
"@noble/hashes": "^2.0.1",
},
"devDependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/server": "workspace:*",
},
"peerDependencies": {
"@opentelemetry/api": ">=1.7.0",
},
"optionalPeers": [
"@opentelemetry/api",
],
},
"packages/shade-observer": { "packages/shade-observer": {
"name": "@shade/observer", "name": "@shade/observer",
"version": "0.1.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/server": "workspace:*", "@shade/server": "workspace:*",
@@ -58,27 +178,115 @@
}, },
"packages/shade-proto": { "packages/shade-proto": {
"name": "@shade/proto", "name": "@shade/proto",
"version": "0.1.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
}, },
}, },
"packages/shade-recovery": {
"name": "@shade/recovery",
"version": "4.8.5",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/sdk": "workspace:*",
},
"devDependencies": {
"@shade/server": "workspace:*",
"fast-check": "^3.22.0",
},
},
"packages/shade-sdk": {
"name": "@shade/sdk",
"version": "4.8.5",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/files": "workspace:*",
"@shade/inbox": "workspace:*",
"@shade/key-transparency": "workspace:*",
"@shade/observability": "workspace:*",
"@shade/observer": "workspace:*",
"@shade/proto": "workspace:*",
"@shade/server": "workspace:*",
"@shade/storage-encrypted": "workspace:*",
"@shade/storage-sqlite": "workspace:*",
"@shade/streams": "workspace:*",
"@shade/transfer": "workspace:*",
"@shade/transport": "workspace:*",
},
"devDependencies": {
"@shade/inbox-server": "workspace:*",
"@shade/transport-webrtc": "workspace:*",
},
"peerDependencies": {
"@shade/transport-webrtc": "workspace:*",
},
"optionalPeers": [
"@shade/transport-webrtc",
],
},
"packages/shade-server": { "packages/shade-server": {
"name": "@shade/server", "name": "@shade/server",
"version": "0.1.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/inbox-server": "workspace:*",
"@shade/key-transparency": "workspace:*",
"@shade/observability": "workspace:*",
"hono": "^4.12.12", "hono": "^4.12.12",
}, },
"devDependencies": { "devDependencies": {
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
}, },
"optionalDependencies": {
"@shade/crypto-web": "workspace:*",
"@shade/observer": "workspace:*",
"@shade/storage-postgres": "workspace:*",
"@shade/storage-sqlite": "workspace:*",
},
},
"packages/shade-storage-encrypted": {
"name": "@shade/storage-encrypted",
"version": "4.8.5",
"dependencies": {
"@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/storage-postgres": "workspace:*",
"@shade/storage-sqlite": "workspace:*",
"idb": "^8.0.3",
"postgres": "^3.4.9",
},
"devDependencies": {
"fake-indexeddb": "^6.0.0",
},
"peerDependencies": {
"@shade/keychain": "workspace:*",
},
"optionalPeers": [
"@shade/keychain",
],
},
"packages/shade-storage-indexeddb": {
"name": "@shade/storage-indexeddb",
"version": "4.8.5",
"dependencies": {
"@shade/core": "workspace:*",
"idb": "^8.0.3",
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
"fake-indexeddb": "^6.0.0",
},
}, },
"packages/shade-storage-postgres": { "packages/shade-storage-postgres": {
"name": "@shade/storage-postgres", "name": "@shade/storage-postgres",
"version": "0.1.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/inbox-server": "workspace:*",
"@shade/key-transparency": "workspace:*",
"@shade/server": "workspace:*", "@shade/server": "workspace:*",
"drizzle-orm": "^0.45.2", "drizzle-orm": "^0.45.2",
"postgres": "^3.4.9", "postgres": "^3.4.9",
@@ -89,29 +297,98 @@
}, },
"packages/shade-storage-sqlite": { "packages/shade-storage-sqlite": {
"name": "@shade/storage-sqlite", "name": "@shade/storage-sqlite",
"version": "0.1.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
"@shade/inbox-server": "workspace:*",
"@shade/server": "workspace:*", "@shade/server": "workspace:*",
}, },
}, },
"packages/shade-transport": { "packages/shade-streams": {
"name": "@shade/transport", "name": "@shade/streams",
"version": "0.1.0", "version": "4.8.5",
"dependencies": {
"@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*",
"@shade/proto": "workspace:*",
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
},
},
"packages/shade-transfer": {
"name": "@shade/transfer",
"version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
"@shade/observability": "workspace:*",
"@shade/proto": "workspace:*",
"@shade/streams": "workspace:*",
},
"peerDependencies": {
"hono": "^4",
},
"optionalPeers": [
"hono",
],
},
"packages/shade-transport": {
"name": "@shade/transport",
"version": "4.8.5",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*",
"@shade/key-transparency": "workspace:*",
"@shade/proto": "workspace:*", "@shade/proto": "workspace:*",
"@shade/server": "workspace:*", "@shade/server": "workspace:*",
}, },
}, },
"packages/shade-transport-bridge": {
"name": "@shade/transport-bridge",
"version": "4.8.5",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/server": "workspace:*",
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
"@shade/inbox-server": "workspace:*",
"hono": "^4.12.12",
},
"optionalDependencies": {
"@shade/inbox-server": "workspace:*",
},
"peerDependencies": {
"hono": "^4",
},
"optionalPeers": [
"hono",
],
},
"packages/shade-transport-webrtc": {
"name": "@shade/transport-webrtc",
"version": "4.8.5",
"dependencies": {
"@shade/core": "workspace:*",
"@shade/streams": "workspace:*",
"@shade/transfer": "workspace:*",
},
},
"packages/shade-widgets": { "packages/shade-widgets": {
"name": "@shade/widgets", "name": "@shade/widgets",
"version": "0.1.0", "version": "4.8.5",
"dependencies": {
"@shade/recovery": "workspace:*",
"@shade/sdk": "workspace:*",
"@shade/streams": "workspace:*",
"@shade/transfer": "workspace:*",
},
"devDependencies": { "devDependencies": {
"@types/react": "^19.2.14", "@types/react": "^19.2.14",
"@types/react-dom": "^19.2.3", "@types/react-dom": "^19.2.3",
"happy-dom": "^15.11.7",
"react": "^19.2.5", "react": "^19.2.5",
"react-dom": "^19.2.5", "react-dom": "^19.2.5",
}, },
@@ -278,24 +555,54 @@
"@rollup/rollup-win32-x64-msvc": ["@rollup/rollup-win32-x64-msvc@4.60.1", "", { "os": "win32", "cpu": "x64" }, "sha512-lWMnixq/QzxyhTV6NjQJ4SFo1J6PvOX8vUx5Wb4bBPsEb+8xZ89Bz6kOXpfXj9ak9AHTQVQzlgzBEc1SyM27xQ=="], "@rollup/rollup-win32-x64-msvc": ["@rollup/rollup-win32-x64-msvc@4.60.1", "", { "os": "win32", "cpu": "x64" }, "sha512-lWMnixq/QzxyhTV6NjQJ4SFo1J6PvOX8vUx5Wb4bBPsEb+8xZ89Bz6kOXpfXj9ak9AHTQVQzlgzBEc1SyM27xQ=="],
"@shade/cli": ["@shade/cli@workspace:packages/shade-cli"],
"@shade/core": ["@shade/core@workspace:packages/shade-core"], "@shade/core": ["@shade/core@workspace:packages/shade-core"],
"@shade/crypto-web": ["@shade/crypto-web@workspace:packages/shade-crypto-web"], "@shade/crypto-web": ["@shade/crypto-web@workspace:packages/shade-crypto-web"],
"@shade/dashboard": ["@shade/dashboard@workspace:packages/shade-dashboard"], "@shade/dashboard": ["@shade/dashboard@workspace:packages/shade-dashboard"],
"@shade/files": ["@shade/files@workspace:packages/shade-files"],
"@shade/inbox": ["@shade/inbox@workspace:packages/shade-inbox"],
"@shade/inbox-server": ["@shade/inbox-server@workspace:packages/shade-inbox-server"],
"@shade/key-transparency": ["@shade/key-transparency@workspace:packages/shade-key-transparency"],
"@shade/keychain": ["@shade/keychain@workspace:packages/shade-keychain"],
"@shade/observability": ["@shade/observability@workspace:packages/shade-observability"],
"@shade/observer": ["@shade/observer@workspace:packages/shade-observer"], "@shade/observer": ["@shade/observer@workspace:packages/shade-observer"],
"@shade/proto": ["@shade/proto@workspace:packages/shade-proto"], "@shade/proto": ["@shade/proto@workspace:packages/shade-proto"],
"@shade/recovery": ["@shade/recovery@workspace:packages/shade-recovery"],
"@shade/sdk": ["@shade/sdk@workspace:packages/shade-sdk"],
"@shade/server": ["@shade/server@workspace:packages/shade-server"], "@shade/server": ["@shade/server@workspace:packages/shade-server"],
"@shade/storage-encrypted": ["@shade/storage-encrypted@workspace:packages/shade-storage-encrypted"],
"@shade/storage-indexeddb": ["@shade/storage-indexeddb@workspace:packages/shade-storage-indexeddb"],
"@shade/storage-postgres": ["@shade/storage-postgres@workspace:packages/shade-storage-postgres"], "@shade/storage-postgres": ["@shade/storage-postgres@workspace:packages/shade-storage-postgres"],
"@shade/storage-sqlite": ["@shade/storage-sqlite@workspace:packages/shade-storage-sqlite"], "@shade/storage-sqlite": ["@shade/storage-sqlite@workspace:packages/shade-storage-sqlite"],
"@shade/streams": ["@shade/streams@workspace:packages/shade-streams"],
"@shade/transfer": ["@shade/transfer@workspace:packages/shade-transfer"],
"@shade/transport": ["@shade/transport@workspace:packages/shade-transport"], "@shade/transport": ["@shade/transport@workspace:packages/shade-transport"],
"@shade/transport-bridge": ["@shade/transport-bridge@workspace:packages/shade-transport-bridge"],
"@shade/transport-webrtc": ["@shade/transport-webrtc@workspace:packages/shade-transport-webrtc"],
"@shade/widgets": ["@shade/widgets@workspace:packages/shade-widgets"], "@shade/widgets": ["@shade/widgets@workspace:packages/shade-widgets"],
"@types/babel__core": ["@types/babel__core@7.20.5", "", { "dependencies": { "@babel/parser": "^7.20.7", "@babel/types": "^7.20.7", "@types/babel__generator": "*", "@types/babel__template": "*", "@types/babel__traverse": "*" } }, "sha512-qoQprZvz5wQFJwMDqeseRXWv3rqMvhgpbXFfVyWhbx9X47POIA6i/+dXefEmZKoAgOaTdaIgNSMqMIU61yRyzA=="], "@types/babel__core": ["@types/babel__core@7.20.5", "", { "dependencies": { "@babel/parser": "^7.20.7", "@babel/types": "^7.20.7", "@types/babel__generator": "*", "@types/babel__template": "*", "@types/babel__traverse": "*" } }, "sha512-qoQprZvz5wQFJwMDqeseRXWv3rqMvhgpbXFfVyWhbx9X47POIA6i/+dXefEmZKoAgOaTdaIgNSMqMIU61yRyzA=="],
@@ -334,18 +641,28 @@
"electron-to-chromium": ["electron-to-chromium@1.5.334", "", {}, "sha512-mgjZAz7Jyx1SRCwEpy9wefDS7GvNPazLthHg8eQMJ76wBdGQQDW33TCrUTvQ4wzpmOrv2zrFoD3oNufMdyMpog=="], "electron-to-chromium": ["electron-to-chromium@1.5.334", "", {}, "sha512-mgjZAz7Jyx1SRCwEpy9wefDS7GvNPazLthHg8eQMJ76wBdGQQDW33TCrUTvQ4wzpmOrv2zrFoD3oNufMdyMpog=="],
"entities": ["entities@4.5.0", "", {}, "sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw=="],
"esbuild": ["esbuild@0.25.12", "", { "optionalDependencies": { "@esbuild/aix-ppc64": "0.25.12", "@esbuild/android-arm": "0.25.12", "@esbuild/android-arm64": "0.25.12", "@esbuild/android-x64": "0.25.12", "@esbuild/darwin-arm64": "0.25.12", "@esbuild/darwin-x64": "0.25.12", "@esbuild/freebsd-arm64": "0.25.12", "@esbuild/freebsd-x64": "0.25.12", "@esbuild/linux-arm": "0.25.12", "@esbuild/linux-arm64": "0.25.12", "@esbuild/linux-ia32": "0.25.12", "@esbuild/linux-loong64": "0.25.12", "@esbuild/linux-mips64el": "0.25.12", "@esbuild/linux-ppc64": "0.25.12", "@esbuild/linux-riscv64": "0.25.12", "@esbuild/linux-s390x": "0.25.12", "@esbuild/linux-x64": "0.25.12", "@esbuild/netbsd-arm64": "0.25.12", "@esbuild/netbsd-x64": "0.25.12", "@esbuild/openbsd-arm64": "0.25.12", "@esbuild/openbsd-x64": "0.25.12", "@esbuild/openharmony-arm64": "0.25.12", "@esbuild/sunos-x64": "0.25.12", "@esbuild/win32-arm64": "0.25.12", "@esbuild/win32-ia32": "0.25.12", "@esbuild/win32-x64": "0.25.12" }, "bin": { "esbuild": "bin/esbuild" } }, "sha512-bbPBYYrtZbkt6Os6FiTLCTFxvq4tt3JKall1vRwshA3fdVztsLAatFaZobhkBC8/BrPetoa0oksYoKXoG4ryJg=="], "esbuild": ["esbuild@0.25.12", "", { "optionalDependencies": { "@esbuild/aix-ppc64": "0.25.12", "@esbuild/android-arm": "0.25.12", "@esbuild/android-arm64": "0.25.12", "@esbuild/android-x64": "0.25.12", "@esbuild/darwin-arm64": "0.25.12", "@esbuild/darwin-x64": "0.25.12", "@esbuild/freebsd-arm64": "0.25.12", "@esbuild/freebsd-x64": "0.25.12", "@esbuild/linux-arm": "0.25.12", "@esbuild/linux-arm64": "0.25.12", "@esbuild/linux-ia32": "0.25.12", "@esbuild/linux-loong64": "0.25.12", "@esbuild/linux-mips64el": "0.25.12", "@esbuild/linux-ppc64": "0.25.12", "@esbuild/linux-riscv64": "0.25.12", "@esbuild/linux-s390x": "0.25.12", "@esbuild/linux-x64": "0.25.12", "@esbuild/netbsd-arm64": "0.25.12", "@esbuild/netbsd-x64": "0.25.12", "@esbuild/openbsd-arm64": "0.25.12", "@esbuild/openbsd-x64": "0.25.12", "@esbuild/openharmony-arm64": "0.25.12", "@esbuild/sunos-x64": "0.25.12", "@esbuild/win32-arm64": "0.25.12", "@esbuild/win32-ia32": "0.25.12", "@esbuild/win32-x64": "0.25.12" }, "bin": { "esbuild": "bin/esbuild" } }, "sha512-bbPBYYrtZbkt6Os6FiTLCTFxvq4tt3JKall1vRwshA3fdVztsLAatFaZobhkBC8/BrPetoa0oksYoKXoG4ryJg=="],
"escalade": ["escalade@3.2.0", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="], "escalade": ["escalade@3.2.0", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="],
"fake-indexeddb": ["fake-indexeddb@6.2.5", "", {}, "sha512-CGnyrvbhPlWYMngksqrSSUT1BAVP49dZocrHuK0SvtR0D5TMs5wP0o3j7jexDJW01KSadjBp1M/71o/KR3nD1w=="],
"fast-check": ["fast-check@3.23.2", "", { "dependencies": { "pure-rand": "^6.1.0" } }, "sha512-h5+1OzzfCC3Ef7VbtKdcv7zsstUQwUDlYpUTvjeUsJAssPgLn7QzbboPtL5ro04Mq0rPOsMzl7q5hIbRs2wD1A=="],
"fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="], "fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="],
"fsevents": ["fsevents@2.3.3", "", { "os": "darwin" }, "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw=="], "fsevents": ["fsevents@2.3.3", "", { "os": "darwin" }, "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw=="],
"gensync": ["gensync@1.0.0-beta.2", "", {}, "sha512-3hN7NaskYvMDLQY55gnW3NQ+mesEAepTqlg+VEbj7zzqEMBVNhzcGYYeqFo/TlYz6eQiFcp1HcsCZO+nGgS8zg=="], "gensync": ["gensync@1.0.0-beta.2", "", {}, "sha512-3hN7NaskYvMDLQY55gnW3NQ+mesEAepTqlg+VEbj7zzqEMBVNhzcGYYeqFo/TlYz6eQiFcp1HcsCZO+nGgS8zg=="],
"happy-dom": ["happy-dom@15.11.7", "", { "dependencies": { "entities": "^4.5.0", "webidl-conversions": "^7.0.0", "whatwg-mimetype": "^3.0.0" } }, "sha512-KyrFvnl+J9US63TEzwoiJOQzZBJY7KgBushJA8X61DMbNsH+2ONkDuLDnCnwUiPTF42tLoEmrPyoqbenVA5zrg=="],
"hono": ["hono@4.12.12", "", {}, "sha512-p1JfQMKaceuCbpJKAPKVqyqviZdS0eUxH9v82oWo1kb9xjQ5wA6iP3FNVAPDFlz5/p7d45lO+BpSk1tuSZMF4Q=="], "hono": ["hono@4.12.12", "", {}, "sha512-p1JfQMKaceuCbpJKAPKVqyqviZdS0eUxH9v82oWo1kb9xjQ5wA6iP3FNVAPDFlz5/p7d45lO+BpSk1tuSZMF4Q=="],
"idb": ["idb@8.0.3", "", {}, "sha512-LtwtVyVYO5BqRvcsKuB2iUMnHwPVByPCXFXOpuU96IZPPoPN6xjOGxZQ74pgSVVLQWtUOYgyeL4GE98BY5D3wg=="],
"js-tokens": ["js-tokens@4.0.0", "", {}, "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ=="], "js-tokens": ["js-tokens@4.0.0", "", {}, "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ=="],
"jsesc": ["jsesc@3.1.0", "", { "bin": { "jsesc": "bin/jsesc" } }, "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA=="], "jsesc": ["jsesc@3.1.0", "", { "bin": { "jsesc": "bin/jsesc" } }, "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA=="],
@@ -368,6 +685,8 @@
"postgres": ["postgres@3.4.9", "", {}, "sha512-GD3qdB0x1z9xgFI6cdRD6xu2Sp2WCOEoe3mtnyB5Ee0XrrL5Pe+e4CCnJrRMnL1zYtRDZmQQVbvOttLnKDLnaw=="], "postgres": ["postgres@3.4.9", "", {}, "sha512-GD3qdB0x1z9xgFI6cdRD6xu2Sp2WCOEoe3mtnyB5Ee0XrrL5Pe+e4CCnJrRMnL1zYtRDZmQQVbvOttLnKDLnaw=="],
"pure-rand": ["pure-rand@6.1.0", "", {}, "sha512-bVWawvoZoBYpp6yIoQtQXHZjmz35RSVHnUOTefl8Vcjr8snTPY1wnpSPMWekcFwbxI6gtmT7rSYPFvz71ldiOA=="],
"react": ["react@19.2.5", "", {}, "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA=="], "react": ["react@19.2.5", "", {}, "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA=="],
"react-dom": ["react-dom@19.2.5", "", { "dependencies": { "scheduler": "^0.27.0" }, "peerDependencies": { "react": "^19.2.5" } }, "sha512-J5bAZz+DXMMwW/wV3xzKke59Af6CHY7G4uYLN1OvBcKEsWOs4pQExj86BBKamxl/Ik5bx9whOrvBlSDfWzgSag=="], "react-dom": ["react-dom@19.2.5", "", { "dependencies": { "scheduler": "^0.27.0" }, "peerDependencies": { "react": "^19.2.5" } }, "sha512-J5bAZz+DXMMwW/wV3xzKke59Af6CHY7G4uYLN1OvBcKEsWOs4pQExj86BBKamxl/Ik5bx9whOrvBlSDfWzgSag=="],
@@ -390,6 +709,12 @@
"vite": ["vite@6.4.2", "", { "dependencies": { "esbuild": "^0.25.0", "fdir": "^6.4.4", "picomatch": "^4.0.2", "postcss": "^8.5.3", "rollup": "^4.34.9", "tinyglobby": "^0.2.13" }, "optionalDependencies": { "fsevents": "~2.3.3" }, "peerDependencies": { "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", "jiti": ">=1.21.0", "less": "*", "lightningcss": "^1.21.0", "sass": "*", "sass-embedded": "*", "stylus": "*", "sugarss": "*", "terser": "^5.16.0", "tsx": "^4.8.1", "yaml": "^2.4.2" }, "optionalPeers": ["@types/node", "jiti", "less", "lightningcss", "sass", "sass-embedded", "stylus", "sugarss", "terser", "tsx", "yaml"], "bin": { "vite": "bin/vite.js" } }, "sha512-2N/55r4JDJ4gdrCvGgINMy+HH3iRpNIz8K6SFwVsA+JbQScLiC+clmAxBgwiSPgcG9U15QmvqCGWzMbqda5zGQ=="], "vite": ["vite@6.4.2", "", { "dependencies": { "esbuild": "^0.25.0", "fdir": "^6.4.4", "picomatch": "^4.0.2", "postcss": "^8.5.3", "rollup": "^4.34.9", "tinyglobby": "^0.2.13" }, "optionalDependencies": { "fsevents": "~2.3.3" }, "peerDependencies": { "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", "jiti": ">=1.21.0", "less": "*", "lightningcss": "^1.21.0", "sass": "*", "sass-embedded": "*", "stylus": "*", "sugarss": "*", "terser": "^5.16.0", "tsx": "^4.8.1", "yaml": "^2.4.2" }, "optionalPeers": ["@types/node", "jiti", "less", "lightningcss", "sass", "sass-embedded", "stylus", "sugarss", "terser", "tsx", "yaml"], "bin": { "vite": "bin/vite.js" } }, "sha512-2N/55r4JDJ4gdrCvGgINMy+HH3iRpNIz8K6SFwVsA+JbQScLiC+clmAxBgwiSPgcG9U15QmvqCGWzMbqda5zGQ=="],
"webidl-conversions": ["webidl-conversions@7.0.0", "", {}, "sha512-VwddBukDzu71offAQR975unBIGqfKZpM+8ZX6ySk8nYhVoo5CYaZyzt3YBvYtRtO+aoGlqxPg/B87NGVZ/fu6g=="],
"whatwg-mimetype": ["whatwg-mimetype@3.0.0", "", {}, "sha512-nt+N2dzIutVRxARx1nghPKGv1xHikU7HKdfafKkLNLindmPU/ch3U31NOCGGA/dmPcmb1VlofO0vnKAcsm0o/Q=="],
"yallist": ["yallist@3.1.1", "", {}, "sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g=="], "yallist": ["yallist@3.1.1", "", {}, "sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g=="],
"zod": ["zod@3.25.76", "", {}, "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ=="],
} }
} }

142
docs/DEPLOYMENT.md Normal file
View File

@@ -0,0 +1,142 @@
# Deploying Shade
Shade ships as a single Docker image that contains the prekey server, observer dashboard, OpenAPI contract, and stale cleanup. You deploy one container per project.
## Quick start
```bash
docker run -d \
--name my-project-shade \
-v my-project-shade:/data \
-p 3900:3900 \
-e SHADE_OBSERVER_TOKEN=change-me-to-at-least-16-chars \
gt.zyon.no/stian/shade-prekey:latest
```
That's it. Your projects can now register identities and exchange prekey bundles via `http://localhost:3900`.
## Why one container per project
Each project is self-contained. Nova doesn't depend on Orchestrator being up. Future projects can be added without touching existing ones. The container is tiny (~260 MB), idle resource usage is near zero, and each container owns its own SQLite volume.
```
Project A Project B Future projects
───────── ───────── ────────────────
app + frontend app + frontend app + frontend
│ │ │
↓ ↓ ↓
shade-a container shade-b container shade-n container
(port 3900) (port 3901) (port 390n)
sqlite volume sqlite volume sqlite volume
```
## Dokploy deployment
1. Go to Dokploy → Projects → New Project → Docker Compose
2. Paste the `docker-compose.yml` from [`examples/05-dokploy-deployment`](../examples/05-dokploy-deployment/docker-compose.yml)
3. Set env vars in the Dokploy UI:
- `SHADE_OBSERVER_TOKEN` (generate a random 32+ char string)
4. Set the container name unique per project (e.g., `nova-shade`, `orchestrator-shade`)
5. Deploy
Dokploy will pull the image, create the volume, and health check the container automatically.
## Volumes and backup
The `/data` volume holds:
- `shade-prekeys.db` — the SQLite database with all identities, prekeys, and activity timestamps
- WAL journal files for crash safety
**Backup:** Copy the `.db` file while the container is stopped, or use SQLite's online backup API:
```bash
docker exec my-project-shade sqlite3 /data/shade-prekeys.db ".backup /data/backup.db"
docker cp my-project-shade:/data/backup.db ./local-backup.db
```
**Restore:** Stop the container, copy the `.db` file into the volume, restart.
## PostgreSQL instead of SQLite
If you want to share a Postgres instance (or need HA), set `SHADE_PREKEY_PG_URL`:
```yaml
environment:
- SHADE_PREKEY_PG_URL=postgres://shade:shade@postgres:5432/shade
```
Tables will be created automatically with the `shade_server_*` prefix, so they coexist cleanly with any other tables in the same database.
## Environment variable reference
| Var | Default | Description |
|-----|---------|-------------|
| `PORT` | `3900` | HTTP port |
| `SHADE_PREKEY_DB_PATH` | `/data/shade-prekeys.db` | SQLite file location |
| `SHADE_PREKEY_PG_URL` | unset | Postgres URL (overrides SQLite) |
| `SHADE_INBOX_DB_PATH` | unset (memory) | SQLite file for the V3.6 inbox relay |
| `SHADE_INBOX_PG_URL` | falls back to `SHADE_PREKEY_PG_URL` | Postgres URL for the inbox relay |
| `SHADE_INBOX_PRUNE_INTERVAL_MINUTES` | `5` | How often expired inbox blobs are dropped |
| `SHADE_OBSERVER_TOKEN` | unset | Enables dashboard at `/shade-observer/dashboard/`. Min 16 chars. |
| `SHADE_STALE_DAYS` | `30` | Purge identities with no activity in N days |
| `SHADE_CLEANUP_INTERVAL_HOURS` | `24` | Cleanup cycle interval |
| `SHADE_LOG_LEVEL` | `info` | `debug` / `info` / `warn` / `error` |
| `SHADE_OTEL_ENABLED` | unset | Set to `1`/`true` to enable OpenTelemetry tracing on `withTracer()`-configured deployments. See [`observability.md`](./observability.md). |
| `SHADE_DISABLE_RATE_LIMIT` | unset | Set to `1` to disable IP rate-limits on every prekey + inbox route. **Single-tenant deployments only** — multi-tenant relays must leave this unset to keep the abuse defenses on. |
## Health and observability
- **Health:** `GET /health` — returns `{"status":"ok"}` when the storage backend is reachable. Docker's HEALTHCHECK uses this.
- **Metrics:** `GET /metrics` — Prometheus format with counters, histograms, and gauges for all routes.
- **Tracing:** Optional OpenTelemetry spans via `@shade/observability`. Off by default; flip `SHADE_OTEL_ENABLED=1` to activate. PII-safe span attributes are documented in [`observability.md`](./observability.md).
- **OpenAPI:** `GET /openapi.yaml` — machine-readable API contract for any language.
- **Redoc viewer:** `GET /docs` — human-readable API reference.
- **Dashboard:** `GET /shade-observer/dashboard/` — live activity viewer (requires token).
## Stale cleanup
Identities with no activity (no bundle fetches, no replenishments, no registration refreshes) for more than `SHADE_STALE_DAYS` days are automatically purged from the database. This keeps the database bounded without manual housekeeping.
The cleanup task runs once at startup and then every `SHADE_CLEANUP_INTERVAL_HOURS` hours. Each cycle logs the number of purged identities.
## Multiple Shade instances on the same host
Run multiple projects side-by-side with different container names and ports:
```yaml
services:
nova-shade:
container_name: nova-shade
image: gt.zyon.no/stian/shade-prekey:latest
ports: ["3900:3900"]
volumes: [nova-shade-data:/data]
environment:
- SHADE_OBSERVER_TOKEN=nova-token-32-chars-minimum-xxx
orch-shade:
container_name: orch-shade
image: gt.zyon.no/stian/shade-prekey:latest
ports: ["3901:3900"]
volumes: [orch-shade-data:/data]
environment:
- SHADE_OBSERVER_TOKEN=orch-token-32-chars-minimum-xxx
volumes:
nova-shade-data:
orch-shade-data:
```
## CI publishing
Tagged releases auto-publish to the Gitea container registry via `.gitea/workflows/docker.yml`. To cut a release:
```bash
bun run version 1.0.1
git push --tags
```
## Security notes
- **Never commit `SHADE_OBSERVER_TOKEN`.** Use Dokploy secrets or environment-specific `.env` files.
- The prekey server stores **public keys only**. No private keys ever touch it.
- Rate limiting is on by default (5 registrations per hour per IP, etc.). Tune via `createPrekeyRoutes` options if embedding, or configure at reverse-proxy level for the container.
- Put the container behind a reverse proxy (Traefik, Caddy) for TLS termination.

View File

@@ -0,0 +1,179 @@
# Shade Production Checklist
A flat punch-list for taking a Shade prekey server from "it boots" to
"production-ready". Every item below is a hard gate — if you can't tick it,
don't ship.
The deeper "why" behind each item lives in `THREAT-MODEL.md`,
`SECURITY.md`, and `docs/DEPLOYMENT.md`. This file is the operator's
checklist.
> Scope: a single Shade prekey container (`@shade/server`) plus any
> consumer apps that talk to it. For E2EE file transfer hardening
> (max-size, retention, quotas), see the **Hardening** and **Retention**
> sections of `docs/streams.md`.
---
## 1. TLS termination
- [ ] Public traffic is **TLS 1.2+ only** — Shade itself speaks plain HTTP
and assumes a reverse proxy (Caddy, Traefik, nginx, Dokploy's
built-in proxy) terminates TLS in front of it.
- [ ] HSTS is on (`Strict-Transport-Security: max-age=15552000`).
- [ ] The proxy is configured to pass the original `Host` header through
so signed payloads bound to the canonical address don't trip the
replay-window check on a mismatch.
- [ ] Internal traffic between consumer apps and the prekey container
runs on a private network (Docker bridge / VPC); the prekey port
is **not** exposed to the public internet without TLS in front.
> **Why:** identity signatures and observer bearer tokens travel in
> request bodies / headers. Without TLS, a network attacker can read
> the observer token and replay it for the full validity window, and
> can read the metadata (who registers, who fetches whose bundle).
> See `THREAT-MODEL.md § 1` (network attacker).
## 2. Backups
- [ ] **SQLite:** scheduled `sqlite3 /data/shade-prekeys.db ".backup ..."`
at least daily. The `.db` file plus `-wal` and `-shm` together is
the recovery unit; never copy the bare `.db` while the container
is running without using the online backup API.
- [ ] **Postgres:** `pg_dump` (or your provider's snapshot) at least
daily; verify a restore at least once per quarter.
- [ ] Backups are stored on different infrastructure than the primary
volume (different host / region / provider).
- [ ] Backups are encrypted at rest (your storage provider's
server-side encryption, age, or restic with a passphrase).
- [ ] **Restore drill:** at least once before going live, restore the
backup into a fresh volume and confirm `/health` is green and a
registered identity is still resolvable.
> **Why:** prekey records contain identity public keys and one-time
> prekeys. Losing them means new sessions can't be established to those
> identities until each user re-registers. Existing sessions keep
> ratcheting on the device-side state.
## 3. Observer token rotation
- [ ] `SHADE_OBSERVER_TOKEN` is set to **≥ 16 chars** of high-entropy
random data (e.g. `openssl rand -hex 32`). The server logs a
warning and disables the observer if the token is shorter.
- [ ] The token is held in your secret manager (Dokploy secret, GitHub
Actions secret, Vault, 1Password CLI), **never** committed to a
compose file or `.env` checked into git.
- [ ] The token is rotated on a schedule (recommended: every 90 days)
and immediately if it has been shared with anyone who no longer
needs access.
- [ ] If you expose the dashboard publicly, you also gate it behind
basic-auth at the proxy layer — bearer tokens are not
revocation-friendly on their own.
> **Why:** the observer dashboard exposes metadata about every active
> identity, registration timestamp, and recent activity. Anyone with
> the token can scrape the entire prekey directory.
## 4. SQLite vs PostgreSQL
Pick one and stick to it.
- [ ] **SQLite** is the default. Use it when **one** Shade container is
enough, you can tolerate downtime during backup snapshots, and
your write rate is below ~500 req/s. Path: `SHADE_PREKEY_DB_PATH`,
default `/data/shade-prekeys.db`.
- [ ] **PostgreSQL** is for multi-replica deployments, shared
infrastructure, or when you already operate a managed Postgres
and want one fewer thing to back up. Path: `SHADE_PREKEY_PG_URL`.
Tables are auto-created with `shade_server_*` prefix.
- [ ] Whichever you pick, the database lives behind TLS for the
connection (`sslmode=require` for Postgres) and on storage that
is itself encrypted (LUKS, EBS encryption, managed-DB encryption).
- [ ] You do **not** mix them in the same deployment. Setting
`SHADE_PREKEY_PG_URL` overrides SQLite silently — pick one in
`compose.yml` and document which.
> **Why:** Shade does **not** encrypt the database itself (V3.2 will).
> Disk-level / volume-level encryption is the operator's responsibility
> until at-rest encryption ships.
## 5. Log level and structured logs
- [ ] `SHADE_LOG_LEVEL` is set to `info` (production) or `warn`
(high-traffic). Avoid `debug` in prod — it logs request bodies
including signed payloads.
- [ ] Logs are shipped to a retention-bounded sink (Loki, CloudWatch,
Datadog) with **redaction of `Authorization` headers and signed
bodies** if your sink doesn't already strip them.
- [ ] You alert on `error`-level logs and on the absence of cleanup
cycles (a stuck cleanup loop = unbounded DB growth).
> **Why:** at `debug` level the server logs signature material. While
> Ed25519 signatures are not secrets per se, leaking them widens the
> replay-window blast radius and reveals timing patterns.
## 6. Stale-identity cleanup parameters
- [ ] `SHADE_STALE_DAYS` is set deliberately for your product. The
default (30 days) is right for "active chat app"; "occasional
use" apps should bump to 90+ to avoid surprise re-registration.
- [ ] `SHADE_CLEANUP_INTERVAL_HOURS` is left at 24 unless you have a
specific reason — running cleanup more often does not free more
space, and running it less often risks one cycle missing a day.
- [ ] You watch the `shade_cleanup_purged_total` metric (Prometheus) and
alert on a sudden 10× spike — that often signals a bug or a
deployment that broke client-side activity timestamps.
> **Why:** stale cleanup is the only thing keeping the prekey directory
> from growing forever. A misconfigured `SHADE_STALE_DAYS = 0` would
> nuke every identity on every cycle. Bound the value at ≥ 1 in your
> deployment config.
## 7. Secret rotation
- [ ] Identity signing keys: each consumer rotates via the documented
identity-rotation flow (7-day grace period for old sessions).
Operators do **not** touch identity keys directly.
- [ ] Observer token: see § 3.
- [ ] Database credentials (Postgres only): rotate per your standard
cadence, with the connection string supplied through the secret
manager.
- [ ] No long-lived API keys or service tokens are stored in the
container image or volume.
## 8. Rate-limit and body-size caps
- [ ] You have not lowered the built-in rate limits below the defaults
(per-IP register/bundle and per-identity replenish/delete).
- [ ] You have not raised the 64 KiB POST body limit. Prekey bundles
fit comfortably; raising the limit only enables abuse.
- [ ] Your reverse proxy enforces an additional connection / request-
rate limit at the edge (Caddy `rate_limit`, Cloudflare, etc.)
so a single noisy IP can't even reach Shade's per-route limits.
## 9. Health checks and metrics scrape
- [ ] Container has a Docker `HEALTHCHECK` (the official image already
ships one against `/health`).
- [ ] `/metrics` is scraped by Prometheus / OpenTelemetry and
retained ≥ 30 days.
- [ ] Alerts are wired for: `/health` failing for > 2 min, request
latency p99 > 1 s, error rate > 1 %, cleanup cycles missing for
> 25 h.
## 10. OpenAPI contract drift
- [ ] CI runs the OpenAPI lint (`bun test packages/shade-server/tests/openapi-lint.test.ts`)
on every PR — the spec must remain valid OpenAPI 3.1 with no
dangling `$ref`s.
- [ ] Generated clients (Python, Go, Kotlin) are regenerated from the
shipped spec on each release; mismatches between server and
client are caught at integration test time, not production.
---
## Pre-flight summary
If you can answer "yes" to every box above, ship it. If you can't,
write down which box and why before you do — that note belongs in your
runbook so the next operator inherits the gap, not the surprise.

116
docs/ROADMAP.md Normal file
View File

@@ -0,0 +1,116 @@
# Shade Roadmap — V3.1 → V5.0
Indeks over versjonsplanene fra V3.1-grunnsteinen via **Shade 4.0 GA** og
videre til **Shade 5.0** (Voice & Video).
- **V4.0 GA** ✅ — alt fra V2.1 / V2.2 / V2.3 og bonus-tracket (sosial
recovery, P2P WebRTC, Pub/Sub, Key Transparency) er merget, testet,
dokumentert og pakket for ekstern review. Wire-formatet er låst.
- **V5.0** = den dedikerte sanntids-releasen. *Alt* VOIP og videostreaming
ligger her — implementert oppå den frosne 4.0-stacken.
Alle V3.x-planer ligger nå under [`docs/archive/`](./archive/) med
`Status: Done`. Aktive planer: [`V5.0.md`](./V5.0.md).
---
## Faser
### Fase 1 — Documentation & Hardening Foundation ✅
| Plan | Tittel | Effort | Status |
|------|--------|--------|--------|
| [V3.1](./archive/V3.1.md) | Documentation & Hardening Foundation | S | **Done** |
### Fase 2 — Sikkerhetsmodning ✅
| Plan | Tittel | Effort | Status |
|------|--------|--------|--------|
| [V3.2](./archive/V3.2.md) | At-Rest Storage Encryption | L | **Done** |
| [V3.3](./archive/V3.3.md) | Fingerprint Gates & Trust UX | M | **Done** |
| [V3.4](./archive/V3.4.md) | Observability v2 (OpenTelemetry) | M | **Done** |
| [V3.5](./archive/V3.5.md) | Android Parity & Cross-Platform CI | XL | **Done** |
### Fase 3 — Plattformutvidelse ✅
| Plan | Tittel | Effort | Status |
|------|--------|--------|--------|
| [V3.6](./archive/V3.6.md) | Async Store-and-Forward (Inbox) | L | **Done** |
| [V3.7](./archive/V3.7.md) | Transport Bridge (SSE / long-poll) | M | **Done** |
| [V3.8](./archive/V3.8.md) | Web Workers Crypto | M-L | **Done** |
| [V3.9](./archive/V3.9.md) | Rich File Metadata & Previews | M | **Done** |
### Fase 4 — Tillit og P2P-transport ✅
| Plan | Tittel | Effort | Status |
|------|--------|--------|--------|
| [V3.10](./archive/V3.10.md) | Social Key Recovery | L | **Done** |
| [V3.11](./archive/V3.11.md) | WebRTC P2P Transport | XL | **Done** |
| [V3.12](./archive/V3.12.md) | Key Transparency | XXL | **Done** |
### Fase 5 — General Availability ✅
| Plan | Tittel | Effort | Status |
|------|--------|--------|--------|
| [V4.0](./archive/V4.0.md) | External Audit, Consolidation, GA | M | **Done** |
### Fase 6 — Sanntid (post-GA)
| Plan | Tittel | Effort | Avhenger av |
|------|--------|--------|-------------|
| [V5.0](./V5.0.md) | Voice & Video | XXL | V4.0 GA + V3.11 |
---
## Effort-nøkkel
| Symbol | Tid |
|--------|-----|
| **S** | 12 uker |
| **M** | 24 uker |
| **L** | 48 uker |
| **XL** | 24 måneder |
| **XXL** | 4+ måneder / multi-quarter |
---
## Avhengighetsgraf
```text
V3.1 ────┬──► V3.2 ──┐
├──► V3.3 ──┼──► V3.10 ──┐
├──► V3.4 ──┘ │
├──► V3.5 ───────────────┼──► V3.12 ──┐
├──► V3.6 ──► V3.7 ──► V3.11 ─────────┤
├──► V3.8 ├──► V4.0 GA ──► V5.0 (Voice & Video)
└──► V3.9 ─────────────────────────────┘
```
---
## Status-konvensjon
Hver plan har et `Status:`-felt øverst. Lov verdier:
- `Idea` — ikke startet, design fortsatt åpent.
- `Design` — designnotat under arbeid eller approved.
- `IMP` — implementasjon pågår.
- `Done` — merget i main, dekket av tester.
Når en plan blir `Done`, flytt fila til `docs/archive/` og oppdater denne tabellen.
---
## Versjonering
- **V3.1 → V3.12** ble trinnvise minor-releases på `0.4.x`-linjen.
- Wire-format-endringer akkumulerte til **V4.0**, men endte med å være
uendret fra 0.4.x — major-bumpen til 4.0 markerer audit-cycle ferdig
og GA-frosset kjerne, ikke en wire-bump.
- **V4.0** er GA — låst kjerne, pakket for ekstern review, ingen
voice/video.
- **V5.0** legger sanntid (voice/video/broadcast) oppå den frosne
4.0-stacken. Bygger på reserverte envelope-typer slik at 4.0-klienter
ignorerer 5.0-trafikk gracefully — ikke breaking.
- Hver `V*`-merge oppdaterer `CHANGELOG.md` og bumper alle pakker via
`bun run version`.

65
docs/SHADE-BY-SCENARIO.md Normal file
View File

@@ -0,0 +1,65 @@
# Shade by scenario — modular E2EE toolkit
This page is for **builders**, not cryptography specialists. Shade is packaged so you can **drop in small pieces** per project instead of importing a heavy “everything” stack.
---
## Plain-language mental model
1. **Identity & session (the hard crypto):** Shade establishes a **secret channel** between two parties using the same kind of cryptographic core as Signal (initial setup + ongoing “ratchet” updates). **You mostly call high-level APIs** (`send`, `receive`, fingerprints) rather than assembling primitives by hand.
2. **Who is “the server”?**
The **prekey server** only helps with **public key material** (so Alice can fetch Bobs public bundle before the first message). It is **not** your general-purpose message relay unless **you** build that separately. Normal **message payloads and file chunks** typically flow over **your** transport (your HTTP routes, websocket, bridge, queue, etc.).
3. **Small vs large payloads:**
Short messages ride the usual ratchet envelopes. Very large payloads use **Streams + Transfer**: secrets are negotiated over the ratchet; ** ciphertext chunks** ship over optimized HTTP/WebSocket transports with parallelism and resume.
4. **Trust:** Strong encryption does **not** replace **verifying who you are talking to**. For high-stakes use, compare **safety numbers** out-of-band (see [THREAT-MODEL.md](../THREAT-MODEL.md)).
---
## Scenario → minimum packages
Pull in **one row** that matches your project; add optional columns only when needed.
| Scenario | What you need | Minimum packages / surface | Where to start |
|----------|----------------|----------------------------|----------------|
| **Backend or Bun service** — encrypted messages between users | Session storage + crypto provider + prekey URL | `@shade/sdk` + `sqlite:` or `@shade/storage-postgres` | `createShade()``send` / `receive` |
| **Browser / frontend** — same, in the client | Web crypto + durable or memory storage | `@shade/sdk` or `@shade/core` + `@shade/crypto-web` (+ storage you provide) | Same APIs; ensure `prekeyServer` is reachable from the browser (CORS, etc.) |
| **Large files** — resumable E2EE upload/download | Above + stream protocol + HTTP (or WS) transport | `@shade/sdk` (re-exports transfer) + mount transfer routes on **your** HTTP server | `shade.upload` / `onIncomingTransfer` — see [streams.md](./streams.md) |
| **React UI** — upload/download widgets | Runtime from SDK + widgets | `@shade/sdk` + `@shade/widgets` | `ShadeRuntimeProvider`, `useShadeUpload` / `useShadeDownload` |
| **Prekey hosting only** — one container per product | No app crypto in the container | Docker image / `@shade/server` | Deploy prekey image; point `prekeyServer` at it from apps |
| **Offline-tolerant messaging** — recipient may be offline | Above + a relay that holds ciphertext blobs | `@shade/inbox` (client) + `@shade/inbox-server` (or the prekey container, which bundles both) | Register address, `inbox.send()` to peer, `inbox.onIncoming(handler)` — see [inbox.md](./inbox.md) |
| **"What if I lose my phone?"** — survive device loss without a recovery agent | Above + Shamir-split shares to `n` guardians; threshold `k` reconstruct | `@shade/recovery` + `@shade/widgets` (`<RecoverySetup />`, `<RecoveryRequest />`, `<RecoveryApprove />`) | `setupRecovery` / `attachGuardian` / `requestRecovery` — see [recovery.md](./recovery.md) |
| **Maximum control** — custom wire, custom transport | Wire + session manager | `@shade/core` + `@shade/proto` (+ your storage + crypto provider) | `ShadeSessionManager`, encode/decode envelopes yourself |
| **HTTP or WebSocket convenience** | Auto-wrap application bytes | `@shade/transport` on top of your stack | Use when you want transport helpers, not a new protocol |
| **Android** | Byte-compatible with TS (cross-vector gated in CI) | `shade-android` module | See [android/shade-android/README.md](../android/shade-android/README.md). Cross-platform vectors live in [`test-vectors/`](../test-vectors/) and are exercised by both runners. |
You can **mix rows**: e.g. backend with `@shade/sdk` + SQLite for sessions, separate service mounting `transfer` routes, browser clients using `@shade/widgets`.
---
## New project checklist (lightweight)
1. Run a **prekey server** (Docker or embedded `@shade/server`) for your environment.
2. Pick **storage** (`sqlite:…`, Postgres, or project-specific adapter implementing the core storage interfaces).
3. Choose **surface**: usually `@shade/sdk` unless you truly need `@shade/core` only.
4. For files: enable **transfer routes** and authenticate chunk uploads using the patterns in the SDK (see streams doc).
5. Run **`shade doctor`** when something fails in production-ish setups (install the CLI as in repository [Quick start](../README.md#quick-start)); the gates that fire are documented in [trust-ux.md](./trust-ux.md) and [PRODUCTION-CHECKLIST.md](./PRODUCTION-CHECKLIST.md).
---
## Related docs & roadmap
| Topic | Doc |
|--------|-----|
| File transfer architecture | [streams.md](./streams.md) |
| Deployment & operations | [DEPLOYMENT.md](./DEPLOYMENT.md) |
| Threat model | [THREAT-MODEL.md](../THREAT-MODEL.md) |
| Planned improvements | [ROADMAP](./ROADMAP.md) — V3.x archive under [`archive/`](./archive/), next milestone [V5.0](./V5.0.md) |
---
## Version note
This file describes how Shade is **intended** to be composed. Package names and re-exports may gain small aliases over time; the **scenario table** should remain the source of truth for “what to install for what job.” Update this page when adding major surfaces (new transport bridges, richer `shade init` templates, etc.).

135
docs/V5.0.md Normal file
View File

@@ -0,0 +1,135 @@
# Shade V5.0 — Voice & Video
**Status:** Idea (post-V4.0 GA)
**Effort:** XXL (4+ måneder)
**Forrige:** V4.0 GA + V3.11 (P2P transport kreves)
**Adresserer:** V2.1-tillegg "ShadeVoiceButton / ShadeVideoCall / ShadeBroadcaster"
V5.0 er den dedikerte sanntids-releasen — alt VOIP og videostreaming
samles her, *etter* at Shade 4.0 er GA-merket. Stacken under
(ratchet, transport, observability, recovery, key transparency,
WebRTC P2P) er låst i 4.0; 5.0 bygger uten å røre kjernekrypto-
revisjonen.
---
## Mål
E2EE sanntidskommunikasjon på Shade-stack: voice-calls, video-calls,
broadcast/streaming — alt som "magic drop-in"-komponenter for konsumerende
apper.
```tsx
<ShadeVoiceButton to={peerAddress} />
<ShadeVideoCall to="device:server-admin" />
<ShadeBroadcaster streamKey="game-stream-1" />
<ShadeViewer streamKey="game-stream-1" />
```
---
## Scope
### Inn
- Ny pakke `@shade/voice` — 1:1 voice over WebRTC P2P.
- Ny pakke `@shade/video` — 1:1 video, deler kjerne med voice.
- Ny pakke `@shade/broadcast` — 1:N broadcast med relay-helper.
- SFrame-style frame encryption — payload-keys ratchet'es per call,
derivert fra Shade-session.
- Codec: Opus (audio), AV1/VP9 (video) — WebRTC standard.
- Widget-komponenter for hvert use case.
- Key-rotation under loss: forward-secrecy per X frames eller hvert N
sekund.
### Ut
- Group-calls (≥ 3 deltakere) som første milestone — krever SFU + group
key agreement; egen sak.
- Replacement for native phone-app — vi tilbyr in-app calls.
- Codec-implementasjon — vi bruker browser/native WebRTC.
---
## Design
### Frame-key derivasjon
```text
callKey = X3DH(A, B) → HKDF("shade-call-v1") → callRatchetKey
frameKey[i] = HKDF(callRatchetKey, "frame" || u64(i))
```
`callRatchetKey` ratcheter forward hver N millisekund eller hver M frames;
kompromittert frame = bare det vinduet eksponert.
### SFrame
Følger IETF MLS/SFrame-mønstre:
- Header er klartekst (codec-metadata).
- Payload er AES-GCM med deterministisk nonce.
- Mottaker dropper frames med out-of-window seq.
### Topologi
- 1:1: P2P via V3.11.
- Broadcast: relay-helper i `@shade/broadcast-relay` distribuerer
ciphertext til subscribers — relay ser aldri plaintext.
---
## Leveranser
### Pakker
- `@shade/voice` + `@shade/video` (delt kjerne i `@shade/realtime-core`).
- `@shade/broadcast` + `@shade/broadcast-relay`.
- Widgets: `<ShadeVoiceButton />`, `<ShadeVideoCall />`,
`<ShadeBroadcaster />`, `<ShadeViewer />`.
### Tester
- Unit: SFrame encrypt/decrypt + tamper.
- Integration: 1:1 video 30 fps i 60 s; > 99 % frames levert; key rotation
observert.
- Loss recovery: 30 % packet loss → quality grace.
- Adversarial: relay-DB-dump avslører ingen plaintext.
### Dokumentasjon
- `docs/voice-video.md` — setup, codec-tradeoffs, broadcast-arkitektur.
---
## Akseptansekriterier
- [ ] 1:1 video 60 fps + 1080p mellom to klienter samme LAN.
- [ ] Frame-key kompromittering blokkerer maks 1 sekund forward data.
- [ ] Broadcast 1:50 viewers fungerer med < 2 s end-to-end latency.
---
## Avhengigheter
- **V4.0 GA** — kjerne-stacken må være ekstern-revidert og frosset før
vi legger sanntid-protokoll oppå.
- V3.11 — P2P transport (kommer i V4.0-vinduet).
- V3.5 — Android-paritet hvis voice/video skal funke på mobile.
---
## Risiko
- **Codec-quirks.** AV1 vs VP9 vs H.264 har ulik browser-støtte.
- **Frame-key sync under loss.** Avansert; SFrame-spec er fortsatt under
standardisering.
- **Latency vs sikkerhet.** Hver ratchet-step legger på µs.
---
## Migrasjon
Nye pakker. Ikke breaking — wire-formatene fra V4.0 holdes uendret;
voice/video legger til egne envelope-typer i et reservert range som
4.0-clients ignorerer.

154
docs/archive/V2.1.md Normal file
View File

@@ -0,0 +1,154 @@
# Shade V2.1 — Improvements (infrastructure, storage, operations, security)
**Status:** Done — superseded by the V3.1 → V3.12 plans, all of which
landed in the 4.0 GA release. This document is preserved as historical
context for the original V2.1 backlog; the concrete deliverables live
under [`docs/archive/V3.*.md`](./).
This document describes **improvements** agreed for next-generation work on Shade: clearer product story, stronger storage, mobile parity, operational hardening, transfer abuse, and a formal security narrative.
---
## 1. Clear “who is the server?” and data flow
**Problem:** New users may think the prekey server is a message hub or that all E2EE traffic goes through the Shade container.
**Goal:** One consistent explanation across the root README, package READMEs, and optional onboarding: **the prekey server distributes public keys and bundles**; **actual messages and (typically) file chunks go through your apps own channel** (your transport, your backend, your URLs).
**Deliverables (proposal):**
- Diagram + short “keys vs payloads” text in the root README and in `@shade/server` README.
- Link to `THREAT-MODEL.md` from the same section (MITM on first contact ↔ safety numbers).
- Optionally one “concept page” (or extend `MIGRATION.md`) with typical architecture: *A ↔ B via app; both talk to the prekey host for X3DH material*.
**Acceptance criteria:** A new developer without domain background understands in one reading *what* goes to the Shade server and *what* does not.
---
## 2. Optional encryption of storage (at-rest)
**Problem:** `THREAT-MODEL.md` states that a stolen DB + filesystem can expose private keys because Shade does not encrypt the storage layer by default.
**Goal:** **Opt-in** protection for sensitive state (identity, session, optional stream resume secrets) with keys that **do not** live in plaintext in the DB — e.g. OS keychain/Keystore, passphrase + KDF, or an explicit device key injected by the app.
**Design principles:**
- Default developer experience (dev, simple demos) stays unchanged or includes a clear “insecure mode” warning in docs.
- APIs implementable per platform (Bun/SQLite, Postgres, web/IndexedDB, Android).
- Document limitations: what remains uncovered (e.g. active memory compromise).
**Acceptance criteria:** Threat model updated for “when encrypted storage is enabled”; at least one reference implementation + migration note.
---
## 3. Android parity and a published roadmap
**Problem:** `shade-android` is under development; drift from the TS SDK undermines the “byte-compatible” promise.
**Goal:** A **published roadmap** (milestones + what counts as parity vs TS-only) and **CI running shared test vectors** as a merge gate before release.
**Deliverables:**
- Roadmap section in `android/shade-android/README.md` or dedicated `ROADMAP-ANDROID.md` with explicit cross-checkpoints: wire format, fingerprints, rotations, streams (`0x11`) where applicable, resume semantics.
- CI job that fails on Kotlin vs TS vector mismatch.
**Acceptance criteria:** Parity coverage is visible and enforceable; the first critical cross-surface (e.g. core ratchet + proto) is green before a “production” label.
---
## 4. Operational hardening — prekey container and production
**Problem:** Many teams deploy the Docker image quickly; mistakes around TLS, backups, and secrets add avoidable risk.
**Goal:** A **production checklist**: TLS termination, volume backup (`/data`), rotation of `SHADE_OBSERVER_TOKEN`, use of `SHADE_PREKEY_PG_URL` vs SQLite, observability hooks, logging levels, meaning of stale cleanup parameters.
**Deliverables:**
- Extend `docs/DEPLOYMENT.md` or add short `docs/PRODUCTION-CHECKLIST.md` with bullet defaults.
- Link from the main README under “Deployment”.
**Acceptance criteria:** A checklist operators can follow without reading the whole codebase first.
---
## 5. Abuse and resource limits on the transfer plane
**Problem:** Parallel lanes and large uploads can be abused for resource or storage if consumer mounts of `createTransferRoutes()` share no coherent policy.
**Goal:** Documented **limits and patterns**: authentication (already an active SDK topic), max stream size, TTL for temporary chunk storage, quotas per identity or IP where sensible.
**Deliverables:**
- Guidelines in `docs/streams.md` or a dedicated “Transfer hardening” section.
- Optional helpers or middleware examples in `@shade/transfer` / server routes for common limits (without forcing every deployment into one DB model).
**Acceptance criteria:** A clear “recommended minimum” for production that teams can copy.
---
## 6. Security review and formal test / narrative
**Problem:** Enterprises and security-conscious users often ask for independent review and a traceable test matrix.
**Goal:** Plan for **independent crypto review** (timing, scope, deliverables) and a **published test / threat matrix** linking `THREAT-MODEL.md` to concrete automated tests (replay, tamper, out-of-order, resume, etc.).
**Deliverables:**
- Internal checklist “preparing for external review” (which files, assumptions, known limits).
- Short section in `SECURITY.md` on review status and how to report findings.
**Acceptance criteria:** One authoritative source for “what is tested automatically” vs “what needs manual/MITM/out-of-band process”.
---
## Dependencies and order (proposal)
| Priority | Topic | Note |
|---------|-------|------|
| Low friction | §1, §4 | Documentation and checklists only |
| Medium | §5 | Design + docs + possibly small API helpers |
| High effort | §2, §3 | Platform work and long-term maintenance |
| Coordinated | §6 | Parallel with maturity in §2§5 |
---
## Document versioning
- **V2.1** — first consolidated plan for the six improvement areas above. Update with dates and short per-section status lines as work starts and completes.
Gitt der du er nå (post-streams 0.2.0), ville jeg prioritert:
1. Sosial nøkkel-recovery — løser "hva skjer hvis jeg mister telefonen?" som er det store hullet i alle
E2EE-systemer
2. P2P WebRTC transport — naturlig kompletterende til streams; gir massiv throughput-gevinst gratis
3. E2EE Pub/Sub — lavt scope, høy nytte, og passer perfekt med din "magic drop-in"-filosofi
Disse tre har alle samme egenskap: de er infrastruktur-features som hver app du bygger fremover vil dra nytte av,
ikke bare Dispatch. Helt i tråd med "Shade som komponent som dras inn i hvilket som helst prosjekt".
/btw Hva med Shade videostreaming, VOIP streaming, og lignende?
Magic drop-in-tilnærming (din filosofi)
Tenk deg at i en Dispatch-app kan du bare gjøre:
<ShadeVideoCall to="device:server-admin" />
<ShadeVoiceButton to={peerAddress} />
<ShadeBroadcaster streamKey="game-stream-1" />
<ShadeViewer streamKey="game-stream-1" />
…og det bare fungerer, E2EE, P2P når mulig, fallback til relay automatisk. Det er det samme mønsteret som streams
0.2.0, bare for sanntid.
Realisme-sjekk
Video/VOIP er det vanskeligste i hele E2EE-verdenen. Signal brukte år på å få det riktig. Du bør:
1. Ferdigstille streams 0.2.0 først (verifiserer crypto-fundamentet)
2. Bygge P2P WebRTC-transport som separat milestone
3. Da har du alle byggeklossene og Voice 0.4.0 blir 70% gjenbruk
Men ja — dette hører absolutt hjemme i Shade. Shade som "alt-i-ett E2EE-platform" er en mye sterkere posisjon enn
"bare messaging + filer". Du kan bli til E2EE hva Twilio er til vanlig kommunikasjon.

128
docs/archive/V2.2.md Normal file
View File

@@ -0,0 +1,128 @@
# Shade V2.2 — Feature plan: product, platform, and developer experience
**Status:** Done — superseded by V3.6 (inbox), V3.7 (bridge), V3.8
(workers), V3.9 (file metadata), V3.10 (recovery), V3.12 (KT). All
landed in the 4.0 GA release; see [`docs/ROADMAP.md`](../ROADMAP.md).
This document gathers **planned features** that extend Shade beyond todays core (X3DH + Double Ratchet + Streams/transfer): groups, asynchronous delivery, richer file UX, web workers, CLI, API docs, and scaffolding.
---
## 1. Groups / multiple participants
**Vision:** Beyond strict 1:1 — multiple identities in the same “conversation” or shared context (messages and possibly shared file/stream policy).
**Challenges:**
- Todays Signal-like core is naturally 1:1; groups need either **pairwise sessions per member**, **sender keys / fan-out**, or a **dedicated group protocol** (larger architectural step).
- Lifecycle: invites, member removal, compromised device, history, and group PCS.
**Goals for early milestones (proposal):**
1. **Document** a recommended pattern for “group-lite” (e.g. coordinator relays ciphertext without decrypting + clients encrypt per-recipient).
2. **Optional** minimal API making fan-out easier in the SDK (without promising full MLS).
**Acceptance criteria (MVP definition):** Explicit scope in docs + one reference architecture; no ambiguous “group is done” without an updated threat model.
---
## 2. Async store-and-forward messaging
**Vision:** Recipient need not be online; **ciphertext** is stored temporarily and fetched when the recipient returns — the server never sees plaintext.
**Distinction from the prekey server:**
- Prekey stays **public keys only** (or extended only under strict policy).
- A **dedicated relay/inbox service** (or app-owned backend) holds **encrypted blobs only** with TTL, idempotency, and authorization (who may list/fetch).
**Deliverables (proposal):**
- Protocol sketch: address registration, `PUT` blob, `GET`/`DELETE` or lease, replay protection at the application layer.
- SDK helpers: outgoing queue, poll/pull, or push-notification hook (without dictating mobile platform).
**Acceptance criteria:** Threat-model section “what the relay sees” + reference implementation or example app.
---
## 3. File metadata and preview
**Vision:** Richer UX **without** leaking sensitive content to the server: filename, MIME type, length where known; **optional** client-generated thumbnails or previews encrypted as separate blocks or small payloads on the control init path.
**Technical considerations:**
- Anything sent must be **E2EE** or omitted; plaintext metadata on the server must be deliberate and minimal.
- Thumbnails should use **format hardening** on the client (size limits, sandboxing in UI).
**Acceptance criteria:** Extended `stream-init` (or sidecar envelope) with optional fields + widget support for “preview when available”.
---
## 4. Web: worker-based crypto and streaming
**Vision:** Large files in the browser without blocking the main thread or blowing RAM — **Web Crypto / noble** inside a **Worker**, **ReadableStream/WritableStream** end-to-end chunk pipeline aligned with `@shade/streams` / transfer.
**Deliverables:**
- `@shade/crypto-web` (or companion) patterns: transferable buffers, lifecycle, errors surfaced to the UI.
- Documented constraints (Safari, chunk sizing, Service Worker vs dedicated worker).
**Acceptance criteria:** E2E demo or test that sends multiMiB through a worker without a blocking UI.
---
## 5. CLI: `shade doctor`
**Vision:** One command that **diagnoses the environment** before production debugging.
**Typical checks (proposal):**
- Reachability of `prekeyServer` (`/health`, optional OpenAPI).
- Local config: storage path, rotation headers, clock skew (relevant for signed requests).
- **Streams:** transfer routes mounted, auth matches expected key, `GET .../state` behaves as expected in test mode.
- CLI / Node/Bun runtime versions and `@shade/*` packages where readable from `package.json`.
**Acceptance criteria:** `shade doctor` with exit codes suitable for CI (warn vs fail).
---
## 6. OpenAPI / docs
**Vision:** All HTTP contracts teams are expected to implement (prekey **and** transfer) appear in **one** OpenAPI story or clearly linked specs — not only README examples.
**Deliverables:**
- Consolidate or cross-reference `openapi.yaml` with transfer endpoints (`/v1/transfer/*`) and security schemes for chunk upload.
- `/docs` (Redoc or similar) or published static artifacts for versioned specs.
**Acceptance criteria:** Generated client (e.g. Python/Go) from spec without manual fixes for the happy path.
---
## 7. `shade init`
**Vision:** Scaffolding from **empty repo** to a **minimal runnable** app with Shade and optional streams.
**Extensions:**
- New or extended template: **minimal Hono/Fastify** app with `Shade.transferRoute()` mounted, auth example matching the SDK authenticator, `.env` template.
- Optional: demo `shade doctor` after init.
**Acceptance criteria:** `shade init …` produces a project that `bun install && bun run start` runs with documented env vars.
---
## Dependencies between items
```text
shade init ─────► doctor (same conventions for URLs and secrets)
openapi/docs ◄── transfer + prekey (single source)
web workers ───► streams UX (large file in browser)
groups ◄──────── store-and-forward (often related socially/technically)
metadata/preview► widgets + proto/control plane
```
---
## Document versioning
- **V2.2** — feature backlog as described. Split into issues/ADR per feature when implementation starts.

106
docs/archive/V2.3.md Normal file
View File

@@ -0,0 +1,106 @@
# Shade V2.3 — Tillit, retention, integrasjon og observability
**Status:** Done — superseded by V3.3 (trust UX), V3.4 (observability),
V3.10 (recovery), V3.12 (KT). Alt levert i 4.0 GA — se
[`docs/ROADMAP.md`](../ROADMAP.md).
Dette dokumentet beskriver **høyere ambisjonsnivå** og **plattformkryssende** arbeid: brukertilfeller der tillit må være **eksplisitt**, data må **utkrympes**/ryddes automatisk, apper må **enkelt koble seg på** kjente transportmønstre, og drift må **observere** uten å lekke innhold.
---
## 1. Key transparency / bundle-attestasjon **eller** kritiske fingerprint-øyeblikk i UI
Dette kan sees som **to spor** samme mål: redusere tillit til én korrumperbar prekey-server uten brukerens merknad.
### Spor A — Key transparency / bundle-attestasjon (ambisiøst)
**Idé:** Logging av **hvilket bundle som ble utlevert når**, kryptografisk forankret og **verifiserbar av klienter** eller tredjeparts-audit (inspirert av KT-litteratur, tilpasset Shade sin trusselmodell).
**Leveranser (målpris høyt):**
- Trusselmodell-oppdatering: *hva* CT/attest løser vs *fortsatt MITM før første verifisering*.
- Designnotat: datastruktur, friskhetsbevis, klient-verifikasjonssteg, operatørkost.
**Risiko:** Kompleks drift og kryptodesign — bør være **valgfritt** lag for motiverte deploys.
### Spor B — Fingerprints i app-UI på kritiske hendelser (pragmatisk)
**Idé:** Gjør **safety numbers** synlige og **handlingspålagte** i presiserte flyt:
- **Før første stor fil** (eller før første stream over terskel i bytes).
- **Før «trust this device»** / backup-importer / ny enhet som gjenbruker identitet.
**Leveranser:**
- `@shade/widgets` + SDK-hooks: modal/sheet med fingerprint, kopier-OOB-tekst, «jeg har verifisert».
- Dokumenterte UX-retningslinjer (unngå «alert fatigue» kun på lave risiko-events).
### Felles akseptansekriterier
Enten spor A eller B må ha **eksplisitt testcoverage** på «blokkerer/handhever verifisering» der det er lovet, og trusselmodellen må nevne kombinasjonen av OOB + tekniske grep.
---
## 2. Retention policies
**Vision:** Standardiserte **TTL- og oppryddingsregler** for data Shade-økosystemet etterlater på server eller i klient-lagring — spesielt:
- **Stream-state** og midlertidige chunk-referanser etter fullførte/avbrutte transfers.
- **Eventuell** inbox/relay-ciphertext (`V2.2`).
- Prekey-server: kobling til eksisterende `SHADE_STALE_DAYS` / cleanup, plus **policy-dokumentasjon** for operatører.
**Leveranser:**
- Default-anbefalinger (f.eks. «ferdige streams: prune etter N dager») i `@shade/storage-*` helpers eller server-factory.
- Konfigurerbare hooks: `maxAge`, `maxBytesPerIdentity`, cron vs on-access prune.
**Akseptansekriterier:** Ingen «uendelig vekst» som default i nye templates; dokumentert adferd i `streams.md` / deployment.
---
## 3. Ferdig «bridge» (transport)
**Vision:** Apper som ikke kan eller vil bruke WebSocket, får **ferdig mønster** for mottak av små meldinger eller kontroll-signaler.
**Eksempler:**
- **Server-Sent Events (SSE)** som **ren ciphertext-pipe** eller som signal «hent fra inbox» uten plaintext på server (kombinasjon med `V2.2`).
- **Lang-poll fallback** dokumentert ved siden av WS i `@shade/transport`.
**Leveranser:**
- Modulær `bridge`-pakke eller tydelig undermodul med få eksponerte typer og én felles `IncomingMessage`-modell på klient.
**Akseptansekriterier:** Ett fungende eksempel + test som viser dekryptering likt eksisterende transport.
---
## 4. Observability
**Vision:** Produksjonsteam får **målepunkter og spor** uten **innholdslekkasjer** eller kryptomatrise i logger.
**Forslag til innhold:**
- **Metrics (Prometheus-stil eller vendor-nøytralt):** opplastings-/nedlastings-varighet, lane-telling, retry counts, abort vs complete rates, HTTP/WS-feilkoder (aggregert).
- **OpenTelemetry:** spans rundt `TransferEngine` og prekey-endepunkter (uten payload-lengde i klartekst som PII — bruk binære størrelser eller binning).
- **Sampling og PII-policy** dokumentert (ikke logg adresser i full hvis compliance krever maskering).
**Akseptansekriterier:** Opt-in flags (default av i lib, på i server-container der det gir mening), og `docs/` avsnitt om hva som **aldri** skal logges.
---
## Prioritering mot V2.1 / V2.2
| Dette dokument (V2.3) | Naturlig forutsetning |
|----------------------|------------------------|
| Retention | Streams/transfer i bruk (`V2.1` §5, `V2.2` metadata) |
| Bridge | `V2.2` store-and-forward eller egen meldingskanal |
| UI fingerprints | Widgets/SDK allerede i bruk |
| KT / attest | Moden trusselmodell + juridisk/operativ vilje |
| Observability | Stabil nok intern API for hooks |
---
## Versjonering
- **V2.3** — første samlet plan for tillit, retention, bridge og observability. Splitt i ADR-er når konkret design er valgt.

100
docs/archive/V3.1.md Normal file
View File

@@ -0,0 +1,100 @@
# Shade V3.1 — Documentation & Hardening Foundation
**Status:** Done
**Effort:** S (12 uker)
**Forrige:** V2.3
**Neste:** V3.2 / V3.3 / V3.4 (kan kjøres parallelt)
---
## Mål
Lukke "lav-friksjon"-gjelden fra V2.1, V2.2 og V2.3 før vi tar fatt på de tunge
sikkerhetsløftene. Dette er pre-arbeidet som låser opp resten av roadmapen:
operatører skal kunne deploye trygt, transfer-konsumenter skal ha klare grenser,
og OpenAPI skal dekke hele HTTP-flaten.
Ingen ny kjernekode — kun docs, OpenAPI-utvidelser, retention-defaults og en
test-/threat-matrise.
---
## Scope
### Inn
- README + `@shade/server`-README: eksplisitt "keys vs payloads"-narrativ med
diagram + lenke til `THREAT-MODEL.md`.
- Ny `docs/PRODUCTION-CHECKLIST.md`: TLS, backup, observer-token-rotering,
SQLite vs PG, log-nivå, stale-params, secret-rotering.
- Hardening-seksjon i `docs/streams.md`: max stream-size, TTL, quota-mønstre —
peker mot `@shade/files`-hooks som referanse.
- `openapi.yaml` utvidet med `/v1/transfer/*` (`chunk`, `state`, `health`) +
sikkerhetsskjema for `ShadeTransferAuthenticator`.
- Retention-defaults i `docs/streams.md` + SDK-template:
`pruneStreamStates`-cron som default — "ferdige streams ryddes etter N
dager".
- `SECURITY.md`-utvidelse: review-status, "hvordan rapportere", lenking fra
`THREAT-MODEL.md`-rader → `tests/security/*` (test-/threat-matrise).
### Ut
- Faktisk crypto-review (det er V4.0).
- Endringer i krypto- eller wire-format.
- Ny kode utenfor SDK-templates.
---
## Leveranser
### Dokumentasjon
- `docs/PRODUCTION-CHECKLIST.md` — ny.
- `docs/streams.md` — utvidet med "Hardening" og "Retention".
- `README.md` — diagram-justering + "Hva som ikke går via Shade-server".
- `packages/shade-server/README.md` — speile narrativet.
- `SECURITY.md` — review-status + threat-/test-matrise.
- `THREAT-MODEL.md` — krysslenker til konkrete tester.
### Kode (kun konfig + templates)
- `packages/shade-server/openapi.yaml``/v1/transfer/*`-paths,
`ShadeTransferAuthenticator` securityScheme.
- `packages/shade-cli/templates/bun-server` — default
`pruneStreamStates`-cron.
### Tester
- Lint-test: OpenAPI-spec validerer fortsatt mot OpenAPI 3.1-skjema.
- Smoke-test for cron i template.
---
## Akseptansekriterier
- [ ] Ny utvikler kan lese README + `PRODUCTION-CHECKLIST.md` og deploye
prod-klar Shade uten å lese hele kodebasen.
- [ ] Generert klient (Python eller Go) fra `openapi.yaml` dekker både
prekey- og transfer-flate uten manuelle fixes for happy path.
- [ ] `THREAT-MODEL.md` linker hver "Mitigations"-rad til minst én test-fil.
- [ ] Default SDK-template `bun-server` prune'r resumable streams uten
manuell konfig.
---
## Avhengigheter
Ingen.
---
## Risiko
Lav. Verste utfall er foreldet docs hvis V3.2+ endrer overflater. Mitiger ved
å skrive små, oppdaterbare seksjoner heller enn lange narrative kapitler.
---
## Migrasjon
Ingen — alt er additivt.

134
docs/archive/V3.10.md Normal file
View File

@@ -0,0 +1,134 @@
# Shade V3.10 — Social Key Recovery
**Status:** Done — landet i `@shade/recovery` 0.4.0, frosset i 4.0 GA.
**Effort:** L (48 uker)
**Forrige:** V3.2 + V3.3
**Adresserer:** V2.1-tillegg "sosial nøkkel-recovery"
---
## Mål
Løs det største UX-hullet i alle E2EE-systemer: **"Hva skjer hvis jeg
mister telefonen?"**. Bruker velger N "guardians" (familie / venner /
jobb-partnere); når bruker mister enheten, kan en threshold-andel av
guardians sammen returnere identity-nøkkelen — uten at noen enkelt guardian
kan gjøre det alene, og uten at server lærer noe.
---
## Scope
### Inn
- Shamir Secret Sharing (k-of-n) over identity private key (eller en
backup-encryption-key).
- Distribusjon av shares via eksisterende 1:1 Shade-sesjoner — guardians
lagrer share lokalt.
- Recovery-flow: ny enhet ber threshold guardians sende sine shares;
rekonstruerer på ny enhet.
- Verifikasjons-step: ny enhet beviser identitet til hver guardian via OOB
safety-number-sammenligning **før** guardian frigjør share.
- UX-guide: hvor mange guardians, hvilken threshold, hvordan rotere når en
guardian mister enhet.
### Ut
- "Cloud guardian" / Shade-driftet recovery — vi tillater ingen sentralisert
komponent som kan gjøre det alene.
- Auto-distribusjon (vi krever eksplisitt valg av guardians).
---
## Design
### Hva deles
```text
shareSecret = AES-256-GCM-encrypt(identityState, recoveryKey)
recoveryKey is Shamir-split(k, n) → shares[i]
shareSecret stored locally + on each guardian
each guardian receives one share via Shade.send
```
`identityState` er det samme som `Shade.exportBackup` (eksisterer i 0.3.x),
men her gjenbrukes formatet.
### Recovery-flow
1. Ny enhet genererer **temporary** identity + safety number.
2. Ny enhet kontakter guardians via prekey-server (OOB verifisering først).
3. Hver guardian godkjenner manuelt og returnerer sin share via
`Shade.send`.
4. Ny enhet rekonstruerer `recoveryKey`, dekrypterer `shareSecret`,
gjenoppretter identity.
5. Original identity roterer (gammel identitet markeres som
"compromised — used for recovery").
### Guardian-UX
- Guardian-app/widget viser:
*"Alice (din venn) har mistet sin enhet og ber om recovery share.
Bekreft fingerprint før du sender."*
- Guardian kan **avslå** uten konsekvens.
---
## Leveranser
### Pakker
- `@shade/recovery` — Shamir + share-distribusjon.
- `@shade/widgets``<RecoverySetup />` (velg guardians) +
`<RecoveryRequest />` (ny enhet ber) + `<RecoveryApprove />` (guardian
godkjenner).
### Tester
- Unit: Shamir split/combine roundtrip; threshold-håndhevelse.
- Integration: full 3-of-5 recovery med 5 mock-guardians.
- Adversarial: 2 guardians koluderer (under threshold) → kan ikke
rekonstruere.
- Adversarial: ondsinnet ny enhet uten safety-number-bekreftelse → ingen
guardian skal frigjøre share.
### Dokumentasjon
- `docs/recovery.md` — full UX + threat model.
- Trusselmodell-utvidelse: kollusjon ≤ k-1, identitetsforfalskning, social
engineering.
---
## Akseptansekriterier
- [ ] 3-of-5 recovery fungerer end-to-end på 2 separate enheter.
- [ ] Ingen koalisjon av (k-1) guardians kan rekonstruere `shareSecret`
(verifisert med fast-check property test).
- [ ] Guardian-side widget krever fingerprint-bekreftelse før send (gate
fra V3.3 forsterket).
---
## Avhengigheter
- V3.2 — nøkkelmateriale at-rest hos guardian skal være kryptert.
- V3.3 — fingerprint-gate på recovery-handshake.
---
## Risiko
- **UX er det vanskeligste.** "Hvem er min guardian?" er sosialt komplekst;
bruker kan velge dårlig.
- **Social engineering.** Angriper imiterer offer over telefon → guardian
gir share. Mitiger med harde fingerprint-gates + cool-down.
- **Dead guardians.** Hvis guardian dør / mister sin enhet uten å være
erstattet, threshold synker. Periodisk "guardian health check"-prompt
anbefales.
---
## Migrasjon
Ny pakke. Apper kan legge til recovery-widget i innstillinger.

124
docs/archive/V3.11.md Normal file
View File

@@ -0,0 +1,124 @@
# Shade V3.11 — WebRTC P2P Transport
**Status:** Done — landet med `@shade/transport-webrtc` 0.4.0,
`MultiTransportFallback` i `@shade/transfer`, og
`shade.configureWebRTC()` i `@shade/sdk`. Se [docs/webrtc.md](../webrtc.md).
**Effort:** XL (24 måneder)
**Forrige:** V3.7
**Adresserer:** V2.1-tillegg "P2P WebRTC transport"
---
## Mål
Direct peer-to-peer datakanal mellom Shade-klienter når NAT/firewall
tillater. Primær gevinst: massiv throughput for `@shade/transfer`
(filer, store payloads) og lav-latens for messaging når begge peere
er online samtidig. E2EE bevart: WebRTC DTLS-SRTP er **transport**
payload er fortsatt Shade ratchet-krypto.
V3.11 lander i V4.0-vinduet og er foundation-only — sanntidsbruken
(voice, video, broadcast) ligger i [V5.0](../V5.0.md) som downstream
konsumer av denne datakanalen.
---
## Scope
### Inn
- Ny pakke `@shade/transport-webrtc`.
- Signaling via Shade control plane (eksisterende kanal — `Shade.send`).
- ICE/STUN: bruk offentlige STUN-servere som default.
- TURN: konfigurerbar TURN-relay som fallback.
- DataChannel for `@shade/transfer`-chunks.
- Auto-fallback: P2P → HTTP (eksisterende stack).
### Ut
- SFU/MCU (mange-til-mange topologi) — broadcast/video er V5.0.
- Voice/video media-tracks — V3.11 er ren datakanal (DataChannel);
audio/video over RTP er V5.0.
- DTLS-fingerprint-binding til Shade-fingerprint (vurderes som hardening,
men ikke krav).
---
## Design
### Connection-flow
```text
A initierer:
1. createOffer() → SDP
2. shade.send(B, { kind: "webrtc-offer", sdp })
3. B mottar over Shade-kanal, createAnswer()
4. shade.send(A, { kind: "webrtc-answer", sdp })
5. ICE-candidates exchange (samme kanal)
6. DataChannel åpen
```
### Wrapping
DataChannel sender ferdige `@shade/transfer`-chunks (allerede E2EE).
WebRTC's egen DTLS-SRTP fungerer som transport-secrecy lag.
### Topologi
- 1:1 P2P direkte når mulig.
- TURN-relay når NAT'er er for strenge (transport-only, ser ikke plaintext).
---
## Leveranser
### Pakker
- `@shade/transport-webrtc` — Connection, DataChannel-wrapper, ICE-config.
- `@shade/transfer` utvides: `WebRTCTransferTransport` som drop-in.
- `FallbackTransferTransport` får ny ledd: P2P → WS → HTTP.
### Tester
- Loopback unit: offer/answer/ICE i Bun via `node-datachannel` eller
`wrtc`.
- Integration: 100 MB transfer over P2P vs HTTP — P2P skal vinne på samme
nettverk.
- Failover: TURN-relay påtvinger relay-modus.
- NAT-emulering (loopback med ulike NAT-typer hvis mulig).
### Dokumentasjon
- `docs/webrtc.md` — setup, STUN/TURN-config, NAT-traversal-håp og
-realiteter.
---
## Akseptansekriterier
- [ ] To klienter på samme LAN: P2P direct uten TURN, throughput > 5x
HTTP-baseline.
- [ ] To klienter bak strenge NAT'er: TURN-relay aktiveres automatisk.
- [ ] Failover P2P-død → HTTP innen 5 s uten meldingstap.
---
## Avhengigheter
- V3.7 — bridge-mønstre + fallback-arkitektur.
---
## Risiko
- **NAT-traversal-helvete.** Mange edge-cases. Mitiger med tidlige
integration-tester på faktiske NAT-konfigurasjoner.
- **Browser-kompatibilitet.** Safari har sine egne RTC-quirks.
- **TURN-koster.** TURN-relay = ekte trafikk gjennom server. Operatør må
vite det.
---
## Migrasjon
Opt-in. Eksisterende HTTP/WS-transport fungerer uendret.

View File

@@ -0,0 +1,557 @@
# V3.12 — Key Transparency: Designnotat
**Status:** Done — implementert i `@shade/key-transparency` 0.4.0, frosset i 4.0 GA.
**Forfatter:** Shade-teamet
**Reviewer-mål:** ekstern crypto-orientert reviewer før produksjons-deploy.
**Implementasjons-target:** `@shade/key-transparency` + utvidelser i
`@shade/server`, `@shade/transport`, `@shade/sdk`.
---
## 1. Mål og ikke-mål
### Mål
Bytt ut "blind tillit til prekey-server" med en **verifiserbar
append-only log**. Når en klient mottar et prekey-bundle skal den ha
kryptografisk bevis for at:
1. Bundlen er **commit'et** i en tidstemplet log (Signed Tree Head).
2. Den eksakte (adresse, identityKey, signedPreKey)-mappingen står i
den loggen — _eller_ den står ikke (fravær-bevis).
3. Loggen har ikke skrevet om historie siden forrige fetch
(konsistens-bevis).
4. Andre klienter ser **samme** log (split-view-deteksjon via
witness-gossip).
Dette er **CT-style transparens** (RFC 6962-prinsipper) tilpasset
prekey-distribusjon.
### Ikke-mål (eksplisitt ut)
- **Federert log mellom flere prekey-servere.** Hver Shade-deployment
har én log (eller ingen). Multi-server gossip er V3.13+.
- **Løse MITM-på-første-kontakt fullstendig.** KT fanger split-view og
re-write, men ikke det at en angriper publiserer en forfalsket
identitet ved første registrering. Det er V3.3 (fingerprint-gate)
+ V3.10 (social recovery).
- **Legal/compliance audit-log.** Loggen er kryptografisk, ikke juridisk.
- **Klient-styrt sletting.** Append-only — DELETE skriver
tombstone-entry, fjerner ikke historikk.
### Beslutningskriterium for implementasjon
Når dette notatet er godkjent _og_ alle åpne spørsmål under §11 har
konkrete svar (ikke bare "vi finner ut av det senere"), kan kode
skrives. Det notatet ligger på når §11 lukkes er det vi bygger.
---
## 2. Trusselmodell-tillegg
Eksisterende THREAT-MODEL.md dekker prekey-server som "honest-but-curious"
+ tilstede TOFU. KT utvider modellen til **fully-malicious server**:
| Angrep | Pre-V3.12 | Post-V3.12 |
|---|---|---|
| Server returnerer feil bundle for én klient | Uoppdaget til OOB-verifisering | Klient kan be om proof; mismatch oppdages |
| Server bytter en allerede registrert identityKey | TOFU-fingerprint endres → V3.3-gate slår inn (men brukerinitiert) | Loggen vil vise to entries med samme adresse → witness oppdager |
| Server gir `alice` ulike identityKeys til Bob og Charlie (split-view) | Uoppdaget til OOB | Witness-gossip avslører to ulike STH-er |
| Server skriver om historikk for å skjule tidligere svik | Mulig | Konsistens-proof feiler → klient varsler |
| Server nekter å publisere ny STH | Mulig | "Stale STH"-detekteres av friskhetsbevis (max age) |
| Server kompromitterer signing-key for STH | KT-trygghet brutt | Witness gossip om gammel STH-kjede; rotasjon krever ny genesis |
KT løser **ikke**:
- Førstegangs-impersonering av en helt ny adresse (intet historisk
bevismateriale).
- Kollusjon mellom server og _alle_ witnesses.
- Klient som glemmer cached STH og må re-bootstrappe.
---
## 3. Datastruktur-valg
Vi velger **RFC 6962-stil append-only Merkle log** + **ekstern
adresse-index** med commitment-bevis. Begrunnelse:
### Vurderte alternativer
1. **Pure CT-log (RFC 6962):** Simple append-only Merkle tree.
Inklusjonsbevis trivielle. Fravær-bevis _ikke_ støttet
nativt (må scanne hele loggen).
2. **CONIKS-tre (sparse Merkle tree over adresser):** Native fravær-bevis,
men mye mer kompleks (epoch-baserte snapshots, prefix-trees,
placeholder-nodes). Overkill for første iterasjon.
3. **Hybrid (RFC 6962 log + side-index):** Loggen er sannhetskilde,
indexen er en _commitment_-mapping `address → leaf_index`. Server
beviser inklusjon via leaf-path, fravær via "denne adressen er ikke
i indexen ved tree_size T" + signert STH.
**Valg: alternativ 3.** Det gir CT-stil enkelthet, samt fravær-bevis
nesten gratis (commitment til indexen er en del av hver STH).
### Konkret format
#### Leaf
Hver leaf representerer én registrering eller revoke:
```
leaf = SHA256(
0x00 || // leaf prefix (RFC 6962)
uint64_be(timestamp_ms) ||
byte(operation) || // 0x01 register, 0x02 replenish, 0x03 delete
uint16_be(len(address)) || address_bytes ||
uint16_be(len(bundle_hash)) || bundle_hash // 32 bytes SHA-256 over canonical bundle
)
```
`bundle_hash` er deterministisk hash av:
```
canonical_bundle = SHA256(
0x01 || // bundle prefix
identitySigningKey (32) ||
identityDHKey (32) ||
uint32_be(signedPreKey.keyId) ||
signedPreKey.publicKey (32) ||
signedPreKey.signature (64)
)
```
One-time prekeys er **ikke** med i bundle-hashen — de er ephemerale og
ville lekket OTP-rotasjons-mønstre.
#### Tree
Merkle-tre over leaf-array, RFC 6962 §2.1:
- `MTH(empty) = SHA256()`
- `MTH({d}) = SHA256(0x00 || d)` (already hashed leaf)
- `MTH(D[n]) = SHA256(0x01 || MTH(D[0:k]) || MTH(D[k:n]))` der
`k` er største 2-potens < n.
#### Signed Tree Head (STH)
```
sth = {
tree_size: uint64,
timestamp: uint64_ms,
root_hash: bytes(32),
index_root: bytes(32), // commitment til adresse-index ved denne tree_size
log_id: bytes(32), // SHA-256 av server-public-key (stabil ID)
signature: bytes(64) // Ed25519 over canonical(rest)
}
```
`canonical(sth)` for signing:
```
0x02 || // sth prefix
uint64_be(tree_size) ||
uint64_be(timestamp) ||
root_hash (32) ||
index_root (32) ||
log_id (32)
```
#### Inklusjons-bevis
Standard RFC 6962 audit-path: liste med søsken-hasher fra leaf til root,
slik at klient re-beregner root og sammenligner med STH.
#### Konsistens-bevis
Standard RFC 6962 §2.1.2: bevis at tree med `tree_size = N1` er prefix
av tree med `tree_size = N2 > N1`. Klient bruker dette for å detektere
re-write.
#### Fravær-bevis
Adresse-indexen er en sortert liste `(address, leaf_index_of_latest)`
serialized og hashet. `index_root` i STH er commitment.
For å bevise fravær av adresse `addr` ved tree_size `N`:
- Server returnerer hele indexen ved tree_size `N` (sortert), eller
- (effektivt:) Returnerer naboparet `(addr_prev, addr_next)` der
`addr_prev < addr < addr_next` lexikografisk, sammen med en
Merkle-path i en sparse Merkle tree over indexen.
Første iterasjon: vi serialiserer hele indexen og lar klienten
laste den (kompakt: <100 KB selv for 100k adresser). Senere
optimaliserer vi til sparse Merkle tree hvis dataset vokser.
---
## 4. Friskhetsbevis (Signed Tree Heads)
### Frekvens
- **Min:** Ny STH ved hver mutasjon (register/replenish/delete) — synkront
i write-pathen.
- **Maks-stale:** Selv uten mutasjoner skal en STH publiseres minst hver
**10. minutt** ("heartbeat STH" — samme tree_size, oppdatert timestamp).
Dette gir klienter mulighet til å detektere "død" log uten å bekymre
seg om hvorvidt logen faktisk har endret seg.
### Klient-akseptansevindue
Klient avviser STH eldre enn `now - 24 timer` (default, konfigurerbar).
Dette beskytter mot replay av gamle STH-er som "skjuler" en mutasjon
gjort i ettertid.
### Stale-STH som soft-fail
Hvis STH er stale men gyldig signert: klient logger advarsel,
returnerer bundle med `proof.staleness = 'warn'` (V1) eller blokkerer
(V2 etter dogfooding). Vi starter med _warn_, eskalerer til _block_
når witness-økosystem er etablert.
---
## 5. Klient-verifikasjonssteg
På hver `fetchBundle(address)`:
1. Server returnerer `{ bundle, proof: { sth, leaf, audit_path, leaf_index, address_index_proof } }`.
2. Klient verifiserer:
- `sth.signature` mot kjent `log_public_key` (pinnet ved første
bootstrap).
- `sth.timestamp >= now - max_age_ms` (default 24t).
- Re-beregner `leaf_hash` fra bundle og sammenligner med `proof.leaf`.
- Re-beregner `root_hash` fra `audit_path + leaf` og sammenligner med
`sth.root_hash`.
- Verifiserer `address_index_proof` mot `sth.index_root`.
3. Hvis klient har en cached forrige STH: sjekk **konsistens-proof**
mellom forrige og denne. Server publiserer dette i
`GET /v1/kt/consistency?from=<size>&to=<size>`.
4. Hvis klient har en cached STH for samme `tree_size` med ulik
`root_hash`**split-view alarm**.
### Probabilistisk vs. obligatorisk verifisering
Vi velger **obligatorisk** ved hver bundle-fetch. Bundle-fetch er sjelden
(per ny peer, ikke per melding) — kostnaden er <100ms. Probabilistisk
verifisering ville la klienter bli lurt av "én dårlig fetch" uten
deteksjon.
### Bootstrap
Første gang en klient møter en log: pinner `log_public_key` etter å ha
hentet det fra et **ut-av-bånd**-pinningendepunkt eller fra `Shade.config`
(operatør sender den med klient-config). Etterfølgende rotasjon krever
ny genesis-STH med eksplisitt break-event signert av forrige nøkkel.
---
## 6. Witness/auditor-rolle
### Hva en witness gjør
- Periodisk poll: `GET /v1/kt/sth` (hent siste STH).
- Lagrer alle observerte STH-er i append-only lokal store.
- Eksponerer `GET /witness/sth?log_id=...&tree_size=...` slik at andre
klienter kan sammenligne hva _denne_ witnessen har sett.
- Verifiserer konsistens mellom hver ny STH og forrige.
### Klient-witness-gossip
Klient-bibliotek kan operere i tre moduser:
1. **Observe-only:** verifiserer kun bundle den selv henter, ingen
gossip.
2. **Light-witness:** poller STH hver `Xt` og lagrer lokalt; sammenligner
med STH levert ved bundle-fetch.
3. **Full-witness:** publiserer signerte STH-observasjoner til en
konfigurert peer-liste eller offentlig endpoint.
V1 leverer 1 og 2. Mode 3 (full-witness publication-protocol) er V2
hvis økosystem trenger det.
### Hvem kjører witnesses?
- Shade-prosjektet kjører **referanse-witness** på offentlig endpoint
(separate-from-prekey-server).
- Power-users / operatører kan kjøre egne via `@shade/key-transparency/witness`-
API.
- Tredjeparts auditors (typisk security-research) er invitert.
Vi krever **ikke** federation/konsensus mellom witnesses i V1 — gossip
er rent "har du sett samme STH som meg?".
---
## 7. Operatørkost
### Lagring
- **Per leaf:** 32 bytes (hash) + ~80 bytes adresse-index entry =
~112 bytes.
- **100k adresser, 1 rotasjon/år, 1 replenish/uke:** ~5.4M leaves =
~600 MB log. Tre-strukturen er beregnet on-demand, ikke lagret.
- **Index:** ~100k × 80B = 8 MB i minne (cacheable).
### CPU
- STH-signing: 1 Ed25519-signering per mutasjon + heartbeat = <1k/dag for
små deployments. Trivielt.
- Audit-path-beregning: O(log N) ved fetch. <1ms.
- Konsistens-proof: O(log N).
### Backup
Logen MÅ aldri miste data — sletting eller corruption ødelegger
integritet permanent. Strategi:
- Loggen lagres som append-only tabell `shade_kt_log` (PG) med
`(leaf_index, leaf_hash, leaf_data_json)`.
- Backup hver time + WAL-shipping anbefalt.
- Ved corruption: se §10 Recovery.
### STH-signing-key
- Genereres ved første KT-aktivering, lagres i operatør-styrt secret
(env, KMS, eller på disk for hjemme-server).
- Rotasjon: **breaking event** — krever ny genesis-STH der ny key
signerer melding "rotated from ${old_key}" med _gammel_ key. Klient
må eksplisitt akseptere rotasjonen.
---
## 8. Migrasjon
### Server-side
KT er **opt-in** på operatør-nivå. `createPrekeyServer({ keyTransparency:
{ enabled, store, signingKey } })`. Når slått på:
1. Server skriver alle eksisterende identiteter inn som genesis-leaves
ved boot.
2. Første STH publiseres med `tree_size = N` der N er antall
eksisterende adresser.
3. Klient som henter bundle får proof; klient som ikke støtter KT
ignorerer proof-felt (forward-compatible).
### Klient-side
`@shade/sdk`-config:
```ts
createShade({
keyTransparency: {
mode: 'observe' | 'light-witness' | 'off',
logPublicKey: '<base64>',
maxStaleMs: 86_400_000,
},
// ...
})
```
`mode: 'off'` (default for backward-compat første release) — ignorerer
proof. Ny SDK med `mode: 'observe'` verifiserer men feiler ikke harde
hvis proof mangler. `mode: 'observe-strict'` (senere) krever proof.
### Eksisterende deployments
Operatør kan rulle KT inn på live server uten klient-update:
1. Skru på KT i server-config → server begynner å produsere proofs.
2. Gamle klienter ignorerer proof-felt (de er additive i bundle-respons).
3. Nye klienter med `mode: 'observe'` begynner å verifisere.
4. Når operatør har testet og publisert log-public-key OOB, kan brukere
skifte til `'light-witness'`.
---
## 9. Akseptansekriterier
- [ ] `@shade/key-transparency` pakke leverer:
- Merkle log core (RFC 6962 hash-funksjoner).
- STH-signing/verifikasjon.
- Inklusjons-bevis generering + verifisering.
- Konsistens-bevis generering + verifisering.
- Adresse-index med commitment.
- Witness-light klient.
- Cross-platform (TS-only, ingen native deps).
- [ ] `@shade/server` integrasjon:
- `KTLogStore`-interface (memory + postgres).
- Routes: `GET /v1/kt/sth`, `GET /v1/kt/sth/:tree_size`,
`GET /v1/kt/consistency`, `GET /v1/kt/inclusion/:address`.
- Bundle-fetch returnerer `{ bundle, proof }` når KT aktivert.
- Heartbeat-STH-publisering hver 10. minutt (configurable).
- [ ] `@shade/transport` `ShadeFetchTransport`:
- Aksepterer optional `keyTransparency`-verifier.
- `fetchBundle()` returnerer `{ bundle, proof?: KTProof }`.
- [ ] `@shade/sdk` `Shade`:
- `keyTransparency`-config.
- Verifiserer proof ved hver bundle-fetch når aktivert.
- Cacher STH for split-view-deteksjon.
- [ ] **End-to-end test: split-view detection.**
- Test-server gir Bob bundle X, Charlie bundle Y for samme adresse `alice`.
- Bob+Charlie kjører som light-witness, gossiper STH-er.
- Test asserter at mismatch detekteres innen N polls.
- [ ] **End-to-end test: log re-write detection.**
- Server skriver om historie (test-only API).
- Konsistens-proof feiler på neste fetch.
- [ ] Operatør-doc dekker recovery-strategi.
- [ ] CHANGELOG, README, ROADMAP oppdatert.
- [ ] Cross-platform vector-test for Merkle hash + STH (Android/TS
paritet, samme som V3.5-tradisjonen).
---
## 10. Recovery
### Log corruption
Hvis log-data tapes (disk-feil før backup): **kan ikke gjenopprettes
uten å miste integritet** — det er hele poenget.
Recovery-prosedyre:
1. Operatør publiserer "log-restart" event signert med STH-keyen.
2. Genesis-STH genereres på nytt med ny `log_id` (= ny offentlig nøkkel
eller eksplisitt versjon).
3. Klienter som har cached STH-er fra gammel log varsles via
eksplisitt diskrepans i `log_id`.
4. Brukere som er bekymret må OOB-verifisere identiteter (V3.3-gate
trigges automatisk for fingerprint-rotasjon).
### Stale signing-key
Hvis STH-keyen lekkes: rotasjon krever break-event (§7). Inntil
brukerne aksepterer ny key, oppfører cient-bibliotek seg som om STH
mangler (soft-fail i `observe`-mode, blokkerer i `observe-strict`).
---
## 11. Åpne spørsmål (lukket før kode)
| Spørsmål | Svar |
|---|---|
| Hvordan distribueres `log_public_key` til klient første gang? | Operatør embedder i `Shade.config` ved app-init. OOB-pinning er fallback. |
| Skal one-time prekeys være med i bundle-hash? | Nei — ephemerale, og deres rotasjon ville støy-fylle loggen. |
| Konflikt: STH ved hver mutasjon vs. batched STH? | Per mutasjon. Heartbeat hver 10 min uansett. Batching vurderes som optimalisering hvis throughput blir et problem (ikke nå). |
| Hva skjer ved replenish (kun OTP-tilført)? | Skriver ikke til log (bundle-hash uendret). Heartbeat-STH dekker friskhet. |
| Hva med DELETE? | Skriver tombstone-leaf med `operation = 0x03`. Identiteten i indexen markeres som "deleted at tree_size N". |
| Sparse Merkle tree for index-proof? | Senere — V1 bruker hele indexen i fravær-proof. <100 KB ved 100k adresser er akseptabelt. |
| Klient-cache eviction-policy for STH? | LRU på `log_id`, last-N (default 100). Klient holder _alltid_ siste sett STH. |
| Witness-publication-protokoll? | V1 har poll-only (`GET /witness/sth`); push-publication er V2. |
Alle åpne spørsmål har konkrete svar. Implementasjon kan starte.
---
## 12. Pakke-struktur
```
packages/shade-key-transparency/
├── package.json # @shade/key-transparency, v0.4.0
├── src/
│ ├── index.ts # Public exports
│ ├── hashes.ts # RFC 6962 leaf/node hashing
│ ├── log.ts # MerkleLog (in-memory) + audit-path
│ ├── consistency.ts # Consistency-proof gen/verify
│ ├── sth.ts # STH sign / verify / canonical bytes
│ ├── index-tree.ts # Address index commitment
│ ├── proof.ts # KTProof type + bundle-proof verifier
│ ├── store.ts # KTLogStore interface (server-side)
│ ├── memory-store.ts # In-memory KTLogStore
│ ├── witness.ts # Light-witness client
│ └── errors.ts # KT-specific error types
└── tests/
├── hashes.test.ts
├── log.test.ts # RFC 6962 test vectors
├── consistency.test.ts
├── sth.test.ts
├── index-tree.test.ts
├── proof.test.ts
└── split-view.test.ts # End-to-end split-view detection
```
Server-integrasjon i `@shade/server`:
```
packages/shade-server/src/
├── kt-routes.ts # /v1/kt/* routes
├── kt-integration.ts # Hook bundle-fetch + register/delete to log
└── ...
```
Postgres-implementasjon i `@shade/storage-postgres`:
```
packages/shade-storage-postgres/src/
├── postgres-kt-store.ts # KTLogStore on PG
└── ...
```
Klient-integrasjon i `@shade/transport` + `@shade/sdk`:
```
packages/shade-transport/src/
├── kt-verifier.ts # Proof-verifier for fetchBundle
└── ...
packages/shade-sdk/src/
├── kt.ts # Shade.keyTransparency config + cache
└── ...
```
---
## 13. Test-strategi
1. **RFC 6962 test-vektorer:** importer kjente vektorer fra
<https://datatracker.ietf.org/doc/html/rfc6962#appendix-A>.
2. **Property-tests (fast-check):** for hver tree_size N og hvert
leaf-index i: `verify(audit_path(i, N), leaf, sth) === true`.
3. **Konsistens-bevis property-tests:** for N1 < N2:
`verify_consistency(proof, sth1, sth2) === true`.
4. **Split-view e2e:** to klienter, ondsinnet test-server, witness
gossip oppdager mismatch.
5. **Re-write-detection e2e:** server muterer log-historie, klient
neste fetch får konsistens-proof som feiler.
6. **Cross-platform:** Android (Kotlin) + TS gir samme leaf-hash for
samme bundle (V3.5-paritet er forutsetning, så dette må også gå
gjennom kotlin-port; for V3.12 første release dekker vi TS — Android
port er V3.13).
7. **Stale STH:** klient avviser STH > max_age.
8. **Bootstrap-pinning:** klient feiler hvis log_public_key ikke matcher.
---
## 14. Sikkerhetsvurdering
- **Falsk trygghet hvis halvveis:** Avhjelpes ved at default-mode er `'off'`,
bare _eksplisitt_ aktivert KT gir hardere garantier. Dokumentasjon
fremhever at `'observe'` er observasjon, ikke obstruksjon, til
økosystemet er etablert.
- **Server-side mutability av historie:** Avhjelpes ved at `KTLogStore`
kun har `append()` — ingen `update()`/`delete()` på historiske leaves.
PG-tabellen har CHECK constraint og BEFORE-triggers for ekstra defense
in depth (se §7).
- **STH-key compromise:** dokumentert §10. Operatør-ansvar.
- **DoS via massive index-proofs:** index-proof er i V1 hele indexen.
100 KB per fetch er overkommelig; rate-limiteren dekker excess.
- **Replay av gammel proof:** STH-timestamp + max_age beskytter.
---
## 15. Approval
Når dette notatet er reviewed (in-tree review er nok for å kommitte
første implementasjon; ekstern crypto-review er pre-deploy-krav per
V3.12 §"Pre-requisite designnotat"), kan implementasjon starte.
**Implementasjon-rekkefølge** (alle commits i samme branch):
1. `@shade/key-transparency` core (Merkle log, STH, proofs).
2. Server-integrasjon (`@shade/server` + memory/postgres KTLogStore).
3. Klient-integrasjon (`@shade/transport` verifier + `@shade/sdk` config).
4. Witness-light + e2e split-view-test.
5. Operatør-doc + CHANGELOG + README + ROADMAP.
— end of design —

99
docs/archive/V3.12.md Normal file
View File

@@ -0,0 +1,99 @@
# Shade V3.12 — Key Transparency
**Status:** Done (0.4.0). Designnotat: `docs/V3.12-DESIGN.md`.
Operatør-/recovery-guide: `docs/key-transparency.md`.
**Effort:** XXL (4+ måneder, multi-quarter)
**Forrige:** V3.5 (hovedplattformene stabile først)
**Adresserer:** V2.3 §1A
---
## Mål
Reduser tillit til prekey-server fra "blind tillit" til "verifiserbar log".
Når serveren utleverer et bundle, skal det være kryptografisk forpliktet i
en **append-only log** som klienter (eller tredjeparts-auditors) kan
verifisere. Et split-view-angrep der serveren viser ulike bundles til ulike
klienter blir fanget av gossip.
---
## Pre-requisite: designnotat
**Ingen kode før dette er review'd og approved:**
- Trusselmodell-tillegg: hva CT/attest faktisk løser, hva som forblir åpent.
- Datastruktur-valg: append-only Merkle log (CT-stil), CONIKS-tre, eller
hybrid.
- Friskhetsbevis: hvor ofte signed tree heads utgis; hva er "stale"?
- Klient-verifikasjonssteg: må klient verifisere på hver bundle-fetch,
eller probabilistisk?
- Witness/auditor-rolle: hvem kjører dem? Hvordan gossip mellom klienter?
- Operatørkost: log-størrelse, signing-frekvens, backup-strategi.
- Migrasjon: eksisterende prekey-server → log-utvidet.
Designnotatet er en `docs/V3.12-DESIGN.md`-PR som må review'es av minst én
ekstern crypto-orientert reviewer.
---
## Mulig scope (etter designnotat)
### Inn (estimat)
- Append-only log som tillegg til prekey-server.
- Inklusjons-bevis ved bundle-fetch (Merkle-path).
- Fravær-bevis for "denne adressen har ikke registrert siden timestamp T".
- Signed tree heads (STH) publisert på fast interval.
- Klient-bibliotek: `@shade/key-transparency` med verifisering.
- Witness-API: tredjeparts-auditor kan hente STH-er og logge gossip.
### Ut (eksplisitt)
- Federated log (multi-server gossip) — for stort for første iterasjon.
- Legal/compliance-side av audit-log.
- "Vi løser MITM-på-første-kontakt-helt" — KT alene fanger split-view, ikke
første-kontakt.
---
## Risiko-vurdering
KT er det **vanskeligste enkeltpunkt** i hele roadmapen:
1. **Halvveis-implementert KT er verre enn ingen KT** — gir falsk trygghet,
brukere slutter å verifisere OOB.
2. Operativt komplekst — log må aldri skrive om historie. En enkelt
restart-bug = ødelagt integritet.
3. Klient-verifikasjons-logikk må kjøre på hver bundle-fetch, eller
risikere at én "gammel" klient blir lurt.
4. Witness-økosystem krever uavhengige aktører — Shade alene kan ikke
garantere det.
**Beslutningskriterium:** Hvis designnotatet etterlater åpne "hvordan
håndterer vi X?"-spørsmål uten klare svar, parker V3.12. Pragmatisk
alternativ er **V3.3 (fingerprint-gate)** + **V3.10 (social recovery)**
som sammen gir 80 % av MITM-beskyttelsen uten KT-kompleksiteten.
---
## Akseptansekriterier (hvis det implementeres)
- [ ] Designnotat passert ekstern review.
- [ ] Klient detekterer split-view i ende-til-ende-test (server gir to
versjoner av samme adresse → klient fanger mismatch).
- [ ] Witness-API testet med minst én ekstern auditor-instans.
- [ ] Operatør-doc dekker recovery hvis log korrumperer.
---
## Avhengigheter
- V3.5 — Android/TS paritet må være solid før vi legger på et nytt
verifikasjons-lag.
---
## Migrasjon
Helt opt-in. Operatører som ikke ønsker KT kjører videre uendret.

146
docs/archive/V3.2.md Normal file
View File

@@ -0,0 +1,146 @@
# Shade V3.2 — At-Rest Storage Encryption
**Status:** Implementert (0.4.0) — `@shade/storage-encrypted`, `@shade/keychain`,
`shade migrate-storage`, `shade rotate-storage-key`
**Effort:** L (48 uker)
**Forrige:** V3.1
**Adresserer:** V2.1 §2
---
## Mål
Opt-in beskyttelse av sensitiv state — identity-nøkler, session-state, valgfri
stream-resume-secret — med nøkler som **ikke** ligger i klartekst i databasen.
Trusselmodellen sier i dag eksplisitt at en stjålet DB eksponerer private
nøkler; dette løser det for deploys som velger å aktivere det.
---
## Scope
### Inn
- Ny `EncryptedStorageProvider`-wrapper som dekorerer `SQLiteStorage` /
`PostgresStorage`.
- Per-rad AES-256-GCM på sensitive felter (`identity_*`, `session_*`,
valgfritt `stream_state.streamSecret`).
- KDF-pluggin (default `scrypt` fra `@noble/hashes`) for passphrase-basert
master-nøkkel.
- Tre nøkkelkilder ut av boksen:
1. **Passphrase + KDF** — utvikler oppgir secret ved oppstart.
2. **OS keychain** — macOS Keychain, Linux libsecret, Windows Credential
Vault (Node-only).
3. **App-injected key** — appens egen kode forsyner 32-byte nøkkel (mest
fleksibel).
- Migrasjons-CLI: `shade migrate-storage --encrypt --key-source=...`.
- Trusselmodell-oppdatering: "når enabled, hva er fortsatt udekket" — memory
compromise, swap, runtime-tap.
### Ut
- Browser/IndexedDB at-rest (egen pakke, vurderes etter V3.8).
- HSM/Secure Enclave (separate driver senere).
- "Always-on by default" — vi flyger opt-in for å ikke bryte eksisterende
deploys.
---
## Design
### Krypteringsenhet
- Per-rad AEAD: `nonce(12) || ciphertext || tag(16)`.
- `nonce = HKDF(rowKey, "shade-row-nonce-v1" || tableName || pk)[..12]`
deterministisk per (tabell, pk) for å unngå nonce-reuse uten å lagre nonce
separat. Endring av (tabell, pk) → re-encryption.
- AAD binder `tableName || columnName || pk` så feltombytting blokkeres.
### Nøkkelhierarki
```text
masterKey (fra kilde — passphrase / keychain / app-injected)
├─ HKDF("shade-storage-v1") → storageKey (32 bytes)
│ │
│ └─ HKDF(storageKey, table || column) → fieldKey
└─ HKDF("shade-storage-version-v1") → versjonsnøkkel (rotasjon)
```
### Migrasjon
1. CLI leser ukryptert DB.
2. Skriver rad-for-rad-kryptering til ny `_v2`-tabell.
3. Atomisk rename + drop gammel.
4. Backup `.bak`-fil etterlatt i samme dir.
### Rotasjon
- `shade rotate-storage-key --new-source=...` re-krypterer med ny masterKey.
- Online ratchet (les med gammel, skriv med ny) for store DB.
---
## Leveranser
### Pakker
- Ny modul: `@shade/storage-encrypted` (re-export over SQLite/PG).
- Utvidelse i `@shade/cli`: `migrate-storage`, `rotate-storage-key`.
- Hjelpe-pakke: `@shade/keychain` (Node-only, valgfri peer-dep) for OS-keychain.
### Tester
- Unit: KDF-derivasjon, nonce-determinisme, AAD-binding.
- Integration: full lifecycle på SQLite + PG; start/stopp; krasj under
migrasjon.
- Tamper: bit-flip i ciphertext / AAD / nonce → dekrypterings-feil.
- Vector-fil: kryss-sjekk masterKey → fieldKey-derivasjon mot
`test-vectors/storage-encryption.json`.
### Dokumentasjon
- `docs/storage-encryption.md` — full guide.
- `THREAT-MODEL.md` — ny kolonne "with at-rest enabled".
- Migrasjonsnotat i `MIGRATION.md`.
---
## Akseptansekriterier
- [ ] Eksisterende ukryptert deploy fortsetter uten endringer (opt-in).
- [ ] `shade migrate-storage --encrypt` migrerer en levende SQLite uten
datatap, verifisert med dump-diff.
- [ ] Rotasjon kan gjøres uten downtime > 5 s for små DB.
- [ ] Wrong passphrase / wrong key → klar feilmelding, ikke krasj.
- [ ] Test-vectors deles med Android-implementasjonen (V3.5 forplikter at
vector-filen kjøres der).
---
## Avhengigheter
- V3.1 — `THREAT-MODEL.md` skal være lenket til testene først, så vi kan
utvide tabellen.
---
## Risiko
**Datatap.** En migrasjon som krasjer halvveis kan etterlate korrupt DB.
Mitigeres ved:
- Atomic-rename + `.bak`-fil.
- Dry-run-modus (`--dry-run` validerer all dekryptering før skriving).
- Refuser å starte hvis WAL har uncommitted writes.
**Nøkkeltap = totaltap.** Hvis bruker mister passphrase = ingen tilgang.
Dokumenter klart, og pek på V3.10 (Social Recovery) som langtidsløsning.
---
## Migrasjon
0.3.x deploys er ukrypterte → fortsatt ukrypterte. Aktivering er én
CLI-kommando. Backwards-kompatibel.

147
docs/archive/V3.3.md Normal file
View File

@@ -0,0 +1,147 @@
# Shade V3.3 — Fingerprint Gates & Trust UX
**Status:** Done
**Effort:** M (24 uker)
**Forrige:** V3.1
**Adresserer:** V2.3 §1B
**Implementert:** se `docs/trust-ux.md`
---
## Mål
Gjør safety numbers **handlingspålagte** — ikke bare synlige — i flyt der
MITM-risikoen er reell. I dag finnes `FingerprintCompare`-widget og
`requireFingerprintVerifiedFor` i `@shade/files`, men hovedkjernen
(`Shade.send`, first-large-file, backup-import) har ingen automatisk gate.
Resultat: alert-fatigue-fri, men også gate-fri.
Dette legger inn **eksplisitt blokkerende verifisering** på et lite antall
kritiske hendelser, plus widget-støtte for å eksponere det i UI.
---
## Scope
### Inn — kritiske hendelser
1. **Før første store fil**`Shade.upload` over en bytes-terskel uten
verifisert peer.
2. **Før backup-import**`Shade.importBackup` blokkerer til peer (eller egen
identitet) er bekreftet.
3. **Ny enhet med rotert identitet**`acceptIdentityChange` blokkerer på
første bruk inntil verifisert.
4. **Før `@shade/inbox` fan-out** (V3.6) — gate per mottaker.
### Inn — APIer
- `Shade.beforeFirstLargeFile(threshold, handler)` — appen får mulighet til å
vise modal og returnere bekreftelse.
- `Shade.beforeBackupImport(handler)` — samme mønster.
- `Shade.beforeNewDeviceTrust(handler)` — ditto.
- `Shade.markPeerVerified(address)` / `Shade.isPeerVerified(address)`
persistent state.
### Inn — widgets
- `<FingerprintGate />` — render-prop wrapper som blokkerer barn til
verifisert.
- `<FingerprintCompare />` utvides med "kopier OOB-tekst" + "jeg har
verifisert".
### Ut
- "Tving alle peers verifisert før hver melding" — alert fatigue.
- Cross-device sync av verified-state (kommer evt. via V3.6 inbox).
---
## Design
### Persistent verified-state
Ny tabell `peer_verifications`:
```sql
CREATE TABLE peer_verifications (
peer_address TEXT PRIMARY KEY,
fingerprint TEXT NOT NULL,
verified_at INTEGER NOT NULL,
verified_by TEXT, -- "user" | "transitive" | "tofu-after-warning"
identity_version INTEGER NOT NULL -- knytter verifikasjon til identity-rotasjon
);
```
Når peer roterer identitet → `identity_version` bumper → verifikasjon "ugyldig"
til ny verifisering.
### Hook-flyt
```text
shade.upload(peer, file)
├─ if !verified(peer) AND file.size > threshold
│ │
│ └─ await beforeFirstLargeFileHandler(peer, fingerprint)
│ ├─ true → markPeerVerified(peer); proceed
│ └─ false → throw FingerprintNotVerifiedError
└─ proceed
```
---
## Leveranser
### Kode
- `@shade/core``peer_verifications`-tabell + storage methods.
- `@shade/sdk` — gate-hooks + `markPeerVerified` / `isPeerVerified`.
- `@shade/widgets``<FingerprintGate />`, utvidet `<FingerprintCompare />`.
### Tester
- Unit: gate kalles, ikke kalles, retur false → throw, retur true → proceed.
- Integration: fil < threshold går gjennom uten gate; fil > threshold
blokkerer.
- Identity-rotasjon ugyldiggjør verifikasjon.
- Backup-import blokkerer.
### Dokumentasjon
- `docs/trust-ux.md` — guide til hvilke gates som finnes og når de bør tunes.
---
## Akseptansekriterier
- [ ] Gate kan ikke bypasses ved å nulle `threshold` ut — minimum gate finnes
alltid for backup-import og new-device.
- [ ] App uten registrerte gates får sane defaults (logger en warning, men
kjører — ikke krasj).
- [ ] Identity-rotasjon resetter verifikasjon i en testet ende-til-ende-flow.
- [ ] Widget kan rendres SSR uten å trigge runtime-gate.
---
## Avhengigheter
- V3.1 — threat-matrise oppdatert til å vise hvilke gates som dekker hvilke
rader.
---
## Risiko
- **Alert fatigue.** Hvis terskler er for lave → bruker klikker blindt.
Mitiger ved å sette default-terskler høyt (10 MiB for first-large-file)
og dokumenter justerings-guide.
- **DX-friksjon.** Apper som ikke vet om gates får uventede prompts. Mitiger
ved å logge tydelig ved første aktivering: "Shade.beforeFirstLargeFile not
configured — using default modal".
---
## Migrasjon
0.3.x apps får defaults aktivert med warning. Ingen breaking change.

124
docs/archive/V3.4.md Normal file
View File

@@ -0,0 +1,124 @@
# Shade V3.4 — Observability v2 (OpenTelemetry)
**Status:** Implementert (2026-05-02) — `@shade/observability` 0.1.0,
hekt inn i sdk/transfer/server/files/core. Off by default; flip
`SHADE_OTEL_ENABLED=1` for å aktivere.
**Effort:** M (24 uker)
**Forrige:** V3.1
**Adresserer:** V2.3 §4
---
## Mål
Gi produksjonsteam **distribuerte spor** rundt `TransferEngine`,
prekey-routes og `@shade/files` — uten å lekke plaintext-adresser, payloads
eller eksakte chunk-størrelser. Bygger videre på Prometheus-metrics som
allerede finnes.
---
## Scope
### Inn
- Opt-in OpenTelemetry-instrumentasjon via `@opentelemetry/api`.
- Spans rundt:
- `TransferEngine.upload` / `.download` (med lane-tags, retry-counts).
- `ShadeSessionManager.encrypt` / `.decrypt` (per-peer mutex-akkvisisjon,
ratchet-step).
- `createPrekeyRoutes` (per route, status-koder).
- `@shade/files` op-handlers (har allerede `onMetric` — utvides til OTel).
- PII-policy-doc: hva som **aldri** logges, hva binnes, hva pseudonymiseres.
- Sample-policy default off; on med `SHADE_OTEL_ENABLED=1`.
### Ut
- Trace-eksport til SaaS-leverandører (det er deploy-konfig, ikke vår kode).
- Logg-aggregering — `@shade/server` har allerede strukturert JSON.
---
## Design
### Span-attributter
| Attribute | Verdi |
|-----------|-------|
| `shade.peer.hash` | `sha256(address).slice(0, 8)` — stabil pseudonym |
| `shade.bytes.bin` | binnet — `"≤4KB"`, `"464KB"`, `"64KB1MB"`, `"≥1MB"` |
| `shade.lane.count` | 1 / 4 / 16 |
| `shade.retry.count` | int |
| `shade.error.code` | `SHADE_*`-kode |
**Aldri:** `shade.peer.address`, `shade.payload`, `shade.bytes.exact`.
### API
```ts
import { withTracer } from '@shade/observability';
const shade = await createShade({
...,
observability: withTracer(myTracer, { sample: 0.1 }),
});
```
`withTracer()` er no-op hvis `tracer` er `undefined` eller
`SHADE_OTEL_ENABLED` ikke er satt.
---
## Leveranser
### Pakker
- Ny submodul `@shade/observability` (peer-dep `@opentelemetry/api`).
- Hooks i `@shade/sdk`, `@shade/transfer`, `@shade/server`, `@shade/files`.
### Tester
- Span emitteres med riktige attributter (mock tracer).
- Sample-rate respekteres.
- Off-by-default verifisert.
- Regex-grep mot recorder fanger plaintext-PII.
### Dokumentasjon
- `docs/observability.md` — setup + PII-policy.
- `docs/DEPLOYMENT.md` — environment-variabler.
---
## Akseptansekriterier
- [x] Default deploy uten OTel: ingen performance-regresjon (`withTracer`
returnerer delt `NOOP_HOOK` når `SHADE_OTEL_ENABLED` ikke er satt).
- [x] Med OTel på: spans for upload/download (`shade.transfer.upload`,
`shade.transfer.download`), prekey-routes (`shade.prekey.request`),
session encrypt/decrypt (`shade.session.{encrypt,decrypt}`), og
`@shade/files` ops (`shade.files.op`).
- [x] Automatisert grep-test fanger plaintext-PII i spans
(`packages/shade-observability/tests/integration-pii.test.ts` +
`packages/shade-transfer/tests/observability.test.ts`,
`safeAttribute()` blokkerer fra-utvikler-introduksert PII).
---
## Avhengigheter
- V3.1 — basis-docs.
---
## Risiko
- **Performance-overhead.** Mitiger ved aggressiv default-off + sampling.
- **PII-lekkasje** hvis utviklere legger til egne attributter. Mitiger ved
å publisere "safe attribute"-helpers og PII-linter.
---
## Migrasjon
Ingen — opt-in.

125
docs/archive/V3.5.md Normal file
View File

@@ -0,0 +1,125 @@
# Shade V3.5 — Android Parity & Cross-Platform CI
**Status:** Done (kryptografisk lag + CI-gate). Android-KeystoreStorage og scrypt/argon2id-paritet er post-GA-arbeid sporet i `android/shade-android/ROADMAP-ANDROID.md` — ikke en 4.0 GA-blocker.
**Effort:** XL (24 måneder, parallelliserbar)
**Forrige:** V3.1
**Adresserer:** V2.1 §3
---
## Mål
Gjør Kotlin-implementasjonen **byte-kompatibel** med TS-implementasjonen, og
forsegle paritet via **CI-gate** som kjører delte test-vectors i begge språk.
Ingen "production"-label på Android før ratchet + proto + streams 0x11 er
grønne.
---
## Scope
### Inn — paritet-sjekkpunkter (eksplisitt)
1. **KDF-chain** — root key + chain key derivasjoner.
Vector: `test-vectors/kdf-chain.json`.
2. **HKDF** — labels for `info`-felt.
Vector: `test-vectors/hkdf.json`.
3. **X3DH** — full agreement med samme bundles.
Vector: `test-vectors/x3dh.json`.
4. **Ratchet message** — encrypt/decrypt roundtrip (legg til vector).
5. **Fingerprint** — 60-digit safety number.
Vector: `test-vectors/fingerprint.json`.
6. **Wire format 0x02** — encode/decode.
Vector: `test-vectors/wire-format.json`.
7. **Streams 0x11** — multi-lane chunk encryption (M-Cross 3, ikke i M-Cross 1).
8. **Backup-format** — passphrase-basert KDF + AES-GCM payload.
### Inn — milestoner
- **M-Cross 1 ✅** — keys + HKDF + X3DH + fingerprint.
- **M-Cross 2 ✅** — ratchet step (encrypt + decrypt roundtrip) + wire 0x02
(RatchetMessage + PreKeyMessage med/uten OTPK). Vector-versjon `2`.
- **M-Cross 3 ✅** — streams 0x11 (KDF, deterministic chunk nonce/AAD, wire 0x11
encode/decode). End-to-end socket interop pending; ikke gating-blokker.
- **M-Cross 4 ✅** — backup-format HKDF + AEAD, gruppe sender-keys
(kdfChainKey + Ed25519 sign(aad ‖ ct)), storage-HKDF (storageKey,
fieldKey, rowNonce). Gjenstående: scrypt master-key (Bouncy Castle),
argon2id-bytte, Android-KeystoreStorage som søsken-modul.
### Inn — CI
- Gitea Actions matrix-job:
- Bun-runner kjører `bun test:vectors` mot `test-vectors/*.json`.
- Gradle-runner kjører `./gradlew vectorTests` mot samme filer.
- PR-gate: begge må passere.
- Vector-genereringsskript (`scripts/generate-vectors.ts`) finnes — utvid
til 7 + 8.
### Ut
- iOS — egen Swift-port er framtidig roadmap, ikke V3.5.
- Native bindings i `shade-android` (vi bruker Tink i JVM-kode).
---
## Leveranser
### Kotlin
- Full ratchet-implementasjon (M-Cross 2).
- Wire 0x02 encode/decode.
- Streams 0x11 (M-Cross 3).
- Tink-storage-adapter med Keystore.
### Test-vectors
- Utvid `scripts/generate-vectors.ts` med ratchet-step + streams + backup.
- Versjons-tag på vector-filer (`{ "version": 2, ... }`).
### CI
- `.gitea/workflows/cross-vectors.yml` — Bun + Gradle matrise.
- Fail-policy: hvis vector-fil endres, **begge** runners må publisere
passing før merge.
### Dokumentasjon
- `android/shade-android/ROADMAP-ANDROID.md` — eksplisitte milestoner +
status per sjekkpunkt.
- `docs/cross-platform.md` — hvordan legge til en ny vector + hvordan
kjøre lokalt.
---
## Akseptansekriterier
- [ ] M-Cross 2: TS-encrypted melding kan dekrypteres av Kotlin-klient og
omvendt, end-to-end-test.
- [ ] CI-jobben feiler innen 60 s ved bevisst byte-divergens.
- [ ] M-Cross 3: 1 MiB streams-fil over 4 lanes mellom TS-server og
Kotlin-klient verifisert.
- [ ] Ingen public release med "production"-label før M-Cross 2 er grønn.
---
## Avhengigheter
- V3.1 — `cross-platform.md` lever der.
---
## Risiko
- **Tink-mismatch.** Tink HKDF-info-encoding kan avvike fra
`@noble/hashes`. Mitiger med tidlig vector-test (M-Cross 1 dekker dette).
- **Endian / encoding.** Wire 0x02 bruker big-endian — Kotlin
`ByteBuffer` default er big-endian, men streams-nonce-konstruksjon må
gjennomgås.
- **Maintainer-kapasitet.** Kotlin-port + TS-port må holdes i sync.
Vector-CI er primær mitigasjon.
---
## Migrasjon
Eksisterende M-Cross 1 scaffold beholdes; alt nytt bygges på den.

123
docs/archive/V3.6.md Normal file
View File

@@ -0,0 +1,123 @@
# Shade V3.6 — Async Store-and-Forward (Inbox)
**Status:** Done
**Effort:** L (48 uker)
**Forrige:** V3.4
**Adresserer:** V2.2 §2
**Implementert:** se `docs/inbox.md`
---
## Mål
Mottaker trenger ikke være online for å motta meldinger eller
kontroll-signaler. En **dedikert relay/inbox-tjeneste** holder
**ciphertext-blobs** med TTL og auth. Server ser aldri plaintext;
prekey-server forblir public-keys-only.
---
## Scope
### Inn
- Ny pakke: `@shade/inbox` (klient) + `@shade/inbox-server` (server).
- HTTP API:
- `POST /v1/inbox/:address` — signed PUT av blob (med TTL).
- `GET /v1/inbox/:address/since/:cursor` — auth'd fetch.
- `DELETE /v1/inbox/:address/:msgId` — leasing/ack.
- Replay-beskyttelse på applikasjonslag (`msgId = sha256(ciphertext)`).
- Push-hook (vendor-nøytral): `inbox.onMessageQueued(handler)`-callback.
- Outgoing queue i klient: lagrer ciphertext lokalt til server bekrefter
PUT.
- Idempotent PUT (samme `msgId` returnerer 200, ikke 409).
### Ut
- Mobile push (FCM / APNs) — utenfor scope; vi eksponerer hook'en.
- Federation mellom inbox-servere — egen sak senere.
- Plaintext-metadata-adresser — vi støtter pseudonyme address-hashes som
privacy-modus.
---
## Design
### Auth
- PUT er **signed** med avsenders Ed25519 (samme som prekey).
- GET krever signed challenge fra mottaker (pull, ikke push).
- Replay-window ±5 min, samme som prekey.
### Wire
- Eksisterende `@shade/proto`-envelope, transportert som body.
- Server lagrer **kun**:
`address || msgId || ciphertext-bytes || expires_at`.
### Lifecycle
1. Avsender encrypter via `Shade.send` → får envelope.
2. Avsender PUT'er envelope til mottaker-inbox med TTL (default 7 dager).
3. Mottaker poller (eller får push-trigger) — fetcher alle siden cursor.
4. Mottaker decrypter; ack'er via DELETE for tidlig prune.
### Storage
- SQLite + Postgres backends (samme mønster som prekey).
- Indeks: `(address, expires_at)`.
- Cron prune.
---
## Leveranser
### Pakker
- `@shade/inbox` — klient + queue.
- `@shade/inbox-server` — Hono routes + storage adapter.
### Tester
- Unit: signed PUT/GET, replay-window, idempotency.
- Integration: full lifecycle 100 msgs, restart server, msgs persisterer.
- Tamper: bit-flip ciphertext → klient-side decrypt feiler (server vet
ikke).
### Dokumentasjon
- `docs/inbox.md` — setup, threat model "what the relay sees", deploy-guide.
- `THREAT-MODEL.md` — ny seksjon om relay.
---
## Akseptansekriterier
- [ ] Avsender → mottaker uten online overlap, payload < 1 MB, ferdig
innen 5 min etter mottakers oppstart.
- [ ] Server-DB-dump avslører **ingen plaintext** og **ingen
avsender-mottaker-graf** utover bytes-pari.
- [ ] Replay av PUT med samme `msgId` returnerer 200 uten å lagre dobbel.
---
## Avhengigheter
- V3.4 — observability hooks for å måle inbox-bruk uten lekkasje.
---
## Risiko
- **Metadata-lekkasje.** Server ser hvem snakker med hvem. Dokumenter klart;
pek på adress-hash som mitigasjon.
- **Storage-DoS.** Ondsinnet avsender fyller mottakers inbox. Mitiger med
per-sender quota + per-address-quota.
- **Privacy-modell.** TTL = 7 dager default, men "uleverte" meldinger er
fortsatt en angrepsflate.
---
## Migrasjon
Ny pakke; ingen breaking change i eksisterende.

127
docs/archive/V3.7.md Normal file
View File

@@ -0,0 +1,127 @@
# Shade V3.7 — Transport Bridge (SSE / long-poll)
**Status:** Implementert
**Effort:** M (24 uker)
**Forrige:** V3.6
**Adresserer:** V2.3 §3
**Leveranse:** `@shade/transport-bridge` 0.1.0 + `createBridgeRoutes` i
`@shade/inbox-server`. Brukerveiledning: [`docs/transport.md`](../transport.md).
---
## Mål
Apper som ikke kan eller vil bruke WebSocket — strenge proxies,
browser-extensions, edge-environments — får **ferdig pattern** for å ta imot
små meldinger og kontroll-signaler. SSE som primær fallback, long-poll som
sekundær.
---
## Scope
### Inn
- `@shade/transport-bridge` — ny submodul i `@shade/transport` (eller egen
pakke).
- SSE-endpoint i `@shade/server` (kombineres med inbox fra V3.6 for "hent
fra inbox uten plaintext").
- Long-poll fallback med konfigurerbar timeout.
- Felles `IncomingMessage`-modell — applikasjonskode behøver ikke vite om
transport.
- Auto-fallback: WS → SSE → long-poll (samme mønster som transfer-transport).
### Ut
- HTTP/2 push.
- WebTransport — browser-støtte fortsatt umoden i 2026.
---
## Design
### Felles type
```ts
interface IncomingMessage {
from: string;
bytes: Uint8Array;
receivedAt: number;
}
interface BridgeTransport {
connect(opts: { onMessage(msg: IncomingMessage): void }): Promise<void>;
disconnect(): Promise<void>;
}
```
### SSE
- Endpoint: `GET /v1/bridge/stream` med `Last-Event-ID` for cursor-resume.
- Server-side: emitterer `envelope-ready`-event når inbox får ny.
- Klient åpner én EventSource; reconnect på drop.
### Long-poll
- Endpoint: `GET /v1/bridge/poll?since=:cursor` blokkerer til melding klar
eller 25 s timeout (under typiske proxy-cutoffs).
- Klient repeterer.
### Fallback
- `FallbackBridgeTransport([WsBridge, SseBridge, LongPollBridge])` prøver i
rekkefølge.
---
## Leveranser
### Kode
- `@shade/transport-bridge` med `WsBridge`, `SseBridge`, `LongPollBridge`,
`FallbackBridgeTransport`.
- Server: SSE og long-poll routes på `@shade/server` eller
`@shade/inbox-server`.
### Tester
- Unit: hver bridge åpner/lukker korrekt; reconnect på drop.
- Integration: WS down → faller til SSE; SSE 502 → long-poll.
- Same `IncomingMessage` shape ut fra alle tre.
### Dokumentasjon
- `docs/transport.md` utvidet med bridge-oversikt.
---
## Akseptansekriterier
- [x] Samme test-suite "send 100 small messages" passer på alle tre
transports.
- [x] Klient som starter med WS og blokkeres av proxy fortsetter
automatisk via SSE uten meldingstap.
- [x] Long-poll-fallback bruker ikke mer enn én outstanding request per
klient.
---
## Avhengigheter
- V3.6 — naturlig komplement; SSE-payload er typisk "envelope er klar i
inbox".
---
## Risiko
- **Reconnect-cykluser.** SSE som flapper kan tape meldinger. Mitiger med
Last-Event-ID + at server beholder kort buffer.
- **Long-poll keepalive.** Proxy-timeouts kan kutte før 30 s; juster
default til 25 s.
---
## Migrasjon
Additivt.

117
docs/archive/V3.8.md Normal file
View File

@@ -0,0 +1,117 @@
# Shade V3.8 — Web Workers Crypto
**Status:** Done
**Effort:** M-L (36 uker)
**Forrige:** V3.1
**Adresserer:** V2.2 §4
**Levert:** `0.4.0`
**Konsumentdokumentasjon:** [`docs/web-workers.md`](../web-workers.md)
---
## Mål
Store filer i nettleseren skal kunne krypteres / dekrypteres uten å blokkere
hovedtråden eller sprenge RAM. Dedikert Worker kjører `@shade/crypto-web` +
`@shade/streams`, koblet til `@shade/transfer` via `ReadableStream` /
`WritableStream`.
---
## Scope
### Inn
- Ny entry: `@shade/crypto-web/worker` — dedikert Web Worker med
`WorkerCryptoProvider`.
- Hovedtråd-proxy: `MainThreadCryptoProvider` som forwarder kall til Worker.
- Stream-pipeline: `ReadableStream<Uint8Array>` → Worker (transferable
buffers) → `@shade/transfer`-chunk-PUTs.
- Lifecycle: spawn-on-demand, idle-timeout, terminate-on-rotate.
- Safari-aware chunk-sizing (Safari har lavere `postMessage`-kapasitet).
### Ut
- Service Workers (background sync) — egen vurdering.
- SharedArrayBuffer (krever COOP/COEP-headers; valgfritt opt-in).
---
## Design
### Provider-API (uendret for konsumenter)
```ts
const crypto = await createWorkerCryptoProvider({
workerUrl: '/shade-crypto.worker.js',
});
const shade = await createShade({ crypto, ... });
```
`WorkerCryptoProvider` implementerer samme `CryptoProvider`-interface som
`SubtleCryptoProvider`. Kall serialiseres med transferable `ArrayBuffer`
minne ikke kopieres.
### Stream-pipeline
```ts
file.stream()
.pipeThrough(shade.encryptStream(peer)) // worker
.pipeThrough(shade.transfer.outboundChunks()) // main → http
.pipeTo(transferSink());
```
Worker-siden av `encryptStream` bruker `MultiLaneSender`.
---
## Leveranser
### Kode
- `@shade/crypto-web` — ny `worker.ts` entrypoint.
- `@shade/sdk``shade.encryptStream` / `decryptStream`.
- Bundler-eksempel for Vite, Webpack og Rollup.
### Tester
- Unit: postMessage roundtrip med transferable buffer.
- Integration: 100 MB fil i nettleser uten frame-drop > 16 ms (P99).
- Safari: chunked `postMessage`-workaround.
### Dokumentasjon
- `docs/web-workers.md` — setup, bundler-kvirks, Safari-notater, COOP/COEP
for SharedArrayBuffer-modus.
---
## Akseptansekriterier
- [x] 100 MB upload i Chrome uten å blokkere main thread > 16 ms i P99
(Performance Observer-måling — verifiseringsoppskrift i
[`docs/web-workers.md`](../web-workers.md#verifying-main-thread-budget)).
- [x] Safari fungerer med default chunk-size (256 KiB postMessage budget,
langt under Safari's transferable-grense).
- [x] Worker termineres innen 30 s etter siste bruk
(`idleTimeoutMs`, default `30_000`).
---
## Avhengigheter
Ingen direkte. Kan kjøres parallelt med V3.2 / V3.4.
---
## Risiko
- **Bundler-helvete.** Vite, Webpack og Rollup behandler Workers ulikt.
Mitiger ved publisert recipe + integration-tester per bundler.
- **Safari postMessage-grenser.** Test tidlig.
---
## Migrasjon
Opt-in. Default forblir `SubtleCryptoProvider`.

137
docs/archive/V3.9.md Normal file
View File

@@ -0,0 +1,137 @@
Start implementasjon, og ikke gi deg før 100% av planen er implementert, alle tester er validert og grønne, samt å ha oppdatert dokumentasjon.
# Shade V3.9 — Rich File Metadata & Previews
**Status:** Implementert (se `docs/streams.md` § Rich file metadata)
**Effort:** M
**Forrige:** V3.1
**Adresserer:** V2.2 §3
---
## Mål
Rikere fil-UX uten å lekke sensitivt innhold til server. Filename,
MIME-type, total length, valgfri thumbnail — alt **E2EE** eller utelatt.
Konsumenter (widgets, files-RPC) kan vise preview før download fullfører.
---
## Scope
### Inn
- Utvid `stream-init` (kontroll-envelope) med valgfrie felt:
- `filename: string` (E2EE, opt-in).
- `mimeType: string` (E2EE, opt-in).
- `totalBytes: number` (alltid OK — bytes-binnet i obs).
- `thumbnailHash: Uint8Array` (sha256 av separat thumbnail-stream).
- Thumbnail som **separat stream** (ikke inline i init) — krypteres med
eget lane.
- Format-hardening på klient: max-size, sandbox i UI.
- Widget-støtte: `<TransferRow showThumbnail />`.
### Ut
- Server-side thumbnail-generering (vi krypterer på klient — server får
aldri klartekst).
- Video preview — separat sak; krever frame-extraction og sandbox.
---
## Design
### Stream-init wire (faktisk implementasjon)
`fileMetadata` er nå et opt-in felt på `StreamMetadata`. Eksisterende
felter er uendret; eldre mottakere ignorerer feltet —
backwards-kompatibelt.
```jsonc
{
"kind": "shade.stream-init/v1",
"streamId": "...",
"streamSecret": "...",
"metadata": {
"chunkSize": 1048576,
"sentAt": 1730000000000,
"userMetadata": { ... }, // eksisterer (V0.3)
"fileMetadata": { // NYTT (V3.9)
"filename": "report.pdf",
"mimeType": "application/pdf",
"thumbnailStreamId": "Ej1z...",
"thumbnailHash": "9a7c...",
"thumbnailMime": "image/webp",
"thumbnailBytes": 18342
}
},
"lanes": [ /* ... */ ]
}
```
### Thumbnail
- Klient genererer 256×256 JPEG/WebP/PNG (browsers via `OffscreenCanvas`
+ `createImageBitmap`).
- Krypteres som **separat stream** med eget `streamId` (referert fra
hoved-strømmens `fileMetadata.thumbnailStreamId`). Den symbolske
konvensjonen `mainStreamId + ".thumb"` er en hjelper; det reelle
streamId er en uavhengig 16-byte verdi.
- Mottaker auto-aksepterer thumbnail-streamen (markert av
`userMetadata.shadeThumbnail = "1"`) inn i `ShadeThumbnailCache`,
som verifiserer sha256 mot deklarert hash før widget rendrer.
---
## Leveranser
### Kode
- `@shade/streams` — utvid `StreamInitMessage`-schema.
- `@shade/sdk``Shade.upload({ ..., generateThumbnail: true })`.
- `@shade/widgets``<TransferRow />` med thumbnail-prop.
### Tester
- Roundtrip: upload med thumbnail, download viser thumbnail før main
ferdig.
- Backwards: 0.3.x-mottaker får stream uten thumbnail og fungerer.
- Format-fuzzing: ondsinnet bilde-fil rendres ikke uten sandbox.
### Dokumentasjon
- `docs/streams.md` utvidet.
- `docs/files.md` — referer til metadata-utvidelsen.
---
## Akseptansekriterier
- [x] Thumbnail leveres som separat E2EE stream som ankommer før main
fullfører (sender shipper preview før hovedstrøm).
- [x] Eldre klient (uten V3.9-støtte) får original stream uten å feile —
dekket av `streams-tests/file-metadata.test.ts` og
`sdk-tests/thumbnail.test.ts` (legacy receiver).
- [x] Thumbnail er aldri synlig i server-DB i klartekst — preview-bytes
rider på en uavhengig AEAD-stream akkurat som hovedstrømmen.
---
## Avhengigheter
- V3.1 — wire-format-utvidelser dokumentert.
---
## Risiko
- **Thumbnail-format-angrep.** Ondsinnet bilde-fil kan kompromittere
preview-renderer. Mitiger ved sandbox-iframe + max-size + format-allowlist.
- **UX-feil.** "Mottaker ser preview før send er ferdig" kan lekke at
avsender prøver å sende noe spesifikt før det er ferdig. Dokumenter for
høy-stakes flows.
---
## Migrasjon
Backwards-kompatibel — alle nye felt er valgfrie.

123
docs/archive/V4.0.md Normal file
View File

@@ -0,0 +1,123 @@
# Shade V4.0 — External Audit, Consolidation, GA
**Status:** Done — tagget som 4.0.0 (2026-05-03)
**Effort:** M (audit-driven)
**Forrige:** V3.1 → V3.12 alle merget
**Adresserer:** V2.1 §6 + samlet GA
> **Scope-merknad:** Voice/Video og all VOIP/streaming-funksjonalitet
> er flyttet til [V5.0](../V5.0.md). 4.0 GA fryser kjerne-stacken
> (ratchet, transport, P2P, recovery, KT) og blir ekstern-revidert
> *uten* sanntid-protokoll i scope. Det lar oss audite én ting av
> gangen — voice/video-frame-keys får sin egen revisjon i 5.0-vinduet.
---
## Mål
Shade 4.0 er **GA-merket release** der alt diskutert i V2.1, V2.2, V2.3
og bonus-track *unntatt* voice/video er i `main`, testet, dokumentert og
review'd. Dette er konsolideringsfasen, ikke ny funksjonsbygging.
Sanntid-laget (voice, video, broadcast) ligger i V5.0 og utvikles oppå
den låste 4.0-stacken.
---
## Scope
### Inn
- **Ekstern crypto-review** av:
- Core (X3DH + ratchet + sender-keys).
- Wire 0x02 + streams 0x11.
- Storage encryption (V3.2).
- Recovery (V3.10).
- WebRTC P2P transport-binding (V3.11).
- Key transparency (V3.12, hvis implementert).
- *(Voice/Video frame keys revideres separat i V5.0-vinduet.)*
- **Migration-guide** 0.3.x → 4.0 — hver wire-bump, schema-endring og
opt-in flagg dokumentert.
- **Soak-testing** — kjør alle pakker i kombinerte stress-tester i 2+
uker.
- **Cross-platform paritet bekreftet** — TS + Kotlin grønne på alle
vector-tester.
- **Dokumentasjons-pass** — README, alle docs/ revidert for 4.0-narrativ.
- **Release-notes + announcement-post.**
### Ut
- Ny krypto.
- Nye pakker.
- Ny wire-format-bump (vi nullstiller her, neste kommer i 4.1+).
---
## Pre-flight checklist
- [ ] V3.1 → V3.12 alle merget.
- [ ] Ingen åpne kritiske eller høy-alvor security issues.
- [ ] Alle test-vectors grønne TS + Kotlin.
- [ ] Production-checklist (V3.1) testet av minst én reell deploy.
- [ ] OpenAPI dekker alle HTTP-flater.
- [ ] Threat model speiler alt nytt (eksklusive sanntid — det er V5.0).
- [ ] Eksisterende 0.3.x → 4.0 migration-CLI testet på reell DB.
---
## Crypto-review-prep
Forberedelse til ekstern reviewer:
1. **Pakke "review-bundle"** — én PR med:
- Linker til alle protokoll-spec-filer.
- Trusselmodellen.
- Antagelser og kjente begrensninger.
- Reproduserbar build-instruksjon.
2. **Scope-dokument** — hvilke deler reviewer ser på (ratchet ja,
build-system nei).
3. **Kontakt-prosess** — hvordan rapportere findings.
4. **Tidslinje** — typisk 48 uker review-vindu.
Anbefalt scope-prioritering:
- **A:** ratchet, X3DH, storage-encryption, recovery (kjerne-protokoll).
- **B:** WebRTC P2P transport-binding, KT-log (hvis implementert).
- **C:** transport-lag, observability (lavere risiko).
- *(Frame-keys er ikke i 4.0-scope — de revideres når V5.0 lander.)*
---
## Akseptansekriterier
- [ ] Ekstern review uten åpne kritiske/høy-alvor findings.
- [ ] Migration-guide brukt vellykket på minst én ekte 0.3.x-deploy.
- [ ] Cross-platform parity verifisert i CI.
- [ ] All `docs/V*.md` arkivert under `docs/archive/` med "DONE"-status.
- [ ] CHANGELOG.md har 4.0-seksjon.
- [ ] Versjon bumpet, alle pakker publisert til Gitea-registry.
- [ ] Docker-image `gt.zyon.no/stian/shade-prekey:4.0.0` publisert.
---
## Etter 4.0
V4.x-serien starter forsiktig: bug-fixes, små features, ingen wire-bump
uten 5.0-vindu.
**[V5.0](../V5.0.md)** er øremerket sanntid: voice (`@shade/voice`),
video (`@shade/video`), 1:N broadcast (`@shade/broadcast`) — alt bygd
oppå den låste 4.0-stacken med SFrame-frame-keys avledet fra
ratchet-sesjonen. V5.0 får sin egen ekstern revisjon av frame-key-
delen før release.
Lengre fram: federation, multi-tenancy, SDK for nye språk (Swift,
Rust) og MLS-overgang for grupper er alle åpne kandidater for V6.0+.
---
## Risiko
- **Audit-findings.** Kan kreve ny implementasjon i siste sekund. Mitiger
ved tidlig review-prep og prioritering av A-scope først.
- **Scope creep.** "Bare en ting til" — V4.0 er låst til konsolidering.
Nye features = V4.1+.

143
docs/audit/REVIEW-BUNDLE.md Normal file
View File

@@ -0,0 +1,143 @@
# Shade 4.0 — External Crypto Review Bundle
This document is the entrypoint for an external cryptographic review of
Shade 4.0. It collects, in one place, every artifact a reviewer needs to
audit the protocol implementation **without** rooting around the
codebase first.
## Tag under review
- **Version:** `4.0.0`
- **Tag:** `v4.0.0`
- **Date:** 2026-05-03
- **Repo:** `https://gt.zyon.no/Stian/Shade` (mirror at the
consumer-app repos that vendor this code)
- **Out-of-scope:** Voice / Video / Broadcast — moved to V5.0 and
reviewed separately.
## What's in scope
Reviewers focus on the protocol-cryptographic core. Each scope cell maps
to one or more packages plus the spec / threat-model section that
describes its design.
### A — Protocol core (highest priority)
| Surface | Spec | Code |
|---------|------|------|
| X3DH initial key agreement | [`docs/archive/V3.1.md`](../archive/V3.1.md), [`THREAT-MODEL.md` §1, §2](../../THREAT-MODEL.md) | [`packages/shade-core/src/x3dh.ts`](../../packages/shade-core/src/x3dh.ts) |
| Double Ratchet | [`docs/archive/V3.1.md`](../archive/V3.1.md), [`THREAT-MODEL.md` §3](../../THREAT-MODEL.md) | [`packages/shade-core/src/ratchet.ts`](../../packages/shade-core/src/ratchet.ts) |
| Sender keys (group ratchet) | [`docs/archive/V3.10.md` § Group send](../archive/V3.10.md) | [`packages/shade-core/src/sender-keys.ts`](../../packages/shade-core/src/sender-keys.ts) |
| Wire envelopes `0x01`, `0x02`, `0x11` | [`packages/shade-proto/README.md`](../../packages/shade-proto/README.md) | [`packages/shade-proto/src/`](../../packages/shade-proto/src/) |
| At-rest storage encryption | [`docs/storage-encryption.md`](../storage-encryption.md), [`THREAT-MODEL.md` §4](../../THREAT-MODEL.md) | [`packages/shade-storage-encrypted/src/`](../../packages/shade-storage-encrypted/src/) |
| Social recovery (Shamir + AEAD-gated reconstruction) | [`docs/recovery.md`](../recovery.md), [`THREAT-MODEL.md` §8](../../THREAT-MODEL.md) | [`packages/shade-recovery/src/`](../../packages/shade-recovery/src/) |
### B — Trust + transport
| Surface | Spec | Code |
|---------|------|------|
| WebRTC P2P transport binding | [`docs/webrtc.md`](../webrtc.md), [`THREAT-MODEL.md` §11](../../THREAT-MODEL.md) | [`packages/shade-transport-webrtc/src/`](../../packages/shade-transport-webrtc/src/) |
| Key Transparency log + verifier | [`docs/key-transparency.md`](../key-transparency.md), [`docs/archive/V3.12-DESIGN.md`](../archive/V3.12-DESIGN.md), [`THREAT-MODEL.md` §2 (mitigated-by-V3.12)](../../THREAT-MODEL.md) | [`packages/shade-key-transparency/src/`](../../packages/shade-key-transparency/src/) |
| Fingerprint gates | [`docs/trust-ux.md`](../trust-ux.md), [`THREAT-MODEL.md` §10](../../THREAT-MODEL.md) | [`packages/shade-sdk/src/fingerprint-gates.ts`](../../packages/shade-sdk/src/fingerprint-gates.ts) |
### C — Lower-priority surfaces
| Surface | Spec | Code |
|---------|------|------|
| Inbox store-and-forward | [`docs/inbox.md`](../inbox.md), [`THREAT-MODEL.md` §6](../../THREAT-MODEL.md) | [`packages/shade-inbox-server/src/`](../../packages/shade-inbox-server/src/), [`packages/shade-inbox/src/`](../../packages/shade-inbox/src/) |
| Bridge transports (SSE / long-poll / WS) | [`docs/transport.md`](../transport.md) | [`packages/shade-transport-bridge/src/`](../../packages/shade-transport-bridge/src/) |
| Web Workers crypto | [`docs/web-workers.md`](../web-workers.md), [`THREAT-MODEL.md` §12](../../THREAT-MODEL.md) | [`packages/shade-crypto-web/src/worker*`](../../packages/shade-crypto-web/src/) |
| Files RPC | [`docs/files.md`](../files.md) | [`packages/shade-files/src/`](../../packages/shade-files/src/) |
| Streams (chunked AEAD over ratchet) | [`docs/streams.md`](../streams.md) | [`packages/shade-streams/src/`](../../packages/shade-streams/src/), [`packages/shade-transfer/src/`](../../packages/shade-transfer/src/) |
| Observability | [`docs/observability.md`](../observability.md) | [`packages/shade-observability/src/`](../../packages/shade-observability/src/) |
## Threat model
The full threat model is at [`THREAT-MODEL.md`](../../THREAT-MODEL.md).
Every numbered "Mitigations" entry ends with a `[tests:]` footnote
linking to the file(s) that holds the mitigation in place. Reviewers
can re-run any individual test in isolation:
```bash
bun test packages/shade-core/tests/ratchet.test.ts
bun test packages/shade-streams/tests/aead.test.ts
bun test packages/shade-key-transparency/tests/manager.test.ts
```
## Cross-platform parity
The wire format and KDF-label corpus are byte-identical between TS
(bun) and Kotlin (gradle). The CI gate that enforces this lives at
[`.gitea/workflows/cross-vectors.yml`](../../.gitea/workflows/cross-vectors.yml).
Vectors are generated by [`scripts/generate-vectors.ts`](../../scripts/generate-vectors.ts);
hand-edits to [`test-vectors/`](../../test-vectors/) are rejected by CI.
```bash
# Re-run the cross-platform vector suite locally:
bun run test:vectors
cd android && ./gradlew :shade-android:test
```
## Build instructions (reproducible)
```bash
git clone https://gt.zyon.no/Stian/Shade
cd Shade
git checkout v4.0.0
bun install --frozen-lockfile
# TS suite
bun test
# Kotlin / vector suite
cd android && ./gradlew :shade-android:test
```
Container image (prekey + transfer + bridge + KT):
```bash
docker pull gt.zyon.no/stian/shade-prekey:4.0.0
docker run --rm -p 3900:3900 \
-e SHADE_PREKEY_PG_URL=postgres://… \
gt.zyon.no/stian/shade-prekey:4.0.0
```
The `Dockerfile` is at [`packages/shade-server/Dockerfile`](../../packages/shade-server/Dockerfile).
Multi-stage; the runtime stage uses a non-root user.
## Assumptions and known limitations
1. The runtime is honest. A malicious Bun / browser engine can defeat
any JS library; we ride the platform's `SubtleCrypto` / `@noble/curves`
for primitives and trust them.
2. `THREAT-MODEL.md` section "Assumptions" is the canonical list; review
the residual-risks table at the bottom of the same file for
intentional gaps.
3. We do **not** claim resistance to power-analysis or fault-injection
side channels.
4. Memory zeroization is best-effort. V8 / JSC may retain freed buffers;
we zero what we can synchronously reach.
## How to report findings
- **Severity-prioritized** (CVSS 3.1 if you can, otherwise plain
language).
- **Reproducer in repo style** — a failing `bun test` is preferred over
prose.
- **Email** the maintainer (`Sterister@live.no`); see
[`SECURITY.md`](../../SECURITY.md) for PGP / age key arrangement.
## Timeline
The 4.0 audit window is open immediately after tag. We aim for a
48-week review cycle (see V4.0 plan). Any **critical** or **high**
severity finding pauses the GA-stable announcement until the fix
ships. Findings ship as `4.0.x` patch releases — wire-format unchanged.
## Out-of-scope (deferred to V5.0)
- Voice (`@shade/voice`) — SFrame-style frame keys, key-rotation policies.
- Video (`@shade/video`) — codec edges (AV1/VP9/H.264).
- Broadcast (`@shade/broadcast`) — relay-helper threat model.
These will get their own review window when V5.0 is ready.

75
docs/audit/SCOPE.md Normal file
View File

@@ -0,0 +1,75 @@
# Shade 4.0 — Audit Scope
A short, structural list a reviewer can scan before opening a single
file. Everything here is a pointer to the deeper material in
[`REVIEW-BUNDLE.md`](./REVIEW-BUNDLE.md) and the package READMEs.
## In scope
- **Protocol primitives**: X3DH, Double Ratchet, sender keys.
- **Wire format**: `0x01` PreKeyMessage, `0x02` RatchetMessage, `0x11`
StreamChunk. Length prefixes (u32) and AAD bindings.
- **Storage encryption** (`@shade/storage-encrypted`): KDF chain,
per-(table,column) DEKs, AEAD AAD layout, online re-key.
- **Recovery** (`@shade/recovery`): Shamir over GF(2^8),
AEAD-authenticated reconstruction, fingerprint gate on guardian
release, share-grant / share-decline envelope schema.
- **WebRTC P2P** (`@shade/transport-webrtc`): SDP/ICE signaling rides
the ratchet; chunk frames AEAD-bound to streamId/laneId/seq; glare
resolution determinism.
- **Key Transparency** (`@shade/key-transparency`): Merkle log over
pre-hashed leaves, address-sorted index, signed STH, witness
cross-check, split-view detection.
- **Inbox** (`@shade/inbox-server`): TOFU registration, per-PUT signed
blobs, idempotent on `(address, msgId)`, replay window.
- **Bridge** (`@shade/transport-bridge`): SSE / long-poll / WS
carriers; signed-query auth (no headers on `EventSource`).
- **Crypto in workers** (`@shade/crypto-web/worker`): key-isolation
boundary, postMessage protocol, idle terminate.
- **Trust UX gates** (`@shade/sdk` `Shade.beforeFirstLargeFile`,
`beforeBackupImport`, `beforeNewDeviceTrust`).
## Out of scope
- **Voice / Video / Broadcast** (`@shade/voice` etc.) — V5.0; reviewed
when the package ships.
- **Build system** (Vite, Rollup, Gradle wiring) — out of crypto scope.
- **App-level UI** (`@shade/widgets`) — re-renders the primitives
above; the cryptographic decisions are in the SDK / core packages
the widgets consume.
- **Browser / native WebRTC stacks** — we ride the platform's
`RTCPeerConnection` and `SubtleCrypto`.
- **Operating system / hardware threat model** — filesystem
encryption, secure-enclave key storage, swap-encryption, coredump
handling. Operator responsibility.
## Methodology suggestions
1. Start with [`THREAT-MODEL.md`](../../THREAT-MODEL.md) — every entry
has a `[tests:]` footnote. Toggle each test off, confirm it fails;
toggle the corresponding mitigation off, confirm it fails.
2. Re-derive every KDF label from the spec; check
[`scripts/generate-vectors.ts`](../../scripts/generate-vectors.ts) and
the recorded vectors in [`test-vectors/`](../../test-vectors/) match.
3. Run the cross-platform suite on **both** TS (bun) and Kotlin
(gradle) — divergence is a vector-format bug.
4. Audit the AEAD AAD construction at every layer:
- Ratchet: header bytes (counter + DH pub) → AES-GCM AAD.
- Streams: `streamId || laneId || seq || isLast` → AES-GCM AAD.
- Storage: `(table, column, pk)` → AES-GCM AAD.
5. Trace the boundary between the worker-side crypto thread and the
main thread — confirm that no handle to a wrapped DEK or a
ratcheted chain key crosses over.
## Open questions for reviewer commentary
- The witness gossip channel for V3.12 is currently in-band over the
ratchet; should we cross-pin against an out-of-band log mirror in
4.x, or wait for a federated relay tier?
- WebRTC peer-glare is resolved by lexicographic address compare — a
reviewer could confirm the equivalent constructions in libsignal or
Matrix and flag if our edge cases match.
- Storage encryption uses AES-GCM with a per-row IV. The IV is
random, not deterministic; reviewers should confirm the
combinatorial-collision threshold matches the per-column row count
bounds.

189
docs/cross-platform.md Normal file
View File

@@ -0,0 +1,189 @@
# Cross-platform parity — adding & running vectors
Shade keeps its TypeScript and Kotlin implementations in lock-step via a
**single source of truth**: `test-vectors/*.json`. Both runners load the
same files and verify their native code produces byte-identical output.
This document covers:
1. How the parity gate works (CI)
2. How to run vectors locally
3. How to add a new vector
## How the gate works
```
┌─────────────────────────────────┐
│ scripts/generate-vectors.ts │
│ (TS reference implementation) │
└────────────────┬────────────────┘
│ writes
┌─────────────────────────────────┐
│ test-vectors/*.json │
│ { version: 2, vectors: [...] }│
└─────┬──────────────────┬────────┘
│ │
│ loaded by │ loaded by
▼ ▼
┌───────────────────────────┐ ┌───────────────────────────┐
│ packages/shade-core/ │ │ android/shade-android/ │
│ tests/cross-platform- │ │ src/test/kotlin/.../ │
│ vectors.test.ts │ │ CrossPlatformVectorTest │
│ (bun) │ │ (gradle JUnit4) │
└───────────────────────────┘ └───────────────────────────┘
│ │
└─────────┬────────┘
both must pass before merge
(.gitea/workflows/cross-vectors.yml)
```
The CI workflow has **two independent jobs**`ts-vectors` and
`kotlin-vectors`. Either failing blocks the merge. The TS job also runs
`bun run vectors:gen` and fails if the result diverges from the committed
files: vector commits must come from the generator, never hand edits.
Vector files have a `version` integer at the top. Bump
`VECTOR_FILE_VERSION` in `scripts/generate-vectors.ts` whenever the
**schema** of any vector file changes (not just the values). Both test
suites assert the version matches their hard-coded expectation.
## Running vectors locally
### TypeScript
```bash
bun run test:vectors
# under the hood:
# bun test packages/shade-core/tests/cross-platform-vectors.test.ts
```
### Kotlin (JVM, no Android SDK required)
```bash
cd android
./gradlew :shade-android:test
```
Requires JDK 17. The wrapper downloads Gradle 8.10.2 on first run. Tink
1.15.0 (JVM JAR) is pulled from Maven Central.
### Regenerating vectors
When the protocol changes (new wire field, new label, new derivation step)
the TS reference is the source of truth. Edit `generate-vectors.ts`, then:
```bash
bun run vectors:gen
git diff test-vectors/ # eyeball the change
bun run test:vectors # confirm TS still agrees
cd android && ./gradlew :shade-android:test # confirm Kotlin still agrees
```
If Kotlin disagrees, **fix Kotlin** — TS is canonical. If both agree but
the diff is unintentional (e.g. you added a field by accident), revert
the generator change.
## Adding a new vector
A new sjekkpunkt has four pieces: generator code, schema, TS test,
Kotlin test. All four must land in the same PR; otherwise the gate
trips on the missing half.
### Step 1 — Add a generator function
In `scripts/generate-vectors.ts`, add a function that:
- Takes deterministic inputs (no randomness — fix every byte)
- Computes the value via the TS reference primitives
- Returns a `Vector[]` with a `description` per case + all inputs and outputs
in hex
Example skeleton:
```ts
async function generateMyVectors(): Promise<Vector[]> {
const input = new Uint8Array(32).fill(0xab);
const output = await someRefImpl(input);
return [{
description: 'My new sjekkpunkt: known input → known output',
input: hex(input),
output: hex(output),
}];
}
```
Wire it up in `main()`:
```ts
['my-vectors.json', { vectors: await generateMyVectors() }],
```
Run `bun run vectors:gen` → you should see `✓ my-vectors.json` and a new
file appears under `test-vectors/`.
### Step 2 — Add a TS test
In `packages/shade-core/tests/cross-platform-vectors.test.ts`:
```ts
test('My vectors match', async () => {
const { vectors } = loadVectors('my-vectors.json');
for (const v of vectors) {
const actual = await someRefImpl(fromHex(v.input));
expect(hex(actual)).toBe(v.output);
}
});
```
`loadVectors` already asserts the version field matches. If you're
introducing a schema-breaking change, bump `EXPECTED_VECTOR_VERSION` and
`VECTOR_FILE_VERSION` together.
### Step 3 — Add the Kotlin equivalent
In
`android/shade-android/src/test/kotlin/no/zyon/shade/CrossPlatformVectorTest.kt`:
```kotlin
@Test
fun myVectorsMatch() {
val vectors = loadVectors("my-vectors.json")
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
val actual = someKotlinImpl(fromHex(v.getString("input")))
assertEquals(v.getString("output"), hex(actual))
}
}
```
If the Kotlin port doesn't yet have `someKotlinImpl`, that's the implementation
work the new vector is gating — write it and re-run the test until it passes.
### Step 4 — Verify the gate trips on divergence
Sanity check: temporarily flip a byte in your Kotlin port and run
`./gradlew :shade-android:test`. The test should fail within 60 seconds
(see `docs/V3.5.md` §Akseptansekriterier). Revert.
## Why a separate generator (vs. golden fixtures)?
Golden test fixtures rot — when the protocol changes, every test file
that pinned a literal hex string needs updating, and it's easy to
"update" Kotlin to match a stale TS-generated value. By centralising
vector generation in one TS script, **the protocol changes in one
place** (the reference impl + `generate-vectors.ts`), the file
regenerates with one command, and any platform that drifts gets caught
by the next CI run.
## Schema versioning
`{ "version": 2, "vectors": [...] }` is the file format. Bump the int
when the **shape** of any vector changes (e.g. you add a field consumers
must read). Both runners hard-code their expected version and refuse to
parse mismatched files — this catches the case where a new vector field
was added in TS but the Kotlin loader silently ignored it.
Schema changes go in the same PR as the bump + the matching loader
update on both sides.

325
docs/files.md Normal file
View File

@@ -0,0 +1,325 @@
# `@shade/files` — E2EE filesystem RPC
`@shade/files` exposes a typed, end-to-end-encrypted RPC surface for
filesystem-style operations between two Shade peers. Both sides ride a
single Double Ratchet session for control envelopes; large-file content
(`> 256 KiB`) flows through `@shade/transfer` lanes, correlated back to
the RPC by an opaque `userMetadata.shadeFilesWriteId` /
`shadeFilesReadStreamId`.
The package is a **primitive**, not a UI: it ships hooks and clients, not
file managers. Consumers (e.g. Dispatch, Mail, Drive-style apps) keep
their own UI and plug `@shade/files` underneath.
## Quick start
### Server (Bob exposes a filesystem)
```ts
import { createShade } from '@shade/sdk';
const bob = await createShade({ prekeyServer, address: 'bob' });
bob.configureTransfers({ resolveBaseUrl: ... });
Bun.serve({ port: 8080, fetch: (await bob.transferRoute()).fetch });
const stop = await bob.files.serve({
list: async (ctx) => ({ entries: await readdirAt(ctx.path), hasMore: false }),
stat: async (ctx) => statAt(ctx.path),
mkdir: async (ctx) => mkdirAt(ctx.path, ctx.args.recursive),
delete: async (ctx) => deleteAt(ctx.path, ctx.args.recursive),
move: async (ctx) => moveAt(ctx.args.src, ctx.args.dst, ctx.args.overwrite),
read: async (ctx) => readAt(ctx.path), // returns inline | streams
write: async (ctx) => writeAt(ctx.args), // receives inline | streams
getThumbnail: async (ctx) => thumbnailAt(ctx.path, ctx.args.size, ctx.args.format),
});
// Later: stop the handler.
await stop();
```
### Client (Alice consumes Bob's filesystem)
```ts
const alice = await createShade({ prekeyServer, address: 'alice' });
alice.configureTransfers({ resolveBaseUrl: ... });
Bun.serve({ port: 8081, fetch: (await alice.transferRoute()).fetch });
const fs = await alice.files.client('bob');
const page = await fs.list('/photos');
await fs.write('/photos/cover.png', new Uint8Array(...)); // auto inline/streams
const result = await fs.read('/photos/cover.png');
if (result.kind === 'inline') console.log(result.bytes.byteLength);
else for await (const _chunk of /* result.stream */) { /* ... */ }
```
## HTTP RPC — browser-friendly request-response (4.1+)
The default `shade.files.client(peer)` requires both peers to be mutually
addressable over HTTP — the response to a `list`/`read` etc. round-trips
through `Shade.deliverControlEnvelope`, which POSTs to the peer's
`/v1/transfer/control` endpoint. **That doesn't work for browsers**
a tab can't host an HTTP server, so the server cannot call back outbound.
`@shade/files` 4.1 ships a parallel **request-response** transport that
lets browser-style clients fully consume the file-RPC surface without
any inbound channel. It mirrors the way `@shade/server`'s
`shade-auth-middleware` works: one POST per RPC, encrypted envelope in
the request body, encrypted response in the same HTTP response.
### Server side — mount the RPC route
```ts
// 1. Register the file handler. `inlineOnly: true` skips the
// streams-bridge (which would require @shade/transfer).
await shade.files.serve(handlerConfig, { inlineOnly: true });
// 2. Mount the route on your Hono app under any base path.
app.route('/api/v1/shade-files', shade.files.rpcRoute());
// ^^^^^^^^^^^^^^
// POST <base>/rpc
```
`rpcRoute()` accepts:
| Option | Default | Purpose |
|---------------------|---------|----------------------------------------------------------------------------------------------------|
| `maxBodyBytes` | 1 MiB | Max request body. The protocol caps inline payloads at 256 KiB; the headroom is for base64 inflation + custom-op envelopes. |
| `acceptFirstMessage`| `false` | Accept `0x01` PreKeyMessage envelopes — required when the RPC route also doubles as the X3DH handshake (browser's first-ever request). |
### Browser client
```ts
import { createShade } from '@shade/sdk';
const shade = await createShade({
prekeyServer: 'https://shade.example.com',
storage: 'memory',
address: 'alice@example.com',
});
const fs = shade.files.httpClient('bob@example.com', {
rpcUrl: 'https://dispatch.example.com/api/v1/shade-files/rpc',
// Optional: thread CSRF / auth tokens, override fetch, etc.
headers: { 'X-CSRF-Token': csrfToken },
});
await fs.mkdir('/photos');
await fs.write('/photos/cover.png', new Uint8Array([/* ... */]), {
contentType: 'image/png',
});
const result = await fs.read('/photos/cover.png');
```
### What works in HTTP-RPC mode
- `list`, `stat`, `mkdir`, `delete`, `move`, `getThumbnail`, `custom<K>` — full parity.
- `write`**inline only** (≤ 256 KiB plaintext). Larger inputs throw `ConflictError`.
- `read`**inline only**. If the server returns a streamed `read` result, the client throws `InternalFileError` directing callers to the stateful pathway.
### Wire contract
```
POST /rpc HTTP/1.1
Content-Type: application/octet-stream
X-Shade-Sender-Address: alice@example.com
<wire-encoded ShadeEnvelope (0x01 first-time, 0x02 after) wrapping
JSON-encoded RpcRequest>
────
200 OK
Content-Type: application/octet-stream
<wire-encoded ShadeEnvelope (0x02) wrapping JSON-encoded RpcResponse | RpcError>
```
Transport-level failures (no session, undecryptable envelope, body too
big) return JSON `{ "error": "..." }` with appropriate 4xx status.
Application-level failures (file not found, permission denied) ship
encrypted `RpcError` envelopes — the client maps them back to typed
`FileError` subclasses (`NotFoundError`, `ConflictError`, etc.).
### Symmetry with `@shade/server`
The shape mirrors `@shade/server`'s shade-auth-middleware: encrypted
envelope rides the request body, server decrypts via the existing
ratchet session, performs the protected operation, returns an encrypted
envelope in the response. No bidirectional channel required, no
WebSocket, no SSE.
### When to use which
| Setup | Use |
|-----------------------------------------------|-----------------------------------------------|
| Browser client ↔ Bun/Hono server | `httpClient()` + `rpcRoute()` |
| Server ↔ server (both can host HTTP) | `client()` (default) — supports streams |
| Service-worker / extension ↔ server | `httpClient()` (no inbound listener) |
| CLI / daemon ↔ daemon | Either; `client()` if you need streams |
## Op surface
| Op | Args | Result |
|----------------|------------------------------------------|-----------------------------------------|
| `list` | `path`, `cursor?`, `pageSize?`, `filter?`| `{ entries, hasMore, nextCursor? }` |
| `stat` | `path` | `FileEntry` |
| `mkdir` | `path`, `recursive?` | `{ entry: FileEntry }` |
| `delete` | `path`, `recursive?` | `{ deletedCount }` |
| `move` | `src`, `dst`, `overwrite?` | `{ entry: FileEntry }` |
| `read` | `path`, `range?`, `preferInline?` | inline `{ bytes, sha256, size }` or streams `{ stream, size, sha256 }` |
| `write` | `path`, `Uint8Array \| Blob \| Stream` | `{ entry: FileEntry }` |
| `getThumbnail` | `path`, `size: 64\|128\|256\|512`, `format?` | `{ bytes, format, width, height, sha256 }` |
| `custom<K>` | typed via `CustomOpsMap[K]` | typed via `CustomOpsMap[K]` |
`MUTATION_OPS = { mkdir, delete, move, write, custom }` — these auto-generate
an idempotency key per logical call so transparent retries under the SDK
don't produce duplicates.
## Inline vs streams
The threshold is `INLINE_THRESHOLD = 256 * 1024` bytes (plaintext). The
client's `decideInline()` runs at `write` time:
* `Uint8Array` / `Blob` with known size → direct comparison.
* Bare `ReadableStream` → peek the first chunks; promote to streams as
soon as the buffered prefix exceeds the threshold.
Streams paths kick `shade.upload(...)` with `userMetadata.shadeFilesWriteId`
in parallel with the RPC envelope. The server's streams-bridge accepts the
inbound transfer immediately into a `TransformStream` and parks the
readable side until the matching `write` RPC arrives.
## Custom ops
Augment `CustomOpsMap` once globally for type-safe consumer-defined ops:
```ts
declare module '@shade/files' {
interface CustomOpsMap {
'app.deploy': CustomOpDef<{ jarPath: string }, { deploymentId: string }>;
}
}
// Server registers a Zod-backed handler:
await shade.files.serve({
custom: {
'app.deploy': {
args: z.object({ jarPath: z.string() }),
response: z.object({ deploymentId: z.string() }),
handler: async (args, ctx) => ({ deploymentId: deploy(args.jarPath) }),
},
},
});
// Client gets typed I/O:
const result = await fs.custom('app.deploy', { jarPath: '/mods/foo.jar' });
// ^? { deploymentId: string }
```
## Production hooks
All hooks are callbacks the consumer plugs in — `@shade/files` enforces
the *mechanism*; the app owns the *policy*.
```ts
await shade.files.serve({
pathPolicy: { rootScope: '/srv/shade-root', forbidTraversal: true },
rateLimits: {
maxOpsPerMinutePerSender: 600,
maxBytesPerHourPerSender: 10 * 1024 ** 3,
opCost: { read: 1, write: 5, delete: 3, default: 1 },
},
idempotency: { ttlMs: 60 * 60 * 1000, maxEntriesPerSender: 10_000 },
requireFingerprintVerifiedFor: (ctx) =>
['delete', 'write', 'mkdir'].includes(String(ctx.op)) ? 'required' : 'optional',
isFingerprintVerified: (sender) => verifiedSet.has(sender),
verifySender: async (sender, canonical, sig) => {
return ed25519.verify(base64ToBytes(sig), canonical, getPubKey(sender));
},
onMetric: (name, value, tags) => prometheus.observe(name, value, tags),
onError: (err, ctx) => logger.error({ op: ctx.op, sender: ctx.sender }, err),
});
```
## React
```tsx
import { ShadeFilesProvider, useFileList } from '@shade/files/react';
function FileBrowser({ shade, peer }: { shade: Shade; peer: string }) {
return (
<ShadeFilesProvider shade={shade}>
<Listing peer={peer} path="/" />
</ShadeFilesProvider>
);
}
function Listing({ peer, path }) {
const { entries, isLoading, hasMore, loadMore, refresh } = useFileList(peer, path);
if (isLoading) return <Spinner />;
return (
<ul>
{entries.map((e) => <li key={e.name}>{e.name} ({e.kind})</li>)}
{hasMore && <button onClick={loadMore}>More</button>}
</ul>
);
}
```
`useFileTransfer` exposes a generic state machine for `BulkTransferHandle`s
returned by `uploadDirectory()` / `downloadDirectory()`:
```tsx
const { start, abort, state, filesDone, filesTotal, bytesDone } = useFileUpload();
const handle = uploadDirectory(fs, localDir, '/uploaded');
useEffect(() => { start(handle); return () => void abort(); }, []);
```
## Path safety
The dispatcher applies `validatePath()` before invoking the user handler:
1. Length check on raw input.
2. Forbidden-bytes check (NUL/CR/LF/DEL/backslash).
3. Percent-decode (defends against `%2e%2e` smuggling).
4. POSIX normalization.
5. `..` traversal rejection.
6. Root-scope boundary check.
7. Consumer-supplied `extra` predicate.
The user handler receives the *normalized* path; using `args.path` raw is a
TOCTOU bug.
## Wire compatibility
`@shade/files` 0.3.0 requires `@shade/proto` 0.3.0+. The proto layer's wire
VERSION was bumped from `0x01` to `0x02` to lift the 64 KiB length-prefix
ceiling that previously capped ratchet payloads. **Sessions are
incompatible across the bump**; both peers must run 0.3.0+.
## Rich file metadata + previews (V3.9)
`stream-init` carries optional E2EE `fileMetadata` (filename, MIME,
thumbnail-stream pointer). `@shade/files` consumers see this on the
incoming-transfer side and can render previews via `<TransferRow
showThumbnail />`. The thumbnail itself rides as a separate AEAD
stream — server never sees preview pixels in plaintext.
See [streams.md § Rich file metadata + previews](streams.md#rich-file-metadata--previews-v39)
for the wire format, format-hardening rules, and renderer trust
model. The pattern integrates seamlessly with `@shade/files`'s own
write/read RPCs — pass `fileMetadata` in the underlying
`shade.upload` and the same `ShadeThumbnailCache` powers previews
across all transfer surfaces.
## Related modules
* `@shade/streams` — chunk encryption, lane key derivation. Indirect dep.
* `@shade/transfer` — multi-lane transport with HTTP / WS fallback.
* `@shade/transport-webrtc` (V3.11, optional) — direct P2P chunk
delivery via `RTCDataChannel`; large `read`/`write` payloads
automatically prefer WebRTC when both peers have called
`shade.configureWebRTC()`.
* `@shade/sdk``Shade.files` getter; `BackgroundHooks.onPruneFiles` for
retention.

317
docs/inbox.md Normal file
View File

@@ -0,0 +1,317 @@
# Shade Inbox — Async Store-and-Forward (V3.6)
A relay that holds **ciphertext blobs with TTL** so senders can deliver
to recipients who happen to be offline. The relay never sees plaintext,
never holds private keys, and never knows who is talking to whom in
plaintext form (only addresses and bytes-per-blob).
This document covers:
- Setup (server side, single-binary)
- Client integration (`@shade/inbox`)
- Threat model — *what the relay actually sees*
- Operational tuning (TTL, quotas, prune cadence)
- Wire-level reference
---
## 1. Server setup
The inbox server is built into the same `@shade/server` standalone
container that ships the prekey server, on the same port. Routes are
namespaced under `/v1/inbox/*`.
### Docker (single binary, both services)
```bash
docker run -d --name shade \
-p 3900:3900 \
-v shade-data:/data \
-e SHADE_PREKEY_DB_PATH=/data/shade-prekeys.db \
-e SHADE_INBOX_DB_PATH=/data/shade-inbox.db \
-e SHADE_INBOX_PRUNE_INTERVAL_MINUTES=5 \
ghcr.io/zyon-no/shade:latest
```
### Postgres (multi-instance / shared infra)
```bash
docker run -d --name shade \
-p 3900:3900 \
-e SHADE_PREKEY_PG_URL='postgres://shade:***@db/shade' \
-e SHADE_INBOX_PG_URL='postgres://shade:***@db/shade' \
ghcr.io/zyon-no/shade:latest
```
Tables are auto-created (`shade_inbox_owners`, `shade_inbox_blobs`,
sequence `shade_inbox_seq`). If you only set `SHADE_PREKEY_PG_URL`, the
inbox falls back to the same database; set
`SHADE_INBOX_PG_URL='-'` to disable that fallback and run the inbox
in-memory (only useful for short-lived test deployments).
### Env vars
| Var | Default | Effect |
| -------------------------------------- | ------------------------ | ----------------------------------- |
| `SHADE_INBOX_DB_PATH` | _(unset → memory)_ | SQLite file path |
| `SHADE_INBOX_PG_URL` | _(unset → falls back)_ | Postgres connection string |
| `SHADE_INBOX_PRUNE_INTERVAL_MINUTES` | `5` | How often expired blobs are dropped |
### Embedding in your own Hono app
```ts
import { Hono } from 'hono';
import { SubtleCryptoProvider } from '@shade/crypto-web';
import { createInboxRoutes, MemoryInboxStore } from '@shade/inbox-server';
const crypto = new SubtleCryptoProvider();
const store = new MemoryInboxStore();
const app = new Hono();
app.route('/', createInboxRoutes(store, crypto));
export default { port: 3901, fetch: app.fetch };
```
---
## 2. Client integration
`@shade/inbox` is the recipient/sender SDK. It composes on top of
`@shade/sdk` — Shade still owns encryption + the ratchet; the inbox
layer is just durable transport.
### Wiring
```ts
import { Shade } from '@shade/sdk';
import { Inbox } from '@shade/inbox';
const shade = new Shade(/* ... */);
await shade.initialize();
// Lift the identity keys we already have.
const identity = await shade.getManager().getIdentityKeyPair();
const inbox = new Inbox({
baseUrl: 'https://inbox.example.com',
ownAddress: shade.myAddress,
crypto: shade.crypto,
signingPrivateKey: identity.signingPrivateKey,
signingPublicKey: identity.signingPublicKey,
pollIntervalMs: 30_000,
});
// Receive: hand each fetched blob to Shade.receive.
inbox.onIncoming(async (raw) => {
const envelope = decodeEnvelope(raw.ciphertext);
// The inbox does not authenticate the sender — Shade.receive does,
// by way of the recipient's session/ratchet/identity-pin.
const senderAddress = /* derive from your own metadata channel */;
await shade.receive(senderAddress, envelope);
return senderAddress;
});
inbox.start(); // registers + begins flush + poll loops
// Send: encrypt with Shade, hand the envelope to the inbox.
const envelope = await shade.send('bob@example.com', 'hi');
await inbox.send({ recipientAddress: 'bob@example.com', envelope });
```
### Push-trigger hook
The inbox is *pull-based* — recipients only see new blobs when they
poll. Most apps want a wake-up nudge when new content lands. Vendor it
yourself (FCM / APNs / email / WebPush):
```ts
inbox.onMessageQueued(async (recipient, msgId) => {
await fcm.send(recipient, { kind: 'shade-inbox', msgId });
});
```
The recipient device wakes, runs `inbox.tick()`, and pulls the blob.
### Durable queue
The default in-memory queue is fine for short-lived processes. For a
service that must survive restart, plug in your own `OutgoingQueueStore`
backed by SQLite/Postgres/IndexedDB:
```ts
const inbox = new Inbox({
// …
queueStore: new MyDurableQueueStore(),
cursorStore: new MyDurableCursorStore(),
});
```
Same idea for the receive cursor — without persistence, every restart
re-downloads everything currently within TTL.
### Errors
- **Decrypt failure** in your handler keeps the blob on the server (no
ack). The next poll re-fetches it — useful when the ratchet temporarily
rejects a message because of out-of-order delivery.
- **`msgId/ciphertext` mismatch** is a relay-tampering canary. The Inbox
client recomputes the hash and emits `inbox.message_decrypt_failed`
*without* acking, so an operator can investigate before the blob
silently expires.
- **Network failure** on PUT keeps the entry in the local queue with an
`attempts` counter; default cap is 10 retries before the entry is
dropped (configurable via `maxAttempts`).
---
## 3. Threat model — what the relay actually sees
| Knows | Doesn't know |
| -------------------------------------------------- | ----------------------------------------- |
| Recipient address (path parameter) | Recipient real identity (it's pseudonymous) |
| Sender's per-PUT signing public key | The mapping sender-pubkey → real identity |
| Number of blobs queued for an address | Plaintext content |
| Approximate ciphertext size | Sender-recipient pair beyond bytes-pari |
| Per-blob TTL (in the row's `expires_at`) | The ratchet/X3DH state |
### Privacy posture
- **Sender-recipient graph leaks at the byte-pari level.** A passive
observer of the relay (or its DB dump) can correlate sender pubkey ↔
recipient address ↔ blob size. Mitigations:
- Recipients can use **address hashes** instead of human-readable
addresses (the address grammar accepts any `[a-zA-Z0-9][a-zA-Z0-9:_\-.]{0,255}`,
so `sha256(real-address || salt)` works).
- Senders can rotate their per-PUT signing key per session; the relay
only verifies the signature and never persists the key.
- **TTL leaks reachability.** A sender's PUT silently dropping after 7
days is itself a signal. Operators can normalize TTLs (clamp every
PUT to a fixed 7-day window) to flatten this.
- **Operator can DoS a recipient** by deleting their queue. Mitigation:
recipient ack happens *after* successful decrypt, so a malicious
delete just forces re-send by the original sender.
### What the relay can NOT do
- **Read plaintext** — the ratchet/AEAD layers run client-side.
- **Forge a sender** — every PUT is Ed25519-signed by the sender's
per-PUT key; the relay rejects bad signatures with 401.
- **Inject a foreign blob** — the recipient client recomputes
`sha256(ciphertext)` and refuses anything that doesn't match the
stored `msgId`.
- **Replay an old PUT** — the signed `signedAt` field has a ±5-minute
window (matches the prekey-server's policy); replays past that window
return 409.
### Storage-DoS
`maxBlobBytes` (default 1 MiB) caps a single PUT.
`maxBlobsPerAddress` (default 1000) caps the recipient's queue depth —
PUTs past the cap return 400 with a structured `inbox.quota_rejected`
event so operators can alert. Combine with per-IP rate limits at the
edge (the built-in token bucket is in-memory and not multi-instance).
---
## 4. Wire reference
All bodies are JSON. Multi-byte fields are base64-standard encoded.
### `POST /v1/inbox/register` (TOFU)
```json
{
"address": "bob",
"signingKey": "<base64 Ed25519 public key>",
"signedAt": 1716057600000,
"signature": "<base64 Ed25519 signature over canonical body>"
}
```
- 200 — registered (or idempotent re-register with same key).
- 401 — different key already owns this address, or signature failed.
### `POST /v1/inbox/:address` (PUT blob)
```json
{
"senderSigningKey": "<base64 sender Ed25519 public key>",
"msgId": "<lowercase hex sha256(ciphertext)>",
"ciphertext": "<base64 wire bytes from encodeEnvelope()>",
"ttlSeconds": 604800,
"signedAt": 1716057600000,
"signature": "<base64 sender signature>"
}
```
- 200 with `{ msgId, receivedAt, idempotent: false }` — first store.
- 200 with `idempotent: true` — duplicate PUT folded into the first row.
- 400 — `msgId` mismatch, ciphertext too big, or address quota exceeded.
- 401 — bad signature or stale `signedAt`.
- 404 — recipient address never registered.
### `POST /v1/inbox/:address/fetch` (signed challenge)
```json
{
"address": "bob",
"sinceCursor": 0,
"signedAt": 1716057600000,
"signature": "<base64 recipient signature>"
}
```
Returns:
```json
{
"blobs": [
{
"msgId": "<hex>",
"ciphertext": "<base64>",
"receivedAt": 1716057601234,
"expiresAt": 1716662401234
}
],
"cursor": 1716057601234,
"hasMore": false
}
```
Pass the returned `cursor` as `sinceCursor` next time. Pages cap at
`fetchPageLimit` (default 100); keep calling with the new cursor while
`hasMore === true`.
### `DELETE /v1/inbox/:address/:msgId` (signed ack)
Body:
```json
{
"address": "bob",
"msgId": "<hex>",
"signedAt": 1716057600000,
"signature": "<base64 recipient signature>"
}
```
- 200 with `{ ok: true }` — row removed.
- 200 with `{ ok: false }` — row was already gone (also idempotent).
- 401 — recipient signature failed.
### `DELETE /v1/inbox/register/:address`
Same auth shape as ack. Drops every queued blob.
---
## 5. Acceptance test mapping
| V3.6 spec criterion | Test |
| ---------------------------------------------------------- | -------------------------------------------------------------- |
| Async delivery without online overlap | `lifecycle.test.ts → "100 messages delivered…"` |
| DB-dump leaks no plaintext / sender-recipient graph | Server stores only `address \|\| msgId \|\| ct \|\| expires_at`; verified by `routes.test.ts` schema asserts |
| Replay PUT with same `msgId` is idempotent | `routes.test.ts → "idempotent on duplicate ciphertext"` |
| Restart preserves blobs | `lifecycle.test.ts → "persistence across restart"` + sqlite-store reopen |
| Bit-flip on stored ciphertext rejected on the client | `lifecycle.test.ts → "Tamper resistance"` + client `client.test.ts → "tamper detection"` |

348
docs/key-transparency.md Normal file
View File

@@ -0,0 +1,348 @@
# Key Transparency (V3.12)
> **Status:** v0.4.0+ — opt-in. Server runs unchanged when KT is off.
> Klient ignorerer proof-felt når KT-config mangler. Trygg å rulle ut
> uten klient-update.
Shades prekey-server er sannhetskilde for hvilket bundle som er
publisert for hver adresse. Uten Key Transparency (KT) kan en
ondsinnet eller kompromittert server bytte ut et bundle uten at noen
oppdager det. Med KT er hvert bundle som leveres **kryptografisk
forpliktet** i en append-only Merkle log som tredjeparts-witnesses kan
auditere.
Se også `docs/V3.12-DESIGN.md` for designnotat med trusselmodell og
beslutningsspor.
---
## Hva KT garanterer
| Angrep | Detektert? |
|---|---|
| Server gir Bob feil bundle for `alice` | **Ja** — inklusjons-proof matcher ikke |
| Server gir Bob og Charlie ulike bundles for `alice` | **Ja** — witness-gossip ser to STH-er på samme `tree_size` |
| Server skriver om historikk for å skjule tidligere svik | **Ja** — konsistens-proof feiler |
| Server signerer "stale" STH for å holde et tidsvindu åpent | **Ja** — klient avviser STH eldre enn `maxStaleMs` (default 24t) |
| Førstegangs-impersonering av en helt ny adresse | **Nei** — KT ser bare etter at adressen er i loggen, ikke at den er "riktig" person. Bruk V3.3 (fingerprint-gate) + V3.10 (social recovery) for det. |
---
## Operatør: skru på KT
KT er opt-in og krever:
1. **Et Ed25519 signing-keypair** for STH-signering. Dette er
*operatørens* nøkkel og må beskyttes som en code-signing-key.
2. **En persistent KTLogStore.** I produksjon: `PostgresKTLogStore`.
I test/dev: `MemoryKTLogStore`.
3. **At klienter pinner samme `logPublicKey`** OOB (typisk via
`Shade.config`-bundling i appen).
### Generere signing-key
```sh
bun run scripts/generate-kt-key.ts > kt-key.json
```
(Eller kjør manuelt: `crypto.generateEd25519KeyPair()` i en Bun REPL.)
Lagre `privateKey` i operatørens secret-store. Distribuér `publicKey`
til klienter sammen med app-config.
### Boot serveren med KT
```ts
import { createPrekeyServerWithKT } from '@shade/server';
import { PostgresPrekeyStore, PostgresKTLogStore } from '@shade/storage-postgres';
import { SubtleCryptoProvider } from '@shade/crypto-web';
const crypto = new SubtleCryptoProvider();
const prekeyStore = await PostgresPrekeyStore.create(process.env.DATABASE_URL!);
const ktStore = await PostgresKTLogStore.create(process.env.DATABASE_URL!);
const { app, kt } = await createPrekeyServerWithKT({
crypto,
store: prekeyStore,
keyTransparency: {
store: ktStore,
signingPrivateKey: loadFromSecret('SHADE_KT_SIGNING_PRIVATE_KEY'),
signingPublicKey: loadFromSecret('SHADE_KT_SIGNING_PUBLIC_KEY'),
heartbeatIntervalMs: 10 * 60 * 1000, // default; 0 = off
},
});
export default { port: 3900, fetch: app.fetch };
```
Når KT er på blir disse rutene tilgjengelig:
| Route | Hva den returnerer |
|---|---|
| `GET /v1/kt/log_id` | `{ logId, publicKey }` (begge base64) |
| `GET /v1/kt/sth` | Siste signed tree head |
| `GET /v1/kt/sth/:treeSize` | Historisk STH for et bestemt tree_size |
| `GET /v1/kt/consistency?from=N1&to=N2` | Konsistens-proof N1 → N2 |
Bundle-fetch (`GET /v1/keys/bundle/:address`) får nå et `ktProof`-felt
i responsen.
### Migrasjon fra ikke-KT
KT er bakoverkompatibel:
1. Skru på KT-config i serveren. Restart.
2. Eksisterende klienter ignorerer proof-feltene (`ktProof`, `ktSth`).
3. Etter hvert som klienter oppgraderes med KT-config (`mode: 'observe'`),
begynner de å verifisere.
4. Når øko-systemet er vant til det, eskalér klienter til
`'observe-strict'` for å avvise prekey-server-svar uten proof.
Ved første boot scanner KT-tjenesten ikke automatisk eksisterende
prekey-store-tilstand inn i loggen. **Re-registrering** av eksisterende
adresser (dvs. en `POST /v1/keys/register`-runde fra hver klient) er
det som backfiller. For et større deployment: anbefalt at en operatør
varsler brukerne om å re-registrere innen et tidsvindue. Klienter som
ikke re-registrerer vil feile `observe-strict`-fetch til de får ny key
fra peer.
---
## Klient: skru på KT
```ts
import { createShade } from '@shade/sdk';
const shade = await createShade({
prekeyServer: 'https://shade.example.com',
address: 'alice',
keyTransparency: {
mode: 'observe-strict', // eller 'observe'
logPublicKey: KT_LOG_PUBLIC_KEY_BASE64, // eller Uint8Array
maxStaleMs: 24 * 60 * 60 * 1000, // default 24t
},
});
```
`shade.getKTWitness()` returnerer `LightWitness`-instansen som
samler observerte STH-er. Bruk `.compare(otherSth)` for manuell
gossip-sjekk mot peers.
### `mode: 'observe'`
- Verifiserer proof når serveren leverer det.
- Skipper verifisering hvis `ktProof` mangler i bundle-respons.
- Anbefalt under første utrulling der ikke alle klienter har
re-registrert ennå.
### `mode: 'observe-strict'`
- Krever proof på hver `200`-respons. Mangler proof → kast `KTVerificationError`.
- Krever proof på hver `404`-respons også (for absence/tombstone-pinning).
- Anbefalt produksjons-modus når KT-økosystemet er etablert.
---
## Witness / auditor
`@shade/key-transparency` eksporterer `LightWitness`. Et CLI-verktøy
eller backend-job kan bruke den slik:
```ts
import { LightWitness } from '@shade/key-transparency';
import { SubtleCryptoProvider } from '@shade/crypto-web';
const crypto = new SubtleCryptoProvider();
const witness = new LightWitness({
crypto,
logPublicKey: KT_LOG_PUBLIC_KEY,
fetcher: {
async fetchLatestSTH() {
const r = await fetch('https://shade.example.com/v1/kt/sth');
return r.json();
},
async fetchConsistencyProof(from, to) {
const r = await fetch(`https://shade.example.com/v1/kt/consistency?from=${from}&to=${to}`);
return r.json();
},
},
});
// Poll periodically (e.g. every 5 minutes)
setInterval(async () => {
try {
const sth = await witness.pollOnce();
console.log(`Observed STH: tree_size=${sth.treeSize}, root=${Buffer.from(sth.rootHash).toString('hex').slice(0, 16)}`);
} catch (err) {
console.error('Witness alarm:', err);
// Send to PagerDuty / Slack / whatever
}
}, 5 * 60 * 1000);
```
Witness-koden detekterer:
- **Stale STH** — server publiserer ikke nye STH-er i tide.
- **Split view** — to STH-er ved samme `tree_size` med ulik root.
- **Re-write** — konsistens-proof feiler.
- **Wrong key** — `log_id` matcher ikke pinnet `logPublicKey`.
---
## Operatørkost (estimat)
For et deployment med:
- **100k registrerte adresser**
- **1 identitets-rotasjon per år** per bruker
- **52 replenish per år** (én i uka, *ikke* committed til loggen — bare register/delete er)
| Ressurs | Per år | Kommentar |
|---|---|---|
| Log-rader | ~100k | bare register/delete |
| Lagring (leaves+index) | ~25 MB | base64-kodet |
| STH-rows | ~52k | én per heartbeat (10 min) |
| STH-storage | ~7 MB | |
| CPU per STH | ~1ms | Ed25519-signing er trivielt |
| Bundle-fetch overhead | <2ms | inkluderer audit-path-bygg |
**Backup:** behandle KT-tabellene som "kan ikke gjenopprettes" data —
`shade_kt_leaves` har en database-trigger som forbyr UPDATE/DELETE i
PostgreSQL-implementasjonen. Backup-strategi:
- Daglig full backup av `shade_kt_*` tabellene.
- WAL-shipping anbefalt (tap < 60 s i verste fall).
- **Test recovery** kvartalsvis. Recovery-prosedyre står under.
---
## Recovery
### Scenario 1 — STH-signing-key tapt eller kompromittert
Loggen forblir konsistent (alle gamle STH-er er allerede signert), men
nye STH-er kan ikke signeres med samme key.
**Steg:**
1. Generer ny Ed25519-keypair.
2. Skriv inn et "rotation breaks here"-leaf i loggen (operasjon = 0x03
på en spesiell `__log__`-adresse) — operasjonen er rent
informativ, men gjør rotasjonen synlig i tree.
3. Re-konfigurer serveren med ny key. Restart.
4. Server publiserer en ny STH; den vil ha en ny `log_id` (siden
`log_id = SHA-256(publicKey)`).
5. **Klienter må eksplisitt akseptere ny key.** Inntil de pinner ny
`logPublicKey`, vil deres `LightWitness` kaste
`KTLogIdMismatchError`. Operatør publiserer ny key OOB med
"rotated from `<gammel logId>`"-melding signert med gammel key
(siste handling før gammel key zeroizes).
### Scenario 2 — KT-database korrumpert / tapt før backup
Dette er **det verste utfallet**. Loggen er per design ikke
gjenopprettbar — å "rekonstruere" den fra prekey-store ville bryte
selve invarianten KT lover.
**Steg:**
1. Stopp serveren.
2. Deklarer en "log-restart event" via offentlig kanal (status-side,
release-notes, Twitter, etc.) — inkluder timestamp, tapte tree_size
(siste backup-bare snapshot om mulig), og ny `logPublicKey`.
3. Generer ny KT-keypair (ikke bruk gamle).
4. Boot serveren tom (tom `shade_kt_*` tabell). Første STH er fra
`tree_size = 0`.
5. Be brukerne om å re-registrere identitetene sine. Klientene vil
trigge V3.3 fingerprint-gate på første re-meldings-flyt etterpå
siden rotasjons-fingerprintet endres.
6. Auditor-organisasjoner kan publisere "vi observerte gammel log
inntil tree_size N, ny log starter på 0 fra T+0" — dette gir
sluttbruker mulighet til å vurdere hvor stort hullet er.
**Beskytt mot dette:** WAL-shipping + off-site backup. Aldri kjør KT
med kun én database-instans uten replicas.
### Scenario 3 — Witness oppdager split-view
Witness kaster `KTSplitViewError` i `LightWitness.observe()` eller
`KTVerificationError` i transport. Dette betyr:
- Operatøren har enten
(a) hatt en software-bug som signerte to ulike STH-er ved samme
tree_size, eller
(b) er kompromittert / ondsinnet.
**Operatør-handling:**
1. Pause `POST /v1/keys/register`, `DELETE`, og bundle-fetch
umiddelbart (return 503).
2. Audit `shade_kt_sths` — hvis du finner to rader med samme
`tree_size` men ulik `root_hash`, har serveren gjort feil. Dette er
alvorlig — finn root cause før du fortsetter.
3. Kommuniser ut til brukerne. Forutsett at en angriper har vært
inne; trigge en bredere reset (recovery scenario 2) hvis det er
mistanke om tampering.
**Klient-handling:**
- `LightWitness` har allerede holdt brukeren tilbake.
- SDK-en surfacer feilen som `KTSplitViewError` til app-koden.
- App-en bør vise advarsel: "Operatørens server kan ikke verifiseres.
Avstå fra sending av sensitive meldinger inntil videre."
---
## Sikkerhets-anbefalinger
1. **Kjør minst én uavhengig witness.** Operatørens egen "witness"
teller ikke — det må være en separat prosess på separate
infrastruktur eid av en separat aktør (community-medlem, security
firm, e.l.).
2. **Pin `logPublicKey` i app-binær eller signert config.** En
man-in-the-middle som kan bytte både prekey-server og KT-key
fanges ikke av KT alene.
3. **Loggrotasjon krever menneske-i-løkken.** Ikke automatiser
key-rotation for KT — den eksplisitte breaking-event er en feature.
4. **`maxStaleMs` bør samsvare med din heartbeat.** 24t default tåler
en heartbeat-pause på opptil et døgn; senk til 14t hvis du har
strenge krav til friskhet.
5. **`observe-strict` bør være standard når økosystemet er etablert.**
Default `'observe'` er en operasjonell overgangsmodus, ikke et
sluttmål.
---
## Kjente begrensninger
- **Federation mellom flere prekey-servere** er ikke støttet i V3.12.
Hver Shade-deployment har én log eller ingen.
- **Sparse Merkle tree for adresse-index** brukes ikke i V3.12 —
fravær-proof er foreløpig nabopar-bevis. <100 KB ved 100k adresser
er akseptabelt; sparse tree blir relevant fra ~10M+ adresser.
- **One-time prekey-rotasjon committes ikke** til loggen. OTP er
ephemerale og inkludering ville støy-fylle loggen. Dette betyr at
en server som svarer med riktig identitet men feil OTP fanges ikke
av KT — forsvar mot dette ligger i V3.3 fingerprint-gate (samme
identitet) + sesjons-etableringens X3DH (feil OTP gir feil shared
secret → første melding feiler decryption).
---
## Tester og test-vektorer
- `packages/shade-key-transparency/tests/` — RFC 6962-kompatibel
Merkle-log + STH + index-proofs (58 tests).
- `packages/shade-server/tests/kt.test.ts` — server-integrasjon (8
tests).
- `packages/shade-transport/tests/kt-transport.test.ts` — klient-
verifikasjon over HTTP (4 tests).
- `packages/shade-transport/tests/kt-split-view-e2e.test.ts`
V3.12-akseptanse split-view-deteksjon (3 tests).
- `packages/shade-sdk/tests/kt.test.ts` — SDK-config + witness wiring
(3 tests).
Totalt 76 tester dedikert til KT.

193
docs/observability.md Normal file
View File

@@ -0,0 +1,193 @@
# Observability v2 — OpenTelemetry tracing
Shade ships an opt-in OpenTelemetry layer that wraps `TransferEngine`,
`ShadeSessionManager`, the prekey HTTP routes, and `@shade/files`
op-handlers in distributed spans. The layer is **off by default** and
PII-safe by construction — span attributes never include peer addresses,
plaintext payloads, or exact byte counts.
This complements the always-on Prometheus metrics exposed by
`@shade/server` and the structural events emitted by `@shade/core`. Use
metrics for aggregate counters and histograms, tracing for per-request
causality and tail-latency hunting.
---
## Quick start
```ts
import { trace } from '@opentelemetry/api';
import { withTracer } from '@shade/observability';
import { createShade } from '@shade/sdk';
// Use the OTel SDK of your choice (NodeSDK + OTLP exporter, Honeycomb,
// Sentry's OTel adapter, …) to register a tracer provider on the
// `@opentelemetry/api` global. Then:
const tracer = trace.getTracer('my-app');
const shade = await createShade({
prekeyServer: 'https://shade.example.com',
storage: 'sqlite:/data/shade.db',
observability: withTracer(tracer, { sample: 0.1 }),
});
```
The hook propagates automatically to:
- `ShadeSessionManager.encrypt` / `.decrypt` (per-peer mutex acquisition,
ratchet step).
- `TransferEngine.upload` / accepted incoming downloads (lane count,
retry count, partition mode).
- `@shade/files` op-handlers (per request, with op + result).
For the prekey server pass the hook to `createPrekeyRoutes`:
```ts
import { createPrekeyRoutes } from '@shade/server';
import { withTracer } from '@shade/observability';
const app = createPrekeyRoutes(store, crypto, {
observability: withTracer(tracer),
});
```
---
## Off-by-default semantics
`withTracer()` returns a no-op hook — the SDK never starts spans — when
**any** of the following are true:
1. The `tracer` argument is `undefined`/`null`.
2. The `SHADE_OTEL_ENABLED` env-var is not set to `1` or `true`. Override
with `withTracer(tracer, { force: true })`, or override the var name
with `withTracer(tracer, { envVar: 'MY_VAR' })`.
3. The configured `sample` rate is `0`.
Per-span sampling (`sample: 0.1` = 10 %) keeps trace volume bounded in
production. Default is `1` (sample everything when the hook is active).
---
## PII policy — what is safe to log, and what isn't
| Category | Status | Why |
|----------|--------|-----|
| **Peer hash** (`shade.peer.hash`) | ✅ allowed | 8-hex-char pseudonym derived via SHA-256. Stable across spans for a given address but does not expose the address itself. |
| **Bytes bin** (`shade.bytes.bin`) | ✅ allowed | One of `≤4KB`, `464KB`, `64KB1MB`, `110MB`, `10100MB`, `100MB1GB`, `≥1GB`. Coarse enough to mask file-size fingerprinting. |
| **Lane count** (`shade.lane.count`) | ✅ allowed | Snapped to `{1, 4, 16, 64}`. |
| **Retry count** (`shade.retry.count`) | ✅ allowed | Integer. |
| **Error code** (`shade.error.code`) | ✅ allowed | `SHADE_*` stable string code — never the full message, which may interpolate user input. |
| **Op kind** (`shade.op`) | ✅ allowed | `list`, `read`, `write`, `custom:foo`, etc. |
| **Route template** (`shade.route`) | ✅ allowed | `/v1/keys/bundle/:address` — the template, never the resolved path. |
| **HTTP status** (`shade.http.status`) | ✅ allowed | Integer status code. |
| **Partition mode** (`shade.partition`) | ✅ allowed | `range` or `round-robin`. |
| **Direction** (`shade.direction`) | ✅ allowed | `upload` or `download`. |
| Plaintext peer addresses | ❌ forbidden | Use `peerHash()`. |
| Plaintext message/file payloads | ❌ forbidden | Encryption boundary — never log. |
| Exact byte counts | ❌ forbidden | Use `bytesBin()`. |
| User identifiers (email, DID, `device:UUID`) | ❌ forbidden | Treat as PII. |
The full attribute-key allow-list is exported from `@shade/observability`
as `ATTR_*` constants. Plug-in authors who want to attach their own tags
should pass each `(key, value)` through `safeAttribute()`, which throws
`UnsafeAttributeError` for any key/value pair that looks like the
forbidden categories above (heuristics: `@`, `device:`, `did:`, key
fragments such as `peer.address` / `bytes.exact`, oversized strings).
---
## Span surface
### `shade.session.encrypt` / `shade.session.decrypt`
Wraps each per-peer `encrypt`/`decrypt` call. Includes the time spent
waiting on the per-peer mutex (`shade.lock.wait_ms`) — handy for
diagnosing ratchet contention under load.
### `shade.transfer.upload` / `shade.transfer.upload.resume`
Wraps an outbound stream transfer end-to-end. Attributes: `peer.hash`,
`bytes.bin`, `lane.count`, `partition`, `retry.count`, `result`,
`error.code`.
### `shade.transfer.download`
Started when the consumer calls `incoming.accept(...)`, ended when the
transfer completes, aborts, or fails an integrity check. Same attribute
set as upload.
### `shade.prekey.request`
One span per HTTP request handled by `@shade/server`'s prekey routes.
Attributes: `route` (the template), `http.status`, `error.code` on
failure. The address path-parameter is **never** placed on the span.
### `shade.files.op`
One span per `@shade/files` RPC. Attributes: `peer.hash`, `op` (the
resolved op kind, e.g. `read` or `custom:foo`), `bytes.bin` (estimated
plaintext size, binned), `result`, `error.code`.
---
## Recording & testing
`@shade/observability` ships a deterministic in-memory recorder for
unit tests:
```ts
import { createRecorder } from '@shade/observability';
const rec = createRecorder();
const shade = await createShade({ ..., observability: rec });
// … exercise code under test …
const hits = rec.scanForPII(['alice@example.com', 'plaintext-secret']);
expect(hits).toHaveLength(0);
```
The Shade test suite runs this recorder over every documented entry
point — see
`packages/shade-observability/tests/integration-pii.test.ts` and
`packages/shade-transfer/tests/observability.test.ts`. Any new
instrumentation must keep the suite green.
---
## Performance characteristics
- With OTel **off** (default): every Shade hook resolves to the shared
`NOOP_HOOK` instance. The cost is one function call + an object
allocation that V8 hoists out in the steady state — measured at
< 1 % overhead vs the pre-V3.4 baseline in the upload roundtrip
benchmark.
- With OTel **on**: cost depends entirely on the configured exporter.
Use `sample: 0.1` (or smaller) on hot paths in production.
---
## Adding new instrumentation
1. Identify a logical operation worth a span — typically anything that
crosses a network/disk boundary or contends on a lock.
2. Add an `observability?: ObservabilityHook` to the relevant config
surface, default to `NOOP_HOOK`.
3. Name the span `shade.<area>.<op>` to keep cardinality bounded.
4. Set attributes via the `ATTR_*` constants from
`@shade/observability`. **Never** introduce a new attribute key
without a PII review — if you must, run the value through
`safeAttribute()`.
5. Add a test that exercises the new instrumentation under the
`createRecorder()` recorder and asserts no PII leaks.
---
## Migration
Previous versions had no tracing — only Prometheus metrics. Adding the
`observability` field to existing configs is fully backwards-compatible
and never required. The `SHADE_OTEL_ENABLED` gate ensures forgetting to
flip the env-var in production won't surprise anyone with unexpected
overhead.

308
docs/recovery.md Normal file
View File

@@ -0,0 +1,308 @@
# Social Key Recovery (`@shade/recovery`)
V3.10 closes the biggest UX hole in any E2EE system: **"What happens
if I lose my phone?"**. Shade's social-recovery flow lets a user
designate `n` guardians (family / friends / co-workers) at setup time
such that any threshold-many `k` of them can together restore the
user's identity onto a new device — without any single guardian
being able to do it alone, and without the prekey server ever seeing
the recovered key material.
The whole flow ships entirely over existing 1:1 Shade sessions; no
server-side recovery agent, no escrow service, no "cloud guardian".
---
## Threat model recap
| # | Adversary | Recovered? |
|---|-----------|------------|
| 1 | Coalition of ≤ k-1 guardians | **No** (information-theoretic, by Shamir construction) |
| 2 | Prekey server alone | **No** (server only relays Double-Ratchet ciphertext) |
| 3 | Single malicious guardian who forges a share | **Detected** — AES-GCM tag mismatch on the backup blob; `requestRecovery` exhaustively tries threshold-sized subsets and rejects when none authenticate |
| 4 | Social engineering (impersonator calls a guardian) | **Mitigated, not eliminated** — guardians MUST OOB-confirm the new device's safety number before approving (see `<RecoveryApprove />`) |
| 5 | Compromised guardian device | **Out of scope** — see "Guardian compromise" below |
| 6 | Compromised primary device at setup time | **Out of scope** — recovery only protects the device; if setup material is exfiltrated, all bets are off |
---
## Setup
### What the user does
1. Pick `n` guardians from their existing peers.
2. Pick a threshold `k` (typically `⌈n/2⌉ + 1` to avoid pure-majority
dominance but still survive losing one or two).
3. Run `setupRecovery(...)`.
4. Print / record a **recovery card** with:
- The user's own address
- `setupId`
- `k` and `n`
- The list of guardian addresses
- Setup-time safety number
The recovery card is the only piece of state the user must remember
out-of-band (or store in a password manager). Without it, the user
cannot drive recovery on a new device — the new device needs to know
who the guardians are.
### What happens cryptographically
```text
recoveryKey = random(32 bytes)
backupBlob = Shade.exportBackup(passphrase = "shade-rk:" + base64url(recoveryKey),
knownAddresses = [...])
shares[i] = Shamir-split(recoveryKey, k, n)
```
For each guardian `i`:
```text
share-deposit envelope:
shadeRecovery: 1
type: "share-deposit"
flowId, setupId, originalAddress
threshold (k), guardianCount (n), shareIndex (i)
shareBytes: base64url( encodeShare(shares[i]) )
backupBlob: Shade.exportBackup output (identical for every guardian)
setupFingerprint, createdAt
```
The envelope rides through `Shade.send` like any other plaintext —
double-ratchet encrypted, AAD-bound, replay-safe.
The `recoveryKey` is **zeroized** on the primary device immediately
after the split returns. The primary therefore retains nothing
except `setupId` and the public roster.
### What each guardian stores
Per (`originalAddress`, `setupId`):
```text
{
shareIndex, // 1..n
shareBytes, // base64url-encoded Shamir share
backupBlob, // identical for every guardian
setupFingerprint, // for sanity-checks at recovery time
guardianCount, threshold,
receivedAt
}
```
The guardian's app provides a `RecoveryStore` implementation. The
package ships `MemoryRecoveryStore` for tests and small one-shot
demos; production guardian apps MUST supply a persistent store
(IndexedDB, AsyncStorage, SQLite, etc.). See "Persistence
recommendations" below.
---
## Recovery
### What the user does on the new device
1. Boot a fresh Shade with a temporary identity.
2. Read the recovery card.
3. In the recovery widget, type / paste:
- `originalAddress`
- `setupId`
- `threshold`
- The guardian roster
4. Read the new device's safety number (the widget displays it
prominently) to each guardian over a side channel — phone call,
in person, whatever they trust.
5. Wait for `≥ k` guardians to approve.
### What happens cryptographically
For each guardian, the new device sends:
```text
recovery-request envelope:
shadeRecovery: 1
type: "recovery-request"
flowId, originalAddress, setupId
requesterFingerprint (= safety number of the temporary identity)
requestedAt
```
Each guardian's `attachGuardian` handler:
1. Looks up its stored deposit by `(originalAddress, setupId)`. If
missing, replies with `share-decline` (`reason = "unknown setup"`).
2. Invokes the `approve` callback with the requester's address +
fingerprint + the original device's setup-time fingerprint. The
callback is the **OOB-confirmation gate** — it MUST require an
explicit user click after they verified the fingerprint. The
`<RecoveryApprove />` widget enforces this with a two-checkbox
gate.
3. On approve → ships `share-grant`. On reject → ships
`share-decline` with a short reason.
The new device collects grants, and as soon as `k` arrive:
1. Combines the `k` shares via Lagrange interpolation at `x = 0` to
reconstruct `recoveryKey`.
2. Derives `passphrase = "shade-rk:" + base64url(recoveryKey)`.
3. Calls `Shade.importBackup(backupBlob, passphrase)` — the
AES-GCM tag in the blob authenticates the reconstruction. **A
forged share is detected here.**
4. If a guardian forged a share, `importBackup` throws. The
reconstruction loop then tries every other threshold-sized subset
of grants until one authenticates (the V3.10 acceptance criterion
"no coalition of (k-1) guardians can rebuild the secret" is the
safety invariant; the AEAD authenticates which subset is
honest).
5. If every subset fails, `RecoveryReconstructionError` is raised
and the user is told that at least one guardian is malicious.
After `importBackup` succeeds, the new device hosts the original
identity and immediately calls `Shade.rotate()` to retire the
recovery-recovered key material from the conversation graph (the
old session keys persisted in the backup blob are now considered
"compromised — used for recovery").
> **The `Shade.beforeBackupImport` gate fires automatically.**
> Without a registered handler the SDK falls back to TOFU-with-warning
> (consistent with the V3.3 contract). Production apps SHOULD register
> a handler that pops the user one more confirmation before the
> identity rotates.
---
## Acceptance criteria status
- [x] **3-of-5 recovery works end-to-end on two separate Shade
instances.** See `tests/integration.test.ts`.
- [x] **No coalition of (k-1) guardians can reconstruct
`recoveryKey`.** Property test asserts this with `fast-check`
across random k/n configurations.
See `tests/shamir.test.ts` and
`tests/adversarial.test.ts`.
- [x] **Guardian-side widget requires fingerprint-confirmation
before sending.** `<RecoveryApprove />` enforces a
two-checkbox gate; `tests/adversarial.test.ts` exercises
both the matching-OOB and rejecting-OOB code paths.
---
## Persistence recommendations
The `RecoveryStore` interface is intentionally small (4 methods).
Pick the implementation that fits your platform:
| Platform | Suggested backing store |
|--------------------------|----------------------------------------|
| Browser (PWA) | IndexedDB (one object store, idb) |
| Browser (extension) | `chrome.storage.local` |
| React Native | AsyncStorage (with crypto-protected blob) |
| Bun / Node server | SQLite via `@shade/storage-sqlite` extension table OR a side file |
| Android (native) | Room / EncryptedSharedPreferences |
Whatever you pick, the records ARE NOT secret on their own — without
threshold-many other guardians' shares they're useless — but they
should still be stored encrypted-at-rest like any other Shade state.
Do not commit them to plaintext logs or network-replicated state.
---
## Guardian-UX guide
### How many guardians?
| n | Survives | Comment |
|---|----------|---------|
| 3, k=2 | 1 lost guardian | Minimum useful — one device away from danger |
| 5, k=3 | 2 lost guardians | Sweet spot for most users |
| 7, k=4 | 3 lost guardians | Suitable when you genuinely have 7+ trustworthy people |
| n=k | 0 lost | DO NOT USE — single point of failure |
The widget defaults to `k = ⌈n/2⌉` which is liberal but
collusion-resistant for `n ≥ 3`. Apps targeting paranoid users may
want to bump that to `⌈2n/3⌉`.
### Replacing a guardian
If a guardian dies, loses their device permanently, or you no longer
trust them:
1. Pick a replacement.
2. Run `setupRecovery` again with the new roster — this generates a
fresh `setupId` and a fresh `recoveryKey`. The old shares become
garbage (no guardian set can use them, because the
`backupBlob` is different).
The widget records the new `setupId` on the recovery card. Treat
this as a hard rotation; the user MUST re-record the card.
### Guardian health checks
Periodically (the V3.10 plan suggests a quarterly prompt), the user
should confirm each guardian is still reachable. Any guardian who
can't be reached in two consecutive prompts SHOULD trigger a
re-setup with a fresh roster. The widget UX track is to be added in
a follow-up release; the primitive is in place.
---
## Wiring example
```ts
import {
setupRecovery,
attachGuardian,
requestRecovery,
MemoryRecoveryStore,
} from '@shade/recovery';
// On the primary device:
const result = await setupRecovery({
shade,
guardians: ['bob', 'carol', 'dan', 'eve', 'faythe'],
threshold: 3,
deliver: async (to, envelope) => {
// wire to your app's existing message-delivery layer
await myMessageOutbox.send(to, envelope);
},
});
console.log(result.setupId);
// On each guardian device:
const stop = attachGuardian({
shade,
store: myPersistentStore, // see "Persistence" above
approve: async (ctx) => {
// Show ctx.requesterFingerprint to the user.
// Block until they confirm OOB and click "Release share".
return await myUI.askApproval(ctx);
},
deliver: myMessageOutbox.send,
});
// On the new device:
const recovered = await requestRecovery({
shade: temporaryShade, // fresh identity for now
originalAddress: 'alice',
setupId: 'sid-from-recovery-card',
threshold: 3,
guardians: ['bob', 'carol', 'dan', 'eve', 'faythe'],
deliver: myMessageOutbox.send,
onProgress: (p) => myUI.showProgress(p),
});
// `temporaryShade` now hosts the original identity.
```
---
## Out of scope (V3.10)
- **Cloud guardian / Shade-operated recovery agent.** Explicit
non-goal; the spec rejects any centralized component that can
recover on its own.
- **Auto-distribution.** The user must explicitly pick guardians.
- **Multi-share-per-guardian.** Each guardian holds exactly one
share. Apps that need redundancy should bump `n`, not give the
same guardian multiple shares.
- **Guardian ZK-proofs of liveness.** A guardian who refuses to
respond is treated as offline; we don't try to compel them.

View File

@@ -0,0 +1,173 @@
# Feature Request — Public accessor for the device's identity public key
**To**: Shade SDK team
**From**: Dispatch (browser-based Shade consumer)
**Target**: Shade SDK 4.4.x (or whichever release vehicle fits)
**Priority**: medium — unblocks real per-device fingerprint binding at
enrollment time; consumers ship with placeholder keys until then
Thanks for shipping `@shade/storage-indexeddb` so quickly — that unblocked
Dispatch's Slice 2.5 (persistent browser identity). During integration we
hit one more gap that's worth raising as a separate FR.
---
## Summary
Expose the local device's Ed25519 identity public key as a public accessor
on `Shade`, so applications can hand it to their own backend at enrollment
time for per-device verification, audit, or peer-fingerprint computation.
Today the SDK exposes `myAddress` and `fingerprint`, but the underlying
identity public key — the cryptographic root that everything else binds
to — is reachable only via the private `this.storage.getIdentityKeyPair()`
call inside `Shade`. Consumers building enrollment flows have no way to
hand the real key over.
## Problem
A common pattern in Shade-using apps is:
1. Browser device generates Shade identity (via `createShade`)
2. User enters an enrollment token
3. Browser POSTs to its backend: `{ token, address, identityPublicKey, ... }`
4. Backend records the device, computes/stores a safety number from the
identity key, opens a Shade session
Step 3 today has nowhere to get a real `identityPublicKey` from. Dispatch
currently sends `crypto.getRandomValues(new Uint8Array(32))` formatted as
hex, with this comment in the source:
```ts
/**
* The browser submits a hex public-key field at enroll time so the schema
* stays stable. Wiring this to the SDK-generated identity key requires a
* Shade SDK addition (no public accessor for the raw identity key today).
*/
export function generatePlaceholderIdentityPublicKey(): string { ... }
```
The backend stores this placeholder but cannot verify anything against it,
because it's not actually the device's key. Real cryptographic binding is
deferred until the SDK exposes the underlying key.
## Why `fingerprint` isn't sufficient
`Shade.fingerprint` returns a 12-groups-of-5-digits safety-number string
designed for human side-channel comparison. That's the right output for a
"compare these digits with your friend" UX, but it's a derived format, not
the raw key. Backends that want to:
- Store the key for later signature verification
- Compute their own safety-number representation (Dispatch uses
`deterministicSafetyNumber(localAddr, peerAddr)` based on the raw bytes)
- Re-derive the fingerprint after an identity rotation
…all need access to the raw 32-byte Ed25519 public key.
## Proposed API
Add a single async accessor on `Shade`:
```ts
class Shade {
/**
* The local device's Ed25519 identity public key (32 bytes).
*
* Stable for the lifetime of the identity. After `rotateIdentity()`
* this returns the new key; the old key is preserved in retired-
* identities storage for the configured grace period.
*/
get identityPublicKey(): Promise<Uint8Array>;
}
```
Internally:
```ts
get identityPublicKey(): Promise<Uint8Array> {
if (!this.initialized) throw new Error('Not initialized');
return this.storage.getIdentityKeyPair().then((kp) => {
if (!kp) throw new Error('Identity not yet generated');
return kp.publicKey;
});
}
```
If returning a getter that produces a Promise feels off, the equivalent
method form is fine:
```ts
async getIdentityPublicKey(): Promise<Uint8Array> { ... }
```
Either shape works for consumers — pick whichever matches existing SDK
conventions.
## Alternative considered: object accessor
Returning an object would leave room to expose other identity-related
fields later without a breaking change:
```ts
get identity(): Promise<{
address: string;
publicKey: Uint8Array;
fingerprint: string;
// future: registrationId, createdAt, etc.
}>;
```
This would mildly duplicate `myAddress` + `fingerprint`, but consolidates
identity-related accessors. Not a blocker for shipping the simpler
single-purpose accessor — just flagging the option in case the SDK is
considering broader API ergonomics.
## What this unblocks
For Dispatch specifically:
- Real device identity binding at enrollment (replaces the placeholder
`crypto.randomUUID()`-derived hex bytes the backend currently stores)
- Server-side `computePeerSafetyNumber()` can use the real key instead of
the deterministic-from-address stand-in (`shade-identity-provider.ts:151`)
- Future signature-based device authentication (sign a challenge with the
device's identity key during enrollment) becomes possible without
another SDK round
For other Shade consumers:
- Any app that hands a device key to its own backend for enrollment —
multi-device pairing flows, contact verification UIs, push-notification
targeting — gets an actual key to work with
## Acceptance criteria
1. New accessor exposed on `Shade` (getter or async method, SDK's
preference)
2. Returns the 32-byte Ed25519 identity public key
3. Returns the **current** key after `rotateIdentity()`
4. Throws (or resolves to a clear error) when called before
`initialize()` completes
5. Documented alongside `myAddress` and `fingerprint` in the SDK reference
6. One unit test in `shade-sdk` confirming the returned bytes match what
`storage.getIdentityKeyPair()` holds
7. Mentioned in the changelog under the next release
## Out of scope
- **Private key access** — consumers should never need it; signature
operations go through `shade.send()` etc. Don't expose the secret half.
- **Cross-peer key lookup** — getting *another* peer's identity key is a
separate concern (related to peer-verification storage), not what this
FR is about. This is strictly the local device's own key.
- **Format conversions** — base64/hex/PEM helpers don't belong in the SDK.
Consumers can encode the raw bytes however their wire format requires.
## Why this can't be done in consumer-land
The identity keypair is generated by `MemoryStorage` / `SQLiteStorage` /
`IndexedDBStorage` and consumed by `ShadeSessionManager`. Consumers can't
reach into either layer without breaching the SDK's encapsulation. A
public accessor is the only path that doesn't require monkey-patching
private fields.

View File

@@ -1,9 +1,9 @@
<!DOCTYPE html> <!DOCTYPE html>
<html lang="no"> <html lang="en">
<head> <head>
<meta charset="utf-8" /> <meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" /> <meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Shade — ende-til-ende kryptering som modul</title> <title>Shade — end-to-end encryption as a module</title>
<link rel="preconnect" href="https://fonts.googleapis.com" /> <link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin /> <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link href="https://fonts.googleapis.com/css2?family=Fraunces:ital,opsz,wght@0,9..144,400;0,9..144,600;0,9..144,700;1,9..144,400&family=Source+Serif+4:ital,opsz,wght@0,8..60,400;0,8..60,600;1,8..60,400&display=swap" rel="stylesheet" /> <link href="https://fonts.googleapis.com/css2?family=Fraunces:ital,opsz,wght@0,9..144,400;0,9..144,600;0,9..144,700;1,9..144,400&family=Source+Serif+4:ital,opsz,wght@0,8..60,400;0,8..60,600;1,8..60,400&display=swap" rel="stylesheet" />
@@ -364,32 +364,32 @@
<header> <header>
<h1>Shade</h1> <h1>Shade</h1>
<p class="lede"> <p class="lede">
En gjenbrukbar modul for <strong style="color: var(--text); font-weight: 600;">ende-til-ende-kryptert</strong> kommunikasjon i egne appermed samme type protokoll som brukes i Signal. A reusable module for <strong style="color: var(--text); font-weight: 600;">end-to-end encrypted</strong> communication in your own appsusing the same kind of protocol as Signal.
</p> </p>
<div class="badge-row"> <div class="badge-row">
<span class="badge">X3DH</span> <span class="badge">X3DH</span>
<span class="badge">Double Ratchet</span> <span class="badge">Double Ratchet</span>
<span class="badge">TypeScript</span> <span class="badge">TypeScript</span>
<span class="badge">Plattformagnostisk crypto</span> <span class="badge">Platform-agnostic crypto</span>
</div> </div>
</header> </header>
<section id="hva"> <section id="what">
<h2>Hva gjør prosjektet?</h2> <h2>What does the project do?</h2>
<p> <p>
<strong>Shade</strong> er et monorepo som implementerer sikker meldingskryptering mellom to parter (for eksempel nettleser og server, eller to klienter). Meldingene er kryptert slik at transportlaget (HTTP, WebSocket, e.l.) bare ser uleselige bytes — ikke innholdet. <strong>Shade</strong> is a monorepo that implements secure messaging encryption between two parties (for example browser and server, or two clients). Messages are encrypted so the transport layer (HTTP, WebSocket, etc.) only sees opaque bytes — not the content.
</p> </p>
<div class="callout"> <div class="callout">
<strong>Kjerneideen:</strong> Du bygger inn <code>ShadeSessionManager</code> (fra <code>@shade/core</code>) sammen med en <code>CryptoProvider</code> (f.eks. Web Crypto i nettleser/Bun) og lagring. Deretter kan du kalle <code>encrypt</code> / <code>decrypt</code> per motpart, akkurat som i demo-koden <code>demo.ts</code>. <strong>Core idea:</strong> Embed <code>ShadeSessionManager</code> (from <code>@shade/core</code>) together with a <code>CryptoProvider</code> (e.g. Web Crypto in the browser/Bun) and storage. Then call <code>encrypt</code> / <code>decrypt</code> per peer, just like in the demo code <code>demo.ts</code>.
</div> </div>
<p> <p>
rste melding til noen ny inneholder nøkkelavtale (X3DH). Etterpå bruker hver melding <em>Double Ratchet</em>: nye meldingsnøkler og periodiske DH-steg gir <strong>forward secrecy</strong> (gamle meldinger overlever ikke nøkkellekkasje) og <strong>post-compromise security</strong> (systemet «helbreder» seg over tid etter kompromittering). The first message to someone new performs key agreement (X3DH). After that each message uses the <em>Double Ratchet</em>: fresh message keys and periodic DH steps provide <strong>forward secrecy</strong> (past messages do not survive key compromise) and <strong>post-compromise security</strong> (the system “recovers” over time after a compromise).
</p> </p>
</section> </section>
<section id="pakker"> <section id="packages">
<h2>Pakkene (hvordan det henger sammen)</h2> <h2>Packages (how they fit)</h2>
<div class="tabs" role="tablist" aria-label="Pakkeoversikt"> <div class="tabs" role="tablist" aria-label="Package overview">
<button type="button" class="tab-btn" role="tab" id="tab-core" aria-selected="true" aria-controls="panel-core">shade-core</button> <button type="button" class="tab-btn" role="tab" id="tab-core" aria-selected="true" aria-controls="panel-core">shade-core</button>
<button type="button" class="tab-btn" role="tab" id="tab-crypto" aria-selected="false" aria-controls="panel-crypto">shade-crypto-web</button> <button type="button" class="tab-btn" role="tab" id="tab-crypto" aria-selected="false" aria-controls="panel-crypto">shade-crypto-web</button>
<button type="button" class="tab-btn" role="tab" id="tab-proto" aria-selected="false" aria-controls="panel-proto">shade-proto</button> <button type="button" class="tab-btn" role="tab" id="tab-proto" aria-selected="false" aria-controls="panel-proto">shade-proto</button>
@@ -397,117 +397,117 @@
<button type="button" class="tab-btn" role="tab" id="tab-server" aria-selected="false" aria-controls="panel-server">shade-server</button> <button type="button" class="tab-btn" role="tab" id="tab-server" aria-selected="false" aria-controls="panel-server">shade-server</button>
</div> </div>
<div id="panel-core" class="tab-panel active" role="tabpanel" aria-labelledby="tab-core"> <div id="panel-core" class="tab-panel active" role="tabpanel" aria-labelledby="tab-core">
<p style="margin-top:0"><strong>Protokollen.</strong> X3DH, Double Ratchet, sesjonstyper, feiltyper. Ingen plattformkrypto her — bare grensesnittet <code>CryptoProvider</code>.</p> <p style="margin-top:0"><strong>The protocol.</strong> X3DH, Double Ratchet, session shapes, errors. No platform crypto hereonly the <code>CryptoProvider</code> interface.</p>
<ul> <ul>
<li><code>ShadeSessionManager</code> — høynivå-API: <code>initialize</code>, <code>createPreKeyBundle</code>, <code>initSessionFromBundle</code>, <code>encrypt</code>, <code>decrypt</code></li> <li><code>ShadeSessionManager</code> — high-level API: <code>initialize</code>, <code>createPreKeyBundle</code>, <code>initSessionFromBundle</code>, <code>encrypt</code>, <code>decrypt</code></li>
<li>Symmetrisk kryptering: <strong>AES-256-GCM</strong> med AAD fra ratchet-header</li> <li>Symmetric encryption: <strong>AES-256-GCM</strong> with AAD from the ratchet header</li>
</ul> </ul>
</div> </div>
<div id="panel-crypto" class="tab-panel" role="tabpanel" aria-labelledby="tab-crypto" hidden> <div id="panel-crypto" class="tab-panel" role="tabpanel" aria-labelledby="tab-crypto" hidden>
<p style="margin-top:0"><strong>Implementasjon av crypto for web/Bun/Node</strong> via SubtleCrypto — X25519, Ed25519, HKDF, HMAC, tilfeldige bytes.</p> <p style="margin-top:0"><strong>Crypto implementation for web/Bun/Node</strong> via SubtleCrypto — X25519, Ed25519, HKDF, HMAC, random bytes.</p>
<ul> <ul>
<li>Gjør det mulig å bruke <code>shade-core</code> i nettleser og i servere som støtter Web Crypto</li> <li>Lets you use <code>shade-core</code> in the browser and on servers that support Web Crypto</li>
<li>Kommentarer i koden peker på fremtidig Android (f.eks. Tink) som egen provider</li> <li>Comments in source point toward future Android (e.g. Tink) as a separate provider</li>
</ul> </ul>
</div> </div>
<div id="panel-proto" class="tab-panel" role="tabpanel" aria-labelledby="tab-proto" hidden> <div id="panel-proto" class="tab-panel" role="tabpanel" aria-labelledby="tab-proto" hidden>
<p style="margin-top:0"><strong>Binært wire-format</strong> for meldinger: versjon + type + lengdeprefiksede felt (big-endian).</p> <p style="margin-top:0"><strong>Binary wire format</strong> for messages: version + type + length-prefixed fields (big-endian).</p>
<ul> <ul>
<li>Type <code>0x01</code> = PreKeyMessage, <code>0x02</code> = RatchetMessage</li> <li>Type <code>0x01</code> = PreKeyMessage, <code>0x02</code> = RatchetMessage</li>
<li>Brukes når du vil serialisere <code>ShadeEnvelope</code> effektivt over nettet</li> <li>Use when you want to serialize <code>ShadeEnvelope</code> efficiently on the wire</li>
</ul> </ul>
</div> </div>
<div id="panel-transport" class="tab-panel" role="tabpanel" aria-labelledby="tab-transport" hidden> <div id="panel-transport" class="tab-panel" role="tabpanel" aria-labelledby="tab-transport" hidden>
<p style="margin-top:0"><strong>Transportadaptere</strong>ikke selve krypteringen, men hvordan du sender bytes (f.eks. fetch eller WebSocket).</p> <p style="margin-top:0"><strong>Transport adapters</strong>not encryption itself, but how you move bytes (e.g. fetch or WebSocket).</p>
<ul> <ul>
<li>Kobler applikasjonen din til den kanalen du allerede bruker</li> <li>Hooks your application to the channel you already use</li>
</ul> </ul>
</div> </div>
<div id="panel-server" class="tab-panel" role="tabpanel" aria-labelledby="tab-server" hidden> <div id="panel-server" class="tab-panel" role="tabpanel" aria-labelledby="tab-server" hidden>
<p style="margin-top:0"><strong>Prekey-server (Hono)</strong>lagrer offentlige nøkler slik at Alice kan starte samtale mens Bob er «offline».</p> <p style="margin-top:0"><strong>Prekey server (Hono)</strong>stores public keys so Alice can start a conversation while Bob is “offline.</p>
<ul> <ul>
<li><code>POST /v1/keys/register</code> — registrer identitet + bundle</li> <li><code>POST /v1/keys/register</code> — register identity + bundle</li>
<li><code>GET /v1/keys/bundle/:address</code>hent bundle (forbruker én engangsnøkkel om tilgjengelig)</li> <li><code>GET /v1/keys/bundle/:address</code>fetch bundle (consumes one-time prekey when available)</li>
<li><code>POST /v1/keys/replenish</code>etterfyll engangsnøkler</li> <li><code>POST /v1/keys/replenish</code>replenish one-time prekeys</li>
</ul> </ul>
</div> </div>
</section> </section>
<section id="nokler"> <section id="keys-in-brief">
<h2>Nøkler i korthet</h2> <h2>Keys at a glance</h2>
<div class="accordion" id="key-acc"> <div class="accordion" id="key-acc">
<div class="acc-item"> <div class="acc-item">
<button type="button" class="acc-trigger" aria-expanded="true" aria-controls="acc-identity" id="btn-identity"> <button type="button" class="acc-trigger" aria-expanded="true" aria-controls="acc-identity" id="btn-identity">
Identitetsnøkkel (langvarig) Identity key (long-term)
</button> </button>
<div class="acc-panel" id="acc-identity" role="region" aria-labelledby="btn-identity"> <div class="acc-panel" id="acc-identity" role="region" aria-labelledby="btn-identity">
<strong>Ed25519</strong> brukes til å signere den «signerte prekeyen». <strong>X25519</strong> brukes i Diffie-Hellman i X3DH og i ratchet. Én identitet per enhet/bruker i typisk oppsett. <strong>Ed25519</strong> signs the “signed prekey. <strong>X25519</strong> is used in DiffieHellman in X3DH and in the ratchet. One identity per device/user is typical.
</div> </div>
</div> </div>
<div class="acc-item"> <div class="acc-item">
<button type="button" class="acc-trigger" aria-expanded="false" aria-controls="acc-spk" id="btn-spk"> <button type="button" class="acc-trigger" aria-expanded="false" aria-controls="acc-spk" id="btn-spk">
Signert prekey (mediumvarig, roteres) Signed prekey (medium-lived, rotated)
</button> </button>
<div class="acc-panel" id="acc-spk" role="region" aria-labelledby="btn-spk" hidden> <div class="acc-panel" id="acc-spk" role="region" aria-labelledby="btn-spk" hidden>
X25519-nøkkel som publiseres og signeres med identiteten. Mottaker verifiserer signaturen før DH. I koden anbefales rotasjon omtrent hver 17 dag. An X25519 key that is published and signed by the identity. The recipient verifies the signature before DH. The codebase recommends rotation on the order of 17 days.
</div> </div>
</div> </div>
<div class="acc-item"> <div class="acc-item">
<button type="button" class="acc-trigger" aria-expanded="false" aria-controls="acc-otpk" id="btn-otpk"> <button type="button" class="acc-trigger" aria-expanded="false" aria-controls="acc-otpk" id="btn-otpk">
Engangsnøkler (one-time prekeys) One-time prekeys
</button> </button>
<div class="acc-panel" id="acc-otpk" role="region" aria-labelledby="btn-otpk" hidden> <div class="acc-panel" id="acc-otpk" role="region" aria-labelledby="btn-otpk" hidden>
Valgfrie, men viktige for ekstra sikkerhet: hver hentet bundle kan inkludere én engangsnøkkel som slettes etter bruk (<code>processPreKeyMessage</code> fjerner den fra lager). Gir bedre beskyttelse mot visse angrep når mange klienter kobler til samme mottaker. Optional but important for stronger security: each fetched bundle can include one one-time key that is removed after use (<code>processPreKeyMessage</code> clears it from storage). Improves protection against certain attacks when many clients connect to the same recipient.
</div> </div>
</div> </div>
</div> </div>
</section> </section>
<section id="flyt"> <section id="flow-demo">
<h2>Interaktiv flyt: fra null til kryptert melding</h2> <h2>Interactive flow: zero to encrypted message</h2>
<p> <p>
Klikk «Neste» for å gå gjennom rekkefølgen slik Shade er bygget. Dette speiler <code>ShadeSessionManager</code> og demoen i repoet. Click Next step” to walk through the sequence as Shade builds it. This mirrors <code>ShadeSessionManager</code> and the demo in the repo.
</p> </p>
<div class="flow"> <div class="flow">
<h3>Sesjon og meldinger</h3> <h3>Sessions and messages</h3>
<div class="flow-steps" id="flow-steps"></div> <div class="flow-steps" id="flow-steps"></div>
<div class="flow-actions"> <div class="flow-actions">
<button type="button" class="btn" id="flow-next">Neste steg</button> <button type="button" class="btn" id="flow-next">Next step</button>
<button type="button" class="btn btn-secondary" id="flow-reset">Start på nytt</button> <button type="button" class="btn btn-secondary" id="flow-reset">Start over</button>
</div> </div>
</div> </div>
</section> </section>
<section id="x3dh-ratchet"> <section id="x3dh-ratchet">
<h2>X3DH og Double Ratchet (kort forklart)</h2> <h2>X3DH and Double Ratchet (brief)</h2>
<p> <p>
<strong>X3DH</strong> løser problemet «jeg vil snakke med Bob nå, men Bob svarer ikke før senere». Bob legger ut en <em>prekey bundle</em> serveren. Alice henter den, gjør 3 eller 4 DH-operasjoner (avhengig av om engangsnøkkel brukes), og deriverer en felles rot-nøkkel som begge kan rekonstruere uten at serveren kjenner hemmeligheten. <strong>X3DH</strong> solves “I want to talk to Bob now, but Bob may not reply until later”. Bob publishes a <em>prekey bundle</em> on the server. Alice fetches it, runs 3 or 4 DH operations (depending on whether a one-time key is used), and derives a shared root key both parties can reconstruct — without the server learning the secret.
</p> </p>
<p> <p>
<strong>Double Ratchet</strong> bruker den roten som startpunkt. For hver melding (eller ved nye DH-nøkler) avledes nye nøkler; meldinger på ledningen er AES-GCM med autentisering (AAD binder kryptoteksten til ratchet-header). Protokollen håndterer også meldinger i feil rekkefølge innenfor grenser (<code>MAX_SKIP</code>). <strong>Double Ratchet</strong> uses that root as a starting point. For each message (or when new DH keys spin), keys are derived; on the wire payloads are AES-GCM with authentication (AAD binds ciphertext to the ratchet header). The protocol also tolerates messages arriving out of order within limits (<code>MAX_SKIP</code>).
</p> </p>
<p> <p>
Spesifikasjoner fra Signal (engelsk): <a href="https://signal.org/docs/specifications/x3dh/" target="_blank" rel="noopener">X3DH</a> · <a href="https://signal.org/docs/specifications/doubleratchet/" target="_blank" rel="noopener">Double Ratchet</a>. Signal specifications (English): <a href="https://signal.org/docs/specifications/x3dh/" target="_blank" rel="noopener">X3DH</a> · <a href="https://signal.org/docs/specifications/doubleratchet/" target="_blank" rel="noopener">Double Ratchet</a>.
</p> </p>
</section> </section>
<section id="gjenbruk"> <section id="reuse">
<h2>Bruke Shade i flere prosjekter</h2> <h2>Using Shade across projects</h2>
<p> <p>
Tenk på Shade som tre lag du kan kombinere etter behov: Treat Shade as three layers you combine as needed:
</p> </p>
<ol> <ol>
<li><strong>Core + crypto-provider + storage</strong>selve E2EE-motoren (kan kjøre i klient eller serverprosess som skal dekryptere).</li> <li><strong>Core + crypto provider + storage</strong>the E2EE engine itself (runs in a client or a server process that must decrypt).</li>
<li><strong>Proto</strong>når du vil ha kompakt binær serialisering.</li> <li><strong>Proto</strong>when you want compact binary serialization.</li>
<li><strong>Transport + prekey-server</strong>når du vil standardisere nøkkelutveksling og kanaler.</li> <li><strong>Transport + prekey server</strong>when you want standardized key discovery and channels.</li>
</ol> </ol>
<p> <p>
Referansekjøring: <code class="mono">bun demo.ts</code> i rotmappen viser frontend/backend-flyt med minnelager og ekte kryptoprimitiver. Reference path: <code class="mono">bun demo.ts</code> from the repo root shows a frontend/backend flow with memory storage and real crypto primitives.
</p> </p>
</section> </section>
<footer> <footer>
<p>Shade — oversikt generert som statisk HTML i <code class="mono">docs/shade-overview.html</code>. Åpne filen direkte i nettleseren eller server den statisk.</p> <p>Shade — overview written as static HTML in <code class="mono">docs/shade-overview.html</code>. Open the file directly in the browser or serve it as static assets.</p>
</footer> </footer>
</div> </div>
@@ -561,28 +561,28 @@
// Flow steps // Flow steps
var steps = [ var steps = [
{ {
title: "Initialiser klient", title: "Initialize client",
body: "Kall initialize(): last eller generer identitetsnøkkel (Ed25519 + X25519), registrationId og signert prekey.", body: "Call initialize(): load or generate the identity keys (Ed25519 + X25519), registrationId, and signed prekey.",
}, },
{ {
title: "Publiser prekey bundle", title: "Publish prekey bundle",
body: "Bygg bundle med createPreKeyBundle() / generateOneTimePreKeys() og registrer prekey-server (eller del ut av band for demo).", body: "Build a bundle with createPreKeyBundle() / generateOneTimePreKeys() and register it on the prekey server (or share out-of-band for a demo).",
}, },
{ {
title: "Start sesjon mot peer", title: "Start session with peer",
body: "Hent motpartens bundle, kjør initSessionFromBundle(address, bundle). X3DH kjører og ratchet-sesjon lagres i StorageProvider.", body: "Fetch the peer bundle, run initSessionFromBundle(address, bundle). X3DH runs and the ratchet session is stored in StorageProvider.",
}, },
{ {
title: "Første encrypt", title: "First encrypt",
body: "encrypt() returnerer ShadeEnvelope type 'prekey': inneholder ephemeral nøkkel, prekey-ID-er og første RatchetMessage (AES-GCM).", body: "encrypt() returns a ShadeEnvelope of type 'prekey': includes ephemeral keys, prekey IDs, and the first RatchetMessage (AES-GCM).",
}, },
{ {
title: "Motpart decrypt", title: "Peer decrypt",
body: "decrypt() PreKeyMessage: gjenskaper samme root key, initReceiverSession, ratchetDecrypt — plaintext ut.", body: "decrypt() on PreKeyMessage: restores the same root key, initReceiverSession, ratchetDecrypt — plaintext out.",
}, },
{ {
title: "Videre meldinger", title: "Further messages",
body: "Neste kall til encrypt() gir type 'ratchet'. DH-ratchet steg gir nye kjeder og forbedret sikkerhet over tid.", body: "The next encrypt() calls yield type 'ratchet'. DH ratchet steps rotate chains and improve security over time.",
}, },
]; ];

View File

@@ -0,0 +1,191 @@
# Feature Request — `@shade/storage-indexeddb`
**To**: Shade SDK team
**From**: Dispatch (browser-based Shade consumer)
**Target**: Shade SDK 4.3.x (or whichever release vehicle fits)
**Priority**: blocks all browser-based Shade apps from achieving session
persistence across tab refresh
---
## Summary
Ship an official IndexedDB-backed `StorageProvider` adapter as a new
workspace package `@shade/storage-indexeddb`, so browser-based Shade SDK
consumers can persist identity, prekeys, sessions, and peer-verification
state across tab refresh and browser restart — the same way `@shade/storage-sqlite`
does for Node and `@shade/storage-postgres` does for server deployments.
## Problem
Today the Shade SDK ships three storage paths:
| spec | adapter | environment |
| --------------------------------- | ------------------------ | ---------------- |
| `"memory"` | `MemoryStorage` (in-SDK) | tests, ephemeral |
| `"sqlite:/path"` | `@shade/storage-sqlite` | Node |
| `{ type: 'postgres', url: '…' }` | `@shade/storage-postgres`| Node servers |
There is **no browser-storage option**. The only way to run Shade in a
browser today is `storage: "memory"`, which means:
- Identity keypair regenerates on every page load
- Sessions reset → re-enrollment after every refresh
- `getLocalRegistrationId()` returns a fresh value → `device:${id}`
address changes → server-side device record orphaned every reload
This forces every browser-based Shade app to either (a) accept the broken
UX, or (b) build their own `StorageProvider` from scratch — duplicating
~25 methods × N consumers, with no shared conformance test surface.
The right place to solve this is at the SDK level, exactly mirroring how
SQLite and Postgres are handled.
## Proposed package
`packages/shade-storage-indexeddb/` — modeled directly after
`packages/shade-storage-sqlite/`. Same package shape, same test layout,
same `@shade/core`-only runtime dependency surface.
### Public API
```ts
// @shade/storage-indexeddb
export class IndexedDBStorage implements StorageProvider {
/**
* Open (or create) the IndexedDB database. Idempotent — repeated calls
* with the same dbName return a connection sharing the same object stores.
*/
static async create(opts?: { dbName?: string }): Promise<IndexedDBStorage>;
/**
* Cleanly close the underlying connection. Future calls will reopen.
* Called by Shade.shutdown() when consumers register cleanup.
*/
async close(): Promise<void>;
// ─── all StorageProvider methods (identity, prekeys, sessions,
// retired identities, peer verifications, optional stream-state) ───
}
```
`dbName` defaults to something like `"shade"`. Consumers like Dispatch
will pass distinct names per app (`"dispatch-dashboard-shade"`,
`"dispatch-host-ui-shade"`) so DevTools' IndexedDB inspector groups them
sensibly, even though origin-isolation already makes the data isolated.
### SDK integration
`@shade/sdk` `resolveStorage()` gets a fourth branch:
```ts
if (typeof spec === 'object' && spec.type === 'indexeddb') {
const moduleId = '@shade/storage-indexeddb';
const mod = (await import(moduleId)) as {
IndexedDBStorage: { create(opts: { dbName?: string }): Promise<StorageProvider> };
};
return mod.IndexedDBStorage.create({ dbName: spec.dbName });
}
```
Dynamic import keeps `@shade/storage-indexeddb` an optional dependency,
matching the Postgres pattern — Node-only consumers don't need to install
a browser-only adapter.
Consumer surface:
```ts
const shade = await createShade({
prekeyServer: 'https://…/shade-prekey',
storage: { type: 'indexeddb', dbName: 'my-app-shade' },
address: 'device:user@example.com', // optional — falls back to device:${registrationId}
});
```
## Implementation guidance (non-prescriptive)
- **IDB wrapper**: suggest `idb` (Jake Archibald's thin wrapper, well-typed,
zero deps). Avoid Dexie or idb-keyval — we want full schema control to
match the SQL adapters' explicit schemas.
- **Object-store layout**: one store per StorageProvider category
(`identity`, `signedPreKeys`, `oneTimePreKeys`, `sessions`,
`trustedIdentities`, `retiredIdentities`, `peerVerifications`,
`streamStates`). Keypaths match the natural keys (`keyId`, `address`,
`streamId`).
- **Schema version**: integer, bumped on every shape change. Migrations
in `db.upgrade(...)` callback. Document schema-history alongside
the SQLite schema.
- **Concurrency**: IndexedDB transactions are auto-committing — the adapter
must keep operations within a single transaction where SQL adapters do.
Particular care for `bumpPeerIdentityVersion` (atomic read-modify-write).
- **Stream-state methods**: implement them. Browser apps will increasingly
use `@shade/transfer` for large file resume, and parity with SQLite's
capabilities matters.
## Test expectations
Mirror `packages/shade-storage-sqlite/tests/`:
- `indexeddb-storage.test.ts` — full StorageProvider surface (identity,
sessions, trusted identities, retired identities)
- `indexeddb-prekey-store.test.ts` — signed + one-time prekey lifecycle
- `peer-verifications.test.ts` — verification CRUD + identity-version
bumping invariants
- `indexeddb-stream-state.test.ts` (if stream-state is implemented)
Use **`fake-indexeddb`** for the Node test environment — it's the
established standard, supports the v3 spec, and lets us run IDB tests
in `bun test` / `vitest` / `jest` without a real browser.
If/when Shade gains a shared `StorageProvider` conformance test suite,
this adapter should consume it directly. Until then, follow the SQLite
adapter's per-method coverage style.
## Acceptance criteria
1. `@shade/storage-indexeddb` published at version 4.3.0 (or whichever
matches the next Shade release)
2. `@shade/sdk` `resolveStorage()` resolves `{ type: 'indexeddb', dbName? }`
via dynamic import
3. Full StorageProvider conformance in tests (identity, prekeys,
sessions, retired identities, peer verifications, stream-state)
4. Documented in Shade docs alongside SQLite/Postgres adapters
5. README example showing browser-app integration
6. Bundle-size note: dynamic-imported IDB module shouldn't pull crypto
dependencies — adapter should be ≤ ~10 KB minified+gzipped
## Out of scope (deferred)
- **Encryption-at-rest** for the IDB contents — separate work item; should
match the deviceKey-AES-GCM pattern Shade already uses for `secretEnc`
in `PersistedStreamState`. This adapter ships unencrypted-at-rest in v1
(consistent with SQLite), with the encryption layer added uniformly to
all adapters later.
- **Cross-tab BroadcastChannel sync** — IDB is shared across same-origin
tabs already; concurrent writes work via IDB transactions. Real-time
notification across tabs (e.g. "session was rotated in another tab") is
a separate concern, not storage-adapter scope.
- **Quota handling** — IDB quota for a Shade keystore is far below realistic
browser quotas. If it ever becomes relevant, add a `QuotaExceededError`
observability hook then.
## Why this can't be done in consumer-land
Building this in Dispatch (or any other consumer) would mean:
- 25+ method `StorageProvider` re-implementation per consumer
- No shared conformance tests
- Schema drift across consumers — each would invent its own object-store
shape, blocking any future cross-app data import/export
- Every Shade SDK update that adjusts `StorageProvider` would force every
consumer to track and patch independently
`@shade/storage-sqlite` and `@shade/storage-postgres` are part of the SDK
for the same reason. IndexedDB belongs alongside them.
## What unblocks
Shipping this unblocks Dispatch's "Slice 2.5" — persistent enrollment
across browser refresh, which today is the largest QA-friction point in
the dev loop. Any future browser-Shade consumer (web dashboard,
contact-list app, browser-extension messenger) gets persistence for free.

160
docs/storage-encryption.md Normal file
View File

@@ -0,0 +1,160 @@
# At-Rest Storage Encryption (V3.2)
**Status:** Implemented in `@shade/storage-encrypted` 0.4.0
**Adresses:** THREAT-MODEL §4 — Compromised device storage
Shade's default `SQLiteStorage` and `PostgresStorage` write private keys and
session state to disk *unencrypted* — the threat model assumes the DB lives
inside a trusted environment. For deployments that need defence in depth,
`@shade/storage-encrypted` adds opt-in at-rest encryption: a stolen DB file
alone yields no usable private key material.
## At a glance
```ts
import { KeyManager, EncryptedSQLiteStorage } from '@shade/storage-encrypted';
const km = await KeyManager.open({
kind: 'passphrase',
passphrase: process.env.SHADE_STORAGE_PASSPHRASE!,
salt: loadSaltFromDisk(), // 16+ bytes, persisted alongside the DB
});
const storage = await EncryptedSQLiteStorage.open({
dbPath: '/data/shade-client.db',
keyManager: km,
});
// Use it exactly like SQLiteStorage — implements the same StorageProvider.
const manager = new ShadeSessionManager(crypto, storage);
```
## What is encrypted
Per-row AEAD over the sensitive payload of every row:
| Table | Encrypted |
|--------------------------------|-----------|
| `identity_enc` | the entire keypair (4× 32-byte keys) |
| `config_enc` | `registrationId` |
| `signed_prekeys_enc` | full `SignedPreKey` (incl. private half) |
| `one_time_prekeys_enc` | full `OneTimePreKey` |
| `sessions_enc` | the Double-Ratchet `SessionState` JSON |
| `trusted_identities_enc` | the trusted peer identity key |
| `retired_identities_enc` | full retired keypair |
| `stream_state_enc.ciphertext` | partition / lane / IO descriptor / streamSecret |
Routing fields on `stream_state_enc` (`stream_id`, `direction`,
`peer_address`, `status`, timestamps) stay plaintext so `listActiveStreamStates()`
remains an indexed query.
## Cryptographic design
```
masterKey (passphrase / keychain / app-injected)
├─ HKDF-SHA-256("shade-storage-v1") → storageKey (32 bytes)
│ └─ HKDF-SHA-256(storageKey, "shade-field-v1:{table}:{column}") → fieldKey (32 bytes)
└─ Used (transitively) for fingerprint checks
```
For each encrypted blob:
- `nonce = HKDF(fieldKey, "shade-row-nonce-v1:{table}:{pk}")[..12]`
deterministic per (key, row), safe because the per-(table, column)
fieldKey is unique. AES-GCM nonce reuse is catastrophic only if the
*same* key is reused with the *same* nonce on different plaintexts;
here every (key, row) pair has a unique nonce.
- `aad = "shade-aad-v1|{table}|{column}|{pk}"` — binds the ciphertext
to its row identity so a row swap or column move triggers decrypt
failure.
- `wire = nonce(12) || ciphertext || tag(16)` — stored as a single
`BLOB`/`BYTEA` column.
## Key sources
`KeyManager.open(...)` accepts three sources:
1. **Passphrase + KDF** — scrypt over `(passphrase, salt)`. Default
parameters: `N=2^17, r=8, p=1, dkLen=32` (~250 ms on a modern laptop).
The salt MUST be persisted alongside the DB (e.g. `<db>.salt`).
2. **OS keychain** — via `@shade/keychain`. Backends:
- macOS: `security` CLI (Keychain).
- Linux: `secret-tool` (libsecret).
- Windows: PowerShell + `CredentialManager` module.
No native deps; `createIfMissing: true` generates and stores a fresh
32-byte key.
3. **App-injected** — caller supplies a 32-byte raw key. Most flexible;
plug your own KMS / HSM / Vault path here.
Wrong-passphrase detection is built in: a fingerprint of the storageKey
is persisted in `shade_meta_enc` on first open and compared on every
subsequent open. A mismatch raises with a clear error — never silently
writing under the wrong key.
## Migration
CLI:
```bash
# Encrypt an existing unencrypted DB (atomic per row, .bak written first).
shade migrate-storage \
--key-source passphrase \
--passphrase "$SHADE_STORAGE_PASSPHRASE" \
--salt-file /data/shade-client.db.salt
# Validate without writing.
shade migrate-storage ... --dry-run
# Keychain mode.
shade migrate-storage --key-source keychain \
--keychain-service shade.storage --keychain-account default
# Inject a raw key (e.g. from your KMS).
shade migrate-storage --key-source injected \
--key-hex "$(cat ~/.shade/storage.key.hex)"
```
The migration is *resumable*: re-running it on a partially-migrated DB
re-writes the same rows under the same key (idempotent). On clean
completion, the unencrypted tables are dropped (use `--keep-original`
to preserve them).
## Rotation
```bash
shade rotate-storage-key \
--key-source passphrase --passphrase "$OLD_PASS" \
--new-key-source passphrase --new-passphrase "$NEW_PASS" \
--new-salt-file /data/shade-client.db.salt.new
```
Reads each encrypted row under the old key, re-seals under the new key.
The DB stays online; brief read-after-write inconsistency for in-flight
readers is acceptable for the supported deployments (CLI tools,
single-process servers). On completion the fingerprint is updated and
the old key no longer opens the DB.
## What this does *not* protect
Even with at-rest enabled:
- A live process holds the storageKey and fieldKeys in memory. An attacker
who can dump process memory (`/proc/<pid>/mem`, swap, hibernation,
coredump) recovers the keys.
- Swap is not encrypted by Shade. Use an encrypted swap device.
- The `.bak` file produced during migration is plaintext during the
migration window. Treat it like the original DB and store securely.
- Lost master key = lost DB. V3.10 (Social Recovery) is the long-term
mitigation.
See `THREAT-MODEL.md` §4 for the full list, including the "with at-rest
enabled" boundary.
## Cross-implementation parity
`test-vectors/storage-encryption.json` pins KDF parameters, info strings,
nonce derivation, and AAD format. The Android implementation (V3.5) MUST
produce byte-identical outputs for the same inputs — covered by
`packages/shade-storage-encrypted/tests/test-vectors.test.ts`.

128
docs/streaming-sessions.md Normal file
View File

@@ -0,0 +1,128 @@
# Streaming Double-Ratchet sub-sessions (V4.11)
`ShadeStream` wraps individual frames on a long-lived, high-frequency,
often one-directional channel (e.g. a server→client console-log
WebSocket) in an **independent** Double Ratchet derived from — but never
mutating — an already-established parent Shade session.
This is the answer to Vyvern FR `shade-ws-streaming-ratchet.md`. It is a
first-class API, *not* the "documented contract that `send`/`receive` is
safe per-frame" fallback: the Double-Ratchet crypto was already safe for
that access pattern, but the `send`/`receive` wrapper layer was not
(per-frame keystore writes; a shared per-peer mutex and a single stored
session row coupling the stream to the HTTP path). `ShadeStream` keeps
the proven ratchet and fixes the wrapper.
## API
Transport-agnostic, exactly like `send`/`receive`: it emits/consumes
wire bytes; you own the WebSocket.
```ts
// Initiator (the side that calls openStream)
const stream = await shade.openStream(peerAddr);
ws.send(stream.handshakeFrame()); // → STREAM_OPEN
// first inbound WS frame is the peer's STREAM_OPEN_ACK:
await stream.handleHandshake(ackBytes); // stream now usable
ws.send(await stream.seal(utf8(logLine))); // outbound frame
onLog(await stream.open(inboundBytes)); // inbound frame
await stream.close(); // on ws close/error
// Responder
const stream = await shade.acceptStream(peerAddr, openBytes); // usable now
ws.send(stream.handshakeFrame()); // → STREAM_OPEN_ACK
// open()/seal() as above
```
Route inbound bytes with `inspectEnvelopeType()`:
`'stream-open' | 'stream-open-ack' | 'stream-frame'`.
## Seeding (no prekey-server round trip)
The stream root key is derived from an identity-bound **3-DH** exchange
— the X3DH pattern minus signed/one-time prekeys, because the peer's
identity is *already* mutually pinned by the parent session's TOFU. Two
ephemerals are exchanged inside the transport (`STREAM_OPEN` /
`STREAM_OPEN_ACK`); no prekey server is involved.
```
slotA = DH(initiatorEphemeral, responderIdentity) — authenticates responder
slotB = DH(initiatorIdentity, responderEphemeral) — authenticates initiator
slotC = DH(initiatorEphemeral, responderEphemeral) — ephemeral forward secrecy
SK = HKDF(ikm = slotA‖slotB‖slotC, salt = streamId, info = "ShadeStream/v1")
```
Both peers compute the identical three scalars regardless of role.
`SK` then bootstraps a textbook Double Ratchet by handing the
responder's ephemeral to `initSenderSession`/`initReceiverSession`
exactly the way X3DH hands its signed prekey to the ratchet — so
`ratchetEncrypt`/`ratchetDecrypt` and every guarantee they carry apply
unchanged.
## Security contract (answers FR R1R7)
- **R1 — same properties as `send`/`receive`.** Each frame is one
`ratchetEncrypt`/`ratchetDecrypt` over the *same* crypto as the HTTP
path: AES-256-GCM confidentiality, per-frame forward secrecy via the
one-way HMAC chain-key KDF with in-place zeroize of the spent chain
key, and replay/rewind rejection (a re-delivered or counter-rewound
frame fails closed). The handshake is mutually authenticated against
the identities the parent session already pinned.
- **R2 — one-directional resilience.** A long server→client burst with
no client traffic only advances the symmetric sending chain (no DH
step until the peer replies — standard Double Ratchet). Forward
secrecy holds per frame in this regime. Over an ordered transport
(WebSocket/TCP) zero keys are skipped per frame.
- **R3 — bounded memory.** Out-of-order arrivals are capped by the
ratchet's `MAX_SKIP` (1000) and `MAX_CACHED_SKIPPED_KEYS` (2000)
with oldest-key eviction. In-order delivery retains nothing. Verified
to stay at zero retained keys across a 5000-frame burst.
- **R4 — browser parity.** Identical API and guarantees in the browser
SDK: `ShadeStream` is on the same `Shade` class over the same
`CryptoProvider` (`SubtleCryptoProvider`), so the IndexedDB-backed
build behaves identically to the `sqlite:` server build. No storage
is touched at all (see R7), so the keystore backend is irrelevant.
- **R5 — independent lifecycle.** The stream ratchet is derived without
reading or writing the stored parent `SessionState`, runs on its own
private op-mutex (not the per-peer `send`/`receive` queues), and is
zeroized on `close()`. Opening, using for thousands of frames, and
closing a stream leaves the parent session byte-identical; the HTTP
path keeps working concurrently against the same peer. Each
`openStream` gets a fresh `streamId` and an independent root, so
concurrent streams to one peer never share key material.
- **R6 — wire framing.** `@shade/proto` defines `STREAM_OPEN` (0x31),
`STREAM_OPEN_ACK` (0x32), `STREAM_FRAME` (0x33). A `STREAM_FRAME`
carries one Double-Ratchet message via the exact ratchet inner codec
the HTTP path uses. One sealed logical frame ⇒ one self-delimiting
wire frame ⇒ one WS text/binary frame.
- **R7 — performance.** The ratchet lives **only in memory and is never
persisted**. There is therefore *zero* per-frame storage I/O — the
per-frame cost is exactly the symmetric KDF + one AES-GCM, the same
primitives the HTTP path runs. This is strictly better than the
"doubled CPU" the Vyvern roadmap budgeted, because the dominant cost
the naive `send`/`receive`-per-frame approach would have paid (a
`saveSession` keystore write per frame) is eliminated, not doubled.
Not persisting is also a *security* property, not a shortcut: writing
evolving per-frame ratchet secrets to disk would defeat forward
secrecy. A dropped/reconnected stream is re-opened with a fresh
handshake, never resumed.
## Double-Ratchet ordering note
A responder cannot `seal()` until it has `open()`ed at least one frame
from the initiator (standard Signal behaviour — the responder has no
sending chain until the first DH step). For a server-heavy stream
either make the bursty data sender the **initiator**, or have the
initiator send one priming frame immediately after the handshake.
## Tests
- `packages/shade-core/tests/stream.test.ts` — handshake agreement,
frame round-trips, 5000-frame one-directional burst (bounded skipped
keys + forward-secrecy zeroize), parent-session independence (R5),
replay/rewind rejection, mutual authentication against pinned
identities, `close()` zeroize/idempotence.
- `packages/shade-proto/tests/stream-wire.test.ts` — wire round-trips
and type-tag/length rejection for all three stream frame kinds.
</content>
</invoke>

370
docs/streams.md Normal file
View File

@@ -0,0 +1,370 @@
# Shade Streams 0.2.0
E2EE chunked upload/download for Shade. Drop into any Shade-using app:
```ts
const handle = await shade.upload({ to: 'bob', input: file });
const result = await handle.done(); // { sha256, bytesSent, durationMs }
```
…and on the receiver:
```ts
shade.onIncomingTransfer(async (incoming) => {
const handle = await incoming.accept({ output: { kind: 'file', path: '/uploads/x' } });
await handle.done();
});
```
Or in React:
```tsx
<ShadeRuntimeProvider runtime={shade}>
<ShadeUploader to="bob" onComplete={(r) => console.log(r.sha256)} />
</ShadeRuntimeProvider>
```
## How it works
A transfer has two planes:
- **Control plane** — `stream-init`, `stream-finish`, `stream-abort`, and
`stream-resume-*` messages, encoded as JSON plaintext and shipped through
the existing Double Ratchet (envelope type `0x02`). One ratchet step
establishes a stream; the rest is per-chunk AEAD.
- **Data plane** — `stream-chunk` envelopes (envelope type `0x11`),
AES-256-GCM-encrypted under a per-lane key, shipped over HTTP POST (or
WebSocket if opted-in). Lanes run in parallel for throughput.
```
Sender Receiver
────── ────────
streamSecret = randomBytes(32)
streamId = randomBytes(16)
streamKey = HKDF(streamSecret, streamId, "shade-stream/v1\0master")
laneKey[i] = HKDF(streamKey, streamId, "...\0lane\0" || u32(i))
[stream-init JSON over Double Ratchet] ─▶
parses streamSecret, derives same keys
spawns L per-lane receivers
[chunk 0x11 over HTTP] ─▶
AES-GCM(laneKey[i], plaintext, nonce=laneId||seq, aad=streamId||laneId||seq||isLast)
decrypts, verifies, writes to sink
(× 4 lanes in parallel)
[stream-finish JSON over Double Ratchet] ─▶
verifies per-lane sha256 + overall sha256
throws TransferIntegrityError on mismatch
```
## Partition strategies
- **Range** (default for known-size inputs) — lane `i` owns bytes
`[i·N/L, (i+1)·N/L)`. Receiver reconstructs by concatenating lane outputs
in laneId order.
- **Round-robin** (default for unknown-size streams) — chunk `i` goes to
lane `i mod L`. Receiver reorders via a per-stream chunk-index buffer.
## Resume
Persistence is opt-in via a `ResumeStore` (memory, SQLite, Postgres,
IndexedDB-ready). State persisted on init; sender's resume queries the
receiver's `lastSeqAcked` per lane via `GET /v1/transfer/:streamId/state`,
then continues from there. The streamSecret is encrypted at rest under a
device-key derived from the local identity's signing private key — a
stolen DB without the identity key cannot resume.
```ts
const handle = await shade.resumeUpload(streamId, sameInputAsBefore);
await handle.done();
```
Resume across **identity rotation** is not supported (rotation invalidates
the device key — by design, to prevent a stolen pre-rotation DB from
deriving keys for any post-rotation transfer). Restart the transfer
manually after rotation.
## Throughput
- Default 4 lanes × 1 MiB chunks × 4 in-flight chunks per lane =
16 MiB peak in-flight per direction.
- Memory-bounded: receivers stream chunks to the configured sink without
buffering the full payload. 1 GB transfer = O(chunkSize) RSS, not O(file).
- AES-GCM is hardware-accelerated via `SubtleCrypto`; SHA-256 streaming via
`@noble/hashes`.
## Security properties
| ID | Property |
|---|---|
| S1 | streamSecret never on the wire in plaintext (Double Ratchet only) |
| S2 | Unique per-(streamKey, laneId, seq) AEAD nonce — no nonce reuse |
| S3 | Tampered chunk header / ciphertext / tag → AEAD reject |
| S4 | Per-lane sha256 + overall sha256 verified at finish |
| S5 | streamKey/laneKey zeroized on abort/finish (`destroy()`) |
| S6 | Concurrent streams have independent lane keys |
| S7 | seq overflow practical-impossible (u64 max) |
| S8 | At-rest streamSecret encrypted under device-key |
## Hardening
`@shade/streams` ships unbounded by default — a peer can declare a
1 PiB transfer and the receiver will dutifully allocate lane state for
it. Production receivers must enforce limits at the boundary. The
`@shade/files` package wires the same patterns up for its filesystem
RPC; copy the shapes that fit your app.
### Per-stream caps
The receiver sees the declared plaintext size in the `stream-init`
control message before it accepts. Reject above your tolerance:
```ts
shade.onIncomingTransfer(async (incoming) => {
if (incoming.metadata.totalBytes > 256 * 1024 * 1024) {
await incoming.decline({ reason: 'stream too large' });
return;
}
await incoming.accept({ output: ... });
});
```
Recommended ceilings (tune to your product, not these):
| Tier | totalBytes ceiling | Rationale |
|------|--------------------|-----------|
| Chat attachment | 25 MiB | matches mobile MMS / Slack expectations |
| Photo / doc share | 256 MiB | covers raw RAW + most desktop docs |
| Backup / dataset | 4 GiB | larger needs explicit operator opt-in |
### Per-chunk cap
`createTransferRoutes` accepts `maxChunkBytes` (default ≈ 16 MiB +
header). Lower it if your sink can't absorb that — the receiver will
413 anything over the limit before the chunk is decrypted, which
keeps DoS cost bounded.
### Per-sender quotas
`@shade/files` ships a `RateLimiter` (`packages/shade-files/src/server/rate-limiter.ts`)
that enforces both ops-per-window and bytes-per-hour caps per sender
address. The same shape is the recommended template for guarding raw
streams: wrap `incoming.accept` in a check that consumes from a token
bucket keyed by `incoming.fromAddress`, and reject with `decline()`
when the bucket is empty. See
`packages/shade-files/tests/security/quota.test.ts` for the test
shape.
### TTL on idle streams
A `paused` stream-state record consumes a row in your storage and an
encrypted streamSecret slot until it expires. Use the **Retention**
defaults below to expire abandoned streams; pair with a metric
(`shade_stream_states_active`) and an alert when the count grows
unbounded. A peer that opens streams and never finishes them is the
dominant abuse pattern for resumable transfer.
### Trust gates
For high-stakes transfers (backups, key material, internal docs),
gate `accept()` on a verified fingerprint. The pattern mirrors
`@shade/files`'s fingerprint gate — see
`packages/shade-files/tests/security/fingerprint-gate.test.ts`.
## Retention
Resumable streams persist a `PersistedStreamState` per in-flight
transfer, encrypted under a device key. Without retention, every
crashed or abandoned upload leaves a row behind forever.
### Defaults
The shipped `bun-server` SDK template (`shade init --template bun-server`)
schedules `pruneStreamStates` on a daily cron with a **14-day**
horizon. That is: any stream-state record whose `updatedAt` is older
than 14 days is removed at the next sweep. If a sender resumes a
14-day-old stream, it will get a "no state" 404 and start over —
which is the right answer for a transfer that has been idle for two
weeks.
### Tuning the horizon
Set `SHADE_STREAM_RETENTION_DAYS` in the template's environment to
override the 14-day default. Recommended ranges:
| Use case | Horizon | Why |
|----------|---------|-----|
| Synchronous chat | 13 days | resume-after-crash, not resume-after-vacation |
| File-share product | 714 days | covers a typical user vacation |
| Cold backup target | 30+ days | deliberate, but plan for storage growth |
### Hooking the prune call manually
If you bring your own server (no `bun-server` template), call the
storage method on your own schedule:
```ts
import { setInterval } from 'node:timers';
const ONE_DAY_MS = 24 * 60 * 60 * 1000;
const HORIZON_MS = 14 * ONE_DAY_MS;
setInterval(async () => {
if (storage.pruneStreamStates !== undefined) {
await storage.pruneStreamStates(Date.now() - HORIZON_MS);
}
}, ONE_DAY_MS);
```
`pruneStreamStates(olderThan)` removes records whose `updatedAt` is
strictly less than `olderThan`. It is idempotent and safe to call
concurrently.
## Rich file metadata + previews (V3.9)
`stream-init` plaintext can carry an optional `fileMetadata` field that
ships filename, MIME-type, and a thumbnail-stream pointer **end-to-end
encrypted**. Older receivers ignore the field — backwards-compatible
with 0.2.x / 0.3.x peers.
```jsonc
{
"kind": "shade.stream-init/v1",
"streamId": "...",
"streamSecret": "...",
"metadata": {
"chunkSize": 1048576,
"sentAt": 1730000000000,
"fileMetadata": {
"filename": "report.pdf",
"mimeType": "application/pdf",
"thumbnailStreamId": "Ej1z...",
"thumbnailHash": "9a7c...",
"thumbnailMime": "image/webp",
"thumbnailBytes": 18342
}
},
"lanes": [ /* ... */ ]
}
```
### What rides where
| Field | Plane | Visible to server? |
|-------|-------|--------------------|
| `filename` | inside Double Ratchet plaintext | no |
| `mimeType` | inside Double Ratchet plaintext | no |
| `thumbnailStreamId` | streamId of companion stream | yes (random ID, no info leak) |
| `thumbnailHash` | sha256 of preview plaintext | base64 hash only, no pixels |
| `thumbnailMime` | one of `image/jpeg / image/webp / image/png` | yes (allowlist enforced) |
| `thumbnailBytes` | declared length, capped at 64 KiB | yes |
| thumbnail bytes themselves | separate AEAD stream, own lane | no |
The thumbnail rides as its **own stream-transfer**, keyed independently
from the main stream. A server compromise leaks neither preview pixels
nor original bytes.
### Sender — attach a preview
```ts
// Pre-computed preview (server-side pipeline path):
await shade.upload({
to: 'bob',
input: pdfBytes,
thumbnail: { bytes: previewWebp, mime: 'image/webp' },
metadata: { fileMetadata: { filename: 'report.pdf', mimeType: 'application/pdf' } },
});
// Browser auto-generation (image File / Blob → 256×256 preview):
await shade.upload({
to: 'bob',
input: imageFile, // a `File` from <input type="file">
generateThumbnail: true, // OffscreenCanvas + createImageBitmap
});
```
`generateThumbnail` is a no-op on runtimes lacking
`OffscreenCanvas + createImageBitmap` (Bun, Node) — those callers should
pre-generate and pass `thumbnail` directly, or skip the preview entirely.
### Receiver — render in widgets
The bundled `@shade/widgets` `useShadeDownload` hook auto-accepts
thumbnail streams (marked by `userMetadata.shadeThumbnail = '1'`) into
an in-memory `ShadeThumbnailCache`. `<TransferRow showThumbnail
fileMetadata={...} />` reads from the same cache and renders inside an
`<img>` element so the browser's image-decoding sandbox is the trust
boundary for format parsing.
```tsx
<ShadeThumbnailProvider>
<TransferRow
handle={handle}
progress={progress}
showThumbnail
fileMetadata={incoming.metadata.fileMetadata}
/>
</ShadeThumbnailProvider>
```
### Format-hardening (sender + receiver)
Both sides enforce the same rules — single source of truth in
`@shade/streams/file-metadata.ts`:
| Rule | Limit |
|------|-------|
| `thumbnailMime` allowlist | `image/jpeg`, `image/webp`, `image/png` |
| `thumbnailBytes` cap | 64 KiB (`THUMBNAIL_MAX_BYTES`) |
| `filename` length | ≤ 1024 chars, no control characters |
| `mimeType` shape | RFC 7231 `type/subtype` token |
| Hash binding | declared `thumbnailHash` = sha256(preview bytes); mismatched bytes are dropped at the cache before any render |
A hostile peer cannot:
- smuggle exotic image formats past the allowlist (envelope parser
rejects at decode-time),
- substitute different bytes for a declared preview (cache verifies
sha256 before exposing bytes to a renderer),
- inflate the cache to OOM the receiver (LRU + 1 MiB total cap).
### Risks consciously accepted
- **Preview-arrival ≠ send completion.** A receiver may see the
thumbnail before the main upload finishes. For high-stakes flows
where "did Alice send X?" is itself sensitive, send the preview
*only* after main completion (set `thumbnail` to `null` and instead
ship a follow-up `stream-init` with the preview). The default
ordering optimizes UX, not metadata-secrecy.
- **Renderer trust.** We render through a Blob-URL `<img>`. A 0-day
in the browser's image decoder would still reach the receiver. Keep
browsers patched; rely on the CSP of your embedding app.
## API surface
See package READMEs:
- `packages/shade-streams/README.md` — crypto + state machines
- `packages/shade-transfer/README.md` — orchestration, transports, persistence
- `packages/shade-transport-webrtc/README.md` — V3.11 P2P transport plug-in
- `packages/shade-sdk/README.md` — magic drop-in
- `packages/shade-widgets/README.md` — React UI
## Transports
`@shade/transfer` ships HTTP + WebSocket chunk transports. V3.11 adds an
opt-in P2P chunk transport via `RTCDataChannel`:
- HTTP — `ShadeTransferHttpTransport`. POST per chunk; the receiver-
side route is `app.route('/v1/transfer', await shade.transferRoute())`.
- WebSocket — `ShadeTransferWsTransport`. One connection per peer,
binary-framed chunks, JSON acks; same wire format inside the frame as
the WebRTC transport.
- WebRTC — `WebRtcTransferTransport` from `@shade/transport-webrtc`.
Wired automatically by `shade.configureWebRTC()` as the primary
layer of a `MultiTransportFallback([webrtc, http])`. See
[docs/webrtc.md](./webrtc.md).
`MultiTransportFallback` is the N-ary generalisation of
`FallbackTransferTransport`: pass an ordered list of named transports
and the engine demotes sticky on `TransferTransportError`.

224
docs/transport.md Normal file
View File

@@ -0,0 +1,224 @@
# Shade Transport — Bridge Layer (V3.7)
> **Looking for V3.11 (peer-to-peer chunk transport via `RTCDataChannel`)?**
> See [docs/webrtc.md](./webrtc.md). This page covers the V3.7 bridge
> layer that ships ciphertext *envelopes* (control plane) over
> WS / SSE / long-poll. The two are orthogonal: the bridge handles
> store-and-forward control envelopes; WebRTC handles direct chunk data.
The bridge layer is the answer to: **"my client is a browser extension /
strict-corp-proxy / edge-runtime / iOS app — I cannot keep a WebSocket
open. How do I receive ciphertext envelopes?"**
It is built on top of the V3.6 inbox: every transport delivers the same
inbox blobs, with the same authentication semantics. Application code
sees a single `IncomingMessage` shape and never branches on transport.
```
┌─────────────────────────────────────────────────────────────────┐
│ application code │
│ │
│ bridge.connect({ onMessage: (m) => decrypt(m.bytes) }) │
└────────────────────────────────┬────────────────────────────────┘
┌─────────────────────────┴──────────────────────────┐
│ FallbackBridgeTransport │
│ (sticky-after-first-success) │
└──┬──────────────────┬─────────────────────────┬────┘
│ │ │
┌──────▼─────┐ ┌──────▼─────┐ ┌──────▼─────┐
│ WsBridge │ │ SseBridge │ │ LongPoll │
│ /v1/ │ │ /v1/ │ │ Bridge │
│ bridge/ws │ │ bridge/ │ │ /v1/bridge │
│ │ │ stream │ │ /poll │
└──────┬─────┘ └──────┬─────┘ └──────┬─────┘
│ │ │
└──────────────────┼─────────────────────────┘
┌─────▼──────┐
│ inbox │ ← the same V3.6 store
│ blobs │ and events
└────────────┘
```
## When to reach for which
| Transport | Latency | Proxy resilience | Browser | Server cost |
|-------------|----------|------------------|---------|-------------|
| WebSocket | ms | breaks under strict CONNECT-blocking proxies | ✓ | one socket per client |
| SSE | ms | passes most HTTP proxies (text/event-stream) | ✓ | one streamed response per client |
| long-poll | ≤ 25 s | passes anything that allows GET | ✓ | one held request per client |
The recommended composition:
```ts
import {
FallbackBridgeTransport,
WsBridge,
SseBridge,
LongPollBridge,
} from '@shade/transport-bridge';
const auth = {
crypto, // CryptoProvider
signingPrivateKey, // recipient's Ed25519 private key
address: 'bob',
};
const bridge = new FallbackBridgeTransport([
new WsBridge({ baseUrl: 'https://relay.example.com', auth }),
new SseBridge({ baseUrl: 'https://relay.example.com', auth }),
new LongPollBridge({ baseUrl: 'https://relay.example.com', auth }),
]);
await bridge.connect({
onMessage: async (msg) => {
// msg.bytes is a Uint8Array — pass it to your decrypt path.
// msg.from is the relay-known sender hint (may be empty); the
// authoritative sender comes from the decrypted envelope.
// msg.msgId is the relay's deterministic message id (sha256(ciphertext)).
const envelope = decodeEnvelope(msg.bytes);
await shade.receive(senderAddress, envelope);
},
});
// Read which transport the fallback chain settled on:
console.log(bridge.activeKind); // "ws" | "sse" | "long-poll"
```
## The IncomingMessage shape
```ts
interface IncomingMessage {
from: string; // relay-side sender hint (may be "")
bytes: Uint8Array; // the ciphertext envelope, exactly as PUT
receivedAt: number; // relay-monotonic cursor — NOT wall-clock arrival
msgId?: string; // sha256(bytes) — useful for ack/dedup
}
```
`from` is intentionally a hint — sender provenance lives inside the
encrypted envelope and is recovered post-decrypt. The bridge layer is
plaintext-blind by design.
## Auth — signed query parameters
Every bridge request signs the canonical
`{address, kind, since, signedAt}` payload with the recipient's Ed25519
signing private key. The server looks up the address-owner key
registered via `/v1/inbox/register` and verifies the signature.
`kind` is bound into the canonical payload so a signature for `/poll`
cannot be replayed against `/stream` or `/ws`.
The browser `EventSource` API does not let callers attach custom
headers; query parameters are the only portable carrier and so the
bridge protocol uses them uniformly across all three transports.
## Server-side — `createBridgeRoutes`
```ts
import { createBridgeRoutes } from '@shade/inbox-server';
import { Hono } from 'hono';
const inbox = new MemoryInboxStore();
const events = new InboxServerEvents();
const bridge = createBridgeRoutes({
store: inbox,
crypto,
events,
longPollTimeoutMs: 25_000, // default — under typical proxy idle limits
heartbeatIntervalMs: 15_000, // SSE keepalive comments
fallbackPollIntervalMs: 1_000, // when no `events` emitter is wired
});
const app = new Hono();
app.route('/', bridge.app);
Bun.serve({
port: 3900,
fetch: (req, srv) => app.fetch(req, srv),
websocket: bridge.websocket as any,
});
```
The bridge subscribes to `InboxServerEvents` (`inbox.blob_stored`) for
push-style delivery — when an event fires for a connected address, the
server fetches new blobs and forwards them. If no events emitter is
wired, the server falls back to a small in-process polling timer at
`fallbackPollIntervalMs` cadence.
## Cursor & resume
Every `IncomingMessage.receivedAt` is the relay's monotonic cursor for
the address. Bridges expose `getCursor()` so applications can persist
the high-water mark and pass it as `startCursor` on the next
`connect()`:
```ts
const sse = new SseBridge({
baseUrl,
auth,
startCursor: await persistedCursor.load(),
});
await sse.connect({
onMessage: async (msg) => {
await persistedCursor.save(msg.receivedAt);
// …
},
});
```
For SSE specifically, the server emits an `id:` field per event; the
bridge sends it back as `Last-Event-ID` plus the `since=` query
parameter on reconnect, so a flapping connection picks up exactly where
it left off without duplicates.
## Reconnect & backoff
| Bridge | Auto-reconnect | Backoff |
|-------------|----------------|----------------------|
| WS | yes (default) | 250 ms → 10 s exponential |
| SSE | yes (default) | 250 ms → 10 s exponential |
| long-poll | always on (the loop *is* the reconnect) | 2 s on hard error |
Pass `disableAutoReconnect: true` (WS / SSE) for tests where you want a
single attempt and immediate surfaced error.
## Long-poll concurrency
The `LongPollBridge` issues exactly one request at a time. The next
request fires after the previous one resolves. This guarantees a
client never holds more than one TCP connection on the server, which
matches the V3.7 acceptance criterion and keeps capacity planning
simple: max in-flight long-poll requests = number of connected clients.
## Failure modes
- **WS handshake rejected (4xxx code).** `WsBridge.connect` rejects.
Caller (or `FallbackBridgeTransport`) moves on.
- **SSE returns non-200.** `SseBridge.connect` throws a `BridgeError`
with `httpStatus`.
- **Long-poll returns non-200.** Same — `BridgeError` with `httpStatus`.
- **Mid-stream error after connect.** WS/SSE auto-reconnect; long-poll
swallows transient errors and continues looping. Errors flow to the
caller's `onError` handler.
## Acceptance test coverage (V3.7)
`packages/shade-transport-bridge/tests/bridge.test.ts` covers:
- "Send 100 small messages" — one test per transport, all pass.
- "WS blocked by proxy → SSE → long-poll" — fallback test boots a
server where the WS endpoint is unreachable and the SSE endpoint
returns 502, verifies the chain falls all the way through to
long-poll without message loss.
- "Long-poll uses ≤ 1 outstanding request" — wraps `fetch` to count
in-flight requests over 1.5 s of steady-state operation.
- Cursor resume — tears down an SSE connection mid-stream, pushes more
blobs, reconnects with the persisted cursor, asserts exactly the new
blobs are delivered (no overlap with the pre-disconnect set).
- Auth rejection — wrong signing key and unregistered address both
produce hard `connect` rejections so the fallback chain advances.

156
docs/trust-ux.md Normal file
View File

@@ -0,0 +1,156 @@
# Trust UX — Fingerprint Gates (V3.3)
> Status: shipped in 0.4.0, GA-frozen in 4.0 — see [V3.3 plan](./archive/V3.3.md).
Shade ships with a small number of **blocking** verification gates that
fire automatically before the operations where MITM risk is highest.
Each gate calls a handler you register on the SDK; until the user (or
your handler) approves, the operation aborts with
`FingerprintNotVerifiedError`.
The point of the gate model is to be alert-fatigue-free: you don't see
a prompt before every chat message, just before the handful of moments
that genuinely matter.
---
## What the gates protect
| Gate | Fires when | Default policy |
|------|------------|----------------|
| `first-large-file` | `Shade.upload(...)` for an unverified peer with a known size at or above the configured threshold. | Threshold `10 MiB`. Below = no gate. |
| `backup-import` | `Shade.importBackup(...)` before any state is written. Handler receives the fingerprint of the identity *embedded in the backup*. | Always fires. |
| `new-device-trust` | `Shade.acceptIdentityChange(...)` after a peer rotates identity. The peer's `identity_version` is bumped first so any prior verification is automatically stale. | Always fires. |
| `inbox-fanout` | Reserved for V3.6 (`@shade/inbox`). Per-recipient hook is wired today so apps can register it now. | Always fires. |
---
## Registering handlers
```ts
const shade = await createShade({
prekeyServer: 'https://prekeys.example.com',
storage: 'sqlite:/data/shade.db',
});
shade.beforeFirstLargeFile(10 * 1024 * 1024, async (ctx) => {
// ctx.peerAddress, ctx.fingerprint, ctx.fileSize
return await ui.confirmFingerprintModal(ctx);
});
shade.beforeBackupImport(async (ctx) => {
// ctx.fingerprint = fingerprint of the identity in the backup blob
return await ui.confirmBackupOwner(ctx);
});
shade.beforeNewDeviceTrust(async (ctx) => {
// ctx.fingerprint = fingerprint of the rotated identity
return await ui.confirmDeviceRotation(ctx);
});
```
Return `true` to allow the operation and persist a `'user'` verification.
Return `false` (or throw) to abort with `FingerprintNotVerifiedError`.
If you don't register a handler, the gate **logs a one-time warning per
peer and proceeds on TOFU**, persisting a `'tofu-after-warning'`
verification. This satisfies the V3.3 acceptance criterion that apps
without registered gates get sane defaults instead of hard-failing — but
it does mean the gate is informational, not a hard wall, in that
configuration. Always register handlers in production.
---
## Manual verification
The handler model assumes your app drives the OOB compare/confirm
flow. If the user verifies through some other path (QR code scan, audio
read-aloud, transitive trust from V3.10), call:
```ts
await shade.markPeerVerified('bob'); // pin current fingerprint
await shade.unmarkPeerVerified('bob'); // revoke
const ok = await shade.isPeerVerified('bob'); // check status
```
`markPeerVerified` reads the peer's *current* fingerprint and pins it
together with the per-peer `identity_version`. When the peer rotates
(`acceptIdentityChange`), the version bumps and the saved verification
goes stale automatically — `isPeerVerified` will return `false` until
the user re-verifies.
---
## Tuning thresholds
The `first-large-file` threshold is the only knob that's customer-tunable
without code changes. The defaults are conservative:
- **Default:** `10 MiB`. Big enough that ordinary chat attachments don't
trigger; small enough that obvious "exfil candidates" do.
- **Lower** (e.g. `1 MiB`) for high-sensitivity deployments — every
document goes through the gate.
- **Raise** (e.g. `100 MiB`) only for use cases where small uploads are
routine and large transfers are deliberate / pre-arranged.
`backup-import` and `new-device-trust` have no threshold by design — the
spec mandates an irremovable minimum gate for both, since each one
either trusts a fresh identity or overwrites pinned trust wholesale.
---
## React widget
Use `<FingerprintGate />` from `@shade/widgets` to block UI on
verification status:
```tsx
import { FingerprintGate } from '@shade/widgets';
<FingerprintGate peerAddress="bob">
<ChatThread peer="bob" />
</FingerprintGate>
```
The default fallback shows the safety number, a "Copy OOB text" button,
and an "I have verified" button that calls `Shade.markPeerVerified`.
Pass a `fallback` render prop to use your own UI, or `onVerified` to
react to the unverified → verified transition.
`<FingerprintCompare />` is the existing observer-dashboard widget; it
now exposes the same Copy-OOB / verify actions when an `onVerified`
prop is wired.
---
## Errors
`FingerprintNotVerifiedError` carries:
- `peerAddress` — the address the gate was protecting.
- `gate``'first-large-file' | 'backup-import' | 'new-device-trust' | 'inbox-fanout'`.
- `code = 'SHADE_FINGERPRINT_NOT_VERIFIED'` — maps to HTTP 403.
Catch it explicitly when wrapping `upload`, `importBackup`, and
`acceptIdentityChange`:
```ts
try {
await shade.upload({ to: 'bob', input: bytes });
} catch (err) {
if (err instanceof FingerprintNotVerifiedError) {
showVerifyFirst(err.peerAddress);
return;
}
throw err;
}
```
---
## Migration from 0.3.x
No breaking changes: existing apps gain warning-mode gates automatically
(see the no-handler note above). To upgrade to hard gates, register
handlers for the operations you use. Your existing `FingerprintCompare`
calls keep working; pass `onVerified` to enable the new actions.

276
docs/web-workers.md Normal file
View File

@@ -0,0 +1,276 @@
# Web Workers Crypto
Status: Implemented (V3.8 — `0.4.0`).
`@shade/crypto-web` ships with an opt-in dedicated Web Worker that keeps
AES-GCM, HKDF, HMAC, X25519 and Ed25519 — and full per-lane stream state —
off the main thread. Big in-browser uploads (100 MB+) stay smooth without
frame drops.
This doc covers:
- [When to use it](#when-to-use-it)
- [Setup](#setup)
- [API](#api)
- [Bundler recipes](#bundler-recipes)
- [Safari notes](#safari-notes)
- [SharedArrayBuffer (COOP/COEP)](#sharedarraybuffer-coopcoep)
- [Lifecycle and rotation](#lifecycle-and-rotation)
- [Threat-model considerations](#threat-model-considerations)
---
## When to use it
The default `SubtleCryptoProvider` runs on whatever thread you give it.
For the SDK that means the main thread. AES-GCM via SubtleCrypto is fast
(hardware-accelerated), but a 100 MB file at 256 KiB chunks is ~400 AEAD
calls — each one queues a microtask on the main thread. Layered on top of
React reflows and large `postMessage` payloads to the network worker, you
*will* see frame drops.
Reach for the Worker pipeline when:
- You upload or download files that don't fit in a single AEAD chunk
(≥ ~1 MB) inside a UI-bearing browser tab.
- You generate or rotate identity / device keys in a UI thread that must
stay interactive.
- You do batch AEAD (e.g. backup export over many records).
You can keep using `SubtleCryptoProvider` for short ops (Signal session
encrypt/decrypt for a chat message). The cost of a `postMessage` round-
trip dwarfs the cost of a single 256-byte AES call.
---
## Setup
`@shade/crypto-web` exposes the worker as a separate subpath, so your
bundler can resolve it through the standard `new Worker(new URL(...,
import.meta.url))` idiom.
```ts
import { createShade } from '@shade/sdk';
const shade = await createShade({ /* ... */ });
shade.configureWorkerCrypto({
workerUrl: new URL('@shade/crypto-web/worker', import.meta.url),
});
```
After `configureWorkerCrypto`, the SDK exposes:
- `shade.encryptStream({ streamId, streamSecret, ... })` — returns a
`TransformStream<Uint8Array, Uint8Array>` and a `laneSha256` promise.
- `shade.decryptStream({ streamId, streamSecret, ... })` — inverse.
- `shade.getWorkerCrypto()` — direct access to the `WorkerCryptoProvider`
for one-off ops (HKDF batches, X25519 batch DH, etc.).
The worker is spawned on first use and self-terminates after
`idleTimeoutMs` (default 30 s) — no manual lifecycle management required.
---
## API
### Stream encryption
```ts
const { stream, laneSha256 } = await shade.encryptStream({
streamId: streamId, // 16 random bytes, agreed with peer
streamSecret: streamSecret,// 32 random bytes, derived via Double Ratchet
laneId: 0, // lane index (use multi-lane for parallel HTTP)
chunkSize: 256 * 1024, // optional; default 256 KiB
});
await file.stream()
.pipeThrough(stream)
.pipeTo(transferSink); // your HTTP-shipping WritableStream
const sha256 = await laneSha256; // for end-to-end integrity proof
```
`stream` consumes plaintext and emits one wire-encoded
`stream-chunk` envelope per write. `flush` always emits a final chunk
with `isLast=true` (even if the trailing slice is empty), so receivers
see a clean termination.
### Stream decryption
```ts
const { stream, laneSha256 } = await shade.decryptStream({
streamId,
streamSecret,
laneId: 0,
});
await incomingChunkStream
.pipeThrough(stream)
.pipeTo(fileSink);
const sha = await laneSha256;
if (!equal(sha, peerLaneSha256)) throw new IntegrityError();
```
Each input chunk MUST be a complete wire envelope. The transport-layer
caller is responsible for framing (one envelope per write). Out-of-order
or replayed chunks reject the stream — the lane key never crosses thread
boundaries, so a man-in-the-middle script in the page can't recover key
material to replay against.
### Direct provider access
```ts
const crypto = await shade.getWorkerCrypto();
// Implements `CryptoProvider` — drop-in replacement for SubtleCryptoProvider
const { ciphertext, nonce } = await crypto.aesGcmEncrypt(key, plaintext);
```
`randomBytes`, `randomUint32`, `constantTimeEqual`, `zeroize` execute on
the calling thread (no round-trip). Async ops forward to the worker.
---
## Bundler recipes
### Vite
```ts
shade.configureWorkerCrypto({
workerUrl: new URL('@shade/crypto-web/worker', import.meta.url),
});
```
Vite resolves the URL via `import.meta.url` and emits a discrete chunk
for the worker. No additional config required for Vite ≥ 5.
If your build complains about `?worker` syntax, use the explicit URL
form (above) — it's the standard Vite idiom.
### Webpack 5 / Rspack
Same idiom — Webpack 5 understands `new URL('./worker.js', import.meta.url)`
natively as long as the source is ESM:
```ts
new Worker(new URL('@shade/crypto-web/worker', import.meta.url), {
type: 'module',
});
```
For Webpack 4 or non-ESM builds, you need `worker-loader` (legacy). We
do not officially support Webpack 4.
### Rollup
Rollup needs `@rollup/plugin-web-worker-loader` or a recent
`rollup-plugin-import-meta-url`. The standard idiom works once the
plugin is wired:
```ts
new URL('@shade/crypto-web/worker', import.meta.url)
```
If your bundler can't resolve `@shade/crypto-web/worker`, copy
`node_modules/@shade/crypto-web/src/worker.ts` (or the compiled `.js`
once we ship dist artefacts) into your `public/` directory and pass an
absolute URL:
```ts
shade.configureWorkerCrypto({ workerUrl: '/shade-crypto.worker.js' });
```
---
## Safari notes
Safari ≤ 17 has a smaller `postMessage` transferable budget than Chrome /
Firefox. Single transfers above ~64 MB occasionally fail silently. The
shipped pipeline already chunks plaintext to 256 KiB before AEAD, so
each `postMessage` carries ≤ ~256 KiB + AEAD overhead — well under any
known Safari limit.
If you override `chunkSize`, keep individual buffers below 16 MiB:
```ts
shade.encryptStream({
streamId, streamSecret,
chunkSize: 8 * 1024 * 1024, // 8 MiB — safe across all browsers
});
```
We do not officially support Safari ≤ 14 (no module workers).
---
## SharedArrayBuffer (COOP/COEP)
The default pipeline uses `ArrayBuffer` transfer (zero-copy ownership
hand-off). It does **not** require COOP/COEP headers.
For multi-lane parallel transfers across multiple workers, you may opt
in to `SharedArrayBuffer` for the AEAD plaintext buffers. That requires
your origin to serve:
```
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
```
`SharedArrayBuffer` support is gated behind a future `useSharedBuffers`
option and is not enabled in V3.8. See `docs/V4.0.md` if/when this lands.
---
## Lifecycle and rotation
```ts
const crypto = await shade.getWorkerCrypto();
await crypto.rotate(); // tear down the current worker, respawn lazily
await crypto.destroy(); // permanent — every subsequent call rejects
```
`shade.shutdown()` calls `destroy()` automatically. The idle-timer fires
30 seconds after the last response (configurable via
`configureWorkerCrypto({ idleTimeoutMs })`); if the timer fires while
calls are pending, it does nothing and reschedules.
---
## Threat-model considerations
- The worker runs in the same origin and the same browsing context as
the main thread. It is **not** a sandbox against a compromised page;
any script that can `eval` in your tab can also `postMessage` to the
worker. The Worker is a *performance* boundary, not a *security*
boundary.
- Lane keys derived inside the worker stay there; they are never
postMessage'd to the main thread. This narrows the window during which
a key sits in main-thread heap, which helps against post-mortem heap
inspection by a curious extension. It does not help against an active
in-page attacker.
- `randomBytes` runs on the calling thread (uses `crypto.getRandomValues`
directly). The worker has its own random source for ops that derive
inside it (nonces are derived deterministically from `(laneId, seq)`).
For the full picture, see `THREAT-MODEL.md`.
---
## Verifying main-thread budget
V3.8 acceptance: 100 MB upload in Chrome without main thread blocked
> 16 ms in P99.
To verify in your app:
1. Open Chrome DevTools → Performance.
2. Record a 100 MB upload.
3. Inspect the main-thread flame chart. Look at "Long Tasks" and
"Self time" of `Shade.encryptStream`.
4. Confirm no contiguous block exceeds ~16 ms (one frame at 60 fps).
If you observe long tasks, lower `chunkSize` (more frequent yields) or
report the trace — see [`docs/archive/V3.8.md`](./archive/V3.8.md) for
the original acceptance criteria.

302
docs/webrtc.md Normal file
View File

@@ -0,0 +1,302 @@
# Shade Transport — WebRTC P2P Layer (V3.11)
`@shade/transport-webrtc` adds a direct peer-to-peer chunk transport on
top of the existing `@shade/transfer` engine. When two clients can reach
each other through NAT/firewall, large transfers (`@shade/files`,
`@shade/transfer`) flow over a single bidirectional `RTCDataChannel`
instead of paying the round-trip cost of HTTP-relayed POSTs. When NAT
traversal fails, the multi-transport fallback automatically demotes the
chain back to HTTP — without losing any chunks already in flight.
The wire payload is unchanged: every chunk is still a Shade ratchet /
streams envelope (AES-256-GCM under HKDF-derived per-lane keys). DTLS-
SRTP is only the WebRTC transport secret; turning a TURN-relay on does
not give the relay operator access to plaintext.
```
┌───────────────────────────────────────────────────────────────┐
│ application code │
│ │
│ shade.upload({ to: 'bob', input: file }) │
└────────────────────────────────┬──────────────────────────────┘
┌─────────▼──────────┐
│ TransferEngine │
└─────────┬──────────┘
│ ITransferTransport
┌─────────▼──────────┐
│ MultiTransport │
│ Fallback (sticky) │
└────┬─────┬─────┬───┘
│ │ │
┌─────────────▼┐ ┌─▼─┐ ┌▼────────────┐
│ WebRtcTransfer│ │WS │ │ ShadeTransfer│
│ Transport │ │… │ │ HttpTransport│
└─────┬─────────┘ └───┘ └──────────────┘
│ DataChannel binary frames
┌─────▼─────────┐
│ WebRtcConn │ ←──── SDP/ICE over Shade.send
│ Manager │ (ratchet-encrypted)
└───────────────┘
```
## When to reach for it
| Scenario | Default (HTTP) | + WebRTC |
|---------------------------------------|----------------|----------------|
| Two clients on the same LAN | server-relayed | direct, P2P |
| One peer behind enterprise NAT only | works | TURN-relay |
| Both peers behind symmetric NAT | works | falls back to HTTP |
| One peer offline | inbox-buffered | inbox-buffered (HTTP path) |
| Browser extension with strict CSP | works | works (uses RTCPeerConnection) |
Use cases:
- `@shade/transfer` upload of multi-MB / multi-GB files
- `@shade/files` `read`/`write` of large inline blobs
- Future: `@shade/streams` real-time channels (V5.0 reuses this same DataChannel)
## Quick start (browser)
```ts
import { createShade } from '@shade/sdk';
import { nativeRtcFactory } from '@shade/transport-webrtc';
const shade = await createShade({ prekeyServer: 'https://prekey.example.com' });
// IMPORTANT: configureWebRTC MUST be called BEFORE the first upload() /
// onIncomingTransfer() / transferRoute() call, because those build the
// transfer engine — and the engine captures its transport stack at
// construction time.
shade.configureWebRTC({
factory: nativeRtcFactory(),
// Optional — defaults to two public Google STUN servers.
iceServers: [
{ urls: 'stun:stun.l.google.com:19302' },
{
urls: 'turn:turn.example.com:3478',
username: 'shade',
credential: 'YOUR_TURN_SECRET',
},
],
});
shade.configureTransfers({
resolveBaseUrl: async (peer) => directory.lookup(peer),
});
await shade.upload({ to: 'bob', input: file }); // → P2P when NAT allows
```
## Quick start (Bun / Node)
Bun does not yet expose `RTCPeerConnection` natively. Use one of:
- [`node-datachannel`](https://github.com/murat-dogan/node-datachannel)
— small, stable, libdatachannel under the hood
- [`@roamhq/wrtc`](https://www.npmjs.com/package/@roamhq/wrtc) — fork of
the Google `wrtc` bindings
Wrap the chosen library behind an `IRtcFactory` (the package only depends
on a narrow surface — `createPeerConnection`, `createDataChannel`,
`addEventListener`):
```ts
import { IRtcFactory, IPeerConnection, IDataChannel } from '@shade/transport-webrtc';
// pseudo-adapter for node-datachannel
class NodeDataChannelFactory implements IRtcFactory {
createPeerConnection(config) { /* ... return adapter wrapping nodeDc PeerConnection */ }
}
shade.configureWebRTC({ factory: new NodeDataChannelFactory(), iceServers });
```
## Connection flow
```
Alice initiates Bob receives
─────────────── ────────────
1. createOffer() → SDP 2. shade.send delivers offer
→ Bob.createAnswer()
3. shade.send delivers answer 4. setRemoteDescription(answer)
5. trickle ICE candidates (both directions) 6. trickle ICE candidates
7. DataChannel onopen (both sides) 7. DataChannel onopen
```
All four signaling kinds (`shade.webrtc-offer/v1`, `shade.webrtc-answer/v1`,
`shade.webrtc-ice/v1`, `shade.webrtc-bye/v1`) ride the existing Shade
ratchet — the relay sees only ciphertext envelopes.
### Glare resolution
If both peers call `getOrCreate()` simultaneously, the manager uses
lexicographic tiebreak: the side with the smaller address wins
caller-role; the side with the larger address closes its outgoing
connection and accepts the inbound offer instead. Both peers ultimately
converge on a single `WebRtcConnection`.
## Backpressure
The `WebRtcTransferTransport` polls `RTCDataChannel.bufferedAmount` and
suspends new sends once the buffer crosses `backpressureThresholdBytes`
(default 4 MiB). This avoids SCTP queue runaway when the application
pushes faster than the network can drain. Tune lower for memory-
constrained clients (mobile / extension contexts).
## Auto-fallback
Configuring WebRTC wires `MultiTransportFallback([webrtc, http])` as the
engine's transport. The chain is sticky-after-first-failure: when WebRTC
raises a `TransferTransportError` (timeout, ICE failed, data channel
closed, frame too large), the fallback advances to HTTP and stays there
for the lifetime of the engine.
For three-tier composition (e.g. WebRTC → WebSocket → HTTP), build the
fallback yourself and pass a custom transport via the engine deps:
```ts
import { MultiTransportFallback } from '@shade/sdk';
const stack = new MultiTransportFallback([
{ name: 'webrtc', transport: rtcTransport },
{ name: 'ws', transport: wsTransport },
{ name: 'http', transport: httpTransport },
]);
stack.onSwitch((from, to) => metrics.observe('shade.transport.demoted', { from, to }));
```
The `WebRtcConnectionManager`'s connect timeout (default 30 s) is the
upper bound on how long the chain dwells on WebRTC before demoting. The
V3.11 acceptance criterion is "P2P-død → HTTP innen 5 s" — set
`connectTimeoutMs: 4_000` in your `configureWebRTC()` call to keep the
upper bound at 4 seconds and meet the SLO with margin.
## ICE server config
| Setting | Default | When to override |
|------------------------|-----------------------------------|------------------|
| `iceServers` | Google public STUN (×2) | Production — pin your own STUN to avoid Google rate limits, plus your TURN credentials |
| `iceTransportPolicy` | `'all'` (host + reflexive + relay)| `'relay'` to mandate TURN-only routing (e.g. inside a corporate network where direct connectivity must never leak) |
| `bundlePolicy` | spec default (`'balanced'`) | rarely |
Public STUN works for ~80% of consumer NATs. The remaining 20% (symmetric
NAT, paranoid corporate proxies, mobile carrier-grade NAT) need TURN.
Run your own [coturn](https://github.com/coturn/coturn) or use a managed
provider — but **TURN traffic is real bandwidth through your server**, so
budget accordingly. Shade's wire format is at least as efficient over
TURN as over HTTPS (no per-request HTTP framing overhead).
## NAT-traversal: hopes and realities
What works without TURN, in our testing:
- Same NAT (LAN): always
- Two clients behind cone NATs: usually
- One client behind symmetric NAT, the other behind any cone NAT: usually
- Two clients behind symmetric NATs: rarely — falls back to TURN
What doesn't work:
- Two clients behind strict carrier-grade NAT (CGNAT): TURN required
- Clients on networks that block UDP entirely: TURN over TCP/443 required
When in doubt, configure TURN over TCP/443 — it impersonates HTTPS and
gets through nearly every middlebox.
## Diagnostics
The SDK exposes the live runtime via `shade.getWebRtcRuntime()`:
```ts
const runtime = shade.getWebRtcRuntime();
if (runtime !== null) {
console.log('active transport:', runtime.fallback.activeName);
console.log('peers:', [...runtime.manager.byPeer ?? []]);
runtime.fallback.onSwitch((from, to) => {
console.warn(`shade transport demoted ${from}${to}`);
});
}
```
The `failures` array on `MultiTransportFallback` records every
demotion's reason — wire it to your observability backend to track
NAT/TURN problems in production.
## Sample code
End-to-end test using `MemoryRtcFactory` (no real network):
```ts
import { MemoryRtcFactory } from '@shade/transport-webrtc';
const factory = new MemoryRtcFactory();
alice.configureWebRTC({ factory });
bob.configureWebRTC({ factory });
await alice.upload({ to: 'bob', input: bytes }); // → P2P loopback
```
See `packages/shade-sdk/tests/webrtc-integration.test.ts` for the full
loopback test, `webrtc-failover.test.ts` for the auto-fallback test, and
`packages/shade-transport-webrtc/tests/` for the unit tests covering
wire format, signaling, glare, and TURN-only configuration.
## Wire format inside the DataChannel
The DataChannel is a single bidirectional pipe shared by every in-flight
stream between two peers. Each frame is a self-describing binary blob:
```
client → server server → client
─────────────── ───────────────
0x01 chunk reqId(16) sid(16) lane(u32) seq(u64) env(...) 0x81 chunk-ack reqId(16) lastSeq(u32) bytesRecv(u32)
0x02 resume-query reqId(16) sid(16) 0x82 resume-state reqId(16) jsonBody(utf-8)
0x03 ping reqId(16) nonce(u64) 0x83 pong reqId(16) nonce(u64)
0xFE error reqId(16) jsonBody(utf-8)
```
`reqId` is a 16-byte random correlation token; the responder echoes it
verbatim so multiple in-flight requests can be matched without a stream
multiplexer on top of SCTP.
The wire matches `ShadeTransferWsTransport` exactly — adapters for
either transport can interoperate by translating between SCTP message-
framing and WS binary frames at the byte level.
## Limits
- Max DataChannel message: **256 KiB** (Chrome's safe ceiling). Configure
`chunkSize` ≤ 256 KiB on uploads that prefer WebRTC. The transport
raises a clear error when an envelope exceeds the cap; the engine then
retries via HTTP.
- One DataChannel per peer pair (label `shade-transfer/v1`). Multiple
in-flight transfers from the same peer pair multiplex via `reqId`.
- No SFU/MCU — group transfers fan out at the application layer.
- DTLS-fingerprint binding to Shade's identity-fingerprint is **not** in
V3.11 (deferred as hardening work — DataChannel is already inside a
ratchet-authenticated session, so the practical exposure window is
limited to in-process MITM scenarios that already require malware).
## Migration
Opt-in. If you don't call `configureWebRTC`, your existing HTTP/WS
transport stack is unchanged.
When you do opt in, the **engine must not be built yet** — the easy way
to ensure this is to call `configureWebRTC` before `configureTransfers`
or before any of `upload` / `onIncomingTransfer` / `transferRoute`.
Receiver-side: the WebRTC manager wires receiver-hooks into the engine
during `engine()` construction, so make sure both sides do `configureWebRTC`
+ `configureTransfers` before the first `transferRoute()` call.
## Related modules
- [`@shade/transfer`](../packages/shade-transfer/) — engine, lane queues,
HTTP transport, multi-fallback wrapper.
- [`@shade/streams`](./streams.md) — chunk encryption + lane key
derivation. Indirect dep.
- [`@shade/transport-bridge`](./transport.md) — V3.7 bridge layer (WS /
SSE / long-poll for control envelopes). Orthogonal to V3.11.
- [V5.0 — real-time channels](./V5.0.md) — downstream consumer of the
same DataChannel for voice/video/broadcast.

View File

@@ -1,9 +1,22 @@
# Shade Prekey Server — Dokploy / Docker Compose deployment
#
# Pulls the published image from Gitea's container registry. Change
# `my-project-shade` to something project-specific so you can run multiple
# Shade instances side-by-side (one per project).
#
# Usage:
# docker compose up -d
#
# To build locally from source instead of pulling, uncomment the `build:`
# section and comment out `image:`.
services: services:
shade-prekey: shade-prekey:
image: shade-prekey-server:latest container_name: my-project-shade
build: image: gt.zyon.no/stian/shade-prekey:latest
context: ../.. # build:
dockerfile: packages/shade-server/Dockerfile # context: ../..
# dockerfile: packages/shade-server/Dockerfile
restart: unless-stopped restart: unless-stopped
ports: ports:
- "3900:3900" - "3900:3900"
@@ -13,6 +26,8 @@ services:
- PORT=3900 - PORT=3900
- SHADE_PREKEY_DB_PATH=/data/shade-prekeys.db - SHADE_PREKEY_DB_PATH=/data/shade-prekeys.db
- SHADE_LOG_LEVEL=info - SHADE_LOG_LEVEL=info
- SHADE_STALE_DAYS=30
- SHADE_CLEANUP_INTERVAL_HOURS=24
# Optional: enable the live observer dashboard at /shade-observer/dashboard/ # Optional: enable the live observer dashboard at /shade-observer/dashboard/
# Token must be at least 16 characters. Use a real secret in production. # Token must be at least 16 characters. Use a real secret in production.
# - SHADE_OBSERVER_TOKEN=change-me-must-be-at-least-16-chars # - SHADE_OBSERVER_TOKEN=change-me-must-be-at-least-16-chars

View File

@@ -0,0 +1,28 @@
# Example 07 — E2EE file upload with Shade Streams
Two Bun processes, both running real `Shade` instances. Alice uploads a file
to Bob; the file is chunked, encrypted per-chunk under a per-lane key, sent
over HTTP across 4 parallel lanes, and verified end-to-end.
## Files
- `prekey-server.ts` — boots a `@shade/server` prekey server on port 9991.
- `bob-receiver.ts` — Bob's Shade runtime + a Bun HTTP server that mounts
`Shade.transferRoute()`. Saves incoming files to `./received/`.
- `alice-sender.ts` — Alice's Shade runtime; uploads a file to Bob.
## Run
```bash
# Terminal 1 — prekey server
bun run examples/07-streams-upload/prekey-server.ts
# Terminal 2 — Bob (receiver)
bun run examples/07-streams-upload/bob-receiver.ts
# Terminal 3 — Alice (sender)
bun run examples/07-streams-upload/alice-sender.ts ./path/to/file
```
The receiver writes the decrypted file to `./received/<filename>` and prints
the matching sha256.

View File

@@ -0,0 +1,54 @@
/**
* Alice — sender. Uploads a local file to Bob using 4 parallel lanes,
* 1 MiB chunks, and the default HTTP transport. Prints sender-side
* progress + the verified sha256 on completion.
*
* Usage:
* bun run examples/07-streams-upload/alice-sender.ts <path/to/file>
*/
import { readFile } from 'node:fs/promises';
import { basename } from 'node:path';
import { createShade } from '@shade/sdk';
const PREKEY_URL = 'http://localhost:9991';
const BOB_BASE_URL = 'http://localhost:9992';
const filePath = process.argv[2];
if (filePath === undefined || filePath === '') {
console.error('usage: alice-sender.ts <path/to/file>');
process.exit(2);
}
const bytes = await readFile(filePath);
const name = basename(filePath);
const alice = await createShade({ prekeyServer: PREKEY_URL, address: 'alice' });
alice.configureTransfers({
resolveBaseUrl: async (peer) => {
if (peer === 'bob') return BOB_BASE_URL;
throw new Error(`unknown peer ${peer}`);
},
});
console.log(`[alice] uploading ${name} (${bytes.length} bytes) → bob`);
const handle = await alice.upload({
to: 'bob',
input: new Uint8Array(bytes),
lanes: 4,
chunkSize: 1024 * 1024,
metadata: { name },
onProgress: (p) => {
const pct = p.percent !== undefined ? `${(p.percent * 100).toFixed(1)}%` : '—';
const mbps = (p.bytesPerSecond / (1024 * 1024)).toFixed(2);
process.stdout.write(`\r[alice] ${pct} ${mbps} MB/s ${p.bytesSent}/${p.bytesTotal ?? '?'} B`);
},
});
const result = await handle.done();
process.stdout.write('\n');
console.log(`[alice] done — ${result.bytesSent} B in ${result.durationMs.toFixed(0)} ms`);
console.log(`[alice] sha256 = ${result.sha256}`);
await alice.shutdown();

Some files were not shown because too many files have changed in this diff Show More