25 Commits
v4.0.0 ... main

Author SHA1 Message Date
3243647aa1 release(v4.11.1): ship pre-built dist/ to npm registry
Some checks failed
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
publish-all.ts now does a tsc → dist/ build per package before pack, then
rewrites package.json's main/types/exports to point at the built artefacts
and ensures `files: ["dist"]` so the tarball ships only the built code.
The in-repo package.json is restored in the finally block so dev/typecheck
keep working without a build pass.

Why: strict-mode consumers (Cyndr) were forced to recompile Shade source
under their own tsconfig and tripped on internal `process.env.X` accesses
and implicit-any parameters. Shipping pre-built `.js` + `.d.ts` makes the
strictness contract live entirely inside Shade.
2026-05-21 13:29:52 +02:00
037f994572 release(v4.11.0): streaming Double-Ratchet sub-sessions (ShadeStream)
Some checks failed
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
Answers Vyvern FR shade-ws-streaming-ratchet.md with a first-class
streaming-session API rather than the documented-contract fallback.
The Double-Ratchet crypto was already safe for high-frequency
one-directional use; the send/receive wrapper was not (per-frame
saveSession keystore write; shared per-peer mutex + single stored
session row coupling reuse to the HTTP path).

- @shade/core: stream.ts — identity-bound 3-DH seeding (X3DH-minus-
  prekeys, no prekey-server round trip, mutually authenticated against
  the parent session's pinned identities), bootstrapStreamSession
  reusing init{Sender,Receiver}Session verbatim, in-memory-only
  StreamRatchet (own op-mutex, never persisted, zeroized on close).
  beginStream/acceptStream on ShadeSessionManager; Stream{Closed,
  Handshake}Error; stream.opened/closed events.
- @shade/proto: STREAM_OPEN/OPEN_ACK/FRAME wire (0x31/0x32/0x33),
  additive; inspectEnvelopeType extended.
- @shade/sdk: Shade.openStream/acceptStream → ShadeStream
  (handshakeFrame/handleHandshake/seal/open/close), transport-
  agnostic, independent of encrypt/decrypt queues + parent session,
  identical server (sqlite:) and browser (IndexedDB) — touches no
  storage.
- Tests: 5000-frame one-directional burst (bounded skipped keys + FS
  zeroize), parent-session independence, replay/rewind rejection,
  mutual-auth, proto wire round-trips. Full suite green (1159 pass).
- docs/streaming-sessions.md (R1–R7 contract); SECURITY.md matrix rows.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 11:29:09 +02:00
188c3db56a android: V4.9 + V4.10 Kotlin ports + KeystoreStorage adapter
Some checks failed
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Test / test (push) Has been cancelled
Pure-JVM additions to shade-android (no Android SDK needed):
- V4.9 blob primitives: BlobKdf (HKDF deriveBlobSlotId/Key/SigningSeed),
  BlobAead (nonce||ct||tag with shade-profile-aad-v1:<slot> AAD),
  BlobClient (java.net.http with hand-written canonical JSON signing
  matching TS signPayload output), Profile high-level namespace.
- V4.10 approval helpers: CanonicalProfileBlob schema with denormalized
  trustedApproverFingerprints, build/sign/verify proxy approvals via
  length-prefixed u16 BE UTF-8 canonical signing payload.
- Password KDFs: scrypt + argon2id via Bouncy Castle, NFKC-normalized.
- SessionStateJson at-rest serializer for persistence layer.

Cross-platform vectors (test-vectors/blob.json, approval.json) gate
byte-identical output between TS and Kotlin, including a TS-signed
Ed25519 signature the Kotlin port verifies and reproduces (Ed25519 is
deterministic).

New shade-android-keystore sibling Gradle module (Android-specific):
- KeystoreMasterKey: hardware-backed AES-256-GCM with BIOMETRIC_STRONG
  gating, StrongBox-backed when available, invalidated on enrollment.
- BiometricUnlock: coroutine wrapper around BiometricPrompt with
  tagged cancellation/failure exceptions.
- KeystoreStorage: StorageProvider over biometric-gated AES-encrypted
  SharedPreferences with AAD-bound row keys.

All 25 SDK packages typecheck clean; 104 SDK tests + 24 new Kotlin
tests + 11 cross-platform vector tests all green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 17:38:15 +02:00
1bd7037a6d release(v4.10.0): cross-host approval routing primitives in @shade/sdk
Some checks failed
Test / test (push) Has been cancelled
Builds on V4.9's encrypted profile blob: ships the canonical
profile-blob schema (hosts/clients/trustedApproverFingerprints) and
the build/sign/verify trio for proxy-approval frames. Headless servers
can now route a `linkRequest` to a trusted-approver phone, verify the
phone's Ed25519 signature against the fresh profile blob, and complete
pairing without a GUI host being available.

Length-prefixed binary signing payload so any platform (Kotlin, Swift,
Go) can produce byte-identical signing input from test vectors. No
relay or transport changes — entirely SDK-level.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 17:09:59 +02:00
80c410f518 release(v4.9.0): relay-side encrypted blob primitive + SDK Profile namespace
Some checks failed
Test / test (push) Has been cancelled
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Ships the Prism FR (encrypted-profile-storage-v4.9.md) as a generic
relay-side encrypted blob primitive: deterministically-located,
AEAD-sealed blobs keyed by a 32-byte slotId derived client-side via
HKDF from the user's master key. Unlocks credential-only bootstrap
of new devices into existing E2EE state — no QR, no physical access.

Server: BlobStore interface + Memory/Sqlite/Postgres impls,
createBlobRoutes for GET/PUT/DELETE /v1/blob/:slotId with TOFU pubkey
auth and If-Match CAS (409/412 semantics). Mounted on the same Hono
app as the inbox; SHADE_BLOB_PG_URL / SHADE_BLOB_DB_PATH /
SHADE_DISABLE_BLOB env-var plumbing in standalone.

SDK: createProfileNamespace high-level wrapper (HKDF derivation,
random-nonce AEAD seal, slotId-bound AAD) + low-level BlobClient.
Cross-platform test vectors in test-vectors/blob-storage.json.

New errors: ConflictError (409), PreconditionFailedError (412).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 02:44:42 +02:00
3c0db14904 release(v4.8.5): kill flushOnce 15s success-backoff + per-recipient parallel drain
Some checks failed
Test / test (push) Has been cancelled
Prism filed a per-recipient-flush-concurrency FR pointing at
serial-per-flush. Investigation surfaced the actual culprit:
`scheduleFlush` was using a 15 s backoff on **both** the success and
failure paths, so envelopes enqueued *during* an in-flight flush
sat ~15 s behind the next drain — visible as "10 s of silence then
25-frame burst" on the receiving side under sustained sender output.

Two fixes:

1. `scheduleFlush` now uses 0 ms delay when `flushOnce` delivered
   ≥1 envelope and more is queued (network healthy → drain
   remainder immediately). 15 s reserved for the actual failure
   case where every attempt this round failed. `flushOnce` returns
   `{ delivered, remaining } | null` so concurrent-flush early
   returns don't double-schedule.

2. `flushOnce` groups the outgoing queue by `recipientAddress` and
   drains buckets via `Promise.all`. Per-peer order preserved
   (sequential within a bucket); a slow POST to recipient A no
   longer head-of-line-blocks frames bound for B.

`Inbox.tick` public shape unchanged. `OutgoingQueueStore`
implementations see the same per-entry list/remove/bumpAttempts/
size contract; only cross-recipient interleaving changes.

Tests cover (1) 25-envelope burst behind a 100 ms slow PUT drains
within 1 s, and (2) carol's PUT lands within 150 ms even when bob's
PUT stalls 200 ms.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 22:56:27 +02:00
a98ea8a1bd release(v4.8.4): server-side cross-channel dedup via BridgeDeliveryLog
Some checks failed
Test / test (push) Has been cancelled
V4.8.3 shipped client-side cross-channel dedup hook
(`Inbox.acceptBridgeFrame`), but recipients that didn't migrate to
the new wiring still observed the same envelope twice — once via
WS bridge push, again ~30 s later via inbox-poll. Prism re-verified
the FR after 4.8.3 and asked for a relay-side enforcement so app
code doesn't have to ack-via-DELETE on every bridge frame.

V4.8.4 adds an in-memory `BridgeDeliveryLog` (default 60 s grace,
8192-per-address cap) that records every successful WS / SSE /
long-poll push of `(address, msgId)`. The `/v1/inbox/:addr/fetch`
route filters out blobs in the log's grace window so a recipient
running both a bridge and the 30 s poll cadence sees exactly one
delivery. Cursor advances over the full fetched window so a poll
that straddles a suppressed blob doesn't stall.

The standalone server auto-wires the log between
`createBridgeRoutes` and `createInboxRoutes`. Custom mounts thread
the same instance through `bridgeDeliveryLog` on both factories.

Tests cover WS-then-poll, SSE-then-poll, and a negative control
(non-bridge-pushed blob still comes through inbox-fetch).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 16:31:42 +02:00
d47774ef1c release(v4.8.3): cross-channel msgId dedup + Shade.aliasSession
Some checks failed
Test / test (push) Has been cancelled
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Two follow-ups to the V4.8.2 duplicate-fan-out fixes Prism filed.

1. `Inbox.acceptBridgeFrame(blob)` + shared 4096-entry msgId LRU.
   The relay durably stores blobs and pushes them to every active
   delivery channel; without a cross-channel ack the bridge frame
   ran first and the next inbox-poll re-dispatched the same blob
   ~30 s later, tripping on consumed prekeys. Bridge consumers now
   plumb pushed frames through `acceptBridgeFrame`, which shares
   the dedup gate + ack path with `pollOnce`. Whichever channel
   delivers first wins; the other acks-and-skips. Inbox records
   the msgId before the ack so a parallel poll can't observe an
   in-flight ack window.

2. `Shade.aliasSession(oldLabel, newLabel)`. First-contact forces
   the receiver to label the new session by the relay's sender
   fingerprint hint (`fp:<senderfp>`); the post-decrypt plaintext
   typically announces the peer's real address. Aliasing moves
   session, trusted identity, peer-verification, and identity-
   version under the canonical label. Holds the per-peer mutex on
   both labels (lexicographic order) so concurrent crypto ops can't
   observe a half-moved state. Refuses to overwrite an existing
   session at the new label.

Wire change: `IncomingMessage.expiresAt?` now surfaces the relay's
expiry so receivers can pass bridge frames straight to
`acceptBridgeFrame` without inventing a TTL.

Tests cover bridge-then-poll, poll-then-bridge, aliasSession happy
path, refuse-to-overwrite, and same-label no-op.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 15:49:36 +02:00
8c606ad498 release(v4.8.2): per-from receive serialization + per-connection bridge dedup
Some checks failed
Test / test (push) Has been cancelled
Two interlocking robustness fixes for the duplicate-fan-out / first-contact
class of failures Prism reported.

1. `Shade.receive(from, env)` now queues its `manager.decrypt` step
   per `from` so concurrent dispatches can't race the SessionManager
   ratchet or the StorageProvider (sqlite "database is locked", IDB
   transaction conflicts). User message handlers run *outside* the
   queue so streams + file-RPC's nested `shade.receive` calls don't
   self-deadlock.

2. Bridge WS + SSE handlers now run a per-connection bounded msgId
   LRU as defense-in-depth against any flushTo re-entry (event-storm,
   future refactor). Pending-flush chains are wrapped in `.catch(() =>
   {})` so a transient `ws.send` rejection no longer poisons the
   connection's flush loop.

Tests: storming `inbox.blob_stored` 10× per PUT yields exactly one WS/
SSE frame; 8 concurrent `bob.receive('alice', envelope)` calls keep
the ratchet intact and never surface "database is locked".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 12:13:46 +02:00
680d6386f3 release(v4.8.1): SHADE_DISABLE_RATE_LIMIT env var for single-tenant deploys
Plumbing fix only — both createPrekeyRoutes and createInboxRoutes
already accepted disableRateLimit; standalone.ts just didn't read
the env. Now SHADE_DISABLE_RATE_LIMIT=1 turns off IP rate-limits on
every prekey + inbox route, with a WARN log on startup so operators
see it.

Single-tenant deployments only — multi-tenant relays must leave it
unset. Documented in docs/DEPLOYMENT.md.

Reported by Prism: ~6 pair attempts/hour from a single dev IP +
the sidecar's register call tripped the 5/hour REGISTER_LIMIT every
dev iteration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 00:55:57 +02:00
1fb59a7076 release(v4.8.0): sender-fingerprint attribution + Inbox.start race fix
Two unblocking changes for first-contact flows.

Sender attribution: relay captures shortHash(senderSigningKey) at
PUT time (after signature verification, no new trust surface) and
surfaces it on bridge push (IncomingMessage.from) + inbox-fetch
(FetchedBlob.from) + DecryptHandler raw arg. Apps receiving a prekey
envelope from a never-before-seen peer can now bootstrap X3DH via
shade.receive('fp:<hex>', env) — pre-4.8 the wire envelope didn't
authenticate the sender and there was no out-of-band hint to use.
Idempotent ALTER TABLE migrations for SQLite + Postgres add a
sender_fp TEXT column; legacy rows surface as from=undefined
(inter-version compat).

Inbox.start() race: pre-4.8 start() called register() fire-and-forget
AND schedulePoll(0) synchronously, so the first poll on a fresh
address often beat the register HTTP RTT and got SHADE_NOT_FOUND.
start() now defers; register() success kicks schedulePoll(0). Manual
tick() is unaffected (deliberate user action, no gating).

Both reported by Prism. Tests cover all five acceptance criteria
from the sender-attribution request (PUT capture, bridge surface,
fetch surface, inter-version compat, end-to-end pair smoke) plus
the three from the race-fix request.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 00:11:59 +02:00
594992a183 release(v4.7.0): peer-presence events for instant BroadcastChannel revoke
Adds the bridge-connection-lifecycle signal that closes Prism's
~45s revoke window down to one server→client round-trip (~50ms).

Server (`@shade/inbox-server`):
- `inbox.peer_connected` / `inbox.peer_disconnected` events on the
  0↔1 boundary across WS + SSE bridges. Long-poll deliberately not
  tracked (every poll boundary would flap; push transports are also
  the only ones where instant revoke matters).
- `PresenceTracker` collapses two parallel bridges (e.g. WS + SSE
  during fallback handover) into one connect/disconnect pair.
- `GET /v1/bridge/presence` SSE endpoint: signed query with
  `kind: 'presence'`, `watched: string[]`; on open streams a
  per-address snapshot, then change frames filtered server-side.
  MAX_WATCHED_ADDRESSES = 64. Subscribing does not itself count as
  a peer-bridge connection.
- `createBridgeRoutes` now returns `{ app, websocket, presence }`.

Client (`@shade/transport-bridge`):
- `PresenceBridge.subscribe({ watch, onPresenceChange })` →
  `{ addPeer, removePeer, watching, unsubscribe }`. addPeer/removePeer
  mutate via reconnect with a fresh signed query.
- `signPresenceQuery` helper for non-PresenceBridge consumers.

Tests cover all four acceptance criteria from the Prism request:
server-event smoke, online→offline subscription, address scoping
(carol invisible to a [alice]-only sub), reconnect, plus an
addPeer/removePeer regression.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 23:16:35 +02:00
8746571d2a release(v4.6.1): bind globalThis.fetch in browser-receiver-sensitive call sites
Browsers' Window.fetch is a WebIDL bound operation; storing it as
this.fetchImpl / this.fetchFn and calling via the instance receiver
threw "Illegal invocation" on the first request. Bind once at
construction in InboxClient, LongPollBridge, and SseBridge. Reported
by Prism (multi-device E2EE terminal), blocking every browser
consumer of the v4.6 transport stack on inbox.start() / bridge.connect().

WsBridge unaffected (uses WebSocket). Node/Bun fetch tolerates a free
receiver, so the bug never surfaced server-side — added regression
tests that install a strict-receiver globalThis.fetch to catch the
issue without an actual browser harness.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 23:00:58 +02:00
2c400d7094 release(v4.6.0): broadcast channels — Signal sender-keys for one-to-many fan-out
Some checks failed
Test / test (push) Has been cancelled
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
Lands the broadcast-channel primitive Prism asked for in
Docs/shade-feature-request-sender-keys.md. The crypto in
@shade/core/sender-keys.ts was already in place; this release wires
it up as a first-class app-facing API, adds the persistence schema
across all six storage backends (memory, sqlite, indexeddb +
encrypted variants), introduces wire type 0x21 in @shade/proto,
and ships Prism's three acceptance tests verbatim.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 15:55:34 +02:00
2b1b4d6630 release(v4.5.0): browser-side encrypted storage + multi-factor unlock
Adds the foundations Prism's web client (and any future browser-based
Shade app) needs: at-rest-encrypted IndexedDB storage that mirrors the
SQLite backend byte-for-byte at the AAD/nonce level, browser-safe
subpath imports so Vite/webpack/esbuild stop hitting bun:sqlite, and
KeyManager support for argon2id and N-factor composite unlock.

@shade/storage-encrypted
- EncryptedIndexedDBStorage (subpath: /idb) — full StorageProvider
  using one object store per _enc table; reuses aeadSeal/aeadOpen +
  row-codec sealers so a row sealed under the SQLite or Postgres
  backend decrypts under IDB given the same KeyManager.
  bumpPeerIdentityVersion is atomic under one IDB transaction.
- KeyManager argon2id source — memory-hard KDF for low-entropy
  secrets (PINs). Backed by @noble/hashes/argon2 (already a transitive
  dep). DEFAULT_ARGON2ID exported (m=64 MiB, t=3, p=1).
- KeyManager composite source — HKDF-combine N sub-sources into one
  master. Every source mandatory; order significant by design;
  composite-of-composite rejected; optional info string for app-level
  domain separation.
- Subpath exports (/crypto, /sqlite, /postgres, /idb) plus a `browser`
  condition on the default import that resolves to a barrel
  excluding the Bun- and Postgres-specific entries. Browser bundles
  no longer pull bun:sqlite transitively.

Tests
- 73 tests in shade-storage-encrypted (was 31). New coverage:
  argon2id determinism + reject paths, composite same-factors → same
  master, wrong-PIN/passphrase/order-swap → different master, info
  domain separation, all 28 StorageProvider methods on
  EncryptedIndexedDBStorage, fingerprint-mismatch rejection, and
  cross-impl roundtrip with EncryptedSQLiteStorage proving the AAD/
  nonce derivation is implementation-agnostic.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 10:58:49 +02:00
dbb3a090d8 release(v4.4.0): public accessor for device identity public key
Expose the local device's 32-byte Ed25519 identity public key on Shade
so apps can hand it to their own backend at enrollment time for
signature verification, key pinning or per-device safety-number
computation. Closes the gap that forced consumers to ship placeholder
random bytes their backend could store but never verify against.

- @shade/sdk Shade.identityPublicKey: Promise<Uint8Array> — getter
  mirrors the existing fingerprint accessor. Throws pre-init,
  reflects the current key after rotate(), retired key preserved in
  retired-identities storage per existing grace-period contract.
  Private key remains unreachable.
- Test in shade-sdk/tests/sdk.test.ts: round-trip match against the
  underlying storage's signingPublicKey, plus value updates after
  rotate().
- Lockstep version bump 4.3.0 → 4.4.0 across all 25 packages.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 17:58:45 +02:00
f5f42fe557 release(v4.3.0): browser persistence via @shade/storage-indexeddb
Ship an official IndexedDB-backed StorageProvider so browser-based Shade
consumers persist identity, prekeys, sessions, retired identities,
peer-verification state and stream-resume rows across tab refresh and
browser restart. Closes the gap that forced browser apps onto
storage:"memory" (regenerated identity each load, orphaned device
records server-side).

- New package @shade/storage-indexeddb (4.3.0): full StorageProvider
  conformance, schema v1, idb-backed; bumpPeerIdentityVersion is wrapped
  in a single readwrite IDB transaction (atomic, vs SQLite's
  read-then-upsert race).
- @shade/sdk resolveStorage() accepts { type: 'indexeddb', dbName? } via
  dynamic import (lazy, optional dep — same pattern as
  @shade/storage-postgres). Named StorageSpec type now reused by
  ResolvedConfig.
- Tests: 16 new tests in shade-storage-indexeddb (StorageProvider
  surface + peer-verifications + full E2EE conversation surviving a
  simulated tab reload). Run on fake-indexeddb.
- Lockstep version bump 4.2.1 → 4.3.0 across all 25 packages.
- Publish scripts updated to include the new package.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 17:35:02 +02:00
b77b7e771c release(v4.2.1): fix concurrent-ratchet desync via OutboundQueue waiter cursor
Some checks failed
Publish / publish (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Pull-mode httpClient + drainer + parallel RPCs against the same peer
deteriorated after ~10s with `DecryptionError`. Two bugs combined:

- `OutboundQueue.enqueue` woke `drain` waiters with a `since=0`
  snapshot, replaying already-processed events into
  `Shade.acceptTransferEnvelope` → `manager.decrypt` twice. The
  duplicate consumed an already-used skipped key and corrupted the
  Double Ratchet receive chain.

- `ratchetDecrypt` then propagated the corruption: a same-DH
  message behind the chain with no cached skipped key fell through
  to `kdfChainKey` on the ahead state and rewound `chain.counter`,
  permanently desyncing the chain.

Fix `OutboundQueue` to honor each waiter's `since`, and harden
`ratchetDecrypt` so any future duplicate fails cleanly without
mutating state. Adds regression coverage at all three layers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 22:58:26 +02:00
7520b11b25 release(v4.2.0): pull-mode streams for browser @shade/files
Some checks failed
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
4.1.0's HTTP RPC for browsers capped at inline payloads (≤ 256 KiB).
4.2.0 unlocks streams: server queues outbound chunks + control
envelopes per peer, browser long-polls the queue. Browser-to-server
writes ride the existing /v1/transfer/<id>/chunk POST routes
unchanged.

For Dispatch this unlocks mod-jar uploads (50 MB) and world-backup
downloads (100+ MB) — the actual reason browser-side @shade/files
matters.

### New API

@shade/sdk:
- shade.transferQueueRoute(opts?) — Hono app with /queue +
  /v1/transfer/* routes. Auto-configures the queue transport.
- shade.configureTransfers extended: transport + envelopeTransport
  override slots; resolveBaseUrl optional when both supplied.

@shade/transfer:
- OutboundQueue — per-peer monotonic event log with long-poll
  semantics, idle-eviction GC, ring-buffered to maxEventsPerPeer.
- QueueTransferTransport — enqueues instead of POSTing.

@shade/files:
- httpClient({ outboundQueueUrl, transferBaseUrl }) — when set,
  starts a long-poll drainer + builds a streams-bridge. fs.read /
  fs.write of >256 KiB work end-to-end.
- startQueueDrainer(shade, opts) — exported helper for advanced
  consumers driving their own drainer.

### Implementation notes

- ClientStreamsBridge's TransformStream had HWM=0 by default which
  stalled the drainer's await chain at chunk 4 (writer.write pended
  before the consumer's reader was attached). Bumped to HWM=64 so
  the receive loop can buffer ahead of the consumer.

### Tests

3 new integration tests in tests/integration/http-rpc-streams.test.ts:
4 MiB streamed read round-trip, inline-only error path, idle-timeout
long-poll behaviour.

Wire-compatible. Source-compatible. Lockstep bump to 4.2.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 23:27:06 +02:00
da93b97cce release(v4.1.0): browser-friendly HTTP RPC for @shade/files
Some checks failed
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
Default shade.files.client(peer) requires both peers to be mutually
addressable over HTTP — the response round-trips through
Shade.deliverControlEnvelope (POST to peer's /v1/transfer/control).
Browser tabs can't host an HTTP server, so they couldn't consume
@shade/files at all. Dispatch's filutforsker (admin-panel browser UI)
is the canonical use-case.

This release adds a parallel request-response transport: one POST per
RPC, encrypted envelope in the body, encrypted response in the same
HTTP response. No inbound channel needed on the client.

### New API

- shade.files.rpcRoute(opts?) — Hono app exposing POST /rpc.
- shade.files.httpClient(peer, opts) — request-response FileClient.
- FilesNamespace.serve(handler, { inlineOnly: true }) — skip streams-
  bridge (and its configureTransfers pre-condition); also skip
  channel-based dispatch so requests aren't double-dispatched.

### Limitations (v1)

Inline only (≤ 256 KiB). Streamed reads/writes throw clear errors
directing to shade.files.client(peer) on a server-to-server deploy.

### Tests

7 integration tests in tests/integration/http-rpc.test.ts covering
round-trip + negative cases (sender header, empty/garbage body,
maxBodyBytes, rpcRoute-without-serve).

### Symmetry

Mirrors @shade/server's shade-auth-middleware: encrypted envelope in
request body, decrypted via existing ratchet, response in same HTTP
roundtrip. No WebSocket, no SSE, no outbound from server.

Wire-compatible. Source-compatible. Lockstep bump to 4.1.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 22:08:14 +02:00
0bdf9e859c release(v4.0.2): consumer-strict reader-shape fixes
Some checks failed
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
4.0.1's typecheck gate compiled each package internally against
lib: ["ES2022"]. That doesn't catch types that only fail when
*consumer* code (lib: ["DOM"] + exactOptionalPropertyTypes) tries to
assign a native browser type into one of our locally-defined narrower
types. Dispatch hit one such case in @shade/files inline-threshold.ts.

This release adds a tests/consumer-strict/ smoke project to the
pre-publish gate. It compiles a tiny "as if I were a downstream app"
TS file against:

  lib: ["ES2022", "DOM", "DOM.Iterable"]
  types: ["bun-types"]
  exactOptionalPropertyTypes: true
  strict: true
  paths → packages/*/src/index.ts

scripts/typecheck-all.ts now runs the smoke after per-package checks.
Both must pass before publish:dry / publish:all proceeds.

### Fixed
- @shade/files inline-threshold.ts: MinimalReader<T> rewritten as the
  explicit disjoint union { done:false, value:T } | { done:true,
  value?: T | undefined } that's assignable from every native reader
  shape (bun, DOM, node:stream/web). Fixes the
  "ReadableStreamReadResult is not assignable" Dispatch reported.
- @shade/files streams-bridge (client + server): stash setTimeout
  return in a local before .unref?.() via { unref?: () => void } cast.
  Fluent .unref?.() failed under lib: ["DOM"] (setTimeout returns
  number there).
- @shade/sdk background.ts: same setInterval .unref?.() fix.

Wire-compatible. No API shape changed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 19:51:46 +02:00
70e319fef8 release(v4.0.1): strict-TS publishability fixes
Some checks failed
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
4.0.0 shipped TypeScript source as published main/types, but several
files only compiled inside the monorepo. Consumer projects (Dispatch,
etc.) running their own strict tsc against our published source hit:

- @shade/key-transparency: 4 noUnusedLocals violations
  (IndexAbsenceProof, IndexInclusionProof, IndexProofWire, nodeHash)
- @shade/sdk: KT verifier callbacks returned Promise<unknown> instead
  of Promise<STHWire> / Promise<{ proof: string[] }>
- @shade/sdk: thumbnail.ts globalThis cast collided with consumer's
  lib.dom-supplied createImageBitmap signature
- @shade/files: cycle with @shade/sdk produced "this is not assignable
  to type 'Shade'" because hoisted node_modules layouts duplicated the
  Shade class. Broken by replacing `import type { Shade }` with a
  local structural ShadeBridge interface.
- @shade/storage-encrypted: KeyUsage (lib.dom) used under
  lib: ["ES2022"]
- @shade/transport-bridge: ReadableStreamDefaultReader<any> ↔
  <Uint8Array> mismatch
- @shade/keychain / @shade/dashboard / @shade/storage-encrypted
  tsconfig rootDir / include hygiene

Tooling: scripts/typecheck-all.ts runs `bunx tsc --noEmit` against
every workspace package's tsconfig and fails on any error. Wired into
publish:dry / publish:all and publish-shade.sh as a hard gate so this
class of bug cannot recur.

All 24 packages bumped to 4.0.1 in lockstep.

Migration: <ShadeFilesProvider> now requires an explicit `files` prop
(pass `shade.files`). Wire format unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 19:36:47 +02:00
f301b391a5 docs(archive): close out Status fields on V2.x backlog + V3.12 design notat
Some checks failed
Test / test (push) Has been cancelled
V4.0 acceptance §"All docs/V*.md arkivert med DONE-status" requires
every archived plan to carry an explicit Status field. V2.1 / V2.2 /
V2.3 inherited their pre-status format; V3.12-DESIGN was still
"Approved". Mark all four as Done with a one-line pointer to where
the work actually landed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 19:04:47 +02:00
40766c60f4 docs(readme): full 4.0 GA refresh — status table, all V3.x surfaces, audit pointers
Some checks failed
Test / test (push) Has been cancelled
- Status table (V3.2 → V3.12 all green; audit + soak harness ready)
- "What you get" expanded: storage encryption, fingerprint gates,
  inbox, bridge, web workers, recovery, KT, observability, soak
- Quick start: opt-in surface examples (gates, workers, WebRTC, KT)
  + at-rest encryption snippet + migrate-storage CLI invocation
- Architecture diagram refreshed: container now exposes /v1/inbox/*,
  /v1/bridge/*, /v1/kt/*, ops endpoints; WebRTC P2P pipe documented
- Packages table: all 24 entries at 4.0.0 (added observability,
  keychain, storage-encrypted, transport-bridge, etc.)
- Publishing section: 4.0.x examples; soak harness section added
- Security properties: V3.2 / V3.3 / V3.6 / V3.8 / V3.10 / V3.11
  rows added alongside the existing forward-secrecy / PCS list
- Documentation: grouped into Operator+integrator, Per-surface,
  Threat-model+audit, Migration+history, Examples
- Container includes: full V4.0 surface (inbox, bridge, KT, ops
  probes); image pinned at gt.zyon.no/stian/shade-prekey:4.0.0

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:58:38 +02:00
de25b19033 fix(publish): break recursion in publish-shade.sh → publish-all.ts
Some checks failed
Test / test (push) Has been cancelled
publish-shade.sh used to call `bun run publish:all`, which in turn was
wired to call publish-shade.sh (after the V4.0 cleanup). Point it
directly at scripts/publish-all.ts so the interactive flow runs the
TS publisher without re-entering itself.

Verified: dry-run from publish-shade.sh now packs all 24 @shade/*@4.0.0
packages cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:53:16 +02:00
186 changed files with 19702 additions and 356 deletions

3
.gitignore vendored
View File

@@ -2,3 +2,6 @@ node_modules/
dist/ dist/
*.tsbuildinfo *.tsbuildinfo
.DS_Store .DS_Store
**/.tmp-*.db
**/.tmp-*.db-shm
**/.tmp-*.db-wal

File diff suppressed because it is too large Load Diff

322
README.md
View File

@@ -3,37 +3,82 @@
End-to-end encryption library implementing the Signal Protocol (X3DH + Double Ratchet) for TypeScript/Bun. Drop into any project — frontend, backend, mobile — to get forward secrecy, post-compromise recovery, and self-healing security. End-to-end encryption library implementing the Signal Protocol (X3DH + Double Ratchet) for TypeScript/Bun. Drop into any project — frontend, backend, mobile — to get forward secrecy, post-compromise recovery, and self-healing security.
> **4.0.0 — General Availability.** All V3.1 → V3.12 work is merged, > **4.0.0 — General Availability.** All V3.1 → V3.12 work is merged,
> the cross-platform vector suite is green on TS + Kotlin, the threat > the cross-platform vector suite is green on TS + Kotlin (1000 / 1000
> model has been refreshed for every new surface, and the core stack > + 11 / 11), the threat model has been refreshed for every new
> (X3DH, ratchet, storage encryption, recovery, WebRTC P2P, Key > surface, and the core stack (X3DH, ratchet, storage encryption,
> Transparency) has been packaged for external review. The wire > recovery, WebRTC P2P, Key Transparency) has been packaged for
> format is **unchanged from 0.4.x** — 4.0 peers interoperate with > external review. The wire format is **unchanged from 0.4.x** — 4.0
> 0.4.x peers byte-for-byte. See > peers interoperate with 0.4.x peers byte-for-byte. See
> [MIGRATION.md § 0.3.x → 4.0](./MIGRATION.md#migrating-from-03x-to-40-ga) > [MIGRATION.md § 0.3.x → 4.0](./MIGRATION.md#migrating-from-03x-to-40-ga)
> for the upgrade path and [CHANGELOG.md § 4.0.0](./CHANGELOG.md) for > for the upgrade path and [CHANGELOG.md § 4.0.0](./CHANGELOG.md) for
> the consolidated release notes. Voice / Video have been moved to > the consolidated release notes. Voice / Video have been moved to
> [V5.0](./docs/V5.0.md), to be built on top of the frozen 4.0 > [V5.0](./docs/V5.0.md), to be built on top of the frozen 4.0
> baseline. > baseline.
## Status
| Area | 4.0 status | Pointers |
|------|-----------|----------|
| Protocol core (X3DH + ratchet + sender keys) | ✅ Done — frozen | [`packages/shade-core`](./packages/shade-core) |
| Storage encryption (V3.2) | ✅ Done — opt-in `EncryptedSQLiteStorage` / `EncryptedPostgresStorage`, key sources: passphrase / OS keychain / app-injected | [`docs/storage-encryption.md`](./docs/storage-encryption.md) |
| Fingerprint gates & trust UX (V3.3) | ✅ Done — `Shade.beforeFirstLargeFile / beforeBackupImport / beforeNewDeviceTrust` | [`docs/trust-ux.md`](./docs/trust-ux.md) |
| Observability v2 (V3.4) | ✅ Done — OpenTelemetry-shaped events, `/metrics`, observer dashboard | [`docs/observability.md`](./docs/observability.md) |
| Android parity & cross-platform CI (V3.5) | ✅ Done — TS + Kotlin vector-gate live; Android `KeystoreStorage` is post-GA | [`android/shade-android/README.md`](./android/shade-android/README.md), [`docs/cross-platform.md`](./docs/cross-platform.md) |
| Async store-and-forward (V3.6) | ✅ Done — `@shade/inbox` + `@shade/inbox-server` | [`docs/inbox.md`](./docs/inbox.md) |
| Transport bridge (V3.7) | ✅ Done — SSE / long-poll / WS adapters | [`docs/transport.md`](./docs/transport.md) |
| Web Workers crypto (V3.8) | ✅ Done — lane keys never cross the thread boundary | [`docs/web-workers.md`](./docs/web-workers.md) |
| Rich file metadata + thumbnails (V3.9) | ✅ Done — in `@shade/files` | [`docs/files.md`](./docs/files.md) |
| Social key recovery (V3.10) | ✅ Done — Shamir + AEAD-gated reconstruction + guardian widgets | [`docs/recovery.md`](./docs/recovery.md) |
| WebRTC P2P transport (V3.11) | ✅ Done — `RTCDataChannel` with `MultiTransportFallback([webrtc, http])` | [`docs/webrtc.md`](./docs/webrtc.md) |
| Key Transparency (V3.12) | ✅ Done — opt-in Merkle log, signed STH, witness gossip | [`docs/key-transparency.md`](./docs/key-transparency.md) |
| External crypto review | 🟡 Bundle ready — review window open after tag | [`docs/audit/REVIEW-BUNDLE.md`](./docs/audit/REVIEW-BUNDLE.md) |
| Soak (≥ 2 weeks under load) | 🟡 Harness shipped — operator runs it | [`scripts/soak.ts`](./scripts/soak.ts) (`bun run soak --hours 336`) |
| Voice / Video / Broadcast | 🔜 V5.0 — built on top of frozen 4.0 stack | [`docs/V5.0.md`](./docs/V5.0.md) |
## What you get ## What you get
**Protocol core**
- **X3DH** initial key agreement (works asynchronously via prekey bundles) - **X3DH** initial key agreement (works asynchronously via prekey bundles)
- **Double Ratchet** for per-message forward secrecy and post-compromise security - **Double Ratchet** for per-message forward secrecy and post-compromise security
- **Self-authenticated prekey server** (Hono, Docker-ready) with rate limiting, metrics, health checks - **Sender keys** for group ratchet (1:N broadcast key derivation)
- **Persistent storage backends**: SQLite (zero-config) and PostgreSQL (Drizzle)
- **Identity rotation** with grace period for old sessions - **Identity rotation** with grace period for old sessions
- **Safety numbers** (Signal-style fingerprints) for out-of-band verification - **Safety numbers** (Signal-style fingerprints) for out-of-band verification
- **Constant-time comparisons** and **memory zeroization** for hardened operation - **Constant-time comparisons** and **memory zeroization** for hardened operation
- **Binary wire format** that's significantly smaller than JSON - **Binary wire format** (`@shade/proto`) — significantly smaller than JSON
**Storage**
- **Persistent backends**: SQLite (zero-config, `bun:sqlite`) and PostgreSQL (Drizzle, FOR UPDATE SKIP LOCKED)
- **Crash-safe** — sessions survive container restarts, power outages, SIGKILL - **Crash-safe** — sessions survive container restarts, power outages, SIGKILL
- **Live observability** — bundled dashboard SPA + embeddable React widgets to see what's happening between every step - **At-rest encryption (V3.2, opt-in)** — AES-256-GCM under per-(table,column) DEKs; key sources: passphrase (scrypt), OS keychain (`@shade/keychain`), or app-injected. Online re-key, no downtime.
- **E2EE file transfers** — multi-lane chunked uploads/downloads with resume, integrity checks, and HTTP/WS fallback (`@shade/streams` + `@shade/transfer`)
- **WebRTC P2P transport (V3.11)** — opt-in `RTCDataChannel` upload path with public-STUN defaults, TURN-relay support, glare-safe peer pool, and automatic `MultiTransportFallback` back to HTTP when NAT traversal fails (`@shade/transport-webrtc`, [docs/webrtc.md](./docs/webrtc.md)) **Servers**
- **Web Workers crypto** — AEAD, HKDF, HMAC, X25519, Ed25519 and per-lane stream state run in a dedicated worker. 100 MB+ uploads stay smooth without frame drops, lane keys never cross the thread boundary (`@shade/crypto-web/worker`, [docs/web-workers.md](./docs/web-workers.md)) - **Self-authenticated prekey server** (`@shade/server`, Hono, Docker-ready) with rate limiting, metrics, health checks
- **E2EE filesystem RPC** — typed `list/stat/mkdir/delete/move/read/write/getThumbnail` + custom ops between peers, with rate-limit, retention, and fingerprint-gate hooks (`@shade/files`) - **Async store-and-forward relay** (`@shade/inbox-server`) — TTL-bound ciphertext blobs, signed PUT/FETCH/ACK, idempotent on `(address, msgId)`, per-recipient quota
- **Async store-and-forward** — deliver to offline recipients via a relay that holds ciphertext-only blobs with TTL, idempotent PUT, signed fetch/ack, and an `onMessageQueued` push-trigger hook (`@shade/inbox` + `@shade/inbox-server`) - **Bridge transports** (`@shade/transport-bridge`) — WS → SSE → long-poll fallback chain for clients that can't keep a WebSocket open. Same `IncomingMessage` shape across all three.
- **Social key recovery** — Shamir-split your identity to `n` guardians; any threshold-many `k` together restore it on a new device. No centralized recovery agent; OOB-fingerprint gate on every guardian release; AES-GCM authenticates the reconstruction (`@shade/recovery` + `<RecoverySetup />` / `<RecoveryRequest />` / `<RecoveryApprove />` widgets, [docs/recovery.md](./docs/recovery.md)) - **Standalone container** — one image bundles prekey + inbox + bridge + transfer + KT + observer
- **Key Transparency (V3.12)** — opt-in append-only Merkle log over the prekey server. Every `register` / `delete` becomes a signed leaf; every bundle-fetch carries an inclusion proof; an Ed25519-signed Tree Head ties roots to a fixed `log_id`. A `LightWitness` cross-checks STHs across clients so a malicious server that splits its view or rewrites history is caught (`@shade/key-transparency`, [docs/key-transparency.md](./docs/key-transparency.md))
**Trust UX**
- **Fingerprint gates (V3.3)** — `Shade.beforeFirstLargeFile(threshold, handler)`, `beforeBackupImport`, `beforeNewDeviceTrust`. Gates raise `FingerprintNotVerifiedError` on the operations that matter, default-warn TOFU otherwise.
- **`<FingerprintCompare />` / `<FingerprintGate />`** widgets for the matching UI side.
**File transfer & filesystem**
- **E2EE file transfers** (`@shade/streams` + `@shade/transfer`) — multi-lane chunked uploads/downloads with resume, integrity checks (per-lane sha256 + overall sha256), HTTP/WS fallback, `MultiTransportFallback` for N-ary demotion
- **WebRTC P2P transport (V3.11, opt-in)** — `RTCDataChannel` chunk path with public-STUN defaults, TURN-relay support, glare-safe peer pool, automatic fallback to HTTP when NAT traversal fails (`@shade/transport-webrtc`)
- **Web Workers crypto (V3.8, opt-in)** — AEAD, HKDF, HMAC, X25519, Ed25519 and per-lane stream state run in a dedicated worker. 100 MB+ uploads stay smooth without frame drops; lane keys never cross the thread boundary (`@shade/crypto-web/worker`)
- **E2EE filesystem RPC** (`@shade/files`) — typed `list/stat/mkdir/delete/move/read/write/getThumbnail` + custom ops, with rate-limit, retention, fingerprint-gate, and metrics hooks. React hooks under `@shade/files/react`.
**Recovery**
- **Social key recovery (V3.10)** — Shamir-split your identity to `n` guardians; any threshold-many `k` together restore it on a new device. No centralized recovery agent; OOB-fingerprint gate per guardian release; AES-GCM-authenticated reconstruction (`@shade/recovery` + `<RecoverySetup />` / `<RecoveryRequest />` / `<RecoveryApprove />`)
**Verifiable distribution**
- **Key Transparency (V3.12, opt-in)** — append-only Merkle log over the prekey server. Every `register` / `delete` becomes a signed leaf; every bundle-fetch carries an inclusion proof; an Ed25519-signed Tree Head ties roots to a fixed `log_id`. A `LightWitness` cross-checks STHs across clients so a malicious server that splits its view or rewrites history is caught (`@shade/key-transparency`).
**Observability**
- **Live observability** — OpenTelemetry-shaped events, bundled dashboard SPA + embeddable React widgets to see what's happening between every step (`@shade/observability` + `@shade/observer` + `@shade/dashboard`)
**Tooling**
- **CLI** — `shade init` scaffolder, `shade migrate-storage` (V3.2), `shade rotate-storage-key`, `shade fingerprint`, `shade rotate`, `shade peer`, `shade dashboard`, `shade doctor`, `shade backup` (`@shade/cli`)
- **Soak harness** — `bun run soak --hours 336` for the 2-week GA-stable window (`scripts/soak.ts`)
## Quick start ## Quick start
@@ -84,7 +129,38 @@ const plaintext = await shade.receive('alice@example.com', incomingEnvelope);
console.log(await shade.fingerprint); console.log(await shade.fingerprint);
``` ```
Need to ship a file or expose a filesystem to a peer? `Shade.files` is the high-level entrypoint: ### Opt-in surfaces (V3.x → 4.0 GA)
All of these are off by default. Wire them only where you need them.
```ts
// V3.3 — Fingerprint gates: enforce verification on the operations that matter
shade.beforeFirstLargeFile(10 * 1024 * 1024, async ({ peer, fingerprint }) => {
return await ui.confirmSafetyNumberMatches(peer, fingerprint);
});
shade.beforeBackupImport(async ({ embeddedFingerprint }) => { /* ... */ });
shade.beforeNewDeviceTrust(async ({ peer, oldFp, newFp }) => { /* ... */ });
// V3.8 — Web Workers crypto: opt-in, lane keys stay off the main thread
shade.configureWorkerCrypto({
workerUrl: new URL('@shade/crypto-web/worker', import.meta.url),
});
// V3.11 — WebRTC P2P transport: file transfers ride RTCDataChannel where NAT allows
import { nativeRtcFactory } from '@shade/transport-webrtc';
shade.configureWebRTC({
factory: nativeRtcFactory(),
iceServers: [{ urls: 'stun:stun.l.google.com:19302' }],
});
// V3.12 — Key Transparency: detect server-side bundle swaps
const shade = await createShade({
prekeyServer: 'https://shade.example.com',
keyTransparency: { mode: 'observe-strict', logPublicKey: PINNED_KEY_BYTES_32 },
});
```
### Files RPC (`@shade/files`)
```ts ```ts
// Server side — Bob exposes a virtual filesystem // Server side — Bob exposes a virtual filesystem
@@ -97,13 +173,13 @@ const stop = await shade.files.serve({
// Client side — Alice consumes Bob's filesystem // Client side — Alice consumes Bob's filesystem
const fs = await shade.files.client('bob'); const fs = await shade.files.client('bob');
await fs.write('/photos/cover.png', new Uint8Array(...)); // auto inline/streams await fs.write('/photos/cover.png', new Uint8Array([/* ... */])); // auto inline/streams
const result = await fs.read('/photos/cover.png'); const result = await fs.read('/photos/cover.png');
``` ```
Files ≤ 256 KiB ride inline in the RPC envelope; larger files automatically promote to multi-lane `@shade/transfer` streams with sha256 integrity. See [`docs/files.md`](./docs/files.md) for the full API. Files ≤ 256 KiB ride inline in the RPC envelope; larger files automatically promote to multi-lane `@shade/transfer` streams with sha256 integrity. See [`docs/files.md`](./docs/files.md) for the full API.
Or use the lower-level packages directly if you need full control: ### Lower-level access
```ts ```ts
import { ShadeSessionManager } from '@shade/core'; import { ShadeSessionManager } from '@shade/core';
@@ -117,20 +193,43 @@ const manager = new ShadeSessionManager(
await manager.initialize(); await manager.initialize();
``` ```
### At-rest encryption (V3.2)
```ts
import { KeyManager, EncryptedSQLiteStorage } from '@shade/storage-encrypted';
const km = await KeyManager.open({
kind: 'passphrase',
passphrase: process.env.SHADE_STORAGE_PASSPHRASE!,
salt: loadSaltFromDisk(),
});
const storage = await EncryptedSQLiteStorage.open({
dbPath: '/data/shade-client.db',
keyManager: km,
});
```
To migrate an existing 0.3.x SQLite DB in place:
```bash
shade migrate-storage \
--key-source passphrase \
--passphrase "$SHADE_STORAGE_PASSPHRASE" \
--salt-file /data/shade-client.db.salt
```
## Architecture — keys vs. payloads ## Architecture — keys vs. payloads
Shade splits the network into two planes. The **prekey server** only Shade splits the network into a **public-key plane** (the prekey
sees public keys; **everything else** rides the encrypted Double server) and an **encrypted plane** (everything else). The prekey
Ratchet between peers. If you remember nothing else from this README, server only sees public key material. If you remember nothing else
remember this picture: from this README, remember this picture:
``` ```
Shade Prekey Server (Hono, public keys only) Shade Prekey Container (Hono public keys only)
POST /v1/keys/register (signed) /v1/keys/* /v1/inbox/* /v1/bridge/* /v1/transfer/*
GET /v1/keys/bundle/:address /v1/kt/* /metrics /healthz /ready
POST /v1/keys/replenish (signed)
DELETE /v1/keys/:address (signed)
┌──────────────────┴──────────────────┐ ┌──────────────────┴──────────────────┐
│ │ │ │
@@ -142,11 +241,20 @@ remember this picture:
│◄── Double Ratchet messages ────────►│ ← end-to-end, │◄── Double Ratchet messages ────────►│ ← end-to-end,
│ (ratchet 0x02 / chunks 0x11) │ never on the │ (ratchet 0x02 / chunks 0x11) │ never on the
│ │ prekey server │ │ prekey server
◄── @shade/transfer chunks ─────────►│
POST /v1/transfer/:id/chunk │ ← peer-to-peer ◄── @shade/transfer chunks ─────────►│ ← peer-to-peer
GET /v1/transfer/:id/state │ HTTP, opaque POST /v1/transfer/:id/chunk │ HTTP, opaque
│ ciphertext GET /v1/transfer/:id/state │ ciphertext
│◄── @shade/inbox blobs (offline) ───►│ ← TTL-bound
│ POST /v1/inbox/:address │ ciphertext-only
│ POST /v1/inbox/:address/fetch │ relay
│◄── @shade/transport-webrtc ────────►│ ← optional P2P
│ RTCDataChannel `shade-transfer/v1` │ `MultiTransportFallback`
│ │ auto-demotes to HTTP
SQLiteStorage / PostgresStorage SQLiteStorage / PostgresStorage SQLiteStorage / PostgresStorage SQLiteStorage / PostgresStorage
+ EncryptedSQLiteStorage (V3.2) + EncryptedSQLiteStorage (V3.2)
(private keys + sessions) (private keys + sessions) (private keys + sessions) (private keys + sessions)
``` ```
@@ -155,17 +263,23 @@ remember this picture:
- Identity public keys (Ed25519 + X25519) - Identity public keys (Ed25519 + X25519)
- Signed prekeys + one-time prekey bundles - Signed prekeys + one-time prekey bundles
- Registration / replenish / delete writes, all Ed25519-signed - Registration / replenish / delete writes, all Ed25519-signed
- Operator-only metrics, `/health`, and the optional observer - (V3.6) Inbox ciphertext blobs with TTL — same container, separate
dashboard routes; the relay only sees `address || msgId || ciphertext-bytes`
- (V3.7) Bridge transports (SSE / long-poll / WS) — also delivered by
the same Hono app for clients that can't hold a WebSocket
- (V3.12, opt-in) KT inclusion proofs + signed tree heads on
`/v1/kt/*` — verifiable distribution
- Operator-only metrics and the optional observer dashboard
### What does **not** go via the prekey server ### What does **not** go via the prekey server
- **Message plaintext, ever.** Encrypted ratchet envelopes flow peer- - **Message plaintext, ever.** Encrypted ratchet envelopes flow peer-
to-peer over whatever transport you choose (HTTP, WebSocket, your to-peer over whatever transport you choose (HTTP, WebSocket, your
own broker). own broker, or the inbox relay above — which carries ciphertext only).
- **File chunks.** `@shade/transfer` POSTs ciphertext directly to the - **File chunks.** `@shade/transfer` POSTs ciphertext directly to the
receiver's `/v1/transfer/:streamId/chunk` route — the prekey server receiver's `/v1/transfer/:streamId/chunk` route — the prekey server
is not involved. is not involved. With V3.11 + `configureWebRTC()`, chunks ride
`RTCDataChannel` peer-to-peer; the relay is bypassed entirely.
- **Identity private keys.** They never leave the device's storage. - **Identity private keys.** They never leave the device's storage.
- **Filesystem RPC.** `@shade/files` rides the Double Ratchet for - **Filesystem RPC.** `@shade/files` rides the Double Ratchet for
control + small payloads, then promotes to direct `@shade/transfer` control + small payloads, then promotes to direct `@shade/transfer`
@@ -176,6 +290,8 @@ remember this picture:
The prekey server is metadata-bearing (see `THREAT-MODEL.md § 2`): The prekey server is metadata-bearing (see `THREAT-MODEL.md § 2`):
it sees who registers, who fetches whose bundle, and when. It does it sees who registers, who fetches whose bundle, and when. It does
**not** see message contents, transfer contents, or session state. **not** see message contents, transfer contents, or session state.
**V3.12 Key Transparency** (opt-in) makes its bundle distribution
*verifiable* so a malicious server that swaps a bundle is caught.
For the full threat model and mitigations, read For the full threat model and mitigations, read
[THREAT-MODEL.md](./THREAT-MODEL.md). For deployment-time guarantees, [THREAT-MODEL.md](./THREAT-MODEL.md). For deployment-time guarantees,
@@ -183,53 +299,84 @@ read [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md).
## Packages ## Packages
All packages publish in lockstep at `4.0.0`.
| Package | Purpose | | Package | Purpose |
|---------|---------| |---------|---------|
| `@shade/core` | Protocol logic (X3DH, Double Ratchet, session manager, errors, events) | | `@shade/core` | Protocol logic (X3DH, Double Ratchet, sender keys, session manager, errors, events) |
| `@shade/proto` | Compact binary wire format (`0x01` PreKeyMessage, `0x02` RatchetMessage, `0x11` StreamChunk) |
| `@shade/crypto-web` | SubtleCrypto + @noble/curves provider, in-memory storage. Includes the V3.8 Web Workers entrypoint (`@shade/crypto-web/worker`) — drop-in `WorkerCryptoProvider` plus `createEncryptStream` / `createDecryptStream` TransformStream factories | | `@shade/crypto-web` | SubtleCrypto + @noble/curves provider, in-memory storage. Includes the V3.8 Web Workers entrypoint (`@shade/crypto-web/worker`) — drop-in `WorkerCryptoProvider` plus `createEncryptStream` / `createDecryptStream` TransformStream factories |
| `@shade/storage-sqlite` | Persistent SQLite storage (zero-config, bun:sqlite) | | `@shade/observability` | OpenTelemetry-shaped event bus consumed by `@shade/observer`, server hooks, and the dashboard |
| `@shade/storage-postgres` | PostgreSQL storage with Drizzle for shared databases | | `@shade/keychain` | OS keychain bindings (libsecret / Keychain / Credential Manager) used by `@shade/storage-encrypted` and the CLI |
| `@shade/server` | Prekey server (Hono routes, auth, rate limit, health, metrics) | | `@shade/key-transparency` | Key Transparency (V3.12) — RFC 6962-style append-only Merkle log, address-index commitment, signed tree heads, and a `LightWitness` for split-view detection. Opt-in on both server and client. |
| `@shade/transport` | HTTP + WebSocket transport wrappers with auto-encryption | | `@shade/storage-sqlite` | Persistent SQLite storage (zero-config, `bun:sqlite`); also ships `SqliteInboxStore` |
| `@shade/storage-postgres` | PostgreSQL storage with Drizzle for shared databases; also ships `PostgresInboxStore` + `PostgresKTLogStore` |
| `@shade/storage-encrypted` | At-rest encryption (V3.2) — `EncryptedSQLiteStorage` / `EncryptedPostgresStorage`, `KeyManager`, online re-key |
| `@shade/streams` | Multi-lane chunk encryption — HKDF-derived per-lane keys, deterministic AES-GCM nonces, streaming SHA-256, file metadata + thumbnails (V3.9) |
| `@shade/transport` | HTTP + WebSocket transport wrappers with auto-encryption; KT-verifying `fetchBundleVerified` |
| `@shade/transport-bridge` | WS → SSE → long-poll fallback chain (V3.7) — single `IncomingMessage` shape across transports for clients that can't keep a WebSocket open | | `@shade/transport-bridge` | WS → SSE → long-poll fallback chain (V3.7) — single `IncomingMessage` shape across transports for clients that can't keep a WebSocket open |
| `@shade/transport-webrtc` | V3.11 P2P chunk transport via `RTCDataChannel`. Plugs into `@shade/transfer` as an `ITransferTransport`; signaling rides Shade's own ratchet. Memory factory + native (`globalThis.RTCPeerConnection`) factory included; `MultiTransportFallback([webrtc, http])` wired automatically when `shade.configureWebRTC()` is called. | | `@shade/transport-webrtc` | V3.11 P2P chunk transport via `RTCDataChannel`. Plugs into `@shade/transfer` as an `ITransferTransport`; signaling rides Shade's own ratchet. Memory factory + native (`globalThis.RTCPeerConnection`) factory included; `MultiTransportFallback([webrtc, http])` wired automatically when `shade.configureWebRTC()` is called. |
| `@shade/proto` | Compact binary wire format (smaller than JSON) | | `@shade/server` | Prekey server (Hono routes, auth, rate limit, health, metrics). `createPrekeyServerWithKT(...)` opts into V3.12 KT mode |
| `@shade/streams` | Multi-lane chunk encryption — HKDF-derived per-lane keys, deterministic AES-GCM nonces, streaming SHA-256 |
| `@shade/transfer` | Transfer engine on top of streams: parallel lanes, resume, HTTP + WS transport with auto-fallback, integrity verification |
| `@shade/files` | Typed E2EE filesystem RPC — list/stat/mkdir/delete/move/read/write/getThumbnail + custom ops, auto inline/streams routing, production hooks (rate limit, retention, fingerprint gate, metrics), React hooks |
| `@shade/recovery` | Social key recovery (V3.10) — Shamir-split identity to `n` guardians; threshold-many `k` reconstruct on a new device. AES-GCM-authenticated reconstruction; OOB-fingerprint gate per guardian release |
| `@shade/key-transparency` | Key Transparency (V3.12) — RFC 6962-style append-only Merkle log, address-index commitment, signed tree heads, and a `LightWitness` for split-view detection. Opt-in on both server and client. See [docs/key-transparency.md](./docs/key-transparency.md) |
| `@shade/inbox-server` | Async store-and-forward relay (V3.6) — Hono routes, signed PUT/FETCH/DELETE, per-recipient TTL + quota, idempotent on `(address, msgId)`. Bundles into the same standalone container as the prekey server | | `@shade/inbox-server` | Async store-and-forward relay (V3.6) — Hono routes, signed PUT/FETCH/DELETE, per-recipient TTL + quota, idempotent on `(address, msgId)`. Bundles into the same standalone container as the prekey server |
| `@shade/inbox` | Inbox client + durable outgoing queue + receive cursor + push-trigger hook (`onMessageQueued`); composes on top of `Shade.send`/`Shade.receive` for offline-recipient delivery | | `@shade/inbox` | Inbox client + durable outgoing queue + receive cursor + push-trigger hook (`onMessageQueued`); composes on top of `Shade.send`/`Shade.receive` for offline-recipient delivery |
| `@shade/transfer` | Transfer engine on top of streams: parallel lanes, resume, HTTP + WS transport with auto-fallback, `MultiTransportFallback` (N-ary demotion), integrity verification |
| `@shade/files` | Typed E2EE filesystem RPC — list/stat/mkdir/delete/move/read/write/getThumbnail + custom ops, auto inline/streams routing, production hooks (rate limit, retention, fingerprint gate, metrics), React hooks under `@shade/files/react` |
| `@shade/recovery` | Social key recovery (V3.10) — Shamir-split identity to `n` guardians; threshold-many `k` reconstruct on a new device. AES-GCM-authenticated reconstruction; OOB-fingerprint gate per guardian release |
| `@shade/observer` | Live debugger backend (snapshot, SSE, dashboard) — see [README](./packages/shade-observer/README.md) | | `@shade/observer` | Live debugger backend (snapshot, SSE, dashboard) — see [README](./packages/shade-observer/README.md) |
| `@shade/widgets` | Embeddable React widgets including transfer uploader/downloader — see [README](./packages/shade-widgets/README.md) |
| `@shade/dashboard` | Standalone dashboard SPA bundled into the observer | | `@shade/dashboard` | Standalone dashboard SPA bundled into the observer |
| `@shade/sdk` | High-level wrapper with `createShade()` one-liner, auto-publish, auto-establish, auto-replenish, `Shade.files` namespace | | `@shade/sdk` | High-level wrapper with `createShade()` one-liner, auto-publish, auto-establish, auto-replenish, `Shade.files` namespace, fingerprint gates, KT integration, WebRTC opt-in |
| `@shade/cli` | `shade init` scaffolder + utilities (fingerprint, rotate, peer, dashboard, doctor) | | `@shade/widgets` | Embeddable React widgets — fingerprint compare/gate, recovery setup/request/approve, transfer uploader/downloader, observer panels |
| `@shade/cli` | `shade init` scaffolder + utilities (fingerprint, rotate, peer, dashboard, doctor, backup, migrate-storage, rotate-storage-key) |
## Shade as a modular toolkit ## Shade as a modular toolkit
Shade is split into packages so each project can depend on **only what it needs**—encrypted messaging, file transfer, prekey hosting, or lower-level building blocks. You do not need one giant stack for every use case. Shade is split into packages so each project can depend on **only what it needs**—encrypted messaging, file transfer, prekey hosting, social recovery, KT verification, or lower-level building blocks. You do not need one giant stack for every use case.
For a **plain-language map** (which packages to add, what the prekey server does vs your own wiring, and where to start in code), see **[docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md)**. For a **plain-language map** (which packages to add, what the prekey server does vs your own wiring, and where to start in code), see **[docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md)**.
## Publishing ## Publishing
All packages publish to a self-hosted Gitea npm registry on `gt.zyon.no`. All packages publish to a self-hosted Gitea npm registry on `gt.zyon.no`. The Docker image of the standalone container ships at `gt.zyon.no/stian/shade-prekey:<tag>`.
```bash ```bash
# Bump all packages in lockstep # Bump all packages in lockstep
bun run version 1.1.0 bun run version 4.0.1
# Dry-run (pack all tarballs without publishing) # Dry-run (pack all tarballs without publishing) — no token required
bun run publish:dry bun run publish:dry
# Real publish (requires GITEA_TOKEN env var) # Real publish — interactive (prompts for GITEA_TOKEN, checks
# registry for conflicts, publishes via scripts/publish-all.ts)
bun run publish:all bun run publish:all
# Or via CI: push a git tag v1.1.0 and .gitea/workflows/publish.yml runs # Build + push the standalone Docker image
bun run scripts/build-docker.ts -- --tag 4.0.1 --push
``` ```
The interactive `scripts/publish-shade.sh` is the human entrypoint;
`scripts/publish-all.ts` is the headless variant used by CI and
`publish:dry`. They share a single `PACKAGES` list (24 entries at
4.0.0) so the two flows can never drift.
## Soak / GA-stable
Before tagging `4.0.0` as `latest` and recommending production
upgrades, run the combined soak harness for ≥ 2 weeks:
```bash
# Full GA window (V4.0 §Soak): 14 days × 24 hours
bun run soak --hours 336
# Smoke (~3 minutes — ratchet ping-pong, integrity check)
bun run soak:smoke
```
The harness fans out N concurrent ratchet pairs, ping-pongs at
~400 ops/sec/pair, and reports cumulative counters every minute.
Any exception in any pair is captured and re-raised at shutdown so
silent failures cannot hide. Wrap it in `systemd-run --user`,
`nohup`, or a Gitea scheduled job for the actual 2-week window.
## Security properties ## Security properties
| Property | Description | | Property | Description |
@@ -242,27 +389,57 @@ bun run publish:all
| **Memory zeroization** | Key material is zeroed after use (best-effort in JS) | | **Memory zeroization** | Key material is zeroed after use (best-effort in JS) |
| **Identity verification** | Safety numbers (60 digits) for out-of-band comparison | | **Identity verification** | Safety numbers (60 digits) for out-of-band comparison |
| **Identity rotation** | 7-day grace period for old sessions during rotation | | **Identity rotation** | 7-day grace period for old sessions during rotation |
| **At-rest encryption** *(V3.2, opt-in)* | AES-256-GCM under per-(table, column) DEKs; AAD binds `(table, column, pk)`; passphrase / OS keychain / app-injected master key; online re-key |
| **Fingerprint gates** *(V3.3)* | `Shade.beforeFirstLargeFile / beforeBackupImport / beforeNewDeviceTrust` raise `FingerprintNotVerifiedError` on the operations that matter; defaults to TOFU + warning when no gate is registered |
| **Async store-and-forward** *(V3.6)* | Relay only sees `address || msgId || ciphertext`; idempotent PUT; signed FETCH/ACK; TTL-bounded |
| **Web-Worker key isolation** *(V3.8)* | Lane keys, identity keys, and ratchet chain keys live inside a dedicated worker; main thread only ferries plaintext via transferable buffers; idle terminate releases worker memory |
| **Social key recovery** *(V3.10)* | Shamir over GF(2^8); AEAD-authenticated reconstruction (forged shares fail); guardian-side fingerprint gate before share release |
| **WebRTC P2P transport** *(V3.11)* | Same Double Ratchet authenticates SDP/ICE signaling; chunk frames AEAD-bound to `streamId/laneId/seq`; deterministic glare resolution; `MultiTransportFallback` auto-demotes to HTTP |
| **Key Transparency** *(V3.12, opt-in)* | Append-only Merkle log + signed tree heads + witness gossip — split-view and history-rewrite are detected by clients | | **Key Transparency** *(V3.12, opt-in)* | Append-only Merkle log + signed tree heads + witness gossip — split-view and history-rewrite are detected by clients |
## Documentation ## Documentation
**Operator + integrator**
- [docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md) — **Modular toolkit**: pick packages by scenario (messages, files, browser, ops) - [docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md) — **Modular toolkit**: pick packages by scenario (messages, files, browser, ops)
- [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md) — Pre-flight gates for going to production - [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md) — Pre-flight gates for going to production
- [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md) — Full deployment guide (Docker, env vars, PostgreSQL, backup, Dokploy)
- [docs/ROADMAP.md](./docs/ROADMAP.md) — V3.x → 4.0 GA → V5.0 trajectory
**Per-surface deep-dives**
- [docs/files.md](./docs/files.md) — `@shade/files` API + design (filesystem RPC, custom ops, hooks, React) - [docs/files.md](./docs/files.md) — `@shade/files` API + design (filesystem RPC, custom ops, hooks, React)
- [docs/recovery.md](./docs/recovery.md) — `@shade/recovery` social key recovery (V3.10): Shamir setup, guardian-side gates, threshold tuning
- [docs/streams.md](./docs/streams.md) — `@shade/streams` + `@shade/transfer` deep dive (incl. hardening + retention) - [docs/streams.md](./docs/streams.md) — `@shade/streams` + `@shade/transfer` deep dive (incl. hardening + retention)
- [docs/inbox.md](./docs/inbox.md) — `@shade/inbox` + `@shade/inbox-server` async store-and-forward relay (V3.6) - [docs/inbox.md](./docs/inbox.md) — `@shade/inbox` + `@shade/inbox-server` async store-and-forward relay (V3.6)
- [docs/transport.md](./docs/transport.md) — `@shade/transport-bridge` SSE / long-poll / WS bridge layer (V3.7) - [docs/transport.md](./docs/transport.md) — `@shade/transport-bridge` SSE / long-poll / WS bridge layer (V3.7)
- [docs/web-workers.md](./docs/web-workers.md) — V3.8 Web Workers crypto: setup, bundler recipes (Vite/Webpack/Rollup), Safari notes, lifecycle, threat-model
- [docs/recovery.md](./docs/recovery.md) — `@shade/recovery` social key recovery (V3.10): Shamir setup, guardian-side gates, threshold tuning
- [docs/webrtc.md](./docs/webrtc.md) — `@shade/transport-webrtc` P2P transport (V3.11): NAT-traversal, TURN config, glare resolution, wire format, multi-fallback wiring - [docs/webrtc.md](./docs/webrtc.md) — `@shade/transport-webrtc` P2P transport (V3.11): NAT-traversal, TURN config, glare resolution, wire format, multi-fallback wiring
- [docs/key-transparency.md](./docs/key-transparency.md) — `@shade/key-transparency` (V3.12): operator + client onboarding, witness role, recovery procedures - [docs/key-transparency.md](./docs/key-transparency.md) — `@shade/key-transparency` (V3.12): operator + client onboarding, witness role, recovery procedures
- [docs/V3.12-DESIGN.md](./docs/V3.12-DESIGN.md) — V3.12 design notat (threat model, RFC 6962 vs CONIKS choice, freshness model, open-questions resolution) - [docs/storage-encryption.md](./docs/storage-encryption.md) — V3.2 at-rest encryption: design, key sources, rotation
- [docs/web-workers.md](./docs/web-workers.md) — V3.8 Web Workers crypto: setup, bundler recipes (Vite/Webpack/Rollup), Safari notes, lifecycle, threat-model - [docs/trust-ux.md](./docs/trust-ux.md) — V3.3 fingerprint gates: when each fires, handler patterns, widget integration
- [docs/observability.md](./docs/observability.md) — V3.4 event bus + dashboard
- [docs/cross-platform.md](./docs/cross-platform.md) — V3.5 Android parity + cross-platform vector regime
**Threat model + audit**
- [SECURITY.md](./SECURITY.md) — Reporting vulnerabilities, security policy, threat-/test-matrix - [SECURITY.md](./SECURITY.md) — Reporting vulnerabilities, security policy, threat-/test-matrix
- [THREAT-MODEL.md](./THREAT-MODEL.md) — Honest threat model and assumptions - [THREAT-MODEL.md](./THREAT-MODEL.md) — Honest threat model and assumptions (12 numbered sections + residual-risks table)
- [docs/audit/REVIEW-BUNDLE.md](./docs/audit/REVIEW-BUNDLE.md) — External crypto-review entrypoint (scope, build instructions, reporting)
- [docs/audit/SCOPE.md](./docs/audit/SCOPE.md) — One-page audit-scope summary
**Migration + history**
- [MIGRATION.md](./MIGRATION.md) — How to replace existing crypto with Shade + the [0.3.x → 4.0 upgrade path](./MIGRATION.md#migrating-from-03x-to-40-ga)
- [CHANGELOG.md](./CHANGELOG.md) — `4.0.0` GA section + every prior release
- [docs/archive/](./docs/archive/) — V2.1 / V2.2 / V2.3 backlog and V3.1 → V3.12 implementation plans (all `Status: Done`)
- [docs/V5.0.md](./docs/V5.0.md) — Voice / Video / Broadcast (post-GA, built on the frozen 4.0 stack)
**Examples**
- [examples/](./examples/) — Runnable example applications, including - [examples/](./examples/) — Runnable example applications, including
[`07-streams-upload`](./examples/07-streams-upload) (multi-lane file transfer) [`07-streams-upload`](./examples/07-streams-upload) (multi-lane file transfer)
and [`08-files-browser`](./examples/08-files-browser) (filesystem RPC) and [`08-files-browser`](./examples/08-files-browser) (filesystem RPC)
- [MIGRATION.md](./MIGRATION.md) — How to replace existing crypto with Shade
## Deployment — one container per project ## Deployment — one container per project
@@ -274,15 +451,30 @@ docker run -d \
-v my-project-shade:/data \ -v my-project-shade:/data \
-p 3900:3900 \ -p 3900:3900 \
-e SHADE_OBSERVER_TOKEN=change-me-to-at-least-16-chars \ -e SHADE_OBSERVER_TOKEN=change-me-to-at-least-16-chars \
gt.zyon.no/stian/shade-prekey:latest gt.zyon.no/stian/shade-prekey:4.0.0
``` ```
The container includes: The container includes:
- **Prekey server** — `/v1/keys/*` REST API - **Prekey server** — `/v1/keys/*` REST API
- **Observer dashboard** — `/shade-observer/dashboard/` (off unless token is set) - **Inbox relay (V3.6)** — `/v1/inbox/*` async store-and-forward; enable
with `SHADE_INBOX_DB_PATH=/data/inbox.db` (or `SHADE_INBOX_PG_URL`).
`SHADE_INBOX_PRUNE_INTERVAL_MINUTES` controls TTL prune cadence.
- **Bridge transports (V3.7)** — `/v1/bridge/{stream,poll,ws}` SSE +
long-poll + WS adapters for clients that can't keep a WebSocket open.
- **Transfer routes** — `/v1/transfer/*` chunk + state + control routes
for `@shade/transfer`.
- **Key Transparency (V3.12)** — `/v1/kt/*` exposes `log_id`, latest +
historical STH, and consistency proofs. Enable with
`SHADE_KT_*` env vars; off by default.
- **Observer dashboard** — `/shade-observer/dashboard/` (off unless
`SHADE_OBSERVER_TOKEN` is set)
- **OpenAPI spec** — `/openapi.yaml` and interactive `/docs` viewer - **OpenAPI spec** — `/openapi.yaml` and interactive `/docs` viewer
(covers all 27 routes — prekey, inbox, bridge, transfer, KT,
observer, `/metrics`, `/healthz`, `/ready`)
- **Prometheus metrics** — `/metrics` - **Prometheus metrics** — `/metrics`
- **Health check** — `/health` - **Health probes** — `/health` (full), `/healthz` (liveness),
`/ready` (readiness)
- **Stale cleanup** — purges inactive identities automatically - **Stale cleanup** — purges inactive identities automatically
See [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md) for the full deployment guide, environment variables, PostgreSQL config, backup strategy, and Dokploy instructions. See [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md) for the full deployment guide, environment variables, PostgreSQL config, backup strategy, and Dokploy instructions.

View File

@@ -100,6 +100,9 @@ matrix row in the same change.
| § 1 Network attacker — replay window | ±5 min `signedAt` enforcement | `packages/shade-server/tests/server.test.ts` (`"rejects registration with stale signedAt"`) | | § 1 Network attacker — replay window | ±5 min `signedAt` enforcement | `packages/shade-server/tests/server.test.ts` (`"rejects registration with stale signedAt"`) |
| § 1 Network attacker — header AAD | Ratchet headers bound to ciphertext | `packages/shade-core/tests/ratchet.test.ts`, `packages/shade-streams/tests/tamper.test.ts`, `packages/shade-streams/tests/aead.test.ts` | | § 1 Network attacker — header AAD | Ratchet headers bound to ciphertext | `packages/shade-core/tests/ratchet.test.ts`, `packages/shade-streams/tests/tamper.test.ts`, `packages/shade-streams/tests/aead.test.ts` |
| § 1 Network attacker — forward secrecy | DH ratchet step + chain-key zeroize | `packages/shade-core/tests/ratchet.test.ts`, `packages/shade-crypto-web/tests/hardening.test.ts` | | § 1 Network attacker — forward secrecy | DH ratchet step + chain-key zeroize | `packages/shade-core/tests/ratchet.test.ts`, `packages/shade-crypto-web/tests/hardening.test.ts` |
| § 1 Network attacker — streaming sub-session FS/replay (V4.11) | Per-frame Double-Ratchet `seal`/`open`; counter-rewind & replay rejected; in-memory-only (never persisted) | `packages/shade-core/tests/stream.test.ts` (`"R1: replayed / rewound frame is rejected"`, `"R2/R3: long one-directional burst stays correct and memory-bounded"`) |
| § 1 Network attacker — streaming handshake auth (V4.11) | Identity-bound 3-DH against parent-session-pinned identities | `packages/shade-core/tests/stream.test.ts` (`"handshake is mutually authenticated against pinned identities"`) |
| § 3 Endpoint compromise — streaming sub-session isolation (V4.11) | Stream ratchet derived without touching the stored parent session; zeroized on close | `packages/shade-core/tests/stream.test.ts` (`"R5: opening/using/closing a stream never touches the parent session"`, `"close() zeroizes and blocks further use; idempotent"`) |
| § 2 Compromised prekey server — public-only storage | Prekey store never accepts a private key | `packages/shade-server/tests/server.test.ts`, `packages/shade-storage-sqlite/tests/sqlite-prekey-store.test.ts` | | § 2 Compromised prekey server — public-only storage | Prekey store never accepts a private key | `packages/shade-server/tests/server.test.ts`, `packages/shade-storage-sqlite/tests/sqlite-prekey-store.test.ts` |
| § 2 Compromised prekey server — signed replenish/delete | Per-identity Ed25519 signature | `packages/shade-server/tests/server.test.ts` | | § 2 Compromised prekey server — signed replenish/delete | Per-identity Ed25519 signature | `packages/shade-server/tests/server.test.ts` |
| § 2 Compromised prekey server — fake-bundle detection | Out-of-band fingerprint comparison | `packages/shade-core/tests/fingerprint-session.test.ts` | | § 2 Compromised prekey server — fake-bundle detection | Out-of-band fingerprint comparison | `packages/shade-core/tests/fingerprint-session.test.ts` |

View File

@@ -1,3 +1,5 @@
plugins { plugins {
kotlin("jvm") version "2.0.20" apply false kotlin("jvm") version "2.0.20" apply false
kotlin("android") version "2.0.20" apply false
id("com.android.library") version "8.7.3" apply false
} }

View File

@@ -2,3 +2,5 @@ org.gradle.jvmargs=-Xmx2g -Dfile.encoding=UTF-8
org.gradle.parallel=true org.gradle.parallel=true
org.gradle.caching=true org.gradle.caching=true
kotlin.code.style=official kotlin.code.style=official
android.useAndroidX=true
android.nonTransitiveRClass=true

View File

@@ -4,6 +4,7 @@ pluginManagement {
repositories { repositories {
gradlePluginPortal() gradlePluginPortal()
mavenCentral() mavenCentral()
google()
} }
} }
@@ -17,3 +18,6 @@ dependencyResolutionManagement {
include(":shade-android") include(":shade-android")
project(":shade-android").projectDir = file("shade-android") project(":shade-android").projectDir = file("shade-android")
include(":shade-android-keystore")
project(":shade-android-keystore").projectDir = file("shade-android-keystore")

View File

@@ -0,0 +1,68 @@
# shade-android-keystore
Android-specific bindings for `shade-android`. Lives as a sibling Gradle module so the JVM-only protocol code can keep running in CI without an Android SDK install.
Provides:
- **`KeystoreMasterKey`** — hardware-backed AES-256-GCM master key in the Android Keystore. Optionally biometric-gated (BIOMETRIC_STRONG only — Class 3 assurance), StrongBox-backed when available, invalidated on new biometric enrollment.
- **`BiometricUnlock`** — coroutine wrapper around `BiometricPrompt` for unlocking a `Cipher` instance bound to the keystore key. Throws `BiometricCancelledException` / `BiometricFailedException` so callers can handle the auth flow without writing custom callbacks.
- **`KeystoreStorage`** — `StorageProvider` implementation that persists session/identity/prekey state to `SharedPreferences`, each row encrypted under the keystore key with the row's preference key bound as AAD.
## Usage
```kotlin
import androidx.fragment.app.FragmentActivity
import no.zyon.shade.ShadeSessionManager
import no.zyon.shade.crypto.TinkProvider
import no.zyon.shade.keystore.BiometricUnlock
import no.zyon.shade.keystore.KeystoreStorage
class MyActivity : FragmentActivity() {
private val crypto = TinkProvider()
private lateinit var storage: KeystoreStorage
private lateinit var manager: ShadeSessionManager
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
storage = KeystoreStorage(this, crypto)
lifecycleScope.launch {
val unlock = BiometricUnlock(
activity = this@MyActivity,
title = "Unlock Shade",
subtitle = "Tap your fingerprint to access your messages",
)
try {
storage.unlock(unlock)
} catch (e: BiometricCancelledException) {
// user backed out — show a "tap to retry" UI
return@launch
}
manager = ShadeSessionManager(crypto, storage)
manager.initialize()
// ... use manager normally
}
}
}
```
For credential-driven bootstrap (V4.9 profile + V4.10 approval), pair this with `no.zyon.shade.blob.createProfileNamespace` and `no.zyon.shade.approval.signProxyApproval` — both pure-JVM (in `:shade-android`).
## Threat model
- **Compromised app process**: cannot read the AES key (it's in the secure environment). Can attempt to use the cipher only after the user has authenticated; biometric re-prompts are required after each biometric event.
- **Stolen device with known PIN**: cannot unlock — `setUserAuthenticationParameters(0, BIOMETRIC_STRONG)` excludes `DEVICE_CREDENTIAL`.
- **Attacker enrolls own biometric**: `setInvalidatedByBiometricEnrollment(true)` invalidates the key on enrollment, forcing a credential rebootstrap (which would need username + password + PIN).
- **Catastrophic recovery**: `forgetEverything()` deletes the master key and clears the SharedPreferences. Pair with `Profile.delete()` for full account erasure.
## Build
Requires an Android SDK. The Gradle build uses Android Gradle Plugin 8.7+, AGP minSdk 28 (Pie+ for BiometricPrompt baseline), targetSdk 35.
```bash
JAVA_HOME=/path/to/jdk-21 ./gradlew :shade-android-keystore:assembleDebug
```
Unit tests: none yet — `KeystoreStorage` requires Android runtime. Robolectric or instrumented tests against an emulator are tracked as a follow-up. The pure-JVM `SessionStateJson` round-trip serializer is tested in `:shade-android` (`SessionStateJsonTest`).

View File

@@ -0,0 +1,59 @@
plugins {
id("com.android.library")
kotlin("android")
}
// V4.10 — Android-specific KeystoreStorage adapter.
//
// Lives as a sibling module to `:shade-android` so the JVM-only
// protocol code can keep running in CI without an Android SDK.
// This module pulls in `:shade-android` for `StorageProvider`,
// `IdentityKeyPair`, etc., and binds those types to a hardware-
// backed Android Keystore master key with biometric gating.
android {
namespace = "no.zyon.shade.keystore"
compileSdk = 35
defaultConfig {
minSdk = 28 // BiometricPrompt + StrongBox baseline
}
compileOptions {
sourceCompatibility = JavaVersion.VERSION_17
targetCompatibility = JavaVersion.VERSION_17
}
kotlinOptions {
jvmTarget = "17"
}
buildTypes {
release {
isMinifyEnabled = false
}
}
testOptions {
unitTests.isReturnDefaultValues = true
}
}
dependencies {
// Sibling: protocol types + StorageProvider interface.
api(project(":shade-android"))
// androidx.biometric — fragment-safe BiometricPrompt wrapper.
// 1.2.0-alpha05 is the latest with stable BiometricPrompt API.
implementation("androidx.biometric:biometric:1.2.0-alpha05")
// androidx.fragment — BiometricPrompt requires FragmentActivity.
implementation("androidx.fragment:fragment-ktx:1.8.5")
// Coroutines for the suspend-function StorageProvider implementation.
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.9.0")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.9.0")
testImplementation("junit:junit:4.13.2")
testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.9.0")
}

View File

@@ -0,0 +1,2 @@
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android" />

View File

@@ -0,0 +1,112 @@
package no.zyon.shade.keystore
import androidx.biometric.BiometricManager
import androidx.biometric.BiometricPrompt
import androidx.fragment.app.FragmentActivity
import javax.crypto.Cipher
import kotlin.coroutines.resume
import kotlin.coroutines.resumeWithException
import kotlinx.coroutines.suspendCancellableCoroutine
/**
* Biometric unlock for a `KeystoreMasterKey`-bound `Cipher`.
*
* The Android keystore enforces that any operation on a
* user-authentication-required key must happen via a
* `BiometricPrompt.CryptoObject`-wrapped `Cipher`. The user sees a
* system biometric prompt; on success the same `Cipher` instance is
* usable for one operation (or one streaming session) before
* needing to re-prompt.
*
* This is a thin coroutine wrapper around `BiometricPrompt` that
* resolves to the authenticated cipher or throws on user
* cancellation. Callers typically run it once at app start to
* unlock the master key for the lifetime of the foreground session.
*/
class BiometricUnlock(
private val activity: FragmentActivity,
private val title: String,
private val subtitle: String? = null,
private val negativeButton: String = "Cancel",
) {
/**
* True if BIOMETRIC_STRONG is currently usable on this device.
* False means the user has no enrolled fingerprint/face that
* meets the class-3 assurance level — fall back to a credential
* recovery flow rather than crashing.
*/
fun canAuthenticate(): Boolean {
val mgr = BiometricManager.from(activity)
return mgr.canAuthenticate(BiometricManager.Authenticators.BIOMETRIC_STRONG) ==
BiometricManager.BIOMETRIC_SUCCESS
}
/**
* Show the biometric prompt and return the authenticated cipher.
*
* Cancellation paths:
* - User taps the negative button → throws `BiometricCancelledException`.
* - System errors out (e.g. too many failures) → throws
* `BiometricFailedException` with the system error code.
*/
suspend fun unlock(cipher: Cipher): Cipher = suspendCancellableCoroutine { cont ->
val callback = object : BiometricPrompt.AuthenticationCallback() {
override fun onAuthenticationSucceeded(result: BiometricPrompt.AuthenticationResult) {
val authedCipher = result.cryptoObject?.cipher
if (authedCipher == null) {
cont.resumeWithException(
BiometricFailedException(-1, "BiometricPrompt returned no cipher"),
)
} else {
cont.resume(authedCipher)
}
}
override fun onAuthenticationError(errorCode: Int, errString: CharSequence) {
if (errorCode == BiometricPrompt.ERROR_USER_CANCELED ||
errorCode == BiometricPrompt.ERROR_NEGATIVE_BUTTON ||
errorCode == BiometricPrompt.ERROR_CANCELED
) {
cont.resumeWithException(BiometricCancelledException(errString.toString()))
} else {
cont.resumeWithException(
BiometricFailedException(errorCode, errString.toString()),
)
}
}
override fun onAuthenticationFailed() {
// A single failed attempt — the prompt stays open and
// gives the user another try. Don't resume the
// continuation; let the system flow continue.
}
}
val prompt = BiometricPrompt(
activity,
activity.mainExecutor,
callback,
)
val info = BiometricPrompt.PromptInfo.Builder()
.setTitle(title)
.apply { if (subtitle != null) setSubtitle(subtitle) }
.setNegativeButtonText(negativeButton)
.setAllowedAuthenticators(BiometricManager.Authenticators.BIOMETRIC_STRONG)
.build()
cont.invokeOnCancellation { prompt.cancelAuthentication() }
prompt.authenticate(info, BiometricPrompt.CryptoObject(cipher))
}
}
/** User cancelled the biometric prompt. */
class BiometricCancelledException(message: String) : RuntimeException(message)
/**
* BiometricPrompt returned a non-cancellation error (lockout, hardware
* unavailable, no enrolled biometrics, etc.). Inspect `errorCode`
* against `BiometricPrompt.ERROR_*` constants to decide UX response.
*/
class BiometricFailedException(val errorCode: Int, message: String) :
RuntimeException("[$errorCode] $message")

View File

@@ -0,0 +1,172 @@
package no.zyon.shade.keystore
import android.os.Build
import android.security.keystore.KeyGenParameterSpec
import android.security.keystore.KeyProperties
import javax.crypto.Cipher
import javax.crypto.KeyGenerator
import javax.crypto.SecretKey
import javax.crypto.spec.GCMParameterSpec
import java.security.KeyStore
/**
* Hardware-backed AES-256-GCM master key in the Android Keystore.
*
* The key never leaves the secure environment — Android's keystore
* implementation enforces that all encrypt/decrypt operations
* happen inside the TEE (or StrongBox if present), and the raw
* key bytes are never returned to userspace.
*
* The key is created on first use with these properties:
*
* - AES-256-GCM, no padding
* - User authentication required: opt-in via the `requireBiometric`
* flag. When true, every encrypt/decrypt operation must be wrapped
* in a `BiometricPrompt.authenticate(CryptoObject(cipher))` call
* that succeeds within the same `Cipher` instance.
* - StrongBox-backed if available (Pixel 3+, most Samsung flagships).
* Falls back to TEE on devices without StrongBox.
* - InvalidatedByBiometricEnrollment(true): a newly enrolled
* fingerprint/face invalidates the key, forcing the user to
* re-bootstrap from credentials. Defends against a thief who
* enrolls their own biometric.
*
* Mirrors the role `KeyManager` plays in `@shade/storage-encrypted`'s
* V4.5 KDF chain: this is the *encryption-at-rest* master key, not
* the X3DH identity key. The Shade protocol's identity keys are
* stored encrypted under THIS key.
*/
class KeystoreMasterKey(
private val alias: String,
private val requireBiometric: Boolean = true,
) {
init {
require(alias.isNotEmpty()) { "alias must be non-empty" }
}
/**
* Build a `Cipher` initialized for encryption with the master key.
*
* If the key requires user auth, the returned cipher is *not yet
* usable* — the caller MUST wrap it in a
* `BiometricPrompt.authenticate(CryptoObject(cipher))` and use
* the cipher exposed by the auth-success callback. Calling
* `cipher.doFinal(...)` before authentication throws
* `UserNotAuthenticatedException`.
*/
fun cipherForEncrypt(): Cipher {
val key = getOrCreateKey()
val cipher = Cipher.getInstance(TRANSFORMATION)
cipher.init(Cipher.ENCRYPT_MODE, key)
return cipher
}
/**
* Build a `Cipher` initialized for decryption with the master key
* and a previously-stored 12-byte nonce. Same authentication
* requirement as `cipherForEncrypt`.
*/
fun cipherForDecrypt(nonce: ByteArray): Cipher {
require(nonce.size == 12) { "GCM nonce must be 12 bytes" }
val key = getOrCreateKey()
val cipher = Cipher.getInstance(TRANSFORMATION)
cipher.init(Cipher.DECRYPT_MODE, key, GCMParameterSpec(128, nonce))
return cipher
}
/** True if the key already exists in the Android Keystore. */
fun exists(): Boolean {
val ks = KeyStore.getInstance(ANDROID_KEYSTORE).apply { load(null) }
return ks.containsAlias(alias)
}
/**
* Delete the master key. Catastrophic — all data encrypted under
* it becomes unrecoverable. Used by the "forget everything" flow
* (paired with `Profile.delete()` in the V4.9 namespace).
*/
fun deleteKey() {
val ks = KeyStore.getInstance(ANDROID_KEYSTORE).apply { load(null) }
if (ks.containsAlias(alias)) ks.deleteEntry(alias)
}
private fun getOrCreateKey(): SecretKey {
val ks = KeyStore.getInstance(ANDROID_KEYSTORE).apply { load(null) }
ks.getEntry(alias, null)?.let { entry ->
return (entry as KeyStore.SecretKeyEntry).secretKey
}
return generateKey()
}
private fun generateKey(): SecretKey {
val builder = KeyGenParameterSpec.Builder(
alias,
KeyProperties.PURPOSE_ENCRYPT or KeyProperties.PURPOSE_DECRYPT,
)
.setBlockModes(KeyProperties.BLOCK_MODE_GCM)
.setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_NONE)
.setKeySize(256)
// Each encrypt operation generates a fresh IV in the secure
// env; we read it back via `cipher.iv` after init.
.setRandomizedEncryptionRequired(true)
if (requireBiometric) {
builder.setUserAuthenticationRequired(true)
// BIOMETRIC_STRONG only — class 3, the highest assurance
// level (Class 3 = false-accept rate < 1/50 000 per BiometricPrompt).
// DEVICE_CREDENTIAL is intentionally NOT included: a stolen
// device with a known PIN should not unlock Shade.
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.R) {
builder.setUserAuthenticationParameters(
/* timeout = */ 0,
KeyProperties.AUTH_BIOMETRIC_STRONG,
)
} else {
@Suppress("DEPRECATION")
builder.setUserAuthenticationValidityDurationSeconds(-1)
}
builder.setInvalidatedByBiometricEnrollment(true)
}
// StrongBox if available — bumps key storage to a dedicated
// tamper-resistant chip on Pixel 3+ / most Samsung flagships.
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) {
builder.setIsStrongBoxBacked(true)
}
val gen = KeyGenerator.getInstance(KeyProperties.KEY_ALGORITHM_AES, ANDROID_KEYSTORE)
return try {
gen.init(builder.build())
gen.generateKey()
} catch (_: Exception) {
// StrongBox not present or full → retry without StrongBox.
// Same for older devices that don't honor
// setUserAuthenticationParameters.
val fallback = KeyGenParameterSpec.Builder(
alias,
KeyProperties.PURPOSE_ENCRYPT or KeyProperties.PURPOSE_DECRYPT,
)
.setBlockModes(KeyProperties.BLOCK_MODE_GCM)
.setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_NONE)
.setKeySize(256)
.setRandomizedEncryptionRequired(true)
.apply {
if (requireBiometric) {
setUserAuthenticationRequired(true)
@Suppress("DEPRECATION")
setUserAuthenticationValidityDurationSeconds(-1)
setInvalidatedByBiometricEnrollment(true)
}
}
.build()
gen.init(fallback)
gen.generateKey()
}
}
companion object {
private const val ANDROID_KEYSTORE = "AndroidKeyStore"
private const val TRANSFORMATION = "AES/GCM/NoPadding"
}
}

View File

@@ -0,0 +1,229 @@
package no.zyon.shade.keystore
import android.content.Context
import android.content.SharedPreferences
import android.util.Base64
import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.withLock
import no.zyon.shade.crypto.CryptoProvider
import no.zyon.shade.serialization.SessionStateJson
import no.zyon.shade.storage.StorageProvider
import no.zyon.shade.types.IdentityKeyPair
import no.zyon.shade.types.OneTimePreKey
import no.zyon.shade.types.SessionState
import no.zyon.shade.types.SignedPreKey
/**
* `StorageProvider` implementation that gates all reads/writes through
* a biometric-locked `KeystoreMasterKey`. Mirrors `MemoryStorage` for
* the API surface but persists state to `SharedPreferences` with each
* row encrypted under the keystore key.
*
* Lifecycle:
*
* 1. App start → construct `KeystoreStorage(context, alias)`.
* 2. `unlock(BiometricUnlock)` runs the system biometric prompt.
* The Android keystore caches the auth state under the key's
* `setUserAuthenticationParameters(0, BIOMETRIC_STRONG)` policy
* until the next biometric event (re-enrollment, etc.).
* 3. While unlocked, `getSession`/`saveSession` etc. work normally.
* 4. `lock()` clears the in-memory unlocked flag so a future
* operation triggers another biometric prompt.
*
* Wire layout per row:
* `<base64(nonce(12))>:<base64(ct||tag)>`
*
* Stored as `String` SharedPreferences entries. AAD = the row's
* preference key (`session:<address>`, `signedPreKey:<id>`, etc.) so
* a substituted-prefs swap fails to open.
*/
class KeystoreStorage(
context: Context,
private val crypto: CryptoProvider,
keyAlias: String = DEFAULT_KEY_ALIAS,
prefsName: String = DEFAULT_PREFS_NAME,
requireBiometric: Boolean = true,
) : StorageProvider {
private val prefs: SharedPreferences =
context.applicationContext.getSharedPreferences(prefsName, Context.MODE_PRIVATE)
private val masterKey = KeystoreMasterKey(keyAlias, requireBiometric = requireBiometric)
private val writeMutex = Mutex()
@Volatile
private var unlocked: Boolean = false
/**
* Unlock the keystore via biometric prompt. Idempotent — calling
* twice without a `lock()` between is a no-op.
*/
suspend fun unlock(unlock: BiometricUnlock) {
if (unlocked) return
// The biometric flow returns an authenticated *encrypt*
// cipher; we discard it after a one-shot probe to confirm
// the master key is reachable. The actual encrypt/decrypt
// ciphers in the I/O path use the authentication state
// established here (Android Keystore caches the auth for
// the user-authentication-required key under the
// `setUserAuthenticationParameters(0, BIOMETRIC_STRONG)`
// policy until the next biometric event).
val probe = masterKey.cipherForEncrypt()
unlock.unlock(probe)
unlocked = true
}
/** Unlock without biometric — only valid for keys constructed with `requireBiometric=false`. */
fun unlockNoBiometric() {
unlocked = true
}
/** Wipe in-memory unlock state. The key itself stays in the keystore. */
fun lock() {
unlocked = false
}
/**
* Catastrophic reset: deletes the master key + all encrypted
* preferences. Used by the "forget everything" / 3-strikes-wipe
* path. The next bootstrap rebuilds from credentials.
*/
fun forgetEverything() {
masterKey.deleteKey()
prefs.edit().clear().apply()
unlocked = false
}
// ─── Identity ──────────────────────────────────────────────
override suspend fun getIdentityKeyPair(): IdentityKeyPair? {
val json = readDecrypted(KEY_IDENTITY) ?: return null
return SessionStateJson.deserializeIdentityKeyPair(json)
}
override suspend fun saveIdentityKeyPair(keyPair: IdentityKeyPair) {
writeEncrypted(KEY_IDENTITY, SessionStateJson.serializeIdentityKeyPair(keyPair))
}
override suspend fun getLocalRegistrationId(): Int {
return readDecrypted(KEY_REGISTRATION_ID)?.toIntOrNull() ?: 0
}
override suspend fun saveLocalRegistrationId(id: Int) {
writeEncrypted(KEY_REGISTRATION_ID, id.toString())
}
// ─── Signed prekeys ────────────────────────────────────────
override suspend fun getSignedPreKey(keyId: Int): SignedPreKey? {
val json = readDecrypted("$KEY_SIGNED_PREKEY:$keyId") ?: return null
return SessionStateJson.deserializeSignedPreKey(json)
}
override suspend fun saveSignedPreKey(key: SignedPreKey) {
writeEncrypted(
"$KEY_SIGNED_PREKEY:${key.keyId}",
SessionStateJson.serializeSignedPreKey(key),
)
}
override suspend fun removeSignedPreKey(keyId: Int) {
writeMutex.withLock { prefs.edit().remove("$KEY_SIGNED_PREKEY:$keyId").apply() }
}
// ─── One-time prekeys ──────────────────────────────────────
override suspend fun getOneTimePreKey(keyId: Int): OneTimePreKey? {
val json = readDecrypted("$KEY_ONETIME_PREKEY:$keyId") ?: return null
return SessionStateJson.deserializeOneTimePreKey(json)
}
override suspend fun saveOneTimePreKey(key: OneTimePreKey) {
writeEncrypted(
"$KEY_ONETIME_PREKEY:${key.keyId}",
SessionStateJson.serializeOneTimePreKey(key),
)
}
override suspend fun removeOneTimePreKey(keyId: Int) {
writeMutex.withLock { prefs.edit().remove("$KEY_ONETIME_PREKEY:$keyId").apply() }
}
override suspend fun getOneTimePreKeyCount(): Int {
return prefs.all.keys.count { it.startsWith("$KEY_ONETIME_PREKEY:") }
}
// ─── Sessions ──────────────────────────────────────────────
override suspend fun getSession(address: String): SessionState? {
val json = readDecrypted("$KEY_SESSION:$address") ?: return null
return SessionStateJson.deserialize(json)
}
override suspend fun saveSession(address: String, state: SessionState) {
writeEncrypted("$KEY_SESSION:$address", SessionStateJson.serialize(state))
}
override suspend fun removeSession(address: String) {
writeMutex.withLock { prefs.edit().remove("$KEY_SESSION:$address").apply() }
}
// ─── Trust ─────────────────────────────────────────────────
override suspend fun isTrustedIdentity(address: String, identityKey: ByteArray): Boolean {
val stored = readDecrypted("$KEY_TRUSTED:$address") ?: return true // TOFU
val storedBytes = Base64.decode(stored, Base64.NO_WRAP)
return crypto.constantTimeEqual(storedBytes, identityKey)
}
override suspend fun saveTrustedIdentity(address: String, identityKey: ByteArray) {
writeEncrypted(
"$KEY_TRUSTED:$address",
Base64.encodeToString(identityKey, Base64.NO_WRAP),
)
}
// ─── Encrypted-row plumbing ────────────────────────────────
private fun ensureUnlocked() {
check(unlocked) {
"KeystoreStorage is locked — call unlock(BiometricUnlock) first"
}
}
private suspend fun readDecrypted(prefKey: String): String? {
ensureUnlocked()
val raw = prefs.getString(prefKey, null) ?: return null
val parts = raw.split(":", limit = 2)
require(parts.size == 2) { "malformed encrypted row at $prefKey" }
val nonce = Base64.decode(parts[0], Base64.NO_WRAP)
val ct = Base64.decode(parts[1], Base64.NO_WRAP)
val cipher = masterKey.cipherForDecrypt(nonce)
cipher.updateAAD(prefKey.toByteArray(Charsets.UTF_8))
return cipher.doFinal(ct).toString(Charsets.UTF_8)
}
private suspend fun writeEncrypted(prefKey: String, plaintext: String) {
ensureUnlocked()
writeMutex.withLock {
val cipher = masterKey.cipherForEncrypt()
cipher.updateAAD(prefKey.toByteArray(Charsets.UTF_8))
val ct = cipher.doFinal(plaintext.toByteArray(Charsets.UTF_8))
val nonce = cipher.iv
val nonceB64 = Base64.encodeToString(nonce, Base64.NO_WRAP)
val ctB64 = Base64.encodeToString(ct, Base64.NO_WRAP)
prefs.edit().putString(prefKey, "$nonceB64:$ctB64").apply()
}
}
companion object {
const val DEFAULT_KEY_ALIAS = "shade-master-v1"
const val DEFAULT_PREFS_NAME = "shade-keystore-storage-v1"
private const val KEY_IDENTITY = "identity"
private const val KEY_REGISTRATION_ID = "registrationId"
private const val KEY_SIGNED_PREKEY = "signedPreKey"
private const val KEY_ONETIME_PREKEY = "oneTimePreKey"
private const val KEY_SESSION = "session"
private const val KEY_TRUSTED = "trusted"
}
}

View File

@@ -7,7 +7,11 @@ Kotlin implementation of the Shade E2EE protocol for Android apps. Byte-for-byte
**M-Cross 1 ✅** — keys + HKDF + X3DH + fingerprint. **M-Cross 1 ✅** — keys + HKDF + X3DH + fingerprint.
**M-Cross 2 ✅** — full ratchet step (encrypt + decrypt roundtrip) + wire 0x02 (RatchetMessage and PreKeyMessage with/without OTPK). **M-Cross 2 ✅** — full ratchet step (encrypt + decrypt roundtrip) + wire 0x02 (RatchetMessage and PreKeyMessage with/without OTPK).
**M-Cross 3 ✅** — streams 0x11 (KDF labels with embedded NULs, deterministic chunk nonce/AAD, wire 0x11 encode/decode). **M-Cross 3 ✅** — streams 0x11 (KDF labels with embedded NULs, deterministic chunk nonce/AAD, wire 0x11 encode/decode).
**M-Cross 4 ✅** — backup-key HKDF + AES-GCM, group sender-key step (`kdfChainKey` + Ed25519 sign over `aad ‖ ct`), storage HKDF (storageKey/fieldKey/rowNonce). Pending: scrypt master-key, argon2id swap, Android KeystoreStorage (sibling module). **M-Cross 4 ✅** — backup-key HKDF + AES-GCM, group sender-key step (`kdfChainKey` + Ed25519 sign over `aad ‖ ct`), storage HKDF (storageKey/fieldKey/rowNonce).
**M-Cross 5 ✅** — V4.9 blob KDF + AEAD (`deriveBlobSlotId / deriveBlobKey / deriveBlobSigningSeed`, AAD-bound seal/open), `BlobClient` HTTP, `Profile` namespace. Cross-platform vectors in `blob.json`.
**M-Cross 6 ✅** — V4.10 cross-host approval routing: canonical profile-blob schema (`hosts[]` / `clients[]` / `trustedApproverFingerprints[]`), build/sign/verify proxy approvals via `canonicalApprovalSigningBytes` (length-prefixed u16 BE UTF-8). Cross-platform vectors in `approval.json`, including a TS-signed Ed25519 signature that the Kotlin port verifies.
**M-Cross 7 ✅** — scrypt + argon2id password-KDF wrappers (Bouncy Castle), NFKC-normalized inputs.
**M-Cross 8 ✅**`:shade-android-keystore` sibling module: `KeystoreMasterKey` (StrongBox-backed AES-256-GCM, BIOMETRIC_STRONG-gated, invalidated on biometric enrollment), `BiometricUnlock`, `KeystoreStorage` (`StorageProvider` over biometric-gated AES-encrypted SharedPreferences).
Cross-platform test vectors in `/test-vectors/` are loaded by both the TS Cross-platform test vectors in `/test-vectors/` are loaded by both the TS
and Kotlin test suites; any byte-divergence fails CI within 60 s. See and Kotlin test suites; any byte-divergence fails CI within 60 s. See

View File

@@ -29,6 +29,11 @@ dependencies {
// The same `subtle.*` API as `tink-android` so the source compiles unchanged. // The same `subtle.*` API as `tink-android` so the source compiles unchanged.
implementation("com.google.crypto.tink:tink:1.15.0") implementation("com.google.crypto.tink:tink:1.15.0")
// Bouncy Castle for scrypt + argon2id. Tink doesn't ship password
// KDFs; @shade/storage-encrypted uses @noble/hashes for both. We
// pin to the JDK18-on artifact so it works on JVM 17 + Android.
implementation("org.bouncycastle:bcprov-jdk18on:1.78.1")
// JSON serialization (session state + test-vector loader on JVM). // JSON serialization (session state + test-vector loader on JVM).
implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.7.3") implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.7.3")

View File

@@ -0,0 +1,273 @@
package no.zyon.shade.approval
import no.zyon.shade.crypto.CryptoProvider
import java.nio.ByteBuffer
import java.nio.ByteOrder
/**
* V4.10 — cross-host approval routing helpers. Mirror
* `@shade/sdk/approval.ts` byte-for-byte.
*
* The frames themselves (`approvalNeeded` / `linkApproveByProxy`) are
* app-defined payloads sent over the existing Shade bilateral E2EE
* channel. This file ships the canonical signing-payload layout, the
* Ed25519 sign step a phone runs after biometric unlock, and the
* verify step a host runs against the freshest profile blob.
*
* The signing payload is length-prefixed binary (u16 BE) so any
* platform — Kotlin, Swift, Go — can produce byte-identical input
* without needing a JSON canonicalizer. Cross-platform parity is
* gated by `test-vectors/blob-storage.json` (signing payload
* fixtures) plus a Kotlin↔TS round-trip in `CrossPlatformVectorTest`.
*/
/** Default domain separator. Apps with their own canonical name (e.g. Prism) override. */
const val DEFAULT_APPROVAL_DOMAIN = "shade-link-approve-v1"
/** Default expiry: 5 minutes after the host issues the request. */
const val DEFAULT_APPROVAL_EXPIRES_IN_MS = 5L * 60 * 1000
/** Information about the device the host received a `linkRequest` from. */
data class ApprovalRequestingDevice(
val fingerprint: String,
val deviceName: String? = null,
val userAgent: String? = null,
val ipHint: String? = null,
val receivedAt: Long,
)
data class ApprovalRequestFrame(
val kind: String = "approvalNeeded",
/** 128-bit hex (32 chars) random idempotency key. */
val requestId: String,
val hostAddress: String,
val hostFingerprint: String,
val requestingDevice: ApprovalRequestingDevice,
val expiresAt: Long,
val domain: String,
)
data class ProxyApprovalFrame(
val kind: String = "linkApproveByProxy",
val requestId: String,
val decision: String,
val approverFingerprint: String,
/** 64-byte Ed25519 signature, lowercase hex (128 chars). */
val signature: String,
val domain: String,
)
/**
* Build a fresh `approvalNeeded` frame with a 128-bit random
* `requestId`. Hosts SHOULD persist the requestId in a pending-set
* keyed by `expiresAt` so a returning `linkApproveByProxy` can be
* matched up — that's app state, the SDK doesn't track it.
*/
fun buildApprovalRequest(
crypto: CryptoProvider,
hostAddress: String,
hostFingerprint: String,
requestingDeviceFingerprint: String,
deviceName: String? = null,
userAgent: String? = null,
ipHint: String? = null,
expiresInMs: Long = DEFAULT_APPROVAL_EXPIRES_IN_MS,
domain: String = DEFAULT_APPROVAL_DOMAIN,
now: Long = System.currentTimeMillis(),
): ApprovalRequestFrame {
val requestId = crypto.randomBytes(16).joinToString("") { "%02x".format(it) }
return ApprovalRequestFrame(
requestId = requestId,
hostAddress = hostAddress,
hostFingerprint = hostFingerprint,
requestingDevice = ApprovalRequestingDevice(
fingerprint = requestingDeviceFingerprint,
deviceName = deviceName,
userAgent = userAgent,
ipHint = ipHint,
receivedAt = now,
),
expiresAt = now + expiresInMs,
domain = domain,
)
}
/**
* Sign a `linkApproveByProxy` frame with the approver's long-term
* Ed25519 identity key. The seed is the 32-byte Ed25519 private key
* (Tink's `Ed25519Sign(seed)` consumes it directly).
*/
fun signProxyApproval(
crypto: CryptoProvider,
request: ApprovalRequestFrame,
decision: String,
approverFingerprint: String,
approverSigningKey: ByteArray,
): ProxyApprovalFrame {
require(decision == "approve" || decision == "reject") {
"decision must be 'approve' or 'reject'"
}
require(approverSigningKey.size == 32) {
"approverSigningKey must be 32 bytes (Ed25519 seed)"
}
val payload = canonicalApprovalSigningBytes(
domain = request.domain,
requestId = request.requestId,
hostFingerprint = request.hostFingerprint,
requestingDeviceFingerprint = request.requestingDevice.fingerprint,
decision = decision,
)
val sig = crypto.sign(approverSigningKey, payload)
return ProxyApprovalFrame(
requestId = request.requestId,
decision = decision,
approverFingerprint = approverFingerprint,
signature = sig.joinToString("") { "%02x".format(it) },
domain = request.domain,
)
}
/** Tagged result of `verifyProxyApproval`. */
sealed class VerifyProxyApprovalResult {
data class Ok(val approver: ProfileClientEntry) : VerifyProxyApprovalResult()
data class Failed(val reason: Reason) : VerifyProxyApprovalResult()
enum class Reason {
REQUEST_ID_MISMATCH,
DOMAIN_MISMATCH,
UNKNOWN_APPROVER,
NOT_TRUSTED,
BAD_SIGNATURE,
EXPIRED,
}
}
/**
* Verify a `linkApproveByProxy` against the originating
* `approvalNeeded` and the host's freshest profile blob. Returns a
* tagged result rather than throwing — callers usually want to log
* the reason before deciding what to surface to the user.
*
* Order of checks:
*
* 1. requestId match (replay defense)
* 2. domain match (cross-app confusion defense)
* 3. approver resolves to a `clients[]` entry
* 4. approver is in `trustedApproverFingerprints[]` AND has the
* `trustedApprover` flag (cross-checked via `isTrustedApprover`)
* 5. expiresAt in the future
* 6. Ed25519 signature verifies against `clients[].identityPublicKey`
*
* Hosts MUST refetch the profile blob fresh before calling this — see
* the FR §5 "approver-revocation propagation" rationale.
*/
fun verifyProxyApproval(
crypto: CryptoProvider,
request: ApprovalRequestFrame,
approval: ProxyApprovalFrame,
profile: CanonicalProfileBlob,
now: Long = System.currentTimeMillis(),
): VerifyProxyApprovalResult {
if (approval.requestId != request.requestId) {
return VerifyProxyApprovalResult.Failed(
VerifyProxyApprovalResult.Reason.REQUEST_ID_MISMATCH,
)
}
if (approval.domain != request.domain) {
return VerifyProxyApprovalResult.Failed(
VerifyProxyApprovalResult.Reason.DOMAIN_MISMATCH,
)
}
val approver = findClientByFingerprint(profile, approval.approverFingerprint)
?: return VerifyProxyApprovalResult.Failed(
VerifyProxyApprovalResult.Reason.UNKNOWN_APPROVER,
)
if (!isTrustedApprover(profile, approval.approverFingerprint)) {
return VerifyProxyApprovalResult.Failed(
VerifyProxyApprovalResult.Reason.NOT_TRUSTED,
)
}
if (now > request.expiresAt) {
return VerifyProxyApprovalResult.Failed(VerifyProxyApprovalResult.Reason.EXPIRED)
}
val pubkey = try {
hexToBytes(approver.identityPublicKey)
} catch (_: Exception) {
return VerifyProxyApprovalResult.Failed(VerifyProxyApprovalResult.Reason.BAD_SIGNATURE)
}
val sig = try {
hexToBytes(approval.signature)
} catch (_: Exception) {
return VerifyProxyApprovalResult.Failed(VerifyProxyApprovalResult.Reason.BAD_SIGNATURE)
}
if (pubkey.size != 32 || sig.size != 64) {
return VerifyProxyApprovalResult.Failed(VerifyProxyApprovalResult.Reason.BAD_SIGNATURE)
}
val payload = canonicalApprovalSigningBytes(
domain = approval.domain,
requestId = approval.requestId,
hostFingerprint = request.hostFingerprint,
requestingDeviceFingerprint = request.requestingDevice.fingerprint,
decision = approval.decision,
)
val ok = crypto.verify(pubkey, payload, sig)
return if (ok) VerifyProxyApprovalResult.Ok(approver)
else VerifyProxyApprovalResult.Failed(VerifyProxyApprovalResult.Reason.BAD_SIGNATURE)
}
/**
* Build the canonical signing payload bytes for a proxy approval.
*
* Format (length-prefixed UTF-8, big-endian u16 lengths):
*
* u16(len(domain)) || domain
* u16(len(requestId)) || requestId
* u16(len(hostFp)) || hostFingerprint
* u16(len(requestFp)) || requestingDeviceFingerprint
* u16(len(decision)) || decision
*
* This is the EXACT byte layout `@shade/sdk`'s
* `canonicalApprovalSigningBytes` produces, ensuring an Android-signed
* approval verifies on a TS host and vice versa.
*/
fun canonicalApprovalSigningBytes(
domain: String,
requestId: String,
hostFingerprint: String,
requestingDeviceFingerprint: String,
decision: String,
): ByteArray {
val fields = listOf(
domain.toByteArray(Charsets.UTF_8),
requestId.toByteArray(Charsets.UTF_8),
hostFingerprint.toByteArray(Charsets.UTF_8),
requestingDeviceFingerprint.toByteArray(Charsets.UTF_8),
decision.toByteArray(Charsets.UTF_8),
)
for (f in fields) {
require(f.size <= 0xFFFF) { "signing field too long: ${f.size} bytes (max 65535)" }
}
val total = fields.sumOf { 2 + it.size }
val buf = ByteBuffer.allocate(total).order(ByteOrder.BIG_ENDIAN)
for (f in fields) {
buf.putShort(f.size.toShort())
buf.put(f)
}
return buf.array()
}
private fun hexToBytes(hex: String): ByteArray {
require(hex.length % 2 == 0) { "hex length must be even" }
require(hex.all { it.isDigit() || it in 'a'..'f' }) { "hex must be lowercase 0-9a-f" }
val out = ByteArray(hex.length / 2)
for (i in out.indices) {
out[i] = ((Character.digit(hex[i * 2], 16) shl 4) +
Character.digit(hex[i * 2 + 1], 16)).toByte()
}
return out
}

View File

@@ -0,0 +1,307 @@
package no.zyon.shade.approval
import org.json.JSONArray
import org.json.JSONObject
/**
* V4.10 — canonical profile-blob schema. Mirror
* `@shade/sdk/approval.ts` byte-for-byte: same field names, same
* JSON shape, same denormalization invariants.
*
* The blob is the AEAD plaintext stored in the V4.9 profile slot. It
* holds the user's list of paired hosts + clients; cross-host
* approval routing reads `clients[]` to find trusted approvers when
* a headless host needs to dispatch a `linkRequest` to a phone.
*
* Mutators (`upsertHost`, `setTrustedApprover`, ...) are immutable —
* they return a new blob and never modify the input. The denormalized
* `trustedApproverFingerprints[]` is rederived on every mutation so it
* can never drift from the per-client `trustedApprover` flag.
*/
/** A host: a device that receives `linkRequest` frames and runs pairing. */
data class ProfileHostEntry(
val address: String,
val name: String,
/** Open enum: `"desktop" | "server" | "laptop" | ...`. */
val kind: String,
val addedAt: Long,
)
/**
* A client: a device that initiates link/approval flows and may
* proxy-approve when `trustedApprover == true`. Stores both the
* 32-byte Ed25519 identity public key (hex) and the safety-number
* fingerprint — the public key is what `verifyProxyApproval` checks
* signatures against; the fingerprint is what UIs display.
*/
data class ProfileClientEntry(
val address: String,
/** 32-byte Ed25519 long-term identity public key, lowercase hex (64 chars). */
val identityPublicKey: String,
/** Safety-number fingerprint of the identity key (computeFingerprint output). */
val identityFingerprint: String,
val name: String,
/** Open enum: `"mobile" | "tablet" | "browser" | ...`. */
val kind: String,
val addedAt: Long,
val trustedApprover: Boolean = false,
)
/**
* Canonical profile blob. `version=1` is the only currently-supported
* shape; bump when an incompatible field is added. Unknown top-level
* fields are dropped on parse — additive changes need a coordinated
* schema bump on both platforms.
*/
data class CanonicalProfileBlob(
val version: Int = 1,
val hosts: List<ProfileHostEntry> = emptyList(),
val clients: List<ProfileClientEntry> = emptyList(),
/** Denormalized list of trusted-approver fingerprints. Rederived on mutate. */
val trustedApproverFingerprints: List<String> = emptyList(),
val updatedAt: Long = 0,
/** Optional hex-encoded pubkey of the writer; informational only. */
val signedBy: String? = null,
)
/** Build a fresh empty profile blob with `updatedAt = now ?? System.currentTimeMillis()`. */
fun emptyCanonicalProfile(now: Long? = null): CanonicalProfileBlob =
CanonicalProfileBlob(updatedAt = now ?: System.currentTimeMillis())
/**
* Decode a profile-blob plaintext (the AEAD-opened bytes) into the
* canonical shape. Throws `IllegalArgumentException` on malformed JSON
* or wrong shape.
*/
fun parseCanonicalProfile(plaintext: ByteArray): CanonicalProfileBlob =
parseCanonicalProfile(plaintext.toString(Charsets.UTF_8))
fun parseCanonicalProfile(plaintext: String): CanonicalProfileBlob {
val obj = try {
JSONObject(plaintext)
} catch (e: Exception) {
throw IllegalArgumentException("profile blob is not valid JSON: ${e.message}")
}
val version = obj.optInt("version", -1)
require(version == 1) { "unsupported profile blob version: $version" }
val hosts = parseArray(obj.optJSONArray("hosts"), "hosts", ::parseHostEntry)
val clients = parseArray(obj.optJSONArray("clients"), "clients", ::parseClientEntry)
val trustedApproverFingerprints = parseStringArray(
obj.optJSONArray("trustedApproverFingerprints"),
"trustedApproverFingerprints",
)
val updatedAt = if (obj.has("updatedAt") && !obj.isNull("updatedAt"))
obj.getLong("updatedAt") else 0L
val signedBy = obj.optString("signedBy", "").takeIf { it.isNotEmpty() }
return CanonicalProfileBlob(
version = 1,
hosts = hosts,
clients = clients,
trustedApproverFingerprints = trustedApproverFingerprints,
updatedAt = updatedAt,
signedBy = signedBy,
)
}
/** Serialize a profile blob to UTF-8 JSON ready for `Profile.put`. */
fun serializeCanonicalProfile(blob: CanonicalProfileBlob): ByteArray {
val json = JSONObject()
json.put("version", blob.version)
json.put("hosts", JSONArray().apply {
blob.hosts.forEach { put(hostEntryToJson(it)) }
})
json.put("clients", JSONArray().apply {
blob.clients.forEach { put(clientEntryToJson(it)) }
})
json.put("trustedApproverFingerprints", JSONArray(blob.trustedApproverFingerprints))
json.put("updatedAt", blob.updatedAt)
if (blob.signedBy != null) json.put("signedBy", blob.signedBy)
return json.toString().toByteArray(Charsets.UTF_8)
}
private fun parseHostEntry(o: JSONObject): ProfileHostEntry =
ProfileHostEntry(
address = o.requireString("address", "hosts"),
name = o.requireString("name", "hosts"),
kind = o.requireString("kind", "hosts"),
addedAt = o.requireLong("addedAt", "hosts"),
)
private fun parseClientEntry(o: JSONObject): ProfileClientEntry {
val identityPublicKey = o.requireString("identityPublicKey", "clients")
require(identityPublicKey.matches(Regex("^[0-9a-f]{64}$"))) {
"clients[].identityPublicKey must be 64 lowercase hex chars"
}
return ProfileClientEntry(
address = o.requireString("address", "clients"),
identityPublicKey = identityPublicKey,
identityFingerprint = o.requireString("identityFingerprint", "clients"),
name = o.requireString("name", "clients"),
kind = o.requireString("kind", "clients"),
addedAt = o.requireLong("addedAt", "clients"),
trustedApprover = o.optBoolean("trustedApprover", false),
)
}
private fun hostEntryToJson(e: ProfileHostEntry): JSONObject = JSONObject().apply {
put("address", e.address)
put("name", e.name)
put("kind", e.kind)
put("addedAt", e.addedAt)
}
private fun clientEntryToJson(e: ProfileClientEntry): JSONObject = JSONObject().apply {
put("address", e.address)
put("identityPublicKey", e.identityPublicKey)
put("identityFingerprint", e.identityFingerprint)
put("name", e.name)
put("kind", e.kind)
put("addedAt", e.addedAt)
if (e.trustedApprover) put("trustedApprover", true)
}
private fun <T> parseArray(
arr: JSONArray?,
field: String,
parse: (JSONObject) -> T,
): List<T> {
if (arr == null) return emptyList()
return (0 until arr.length()).map { i ->
val item = arr.opt(i)
require(item is JSONObject) { "$field[$i] must be an object" }
parse(item)
}
}
private fun parseStringArray(arr: JSONArray?, field: String): List<String> {
if (arr == null) return emptyList()
return (0 until arr.length()).map { i ->
val item = arr.opt(i)
require(item is String) { "$field[$i] must be a string" }
item
}
}
private fun JSONObject.requireString(key: String, ctx: String): String {
require(has(key) && !isNull(key)) { "$ctx[].$key is required" }
val v = get(key)
require(v is String) { "$ctx[].$key must be a string" }
return v
}
private fun JSONObject.requireLong(key: String, ctx: String): Long {
require(has(key) && !isNull(key)) { "$ctx[].$key is required" }
return when (val v = get(key)) {
is Number -> v.toLong()
else -> throw IllegalArgumentException("$ctx[].$key must be a number")
}
}
// ─── Mutators (immutable; return new blob, never mutate input) ──
/** Insert or replace a host entry by address. Bumps `updatedAt`. */
fun upsertHost(
blob: CanonicalProfileBlob,
host: ProfileHostEntry,
now: Long? = null,
): CanonicalProfileBlob {
val hosts = blob.hosts.filter { it.address != host.address } + host
return blob.copy(hosts = hosts, updatedAt = now ?: System.currentTimeMillis())
}
/** Remove the host with the given address, if any. */
fun removeHost(
blob: CanonicalProfileBlob,
address: String,
now: Long? = null,
): CanonicalProfileBlob {
val hosts = blob.hosts.filter { it.address != address }
if (hosts.size == blob.hosts.size) return blob
return blob.copy(hosts = hosts, updatedAt = now ?: System.currentTimeMillis())
}
/**
* Insert or replace a client entry by `identityFingerprint`. Re-derives
* `trustedApproverFingerprints` from the resulting `clients[]` so the
* denormalized list never drifts.
*/
fun upsertClient(
blob: CanonicalProfileBlob,
client: ProfileClientEntry,
now: Long? = null,
): CanonicalProfileBlob {
val clients = blob.clients
.filter { it.identityFingerprint != client.identityFingerprint } + client
return blob.copy(
clients = clients,
trustedApproverFingerprints = deriveTrustedApprovers(clients),
updatedAt = now ?: System.currentTimeMillis(),
)
}
/** Remove the client with the given identityFingerprint, if any. */
fun removeClient(
blob: CanonicalProfileBlob,
identityFingerprint: String,
now: Long? = null,
): CanonicalProfileBlob {
val clients = blob.clients.filter { it.identityFingerprint != identityFingerprint }
if (clients.size == blob.clients.size) return blob
return blob.copy(
clients = clients,
trustedApproverFingerprints = deriveTrustedApprovers(clients),
updatedAt = now ?: System.currentTimeMillis(),
)
}
/**
* Toggle the `trustedApprover` flag on a client by fingerprint.
* Returns the input unchanged if fingerprint isn't found OR the
* desired state already matches (no spurious updatedAt bump).
*/
fun setTrustedApprover(
blob: CanonicalProfileBlob,
identityFingerprint: String,
trusted: Boolean,
now: Long? = null,
): CanonicalProfileBlob {
var touched = false
val clients = blob.clients.map { c ->
if (c.identityFingerprint != identityFingerprint) c
else if (c.trustedApprover == trusted) c
else {
touched = true
c.copy(trustedApprover = trusted)
}
}
if (!touched) return blob
return blob.copy(
clients = clients,
trustedApproverFingerprints = deriveTrustedApprovers(clients),
updatedAt = now ?: System.currentTimeMillis(),
)
}
/**
* True iff the given fingerprint resolves to a client with both
* `trustedApprover == true` AND an entry in `trustedApproverFingerprints[]`.
*/
fun isTrustedApprover(blob: CanonicalProfileBlob, identityFingerprint: String): Boolean {
if (!blob.trustedApproverFingerprints.contains(identityFingerprint)) return false
val c = findClientByFingerprint(blob, identityFingerprint) ?: return false
return c.trustedApprover
}
fun findClientByFingerprint(
blob: CanonicalProfileBlob,
identityFingerprint: String,
): ProfileClientEntry? = blob.clients.firstOrNull { it.identityFingerprint == identityFingerprint }
fun findClientByAddress(blob: CanonicalProfileBlob, address: String): ProfileClientEntry? =
blob.clients.firstOrNull { it.address == address }
private fun deriveTrustedApprovers(clients: List<ProfileClientEntry>): List<String> =
clients.filter { it.trustedApprover }.map { it.identityFingerprint }

View File

@@ -0,0 +1,94 @@
package no.zyon.shade.blob
import javax.crypto.Cipher
import javax.crypto.spec.GCMParameterSpec
import javax.crypto.spec.SecretKeySpec
/**
* AEAD wrapper for the V4.9 profile blob.
*
* Wire format for one ciphertext blob:
* `nonce(12) || ciphertext(N) || tag(16)`
*
* Mirror `@shade/storage-encrypted/crypto/aead.ts` byte-for-byte. The
* relay stores this as a single opaque BLOB column; AAD is reconstructed
* at read-time as `"shade-profile-aad-v1:" + slotIdHex` and is NOT
* stored on the relay.
*/
const val BLOB_AEAD_NONCE_LEN = 12
const val BLOB_AEAD_TAG_LEN = 16
private const val MIN_CIPHERTEXT_LEN = BLOB_AEAD_NONCE_LEN + BLOB_AEAD_TAG_LEN
/**
* Seal a plaintext blob. Returns `nonce || ct||tag` ready for direct
* blob storage. The caller supplies the nonce so this function is
* deterministic — the high-level Profile namespace generates a fresh
* 12-byte random nonce per write to keep (key, nonce, plaintext)
* unique across re-uploads.
*/
fun aeadSeal(
key: ByteArray,
nonce: ByteArray,
plaintext: ByteArray,
aad: ByteArray,
): ByteArray {
require(key.size == 32) { "AES-256-GCM key must be 32 bytes" }
require(nonce.size == BLOB_AEAD_NONCE_LEN) {
"nonce must be $BLOB_AEAD_NONCE_LEN bytes"
}
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val spec = GCMParameterSpec(128, nonce)
cipher.init(Cipher.ENCRYPT_MODE, SecretKeySpec(key, "AES"), spec)
cipher.updateAAD(aad)
val ctTag = cipher.doFinal(plaintext)
val out = ByteArray(BLOB_AEAD_NONCE_LEN + ctTag.size)
System.arraycopy(nonce, 0, out, 0, BLOB_AEAD_NONCE_LEN)
System.arraycopy(ctTag, 0, out, BLOB_AEAD_NONCE_LEN, ctTag.size)
return out
}
/**
* Open a `nonce || ct||tag` blob and return the plaintext. Throws on
* tamper (AEAD tag mismatch) or short input. The caller may pass an
* `expectedNonce` to enforce a deterministic nonce — mismatch throws
* before the AEAD even runs (defense-in-depth against a relay returning
* the wrong slot's blob).
*/
fun aeadOpen(
key: ByteArray,
blob: ByteArray,
aad: ByteArray,
expectedNonce: ByteArray? = null,
): ByteArray {
require(key.size == 32) { "AES-256-GCM key must be 32 bytes" }
require(blob.size >= MIN_CIPHERTEXT_LEN) { "ciphertext blob too short" }
val nonce = blob.copyOfRange(0, BLOB_AEAD_NONCE_LEN)
if (expectedNonce != null && !ctEqual(nonce, expectedNonce)) {
throw IllegalArgumentException(
"nonce mismatch — ciphertext blob has been tampered or row identity changed",
)
}
val ctTag = blob.copyOfRange(BLOB_AEAD_NONCE_LEN, blob.size)
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val spec = GCMParameterSpec(128, nonce)
cipher.init(Cipher.DECRYPT_MODE, SecretKeySpec(key, "AES"), spec)
cipher.updateAAD(aad)
return cipher.doFinal(ctTag)
}
private fun ctEqual(a: ByteArray, b: ByteArray): Boolean {
if (a.size != b.size) return false
var diff = 0
for (i in a.indices) {
diff = diff or (a[i].toInt() xor b[i].toInt())
}
return diff == 0
}
/** Build the AAD for a given slotId hex string. */
fun blobAadForSlot(slotIdHex: String): ByteArray {
require(slotIdHex.length == 64) { "slotIdHex must be 64 hex chars" }
return "shade-profile-aad-v1:$slotIdHex".toByteArray(Charsets.UTF_8)
}

View File

@@ -0,0 +1,250 @@
package no.zyon.shade.blob
import no.zyon.shade.crypto.CryptoProvider
import org.json.JSONObject
import java.net.URI
import java.net.http.HttpClient
import java.net.http.HttpRequest
import java.net.http.HttpResponse
import java.time.Duration
import java.util.Base64
/**
* Low-level HTTP client for the V4.9 encrypted-blob primitive
* (`/v1/blob/<slotId>`). Mirror `@shade/inbox`'s `BlobClient` —
* stateless, reusable, and protocol-compatible with the TypeScript
* relay endpoints.
*
* The client doesn't care what the blob bytes mean — it just
* transports them. Higher-level wrappers (e.g. `Profile`) compose
* this client with AEAD-sealing of the actual payload.
*
* Auth model: every PUT/DELETE carries a detached Ed25519 signature
* (base64) over a canonical-JSON form of the request body. The
* canonicalization is deterministic — sorted keys, compact JSON, no
* trailing whitespace — so signatures generated on Kotlin verify on
* the TS server.
*/
class BlobClient(
private val baseUrl: String,
private val crypto: CryptoProvider,
private val httpClient: HttpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(15))
.build(),
) {
data class GetResult(
val blob: ByteArray,
val etag: String,
val updatedAt: Long,
)
data class PutResult(
val created: Boolean,
val etag: String,
val updatedAt: Long,
)
/**
* Read a slot. Returns null if no blob has ever been written there
* (or if it was DELETE'd). GET is unauthenticated by design — the
* slotId is itself a 256-bit secret derived from the master key.
*/
fun get(slotIdHex: String): GetResult? {
validateSlotIdHex(slotIdHex)
val req = HttpRequest.newBuilder(URI.create(joinUrl(baseUrl, "/v1/blob/$slotIdHex")))
.GET()
.build()
val res = httpClient.send(req, HttpResponse.BodyHandlers.ofString())
if (res.statusCode() == 404) return null
val json = parseJson(res, "GET")
val blob = Base64.getDecoder().decode(json.getString("blob"))
return GetResult(
blob = blob,
etag = json.getString("etag"),
updatedAt = json.getLong("updatedAt"),
)
}
/**
* Create or update a slot.
*
* `ifMatch` semantics:
* - `null`: create-only. Slot must be empty (else 409).
* - `<etag-string>`: compare-and-swap. Must match (else 412).
* - `"*"`: unconditional overwrite. Slot must already exist (else 412).
*/
fun put(
slotIdHex: String,
blob: ByteArray,
signingSeed: ByteArray,
ownerPubkey: ByteArray,
ifMatch: String? = null,
): PutResult {
validateSlotIdHex(slotIdHex)
require(blob.isNotEmpty()) { "Empty blob" }
require(ownerPubkey.size == 32) { "ownerPubkey must be 32 bytes (Ed25519)" }
require(signingSeed.size == 32) { "signingSeed must be 32 bytes (Ed25519)" }
// Canonical form for signing: sorted keys, slotId included,
// signature field absent. The wire body strips slotId (it's
// in the URL) but the signature is computed over the
// slotId-bearing form.
val signedAt = System.currentTimeMillis()
val canonical = sortedMapOf<String, Any>().apply {
put("blob", Base64.getEncoder().encodeToString(blob))
if (ifMatch != null) put("ifMatch", ifMatch)
put("ownerPubkey", Base64.getEncoder().encodeToString(ownerPubkey))
put("signedAt", signedAt)
put("slotId", slotIdHex)
}
val canonicalBytes = canonicalJson(canonical).toByteArray(Charsets.UTF_8)
val sig = crypto.sign(signingSeed, canonicalBytes)
// Wire body: same as canonical minus slotId, plus signature.
val wire = JSONObject()
wire.put("blob", Base64.getEncoder().encodeToString(blob))
if (ifMatch != null) wire.put("ifMatch", ifMatch)
wire.put("ownerPubkey", Base64.getEncoder().encodeToString(ownerPubkey))
wire.put("signedAt", signedAt)
wire.put("signature", Base64.getEncoder().encodeToString(sig))
val req = HttpRequest.newBuilder(URI.create(joinUrl(baseUrl, "/v1/blob/$slotIdHex")))
.header("content-type", "application/json")
.PUT(HttpRequest.BodyPublishers.ofString(wire.toString()))
.build()
val res = httpClient.send(req, HttpResponse.BodyHandlers.ofString())
val json = parseJson(res, "PUT")
return PutResult(
created = json.optBoolean("created"),
etag = json.getString("etag"),
updatedAt = json.getLong("updatedAt"),
)
}
/**
* Delete a slot. The next PUT TOFU-claims it again, possibly under
* a fresh signing key (e.g. after rotation). Used by "forget
* everything" flows.
*/
fun delete(slotIdHex: String, signingSeed: ByteArray): Boolean {
validateSlotIdHex(slotIdHex)
require(signingSeed.size == 32) { "signingSeed must be 32 bytes (Ed25519)" }
val signedAt = System.currentTimeMillis()
val canonical = sortedMapOf<String, Any>().apply {
put("signedAt", signedAt)
put("slotId", slotIdHex)
}
val canonicalBytes = canonicalJson(canonical).toByteArray(Charsets.UTF_8)
val sig = crypto.sign(signingSeed, canonicalBytes)
val wire = JSONObject()
wire.put("signedAt", signedAt)
wire.put("signature", Base64.getEncoder().encodeToString(sig))
val req = HttpRequest.newBuilder(URI.create(joinUrl(baseUrl, "/v1/blob/$slotIdHex")))
.header("content-type", "application/json")
.method("DELETE", HttpRequest.BodyPublishers.ofString(wire.toString()))
.build()
val res = httpClient.send(req, HttpResponse.BodyHandlers.ofString())
val json = parseJson(res, "DELETE")
return json.optBoolean("ok", false)
}
private fun parseJson(res: HttpResponse<String>, op: String): JSONObject {
val text = res.body() ?: ""
val json = if (text.isEmpty()) JSONObject() else try {
JSONObject(text)
} catch (e: Exception) {
throw BlobClientException(
code = "SHADE_NETWORK",
statusCode = res.statusCode(),
message = "Blob $op response not JSON: ${text.take(200)}",
)
}
if (res.statusCode() !in 200..299) {
throw BlobClientException(
code = json.optString("code", "SHADE_NETWORK"),
statusCode = res.statusCode(),
message = json.optString("message", text),
)
}
return json
}
private fun validateSlotIdHex(s: String) {
require(s.matches(Regex("^[0-9a-f]{64}$"))) {
"slotIdHex must be 64 lowercase hex chars (32 bytes)"
}
}
private fun joinUrl(base: String, path: String): String =
when {
base.endsWith("/") && path.startsWith("/") -> base + path.substring(1)
!base.endsWith("/") && !path.startsWith("/") -> "$base/$path"
else -> base + path
}
}
/**
* Mirror of TS `signPayload`'s canonicalization: sorted keys, compact
* JSON, signature field absent. Only handles the subset of types we
* need (strings + longs + base64 strings) — keeping the implementation
* narrow so it can't accidentally diverge from `JSON.stringify` on
* structurally-different inputs.
*/
internal fun canonicalJson(map: Map<String, Any>): String {
val sb = StringBuilder()
sb.append('{')
var first = true
for ((key, value) in map.toSortedMap()) {
if (!first) sb.append(',')
first = false
appendJsonString(sb, key)
sb.append(':')
appendJsonValue(sb, value)
}
sb.append('}')
return sb.toString()
}
private fun appendJsonValue(sb: StringBuilder, value: Any) {
when (value) {
is String -> appendJsonString(sb, value)
is Long, is Int -> sb.append(value.toString())
is Boolean -> sb.append(value.toString())
else -> throw IllegalArgumentException(
"canonicalJson: unsupported value type ${value::class.java}",
)
}
}
private fun appendJsonString(sb: StringBuilder, s: String) {
sb.append('"')
for (c in s) {
when (c) {
'\\' -> sb.append("\\\\")
'"' -> sb.append("\\\"")
'\b' -> sb.append("\\b")
'\u000C' -> sb.append("\\f")
'\n' -> sb.append("\\n")
'\r' -> sb.append("\\r")
'\t' -> sb.append("\\t")
else -> {
if (c.code < 0x20) {
sb.append("\\u").append("%04x".format(c.code))
} else {
sb.append(c)
}
}
}
}
sb.append('"')
}
class BlobClientException(
val code: String,
val statusCode: Int,
message: String,
) : RuntimeException("[$code @ $statusCode] $message")

View File

@@ -0,0 +1,88 @@
package no.zyon.shade.blob
import com.google.crypto.tink.subtle.Ed25519Sign
import no.zyon.shade.crypto.CryptoProvider
/**
* V4.9 — relay-side encrypted blob primitive: deterministic
* derivations from a 32-byte master key + per-app namespace string.
*
* Mirror `@shade/storage-encrypted/crypto/kdf.ts` byte-for-byte.
* Reference vectors: `test-vectors/blob-storage.json`.
*
* Three independent 32-byte derivations:
*
* slotId = HKDF(masterKey, info=`shade-blob-slot-v1:<app>`) // relay-visible opaque ID
* blobKey = HKDF(masterKey, info=`shade-blob-key-v1:<app>`) // AEAD key for the blob
* sigSeed = HKDF(masterKey, info=`shade-blob-sig-v1:<app>`) // Ed25519 owner signing seed
*
* `app` is a caller-supplied namespace string — distinct apps under
* the same master MUST pass different values (e.g. `prism-profile-v1`)
* so they don't collide on the same slot.
*
* The signing seed is an Ed25519 *seed* in the @noble/curves convention:
* `pubkey = Ed25519.publicFromSeed(seed)` is what the relay TOFU-stores
* on the first PUT and verifies subsequent writes against.
*/
private const val SLOT_INFO_PREFIX = "shade-blob-slot-v1:"
private const val BLOB_KEY_INFO_PREFIX = "shade-blob-key-v1:"
private const val SIG_SEED_INFO_PREFIX = "shade-blob-sig-v1:"
private const val DERIVED_LEN = 32
/** Lower-hex 64-char slotId derived deterministically from the master key. */
fun deriveBlobSlotId(crypto: CryptoProvider, masterKey: ByteArray, app: String): ByteArray {
require(masterKey.size == 32) { "masterKey must be 32 bytes" }
require(app.isNotEmpty()) { "app namespace must be non-empty" }
return crypto.hkdf(
masterKey,
ByteArray(0),
(SLOT_INFO_PREFIX + app).toByteArray(Charsets.UTF_8),
DERIVED_LEN,
)
}
/** AEAD key for sealing/opening the blob. The slotId hex is bound as AAD. */
fun deriveBlobKey(crypto: CryptoProvider, masterKey: ByteArray, app: String): ByteArray {
require(masterKey.size == 32) { "masterKey must be 32 bytes" }
require(app.isNotEmpty()) { "app namespace must be non-empty" }
return crypto.hkdf(
masterKey,
ByteArray(0),
(BLOB_KEY_INFO_PREFIX + app).toByteArray(Charsets.UTF_8),
DERIVED_LEN,
)
}
/**
* 32-byte Ed25519 signing seed. The pubkey, derived deterministically
* from the seed, is what the relay TOFU-stores on the first PUT.
*/
fun deriveBlobSigningSeed(crypto: CryptoProvider, masterKey: ByteArray, app: String): ByteArray {
require(masterKey.size == 32) { "masterKey must be 32 bytes" }
require(app.isNotEmpty()) { "app namespace must be non-empty" }
return crypto.hkdf(
masterKey,
ByteArray(0),
(SIG_SEED_INFO_PREFIX + app).toByteArray(Charsets.UTF_8),
DERIVED_LEN,
)
}
/**
* Recover the Ed25519 public key from a 32-byte seed. Mirrors
* `@shade/crypto-web`'s `ed25519PublicKeyFromSeed`. Tink's
* `Ed25519Sign.KeyPair.newKeyPairFromSeed(seed)` exposes both halves;
* we discard the private half here.
*/
fun ed25519PublicKeyFromSeed(seed: ByteArray): ByteArray {
require(seed.size == 32) { "Ed25519 seed must be 32 bytes" }
return Ed25519Sign.KeyPair.newKeyPairFromSeed(seed).publicKey
}
/** Convert a 32-byte slotId into the lowercase-hex wire form (64 chars). */
fun slotIdToHex(slotId: ByteArray): String {
require(slotId.size == 32) { "slotId must be 32 bytes" }
return slotId.joinToString("") { "%02x".format(it) }
}

View File

@@ -0,0 +1,118 @@
package no.zyon.shade.blob
import no.zyon.shade.crypto.CryptoProvider
/**
* V4.9 — high-level profile namespace. Mirror
* `@shade/sdk`'s `createProfileNamespace`. The relay never sees
* plaintext; AAD binds the slotId so a relay returning the wrong
* slot's blob fails to open.
*
* Usage:
*
* val crypto = TinkProvider()
* val masterKey = deriveMasterKey("password", salt) // V4.5 KDF chain
* val profile = createProfileNamespace(
* baseUrl = "https://shade.example/",
* crypto = crypto,
* masterKey = masterKey,
* app = "prism-profile-v1",
* )
*
* val current = profile.get() // null if no blob yet
* profile.put(serializeCanonicalProfile(...), ifMatch = current?.etag)
* profile.delete()
*
* Apps with the same master key + app namespace converge on the same
* slot — that's the whole point: a brand new device with the right
* credentials can locate, decrypt, and update the blob.
*/
class ProfileNamespace internal constructor(
/** Lower-hex 64-char slotId. Stable per (master, app). */
val slotIdHex: String,
private val blobKey: ByteArray,
private val signingSeed: ByteArray,
private val ownerPubkey: ByteArray,
private val aad: ByteArray,
private val client: BlobClient,
private val crypto: CryptoProvider,
) {
data class GetResult(
val plaintext: ByteArray,
val etag: String,
val updatedAt: Long,
)
data class PutResult(
val created: Boolean,
val etag: String,
val updatedAt: Long,
)
/** Returns null when the slot has never been written (or was deleted). */
fun get(): GetResult? {
val raw = client.get(slotIdHex) ?: return null
val plaintext = aeadOpen(blobKey, raw.blob, aad)
return GetResult(plaintext = plaintext, etag = raw.etag, updatedAt = raw.updatedAt)
}
/**
* Create or update. `ifMatch`:
* - null: create-only (fails with 409 if slot populated).
* - "<etag>": CAS (fails with 412 on stale).
* - "*": unconditional overwrite.
*/
fun put(plaintext: ByteArray, ifMatch: String? = null): PutResult {
// Fresh random nonce per write — see `BlobAead`. Re-uploading
// the same plaintext after a transient error reuses neither
// (key, nonce, plaintext) nor (key, nonce).
val nonce = crypto.randomBytes(BLOB_AEAD_NONCE_LEN)
val sealed = aeadSeal(blobKey, nonce, plaintext, aad)
val r = client.put(
slotIdHex = slotIdHex,
blob = sealed,
signingSeed = signingSeed,
ownerPubkey = ownerPubkey,
ifMatch = ifMatch,
)
return PutResult(created = r.created, etag = r.etag, updatedAt = r.updatedAt)
}
/** "Forget everything" path — the next PUT TOFU-claims it again. */
fun delete(): Boolean = client.delete(slotIdHex, signingSeed)
}
/**
* Build a Profile namespace bound to a (master key, app) pair. The
* derivations are deterministic: any device with the same master
* key + app namespace produces the same slot, so a fresh device
* after credential entry can locate the existing profile blob.
*/
fun createProfileNamespace(
baseUrl: String,
crypto: CryptoProvider,
masterKey: ByteArray,
app: String,
): ProfileNamespace {
require(masterKey.size == 32) { "masterKey must be 32 bytes" }
require(app.isNotEmpty()) { "app namespace must be non-empty" }
val slotIdBytes = deriveBlobSlotId(crypto, masterKey, app)
val slotIdHex = slotIdToHex(slotIdBytes)
val blobKey = deriveBlobKey(crypto, masterKey, app)
val signingSeed = deriveBlobSigningSeed(crypto, masterKey, app)
val ownerPubkey = ed25519PublicKeyFromSeed(signingSeed)
val aad = blobAadForSlot(slotIdHex)
val client = BlobClient(baseUrl = baseUrl, crypto = crypto)
return ProfileNamespace(
slotIdHex = slotIdHex,
blobKey = blobKey,
signingSeed = signingSeed,
ownerPubkey = ownerPubkey,
aad = aad,
client = client,
crypto = crypto,
)
}

View File

@@ -0,0 +1,133 @@
package no.zyon.shade.serialization
import no.zyon.shade.types.ChainState
import no.zyon.shade.types.IdentityKeyPair
import no.zyon.shade.types.KeyPair
import no.zyon.shade.types.OneTimePreKey
import no.zyon.shade.types.SessionState
import no.zyon.shade.types.SignedPreKey
import org.json.JSONObject
import java.util.Base64 as JdkBase64
/**
* Plain-JSON serialization for the persisted protocol state types
* (`IdentityKeyPair`, `SignedPreKey`, `OneTimePreKey`, `SessionState`).
*
* The on-disk shape is for at-rest storage only — it does NOT need
* to round-trip across platforms (TS uses its own JSON shape via
* `@shade/core/serialization`). What matters is that the Kotlin
* round-trip (`serialize` then `deserialize`) preserves every byte.
*
* Both Android-targeted (`shade-android-keystore`) and pure-JVM
* (`shade-android` tests) callers use this — the function works
* without any `android.*` imports so it compiles in both.
*/
object SessionStateJson {
fun serialize(state: SessionState): String {
val o = JSONObject()
o.put("remoteIdentityKey", b64(state.remoteIdentityKey))
o.put("rootKey", b64(state.rootKey))
o.put("sendChain", chainToJson(state.sendChain))
if (state.receiveChain != null) o.put("receiveChain", chainToJson(state.receiveChain!!))
o.put("dhSend", keyPairToJson(state.dhSend))
if (state.dhReceive != null) o.put("dhReceive", b64(state.dhReceive!!))
o.put("previousSendCounter", state.previousSendCounter)
val skipped = JSONObject()
for ((k, v) in state.skippedKeys) skipped.put(k, b64(v))
o.put("skippedKeys", skipped)
return o.toString()
}
fun deserialize(s: String): SessionState {
val o = JSONObject(s)
val skipped = mutableMapOf<String, ByteArray>()
val skJson = o.optJSONObject("skippedKeys")
if (skJson != null) {
val it = skJson.keys()
while (it.hasNext()) {
val k = it.next()
skipped[k] = fb64(skJson.getString(k))
}
}
return SessionState(
remoteIdentityKey = fb64(o.getString("remoteIdentityKey")),
rootKey = fb64(o.getString("rootKey")),
sendChain = chainFromJson(o.getJSONObject("sendChain")),
receiveChain = if (o.has("receiveChain"))
chainFromJson(o.getJSONObject("receiveChain")) else null,
dhSend = keyPairFromJson(o.getJSONObject("dhSend")),
dhReceive = if (o.has("dhReceive")) fb64(o.getString("dhReceive")) else null,
previousSendCounter = o.getInt("previousSendCounter"),
skippedKeys = skipped,
)
}
fun serializeIdentityKeyPair(k: IdentityKeyPair): String = JSONObject().apply {
put("signingPublicKey", b64(k.signingPublicKey))
put("signingPrivateKey", b64(k.signingPrivateKey))
put("dhPublicKey", b64(k.dhPublicKey))
put("dhPrivateKey", b64(k.dhPrivateKey))
}.toString()
fun deserializeIdentityKeyPair(s: String): IdentityKeyPair = JSONObject(s).run {
IdentityKeyPair(
signingPublicKey = fb64(getString("signingPublicKey")),
signingPrivateKey = fb64(getString("signingPrivateKey")),
dhPublicKey = fb64(getString("dhPublicKey")),
dhPrivateKey = fb64(getString("dhPrivateKey")),
)
}
fun serializeSignedPreKey(k: SignedPreKey): String = JSONObject().apply {
put("keyId", k.keyId)
put("keyPair", keyPairToJson(k.keyPair))
put("signature", b64(k.signature))
put("timestamp", k.timestamp)
}.toString()
fun deserializeSignedPreKey(s: String): SignedPreKey = JSONObject(s).run {
SignedPreKey(
keyId = getInt("keyId"),
keyPair = keyPairFromJson(getJSONObject("keyPair")),
signature = fb64(getString("signature")),
timestamp = getLong("timestamp"),
)
}
fun serializeOneTimePreKey(k: OneTimePreKey): String = JSONObject().apply {
put("keyId", k.keyId)
put("keyPair", keyPairToJson(k.keyPair))
}.toString()
fun deserializeOneTimePreKey(s: String): OneTimePreKey = JSONObject(s).run {
OneTimePreKey(
keyId = getInt("keyId"),
keyPair = keyPairFromJson(getJSONObject("keyPair")),
)
}
private fun chainToJson(c: ChainState): JSONObject = JSONObject().apply {
put("chainKey", b64(c.chainKey))
put("counter", c.counter)
}
private fun chainFromJson(o: JSONObject): ChainState =
ChainState(chainKey = fb64(o.getString("chainKey")), counter = o.getInt("counter"))
private fun keyPairToJson(k: KeyPair): JSONObject = JSONObject().apply {
put("publicKey", b64(k.publicKey))
put("privateKey", b64(k.privateKey))
}
private fun keyPairFromJson(o: JSONObject): KeyPair = KeyPair(
publicKey = fb64(o.getString("publicKey")),
privateKey = fb64(o.getString("privateKey")),
)
// android.util.Base64 isn't on the JVM classpath; java.util.Base64
// is available on both modern JVM and Android API 26+. Use JDK
// Base64 throughout — it's present on both targets.
private fun b64(b: ByteArray): String = JdkBase64.getEncoder().encodeToString(b)
private fun fb64(s: String): ByteArray = JdkBase64.getDecoder().decode(s)
}

View File

@@ -0,0 +1,107 @@
package no.zyon.shade.storage
import org.bouncycastle.crypto.generators.Argon2BytesGenerator
import org.bouncycastle.crypto.generators.SCrypt
import org.bouncycastle.crypto.params.Argon2Parameters
import java.text.Normalizer
/**
* Password / PIN key-derivation primitives. Mirror
* `@shade/storage-encrypted/crypto/kdf` (`deriveMasterKey` /
* `deriveMasterKeyArgon2id`) byte-for-byte — Tink doesn't ship password
* KDFs so we wrap Bouncy Castle.
*
* Both functions normalize string passphrases to NFKC before hashing,
* matching the TS implementation's `passphrase.normalize('NFKC')`.
* This ensures the same password typed on different OSes/keyboards
* produces the same master key regardless of which compatibility-form
* the input arrived in.
*
* The reference test-vector lives in `test-vectors/storage-encryption.json`
* and `test-vectors/blob-storage.json`. Cross-platform parity is gated
* by `CrossPlatformVectorTest`.
*/
/** scrypt parameters. Defaults match `DEFAULT_SCRYPT` in TS. */
data class ScryptParams(
/** CPU/memory cost. Must be a power of 2. */
val n: Int = 1 shl 17,
/** Block size. */
val r: Int = 8,
/** Parallelization. */
val p: Int = 1,
/** Output length in bytes. */
val dkLen: Int = 32,
)
/** Argon2id parameters. Defaults match `DEFAULT_ARGON2ID` in TS. */
data class Argon2idParams(
/** Memory cost in KiB. Default 64 MiB. */
val m: Int = 64 * 1024,
/** Time cost (iterations). Default 3. */
val t: Int = 3,
/** Parallelism. Default 1. */
val p: Int = 1,
/** Output length in bytes. Default 32. */
val dkLen: Int = 32,
)
/**
* Derive a 32-byte master key from a passphrase + salt via scrypt.
* Salt MUST be at least 16 bytes and persisted alongside the
* encrypted database. Throws on empty passphrase.
*/
fun deriveMasterKey(
passphrase: String,
salt: ByteArray,
params: ScryptParams = ScryptParams(),
): ByteArray {
require(passphrase.isNotEmpty()) { "passphrase must be non-empty" }
require(salt.size >= 16) { "salt must be at least 16 bytes" }
val nfkc = Normalizer.normalize(passphrase, Normalizer.Form.NFKC)
val pwBytes = nfkc.toByteArray(Charsets.UTF_8)
return SCrypt.generate(pwBytes, salt, params.n, params.r, params.p, params.dkLen)
}
/**
* Derive a 32-byte master key from a low-entropy secret (PIN) + salt
* via argon2id. Salt MUST be at least 16 bytes. The Bouncy Castle
* `Argon2BytesGenerator` parameters mirror RFC 9106's argon2id mode
* with version 1.3 (`Argon2Parameters.ARGON2_VERSION_13`), which is
* what `@noble/hashes/argon2` produces — keeping cross-platform parity.
*/
fun deriveMasterKeyArgon2id(
secret: String,
salt: ByteArray,
params: Argon2idParams = Argon2idParams(),
): ByteArray {
require(secret.isNotEmpty()) { "argon2id secret must be non-empty" }
require(salt.size >= 16) { "salt must be at least 16 bytes" }
val nfkc = Normalizer.normalize(secret, Normalizer.Form.NFKC)
return deriveMasterKeyArgon2id(nfkc.toByteArray(Charsets.UTF_8), salt, params)
}
/**
* Byte-array overload — useful when the secret is already binary
* (e.g. derived from a hardware token rather than typed) and
* shouldn't be NFKC-normalized as text.
*/
fun deriveMasterKeyArgon2id(
secret: ByteArray,
salt: ByteArray,
params: Argon2idParams = Argon2idParams(),
): ByteArray {
require(secret.isNotEmpty()) { "argon2id secret must be non-empty" }
require(salt.size >= 16) { "salt must be at least 16 bytes" }
val builder = Argon2Parameters.Builder(Argon2Parameters.ARGON2_id)
.withVersion(Argon2Parameters.ARGON2_VERSION_13)
.withIterations(params.t)
.withMemoryAsKB(params.m)
.withParallelism(params.p)
.withSalt(salt)
val gen = Argon2BytesGenerator()
gen.init(builder.build())
val out = ByteArray(params.dkLen)
gen.generateBytes(secret, out)
return out
}

View File

@@ -0,0 +1,593 @@
package no.zyon.shade
import no.zyon.shade.approval.ApprovalRequestFrame
import no.zyon.shade.approval.ApprovalRequestingDevice
import no.zyon.shade.approval.CanonicalProfileBlob
import no.zyon.shade.approval.DEFAULT_APPROVAL_DOMAIN
import no.zyon.shade.approval.ProfileClientEntry
import no.zyon.shade.approval.ProfileHostEntry
import no.zyon.shade.approval.ProxyApprovalFrame
import no.zyon.shade.approval.VerifyProxyApprovalResult
import no.zyon.shade.approval.buildApprovalRequest
import no.zyon.shade.approval.canonicalApprovalSigningBytes
import no.zyon.shade.approval.emptyCanonicalProfile
import no.zyon.shade.approval.findClientByAddress
import no.zyon.shade.approval.findClientByFingerprint
import no.zyon.shade.approval.isTrustedApprover
import no.zyon.shade.approval.parseCanonicalProfile
import no.zyon.shade.approval.removeClient
import no.zyon.shade.approval.serializeCanonicalProfile
import no.zyon.shade.approval.setTrustedApprover
import no.zyon.shade.approval.signProxyApproval
import no.zyon.shade.approval.upsertClient
import no.zyon.shade.approval.upsertHost
import no.zyon.shade.approval.verifyProxyApproval
import no.zyon.shade.blob.aeadOpen
import no.zyon.shade.blob.aeadSeal
import no.zyon.shade.blob.blobAadForSlot
import no.zyon.shade.blob.deriveBlobKey
import no.zyon.shade.blob.deriveBlobSigningSeed
import no.zyon.shade.blob.deriveBlobSlotId
import no.zyon.shade.blob.ed25519PublicKeyFromSeed
import no.zyon.shade.blob.slotIdToHex
import no.zyon.shade.crypto.TinkProvider
import no.zyon.shade.storage.Argon2idParams
import no.zyon.shade.storage.ScryptParams
import no.zyon.shade.storage.deriveMasterKey
import no.zyon.shade.storage.deriveMasterKeyArgon2id
import org.junit.Assert.assertArrayEquals
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertNotNull
import org.junit.Assert.assertNull
import org.junit.Assert.assertNotEquals
import org.junit.Assert.assertTrue
import org.junit.Test
/**
* Unit tests for the V4.9 blob primitive ports + V4.10 approval
* helpers + scrypt/argon2id wrappers. Cross-platform vector parity
* lives in `CrossPlatformVectorTest`; this file tests Kotlin-side
* round-trip behavior independent of the TS reference.
*/
class BlobAndApprovalTest {
private val crypto = TinkProvider()
private fun hex(bytes: ByteArray): String =
bytes.joinToString("") { "%02x".format(it) }
// ─── V4.9 blob KDF ─────────────────────────────────────────
@Test
fun deriveBlobSlotIdIsDeterministicPerMasterAndApp() {
val km = ByteArray(32) { it.toByte() }
val a1 = deriveBlobSlotId(crypto, km, "foo")
val a2 = deriveBlobSlotId(crypto, km, "foo")
assertArrayEquals(a1, a2)
val b = deriveBlobSlotId(crypto, km, "bar")
assertFalse(a1.contentEquals(b))
val km2 = ByteArray(32) { (it + 1).toByte() }
val c = deriveBlobSlotId(crypto, km2, "foo")
assertFalse(a1.contentEquals(c))
}
@Test
fun blobKdfHelpersAreIndependent() {
val km = ByteArray(32) { it.toByte() }
val slot = deriveBlobSlotId(crypto, km, "x")
val key = deriveBlobKey(crypto, km, "x")
val seed = deriveBlobSigningSeed(crypto, km, "x")
assertFalse(slot.contentEquals(key))
assertFalse(slot.contentEquals(seed))
assertFalse(key.contentEquals(seed))
}
@Test
fun ed25519PublicKeyFromSeedIsDeterministic() {
val seed = ByteArray(32) { it.toByte() }
val pk1 = ed25519PublicKeyFromSeed(seed)
val pk2 = ed25519PublicKeyFromSeed(seed)
assertArrayEquals(pk1, pk2)
assertEquals(32, pk1.size)
}
@Test
fun slotIdToHexProducesLowercase64Chars() {
val s = ByteArray(32) { 0xab.toByte() }
val hex = slotIdToHex(s)
assertEquals(64, hex.length)
assertEquals("a".repeat(0) + "ab".repeat(32), hex)
}
// ─── V4.9 AEAD round-trip ──────────────────────────────────
@Test
fun aeadSealOpenRoundTrip() {
val key = crypto.randomBytes(32)
val nonce = crypto.randomBytes(12)
val pt = "hello".toByteArray()
val aad = blobAadForSlot("00".repeat(32))
val sealed = aeadSeal(key, nonce, pt, aad)
val opened = aeadOpen(key, sealed, aad)
assertArrayEquals(pt, opened)
}
@Test
fun aeadOpenWithWrongAadFails() {
val key = crypto.randomBytes(32)
val nonce = crypto.randomBytes(12)
val pt = "hello".toByteArray()
val aad1 = blobAadForSlot("00".repeat(32))
val aad2 = blobAadForSlot("ff".repeat(32))
val sealed = aeadSeal(key, nonce, pt, aad1)
try {
aeadOpen(key, sealed, aad2)
org.junit.Assert.fail("expected AEAD to reject wrong AAD")
} catch (_: Exception) {
// expected
}
}
@Test
fun aeadOpenWithExpectedNonceMismatchFails() {
val key = crypto.randomBytes(32)
val nonce = crypto.randomBytes(12)
val wrongNonce = crypto.randomBytes(12)
val pt = "hello".toByteArray()
val aad = blobAadForSlot("00".repeat(32))
val sealed = aeadSeal(key, nonce, pt, aad)
try {
aeadOpen(key, sealed, aad, expectedNonce = wrongNonce)
org.junit.Assert.fail("expected expectedNonce check to reject")
} catch (_: IllegalArgumentException) {
// expected
}
}
// ─── Password KDFs ─────────────────────────────────────────
@Test
fun scryptDerivesDeterministically() {
val pw = "correct-horse-battery-staple"
val salt = ByteArray(16) { 0x42.toByte() }
val k1 = deriveMasterKey(pw, salt, ScryptParams(n = 1024))
val k2 = deriveMasterKey(pw, salt, ScryptParams(n = 1024))
assertArrayEquals(k1, k2)
assertEquals(32, k1.size)
}
@Test
fun argon2idDerivesDeterministically() {
val pw = "1234"
val salt = ByteArray(16) { 0x55.toByte() }
val k1 = deriveMasterKeyArgon2id(pw, salt, Argon2idParams(m = 256, t = 2))
val k2 = deriveMasterKeyArgon2id(pw, salt, Argon2idParams(m = 256, t = 2))
assertArrayEquals(k1, k2)
assertEquals(32, k1.size)
}
@Test
fun nfkcNormalizationMakesEquivalentInputsConverge() {
// "café" can be encoded either as 'c','a','f','é' (NFC) or
// 'c','a','f','e','́' (NFD). NFKC normalization on both
// should converge to the same bytes.
val nfc = "café"
val nfd = "café"
assertNotEquals(nfc, nfd)
val salt = ByteArray(16) { 1.toByte() }
val k1 = deriveMasterKey(nfc, salt, ScryptParams(n = 1024))
val k2 = deriveMasterKey(nfd, salt, ScryptParams(n = 1024))
assertArrayEquals(k1, k2)
}
// ─── Canonical profile schema ──────────────────────────────
private fun makeClient(name: String, trusted: Boolean = false): Pair<ProfileClientEntry, ByteArray> {
val seed = crypto.randomBytes(32)
val pubkey = ed25519PublicKeyFromSeed(seed)
val fp = "fp-$name-${hex(pubkey).take(8)}"
return ProfileClientEntry(
address = "device:$name",
identityPublicKey = hex(pubkey),
identityFingerprint = fp,
name = name,
kind = "mobile",
addedAt = 1_700_000_000_000L,
trustedApprover = trusted,
) to seed
}
private fun makeHost(): ProfileHostEntry = ProfileHostEntry(
address = "device:host-server",
name = "Server",
kind = "server",
addedAt = 1_700_000_000_000L,
)
@Test
fun emptyCanonicalProfileRoundTrips() {
val blob = emptyCanonicalProfile(now = 123L)
val bytes = serializeCanonicalProfile(blob)
val parsed = parseCanonicalProfile(bytes)
assertEquals(1, parsed.version)
assertTrue(parsed.hosts.isEmpty())
assertTrue(parsed.clients.isEmpty())
assertTrue(parsed.trustedApproverFingerprints.isEmpty())
assertEquals(123L, parsed.updatedAt)
}
@Test
fun upsertClientDenormalizesTrustedApprovers() {
var blob = emptyCanonicalProfile(0)
val (a, _) = makeClient("phone-a", trusted = true)
val (b, _) = makeClient("phone-b", trusted = false)
blob = upsertClient(blob, a)
blob = upsertClient(blob, b)
assertEquals(2, blob.clients.size)
assertEquals(listOf(a.identityFingerprint), blob.trustedApproverFingerprints)
assertTrue(isTrustedApprover(blob, a.identityFingerprint))
assertFalse(isTrustedApprover(blob, b.identityFingerprint))
}
@Test
fun setTrustedApproverIsIdempotentNoOpReturnsSameInstance() {
var blob = emptyCanonicalProfile(0)
val (c, _) = makeClient("phone", trusted = false)
blob = upsertClient(blob, c)
val before = blob
blob = setTrustedApprover(blob, c.identityFingerprint, false, now = 999L)
assertTrue(blob === before)
}
@Test
fun setTrustedApproverFlipsFlagAndDenormalizedList() {
var blob = emptyCanonicalProfile(0)
val (c, _) = makeClient("phone", trusted = false)
blob = upsertClient(blob, c)
blob = setTrustedApprover(blob, c.identityFingerprint, true, now = 100L)
assertEquals(listOf(c.identityFingerprint), blob.trustedApproverFingerprints)
assertEquals(true, blob.clients[0].trustedApprover)
blob = setTrustedApprover(blob, c.identityFingerprint, false, now = 200L)
assertTrue(blob.trustedApproverFingerprints.isEmpty())
assertEquals(false, blob.clients[0].trustedApprover)
}
@Test
fun removeClientCleansUpDenormalizedList() {
var blob = emptyCanonicalProfile(0)
val (c, _) = makeClient("phone", trusted = true)
blob = upsertClient(blob, c)
blob = removeClient(blob, c.identityFingerprint)
assertTrue(blob.clients.isEmpty())
assertTrue(blob.trustedApproverFingerprints.isEmpty())
}
@Test
fun findClientByFingerprintAndAddress() {
var blob = emptyCanonicalProfile(0)
val (c, _) = makeClient("phone")
blob = upsertClient(blob, c)
assertEquals(c.address, findClientByFingerprint(blob, c.identityFingerprint)?.address)
assertEquals(
c.identityFingerprint,
findClientByAddress(blob, c.address)?.identityFingerprint,
)
assertNull(findClientByFingerprint(blob, "unknown"))
assertNull(findClientByAddress(blob, "unknown"))
}
@Test
fun parseRejectsMalformedProfile() {
try {
parseCanonicalProfile("not json")
org.junit.Assert.fail("expected throw")
} catch (_: IllegalArgumentException) {}
try {
parseCanonicalProfile("""{"version":2}""")
org.junit.Assert.fail("expected throw")
} catch (_: IllegalArgumentException) {}
try {
parseCanonicalProfile(
"""{"version":1,"clients":[{"address":"x","name":"x","kind":"m","addedAt":0}]}""",
)
org.junit.Assert.fail("expected throw — missing identityPublicKey")
} catch (_: IllegalArgumentException) {}
try {
parseCanonicalProfile(
"""{"version":1,"clients":[{"address":"x","identityPublicKey":"NOTHEX","identityFingerprint":"x","name":"x","kind":"m","addedAt":0}]}""",
)
org.junit.Assert.fail("expected throw — bad pubkey hex")
} catch (_: IllegalArgumentException) {}
}
@Test
fun fullProfileSerializeParsePreservesAllFields() {
var blob = emptyCanonicalProfile(1L)
blob = upsertHost(blob, makeHost(), now = 2L)
val (c, _) = makeClient("phone", trusted = true)
blob = upsertClient(blob, c, now = 3L)
blob = blob.copy(signedBy = "aabbccdd")
val bytes = serializeCanonicalProfile(blob)
val parsed = parseCanonicalProfile(bytes)
assertEquals(blob, parsed)
}
// ─── Approval signing payload ──────────────────────────────
@Test
fun canonicalApprovalSigningBytesIsDeterministic() {
val a = canonicalApprovalSigningBytes(
domain = DEFAULT_APPROVAL_DOMAIN,
requestId = "aabbccddeeff00112233445566778899",
hostFingerprint = "11111 22222 33333 44444",
requestingDeviceFingerprint = "55555 66666 77777 88888",
decision = "approve",
)
val b = canonicalApprovalSigningBytes(
domain = DEFAULT_APPROVAL_DOMAIN,
requestId = "aabbccddeeff00112233445566778899",
hostFingerprint = "11111 22222 33333 44444",
requestingDeviceFingerprint = "55555 66666 77777 88888",
decision = "approve",
)
assertArrayEquals(a, b)
}
@Test
fun differentDecisionProducesDifferentSigningBytes() {
val approve = canonicalApprovalSigningBytes(
DEFAULT_APPROVAL_DOMAIN, "r", "h", "d", "approve",
)
val reject = canonicalApprovalSigningBytes(
DEFAULT_APPROVAL_DOMAIN, "r", "h", "d", "reject",
)
assertFalse(approve.contentEquals(reject))
}
@Test
fun differentDomainProducesDifferentSigningBytes() {
val a = canonicalApprovalSigningBytes("shade-link-approve-v1", "r", "h", "d", "approve")
val b = canonicalApprovalSigningBytes("prism-link-approve-v1", "r", "h", "d", "approve")
assertFalse(a.contentEquals(b))
}
// ─── Build / sign / verify ─────────────────────────────────
private data class Scenario(
val phone: ProfileClientEntry,
val phoneSeed: ByteArray,
val profile: CanonicalProfileBlob,
val request: ApprovalRequestFrame,
)
private fun buildScenario(): Scenario {
val (phone, seed) = makeClient("phone", trusted = true)
var profile = emptyCanonicalProfile(0)
profile = upsertHost(profile, makeHost())
profile = upsertClient(profile, phone)
val request = buildApprovalRequest(
crypto = crypto,
hostAddress = "device:host-server",
hostFingerprint = "host-fp-12345",
requestingDeviceFingerprint = "cafe-laptop-fp-67890",
deviceName = "cafe-laptop",
userAgent = "Mozilla/5.0",
ipHint = "203.0.113.7",
)
return Scenario(phone, seed, profile, request)
}
@Test
fun happyPathApproveVerifies() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request,
decision = "approve",
approverFingerprint = s.phone.identityFingerprint,
approverSigningKey = s.phoneSeed,
)
assertEquals("linkApproveByProxy", approval.kind)
assertEquals(s.request.requestId, approval.requestId)
assertEquals(128, approval.signature.length)
val r = verifyProxyApproval(crypto, s.request, approval, s.profile)
assertTrue(r is VerifyProxyApprovalResult.Ok)
assertEquals(s.phone.address, (r as VerifyProxyApprovalResult.Ok).approver.address)
}
@Test
fun happyPathRejectVerifies() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "reject",
s.phone.identityFingerprint, s.phoneSeed,
)
val r = verifyProxyApproval(crypto, s.request, approval, s.profile)
assertTrue(r is VerifyProxyApprovalResult.Ok)
}
@Test
fun replayAgainstDifferentRequestFails() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
val other = s.request.copy(requestId = "f".repeat(32))
val r = verifyProxyApproval(crypto, other, approval, s.profile)
assertEquals(
VerifyProxyApprovalResult.Reason.REQUEST_ID_MISMATCH,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun decisionTamperingFails() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
val tampered = approval.copy(decision = "reject")
val r = verifyProxyApproval(crypto, s.request, tampered, s.profile)
assertEquals(
VerifyProxyApprovalResult.Reason.BAD_SIGNATURE,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun hostFingerprintSwapFails() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
val swapped = s.request.copy(hostFingerprint = "evil-host-fp")
val r = verifyProxyApproval(crypto, swapped, approval, s.profile)
assertEquals(
VerifyProxyApprovalResult.Reason.BAD_SIGNATURE,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun domainMismatchIsRejectedBeforeSignature() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
val r = verifyProxyApproval(
crypto, s.request, approval.copy(domain = "prism-link-approve-v1"), s.profile,
)
assertEquals(
VerifyProxyApprovalResult.Reason.DOMAIN_MISMATCH,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun unknownApproverFails() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
val lying = approval.copy(approverFingerprint = "no-such-fingerprint")
val r = verifyProxyApproval(crypto, s.request, lying, s.profile)
assertEquals(
VerifyProxyApprovalResult.Reason.UNKNOWN_APPROVER,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun revokedApproverFailsWithNotTrusted() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
val revoked = setTrustedApprover(s.profile, s.phone.identityFingerprint, false)
val r = verifyProxyApproval(crypto, s.request, approval, revoked)
assertEquals(
VerifyProxyApprovalResult.Reason.NOT_TRUSTED,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun expiredRequestIsRejected() {
val s = buildScenario()
val approval = signProxyApproval(
crypto, s.request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
val r = verifyProxyApproval(crypto, s.request, approval, s.profile, now = s.request.expiresAt + 1)
assertEquals(
VerifyProxyApprovalResult.Reason.EXPIRED,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun signatureWithWrongKeyFails() {
val s = buildScenario()
val wrongSeed = crypto.randomBytes(32)
val approval = signProxyApproval(
crypto, s.request, "approve",
approverFingerprint = s.phone.identityFingerprint, // claim phone
approverSigningKey = wrongSeed, // sign with different key
)
val r = verifyProxyApproval(crypto, s.request, approval, s.profile)
assertEquals(
VerifyProxyApprovalResult.Reason.BAD_SIGNATURE,
(r as VerifyProxyApprovalResult.Failed).reason,
)
}
@Test
fun customDomainSurvivesRoundTrip() {
val s = buildScenario()
val request = s.request.copy(domain = "prism-link-approve-v1")
val approval = signProxyApproval(
crypto, request, "approve",
s.phone.identityFingerprint, s.phoneSeed,
)
assertEquals("prism-link-approve-v1", approval.domain)
val r = verifyProxyApproval(crypto, request, approval, s.profile)
assertTrue(r is VerifyProxyApprovalResult.Ok)
}
@Test
fun requestIdIs32LowercaseHexChars() {
val r = buildApprovalRequest(
crypto, "device:h", "h", "r",
)
assertTrue(r.requestId.matches(Regex("^[0-9a-f]{32}$")))
}
@Test
fun consecutiveBuildsProduceDistinctRequestIds() {
val a = buildApprovalRequest(crypto, "device:h", "h", "r")
val b = buildApprovalRequest(crypto, "device:h", "h", "r")
assertNotEquals(a.requestId, b.requestId)
}
// ─── Sanity glue: TS-side reference frame parses on Kotlin ──
@Test
fun tsStyleProxyApprovalFrameParsesAndStructurallyMatches() {
// Construct a frame the way @shade/sdk would emit it via JSON,
// and check our Kotlin types accept the same field names.
val expected = ProxyApprovalFrame(
requestId = "00112233445566778899aabbccddeeff",
decision = "approve",
approverFingerprint = "fp",
signature = "ab".repeat(64),
domain = DEFAULT_APPROVAL_DOMAIN,
)
assertEquals("linkApproveByProxy", expected.kind)
assertEquals("approve", expected.decision)
val req = ApprovalRequestFrame(
requestId = "00112233445566778899aabbccddeeff",
hostAddress = "device:h",
hostFingerprint = "host-fp",
requestingDevice = ApprovalRequestingDevice(
fingerprint = "req-fp",
receivedAt = 1L,
),
expiresAt = 2L,
domain = DEFAULT_APPROVAL_DOMAIN,
)
assertNotNull(req)
}
}

View File

@@ -1,6 +1,13 @@
package no.zyon.shade package no.zyon.shade
import no.zyon.shade.approval.canonicalApprovalSigningBytes
import no.zyon.shade.backup.deriveBackupKey import no.zyon.shade.backup.deriveBackupKey
import no.zyon.shade.blob.aeadOpen
import no.zyon.shade.blob.blobAadForSlot
import no.zyon.shade.blob.deriveBlobKey
import no.zyon.shade.blob.deriveBlobSigningSeed
import no.zyon.shade.blob.deriveBlobSlotId
import no.zyon.shade.blob.ed25519PublicKeyFromSeed
import no.zyon.shade.crypto.TinkProvider import no.zyon.shade.crypto.TinkProvider
import no.zyon.shade.fingerprint.computeFingerprint import no.zyon.shade.fingerprint.computeFingerprint
import no.zyon.shade.group.encodeSenderHeader import no.zyon.shade.group.encodeSenderHeader
@@ -447,6 +454,92 @@ class CrossPlatformVectorTest {
} }
} }
@Test
fun blobKdfAndAeadVectorsMatch() {
val vectors = loadVectors("blob.json")
var kdfMatched = 0
var aeadMatched = 0
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
val desc = v.getString("description")
if (desc.startsWith("V4.9 blob KDF")) {
kdfMatched++
val masterKey = fromHex(v.getString("masterKey"))
val app = v.getString("app")
val slotId = deriveBlobSlotId(crypto, masterKey, app)
assertEquals(v.getString("slotId"), hex(slotId))
assertEquals(v.getString("blobKey"), hex(deriveBlobKey(crypto, masterKey, app)))
val seed = deriveBlobSigningSeed(crypto, masterKey, app)
assertEquals(v.getString("signingSeed"), hex(seed))
assertEquals(v.getString("ownerPubkey"), hex(ed25519PublicKeyFromSeed(seed)))
} else if (desc.startsWith("V4.9 blob AEAD")) {
aeadMatched++
val key = fromHex(v.getString("key"))
val slotIdHex = v.getString("slotIdHex")
val expectedPlaintext = fromHex(v.getString("plaintext"))
val wire = fromHex(v.getString("wire"))
val aad = blobAadForSlot(slotIdHex)
val opened = aeadOpen(key, wire, aad)
assertEquals(hex(expectedPlaintext), hex(opened))
}
}
assertTrue("KDF vectors expected", kdfMatched >= 3)
assertTrue("AEAD vectors expected", aeadMatched >= 2)
}
@Test
fun approvalSigningPayloadVectorsMatch() {
val vectors = loadVectors("approval.json")
var payloadMatched = 0
var e2eMatched = 0
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
val desc = v.getString("description")
if (desc.startsWith("V4.10 approval signing payload")) {
payloadMatched++
val out = canonicalApprovalSigningBytes(
domain = v.getString("domain"),
requestId = v.getString("requestId"),
hostFingerprint = v.getString("hostFingerprint"),
requestingDeviceFingerprint = v.getString("requestingDeviceFingerprint"),
decision = v.getString("decision"),
)
assertEquals(v.getString("signingPayload"), hex(out))
} else if (desc.startsWith("V4.10 approval Ed25519 sign/verify")) {
e2eMatched++
val seed = fromHex(v.getString("seed"))
val pubkey = fromHex(v.getString("publicKey"))
assertEquals(hex(pubkey), hex(ed25519PublicKeyFromSeed(seed)))
val req = v.getJSONObject("request")
val payload = canonicalApprovalSigningBytes(
domain = req.getString("domain"),
requestId = req.getString("requestId"),
hostFingerprint = req.getString("hostFingerprint"),
requestingDeviceFingerprint = req.getString("requestingDeviceFingerprint"),
decision = req.getString("decision"),
)
assertEquals(v.getString("signingPayload"), hex(payload))
// Verify the TS-generated signature against our pubkey + payload.
// This is the load-bearing parity check: a Kotlin-implemented
// verifyProxyApproval running against a TS-signed approval
// succeeds.
val sig = fromHex(v.getString("signature"))
val ok = crypto.verify(pubkey, payload, sig)
assertTrue("Ed25519 verify of TS-signed approval failed", ok)
// And: Kotlin signs the same payload with the same seed and
// produces a sig the TS pubkey verifies. Ed25519 is
// deterministic, so the sig bytes also match exactly.
val mySig = crypto.sign(seed, payload)
assertEquals(v.getString("signature"), hex(mySig))
}
}
assertTrue("payload vectors expected", payloadMatched >= 3)
assertTrue("e2e sign/verify vector expected", e2eMatched >= 1)
}
@Test @Test
fun ratchetStepRoundtripMatches() { fun ratchetStepRoundtripMatches() {
val vectors = loadVectors("ratchet-step.json") val vectors = loadVectors("ratchet-step.json")

View File

@@ -0,0 +1,123 @@
package no.zyon.shade
import no.zyon.shade.serialization.SessionStateJson
import no.zyon.shade.types.ChainState
import no.zyon.shade.types.IdentityKeyPair
import no.zyon.shade.types.KeyPair
import no.zyon.shade.types.OneTimePreKey
import no.zyon.shade.types.SessionState
import no.zyon.shade.types.SignedPreKey
import org.junit.Assert.assertArrayEquals
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNotNull
import org.junit.Assert.assertNull
import org.junit.Test
/**
* Round-trip tests for the at-rest JSON serialization used by
* `KeystoreStorage`. The format isn't cross-platform (TS uses its
* own shape) — what matters is `serialize → deserialize` preserves
* every byte of every key.
*/
class SessionStateJsonTest {
private fun bytes(n: Int, fill: Byte): ByteArray = ByteArray(n) { fill }
@Test
fun identityKeyPairRoundTrip() {
val k = IdentityKeyPair(
signingPublicKey = bytes(32, 0x11),
signingPrivateKey = bytes(32, 0x22),
dhPublicKey = bytes(32, 0x33),
dhPrivateKey = bytes(32, 0x44),
)
val s = SessionStateJson.serializeIdentityKeyPair(k)
val d = SessionStateJson.deserializeIdentityKeyPair(s)
assertArrayEquals(k.signingPublicKey, d.signingPublicKey)
assertArrayEquals(k.signingPrivateKey, d.signingPrivateKey)
assertArrayEquals(k.dhPublicKey, d.dhPublicKey)
assertArrayEquals(k.dhPrivateKey, d.dhPrivateKey)
}
@Test
fun signedPreKeyRoundTrip() {
val k = SignedPreKey(
keyId = 42,
keyPair = KeyPair(publicKey = bytes(32, 0x55), privateKey = bytes(32, 0x66)),
signature = bytes(64, 0x77),
timestamp = 1_700_000_000_000L,
)
val s = SessionStateJson.serializeSignedPreKey(k)
val d = SessionStateJson.deserializeSignedPreKey(s)
assertEquals(k.keyId, d.keyId)
assertArrayEquals(k.keyPair.publicKey, d.keyPair.publicKey)
assertArrayEquals(k.keyPair.privateKey, d.keyPair.privateKey)
assertArrayEquals(k.signature, d.signature)
assertEquals(k.timestamp, d.timestamp)
}
@Test
fun oneTimePreKeyRoundTrip() {
val k = OneTimePreKey(
keyId = 7,
keyPair = KeyPair(publicKey = bytes(32, 0x88.toByte()), privateKey = bytes(32, 0x99.toByte())),
)
val s = SessionStateJson.serializeOneTimePreKey(k)
val d = SessionStateJson.deserializeOneTimePreKey(s)
assertEquals(k.keyId, d.keyId)
assertArrayEquals(k.keyPair.publicKey, d.keyPair.publicKey)
assertArrayEquals(k.keyPair.privateKey, d.keyPair.privateKey)
}
@Test
fun sessionStateRoundTripFullPopulated() {
val state = SessionState(
remoteIdentityKey = bytes(32, 0x01),
rootKey = bytes(32, 0x02),
sendChain = ChainState(chainKey = bytes(32, 0x03), counter = 5),
receiveChain = ChainState(chainKey = bytes(32, 0x04), counter = 3),
dhSend = KeyPair(publicKey = bytes(32, 0x05), privateKey = bytes(32, 0x06)),
dhReceive = bytes(32, 0x07),
previousSendCounter = 9,
skippedKeys = mutableMapOf(
"remote:1" to bytes(32, 0x0A),
"remote:2" to bytes(32, 0x0B),
),
)
val s = SessionStateJson.serialize(state)
val d = SessionStateJson.deserialize(s)
assertArrayEquals(state.remoteIdentityKey, d.remoteIdentityKey)
assertArrayEquals(state.rootKey, d.rootKey)
assertArrayEquals(state.sendChain.chainKey, d.sendChain.chainKey)
assertEquals(state.sendChain.counter, d.sendChain.counter)
assertNotNull(d.receiveChain)
assertArrayEquals(state.receiveChain!!.chainKey, d.receiveChain!!.chainKey)
assertArrayEquals(state.dhSend.publicKey, d.dhSend.publicKey)
assertArrayEquals(state.dhSend.privateKey, d.dhSend.privateKey)
assertArrayEquals(state.dhReceive, d.dhReceive)
assertEquals(state.previousSendCounter, d.previousSendCounter)
assertEquals(state.skippedKeys.size, d.skippedKeys.size)
for ((k, v) in state.skippedKeys) {
assertArrayEquals(v, d.skippedKeys[k])
}
}
@Test
fun sessionStateRoundTripWithNullableFields() {
val state = SessionState(
remoteIdentityKey = bytes(32, 0x01),
rootKey = bytes(32, 0x02),
sendChain = ChainState(chainKey = bytes(32, 0x03), counter = 0),
receiveChain = null,
dhSend = KeyPair(publicKey = bytes(32, 0x05), privateKey = bytes(32, 0x06)),
dhReceive = null,
previousSendCounter = 0,
skippedKeys = mutableMapOf(),
)
val s = SessionStateJson.serialize(state)
val d = SessionStateJson.deserialize(s)
assertNull(d.receiveChain)
assertNull(d.dhReceive)
assertEquals(0, d.skippedKeys.size)
}
}

View File

@@ -17,7 +17,7 @@
}, },
"packages/shade-cli": { "packages/shade-cli": {
"name": "@shade/cli", "name": "@shade/cli",
"version": "0.4.0", "version": "4.8.5",
"bin": { "bin": {
"shade": "src/cli.ts", "shade": "src/cli.ts",
}, },
@@ -36,7 +36,7 @@
}, },
"packages/shade-core": { "packages/shade-core": {
"name": "@shade/core", "name": "@shade/core",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/observability": "workspace:*", "@shade/observability": "workspace:*",
}, },
@@ -49,7 +49,7 @@
}, },
"packages/shade-crypto-web": { "packages/shade-crypto-web": {
"name": "@shade/crypto-web", "name": "@shade/crypto-web",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@noble/curves": "^2.0.1", "@noble/curves": "^2.0.1",
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
@@ -59,7 +59,7 @@
}, },
"packages/shade-dashboard": { "packages/shade-dashboard": {
"name": "@shade/dashboard", "name": "@shade/dashboard",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/widgets": "workspace:*", "@shade/widgets": "workspace:*",
"react": "^19.0.0", "react": "^19.0.0",
@@ -74,7 +74,7 @@
}, },
"packages/shade-files": { "packages/shade-files": {
"name": "@shade/files", "name": "@shade/files",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -101,7 +101,7 @@
}, },
"packages/shade-inbox": { "packages/shade-inbox": {
"name": "@shade/inbox", "name": "@shade/inbox",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/proto": "workspace:*", "@shade/proto": "workspace:*",
@@ -114,7 +114,7 @@
}, },
"packages/shade-inbox-server": { "packages/shade-inbox-server": {
"name": "@shade/inbox-server", "name": "@shade/inbox-server",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/observability": "workspace:*", "@shade/observability": "workspace:*",
@@ -132,7 +132,7 @@
}, },
"packages/shade-key-transparency": { "packages/shade-key-transparency": {
"name": "@shade/key-transparency", "name": "@shade/key-transparency",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
@@ -144,11 +144,11 @@
}, },
"packages/shade-keychain": { "packages/shade-keychain": {
"name": "@shade/keychain", "name": "@shade/keychain",
"version": "0.4.0", "version": "4.8.5",
}, },
"packages/shade-observability": { "packages/shade-observability": {
"name": "@shade/observability", "name": "@shade/observability",
"version": "0.1.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
}, },
@@ -166,7 +166,7 @@
}, },
"packages/shade-observer": { "packages/shade-observer": {
"name": "@shade/observer", "name": "@shade/observer",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/server": "workspace:*", "@shade/server": "workspace:*",
@@ -178,14 +178,14 @@
}, },
"packages/shade-proto": { "packages/shade-proto": {
"name": "@shade/proto", "name": "@shade/proto",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
}, },
}, },
"packages/shade-recovery": { "packages/shade-recovery": {
"name": "@shade/recovery", "name": "@shade/recovery",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -198,22 +198,25 @@
}, },
"packages/shade-sdk": { "packages/shade-sdk": {
"name": "@shade/sdk", "name": "@shade/sdk",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
"@shade/files": "workspace:*", "@shade/files": "workspace:*",
"@shade/inbox": "workspace:*",
"@shade/key-transparency": "workspace:*", "@shade/key-transparency": "workspace:*",
"@shade/observability": "workspace:*", "@shade/observability": "workspace:*",
"@shade/observer": "workspace:*", "@shade/observer": "workspace:*",
"@shade/proto": "workspace:*", "@shade/proto": "workspace:*",
"@shade/server": "workspace:*", "@shade/server": "workspace:*",
"@shade/storage-encrypted": "workspace:*",
"@shade/storage-sqlite": "workspace:*", "@shade/storage-sqlite": "workspace:*",
"@shade/streams": "workspace:*", "@shade/streams": "workspace:*",
"@shade/transfer": "workspace:*", "@shade/transfer": "workspace:*",
"@shade/transport": "workspace:*", "@shade/transport": "workspace:*",
}, },
"devDependencies": { "devDependencies": {
"@shade/inbox-server": "workspace:*",
"@shade/transport-webrtc": "workspace:*", "@shade/transport-webrtc": "workspace:*",
}, },
"peerDependencies": { "peerDependencies": {
@@ -225,7 +228,7 @@
}, },
"packages/shade-server": { "packages/shade-server": {
"name": "@shade/server", "name": "@shade/server",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/inbox-server": "workspace:*", "@shade/inbox-server": "workspace:*",
@@ -245,15 +248,19 @@
}, },
"packages/shade-storage-encrypted": { "packages/shade-storage-encrypted": {
"name": "@shade/storage-encrypted", "name": "@shade/storage-encrypted",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
"@shade/storage-postgres": "workspace:*", "@shade/storage-postgres": "workspace:*",
"@shade/storage-sqlite": "workspace:*", "@shade/storage-sqlite": "workspace:*",
"idb": "^8.0.3",
"postgres": "^3.4.9", "postgres": "^3.4.9",
}, },
"devDependencies": {
"fake-indexeddb": "^6.0.0",
},
"peerDependencies": { "peerDependencies": {
"@shade/keychain": "workspace:*", "@shade/keychain": "workspace:*",
}, },
@@ -261,9 +268,21 @@
"@shade/keychain", "@shade/keychain",
], ],
}, },
"packages/shade-storage-indexeddb": {
"name": "@shade/storage-indexeddb",
"version": "4.8.5",
"dependencies": {
"@shade/core": "workspace:*",
"idb": "^8.0.3",
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
"fake-indexeddb": "^6.0.0",
},
},
"packages/shade-storage-postgres": { "packages/shade-storage-postgres": {
"name": "@shade/storage-postgres", "name": "@shade/storage-postgres",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/inbox-server": "workspace:*", "@shade/inbox-server": "workspace:*",
@@ -278,7 +297,7 @@
}, },
"packages/shade-storage-sqlite": { "packages/shade-storage-sqlite": {
"name": "@shade/storage-sqlite", "name": "@shade/storage-sqlite",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -288,7 +307,7 @@
}, },
"packages/shade-streams": { "packages/shade-streams": {
"name": "@shade/streams", "name": "@shade/streams",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
@@ -300,7 +319,7 @@
}, },
"packages/shade-transfer": { "packages/shade-transfer": {
"name": "@shade/transfer", "name": "@shade/transfer",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -317,7 +336,7 @@
}, },
"packages/shade-transport": { "packages/shade-transport": {
"name": "@shade/transport", "name": "@shade/transport",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -328,7 +347,7 @@
}, },
"packages/shade-transport-bridge": { "packages/shade-transport-bridge": {
"name": "@shade/transport-bridge", "name": "@shade/transport-bridge",
"version": "0.1.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/server": "workspace:*", "@shade/server": "workspace:*",
@@ -350,7 +369,7 @@
}, },
"packages/shade-transport-webrtc": { "packages/shade-transport-webrtc": {
"name": "@shade/transport-webrtc", "name": "@shade/transport-webrtc",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/streams": "workspace:*", "@shade/streams": "workspace:*",
@@ -359,7 +378,7 @@
}, },
"packages/shade-widgets": { "packages/shade-widgets": {
"name": "@shade/widgets", "name": "@shade/widgets",
"version": "0.4.0", "version": "4.8.5",
"dependencies": { "dependencies": {
"@shade/recovery": "workspace:*", "@shade/recovery": "workspace:*",
"@shade/sdk": "workspace:*", "@shade/sdk": "workspace:*",
@@ -568,6 +587,8 @@
"@shade/storage-encrypted": ["@shade/storage-encrypted@workspace:packages/shade-storage-encrypted"], "@shade/storage-encrypted": ["@shade/storage-encrypted@workspace:packages/shade-storage-encrypted"],
"@shade/storage-indexeddb": ["@shade/storage-indexeddb@workspace:packages/shade-storage-indexeddb"],
"@shade/storage-postgres": ["@shade/storage-postgres@workspace:packages/shade-storage-postgres"], "@shade/storage-postgres": ["@shade/storage-postgres@workspace:packages/shade-storage-postgres"],
"@shade/storage-sqlite": ["@shade/storage-sqlite@workspace:packages/shade-storage-sqlite"], "@shade/storage-sqlite": ["@shade/storage-sqlite@workspace:packages/shade-storage-sqlite"],
@@ -626,6 +647,8 @@
"escalade": ["escalade@3.2.0", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="], "escalade": ["escalade@3.2.0", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="],
"fake-indexeddb": ["fake-indexeddb@6.2.5", "", {}, "sha512-CGnyrvbhPlWYMngksqrSSUT1BAVP49dZocrHuK0SvtR0D5TMs5wP0o3j7jexDJW01KSadjBp1M/71o/KR3nD1w=="],
"fast-check": ["fast-check@3.23.2", "", { "dependencies": { "pure-rand": "^6.1.0" } }, "sha512-h5+1OzzfCC3Ef7VbtKdcv7zsstUQwUDlYpUTvjeUsJAssPgLn7QzbboPtL5ro04Mq0rPOsMzl7q5hIbRs2wD1A=="], "fast-check": ["fast-check@3.23.2", "", { "dependencies": { "pure-rand": "^6.1.0" } }, "sha512-h5+1OzzfCC3Ef7VbtKdcv7zsstUQwUDlYpUTvjeUsJAssPgLn7QzbboPtL5ro04Mq0rPOsMzl7q5hIbRs2wD1A=="],
"fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="], "fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="],
@@ -638,6 +661,8 @@
"hono": ["hono@4.12.12", "", {}, "sha512-p1JfQMKaceuCbpJKAPKVqyqviZdS0eUxH9v82oWo1kb9xjQ5wA6iP3FNVAPDFlz5/p7d45lO+BpSk1tuSZMF4Q=="], "hono": ["hono@4.12.12", "", {}, "sha512-p1JfQMKaceuCbpJKAPKVqyqviZdS0eUxH9v82oWo1kb9xjQ5wA6iP3FNVAPDFlz5/p7d45lO+BpSk1tuSZMF4Q=="],
"idb": ["idb@8.0.3", "", {}, "sha512-LtwtVyVYO5BqRvcsKuB2iUMnHwPVByPCXFXOpuU96IZPPoPN6xjOGxZQ74pgSVVLQWtUOYgyeL4GE98BY5D3wg=="],
"js-tokens": ["js-tokens@4.0.0", "", {}, "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ=="], "js-tokens": ["js-tokens@4.0.0", "", {}, "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ=="],
"jsesc": ["jsesc@3.1.0", "", { "bin": { "jsesc": "bin/jsesc" } }, "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA=="], "jsesc": ["jsesc@3.1.0", "", { "bin": { "jsesc": "bin/jsesc" } }, "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA=="],

View File

@@ -81,6 +81,7 @@ Tables will be created automatically with the `shade_server_*` prefix, so they c
| `SHADE_CLEANUP_INTERVAL_HOURS` | `24` | Cleanup cycle interval | | `SHADE_CLEANUP_INTERVAL_HOURS` | `24` | Cleanup cycle interval |
| `SHADE_LOG_LEVEL` | `info` | `debug` / `info` / `warn` / `error` | | `SHADE_LOG_LEVEL` | `info` | `debug` / `info` / `warn` / `error` |
| `SHADE_OTEL_ENABLED` | unset | Set to `1`/`true` to enable OpenTelemetry tracing on `withTracer()`-configured deployments. See [`observability.md`](./observability.md). | | `SHADE_OTEL_ENABLED` | unset | Set to `1`/`true` to enable OpenTelemetry tracing on `withTracer()`-configured deployments. See [`observability.md`](./observability.md). |
| `SHADE_DISABLE_RATE_LIMIT` | unset | Set to `1` to disable IP rate-limits on every prekey + inbox route. **Single-tenant deployments only** — multi-tenant relays must leave this unset to keep the abuse defenses on. |
## Health and observability ## Health and observability

View File

@@ -1,8 +1,11 @@
# Shade V2.1 — Improvements (infrastructure, storage, operations, security) # Shade V2.1 — Improvements (infrastructure, storage, operations, security)
This document describes **improvements** agreed for next-generation work on Shade: clearer product story, stronger storage, mobile parity, operational hardening, transfer abuse, and a formal security narrative. **Status:** Done — superseded by the V3.1 → V3.12 plans, all of which
landed in the 4.0 GA release. This document is preserved as historical
context for the original V2.1 backlog; the concrete deliverables live
under [`docs/archive/V3.*.md`](./).
**Audience:** **Maintainers and contributors** implementing the changes. Add status fields as items land in code/docs. This document describes **improvements** agreed for next-generation work on Shade: clearer product story, stronger storage, mobile parity, operational hardening, transfer abuse, and a formal security narrative.
--- ---

View File

@@ -1,8 +1,10 @@
# Shade V2.2 — Feature plan: product, platform, and developer experience # Shade V2.2 — Feature plan: product, platform, and developer experience
This document gathers **planned features** that extend Shade beyond todays core (X3DH + Double Ratchet + Streams/transfer): groups, asynchronous delivery, richer file UX, web workers, CLI, API docs, and scaffolding. **Status:** Done — superseded by V3.6 (inbox), V3.7 (bridge), V3.8
(workers), V3.9 (file metadata), V3.10 (recovery), V3.12 (KT). All
landed in the 4.0 GA release; see [`docs/ROADMAP.md`](../ROADMAP.md).
Add optional per-feature status (Idea / Design / IMP / Done). This document gathers **planned features** that extend Shade beyond todays core (X3DH + Double Ratchet + Streams/transfer): groups, asynchronous delivery, richer file UX, web workers, CLI, API docs, and scaffolding.
--- ---

View File

@@ -1,5 +1,9 @@
# Shade V2.3 — Tillit, retention, integrasjon og observability # Shade V2.3 — Tillit, retention, integrasjon og observability
**Status:** Done — superseded by V3.3 (trust UX), V3.4 (observability),
V3.10 (recovery), V3.12 (KT). Alt levert i 4.0 GA — se
[`docs/ROADMAP.md`](../ROADMAP.md).
Dette dokumentet beskriver **høyere ambisjonsnivå** og **plattformkryssende** arbeid: brukertilfeller der tillit må være **eksplisitt**, data må **utkrympes**/ryddes automatisk, apper må **enkelt koble seg på** kjente transportmønstre, og drift må **observere** uten å lekke innhold. Dette dokumentet beskriver **høyere ambisjonsnivå** og **plattformkryssende** arbeid: brukertilfeller der tillit må være **eksplisitt**, data må **utkrympes**/ryddes automatisk, apper må **enkelt koble seg på** kjente transportmønstre, og drift må **observere** uten å lekke innhold.
--- ---

View File

@@ -1,6 +1,6 @@
# V3.12 — Key Transparency: Designnotat # V3.12 — Key Transparency: Designnotat
**Status:** Approved (in-tree review — markeres `Design` i ROADMAP) **Status:** Done — implementert i `@shade/key-transparency` 0.4.0, frosset i 4.0 GA.
**Forfatter:** Shade-teamet **Forfatter:** Shade-teamet
**Reviewer-mål:** ekstern crypto-orientert reviewer før produksjons-deploy. **Reviewer-mål:** ekstern crypto-orientert reviewer før produksjons-deploy.
**Implementasjons-target:** `@shade/key-transparency` + utvidelser i **Implementasjons-target:** `@shade/key-transparency` + utvidelser i

View File

@@ -53,6 +53,111 @@ if (result.kind === 'inline') console.log(result.bytes.byteLength);
else for await (const _chunk of /* result.stream */) { /* ... */ } else for await (const _chunk of /* result.stream */) { /* ... */ }
``` ```
## HTTP RPC — browser-friendly request-response (4.1+)
The default `shade.files.client(peer)` requires both peers to be mutually
addressable over HTTP — the response to a `list`/`read` etc. round-trips
through `Shade.deliverControlEnvelope`, which POSTs to the peer's
`/v1/transfer/control` endpoint. **That doesn't work for browsers**
a tab can't host an HTTP server, so the server cannot call back outbound.
`@shade/files` 4.1 ships a parallel **request-response** transport that
lets browser-style clients fully consume the file-RPC surface without
any inbound channel. It mirrors the way `@shade/server`'s
`shade-auth-middleware` works: one POST per RPC, encrypted envelope in
the request body, encrypted response in the same HTTP response.
### Server side — mount the RPC route
```ts
// 1. Register the file handler. `inlineOnly: true` skips the
// streams-bridge (which would require @shade/transfer).
await shade.files.serve(handlerConfig, { inlineOnly: true });
// 2. Mount the route on your Hono app under any base path.
app.route('/api/v1/shade-files', shade.files.rpcRoute());
// ^^^^^^^^^^^^^^
// POST <base>/rpc
```
`rpcRoute()` accepts:
| Option | Default | Purpose |
|---------------------|---------|----------------------------------------------------------------------------------------------------|
| `maxBodyBytes` | 1 MiB | Max request body. The protocol caps inline payloads at 256 KiB; the headroom is for base64 inflation + custom-op envelopes. |
| `acceptFirstMessage`| `false` | Accept `0x01` PreKeyMessage envelopes — required when the RPC route also doubles as the X3DH handshake (browser's first-ever request). |
### Browser client
```ts
import { createShade } from '@shade/sdk';
const shade = await createShade({
prekeyServer: 'https://shade.example.com',
storage: 'memory',
address: 'alice@example.com',
});
const fs = shade.files.httpClient('bob@example.com', {
rpcUrl: 'https://dispatch.example.com/api/v1/shade-files/rpc',
// Optional: thread CSRF / auth tokens, override fetch, etc.
headers: { 'X-CSRF-Token': csrfToken },
});
await fs.mkdir('/photos');
await fs.write('/photos/cover.png', new Uint8Array([/* ... */]), {
contentType: 'image/png',
});
const result = await fs.read('/photos/cover.png');
```
### What works in HTTP-RPC mode
- `list`, `stat`, `mkdir`, `delete`, `move`, `getThumbnail`, `custom<K>` — full parity.
- `write`**inline only** (≤ 256 KiB plaintext). Larger inputs throw `ConflictError`.
- `read`**inline only**. If the server returns a streamed `read` result, the client throws `InternalFileError` directing callers to the stateful pathway.
### Wire contract
```
POST /rpc HTTP/1.1
Content-Type: application/octet-stream
X-Shade-Sender-Address: alice@example.com
<wire-encoded ShadeEnvelope (0x01 first-time, 0x02 after) wrapping
JSON-encoded RpcRequest>
────
200 OK
Content-Type: application/octet-stream
<wire-encoded ShadeEnvelope (0x02) wrapping JSON-encoded RpcResponse | RpcError>
```
Transport-level failures (no session, undecryptable envelope, body too
big) return JSON `{ "error": "..." }` with appropriate 4xx status.
Application-level failures (file not found, permission denied) ship
encrypted `RpcError` envelopes — the client maps them back to typed
`FileError` subclasses (`NotFoundError`, `ConflictError`, etc.).
### Symmetry with `@shade/server`
The shape mirrors `@shade/server`'s shade-auth-middleware: encrypted
envelope rides the request body, server decrypts via the existing
ratchet session, performs the protected operation, returns an encrypted
envelope in the response. No bidirectional channel required, no
WebSocket, no SSE.
### When to use which
| Setup | Use |
|-----------------------------------------------|-----------------------------------------------|
| Browser client ↔ Bun/Hono server | `httpClient()` + `rpcRoute()` |
| Server ↔ server (both can host HTTP) | `client()` (default) — supports streams |
| Service-worker / extension ↔ server | `httpClient()` (no inbound listener) |
| CLI / daemon ↔ daemon | Either; `client()` if you need streams |
## Op surface ## Op surface
| Op | Args | Result | | Op | Args | Result |

View File

@@ -0,0 +1,173 @@
# Feature Request — Public accessor for the device's identity public key
**To**: Shade SDK team
**From**: Dispatch (browser-based Shade consumer)
**Target**: Shade SDK 4.4.x (or whichever release vehicle fits)
**Priority**: medium — unblocks real per-device fingerprint binding at
enrollment time; consumers ship with placeholder keys until then
Thanks for shipping `@shade/storage-indexeddb` so quickly — that unblocked
Dispatch's Slice 2.5 (persistent browser identity). During integration we
hit one more gap that's worth raising as a separate FR.
---
## Summary
Expose the local device's Ed25519 identity public key as a public accessor
on `Shade`, so applications can hand it to their own backend at enrollment
time for per-device verification, audit, or peer-fingerprint computation.
Today the SDK exposes `myAddress` and `fingerprint`, but the underlying
identity public key — the cryptographic root that everything else binds
to — is reachable only via the private `this.storage.getIdentityKeyPair()`
call inside `Shade`. Consumers building enrollment flows have no way to
hand the real key over.
## Problem
A common pattern in Shade-using apps is:
1. Browser device generates Shade identity (via `createShade`)
2. User enters an enrollment token
3. Browser POSTs to its backend: `{ token, address, identityPublicKey, ... }`
4. Backend records the device, computes/stores a safety number from the
identity key, opens a Shade session
Step 3 today has nowhere to get a real `identityPublicKey` from. Dispatch
currently sends `crypto.getRandomValues(new Uint8Array(32))` formatted as
hex, with this comment in the source:
```ts
/**
* The browser submits a hex public-key field at enroll time so the schema
* stays stable. Wiring this to the SDK-generated identity key requires a
* Shade SDK addition (no public accessor for the raw identity key today).
*/
export function generatePlaceholderIdentityPublicKey(): string { ... }
```
The backend stores this placeholder but cannot verify anything against it,
because it's not actually the device's key. Real cryptographic binding is
deferred until the SDK exposes the underlying key.
## Why `fingerprint` isn't sufficient
`Shade.fingerprint` returns a 12-groups-of-5-digits safety-number string
designed for human side-channel comparison. That's the right output for a
"compare these digits with your friend" UX, but it's a derived format, not
the raw key. Backends that want to:
- Store the key for later signature verification
- Compute their own safety-number representation (Dispatch uses
`deterministicSafetyNumber(localAddr, peerAddr)` based on the raw bytes)
- Re-derive the fingerprint after an identity rotation
…all need access to the raw 32-byte Ed25519 public key.
## Proposed API
Add a single async accessor on `Shade`:
```ts
class Shade {
/**
* The local device's Ed25519 identity public key (32 bytes).
*
* Stable for the lifetime of the identity. After `rotateIdentity()`
* this returns the new key; the old key is preserved in retired-
* identities storage for the configured grace period.
*/
get identityPublicKey(): Promise<Uint8Array>;
}
```
Internally:
```ts
get identityPublicKey(): Promise<Uint8Array> {
if (!this.initialized) throw new Error('Not initialized');
return this.storage.getIdentityKeyPair().then((kp) => {
if (!kp) throw new Error('Identity not yet generated');
return kp.publicKey;
});
}
```
If returning a getter that produces a Promise feels off, the equivalent
method form is fine:
```ts
async getIdentityPublicKey(): Promise<Uint8Array> { ... }
```
Either shape works for consumers — pick whichever matches existing SDK
conventions.
## Alternative considered: object accessor
Returning an object would leave room to expose other identity-related
fields later without a breaking change:
```ts
get identity(): Promise<{
address: string;
publicKey: Uint8Array;
fingerprint: string;
// future: registrationId, createdAt, etc.
}>;
```
This would mildly duplicate `myAddress` + `fingerprint`, but consolidates
identity-related accessors. Not a blocker for shipping the simpler
single-purpose accessor — just flagging the option in case the SDK is
considering broader API ergonomics.
## What this unblocks
For Dispatch specifically:
- Real device identity binding at enrollment (replaces the placeholder
`crypto.randomUUID()`-derived hex bytes the backend currently stores)
- Server-side `computePeerSafetyNumber()` can use the real key instead of
the deterministic-from-address stand-in (`shade-identity-provider.ts:151`)
- Future signature-based device authentication (sign a challenge with the
device's identity key during enrollment) becomes possible without
another SDK round
For other Shade consumers:
- Any app that hands a device key to its own backend for enrollment —
multi-device pairing flows, contact verification UIs, push-notification
targeting — gets an actual key to work with
## Acceptance criteria
1. New accessor exposed on `Shade` (getter or async method, SDK's
preference)
2. Returns the 32-byte Ed25519 identity public key
3. Returns the **current** key after `rotateIdentity()`
4. Throws (or resolves to a clear error) when called before
`initialize()` completes
5. Documented alongside `myAddress` and `fingerprint` in the SDK reference
6. One unit test in `shade-sdk` confirming the returned bytes match what
`storage.getIdentityKeyPair()` holds
7. Mentioned in the changelog under the next release
## Out of scope
- **Private key access** — consumers should never need it; signature
operations go through `shade.send()` etc. Don't expose the secret half.
- **Cross-peer key lookup** — getting *another* peer's identity key is a
separate concern (related to peer-verification storage), not what this
FR is about. This is strictly the local device's own key.
- **Format conversions** — base64/hex/PEM helpers don't belong in the SDK.
Consumers can encode the raw bytes however their wire format requires.
## Why this can't be done in consumer-land
The identity keypair is generated by `MemoryStorage` / `SQLiteStorage` /
`IndexedDBStorage` and consumed by `ShadeSessionManager`. Consumers can't
reach into either layer without breaching the SDK's encapsulation. A
public accessor is the only path that doesn't require monkey-patching
private fields.

View File

@@ -0,0 +1,191 @@
# Feature Request — `@shade/storage-indexeddb`
**To**: Shade SDK team
**From**: Dispatch (browser-based Shade consumer)
**Target**: Shade SDK 4.3.x (or whichever release vehicle fits)
**Priority**: blocks all browser-based Shade apps from achieving session
persistence across tab refresh
---
## Summary
Ship an official IndexedDB-backed `StorageProvider` adapter as a new
workspace package `@shade/storage-indexeddb`, so browser-based Shade SDK
consumers can persist identity, prekeys, sessions, and peer-verification
state across tab refresh and browser restart — the same way `@shade/storage-sqlite`
does for Node and `@shade/storage-postgres` does for server deployments.
## Problem
Today the Shade SDK ships three storage paths:
| spec | adapter | environment |
| --------------------------------- | ------------------------ | ---------------- |
| `"memory"` | `MemoryStorage` (in-SDK) | tests, ephemeral |
| `"sqlite:/path"` | `@shade/storage-sqlite` | Node |
| `{ type: 'postgres', url: '…' }` | `@shade/storage-postgres`| Node servers |
There is **no browser-storage option**. The only way to run Shade in a
browser today is `storage: "memory"`, which means:
- Identity keypair regenerates on every page load
- Sessions reset → re-enrollment after every refresh
- `getLocalRegistrationId()` returns a fresh value → `device:${id}`
address changes → server-side device record orphaned every reload
This forces every browser-based Shade app to either (a) accept the broken
UX, or (b) build their own `StorageProvider` from scratch — duplicating
~25 methods × N consumers, with no shared conformance test surface.
The right place to solve this is at the SDK level, exactly mirroring how
SQLite and Postgres are handled.
## Proposed package
`packages/shade-storage-indexeddb/` — modeled directly after
`packages/shade-storage-sqlite/`. Same package shape, same test layout,
same `@shade/core`-only runtime dependency surface.
### Public API
```ts
// @shade/storage-indexeddb
export class IndexedDBStorage implements StorageProvider {
/**
* Open (or create) the IndexedDB database. Idempotent — repeated calls
* with the same dbName return a connection sharing the same object stores.
*/
static async create(opts?: { dbName?: string }): Promise<IndexedDBStorage>;
/**
* Cleanly close the underlying connection. Future calls will reopen.
* Called by Shade.shutdown() when consumers register cleanup.
*/
async close(): Promise<void>;
// ─── all StorageProvider methods (identity, prekeys, sessions,
// retired identities, peer verifications, optional stream-state) ───
}
```
`dbName` defaults to something like `"shade"`. Consumers like Dispatch
will pass distinct names per app (`"dispatch-dashboard-shade"`,
`"dispatch-host-ui-shade"`) so DevTools' IndexedDB inspector groups them
sensibly, even though origin-isolation already makes the data isolated.
### SDK integration
`@shade/sdk` `resolveStorage()` gets a fourth branch:
```ts
if (typeof spec === 'object' && spec.type === 'indexeddb') {
const moduleId = '@shade/storage-indexeddb';
const mod = (await import(moduleId)) as {
IndexedDBStorage: { create(opts: { dbName?: string }): Promise<StorageProvider> };
};
return mod.IndexedDBStorage.create({ dbName: spec.dbName });
}
```
Dynamic import keeps `@shade/storage-indexeddb` an optional dependency,
matching the Postgres pattern — Node-only consumers don't need to install
a browser-only adapter.
Consumer surface:
```ts
const shade = await createShade({
prekeyServer: 'https://…/shade-prekey',
storage: { type: 'indexeddb', dbName: 'my-app-shade' },
address: 'device:user@example.com', // optional — falls back to device:${registrationId}
});
```
## Implementation guidance (non-prescriptive)
- **IDB wrapper**: suggest `idb` (Jake Archibald's thin wrapper, well-typed,
zero deps). Avoid Dexie or idb-keyval — we want full schema control to
match the SQL adapters' explicit schemas.
- **Object-store layout**: one store per StorageProvider category
(`identity`, `signedPreKeys`, `oneTimePreKeys`, `sessions`,
`trustedIdentities`, `retiredIdentities`, `peerVerifications`,
`streamStates`). Keypaths match the natural keys (`keyId`, `address`,
`streamId`).
- **Schema version**: integer, bumped on every shape change. Migrations
in `db.upgrade(...)` callback. Document schema-history alongside
the SQLite schema.
- **Concurrency**: IndexedDB transactions are auto-committing — the adapter
must keep operations within a single transaction where SQL adapters do.
Particular care for `bumpPeerIdentityVersion` (atomic read-modify-write).
- **Stream-state methods**: implement them. Browser apps will increasingly
use `@shade/transfer` for large file resume, and parity with SQLite's
capabilities matters.
## Test expectations
Mirror `packages/shade-storage-sqlite/tests/`:
- `indexeddb-storage.test.ts` — full StorageProvider surface (identity,
sessions, trusted identities, retired identities)
- `indexeddb-prekey-store.test.ts` — signed + one-time prekey lifecycle
- `peer-verifications.test.ts` — verification CRUD + identity-version
bumping invariants
- `indexeddb-stream-state.test.ts` (if stream-state is implemented)
Use **`fake-indexeddb`** for the Node test environment — it's the
established standard, supports the v3 spec, and lets us run IDB tests
in `bun test` / `vitest` / `jest` without a real browser.
If/when Shade gains a shared `StorageProvider` conformance test suite,
this adapter should consume it directly. Until then, follow the SQLite
adapter's per-method coverage style.
## Acceptance criteria
1. `@shade/storage-indexeddb` published at version 4.3.0 (or whichever
matches the next Shade release)
2. `@shade/sdk` `resolveStorage()` resolves `{ type: 'indexeddb', dbName? }`
via dynamic import
3. Full StorageProvider conformance in tests (identity, prekeys,
sessions, retired identities, peer verifications, stream-state)
4. Documented in Shade docs alongside SQLite/Postgres adapters
5. README example showing browser-app integration
6. Bundle-size note: dynamic-imported IDB module shouldn't pull crypto
dependencies — adapter should be ≤ ~10 KB minified+gzipped
## Out of scope (deferred)
- **Encryption-at-rest** for the IDB contents — separate work item; should
match the deviceKey-AES-GCM pattern Shade already uses for `secretEnc`
in `PersistedStreamState`. This adapter ships unencrypted-at-rest in v1
(consistent with SQLite), with the encryption layer added uniformly to
all adapters later.
- **Cross-tab BroadcastChannel sync** — IDB is shared across same-origin
tabs already; concurrent writes work via IDB transactions. Real-time
notification across tabs (e.g. "session was rotated in another tab") is
a separate concern, not storage-adapter scope.
- **Quota handling** — IDB quota for a Shade keystore is far below realistic
browser quotas. If it ever becomes relevant, add a `QuotaExceededError`
observability hook then.
## Why this can't be done in consumer-land
Building this in Dispatch (or any other consumer) would mean:
- 25+ method `StorageProvider` re-implementation per consumer
- No shared conformance tests
- Schema drift across consumers — each would invent its own object-store
shape, blocking any future cross-app data import/export
- Every Shade SDK update that adjusts `StorageProvider` would force every
consumer to track and patch independently
`@shade/storage-sqlite` and `@shade/storage-postgres` are part of the SDK
for the same reason. IndexedDB belongs alongside them.
## What unblocks
Shipping this unblocks Dispatch's "Slice 2.5" — persistent enrollment
across browser refresh, which today is the largest QA-friction point in
the dev loop. Any future browser-Shade consumer (web dashboard,
contact-list app, browser-extension messenger) gets persistence for free.

128
docs/streaming-sessions.md Normal file
View File

@@ -0,0 +1,128 @@
# Streaming Double-Ratchet sub-sessions (V4.11)
`ShadeStream` wraps individual frames on a long-lived, high-frequency,
often one-directional channel (e.g. a server→client console-log
WebSocket) in an **independent** Double Ratchet derived from — but never
mutating — an already-established parent Shade session.
This is the answer to Vyvern FR `shade-ws-streaming-ratchet.md`. It is a
first-class API, *not* the "documented contract that `send`/`receive` is
safe per-frame" fallback: the Double-Ratchet crypto was already safe for
that access pattern, but the `send`/`receive` wrapper layer was not
(per-frame keystore writes; a shared per-peer mutex and a single stored
session row coupling the stream to the HTTP path). `ShadeStream` keeps
the proven ratchet and fixes the wrapper.
## API
Transport-agnostic, exactly like `send`/`receive`: it emits/consumes
wire bytes; you own the WebSocket.
```ts
// Initiator (the side that calls openStream)
const stream = await shade.openStream(peerAddr);
ws.send(stream.handshakeFrame()); // → STREAM_OPEN
// first inbound WS frame is the peer's STREAM_OPEN_ACK:
await stream.handleHandshake(ackBytes); // stream now usable
ws.send(await stream.seal(utf8(logLine))); // outbound frame
onLog(await stream.open(inboundBytes)); // inbound frame
await stream.close(); // on ws close/error
// Responder
const stream = await shade.acceptStream(peerAddr, openBytes); // usable now
ws.send(stream.handshakeFrame()); // → STREAM_OPEN_ACK
// open()/seal() as above
```
Route inbound bytes with `inspectEnvelopeType()`:
`'stream-open' | 'stream-open-ack' | 'stream-frame'`.
## Seeding (no prekey-server round trip)
The stream root key is derived from an identity-bound **3-DH** exchange
— the X3DH pattern minus signed/one-time prekeys, because the peer's
identity is *already* mutually pinned by the parent session's TOFU. Two
ephemerals are exchanged inside the transport (`STREAM_OPEN` /
`STREAM_OPEN_ACK`); no prekey server is involved.
```
slotA = DH(initiatorEphemeral, responderIdentity) — authenticates responder
slotB = DH(initiatorIdentity, responderEphemeral) — authenticates initiator
slotC = DH(initiatorEphemeral, responderEphemeral) — ephemeral forward secrecy
SK = HKDF(ikm = slotA‖slotB‖slotC, salt = streamId, info = "ShadeStream/v1")
```
Both peers compute the identical three scalars regardless of role.
`SK` then bootstraps a textbook Double Ratchet by handing the
responder's ephemeral to `initSenderSession`/`initReceiverSession`
exactly the way X3DH hands its signed prekey to the ratchet — so
`ratchetEncrypt`/`ratchetDecrypt` and every guarantee they carry apply
unchanged.
## Security contract (answers FR R1R7)
- **R1 — same properties as `send`/`receive`.** Each frame is one
`ratchetEncrypt`/`ratchetDecrypt` over the *same* crypto as the HTTP
path: AES-256-GCM confidentiality, per-frame forward secrecy via the
one-way HMAC chain-key KDF with in-place zeroize of the spent chain
key, and replay/rewind rejection (a re-delivered or counter-rewound
frame fails closed). The handshake is mutually authenticated against
the identities the parent session already pinned.
- **R2 — one-directional resilience.** A long server→client burst with
no client traffic only advances the symmetric sending chain (no DH
step until the peer replies — standard Double Ratchet). Forward
secrecy holds per frame in this regime. Over an ordered transport
(WebSocket/TCP) zero keys are skipped per frame.
- **R3 — bounded memory.** Out-of-order arrivals are capped by the
ratchet's `MAX_SKIP` (1000) and `MAX_CACHED_SKIPPED_KEYS` (2000)
with oldest-key eviction. In-order delivery retains nothing. Verified
to stay at zero retained keys across a 5000-frame burst.
- **R4 — browser parity.** Identical API and guarantees in the browser
SDK: `ShadeStream` is on the same `Shade` class over the same
`CryptoProvider` (`SubtleCryptoProvider`), so the IndexedDB-backed
build behaves identically to the `sqlite:` server build. No storage
is touched at all (see R7), so the keystore backend is irrelevant.
- **R5 — independent lifecycle.** The stream ratchet is derived without
reading or writing the stored parent `SessionState`, runs on its own
private op-mutex (not the per-peer `send`/`receive` queues), and is
zeroized on `close()`. Opening, using for thousands of frames, and
closing a stream leaves the parent session byte-identical; the HTTP
path keeps working concurrently against the same peer. Each
`openStream` gets a fresh `streamId` and an independent root, so
concurrent streams to one peer never share key material.
- **R6 — wire framing.** `@shade/proto` defines `STREAM_OPEN` (0x31),
`STREAM_OPEN_ACK` (0x32), `STREAM_FRAME` (0x33). A `STREAM_FRAME`
carries one Double-Ratchet message via the exact ratchet inner codec
the HTTP path uses. One sealed logical frame ⇒ one self-delimiting
wire frame ⇒ one WS text/binary frame.
- **R7 — performance.** The ratchet lives **only in memory and is never
persisted**. There is therefore *zero* per-frame storage I/O — the
per-frame cost is exactly the symmetric KDF + one AES-GCM, the same
primitives the HTTP path runs. This is strictly better than the
"doubled CPU" the Vyvern roadmap budgeted, because the dominant cost
the naive `send`/`receive`-per-frame approach would have paid (a
`saveSession` keystore write per frame) is eliminated, not doubled.
Not persisting is also a *security* property, not a shortcut: writing
evolving per-frame ratchet secrets to disk would defeat forward
secrecy. A dropped/reconnected stream is re-opened with a fresh
handshake, never resumed.
## Double-Ratchet ordering note
A responder cannot `seal()` until it has `open()`ed at least one frame
from the initiator (standard Signal behaviour — the responder has no
sending chain until the first DH step). For a server-heavy stream
either make the bursty data sender the **initiator**, or have the
initiator send one priming frame immediately after the handshake.
## Tests
- `packages/shade-core/tests/stream.test.ts` — handshake agreement,
frame round-trips, 5000-frame one-directional burst (bounded skipped
keys + forward-secrecy zeroize), parent-session independence (R5),
replay/rewind rejection, mutual authentication against pinned
identities, `close()` zeroize/idempotence.
- `packages/shade-proto/tests/stream-wire.test.ts` — wire round-trips
and type-tag/length rejection for all three stream frame kinds.
</content>
</invoke>

View File

@@ -16,8 +16,10 @@
"version": "bun run scripts/bump-version.ts", "version": "bun run scripts/bump-version.ts",
"soak": "bun run scripts/soak.ts", "soak": "bun run scripts/soak.ts",
"soak:smoke": "bun run scripts/soak.ts --hours 0.05 --pairs 4", "soak:smoke": "bun run scripts/soak.ts --hours 0.05 --pairs 4",
"publish:dry": "DRY_RUN=1 bun run scripts/publish-all.ts", "typecheck": "bun run scripts/typecheck-all.ts",
"publish:all": "bash scripts/publish-shade.sh", "prepublish:check": "bun run typecheck",
"publish:dry": "bun run prepublish:check && DRY_RUN=1 bun run scripts/publish-all.ts",
"publish:all": "bun run prepublish:check && bash scripts/publish-shade.sh",
"build:docker": "bun run scripts/build-docker.ts", "build:docker": "bun run scripts/build-docker.ts",
"publish:docker": "bun run scripts/build-docker.ts -- --push" "publish:docker": "bun run scripts/build-docker.ts -- --push"
}, },

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/cli", "name": "@shade/cli",
"version": "4.0.0", "version": "4.11.1",
"type": "module", "type": "module",
"main": "src/cli.ts", "main": "src/cli.ts",
"bin": { "bin": {

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/core", "name": "@shade/core",
"version": "4.0.0", "version": "4.11.1",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -107,6 +107,31 @@ export class FingerprintNotVerifiedError extends ShadeError {
} }
} }
/**
* Thrown when `seal()` / `open()` is called on a {@link StreamRatchet}
* that has already been torn down via `close()`. The stream's ratchet
* secrets have been zeroized and cannot be revived — open a fresh
* stream instead.
*/
export class StreamClosedError extends ShadeError {
constructor(message = 'Stream is closed') {
super('SHADE_STREAM_CLOSED', message);
this.name = 'StreamClosedError';
}
}
/**
* Thrown when a stream handshake frame is malformed, arrives in the
* wrong order, or references a streamId that does not match the stream
* it was fed to.
*/
export class StreamHandshakeError extends ShadeError {
constructor(message = 'Stream handshake failed') {
super('SHADE_STREAM_HANDSHAKE', message);
this.name = 'StreamHandshakeError';
}
}
// ─── Infrastructure Errors ─────────────────────────────────── // ─── Infrastructure Errors ───────────────────────────────────
export class NetworkError extends ShadeError { export class NetworkError extends ShadeError {
@@ -158,6 +183,30 @@ export class UnauthorizedError extends ShadeError {
} }
} }
/**
* 409 Conflict — caller wrote to a resource that already exists without
* supplying an If-Match precondition. V4.9: the encrypted blob primitive
* uses this to force read-then-write on already-occupied slots.
*/
export class ConflictError extends ShadeError {
constructor(message = 'Conflict') {
super('SHADE_CONFLICT', message);
this.name = 'ConflictError';
}
}
/**
* 412 Precondition Failed — caller supplied an If-Match etag that does
* not match the current state. V4.9: the encrypted blob primitive uses
* this to surface stale-CAS so clients can re-read, merge, and retry.
*/
export class PreconditionFailedError extends ShadeError {
constructor(message = 'Precondition failed') {
super('SHADE_PRECONDITION_FAILED', message);
this.name = 'PreconditionFailedError';
}
}
// ─── Error → HTTP Status Mapping ──────────────────────────── // ─── Error → HTTP Status Mapping ────────────────────────────
/** /**
@@ -180,7 +229,10 @@ export function errorToHttpStatus(error: unknown): number {
return 400; return 400;
case 'SHADE_REPLAY': case 'SHADE_REPLAY':
case 'SHADE_DUPLICATE_MESSAGE': case 'SHADE_DUPLICATE_MESSAGE':
case 'SHADE_CONFLICT':
return 409; return 409;
case 'SHADE_PRECONDITION_FAILED':
return 412;
case 'SHADE_RATE_LIMIT': case 'SHADE_RATE_LIMIT':
return 429; return 429;
case 'SHADE_TIMEOUT': case 'SHADE_TIMEOUT':

View File

@@ -26,6 +26,8 @@ export interface ShadeEventMap {
'identity.rotated': { newFingerprint: string }; 'identity.rotated': { newFingerprint: string };
'session.created': { address: string; remoteIdentityKeyHash: string }; 'session.created': { address: string; remoteIdentityKeyHash: string };
'session.removed': { address: string }; 'session.removed': { address: string };
/** V4.8.3 — emitted when `aliasSession` moves a peer's per-peer state. */
'session.aliased': { oldLabel: string; newLabel: string };
'message.encrypted': { address: string; counter: number; ciphertextSize: number }; 'message.encrypted': { address: string; counter: number; ciphertextSize: number };
'message.decrypted': { address: string; counter: number; plaintextSize: number }; 'message.decrypted': { address: string; counter: number; plaintextSize: number };
'ratchet.dh_step': { address: string }; 'ratchet.dh_step': { address: string };
@@ -34,6 +36,10 @@ export interface ShadeEventMap {
'signed_prekey.rotated': { oldKeyId: number; newKeyId: number }; 'signed_prekey.rotated': { oldKeyId: number; newKeyId: number };
'trust.pinned': { address: string; identityKeyHash: string }; 'trust.pinned': { address: string; identityKeyHash: string };
'trust.changed': { address: string; oldKeyHash: string; newKeyHash: string }; 'trust.changed': { address: string; oldKeyHash: string; newKeyHash: string };
/** V4.11 — a streaming sub-ratchet handshake completed. */
'stream.opened': { address: string; role: 'initiator' | 'responder' };
/** V4.11 — a streaming sub-ratchet was torn down and zeroized. */
'stream.closed': { address: string };
} }
export type ShadeEventName = keyof ShadeEventMap; export type ShadeEventName = keyof ShadeEventMap;

View File

@@ -5,6 +5,7 @@ export * from './keys.js';
export * from './errors.js'; export * from './errors.js';
export * from './x3dh.js'; export * from './x3dh.js';
export * from './ratchet.js'; export * from './ratchet.js';
export * from './stream.js';
export { ShadeSessionManager, GRACE_PERIOD_MS } from './session.js'; export { ShadeSessionManager, GRACE_PERIOD_MS } from './session.js';
export * from './serialization.js'; export * from './serialization.js';
export * from './fingerprint.js'; export * from './fingerprint.js';

View File

@@ -185,6 +185,22 @@ export async function ratchetDecrypt(
if (!session.receiveChain) { if (!session.receiveChain) {
throw new DecryptionError('No receiving chain available'); throw new DecryptionError('No receiving chain available');
} }
// Defense-in-depth: a same-DH message whose counter is already
// behind the chain — and that did NOT match a cached skipped key —
// is either a duplicate we already decrypted (skipped key was
// consumed) or one whose key was evicted under cache pressure.
// Falling through would call kdfChainKey on the *current* (ahead)
// chainKey and then rewind `chain.counter = message.counter + 1`,
// permanently desyncing the chain so every subsequent decrypt
// returns wrong-key. Reject without mutating state instead.
if (
!isNewRatchet &&
message.counter < session.receiveChain.counter
) {
throw new DecryptionError(
'Failed to decrypt message — wrong key or tampered data',
);
}
await skipMessageKeys(crypto, session, message.dhPublicKey, session.receiveChain, message.counter); await skipMessageKeys(crypto, session, message.dhPublicKey, session.receiveChain, message.counter);
// Advance the receiving chain one more step to get this message's key // Advance the receiving chain one more step to get this message's key

View File

@@ -23,7 +23,14 @@ import {
ratchetEncrypt, ratchetEncrypt,
ratchetDecrypt, ratchetDecrypt,
} from './ratchet.js'; } from './ratchet.js';
import { NoSessionError } from './errors.js'; import {
deriveStreamRootKey,
bootstrapStreamSession,
StreamRatchet,
STREAM_ID_BYTES,
STREAM_EPHEMERAL_BYTES,
} from './stream.js';
import { NoSessionError, StreamHandshakeError } from './errors.js';
import { computeFingerprint, shortFingerprint } from './fingerprint.js'; import { computeFingerprint, shortFingerprint } from './fingerprint.js';
import { ShadeEventEmitter, shortHash } from './events.js'; import { ShadeEventEmitter, shortHash } from './events.js';
import { import {
@@ -265,6 +272,73 @@ export class ShadeSessionManager {
// Note: we keep the trusted identity; new session will verify against it. // Note: we keep the trusted identity; new session will verify against it.
} }
/**
* Move every per-peer storage row for `oldLabel` (session, trusted
* identity, peer-verification, identity-version counter) to
* `newLabel`. Used to canonicalize sessions when first-contact
* forces the receiver to label by sender-fingerprint hint
* (`fp:<hex>`) and a later in-band announcement reveals the peer's
* canonical address.
*
* Holds the per-peer mutex on **both** labels for the whole
* migration so concurrent encrypt/decrypt for either label can't
* observe a half-moved state. Locks are taken in lexicographic
* order to avoid deadlocks if two callers alias in opposite
* directions.
*
* Throws if no session exists for `oldLabel`. Throws (refuses to
* overwrite) if a session already exists for `newLabel`. No-ops
* when `oldLabel === newLabel`.
*
* V4.8.3 — Prism FR `session-label-asymmetry-v4.8.2.md`.
*/
async aliasSession(oldLabel: string, newLabel: string): Promise<void> {
if (oldLabel === newLabel) return;
const [first, second] = oldLabel < newLabel ? [oldLabel, newLabel] : [newLabel, oldLabel];
await this.runUnderPeerLock(first, () =>
this.runUnderPeerLock(second, () => this.aliasUnderLock(oldLabel, newLabel)),
);
}
private async aliasUnderLock(oldLabel: string, newLabel: string): Promise<void> {
const session = await this.storage.getSession(oldLabel);
if (!session) throw new NoSessionError(oldLabel);
const collision = await this.storage.getSession(newLabel);
if (collision) {
throw new Error(
`aliasSession: refusing to overwrite an existing session for "${newLabel}". ` +
`If you want to replace it, call resetSession("${newLabel}") first.`,
);
}
// Move the session.
await this.storage.saveSession(newLabel, session);
// Re-pin trust under the new label using the session's stored DH
// identity key — `saveTrustedIdentity` is the same primitive that
// the X3DH initiator/responder uses, and the DH key in `session`
// is the value that was pinned at session-establish time. The old
// pin under `oldLabel` is harmless leftover (the storage interface
// has no remove for trust pins) and would only be re-checked if a
// fresh X3DH against `oldLabel` somehow happened later.
await this.storage.saveTrustedIdentity(newLabel, session.remoteIdentityKey);
// Migrate the peer-verification record if present.
const verification = await this.storage.getPeerVerification(oldLabel);
if (verification) {
await this.storage.savePeerVerification({
...verification,
peerAddress: newLabel,
});
await this.storage.removePeerVerification(oldLabel);
}
// Carry the identity-version counter forward so peer rotation
// history is preserved.
const oldVersion = await this.storage.getPeerIdentityVersion(oldLabel);
for (let i = 1; i < oldVersion; i++) {
await this.storage.bumpPeerIdentityVersion(newLabel);
}
await this.storage.removeSession(oldLabel);
this.events?.emit('session.aliased', { oldLabel, newLabel });
}
/** /**
* Accept a changed remote identity. This should only be called after * Accept a changed remote identity. This should only be called after
* verifying the new identity out-of-band (e.g., comparing fingerprints). * verifying the new identity out-of-band (e.g., comparing fingerprints).
@@ -559,6 +633,121 @@ export class ShadeSessionManager {
return dec.decode(plaintext); return dec.decode(plaintext);
} }
// ─── Streaming sub-sessions (V4.11) ────────────────────────
/**
* Resolve the peer's pinned identity X25519 key for a stream
* handshake. Requires an *already established* parent session — the
* stream is explicitly a "second channel on a known peer", never a
* first contact (so it needs no prekey-server round trip and inherits
* the parent's TOFU pin).
*/
private async streamIdentityMaterial(
address: string,
): Promise<{ selfIdentityDHPriv: Uint8Array; peerIdentityDHPub: Uint8Array }> {
if (!this.identity) throw new Error('Not initialized');
const session = await this.storage.getSession(address);
if (!session) throw new NoSessionError(address);
return {
selfIdentityDHPriv: this.identity.dhPrivateKey,
peerIdentityDHPub: session.remoteIdentityKey,
};
}
/**
* Initiator side of a stream handshake. Generates the streamId and
* this side's ephemeral, and returns a `complete` continuation that
* derives the sub-ratchet once the responder's ephemeral arrives in
* the `STREAM_OPEN_ACK`.
*
* Touches neither the stored parent session nor the per-peer op
* queues (R5).
*/
async beginStream(address: string): Promise<{
streamId: Uint8Array;
ephemeralPublicKey: Uint8Array;
complete: (peerEphemeralPub: Uint8Array) => Promise<StreamRatchet>;
}> {
const { selfIdentityDHPriv, peerIdentityDHPub } =
await this.streamIdentityMaterial(address);
const streamId = this.crypto.randomBytes(STREAM_ID_BYTES);
const ephemeral = await this.crypto.generateX25519KeyPair();
const complete = async (peerEphemeralPub: Uint8Array): Promise<StreamRatchet> => {
if (peerEphemeralPub.length !== STREAM_EPHEMERAL_BYTES) {
throw new StreamHandshakeError(
`responder ephemeral must be ${STREAM_EPHEMERAL_BYTES} bytes`,
);
}
const sk = await deriveStreamRootKey(
this.crypto,
'initiator',
streamId,
selfIdentityDHPriv,
peerIdentityDHPub,
ephemeral.privateKey,
peerEphemeralPub,
);
const session = await bootstrapStreamSession(this.crypto, 'initiator', sk, peerIdentityDHPub, {
publicKey: peerEphemeralPub,
privateKey: new Uint8Array(0),
});
this.crypto.zeroize(sk);
this.crypto.zeroize(ephemeral.privateKey);
this.events?.emit('stream.opened', { address, role: 'initiator' });
return new StreamRatchet(this.crypto, session, streamId);
};
return { streamId, ephemeralPublicKey: ephemeral.publicKey, complete };
}
/**
* Responder side of a stream handshake. Given the initiator's
* `STREAM_OPEN` (its streamId + ephemeral), derives the sub-ratchet
* immediately and returns this side's ephemeral for the
* `STREAM_OPEN_ACK`.
*/
async acceptStream(
address: string,
streamId: Uint8Array,
initiatorEphemeralPub: Uint8Array,
): Promise<{ ephemeralPublicKey: Uint8Array; stream: StreamRatchet }> {
if (streamId.length !== STREAM_ID_BYTES) {
throw new StreamHandshakeError(`streamId must be ${STREAM_ID_BYTES} bytes`);
}
if (initiatorEphemeralPub.length !== STREAM_EPHEMERAL_BYTES) {
throw new StreamHandshakeError(
`initiator ephemeral must be ${STREAM_EPHEMERAL_BYTES} bytes`,
);
}
const { selfIdentityDHPriv, peerIdentityDHPub } =
await this.streamIdentityMaterial(address);
const ephemeral = await this.crypto.generateX25519KeyPair();
const sk = await deriveStreamRootKey(
this.crypto,
'responder',
streamId,
selfIdentityDHPriv,
peerIdentityDHPub,
ephemeral.privateKey,
initiatorEphemeralPub,
);
const session = await bootstrapStreamSession(
this.crypto,
'responder',
sk,
peerIdentityDHPub,
ephemeral,
);
this.crypto.zeroize(sk);
this.events?.emit('stream.opened', { address, role: 'responder' });
return {
ephemeralPublicKey: ephemeral.publicKey,
stream: new StreamRatchet(this.crypto, session, streamId),
};
}
} }
function arraysEqual(a: Uint8Array, b: Uint8Array): boolean { function arraysEqual(a: Uint8Array, b: Uint8Array): boolean {

View File

@@ -46,6 +46,50 @@ export interface PersistedStreamState {
*/ */
export type PeerVerificationSource = 'user' | 'transitive' | 'tofu-after-warning'; export type PeerVerificationSource = 'user' | 'transitive' | 'tofu-after-warning';
/**
* Persisted broadcast-channel state (V4.6). Holds the sender-key chain that
* lets the channel owner encrypt a single ciphertext and fan it out to all
* paired members.
*
* - `ownerRole === 'sender'`: this device created the channel; the record
* carries the full chain (chainKey + iteration + signing keypair).
* - `ownerRole === 'receiver'`: this device joined a channel that
* `ownerAddress` owns; we hold a tracking copy of the chain (chainKey +
* iteration + signing public key only — no private signing key).
*
* `generation` is bumped each time `removeMember` rotates the chain. Stale
* broadcasts at lower generations are silently dropped on receive.
*/
export interface BroadcastChannelRecord {
/** Opaque, stable across restarts. UUID-style. */
channelId: string;
ownerRole: 'sender' | 'receiver';
/** The address that owns the channel (i.e. the only sender). */
ownerAddress: string;
label?: string;
/** Sender-key generation — bumped on rotation. Starts at 0. */
generation: number;
/** Current chain key (32 bytes). */
chainKey: Uint8Array;
/** Counter advanced by `senderKeyEncrypt` / `senderKeyDecrypt`. */
iteration: number;
/** Owner's Ed25519 signing public key (32 bytes). */
signingPublicKey: Uint8Array;
/** Owner's Ed25519 signing private key. Present iff `ownerRole === 'sender'`. */
signingPrivateKey?: Uint8Array;
createdAt: number;
updatedAt: number;
}
/** Membership row for a broadcast channel (sender-side only). */
export interface BroadcastMemberRecord {
channelId: string;
peerAddress: string;
joinedAt: number;
/** When this peer was revoked. `null` while still active. */
removedAt: number | null;
}
/** /**
* Persistent record that a peer's safety number was verified at a point * Persistent record that a peer's safety number was verified at a point
* in time. `identityVersion` is the local counter for that peer's identity: * in time. `identityVersion` is the local counter for that peer's identity:
@@ -188,4 +232,35 @@ export interface StorageProvider {
/** Prune stream-state rows in `'finished' | 'aborted'` status older than `olderThan`. */ /** Prune stream-state rows in `'finished' | 'aborted'` status older than `olderThan`. */
pruneStreamStates?(olderThan: number): Promise<void>; pruneStreamStates?(olderThan: number): Promise<void>;
// ─── Broadcast channels (V4.6) — optional ─────────────────
/**
* Persist or replace the broadcast-channel record. Idempotent upsert on
* `channelId`. Backends that don't implement broadcast-channel support
* may omit this; the SDK throws a clear error when the app tries to
* call `createBroadcastChannel` on such a backend.
*/
saveBroadcastChannel?(channel: BroadcastChannelRecord): Promise<void>;
/** Look up a broadcast channel by id. */
getBroadcastChannel?(channelId: string): Promise<BroadcastChannelRecord | null>;
/** Enumerate all broadcast channels persisted on this device. */
listBroadcastChannels?(): Promise<BroadcastChannelRecord[]>;
/** Drop a broadcast channel and all its membership rows. */
removeBroadcastChannel?(channelId: string): Promise<void>;
/** Persist or replace a broadcast-membership row (sender-side only). */
saveBroadcastMember?(member: BroadcastMemberRecord): Promise<void>;
/**
* List membership rows for a channel. Includes revoked members (with
* `removedAt !== null`); callers filter as needed.
*/
getBroadcastMembers?(channelId: string): Promise<BroadcastMemberRecord[]>;
/** Hard-delete a single membership row. */
removeBroadcastMember?(channelId: string, peerAddress: string): Promise<void>;
} }

View File

@@ -0,0 +1,233 @@
import type { CryptoProvider } from './crypto.js';
import type { KeyPair, RatchetMessage, SessionState } from './types.js';
import {
initSenderSession,
initReceiverSession,
ratchetEncrypt,
ratchetDecrypt,
} from './ratchet.js';
import { StreamClosedError } from './errors.js';
/**
* Streaming Double-Ratchet sub-sessions (V4.11).
*
* Wraps a long-lived, high-frequency, often-one-directional channel
* (e.g. a server→client WebSocket log burst) in an *independent* Double
* Ratchet that is derived from — but never mutates — an already
* established parent Shade session.
*
* Why a sub-ratchet rather than reusing `ShadeSessionManager`:
*
* - **Independence (R5).** A stream gets its own root key, chains, DH
* ratchet and op-mutex. Opening/closing it never touches the stored
* parent `SessionState` nor serialises against the HTTP send/receive
* queue.
* - **Performance (R7).** The stream ratchet lives only in memory and
* is *never* written to the keystore. There is therefore zero
* per-frame storage I/O — the cost is purely the symmetric KDF +
* AES-GCM, the same primitives the HTTP path uses.
* - **Forward secrecy.** Not persisting the evolving ratchet state is
* a feature, not a shortcut: writing per-frame secrets to disk would
* actively defeat the forward-secrecy guarantee. A dropped/reconnected
* stream is re-opened with a fresh handshake, not resumed.
*
* ## Seeding (no prekey-server round trip)
*
* The stream root key is derived from an identity-bound 3-DH exchange —
* the X3DH pattern minus the signed / one-time prekeys, because the
* peer's identity is *already* mutually pinned by the parent session's
* TOFU. Two ephemeral keys are exchanged inside the transport itself
* (`STREAM_OPEN` / `STREAM_OPEN_ACK`); no prekey server is involved.
*
* slotA = DH(initiatorEphemeral, responderIdentity) — auth of responder
* slotB = DH(initiatorIdentity, responderEphemeral) — auth of initiator
* slotC = DH(initiatorEphemeral, responderEphemeral) — ephemeral FS
*
* SK = HKDF(ikm = slotA‖slotB‖slotC, salt = streamId, info = "ShadeStream/v1")
*
* Both peers compute the identical three scalars regardless of role, so
* `SK` agrees. An attacker lacking the responder's identity private key
* cannot form slotA; one lacking the initiator's cannot form slotB —
* the handshake is therefore mutually authenticated against the same
* identities the parent session already trusts.
*
* `SK` then bootstraps a textbook Double Ratchet by handing the
* responder's ephemeral to {@link initSenderSession} /
* {@link initReceiverSession} exactly the way X3DH hands its signed
* prekey to the ratchet — so `ratchetEncrypt` / `ratchetDecrypt` (and
* thus every R1R3 guarantee they already carry) apply unchanged.
*/
export type StreamRole = 'initiator' | 'responder';
/** Stream identifier length (bytes). 128 bits of collision resistance. */
export const STREAM_ID_BYTES = 16;
/** Ephemeral X25519 public-key length carried in the handshake. */
export const STREAM_EPHEMERAL_BYTES = 32;
const STREAM_KDF_INFO = new TextEncoder().encode('ShadeStream/v1');
/**
* Derive the stream's independent root key from the identity-bound 3-DH
* exchange. Pure: never reads or mutates any `SessionState`.
*
* @param role which end of the handshake we are
* @param streamId 16-byte stream id (HKDF salt; binds the
* derivation so two concurrent streams to the
* same peer never share a root key)
* @param selfIdentityDHPriv our long-term identity X25519 private key
* @param peerIdentityDHPub peer's pinned identity X25519 public key
* (the value the parent session pinned)
* @param selfEphemeralPriv our per-stream ephemeral X25519 private key
* @param peerEphemeralPub peer's per-stream ephemeral X25519 public key
*/
export async function deriveStreamRootKey(
crypto: CryptoProvider,
role: StreamRole,
streamId: Uint8Array,
selfIdentityDHPriv: Uint8Array,
peerIdentityDHPub: Uint8Array,
selfEphemeralPriv: Uint8Array,
peerEphemeralPub: Uint8Array,
): Promise<Uint8Array> {
// Each slot is pinned to a fixed semantic (not to local role) so both
// sides feed HKDF the identical ikm:
// slotA = DH(initiatorEphemeral, responderIdentity)
// slotB = DH(initiatorIdentity, responderEphemeral)
// slotC = DH(initiatorEphemeral, responderEphemeral)
let slotA: Uint8Array;
let slotB: Uint8Array;
let slotC: Uint8Array;
if (role === 'initiator') {
slotA = await crypto.x25519(selfEphemeralPriv, peerIdentityDHPub);
slotB = await crypto.x25519(selfIdentityDHPriv, peerEphemeralPub);
slotC = await crypto.x25519(selfEphemeralPriv, peerEphemeralPub);
} else {
slotA = await crypto.x25519(selfIdentityDHPriv, peerEphemeralPub);
slotB = await crypto.x25519(selfEphemeralPriv, peerIdentityDHPub);
slotC = await crypto.x25519(selfEphemeralPriv, peerEphemeralPub);
}
const ikm = new Uint8Array(96);
ikm.set(slotA, 0);
ikm.set(slotB, 32);
ikm.set(slotC, 64);
const sk = await crypto.hkdf(ikm, streamId, STREAM_KDF_INFO, 32);
crypto.zeroize(slotA);
crypto.zeroize(slotB);
crypto.zeroize(slotC);
crypto.zeroize(ikm);
return sk;
}
/**
* Bootstrap a fresh Double Ratchet `SessionState` from the derived
* stream root key. The responder's ephemeral plays exactly the role
* X3DH's signed prekey plays in {@link initSenderSession} /
* {@link initReceiverSession}, so the ratchet handoff is identical to
* the proven HTTP path.
*
* On the initiator only `responderEphemeral.publicKey` is needed; the
* responder must pass its full ephemeral keypair.
*
* `peerIdentityDHPub` is recorded as the session's `remoteIdentityKey`
* so stream fingerprints stay meaningful and consistent with the parent.
*/
export async function bootstrapStreamSession(
crypto: CryptoProvider,
role: StreamRole,
sk: Uint8Array,
peerIdentityDHPub: Uint8Array,
responderEphemeral: KeyPair,
): Promise<SessionState> {
if (role === 'initiator') {
// initSenderSession derives a fresh root via kdfRootKey and does not
// retain `sk`, so the caller may safely zeroize it afterwards.
return initSenderSession(crypto, sk, peerIdentityDHPub, responderEphemeral.publicKey);
}
// initReceiverSession stores the root key BY REFERENCE. Hand it an
// independent copy so the caller zeroizing its `sk` scratch buffer
// can't wipe the live session root.
return initReceiverSession(new Uint8Array(sk), peerIdentityDHPub, responderEphemeral);
}
/** Zeroize every secret a stream session holds, then drop the chains. */
function zeroizeSession(crypto: CryptoProvider, s: SessionState): void {
crypto.zeroize(s.rootKey);
if (s.sendChain.chainKey.length > 0) crypto.zeroize(s.sendChain.chainKey);
if (s.receiveChain && s.receiveChain.chainKey.length > 0) {
crypto.zeroize(s.receiveChain.chainKey);
}
if (s.dhSend.privateKey.length > 0) crypto.zeroize(s.dhSend.privateKey);
for (const mk of s.skippedKeys.values()) crypto.zeroize(mk);
s.skippedKeys.clear();
}
/**
* In-memory holder for a stream's Double Ratchet. Serialises its own
* `seal`/`open`/`close` on a private promise chain (independent of the
* SDK's per-peer encrypt/decrypt queues — R5) so per-frame ratchet
* mutations never interleave, while staying fully concurrent with the
* parent session and with other streams.
*
* Never persisted: the ratchet exists only for the lifetime of the
* stream and is zeroized on `close()`.
*/
export class StreamRatchet {
private session: SessionState | null;
private opChain: Promise<unknown> = Promise.resolve();
constructor(
private readonly crypto: CryptoProvider,
session: SessionState,
/** 16-byte stream id this ratchet is bound to. */
public readonly streamId: Uint8Array,
) {
this.session = session;
}
/** True once {@link close} has run; `seal`/`open` will throw. */
get closed(): boolean {
return this.session === null;
}
private run<T>(fn: (s: SessionState) => Promise<T>): Promise<T> {
const next = this.opChain.catch(() => undefined).then(() => {
if (!this.session) throw new StreamClosedError();
return fn(this.session);
});
// Keep a never-rejecting tail so a failed frame doesn't poison the
// next one (a single bad inbound frame must not wedge the stream).
this.opChain = next.catch(() => undefined);
return next;
}
/** Wrap one logical frame. Advances the sending chain by one step. */
seal(plaintext: Uint8Array): Promise<RatchetMessage> {
return this.run((s) => ratchetEncrypt(this.crypto, s, plaintext));
}
/**
* Unwrap one inbound frame. Correct and memory-bounded across long
* one-directional runs from the peer: ordered transport delivery
* skips zero keys per frame, and out-of-order arrivals are still
* capped by the ratchet's `MAX_SKIP` / `MAX_CACHED_SKIPPED_KEYS`.
*/
open(message: RatchetMessage): Promise<Uint8Array> {
return this.run((s) => ratchetDecrypt(this.crypto, s, message));
}
/** Zeroize and drop the ratchet. Idempotent. */
close(): Promise<void> {
return this.opChain
.catch(() => undefined)
.then(() => {
if (this.session) {
zeroizeSession(this.crypto, this.session);
this.session = null;
}
});
}
}

View File

@@ -281,6 +281,41 @@ describe('Double Ratchet', () => {
expect(ratchetDecrypt(crypto, bob, msg)).rejects.toThrow(); expect(ratchetDecrypt(crypto, bob, msg)).rejects.toThrow();
}); });
/**
* Regression — the v4.2.0 OutboundQueue waiter-since bug delivered
* the same envelope twice to `manager.decrypt`. The first decrypt
* succeeded via a cached skipped key; the second one fell into the
* `message.counter < chain.counter` path with no skipped key
* available, advanced the chainKey ONCE and rewound `chain.counter`
* to `message.counter + 1`, leaving the ratchet permanently
* desynced. ratchetDecrypt now rejects without mutating state when
* a same-DH message is behind the chain and not in skippedKeys, so
* a downstream replay (transport bug, retry, etc.) cannot poison
* the session for everyone else.
*/
test('same-DH stale message after consumed skipped key fails without corrupting state', async () => {
const { alice, bob } = await setupPair();
// Alice sends 3 messages on the same DH chain.
const m0 = await ratchetEncrypt(crypto, alice, enc.encode('m0'));
const m1 = await ratchetEncrypt(crypto, alice, enc.encode('m1'));
const m2 = await ratchetEncrypt(crypto, alice, enc.encode('m2'));
// Bob receives m1 first, caching m0's key. Then m0 (delivered
// via the cache). After this, m0's skipped key is consumed.
expect(dec.decode(await ratchetDecrypt(crypto, bob, m1))).toBe('m1');
expect(dec.decode(await ratchetDecrypt(crypto, bob, m0))).toBe('m0');
// Replay of m0: skippedKey is gone, chain.counter is past m0.
// Pre-fix: this would corrupt Bob's chain state; post-fix it
// throws cleanly.
await expect(ratchetDecrypt(crypto, bob, m0)).rejects.toThrow(DecryptionError);
// Bob can still decrypt the remaining valid message — chain
// state was NOT mutated by the rejected replay.
expect(dec.decode(await ratchetDecrypt(crypto, bob, m2))).toBe('m2');
});
}); });
// ─── Long Conversation ──────────────────────────────────── // ─── Long Conversation ────────────────────────────────────

View File

@@ -0,0 +1,176 @@
import { describe, test, expect, beforeEach } from 'bun:test';
import { SubtleCryptoProvider, MemoryStorage } from '@shade/crypto-web';
import {
ShadeSessionManager,
StreamRatchet,
StreamClosedError,
DecryptionError,
} from '../src/index.js';
const crypto = new SubtleCryptoProvider();
const enc = new TextEncoder();
const dec = new TextDecoder();
/**
* Establish a *bidirectional* parent session: Alice→Bob X3DH, then one
* Alice→Bob message Bob decrypts so Bob also has a session for 'alice'.
* Both sides then hold the peer's pinned identity DH key — the input the
* stream handshake derives from.
*/
async function bidirectionalPair() {
const aliceStorage = new MemoryStorage();
const bobStorage = new MemoryStorage();
const alice = new ShadeSessionManager(crypto, aliceStorage);
const bob = new ShadeSessionManager(crypto, bobStorage);
await alice.initialize();
await bob.initialize();
const otpks = await bob.generateOneTimePreKeys(4);
const bundle = await bob.createPreKeyBundle();
bundle.oneTimePreKey = { keyId: otpks[0]!.keyId, publicKey: otpks[0]!.keyPair.publicKey };
await alice.initSessionFromBundle('bob', bundle);
const hello = await alice.encrypt('bob', 'parent-hello');
expect(await bob.decrypt('alice', hello)).toBe('parent-hello');
return { alice, bob, aliceStorage, bobStorage };
}
/** Run the full STREAM_OPEN / STREAM_OPEN_ACK handshake between managers. */
async function openStreamPair(alice: ShadeSessionManager, bob: ShadeSessionManager) {
const begun = await alice.beginStream('bob'); // initiator
const accepted = await bob.acceptStream('alice', begun.streamId, begun.ephemeralPublicKey);
const aliceStream = await begun.complete(accepted.ephemeralPublicKey);
return { aliceStream, bobStream: accepted.stream, streamId: begun.streamId };
}
describe('streaming sub-ratchet (V4.11)', () => {
let alice: ShadeSessionManager;
let bob: ShadeSessionManager;
let aliceStorage: MemoryStorage;
beforeEach(async () => {
({ alice, bob, aliceStorage } = await bidirectionalPair());
});
test('both sides derive the same stream root (round-trips frames)', async () => {
const { aliceStream, bobStream } = await openStreamPair(alice, bob);
// Initiator → responder (first frame triggers responder DH step).
const f1 = await aliceStream.seal(enc.encode('log line 1'));
expect(dec.decode(await bobStream.open(f1))).toBe('log line 1');
// Responder → initiator (now responder may seal).
const r1 = await bobStream.seal(enc.encode('command-response 1'));
expect(dec.decode(await aliceStream.open(r1))).toBe('command-response 1');
});
test('two streams to the same peer get independent roots', async () => {
const s1 = await openStreamPair(alice, bob);
const s2 = await openStreamPair(alice, bob);
expect(s1.streamId).not.toEqual(s2.streamId);
const a = await s1.aliceStream.seal(enc.encode('on stream 1'));
// A frame from stream 1 must not decrypt on stream 2's ratchet.
await expect(s2.bobStream.open(a)).rejects.toBeInstanceOf(DecryptionError);
// …but does on its own.
expect(dec.decode(await s1.bobStream.open(a))).toBe('on stream 1');
});
test('R2/R3: long one-directional burst stays correct and memory-bounded', async () => {
const { aliceStream, bobStream } = await openStreamPair(alice, bob);
const N = 5000;
// Capture a live receive-chain key buffer to prove forward secrecy:
// ratchetDecrypt zeroizes the previous chain key in place.
await bobStream.open(await aliceStream.seal(enc.encode('frame-0')));
const bobSession = (bobStream as unknown as { session: { receiveChain: { chainKey: Uint8Array }; skippedKeys: Map<string, Uint8Array> } }).session;
const staleChainKey = bobSession.receiveChain.chainKey;
const staleCopy = staleChainKey.slice();
expect(staleCopy.some((b) => b !== 0)).toBe(true);
for (let i = 1; i < N; i++) {
const wire = await aliceStream.seal(enc.encode(`frame-${i}`));
expect(dec.decode(await bobStream.open(wire))).toBe(`frame-${i}`);
}
// In-order delivery ⇒ zero skipped keys retained across 5k frames.
expect(bobSession.skippedKeys.size).toBe(0);
// The chain key in use at frame 0 was overwritten (forward secrecy).
expect(staleChainKey.every((b) => b === 0)).toBe(true);
});
test('R5: opening/using/closing a stream never touches the parent session', async () => {
const before = await aliceStorage.getSession('bob');
const snapshot = JSON.stringify({
root: Array.from(before!.rootKey),
sendCtr: before!.sendChain.counter,
prevCtr: before!.previousSendCounter,
});
const { aliceStream, bobStream } = await openStreamPair(alice, bob);
for (let i = 0; i < 200; i++) {
await bobStream.open(await aliceStream.seal(enc.encode(`x${i}`)));
}
await aliceStream.close();
await bobStream.close();
const after = await aliceStorage.getSession('bob');
expect(
JSON.stringify({
root: Array.from(after!.rootKey),
sendCtr: after!.sendChain.counter,
prevCtr: after!.previousSendCounter,
}),
).toBe(snapshot);
// Parent HTTP path still works after the stream lifecycle.
const env = await alice.encrypt('bob', 'after-stream');
expect(await bob.decrypt('alice', env)).toBe('after-stream');
});
test('R1: replayed / rewound frame is rejected', async () => {
const { aliceStream, bobStream } = await openStreamPair(alice, bob);
const f1 = await aliceStream.seal(enc.encode('once'));
expect(dec.decode(await bobStream.open(f1))).toBe('once');
// Re-delivering the exact same sealed frame must fail.
await expect(bobStream.open(f1)).rejects.toBeInstanceOf(DecryptionError);
});
test('close() zeroizes and blocks further use; idempotent', async () => {
const { aliceStream, bobStream } = await openStreamPair(alice, bob);
await aliceStream.close();
await aliceStream.close(); // idempotent
expect(aliceStream.closed).toBe(true);
await expect(aliceStream.seal(enc.encode('nope'))).rejects.toBeInstanceOf(
StreamClosedError,
);
// The peer end is unaffected by our local close.
expect(bobStream.closed).toBe(false);
});
test('handshake is mutually authenticated against pinned identities', async () => {
// A third party (mallory) with its own identity cannot stand in for
// bob: alice derives against bob's pinned identity key, so a
// handshake completed with mallory's ephemeral yields a different
// root and frames fail to open.
const mStorage = new MemoryStorage();
const mallory = new ShadeSessionManager(crypto, mStorage);
await mallory.initialize();
// Give mallory a parent session label so acceptStream has identity
// material, but pinned to the WRONG (alice) identity vs what alice
// pinned for 'bob'.
const otpks = await mallory.generateOneTimePreKeys(2);
const mb = await mallory.createPreKeyBundle();
mb.oneTimePreKey = { keyId: otpks[0]!.keyId, publicKey: otpks[0]!.keyPair.publicKey };
await alice.initSessionFromBundle('mallory', mb);
const helo = await alice.encrypt('mallory', 'hi');
await mallory.decrypt('alice', helo);
const begun = await alice.beginStream('bob');
const mAccept = await mallory.acceptStream('alice', begun.streamId, begun.ephemeralPublicKey);
const aliceStream = await begun.complete(mAccept.ephemeralPublicKey);
const frame = await aliceStream.seal(enc.encode('secret'));
await expect(mAccept.stream.open(frame)).rejects.toBeInstanceOf(DecryptionError);
});
});

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/crypto-web", "name": "@shade/crypto-web",
"version": "4.0.0", "version": "4.11.1",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -0,0 +1,18 @@
import { ed25519 } from '@noble/curves/ed25519.js';
/**
* Deterministically derive an Ed25519 public key from a 32-byte seed.
*
* In the @noble/curves convention the "private key" *is* the seed —
* `sign(seed, msg)` works directly, and `getPublicKey(seed)` recovers
* the matching public key. V4.9's encrypted-blob primitive uses this
* to mint a per-slot signing keypair from an HKDF output rooted at the
* user's master key, so the same credentials always reproduce the same
* keypair.
*/
export function ed25519PublicKeyFromSeed(seed: Uint8Array): Uint8Array {
if (seed.length !== 32) {
throw new Error(`Ed25519 seed must be 32 bytes, got ${seed.length}`);
}
return ed25519.getPublicKey(seed);
}

View File

@@ -1,5 +1,6 @@
export { SubtleCryptoProvider } from './provider.js'; export { SubtleCryptoProvider } from './provider.js';
export { MemoryStorage } from './memory-storage.js'; export { MemoryStorage } from './memory-storage.js';
export { ed25519PublicKeyFromSeed } from './ed25519-derive.js';
// ─── Web Workers crypto (V3.8) ──────────────────────────── // ─── Web Workers crypto (V3.8) ────────────────────────────
export { export {

View File

@@ -1,4 +1,8 @@
import type { StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey, SessionState, RetiredIdentity, PersistedStreamState, PeerVerification } from '@shade/core'; import type {
StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey,
SessionState, RetiredIdentity, PersistedStreamState, PeerVerification,
BroadcastChannelRecord, BroadcastMemberRecord,
} from '@shade/core';
import { constantTimeEqual } from '@shade/core'; import { constantTimeEqual } from '@shade/core';
/** /**
@@ -167,4 +171,62 @@ export class MemoryStorage implements StorageProvider {
} }
} }
} }
// ─── Broadcast channels (V4.6) ────────────────────────────
private broadcastChannels = new Map<string, BroadcastChannelRecord>();
private broadcastMembers = new Map<string, Map<string, BroadcastMemberRecord>>();
async saveBroadcastChannel(channel: BroadcastChannelRecord): Promise<void> {
this.broadcastChannels.set(channel.channelId, cloneChannel(channel));
}
async getBroadcastChannel(channelId: string): Promise<BroadcastChannelRecord | null> {
const v = this.broadcastChannels.get(channelId);
return v ? cloneChannel(v) : null;
}
async listBroadcastChannels(): Promise<BroadcastChannelRecord[]> {
return [...this.broadcastChannels.values()].map(cloneChannel);
}
async removeBroadcastChannel(channelId: string): Promise<void> {
this.broadcastChannels.delete(channelId);
this.broadcastMembers.delete(channelId);
}
async saveBroadcastMember(member: BroadcastMemberRecord): Promise<void> {
let inner = this.broadcastMembers.get(member.channelId);
if (!inner) {
inner = new Map();
this.broadcastMembers.set(member.channelId, inner);
}
inner.set(member.peerAddress, { ...member });
}
async getBroadcastMembers(channelId: string): Promise<BroadcastMemberRecord[]> {
const inner = this.broadcastMembers.get(channelId);
return inner ? [...inner.values()].map((m) => ({ ...m })) : [];
}
async removeBroadcastMember(channelId: string, peerAddress: string): Promise<void> {
this.broadcastMembers.get(channelId)?.delete(peerAddress);
}
}
function cloneChannel(c: BroadcastChannelRecord): BroadcastChannelRecord {
const out: BroadcastChannelRecord = {
channelId: c.channelId,
ownerRole: c.ownerRole,
ownerAddress: c.ownerAddress,
generation: c.generation,
chainKey: new Uint8Array(c.chainKey),
iteration: c.iteration,
signingPublicKey: new Uint8Array(c.signingPublicKey),
createdAt: c.createdAt,
updatedAt: c.updatedAt,
};
if (c.label !== undefined) out.label = c.label;
if (c.signingPrivateKey !== undefined) out.signingPrivateKey = new Uint8Array(c.signingPrivateKey);
return out;
} }

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/dashboard", "name": "@shade/dashboard",
"version": "4.0.0", "version": "4.11.1",
"type": "module", "type": "module",
"scripts": { "scripts": {
"dev": "vite", "dev": "vite",

View File

@@ -8,5 +8,5 @@
"module": "ESNext", "module": "ESNext",
"moduleResolution": "bundler" "moduleResolution": "bundler"
}, },
"include": ["src", "vite.config.ts"] "include": ["src"]
} }

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/files", "name": "@shade/files",
"version": "4.0.0", "version": "4.11.1",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,4 +1,4 @@
import type { Shade } from '@shade/sdk'; import type { ShadeBridge } from '../integration/shade-bridge.js';
import { import {
KIND_CUSTOM_V1, KIND_CUSTOM_V1,
KIND_DELETE_V1, KIND_DELETE_V1,
@@ -184,7 +184,7 @@ export interface CreateFileClientOptions {
* transfers that carry the actual bytes. * transfers that carry the actual bytes.
*/ */
export function createFileClient( export function createFileClient(
shade: Shade, shade: ShadeBridge,
channel: ShadeFileRpcChannel, channel: ShadeFileRpcChannel,
pending: PendingRpcRegistry, pending: PendingRpcRegistry,
peerAddress: string, peerAddress: string,

View File

@@ -0,0 +1,590 @@
/**
* Browser-friendly request-response `FileClient` for `@shade/files`.
*
* The default `shade.files.client(peer)` ships RPC envelopes via
* `Shade.send` + `Shade.deliverControlEnvelope`, which means the
* server has to be able to call back outbound to the client. That
* doesn't work for browser tabs (no inbound HTTP listener). This
* client posts each RPC envelope to a single server endpoint and
* reads the encrypted response from the same HTTP response — pure
* request-response, no inbound channel required.
*
* Inline payloads only (≤ 256 KiB). For larger reads/writes, use the
* stateful path: `shade.files.client(peer)` server-to-server, with
* `@shade/transfer` chunk routes for content I/O.
*
* @see {@link createFilesRpcRoute} for the matching server-side route.
*/
import type { ZodTypeAny } from 'zod';
import { decodeEnvelope, encodeEnvelope as encodeWireEnvelope } from '@shade/proto';
import type { ShadeBridge } from '../integration/shade-bridge.js';
import {
encodeEnvelope as encodeRpcEnvelope,
tryParseEnvelope,
} from '../protocol/envelope-codec.js';
import {
KIND_CUSTOM_V1,
KIND_DELETE_V1,
KIND_GET_THUMBNAIL_V1,
KIND_LIST_V1,
KIND_MKDIR_V1,
KIND_MOVE_V1,
KIND_READ_V1,
KIND_STAT_V1,
KIND_WRITE_V1,
} from '../protocol/kinds.js';
import {
CustomArgsSchema,
CustomResultSchema,
DeleteArgsSchema,
DeleteResultSchema,
GetThumbnailArgsSchema,
GetThumbnailResultSchema,
ListArgsSchema,
ListResultSchema,
MkdirArgsSchema,
MkdirResultSchema,
MoveArgsSchema,
MoveResultSchema,
ReadArgsSchema,
ReadResultSchema,
StatArgsSchema,
StatResultSchema,
WriteArgsSchema,
WriteResultSchema,
type ListResult,
type MkdirResult,
type DeleteResult,
type MoveResult,
type StatResult,
type ThumbnailSize,
type WriteResult,
} from '../schemas/ops.js';
import {
fileErrorFromPayload,
CancelledError,
InternalFileError,
ConflictError,
} from '../schemas/errors.js';
import { buildRpcRequest } from '../protocol/rpc-builder.js';
import { decideInline, INLINE_THRESHOLD, type WriteSource } from './inline-threshold.js';
import { base64ToBytes, bytesToBase64 } from '../protocol/canonical.js';
import { startQueueDrainer, type QueueDrainerHandle } from './queue-drainer.js';
import {
createClientStreamsBridge,
type ClientStreamsBridge,
} from './streams-bridge.js';
import type {
FileClient,
ReadOpts,
ReadOutput,
ThumbnailResult,
WriteOpts,
CreateFileClientOptions,
BaseOpts,
} from './client.js';
export interface FilesHttpClientOptions
extends Omit<CreateFileClientOptions, 'streamsBridge'> {
/**
* Server endpoint that hosts `createFilesRpcRoute(...)`. Typically:
* `https://server.example.com/api/v1/shade-files/rpc`.
*/
rpcUrl: string;
/**
* Optional `fetch` override. Defaults to `globalThis.fetch`. Wire a
* custom `fetch` to thread auth-cookies, CSRF tokens, or
* service-worker interception.
*/
fetch?: typeof globalThis.fetch;
/**
* Extra HTTP headers applied to every RPC POST. Useful for app-level
* auth (CSRF, session cookies via custom header, etc.) — these are
* orthogonal to the ratchet authentication on the envelope itself.
*/
headers?: Record<string, string>;
/**
* Server endpoint that hosts `transferQueueRoute()`'s long-poll
* endpoint. Typically:
* `https://server.example.com/api/v1/shade-files/queue`.
*
* When supplied, the client starts a background long-poll that
* drains queued envelopes + chunks from the server and dispatches
* them via `shade.acceptTransferEnvelope`. This unlocks
* **streamed reads** (>256 KiB) for browser-style consumers.
*/
outboundQueueUrl?: string;
/**
* Base URL for outbound transfer routes (browser → server). Required
* alongside `outboundQueueUrl` to enable streamed writes. Typically:
* `https://server.example.com/api/v1/shade-files`.
*
* The client POSTs:
* - chunks to `<base>/v1/transfer/<streamId>/chunk`
* - control envelopes to `<base>/v1/transfer/control`
*/
transferBaseUrl?: string;
/**
* Long-poll block timeout, milliseconds. Default 30_000. Server
* clamps to its own `maxBlockMs` (default 55_000).
*/
queueBlockMs?: number;
}
interface RoundTripOpts {
signal?: AbortSignal;
timeoutMs?: number;
idempotencyKey?: string;
}
/**
* Create a request-response `FileClient` bound to `peerAddress` and a
* server-side RPC URL. The session must already be established
* (via `shade.initSessionFromBundle(peerAddress, bundle)` or an
* incoming first-message). Otherwise the first RPC will fail with
* "decrypt failed: no session for peer".
*
* When `outboundQueueUrl` + `transferBaseUrl` are supplied, the
* client also unlocks **streamed reads/writes** for files larger than
* the inline threshold (256 KiB). The browser polls the server's
* outbound queue for chunks/envelopes and POSTs its own outbound
* chunks to the server's transfer-receive routes.
*/
export function createFilesHttpClient(
shade: ShadeBridge,
peerAddress: string,
options: FilesHttpClientOptions,
): FileClient {
const rpcUrl = options.rpcUrl;
const fetchFn = options.fetch ?? globalThis.fetch.bind(globalThis);
const extraHeaders = options.headers ?? {};
const defaultTimeoutMs = options.defaultTimeoutMs ?? 30_000;
const ioTimeoutMs = options.ioTimeoutMs ?? 60_000;
const signRequest = options.signRequest;
const senderAddress = shade.myAddress;
// ─── Streamed-mode bootstrap ─────────────────────────────────
//
// When `outboundQueueUrl` is supplied, the client:
// 1. Configures `shade.configureTransfers(...)` so outbound
// chunks POST to `<transferBaseUrl>/v1/transfer/<streamId>/chunk`
// and outbound control envelopes POST to
// `<transferBaseUrl>/v1/transfer/control`.
// 2. Spawns a streams-bridge so streamed reads can be awaited.
// 3. Starts a long-poll drainer that pulls queued envelopes +
// chunks from the server and dispatches via
// `shade.acceptTransferEnvelope`.
let drainer: QueueDrainerHandle | null = null;
let streamsBridgePromise: Promise<ClientStreamsBridge> | null = null;
let streamsBridge: ClientStreamsBridge | null = null;
if (options.outboundQueueUrl !== undefined) {
const outboundQueueUrl = options.outboundQueueUrl;
if (options.transferBaseUrl === undefined) {
throw new Error(
'createFilesHttpClient: outboundQueueUrl was supplied without transferBaseUrl. Pass `transferBaseUrl` (the server prefix that hosts /v1/transfer/...) so outbound chunks have a destination.',
);
}
if (shade.configureTransfers === undefined) {
throw new Error(
'createFilesHttpClient: shade.configureTransfers is required for streamed mode (the underlying ShadeBridge must surface it).',
);
}
const transferBaseUrl = options.transferBaseUrl.replace(/\/$/, '');
shade.configureTransfers({
resolveBaseUrl: async (peer) => {
if (peer !== peerAddress) {
throw new Error(
`httpClient is bound to peer "${peerAddress}" — refusing to resolve outgoing chunks for "${peer}" without a multi-peer registry. Use shade.files.client(peer) for server-to-server multi-peer.`,
);
}
return transferBaseUrl;
},
});
// Build the streams-bridge eagerly. The engine's incoming-transfer
// subscription has to be in place BEFORE the drainer dispatches the
// first stream-init envelope, otherwise the engine emits the
// IncomingTransfer to zero handlers and the read silently never
// accepts. We kick off the drainer once the bridge has subscribed.
streamsBridgePromise = createClientStreamsBridge(shade).then((bridge) => {
streamsBridge = bridge;
drainer = startQueueDrainer(shade, {
outboundQueueUrl,
peerAddress,
senderAddress,
...(options.fetch !== undefined ? { fetch: options.fetch } : {}),
...(options.headers !== undefined ? { headers: options.headers } : {}),
...(options.queueBlockMs !== undefined ? { blockMs: options.queueBlockMs } : {}),
});
return bridge;
});
// Surface bridge-construction failures eagerly via a rejected
// promise the next read/write picks up.
streamsBridgePromise.catch(() => {
/* observed via getStreamsBridge() */
});
}
async function getStreamsBridge(): Promise<ClientStreamsBridge> {
if (streamsBridge !== null) return streamsBridge;
if (streamsBridgePromise === null) {
throw new ConflictError(
`http RPC client supports inline writes/reads only (≤ ${INLINE_THRESHOLD} bytes) — pass { outboundQueueUrl, transferBaseUrl } to enable streamed transfers.`,
);
}
streamsBridge = await streamsBridgePromise;
return streamsBridge;
}
/**
* Encrypt + POST + decrypt + parse one RPC round-trip.
*
* Throws a typed `FileError` subclass when the server returns an
* encrypted `RpcError`, or `InternalFileError` for transport-level
* failures (network, 4xx/5xx, malformed body).
*/
async function roundTrip<TResult>(
kind: string,
op: 'list' | 'stat' | 'mkdir' | 'delete' | 'move' | 'read' | 'write' | 'getThumbnail' | 'custom',
args: unknown,
resultSchema: ZodTypeAny,
opts: RoundTripOpts | undefined,
): Promise<TResult> {
const requestEnv = await buildRpcRequest({
senderAddress,
kind,
op,
args,
...(opts?.idempotencyKey !== undefined ? { idempotencyKey: opts.idempotencyKey } : {}),
...(signRequest !== undefined ? { signRequest } : {}),
});
const plaintext = encodeRpcEnvelope(requestEnv);
const ratchetEnvelope = await shade.send(peerAddress, plaintext);
const wireBytes = encodeWireEnvelope(ratchetEnvelope);
const ac = new AbortController();
const timeoutMs = opts?.timeoutMs ?? defaultTimeoutMs;
const timer = setTimeout(
() => ac.abort(new Error(`RPC timeout after ${timeoutMs}ms`)),
timeoutMs,
);
(timer as unknown as { unref?: () => void }).unref?.();
if (opts?.signal !== undefined) {
const userSignal = opts.signal;
if (userSignal.aborted) ac.abort(userSignal.reason);
else userSignal.addEventListener('abort', () => ac.abort(userSignal.reason), { once: true });
}
let response: Response;
try {
// Wrap the wire bytes in a Blob so the body type satisfies the
// common-denominator `BodyInit` across DOM, Bun, and node-fetch
// (some runtimes accept `Uint8Array` directly, others don't).
// Cast through `unknown` because TS's `bun-types` and `lib.dom`
// disagree about whether `Uint8Array<ArrayBufferLike>` is itself
// a `BlobPart`; the runtime accepts it on every platform.
response = await fetchFn(rpcUrl, {
method: 'POST',
body: new Blob([wireBytes as unknown as ArrayBuffer]),
signal: ac.signal,
headers: {
'Content-Type': 'application/octet-stream',
'X-Shade-Sender-Address': senderAddress,
...extraHeaders,
},
});
} catch (err) {
clearTimeout(timer);
if ((err as Error).name === 'AbortError') {
throw new CancelledError(`RPC ${kind} aborted: ${(err as Error).message}`);
}
throw new InternalFileError(`RPC ${kind} fetch failed: ${(err as Error).message}`);
}
clearTimeout(timer);
if (!response.ok) {
let body: { error?: string } | null = null;
try {
body = (await response.json()) as { error?: string };
} catch {
/* server emitted non-JSON body */
}
throw new InternalFileError(
`RPC ${kind}${response.status} ${response.statusText}: ${
body?.error ?? '(no error body)'
}`,
);
}
const ab = await response.arrayBuffer();
if (ab.byteLength === 0) {
throw new InternalFileError(`RPC ${kind}: empty response body`);
}
let responseRatchet;
try {
responseRatchet = decodeEnvelope(new Uint8Array(ab));
} catch (err) {
throw new InternalFileError(
`RPC ${kind}: response body is not a valid wire envelope: ${(err as Error).message}`,
);
}
let responsePlaintext: string;
try {
responsePlaintext = await shade.receive(peerAddress, responseRatchet);
} catch (err) {
throw new InternalFileError(
`RPC ${kind}: response decrypt failed: ${(err as Error).message}`,
);
}
const classified = tryParseEnvelope(responsePlaintext);
if (classified === null) {
throw new InternalFileError(
`RPC ${kind}: response plaintext is not a valid @shade/files envelope`,
);
}
if (classified.kind === 'error') {
throw fileErrorFromPayload(classified.envelope.error);
}
if (classified.kind !== 'response') {
throw new InternalFileError(
`RPC ${kind}: unexpected response envelope kind: ${classified.kind}`,
);
}
if (classified.envelope.id !== requestEnv.id) {
throw new InternalFileError(
`RPC ${kind}: response correlation id mismatch (got ${classified.envelope.id}, expected ${requestEnv.id})`,
);
}
return resultSchema.parse(classified.envelope.result) as TResult;
}
return {
async list(path, opts): Promise<ListResult> {
const args = ListArgsSchema.parse({
path,
...(opts?.cursor !== undefined ? { cursor: opts.cursor } : {}),
...(opts?.pageSize !== undefined ? { pageSize: opts.pageSize } : {}),
...(opts?.filter !== undefined ? { filter: opts.filter } : {}),
});
return await roundTrip<ListResult>(
KIND_LIST_V1,
'list',
args,
ListResultSchema,
opts,
);
},
async stat(path, opts): Promise<StatResult> {
const args = StatArgsSchema.parse({ path });
return await roundTrip<StatResult>(KIND_STAT_V1, 'stat', args, StatResultSchema, opts);
},
async mkdir(path, opts): Promise<MkdirResult> {
const args = MkdirArgsSchema.parse({
path,
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
});
return await roundTrip<MkdirResult>(
KIND_MKDIR_V1,
'mkdir',
args,
MkdirResultSchema,
opts,
);
},
async delete(path, opts): Promise<DeleteResult> {
const args = DeleteArgsSchema.parse({
path,
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
});
return await roundTrip<DeleteResult>(
KIND_DELETE_V1,
'delete',
args,
DeleteResultSchema,
opts,
);
},
async move(src, dst, opts): Promise<MoveResult> {
const args = MoveArgsSchema.parse({
src,
dst,
...(opts?.overwrite !== undefined ? { overwrite: opts.overwrite } : {}),
});
return await roundTrip<MoveResult>(KIND_MOVE_V1, 'move', args, MoveResultSchema, opts);
},
async read(path, opts: ReadOpts = {}): Promise<ReadOutput> {
const args = ReadArgsSchema.parse({
path,
...(opts.range !== undefined ? { range: opts.range } : {}),
...(opts.preferInline !== undefined ? { preferInline: opts.preferInline } : {}),
});
const wire = await roundTrip<import('../schemas/ops.js').ReadResult>(
KIND_READ_V1,
'read',
args,
ReadResultSchema,
opts,
);
if (wire.kind === 'inline') {
const bytes = base64ToBytes(wire.bytesB64);
const out: ReadOutput = {
kind: 'inline',
bytes,
size: wire.size,
sha256: wire.sha256,
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
};
return out;
}
// Streamed read — only supported when the queue drainer is wired.
if (drainer === null) {
throw new InternalFileError(
`http RPC client received a streamed read (size ${wire.size}) but is in inline-only mode. Pass { outboundQueueUrl, transferBaseUrl } when constructing the client to enable streamed reads.`,
);
}
const bridge = await getStreamsBridge();
const bridgeSignal = opts.signal ?? new AbortController().signal;
const parked = await bridge.awaitRead(wire.streamId, {
expectedFrom: peerAddress,
signal: bridgeSignal,
timeoutMs: ioTimeoutMs,
});
const out: ReadOutput = {
kind: 'streams',
stream: parked.readable,
size: wire.size,
sha256: wire.sha256,
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
done: async () => {
await parked.done;
},
};
return out;
},
async write(path, input: WriteSource, opts: WriteOpts = {}): Promise<WriteResult> {
const decision = await decideInline(input);
const overwrite = opts.overwrite ?? false;
const contentType = opts.contentType ?? decision.contentType;
if (decision.kind === 'inline' || opts.forceInline === true) {
const bytes = decision.kind === 'inline' ? decision.bytes : null;
if (bytes === null) {
// forceInline === true with a streams-typed decision —
// decideInline always produced a `streams` shape because the
// input was a bare ReadableStream. We can't drain a stream
// synchronously here without a streams-bridge.
throw new ConflictError(
'http RPC client cannot forceInline a streamed input — pass a Uint8Array / Blob, or pre-buffer the stream.',
);
}
if (bytes.byteLength > INLINE_THRESHOLD) {
throw new ConflictError(
`inline write exceeds ${INLINE_THRESHOLD}-byte threshold (got ${bytes.byteLength}); pass forceInline=true to override`,
);
}
const args = WriteArgsSchema.parse({
kind: 'inline',
path,
bytesB64: bytesToBase64(bytes),
...(contentType !== undefined ? { contentType } : {}),
overwrite,
});
return await roundTrip<WriteResult>(
KIND_WRITE_V1,
'write',
args,
WriteResultSchema,
opts,
);
}
// Streamed write — requires the queue drainer + streams-bridge.
if (drainer === null) {
throw new ConflictError(
`http RPC client supports inline writes only (≤ ${INLINE_THRESHOLD} bytes). The supplied input was promoted to streams (size ${decision.size ?? 'unknown'}). Pass { outboundQueueUrl, transferBaseUrl } to enable streamed writes.`,
);
}
const bridge = await getStreamsBridge();
const size = decision.size;
if (size === undefined) {
throw new ConflictError(
'streams write requires a known plaintext size; pass `{ stream, size }` instead of a bare ReadableStream',
);
}
const { writeId, handle } = await bridge.initiateWrite({
peer: peerAddress,
stream: decision.stream,
size,
...(contentType !== undefined ? { contentType } : {}),
name: path,
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
});
const args = WriteArgsSchema.parse({
kind: 'streams',
path,
size,
...(contentType !== undefined ? { contentType } : {}),
overwrite,
writeId,
});
try {
const [result] = await Promise.all([
roundTrip<WriteResult>(KIND_WRITE_V1, 'write', args, WriteResultSchema, opts),
handle.done(),
]);
return result;
} catch (err) {
await handle.abort('rpc-failed').catch(() => undefined);
throw err;
}
},
async getThumbnail(path, size: ThumbnailSize, opts): Promise<ThumbnailResult> {
const args = GetThumbnailArgsSchema.parse({
path,
size,
...(opts?.format !== undefined ? { format: opts.format } : {}),
});
const raw = await roundTrip<import('../schemas/ops.js').GetThumbnailResult>(
KIND_GET_THUMBNAIL_V1,
'getThumbnail',
args,
GetThumbnailResultSchema,
opts,
);
return {
bytes: base64ToBytes(raw.bytesB64),
format: raw.format,
width: raw.width,
height: raw.height,
sha256: raw.sha256,
};
},
async custom(name, args, opts?: BaseOpts): Promise<unknown> {
const wireArgs = CustomArgsSchema.parse({ name, args });
return await roundTrip(KIND_CUSTOM_V1, 'custom', wireArgs, CustomResultSchema, opts);
},
close(): void {
// Stop the long-poll drainer + tear down the streams-bridge if
// we built one. Idempotent — safe to call multiple times.
drainer?.stop();
drainer = null;
if (streamsBridge !== null) {
void streamsBridge.destroy().catch(() => undefined);
streamsBridge = null;
}
streamsBridgePromise = null;
},
} as FileClient;
}

View File

@@ -161,15 +161,33 @@ async function peekStream(stream: ReadableStream<Uint8Array>): Promise<InlineDec
} }
} }
interface MinimalReader { /**
read(): Promise<{ value: Uint8Array | undefined; done: boolean }>; * Structural mirror of WHATWG `ReadableStreamDefaultReader<Uint8Array>`.
*
* The disjoint union shape with `value?: T | undefined` is the lowest
* common denominator across every lib environment we care about:
* - `bun-types` emits `{ done: true; value?: undefined }`
* - `lib.dom` emits `{ done: true; value?: T }`
* - `node:stream/web` emits the union form
*
* `value?: T | undefined` is assignable from all three. A flat
* `{ value?: T; done: boolean }` is rejected by
* `exactOptionalPropertyTypes` because the present branches require
* `value: T`. Defining it as an explicit union avoids the trap.
*/
type MinimalReadResult<T> =
| { done: false; value: T }
| { done: true; value?: T | undefined };
interface MinimalReader<T> {
read(): Promise<MinimalReadResult<T>>;
cancel(reason?: unknown): Promise<void>; cancel(reason?: unknown): Promise<void>;
releaseLock(): void; releaseLock(): void;
} }
function reconstructStream( function reconstructStream(
prefix: Uint8Array[], prefix: Uint8Array[],
reader: MinimalReader, reader: MinimalReader<Uint8Array>,
): ReadableStream<Uint8Array> { ): ReadableStream<Uint8Array> {
let prefixIdx = 0; let prefixIdx = 0;
return new ReadableStream<Uint8Array>({ return new ReadableStream<Uint8Array>({

View File

@@ -0,0 +1,172 @@
/**
* Browser-side drainer for the pull-mode outbound queue.
*
* Background task that long-polls the server's `/queue` endpoint,
* decodes each event, and dispatches it into the consumer's Shade
* instance via `shade.acceptTransferEnvelope`. Same wire-shape as the
* server-to-server case where the engine receives chunks via direct
* HTTP POSTs — we just flip the direction so the browser pulls
* instead of accepts.
*/
import type { ShadeBridge } from '../integration/shade-bridge.js';
export interface QueueDrainerOptions {
/**
* Server endpoint that hosts `transferQueueRoute()`. Typically:
* `https://server.example.com/api/v1/shade-files/queue`.
*/
outboundQueueUrl: string;
/** Peer the queue is pulled FROM (the server's address). */
peerAddress: string;
/** Address we identify ourselves with on the long-poll. */
senderAddress: string;
/** Optional `fetch` override. Default `globalThis.fetch`. */
fetch?: typeof globalThis.fetch;
/** Extra headers applied to every poll request. */
headers?: Record<string, string>;
/**
* Long-poll request timeout (server-side block). Default 30_000.
* Server clamps to its own `maxBlockMs` (default 55_000).
*/
blockMs?: number;
/**
* Backoff after a network error before re-polling. Default 2_000.
* Doubles up to `maxBackoffMs` on consecutive failures.
*/
initialBackoffMs?: number;
maxBackoffMs?: number;
/**
* Called when a poll cycle fails. Defaults to logging via `console.error`.
* Throwing from this hook does NOT stop the drainer — it backs off
* and retries.
*/
onError?: (err: unknown) => void;
}
export interface QueueDrainerHandle {
/** Stop the drainer. Pending fetch is aborted; the loop exits. */
stop(): void;
/** Promise that resolves once the drainer has fully stopped. */
stopped: Promise<void>;
}
interface PolledEvent {
id: number;
timestampMs: number;
kind: 'envelope' | 'chunk';
bytesB64: string;
meta?: { streamId: string; laneId: number; seq: number };
}
interface PollResponseBody {
events: PolledEvent[];
nextSince: number;
}
const DEFAULT_BLOCK_MS = 30_000;
const DEFAULT_INITIAL_BACKOFF_MS = 2_000;
const DEFAULT_MAX_BACKOFF_MS = 30_000;
/**
* Start a long-poll loop that drains queued envelopes + chunks from
* the server and dispatches them into the local Shade instance.
*
* Returns a handle the caller can use to stop the drainer when the
* `httpClient` is closed (e.g. on tab unload).
*/
export function startQueueDrainer(
shade: ShadeBridge,
options: QueueDrainerOptions,
): QueueDrainerHandle {
if (shade.acceptTransferEnvelope === undefined) {
throw new Error(
'startQueueDrainer: shade.acceptTransferEnvelope is required for pull-mode streams. The supplied ShadeBridge implementation must surface it.',
);
}
const accept = shade.acceptTransferEnvelope.bind(shade);
const fetchFn = options.fetch ?? globalThis.fetch.bind(globalThis);
const blockMs = options.blockMs ?? DEFAULT_BLOCK_MS;
const onError = options.onError ?? ((err: unknown) => console.error('[shade.files queue-drainer]', err));
const initialBackoffMs = options.initialBackoffMs ?? DEFAULT_INITIAL_BACKOFF_MS;
const maxBackoffMs = options.maxBackoffMs ?? DEFAULT_MAX_BACKOFF_MS;
const ac = new AbortController();
let stopped = false;
let resolveStopped!: () => void;
const stoppedPromise = new Promise<void>((r) => {
resolveStopped = r;
});
void (async () => {
let since = 0;
let backoff = initialBackoffMs;
while (!stopped) {
try {
const response = await fetchFn(options.outboundQueueUrl, {
method: 'POST',
signal: ac.signal,
headers: {
'Content-Type': 'application/json',
'X-Shade-Sender-Address': options.senderAddress,
...(options.headers ?? {}),
},
body: JSON.stringify({ since, blockMs }),
});
if (!response.ok) {
throw new Error(`queue poll → ${response.status} ${response.statusText}`);
}
const body = (await response.json()) as PollResponseBody;
if (Array.isArray(body.events) && body.events.length > 0) {
for (const event of body.events) {
if (stopped) break;
try {
const bytes = base64ToBytes(event.bytesB64);
await accept(options.peerAddress, bytes);
} catch (err) {
// Per-event dispatch failure should not kill the loop —
// resume picks up missing chunks via @shade/transfer's
// built-in lane-resume protocol.
onError(err);
}
}
}
since = typeof body.nextSince === 'number' ? body.nextSince : since;
backoff = initialBackoffMs;
} catch (err) {
if (stopped || ac.signal.aborted) break;
onError(err);
// Exponential backoff with jitter — caps at maxBackoffMs.
const jitter = Math.floor(Math.random() * Math.min(backoff, 1_000));
await new Promise<void>((r) => {
const t = setTimeout(r, backoff + jitter);
(t as unknown as { unref?: () => void }).unref?.();
ac.signal.addEventListener(
'abort',
() => {
clearTimeout(t);
r();
},
{ once: true },
);
});
backoff = Math.min(maxBackoffMs, backoff * 2);
}
}
resolveStopped();
})();
return {
stop(): void {
if (stopped) return;
stopped = true;
ac.abort(new Error('queue drainer stopped'));
},
stopped: stoppedPromise,
};
}
function base64ToBytes(b64: string): Uint8Array {
const bin = atob(b64);
const out = new Uint8Array(bin.length);
for (let i = 0; i < bin.length; i++) out[i] = bin.charCodeAt(i);
return out;
}

View File

@@ -102,7 +102,16 @@ export async function createClientStreamsBridge(
const readStreamId = incoming.metadata.userMetadata?.[META_KEY_READ_STREAM_ID]; const readStreamId = incoming.metadata.userMetadata?.[META_KEY_READ_STREAM_ID];
if (readStreamId === undefined) return; if (readStreamId === undefined) return;
const ts = new TransformStream<Uint8Array, Uint8Array>(); // Generous HWM so the receiver-side write loop (drainer →
// engine.receiveChunk → sink.write) doesn't stall on backpressure
// before the consumer's reader is wired up. The reader still
// applies its own backpressure once it's consuming, but we no
// longer race fs.read's await on stream-init against the consumer
// attaching its reader.
const ts = new TransformStream<Uint8Array, Uint8Array>(undefined, undefined, {
highWaterMark: 64,
size: (chunk?: Uint8Array) => (chunk === undefined ? 0 : 1),
});
let handle: TransferHandle; let handle: TransferHandle;
try { try {
handle = await incoming.accept({ handle = await incoming.accept({
@@ -142,13 +151,14 @@ export async function createClientStreamsBridge(
} }
parked.set(readStreamId, arrival); parked.set(readStreamId, arrival);
setTimeout(() => { const t = setTimeout(() => {
const stale = parked.get(readStreamId); const stale = parked.get(readStreamId);
if (stale === arrival) { if (stale === arrival) {
parked.delete(readStreamId); parked.delete(readStreamId);
void handle.abort('rpc-timeout').catch(() => undefined); void handle.abort('rpc-timeout').catch(() => undefined);
} }
}, parkedReadTtlMs).unref?.(); }, parkedReadTtlMs);
(t as unknown as { unref?: () => void }).unref?.();
}); });
function cleanupWaiter(w: PendingReadWaiter): void { function cleanupWaiter(w: PendingReadWaiter): void {

View File

@@ -183,6 +183,24 @@ export { MAX_SIGNATURE_AGE_MS } from './server/handler.js';
export { createFilesNamespace } from './integration/files-namespace.js'; export { createFilesNamespace } from './integration/files-namespace.js';
export type { FilesNamespace } from './integration/files-namespace.js'; export type { FilesNamespace } from './integration/files-namespace.js';
// Request-response HTTP transport — for browser-style consumers
// (one HTTP POST per RPC, no inbound channel needed). See
// `docs/files.md § HTTP RPC`.
export { createFilesRpcRoute } from './server/rpc-route.js';
export type { FilesRpcRouteOptions } from './server/rpc-route.js';
export { createFilesHttpClient } from './client/http-client.js';
export type { FilesHttpClientOptions } from './client/http-client.js';
export { startQueueDrainer } from './client/queue-drainer.js';
export type {
QueueDrainerHandle,
QueueDrainerOptions,
} from './client/queue-drainer.js';
// Shared structural surface @shade/files needs from a Shade instance —
// exposed so consumers building custom Shade-shaped bridges can verify
// they implement every required member.
export type { ShadeBridge } from './integration/shade-bridge.js';
// Integration helpers — wire handler + pending registry onto a channel // Integration helpers — wire handler + pending registry onto a channel
export { attachFileHandler } from './integration/wire-server.js'; export { attachFileHandler } from './integration/wire-server.js';
export { attachClientRouting } from './integration/wire-client.js'; export { attachClientRouting } from './integration/wire-client.js';

View File

@@ -4,13 +4,16 @@
* so a single Shade can simultaneously serve files AND consume them from * so a single Shade can simultaneously serve files AND consume them from
* peers without paying the setup cost twice. * peers without paying the setup cost twice.
*/ */
import type { Shade } from '@shade/sdk'; import type { Hono } from 'hono';
import type { ShadeBridge } from './shade-bridge.js';
import { import {
attachClientRouting, attachClientRouting,
attachFileHandler, attachFileHandler,
createClientStreamsBridge, createClientStreamsBridge,
createFileClient, createFileClient,
createFileHandler, createFileHandler,
createFilesHttpClient,
createFilesRpcRoute,
createServerStreamsBridge, createServerStreamsBridge,
PendingRpcRegistry, PendingRpcRegistry,
ShadeFileRpcChannel, ShadeFileRpcChannel,
@@ -19,22 +22,79 @@ import {
type FileClient, type FileClient,
type FileHandler, type FileHandler,
type FileHandlerConfig, type FileHandlerConfig,
type FilesHttpClientOptions,
type FilesRpcRouteOptions,
type ServerStreamsBridge, type ServerStreamsBridge,
} from '../index.js'; } from '../index.js';
import { IdempotencyCache } from '../server/idempotency-cache.js'; import { IdempotencyCache } from '../server/idempotency-cache.js';
export interface ServeOptions {
/**
* Skip the streams bridge setup. Required for deployments that only
* use the HTTP RPC route ({@link FilesNamespace.rpcRoute}) — those
* deployments don't need to configure `@shade/transfer` because the
* RPC route only services inline payloads (≤ 256 KiB). Without this
* flag, `serve()` calls `createServerStreamsBridge(shade)` which
* eagerly instantiates the transfer engine and fails when
* `configureTransfers({ resolveBaseUrl })` has not been called.
*
* Default: `false` (build the streams bridge — full server-to-server
* stack with streamed reads/writes).
*/
inlineOnly?: boolean;
}
export interface FilesNamespace { export interface FilesNamespace {
/** /**
* Register a file handler. Throws if a handler is already attached on * Register a file handler. Throws if a handler is already attached on
* this Shade — only one server per Shade. The returned function detaches * this Shade — only one server per Shade. The returned function detaches
* the handler and tears down its idempotency / retention timers. * the handler and tears down its idempotency / retention timers.
*
* Pass `{ inlineOnly: true }` for HTTP-RPC-only deployments to skip
* the streams-bridge setup (and the implied `configureTransfers`
* pre-condition).
*/ */
serve(handler: FileHandlerConfig): Promise<() => Promise<void>>; serve(
handler: FileHandlerConfig,
options?: ServeOptions,
): Promise<() => Promise<void>>;
/** /**
* Build a typed file client for `peer`. Multiple concurrent clients to * Build a typed file client for `peer`. Multiple concurrent clients to
* different peers share the same channel + streams bridge. * different peers share the same channel + streams bridge.
*
* Use this for **server-to-server** deployments where both peers can
* receive inbound HTTP. For browser clients (no inbound listener),
* use {@link httpClient} instead.
*/ */
client(peer: string, opts?: Omit<CreateFileClientOptions, 'streamsBridge'>): Promise<FileClient>; client(peer: string, opts?: Omit<CreateFileClientOptions, 'streamsBridge'>): Promise<FileClient>;
/**
* Build a request-response `FileClient` for browser-style consumers.
* Each RPC is one HTTP POST to the supplied `rpcUrl`; the encrypted
* response rides back in the same response body. No inbound channel
* required on the client side.
*
* Inline payloads only (≤ 256 KiB). Streamed reads/writes throw a
* clear error directing callers to {@link client} instead.
*
* Pre-condition: the session for `peer` must already be established
* (typically via `shade.initSessionFromBundle(peer, bundle)`).
*/
httpClient(peer: string, opts: FilesHttpClientOptions): FileClient;
/**
* Mount the server-side request-response RPC route. Returns a Hono
* app exposing `POST /rpc` that accepts encrypted file-RPC envelopes
* and returns encrypted responses in the same HTTP roundtrip.
*
* Mount under any base path:
* ```ts
* app.route('/api/v1/shade-files', shade.files.rpcRoute());
* ```
*
* Requires `shade.files.serve(...)` to have been called first —
* the route dispatches incoming requests through the attached
* handler.
*/
rpcRoute(opts?: FilesRpcRouteOptions): Hono;
/** Tear down channel + bridges. After destroy(), serve()/client() throw. */ /** Tear down channel + bridges. After destroy(), serve()/client() throw. */
destroy(): Promise<void>; destroy(): Promise<void>;
} }
@@ -54,7 +114,7 @@ interface NamespaceState {
* Construct a `FilesNamespace` bound to a Shade instance. The SDK's * Construct a `FilesNamespace` bound to a Shade instance. The SDK's
* `Shade.files` getter calls this lazily and memoizes the result. * `Shade.files` getter calls this lazily and memoizes the result.
*/ */
export function createFilesNamespace(shade: Shade): FilesNamespace { export function createFilesNamespace(shade: ShadeBridge): FilesNamespace {
const state: NamespaceState = { const state: NamespaceState = {
channel: new ShadeFileRpcChannel(shade), channel: new ShadeFileRpcChannel(shade),
pending: new PendingRpcRegistry(), pending: new PendingRpcRegistry(),
@@ -71,24 +131,39 @@ export function createFilesNamespace(shade: Shade): FilesNamespace {
} }
return { return {
async serve(handlerConfig) { async serve(handlerConfig, options = {}) {
ensureAlive(); ensureAlive();
if (state.serverHandler !== null) { if (state.serverHandler !== null) {
throw new Error('FilesNamespace: a handler is already registered (one per Shade)'); throw new Error('FilesNamespace: a handler is already registered (one per Shade)');
} }
// Lazy server-side streams bridge. // Lazy server-side streams bridge — skip when the deployment is
if (state.serverBridge === null) { // HTTP-RPC-only and does not need `@shade/transfer` wired up.
if (!options.inlineOnly && state.serverBridge === null) {
state.serverBridge = await createServerStreamsBridge(shade); state.serverBridge = await createServerStreamsBridge(shade);
} }
const inheritedObservability = shade.getObservability?.(); const inheritedObservability = shade.getObservability?.();
const handler = createFileHandler(shade, { const handler = createFileHandler(shade, {
...handlerConfig, ...handlerConfig,
streamsBridge: state.serverBridge, ...(state.serverBridge !== null
? { streamsBridge: state.serverBridge }
: {}),
...(handlerConfig.observability === undefined && inheritedObservability !== undefined ...(handlerConfig.observability === undefined && inheritedObservability !== undefined
? { observability: inheritedObservability } ? { observability: inheritedObservability }
: {}), : {}),
}); });
const detach = attachFileHandler(state.channel, handler); // In inlineOnly mode, the rpc-route is the sole inbound path —
// do NOT also subscribe the channel's onMessage handler to this
// file handler, because that would cause every incoming request
// to be dispatched twice (once by the rpc-route's direct call,
// once by the channel's onMessage handler) and the channel-side
// response would attempt an outbound POST via
// `deliverControlEnvelope`, which is exactly the path that fails
// for browser clients.
const detach: () => void = options.inlineOnly
? () => {
/* no channel subscription to detach */
}
: attachFileHandler(state.channel, handler);
state.serverHandler = handler; state.serverHandler = handler;
state.serverDetach = detach; state.serverDetach = detach;
@@ -131,6 +206,21 @@ export function createFilesNamespace(shade: Shade): FilesNamespace {
}); });
}, },
httpClient(peer, opts) {
ensureAlive();
return createFilesHttpClient(shade, peer, opts);
},
rpcRoute(opts = {}) {
ensureAlive();
if (state.serverHandler === null) {
throw new Error(
'FilesNamespace.rpcRoute(): no handler attached. Call shade.files.serve(...) before mounting the RPC route.',
);
}
return createFilesRpcRoute(shade, state.serverHandler, opts);
},
async destroy() { async destroy() {
if (state.destroyed) return; if (state.destroyed) return;
state.destroyed = true; state.destroyed = true;

View File

@@ -0,0 +1,94 @@
/**
* Structural surface @shade/files needs from a Shade instance.
*
* Defining this locally — instead of `import type { Shade } from '@shade/sdk'`
* — breaks the @shade/sdk ↔ @shade/files dependency cycle. Without this
* break, a consumer that installs @shade/sdk from a registry ends up with
* two distinct `Shade` classes in `node_modules` (one from
* `@shade/sdk/node_modules/@shade/files/.../Shade`, one from
* `@shade/sdk/Shade`). TypeScript treats them as nominally different types,
* raising `this is not assignable to Shade` from inside SDK methods that
* pass `this` into `createFilesNamespace`.
*
* The Shade class structurally implements every member listed below, so
* `createFilesNamespace(this)` from the SDK side compiles regardless of
* how many copies of @shade/sdk a consumer's package manager installs.
*
* Member signatures match Shade's exactly so this is a structural
* subtype, not a parallel API.
*/
import type { ShadeEnvelope } from '@shade/core';
import type {
IncomingTransfer,
TransferHandle,
TransferOptions,
} from '@shade/transfer';
import type { ObservabilityHook } from '@shade/observability';
export interface ShadeBridge {
/** Address that names this Shade instance to peers. */
readonly myAddress: string;
/** Encrypt + send `plaintext` to `peer`; returns the wire envelope. */
send(peer: string, plaintext: string): Promise<ShadeEnvelope>;
/**
* Decrypt an inbound envelope from `peer` and return the plaintext.
* Used by the request-response RPC route on the server side.
*/
receive(peer: string, envelope: ShadeEnvelope): Promise<string>;
/**
* Subscribe to incoming ratchet plaintext. Returns an unsubscribe.
* Handlers may be sync or async; async handlers are awaited in
* registration order.
*/
onMessage(
handler: (from: string, plaintext: string) => void | Promise<void>,
): () => void;
/**
* Upload bytes via the SDK's transfer engine. Required when the bridge
* is used with `streams` content I/O (read/write > 256 KiB).
*/
upload(opts: TransferOptions): Promise<TransferHandle>;
/** Subscribe to incoming transfers initiated by a peer. */
onIncomingTransfer(
handler: (incoming: IncomingTransfer) => void | Promise<void>,
): Promise<() => void>;
/** Fingerprint accessor for the trust-gate hooks. */
getFingerprintFor(peer: string): Promise<string>;
/**
* Optional inheritable observability bus. Files inherits the bus when
* the SDK passes one in via the namespace; otherwise files runs without
* observability hooks.
*/
getObservability?(): ObservabilityHook | undefined;
/** Optional control-envelope passthrough used by the WebRTC bridge. */
deliverControlEnvelope?(peer: string, envelope: ShadeEnvelope): Promise<void>;
/**
* Hand a freshly-decoded wire envelope (control or chunk) to the
* transfer engine. Required by the pull-mode HTTP client when it
* drains queued events from the server: each polled chunk / control
* envelope is dispatched here so the engine sees it just as if it
* had arrived via an HTTP POST on `/v1/transfer/...`.
*/
acceptTransferEnvelope?(from: string, env: ShadeEnvelope | Uint8Array): Promise<void>;
/**
* Configure the transfer stack. Called by the pull-mode HTTP client
* to point the browser's outgoing chunks + control envelopes at the
* server's transferQueueRoute mount. Optional because the
* server-to-server path uses a separate, app-driven configuration.
*/
configureTransfers?(opts: {
resolveBaseUrl?: (peerAddress: string) => Promise<string>;
transport?: unknown;
envelopeTransport?: unknown;
}): void;
}

View File

@@ -0,0 +1,126 @@
/**
* Shared RPC-request construction.
*
* Both the channel-based `FileClient` (`createFileClient`) and the
* HTTP-based `FilesHttpClient` build identical `RpcRequest` envelopes
* — they differ only in *transport* (channel.send → ratchet via
* Shade.send/onMessage vs HTTP POST → ratchet via single
* request-response). This module is the single source of truth for
* the wire shape so the two clients can never drift.
*/
import type { ZodTypeAny } from 'zod';
import {
KIND_DELETE_V1,
KIND_GET_THUMBNAIL_V1,
KIND_LIST_V1,
KIND_MKDIR_V1,
KIND_MOVE_V1,
KIND_READ_V1,
KIND_STAT_V1,
KIND_WRITE_V1,
MUTATION_OPS,
type StandardOp,
} from './kinds.js';
import { generateIdempotencyKey, generateRequestId } from './correlate.js';
import { canonicalRpcBytes, hashArgs } from './canonical.js';
import type { RpcRequest } from '../schemas/envelope.js';
export const KIND_BY_OP: Record<StandardOp, string> = {
list: KIND_LIST_V1,
stat: KIND_STAT_V1,
mkdir: KIND_MKDIR_V1,
delete: KIND_DELETE_V1,
move: KIND_MOVE_V1,
read: KIND_READ_V1,
write: KIND_WRITE_V1,
getThumbnail: KIND_GET_THUMBNAIL_V1,
};
export type SignRequest = (canonicalBytes: Uint8Array) => Promise<string> | string;
export interface BuildRpcRequestOptions {
/** Address that this RPC call originates from. */
senderAddress: string;
/** RPC kind — `KIND_*_V1` constants for standard ops, `KIND_CUSTOM_V1` otherwise. */
kind: string;
/** Op classifier so mutations get an auto-generated idempotency key. */
op: StandardOp | 'custom';
/** Validated args object. The caller is responsible for `Zod.parse(args)`. */
args: unknown;
/** Caller-supplied idempotency key. Mutations get one auto-generated. */
idempotencyKey?: string;
/** Optional Ed25519-style signer over the canonical bytes. */
signRequest?: SignRequest;
}
/**
* Build a single `RpcRequest` envelope. Generates `id`, `signedAt`,
* idempotency key (mutations only), and the canonical signature.
*/
export async function buildRpcRequest(
options: BuildRpcRequestOptions,
): Promise<RpcRequest> {
const { senderAddress, kind, op, args, signRequest } = options;
const requestId = generateRequestId();
const isMutation = MUTATION_OPS.has(op);
const idempotencyKey =
options.idempotencyKey ?? (isMutation ? generateIdempotencyKey() : undefined);
const signedAt = Date.now();
let sig = 'unsigned';
if (signRequest !== undefined) {
const canonical = canonicalRpcBytes({
address: senderAddress,
signedAt,
kind,
id: requestId,
argsHash: hashArgs(args),
});
sig = await signRequest(canonical);
}
const env: RpcRequest = {
kind,
id: requestId,
args,
...(idempotencyKey !== undefined ? { idempotencyKey } : {}),
sig,
signedAt,
};
return env;
}
/**
* Helper for the typed standard ops: validates args via the supplied Zod
* schema, calls {@link buildRpcRequest}, and forwards the result.
*
* The HTTP and channel clients both call this from each op-method to
* guarantee identical wire-shape. Custom ops use {@link buildRpcRequest}
* directly because they ship `KIND_CUSTOM_V1` plus runtime-validated args.
*/
export async function buildStandardRpcRequest<TArgs>(
schema: { parse(input: unknown): TArgs },
rawArgs: unknown,
options: Omit<BuildRpcRequestOptions, 'kind' | 'op' | 'args'> & {
op: StandardOp;
},
): Promise<{ request: RpcRequest; args: TArgs }> {
const args = schema.parse(rawArgs) as TArgs;
const kind = KIND_BY_OP[options.op];
const request = await buildRpcRequest({
...options,
kind,
op: options.op,
args,
});
return { request, args };
}
/**
* Validate a returned RpcResponse `result` against the supplied schema.
* Centralised so the HTTP and channel clients fail-loudly with the same
* error type when a server returns an off-spec response.
*/
export function parseResult<T>(schema: ZodTypeAny, raw: unknown): T {
return schema.parse(raw) as T;
}

View File

@@ -1,17 +1,23 @@
import React, { createContext, useContext, useMemo } from 'react'; import React, { createContext, useContext, useMemo } from 'react';
import type { Shade } from '@shade/sdk'; import type { ShadeBridge } from '../integration/shade-bridge.js';
import type { FilesNamespace } from '../integration/files-namespace.js'; import type { FilesNamespace } from '../integration/files-namespace.js';
export interface ShadeFilesContextValue { export interface ShadeFilesContextValue {
shade: Shade; shade: ShadeBridge;
files: FilesNamespace; files: FilesNamespace;
} }
const ShadeFilesContext = createContext<ShadeFilesContextValue | null>(null); const ShadeFilesContext = createContext<ShadeFilesContextValue | null>(null);
export interface ShadeFilesProviderProps { export interface ShadeFilesProviderProps {
/** Initialized `Shade` instance. `files` namespace is read off it lazily. */ /** Initialized `Shade` instance (or any `ShadeBridge`-shaped object). */
shade: Shade; shade: ShadeBridge;
/**
* The `FilesNamespace` to expose to children. Pass `shade.files` from
* `@shade/sdk`, or a `createFilesNamespace(...)` result for tests /
* custom bridges.
*/
files: FilesNamespace;
children: React.ReactNode; children: React.ReactNode;
} }
@@ -20,8 +26,8 @@ export interface ShadeFilesProviderProps {
* `<ShadeRuntimeProvider>` in `@shade/widgets` so file-RPC consumers * `<ShadeRuntimeProvider>` in `@shade/widgets` so file-RPC consumers
* don't pull in the widget tree. * don't pull in the widget tree.
*/ */
export function ShadeFilesProvider({ shade, children }: ShadeFilesProviderProps): React.ReactElement { export function ShadeFilesProvider({ shade, files, children }: ShadeFilesProviderProps): React.ReactElement {
const value = useMemo<ShadeFilesContextValue>(() => ({ shade, files: shade.files }), [shade]); const value = useMemo<ShadeFilesContextValue>(() => ({ shade, files }), [shade, files]);
return React.createElement(ShadeFilesContext.Provider, { value }, children); return React.createElement(ShadeFilesContext.Provider, { value }, children);
} }

View File

@@ -1,4 +1,4 @@
import type { Shade } from '@shade/sdk'; import type { ShadeBridge } from '../integration/shade-bridge.js';
import { import {
encodeEnvelope, encodeEnvelope,
looksLikeFileEnvelope, looksLikeFileEnvelope,
@@ -35,7 +35,7 @@ export class ShadeFileRpcChannel {
private readonly unsubscribe: () => void; private readonly unsubscribe: () => void;
private destroyed = false; private destroyed = false;
constructor(private readonly shade: Shade) { constructor(private readonly shade: ShadeBridge) {
this.unsubscribe = shade.onMessage(async (from, plaintext) => { this.unsubscribe = shade.onMessage(async (from, plaintext) => {
if (!looksLikeFileEnvelope(plaintext)) return; if (!looksLikeFileEnvelope(plaintext)) return;
const classified = tryParseEnvelope(plaintext); const classified = tryParseEnvelope(plaintext);
@@ -72,6 +72,11 @@ export class ShadeFileRpcChannel {
if (this.destroyed) throw new Error('ShadeFileRpcChannel: destroyed'); if (this.destroyed) throw new Error('ShadeFileRpcChannel: destroyed');
const plaintext = encodeEnvelope(envelope); const plaintext = encodeEnvelope(envelope);
const ratchetEnvelope = await this.shade.send(peerAddress, plaintext); const ratchetEnvelope = await this.shade.send(peerAddress, plaintext);
if (this.shade.deliverControlEnvelope === undefined) {
throw new Error(
'ShadeFileRpcChannel: shade.deliverControlEnvelope is required — call shade.configureTransfers({ resolveBaseUrl }) before using the files namespace.',
);
}
await this.shade.deliverControlEnvelope(peerAddress, ratchetEnvelope); await this.shade.deliverControlEnvelope(peerAddress, ratchetEnvelope);
} }

View File

@@ -1,4 +1,4 @@
import type { Shade } from '@shade/sdk'; import type { ShadeBridge } from '../integration/shade-bridge.js';
import type { StandardOp } from '../protocol/kinds.js'; import type { StandardOp } from '../protocol/kinds.js';
export type OpKind = StandardOp | `custom:${string}`; export type OpKind = StandardOp | `custom:${string}`;
@@ -42,7 +42,7 @@ export function buildOpContext<TArgs>(args: {
signal: AbortSignal; signal: AbortSignal;
idempotencyKey: string | undefined; idempotencyKey: string | undefined;
attemptNumber: number; attemptNumber: number;
shade: Shade; shade: ShadeBridge;
}): OpContext<TArgs> { }): OpContext<TArgs> {
return { return {
op: args.op, op: args.op,

View File

@@ -1,4 +1,4 @@
import type { Shade } from '@shade/sdk'; import type { ShadeBridge } from '../integration/shade-bridge.js';
import type { ZodTypeAny } from 'zod'; import type { ZodTypeAny } from 'zod';
import { import {
MUTATION_OPS, MUTATION_OPS,
@@ -215,7 +215,7 @@ const OP_SCHEMAS: Record<StandardOp, OpSchemaPair> = {
* via `Shade.files.serve(...)` in the SDK). * via `Shade.files.serve(...)` in the SDK).
*/ */
export function createFileHandler( export function createFileHandler(
shade: Shade, shade: ShadeBridge,
config: FileHandlerConfig, config: FileHandlerConfig,
): FileHandler { ): FileHandler {
const idempotency = new IdempotencyCache(config.idempotency); const idempotency = new IdempotencyCache(config.idempotency);

View File

@@ -0,0 +1,218 @@
/**
* Request-response RPC route for `@shade/files`.
*
* Mounts a single `POST /rpc` Hono endpoint that accepts an encrypted
* `RpcRequest` envelope, dispatches it through the file handler, and
* returns the encrypted `RpcResponse` (or `RpcError`) envelope in the
* SAME HTTP response.
*
* This is the browser-friendly transport: the server never needs to
* make outbound calls back to the client, so a browser tab — which
* cannot host an HTTP server — can fully consume `@shade/files`.
*
* ### Wire contract
*
* Request:
* ```
* POST <mount>/rpc HTTP/1.1
* Content-Type: application/octet-stream
* X-Shade-Sender-Address: <peer address>
*
* <wire-encoded ShadeEnvelope (0x01 PreKeyMessage or 0x02 RatchetMessage)
* containing JSON-encoded RpcRequest>
* ```
*
* Response (success):
* ```
* 200 OK
* Content-Type: application/octet-stream
*
* <wire-encoded ShadeEnvelope (0x02 RatchetMessage)
* containing JSON-encoded RpcResponse | RpcError>
* ```
*
* Response (transport-level failure — no session, undecryptable, etc.):
* ```
* 4xx
* Content-Type: application/json
*
* { "error": "..." }
* ```
*
* ### Symmetry with shade-auth-middleware
*
* The shape mirrors `@shade/server`'s shade-auth-middleware: an
* encrypted envelope rides the request body, the server decrypts via
* the existing ratchet session, performs the protected operation,
* and returns an encrypted envelope in the response. No bidirectional
* channel required.
*
* @see {@link createFilesHttpClient} for the matching browser client.
*/
import { Hono } from 'hono';
import { decodeEnvelope, encodeEnvelope } from '@shade/proto';
import type { ShadeBridge } from '../integration/shade-bridge.js';
import {
encodeEnvelope as encodeRpcEnvelope,
tryParseEnvelope,
} from '../protocol/envelope-codec.js';
import type { FileHandler } from './handler.js';
import type { RpcError, RpcRequest, RpcResponse } from '../schemas/envelope.js';
import { KIND_ERROR_V1 } from '../protocol/kinds.js';
export interface FilesRpcRouteOptions {
/**
* Maximum request body size in bytes. Default 1 MiB. Inline payloads
* are capped at 256 KiB by the protocol; the headroom is for
* custom-op payloads and base64 inflation.
*/
maxBodyBytes?: number;
/**
* Allow this server to accept the very first message (PreKeyMessage,
* `0x01`) over the RPC route. Disabled by default — most browser
* clients establish a session via `shade.initSessionFromBundle`
* before the first RPC. Enable when you want the RPC route to also
* be the X3DH carrier (uncommon but supported).
*/
acceptFirstMessage?: boolean;
}
const DEFAULT_MAX_BODY_BYTES = 1 * 1024 * 1024;
/**
* Build a Hono app with a single `POST /rpc` route. Mount under any
* base path: `app.route('/api/v1/shade-files', shade.files.rpcRoute())`.
*
* The `handler` must already be attached (typically via
* `shade.files.serve(handlerConfig)`); this route only ships the
* transport — it does not register a new file handler.
*/
export function createFilesRpcRoute(
shade: ShadeBridge,
handler: FileHandler,
options: FilesRpcRouteOptions = {},
): Hono {
const maxBodyBytes = options.maxBodyBytes ?? DEFAULT_MAX_BODY_BYTES;
const app = new Hono();
app.post('/rpc', async (c) => {
const senderAddress = c.req.header('X-Shade-Sender-Address');
if (senderAddress === undefined || senderAddress === '') {
return c.json({ error: 'missing X-Shade-Sender-Address header' }, 400);
}
const contentLengthHeader = c.req.header('Content-Length');
if (contentLengthHeader !== undefined) {
const contentLength = Number.parseInt(contentLengthHeader, 10);
if (Number.isFinite(contentLength) && contentLength > maxBodyBytes) {
return c.json(
{ error: `body exceeds maxBodyBytes (${contentLength} > ${maxBodyBytes})` },
413,
);
}
}
let bodyBytes: Uint8Array;
try {
const ab = await c.req.arrayBuffer();
if (ab.byteLength > maxBodyBytes) {
return c.json(
{ error: `body exceeds maxBodyBytes (${ab.byteLength} > ${maxBodyBytes})` },
413,
);
}
bodyBytes = new Uint8Array(ab);
} catch (err) {
return c.json({ error: `failed to read request body: ${(err as Error).message}` }, 400);
}
if (bodyBytes.byteLength === 0) {
return c.json({ error: 'empty request body' }, 400);
}
// Decode the wire envelope. `decodeEnvelope` handles both `0x01`
// PreKeyMessage and `0x02` RatchetMessage shapes.
let plaintext: string;
try {
const envelope = decodeEnvelope(bodyBytes);
// First-message gate: only allow `prekey` envelopes when the
// operator has explicitly opted in.
if (options.acceptFirstMessage !== true && envelope.type === 'prekey') {
return c.json(
{
error:
'PreKeyMessage envelopes are not accepted on this RPC route — establish the session first via shade.initSessionFromBundle, or set acceptFirstMessage: true',
},
400,
);
}
plaintext = await shade.receive(senderAddress, envelope);
} catch (err) {
// Decryption failure — could be no session, corrupted envelope,
// or sender address mismatch. Treat as 401 since the envelope is
// self-authenticating: a valid sender would decrypt cleanly.
return c.json({ error: `decrypt failed: ${(err as Error).message}` }, 401);
}
// Parse the plaintext as an RpcRequest.
const classified = tryParseEnvelope(plaintext);
if (classified === null) {
return c.json({ error: 'plaintext is not a valid @shade/files envelope' }, 400);
}
if (classified.kind !== 'request') {
// Cancel envelopes are silently dropped — RPC route is request/
// response only. Cancellation across HTTP is achieved via
// AbortController on the client side, not protocol-level.
if (classified.kind === 'cancel') {
handler.handleCancel(senderAddress, classified.envelope);
// No response body — the cancel was best-effort.
return new Response(null, { status: 204 });
}
return c.json(
{ error: `unexpected envelope kind on RPC route: ${classified.kind}` },
400,
);
}
const request: RpcRequest = classified.envelope;
// Dispatch through the file handler.
let result: RpcResponse | RpcError;
try {
result = await handler.handleRequest(senderAddress, request);
} catch (err) {
// Should never happen — handler.handleRequest catches its own
// errors and returns RpcError. If it didn't, that's a bug; emit
// a generic transport-level RpcError so the client can surface
// it deterministically.
result = {
kind: KIND_ERROR_V1,
id: request.id,
error: {
code: 'INTERNAL',
message: `handler raised: ${(err as Error).message}`,
},
};
}
// Encrypt the response and return it as wire bytes.
let responseBytes: Uint8Array;
try {
const responsePlaintext = encodeRpcEnvelope(result);
const responseEnvelope = await shade.send(senderAddress, responsePlaintext);
responseBytes = encodeEnvelope(responseEnvelope);
} catch (err) {
return c.json(
{ error: `failed to encrypt response: ${(err as Error).message}` },
500,
);
}
return new Response(new Blob([responseBytes as unknown as ArrayBuffer]), {
status: 200,
headers: { 'Content-Type': 'application/octet-stream' },
});
});
return app;
}

View File

@@ -168,13 +168,14 @@ export async function createServerStreamsBridge(
// No waiter yet — park. // No waiter yet — park.
parked.set(writeId, arrived); parked.set(writeId, arrived);
setTimeout(() => { const parkTimer = setTimeout(() => {
const stale = parked.get(writeId); const stale = parked.get(writeId);
if (stale === arrived) { if (stale === arrived) {
parked.delete(writeId); parked.delete(writeId);
void handle.abort('rpc-timeout').catch(() => undefined); void handle.abort('rpc-timeout').catch(() => undefined);
} }
}, parkedWriteTtlMs).unref?.(); }, parkedWriteTtlMs);
(parkTimer as unknown as { unref?: () => void }).unref?.();
}); });
function cleanupWaiter(w: PendingWaiter): void { function cleanupWaiter(w: PendingWaiter): void {

View File

@@ -0,0 +1,184 @@
import { describe, expect, test } from 'bun:test';
import { createShade } from '@shade/sdk';
import {
createPrekeyServer,
MemoryPrekeyStore,
PrekeyServerEvents,
} from '@shade/server';
import { SubtleCryptoProvider } from '@shade/crypto-web';
import { Hono } from 'hono';
const crypto = new SubtleCryptoProvider();
/**
* Concurrent-ratchet hardening tests.
*
* Reproduces the scenario described in the v4.2.0 ratchet-desync bug
* report: with the queue-drainer running on Alice and many concurrent
* `shade.send`/RPC operations against the same peer, do
* encrypt/decrypt paths share the per-peer mutex on
* `ShadeSessionManager` so that no path observes a stale ratchet
* state?
*
* If the lock coverage regresses (a future change re-introduces a
* sidekanal bypass), one of these tests will fail with
* `DecryptionError: Failed to decrypt message — wrong key or
* tampered data`.
*/
async function setupPullRig() {
const prekey = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
events: new PrekeyServerEvents(),
});
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
const prekeyUrl = `http://localhost:${prekeyServer.port}`;
const alice = await createShade({ prekeyServer: prekeyUrl, address: 'alice' });
const bob = await createShade({ prekeyServer: prekeyUrl, address: 'bob' });
const queueRoute = await bob.transferQueueRoute({ blockMs: 500 });
await bob.files.serve({
stat: async () => ({
name: '_',
kind: 'dir' as const,
size: 0,
mtime: 0,
metadata: {},
}),
});
const rpcRoute = bob.files.rpcRoute({ acceptFirstMessage: true });
const app = new Hono();
app.route('/', queueRoute);
app.route('/', rpcRoute);
const bobServer = Bun.serve({ port: 0, fetch: app.fetch });
const baseUrl = `http://localhost:${bobServer.port}`;
const fs = alice.files.httpClient('bob', {
rpcUrl: `${baseUrl}/rpc`,
outboundQueueUrl: `${baseUrl}/queue`,
transferBaseUrl: baseUrl,
defaultTimeoutMs: 10_000,
queueBlockMs: 500,
});
return {
alice,
bob,
fs,
baseUrl,
teardown: async () => {
fs.close();
await alice.shutdown();
await bob.shutdown();
bobServer.stop();
prekeyServer.stop();
},
};
}
describe('@shade/files — concurrent ratchet under drainer', () => {
test('100 parallel httpClient RPCs while drainer runs — no DecryptionError', async () => {
const rig = await setupPullRig();
try {
// Warm-up: establishes the X3DH session (Alice → Bob first message
// is a PreKeyMessage; subsequent messages are pure ratchet).
const first = await rig.fs.stat('/');
expect(first.kind).toBe('dir');
// Fire 100 concurrent stat RPCs. Each one is a full ratchet
// round-trip: encrypt request, POST, decrypt response. They all
// contend for `manager.peerOpChains["bob"]` on Alice's side
// (encrypt + decrypt) and `manager.peerOpChains["alice"]` on
// Bob's side. Drainer is running in the background polling
// Bob's queue — its decrypt path also funnels through the same
// per-peer lock.
// 100 concurrent — minimal repro (after warm-up only).
const N = 100;
const results = await Promise.allSettled(
Array.from({ length: N }, () => rig.fs.stat('/')),
);
const failures = results.filter((s) => s.status === 'rejected') as Array<
PromiseRejectedResult
>;
if (failures.length > 0) {
const sample = failures.slice(0, 1).map((f) => String(f.reason));
throw new Error(`${failures.length}/${N} concurrent RPCs failed: ${sample[0]}`);
}
} finally {
await rig.teardown();
}
}, 30_000);
test('parallel shade.send + drainer + RPCs — ratchet stays in sync', async () => {
const rig = await setupPullRig();
try {
// Establish session via one warm-up RPC.
await rig.fs.stat('/');
// Subscribe Bob to inbound plaintext from Alice — when Alice's
// raw `shade.send` plaintext arrives, Bob echoes a reply back
// through `shade.send` + `deliverControlEnvelope`, which the
// pull-mode envelope transport enqueues for Alice's drainer.
// This injects extra inbound traffic into Alice's drainer in
// parallel with her ongoing RPCs.
const echoes: string[] = [];
rig.bob.onMessage(async (from, plaintext) => {
if (from !== 'alice') return;
if (!plaintext.startsWith('ping:')) return;
echoes.push(plaintext);
const reply = `pong:${plaintext.slice('ping:'.length)}`;
const env = await rig.bob.send('alice', reply);
await rig.bob.deliverControlEnvelope('alice', env);
});
const inboundDrained: string[] = [];
rig.alice.onMessage((from, plaintext) => {
if (from !== 'bob') return;
if (plaintext.startsWith('pong:')) inboundDrained.push(plaintext);
});
// Mix three concurrent workloads against the same peer:
// - 50 inline file RPCs through httpClient (encrypt + decrypt)
// - 50 raw `shade.send` deliveries via control envelope
// - drainer pulling Bob's responses + echoes
const N = 50;
const rpcs = Array.from({ length: N }, () => rig.fs.stat('/'));
const sends = Array.from({ length: N }, async (_, i) => {
const env = await rig.alice.send('bob', `ping:${i}`);
await rig.alice.deliverControlEnvelope('bob', env);
});
const settled = await Promise.allSettled([...rpcs, ...sends]);
const failures = settled.filter((s) => s.status === 'rejected') as Array<
PromiseRejectedResult
>;
if (failures.length > 0) {
const sample = failures.slice(0, 3).map((f) => String(f.reason));
throw new Error(
`${failures.length}/${settled.length} concurrent ops failed: ${sample.join(' | ')}`,
);
}
// Give Bob's queue + Alice's drainer a beat to drain pongs back.
// Echoes round-trip Alice → Bob (control envelope) → Bob's
// onMessage → Bob.send + deliver (queue) → Alice's drainer →
// Alice's onMessage. We just verify some make it back without
// any DecryptionError surfacing.
const deadline = Date.now() + 5_000;
while (inboundDrained.length < N && Date.now() < deadline) {
await new Promise((r) => setTimeout(r, 50));
}
// Don't gate on every echo arriving — the long-poll cadence and
// bun's serve/abort timing can lag a few. We only care that the
// ratchet didn't desync; if it had, every subsequent op would
// throw DecryptionError above.
expect(echoes.length).toBe(N);
expect(inboundDrained.length).toBeGreaterThan(0);
} finally {
await rig.teardown();
}
}, 30_000);
});

View File

@@ -0,0 +1,208 @@
import { describe, expect, test } from 'bun:test';
import { createShade } from '@shade/sdk';
import {
createPrekeyServer,
MemoryPrekeyStore,
PrekeyServerEvents,
} from '@shade/server';
import { SubtleCryptoProvider } from '@shade/crypto-web';
import { Hono } from 'hono';
const crypto = new SubtleCryptoProvider();
/**
* Stand up the full pull-mode rig:
* - Prekey server (for X3DH)
* - Bob: file handler + rpcRoute + transferQueueRoute, all on one server
* - Alice: httpClient with outboundQueueUrl + transferBaseUrl wired
*
* Returns Alice's `FileClient`, which speaks browser-style: ONE base URL,
* no inbound listener, streams supported via long-poll.
*/
async function setupPullRig(opts: {
bobHandler: Parameters<NonNullable<Awaited<ReturnType<typeof createShade>>['files']>['serve']>[0];
}) {
const prekey = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
events: new PrekeyServerEvents(),
});
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
const prekeyUrl = `http://localhost:${prekeyServer.port}`;
const alice = await createShade({ prekeyServer: prekeyUrl, address: 'alice' });
const bob = await createShade({ prekeyServer: prekeyUrl, address: 'bob' });
// Bob: queue-route FIRST (configures bob's transports), then files.serve.
const queueRoute = await bob.transferQueueRoute({ blockMs: 1_500 });
await bob.files.serve(opts.bobHandler);
const rpcRoute = bob.files.rpcRoute({ acceptFirstMessage: true });
const app = new Hono();
app.route('/', queueRoute);
app.route('/', rpcRoute);
const bobServer = Bun.serve({ port: 0, fetch: app.fetch });
const baseUrl = `http://localhost:${bobServer.port}`;
const fs = alice.files.httpClient('bob', {
rpcUrl: `${baseUrl}/rpc`,
outboundQueueUrl: `${baseUrl}/queue`,
transferBaseUrl: baseUrl,
defaultTimeoutMs: 10_000,
queueBlockMs: 1_000,
});
return {
alice,
bob,
fs,
baseUrl,
teardown: async () => {
fs.close();
await alice.shutdown();
await bob.shutdown();
bobServer.stop();
prekeyServer.stop();
},
};
}
describe('@shade/files HTTP RPC — pull-mode streams', () => {
test('streamed read (4 MiB) via long-poll queue', async () => {
const payload = new Uint8Array(4 * 1024 * 1024);
for (let i = 0; i < payload.length; i++) payload[i] = (i * 97) & 0xff;
const rig = await setupPullRig({
bobHandler: {
read: async () => {
// Return the payload as a streamed read so the rpc-handler
// promotes it via the streams-bridge into a transfer.
const stream = new ReadableStream<Uint8Array>({
start(controller) {
const CHUNK = 256 * 1024;
for (let off = 0; off < payload.byteLength; off += CHUNK) {
controller.enqueue(payload.slice(off, Math.min(off + CHUNK, payload.byteLength)));
}
controller.close();
},
});
// Need a precomputed sha256 for streamed reads. Use the
// crypto provider's sha256 directly.
const digest = new Uint8Array(await globalThis.crypto.subtle.digest('SHA-256', payload));
const sha256Hex = Array.from(digest, (b) => b.toString(16).padStart(2, '0')).join('');
return {
kind: 'streams' as const,
stream,
size: payload.byteLength,
sha256: sha256Hex,
contentType: 'application/octet-stream',
};
},
},
});
try {
const result = await rig.fs.read('/big.bin');
expect(result.kind).toBe('streams');
if (result.kind !== 'streams') return;
// Drain the stream and compare.
const reader = result.stream.getReader();
const got = new Uint8Array(payload.byteLength);
let offset = 0;
while (true) {
const { value, done } = await reader.read();
if (done) break;
if (value !== undefined) {
got.set(value, offset);
offset += value.byteLength;
}
}
reader.releaseLock();
await result.done();
expect(offset).toBe(payload.byteLength);
// Compare in 64KiB strides for speed.
let mismatch = -1;
for (let i = 0; i < payload.byteLength; i++) {
if (got[i] !== payload[i]) {
mismatch = i;
break;
}
}
expect(mismatch).toBe(-1);
} finally {
await rig.teardown();
}
}, 30_000);
test('streamed read fails with clear error when outboundQueueUrl is omitted', async () => {
const rig = await setupPullRig({
bobHandler: {
read: async () => {
const stream = new ReadableStream<Uint8Array>({
start(c) {
c.enqueue(new Uint8Array(512 * 1024));
c.close();
},
});
const digest = new Uint8Array(await globalThis.crypto.subtle.digest('SHA-256', new Uint8Array(512 * 1024)));
const sha256Hex = Array.from(digest, (b) => b.toString(16).padStart(2, '0')).join('');
return {
kind: 'streams' as const,
stream,
size: 512 * 1024,
sha256: sha256Hex,
};
},
},
});
// Tear down the rig's drainer so we can construct an inline-only client
rig.fs.close();
const inlineOnly = rig.alice.files.httpClient('bob', {
rpcUrl: `${rig.baseUrl}/rpc`,
defaultTimeoutMs: 10_000,
});
try {
await expect(inlineOnly.read('/big.bin')).rejects.toThrow(/streamed read/);
} finally {
inlineOnly.close();
await rig.teardown();
}
}, 15_000);
test('long-poll returns empty events on idle timeout', async () => {
const rig = await setupPullRig({
bobHandler: {
stat: async () => ({
name: '_',
kind: 'dir' as const,
size: 0,
mtime: 0,
metadata: {},
}),
},
});
try {
// Direct poll without any pending events — should return after blockMs.
const start = Date.now();
const res = await fetch(`${rig.baseUrl}/queue`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-Shade-Sender-Address': 'alice',
},
body: JSON.stringify({ since: 0, blockMs: 500 }),
});
expect(res.status).toBe(200);
const body = (await res.json()) as { events: unknown[]; nextSince: number };
expect(body.events).toHaveLength(0);
expect(Date.now() - start).toBeGreaterThanOrEqual(400);
} finally {
await rig.teardown();
}
}, 10_000);
});

View File

@@ -0,0 +1,321 @@
import { describe, expect, test } from 'bun:test';
import { createShade } from '@shade/sdk';
import {
createPrekeyServer,
MemoryPrekeyStore,
PrekeyServerEvents,
} from '@shade/server';
import { SubtleCryptoProvider } from '@shade/crypto-web';
const crypto = new SubtleCryptoProvider();
/**
* Stand up a prekey server + two Shades + Bob's file handler + RPC route
* mounted on Bun.serve, then return Alice's HTTP-only `FileClient`.
*
* Mirrors the request-response setup a browser client would use against a
* Bun-style server.
*/
async function setupHttpRig(opts: {
bobHandler: Parameters<NonNullable<Awaited<ReturnType<typeof createShade>>['files']>['serve']>[0];
}) {
// 1. Prekey server.
const prekeyEvents = new PrekeyServerEvents();
const prekey = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
events: prekeyEvents,
});
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
const prekeyUrl = `http://localhost:${prekeyServer.port}`;
// 2. Two Shades. Alice plays the browser client (no transferRoute);
// Bob is the server.
const alice = await createShade({ prekeyServer: prekeyUrl, address: 'alice' });
const bob = await createShade({ prekeyServer: prekeyUrl, address: 'bob' });
// 3. Bob: register file handler (HTTP-only — no streams) + mount
// the RPC route.
await bob.files.serve(opts.bobHandler, { inlineOnly: true });
const rpcRoute = bob.files.rpcRoute({ acceptFirstMessage: true });
const bobServer = Bun.serve({ port: 0, fetch: rpcRoute.fetch });
const rpcUrl = `http://localhost:${bobServer.port}/rpc`;
// 5. Alice: build the HTTP-only file client.
const fs = alice.files.httpClient('bob', { rpcUrl, defaultTimeoutMs: 5000 });
return {
alice,
bob,
fs,
rpcUrl,
teardown: async () => {
await alice.shutdown();
await bob.shutdown();
bobServer.stop();
prekeyServer.stop();
},
};
}
describe('@shade/files HTTP RPC — round-trip', () => {
test('list → mkdir → stat → write inline → read inline → delete via httpClient', async () => {
interface VfsEntry {
kind: 'file' | 'dir';
bytes?: Uint8Array;
contentType?: string;
}
const vfs = new Map<string, VfsEntry>([
['/', { kind: 'dir' }],
['/photos', { kind: 'dir' }],
]);
const rig = await setupHttpRig({
bobHandler: {
list: async (ctx) => {
const prefix = ctx.path.endsWith('/') ? ctx.path : `${ctx.path}/`;
const entries = Array.from(vfs.entries())
.filter(([p]) => p.startsWith(prefix) && p !== ctx.path && !p.slice(prefix.length).includes('/'))
.map(([p, e]) => ({
name: p.slice(prefix.length) || p,
kind: e.kind,
size: e.bytes?.byteLength ?? 0,
mtime: 0,
metadata: {},
}));
return { entries, hasMore: false };
},
stat: async (ctx) => {
const e = vfs.get(ctx.path);
if (!e) throw new (await import('../../src/index.js')).NotFoundError(`stat ${ctx.path}`);
return {
name: ctx.path.split('/').pop() ?? ctx.path,
kind: e.kind,
size: e.bytes?.byteLength ?? 0,
mtime: 0,
metadata: {},
...(e.contentType !== undefined ? { contentType: e.contentType } : {}),
};
},
mkdir: async (ctx) => {
vfs.set(ctx.path, { kind: 'dir' });
return { entry: { name: ctx.path.split('/').pop() ?? ctx.path, kind: 'dir' as const, size: 0, mtime: 0, metadata: {} } };
},
delete: async (ctx) => {
if (!vfs.has(ctx.path)) {
throw new (await import('../../src/index.js')).NotFoundError(`delete ${ctx.path}`);
}
vfs.delete(ctx.path);
return { deletedCount: 1 };
},
read: async (ctx) => {
const e = vfs.get(ctx.path);
if (!e || e.kind !== 'file' || !e.bytes) {
throw new (await import('../../src/index.js')).NotFoundError(`read ${ctx.path}`);
}
// Omit sha256 — dispatcher computes it from the bytes.
return {
kind: 'inline' as const,
bytes: e.bytes,
...(e.contentType !== undefined ? { contentType: e.contentType } : {}),
};
},
write: async (ctx) => {
if (ctx.args.content.kind !== 'inline') {
throw new (await import('../../src/index.js')).ConflictError('streams not supported in this test handler');
}
vfs.set(ctx.args.path, {
kind: 'file',
bytes: ctx.args.content.bytes,
...(ctx.args.contentType !== undefined ? { contentType: ctx.args.contentType } : {}),
});
return {
entry: {
name: ctx.args.path.split('/').pop() ?? ctx.args.path,
kind: 'file' as const,
size: ctx.args.content.bytes.byteLength,
mtime: 0,
metadata: {},
...(ctx.args.contentType !== undefined ? { contentType: ctx.args.contentType } : {}),
},
};
},
},
});
try {
// list
const listed = await rig.fs.list('/');
expect(listed.entries.map((e) => e.name).sort()).toContain('photos');
// mkdir
await rig.fs.mkdir('/docs');
const stat = await rig.fs.stat('/docs');
expect(stat.kind).toBe('dir');
// write inline
const payload = new TextEncoder().encode('hello browser-friendly world');
const writeResult = await rig.fs.write('/docs/greeting.txt', payload, {
contentType: 'text/plain',
});
expect(writeResult.entry.size).toBe(payload.byteLength);
// read inline
const readResult = await rig.fs.read('/docs/greeting.txt');
expect(readResult.kind).toBe('inline');
if (readResult.kind === 'inline') {
expect(new TextDecoder().decode(readResult.bytes)).toBe('hello browser-friendly world');
expect(readResult.contentType).toBe('text/plain');
}
// delete
const del = await rig.fs.delete('/docs/greeting.txt');
expect(del.deletedCount).toBe(1);
// stat the deleted path → typed NotFoundError
const { NotFoundError } = await import('../../src/index.js');
await expect(rig.fs.stat('/docs/greeting.txt')).rejects.toBeInstanceOf(NotFoundError);
} finally {
await rig.teardown();
}
});
test('streamed write (> 256 KiB) is rejected with a clear error', async () => {
const rig = await setupHttpRig({
bobHandler: {
write: async () => ({
entry: { name: 'unused', kind: 'file' as const, size: 0, mtime: 0, metadata: {} },
}),
},
});
try {
const big = new Uint8Array(257 * 1024);
const { ConflictError } = await import('../../src/index.js');
await expect(rig.fs.write('/big.bin', big)).rejects.toBeInstanceOf(ConflictError);
} finally {
await rig.teardown();
}
});
test('rpcRoute() throws when no handler is attached', async () => {
// Don't call shade.files.serve(...) — rpcRoute() should refuse.
const prekeyEvents = new PrekeyServerEvents();
const prekey = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
events: prekeyEvents,
});
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
const bob = await createShade({
prekeyServer: `http://localhost:${prekeyServer.port}`,
address: 'bob',
});
try {
expect(() => bob.files.rpcRoute()).toThrow(/no handler attached/);
} finally {
await bob.shutdown();
prekeyServer.stop();
}
});
test('missing X-Shade-Sender-Address header → 400', async () => {
const rig = await setupHttpRig({
bobHandler: {
stat: async () => ({
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
}),
},
});
try {
const res = await fetch(rig.rpcUrl, {
method: 'POST',
body: new Uint8Array([0]),
});
expect(res.status).toBe(400);
const body = (await res.json()) as { error: string };
expect(body.error).toMatch(/X-Shade-Sender-Address/);
} finally {
await rig.teardown();
}
});
test('empty body → 400', async () => {
const rig = await setupHttpRig({
bobHandler: {
stat: async () => ({
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
}),
},
});
try {
const res = await fetch(rig.rpcUrl, {
method: 'POST',
headers: { 'X-Shade-Sender-Address': 'alice' },
});
expect(res.status).toBe(400);
} finally {
await rig.teardown();
}
});
test('garbage body → 401 decrypt failure', async () => {
const rig = await setupHttpRig({
bobHandler: {
stat: async () => ({
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
}),
},
});
try {
const res = await fetch(rig.rpcUrl, {
method: 'POST',
headers: { 'X-Shade-Sender-Address': 'alice' },
body: new Uint8Array([0x02, 0xff, 0xff, 0xff]),
});
// 400 from envelope decode failure or 401 from decrypt failure.
expect([400, 401]).toContain(res.status);
} finally {
await rig.teardown();
}
});
test('body past maxBodyBytes → 413', async () => {
const prekeyEvents = new PrekeyServerEvents();
const prekey = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
events: prekeyEvents,
});
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
const bob = await createShade({
prekeyServer: `http://localhost:${prekeyServer.port}`,
address: 'bob',
});
await bob.files.serve(
{
stat: async () => ({
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
}),
},
{ inlineOnly: true },
);
const route = bob.files.rpcRoute({ maxBodyBytes: 1024 });
const server = Bun.serve({ port: 0, fetch: route.fetch });
try {
const big = new Uint8Array(2048);
const res = await fetch(`http://localhost:${server.port}/rpc`, {
method: 'POST',
headers: { 'X-Shade-Sender-Address': 'alice' },
body: big,
});
expect(res.status).toBe(413);
} finally {
await bob.shutdown();
server.stop();
prekeyServer.stop();
}
});
});

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/inbox-server", "name": "@shade/inbox-server",
"version": "4.0.0", "version": "4.11.1",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -0,0 +1,268 @@
import { Hono } from 'hono';
import type { CryptoProvider } from '@shade/core';
import {
errorToHttpStatus,
ShadeError,
ValidationError,
UnauthorizedError,
fromBase64,
toBase64,
constantTimeEqual,
} from '@shade/core';
import {
verifyPayload,
RateLimiter,
MemoryRateLimitStore,
type RateLimitConfig,
} from '@shade/server';
import {
ATTR_ERROR_CODE,
ATTR_HTTP_STATUS,
ATTR_ROUTE,
NOOP_HOOK,
type ObservabilityHook,
} from '@shade/observability';
import type { BlobStore } from './blob-store.js';
/**
* Wire-level wrapper around the V4.9 BlobStore primitive.
*
* Endpoints:
* GET /v1/blob/:slotId → { blob, etag } | 404
* PUT /v1/blob/:slotId → { etag, created } | 409 | 412
* DELETE /v1/blob/:slotId → { ok }
*
* SlotId is 64 lowercase hex chars (the HKDF output, 32 bytes). Payloads
* are base64-encoded ciphertext; the relay never decrypts. Auth uses
* `signPayload` / `verifyPayload` (same canonical-JSON-and-Ed25519
* scheme as the inbox routes), keyed off the per-slot pubkey stored
* TOFU on the first PUT.
*
* Quota: a single slot holds one blob. `MAX_BLOB_BYTES` (64 KiB) is
* sized for Prism's profile use-case (a few hundred host entries) with
* plenty of headroom; future apps can override via `BlobRoutesOptions`.
*/
const SLOT_ID_REGEX = /^[0-9a-f]{64}$/;
const MAX_META_BODY_SIZE = 64 * 1024;
/** Default per-slot blob ceiling. Sized for ~500 host entries in JSON form. */
export const DEFAULT_MAX_BLOB_BYTES = 64 * 1024;
const PUT_LIMIT: RateLimitConfig = { capacity: 60, refillPerSecond: 1 };
const GET_LIMIT: RateLimitConfig = { capacity: 120, refillPerSecond: 2 };
const DELETE_LIMIT: RateLimitConfig = { capacity: 30, refillPerSecond: 1 };
export interface BlobRoutesOptions {
disableRateLimit?: boolean;
observability?: ObservabilityHook;
/** Per-blob byte ceiling. Defaults to 64 KiB. */
maxBlobBytes?: number;
}
export function createBlobRoutes(
store: BlobStore,
crypto: CryptoProvider,
options: BlobRoutesOptions = {},
): Hono {
const app = new Hono();
const observability = options.observability ?? NOOP_HOOK;
const maxBlobBytes = options.maxBlobBytes ?? DEFAULT_MAX_BLOB_BYTES;
app.use('*', async (c, next) => {
const route = c.req.routePath ?? c.req.path ?? '<unknown>';
const span = observability.startSpan('shade.blob.request', {
[ATTR_ROUTE]: route,
});
try {
await next();
span.setAttribute(ATTR_HTTP_STATUS, c.res.status);
span.setStatus(c.res.status >= 500 ? 'error' : 'ok');
} catch (err) {
const code =
err instanceof ShadeError ? err.code ?? 'SHADE_ERROR' : 'SHADE_INTERNAL';
span.setAttribute(ATTR_ERROR_CODE, code);
span.recordException(err);
span.setStatus('error', code);
throw err;
} finally {
span.end();
}
});
const rlStore = new MemoryRateLimitStore();
const putRL = new RateLimiter(rlStore, PUT_LIMIT);
const getRL = new RateLimiter(rlStore, GET_LIMIT);
const deleteRL = new RateLimiter(rlStore, DELETE_LIMIT);
const rateLimitEnabled = !options.disableRateLimit;
const getClientIp = (c: any): string =>
c.req.header('x-forwarded-for')?.split(',')[0]?.trim() ??
c.req.header('x-real-ip') ??
'unknown';
app.onError((err, c) => {
if (err instanceof ShadeError) {
const status = errorToHttpStatus(err);
const body: any = err.toJSON();
if ((err as any).retryAfterSeconds) {
c.header('Retry-After', String((err as any).retryAfterSeconds));
}
return c.json(body, status as any);
}
console.error('[Shade] Unhandled blob error:', err);
return c.json({ error: 'Internal server error' }, 500);
});
function validateSlotId(raw: string | undefined): string {
if (typeof raw !== 'string' || !SLOT_ID_REGEX.test(raw)) {
throw new ValidationError(
'slotId must be 64 lowercase hex chars (32 bytes)',
'slotId',
);
}
return raw;
}
// ─── GET ─────────────────────────────────────────────────────
// Unauthenticated. SlotId is itself a 256-bit secret derived from the
// master key — knowing it implies you derived the master, which is
// equivalent to holding the credentials. The blob is AEAD-sealed, so
// a relay-side leak of slotId still cannot decrypt the contents.
app.get('/v1/blob/:slotId', async (c) => {
const slotId = validateSlotId(c.req.param('slotId'));
if (rateLimitEnabled) await getRL.consume(`blob-get:${getClientIp(c)}`);
const row = await store.get(slotId);
if (!row) {
return c.json({ error: 'Slot not found', code: 'SHADE_NOT_FOUND' }, 404);
}
return c.json({
blob: toBase64(row.blob),
etag: String(row.etag),
updatedAt: row.updatedAt,
});
});
// ─── PUT ─────────────────────────────────────────────────────
// Body format:
// {
// ownerPubkey: b64, // Ed25519 pubkey deterministically
// // derived from the master via HKDF.
// blob: b64,
// ifMatch?: string, // "<etag>" | "*" | undefined
// signedAt: number,
// signature: b64 // over the canonical body sans signature
// }
//
// First write to a slot is TOFU: we record `ownerPubkey` and require
// any future write to verify against it. A different key trying to
// overwrite an existing slot is rejected with UnauthorizedError.
app.put('/v1/blob/:slotId', async (c) => {
const slotId = validateSlotId(c.req.param('slotId'));
if (rateLimitEnabled) await putRL.consume(`blob-put:${getClientIp(c)}`);
const rawBody = await c.req.text();
const hardLimit = Math.ceil(maxBlobBytes * 1.4) + MAX_META_BODY_SIZE;
if (rawBody.length > hardLimit) {
throw new ValidationError(`Request body too large`);
}
const body = JSON.parse(rawBody);
const { ownerPubkey, blob, ifMatch } = body;
if (typeof ownerPubkey !== 'string') {
throw new ValidationError('Missing ownerPubkey', 'ownerPubkey');
}
if (typeof blob !== 'string') {
throw new ValidationError('Missing blob', 'blob');
}
const claimedKey = fromBase64(ownerPubkey);
if (claimedKey.length !== 32) {
throw new ValidationError('ownerPubkey must be 32 bytes (Ed25519)', 'ownerPubkey');
}
const blobBytes = fromBase64(blob);
if (blobBytes.length === 0) {
throw new ValidationError('blob is empty', 'blob');
}
if (blobBytes.length > maxBlobBytes) {
throw new ValidationError(
`blob exceeds maxBlobBytes (${blobBytes.length} > ${maxBlobBytes})`,
'blob',
);
}
let expectedEtag: number | '*' | undefined;
if (ifMatch === undefined) {
expectedEtag = undefined;
} else if (typeof ifMatch !== 'string') {
throw new ValidationError('ifMatch must be a string when present', 'ifMatch');
} else if (ifMatch === '*') {
expectedEtag = '*';
} else {
const n = Number(ifMatch);
if (!Number.isFinite(n) || !Number.isInteger(n) || n < 0) {
throw new ValidationError('ifMatch must be a non-negative integer or "*"', 'ifMatch');
}
expectedEtag = n;
}
// Existing slot: caller must sign with the original owner key. Use
// the stored pubkey for verification. The body's `ownerPubkey` is
// bound by the signature too, so an attacker cannot trick us into
// verifying with a key they control — the canonicalization includes
// every field but `signature`.
const existing = await store.get(slotId);
const verifyKey = existing ? existing.ownerPubkey : claimedKey;
// Bind slotId into the signed payload so a signature for slot A
// can't be replayed against slot B (the URL is otherwise outside
// the signed bytes).
await verifyPayload(crypto, verifyKey, { ...body, slotId });
if (existing && !constantTimeEqual(existing.ownerPubkey, claimedKey)) {
throw new UnauthorizedError(
`Slot ${slotId} is owned by a different signing key`,
);
}
const result = await store.put({
slotId,
blob: blobBytes,
ownerPubkey: claimedKey,
expectedEtag,
now: Date.now(),
});
return c.json({
ok: true,
created: result.created,
etag: String(result.etag),
updatedAt: result.updatedAt,
});
});
// ─── DELETE ──────────────────────────────────────────────────
// Body format: { signedAt, signature }. Signed by the owner pubkey
// recorded on the first PUT. After deletion, the slot is fully gone —
// the next PUT TOFU-claims it again (potentially under a different
// signing key, e.g. after a rotation).
app.delete('/v1/blob/:slotId', async (c) => {
const slotId = validateSlotId(c.req.param('slotId'));
if (rateLimitEnabled) await deleteRL.consume(`blob-delete:${getClientIp(c)}`);
const existing = await store.get(slotId);
if (!existing) {
return c.json({ error: 'Slot not found', code: 'SHADE_NOT_FOUND' }, 404);
}
const rawBody = await c.req.text();
if (rawBody.length > MAX_META_BODY_SIZE) {
throw new ValidationError(`Request body too large`);
}
const body = JSON.parse(rawBody);
await verifyPayload(crypto, existing.ownerPubkey, { ...body, slotId });
const removed = await store.delete(slotId);
return c.json({ ok: removed });
});
return app;
}

View File

@@ -0,0 +1,86 @@
/**
* BlobStore — server-side storage interface for the V4.9 encrypted-blob
* primitive. A "slot" is a single AEAD-sealed blob keyed by a
* deterministic 32-byte slotId derived client-side via HKDF from a
* master key. The relay never sees plaintext, never holds private keys,
* and never decrypts.
*
* Auth model (TOFU per slot, mirrors the inbox-owner pattern):
* - First PUT to an empty slot stores the caller's Ed25519 signing
* pubkey alongside the blob. Subsequent writes must produce a valid
* signature verifiable by that pubkey.
* - GET is unauthenticated — slotId is itself a 256-bit secret derived
* from the master key, so knowing it implies you derived the master.
* - DELETE clears the blob AND the owner pubkey, allowing future TOFU
* re-claim by a fresh signing key derived from the same master (e.g.
* after a rotation).
*
* CAS / etag semantics:
* - Every successful PUT bumps a per-slot monotonic etag (returned to
* the caller as a string).
* - A stale `ifMatch` triggers `PreconditionFailedError` (HTTP 412).
* - `ifMatch === undefined` against a populated slot triggers
* `ConflictError` (HTTP 409) — clients must read-then-write.
* - `ifMatch === '*'` against a populated slot is unconditional
* overwrite (escape hatch). Against an empty slot it's still 412
* per RFC 7232 (no entity to match).
*/
export interface BlobSlotRecord {
/** Lower-hex 64-char slotId (32 bytes). */
slotId: string;
/** Raw AEAD ciphertext (bytes). The relay never decrypts. */
blob: Uint8Array;
/** Owner Ed25519 signing pubkey, established TOFU on the first PUT. */
ownerPubkey: Uint8Array;
/** Monotonic per-slot version. Used as the ETag on the wire. */
etag: number;
/** Wall-clock ms of the last successful write. */
updatedAt: number;
}
/** Returned to the route layer after a successful PUT. */
export interface PutBlobResult {
/** Whether the slot was created (true) or updated in place (false). */
created: boolean;
/** New etag after the write. */
etag: number;
/** Wall-clock ms of the write. */
updatedAt: number;
}
export interface BlobStore {
/** Read a slot, or null if it has never been written (or was deleted). */
get(slotId: string): Promise<BlobSlotRecord | null>;
/**
* Create or update a slot.
*
* Implementations MUST treat `(slotId, ownerPubkey)` atomically: the
* route layer has already verified the signature, but the store is the
* authority on whether the slot exists and what etag it has. Callers
* pass the verified `ownerPubkey` (used on first-write to record the
* owner; ignored on subsequent writes — the existing pubkey is the
* source of truth for who's allowed to write).
*
* `expectedEtag` semantics (mirror the wire-level If-Match):
* - `undefined` : create-only. Slot must be empty.
* - `<number>` : compare-and-swap. Must equal the current etag.
* - `'*'` : unconditional overwrite. Slot must already exist.
*
* On precondition mismatch the store throws `PreconditionFailedError`
* (stale etag) or `ConflictError` (slot exists, no ifMatch).
*/
put(args: {
slotId: string;
blob: Uint8Array;
ownerPubkey: Uint8Array;
expectedEtag: number | '*' | undefined;
now: number;
}): Promise<PutBlobResult>;
/**
* Delete a slot. Authentication has already been checked by the route
* layer. Returns true if a row was removed (i.e. the slot existed).
*/
delete(slotId: string): Promise<boolean>;
}

View File

@@ -0,0 +1,140 @@
/**
* BridgeDeliveryLog — V4.8.4 cross-channel dedup.
*
* Records per-`(address, msgId)` "delivered via bridge push" timestamps so
* the inbox-fetch route can filter out blobs the bridge has already pushed
* to the recipient. The relay's contract becomes:
*
* one `Inbox.send` ⇒ one observable delivery on the recipient
*
* even when the recipient runs both a bridge subscription (WS / SSE) AND
* the regular inbox-poll. Without it, bridge-push and inbox-poll are
* independent paths against the same store and the recipient gets the
* same envelope twice — bridge-first, then ~30 s later via the next poll
* — tripping on already-consumed prekeys (`one-time prekey not found`)
* or surfacing as duplicate `shade.receive` work.
*
* The log is in-memory per process and intentionally bounded: each entry
* lives for `graceMs` (default 60 s, well past a typical `pollIntervalMs`
* of 30 s). After grace, the entry is forgotten and inbox-poll falls back
* to delivering the blob — that's the legitimate "bridge dropped the
* frame, poll picked up" recovery path. If the recipient explicitly
* acks the blob (HTTP `DELETE /v1/inbox/:addr/:msgId`), the blob is gone
* from storage and the log entry is moot.
*
* Multi-bridge per address (e.g. WS + SSE redundancy on the same client,
* or two devices sharing one signing key) is preserved: every bridge
* connection still fetches + pushes the blob — each push records its own
* timestamp — so each connected bridge gets the frame. Only the *poll*
* fetch is filtered, not the bridge fetches themselves.
*
* @see Prism FR `cross-channel-duplicate-fanout-v4.8.2.md`.
*/
const DEFAULT_GRACE_MS = 60_000;
export interface BridgeDeliveryLogOptions {
/**
* How long a `(address, msgId)` mark suppresses inbox-poll delivery.
* Defaults to 60_000ms — twice the default `pollIntervalMs` of the
* `@shade/inbox` orchestrator, so a poll cycle that races a bridge
* push always sees the mark, but a stuck recipient still gets the
* blob via poll within ~minutes.
*/
graceMs?: number;
/**
* Maximum entries per address. Bounds memory under a busy address.
* Oldest entries (by recorded timestamp) are evicted first. Default
* 8192 — comfortably above any realistic backlog.
*/
maxPerAddress?: number;
}
export class BridgeDeliveryLog {
private readonly log = new Map<string, Map<string, number>>();
private readonly graceMs: number;
private readonly maxPerAddress: number;
constructor(options: BridgeDeliveryLogOptions = {}) {
this.graceMs = options.graceMs ?? DEFAULT_GRACE_MS;
this.maxPerAddress = options.maxPerAddress ?? 8192;
}
/** Mark `(address, msgId)` as bridge-delivered at `now`. */
recordDelivered(address: string, msgId: string, now: number): void {
let inner = this.log.get(address);
if (!inner) {
inner = new Map();
this.log.set(address, inner);
}
inner.set(msgId, now);
// Lazy cleanup: drop entries past 2× grace so the map stays bounded
// without a separate timer. Bound by `maxPerAddress` as a fallback
// for pathological burst scenarios.
if (inner.size > this.maxPerAddress) {
const cutoff = now - this.graceMs * 2;
for (const [id, ts] of inner) {
if (ts < cutoff) inner.delete(id);
}
// Still over cap? Drop the oldest.
if (inner.size > this.maxPerAddress) {
const sorted = Array.from(inner.entries()).sort((a, b) => a[1] - b[1]);
const toDrop = sorted.slice(0, inner.size - this.maxPerAddress);
for (const [id] of toDrop) inner.delete(id);
}
}
}
/**
* Returns true if `(address, msgId)` was bridge-delivered within the
* grace window.
*/
isRecentlyDelivered(address: string, msgId: string, now: number): boolean {
const inner = this.log.get(address);
if (!inner) return false;
const ts = inner.get(msgId);
if (ts === undefined) return false;
if (now - ts > this.graceMs) {
inner.delete(msgId); // tombstone the stale entry
return false;
}
return true;
}
/**
* Filter `blobs` down to those not currently in the bridge-delivered
* grace window. Used by the inbox-fetch route to suppress duplicates.
*/
filterRecent<T extends { msgId: string }>(
address: string,
blobs: T[],
now: number,
): T[] {
const inner = this.log.get(address);
if (!inner || inner.size === 0) return blobs;
return blobs.filter((b) => {
const ts = inner.get(b.msgId);
if (ts === undefined) return true;
if (now - ts > this.graceMs) {
inner.delete(b.msgId);
return true;
}
return false;
});
}
/** Drop the entry for `(address, msgId)`. Called from blob-delete paths. */
forget(address: string, msgId: string): void {
this.log.get(address)?.delete(msgId);
}
/** Drop every entry for `address`. Called from address-delete paths. */
forgetAddress(address: string): void {
this.log.delete(address);
}
/** Test-only inspection. */
size(address: string): number {
return this.log.get(address)?.size ?? 0;
}
}

View File

@@ -38,8 +38,16 @@ import {
import { verifyPayload, validateAddress } from '@shade/server'; import { verifyPayload, validateAddress } from '@shade/server';
import type { InboxStore } from './store.js'; import type { InboxStore } from './store.js';
import type { InboxServerEvents } from './events.js'; import type { InboxServerEvents } from './events.js';
import { PresenceTracker, type TrackedBridgeKind } from './presence.js';
import { BridgeDeliveryLog } from './bridge-delivery-log.js';
export type BridgeKind = 'stream' | 'poll' | 'ws'; export type BridgeKind = 'stream' | 'poll' | 'ws';
/**
* Wire-protocol kind tag for `/v1/bridge/presence`. Distinct from
* `BridgeKind` because the canonical signed payload is shaped
* differently (`watched: string[]` instead of `since: number`).
*/
export type PresenceKind = 'presence';
export interface BridgeRoutesOptions { export interface BridgeRoutesOptions {
store: InboxStore; store: InboxStore;
@@ -60,6 +68,23 @@ export interface BridgeRoutesOptions {
* Default 1_000. * Default 1_000.
*/ */
fallbackPollIntervalMs?: number; fallbackPollIntervalMs?: number;
/**
* Inject an existing presence tracker. Useful when multiple
* `createBridgeRoutes` calls need to share state (e.g. mounting the
* routes under several hostnames in a single process). When omitted,
* the bridge auto-creates an internal tracker bound to `events`.
*/
presenceTracker?: PresenceTracker;
/**
* V4.8.4 — shared bridge delivery log. After every successful WS /
* SSE push we record `(address, msgId, now)` here so the inbox-fetch
* route can suppress the same blob from a subsequent inbox-poll
* within the log's grace window. Pass the same instance to
* `createInboxRoutes` (or use the auto-created one returned in
* `bridgeRoutes.bridgeDeliveryLog`). When omitted, the bridge
* auto-creates its own log.
*/
bridgeDeliveryLog?: BridgeDeliveryLog;
} }
interface VerifiedBridgeRequest { interface VerifiedBridgeRequest {
@@ -68,6 +93,13 @@ interface VerifiedBridgeRequest {
since: number; since: number;
} }
interface VerifiedPresenceRequest {
/** The watcher's address (signer of the request). */
address: string;
/** Addresses whose presence the watcher is asking to track. */
watched: string[];
}
/** /**
* Build the bridge Hono router and a paired Bun-WebSocket handler. * Build the bridge Hono router and a paired Bun-WebSocket handler.
* *
@@ -80,6 +112,15 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
app: Hono; app: Hono;
/** Pass to `Bun.serve({ websocket })`. Undefined if Bun adapter is missing. */ /** Pass to `Bun.serve({ websocket })`. Undefined if Bun adapter is missing. */
websocket: unknown; websocket: unknown;
/** Live presence tracker. Tests + observers can read it; routes update it. */
presence: PresenceTracker;
/**
* V4.8.4 — the shared bridge-delivery log this router writes to on
* every successful push. Wire the same instance into
* `createInboxRoutes({ bridgeDeliveryLog })` so the inbox-fetch route
* can suppress recently-pushed blobs.
*/
bridgeDeliveryLog: BridgeDeliveryLog;
} { } {
const app = new Hono(); const app = new Hono();
const pageLimit = opts.pageLimit ?? 50; const pageLimit = opts.pageLimit ?? 50;
@@ -87,6 +128,8 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
const longPollDefault = opts.longPollTimeoutMs ?? 25_000; const longPollDefault = opts.longPollTimeoutMs ?? 25_000;
const longPollMax = opts.longPollMaxTimeoutMs ?? 55_000; const longPollMax = opts.longPollMaxTimeoutMs ?? 55_000;
const fallbackPollIntervalMs = opts.fallbackPollIntervalMs ?? 1_000; const fallbackPollIntervalMs = opts.fallbackPollIntervalMs ?? 1_000;
const presence = opts.presenceTracker ?? new PresenceTracker(opts.events ?? null);
const bridgeDeliveryLog = opts.bridgeDeliveryLog ?? new BridgeDeliveryLog();
app.onError((err, c) => { app.onError((err, c) => {
if (err instanceof ShadeError) { if (err instanceof ShadeError) {
@@ -102,17 +145,37 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
const verified = await verifyBridgeAuth(c, opts, 'stream'); const verified = await verifyBridgeAuth(c, opts, 'stream');
return streamSSE(c, async (stream) => { return streamSSE(c, async (stream) => {
const address = verified.address; const address = verified.address;
const connId = presence.newConnectionId();
presence.markConnected(address, 'sse', connId);
let presenceClosed = false;
const closePresence = (reason: 'closed' | 'error'): void => {
if (presenceClosed) return;
presenceClosed = true;
presence.markDisconnected(address, 'sse', connId, reason);
};
let cursor = verified.since; let cursor = verified.since;
const writer = makeBlobWriter(opts.store, pageLimit); const writer = makeBlobWriter(opts.store, pageLimit);
const delivered = new DeliveredIdLru();
const recordPush = (msgId: string): void => {
bridgeDeliveryLog.recordDelivered(address, msgId, Date.now());
};
// Initial backlog drain. // Initial backlog drain.
const flushed = await flushTo(writer, address, cursor, async (blob) => { const flushed = await flushTo(
writer,
address,
cursor,
async (blob) => {
await stream.writeSSE({ await stream.writeSSE({
id: String(blob.receivedAt), id: String(blob.receivedAt),
event: 'envelope', event: 'envelope',
data: JSON.stringify(serializeBlob(blob)), data: JSON.stringify(serializeBlob(blob)),
}); });
}); recordPush(blob.msgId);
},
delivered,
);
cursor = Math.max(cursor, flushed); cursor = Math.max(cursor, flushed);
// Hook up event-driven push if available, else fall back to a poll // Hook up event-driven push if available, else fall back to a poll
@@ -124,19 +187,32 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
const triggerFlush = (): void => { const triggerFlush = (): void => {
signalled = true; signalled = true;
// Serialize fan-in so concurrent triggers don't double-fetch. // Serialize fan-in so concurrent triggers don't double-fetch.
pendingFlushPromise = pendingFlushPromise.then(async () => { // `.catch(() => {})` keeps the chain alive across transient
// emit failures (e.g. a closed SSE write throws) — without it
// one rejection silently kills every future flush on this
// connection.
pendingFlushPromise = pendingFlushPromise
.then(async () => {
while (signalled) { while (signalled) {
signalled = false; signalled = false;
const drained = await flushTo(writer, address, cursor, async (blob) => { const drained = await flushTo(
writer,
address,
cursor,
async (blob) => {
await stream.writeSSE({ await stream.writeSSE({
id: String(blob.receivedAt), id: String(blob.receivedAt),
event: 'envelope', event: 'envelope',
data: JSON.stringify(serializeBlob(blob)), data: JSON.stringify(serializeBlob(blob)),
}); });
}); recordPush(blob.msgId);
},
delivered,
);
if (drained > cursor) cursor = drained; if (drained > cursor) cursor = drained;
} }
}); })
.catch(() => {});
}; };
if (opts.events) { if (opts.events) {
@@ -163,6 +239,68 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
clearInterval(fallbackTimer); clearInterval(fallbackTimer);
clearInterval(heartbeat); clearInterval(heartbeat);
await pendingFlushPromise.catch(() => {}); await pendingFlushPromise.catch(() => {});
closePresence('closed');
});
});
// ─── Presence (V4.7) ──────────────────────────────────────────
// SSE feed of `peer_connected` / `peer_disconnected` events filtered
// by a watcher-supplied address list. Subscribing does NOT count as
// a peer-bridge connection (it doesn't call `markConnected`) so
// monitoring presence doesn't make you appear online to others.
app.get('/v1/bridge/presence', async (c) => {
const verified = await verifyPresenceAuth(c, opts);
return streamSSE(c, async (stream) => {
const watched = new Set(verified.watched);
// Initial snapshot — one frame per watched address with current
// status. Lets subscribers render UI immediately rather than
// waiting for the next state change.
const now = Date.now();
for (const addr of verified.watched) {
await stream.writeSSE({
event: 'presence',
data: JSON.stringify({
address: addr,
status: presence.isOnline(addr) ? 'online' : 'offline',
at: now,
}),
});
}
let unsubscribe: (() => void) | null = null;
if (opts.events) {
unsubscribe = opts.events.on((e) => {
if (e.name !== 'inbox.peer_connected' && e.name !== 'inbox.peer_disconnected') return;
const data = e.data as { address: string; bridgeKind: TrackedBridgeKind };
if (!watched.has(data.address)) return;
const status = e.name === 'inbox.peer_connected' ? 'online' : 'offline';
// Fire-and-forget: drop the frame if the stream has gone away.
void stream
.writeSSE({
event: 'presence',
data: JSON.stringify({
address: data.address,
status,
at: e.timestamp,
via: data.bridgeKind,
}),
})
.catch(() => {});
});
}
const heartbeat = setInterval(() => {
stream.write(`: ping ${Date.now()}\n\n`).catch(() => {});
}, heartbeatIntervalMs);
await new Promise<void>((resolve) => {
const sig = c.req.raw.signal;
if (sig.aborted) return resolve();
sig.addEventListener('abort', () => resolve(), { once: true });
});
unsubscribe?.();
clearInterval(heartbeat);
}); });
}); });
@@ -183,6 +321,8 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
limit: pageLimit, limit: pageLimit,
}); });
if (blobs.length > 0) { if (blobs.length > 0) {
const now = Date.now();
for (const b of blobs) bridgeDeliveryLog.recordDelivered(verified.address, b.msgId, now);
return c.json(buildPollResponse(blobs, verified.since)); return c.json(buildPollResponse(blobs, verified.since));
} }
@@ -197,6 +337,8 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
fallbackPollIntervalMs, fallbackPollIntervalMs,
abortSignal: c.req.raw.signal, abortSignal: c.req.raw.signal,
}); });
const now = Date.now();
for (const b of blobs) bridgeDeliveryLog.recordDelivered(verified.address, b.msgId, now);
return c.json(buildPollResponse(blobs, verified.since)); return c.json(buildPollResponse(blobs, verified.since));
}); });
@@ -230,30 +372,51 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
} }
const address = verified.address; const address = verified.address;
const connId = presence.newConnectionId();
let cursor = verified.since; let cursor = verified.since;
const writer = makeBlobWriter(opts.store, pageLimit); const writer = makeBlobWriter(opts.store, pageLimit);
const delivered = new DeliveredIdLru();
let unsubscribe: (() => void) | null = null; let unsubscribe: (() => void) | null = null;
let fallbackTimer: ReturnType<typeof setInterval> | null = null; let fallbackTimer: ReturnType<typeof setInterval> | null = null;
let pendingFlushPromise: Promise<void> = Promise.resolve(); let pendingFlushPromise: Promise<void> = Promise.resolve();
let signalled = false; let signalled = false;
let connected = true; let connected = true;
let presenceClosed = false;
const closePresence = (reason: 'closed' | 'error'): void => {
if (presenceClosed) return;
presenceClosed = true;
presence.markDisconnected(address, 'ws', connId, reason);
};
return { return {
onOpen(_evt: unknown, ws: { onOpen(_evt: unknown, ws: {
send: (data: string) => void; send: (data: string) => void;
close: (code?: number, reason?: string) => void; close: (code?: number, reason?: string) => void;
}) { }) {
presence.markConnected(address, 'ws', connId);
const triggerFlush = (): void => { const triggerFlush = (): void => {
signalled = true; signalled = true;
pendingFlushPromise = pendingFlushPromise.then(async () => { // `.catch(() => {})` mirrors the SSE chain — keeps the
// pending-flush queue alive across transient ws.send errors
// (e.g. partial close, backpressure overflow).
pendingFlushPromise = pendingFlushPromise
.then(async () => {
while (signalled && connected) { while (signalled && connected) {
signalled = false; signalled = false;
const drained = await flushTo(writer, address, cursor, async (blob) => { const drained = await flushTo(
writer,
address,
cursor,
async (blob) => {
ws.send(JSON.stringify(serializeBlob(blob))); ws.send(JSON.stringify(serializeBlob(blob)));
}); bridgeDeliveryLog.recordDelivered(address, blob.msgId, Date.now());
},
delivered,
);
if (drained > cursor) cursor = drained; if (drained > cursor) cursor = drained;
} }
}); })
.catch(() => {});
}; };
if (opts.events) { if (opts.events) {
unsubscribe = opts.events.on((e) => { unsubscribe = opts.events.on((e) => {
@@ -269,12 +432,19 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
connected = false; connected = false;
unsubscribe?.(); unsubscribe?.();
if (fallbackTimer) clearInterval(fallbackTimer); if (fallbackTimer) clearInterval(fallbackTimer);
closePresence('closed');
},
onError() {
connected = false;
unsubscribe?.();
if (fallbackTimer) clearInterval(fallbackTimer);
closePresence('error');
}, },
}; };
}), }),
); );
return { app, websocket }; return { app, websocket, presence, bridgeDeliveryLog };
} }
// ─── helpers ────────────────────────────────────────────────── // ─── helpers ──────────────────────────────────────────────────
@@ -321,11 +491,75 @@ async function verifyBridgeAuth(
return { address, kind: kind as BridgeKind, since }; return { address, kind: kind as BridgeKind, since };
} }
const MAX_WATCHED_ADDRESSES = 64;
/**
* Verify a `/v1/bridge/presence` request.
*
* Signed canonical payload: `{address, kind: 'presence', watched: string[],
* signedAt}`. The watcher's address must be a registered inbox; the
* signature is verified against the registered owner key for that
* address. The `watched` list bounds what the subscription will
* receive — server-side filtering is enforced inside the handler.
*/
async function verifyPresenceAuth(
c: Context,
opts: BridgeRoutesOptions,
): Promise<VerifiedPresenceRequest> {
const url = new URL(c.req.url);
const qs = url.searchParams;
const address = validateAddress(qs.get('address'));
const kind = qs.get('kind');
if (kind !== 'presence') {
throw new ValidationError(`bridge kind mismatch: expected presence`, 'kind');
}
const watchedRaw = qs.get('watched');
if (watchedRaw === null) throw new ValidationError('missing watched', 'watched');
// Empty subscription is allowed (subscribe to nothing — useful for a
// client that intends to call addPeer right after open). A null/
// missing param is still rejected so the canonicalization is
// unambiguous.
const watched =
watchedRaw === ''
? []
: watchedRaw.split(',').map((a) => validateAddress(a));
if (watched.length > MAX_WATCHED_ADDRESSES) {
throw new ValidationError(
`watched list too large: ${watched.length} > ${MAX_WATCHED_ADDRESSES}`,
'watched',
);
}
const signedAtStr = qs.get('signedAt');
const signature = qs.get('signature');
if (signedAtStr === null) throw new ValidationError('missing signedAt', 'signedAt');
if (!signature) throw new ValidationError('missing signature', 'signature');
const signedAt = Number(signedAtStr);
if (!Number.isFinite(signedAt)) {
throw new ValidationError('signedAt must be a number', 'signedAt');
}
const owner = await opts.store.getAddressOwner(address);
if (!owner) {
throw new UnauthorizedError(`address ${address} is not registered`);
}
await verifyPayload(opts.crypto, owner, {
address,
kind,
watched,
signedAt,
signature,
});
return { address, watched };
}
interface BlobRow { interface BlobRow {
msgId: string; msgId: string;
ciphertext: Uint8Array; ciphertext: Uint8Array;
receivedAt: number; receivedAt: number;
expiresAt: number; expiresAt: number;
/** V4.8 — relay-captured sender fingerprint. Optional for legacy rows. */
senderFp?: string;
} }
interface BlobWriter { interface BlobWriter {
@@ -345,11 +579,41 @@ function makeBlobWriter(store: InboxStore, pageLimit: number): BlobWriter {
}; };
} }
/**
* Per-connection bounded msgId tracker — defense in depth against duplicate
* delivery of the same blob to the same bridge socket. Cursor pagination
* already guarantees uniqueness in the happy path, but a dedup gate at the
* emit boundary catches any subtle bug (e.g. a flushTo race, a future
* refactor, an event-emit retry) without changing wire semantics.
*
* The cap is intentionally large enough to cover any realistic bridge
* pageLimit and small enough to bound memory under long-running streams.
*/
const DELIVERED_LRU_CAP = 4096;
class DeliveredIdLru {
private readonly seen = new Set<string>();
private readonly order: string[] = [];
/** Returns true if `msgId` has not been seen on this connection yet. */
add(msgId: string): boolean {
if (this.seen.has(msgId)) return false;
this.seen.add(msgId);
this.order.push(msgId);
if (this.order.length > DELIVERED_LRU_CAP) {
const evicted = this.order.shift()!;
this.seen.delete(evicted);
}
return true;
}
}
async function flushTo( async function flushTo(
writer: BlobWriter, writer: BlobWriter,
address: string, address: string,
startCursor: number, startCursor: number,
emit: (blob: BlobRow) => Promise<void>, emit: (blob: BlobRow) => Promise<void>,
delivered?: DeliveredIdLru,
): Promise<number> { ): Promise<number> {
let cursor = startCursor; let cursor = startCursor;
// Drain page-by-page so a backlog larger than `pageLimit` still flushes. // Drain page-by-page so a backlog larger than `pageLimit` still flushes.
@@ -358,7 +622,12 @@ async function flushTo(
const page = await writer.fetchPage(address, cursor); const page = await writer.fetchPage(address, cursor);
if (page.length === 0) break; if (page.length === 0) break;
for (const row of page) { for (const row of page) {
// Per-connection dedup gate — prevents the same msgId from being
// emitted twice if flushTo is somehow re-entered before the cursor
// catches up. See comment on `DeliveredIdLru`.
if (!delivered || delivered.add(row.msgId)) {
await emit(row); await emit(row);
}
if (row.receivedAt > cursor) cursor = row.receivedAt; if (row.receivedAt > cursor) cursor = row.receivedAt;
} }
if (page.length === 0) break; if (page.length === 0) break;
@@ -371,13 +640,26 @@ function serializeBlob(blob: BlobRow): {
ciphertext: string; ciphertext: string;
receivedAt: number; receivedAt: number;
expiresAt: number; expiresAt: number;
from?: string;
} { } {
return { const out: {
msgId: string;
ciphertext: string;
receivedAt: number;
expiresAt: number;
from?: string;
} = {
msgId: blob.msgId, msgId: blob.msgId,
ciphertext: toBase64(blob.ciphertext), ciphertext: toBase64(blob.ciphertext),
receivedAt: blob.receivedAt, receivedAt: blob.receivedAt,
expiresAt: blob.expiresAt, expiresAt: blob.expiresAt,
}; };
// V4.8 — relay-captured sender fingerprint. The transport-bridge
// wire format already accepts `from`; populating it lets receivers
// bootstrap unknown-sender first-contact via `shade.receive('fp:<hex>',
// env)` without requiring an out-of-band sender hint.
if (blob.senderFp) out.from = blob.senderFp;
return out;
} }
function buildPollResponse(blobs: BlobRow[], sinceFallback: number): { function buildPollResponse(blobs: BlobRow[], sinceFallback: number): {

View File

@@ -21,6 +21,19 @@ export interface InboxServerEventMap {
'inbox.expired_purged': { count: number }; 'inbox.expired_purged': { count: number };
'inbox.rate_limited': { route: string; key: string }; 'inbox.rate_limited': { route: string; key: string };
'inbox.quota_rejected': { address: string; reason: 'address-quota' | 'sender-quota' | 'body-too-large' }; 'inbox.quota_rejected': { address: string; reason: 'address-quota' | 'sender-quota' | 'body-too-large' };
// V4.7 — bridge presence transitions. Emitted on the 0↔1 boundary
// across tracked transports for a given address. Long-poll is
// intentionally NOT tracked: an LP client toggles in/out of a request
// every few seconds, and the resulting flapping would dominate the
// event stream. Push transports (WS, SSE) are also the only ones
// where the ~50ms revoke window for `BroadcastChannel.removeMember`
// matters — long-poll users are already on a slow path.
'inbox.peer_connected': { address: string; bridgeKind: 'ws' | 'sse' };
'inbox.peer_disconnected': {
address: string;
bridgeKind: 'ws' | 'sse';
reason: 'closed' | 'error';
};
} }
export type InboxServerEventName = keyof InboxServerEventMap; export type InboxServerEventName = keyof InboxServerEventMap;

View File

@@ -1,9 +1,12 @@
import type { Hono } from 'hono'; import { Hono } from 'hono';
import type { CryptoProvider } from '@shade/core'; import type { CryptoProvider } from '@shade/core';
import { createInboxRoutes, type InboxRoutesOptions } from './routes.js'; import { createInboxRoutes, type InboxRoutesOptions } from './routes.js';
import { MemoryInboxStore } from './memory-store.js'; import { MemoryInboxStore } from './memory-store.js';
import type { InboxStore } from './store.js'; import type { InboxStore } from './store.js';
import { InboxServerEvents } from './events.js'; import { InboxServerEvents } from './events.js';
import { createBlobRoutes, type BlobRoutesOptions } from './blob-routes.js';
import { MemoryBlobStore } from './memory-blob-store.js';
import type { BlobStore } from './blob-store.js';
export { createInboxRoutes } from './routes.js'; export { createInboxRoutes } from './routes.js';
export type { InboxRoutesOptions } from './routes.js'; export type { InboxRoutesOptions } from './routes.js';
@@ -32,6 +35,14 @@ export {
export type { InboxQuotaConfig } from './quota.js'; export type { InboxQuotaConfig } from './quota.js';
export { createBridgeRoutes } from './bridge.js'; export { createBridgeRoutes } from './bridge.js';
export type { BridgeRoutesOptions, BridgeKind } from './bridge.js'; export type { BridgeRoutesOptions, BridgeKind } from './bridge.js';
export { PresenceTracker } from './presence.js';
export type { TrackedBridgeKind } from './presence.js';
export { BridgeDeliveryLog } from './bridge-delivery-log.js';
export type { BridgeDeliveryLogOptions } from './bridge-delivery-log.js';
export { createBlobRoutes, DEFAULT_MAX_BLOB_BYTES } from './blob-routes.js';
export type { BlobRoutesOptions } from './blob-routes.js';
export { MemoryBlobStore } from './memory-blob-store.js';
export type { BlobStore, BlobSlotRecord, PutBlobResult } from './blob-store.js';
/** /**
* Create a standalone Shade Inbox Server. * Create a standalone Shade Inbox Server.
@@ -44,17 +55,45 @@ export type { BridgeRoutesOptions, BridgeKind } from './bridge.js';
* const app = new Hono(); * const app = new Hono();
* app.route('/', createInboxServer({ crypto })); * app.route('/', createInboxServer({ crypto }));
*/ */
export function createInboxServer(options: { export function createInboxServer(
options: {
crypto: CryptoProvider; crypto: CryptoProvider;
store?: InboxStore; store?: InboxStore;
disableRateLimit?: boolean; disableRateLimit?: boolean;
events?: InboxServerEvents; events?: InboxServerEvents;
} & Pick<InboxRoutesOptions, 'observability' | 'quota'>): Hono { /**
* V4.9 — when supplied, mounts the encrypted-blob primitive
* (`/v1/blob/<slotId>`) on the same Hono app. Pass `null` to
* explicitly opt out; omit to default to a `MemoryBlobStore`.
*/
blobStore?: BlobStore | null;
blobOptions?: Pick<BlobRoutesOptions, 'maxBlobBytes'>;
} & Pick<InboxRoutesOptions, 'observability' | 'quota' | 'bridgeDeliveryLog'>,
): Hono {
const store = options.store ?? new MemoryInboxStore(); const store = options.store ?? new MemoryInboxStore();
const routesOptions: InboxRoutesOptions = {}; const routesOptions: InboxRoutesOptions = {};
if (options.disableRateLimit !== undefined) routesOptions.disableRateLimit = options.disableRateLimit; if (options.disableRateLimit !== undefined) routesOptions.disableRateLimit = options.disableRateLimit;
if (options.events !== undefined) routesOptions.events = options.events; if (options.events !== undefined) routesOptions.events = options.events;
if (options.observability !== undefined) routesOptions.observability = options.observability; if (options.observability !== undefined) routesOptions.observability = options.observability;
if (options.quota !== undefined) routesOptions.quota = options.quota; if (options.quota !== undefined) routesOptions.quota = options.quota;
return createInboxRoutes(store, options.crypto, routesOptions); if (options.bridgeDeliveryLog !== undefined) routesOptions.bridgeDeliveryLog = options.bridgeDeliveryLog;
const inboxApp = createInboxRoutes(store, options.crypto, routesOptions);
// Compose with the blob primitive unless explicitly disabled. The
// blob routes share the same Hono app so a single port serves both.
if (options.blobStore === null) return inboxApp;
const blobStore = options.blobStore ?? new MemoryBlobStore();
const blobRoutesOptions: BlobRoutesOptions = {};
if (options.disableRateLimit !== undefined) blobRoutesOptions.disableRateLimit = options.disableRateLimit;
if (options.observability !== undefined) blobRoutesOptions.observability = options.observability;
if (options.blobOptions?.maxBlobBytes !== undefined) {
blobRoutesOptions.maxBlobBytes = options.blobOptions.maxBlobBytes;
}
const blobApp = createBlobRoutes(blobStore, options.crypto, blobRoutesOptions);
const composed = new Hono();
composed.route('/', inboxApp);
composed.route('/', blobApp);
return composed;
} }

View File

@@ -0,0 +1,85 @@
import { ConflictError, PreconditionFailedError } from '@shade/core';
import type { BlobStore, BlobSlotRecord, PutBlobResult } from './blob-store.js';
/**
* In-memory BlobStore — used in tests and as the default fallback when
* no SQLite/Postgres URL is configured. Rows are kept in a single Map.
*
* Etag is a strictly-monotonic per-process counter — guarantees a total
* order across writes even when many land in the same millisecond. (We
* could scope it per-slot, but a global counter keeps the implementation
* trivial and the etag values still uniquely identify the write that
* produced them, which is all CAS needs.)
*/
export class MemoryBlobStore implements BlobStore {
private slots = new Map<string, BlobSlotRecord>();
private nextEtag = 0;
async get(slotId: string): Promise<BlobSlotRecord | null> {
const r = this.slots.get(slotId);
if (!r) return null;
return {
slotId: r.slotId,
blob: new Uint8Array(r.blob),
ownerPubkey: new Uint8Array(r.ownerPubkey),
etag: r.etag,
updatedAt: r.updatedAt,
};
}
async put(args: {
slotId: string;
blob: Uint8Array;
ownerPubkey: Uint8Array;
expectedEtag: number | '*' | undefined;
now: number;
}): Promise<PutBlobResult> {
const existing = this.slots.get(args.slotId);
if (!existing) {
// Empty slot. `ifMatch: '*'` per RFC 7232 still fails — there is
// no entity to match. A numeric etag also fails (we have nothing
// to compare against).
if (args.expectedEtag !== undefined) {
throw new PreconditionFailedError(
`Slot ${args.slotId} does not exist; cannot match ifMatch=${String(args.expectedEtag)}`,
);
}
this.nextEtag = Math.max(this.nextEtag + 1, args.now);
const etag = this.nextEtag;
this.slots.set(args.slotId, {
slotId: args.slotId,
blob: new Uint8Array(args.blob),
ownerPubkey: new Uint8Array(args.ownerPubkey),
etag,
updatedAt: args.now,
});
return { created: true, etag, updatedAt: args.now };
}
// Slot exists. Pubkey check is the route layer's job — by the time
// we're here the signature has already been verified against
// `existing.ownerPubkey`.
if (args.expectedEtag === undefined) {
throw new ConflictError(
`Slot ${args.slotId} already exists; supply ifMatch to update`,
);
}
if (args.expectedEtag !== '*' && args.expectedEtag !== existing.etag) {
throw new PreconditionFailedError(
`Slot ${args.slotId} etag mismatch: expected=${args.expectedEtag} actual=${existing.etag}`,
);
}
this.nextEtag = Math.max(this.nextEtag + 1, args.now);
const etag = this.nextEtag;
existing.blob = new Uint8Array(args.blob);
existing.etag = etag;
existing.updatedAt = args.now;
return { created: false, etag, updatedAt: args.now };
}
async delete(slotId: string): Promise<boolean> {
return this.slots.delete(slotId);
}
}

View File

@@ -5,6 +5,7 @@ interface BlobRow {
ciphertext: Uint8Array; ciphertext: Uint8Array;
receivedAt: number; receivedAt: number;
expiresAt: number; expiresAt: number;
senderFp?: string;
} }
/** /**
@@ -33,6 +34,7 @@ export class MemoryInboxStore implements InboxStore {
msgId: string; msgId: string;
ciphertext: Uint8Array; ciphertext: Uint8Array;
expiresAt: number; expiresAt: number;
senderFp?: string;
}): Promise<{ created: boolean; receivedAt: number }> { }): Promise<{ created: boolean; receivedAt: number }> {
const list = this.blobs.get(args.address) ?? []; const list = this.blobs.get(args.address) ?? [];
const existing = list.find((r) => r.msgId === args.msgId); const existing = list.find((r) => r.msgId === args.msgId);
@@ -41,12 +43,14 @@ export class MemoryInboxStore implements InboxStore {
// multiple blobs land in the same millisecond. // multiple blobs land in the same millisecond.
const receivedAt = Math.max(this.nextReceivedAt + 1, Date.now()); const receivedAt = Math.max(this.nextReceivedAt + 1, Date.now());
this.nextReceivedAt = receivedAt; this.nextReceivedAt = receivedAt;
list.push({ const row: BlobRow = {
msgId: args.msgId, msgId: args.msgId,
ciphertext: new Uint8Array(args.ciphertext), ciphertext: new Uint8Array(args.ciphertext),
receivedAt, receivedAt,
expiresAt: args.expiresAt, expiresAt: args.expiresAt,
}); };
if (args.senderFp !== undefined) row.senderFp = args.senderFp;
list.push(row);
this.blobs.set(args.address, list); this.blobs.set(args.address, list);
return { created: true, receivedAt }; return { created: true, receivedAt };
} }
@@ -56,18 +60,36 @@ export class MemoryInboxStore implements InboxStore {
sinceCursor: number; sinceCursor: number;
now: number; now: number;
limit: number; limit: number;
}): Promise<Array<{ msgId: string; ciphertext: Uint8Array; receivedAt: number; expiresAt: number }>> { }): Promise<
Array<{
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
senderFp?: string;
}>
> {
const list = this.blobs.get(args.address) ?? []; const list = this.blobs.get(args.address) ?? [];
return list return list
.filter((r) => r.receivedAt > args.sinceCursor && r.expiresAt > args.now) .filter((r) => r.receivedAt > args.sinceCursor && r.expiresAt > args.now)
.sort((a, b) => a.receivedAt - b.receivedAt) .sort((a, b) => a.receivedAt - b.receivedAt)
.slice(0, args.limit) .slice(0, args.limit)
.map((r) => ({ .map((r) => {
const out: {
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
senderFp?: string;
} = {
msgId: r.msgId, msgId: r.msgId,
ciphertext: new Uint8Array(r.ciphertext), ciphertext: new Uint8Array(r.ciphertext),
receivedAt: r.receivedAt, receivedAt: r.receivedAt,
expiresAt: r.expiresAt, expiresAt: r.expiresAt,
})); };
if (r.senderFp !== undefined) out.senderFp = r.senderFp;
return out;
});
} }
async deleteBlob(address: string, msgId: string): Promise<boolean> { async deleteBlob(address: string, msgId: string): Promise<boolean> {

View File

@@ -0,0 +1,75 @@
/**
* V4.7 — bridge-connection presence tracking.
*
* The bridge handlers (`/v1/bridge/stream` and `/v1/bridge/ws`) call
* `markConnected` on open and `markDisconnected` on close. The tracker
* keeps a per-address set of connection ids; the `inbox.peer_connected`
* / `inbox.peer_disconnected` events fire only on the 0↔1 boundary so
* that two simultaneous bridges (e.g. SSE + WS during a transport-
* fallback handover) collapse into a single connected/disconnected
* pair from the consumer's point of view.
*
* Long-poll (`/v1/bridge/poll`) is intentionally NOT tracked — see the
* note on `InboxServerEventMap` in `events.ts`.
*/
import type { InboxServerEvents } from './events.js';
export type TrackedBridgeKind = 'ws' | 'sse';
export class PresenceTracker {
private readonly connections = new Map<string, Set<string>>();
private nextConnId = 1;
constructor(private readonly events: InboxServerEvents | null) {}
/** Allocate a fresh connection id for `markConnected` / `markDisconnected`. */
newConnectionId(): string {
return `c${this.nextConnId++}`;
}
/**
* Snapshot: is `address` currently connected over any tracked transport?
* Used by `/v1/bridge/presence` to push the initial state to a new
* subscriber.
*/
isOnline(address: string): boolean {
const set = this.connections.get(address);
return set !== undefined && set.size > 0;
}
markConnected(address: string, bridgeKind: TrackedBridgeKind, connectionId: string): void {
let set = this.connections.get(address);
const wasOnline = set !== undefined && set.size > 0;
if (!set) {
set = new Set();
this.connections.set(address, set);
}
set.add(connectionId);
if (!wasOnline) {
this.events?.emit('inbox.peer_connected', { address, bridgeKind });
}
}
markDisconnected(
address: string,
bridgeKind: TrackedBridgeKind,
connectionId: string,
reason: 'closed' | 'error',
): void {
const set = this.connections.get(address);
if (!set) return;
if (!set.delete(connectionId)) return;
if (set.size === 0) {
this.connections.delete(address);
this.events?.emit('inbox.peer_disconnected', { address, bridgeKind, reason });
}
}
/** Inspect the underlying map. Test/observability use only. */
snapshot(): Map<string, ReadonlySet<string>> {
return new Map(
Array.from(this.connections.entries(), ([k, v]) => [k, v as ReadonlySet<string>]),
);
}
}

View File

@@ -28,6 +28,7 @@ import type { InboxStore } from './store.js';
import { InboxServerEvents, shortHash } from './events.js'; import { InboxServerEvents, shortHash } from './events.js';
import { computeMsgId, isValidMsgId, constantTimeStringEqual } from './msg-id.js'; import { computeMsgId, isValidMsgId, constantTimeStringEqual } from './msg-id.js';
import { clampTtl, DEFAULT_INBOX_QUOTA, type InboxQuotaConfig } from './quota.js'; import { clampTtl, DEFAULT_INBOX_QUOTA, type InboxQuotaConfig } from './quota.js';
import type { BridgeDeliveryLog } from './bridge-delivery-log.js';
/** Max metadata-only body size (no ciphertext). 64 KB is enough headroom. */ /** Max metadata-only body size (no ciphertext). 64 KB is enough headroom. */
const MAX_META_BODY_SIZE = 64 * 1024; const MAX_META_BODY_SIZE = 64 * 1024;
@@ -54,6 +55,16 @@ export interface InboxRoutesOptions {
observability?: ObservabilityHook; observability?: ObservabilityHook;
/** Override quota policy. */ /** Override quota policy. */
quota?: Partial<InboxQuotaConfig>; quota?: Partial<InboxQuotaConfig>;
/**
* V4.8.4 — shared bridge delivery log. When provided (and the same
* instance is wired into `createBridgeRoutes`), the inbox-fetch route
* filters out blobs already pushed via bridge within the log's grace
* window. Without this, a recipient that runs both a bridge
* subscription and inbox-poll receives the same envelope twice.
* Optional — leaving it unset preserves the pre-V4.8.4 behavior of
* always returning every blob the cursor matches.
*/
bridgeDeliveryLog?: BridgeDeliveryLog;
} }
export function createInboxRoutes( export function createInboxRoutes(
@@ -171,6 +182,7 @@ export function createInboxRoutes(
await verifyPayload(crypto, owner, { ...body, address }); await verifyPayload(crypto, owner, { ...body, address });
await store.deleteAddress(address); await store.deleteAddress(address);
options.bridgeDeliveryLog?.forgetAddress(address);
events?.emit('inbox.address_deleted', { address }); events?.emit('inbox.address_deleted', { address });
return c.json({ ok: true }); return c.json({ ok: true });
}); });
@@ -255,11 +267,19 @@ export function createInboxRoutes(
); );
} }
// V4.8: capture sender fingerprint at PUT time. The sender's
// signing key was just verified for this request, so the fingerprint
// is bound to the same authentication path that authorized the
// store. Surfaced on bridge push + inbox-fetch responses to
// bootstrap unknown-sender first-contact (X3DH pair handshake).
const senderFp = await shortHash(senderKey);
const result = await store.putBlob({ const result = await store.putBlob({
address, address,
msgId, msgId,
ciphertext: ctBytes, ciphertext: ctBytes,
expiresAt, expiresAt,
senderFp,
}); });
if (result.created) { if (result.created) {
events?.emit('inbox.blob_stored', { events?.emit('inbox.blob_stored', {
@@ -309,31 +329,63 @@ export function createInboxRoutes(
await verifyPayload(crypto, owner, { ...body, address }); await verifyPayload(crypto, owner, { ...body, address });
const now = Date.now(); const now = Date.now();
const rows = await store.fetchBlobs({ const rawRows = await store.fetchBlobs({
address, address,
sinceCursor, sinceCursor,
now, now,
limit: quota.fetchPageLimit, limit: quota.fetchPageLimit,
}); });
// V4.8.4 — drop blobs the bridge has already pushed to this address
// within the grace window. This is the cross-channel dedup gate that
// makes "one inbox.send ⇒ one observable delivery" hold even when
// the recipient runs both a bridge subscription and inbox-poll. The
// cursor still advances over the whole `rawRows` window so the
// client doesn't get stuck behind suppressed blobs — pollOnce uses
// `nextCursor` (max receivedAt seen by the server, suppressed or
// not) for the next fetch.
const rows = options.bridgeDeliveryLog
? options.bridgeDeliveryLog.filterRecent(address, rawRows, now)
: rawRows;
let bytes = 0; let bytes = 0;
const blobs = rows.map((r) => { const blobs = rows.map((r) => {
bytes += r.ciphertext.length; bytes += r.ciphertext.length;
return { const out: {
msgId: string;
ciphertext: string;
receivedAt: number;
expiresAt: number;
from?: string;
} = {
msgId: r.msgId, msgId: r.msgId,
ciphertext: toBase64(r.ciphertext), ciphertext: toBase64(r.ciphertext),
receivedAt: r.receivedAt, receivedAt: r.receivedAt,
expiresAt: r.expiresAt, expiresAt: r.expiresAt,
}; };
// V4.8: surface sender fingerprint when present. Empty for blobs
// persisted by a pre-4.8 relay that didn't track sender provenance.
if (r.senderFp) out.from = r.senderFp;
return out;
}); });
const nextCursor = rows.length > 0 ? rows[rows.length - 1]!.receivedAt : sinceCursor; // Advance the cursor past the FULL rawRows window — including blobs
// we suppressed because the bridge already pushed them. If we
// anchored the cursor on `rows` only, suppressed blobs in the
// middle of the window would block all subsequent fetches forever
// (re-fetched on every poll, re-suppressed, no progress). The
// bridge-delivery contract is "the bridge frame is the canonical
// delivery"; if the recipient missed processing it, they fall back
// to ack-via-DELETE or the blob ages out at TTL — same as a
// recipient that crashes mid-handler in the no-bridge case.
const cursorAnchor = rawRows.length > 0 ? rawRows[rawRows.length - 1]!.receivedAt : sinceCursor;
const nextCursor = cursorAnchor;
events?.emit('inbox.blob_fetched', { address, count: blobs.length, bytes }); events?.emit('inbox.blob_fetched', { address, count: blobs.length, bytes });
return c.json({ return c.json({
blobs, blobs,
cursor: nextCursor, cursor: nextCursor,
hasMore: rows.length === quota.fetchPageLimit, hasMore: rawRows.length === quota.fetchPageLimit,
}); });
}); });
@@ -359,6 +411,10 @@ export function createInboxRoutes(
if (removed) { if (removed) {
events?.emit('inbox.blob_acked', { address, msgId }); events?.emit('inbox.blob_acked', { address, msgId });
} }
// Drop any bridge-delivery mark — keeps the log bounded under
// sustained traffic (otherwise long-lived addresses accumulate
// entries even after the underlying blob is gone).
options.bridgeDeliveryLog?.forget(address, msgId);
return c.json({ ok: removed }); return c.json({ ok: removed });
}); });

View File

@@ -36,12 +36,20 @@ export interface InboxStore {
* **Idempotent**: if a row already exists for `(address, msgId)` the * **Idempotent**: if a row already exists for `(address, msgId)` the
* implementation MUST return `{ created: false }` and leave the existing * implementation MUST return `{ created: false }` and leave the existing
* row untouched. A fresh insert returns `{ created: true, receivedAt }`. * row untouched. A fresh insert returns `{ created: true, receivedAt }`.
*
* `senderFp` (V4.8+) is the 8-byte hex of SHA-256(senderSigningKey)
* — captured at PUT time when the relay verified the sender's
* signature. Optional so legacy 4.7 callers compile, but populated by
* `createInboxRoutes` from 4.8 onward and surfaced on bridge push +
* inbox-fetch responses to bootstrap unknown-sender first-contact
* (X3DH pair handshake).
*/ */
putBlob(args: { putBlob(args: {
address: string; address: string;
msgId: string; msgId: string;
ciphertext: Uint8Array; ciphertext: Uint8Array;
expiresAt: number; expiresAt: number;
senderFp?: string;
}): Promise<{ created: boolean; receivedAt: number }>; }): Promise<{ created: boolean; receivedAt: number }>;
/** /**
@@ -57,7 +65,20 @@ export interface InboxStore {
sinceCursor: number; sinceCursor: number;
now: number; now: number;
limit: number; limit: number;
}): Promise<Array<{ msgId: string; ciphertext: Uint8Array; receivedAt: number; expiresAt: number }>>; }): Promise<
Array<{
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
/**
* Sender fingerprint — 8-byte hex of SHA-256(senderSigningKey)
* captured at PUT time. Empty/undefined for blobs persisted by a
* pre-4.8 relay that didn't track sender provenance.
*/
senderFp?: string;
}>
>;
/** /**
* Delete a single blob by `(address, msgId)`. Returns true if a row was * Delete a single blob by `(address, msgId)`. Returns true if a row was

View File

@@ -0,0 +1,295 @@
import { describe, test, expect, beforeEach } from 'bun:test';
import { Hono } from 'hono';
import {
createBlobRoutes,
MemoryBlobStore,
type BlobStore,
} from '../src/index.js';
import { signPayload } from '@shade/server';
import { SubtleCryptoProvider, ed25519PublicKeyFromSeed } from '@shade/crypto-web';
import { toBase64, fromBase64 } from '@shade/core';
const crypto = new SubtleCryptoProvider();
function randBytes(n: number): Uint8Array {
const buf = new Uint8Array(n);
globalThis.crypto.getRandomValues(buf);
return buf;
}
function hex(bytes: Uint8Array): string {
let s = '';
for (const b of bytes) s += b.toString(16).padStart(2, '0');
return s;
}
describe('Shade Blob Routes (V4.9)', () => {
let store: BlobStore;
let app: Hono;
beforeEach(() => {
store = new MemoryBlobStore();
app = createBlobRoutes(store, crypto, { disableRateLimit: true });
});
async function makeOwner() {
const seed = randBytes(32);
const pubkey = ed25519PublicKeyFromSeed(seed);
return { seed, pubkey };
}
function makeSlotId(): string {
return hex(randBytes(32));
}
async function signedPut(args: {
slotId: string;
blob: Uint8Array;
seed: Uint8Array;
pubkey: Uint8Array;
ifMatch?: string;
}) {
const payload: Record<string, unknown> = {
ownerPubkey: toBase64(args.pubkey),
blob: toBase64(args.blob),
slotId: args.slotId,
};
if (args.ifMatch !== undefined) payload.ifMatch = args.ifMatch;
const signed = await signPayload(crypto, args.seed, payload);
const { slotId: _omit, ...wire } = signed as Record<string, unknown>;
return app.request(`/v1/blob/${args.slotId}`, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(wire),
});
}
async function signedDelete(args: {
slotId: string;
seed: Uint8Array;
}) {
const signed = await signPayload(crypto, args.seed, {
slotId: args.slotId,
});
const { slotId: _omit, ...wire } = signed as Record<string, unknown>;
return app.request(`/v1/blob/${args.slotId}`, {
method: 'DELETE',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(wire),
});
}
// ─── GET ─────────────────────────────────────────────────────
test('GET on missing slot returns 404', async () => {
const slotId = makeSlotId();
const res = await app.request(`/v1/blob/${slotId}`);
expect(res.status).toBe(404);
});
test('GET requires lowercase 64-hex slotId', async () => {
const res = await app.request('/v1/blob/notahex');
expect(res.status).toBe(400);
const res2 = await app.request(`/v1/blob/${'A'.repeat(64)}`);
expect(res2.status).toBe(400);
});
// ─── PUT (TOFU) ──────────────────────────────────────────────
test('first PUT creates slot and returns etag', async () => {
const slotId = makeSlotId();
const owner = await makeOwner();
const blob = randBytes(128);
const res = await signedPut({ slotId, blob, ...owner });
expect(res.status).toBe(200);
const json = (await res.json()) as { created: boolean; etag: string };
expect(json.created).toBe(true);
expect(typeof json.etag).toBe('string');
const got = await app.request(`/v1/blob/${slotId}`);
expect(got.status).toBe(200);
const back = (await got.json()) as { blob: string; etag: string };
expect(fromBase64(back.blob)).toEqual(blob);
expect(back.etag).toBe(json.etag);
});
test('PUT without ifMatch on populated slot returns 409', async () => {
const slotId = makeSlotId();
const owner = await makeOwner();
await signedPut({ slotId, blob: randBytes(64), ...owner });
const res = await signedPut({ slotId, blob: randBytes(64), ...owner });
expect(res.status).toBe(409);
const json = (await res.json()) as { code: string };
expect(json.code).toBe('SHADE_CONFLICT');
});
test('PUT with stale ifMatch returns 412', async () => {
const slotId = makeSlotId();
const owner = await makeOwner();
const r1 = await signedPut({ slotId, blob: randBytes(64), ...owner });
const j1 = (await r1.json()) as { etag: string };
// Use an etag we know does not match.
const stale = String(Number(j1.etag) - 999);
const res = await signedPut({
slotId,
blob: randBytes(64),
...owner,
ifMatch: stale,
});
expect(res.status).toBe(412);
const json = (await res.json()) as { code: string };
expect(json.code).toBe('SHADE_PRECONDITION_FAILED');
});
test('PUT with matching ifMatch updates and bumps etag', async () => {
const slotId = makeSlotId();
const owner = await makeOwner();
const r1 = await signedPut({ slotId, blob: randBytes(64), ...owner });
const j1 = (await r1.json()) as { etag: string };
const r2 = await signedPut({
slotId,
blob: randBytes(64),
...owner,
ifMatch: j1.etag,
});
expect(r2.status).toBe(200);
const j2 = (await r2.json()) as { created: boolean; etag: string };
expect(j2.created).toBe(false);
expect(Number(j2.etag)).toBeGreaterThan(Number(j1.etag));
});
test('PUT with ifMatch="*" unconditionally overwrites existing slot', async () => {
const slotId = makeSlotId();
const owner = await makeOwner();
await signedPut({ slotId, blob: randBytes(64), ...owner });
const res = await signedPut({
slotId,
blob: randBytes(64),
...owner,
ifMatch: '*',
});
expect(res.status).toBe(200);
});
test('PUT with ifMatch="*" on empty slot returns 412', async () => {
const slotId = makeSlotId();
const owner = await makeOwner();
const res = await signedPut({
slotId,
blob: randBytes(64),
...owner,
ifMatch: '*',
});
expect(res.status).toBe(412);
});
test('PUT by a different owner key on existing slot is rejected', async () => {
const slotId = makeSlotId();
const ownerA = await makeOwner();
await signedPut({ slotId, blob: randBytes(64), ...ownerA });
const ownerB = await makeOwner();
const res = await signedPut({
slotId,
blob: randBytes(64),
...ownerB,
ifMatch: '*',
});
expect(res.status).toBe(401);
});
test('PUT with bad signature is rejected', async () => {
const slotId = makeSlotId();
const owner = await makeOwner();
// Sign the payload, then mutate the blob bytes — signature no
// longer matches the canonicalized body.
const blob = randBytes(64);
const payload = {
ownerPubkey: toBase64(owner.pubkey),
blob: toBase64(blob),
slotId,
};
const signed = await signPayload(crypto, owner.seed, payload);
(signed as any).blob = toBase64(randBytes(64));
const { slotId: _omit, ...wire } = signed as Record<string, unknown>;
const res = await app.request(`/v1/blob/${slotId}`, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(wire),
});
expect(res.status).toBe(401);
});
test('PUT rejects empty blob and oversized blob', async () => {
const slotId = makeSlotId();
const owner = await makeOwner();
const empty = await signedPut({ slotId, blob: new Uint8Array(0), ...owner });
expect(empty.status).toBe(400);
const tooBig = await signedPut({
slotId,
blob: randBytes(70 * 1024),
...owner,
});
expect(tooBig.status).toBe(400);
});
// ─── DELETE ──────────────────────────────────────────────────
test('DELETE clears slot and lets a fresh key TOFU re-claim', async () => {
const slotId = makeSlotId();
const ownerA = await makeOwner();
await signedPut({ slotId, blob: randBytes(64), ...ownerA });
const del = await signedDelete({ slotId, seed: ownerA.seed });
expect(del.status).toBe(200);
// Slot is gone.
const gone = await app.request(`/v1/blob/${slotId}`);
expect(gone.status).toBe(404);
// A fresh owner can now claim it.
const ownerB = await makeOwner();
const claim = await signedPut({ slotId, blob: randBytes(64), ...ownerB });
expect(claim.status).toBe(200);
});
test('DELETE by a different key is rejected', async () => {
const slotId = makeSlotId();
const ownerA = await makeOwner();
await signedPut({ slotId, blob: randBytes(64), ...ownerA });
const ownerB = await makeOwner();
const res = await signedDelete({ slotId, seed: ownerB.seed });
expect(res.status).toBe(401);
});
test('DELETE on missing slot returns 404', async () => {
const slotId = makeSlotId();
const owner = await makeOwner();
const res = await signedDelete({ slotId, seed: owner.seed });
expect(res.status).toBe(404);
});
// ─── Cross-slot replay ───────────────────────────────────────
test('PUT signed for slot A is rejected against slot B', async () => {
const slotA = makeSlotId();
const slotB = makeSlotId();
const owner = await makeOwner();
const blob = randBytes(64);
// Sign for slotA, send to slotB (URL).
const payload = {
ownerPubkey: toBase64(owner.pubkey),
blob: toBase64(blob),
slotId: slotA,
};
const signed = await signPayload(crypto, owner.seed, payload);
const { slotId: _omit, ...wire } = signed as Record<string, unknown>;
const res = await app.request(`/v1/blob/${slotB}`, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(wire),
});
expect(res.status).toBe(401);
});
});

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/inbox", "name": "@shade/inbox",
"version": "4.0.0", "version": "4.11.1",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -0,0 +1,208 @@
import type { CryptoProvider } from '@shade/core';
import {
NetworkError,
toBase64,
fromBase64,
ShadeError,
ValidationError,
} from '@shade/core';
import { signPayload } from '@shade/server';
/**
* Low-level HTTP client for the V4.9 encrypted-blob primitive
* (`/v1/blob/<slotId>`). Stateless and reusable; higher-level wrappers
* (e.g. `Profile` in `@shade/sdk`) compose this client.
*
* The client doesn't care what the blob bytes mean — it just transports
* them. Callers are responsible for AEAD-sealing/opening, deriving the
* slotId from the master key, and managing the signing key.
*/
export interface BlobClientOptions {
baseUrl: string;
crypto: CryptoProvider;
/** Optional fetch override (defaults to globalThis.fetch). */
fetch?: typeof fetch;
}
export interface BlobGetResult {
blob: Uint8Array;
/** ETag string — pass back as `ifMatch` to do a CAS update. */
etag: string;
updatedAt: number;
}
export interface BlobPutResult {
/** True if this PUT created the slot, false if it updated an existing one. */
created: boolean;
/** New ETag after the write. */
etag: string;
updatedAt: number;
}
export class BlobClient {
private readonly fetchImpl: typeof fetch;
constructor(private readonly options: BlobClientOptions) {
const f = options.fetch ?? globalThis.fetch;
this.fetchImpl = f.bind(globalThis);
}
/**
* Read a slot. Returns null if no blob has ever been written there
* (or if it was DELETE'd). GET is unauthenticated — see the
* `BlobStore` JSDoc for the threat-model rationale.
*/
async get(slotIdHex: string): Promise<BlobGetResult | null> {
validateSlotIdHex(slotIdHex);
const url = joinUrl(this.options.baseUrl, `/v1/blob/${slotIdHex}`);
let res: Response;
try {
res = await this.fetchImpl(url, { method: 'GET' });
} catch (err) {
throw new NetworkError(`Blob GET failed: ${(err as Error).message}`);
}
if (res.status === 404) return null;
const text = await res.text();
let json: any;
try {
json = text.length > 0 ? JSON.parse(text) : {};
} catch {
throw new NetworkError(`Blob response not JSON: ${text.slice(0, 200)}`, res.status);
}
if (!res.ok) {
throw new ShadeError(
String(json.code ?? 'SHADE_NETWORK'),
String(json.message ?? text),
);
}
return {
blob: fromBase64(String(json.blob)),
etag: String(json.etag),
updatedAt: Number(json.updatedAt),
};
}
/**
* Create or update a slot.
*
* `ifMatch` semantics:
* - `undefined`: create-only. Slot must be empty (else 409).
* - `<etag-string>`: compare-and-swap. Must match (else 412).
* - `'*'`: unconditional overwrite. Slot must already exist (else 412).
*/
async put(args: {
slotIdHex: string;
blob: Uint8Array;
/** 32-byte Ed25519 seed (== `signingPrivateKey`). */
signingPrivateKey: Uint8Array;
/** Pubkey paired to `signingPrivateKey`. */
ownerPubkey: Uint8Array;
ifMatch?: string;
}): Promise<BlobPutResult> {
validateSlotIdHex(args.slotIdHex);
if (args.blob.length === 0) {
throw new ValidationError('Empty blob');
}
if (args.ownerPubkey.length !== 32) {
throw new ValidationError('ownerPubkey must be 32 bytes (Ed25519)');
}
const payload: Record<string, unknown> = {
ownerPubkey: toBase64(args.ownerPubkey),
blob: toBase64(args.blob),
slotId: args.slotIdHex,
};
if (args.ifMatch !== undefined) payload.ifMatch = args.ifMatch;
const signed = await signPayload(
this.options.crypto,
args.signingPrivateKey,
payload,
);
// `slotId` was used for the signature canonicalization to bind it
// into the payload; the server rebuilds the same canonical form
// by mixing the URL slotId back in. Strip it from the wire body
// so we don't send it twice (URL is the path param).
const { slotId: _omit, ...wireBody } = signed as Record<string, unknown>;
const url = joinUrl(this.options.baseUrl, `/v1/blob/${args.slotIdHex}`);
const json = await this.requestJson('PUT', url, wireBody);
return {
created: Boolean(json.created),
etag: String(json.etag),
updatedAt: Number(json.updatedAt),
};
}
/**
* Delete a slot — the next PUT TOFU-claims it again, possibly under
* a fresh signing key (e.g. after a rotation). Used by the "forget
* everything" path.
*/
async delete(args: {
slotIdHex: string;
signingPrivateKey: Uint8Array;
}): Promise<boolean> {
validateSlotIdHex(args.slotIdHex);
const signed = await signPayload(this.options.crypto, args.signingPrivateKey, {
slotId: args.slotIdHex,
});
const { slotId: _omit, ...wireBody } = signed as Record<string, unknown>;
const url = joinUrl(this.options.baseUrl, `/v1/blob/${args.slotIdHex}`);
const json = await this.requestJson('DELETE', url, wireBody);
return Boolean(json.ok);
}
// ─── HTTP plumbing ──────────────────────────────────────────
private async requestJson(method: string, url: string, body: unknown): Promise<any> {
let res: Response;
try {
res = await this.fetchImpl(url, {
method,
headers: { 'content-type': 'application/json' },
body: JSON.stringify(body),
});
} catch (err) {
throw new NetworkError(`Blob request failed: ${(err as Error).message}`);
}
const text = await res.text();
let json: any;
try {
json = text.length > 0 ? JSON.parse(text) : {};
} catch {
throw new NetworkError(`Blob response not JSON: ${text.slice(0, 200)}`, res.status);
}
if (!res.ok) {
throw new ShadeError(
String(json.code ?? 'SHADE_NETWORK'),
String(json.message ?? text),
);
}
return json;
}
}
function validateSlotIdHex(s: string): void {
if (!/^[0-9a-f]{64}$/.test(s)) {
throw new ValidationError('slotIdHex must be 64 lowercase hex chars (32 bytes)');
}
}
function joinUrl(base: string, path: string): string {
if (base.endsWith('/') && path.startsWith('/')) return base + path.slice(1);
if (!base.endsWith('/') && !path.startsWith('/')) return base + '/' + path;
return base + path;
}
/** Convert a 32-byte slotId Uint8Array into the lowercase-hex wire form. */
export function slotIdToHex(slotId: Uint8Array): string {
if (slotId.length !== 32) {
throw new ValidationError('slotId must be 32 bytes');
}
let s = '';
for (let i = 0; i < slotId.length; i++) {
s += slotId[i]!.toString(16).padStart(2, '0');
}
return s;
}

View File

@@ -40,6 +40,16 @@ export interface FetchedBlob {
receivedAt: number; receivedAt: number;
/** Absolute expiry time (ms since epoch) reported by the server. */ /** Absolute expiry time (ms since epoch) reported by the server. */
expiresAt: number; expiresAt: number;
/**
* Sender fingerprint — 8-byte hex of SHA-256(senderSigningKey),
* captured by the relay at PUT time when the sender's signature was
* verified. Empty/undefined when the relay is pre-4.8 or the blob
* predates sender-fingerprint tracking. Use as an unknown-sender
* bootstrap label (`fp:<hex>`) for X3DH first-contact; the
* authoritative sender identity is recovered post-decrypt from the
* envelope itself, so `from` is a hint, not a trust anchor.
*/
from?: string;
} }
export interface FetchResult { export interface FetchResult {
@@ -52,7 +62,14 @@ export class InboxClient {
private readonly fetchImpl: typeof fetch; private readonly fetchImpl: typeof fetch;
constructor(private readonly options: InboxClientOptions) { constructor(private readonly options: InboxClientOptions) {
this.fetchImpl = options.fetch ?? globalThis.fetch; // Bind once. The browser's `globalThis.fetch` is a WebIDL bound
// operation that throws "Illegal invocation" when called as a method
// on another object (which is what `this.fetchImpl(...)` does).
// Node/Bun fetch tolerates a free receiver, but binding is harmless.
// A consumer-supplied `options.fetch` is bound to the global too —
// a fetch that requires a specific receiver must bind itself.
const f = options.fetch ?? globalThis.fetch;
this.fetchImpl = f.bind(globalThis);
} }
/** /**
@@ -144,12 +161,18 @@ export class InboxClient {
); );
const blobs = Array.isArray(json.blobs) ? json.blobs : []; const blobs = Array.isArray(json.blobs) ? json.blobs : [];
return { return {
blobs: blobs.map((b: any) => ({ blobs: blobs.map((b: any): FetchedBlob => {
const out: FetchedBlob = {
msgId: String(b.msgId), msgId: String(b.msgId),
ciphertext: fromBase64(String(b.ciphertext)), ciphertext: fromBase64(String(b.ciphertext)),
receivedAt: Number(b.receivedAt), receivedAt: Number(b.receivedAt),
expiresAt: Number(b.expiresAt), expiresAt: Number(b.expiresAt),
})), };
// V4.8 — relay-supplied sender fingerprint hint. Optional; absent
// on pre-4.8 relays or for blobs persisted before tracking landed.
if (typeof b.from === 'string' && b.from.length > 0) out.from = b.from;
return out;
}),
cursor: Number(json.cursor ?? sinceCursor), cursor: Number(json.cursor ?? sinceCursor),
hasMore: Boolean(json.hasMore), hasMore: Boolean(json.hasMore),
}; };

View File

@@ -16,9 +16,20 @@ import { InboxClientEvents, type InboxClientListener } from './events.js';
* decrypt path it owns) and either return a sender-hint for telemetry * decrypt path it owns) and either return a sender-hint for telemetry
* (the address the SDK extracted, or `null`) or throw to keep the blob * (the address the SDK extracted, or `null`) or throw to keep the blob
* on the server for a later retry. * on the server for a later retry.
*
* V4.8: `raw.from` is the relay-captured sender fingerprint (8-byte hex
* of SHA-256 over the sender's signing key). Empty when the relay is
* pre-4.8 or didn't track the sender. Use it as the `fp:<hex>` bootstrap
* label for X3DH first-contact handshakes.
*/ */
export type DecryptHandler = ( export type DecryptHandler = (
raw: { msgId: string; ciphertext: Uint8Array; receivedAt: number; expiresAt: number }, raw: {
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
from?: string;
},
) => Promise<string | null | undefined> | string | null | undefined; ) => Promise<string | null | undefined> | string | null | undefined;
export interface InboxOptions { export interface InboxOptions {
@@ -52,6 +63,14 @@ export interface InboxOptions {
const DEFAULT_TTL_SECONDS = 7 * 24 * 60 * 60; const DEFAULT_TTL_SECONDS = 7 * 24 * 60 * 60;
const DEFAULT_POLL_INTERVAL_MS = 30_000; const DEFAULT_POLL_INTERVAL_MS = 30_000;
const DEFAULT_MAX_ATTEMPTS = 10; const DEFAULT_MAX_ATTEMPTS = 10;
/**
* Cap for the cross-channel msgId dedup LRU. Each entry is a 64-char hex
* string; 4096 entries ≈ 256 KiB of overhead, plenty of headroom for
* bursty traffic (the LRU only needs to span the window between a bridge
* push and the next inbox-poll catching up — typically 30 s × the
* recipient's throughput).
*/
const DEFAULT_DEDUP_LRU_CAP = 4096;
/** /**
* High-level inbox orchestrator. * High-level inbox orchestrator.
@@ -94,6 +113,23 @@ export class Inbox {
private started = false; private started = false;
private registered = false; private registered = false;
/**
* Bounded msgId dedup window. Used by both the inbox-poll path
* (`pollOnce` → `handleBlob`) and the bridge-push path
* (`acceptBridgeFrame`). The relay stores blobs durably and pushes
* them to every active delivery channel; without a shared dedup gate
* here the recipient processes the same envelope twice — once from
* the bridge, again from the next inbox-poll. The duplicate receive
* trips on consumed one-time prekeys ("OPK not found") and pollutes
* logs even when the canonical first delivery succeeded. See V4.8.3
* Prism FR `cross-channel-duplicate-fanout-v4.8.2.md`.
*
* Insertion order is FIFO; the oldest msgId is evicted once the LRU
* exceeds `DEFAULT_DEDUP_LRU_CAP`.
*/
private readonly deliveredIds = new Set<string>();
private readonly deliveredOrder: string[] = [];
constructor(private readonly options: InboxOptions) { constructor(private readonly options: InboxOptions) {
const clientOptions: ConstructorParameters<typeof InboxClient>[0] = { const clientOptions: ConstructorParameters<typeof InboxClient>[0] = {
baseUrl: options.baseUrl, baseUrl: options.baseUrl,
@@ -149,6 +185,14 @@ export class Inbox {
signingKey: this.options.signingPublicKey, signingKey: this.options.signingPublicKey,
}); });
this.registered = true; this.registered = true;
// V4.8: gate the first poll on register success. `start()` calls
// `register()` fire-and-forget; without this kick, the very first
// `pollOnce()` (scheduled synchronously alongside register) would
// race the register HTTP RTT and return SHADE_NOT_FOUND. The
// pollOnce() guard skips polls until `registered === true`; this
// immediate schedule ensures we don't wait the full pollIntervalMs
// for the next attempt once register lands.
if (this.started) this.schedulePoll(0);
} }
/** Drop the address from the server. Local queue/cursor are preserved. */ /** Drop the address from the server. Local queue/cursor are preserved. */
@@ -203,7 +247,10 @@ export class Inbox {
* after a push-trigger arrives). Does not throw on transient errors. * after a push-trigger arrives). Does not throw on transient errors.
*/ */
async tick(): Promise<{ flushed: number; received: number }> { async tick(): Promise<{ flushed: number; received: number }> {
const flushed = await this.flushOnce(); const flushResult = await this.flushOnce();
// `null` means another flush was concurrent; report 0 newly-flushed
// for this caller (the other flush counted them).
const flushed = flushResult?.delivered ?? 0;
const received = await this.pollOnce(); const received = await this.pollOnce();
return { flushed, received }; return { flushed, received };
} }
@@ -217,7 +264,11 @@ export class Inbox {
this.scheduleRegisterRetry(); this.scheduleRegisterRetry();
}); });
this.scheduleFlush(); this.scheduleFlush();
this.schedulePoll(0); // V4.8: do NOT schedule the first poll synchronously. `register()`
// success kicks `schedulePoll(0)` so the first poll fires after the
// server has acknowledged the address. Pre-fix this raced register
// and burned a 404 on every fresh-address `start()`.
if (this.registered) this.schedulePoll(0);
} }
/** Stop background timers. Pending entries remain in the queue. */ /** Stop background timers. Pending entries remain in the queue. */
@@ -253,11 +304,27 @@ export class Inbox {
this.flushTimer = setTimeout(() => { this.flushTimer = setTimeout(() => {
this.flushTimer = null; this.flushTimer = null;
this.flushOnce() this.flushOnce()
.then(() => { .then((result) => {
// If anything is still queued, retry with backoff. // `result === null` means another flush was already in flight
this.queueStore.size().then((n) => { // and this call early-returned via the `flushing` guard. The
if (n > 0 && this.started) this.scheduleFlush(15_000); // already-running flush will reschedule itself when it
}); // finishes; do not double-schedule from here.
if (result === null) return;
if (result.remaining === 0) return;
// V4.8.5 — distinguish healthy-drain-but-more-queued from
// all-attempts-failed. Pre-fix, both cases used a 15 s
// backoff. Under sustained traffic (Prism's typing-into-a-
// chatty-shell pattern), bursts of envelopes enqueued
// *during* a flush would sit ~1015 s behind the backoff
// timer before the next drain — visible to the receiver as a
// "10 s silence then 25-frame burst" wave. Healthy drain
// (delivered > 0) means the network is fine and we should
// immediately drain whatever piled up; reserve the 15 s
// retry for the case where every attempt this round failed.
if (this.started) {
const delay = result.delivered > 0 ? 0 : 15_000;
this.scheduleFlush(delay);
}
}) })
.catch(() => { .catch(() => {
if (this.started) this.scheduleFlush(15_000); if (this.started) this.scheduleFlush(15_000);
@@ -276,13 +343,47 @@ export class Inbox {
}, delayMs); }, delayMs);
} }
private async flushOnce(): Promise<number> { /**
if (this.flushing) return 0; * Drain the outgoing queue. Returns `null` if another flush is already
* in flight (the running flush owns the rescheduling); otherwise
* returns the count of newly-delivered envelopes and the queue size
* after the drain so the caller can decide whether to immediately
* re-flush (more piled up during the drain — healthy network) or
* back off (everything failed).
*
* V4.8.5: drain is parallel-per-recipient. Each `recipientAddress`
* gets its own sequential worker (so per-peer order is preserved),
* but distinct recipients run concurrently. Pre-fix, a single slow
* POST head-of-line-blocked the entire queue — including small
* frames bound for unrelated peers. See Prism FR
* `per-recipient-flush-concurrency-v4.8.md`.
*/
private async flushOnce(): Promise<{ delivered: number; remaining: number } | null> {
if (this.flushing) return null;
this.flushing = true; this.flushing = true;
let delivered = 0; let delivered = 0;
try { try {
const entries = await this.queueStore.list(); const entries = await this.queueStore.list();
// Group by recipient. Within a bucket we drain sequentially so
// per-peer message order matches enqueue order (the relay
// assigns `receivedAt` on PUT arrival; concurrent POSTs to the
// same peer would let the second arrive first and the recipient
// would observe out-of-order envelopes). Across buckets, no
// ordering guarantee exists in Shade's wire model anyway, so
// parallel drain is safe.
const buckets = new Map<string, OutgoingEntry[]>();
for (const entry of entries) { for (const entry of entries) {
let bucket = buckets.get(entry.recipientAddress);
if (!bucket) {
bucket = [];
buckets.set(entry.recipientAddress, bucket);
}
bucket.push(entry);
}
const drainBucket = async (bucket: OutgoingEntry[]): Promise<number> => {
let count = 0;
for (const entry of bucket) {
try { try {
const result = await this.client.put({ const result = await this.client.put({
recipientAddress: entry.recipientAddress, recipientAddress: entry.recipientAddress,
@@ -291,7 +392,7 @@ export class Inbox {
ttlSeconds: entry.ttlSeconds, ttlSeconds: entry.ttlSeconds,
}); });
await this.queueStore.remove(entry.recipientAddress, entry.msgId); await this.queueStore.remove(entry.recipientAddress, entry.msgId);
delivered++; count++;
this.events.emit('inbox.message_delivered', { this.events.emit('inbox.message_delivered', {
recipientAddress: entry.recipientAddress, recipientAddress: entry.recipientAddress,
msgId: result.msgId, msgId: result.msgId,
@@ -311,10 +412,18 @@ export class Inbox {
} }
} }
} }
return count;
};
const counts = await Promise.all(
Array.from(buckets.values(), drainBucket),
);
delivered = counts.reduce((a, b) => a + b, 0);
} finally { } finally {
this.flushing = false; this.flushing = false;
} }
return delivered; const remaining = await this.queueStore.size();
return { delivered, remaining };
} }
private async pollOnce(): Promise<number> { private async pollOnce(): Promise<number> {
@@ -357,9 +466,52 @@ export class Inbox {
return total; return total;
} }
/**
* Feed a blob delivered by a bridge transport (WS / SSE / long-poll
* push) into the same dispatch + ack pipeline that `pollOnce` uses.
*
* Wire-up pattern:
* ```ts
* const bridge = new FallbackBridgeTransport([...]);
* await bridge.connect({
* onMessage: async (msg) => {
* await inbox.acceptBridgeFrame({
* msgId: msg.msgId!, // present on v4.8+ relays
* ciphertext: msg.bytes,
* receivedAt: msg.receivedAt,
* expiresAt: msg.expiresAt ?? Date.now() + 7 * 24 * 3600 * 1000,
* ...(msg.from !== undefined ? { from: msg.from } : {}),
* });
* },
* });
* ```
*
* The Inbox's bounded msgId LRU is shared between this path and
* `pollOnce`, so whichever channel delivers first wins; the
* other channel acks-and-skips when the same msgId comes back
* around. Both paths also DELETE the blob from the relay on success
* so subsequent polls don't see it either.
*
* Returns `true` if the blob was newly dispatched, `false` if it
* was a duplicate or rejected by the handler (handler still gets a
* chance to retry on the next poll if it threw).
*/
async acceptBridgeFrame(blob: FetchedBlob): Promise<boolean> {
return this.handleBlob(blob);
}
private async handleBlob(blob: FetchedBlob): Promise<boolean> { private async handleBlob(blob: FetchedBlob): Promise<boolean> {
if (!this.incomingHandler) return false; if (!this.incomingHandler) return false;
// Cross-channel msgId dedup. If the bridge already delivered this
// blob, the inbox-poll copy must not re-dispatch (would re-trigger
// X3DH / consume an OPK we no longer have). We still ack so the
// relay drops the now-redundant copy.
if (this.deliveredIds.has(blob.msgId)) {
await this.ackQuietly(blob.msgId);
return false;
}
// Defense-in-depth: verify msgId ↔ ciphertext at the client too. A // Defense-in-depth: verify msgId ↔ ciphertext at the client too. A
// server bug or malicious operator can't sneak a different blob past // server bug or malicious operator can't sneak a different blob past
// the client's hash check. // the client's hash check.
@@ -375,12 +527,20 @@ export class Inbox {
let senderHint: string | null = null; let senderHint: string | null = null;
try { try {
const result = await this.incomingHandler({ const raw: {
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
from?: string;
} = {
msgId: blob.msgId, msgId: blob.msgId,
ciphertext: blob.ciphertext, ciphertext: blob.ciphertext,
receivedAt: blob.receivedAt, receivedAt: blob.receivedAt,
expiresAt: blob.expiresAt, expiresAt: blob.expiresAt,
}); };
if (blob.from !== undefined) raw.from = blob.from;
const result = await this.incomingHandler(raw);
senderHint = result ?? null; senderHint = result ?? null;
} catch (err) { } catch (err) {
this.events.emit('inbox.message_decrypt_failed', { this.events.emit('inbox.message_decrypt_failed', {
@@ -391,17 +551,34 @@ export class Inbox {
return false; return false;
} }
try { // Mark before the ack so a slow-network ack doesn't leave a window
await this.client.ack({ address: this.options.ownAddress, msgId: blob.msgId }); // where a parallel pollOnce sees the same msgId and re-dispatches.
} catch (err) { this.recordDelivered(blob.msgId);
// Decryption succeeded; ack just failed. Will be retried later, and await this.ackQuietly(blob.msgId);
// the duplicate-message ratchet check on `Shade.receive` will dedupe.
console.warn('[Shade] Inbox ack failed (will retry):', (err as Error).message);
}
this.events.emit('inbox.message_received', { this.events.emit('inbox.message_received', {
senderHint, senderHint,
msgId: blob.msgId, msgId: blob.msgId,
}); });
return true; return true;
} }
private async ackQuietly(msgId: string): Promise<void> {
try {
await this.client.ack({ address: this.options.ownAddress, msgId });
} catch (err) {
// Dispatch (or skip) succeeded; the ack just failed. Next poll
// will see the blob again and the dedup gate above will skip it.
console.warn('[Shade] Inbox ack failed (will retry):', (err as Error).message);
}
}
private recordDelivered(msgId: string): void {
if (this.deliveredIds.has(msgId)) return;
this.deliveredIds.add(msgId);
this.deliveredOrder.push(msgId);
if (this.deliveredOrder.length > DEFAULT_DEDUP_LRU_CAP) {
const evicted = this.deliveredOrder.shift()!;
this.deliveredIds.delete(evicted);
}
}
} }

View File

@@ -43,3 +43,11 @@ export type {
} from './events.js'; } from './events.js';
export { computeMsgId } from './msg-id.js'; export { computeMsgId } from './msg-id.js';
// V4.9 — encrypted-blob primitive (`/v1/blob/<slotId>`).
export { BlobClient, slotIdToHex } from './blob-client.js';
export type {
BlobClientOptions,
BlobGetResult,
BlobPutResult,
} from './blob-client.js';

View File

@@ -170,6 +170,255 @@ describe('Inbox orchestrator', () => {
expect(seen[1]!.to).toBe('carol'); expect(seen[1]!.to).toBe('carol');
}); });
test('cross-channel dedup: acceptBridgeFrame + pollOnce never re-dispatch the same msgId (V4.8.3)', async () => {
// Reproduces the Prism FR `cross-channel-duplicate-fanout-v4.8.2`:
// a single relay PUT was being delivered twice — once via WS bridge
// push, again ~30 s later via inbox-poll catching up. Both copies
// would dispatch `shade.receive`, the second one tripping on
// already-consumed prekeys. The cross-channel msgId LRU inside
// Inbox is the dedup gate; this test exercises it directly via
// `acceptBridgeFrame` followed by `pollOnce`.
const store = new MemoryInboxStore();
const app = createInboxServer({ crypto, store, disableRateLimit: true });
const bob = await makeIdentity();
const alice = await makeIdentity();
const bobInbox = new Inbox({
baseUrl: 'http://localhost',
ownAddress: 'bob',
crypto,
signingPrivateKey: bob.signingPrivateKey,
signingPublicKey: bob.signingPublicKey,
pollIntervalMs: 0,
fetch: honoFetch(app),
});
await bobInbox.register();
// Alice PUTs a blob via the relay HTTP API.
const ct = randBytes(64);
const msgId = await computeMsgId(ct);
const aliceClient = new InboxClient({
baseUrl: 'http://localhost',
crypto,
signingPrivateKey: alice.signingPrivateKey,
fetch: honoFetch(app),
});
const putResult = await aliceClient.put({
recipientAddress: 'bob',
senderSigningKey: alice.signingPublicKey,
envelope: ct,
});
expect(putResult.idempotent).toBe(false);
const dispatched: string[] = [];
bobInbox.onIncoming(async (raw) => {
dispatched.push(raw.msgId);
return null;
});
// Simulate the bridge push arriving first.
await bobInbox.acceptBridgeFrame({
msgId,
ciphertext: ct,
receivedAt: putResult.receivedAt,
expiresAt: Date.now() + 60_000,
});
expect(dispatched).toEqual([msgId]);
// The inbox-poll path catches up next — without dedup it would
// re-dispatch. With the LRU it acks-and-skips.
const polled = await bobInbox.tick();
expect(polled.received).toBe(0);
expect(dispatched).toEqual([msgId]); // still one entry
});
test('cross-channel dedup also covers poll-first then bridge-second order', async () => {
const store = new MemoryInboxStore();
const app = createInboxServer({ crypto, store, disableRateLimit: true });
const bob = await makeIdentity();
const alice = await makeIdentity();
const bobInbox = new Inbox({
baseUrl: 'http://localhost',
ownAddress: 'bob',
crypto,
signingPrivateKey: bob.signingPrivateKey,
signingPublicKey: bob.signingPublicKey,
pollIntervalMs: 0,
fetch: honoFetch(app),
});
await bobInbox.register();
const ct = randBytes(48);
const msgId = await computeMsgId(ct);
const aliceClient = new InboxClient({
baseUrl: 'http://localhost',
crypto,
signingPrivateKey: alice.signingPrivateKey,
fetch: honoFetch(app),
});
const putRes = await aliceClient.put({
recipientAddress: 'bob',
senderSigningKey: alice.signingPublicKey,
envelope: ct,
});
const dispatched: string[] = [];
bobInbox.onIncoming(async (raw) => {
dispatched.push(raw.msgId);
return null;
});
// Poll first.
const polled = await bobInbox.tick();
expect(polled.received).toBe(1);
// Bridge frame for the same msgId arrives after the poll already
// dispatched + ack'd it — must be a no-op.
const handled = await bobInbox.acceptBridgeFrame({
msgId,
ciphertext: ct,
receivedAt: putRes.receivedAt,
expiresAt: Date.now() + 60_000,
});
expect(handled).toBe(false);
expect(dispatched).toEqual([msgId]);
});
test('burst enqueued during a flush drains immediately, not after 15 s backoff (V4.8.5)', async () => {
// Reproduces Prism FR `per-recipient-flush-concurrency-v4.8`: a
// burst of envelopes enqueued *during* a slow POST used to sit
// ~15 s behind the next flush because both the success path and
// the failure path of `flushOnce` rescheduled with the same 15 s
// backoff. The fix uses 0 ms when the round delivered something
// (network is healthy — drain remainder) and reserves 15 s for
// the all-attempts-failed case.
const store = new MemoryInboxStore();
const app = createInboxServer({ crypto, store, disableRateLimit: true });
const bob = await makeIdentity();
const alice = await makeIdentity();
const bobClient = new InboxClient({
baseUrl: 'http://localhost',
crypto,
signingPrivateKey: bob.signingPrivateKey,
fetch: honoFetch(app),
});
await bobClient.register({ address: 'bob', signingKey: bob.signingPublicKey });
// Wrap fetch so the FIRST PUT (only) takes 100 ms — long enough
// for many enqueues to land while it's in flight.
let firstPutSeen = false;
const slowFirstFetch: typeof fetch = (async (input, init) => {
const u =
typeof input === 'string'
? input
: input instanceof URL
? input.toString()
: (input as Request).url;
const isPut = u.includes('/v1/inbox/bob') && !u.includes('/fetch');
if (isPut && !firstPutSeen) {
firstPutSeen = true;
await new Promise((r) => setTimeout(r, 100));
}
return honoFetch(app)(input, init);
}) as typeof fetch;
const aliceInbox = new Inbox({
baseUrl: 'http://localhost',
ownAddress: 'alice',
crypto,
signingPrivateKey: alice.signingPrivateKey,
signingPublicKey: alice.signingPublicKey,
pollIntervalMs: 0,
fetch: slowFirstFetch,
});
aliceInbox.start();
// First send — this kicks the slow-PUT path.
await aliceInbox.send({ recipientAddress: 'bob', envelope: randBytes(20) });
// Pile 24 more on top while the first PUT is still in flight. The
// first PUT will finish at ~T+100 ms; the subsequent 24 should
// drain immediately after, NOT after a 15 s backoff.
for (let i = 0; i < 24; i++) {
await aliceInbox.send({ recipientAddress: 'bob', envelope: randBytes(20) });
}
// Wait long enough for the slow first PUT + the immediate
// reschedule + the 24-envelope drain. Pre-fix this would still
// have ≥1 entry pending after 1 s (waiting for the 15 s timer).
await new Promise((r) => setTimeout(r, 1_000));
expect(await aliceInbox.pendingCount()).toBe(0);
aliceInbox.stop();
});
test('per-recipient parallel drain — slow POST to A does not block POSTs to B (V4.8.5)', async () => {
const store = new MemoryInboxStore();
const app = createInboxServer({ crypto, store, disableRateLimit: true });
const alice = await makeIdentity();
const bob = await makeIdentity();
const carol = await makeIdentity();
// Register bob + carol.
const reg = async (name: string, kp: { signingPrivateKey: Uint8Array; signingPublicKey: Uint8Array }) => {
const c = new InboxClient({
baseUrl: 'http://localhost',
crypto,
signingPrivateKey: kp.signingPrivateKey,
fetch: honoFetch(app),
});
await c.register({ address: name, signingKey: kp.signingPublicKey });
};
await reg('bob', bob);
await reg('carol', carol);
// bob's PUT route stalls 200 ms; carol's is instant. Pre-fix this
// would head-of-line block carol behind bob.
const slowedFetch: typeof fetch = (async (input, init) => {
const u =
typeof input === 'string'
? input
: input instanceof URL
? input.toString()
: (input as Request).url;
const m = (init as RequestInit | undefined)?.method ?? 'GET';
if (m === 'POST' && u.includes('/v1/inbox/bob') && !u.includes('/fetch')) {
await new Promise((r) => setTimeout(r, 200));
}
return honoFetch(app)(input, init);
}) as typeof fetch;
const aliceInbox = new Inbox({
baseUrl: 'http://localhost',
ownAddress: 'alice',
crypto,
signingPrivateKey: alice.signingPrivateKey,
signingPublicKey: alice.signingPublicKey,
pollIntervalMs: 0,
fetch: slowedFetch,
});
const carolDeliveredAt = new Promise<number>((resolve) => {
aliceInbox.on((e) => {
if (e.name === 'inbox.message_delivered' && e.data.recipientAddress === 'carol') {
resolve(Date.now());
}
});
});
const t0 = Date.now();
// Bob queue first, carol second — pre-fix carol would wait 200 ms
// behind bob's slow PUT. With per-recipient parallelism, carol's
// PUT runs concurrently and lands first.
await aliceInbox.send({ recipientAddress: 'bob', envelope: randBytes(20) });
await aliceInbox.send({ recipientAddress: 'carol', envelope: randBytes(20) });
aliceInbox.start();
const carolAt = await carolDeliveredAt;
const carolElapsed = carolAt - t0;
expect(carolElapsed).toBeLessThan(150);
aliceInbox.stop();
});
test('flush retries on transient server failure', async () => { test('flush retries on transient server failure', async () => {
const store = new MemoryInboxStore(); const store = new MemoryInboxStore();
const app = createInboxServer({ crypto, store, disableRateLimit: true }); const app = createInboxServer({ crypto, store, disableRateLimit: true });
@@ -281,3 +530,195 @@ describe('tamper detection', () => {
expect(result.received).toBe(0); expect(result.received).toBe(0);
}); });
}); });
describe('InboxClient — default fetch is bound to globalThis', () => {
// Regression: browsers' `fetch` is a WebIDL bound operation that throws
// "Illegal invocation" when called as a method on another object. The
// class stores `fetchImpl` and calls `this.fetchImpl(...)`, which strips
// the Window receiver. Constructor must `bind(globalThis)`.
test('default path passes globalThis as `this` (no Illegal invocation)', async () => {
const realFetch = globalThis.fetch;
let observedReceiver: unknown = 'unset';
function strictFetch(this: unknown, _input: unknown, _init?: unknown): Promise<Response> {
observedReceiver = this;
if (this !== globalThis) {
throw new TypeError("Failed to execute 'fetch' on 'Window': Illegal invocation");
}
return Promise.resolve(
new Response('{}', {
status: 200,
headers: { 'content-type': 'application/json' },
}),
);
}
Object.defineProperty(globalThis, 'fetch', {
configurable: true,
writable: true,
value: strictFetch,
});
try {
const id = await makeIdentity();
const client = new InboxClient({
baseUrl: 'http://example.invalid',
crypto,
signingPrivateKey: id.signingPrivateKey,
// No `fetch` override on purpose — this exercises the default path.
});
await client.register({ address: 'whoever', signingKey: id.signingPublicKey });
expect(observedReceiver).toBe(globalThis);
} finally {
Object.defineProperty(globalThis, 'fetch', {
configurable: true,
writable: true,
value: realFetch,
});
}
});
});
describe('Inbox.start() — fresh-address register/poll race (V4.8)', () => {
// Regression: pre-4.8 `start()` called `register()` fire-and-forget AND
// `schedulePoll(0)` synchronously, so the first poll often beat the
// register HTTP RTT and got SHADE_NOT_FOUND on a fresh address. Fix:
// start() defers the first poll; register() success kicks it.
test('fresh address: no fetch fires before register completes', async () => {
const store = new MemoryInboxStore();
const app = createInboxServer({ crypto, store, disableRateLimit: true });
const alice = await makeIdentity();
// Order observed by the server: must be register-then-fetch, never
// fetch-then-register.
const calls: Array<'register' | 'fetch' | 'put'> = [];
let registerArrived = false;
const recordingFetch: typeof fetch = (async (input, init) => {
const u =
typeof input === 'string'
? input
: input instanceof URL
? input.toString()
: (input as Request).url;
if (u.includes('/v1/inbox/register')) {
calls.push('register');
// Hold register for a tick to widen the race window.
await new Promise((r) => setTimeout(r, 25));
registerArrived = true;
} else if (u.endsWith('/fetch')) {
// Any fetch arriving before register is the race we're guarding
// against.
if (!registerArrived) {
throw new Error('fetch fired before register completed (race not fixed)');
}
calls.push('fetch');
} else if (u.includes('/v1/inbox/')) {
calls.push('put');
}
return honoFetch(app)(input, init);
}) as typeof fetch;
const inbox = new Inbox({
baseUrl: 'http://localhost',
ownAddress: 'alice',
crypto,
signingPrivateKey: alice.signingPrivateKey,
signingPublicKey: alice.signingPublicKey,
pollIntervalMs: 30_000, // Long enough that only register's kick triggers.
fetch: recordingFetch,
});
inbox.onIncoming(() => null);
inbox.start();
// Wait until register has completed and the success-kick poll lands.
await new Promise((r) => setTimeout(r, 100));
inbox.stop();
expect(calls[0]).toBe('register');
// First fetch (if any) must be after register.
const firstFetchIdx = calls.indexOf('fetch');
if (firstFetchIdx !== -1) {
expect(firstFetchIdx).toBeGreaterThan(calls.indexOf('register'));
}
});
});
describe('FetchedBlob.from — relay-supplied sender fingerprint (V4.8)', () => {
test('inbox-fetch response carries from = 8-byte hex of SHA-256(senderSigningKey)', async () => {
const store = new MemoryInboxStore();
const app = createInboxServer({ crypto, store, disableRateLimit: true });
const bob = await makeIdentity();
const alice = await makeIdentity();
const bobClient = new InboxClient({
baseUrl: 'http://localhost',
crypto,
signingPrivateKey: bob.signingPrivateKey,
fetch: honoFetch(app),
});
const aliceClient = new InboxClient({
baseUrl: 'http://localhost',
crypto,
signingPrivateKey: alice.signingPrivateKey,
fetch: honoFetch(app),
});
await bobClient.register({ address: 'bob', signingKey: bob.signingPublicKey });
await aliceClient.put({
recipientAddress: 'bob',
senderSigningKey: alice.signingPublicKey,
envelope: randBytes(64),
});
const fetched = await bobClient.fetch({ address: 'bob' });
expect(fetched.blobs.length).toBe(1);
const fp = fetched.blobs[0]!.from;
expect(fp).toBeDefined();
expect(fp).toMatch(/^[0-9a-f]{16}$/);
// Must be reproducible: SHA-256(alice.signingPublicKey) → first 8 bytes hex.
const digest = await globalThis.crypto.subtle.digest(
'SHA-256',
alice.signingPublicKey as unknown as ArrayBuffer,
);
const expected = Array.from(new Uint8Array(digest).slice(0, 8), (b) =>
b.toString(16).padStart(2, '0'),
).join('');
expect(fp).toBe(expected);
});
test('DecryptHandler raw arg propagates from to the app', async () => {
const store = new MemoryInboxStore();
const app = createInboxServer({ crypto, store, disableRateLimit: true });
const bob = await makeIdentity();
const alice = await makeIdentity();
const aliceClient = new InboxClient({
baseUrl: 'http://localhost',
crypto,
signingPrivateKey: alice.signingPrivateKey,
fetch: honoFetch(app),
});
const bobInbox = new Inbox({
baseUrl: 'http://localhost',
ownAddress: 'bob',
crypto,
signingPrivateKey: bob.signingPrivateKey,
signingPublicKey: bob.signingPublicKey,
pollIntervalMs: 0,
fetch: honoFetch(app),
});
await bobInbox.register();
await aliceClient.put({
recipientAddress: 'bob',
senderSigningKey: alice.signingPublicKey,
envelope: randBytes(40),
});
let observed: string | undefined = undefined;
bobInbox.onIncoming((raw) => {
observed = raw.from;
return null;
});
await bobInbox.tick();
expect(observed).toMatch(/^[0-9a-f]{16}$/);
});
});

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/key-transparency", "name": "@shade/key-transparency",
"version": "4.0.0", "version": "4.11.1",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -21,7 +21,7 @@
* the dataset grows enough that flat re-hash becomes a bottleneck. * the dataset grows enough that flat re-hash becomes a bottleneck.
*/ */
import { leafHash, nodeHash, emptyRootHash } from './hashes.js'; import { leafHash, emptyRootHash } from './hashes.js';
import { sha256Sync } from './sha256.js'; import { sha256Sync } from './sha256.js';
import { constantTimeEqual } from './util.js'; import { constantTimeEqual } from './util.js';
import { mth, auditPath, recomputeRootFromAuditPath } from './log.js'; import { mth, auditPath, recomputeRootFromAuditPath } from './log.js';

View File

@@ -24,8 +24,6 @@ import { MerkleLog, auditPath } from './log.js';
import { import {
AddressIndex, AddressIndex,
type AddressIndexEntry, type AddressIndexEntry,
type IndexAbsenceProof,
type IndexInclusionProof,
} from './index-tree.js'; } from './index-tree.js';
import { import {
type SignedTreeHead, type SignedTreeHead,

View File

@@ -111,7 +111,7 @@ interface IndexAbsenceWire {
} | null; } | null;
} }
type IndexProofWire = IndexInclusionWire | IndexAbsenceWire; export type IndexProofWire = IndexInclusionWire | IndexAbsenceWire;
interface BundleInclusionWire { interface BundleInclusionWire {
kind: 'inclusion' | 'tombstone'; kind: 'inclusion' | 'tombstone';

View File

@@ -0,0 +1,5 @@
{
"extends": "../../tsconfig.json",
"compilerOptions": { "outDir": "dist", "rootDir": "src" },
"include": ["src"]
}

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/keychain", "name": "@shade/keychain",
"version": "4.0.0", "version": "4.11.1",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,4 +1,5 @@
{ {
"extends": "../../tsconfig.json", "extends": "../../tsconfig.json",
"include": ["src/**/*", "tests/**/*"] "compilerOptions": { "outDir": "dist", "rootDir": "src" },
"include": ["src"]
} }

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/observability", "name": "@shade/observability",
"version": "4.0.0", "version": "4.11.1",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

Some files were not shown because too many files have changed in this diff Show More