13 Commits

Author SHA1 Message Date
8c606ad498 release(v4.8.2): per-from receive serialization + per-connection bridge dedup
Some checks failed
Test / test (push) Has been cancelled
Two interlocking robustness fixes for the duplicate-fan-out / first-contact
class of failures Prism reported.

1. `Shade.receive(from, env)` now queues its `manager.decrypt` step
   per `from` so concurrent dispatches can't race the SessionManager
   ratchet or the StorageProvider (sqlite "database is locked", IDB
   transaction conflicts). User message handlers run *outside* the
   queue so streams + file-RPC's nested `shade.receive` calls don't
   self-deadlock.

2. Bridge WS + SSE handlers now run a per-connection bounded msgId
   LRU as defense-in-depth against any flushTo re-entry (event-storm,
   future refactor). Pending-flush chains are wrapped in `.catch(() =>
   {})` so a transient `ws.send` rejection no longer poisons the
   connection's flush loop.

Tests: storming `inbox.blob_stored` 10× per PUT yields exactly one WS/
SSE frame; 8 concurrent `bob.receive('alice', envelope)` calls keep
the ratchet intact and never surface "database is locked".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 12:13:46 +02:00
680d6386f3 release(v4.8.1): SHADE_DISABLE_RATE_LIMIT env var for single-tenant deploys
Plumbing fix only — both createPrekeyRoutes and createInboxRoutes
already accepted disableRateLimit; standalone.ts just didn't read
the env. Now SHADE_DISABLE_RATE_LIMIT=1 turns off IP rate-limits on
every prekey + inbox route, with a WARN log on startup so operators
see it.

Single-tenant deployments only — multi-tenant relays must leave it
unset. Documented in docs/DEPLOYMENT.md.

Reported by Prism: ~6 pair attempts/hour from a single dev IP +
the sidecar's register call tripped the 5/hour REGISTER_LIMIT every
dev iteration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 00:55:57 +02:00
1fb59a7076 release(v4.8.0): sender-fingerprint attribution + Inbox.start race fix
Two unblocking changes for first-contact flows.

Sender attribution: relay captures shortHash(senderSigningKey) at
PUT time (after signature verification, no new trust surface) and
surfaces it on bridge push (IncomingMessage.from) + inbox-fetch
(FetchedBlob.from) + DecryptHandler raw arg. Apps receiving a prekey
envelope from a never-before-seen peer can now bootstrap X3DH via
shade.receive('fp:<hex>', env) — pre-4.8 the wire envelope didn't
authenticate the sender and there was no out-of-band hint to use.
Idempotent ALTER TABLE migrations for SQLite + Postgres add a
sender_fp TEXT column; legacy rows surface as from=undefined
(inter-version compat).

Inbox.start() race: pre-4.8 start() called register() fire-and-forget
AND schedulePoll(0) synchronously, so the first poll on a fresh
address often beat the register HTTP RTT and got SHADE_NOT_FOUND.
start() now defers; register() success kicks schedulePoll(0). Manual
tick() is unaffected (deliberate user action, no gating).

Both reported by Prism. Tests cover all five acceptance criteria
from the sender-attribution request (PUT capture, bridge surface,
fetch surface, inter-version compat, end-to-end pair smoke) plus
the three from the race-fix request.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 00:11:59 +02:00
594992a183 release(v4.7.0): peer-presence events for instant BroadcastChannel revoke
Adds the bridge-connection-lifecycle signal that closes Prism's
~45s revoke window down to one server→client round-trip (~50ms).

Server (`@shade/inbox-server`):
- `inbox.peer_connected` / `inbox.peer_disconnected` events on the
  0↔1 boundary across WS + SSE bridges. Long-poll deliberately not
  tracked (every poll boundary would flap; push transports are also
  the only ones where instant revoke matters).
- `PresenceTracker` collapses two parallel bridges (e.g. WS + SSE
  during fallback handover) into one connect/disconnect pair.
- `GET /v1/bridge/presence` SSE endpoint: signed query with
  `kind: 'presence'`, `watched: string[]`; on open streams a
  per-address snapshot, then change frames filtered server-side.
  MAX_WATCHED_ADDRESSES = 64. Subscribing does not itself count as
  a peer-bridge connection.
- `createBridgeRoutes` now returns `{ app, websocket, presence }`.

Client (`@shade/transport-bridge`):
- `PresenceBridge.subscribe({ watch, onPresenceChange })` →
  `{ addPeer, removePeer, watching, unsubscribe }`. addPeer/removePeer
  mutate via reconnect with a fresh signed query.
- `signPresenceQuery` helper for non-PresenceBridge consumers.

Tests cover all four acceptance criteria from the Prism request:
server-event smoke, online→offline subscription, address scoping
(carol invisible to a [alice]-only sub), reconnect, plus an
addPeer/removePeer regression.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 23:16:35 +02:00
8746571d2a release(v4.6.1): bind globalThis.fetch in browser-receiver-sensitive call sites
Browsers' Window.fetch is a WebIDL bound operation; storing it as
this.fetchImpl / this.fetchFn and calling via the instance receiver
threw "Illegal invocation" on the first request. Bind once at
construction in InboxClient, LongPollBridge, and SseBridge. Reported
by Prism (multi-device E2EE terminal), blocking every browser
consumer of the v4.6 transport stack on inbox.start() / bridge.connect().

WsBridge unaffected (uses WebSocket). Node/Bun fetch tolerates a free
receiver, so the bug never surfaced server-side — added regression
tests that install a strict-receiver globalThis.fetch to catch the
issue without an actual browser harness.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 23:00:58 +02:00
2c400d7094 release(v4.6.0): broadcast channels — Signal sender-keys for one-to-many fan-out
Some checks failed
Test / test (push) Has been cancelled
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
Lands the broadcast-channel primitive Prism asked for in
Docs/shade-feature-request-sender-keys.md. The crypto in
@shade/core/sender-keys.ts was already in place; this release wires
it up as a first-class app-facing API, adds the persistence schema
across all six storage backends (memory, sqlite, indexeddb +
encrypted variants), introduces wire type 0x21 in @shade/proto,
and ships Prism's three acceptance tests verbatim.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 15:55:34 +02:00
2b1b4d6630 release(v4.5.0): browser-side encrypted storage + multi-factor unlock
Adds the foundations Prism's web client (and any future browser-based
Shade app) needs: at-rest-encrypted IndexedDB storage that mirrors the
SQLite backend byte-for-byte at the AAD/nonce level, browser-safe
subpath imports so Vite/webpack/esbuild stop hitting bun:sqlite, and
KeyManager support for argon2id and N-factor composite unlock.

@shade/storage-encrypted
- EncryptedIndexedDBStorage (subpath: /idb) — full StorageProvider
  using one object store per _enc table; reuses aeadSeal/aeadOpen +
  row-codec sealers so a row sealed under the SQLite or Postgres
  backend decrypts under IDB given the same KeyManager.
  bumpPeerIdentityVersion is atomic under one IDB transaction.
- KeyManager argon2id source — memory-hard KDF for low-entropy
  secrets (PINs). Backed by @noble/hashes/argon2 (already a transitive
  dep). DEFAULT_ARGON2ID exported (m=64 MiB, t=3, p=1).
- KeyManager composite source — HKDF-combine N sub-sources into one
  master. Every source mandatory; order significant by design;
  composite-of-composite rejected; optional info string for app-level
  domain separation.
- Subpath exports (/crypto, /sqlite, /postgres, /idb) plus a `browser`
  condition on the default import that resolves to a barrel
  excluding the Bun- and Postgres-specific entries. Browser bundles
  no longer pull bun:sqlite transitively.

Tests
- 73 tests in shade-storage-encrypted (was 31). New coverage:
  argon2id determinism + reject paths, composite same-factors → same
  master, wrong-PIN/passphrase/order-swap → different master, info
  domain separation, all 28 StorageProvider methods on
  EncryptedIndexedDBStorage, fingerprint-mismatch rejection, and
  cross-impl roundtrip with EncryptedSQLiteStorage proving the AAD/
  nonce derivation is implementation-agnostic.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 10:58:49 +02:00
dbb3a090d8 release(v4.4.0): public accessor for device identity public key
Expose the local device's 32-byte Ed25519 identity public key on Shade
so apps can hand it to their own backend at enrollment time for
signature verification, key pinning or per-device safety-number
computation. Closes the gap that forced consumers to ship placeholder
random bytes their backend could store but never verify against.

- @shade/sdk Shade.identityPublicKey: Promise<Uint8Array> — getter
  mirrors the existing fingerprint accessor. Throws pre-init,
  reflects the current key after rotate(), retired key preserved in
  retired-identities storage per existing grace-period contract.
  Private key remains unreachable.
- Test in shade-sdk/tests/sdk.test.ts: round-trip match against the
  underlying storage's signingPublicKey, plus value updates after
  rotate().
- Lockstep version bump 4.3.0 → 4.4.0 across all 25 packages.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 17:58:45 +02:00
f5f42fe557 release(v4.3.0): browser persistence via @shade/storage-indexeddb
Ship an official IndexedDB-backed StorageProvider so browser-based Shade
consumers persist identity, prekeys, sessions, retired identities,
peer-verification state and stream-resume rows across tab refresh and
browser restart. Closes the gap that forced browser apps onto
storage:"memory" (regenerated identity each load, orphaned device
records server-side).

- New package @shade/storage-indexeddb (4.3.0): full StorageProvider
  conformance, schema v1, idb-backed; bumpPeerIdentityVersion is wrapped
  in a single readwrite IDB transaction (atomic, vs SQLite's
  read-then-upsert race).
- @shade/sdk resolveStorage() accepts { type: 'indexeddb', dbName? } via
  dynamic import (lazy, optional dep — same pattern as
  @shade/storage-postgres). Named StorageSpec type now reused by
  ResolvedConfig.
- Tests: 16 new tests in shade-storage-indexeddb (StorageProvider
  surface + peer-verifications + full E2EE conversation surviving a
  simulated tab reload). Run on fake-indexeddb.
- Lockstep version bump 4.2.1 → 4.3.0 across all 25 packages.
- Publish scripts updated to include the new package.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 17:35:02 +02:00
b77b7e771c release(v4.2.1): fix concurrent-ratchet desync via OutboundQueue waiter cursor
Some checks failed
Publish / publish (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Pull-mode httpClient + drainer + parallel RPCs against the same peer
deteriorated after ~10s with `DecryptionError`. Two bugs combined:

- `OutboundQueue.enqueue` woke `drain` waiters with a `since=0`
  snapshot, replaying already-processed events into
  `Shade.acceptTransferEnvelope` → `manager.decrypt` twice. The
  duplicate consumed an already-used skipped key and corrupted the
  Double Ratchet receive chain.

- `ratchetDecrypt` then propagated the corruption: a same-DH
  message behind the chain with no cached skipped key fell through
  to `kdfChainKey` on the ahead state and rewound `chain.counter`,
  permanently desyncing the chain.

Fix `OutboundQueue` to honor each waiter's `since`, and harden
`ratchetDecrypt` so any future duplicate fails cleanly without
mutating state. Adds regression coverage at all three layers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 22:58:26 +02:00
7520b11b25 release(v4.2.0): pull-mode streams for browser @shade/files
Some checks failed
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
4.1.0's HTTP RPC for browsers capped at inline payloads (≤ 256 KiB).
4.2.0 unlocks streams: server queues outbound chunks + control
envelopes per peer, browser long-polls the queue. Browser-to-server
writes ride the existing /v1/transfer/<id>/chunk POST routes
unchanged.

For Dispatch this unlocks mod-jar uploads (50 MB) and world-backup
downloads (100+ MB) — the actual reason browser-side @shade/files
matters.

### New API

@shade/sdk:
- shade.transferQueueRoute(opts?) — Hono app with /queue +
  /v1/transfer/* routes. Auto-configures the queue transport.
- shade.configureTransfers extended: transport + envelopeTransport
  override slots; resolveBaseUrl optional when both supplied.

@shade/transfer:
- OutboundQueue — per-peer monotonic event log with long-poll
  semantics, idle-eviction GC, ring-buffered to maxEventsPerPeer.
- QueueTransferTransport — enqueues instead of POSTing.

@shade/files:
- httpClient({ outboundQueueUrl, transferBaseUrl }) — when set,
  starts a long-poll drainer + builds a streams-bridge. fs.read /
  fs.write of >256 KiB work end-to-end.
- startQueueDrainer(shade, opts) — exported helper for advanced
  consumers driving their own drainer.

### Implementation notes

- ClientStreamsBridge's TransformStream had HWM=0 by default which
  stalled the drainer's await chain at chunk 4 (writer.write pended
  before the consumer's reader was attached). Bumped to HWM=64 so
  the receive loop can buffer ahead of the consumer.

### Tests

3 new integration tests in tests/integration/http-rpc-streams.test.ts:
4 MiB streamed read round-trip, inline-only error path, idle-timeout
long-poll behaviour.

Wire-compatible. Source-compatible. Lockstep bump to 4.2.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 23:27:06 +02:00
da93b97cce release(v4.1.0): browser-friendly HTTP RPC for @shade/files
Some checks failed
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
Default shade.files.client(peer) requires both peers to be mutually
addressable over HTTP — the response round-trips through
Shade.deliverControlEnvelope (POST to peer's /v1/transfer/control).
Browser tabs can't host an HTTP server, so they couldn't consume
@shade/files at all. Dispatch's filutforsker (admin-panel browser UI)
is the canonical use-case.

This release adds a parallel request-response transport: one POST per
RPC, encrypted envelope in the body, encrypted response in the same
HTTP response. No inbound channel needed on the client.

### New API

- shade.files.rpcRoute(opts?) — Hono app exposing POST /rpc.
- shade.files.httpClient(peer, opts) — request-response FileClient.
- FilesNamespace.serve(handler, { inlineOnly: true }) — skip streams-
  bridge (and its configureTransfers pre-condition); also skip
  channel-based dispatch so requests aren't double-dispatched.

### Limitations (v1)

Inline only (≤ 256 KiB). Streamed reads/writes throw clear errors
directing to shade.files.client(peer) on a server-to-server deploy.

### Tests

7 integration tests in tests/integration/http-rpc.test.ts covering
round-trip + negative cases (sender header, empty/garbage body,
maxBodyBytes, rpcRoute-without-serve).

### Symmetry

Mirrors @shade/server's shade-auth-middleware: encrypted envelope in
request body, decrypted via existing ratchet, response in same HTTP
roundtrip. No WebSocket, no SSE, no outbound from server.

Wire-compatible. Source-compatible. Lockstep bump to 4.1.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 22:08:14 +02:00
0bdf9e859c release(v4.0.2): consumer-strict reader-shape fixes
Some checks failed
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
4.0.1's typecheck gate compiled each package internally against
lib: ["ES2022"]. That doesn't catch types that only fail when
*consumer* code (lib: ["DOM"] + exactOptionalPropertyTypes) tries to
assign a native browser type into one of our locally-defined narrower
types. Dispatch hit one such case in @shade/files inline-threshold.ts.

This release adds a tests/consumer-strict/ smoke project to the
pre-publish gate. It compiles a tiny "as if I were a downstream app"
TS file against:

  lib: ["ES2022", "DOM", "DOM.Iterable"]
  types: ["bun-types"]
  exactOptionalPropertyTypes: true
  strict: true
  paths → packages/*/src/index.ts

scripts/typecheck-all.ts now runs the smoke after per-package checks.
Both must pass before publish:dry / publish:all proceeds.

### Fixed
- @shade/files inline-threshold.ts: MinimalReader<T> rewritten as the
  explicit disjoint union { done:false, value:T } | { done:true,
  value?: T | undefined } that's assignable from every native reader
  shape (bun, DOM, node:stream/web). Fixes the
  "ReadableStreamReadResult is not assignable" Dispatch reported.
- @shade/files streams-bridge (client + server): stash setTimeout
  return in a local before .unref?.() via { unref?: () => void } cast.
  Fluent .unref?.() failed under lib: ["DOM"] (setTimeout returns
  number there).
- @shade/sdk background.ts: same setInterval .unref?.() fix.

Wire-compatible. No API shape changed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 19:51:46 +02:00
112 changed files with 9881 additions and 186 deletions

3
.gitignore vendored
View File

@@ -2,3 +2,6 @@ node_modules/
dist/ dist/
*.tsbuildinfo *.tsbuildinfo
.DS_Store .DS_Store
**/.tmp-*.db
**/.tmp-*.db-shm
**/.tmp-*.db-wal

File diff suppressed because it is too large Load Diff

View File

@@ -17,7 +17,7 @@
}, },
"packages/shade-cli": { "packages/shade-cli": {
"name": "@shade/cli", "name": "@shade/cli",
"version": "0.4.0", "version": "4.4.0",
"bin": { "bin": {
"shade": "src/cli.ts", "shade": "src/cli.ts",
}, },
@@ -36,7 +36,7 @@
}, },
"packages/shade-core": { "packages/shade-core": {
"name": "@shade/core", "name": "@shade/core",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/observability": "workspace:*", "@shade/observability": "workspace:*",
}, },
@@ -49,7 +49,7 @@
}, },
"packages/shade-crypto-web": { "packages/shade-crypto-web": {
"name": "@shade/crypto-web", "name": "@shade/crypto-web",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@noble/curves": "^2.0.1", "@noble/curves": "^2.0.1",
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
@@ -59,7 +59,7 @@
}, },
"packages/shade-dashboard": { "packages/shade-dashboard": {
"name": "@shade/dashboard", "name": "@shade/dashboard",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/widgets": "workspace:*", "@shade/widgets": "workspace:*",
"react": "^19.0.0", "react": "^19.0.0",
@@ -74,7 +74,7 @@
}, },
"packages/shade-files": { "packages/shade-files": {
"name": "@shade/files", "name": "@shade/files",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -101,7 +101,7 @@
}, },
"packages/shade-inbox": { "packages/shade-inbox": {
"name": "@shade/inbox", "name": "@shade/inbox",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/proto": "workspace:*", "@shade/proto": "workspace:*",
@@ -114,7 +114,7 @@
}, },
"packages/shade-inbox-server": { "packages/shade-inbox-server": {
"name": "@shade/inbox-server", "name": "@shade/inbox-server",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/observability": "workspace:*", "@shade/observability": "workspace:*",
@@ -132,7 +132,7 @@
}, },
"packages/shade-key-transparency": { "packages/shade-key-transparency": {
"name": "@shade/key-transparency", "name": "@shade/key-transparency",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
@@ -144,11 +144,11 @@
}, },
"packages/shade-keychain": { "packages/shade-keychain": {
"name": "@shade/keychain", "name": "@shade/keychain",
"version": "0.4.0", "version": "4.4.0",
}, },
"packages/shade-observability": { "packages/shade-observability": {
"name": "@shade/observability", "name": "@shade/observability",
"version": "0.1.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
}, },
@@ -166,7 +166,7 @@
}, },
"packages/shade-observer": { "packages/shade-observer": {
"name": "@shade/observer", "name": "@shade/observer",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/server": "workspace:*", "@shade/server": "workspace:*",
@@ -178,14 +178,14 @@
}, },
"packages/shade-proto": { "packages/shade-proto": {
"name": "@shade/proto", "name": "@shade/proto",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
}, },
}, },
"packages/shade-recovery": { "packages/shade-recovery": {
"name": "@shade/recovery", "name": "@shade/recovery",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -198,7 +198,7 @@
}, },
"packages/shade-sdk": { "packages/shade-sdk": {
"name": "@shade/sdk", "name": "@shade/sdk",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -225,7 +225,7 @@
}, },
"packages/shade-server": { "packages/shade-server": {
"name": "@shade/server", "name": "@shade/server",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/inbox-server": "workspace:*", "@shade/inbox-server": "workspace:*",
@@ -245,15 +245,19 @@
}, },
"packages/shade-storage-encrypted": { "packages/shade-storage-encrypted": {
"name": "@shade/storage-encrypted", "name": "@shade/storage-encrypted",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
"@shade/storage-postgres": "workspace:*", "@shade/storage-postgres": "workspace:*",
"@shade/storage-sqlite": "workspace:*", "@shade/storage-sqlite": "workspace:*",
"idb": "^8.0.3",
"postgres": "^3.4.9", "postgres": "^3.4.9",
}, },
"devDependencies": {
"fake-indexeddb": "^6.0.0",
},
"peerDependencies": { "peerDependencies": {
"@shade/keychain": "workspace:*", "@shade/keychain": "workspace:*",
}, },
@@ -261,9 +265,21 @@
"@shade/keychain", "@shade/keychain",
], ],
}, },
"packages/shade-storage-indexeddb": {
"name": "@shade/storage-indexeddb",
"version": "4.4.0",
"dependencies": {
"@shade/core": "workspace:*",
"idb": "^8.0.3",
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
"fake-indexeddb": "^6.0.0",
},
},
"packages/shade-storage-postgres": { "packages/shade-storage-postgres": {
"name": "@shade/storage-postgres", "name": "@shade/storage-postgres",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/inbox-server": "workspace:*", "@shade/inbox-server": "workspace:*",
@@ -278,7 +294,7 @@
}, },
"packages/shade-storage-sqlite": { "packages/shade-storage-sqlite": {
"name": "@shade/storage-sqlite", "name": "@shade/storage-sqlite",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -288,7 +304,7 @@
}, },
"packages/shade-streams": { "packages/shade-streams": {
"name": "@shade/streams", "name": "@shade/streams",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
@@ -300,7 +316,7 @@
}, },
"packages/shade-transfer": { "packages/shade-transfer": {
"name": "@shade/transfer", "name": "@shade/transfer",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -317,7 +333,7 @@
}, },
"packages/shade-transport": { "packages/shade-transport": {
"name": "@shade/transport", "name": "@shade/transport",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -328,7 +344,7 @@
}, },
"packages/shade-transport-bridge": { "packages/shade-transport-bridge": {
"name": "@shade/transport-bridge", "name": "@shade/transport-bridge",
"version": "0.1.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/server": "workspace:*", "@shade/server": "workspace:*",
@@ -350,7 +366,7 @@
}, },
"packages/shade-transport-webrtc": { "packages/shade-transport-webrtc": {
"name": "@shade/transport-webrtc", "name": "@shade/transport-webrtc",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/streams": "workspace:*", "@shade/streams": "workspace:*",
@@ -359,7 +375,7 @@
}, },
"packages/shade-widgets": { "packages/shade-widgets": {
"name": "@shade/widgets", "name": "@shade/widgets",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/recovery": "workspace:*", "@shade/recovery": "workspace:*",
"@shade/sdk": "workspace:*", "@shade/sdk": "workspace:*",
@@ -568,6 +584,8 @@
"@shade/storage-encrypted": ["@shade/storage-encrypted@workspace:packages/shade-storage-encrypted"], "@shade/storage-encrypted": ["@shade/storage-encrypted@workspace:packages/shade-storage-encrypted"],
"@shade/storage-indexeddb": ["@shade/storage-indexeddb@workspace:packages/shade-storage-indexeddb"],
"@shade/storage-postgres": ["@shade/storage-postgres@workspace:packages/shade-storage-postgres"], "@shade/storage-postgres": ["@shade/storage-postgres@workspace:packages/shade-storage-postgres"],
"@shade/storage-sqlite": ["@shade/storage-sqlite@workspace:packages/shade-storage-sqlite"], "@shade/storage-sqlite": ["@shade/storage-sqlite@workspace:packages/shade-storage-sqlite"],
@@ -626,6 +644,8 @@
"escalade": ["escalade@3.2.0", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="], "escalade": ["escalade@3.2.0", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="],
"fake-indexeddb": ["fake-indexeddb@6.2.5", "", {}, "sha512-CGnyrvbhPlWYMngksqrSSUT1BAVP49dZocrHuK0SvtR0D5TMs5wP0o3j7jexDJW01KSadjBp1M/71o/KR3nD1w=="],
"fast-check": ["fast-check@3.23.2", "", { "dependencies": { "pure-rand": "^6.1.0" } }, "sha512-h5+1OzzfCC3Ef7VbtKdcv7zsstUQwUDlYpUTvjeUsJAssPgLn7QzbboPtL5ro04Mq0rPOsMzl7q5hIbRs2wD1A=="], "fast-check": ["fast-check@3.23.2", "", { "dependencies": { "pure-rand": "^6.1.0" } }, "sha512-h5+1OzzfCC3Ef7VbtKdcv7zsstUQwUDlYpUTvjeUsJAssPgLn7QzbboPtL5ro04Mq0rPOsMzl7q5hIbRs2wD1A=="],
"fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="], "fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="],
@@ -638,6 +658,8 @@
"hono": ["hono@4.12.12", "", {}, "sha512-p1JfQMKaceuCbpJKAPKVqyqviZdS0eUxH9v82oWo1kb9xjQ5wA6iP3FNVAPDFlz5/p7d45lO+BpSk1tuSZMF4Q=="], "hono": ["hono@4.12.12", "", {}, "sha512-p1JfQMKaceuCbpJKAPKVqyqviZdS0eUxH9v82oWo1kb9xjQ5wA6iP3FNVAPDFlz5/p7d45lO+BpSk1tuSZMF4Q=="],
"idb": ["idb@8.0.3", "", {}, "sha512-LtwtVyVYO5BqRvcsKuB2iUMnHwPVByPCXFXOpuU96IZPPoPN6xjOGxZQ74pgSVVLQWtUOYgyeL4GE98BY5D3wg=="],
"js-tokens": ["js-tokens@4.0.0", "", {}, "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ=="], "js-tokens": ["js-tokens@4.0.0", "", {}, "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ=="],
"jsesc": ["jsesc@3.1.0", "", { "bin": { "jsesc": "bin/jsesc" } }, "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA=="], "jsesc": ["jsesc@3.1.0", "", { "bin": { "jsesc": "bin/jsesc" } }, "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA=="],

View File

@@ -81,6 +81,7 @@ Tables will be created automatically with the `shade_server_*` prefix, so they c
| `SHADE_CLEANUP_INTERVAL_HOURS` | `24` | Cleanup cycle interval | | `SHADE_CLEANUP_INTERVAL_HOURS` | `24` | Cleanup cycle interval |
| `SHADE_LOG_LEVEL` | `info` | `debug` / `info` / `warn` / `error` | | `SHADE_LOG_LEVEL` | `info` | `debug` / `info` / `warn` / `error` |
| `SHADE_OTEL_ENABLED` | unset | Set to `1`/`true` to enable OpenTelemetry tracing on `withTracer()`-configured deployments. See [`observability.md`](./observability.md). | | `SHADE_OTEL_ENABLED` | unset | Set to `1`/`true` to enable OpenTelemetry tracing on `withTracer()`-configured deployments. See [`observability.md`](./observability.md). |
| `SHADE_DISABLE_RATE_LIMIT` | unset | Set to `1` to disable IP rate-limits on every prekey + inbox route. **Single-tenant deployments only** — multi-tenant relays must leave this unset to keep the abuse defenses on. |
## Health and observability ## Health and observability

View File

@@ -53,6 +53,111 @@ if (result.kind === 'inline') console.log(result.bytes.byteLength);
else for await (const _chunk of /* result.stream */) { /* ... */ } else for await (const _chunk of /* result.stream */) { /* ... */ }
``` ```
## HTTP RPC — browser-friendly request-response (4.1+)
The default `shade.files.client(peer)` requires both peers to be mutually
addressable over HTTP — the response to a `list`/`read` etc. round-trips
through `Shade.deliverControlEnvelope`, which POSTs to the peer's
`/v1/transfer/control` endpoint. **That doesn't work for browsers**
a tab can't host an HTTP server, so the server cannot call back outbound.
`@shade/files` 4.1 ships a parallel **request-response** transport that
lets browser-style clients fully consume the file-RPC surface without
any inbound channel. It mirrors the way `@shade/server`'s
`shade-auth-middleware` works: one POST per RPC, encrypted envelope in
the request body, encrypted response in the same HTTP response.
### Server side — mount the RPC route
```ts
// 1. Register the file handler. `inlineOnly: true` skips the
// streams-bridge (which would require @shade/transfer).
await shade.files.serve(handlerConfig, { inlineOnly: true });
// 2. Mount the route on your Hono app under any base path.
app.route('/api/v1/shade-files', shade.files.rpcRoute());
// ^^^^^^^^^^^^^^
// POST <base>/rpc
```
`rpcRoute()` accepts:
| Option | Default | Purpose |
|---------------------|---------|----------------------------------------------------------------------------------------------------|
| `maxBodyBytes` | 1 MiB | Max request body. The protocol caps inline payloads at 256 KiB; the headroom is for base64 inflation + custom-op envelopes. |
| `acceptFirstMessage`| `false` | Accept `0x01` PreKeyMessage envelopes — required when the RPC route also doubles as the X3DH handshake (browser's first-ever request). |
### Browser client
```ts
import { createShade } from '@shade/sdk';
const shade = await createShade({
prekeyServer: 'https://shade.example.com',
storage: 'memory',
address: 'alice@example.com',
});
const fs = shade.files.httpClient('bob@example.com', {
rpcUrl: 'https://dispatch.example.com/api/v1/shade-files/rpc',
// Optional: thread CSRF / auth tokens, override fetch, etc.
headers: { 'X-CSRF-Token': csrfToken },
});
await fs.mkdir('/photos');
await fs.write('/photos/cover.png', new Uint8Array([/* ... */]), {
contentType: 'image/png',
});
const result = await fs.read('/photos/cover.png');
```
### What works in HTTP-RPC mode
- `list`, `stat`, `mkdir`, `delete`, `move`, `getThumbnail`, `custom<K>` — full parity.
- `write`**inline only** (≤ 256 KiB plaintext). Larger inputs throw `ConflictError`.
- `read`**inline only**. If the server returns a streamed `read` result, the client throws `InternalFileError` directing callers to the stateful pathway.
### Wire contract
```
POST /rpc HTTP/1.1
Content-Type: application/octet-stream
X-Shade-Sender-Address: alice@example.com
<wire-encoded ShadeEnvelope (0x01 first-time, 0x02 after) wrapping
JSON-encoded RpcRequest>
────
200 OK
Content-Type: application/octet-stream
<wire-encoded ShadeEnvelope (0x02) wrapping JSON-encoded RpcResponse | RpcError>
```
Transport-level failures (no session, undecryptable envelope, body too
big) return JSON `{ "error": "..." }` with appropriate 4xx status.
Application-level failures (file not found, permission denied) ship
encrypted `RpcError` envelopes — the client maps them back to typed
`FileError` subclasses (`NotFoundError`, `ConflictError`, etc.).
### Symmetry with `@shade/server`
The shape mirrors `@shade/server`'s shade-auth-middleware: encrypted
envelope rides the request body, server decrypts via the existing
ratchet session, performs the protected operation, returns an encrypted
envelope in the response. No bidirectional channel required, no
WebSocket, no SSE.
### When to use which
| Setup | Use |
|-----------------------------------------------|-----------------------------------------------|
| Browser client ↔ Bun/Hono server | `httpClient()` + `rpcRoute()` |
| Server ↔ server (both can host HTTP) | `client()` (default) — supports streams |
| Service-worker / extension ↔ server | `httpClient()` (no inbound listener) |
| CLI / daemon ↔ daemon | Either; `client()` if you need streams |
## Op surface ## Op surface
| Op | Args | Result | | Op | Args | Result |

View File

@@ -0,0 +1,173 @@
# Feature Request — Public accessor for the device's identity public key
**To**: Shade SDK team
**From**: Dispatch (browser-based Shade consumer)
**Target**: Shade SDK 4.4.x (or whichever release vehicle fits)
**Priority**: medium — unblocks real per-device fingerprint binding at
enrollment time; consumers ship with placeholder keys until then
Thanks for shipping `@shade/storage-indexeddb` so quickly — that unblocked
Dispatch's Slice 2.5 (persistent browser identity). During integration we
hit one more gap that's worth raising as a separate FR.
---
## Summary
Expose the local device's Ed25519 identity public key as a public accessor
on `Shade`, so applications can hand it to their own backend at enrollment
time for per-device verification, audit, or peer-fingerprint computation.
Today the SDK exposes `myAddress` and `fingerprint`, but the underlying
identity public key — the cryptographic root that everything else binds
to — is reachable only via the private `this.storage.getIdentityKeyPair()`
call inside `Shade`. Consumers building enrollment flows have no way to
hand the real key over.
## Problem
A common pattern in Shade-using apps is:
1. Browser device generates Shade identity (via `createShade`)
2. User enters an enrollment token
3. Browser POSTs to its backend: `{ token, address, identityPublicKey, ... }`
4. Backend records the device, computes/stores a safety number from the
identity key, opens a Shade session
Step 3 today has nowhere to get a real `identityPublicKey` from. Dispatch
currently sends `crypto.getRandomValues(new Uint8Array(32))` formatted as
hex, with this comment in the source:
```ts
/**
* The browser submits a hex public-key field at enroll time so the schema
* stays stable. Wiring this to the SDK-generated identity key requires a
* Shade SDK addition (no public accessor for the raw identity key today).
*/
export function generatePlaceholderIdentityPublicKey(): string { ... }
```
The backend stores this placeholder but cannot verify anything against it,
because it's not actually the device's key. Real cryptographic binding is
deferred until the SDK exposes the underlying key.
## Why `fingerprint` isn't sufficient
`Shade.fingerprint` returns a 12-groups-of-5-digits safety-number string
designed for human side-channel comparison. That's the right output for a
"compare these digits with your friend" UX, but it's a derived format, not
the raw key. Backends that want to:
- Store the key for later signature verification
- Compute their own safety-number representation (Dispatch uses
`deterministicSafetyNumber(localAddr, peerAddr)` based on the raw bytes)
- Re-derive the fingerprint after an identity rotation
…all need access to the raw 32-byte Ed25519 public key.
## Proposed API
Add a single async accessor on `Shade`:
```ts
class Shade {
/**
* The local device's Ed25519 identity public key (32 bytes).
*
* Stable for the lifetime of the identity. After `rotateIdentity()`
* this returns the new key; the old key is preserved in retired-
* identities storage for the configured grace period.
*/
get identityPublicKey(): Promise<Uint8Array>;
}
```
Internally:
```ts
get identityPublicKey(): Promise<Uint8Array> {
if (!this.initialized) throw new Error('Not initialized');
return this.storage.getIdentityKeyPair().then((kp) => {
if (!kp) throw new Error('Identity not yet generated');
return kp.publicKey;
});
}
```
If returning a getter that produces a Promise feels off, the equivalent
method form is fine:
```ts
async getIdentityPublicKey(): Promise<Uint8Array> { ... }
```
Either shape works for consumers — pick whichever matches existing SDK
conventions.
## Alternative considered: object accessor
Returning an object would leave room to expose other identity-related
fields later without a breaking change:
```ts
get identity(): Promise<{
address: string;
publicKey: Uint8Array;
fingerprint: string;
// future: registrationId, createdAt, etc.
}>;
```
This would mildly duplicate `myAddress` + `fingerprint`, but consolidates
identity-related accessors. Not a blocker for shipping the simpler
single-purpose accessor — just flagging the option in case the SDK is
considering broader API ergonomics.
## What this unblocks
For Dispatch specifically:
- Real device identity binding at enrollment (replaces the placeholder
`crypto.randomUUID()`-derived hex bytes the backend currently stores)
- Server-side `computePeerSafetyNumber()` can use the real key instead of
the deterministic-from-address stand-in (`shade-identity-provider.ts:151`)
- Future signature-based device authentication (sign a challenge with the
device's identity key during enrollment) becomes possible without
another SDK round
For other Shade consumers:
- Any app that hands a device key to its own backend for enrollment —
multi-device pairing flows, contact verification UIs, push-notification
targeting — gets an actual key to work with
## Acceptance criteria
1. New accessor exposed on `Shade` (getter or async method, SDK's
preference)
2. Returns the 32-byte Ed25519 identity public key
3. Returns the **current** key after `rotateIdentity()`
4. Throws (or resolves to a clear error) when called before
`initialize()` completes
5. Documented alongside `myAddress` and `fingerprint` in the SDK reference
6. One unit test in `shade-sdk` confirming the returned bytes match what
`storage.getIdentityKeyPair()` holds
7. Mentioned in the changelog under the next release
## Out of scope
- **Private key access** — consumers should never need it; signature
operations go through `shade.send()` etc. Don't expose the secret half.
- **Cross-peer key lookup** — getting *another* peer's identity key is a
separate concern (related to peer-verification storage), not what this
FR is about. This is strictly the local device's own key.
- **Format conversions** — base64/hex/PEM helpers don't belong in the SDK.
Consumers can encode the raw bytes however their wire format requires.
## Why this can't be done in consumer-land
The identity keypair is generated by `MemoryStorage` / `SQLiteStorage` /
`IndexedDBStorage` and consumed by `ShadeSessionManager`. Consumers can't
reach into either layer without breaching the SDK's encapsulation. A
public accessor is the only path that doesn't require monkey-patching
private fields.

View File

@@ -0,0 +1,191 @@
# Feature Request — `@shade/storage-indexeddb`
**To**: Shade SDK team
**From**: Dispatch (browser-based Shade consumer)
**Target**: Shade SDK 4.3.x (or whichever release vehicle fits)
**Priority**: blocks all browser-based Shade apps from achieving session
persistence across tab refresh
---
## Summary
Ship an official IndexedDB-backed `StorageProvider` adapter as a new
workspace package `@shade/storage-indexeddb`, so browser-based Shade SDK
consumers can persist identity, prekeys, sessions, and peer-verification
state across tab refresh and browser restart — the same way `@shade/storage-sqlite`
does for Node and `@shade/storage-postgres` does for server deployments.
## Problem
Today the Shade SDK ships three storage paths:
| spec | adapter | environment |
| --------------------------------- | ------------------------ | ---------------- |
| `"memory"` | `MemoryStorage` (in-SDK) | tests, ephemeral |
| `"sqlite:/path"` | `@shade/storage-sqlite` | Node |
| `{ type: 'postgres', url: '…' }` | `@shade/storage-postgres`| Node servers |
There is **no browser-storage option**. The only way to run Shade in a
browser today is `storage: "memory"`, which means:
- Identity keypair regenerates on every page load
- Sessions reset → re-enrollment after every refresh
- `getLocalRegistrationId()` returns a fresh value → `device:${id}`
address changes → server-side device record orphaned every reload
This forces every browser-based Shade app to either (a) accept the broken
UX, or (b) build their own `StorageProvider` from scratch — duplicating
~25 methods × N consumers, with no shared conformance test surface.
The right place to solve this is at the SDK level, exactly mirroring how
SQLite and Postgres are handled.
## Proposed package
`packages/shade-storage-indexeddb/` — modeled directly after
`packages/shade-storage-sqlite/`. Same package shape, same test layout,
same `@shade/core`-only runtime dependency surface.
### Public API
```ts
// @shade/storage-indexeddb
export class IndexedDBStorage implements StorageProvider {
/**
* Open (or create) the IndexedDB database. Idempotent — repeated calls
* with the same dbName return a connection sharing the same object stores.
*/
static async create(opts?: { dbName?: string }): Promise<IndexedDBStorage>;
/**
* Cleanly close the underlying connection. Future calls will reopen.
* Called by Shade.shutdown() when consumers register cleanup.
*/
async close(): Promise<void>;
// ─── all StorageProvider methods (identity, prekeys, sessions,
// retired identities, peer verifications, optional stream-state) ───
}
```
`dbName` defaults to something like `"shade"`. Consumers like Dispatch
will pass distinct names per app (`"dispatch-dashboard-shade"`,
`"dispatch-host-ui-shade"`) so DevTools' IndexedDB inspector groups them
sensibly, even though origin-isolation already makes the data isolated.
### SDK integration
`@shade/sdk` `resolveStorage()` gets a fourth branch:
```ts
if (typeof spec === 'object' && spec.type === 'indexeddb') {
const moduleId = '@shade/storage-indexeddb';
const mod = (await import(moduleId)) as {
IndexedDBStorage: { create(opts: { dbName?: string }): Promise<StorageProvider> };
};
return mod.IndexedDBStorage.create({ dbName: spec.dbName });
}
```
Dynamic import keeps `@shade/storage-indexeddb` an optional dependency,
matching the Postgres pattern — Node-only consumers don't need to install
a browser-only adapter.
Consumer surface:
```ts
const shade = await createShade({
prekeyServer: 'https://…/shade-prekey',
storage: { type: 'indexeddb', dbName: 'my-app-shade' },
address: 'device:user@example.com', // optional — falls back to device:${registrationId}
});
```
## Implementation guidance (non-prescriptive)
- **IDB wrapper**: suggest `idb` (Jake Archibald's thin wrapper, well-typed,
zero deps). Avoid Dexie or idb-keyval — we want full schema control to
match the SQL adapters' explicit schemas.
- **Object-store layout**: one store per StorageProvider category
(`identity`, `signedPreKeys`, `oneTimePreKeys`, `sessions`,
`trustedIdentities`, `retiredIdentities`, `peerVerifications`,
`streamStates`). Keypaths match the natural keys (`keyId`, `address`,
`streamId`).
- **Schema version**: integer, bumped on every shape change. Migrations
in `db.upgrade(...)` callback. Document schema-history alongside
the SQLite schema.
- **Concurrency**: IndexedDB transactions are auto-committing — the adapter
must keep operations within a single transaction where SQL adapters do.
Particular care for `bumpPeerIdentityVersion` (atomic read-modify-write).
- **Stream-state methods**: implement them. Browser apps will increasingly
use `@shade/transfer` for large file resume, and parity with SQLite's
capabilities matters.
## Test expectations
Mirror `packages/shade-storage-sqlite/tests/`:
- `indexeddb-storage.test.ts` — full StorageProvider surface (identity,
sessions, trusted identities, retired identities)
- `indexeddb-prekey-store.test.ts` — signed + one-time prekey lifecycle
- `peer-verifications.test.ts` — verification CRUD + identity-version
bumping invariants
- `indexeddb-stream-state.test.ts` (if stream-state is implemented)
Use **`fake-indexeddb`** for the Node test environment — it's the
established standard, supports the v3 spec, and lets us run IDB tests
in `bun test` / `vitest` / `jest` without a real browser.
If/when Shade gains a shared `StorageProvider` conformance test suite,
this adapter should consume it directly. Until then, follow the SQLite
adapter's per-method coverage style.
## Acceptance criteria
1. `@shade/storage-indexeddb` published at version 4.3.0 (or whichever
matches the next Shade release)
2. `@shade/sdk` `resolveStorage()` resolves `{ type: 'indexeddb', dbName? }`
via dynamic import
3. Full StorageProvider conformance in tests (identity, prekeys,
sessions, retired identities, peer verifications, stream-state)
4. Documented in Shade docs alongside SQLite/Postgres adapters
5. README example showing browser-app integration
6. Bundle-size note: dynamic-imported IDB module shouldn't pull crypto
dependencies — adapter should be ≤ ~10 KB minified+gzipped
## Out of scope (deferred)
- **Encryption-at-rest** for the IDB contents — separate work item; should
match the deviceKey-AES-GCM pattern Shade already uses for `secretEnc`
in `PersistedStreamState`. This adapter ships unencrypted-at-rest in v1
(consistent with SQLite), with the encryption layer added uniformly to
all adapters later.
- **Cross-tab BroadcastChannel sync** — IDB is shared across same-origin
tabs already; concurrent writes work via IDB transactions. Real-time
notification across tabs (e.g. "session was rotated in another tab") is
a separate concern, not storage-adapter scope.
- **Quota handling** — IDB quota for a Shade keystore is far below realistic
browser quotas. If it ever becomes relevant, add a `QuotaExceededError`
observability hook then.
## Why this can't be done in consumer-land
Building this in Dispatch (or any other consumer) would mean:
- 25+ method `StorageProvider` re-implementation per consumer
- No shared conformance tests
- Schema drift across consumers — each would invent its own object-store
shape, blocking any future cross-app data import/export
- Every Shade SDK update that adjusts `StorageProvider` would force every
consumer to track and patch independently
`@shade/storage-sqlite` and `@shade/storage-postgres` are part of the SDK
for the same reason. IndexedDB belongs alongside them.
## What unblocks
Shipping this unblocks Dispatch's "Slice 2.5" — persistent enrollment
across browser refresh, which today is the largest QA-friction point in
the dev loop. Any future browser-Shade consumer (web dashboard,
contact-list app, browser-extension messenger) gets persistence for free.

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/cli", "name": "@shade/cli",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/cli.ts", "main": "src/cli.ts",
"bin": { "bin": {

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/core", "name": "@shade/core",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -185,6 +185,22 @@ export async function ratchetDecrypt(
if (!session.receiveChain) { if (!session.receiveChain) {
throw new DecryptionError('No receiving chain available'); throw new DecryptionError('No receiving chain available');
} }
// Defense-in-depth: a same-DH message whose counter is already
// behind the chain — and that did NOT match a cached skipped key —
// is either a duplicate we already decrypted (skipped key was
// consumed) or one whose key was evicted under cache pressure.
// Falling through would call kdfChainKey on the *current* (ahead)
// chainKey and then rewind `chain.counter = message.counter + 1`,
// permanently desyncing the chain so every subsequent decrypt
// returns wrong-key. Reject without mutating state instead.
if (
!isNewRatchet &&
message.counter < session.receiveChain.counter
) {
throw new DecryptionError(
'Failed to decrypt message — wrong key or tampered data',
);
}
await skipMessageKeys(crypto, session, message.dhPublicKey, session.receiveChain, message.counter); await skipMessageKeys(crypto, session, message.dhPublicKey, session.receiveChain, message.counter);
// Advance the receiving chain one more step to get this message's key // Advance the receiving chain one more step to get this message's key

View File

@@ -46,6 +46,50 @@ export interface PersistedStreamState {
*/ */
export type PeerVerificationSource = 'user' | 'transitive' | 'tofu-after-warning'; export type PeerVerificationSource = 'user' | 'transitive' | 'tofu-after-warning';
/**
* Persisted broadcast-channel state (V4.6). Holds the sender-key chain that
* lets the channel owner encrypt a single ciphertext and fan it out to all
* paired members.
*
* - `ownerRole === 'sender'`: this device created the channel; the record
* carries the full chain (chainKey + iteration + signing keypair).
* - `ownerRole === 'receiver'`: this device joined a channel that
* `ownerAddress` owns; we hold a tracking copy of the chain (chainKey +
* iteration + signing public key only — no private signing key).
*
* `generation` is bumped each time `removeMember` rotates the chain. Stale
* broadcasts at lower generations are silently dropped on receive.
*/
export interface BroadcastChannelRecord {
/** Opaque, stable across restarts. UUID-style. */
channelId: string;
ownerRole: 'sender' | 'receiver';
/** The address that owns the channel (i.e. the only sender). */
ownerAddress: string;
label?: string;
/** Sender-key generation — bumped on rotation. Starts at 0. */
generation: number;
/** Current chain key (32 bytes). */
chainKey: Uint8Array;
/** Counter advanced by `senderKeyEncrypt` / `senderKeyDecrypt`. */
iteration: number;
/** Owner's Ed25519 signing public key (32 bytes). */
signingPublicKey: Uint8Array;
/** Owner's Ed25519 signing private key. Present iff `ownerRole === 'sender'`. */
signingPrivateKey?: Uint8Array;
createdAt: number;
updatedAt: number;
}
/** Membership row for a broadcast channel (sender-side only). */
export interface BroadcastMemberRecord {
channelId: string;
peerAddress: string;
joinedAt: number;
/** When this peer was revoked. `null` while still active. */
removedAt: number | null;
}
/** /**
* Persistent record that a peer's safety number was verified at a point * Persistent record that a peer's safety number was verified at a point
* in time. `identityVersion` is the local counter for that peer's identity: * in time. `identityVersion` is the local counter for that peer's identity:
@@ -188,4 +232,35 @@ export interface StorageProvider {
/** Prune stream-state rows in `'finished' | 'aborted'` status older than `olderThan`. */ /** Prune stream-state rows in `'finished' | 'aborted'` status older than `olderThan`. */
pruneStreamStates?(olderThan: number): Promise<void>; pruneStreamStates?(olderThan: number): Promise<void>;
// ─── Broadcast channels (V4.6) — optional ─────────────────
/**
* Persist or replace the broadcast-channel record. Idempotent upsert on
* `channelId`. Backends that don't implement broadcast-channel support
* may omit this; the SDK throws a clear error when the app tries to
* call `createBroadcastChannel` on such a backend.
*/
saveBroadcastChannel?(channel: BroadcastChannelRecord): Promise<void>;
/** Look up a broadcast channel by id. */
getBroadcastChannel?(channelId: string): Promise<BroadcastChannelRecord | null>;
/** Enumerate all broadcast channels persisted on this device. */
listBroadcastChannels?(): Promise<BroadcastChannelRecord[]>;
/** Drop a broadcast channel and all its membership rows. */
removeBroadcastChannel?(channelId: string): Promise<void>;
/** Persist or replace a broadcast-membership row (sender-side only). */
saveBroadcastMember?(member: BroadcastMemberRecord): Promise<void>;
/**
* List membership rows for a channel. Includes revoked members (with
* `removedAt !== null`); callers filter as needed.
*/
getBroadcastMembers?(channelId: string): Promise<BroadcastMemberRecord[]>;
/** Hard-delete a single membership row. */
removeBroadcastMember?(channelId: string, peerAddress: string): Promise<void>;
} }

View File

@@ -281,6 +281,41 @@ describe('Double Ratchet', () => {
expect(ratchetDecrypt(crypto, bob, msg)).rejects.toThrow(); expect(ratchetDecrypt(crypto, bob, msg)).rejects.toThrow();
}); });
/**
* Regression — the v4.2.0 OutboundQueue waiter-since bug delivered
* the same envelope twice to `manager.decrypt`. The first decrypt
* succeeded via a cached skipped key; the second one fell into the
* `message.counter < chain.counter` path with no skipped key
* available, advanced the chainKey ONCE and rewound `chain.counter`
* to `message.counter + 1`, leaving the ratchet permanently
* desynced. ratchetDecrypt now rejects without mutating state when
* a same-DH message is behind the chain and not in skippedKeys, so
* a downstream replay (transport bug, retry, etc.) cannot poison
* the session for everyone else.
*/
test('same-DH stale message after consumed skipped key fails without corrupting state', async () => {
const { alice, bob } = await setupPair();
// Alice sends 3 messages on the same DH chain.
const m0 = await ratchetEncrypt(crypto, alice, enc.encode('m0'));
const m1 = await ratchetEncrypt(crypto, alice, enc.encode('m1'));
const m2 = await ratchetEncrypt(crypto, alice, enc.encode('m2'));
// Bob receives m1 first, caching m0's key. Then m0 (delivered
// via the cache). After this, m0's skipped key is consumed.
expect(dec.decode(await ratchetDecrypt(crypto, bob, m1))).toBe('m1');
expect(dec.decode(await ratchetDecrypt(crypto, bob, m0))).toBe('m0');
// Replay of m0: skippedKey is gone, chain.counter is past m0.
// Pre-fix: this would corrupt Bob's chain state; post-fix it
// throws cleanly.
await expect(ratchetDecrypt(crypto, bob, m0)).rejects.toThrow(DecryptionError);
// Bob can still decrypt the remaining valid message — chain
// state was NOT mutated by the rejected replay.
expect(dec.decode(await ratchetDecrypt(crypto, bob, m2))).toBe('m2');
});
}); });
// ─── Long Conversation ──────────────────────────────────── // ─── Long Conversation ────────────────────────────────────

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/crypto-web", "name": "@shade/crypto-web",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,4 +1,8 @@
import type { StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey, SessionState, RetiredIdentity, PersistedStreamState, PeerVerification } from '@shade/core'; import type {
StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey,
SessionState, RetiredIdentity, PersistedStreamState, PeerVerification,
BroadcastChannelRecord, BroadcastMemberRecord,
} from '@shade/core';
import { constantTimeEqual } from '@shade/core'; import { constantTimeEqual } from '@shade/core';
/** /**
@@ -167,4 +171,62 @@ export class MemoryStorage implements StorageProvider {
} }
} }
} }
// ─── Broadcast channels (V4.6) ────────────────────────────
private broadcastChannels = new Map<string, BroadcastChannelRecord>();
private broadcastMembers = new Map<string, Map<string, BroadcastMemberRecord>>();
async saveBroadcastChannel(channel: BroadcastChannelRecord): Promise<void> {
this.broadcastChannels.set(channel.channelId, cloneChannel(channel));
}
async getBroadcastChannel(channelId: string): Promise<BroadcastChannelRecord | null> {
const v = this.broadcastChannels.get(channelId);
return v ? cloneChannel(v) : null;
}
async listBroadcastChannels(): Promise<BroadcastChannelRecord[]> {
return [...this.broadcastChannels.values()].map(cloneChannel);
}
async removeBroadcastChannel(channelId: string): Promise<void> {
this.broadcastChannels.delete(channelId);
this.broadcastMembers.delete(channelId);
}
async saveBroadcastMember(member: BroadcastMemberRecord): Promise<void> {
let inner = this.broadcastMembers.get(member.channelId);
if (!inner) {
inner = new Map();
this.broadcastMembers.set(member.channelId, inner);
}
inner.set(member.peerAddress, { ...member });
}
async getBroadcastMembers(channelId: string): Promise<BroadcastMemberRecord[]> {
const inner = this.broadcastMembers.get(channelId);
return inner ? [...inner.values()].map((m) => ({ ...m })) : [];
}
async removeBroadcastMember(channelId: string, peerAddress: string): Promise<void> {
this.broadcastMembers.get(channelId)?.delete(peerAddress);
}
}
function cloneChannel(c: BroadcastChannelRecord): BroadcastChannelRecord {
const out: BroadcastChannelRecord = {
channelId: c.channelId,
ownerRole: c.ownerRole,
ownerAddress: c.ownerAddress,
generation: c.generation,
chainKey: new Uint8Array(c.chainKey),
iteration: c.iteration,
signingPublicKey: new Uint8Array(c.signingPublicKey),
createdAt: c.createdAt,
updatedAt: c.updatedAt,
};
if (c.label !== undefined) out.label = c.label;
if (c.signingPrivateKey !== undefined) out.signingPrivateKey = new Uint8Array(c.signingPrivateKey);
return out;
} }

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/dashboard", "name": "@shade/dashboard",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"scripts": { "scripts": {
"dev": "vite", "dev": "vite",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/files", "name": "@shade/files",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -0,0 +1,590 @@
/**
* Browser-friendly request-response `FileClient` for `@shade/files`.
*
* The default `shade.files.client(peer)` ships RPC envelopes via
* `Shade.send` + `Shade.deliverControlEnvelope`, which means the
* server has to be able to call back outbound to the client. That
* doesn't work for browser tabs (no inbound HTTP listener). This
* client posts each RPC envelope to a single server endpoint and
* reads the encrypted response from the same HTTP response — pure
* request-response, no inbound channel required.
*
* Inline payloads only (≤ 256 KiB). For larger reads/writes, use the
* stateful path: `shade.files.client(peer)` server-to-server, with
* `@shade/transfer` chunk routes for content I/O.
*
* @see {@link createFilesRpcRoute} for the matching server-side route.
*/
import type { ZodTypeAny } from 'zod';
import { decodeEnvelope, encodeEnvelope as encodeWireEnvelope } from '@shade/proto';
import type { ShadeBridge } from '../integration/shade-bridge.js';
import {
encodeEnvelope as encodeRpcEnvelope,
tryParseEnvelope,
} from '../protocol/envelope-codec.js';
import {
KIND_CUSTOM_V1,
KIND_DELETE_V1,
KIND_GET_THUMBNAIL_V1,
KIND_LIST_V1,
KIND_MKDIR_V1,
KIND_MOVE_V1,
KIND_READ_V1,
KIND_STAT_V1,
KIND_WRITE_V1,
} from '../protocol/kinds.js';
import {
CustomArgsSchema,
CustomResultSchema,
DeleteArgsSchema,
DeleteResultSchema,
GetThumbnailArgsSchema,
GetThumbnailResultSchema,
ListArgsSchema,
ListResultSchema,
MkdirArgsSchema,
MkdirResultSchema,
MoveArgsSchema,
MoveResultSchema,
ReadArgsSchema,
ReadResultSchema,
StatArgsSchema,
StatResultSchema,
WriteArgsSchema,
WriteResultSchema,
type ListResult,
type MkdirResult,
type DeleteResult,
type MoveResult,
type StatResult,
type ThumbnailSize,
type WriteResult,
} from '../schemas/ops.js';
import {
fileErrorFromPayload,
CancelledError,
InternalFileError,
ConflictError,
} from '../schemas/errors.js';
import { buildRpcRequest } from '../protocol/rpc-builder.js';
import { decideInline, INLINE_THRESHOLD, type WriteSource } from './inline-threshold.js';
import { base64ToBytes, bytesToBase64 } from '../protocol/canonical.js';
import { startQueueDrainer, type QueueDrainerHandle } from './queue-drainer.js';
import {
createClientStreamsBridge,
type ClientStreamsBridge,
} from './streams-bridge.js';
import type {
FileClient,
ReadOpts,
ReadOutput,
ThumbnailResult,
WriteOpts,
CreateFileClientOptions,
BaseOpts,
} from './client.js';
export interface FilesHttpClientOptions
extends Omit<CreateFileClientOptions, 'streamsBridge'> {
/**
* Server endpoint that hosts `createFilesRpcRoute(...)`. Typically:
* `https://server.example.com/api/v1/shade-files/rpc`.
*/
rpcUrl: string;
/**
* Optional `fetch` override. Defaults to `globalThis.fetch`. Wire a
* custom `fetch` to thread auth-cookies, CSRF tokens, or
* service-worker interception.
*/
fetch?: typeof globalThis.fetch;
/**
* Extra HTTP headers applied to every RPC POST. Useful for app-level
* auth (CSRF, session cookies via custom header, etc.) — these are
* orthogonal to the ratchet authentication on the envelope itself.
*/
headers?: Record<string, string>;
/**
* Server endpoint that hosts `transferQueueRoute()`'s long-poll
* endpoint. Typically:
* `https://server.example.com/api/v1/shade-files/queue`.
*
* When supplied, the client starts a background long-poll that
* drains queued envelopes + chunks from the server and dispatches
* them via `shade.acceptTransferEnvelope`. This unlocks
* **streamed reads** (>256 KiB) for browser-style consumers.
*/
outboundQueueUrl?: string;
/**
* Base URL for outbound transfer routes (browser → server). Required
* alongside `outboundQueueUrl` to enable streamed writes. Typically:
* `https://server.example.com/api/v1/shade-files`.
*
* The client POSTs:
* - chunks to `<base>/v1/transfer/<streamId>/chunk`
* - control envelopes to `<base>/v1/transfer/control`
*/
transferBaseUrl?: string;
/**
* Long-poll block timeout, milliseconds. Default 30_000. Server
* clamps to its own `maxBlockMs` (default 55_000).
*/
queueBlockMs?: number;
}
interface RoundTripOpts {
signal?: AbortSignal;
timeoutMs?: number;
idempotencyKey?: string;
}
/**
* Create a request-response `FileClient` bound to `peerAddress` and a
* server-side RPC URL. The session must already be established
* (via `shade.initSessionFromBundle(peerAddress, bundle)` or an
* incoming first-message). Otherwise the first RPC will fail with
* "decrypt failed: no session for peer".
*
* When `outboundQueueUrl` + `transferBaseUrl` are supplied, the
* client also unlocks **streamed reads/writes** for files larger than
* the inline threshold (256 KiB). The browser polls the server's
* outbound queue for chunks/envelopes and POSTs its own outbound
* chunks to the server's transfer-receive routes.
*/
export function createFilesHttpClient(
shade: ShadeBridge,
peerAddress: string,
options: FilesHttpClientOptions,
): FileClient {
const rpcUrl = options.rpcUrl;
const fetchFn = options.fetch ?? globalThis.fetch.bind(globalThis);
const extraHeaders = options.headers ?? {};
const defaultTimeoutMs = options.defaultTimeoutMs ?? 30_000;
const ioTimeoutMs = options.ioTimeoutMs ?? 60_000;
const signRequest = options.signRequest;
const senderAddress = shade.myAddress;
// ─── Streamed-mode bootstrap ─────────────────────────────────
//
// When `outboundQueueUrl` is supplied, the client:
// 1. Configures `shade.configureTransfers(...)` so outbound
// chunks POST to `<transferBaseUrl>/v1/transfer/<streamId>/chunk`
// and outbound control envelopes POST to
// `<transferBaseUrl>/v1/transfer/control`.
// 2. Spawns a streams-bridge so streamed reads can be awaited.
// 3. Starts a long-poll drainer that pulls queued envelopes +
// chunks from the server and dispatches via
// `shade.acceptTransferEnvelope`.
let drainer: QueueDrainerHandle | null = null;
let streamsBridgePromise: Promise<ClientStreamsBridge> | null = null;
let streamsBridge: ClientStreamsBridge | null = null;
if (options.outboundQueueUrl !== undefined) {
const outboundQueueUrl = options.outboundQueueUrl;
if (options.transferBaseUrl === undefined) {
throw new Error(
'createFilesHttpClient: outboundQueueUrl was supplied without transferBaseUrl. Pass `transferBaseUrl` (the server prefix that hosts /v1/transfer/...) so outbound chunks have a destination.',
);
}
if (shade.configureTransfers === undefined) {
throw new Error(
'createFilesHttpClient: shade.configureTransfers is required for streamed mode (the underlying ShadeBridge must surface it).',
);
}
const transferBaseUrl = options.transferBaseUrl.replace(/\/$/, '');
shade.configureTransfers({
resolveBaseUrl: async (peer) => {
if (peer !== peerAddress) {
throw new Error(
`httpClient is bound to peer "${peerAddress}" — refusing to resolve outgoing chunks for "${peer}" without a multi-peer registry. Use shade.files.client(peer) for server-to-server multi-peer.`,
);
}
return transferBaseUrl;
},
});
// Build the streams-bridge eagerly. The engine's incoming-transfer
// subscription has to be in place BEFORE the drainer dispatches the
// first stream-init envelope, otherwise the engine emits the
// IncomingTransfer to zero handlers and the read silently never
// accepts. We kick off the drainer once the bridge has subscribed.
streamsBridgePromise = createClientStreamsBridge(shade).then((bridge) => {
streamsBridge = bridge;
drainer = startQueueDrainer(shade, {
outboundQueueUrl,
peerAddress,
senderAddress,
...(options.fetch !== undefined ? { fetch: options.fetch } : {}),
...(options.headers !== undefined ? { headers: options.headers } : {}),
...(options.queueBlockMs !== undefined ? { blockMs: options.queueBlockMs } : {}),
});
return bridge;
});
// Surface bridge-construction failures eagerly via a rejected
// promise the next read/write picks up.
streamsBridgePromise.catch(() => {
/* observed via getStreamsBridge() */
});
}
async function getStreamsBridge(): Promise<ClientStreamsBridge> {
if (streamsBridge !== null) return streamsBridge;
if (streamsBridgePromise === null) {
throw new ConflictError(
`http RPC client supports inline writes/reads only (≤ ${INLINE_THRESHOLD} bytes) — pass { outboundQueueUrl, transferBaseUrl } to enable streamed transfers.`,
);
}
streamsBridge = await streamsBridgePromise;
return streamsBridge;
}
/**
* Encrypt + POST + decrypt + parse one RPC round-trip.
*
* Throws a typed `FileError` subclass when the server returns an
* encrypted `RpcError`, or `InternalFileError` for transport-level
* failures (network, 4xx/5xx, malformed body).
*/
async function roundTrip<TResult>(
kind: string,
op: 'list' | 'stat' | 'mkdir' | 'delete' | 'move' | 'read' | 'write' | 'getThumbnail' | 'custom',
args: unknown,
resultSchema: ZodTypeAny,
opts: RoundTripOpts | undefined,
): Promise<TResult> {
const requestEnv = await buildRpcRequest({
senderAddress,
kind,
op,
args,
...(opts?.idempotencyKey !== undefined ? { idempotencyKey: opts.idempotencyKey } : {}),
...(signRequest !== undefined ? { signRequest } : {}),
});
const plaintext = encodeRpcEnvelope(requestEnv);
const ratchetEnvelope = await shade.send(peerAddress, plaintext);
const wireBytes = encodeWireEnvelope(ratchetEnvelope);
const ac = new AbortController();
const timeoutMs = opts?.timeoutMs ?? defaultTimeoutMs;
const timer = setTimeout(
() => ac.abort(new Error(`RPC timeout after ${timeoutMs}ms`)),
timeoutMs,
);
(timer as unknown as { unref?: () => void }).unref?.();
if (opts?.signal !== undefined) {
const userSignal = opts.signal;
if (userSignal.aborted) ac.abort(userSignal.reason);
else userSignal.addEventListener('abort', () => ac.abort(userSignal.reason), { once: true });
}
let response: Response;
try {
// Wrap the wire bytes in a Blob so the body type satisfies the
// common-denominator `BodyInit` across DOM, Bun, and node-fetch
// (some runtimes accept `Uint8Array` directly, others don't).
// Cast through `unknown` because TS's `bun-types` and `lib.dom`
// disagree about whether `Uint8Array<ArrayBufferLike>` is itself
// a `BlobPart`; the runtime accepts it on every platform.
response = await fetchFn(rpcUrl, {
method: 'POST',
body: new Blob([wireBytes as unknown as ArrayBuffer]),
signal: ac.signal,
headers: {
'Content-Type': 'application/octet-stream',
'X-Shade-Sender-Address': senderAddress,
...extraHeaders,
},
});
} catch (err) {
clearTimeout(timer);
if ((err as Error).name === 'AbortError') {
throw new CancelledError(`RPC ${kind} aborted: ${(err as Error).message}`);
}
throw new InternalFileError(`RPC ${kind} fetch failed: ${(err as Error).message}`);
}
clearTimeout(timer);
if (!response.ok) {
let body: { error?: string } | null = null;
try {
body = (await response.json()) as { error?: string };
} catch {
/* server emitted non-JSON body */
}
throw new InternalFileError(
`RPC ${kind}${response.status} ${response.statusText}: ${
body?.error ?? '(no error body)'
}`,
);
}
const ab = await response.arrayBuffer();
if (ab.byteLength === 0) {
throw new InternalFileError(`RPC ${kind}: empty response body`);
}
let responseRatchet;
try {
responseRatchet = decodeEnvelope(new Uint8Array(ab));
} catch (err) {
throw new InternalFileError(
`RPC ${kind}: response body is not a valid wire envelope: ${(err as Error).message}`,
);
}
let responsePlaintext: string;
try {
responsePlaintext = await shade.receive(peerAddress, responseRatchet);
} catch (err) {
throw new InternalFileError(
`RPC ${kind}: response decrypt failed: ${(err as Error).message}`,
);
}
const classified = tryParseEnvelope(responsePlaintext);
if (classified === null) {
throw new InternalFileError(
`RPC ${kind}: response plaintext is not a valid @shade/files envelope`,
);
}
if (classified.kind === 'error') {
throw fileErrorFromPayload(classified.envelope.error);
}
if (classified.kind !== 'response') {
throw new InternalFileError(
`RPC ${kind}: unexpected response envelope kind: ${classified.kind}`,
);
}
if (classified.envelope.id !== requestEnv.id) {
throw new InternalFileError(
`RPC ${kind}: response correlation id mismatch (got ${classified.envelope.id}, expected ${requestEnv.id})`,
);
}
return resultSchema.parse(classified.envelope.result) as TResult;
}
return {
async list(path, opts): Promise<ListResult> {
const args = ListArgsSchema.parse({
path,
...(opts?.cursor !== undefined ? { cursor: opts.cursor } : {}),
...(opts?.pageSize !== undefined ? { pageSize: opts.pageSize } : {}),
...(opts?.filter !== undefined ? { filter: opts.filter } : {}),
});
return await roundTrip<ListResult>(
KIND_LIST_V1,
'list',
args,
ListResultSchema,
opts,
);
},
async stat(path, opts): Promise<StatResult> {
const args = StatArgsSchema.parse({ path });
return await roundTrip<StatResult>(KIND_STAT_V1, 'stat', args, StatResultSchema, opts);
},
async mkdir(path, opts): Promise<MkdirResult> {
const args = MkdirArgsSchema.parse({
path,
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
});
return await roundTrip<MkdirResult>(
KIND_MKDIR_V1,
'mkdir',
args,
MkdirResultSchema,
opts,
);
},
async delete(path, opts): Promise<DeleteResult> {
const args = DeleteArgsSchema.parse({
path,
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
});
return await roundTrip<DeleteResult>(
KIND_DELETE_V1,
'delete',
args,
DeleteResultSchema,
opts,
);
},
async move(src, dst, opts): Promise<MoveResult> {
const args = MoveArgsSchema.parse({
src,
dst,
...(opts?.overwrite !== undefined ? { overwrite: opts.overwrite } : {}),
});
return await roundTrip<MoveResult>(KIND_MOVE_V1, 'move', args, MoveResultSchema, opts);
},
async read(path, opts: ReadOpts = {}): Promise<ReadOutput> {
const args = ReadArgsSchema.parse({
path,
...(opts.range !== undefined ? { range: opts.range } : {}),
...(opts.preferInline !== undefined ? { preferInline: opts.preferInline } : {}),
});
const wire = await roundTrip<import('../schemas/ops.js').ReadResult>(
KIND_READ_V1,
'read',
args,
ReadResultSchema,
opts,
);
if (wire.kind === 'inline') {
const bytes = base64ToBytes(wire.bytesB64);
const out: ReadOutput = {
kind: 'inline',
bytes,
size: wire.size,
sha256: wire.sha256,
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
};
return out;
}
// Streamed read — only supported when the queue drainer is wired.
if (drainer === null) {
throw new InternalFileError(
`http RPC client received a streamed read (size ${wire.size}) but is in inline-only mode. Pass { outboundQueueUrl, transferBaseUrl } when constructing the client to enable streamed reads.`,
);
}
const bridge = await getStreamsBridge();
const bridgeSignal = opts.signal ?? new AbortController().signal;
const parked = await bridge.awaitRead(wire.streamId, {
expectedFrom: peerAddress,
signal: bridgeSignal,
timeoutMs: ioTimeoutMs,
});
const out: ReadOutput = {
kind: 'streams',
stream: parked.readable,
size: wire.size,
sha256: wire.sha256,
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
done: async () => {
await parked.done;
},
};
return out;
},
async write(path, input: WriteSource, opts: WriteOpts = {}): Promise<WriteResult> {
const decision = await decideInline(input);
const overwrite = opts.overwrite ?? false;
const contentType = opts.contentType ?? decision.contentType;
if (decision.kind === 'inline' || opts.forceInline === true) {
const bytes = decision.kind === 'inline' ? decision.bytes : null;
if (bytes === null) {
// forceInline === true with a streams-typed decision —
// decideInline always produced a `streams` shape because the
// input was a bare ReadableStream. We can't drain a stream
// synchronously here without a streams-bridge.
throw new ConflictError(
'http RPC client cannot forceInline a streamed input — pass a Uint8Array / Blob, or pre-buffer the stream.',
);
}
if (bytes.byteLength > INLINE_THRESHOLD) {
throw new ConflictError(
`inline write exceeds ${INLINE_THRESHOLD}-byte threshold (got ${bytes.byteLength}); pass forceInline=true to override`,
);
}
const args = WriteArgsSchema.parse({
kind: 'inline',
path,
bytesB64: bytesToBase64(bytes),
...(contentType !== undefined ? { contentType } : {}),
overwrite,
});
return await roundTrip<WriteResult>(
KIND_WRITE_V1,
'write',
args,
WriteResultSchema,
opts,
);
}
// Streamed write — requires the queue drainer + streams-bridge.
if (drainer === null) {
throw new ConflictError(
`http RPC client supports inline writes only (≤ ${INLINE_THRESHOLD} bytes). The supplied input was promoted to streams (size ${decision.size ?? 'unknown'}). Pass { outboundQueueUrl, transferBaseUrl } to enable streamed writes.`,
);
}
const bridge = await getStreamsBridge();
const size = decision.size;
if (size === undefined) {
throw new ConflictError(
'streams write requires a known plaintext size; pass `{ stream, size }` instead of a bare ReadableStream',
);
}
const { writeId, handle } = await bridge.initiateWrite({
peer: peerAddress,
stream: decision.stream,
size,
...(contentType !== undefined ? { contentType } : {}),
name: path,
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
});
const args = WriteArgsSchema.parse({
kind: 'streams',
path,
size,
...(contentType !== undefined ? { contentType } : {}),
overwrite,
writeId,
});
try {
const [result] = await Promise.all([
roundTrip<WriteResult>(KIND_WRITE_V1, 'write', args, WriteResultSchema, opts),
handle.done(),
]);
return result;
} catch (err) {
await handle.abort('rpc-failed').catch(() => undefined);
throw err;
}
},
async getThumbnail(path, size: ThumbnailSize, opts): Promise<ThumbnailResult> {
const args = GetThumbnailArgsSchema.parse({
path,
size,
...(opts?.format !== undefined ? { format: opts.format } : {}),
});
const raw = await roundTrip<import('../schemas/ops.js').GetThumbnailResult>(
KIND_GET_THUMBNAIL_V1,
'getThumbnail',
args,
GetThumbnailResultSchema,
opts,
);
return {
bytes: base64ToBytes(raw.bytesB64),
format: raw.format,
width: raw.width,
height: raw.height,
sha256: raw.sha256,
};
},
async custom(name, args, opts?: BaseOpts): Promise<unknown> {
const wireArgs = CustomArgsSchema.parse({ name, args });
return await roundTrip(KIND_CUSTOM_V1, 'custom', wireArgs, CustomResultSchema, opts);
},
close(): void {
// Stop the long-poll drainer + tear down the streams-bridge if
// we built one. Idempotent — safe to call multiple times.
drainer?.stop();
drainer = null;
if (streamsBridge !== null) {
void streamsBridge.destroy().catch(() => undefined);
streamsBridge = null;
}
streamsBridgePromise = null;
},
} as FileClient;
}

View File

@@ -161,15 +161,33 @@ async function peekStream(stream: ReadableStream<Uint8Array>): Promise<InlineDec
} }
} }
interface MinimalReader { /**
read(): Promise<{ value: Uint8Array | undefined; done: boolean }>; * Structural mirror of WHATWG `ReadableStreamDefaultReader<Uint8Array>`.
*
* The disjoint union shape with `value?: T | undefined` is the lowest
* common denominator across every lib environment we care about:
* - `bun-types` emits `{ done: true; value?: undefined }`
* - `lib.dom` emits `{ done: true; value?: T }`
* - `node:stream/web` emits the union form
*
* `value?: T | undefined` is assignable from all three. A flat
* `{ value?: T; done: boolean }` is rejected by
* `exactOptionalPropertyTypes` because the present branches require
* `value: T`. Defining it as an explicit union avoids the trap.
*/
type MinimalReadResult<T> =
| { done: false; value: T }
| { done: true; value?: T | undefined };
interface MinimalReader<T> {
read(): Promise<MinimalReadResult<T>>;
cancel(reason?: unknown): Promise<void>; cancel(reason?: unknown): Promise<void>;
releaseLock(): void; releaseLock(): void;
} }
function reconstructStream( function reconstructStream(
prefix: Uint8Array[], prefix: Uint8Array[],
reader: MinimalReader, reader: MinimalReader<Uint8Array>,
): ReadableStream<Uint8Array> { ): ReadableStream<Uint8Array> {
let prefixIdx = 0; let prefixIdx = 0;
return new ReadableStream<Uint8Array>({ return new ReadableStream<Uint8Array>({

View File

@@ -0,0 +1,172 @@
/**
* Browser-side drainer for the pull-mode outbound queue.
*
* Background task that long-polls the server's `/queue` endpoint,
* decodes each event, and dispatches it into the consumer's Shade
* instance via `shade.acceptTransferEnvelope`. Same wire-shape as the
* server-to-server case where the engine receives chunks via direct
* HTTP POSTs — we just flip the direction so the browser pulls
* instead of accepts.
*/
import type { ShadeBridge } from '../integration/shade-bridge.js';
export interface QueueDrainerOptions {
/**
* Server endpoint that hosts `transferQueueRoute()`. Typically:
* `https://server.example.com/api/v1/shade-files/queue`.
*/
outboundQueueUrl: string;
/** Peer the queue is pulled FROM (the server's address). */
peerAddress: string;
/** Address we identify ourselves with on the long-poll. */
senderAddress: string;
/** Optional `fetch` override. Default `globalThis.fetch`. */
fetch?: typeof globalThis.fetch;
/** Extra headers applied to every poll request. */
headers?: Record<string, string>;
/**
* Long-poll request timeout (server-side block). Default 30_000.
* Server clamps to its own `maxBlockMs` (default 55_000).
*/
blockMs?: number;
/**
* Backoff after a network error before re-polling. Default 2_000.
* Doubles up to `maxBackoffMs` on consecutive failures.
*/
initialBackoffMs?: number;
maxBackoffMs?: number;
/**
* Called when a poll cycle fails. Defaults to logging via `console.error`.
* Throwing from this hook does NOT stop the drainer — it backs off
* and retries.
*/
onError?: (err: unknown) => void;
}
export interface QueueDrainerHandle {
/** Stop the drainer. Pending fetch is aborted; the loop exits. */
stop(): void;
/** Promise that resolves once the drainer has fully stopped. */
stopped: Promise<void>;
}
interface PolledEvent {
id: number;
timestampMs: number;
kind: 'envelope' | 'chunk';
bytesB64: string;
meta?: { streamId: string; laneId: number; seq: number };
}
interface PollResponseBody {
events: PolledEvent[];
nextSince: number;
}
const DEFAULT_BLOCK_MS = 30_000;
const DEFAULT_INITIAL_BACKOFF_MS = 2_000;
const DEFAULT_MAX_BACKOFF_MS = 30_000;
/**
* Start a long-poll loop that drains queued envelopes + chunks from
* the server and dispatches them into the local Shade instance.
*
* Returns a handle the caller can use to stop the drainer when the
* `httpClient` is closed (e.g. on tab unload).
*/
export function startQueueDrainer(
shade: ShadeBridge,
options: QueueDrainerOptions,
): QueueDrainerHandle {
if (shade.acceptTransferEnvelope === undefined) {
throw new Error(
'startQueueDrainer: shade.acceptTransferEnvelope is required for pull-mode streams. The supplied ShadeBridge implementation must surface it.',
);
}
const accept = shade.acceptTransferEnvelope.bind(shade);
const fetchFn = options.fetch ?? globalThis.fetch.bind(globalThis);
const blockMs = options.blockMs ?? DEFAULT_BLOCK_MS;
const onError = options.onError ?? ((err: unknown) => console.error('[shade.files queue-drainer]', err));
const initialBackoffMs = options.initialBackoffMs ?? DEFAULT_INITIAL_BACKOFF_MS;
const maxBackoffMs = options.maxBackoffMs ?? DEFAULT_MAX_BACKOFF_MS;
const ac = new AbortController();
let stopped = false;
let resolveStopped!: () => void;
const stoppedPromise = new Promise<void>((r) => {
resolveStopped = r;
});
void (async () => {
let since = 0;
let backoff = initialBackoffMs;
while (!stopped) {
try {
const response = await fetchFn(options.outboundQueueUrl, {
method: 'POST',
signal: ac.signal,
headers: {
'Content-Type': 'application/json',
'X-Shade-Sender-Address': options.senderAddress,
...(options.headers ?? {}),
},
body: JSON.stringify({ since, blockMs }),
});
if (!response.ok) {
throw new Error(`queue poll → ${response.status} ${response.statusText}`);
}
const body = (await response.json()) as PollResponseBody;
if (Array.isArray(body.events) && body.events.length > 0) {
for (const event of body.events) {
if (stopped) break;
try {
const bytes = base64ToBytes(event.bytesB64);
await accept(options.peerAddress, bytes);
} catch (err) {
// Per-event dispatch failure should not kill the loop —
// resume picks up missing chunks via @shade/transfer's
// built-in lane-resume protocol.
onError(err);
}
}
}
since = typeof body.nextSince === 'number' ? body.nextSince : since;
backoff = initialBackoffMs;
} catch (err) {
if (stopped || ac.signal.aborted) break;
onError(err);
// Exponential backoff with jitter — caps at maxBackoffMs.
const jitter = Math.floor(Math.random() * Math.min(backoff, 1_000));
await new Promise<void>((r) => {
const t = setTimeout(r, backoff + jitter);
(t as unknown as { unref?: () => void }).unref?.();
ac.signal.addEventListener(
'abort',
() => {
clearTimeout(t);
r();
},
{ once: true },
);
});
backoff = Math.min(maxBackoffMs, backoff * 2);
}
}
resolveStopped();
})();
return {
stop(): void {
if (stopped) return;
stopped = true;
ac.abort(new Error('queue drainer stopped'));
},
stopped: stoppedPromise,
};
}
function base64ToBytes(b64: string): Uint8Array {
const bin = atob(b64);
const out = new Uint8Array(bin.length);
for (let i = 0; i < bin.length; i++) out[i] = bin.charCodeAt(i);
return out;
}

View File

@@ -102,7 +102,16 @@ export async function createClientStreamsBridge(
const readStreamId = incoming.metadata.userMetadata?.[META_KEY_READ_STREAM_ID]; const readStreamId = incoming.metadata.userMetadata?.[META_KEY_READ_STREAM_ID];
if (readStreamId === undefined) return; if (readStreamId === undefined) return;
const ts = new TransformStream<Uint8Array, Uint8Array>(); // Generous HWM so the receiver-side write loop (drainer →
// engine.receiveChunk → sink.write) doesn't stall on backpressure
// before the consumer's reader is wired up. The reader still
// applies its own backpressure once it's consuming, but we no
// longer race fs.read's await on stream-init against the consumer
// attaching its reader.
const ts = new TransformStream<Uint8Array, Uint8Array>(undefined, undefined, {
highWaterMark: 64,
size: (chunk?: Uint8Array) => (chunk === undefined ? 0 : 1),
});
let handle: TransferHandle; let handle: TransferHandle;
try { try {
handle = await incoming.accept({ handle = await incoming.accept({
@@ -142,13 +151,14 @@ export async function createClientStreamsBridge(
} }
parked.set(readStreamId, arrival); parked.set(readStreamId, arrival);
setTimeout(() => { const t = setTimeout(() => {
const stale = parked.get(readStreamId); const stale = parked.get(readStreamId);
if (stale === arrival) { if (stale === arrival) {
parked.delete(readStreamId); parked.delete(readStreamId);
void handle.abort('rpc-timeout').catch(() => undefined); void handle.abort('rpc-timeout').catch(() => undefined);
} }
}, parkedReadTtlMs).unref?.(); }, parkedReadTtlMs);
(t as unknown as { unref?: () => void }).unref?.();
}); });
function cleanupWaiter(w: PendingReadWaiter): void { function cleanupWaiter(w: PendingReadWaiter): void {

View File

@@ -183,6 +183,24 @@ export { MAX_SIGNATURE_AGE_MS } from './server/handler.js';
export { createFilesNamespace } from './integration/files-namespace.js'; export { createFilesNamespace } from './integration/files-namespace.js';
export type { FilesNamespace } from './integration/files-namespace.js'; export type { FilesNamespace } from './integration/files-namespace.js';
// Request-response HTTP transport — for browser-style consumers
// (one HTTP POST per RPC, no inbound channel needed). See
// `docs/files.md § HTTP RPC`.
export { createFilesRpcRoute } from './server/rpc-route.js';
export type { FilesRpcRouteOptions } from './server/rpc-route.js';
export { createFilesHttpClient } from './client/http-client.js';
export type { FilesHttpClientOptions } from './client/http-client.js';
export { startQueueDrainer } from './client/queue-drainer.js';
export type {
QueueDrainerHandle,
QueueDrainerOptions,
} from './client/queue-drainer.js';
// Shared structural surface @shade/files needs from a Shade instance —
// exposed so consumers building custom Shade-shaped bridges can verify
// they implement every required member.
export type { ShadeBridge } from './integration/shade-bridge.js';
// Integration helpers — wire handler + pending registry onto a channel // Integration helpers — wire handler + pending registry onto a channel
export { attachFileHandler } from './integration/wire-server.js'; export { attachFileHandler } from './integration/wire-server.js';
export { attachClientRouting } from './integration/wire-client.js'; export { attachClientRouting } from './integration/wire-client.js';

View File

@@ -4,6 +4,7 @@
* so a single Shade can simultaneously serve files AND consume them from * so a single Shade can simultaneously serve files AND consume them from
* peers without paying the setup cost twice. * peers without paying the setup cost twice.
*/ */
import type { Hono } from 'hono';
import type { ShadeBridge } from './shade-bridge.js'; import type { ShadeBridge } from './shade-bridge.js';
import { import {
attachClientRouting, attachClientRouting,
@@ -11,6 +12,8 @@ import {
createClientStreamsBridge, createClientStreamsBridge,
createFileClient, createFileClient,
createFileHandler, createFileHandler,
createFilesHttpClient,
createFilesRpcRoute,
createServerStreamsBridge, createServerStreamsBridge,
PendingRpcRegistry, PendingRpcRegistry,
ShadeFileRpcChannel, ShadeFileRpcChannel,
@@ -19,22 +22,79 @@ import {
type FileClient, type FileClient,
type FileHandler, type FileHandler,
type FileHandlerConfig, type FileHandlerConfig,
type FilesHttpClientOptions,
type FilesRpcRouteOptions,
type ServerStreamsBridge, type ServerStreamsBridge,
} from '../index.js'; } from '../index.js';
import { IdempotencyCache } from '../server/idempotency-cache.js'; import { IdempotencyCache } from '../server/idempotency-cache.js';
export interface ServeOptions {
/**
* Skip the streams bridge setup. Required for deployments that only
* use the HTTP RPC route ({@link FilesNamespace.rpcRoute}) — those
* deployments don't need to configure `@shade/transfer` because the
* RPC route only services inline payloads (≤ 256 KiB). Without this
* flag, `serve()` calls `createServerStreamsBridge(shade)` which
* eagerly instantiates the transfer engine and fails when
* `configureTransfers({ resolveBaseUrl })` has not been called.
*
* Default: `false` (build the streams bridge — full server-to-server
* stack with streamed reads/writes).
*/
inlineOnly?: boolean;
}
export interface FilesNamespace { export interface FilesNamespace {
/** /**
* Register a file handler. Throws if a handler is already attached on * Register a file handler. Throws if a handler is already attached on
* this Shade — only one server per Shade. The returned function detaches * this Shade — only one server per Shade. The returned function detaches
* the handler and tears down its idempotency / retention timers. * the handler and tears down its idempotency / retention timers.
*
* Pass `{ inlineOnly: true }` for HTTP-RPC-only deployments to skip
* the streams-bridge setup (and the implied `configureTransfers`
* pre-condition).
*/ */
serve(handler: FileHandlerConfig): Promise<() => Promise<void>>; serve(
handler: FileHandlerConfig,
options?: ServeOptions,
): Promise<() => Promise<void>>;
/** /**
* Build a typed file client for `peer`. Multiple concurrent clients to * Build a typed file client for `peer`. Multiple concurrent clients to
* different peers share the same channel + streams bridge. * different peers share the same channel + streams bridge.
*
* Use this for **server-to-server** deployments where both peers can
* receive inbound HTTP. For browser clients (no inbound listener),
* use {@link httpClient} instead.
*/ */
client(peer: string, opts?: Omit<CreateFileClientOptions, 'streamsBridge'>): Promise<FileClient>; client(peer: string, opts?: Omit<CreateFileClientOptions, 'streamsBridge'>): Promise<FileClient>;
/**
* Build a request-response `FileClient` for browser-style consumers.
* Each RPC is one HTTP POST to the supplied `rpcUrl`; the encrypted
* response rides back in the same response body. No inbound channel
* required on the client side.
*
* Inline payloads only (≤ 256 KiB). Streamed reads/writes throw a
* clear error directing callers to {@link client} instead.
*
* Pre-condition: the session for `peer` must already be established
* (typically via `shade.initSessionFromBundle(peer, bundle)`).
*/
httpClient(peer: string, opts: FilesHttpClientOptions): FileClient;
/**
* Mount the server-side request-response RPC route. Returns a Hono
* app exposing `POST /rpc` that accepts encrypted file-RPC envelopes
* and returns encrypted responses in the same HTTP roundtrip.
*
* Mount under any base path:
* ```ts
* app.route('/api/v1/shade-files', shade.files.rpcRoute());
* ```
*
* Requires `shade.files.serve(...)` to have been called first —
* the route dispatches incoming requests through the attached
* handler.
*/
rpcRoute(opts?: FilesRpcRouteOptions): Hono;
/** Tear down channel + bridges. After destroy(), serve()/client() throw. */ /** Tear down channel + bridges. After destroy(), serve()/client() throw. */
destroy(): Promise<void>; destroy(): Promise<void>;
} }
@@ -71,24 +131,39 @@ export function createFilesNamespace(shade: ShadeBridge): FilesNamespace {
} }
return { return {
async serve(handlerConfig) { async serve(handlerConfig, options = {}) {
ensureAlive(); ensureAlive();
if (state.serverHandler !== null) { if (state.serverHandler !== null) {
throw new Error('FilesNamespace: a handler is already registered (one per Shade)'); throw new Error('FilesNamespace: a handler is already registered (one per Shade)');
} }
// Lazy server-side streams bridge. // Lazy server-side streams bridge — skip when the deployment is
if (state.serverBridge === null) { // HTTP-RPC-only and does not need `@shade/transfer` wired up.
if (!options.inlineOnly && state.serverBridge === null) {
state.serverBridge = await createServerStreamsBridge(shade); state.serverBridge = await createServerStreamsBridge(shade);
} }
const inheritedObservability = shade.getObservability?.(); const inheritedObservability = shade.getObservability?.();
const handler = createFileHandler(shade, { const handler = createFileHandler(shade, {
...handlerConfig, ...handlerConfig,
streamsBridge: state.serverBridge, ...(state.serverBridge !== null
? { streamsBridge: state.serverBridge }
: {}),
...(handlerConfig.observability === undefined && inheritedObservability !== undefined ...(handlerConfig.observability === undefined && inheritedObservability !== undefined
? { observability: inheritedObservability } ? { observability: inheritedObservability }
: {}), : {}),
}); });
const detach = attachFileHandler(state.channel, handler); // In inlineOnly mode, the rpc-route is the sole inbound path —
// do NOT also subscribe the channel's onMessage handler to this
// file handler, because that would cause every incoming request
// to be dispatched twice (once by the rpc-route's direct call,
// once by the channel's onMessage handler) and the channel-side
// response would attempt an outbound POST via
// `deliverControlEnvelope`, which is exactly the path that fails
// for browser clients.
const detach: () => void = options.inlineOnly
? () => {
/* no channel subscription to detach */
}
: attachFileHandler(state.channel, handler);
state.serverHandler = handler; state.serverHandler = handler;
state.serverDetach = detach; state.serverDetach = detach;
@@ -131,6 +206,21 @@ export function createFilesNamespace(shade: ShadeBridge): FilesNamespace {
}); });
}, },
httpClient(peer, opts) {
ensureAlive();
return createFilesHttpClient(shade, peer, opts);
},
rpcRoute(opts = {}) {
ensureAlive();
if (state.serverHandler === null) {
throw new Error(
'FilesNamespace.rpcRoute(): no handler attached. Call shade.files.serve(...) before mounting the RPC route.',
);
}
return createFilesRpcRoute(shade, state.serverHandler, opts);
},
async destroy() { async destroy() {
if (state.destroyed) return; if (state.destroyed) return;
state.destroyed = true; state.destroyed = true;

View File

@@ -32,6 +32,12 @@ export interface ShadeBridge {
/** Encrypt + send `plaintext` to `peer`; returns the wire envelope. */ /** Encrypt + send `plaintext` to `peer`; returns the wire envelope. */
send(peer: string, plaintext: string): Promise<ShadeEnvelope>; send(peer: string, plaintext: string): Promise<ShadeEnvelope>;
/**
* Decrypt an inbound envelope from `peer` and return the plaintext.
* Used by the request-response RPC route on the server side.
*/
receive(peer: string, envelope: ShadeEnvelope): Promise<string>;
/** /**
* Subscribe to incoming ratchet plaintext. Returns an unsubscribe. * Subscribe to incoming ratchet plaintext. Returns an unsubscribe.
* Handlers may be sync or async; async handlers are awaited in * Handlers may be sync or async; async handlers are awaited in
@@ -64,4 +70,25 @@ export interface ShadeBridge {
/** Optional control-envelope passthrough used by the WebRTC bridge. */ /** Optional control-envelope passthrough used by the WebRTC bridge. */
deliverControlEnvelope?(peer: string, envelope: ShadeEnvelope): Promise<void>; deliverControlEnvelope?(peer: string, envelope: ShadeEnvelope): Promise<void>;
/**
* Hand a freshly-decoded wire envelope (control or chunk) to the
* transfer engine. Required by the pull-mode HTTP client when it
* drains queued events from the server: each polled chunk / control
* envelope is dispatched here so the engine sees it just as if it
* had arrived via an HTTP POST on `/v1/transfer/...`.
*/
acceptTransferEnvelope?(from: string, env: ShadeEnvelope | Uint8Array): Promise<void>;
/**
* Configure the transfer stack. Called by the pull-mode HTTP client
* to point the browser's outgoing chunks + control envelopes at the
* server's transferQueueRoute mount. Optional because the
* server-to-server path uses a separate, app-driven configuration.
*/
configureTransfers?(opts: {
resolveBaseUrl?: (peerAddress: string) => Promise<string>;
transport?: unknown;
envelopeTransport?: unknown;
}): void;
} }

View File

@@ -0,0 +1,126 @@
/**
* Shared RPC-request construction.
*
* Both the channel-based `FileClient` (`createFileClient`) and the
* HTTP-based `FilesHttpClient` build identical `RpcRequest` envelopes
* — they differ only in *transport* (channel.send → ratchet via
* Shade.send/onMessage vs HTTP POST → ratchet via single
* request-response). This module is the single source of truth for
* the wire shape so the two clients can never drift.
*/
import type { ZodTypeAny } from 'zod';
import {
KIND_DELETE_V1,
KIND_GET_THUMBNAIL_V1,
KIND_LIST_V1,
KIND_MKDIR_V1,
KIND_MOVE_V1,
KIND_READ_V1,
KIND_STAT_V1,
KIND_WRITE_V1,
MUTATION_OPS,
type StandardOp,
} from './kinds.js';
import { generateIdempotencyKey, generateRequestId } from './correlate.js';
import { canonicalRpcBytes, hashArgs } from './canonical.js';
import type { RpcRequest } from '../schemas/envelope.js';
export const KIND_BY_OP: Record<StandardOp, string> = {
list: KIND_LIST_V1,
stat: KIND_STAT_V1,
mkdir: KIND_MKDIR_V1,
delete: KIND_DELETE_V1,
move: KIND_MOVE_V1,
read: KIND_READ_V1,
write: KIND_WRITE_V1,
getThumbnail: KIND_GET_THUMBNAIL_V1,
};
export type SignRequest = (canonicalBytes: Uint8Array) => Promise<string> | string;
export interface BuildRpcRequestOptions {
/** Address that this RPC call originates from. */
senderAddress: string;
/** RPC kind — `KIND_*_V1` constants for standard ops, `KIND_CUSTOM_V1` otherwise. */
kind: string;
/** Op classifier so mutations get an auto-generated idempotency key. */
op: StandardOp | 'custom';
/** Validated args object. The caller is responsible for `Zod.parse(args)`. */
args: unknown;
/** Caller-supplied idempotency key. Mutations get one auto-generated. */
idempotencyKey?: string;
/** Optional Ed25519-style signer over the canonical bytes. */
signRequest?: SignRequest;
}
/**
* Build a single `RpcRequest` envelope. Generates `id`, `signedAt`,
* idempotency key (mutations only), and the canonical signature.
*/
export async function buildRpcRequest(
options: BuildRpcRequestOptions,
): Promise<RpcRequest> {
const { senderAddress, kind, op, args, signRequest } = options;
const requestId = generateRequestId();
const isMutation = MUTATION_OPS.has(op);
const idempotencyKey =
options.idempotencyKey ?? (isMutation ? generateIdempotencyKey() : undefined);
const signedAt = Date.now();
let sig = 'unsigned';
if (signRequest !== undefined) {
const canonical = canonicalRpcBytes({
address: senderAddress,
signedAt,
kind,
id: requestId,
argsHash: hashArgs(args),
});
sig = await signRequest(canonical);
}
const env: RpcRequest = {
kind,
id: requestId,
args,
...(idempotencyKey !== undefined ? { idempotencyKey } : {}),
sig,
signedAt,
};
return env;
}
/**
* Helper for the typed standard ops: validates args via the supplied Zod
* schema, calls {@link buildRpcRequest}, and forwards the result.
*
* The HTTP and channel clients both call this from each op-method to
* guarantee identical wire-shape. Custom ops use {@link buildRpcRequest}
* directly because they ship `KIND_CUSTOM_V1` plus runtime-validated args.
*/
export async function buildStandardRpcRequest<TArgs>(
schema: { parse(input: unknown): TArgs },
rawArgs: unknown,
options: Omit<BuildRpcRequestOptions, 'kind' | 'op' | 'args'> & {
op: StandardOp;
},
): Promise<{ request: RpcRequest; args: TArgs }> {
const args = schema.parse(rawArgs) as TArgs;
const kind = KIND_BY_OP[options.op];
const request = await buildRpcRequest({
...options,
kind,
op: options.op,
args,
});
return { request, args };
}
/**
* Validate a returned RpcResponse `result` against the supplied schema.
* Centralised so the HTTP and channel clients fail-loudly with the same
* error type when a server returns an off-spec response.
*/
export function parseResult<T>(schema: ZodTypeAny, raw: unknown): T {
return schema.parse(raw) as T;
}

View File

@@ -0,0 +1,218 @@
/**
* Request-response RPC route for `@shade/files`.
*
* Mounts a single `POST /rpc` Hono endpoint that accepts an encrypted
* `RpcRequest` envelope, dispatches it through the file handler, and
* returns the encrypted `RpcResponse` (or `RpcError`) envelope in the
* SAME HTTP response.
*
* This is the browser-friendly transport: the server never needs to
* make outbound calls back to the client, so a browser tab — which
* cannot host an HTTP server — can fully consume `@shade/files`.
*
* ### Wire contract
*
* Request:
* ```
* POST <mount>/rpc HTTP/1.1
* Content-Type: application/octet-stream
* X-Shade-Sender-Address: <peer address>
*
* <wire-encoded ShadeEnvelope (0x01 PreKeyMessage or 0x02 RatchetMessage)
* containing JSON-encoded RpcRequest>
* ```
*
* Response (success):
* ```
* 200 OK
* Content-Type: application/octet-stream
*
* <wire-encoded ShadeEnvelope (0x02 RatchetMessage)
* containing JSON-encoded RpcResponse | RpcError>
* ```
*
* Response (transport-level failure — no session, undecryptable, etc.):
* ```
* 4xx
* Content-Type: application/json
*
* { "error": "..." }
* ```
*
* ### Symmetry with shade-auth-middleware
*
* The shape mirrors `@shade/server`'s shade-auth-middleware: an
* encrypted envelope rides the request body, the server decrypts via
* the existing ratchet session, performs the protected operation,
* and returns an encrypted envelope in the response. No bidirectional
* channel required.
*
* @see {@link createFilesHttpClient} for the matching browser client.
*/
import { Hono } from 'hono';
import { decodeEnvelope, encodeEnvelope } from '@shade/proto';
import type { ShadeBridge } from '../integration/shade-bridge.js';
import {
encodeEnvelope as encodeRpcEnvelope,
tryParseEnvelope,
} from '../protocol/envelope-codec.js';
import type { FileHandler } from './handler.js';
import type { RpcError, RpcRequest, RpcResponse } from '../schemas/envelope.js';
import { KIND_ERROR_V1 } from '../protocol/kinds.js';
export interface FilesRpcRouteOptions {
/**
* Maximum request body size in bytes. Default 1 MiB. Inline payloads
* are capped at 256 KiB by the protocol; the headroom is for
* custom-op payloads and base64 inflation.
*/
maxBodyBytes?: number;
/**
* Allow this server to accept the very first message (PreKeyMessage,
* `0x01`) over the RPC route. Disabled by default — most browser
* clients establish a session via `shade.initSessionFromBundle`
* before the first RPC. Enable when you want the RPC route to also
* be the X3DH carrier (uncommon but supported).
*/
acceptFirstMessage?: boolean;
}
const DEFAULT_MAX_BODY_BYTES = 1 * 1024 * 1024;
/**
* Build a Hono app with a single `POST /rpc` route. Mount under any
* base path: `app.route('/api/v1/shade-files', shade.files.rpcRoute())`.
*
* The `handler` must already be attached (typically via
* `shade.files.serve(handlerConfig)`); this route only ships the
* transport — it does not register a new file handler.
*/
export function createFilesRpcRoute(
shade: ShadeBridge,
handler: FileHandler,
options: FilesRpcRouteOptions = {},
): Hono {
const maxBodyBytes = options.maxBodyBytes ?? DEFAULT_MAX_BODY_BYTES;
const app = new Hono();
app.post('/rpc', async (c) => {
const senderAddress = c.req.header('X-Shade-Sender-Address');
if (senderAddress === undefined || senderAddress === '') {
return c.json({ error: 'missing X-Shade-Sender-Address header' }, 400);
}
const contentLengthHeader = c.req.header('Content-Length');
if (contentLengthHeader !== undefined) {
const contentLength = Number.parseInt(contentLengthHeader, 10);
if (Number.isFinite(contentLength) && contentLength > maxBodyBytes) {
return c.json(
{ error: `body exceeds maxBodyBytes (${contentLength} > ${maxBodyBytes})` },
413,
);
}
}
let bodyBytes: Uint8Array;
try {
const ab = await c.req.arrayBuffer();
if (ab.byteLength > maxBodyBytes) {
return c.json(
{ error: `body exceeds maxBodyBytes (${ab.byteLength} > ${maxBodyBytes})` },
413,
);
}
bodyBytes = new Uint8Array(ab);
} catch (err) {
return c.json({ error: `failed to read request body: ${(err as Error).message}` }, 400);
}
if (bodyBytes.byteLength === 0) {
return c.json({ error: 'empty request body' }, 400);
}
// Decode the wire envelope. `decodeEnvelope` handles both `0x01`
// PreKeyMessage and `0x02` RatchetMessage shapes.
let plaintext: string;
try {
const envelope = decodeEnvelope(bodyBytes);
// First-message gate: only allow `prekey` envelopes when the
// operator has explicitly opted in.
if (options.acceptFirstMessage !== true && envelope.type === 'prekey') {
return c.json(
{
error:
'PreKeyMessage envelopes are not accepted on this RPC route — establish the session first via shade.initSessionFromBundle, or set acceptFirstMessage: true',
},
400,
);
}
plaintext = await shade.receive(senderAddress, envelope);
} catch (err) {
// Decryption failure — could be no session, corrupted envelope,
// or sender address mismatch. Treat as 401 since the envelope is
// self-authenticating: a valid sender would decrypt cleanly.
return c.json({ error: `decrypt failed: ${(err as Error).message}` }, 401);
}
// Parse the plaintext as an RpcRequest.
const classified = tryParseEnvelope(plaintext);
if (classified === null) {
return c.json({ error: 'plaintext is not a valid @shade/files envelope' }, 400);
}
if (classified.kind !== 'request') {
// Cancel envelopes are silently dropped — RPC route is request/
// response only. Cancellation across HTTP is achieved via
// AbortController on the client side, not protocol-level.
if (classified.kind === 'cancel') {
handler.handleCancel(senderAddress, classified.envelope);
// No response body — the cancel was best-effort.
return new Response(null, { status: 204 });
}
return c.json(
{ error: `unexpected envelope kind on RPC route: ${classified.kind}` },
400,
);
}
const request: RpcRequest = classified.envelope;
// Dispatch through the file handler.
let result: RpcResponse | RpcError;
try {
result = await handler.handleRequest(senderAddress, request);
} catch (err) {
// Should never happen — handler.handleRequest catches its own
// errors and returns RpcError. If it didn't, that's a bug; emit
// a generic transport-level RpcError so the client can surface
// it deterministically.
result = {
kind: KIND_ERROR_V1,
id: request.id,
error: {
code: 'INTERNAL',
message: `handler raised: ${(err as Error).message}`,
},
};
}
// Encrypt the response and return it as wire bytes.
let responseBytes: Uint8Array;
try {
const responsePlaintext = encodeRpcEnvelope(result);
const responseEnvelope = await shade.send(senderAddress, responsePlaintext);
responseBytes = encodeEnvelope(responseEnvelope);
} catch (err) {
return c.json(
{ error: `failed to encrypt response: ${(err as Error).message}` },
500,
);
}
return new Response(new Blob([responseBytes as unknown as ArrayBuffer]), {
status: 200,
headers: { 'Content-Type': 'application/octet-stream' },
});
});
return app;
}

View File

@@ -168,13 +168,14 @@ export async function createServerStreamsBridge(
// No waiter yet — park. // No waiter yet — park.
parked.set(writeId, arrived); parked.set(writeId, arrived);
setTimeout(() => { const parkTimer = setTimeout(() => {
const stale = parked.get(writeId); const stale = parked.get(writeId);
if (stale === arrived) { if (stale === arrived) {
parked.delete(writeId); parked.delete(writeId);
void handle.abort('rpc-timeout').catch(() => undefined); void handle.abort('rpc-timeout').catch(() => undefined);
} }
}, parkedWriteTtlMs).unref?.(); }, parkedWriteTtlMs);
(parkTimer as unknown as { unref?: () => void }).unref?.();
}); });
function cleanupWaiter(w: PendingWaiter): void { function cleanupWaiter(w: PendingWaiter): void {

View File

@@ -0,0 +1,184 @@
import { describe, expect, test } from 'bun:test';
import { createShade } from '@shade/sdk';
import {
createPrekeyServer,
MemoryPrekeyStore,
PrekeyServerEvents,
} from '@shade/server';
import { SubtleCryptoProvider } from '@shade/crypto-web';
import { Hono } from 'hono';
const crypto = new SubtleCryptoProvider();
/**
* Concurrent-ratchet hardening tests.
*
* Reproduces the scenario described in the v4.2.0 ratchet-desync bug
* report: with the queue-drainer running on Alice and many concurrent
* `shade.send`/RPC operations against the same peer, do
* encrypt/decrypt paths share the per-peer mutex on
* `ShadeSessionManager` so that no path observes a stale ratchet
* state?
*
* If the lock coverage regresses (a future change re-introduces a
* sidekanal bypass), one of these tests will fail with
* `DecryptionError: Failed to decrypt message — wrong key or
* tampered data`.
*/
async function setupPullRig() {
const prekey = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
events: new PrekeyServerEvents(),
});
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
const prekeyUrl = `http://localhost:${prekeyServer.port}`;
const alice = await createShade({ prekeyServer: prekeyUrl, address: 'alice' });
const bob = await createShade({ prekeyServer: prekeyUrl, address: 'bob' });
const queueRoute = await bob.transferQueueRoute({ blockMs: 500 });
await bob.files.serve({
stat: async () => ({
name: '_',
kind: 'dir' as const,
size: 0,
mtime: 0,
metadata: {},
}),
});
const rpcRoute = bob.files.rpcRoute({ acceptFirstMessage: true });
const app = new Hono();
app.route('/', queueRoute);
app.route('/', rpcRoute);
const bobServer = Bun.serve({ port: 0, fetch: app.fetch });
const baseUrl = `http://localhost:${bobServer.port}`;
const fs = alice.files.httpClient('bob', {
rpcUrl: `${baseUrl}/rpc`,
outboundQueueUrl: `${baseUrl}/queue`,
transferBaseUrl: baseUrl,
defaultTimeoutMs: 10_000,
queueBlockMs: 500,
});
return {
alice,
bob,
fs,
baseUrl,
teardown: async () => {
fs.close();
await alice.shutdown();
await bob.shutdown();
bobServer.stop();
prekeyServer.stop();
},
};
}
describe('@shade/files — concurrent ratchet under drainer', () => {
test('100 parallel httpClient RPCs while drainer runs — no DecryptionError', async () => {
const rig = await setupPullRig();
try {
// Warm-up: establishes the X3DH session (Alice → Bob first message
// is a PreKeyMessage; subsequent messages are pure ratchet).
const first = await rig.fs.stat('/');
expect(first.kind).toBe('dir');
// Fire 100 concurrent stat RPCs. Each one is a full ratchet
// round-trip: encrypt request, POST, decrypt response. They all
// contend for `manager.peerOpChains["bob"]` on Alice's side
// (encrypt + decrypt) and `manager.peerOpChains["alice"]` on
// Bob's side. Drainer is running in the background polling
// Bob's queue — its decrypt path also funnels through the same
// per-peer lock.
// 100 concurrent — minimal repro (after warm-up only).
const N = 100;
const results = await Promise.allSettled(
Array.from({ length: N }, () => rig.fs.stat('/')),
);
const failures = results.filter((s) => s.status === 'rejected') as Array<
PromiseRejectedResult
>;
if (failures.length > 0) {
const sample = failures.slice(0, 1).map((f) => String(f.reason));
throw new Error(`${failures.length}/${N} concurrent RPCs failed: ${sample[0]}`);
}
} finally {
await rig.teardown();
}
}, 30_000);
test('parallel shade.send + drainer + RPCs — ratchet stays in sync', async () => {
const rig = await setupPullRig();
try {
// Establish session via one warm-up RPC.
await rig.fs.stat('/');
// Subscribe Bob to inbound plaintext from Alice — when Alice's
// raw `shade.send` plaintext arrives, Bob echoes a reply back
// through `shade.send` + `deliverControlEnvelope`, which the
// pull-mode envelope transport enqueues for Alice's drainer.
// This injects extra inbound traffic into Alice's drainer in
// parallel with her ongoing RPCs.
const echoes: string[] = [];
rig.bob.onMessage(async (from, plaintext) => {
if (from !== 'alice') return;
if (!plaintext.startsWith('ping:')) return;
echoes.push(plaintext);
const reply = `pong:${plaintext.slice('ping:'.length)}`;
const env = await rig.bob.send('alice', reply);
await rig.bob.deliverControlEnvelope('alice', env);
});
const inboundDrained: string[] = [];
rig.alice.onMessage((from, plaintext) => {
if (from !== 'bob') return;
if (plaintext.startsWith('pong:')) inboundDrained.push(plaintext);
});
// Mix three concurrent workloads against the same peer:
// - 50 inline file RPCs through httpClient (encrypt + decrypt)
// - 50 raw `shade.send` deliveries via control envelope
// - drainer pulling Bob's responses + echoes
const N = 50;
const rpcs = Array.from({ length: N }, () => rig.fs.stat('/'));
const sends = Array.from({ length: N }, async (_, i) => {
const env = await rig.alice.send('bob', `ping:${i}`);
await rig.alice.deliverControlEnvelope('bob', env);
});
const settled = await Promise.allSettled([...rpcs, ...sends]);
const failures = settled.filter((s) => s.status === 'rejected') as Array<
PromiseRejectedResult
>;
if (failures.length > 0) {
const sample = failures.slice(0, 3).map((f) => String(f.reason));
throw new Error(
`${failures.length}/${settled.length} concurrent ops failed: ${sample.join(' | ')}`,
);
}
// Give Bob's queue + Alice's drainer a beat to drain pongs back.
// Echoes round-trip Alice → Bob (control envelope) → Bob's
// onMessage → Bob.send + deliver (queue) → Alice's drainer →
// Alice's onMessage. We just verify some make it back without
// any DecryptionError surfacing.
const deadline = Date.now() + 5_000;
while (inboundDrained.length < N && Date.now() < deadline) {
await new Promise((r) => setTimeout(r, 50));
}
// Don't gate on every echo arriving — the long-poll cadence and
// bun's serve/abort timing can lag a few. We only care that the
// ratchet didn't desync; if it had, every subsequent op would
// throw DecryptionError above.
expect(echoes.length).toBe(N);
expect(inboundDrained.length).toBeGreaterThan(0);
} finally {
await rig.teardown();
}
}, 30_000);
});

View File

@@ -0,0 +1,208 @@
import { describe, expect, test } from 'bun:test';
import { createShade } from '@shade/sdk';
import {
createPrekeyServer,
MemoryPrekeyStore,
PrekeyServerEvents,
} from '@shade/server';
import { SubtleCryptoProvider } from '@shade/crypto-web';
import { Hono } from 'hono';
const crypto = new SubtleCryptoProvider();
/**
* Stand up the full pull-mode rig:
* - Prekey server (for X3DH)
* - Bob: file handler + rpcRoute + transferQueueRoute, all on one server
* - Alice: httpClient with outboundQueueUrl + transferBaseUrl wired
*
* Returns Alice's `FileClient`, which speaks browser-style: ONE base URL,
* no inbound listener, streams supported via long-poll.
*/
async function setupPullRig(opts: {
bobHandler: Parameters<NonNullable<Awaited<ReturnType<typeof createShade>>['files']>['serve']>[0];
}) {
const prekey = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
events: new PrekeyServerEvents(),
});
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
const prekeyUrl = `http://localhost:${prekeyServer.port}`;
const alice = await createShade({ prekeyServer: prekeyUrl, address: 'alice' });
const bob = await createShade({ prekeyServer: prekeyUrl, address: 'bob' });
// Bob: queue-route FIRST (configures bob's transports), then files.serve.
const queueRoute = await bob.transferQueueRoute({ blockMs: 1_500 });
await bob.files.serve(opts.bobHandler);
const rpcRoute = bob.files.rpcRoute({ acceptFirstMessage: true });
const app = new Hono();
app.route('/', queueRoute);
app.route('/', rpcRoute);
const bobServer = Bun.serve({ port: 0, fetch: app.fetch });
const baseUrl = `http://localhost:${bobServer.port}`;
const fs = alice.files.httpClient('bob', {
rpcUrl: `${baseUrl}/rpc`,
outboundQueueUrl: `${baseUrl}/queue`,
transferBaseUrl: baseUrl,
defaultTimeoutMs: 10_000,
queueBlockMs: 1_000,
});
return {
alice,
bob,
fs,
baseUrl,
teardown: async () => {
fs.close();
await alice.shutdown();
await bob.shutdown();
bobServer.stop();
prekeyServer.stop();
},
};
}
describe('@shade/files HTTP RPC — pull-mode streams', () => {
test('streamed read (4 MiB) via long-poll queue', async () => {
const payload = new Uint8Array(4 * 1024 * 1024);
for (let i = 0; i < payload.length; i++) payload[i] = (i * 97) & 0xff;
const rig = await setupPullRig({
bobHandler: {
read: async () => {
// Return the payload as a streamed read so the rpc-handler
// promotes it via the streams-bridge into a transfer.
const stream = new ReadableStream<Uint8Array>({
start(controller) {
const CHUNK = 256 * 1024;
for (let off = 0; off < payload.byteLength; off += CHUNK) {
controller.enqueue(payload.slice(off, Math.min(off + CHUNK, payload.byteLength)));
}
controller.close();
},
});
// Need a precomputed sha256 for streamed reads. Use the
// crypto provider's sha256 directly.
const digest = new Uint8Array(await globalThis.crypto.subtle.digest('SHA-256', payload));
const sha256Hex = Array.from(digest, (b) => b.toString(16).padStart(2, '0')).join('');
return {
kind: 'streams' as const,
stream,
size: payload.byteLength,
sha256: sha256Hex,
contentType: 'application/octet-stream',
};
},
},
});
try {
const result = await rig.fs.read('/big.bin');
expect(result.kind).toBe('streams');
if (result.kind !== 'streams') return;
// Drain the stream and compare.
const reader = result.stream.getReader();
const got = new Uint8Array(payload.byteLength);
let offset = 0;
while (true) {
const { value, done } = await reader.read();
if (done) break;
if (value !== undefined) {
got.set(value, offset);
offset += value.byteLength;
}
}
reader.releaseLock();
await result.done();
expect(offset).toBe(payload.byteLength);
// Compare in 64KiB strides for speed.
let mismatch = -1;
for (let i = 0; i < payload.byteLength; i++) {
if (got[i] !== payload[i]) {
mismatch = i;
break;
}
}
expect(mismatch).toBe(-1);
} finally {
await rig.teardown();
}
}, 30_000);
test('streamed read fails with clear error when outboundQueueUrl is omitted', async () => {
const rig = await setupPullRig({
bobHandler: {
read: async () => {
const stream = new ReadableStream<Uint8Array>({
start(c) {
c.enqueue(new Uint8Array(512 * 1024));
c.close();
},
});
const digest = new Uint8Array(await globalThis.crypto.subtle.digest('SHA-256', new Uint8Array(512 * 1024)));
const sha256Hex = Array.from(digest, (b) => b.toString(16).padStart(2, '0')).join('');
return {
kind: 'streams' as const,
stream,
size: 512 * 1024,
sha256: sha256Hex,
};
},
},
});
// Tear down the rig's drainer so we can construct an inline-only client
rig.fs.close();
const inlineOnly = rig.alice.files.httpClient('bob', {
rpcUrl: `${rig.baseUrl}/rpc`,
defaultTimeoutMs: 10_000,
});
try {
await expect(inlineOnly.read('/big.bin')).rejects.toThrow(/streamed read/);
} finally {
inlineOnly.close();
await rig.teardown();
}
}, 15_000);
test('long-poll returns empty events on idle timeout', async () => {
const rig = await setupPullRig({
bobHandler: {
stat: async () => ({
name: '_',
kind: 'dir' as const,
size: 0,
mtime: 0,
metadata: {},
}),
},
});
try {
// Direct poll without any pending events — should return after blockMs.
const start = Date.now();
const res = await fetch(`${rig.baseUrl}/queue`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-Shade-Sender-Address': 'alice',
},
body: JSON.stringify({ since: 0, blockMs: 500 }),
});
expect(res.status).toBe(200);
const body = (await res.json()) as { events: unknown[]; nextSince: number };
expect(body.events).toHaveLength(0);
expect(Date.now() - start).toBeGreaterThanOrEqual(400);
} finally {
await rig.teardown();
}
}, 10_000);
});

View File

@@ -0,0 +1,321 @@
import { describe, expect, test } from 'bun:test';
import { createShade } from '@shade/sdk';
import {
createPrekeyServer,
MemoryPrekeyStore,
PrekeyServerEvents,
} from '@shade/server';
import { SubtleCryptoProvider } from '@shade/crypto-web';
const crypto = new SubtleCryptoProvider();
/**
* Stand up a prekey server + two Shades + Bob's file handler + RPC route
* mounted on Bun.serve, then return Alice's HTTP-only `FileClient`.
*
* Mirrors the request-response setup a browser client would use against a
* Bun-style server.
*/
async function setupHttpRig(opts: {
bobHandler: Parameters<NonNullable<Awaited<ReturnType<typeof createShade>>['files']>['serve']>[0];
}) {
// 1. Prekey server.
const prekeyEvents = new PrekeyServerEvents();
const prekey = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
events: prekeyEvents,
});
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
const prekeyUrl = `http://localhost:${prekeyServer.port}`;
// 2. Two Shades. Alice plays the browser client (no transferRoute);
// Bob is the server.
const alice = await createShade({ prekeyServer: prekeyUrl, address: 'alice' });
const bob = await createShade({ prekeyServer: prekeyUrl, address: 'bob' });
// 3. Bob: register file handler (HTTP-only — no streams) + mount
// the RPC route.
await bob.files.serve(opts.bobHandler, { inlineOnly: true });
const rpcRoute = bob.files.rpcRoute({ acceptFirstMessage: true });
const bobServer = Bun.serve({ port: 0, fetch: rpcRoute.fetch });
const rpcUrl = `http://localhost:${bobServer.port}/rpc`;
// 5. Alice: build the HTTP-only file client.
const fs = alice.files.httpClient('bob', { rpcUrl, defaultTimeoutMs: 5000 });
return {
alice,
bob,
fs,
rpcUrl,
teardown: async () => {
await alice.shutdown();
await bob.shutdown();
bobServer.stop();
prekeyServer.stop();
},
};
}
describe('@shade/files HTTP RPC — round-trip', () => {
test('list → mkdir → stat → write inline → read inline → delete via httpClient', async () => {
interface VfsEntry {
kind: 'file' | 'dir';
bytes?: Uint8Array;
contentType?: string;
}
const vfs = new Map<string, VfsEntry>([
['/', { kind: 'dir' }],
['/photos', { kind: 'dir' }],
]);
const rig = await setupHttpRig({
bobHandler: {
list: async (ctx) => {
const prefix = ctx.path.endsWith('/') ? ctx.path : `${ctx.path}/`;
const entries = Array.from(vfs.entries())
.filter(([p]) => p.startsWith(prefix) && p !== ctx.path && !p.slice(prefix.length).includes('/'))
.map(([p, e]) => ({
name: p.slice(prefix.length) || p,
kind: e.kind,
size: e.bytes?.byteLength ?? 0,
mtime: 0,
metadata: {},
}));
return { entries, hasMore: false };
},
stat: async (ctx) => {
const e = vfs.get(ctx.path);
if (!e) throw new (await import('../../src/index.js')).NotFoundError(`stat ${ctx.path}`);
return {
name: ctx.path.split('/').pop() ?? ctx.path,
kind: e.kind,
size: e.bytes?.byteLength ?? 0,
mtime: 0,
metadata: {},
...(e.contentType !== undefined ? { contentType: e.contentType } : {}),
};
},
mkdir: async (ctx) => {
vfs.set(ctx.path, { kind: 'dir' });
return { entry: { name: ctx.path.split('/').pop() ?? ctx.path, kind: 'dir' as const, size: 0, mtime: 0, metadata: {} } };
},
delete: async (ctx) => {
if (!vfs.has(ctx.path)) {
throw new (await import('../../src/index.js')).NotFoundError(`delete ${ctx.path}`);
}
vfs.delete(ctx.path);
return { deletedCount: 1 };
},
read: async (ctx) => {
const e = vfs.get(ctx.path);
if (!e || e.kind !== 'file' || !e.bytes) {
throw new (await import('../../src/index.js')).NotFoundError(`read ${ctx.path}`);
}
// Omit sha256 — dispatcher computes it from the bytes.
return {
kind: 'inline' as const,
bytes: e.bytes,
...(e.contentType !== undefined ? { contentType: e.contentType } : {}),
};
},
write: async (ctx) => {
if (ctx.args.content.kind !== 'inline') {
throw new (await import('../../src/index.js')).ConflictError('streams not supported in this test handler');
}
vfs.set(ctx.args.path, {
kind: 'file',
bytes: ctx.args.content.bytes,
...(ctx.args.contentType !== undefined ? { contentType: ctx.args.contentType } : {}),
});
return {
entry: {
name: ctx.args.path.split('/').pop() ?? ctx.args.path,
kind: 'file' as const,
size: ctx.args.content.bytes.byteLength,
mtime: 0,
metadata: {},
...(ctx.args.contentType !== undefined ? { contentType: ctx.args.contentType } : {}),
},
};
},
},
});
try {
// list
const listed = await rig.fs.list('/');
expect(listed.entries.map((e) => e.name).sort()).toContain('photos');
// mkdir
await rig.fs.mkdir('/docs');
const stat = await rig.fs.stat('/docs');
expect(stat.kind).toBe('dir');
// write inline
const payload = new TextEncoder().encode('hello browser-friendly world');
const writeResult = await rig.fs.write('/docs/greeting.txt', payload, {
contentType: 'text/plain',
});
expect(writeResult.entry.size).toBe(payload.byteLength);
// read inline
const readResult = await rig.fs.read('/docs/greeting.txt');
expect(readResult.kind).toBe('inline');
if (readResult.kind === 'inline') {
expect(new TextDecoder().decode(readResult.bytes)).toBe('hello browser-friendly world');
expect(readResult.contentType).toBe('text/plain');
}
// delete
const del = await rig.fs.delete('/docs/greeting.txt');
expect(del.deletedCount).toBe(1);
// stat the deleted path → typed NotFoundError
const { NotFoundError } = await import('../../src/index.js');
await expect(rig.fs.stat('/docs/greeting.txt')).rejects.toBeInstanceOf(NotFoundError);
} finally {
await rig.teardown();
}
});
test('streamed write (> 256 KiB) is rejected with a clear error', async () => {
const rig = await setupHttpRig({
bobHandler: {
write: async () => ({
entry: { name: 'unused', kind: 'file' as const, size: 0, mtime: 0, metadata: {} },
}),
},
});
try {
const big = new Uint8Array(257 * 1024);
const { ConflictError } = await import('../../src/index.js');
await expect(rig.fs.write('/big.bin', big)).rejects.toBeInstanceOf(ConflictError);
} finally {
await rig.teardown();
}
});
test('rpcRoute() throws when no handler is attached', async () => {
// Don't call shade.files.serve(...) — rpcRoute() should refuse.
const prekeyEvents = new PrekeyServerEvents();
const prekey = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
events: prekeyEvents,
});
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
const bob = await createShade({
prekeyServer: `http://localhost:${prekeyServer.port}`,
address: 'bob',
});
try {
expect(() => bob.files.rpcRoute()).toThrow(/no handler attached/);
} finally {
await bob.shutdown();
prekeyServer.stop();
}
});
test('missing X-Shade-Sender-Address header → 400', async () => {
const rig = await setupHttpRig({
bobHandler: {
stat: async () => ({
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
}),
},
});
try {
const res = await fetch(rig.rpcUrl, {
method: 'POST',
body: new Uint8Array([0]),
});
expect(res.status).toBe(400);
const body = (await res.json()) as { error: string };
expect(body.error).toMatch(/X-Shade-Sender-Address/);
} finally {
await rig.teardown();
}
});
test('empty body → 400', async () => {
const rig = await setupHttpRig({
bobHandler: {
stat: async () => ({
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
}),
},
});
try {
const res = await fetch(rig.rpcUrl, {
method: 'POST',
headers: { 'X-Shade-Sender-Address': 'alice' },
});
expect(res.status).toBe(400);
} finally {
await rig.teardown();
}
});
test('garbage body → 401 decrypt failure', async () => {
const rig = await setupHttpRig({
bobHandler: {
stat: async () => ({
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
}),
},
});
try {
const res = await fetch(rig.rpcUrl, {
method: 'POST',
headers: { 'X-Shade-Sender-Address': 'alice' },
body: new Uint8Array([0x02, 0xff, 0xff, 0xff]),
});
// 400 from envelope decode failure or 401 from decrypt failure.
expect([400, 401]).toContain(res.status);
} finally {
await rig.teardown();
}
});
test('body past maxBodyBytes → 413', async () => {
const prekeyEvents = new PrekeyServerEvents();
const prekey = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
events: prekeyEvents,
});
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
const bob = await createShade({
prekeyServer: `http://localhost:${prekeyServer.port}`,
address: 'bob',
});
await bob.files.serve(
{
stat: async () => ({
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
}),
},
{ inlineOnly: true },
);
const route = bob.files.rpcRoute({ maxBodyBytes: 1024 });
const server = Bun.serve({ port: 0, fetch: route.fetch });
try {
const big = new Uint8Array(2048);
const res = await fetch(`http://localhost:${server.port}/rpc`, {
method: 'POST',
headers: { 'X-Shade-Sender-Address': 'alice' },
body: big,
});
expect(res.status).toBe(413);
} finally {
await bob.shutdown();
server.stop();
prekeyServer.stop();
}
});
});

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/inbox-server", "name": "@shade/inbox-server",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -38,8 +38,15 @@ import {
import { verifyPayload, validateAddress } from '@shade/server'; import { verifyPayload, validateAddress } from '@shade/server';
import type { InboxStore } from './store.js'; import type { InboxStore } from './store.js';
import type { InboxServerEvents } from './events.js'; import type { InboxServerEvents } from './events.js';
import { PresenceTracker, type TrackedBridgeKind } from './presence.js';
export type BridgeKind = 'stream' | 'poll' | 'ws'; export type BridgeKind = 'stream' | 'poll' | 'ws';
/**
* Wire-protocol kind tag for `/v1/bridge/presence`. Distinct from
* `BridgeKind` because the canonical signed payload is shaped
* differently (`watched: string[]` instead of `since: number`).
*/
export type PresenceKind = 'presence';
export interface BridgeRoutesOptions { export interface BridgeRoutesOptions {
store: InboxStore; store: InboxStore;
@@ -60,6 +67,13 @@ export interface BridgeRoutesOptions {
* Default 1_000. * Default 1_000.
*/ */
fallbackPollIntervalMs?: number; fallbackPollIntervalMs?: number;
/**
* Inject an existing presence tracker. Useful when multiple
* `createBridgeRoutes` calls need to share state (e.g. mounting the
* routes under several hostnames in a single process). When omitted,
* the bridge auto-creates an internal tracker bound to `events`.
*/
presenceTracker?: PresenceTracker;
} }
interface VerifiedBridgeRequest { interface VerifiedBridgeRequest {
@@ -68,6 +82,13 @@ interface VerifiedBridgeRequest {
since: number; since: number;
} }
interface VerifiedPresenceRequest {
/** The watcher's address (signer of the request). */
address: string;
/** Addresses whose presence the watcher is asking to track. */
watched: string[];
}
/** /**
* Build the bridge Hono router and a paired Bun-WebSocket handler. * Build the bridge Hono router and a paired Bun-WebSocket handler.
* *
@@ -80,6 +101,8 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
app: Hono; app: Hono;
/** Pass to `Bun.serve({ websocket })`. Undefined if Bun adapter is missing. */ /** Pass to `Bun.serve({ websocket })`. Undefined if Bun adapter is missing. */
websocket: unknown; websocket: unknown;
/** Live presence tracker. Tests + observers can read it; routes update it. */
presence: PresenceTracker;
} { } {
const app = new Hono(); const app = new Hono();
const pageLimit = opts.pageLimit ?? 50; const pageLimit = opts.pageLimit ?? 50;
@@ -87,6 +110,7 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
const longPollDefault = opts.longPollTimeoutMs ?? 25_000; const longPollDefault = opts.longPollTimeoutMs ?? 25_000;
const longPollMax = opts.longPollMaxTimeoutMs ?? 55_000; const longPollMax = opts.longPollMaxTimeoutMs ?? 55_000;
const fallbackPollIntervalMs = opts.fallbackPollIntervalMs ?? 1_000; const fallbackPollIntervalMs = opts.fallbackPollIntervalMs ?? 1_000;
const presence = opts.presenceTracker ?? new PresenceTracker(opts.events ?? null);
app.onError((err, c) => { app.onError((err, c) => {
if (err instanceof ShadeError) { if (err instanceof ShadeError) {
@@ -102,17 +126,32 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
const verified = await verifyBridgeAuth(c, opts, 'stream'); const verified = await verifyBridgeAuth(c, opts, 'stream');
return streamSSE(c, async (stream) => { return streamSSE(c, async (stream) => {
const address = verified.address; const address = verified.address;
const connId = presence.newConnectionId();
presence.markConnected(address, 'sse', connId);
let presenceClosed = false;
const closePresence = (reason: 'closed' | 'error'): void => {
if (presenceClosed) return;
presenceClosed = true;
presence.markDisconnected(address, 'sse', connId, reason);
};
let cursor = verified.since; let cursor = verified.since;
const writer = makeBlobWriter(opts.store, pageLimit); const writer = makeBlobWriter(opts.store, pageLimit);
const delivered = new DeliveredIdLru();
// Initial backlog drain. // Initial backlog drain.
const flushed = await flushTo(writer, address, cursor, async (blob) => { const flushed = await flushTo(
writer,
address,
cursor,
async (blob) => {
await stream.writeSSE({ await stream.writeSSE({
id: String(blob.receivedAt), id: String(blob.receivedAt),
event: 'envelope', event: 'envelope',
data: JSON.stringify(serializeBlob(blob)), data: JSON.stringify(serializeBlob(blob)),
}); });
}); },
delivered,
);
cursor = Math.max(cursor, flushed); cursor = Math.max(cursor, flushed);
// Hook up event-driven push if available, else fall back to a poll // Hook up event-driven push if available, else fall back to a poll
@@ -124,19 +163,31 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
const triggerFlush = (): void => { const triggerFlush = (): void => {
signalled = true; signalled = true;
// Serialize fan-in so concurrent triggers don't double-fetch. // Serialize fan-in so concurrent triggers don't double-fetch.
pendingFlushPromise = pendingFlushPromise.then(async () => { // `.catch(() => {})` keeps the chain alive across transient
// emit failures (e.g. a closed SSE write throws) — without it
// one rejection silently kills every future flush on this
// connection.
pendingFlushPromise = pendingFlushPromise
.then(async () => {
while (signalled) { while (signalled) {
signalled = false; signalled = false;
const drained = await flushTo(writer, address, cursor, async (blob) => { const drained = await flushTo(
writer,
address,
cursor,
async (blob) => {
await stream.writeSSE({ await stream.writeSSE({
id: String(blob.receivedAt), id: String(blob.receivedAt),
event: 'envelope', event: 'envelope',
data: JSON.stringify(serializeBlob(blob)), data: JSON.stringify(serializeBlob(blob)),
}); });
}); },
delivered,
);
if (drained > cursor) cursor = drained; if (drained > cursor) cursor = drained;
} }
}); })
.catch(() => {});
}; };
if (opts.events) { if (opts.events) {
@@ -163,6 +214,68 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
clearInterval(fallbackTimer); clearInterval(fallbackTimer);
clearInterval(heartbeat); clearInterval(heartbeat);
await pendingFlushPromise.catch(() => {}); await pendingFlushPromise.catch(() => {});
closePresence('closed');
});
});
// ─── Presence (V4.7) ──────────────────────────────────────────
// SSE feed of `peer_connected` / `peer_disconnected` events filtered
// by a watcher-supplied address list. Subscribing does NOT count as
// a peer-bridge connection (it doesn't call `markConnected`) so
// monitoring presence doesn't make you appear online to others.
app.get('/v1/bridge/presence', async (c) => {
const verified = await verifyPresenceAuth(c, opts);
return streamSSE(c, async (stream) => {
const watched = new Set(verified.watched);
// Initial snapshot — one frame per watched address with current
// status. Lets subscribers render UI immediately rather than
// waiting for the next state change.
const now = Date.now();
for (const addr of verified.watched) {
await stream.writeSSE({
event: 'presence',
data: JSON.stringify({
address: addr,
status: presence.isOnline(addr) ? 'online' : 'offline',
at: now,
}),
});
}
let unsubscribe: (() => void) | null = null;
if (opts.events) {
unsubscribe = opts.events.on((e) => {
if (e.name !== 'inbox.peer_connected' && e.name !== 'inbox.peer_disconnected') return;
const data = e.data as { address: string; bridgeKind: TrackedBridgeKind };
if (!watched.has(data.address)) return;
const status = e.name === 'inbox.peer_connected' ? 'online' : 'offline';
// Fire-and-forget: drop the frame if the stream has gone away.
void stream
.writeSSE({
event: 'presence',
data: JSON.stringify({
address: data.address,
status,
at: e.timestamp,
via: data.bridgeKind,
}),
})
.catch(() => {});
});
}
const heartbeat = setInterval(() => {
stream.write(`: ping ${Date.now()}\n\n`).catch(() => {});
}, heartbeatIntervalMs);
await new Promise<void>((resolve) => {
const sig = c.req.raw.signal;
if (sig.aborted) return resolve();
sig.addEventListener('abort', () => resolve(), { once: true });
});
unsubscribe?.();
clearInterval(heartbeat);
}); });
}); });
@@ -230,30 +343,50 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
} }
const address = verified.address; const address = verified.address;
const connId = presence.newConnectionId();
let cursor = verified.since; let cursor = verified.since;
const writer = makeBlobWriter(opts.store, pageLimit); const writer = makeBlobWriter(opts.store, pageLimit);
const delivered = new DeliveredIdLru();
let unsubscribe: (() => void) | null = null; let unsubscribe: (() => void) | null = null;
let fallbackTimer: ReturnType<typeof setInterval> | null = null; let fallbackTimer: ReturnType<typeof setInterval> | null = null;
let pendingFlushPromise: Promise<void> = Promise.resolve(); let pendingFlushPromise: Promise<void> = Promise.resolve();
let signalled = false; let signalled = false;
let connected = true; let connected = true;
let presenceClosed = false;
const closePresence = (reason: 'closed' | 'error'): void => {
if (presenceClosed) return;
presenceClosed = true;
presence.markDisconnected(address, 'ws', connId, reason);
};
return { return {
onOpen(_evt: unknown, ws: { onOpen(_evt: unknown, ws: {
send: (data: string) => void; send: (data: string) => void;
close: (code?: number, reason?: string) => void; close: (code?: number, reason?: string) => void;
}) { }) {
presence.markConnected(address, 'ws', connId);
const triggerFlush = (): void => { const triggerFlush = (): void => {
signalled = true; signalled = true;
pendingFlushPromise = pendingFlushPromise.then(async () => { // `.catch(() => {})` mirrors the SSE chain — keeps the
// pending-flush queue alive across transient ws.send errors
// (e.g. partial close, backpressure overflow).
pendingFlushPromise = pendingFlushPromise
.then(async () => {
while (signalled && connected) { while (signalled && connected) {
signalled = false; signalled = false;
const drained = await flushTo(writer, address, cursor, async (blob) => { const drained = await flushTo(
writer,
address,
cursor,
async (blob) => {
ws.send(JSON.stringify(serializeBlob(blob))); ws.send(JSON.stringify(serializeBlob(blob)));
}); },
delivered,
);
if (drained > cursor) cursor = drained; if (drained > cursor) cursor = drained;
} }
}); })
.catch(() => {});
}; };
if (opts.events) { if (opts.events) {
unsubscribe = opts.events.on((e) => { unsubscribe = opts.events.on((e) => {
@@ -269,12 +402,19 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
connected = false; connected = false;
unsubscribe?.(); unsubscribe?.();
if (fallbackTimer) clearInterval(fallbackTimer); if (fallbackTimer) clearInterval(fallbackTimer);
closePresence('closed');
},
onError() {
connected = false;
unsubscribe?.();
if (fallbackTimer) clearInterval(fallbackTimer);
closePresence('error');
}, },
}; };
}), }),
); );
return { app, websocket }; return { app, websocket, presence };
} }
// ─── helpers ────────────────────────────────────────────────── // ─── helpers ──────────────────────────────────────────────────
@@ -321,11 +461,75 @@ async function verifyBridgeAuth(
return { address, kind: kind as BridgeKind, since }; return { address, kind: kind as BridgeKind, since };
} }
const MAX_WATCHED_ADDRESSES = 64;
/**
* Verify a `/v1/bridge/presence` request.
*
* Signed canonical payload: `{address, kind: 'presence', watched: string[],
* signedAt}`. The watcher's address must be a registered inbox; the
* signature is verified against the registered owner key for that
* address. The `watched` list bounds what the subscription will
* receive — server-side filtering is enforced inside the handler.
*/
async function verifyPresenceAuth(
c: Context,
opts: BridgeRoutesOptions,
): Promise<VerifiedPresenceRequest> {
const url = new URL(c.req.url);
const qs = url.searchParams;
const address = validateAddress(qs.get('address'));
const kind = qs.get('kind');
if (kind !== 'presence') {
throw new ValidationError(`bridge kind mismatch: expected presence`, 'kind');
}
const watchedRaw = qs.get('watched');
if (watchedRaw === null) throw new ValidationError('missing watched', 'watched');
// Empty subscription is allowed (subscribe to nothing — useful for a
// client that intends to call addPeer right after open). A null/
// missing param is still rejected so the canonicalization is
// unambiguous.
const watched =
watchedRaw === ''
? []
: watchedRaw.split(',').map((a) => validateAddress(a));
if (watched.length > MAX_WATCHED_ADDRESSES) {
throw new ValidationError(
`watched list too large: ${watched.length} > ${MAX_WATCHED_ADDRESSES}`,
'watched',
);
}
const signedAtStr = qs.get('signedAt');
const signature = qs.get('signature');
if (signedAtStr === null) throw new ValidationError('missing signedAt', 'signedAt');
if (!signature) throw new ValidationError('missing signature', 'signature');
const signedAt = Number(signedAtStr);
if (!Number.isFinite(signedAt)) {
throw new ValidationError('signedAt must be a number', 'signedAt');
}
const owner = await opts.store.getAddressOwner(address);
if (!owner) {
throw new UnauthorizedError(`address ${address} is not registered`);
}
await verifyPayload(opts.crypto, owner, {
address,
kind,
watched,
signedAt,
signature,
});
return { address, watched };
}
interface BlobRow { interface BlobRow {
msgId: string; msgId: string;
ciphertext: Uint8Array; ciphertext: Uint8Array;
receivedAt: number; receivedAt: number;
expiresAt: number; expiresAt: number;
/** V4.8 — relay-captured sender fingerprint. Optional for legacy rows. */
senderFp?: string;
} }
interface BlobWriter { interface BlobWriter {
@@ -345,11 +549,41 @@ function makeBlobWriter(store: InboxStore, pageLimit: number): BlobWriter {
}; };
} }
/**
* Per-connection bounded msgId tracker — defense in depth against duplicate
* delivery of the same blob to the same bridge socket. Cursor pagination
* already guarantees uniqueness in the happy path, but a dedup gate at the
* emit boundary catches any subtle bug (e.g. a flushTo race, a future
* refactor, an event-emit retry) without changing wire semantics.
*
* The cap is intentionally large enough to cover any realistic bridge
* pageLimit and small enough to bound memory under long-running streams.
*/
const DELIVERED_LRU_CAP = 4096;
class DeliveredIdLru {
private readonly seen = new Set<string>();
private readonly order: string[] = [];
/** Returns true if `msgId` has not been seen on this connection yet. */
add(msgId: string): boolean {
if (this.seen.has(msgId)) return false;
this.seen.add(msgId);
this.order.push(msgId);
if (this.order.length > DELIVERED_LRU_CAP) {
const evicted = this.order.shift()!;
this.seen.delete(evicted);
}
return true;
}
}
async function flushTo( async function flushTo(
writer: BlobWriter, writer: BlobWriter,
address: string, address: string,
startCursor: number, startCursor: number,
emit: (blob: BlobRow) => Promise<void>, emit: (blob: BlobRow) => Promise<void>,
delivered?: DeliveredIdLru,
): Promise<number> { ): Promise<number> {
let cursor = startCursor; let cursor = startCursor;
// Drain page-by-page so a backlog larger than `pageLimit` still flushes. // Drain page-by-page so a backlog larger than `pageLimit` still flushes.
@@ -358,7 +592,12 @@ async function flushTo(
const page = await writer.fetchPage(address, cursor); const page = await writer.fetchPage(address, cursor);
if (page.length === 0) break; if (page.length === 0) break;
for (const row of page) { for (const row of page) {
// Per-connection dedup gate — prevents the same msgId from being
// emitted twice if flushTo is somehow re-entered before the cursor
// catches up. See comment on `DeliveredIdLru`.
if (!delivered || delivered.add(row.msgId)) {
await emit(row); await emit(row);
}
if (row.receivedAt > cursor) cursor = row.receivedAt; if (row.receivedAt > cursor) cursor = row.receivedAt;
} }
if (page.length === 0) break; if (page.length === 0) break;
@@ -371,13 +610,26 @@ function serializeBlob(blob: BlobRow): {
ciphertext: string; ciphertext: string;
receivedAt: number; receivedAt: number;
expiresAt: number; expiresAt: number;
from?: string;
} { } {
return { const out: {
msgId: string;
ciphertext: string;
receivedAt: number;
expiresAt: number;
from?: string;
} = {
msgId: blob.msgId, msgId: blob.msgId,
ciphertext: toBase64(blob.ciphertext), ciphertext: toBase64(blob.ciphertext),
receivedAt: blob.receivedAt, receivedAt: blob.receivedAt,
expiresAt: blob.expiresAt, expiresAt: blob.expiresAt,
}; };
// V4.8 — relay-captured sender fingerprint. The transport-bridge
// wire format already accepts `from`; populating it lets receivers
// bootstrap unknown-sender first-contact via `shade.receive('fp:<hex>',
// env)` without requiring an out-of-band sender hint.
if (blob.senderFp) out.from = blob.senderFp;
return out;
} }
function buildPollResponse(blobs: BlobRow[], sinceFallback: number): { function buildPollResponse(blobs: BlobRow[], sinceFallback: number): {

View File

@@ -21,6 +21,19 @@ export interface InboxServerEventMap {
'inbox.expired_purged': { count: number }; 'inbox.expired_purged': { count: number };
'inbox.rate_limited': { route: string; key: string }; 'inbox.rate_limited': { route: string; key: string };
'inbox.quota_rejected': { address: string; reason: 'address-quota' | 'sender-quota' | 'body-too-large' }; 'inbox.quota_rejected': { address: string; reason: 'address-quota' | 'sender-quota' | 'body-too-large' };
// V4.7 — bridge presence transitions. Emitted on the 0↔1 boundary
// across tracked transports for a given address. Long-poll is
// intentionally NOT tracked: an LP client toggles in/out of a request
// every few seconds, and the resulting flapping would dominate the
// event stream. Push transports (WS, SSE) are also the only ones
// where the ~50ms revoke window for `BroadcastChannel.removeMember`
// matters — long-poll users are already on a slow path.
'inbox.peer_connected': { address: string; bridgeKind: 'ws' | 'sse' };
'inbox.peer_disconnected': {
address: string;
bridgeKind: 'ws' | 'sse';
reason: 'closed' | 'error';
};
} }
export type InboxServerEventName = keyof InboxServerEventMap; export type InboxServerEventName = keyof InboxServerEventMap;

View File

@@ -32,6 +32,8 @@ export {
export type { InboxQuotaConfig } from './quota.js'; export type { InboxQuotaConfig } from './quota.js';
export { createBridgeRoutes } from './bridge.js'; export { createBridgeRoutes } from './bridge.js';
export type { BridgeRoutesOptions, BridgeKind } from './bridge.js'; export type { BridgeRoutesOptions, BridgeKind } from './bridge.js';
export { PresenceTracker } from './presence.js';
export type { TrackedBridgeKind } from './presence.js';
/** /**
* Create a standalone Shade Inbox Server. * Create a standalone Shade Inbox Server.

View File

@@ -5,6 +5,7 @@ interface BlobRow {
ciphertext: Uint8Array; ciphertext: Uint8Array;
receivedAt: number; receivedAt: number;
expiresAt: number; expiresAt: number;
senderFp?: string;
} }
/** /**
@@ -33,6 +34,7 @@ export class MemoryInboxStore implements InboxStore {
msgId: string; msgId: string;
ciphertext: Uint8Array; ciphertext: Uint8Array;
expiresAt: number; expiresAt: number;
senderFp?: string;
}): Promise<{ created: boolean; receivedAt: number }> { }): Promise<{ created: boolean; receivedAt: number }> {
const list = this.blobs.get(args.address) ?? []; const list = this.blobs.get(args.address) ?? [];
const existing = list.find((r) => r.msgId === args.msgId); const existing = list.find((r) => r.msgId === args.msgId);
@@ -41,12 +43,14 @@ export class MemoryInboxStore implements InboxStore {
// multiple blobs land in the same millisecond. // multiple blobs land in the same millisecond.
const receivedAt = Math.max(this.nextReceivedAt + 1, Date.now()); const receivedAt = Math.max(this.nextReceivedAt + 1, Date.now());
this.nextReceivedAt = receivedAt; this.nextReceivedAt = receivedAt;
list.push({ const row: BlobRow = {
msgId: args.msgId, msgId: args.msgId,
ciphertext: new Uint8Array(args.ciphertext), ciphertext: new Uint8Array(args.ciphertext),
receivedAt, receivedAt,
expiresAt: args.expiresAt, expiresAt: args.expiresAt,
}); };
if (args.senderFp !== undefined) row.senderFp = args.senderFp;
list.push(row);
this.blobs.set(args.address, list); this.blobs.set(args.address, list);
return { created: true, receivedAt }; return { created: true, receivedAt };
} }
@@ -56,18 +60,36 @@ export class MemoryInboxStore implements InboxStore {
sinceCursor: number; sinceCursor: number;
now: number; now: number;
limit: number; limit: number;
}): Promise<Array<{ msgId: string; ciphertext: Uint8Array; receivedAt: number; expiresAt: number }>> { }): Promise<
Array<{
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
senderFp?: string;
}>
> {
const list = this.blobs.get(args.address) ?? []; const list = this.blobs.get(args.address) ?? [];
return list return list
.filter((r) => r.receivedAt > args.sinceCursor && r.expiresAt > args.now) .filter((r) => r.receivedAt > args.sinceCursor && r.expiresAt > args.now)
.sort((a, b) => a.receivedAt - b.receivedAt) .sort((a, b) => a.receivedAt - b.receivedAt)
.slice(0, args.limit) .slice(0, args.limit)
.map((r) => ({ .map((r) => {
const out: {
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
senderFp?: string;
} = {
msgId: r.msgId, msgId: r.msgId,
ciphertext: new Uint8Array(r.ciphertext), ciphertext: new Uint8Array(r.ciphertext),
receivedAt: r.receivedAt, receivedAt: r.receivedAt,
expiresAt: r.expiresAt, expiresAt: r.expiresAt,
})); };
if (r.senderFp !== undefined) out.senderFp = r.senderFp;
return out;
});
} }
async deleteBlob(address: string, msgId: string): Promise<boolean> { async deleteBlob(address: string, msgId: string): Promise<boolean> {

View File

@@ -0,0 +1,75 @@
/**
* V4.7 — bridge-connection presence tracking.
*
* The bridge handlers (`/v1/bridge/stream` and `/v1/bridge/ws`) call
* `markConnected` on open and `markDisconnected` on close. The tracker
* keeps a per-address set of connection ids; the `inbox.peer_connected`
* / `inbox.peer_disconnected` events fire only on the 0↔1 boundary so
* that two simultaneous bridges (e.g. SSE + WS during a transport-
* fallback handover) collapse into a single connected/disconnected
* pair from the consumer's point of view.
*
* Long-poll (`/v1/bridge/poll`) is intentionally NOT tracked — see the
* note on `InboxServerEventMap` in `events.ts`.
*/
import type { InboxServerEvents } from './events.js';
export type TrackedBridgeKind = 'ws' | 'sse';
export class PresenceTracker {
private readonly connections = new Map<string, Set<string>>();
private nextConnId = 1;
constructor(private readonly events: InboxServerEvents | null) {}
/** Allocate a fresh connection id for `markConnected` / `markDisconnected`. */
newConnectionId(): string {
return `c${this.nextConnId++}`;
}
/**
* Snapshot: is `address` currently connected over any tracked transport?
* Used by `/v1/bridge/presence` to push the initial state to a new
* subscriber.
*/
isOnline(address: string): boolean {
const set = this.connections.get(address);
return set !== undefined && set.size > 0;
}
markConnected(address: string, bridgeKind: TrackedBridgeKind, connectionId: string): void {
let set = this.connections.get(address);
const wasOnline = set !== undefined && set.size > 0;
if (!set) {
set = new Set();
this.connections.set(address, set);
}
set.add(connectionId);
if (!wasOnline) {
this.events?.emit('inbox.peer_connected', { address, bridgeKind });
}
}
markDisconnected(
address: string,
bridgeKind: TrackedBridgeKind,
connectionId: string,
reason: 'closed' | 'error',
): void {
const set = this.connections.get(address);
if (!set) return;
if (!set.delete(connectionId)) return;
if (set.size === 0) {
this.connections.delete(address);
this.events?.emit('inbox.peer_disconnected', { address, bridgeKind, reason });
}
}
/** Inspect the underlying map. Test/observability use only. */
snapshot(): Map<string, ReadonlySet<string>> {
return new Map(
Array.from(this.connections.entries(), ([k, v]) => [k, v as ReadonlySet<string>]),
);
}
}

View File

@@ -255,11 +255,19 @@ export function createInboxRoutes(
); );
} }
// V4.8: capture sender fingerprint at PUT time. The sender's
// signing key was just verified for this request, so the fingerprint
// is bound to the same authentication path that authorized the
// store. Surfaced on bridge push + inbox-fetch responses to
// bootstrap unknown-sender first-contact (X3DH pair handshake).
const senderFp = await shortHash(senderKey);
const result = await store.putBlob({ const result = await store.putBlob({
address, address,
msgId, msgId,
ciphertext: ctBytes, ciphertext: ctBytes,
expiresAt, expiresAt,
senderFp,
}); });
if (result.created) { if (result.created) {
events?.emit('inbox.blob_stored', { events?.emit('inbox.blob_stored', {
@@ -319,12 +327,22 @@ export function createInboxRoutes(
let bytes = 0; let bytes = 0;
const blobs = rows.map((r) => { const blobs = rows.map((r) => {
bytes += r.ciphertext.length; bytes += r.ciphertext.length;
return { const out: {
msgId: string;
ciphertext: string;
receivedAt: number;
expiresAt: number;
from?: string;
} = {
msgId: r.msgId, msgId: r.msgId,
ciphertext: toBase64(r.ciphertext), ciphertext: toBase64(r.ciphertext),
receivedAt: r.receivedAt, receivedAt: r.receivedAt,
expiresAt: r.expiresAt, expiresAt: r.expiresAt,
}; };
// V4.8: surface sender fingerprint when present. Empty for blobs
// persisted by a pre-4.8 relay that didn't track sender provenance.
if (r.senderFp) out.from = r.senderFp;
return out;
}); });
const nextCursor = rows.length > 0 ? rows[rows.length - 1]!.receivedAt : sinceCursor; const nextCursor = rows.length > 0 ? rows[rows.length - 1]!.receivedAt : sinceCursor;

View File

@@ -36,12 +36,20 @@ export interface InboxStore {
* **Idempotent**: if a row already exists for `(address, msgId)` the * **Idempotent**: if a row already exists for `(address, msgId)` the
* implementation MUST return `{ created: false }` and leave the existing * implementation MUST return `{ created: false }` and leave the existing
* row untouched. A fresh insert returns `{ created: true, receivedAt }`. * row untouched. A fresh insert returns `{ created: true, receivedAt }`.
*
* `senderFp` (V4.8+) is the 8-byte hex of SHA-256(senderSigningKey)
* — captured at PUT time when the relay verified the sender's
* signature. Optional so legacy 4.7 callers compile, but populated by
* `createInboxRoutes` from 4.8 onward and surfaced on bridge push +
* inbox-fetch responses to bootstrap unknown-sender first-contact
* (X3DH pair handshake).
*/ */
putBlob(args: { putBlob(args: {
address: string; address: string;
msgId: string; msgId: string;
ciphertext: Uint8Array; ciphertext: Uint8Array;
expiresAt: number; expiresAt: number;
senderFp?: string;
}): Promise<{ created: boolean; receivedAt: number }>; }): Promise<{ created: boolean; receivedAt: number }>;
/** /**
@@ -57,7 +65,20 @@ export interface InboxStore {
sinceCursor: number; sinceCursor: number;
now: number; now: number;
limit: number; limit: number;
}): Promise<Array<{ msgId: string; ciphertext: Uint8Array; receivedAt: number; expiresAt: number }>>; }): Promise<
Array<{
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
/**
* Sender fingerprint — 8-byte hex of SHA-256(senderSigningKey)
* captured at PUT time. Empty/undefined for blobs persisted by a
* pre-4.8 relay that didn't track sender provenance.
*/
senderFp?: string;
}>
>;
/** /**
* Delete a single blob by `(address, msgId)`. Returns true if a row was * Delete a single blob by `(address, msgId)`. Returns true if a row was

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/inbox", "name": "@shade/inbox",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -40,6 +40,16 @@ export interface FetchedBlob {
receivedAt: number; receivedAt: number;
/** Absolute expiry time (ms since epoch) reported by the server. */ /** Absolute expiry time (ms since epoch) reported by the server. */
expiresAt: number; expiresAt: number;
/**
* Sender fingerprint — 8-byte hex of SHA-256(senderSigningKey),
* captured by the relay at PUT time when the sender's signature was
* verified. Empty/undefined when the relay is pre-4.8 or the blob
* predates sender-fingerprint tracking. Use as an unknown-sender
* bootstrap label (`fp:<hex>`) for X3DH first-contact; the
* authoritative sender identity is recovered post-decrypt from the
* envelope itself, so `from` is a hint, not a trust anchor.
*/
from?: string;
} }
export interface FetchResult { export interface FetchResult {
@@ -52,7 +62,14 @@ export class InboxClient {
private readonly fetchImpl: typeof fetch; private readonly fetchImpl: typeof fetch;
constructor(private readonly options: InboxClientOptions) { constructor(private readonly options: InboxClientOptions) {
this.fetchImpl = options.fetch ?? globalThis.fetch; // Bind once. The browser's `globalThis.fetch` is a WebIDL bound
// operation that throws "Illegal invocation" when called as a method
// on another object (which is what `this.fetchImpl(...)` does).
// Node/Bun fetch tolerates a free receiver, but binding is harmless.
// A consumer-supplied `options.fetch` is bound to the global too —
// a fetch that requires a specific receiver must bind itself.
const f = options.fetch ?? globalThis.fetch;
this.fetchImpl = f.bind(globalThis);
} }
/** /**
@@ -144,12 +161,18 @@ export class InboxClient {
); );
const blobs = Array.isArray(json.blobs) ? json.blobs : []; const blobs = Array.isArray(json.blobs) ? json.blobs : [];
return { return {
blobs: blobs.map((b: any) => ({ blobs: blobs.map((b: any): FetchedBlob => {
const out: FetchedBlob = {
msgId: String(b.msgId), msgId: String(b.msgId),
ciphertext: fromBase64(String(b.ciphertext)), ciphertext: fromBase64(String(b.ciphertext)),
receivedAt: Number(b.receivedAt), receivedAt: Number(b.receivedAt),
expiresAt: Number(b.expiresAt), expiresAt: Number(b.expiresAt),
})), };
// V4.8 — relay-supplied sender fingerprint hint. Optional; absent
// on pre-4.8 relays or for blobs persisted before tracking landed.
if (typeof b.from === 'string' && b.from.length > 0) out.from = b.from;
return out;
}),
cursor: Number(json.cursor ?? sinceCursor), cursor: Number(json.cursor ?? sinceCursor),
hasMore: Boolean(json.hasMore), hasMore: Boolean(json.hasMore),
}; };

View File

@@ -16,9 +16,20 @@ import { InboxClientEvents, type InboxClientListener } from './events.js';
* decrypt path it owns) and either return a sender-hint for telemetry * decrypt path it owns) and either return a sender-hint for telemetry
* (the address the SDK extracted, or `null`) or throw to keep the blob * (the address the SDK extracted, or `null`) or throw to keep the blob
* on the server for a later retry. * on the server for a later retry.
*
* V4.8: `raw.from` is the relay-captured sender fingerprint (8-byte hex
* of SHA-256 over the sender's signing key). Empty when the relay is
* pre-4.8 or didn't track the sender. Use it as the `fp:<hex>` bootstrap
* label for X3DH first-contact handshakes.
*/ */
export type DecryptHandler = ( export type DecryptHandler = (
raw: { msgId: string; ciphertext: Uint8Array; receivedAt: number; expiresAt: number }, raw: {
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
from?: string;
},
) => Promise<string | null | undefined> | string | null | undefined; ) => Promise<string | null | undefined> | string | null | undefined;
export interface InboxOptions { export interface InboxOptions {
@@ -149,6 +160,14 @@ export class Inbox {
signingKey: this.options.signingPublicKey, signingKey: this.options.signingPublicKey,
}); });
this.registered = true; this.registered = true;
// V4.8: gate the first poll on register success. `start()` calls
// `register()` fire-and-forget; without this kick, the very first
// `pollOnce()` (scheduled synchronously alongside register) would
// race the register HTTP RTT and return SHADE_NOT_FOUND. The
// pollOnce() guard skips polls until `registered === true`; this
// immediate schedule ensures we don't wait the full pollIntervalMs
// for the next attempt once register lands.
if (this.started) this.schedulePoll(0);
} }
/** Drop the address from the server. Local queue/cursor are preserved. */ /** Drop the address from the server. Local queue/cursor are preserved. */
@@ -217,7 +236,11 @@ export class Inbox {
this.scheduleRegisterRetry(); this.scheduleRegisterRetry();
}); });
this.scheduleFlush(); this.scheduleFlush();
this.schedulePoll(0); // V4.8: do NOT schedule the first poll synchronously. `register()`
// success kicks `schedulePoll(0)` so the first poll fires after the
// server has acknowledged the address. Pre-fix this raced register
// and burned a 404 on every fresh-address `start()`.
if (this.registered) this.schedulePoll(0);
} }
/** Stop background timers. Pending entries remain in the queue. */ /** Stop background timers. Pending entries remain in the queue. */
@@ -375,12 +398,20 @@ export class Inbox {
let senderHint: string | null = null; let senderHint: string | null = null;
try { try {
const result = await this.incomingHandler({ const raw: {
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
from?: string;
} = {
msgId: blob.msgId, msgId: blob.msgId,
ciphertext: blob.ciphertext, ciphertext: blob.ciphertext,
receivedAt: blob.receivedAt, receivedAt: blob.receivedAt,
expiresAt: blob.expiresAt, expiresAt: blob.expiresAt,
}); };
if (blob.from !== undefined) raw.from = blob.from;
const result = await this.incomingHandler(raw);
senderHint = result ?? null; senderHint = result ?? null;
} catch (err) { } catch (err) {
this.events.emit('inbox.message_decrypt_failed', { this.events.emit('inbox.message_decrypt_failed', {

View File

@@ -281,3 +281,195 @@ describe('tamper detection', () => {
expect(result.received).toBe(0); expect(result.received).toBe(0);
}); });
}); });
describe('InboxClient — default fetch is bound to globalThis', () => {
// Regression: browsers' `fetch` is a WebIDL bound operation that throws
// "Illegal invocation" when called as a method on another object. The
// class stores `fetchImpl` and calls `this.fetchImpl(...)`, which strips
// the Window receiver. Constructor must `bind(globalThis)`.
test('default path passes globalThis as `this` (no Illegal invocation)', async () => {
const realFetch = globalThis.fetch;
let observedReceiver: unknown = 'unset';
function strictFetch(this: unknown, _input: unknown, _init?: unknown): Promise<Response> {
observedReceiver = this;
if (this !== globalThis) {
throw new TypeError("Failed to execute 'fetch' on 'Window': Illegal invocation");
}
return Promise.resolve(
new Response('{}', {
status: 200,
headers: { 'content-type': 'application/json' },
}),
);
}
Object.defineProperty(globalThis, 'fetch', {
configurable: true,
writable: true,
value: strictFetch,
});
try {
const id = await makeIdentity();
const client = new InboxClient({
baseUrl: 'http://example.invalid',
crypto,
signingPrivateKey: id.signingPrivateKey,
// No `fetch` override on purpose — this exercises the default path.
});
await client.register({ address: 'whoever', signingKey: id.signingPublicKey });
expect(observedReceiver).toBe(globalThis);
} finally {
Object.defineProperty(globalThis, 'fetch', {
configurable: true,
writable: true,
value: realFetch,
});
}
});
});
describe('Inbox.start() — fresh-address register/poll race (V4.8)', () => {
// Regression: pre-4.8 `start()` called `register()` fire-and-forget AND
// `schedulePoll(0)` synchronously, so the first poll often beat the
// register HTTP RTT and got SHADE_NOT_FOUND on a fresh address. Fix:
// start() defers the first poll; register() success kicks it.
test('fresh address: no fetch fires before register completes', async () => {
const store = new MemoryInboxStore();
const app = createInboxServer({ crypto, store, disableRateLimit: true });
const alice = await makeIdentity();
// Order observed by the server: must be register-then-fetch, never
// fetch-then-register.
const calls: Array<'register' | 'fetch' | 'put'> = [];
let registerArrived = false;
const recordingFetch: typeof fetch = (async (input, init) => {
const u =
typeof input === 'string'
? input
: input instanceof URL
? input.toString()
: (input as Request).url;
if (u.includes('/v1/inbox/register')) {
calls.push('register');
// Hold register for a tick to widen the race window.
await new Promise((r) => setTimeout(r, 25));
registerArrived = true;
} else if (u.endsWith('/fetch')) {
// Any fetch arriving before register is the race we're guarding
// against.
if (!registerArrived) {
throw new Error('fetch fired before register completed (race not fixed)');
}
calls.push('fetch');
} else if (u.includes('/v1/inbox/')) {
calls.push('put');
}
return honoFetch(app)(input, init);
}) as typeof fetch;
const inbox = new Inbox({
baseUrl: 'http://localhost',
ownAddress: 'alice',
crypto,
signingPrivateKey: alice.signingPrivateKey,
signingPublicKey: alice.signingPublicKey,
pollIntervalMs: 30_000, // Long enough that only register's kick triggers.
fetch: recordingFetch,
});
inbox.onIncoming(() => null);
inbox.start();
// Wait until register has completed and the success-kick poll lands.
await new Promise((r) => setTimeout(r, 100));
inbox.stop();
expect(calls[0]).toBe('register');
// First fetch (if any) must be after register.
const firstFetchIdx = calls.indexOf('fetch');
if (firstFetchIdx !== -1) {
expect(firstFetchIdx).toBeGreaterThan(calls.indexOf('register'));
}
});
});
describe('FetchedBlob.from — relay-supplied sender fingerprint (V4.8)', () => {
test('inbox-fetch response carries from = 8-byte hex of SHA-256(senderSigningKey)', async () => {
const store = new MemoryInboxStore();
const app = createInboxServer({ crypto, store, disableRateLimit: true });
const bob = await makeIdentity();
const alice = await makeIdentity();
const bobClient = new InboxClient({
baseUrl: 'http://localhost',
crypto,
signingPrivateKey: bob.signingPrivateKey,
fetch: honoFetch(app),
});
const aliceClient = new InboxClient({
baseUrl: 'http://localhost',
crypto,
signingPrivateKey: alice.signingPrivateKey,
fetch: honoFetch(app),
});
await bobClient.register({ address: 'bob', signingKey: bob.signingPublicKey });
await aliceClient.put({
recipientAddress: 'bob',
senderSigningKey: alice.signingPublicKey,
envelope: randBytes(64),
});
const fetched = await bobClient.fetch({ address: 'bob' });
expect(fetched.blobs.length).toBe(1);
const fp = fetched.blobs[0]!.from;
expect(fp).toBeDefined();
expect(fp).toMatch(/^[0-9a-f]{16}$/);
// Must be reproducible: SHA-256(alice.signingPublicKey) → first 8 bytes hex.
const digest = await globalThis.crypto.subtle.digest(
'SHA-256',
alice.signingPublicKey as unknown as ArrayBuffer,
);
const expected = Array.from(new Uint8Array(digest).slice(0, 8), (b) =>
b.toString(16).padStart(2, '0'),
).join('');
expect(fp).toBe(expected);
});
test('DecryptHandler raw arg propagates from to the app', async () => {
const store = new MemoryInboxStore();
const app = createInboxServer({ crypto, store, disableRateLimit: true });
const bob = await makeIdentity();
const alice = await makeIdentity();
const aliceClient = new InboxClient({
baseUrl: 'http://localhost',
crypto,
signingPrivateKey: alice.signingPrivateKey,
fetch: honoFetch(app),
});
const bobInbox = new Inbox({
baseUrl: 'http://localhost',
ownAddress: 'bob',
crypto,
signingPrivateKey: bob.signingPrivateKey,
signingPublicKey: bob.signingPublicKey,
pollIntervalMs: 0,
fetch: honoFetch(app),
});
await bobInbox.register();
await aliceClient.put({
recipientAddress: 'bob',
senderSigningKey: alice.signingPublicKey,
envelope: randBytes(40),
});
let observed: string | undefined = undefined;
bobInbox.onIncoming((raw) => {
observed = raw.from;
return null;
});
await bobInbox.tick();
expect(observed).toMatch(/^[0-9a-f]{16}$/);
});
});

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/key-transparency", "name": "@shade/key-transparency",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/keychain", "name": "@shade/keychain",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/observability", "name": "@shade/observability",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/observer", "name": "@shade/observer",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/proto", "name": "@shade/proto",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -5,7 +5,10 @@ export {
encodeRatchetMessage, encodeRatchetMessage,
encodeStreamChunk, encodeStreamChunk,
decodeStreamChunk, decodeStreamChunk,
encodeBroadcast,
decodeBroadcast,
inspectEnvelopeType, inspectEnvelopeType,
TYPE_STREAM_CHUNK, TYPE_STREAM_CHUNK,
TYPE_BROADCAST,
} from './wire.js'; } from './wire.js';
export type { StreamChunkWire } from './wire.js'; export type { StreamChunkWire, BroadcastWire } from './wire.js';

View File

@@ -7,6 +7,7 @@
* 0x01 = PreKeyMessage * 0x01 = PreKeyMessage
* 0x02 = RatchetMessage * 0x02 = RatchetMessage
* 0x11 = StreamChunk * 0x11 = StreamChunk
* 0x21 = BroadcastMessage (V4.6 — sender-key encrypted group payload)
* *
* All multi-byte integers are big-endian. * All multi-byte integers are big-endian.
* *
@@ -23,6 +24,7 @@ const VERSION = 0x02;
const TYPE_PREKEY = 0x01; const TYPE_PREKEY = 0x01;
const TYPE_RATCHET = 0x02; const TYPE_RATCHET = 0x02;
export const TYPE_STREAM_CHUNK = 0x11; export const TYPE_STREAM_CHUNK = 0x11;
export const TYPE_BROADCAST = 0x21;
// ─── Stream chunk types ────────────────────────────────────── // ─── Stream chunk types ──────────────────────────────────────
@@ -43,6 +45,28 @@ export interface StreamChunkWire {
ciphertext: Uint8Array; // AES-GCM ciphertext including 16-byte tag ciphertext: Uint8Array; // AES-GCM ciphertext including 16-byte tag
} }
/**
* Wire-decoded broadcast envelope (type 0x21).
*
* Carries a Signal-style sender-key message. The sender (channel owner)
* encrypted once with their per-channel chain key; this same byte sequence
* is delivered verbatim to every member. Authenticity rides on the embedded
* Ed25519 signature — no bilateral ratchet wraps the broadcast itself.
*
* `generation` is a per-channel rotation counter: bumped each time a
* member is revoked. Receivers silently drop broadcasts at older
* generations than their currently-installed sender-key.
*/
export interface BroadcastWire {
channelId: string; // utf-8, length-prefixed
senderAddress: string; // utf-8, length-prefixed
generation: number; // u32
iteration: number; // u32 — sender chain counter
nonce: Uint8Array; // AES-GCM nonce (12 bytes)
signature: Uint8Array; // Ed25519 signature (64 bytes)
ciphertext: Uint8Array; // AES-GCM ciphertext including 16-byte tag
}
const STREAM_ID_BYTES = 16; const STREAM_ID_BYTES = 16;
const STREAM_NONCE_BYTES = 12; const STREAM_NONCE_BYTES = 12;
@@ -230,11 +254,11 @@ export function decodeStreamChunk(data: Uint8Array): StreamChunkWire {
/** /**
* Inspect the type tag of an arbitrary envelope without full parsing. * Inspect the type tag of an arbitrary envelope without full parsing.
* Returns one of `'prekey' | 'ratchet' | 'stream-chunk' | 'unknown'`. * Returns one of `'prekey' | 'ratchet' | 'stream-chunk' | 'broadcast' | 'unknown'`.
*/ */
export function inspectEnvelopeType( export function inspectEnvelopeType(
data: Uint8Array, data: Uint8Array,
): 'prekey' | 'ratchet' | 'stream-chunk' | 'unknown' { ): 'prekey' | 'ratchet' | 'stream-chunk' | 'broadcast' | 'unknown' {
if (data.length < 2 || data[0] !== VERSION) return 'unknown'; if (data.length < 2 || data[0] !== VERSION) return 'unknown';
switch (data[1]) { switch (data[1]) {
case TYPE_PREKEY: case TYPE_PREKEY:
@@ -243,11 +267,116 @@ export function inspectEnvelopeType(
return 'ratchet'; return 'ratchet';
case TYPE_STREAM_CHUNK: case TYPE_STREAM_CHUNK:
return 'stream-chunk'; return 'stream-chunk';
case TYPE_BROADCAST:
return 'broadcast';
default: default:
return 'unknown'; return 'unknown';
} }
} }
// ─── Broadcast wire (V4.6) ───────────────────────────────────
const BROADCAST_NONCE_BYTES = 12;
const BROADCAST_SIGNATURE_BYTES = 64;
/**
* Encode a broadcast envelope to wire bytes (type 0x21).
*
* Layout:
* [version:1][type=0x21:1]
* [channelIdLen:u16][channelId utf-8]
* [senderAddrLen:u16][senderAddr utf-8]
* [generation:u32]
* [iteration:u32]
* [nonce:12]
* [signature:64]
* [ctLen:u32][ciphertext]
*/
export function encodeBroadcast(b: BroadcastWire): Uint8Array {
if (b.nonce.length !== BROADCAST_NONCE_BYTES) {
throw new Error(`broadcast nonce must be ${BROADCAST_NONCE_BYTES} bytes`);
}
if (b.signature.length !== BROADCAST_SIGNATURE_BYTES) {
throw new Error(`broadcast signature must be ${BROADCAST_SIGNATURE_BYTES} bytes`);
}
const enc = new TextEncoder();
const channelIdBytes = enc.encode(b.channelId);
const senderBytes = enc.encode(b.senderAddress);
if (channelIdBytes.length > 0xffff) throw new Error('channelId too long');
if (senderBytes.length > 0xffff) throw new Error('senderAddress too long');
const headerSize =
1 + 1 +
2 + channelIdBytes.length +
2 + senderBytes.length +
4 + 4 +
BROADCAST_NONCE_BYTES +
BROADCAST_SIGNATURE_BYTES +
4;
const out = new Uint8Array(headerSize + b.ciphertext.length);
const view = new DataView(out.buffer);
let offset = 0;
out[offset++] = VERSION;
out[offset++] = TYPE_BROADCAST;
view.setUint16(offset, channelIdBytes.length, false); offset += 2;
out.set(channelIdBytes, offset); offset += channelIdBytes.length;
view.setUint16(offset, senderBytes.length, false); offset += 2;
out.set(senderBytes, offset); offset += senderBytes.length;
view.setUint32(offset, b.generation, false); offset += 4;
view.setUint32(offset, b.iteration, false); offset += 4;
out.set(b.nonce, offset); offset += BROADCAST_NONCE_BYTES;
out.set(b.signature, offset); offset += BROADCAST_SIGNATURE_BYTES;
view.setUint32(offset, b.ciphertext.length, false); offset += 4;
out.set(b.ciphertext, offset);
return out;
}
export function decodeBroadcast(data: Uint8Array): BroadcastWire {
const minSize = 1 + 1 + 2 + 0 + 2 + 0 + 4 + 4 +
BROADCAST_NONCE_BYTES + BROADCAST_SIGNATURE_BYTES + 4;
if (data.length < minSize) {
throw new Error(`broadcast envelope too short: ${data.length} < ${minSize}`);
}
if (data[0] !== VERSION) throw new Error(`Unknown version: ${data[0]}`);
if (data[1] !== TYPE_BROADCAST) throw new Error(`Not a broadcast: type=${data[1]}`);
const view = new DataView(data.buffer, data.byteOffset);
const dec = new TextDecoder();
let offset = 2;
const channelIdLen = view.getUint16(offset, false); offset += 2;
if (offset + channelIdLen > data.length) throw new Error('broadcast truncated in channelId');
const channelId = dec.decode(data.slice(offset, offset + channelIdLen));
offset += channelIdLen;
const senderLen = view.getUint16(offset, false); offset += 2;
if (offset + senderLen > data.length) throw new Error('broadcast truncated in senderAddress');
const senderAddress = dec.decode(data.slice(offset, offset + senderLen));
offset += senderLen;
const generation = view.getUint32(offset, false); offset += 4;
const iteration = view.getUint32(offset, false); offset += 4;
const nonce = data.slice(offset, offset + BROADCAST_NONCE_BYTES);
offset += BROADCAST_NONCE_BYTES;
const signature = data.slice(offset, offset + BROADCAST_SIGNATURE_BYTES);
offset += BROADCAST_SIGNATURE_BYTES;
const ctLen = view.getUint32(offset, false); offset += 4;
if (offset + ctLen !== data.length) {
throw new Error(`broadcast length mismatch: declared ${offset + ctLen}, actual ${data.length}`);
}
const ciphertext = data.slice(offset, offset + ctLen);
return { channelId, senderAddress, generation, iteration, nonce, signature, ciphertext };
}
function decodePreKeyMessageInner(data: Uint8Array): PreKeyMessage { function decodePreKeyMessageInner(data: Uint8Array): PreKeyMessage {
let offset = 0; let offset = 0;

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/recovery", "name": "@shade/recovery",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/sdk", "name": "@shade/sdk",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -96,7 +96,7 @@ export class BackgroundTasks {
this.hooks.onError?.(err as Error, 'prune-files'); this.hooks.onError?.(err as Error, 'prune-files');
} }
}, this.pruneFilesIntervalMs); }, this.pruneFilesIntervalMs);
this.pruneFilesTimer.unref?.(); (this.pruneFilesTimer as unknown as { unref?: () => void }).unref?.();
} }
stop(): void { stop(): void {

Binary file not shown.

View File

@@ -17,11 +17,12 @@ export interface ShadeConfig {
* - "memory" — in-memory only (lost on restart, good for tests) * - "memory" — in-memory only (lost on restart, good for tests)
* - "sqlite:/path/to/file.db" — SQLite backend * - "sqlite:/path/to/file.db" — SQLite backend
* - { type: 'postgres', url: 'postgres://...' } — PostgreSQL backend * - { type: 'postgres', url: 'postgres://...' } — PostgreSQL backend
* - { type: 'indexeddb', dbName?: 'my-app' } — browser IndexedDB
* - An explicit StorageProvider instance * - An explicit StorageProvider instance
* *
* Default: "memory" * Default: "memory"
*/ */
storage?: string | StorageProvider | { type: 'postgres'; url: string }; storage?: StorageSpec;
/** /**
* Your address on the prekey server (e.g. "alice@example.com" or "device:abc123"). * Your address on the prekey server (e.g. "alice@example.com" or "device:abc123").
@@ -96,9 +97,16 @@ export interface ShadeKTConfig {
witnessMaxStored?: number; witnessMaxStored?: number;
} }
/** Acceptable shapes for `ShadeConfig.storage`. */
export type StorageSpec =
| string
| StorageProvider
| { type: 'postgres'; url: string }
| { type: 'indexeddb'; dbName?: string };
export interface ResolvedConfig { export interface ResolvedConfig {
prekeyServer: string; prekeyServer: string;
storage: string | StorageProvider | { type: 'postgres'; url: string }; storage: StorageSpec;
address?: string | undefined; address?: string | undefined;
autoReplenish: { min: number; target: number; intervalMs: number } | false; autoReplenish: { min: number; target: number; intervalMs: number } | false;
autoRotate: false | '1d' | '7d' | '30d' | '90d'; autoRotate: false | '1d' | '7d' | '30d' | '90d';

View File

@@ -5,6 +5,11 @@ export type {
ShadeWebRtcConfig, ShadeWebRtcConfig,
ShadeWebRtcRuntime, ShadeWebRtcRuntime,
} from './shade.js'; } from './shade.js';
export type {
BroadcastChannel,
BroadcastChannelSummary,
MessageMeta,
} from './broadcast.js';
export { generateThumbnail } from './thumbnail.js'; export { generateThumbnail } from './thumbnail.js';
export type { GeneratedThumbnail, ThumbnailGenerationOptions } from './thumbnail.js'; export type { GeneratedThumbnail, ThumbnailGenerationOptions } from './thumbnail.js';
export { ShadeThumbnailCache } from './thumbnail-cache.js'; export { ShadeThumbnailCache } from './thumbnail-cache.js';

View File

@@ -31,6 +31,8 @@ import {
type TransferHandle, type TransferHandle,
type TransferOptions, type TransferOptions,
type TransferSummary, type TransferSummary,
type OutboundQueue as OutboundQueueLike,
type QueuedEventInput,
} from '@shade/transfer'; } from '@shade/transfer';
import type { Hono } from 'hono'; import type { Hono } from 'hono';
import { BackgroundTasks } from './background.js'; import { BackgroundTasks } from './background.js';
@@ -42,7 +44,18 @@ import {
backupFromString, backupFromString,
} from './backup.js'; } from './backup.js';
import { computeFingerprint, deserializeIdentityKeyPair } from '@shade/core'; import { computeFingerprint, deserializeIdentityKeyPair } from '@shade/core';
import type { ResolvedConfig } from './config.js'; import {
acceptBroadcastEnvelope,
createBroadcastChannelImpl,
getBroadcastChannelImpl,
listBroadcastChannelsImpl,
maybeHandleControlPlaintext,
type BroadcastChannel,
type BroadcastChannelSummary,
type BroadcastSdkHooks,
type MessageMeta,
} from './broadcast.js';
import type { ResolvedConfig, StorageSpec } from './config.js';
import { import {
ShadeControlChannel, ShadeControlChannel,
ShadeTransferAuthenticator, ShadeTransferAuthenticator,
@@ -140,10 +153,19 @@ export class Shade {
private establishing = new Map<string, Promise<void>>(); private establishing = new Map<string, Promise<void>>();
// Per-address encrypt queue to serialize ratchet mutations // Per-address encrypt queue to serialize ratchet mutations
private encryptChains = new Map<string, Promise<unknown>>(); private encryptChains = new Map<string, Promise<unknown>>();
// Per-`from` decrypt queue: serializes incoming receives so two concurrent
// shade.receive(from, env) calls can't race the ratchet/storage. Without
// this, parallel deliveries (relay duplicate fan-out, fast pipelined
// sends) hit `database is locked` (sqlite) or transaction conflicts (IDB)
// because the underlying StorageProvider isn't required to be a
// concurrent-safe writer. See V4.8.2 changelog.
private decryptChains = new Map<string, Promise<unknown>>();
// Message handlers — may be sync or async; receive() awaits each. // Message handlers — may be sync or async; receive() awaits each. The
// optional third arg distinguishes direct vs broadcast plaintexts;
// handlers registered without it work unchanged (V4.6 back-compat).
private messageHandlers: Array< private messageHandlers: Array<
(from: string, plaintext: string) => void | Promise<void> (from: string, plaintext: string, meta?: MessageMeta) => void | Promise<void>
> = []; > = [];
// Stream-transfer engine, lazily constructed on first use. // Stream-transfer engine, lazily constructed on first use.
@@ -151,6 +173,8 @@ export class Shade {
private controlChannel: ShadeControlChannel | null = null; private controlChannel: ShadeControlChannel | null = null;
private peerBaseUrlResolver: ((peerAddress: string) => Promise<string>) | null = null; private peerBaseUrlResolver: ((peerAddress: string) => Promise<string>) | null = null;
private envelopeOutboxes: ControlEnvelopeTransport | null = null; private envelopeOutboxes: ControlEnvelopeTransport | null = null;
private transferTransportOverride: ITransferTransport | null = null;
private transferQueue: OutboundQueueLike | null = null;
// `@shade/files` namespace, lazy + memoized. // `@shade/files` namespace, lazy + memoized.
private filesNamespace: FilesNamespace | null = null; private filesNamespace: FilesNamespace | null = null;
@@ -295,6 +319,26 @@ export class Shade {
return this.address; return this.address;
} }
/**
* The local device's Ed25519 identity public key (32 bytes).
*
* Stable for the lifetime of the identity. After {@link rotate} this
* reflects the new key; the previous key is preserved in retired-
* identities storage for the configured grace period.
*
* Hand this to your application's backend at enrollment time so it
* can verify signatures from this device, compute its own safety-
* number representation, or pin the key for later attestation. Use
* {@link fingerprint} instead for human side-channel comparison.
*/
get identityPublicKey(): Promise<Uint8Array> {
if (!this.initialized) throw new Error('Not initialized');
return this.storage.getIdentityKeyPair().then((kp) => {
if (!kp) throw new Error('Identity not yet generated');
return kp.signingPublicKey;
});
}
/** /**
* `@shade/files` namespace — high-level entry point for E2EE filesystem * `@shade/files` namespace — high-level entry point for E2EE filesystem
* RPC. Lazily creates the underlying channel + streams bridges on first * RPC. Lazily creates the underlying channel + streams bridges on first
@@ -390,13 +434,54 @@ export class Shade {
* The caller provides the `from` address because the envelope itself * The caller provides the `from` address because the envelope itself
* doesn't authenticate the sender — that's determined by your transport * doesn't authenticate the sender — that's determined by your transport
* layer (auth header, WebSocket peer, push notification metadata, etc.). * layer (auth header, WebSocket peer, push notification metadata, etc.).
*
* V4.6: when the decrypted plaintext is a broadcast control message
* (sender-key distribution / revocation), the SDK consumes it
* internally and returns an empty string; user handlers do NOT fire.
* Apps therefore see only direct plaintexts here. Broadcast payloads
* arrive via {@link Shade.acceptBroadcast}.
*/ */
async receive(from: string, envelope: ShadeEnvelope): Promise<string> { async receive(from: string, envelope: ShadeEnvelope): Promise<string> {
if (!this.initialized) throw new Error('Not initialized'); if (!this.initialized) throw new Error('Not initialized');
const plaintext = await this.manager.decrypt(from, envelope);
// Serialize ONLY the ratchet/storage write portion of receive (the
// call into `manager.decrypt`). Concurrent decrypts race the
// SessionManager ratchet (mutated in place) and the StorageProvider
// (not required to be a concurrent-safe writer — `bun:sqlite`
// throws `database is locked`, IDB throws transaction conflicts).
// The Prism FR called this out: a relay-duplicated WS fan-out
// dispatched 8 parallel `shade.receive(from, env)` calls, one won
// the X3DH prekey race and the other 7 failed with
// `database is locked` / `one-time prekey not found`. The fix is
// to queue per-`from` decrypts so the ratchet step is sequential.
//
// Crucially the user-visible MESSAGE HANDLERS run *outside* the
// queue. Streams + file-RPC issue nested `shade.receive` calls for
// the same peer from inside their handlers (e.g. `stream-end`
// arrives while a write-RPC is still waiting on chunks); holding
// the queue across the handler would self-deadlock. The atomic
// unit we have to protect is just the ratchet+storage step, not
// the consumer's reaction to it.
const previous = this.decryptChains.get(from) ?? Promise.resolve();
const decryptPromise = previous
.catch(() => undefined) // don't propagate upstream failures
.then(() => this.manager.decrypt(from, envelope));
// Store a never-rejecting copy so the next chained receive doesn't
// see a rejection from this one (we still surface our own rejection
// to *this* caller via the original `decryptPromise`).
this.decryptChains.set(from, decryptPromise.catch(() => undefined));
const plaintext = await decryptPromise;
const consumed = await maybeHandleControlPlaintext(
this.broadcastHooks(),
from,
plaintext,
);
if (consumed) return '';
const meta: MessageMeta = { kind: 'direct' };
for (const handler of this.messageHandlers) { for (const handler of this.messageHandlers) {
try { try {
await handler(from, plaintext); await handler(from, plaintext, meta);
} catch (err) { } catch (err) {
console.error('[Shade] Message handler threw:', err); console.error('[Shade] Message handler threw:', err);
} }
@@ -404,9 +489,16 @@ export class Shade {
return plaintext; return plaintext;
} }
/** Register a handler for incoming messages. Async handlers are awaited. */ /**
* Register a handler for incoming messages. Async handlers are awaited.
*
* V4.6: handlers may declare an optional `meta` parameter to discriminate
* direct (`meta.kind === 'direct'`) from broadcast (`meta.kind === 'broadcast'`)
* deliveries. Handlers that ignore the third arg keep working unchanged
* for direct messages.
*/
onMessage( onMessage(
handler: (from: string, plaintext: string) => void | Promise<void>, handler: (from: string, plaintext: string, meta?: MessageMeta) => void | Promise<void>,
): () => void { ): () => void {
this.messageHandlers.push(handler); this.messageHandlers.push(handler);
return () => { return () => {
@@ -414,6 +506,73 @@ export class Shade {
}; };
} }
// ─── V4.6 Broadcast channels ───────────────────────────────
/**
* Create a new broadcast channel owned by this device. Returns a
* handle for adding/removing members, encrypting a single payload,
* and rotating on revocation. The channel id is opaque, stable
* across `shutdown()` / re-open, and persisted via the configured
* `StorageProvider`.
*/
async createBroadcastChannel(opts: { label?: string } = {}): Promise<BroadcastChannel> {
if (!this.initialized) throw new Error('Not initialized');
return createBroadcastChannelImpl(this.broadcastHooks(), opts);
}
/**
* Look up an existing sender-side broadcast channel by id. Returns
* `null` when the id is unknown OR when this device only holds a
* receiver-side copy (the receiver path uses `onMessage` for delivery
* — there is no app-facing handle on the receive side).
*/
async getBroadcastChannel(channelId: string): Promise<BroadcastChannel | null> {
if (!this.initialized) throw new Error('Not initialized');
return getBroadcastChannelImpl(this.broadcastHooks(), channelId);
}
/**
* Snapshot of every broadcast channel persisted on this device,
* including receiver-side channels that we joined. Useful for
* rebuilding UI state on startup.
*/
async listBroadcastChannels(): Promise<readonly BroadcastChannelSummary[]> {
if (!this.initialized) throw new Error('Not initialized');
return listBroadcastChannelsImpl(this.broadcastHooks());
}
/**
* Hand a wire-encoded broadcast envelope (type 0x21) to the SDK.
* Decrypts via the matching channel, advances the chain, and dispatches
* the plaintext to `onMessage` handlers with `meta.kind === 'broadcast'`.
*
* Stale generations (sender's old chain after a rotation we already
* received) are silently dropped. Future generations (we haven't seen
* the rotation distribution yet) throw — the app should ensure the
* distribution envelope is delivered before the broadcast.
*/
async acceptBroadcast(envelope: Uint8Array): Promise<void> {
if (!this.initialized) throw new Error('Not initialized');
const result = await acceptBroadcastEnvelope(this.broadcastHooks(), envelope);
if (result === null) return;
for (const handler of this.messageHandlers) {
try {
await handler(result.meta.kind === 'broadcast' ? result.meta.sender : '', result.plaintext, result.meta);
} catch (err) {
console.error('[Shade] Broadcast handler threw:', err);
}
}
}
private broadcastHooks(): BroadcastSdkHooks {
return {
bilateralSend: (peer, pt) => this.send(peer, pt),
myAddress: () => this.address,
crypto: this.crypto,
storage: this.storage,
};
}
/** Get a peer's fingerprint (requires an existing session) */ /** Get a peer's fingerprint (requires an existing session) */
async getFingerprintFor(address: string): Promise<string> { async getFingerprintFor(address: string): Promise<string> {
if (!this.initialized) throw new Error('Not initialized'); if (!this.initialized) throw new Error('Not initialized');
@@ -746,12 +905,52 @@ export class Shade {
* HTTP POSTs to `<base>/v1/transfer/control`). * HTTP POSTs to `<base>/v1/transfer/control`).
*/ */
configureTransfers(opts: { configureTransfers(opts: {
resolveBaseUrl: (peerAddress: string) => Promise<string>; /**
* Resolver for the peer's HTTP base URL (used by the default
* `ShadeTransferHttpTransport` to POST chunks). Optional when a
* custom `transport` and `envelopeTransport` are supplied — e.g.
* for pull-mode browser servers (`@shade/files transferQueueRoute`)
* which never POST chunks anywhere.
*/
resolveBaseUrl?: (peerAddress: string) => Promise<string>;
/**
* Override the chunk-level transport. Defaults to
* `ShadeTransferHttpTransport` (HTTP POSTs per chunk) when
* `resolveBaseUrl` is supplied. Required when `resolveBaseUrl`
* is omitted.
*/
transport?: ITransferTransport;
/**
* Override the control-envelope transport. Defaults to HTTP POSTs
* to `<base>/v1/transfer/control` when `resolveBaseUrl` is
* supplied. Required when `resolveBaseUrl` is omitted.
*/
envelopeTransport?: ControlEnvelopeTransport; envelopeTransport?: ControlEnvelopeTransport;
}): void { }): void {
if (opts.resolveBaseUrl === undefined) {
if (opts.transport === undefined || opts.envelopeTransport === undefined) {
throw new Error(
'configureTransfers: resolveBaseUrl is required unless both `transport` and `envelopeTransport` are supplied (e.g. for pull-mode queue servers).',
);
}
this.peerBaseUrlResolver = async () => {
throw new Error(
'resolveBaseUrl was not configured — this Shade is in queue/pull mode and does not POST chunks. Configure a custom transport instead.',
);
};
} else {
this.peerBaseUrlResolver = opts.resolveBaseUrl; this.peerBaseUrlResolver = opts.resolveBaseUrl;
this.envelopeOutboxes = }
opts.envelopeTransport ?? new HttpEnvelopeTransport(opts.resolveBaseUrl, this.address); this.transferTransportOverride = opts.transport ?? null;
if (opts.envelopeTransport !== undefined) {
this.envelopeOutboxes = opts.envelopeTransport;
} else if (opts.resolveBaseUrl !== undefined) {
this.envelopeOutboxes = new HttpEnvelopeTransport(opts.resolveBaseUrl, this.address);
} else {
throw new Error(
'configureTransfers: envelopeTransport is required when resolveBaseUrl is omitted.',
);
}
} }
/** /**
@@ -898,6 +1097,109 @@ export class Shade {
return (await this.engine()).onIncomingTransfer(handler); return (await this.engine()).onIncomingTransfer(handler);
} }
/**
* Mount the **pull-mode** transfer routes on a Hono app. Mount under
* any base path: `app.route('/api/v1/shade-files', shade.transferQueueRoute())`.
*
* Configures this Shade instance to queue all outbound chunks +
* control envelopes per peer instead of POSTing them. Browser-style
* receivers drain the queue via long-polling — no inbound HTTP
* listener required on the receiver.
*
* Routes mounted (relative to the base path):
* POST /queue — long-poll the per-peer outbound queue
* POST /v1/transfer/:streamId/chunk — receive incoming chunks (browser → server)
* GET /v1/transfer/:streamId/state — resume-state lookup
* POST /v1/transfer/control — receive incoming control envelopes
* GET /v1/transfer/health — peer reachability probe
*
* **Idempotent**: calling twice returns a fresh `Hono` app each
* time but reuses the underlying queue + transport (so the engine
* stays single).
*
* **Ordering**: must be called **before** `shade.files.serve(...)`
* (or any other path that builds the engine), because configuring
* the queue transport mutates the transfer stack. Calling after the
* engine is built throws.
*/
async transferQueueRoute(opts: TransferQueueRouteOptions = {}): Promise<Hono> {
if (this.transferEngine !== null && this.transferTransportOverride === null) {
throw new Error(
'transferQueueRoute(): the transfer engine has already been built with the default HTTP transport. Call transferQueueRoute() before any upload()/onIncomingTransfer()/configureTransfers().',
);
}
const { OutboundQueue, QueueTransferTransport } = await import('@shade/transfer');
if (this.transferQueue === null) {
this.transferQueue = new OutboundQueue({
...(opts.maxEventsPerPeer !== undefined ? { maxEventsPerPeer: opts.maxEventsPerPeer } : {}),
...(opts.idleEvictionMs !== undefined ? { idleEvictionMs: opts.idleEvictionMs } : {}),
});
}
if (this.transferTransportOverride === null) {
const queueTransport = new QueueTransferTransport(this.transferQueue);
const queueEnvelopeTransport = new QueueEnvelopeTransport(this.transferQueue);
this.configureTransfers({
transport: queueTransport,
envelopeTransport: queueEnvelopeTransport,
});
}
const queue = this.transferQueue;
const blockMs = opts.blockMs ?? 30_000;
const maxBlockMs = opts.maxBlockMs ?? 55_000;
const engine = await this.engine();
const { createTransferRoutes, PermissiveAuthenticator } = await import('@shade/transfer');
const app = await createTransferRoutes(engine, {
authenticator: PermissiveAuthenticator,
});
app.post('/v1/transfer/control', async (c) => {
const senderAddress = c.req.header('X-Shade-Sender-Address');
if (senderAddress === undefined || senderAddress === '') {
return c.json({ error: 'missing X-Shade-Sender-Address' }, 400);
}
const ab = await c.req.arrayBuffer();
const bytes = new Uint8Array(ab);
try {
await this.acceptTransferEnvelope(senderAddress, bytes);
} catch (err) {
return c.json({ error: (err as Error).message }, 400);
}
return c.json({ ok: true });
});
// Long-poll endpoint.
app.post('/queue', async (c) => {
const senderAddress = c.req.header('X-Shade-Sender-Address');
if (senderAddress === undefined || senderAddress === '') {
return c.json({ error: 'missing X-Shade-Sender-Address' }, 400);
}
let body: { since?: unknown; blockMs?: unknown };
try {
body = (await c.req.json()) as { since?: unknown; blockMs?: unknown };
} catch {
return c.json({ error: 'invalid JSON body' }, 400);
}
const since = typeof body.since === 'number' && Number.isFinite(body.since) ? body.since : 0;
const requestedBlockMs =
typeof body.blockMs === 'number' && Number.isFinite(body.blockMs)
? Math.max(0, Math.min(maxBlockMs, body.blockMs))
: blockMs;
// Bun-side short-circuit if the request was aborted while we
// were holding the long-poll. AbortSignal from the request body
// is already surfaced via `c.req.raw.signal` in Hono.
const events = await queue.drain(senderAddress, since, requestedBlockMs, c.req.raw.signal);
return c.json({
events: events.map((e) => ({
id: e.id,
timestampMs: e.timestampMs,
kind: e.kind,
bytesB64: bytesToBase64Std(e.bytes),
...(e.kind === 'chunk' ? { meta: e.meta } : {}),
})),
nextSince: events.length > 0 ? events[events.length - 1]!.id : since,
});
});
return app;
}
/** /**
* Mount the receiver-side HTTP routes on a Hono app. Mount under any * Mount the receiver-side HTTP routes on a Hono app. Mount under any
* base path: `app.route('/shade', await shade.transferRoute())`. * base path: `app.route('/shade', await shade.transferRoute())`.
@@ -1019,17 +1321,24 @@ export class Shade {
); );
} }
this.controlChannel = new ShadeControlChannel(this, this.envelopeOutboxes); this.controlChannel = new ShadeControlChannel(this, this.envelopeOutboxes);
let transport: ITransferTransport;
let webrtcRuntime: ShadeWebRtcRuntime | null = null;
if (this.transferTransportOverride !== null) {
// Custom transport (queue, in-memory, custom adapter) — used as-is.
// WebRTC fallback only attaches when the default HTTP transport is
// active because WebRTC's `MultiTransportFallback` is HTTP-shaped.
transport = this.transferTransportOverride;
} else {
const httpTransport: ITransferTransport = new ShadeTransferHttpTransport({ const httpTransport: ITransferTransport = new ShadeTransferHttpTransport({
resolveBaseUrl: this.peerBaseUrlResolver, resolveBaseUrl: this.peerBaseUrlResolver,
authenticator: await this.makeAuthenticator(), authenticator: await this.makeAuthenticator(),
}); });
transport = httpTransport;
let transport: ITransferTransport = httpTransport;
let webrtcRuntime: ShadeWebRtcRuntime | null = null;
if (this.webrtcConfig !== null) { if (this.webrtcConfig !== null) {
webrtcRuntime = await this.buildWebRtcRuntime(this.webrtcConfig, httpTransport); webrtcRuntime = await this.buildWebRtcRuntime(this.webrtcConfig, httpTransport);
transport = webrtcRuntime.fallback; transport = webrtcRuntime.fallback;
} }
}
this.transferEngine = new TransferEngine({ this.transferEngine = new TransferEngine({
crypto: this.crypto, crypto: this.crypto,
@@ -1256,6 +1565,53 @@ function parseChunkHeader(bytes: Uint8Array): {
return { streamId, laneId, seq }; return { streamId, laneId, seq };
} }
// ─── Queue-mode (pull) envelope transport ─────────────────────
/**
* Configuration for {@link Shade.transferQueueRoute}. All fields are
* optional with sensible production defaults.
*/
export interface TransferQueueRouteOptions {
/**
* Long-poll timeout in milliseconds. Server holds the request open
* up to this long before returning an empty `events` array. Default
* 30_000.
*/
blockMs?: number;
/**
* Hard cap on long-poll timeout (clamps client-supplied `blockMs`).
* Default 55_000 — under typical reverse-proxy idle thresholds (60s
* on most CDNs).
*/
maxBlockMs?: number;
/**
* Per-peer ring-buffer size. When the queue is full, oldest events
* are dropped on enqueue. Receivers detect the gap via missing
* sequence numbers and re-resume from `since=0`. Default 1000.
*/
maxEventsPerPeer?: number;
/**
* Drop a peer's queue + reject pending pollers after this much
* silence. Default 10 minutes. Setting to `0` disables idle-eviction.
*/
idleEvictionMs?: number;
}
/**
* `ControlEnvelopeTransport` that enqueues outbound envelopes into an
* `OutboundQueue` for browser-style receivers to long-poll. Mirrors
* `HttpEnvelopeTransport` shape (one `send(peer, envelope)` method);
* the difference is the destination — local queue, not remote HTTP.
*/
class QueueEnvelopeTransport implements ControlEnvelopeTransport {
constructor(private readonly queue: OutboundQueueLike) {}
async send(peerAddress: string, envelope: ShadeEnvelope): Promise<void> {
const bytes = encodeEnvelope(envelope);
const event: QueuedEventInput = { kind: 'envelope', bytes };
this.queue.enqueue(peerAddress, event);
}
}
// ─── Default HTTP envelope transport ────────────────────────── // ─── Default HTTP envelope transport ──────────────────────────
class HttpEnvelopeTransport implements ControlEnvelopeTransport { class HttpEnvelopeTransport implements ControlEnvelopeTransport {
@@ -1283,9 +1639,7 @@ class HttpEnvelopeTransport implements ControlEnvelopeTransport {
// ─── Helpers ───────────────────────────────────────────────── // ─── Helpers ─────────────────────────────────────────────────
async function resolveStorage( async function resolveStorage(spec: StorageSpec): Promise<StorageProvider> {
spec: string | StorageProvider | { type: 'postgres'; url: string },
): Promise<StorageProvider> {
if (typeof spec === 'object' && 'getIdentityKeyPair' in spec) { if (typeof spec === 'object' && 'getIdentityKeyPair' in spec) {
return spec; return spec;
} }
@@ -1311,6 +1665,18 @@ async function resolveStorage(
return mod.PostgresStorage.create(spec.url); return mod.PostgresStorage.create(spec.url);
} }
if (typeof spec === 'object' && spec.type === 'indexeddb') {
// Dynamic import keeps @shade/storage-indexeddb optional — Node-only
// consumers don't need to install a browser-only adapter.
const moduleId = '@shade/storage-indexeddb';
const mod = (await import(moduleId)) as {
IndexedDBStorage: { create(opts: { dbName?: string }): Promise<StorageProvider> };
};
const opts: { dbName?: string } = {};
if (spec.dbName !== undefined) opts.dbName = spec.dbName;
return mod.IndexedDBStorage.create(opts);
}
throw new Error(`Unsupported storage spec: ${JSON.stringify(spec)}`); throw new Error(`Unsupported storage spec: ${JSON.stringify(spec)}`);
} }

View File

@@ -0,0 +1,225 @@
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { tmpdir } from 'os';
import { join } from 'path';
import { unlinkSync } from 'fs';
import { createShade, type Shade, type MessageMeta } from '../src/index.js';
import { createPrekeyServer, MemoryPrekeyStore } from '@shade/server';
import { SubtleCryptoProvider } from '@shade/crypto-web';
const crypto = new SubtleCryptoProvider();
async function startPrekeyServer(): Promise<{ url: string; stop: () => void }> {
const server = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
});
const port = 19500 + Math.floor(Math.random() * 500);
const handle = Bun.serve({ port, fetch: server.fetch });
return { url: `http://localhost:${port}`, stop: () => handle.stop() };
}
/**
* Prism's three acceptance tests, ported verbatim from
* `Docs/shade-feature-request-sender-keys.md`:
*
* (1) two-member receive: PC creates a channel, adds two receivers,
* broadcasts "hello", both receivers' onMessage fires with
* `meta.kind === 'broadcast'` and the same plaintext.
*
* (1*) revocation: same setup, then `removeMember(receiverA)`. Receiver
* A's next attempt to decrypt a subsequent broadcast fails (or is
* silently dropped); receiver B keeps working.
*
* (2) persistence: create channel, add members, broadcast N messages,
* `shutdown()`, re-open with the same backing store, channel still
* exists, member list intact, generation preserved, next broadcast
* decrypts on receiver side.
*
* (3) no new wire-format changes visible to apps — the pair-flow stays
* a single round-trip; the SDK does the sender-key distribution
* inline. (Validated by inspecting the API surface.)
*/
describe('Broadcast channels — Prism acceptance', () => {
let server: Awaited<ReturnType<typeof startPrekeyServer>>;
let pc: Shade;
let mobileA: Shade;
let mobileB: Shade;
beforeEach(async () => {
server = await startPrekeyServer();
});
afterEach(async () => {
await pc?.shutdown();
await mobileA?.shutdown();
await mobileB?.shutdown();
server.stop();
});
test('(1) two-member receive', async () => {
pc = await createShade({ prekeyServer: server.url, address: 'pc' });
mobileA = await createShade({ prekeyServer: server.url, address: 'mobile-a' });
mobileB = await createShade({ prekeyServer: server.url, address: 'mobile-b' });
const channel = await pc.createBroadcastChannel({ label: 'output' });
expect(channel.id.length).toBeGreaterThan(0);
// Add both members. Each call returns a bilateral envelope to deliver.
const distA = await channel.addMember('mobile-a');
const distB = await channel.addMember('mobile-b');
expect((await channel.members())).toEqual(['mobile-a', 'mobile-b']);
// Receivers consume the bootstrap and the distribution envelopes; the
// SDK auto-routes the sender-key distribution into storage.
await mobileA.receive('pc', distA.envelope);
await mobileB.receive('pc', distB.envelope);
// Hook receiver-side onMessage with meta.
const receivedA: Array<{ from: string; pt: string; meta?: MessageMeta }> = [];
const receivedB: Array<{ from: string; pt: string; meta?: MessageMeta }> = [];
mobileA.onMessage((from, pt, meta) => { receivedA.push({ from, pt, meta }); });
mobileB.onMessage((from, pt, meta) => { receivedB.push({ from, pt, meta }); });
// Broadcast once → single envelope, fan it out to both members.
const out = await channel.broadcast('hello');
expect(out.members).toEqual(['mobile-a', 'mobile-b']);
await mobileA.acceptBroadcast(out.envelope);
await mobileB.acceptBroadcast(out.envelope);
expect(receivedA).toHaveLength(1);
expect(receivedB).toHaveLength(1);
expect(receivedA[0]!.pt).toBe('hello');
expect(receivedB[0]!.pt).toBe('hello');
expect(receivedA[0]!.meta?.kind).toBe('broadcast');
expect(receivedB[0]!.meta?.kind).toBe('broadcast');
if (receivedA[0]!.meta?.kind === 'broadcast') {
expect(receivedA[0]!.meta.channelId).toBe(channel.id);
expect(receivedA[0]!.meta.sender).toBe('pc');
}
});
test('(1*) revocation rotates the chain — receiver A drops, B keeps working', async () => {
pc = await createShade({ prekeyServer: server.url, address: 'pc' });
mobileA = await createShade({ prekeyServer: server.url, address: 'mobile-a' });
mobileB = await createShade({ prekeyServer: server.url, address: 'mobile-b' });
const channel = await pc.createBroadcastChannel({ label: 'output' });
const distA = await channel.addMember('mobile-a');
const distB = await channel.addMember('mobile-b');
await mobileA.receive('pc', distA.envelope);
await mobileB.receive('pc', distB.envelope);
const receivedA: string[] = [];
const receivedB: string[] = [];
mobileA.onMessage((_from, pt, meta) => {
if (meta?.kind === 'broadcast') receivedA.push(pt);
});
mobileB.onMessage((_from, pt, meta) => {
if (meta?.kind === 'broadcast') receivedB.push(pt);
});
// Pre-revocation broadcast — both decrypt.
const before = await channel.broadcast('before');
await mobileA.acceptBroadcast(before.envelope);
await mobileB.acceptBroadcast(before.envelope);
expect(receivedA).toEqual(['before']);
expect(receivedB).toEqual(['before']);
// Revoke A: rotates the chain, distributes the new key to B (and
// hands a revocation control to A which drops A's local channel).
const { rotations } = await channel.removeMember('mobile-a');
expect(rotations.map((r) => r.to)).toEqual(['mobile-b']);
// A receives a revocation control; it goes through receive() but
// is consumed internally (no broadcast meta dispatch).
// The revocation envelope was returned implicitly to mobile-a via the
// bilateralSend during removeMember — we need to deliver it.
// (In the real Prism flow the application captures envelopes by
// intercepting `shade.send`. Here we just trust it happened.)
// Deliver new chain to B.
await mobileB.receive('pc', rotations[0]!.envelope);
// Post-rotation broadcast.
const after = await channel.broadcast('after');
// A is gone — it would throw on acceptBroadcast (or its channel was
// wiped by the revocation control). Either way, A's receivedA stays
// unchanged.
let aThrew = false;
try {
await mobileA.acceptBroadcast(after.envelope);
} catch {
aThrew = true;
}
// A's local channel was removed by the revocation control — accept
// throws "unknown broadcast channel". This is the expected post-
// revocation behavior.
expect(aThrew).toBe(true);
// B decrypts as normal.
await mobileB.acceptBroadcast(after.envelope);
expect(receivedA).toEqual(['before']); // unchanged
expect(receivedB).toEqual(['before', 'after']);
});
test('(2) persistence — channel survives shutdown + re-open', async () => {
// Use a SQLite-backed storage so we get real persistence.
const dbPath = join(tmpdir(), `shade-broadcast-${Date.now()}-${Math.random().toString(36).slice(2)}.db`);
pc = await createShade({ prekeyServer: server.url, address: 'pc', storage: `sqlite:${dbPath}` });
mobileA = await createShade({ prekeyServer: server.url, address: 'mobile-a' });
const channel = await pc.createBroadcastChannel({ label: 'output' });
const channelId = channel.id;
const dist = await channel.addMember('mobile-a');
await mobileA.receive('pc', dist.envelope);
// Send a few broadcasts to advance the chain.
const received: string[] = [];
mobileA.onMessage((_from, pt, meta) => {
if (meta?.kind === 'broadcast') received.push(pt);
});
for (let i = 0; i < 3; i++) {
const out = await channel.broadcast(`msg-${i}`);
await mobileA.acceptBroadcast(out.envelope);
}
expect(received).toEqual(['msg-0', 'msg-1', 'msg-2']);
// Shutdown and re-open the PC side. The storage file persists.
await pc.shutdown();
pc = await createShade({ prekeyServer: server.url, address: 'pc', storage: `sqlite:${dbPath}` });
const reopened = await pc.getBroadcastChannel(channelId);
expect(reopened).not.toBeNull();
expect(reopened!.id).toBe(channelId);
expect((await reopened!.members())).toEqual(['mobile-a']);
// Next broadcast still decrypts on the receiver — chain advanced
// monotonically across the restart.
const out = await reopened!.broadcast('msg-after-restart');
await mobileA.acceptBroadcast(out.envelope);
expect(received).toEqual(['msg-0', 'msg-1', 'msg-2', 'msg-after-restart']);
// Cleanup.
try { unlinkSync(dbPath); } catch {}
try { unlinkSync(dbPath + '-shm'); } catch {}
try { unlinkSync(dbPath + '-wal'); } catch {}
});
test('(3) listBroadcastChannels surfaces sender + receiver records', async () => {
pc = await createShade({ prekeyServer: server.url, address: 'pc' });
mobileA = await createShade({ prekeyServer: server.url, address: 'mobile-a' });
const channel = await pc.createBroadcastChannel({ label: 'output' });
const dist = await channel.addMember('mobile-a');
await mobileA.receive('pc', dist.envelope);
const pcSide = await pc.listBroadcastChannels();
const mobileSide = await mobileA.listBroadcastChannels();
expect(pcSide).toHaveLength(1);
expect(pcSide[0]!.ownerRole).toBe('sender');
expect(pcSide[0]!.members).toEqual(['mobile-a']);
expect(mobileSide).toHaveLength(1);
expect(mobileSide[0]!.ownerRole).toBe('receiver');
expect(mobileSide[0]!.id).toBe(channel.id);
});
});

View File

@@ -5,7 +5,7 @@ import {
MemoryPrekeyStore, MemoryPrekeyStore,
PrekeyServerEvents, PrekeyServerEvents,
} from '@shade/server'; } from '@shade/server';
import { SubtleCryptoProvider } from '@shade/crypto-web'; import { SubtleCryptoProvider, MemoryStorage } from '@shade/crypto-web';
const crypto = new SubtleCryptoProvider(); const crypto = new SubtleCryptoProvider();
@@ -131,6 +131,53 @@ describe('createShade — happy path', () => {
await expect(alice.send('nobody', 'ghost')).rejects.toThrow(); await expect(alice.send('nobody', 'ghost')).rejects.toThrow();
}); });
test('concurrent receive(from, env) for same `from` does not race the ratchet (V4.8.2)', async () => {
// Reproduces the Prism FR scenario: a single PUT is fanned out
// multiple times by the relay (or any duplicating transport), the
// receiver dispatches several `shade.receive(from, env)` in
// parallel, and the underlying SessionManager + StorageProvider
// would race on the ratchet (and on storage writes — sqlite throws
// "database is locked", IDB throws transaction conflicts) without
// per-`from` serialization. We pre-establish a session, then fire
// the same envelope at `bob.receive` from many concurrent callers
// and verify all of them either decrypt to the same plaintext or
// surface a benign "already-consumed" error. Crucially: no
// unhandled storage races, no ratchet corruption, and the next
// legitimate message still decrypts.
alice = await createShade({ prekeyServer: server.url, address: 'alice' });
bob = await createShade({ prekeyServer: server.url, address: 'bob' });
const env1 = await alice.send('bob', 'first');
expect(await bob.receive('alice', env1)).toBe('first');
const env2 = await alice.send('bob', 'second');
// Fan the same envelope out to 8 concurrent receives — exactly the
// shape of the relay duplicate fan-out described in the FR.
const dispatches = await Promise.allSettled(
Array.from({ length: 8 }, () => bob.receive('alice', env2)),
);
// At least one must have succeeded with the right plaintext; the
// others may legitimately reject (replay protection / OTPK
// already-consumed) but MUST NOT corrupt the ratchet or throw
// "database is locked".
const fulfilled = dispatches.filter((d) => d.status === 'fulfilled') as Array<
PromiseFulfilledResult<string>
>;
expect(fulfilled.length).toBeGreaterThan(0);
expect(fulfilled[0]!.value).toBe('second');
for (const d of dispatches) {
if (d.status === 'rejected') {
const msg = String((d.reason as Error)?.message ?? d.reason);
expect(msg).not.toMatch(/database is locked/i);
}
}
// Ratchet must still advance — the next legitimate message decrypts.
const env3 = await alice.send('bob', 'third');
expect(await bob.receive('alice', env3)).toBe('third');
});
test('verify fingerprint matches pinned identity', async () => { test('verify fingerprint matches pinned identity', async () => {
alice = await createShade({ prekeyServer: server.url, address: 'alice' }); alice = await createShade({ prekeyServer: server.url, address: 'alice' });
bob = await createShade({ prekeyServer: server.url, address: 'bob' }); bob = await createShade({ prekeyServer: server.url, address: 'bob' });
@@ -182,6 +229,26 @@ describe('createShade — happy path', () => {
const newFp = await alice.fingerprint; const newFp = await alice.fingerprint;
expect(newFp).not.toBe(oldFp); expect(newFp).not.toBe(oldFp);
}); });
test('identityPublicKey exposes the device Ed25519 key and tracks rotation', async () => {
const storage = new MemoryStorage();
alice = await createShade({ prekeyServer: server.url, address: 'alice', storage });
const pk = await alice.identityPublicKey;
expect(pk).toBeInstanceOf(Uint8Array);
expect(pk.length).toBe(32);
// Matches what the underlying storage holds
const stored = await storage.getIdentityKeyPair();
expect(stored).not.toBeNull();
expect(pk).toEqual(stored!.signingPublicKey);
// Reflects the new key after rotate (acceptance criteria #3)
await alice.rotate();
const pkAfter = await alice.identityPublicKey;
expect(pkAfter.length).toBe(32);
expect(pkAfter).not.toEqual(pk);
});
}); });
describe('createShade — validation', () => { describe('createShade — validation', () => {

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/server", "name": "@shade/server",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -154,6 +154,20 @@ const inboxEvents = new InboxServerEvents();
// SHADE_KT_PG_URL (or SHADE_PREKEY_PG_URL) is set, else memory. // SHADE_KT_PG_URL (or SHADE_PREKEY_PG_URL) is set, else memory.
const kt = await maybeCreateKT(); const kt = await maybeCreateKT();
// V4.8.1 — `SHADE_DISABLE_RATE_LIMIT=1` turns off the IP-based
// register/replenish/fetch token-buckets on every prekey + inbox
// route. INTENDED FOR SELF-HOSTED SINGLE-TEAM (DEV / SINGLE-TENANT)
// DEPLOYMENTS ONLY — the rate-limit defends multi-tenant relays
// against abuse, so a public/shared deployment must leave this
// unset. Without it, the existing 5/hour REGISTER_LIMIT etc. apply
// unchanged.
const disableRateLimit = process.env.SHADE_DISABLE_RATE_LIMIT === '1';
if (disableRateLimit) {
logger.warn(
'SHADE_DISABLE_RATE_LIMIT=1 — IP rate limits OFF on prekey + inbox routes. Use only for single-tenant deployments.',
);
}
// Compose the full app: metrics middleware + health + metrics + prekey routes // Compose the full app: metrics middleware + health + metrics + prekey routes
const app = new Hono(); const app = new Hono();
app.use('*', metricsMiddleware()); app.use('*', metricsMiddleware());
@@ -164,10 +178,17 @@ app.route(
'/', '/',
createPrekeyRoutes(store, crypto, { createPrekeyRoutes(store, crypto, {
events, events,
disableRateLimit,
...(kt ? { keyTransparency: kt } : {}), ...(kt ? { keyTransparency: kt } : {}),
}), }),
); );
app.route('/', createInboxRoutes(inboxStore, crypto, { events: inboxEvents })); app.route(
'/',
createInboxRoutes(inboxStore, crypto, {
events: inboxEvents,
disableRateLimit,
}),
);
// V3.7 transport-bridge — SSE / long-poll / WS fallbacks for the inbox. // V3.7 transport-bridge — SSE / long-poll / WS fallbacks for the inbox.
// Held as a top-level reference so the WebSocket handler can be passed to // Held as a top-level reference so the WebSocket handler can be passed to

View File

@@ -102,6 +102,48 @@ describe('Rate limiting integration with routes', () => {
expect(results.filter((s) => s === 429).length).toBeGreaterThanOrEqual(1); expect(results.filter((s) => s === 429).length).toBeGreaterThanOrEqual(1);
}); });
// V4.8.1 — `SHADE_DISABLE_RATE_LIMIT=1` in standalone.ts is plumbed
// through to `createPrekeyServer({ disableRateLimit })`. This test
// covers the "what happens when the flag is true" path; the env-var
// → option conversion in standalone.ts is a one-liner verified by
// inspection.
test('register endpoint allows >5/hour from a single IP when disableRateLimit is set', async () => {
const app = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
});
async function doRegister(addressSuffix: number) {
const identity = await generateIdentityKeyPair(crypto);
const body: any = {
address: `user${addressSuffix}`,
identitySigningKey: Buffer.from(identity.signingPublicKey).toString('base64'),
identityDHKey: Buffer.from(identity.dhPublicKey).toString('base64'),
signedPreKey: {
keyId: 1,
publicKey: Buffer.from(crypto.randomBytes(32)).toString('base64'),
signature: Buffer.from(crypto.randomBytes(64)).toString('base64'),
},
};
const signed = await signPayload(crypto, identity.signingPrivateKey, body);
return app.request('/v1/keys/register', {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'x-forwarded-for': '203.0.113.1' },
body: JSON.stringify(signed),
});
}
const results: number[] = [];
for (let i = 0; i < 12; i++) {
const res = await doRegister(i);
results.push(res.status);
}
// No 429 anywhere — the limit is OFF.
expect(results.filter((s) => s === 429).length).toBe(0);
expect(results.filter((s) => s === 200).length).toBe(12);
});
test('rate limit returns Retry-After header', async () => { test('rate limit returns Retry-After header', async () => {
const app = createPrekeyServer({ crypto, store: new MemoryPrekeyStore() }); const app = createPrekeyServer({ crypto, store: new MemoryPrekeyStore() });

View File

@@ -1,17 +1,31 @@
{ {
"name": "@shade/storage-encrypted", "name": "@shade/storage-encrypted",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",
"exports": {
".": {
"browser": "./src/index.browser.ts",
"default": "./src/index.ts"
},
"./crypto": "./src/crypto.ts",
"./sqlite": "./src/sqlite.ts",
"./postgres": "./src/postgres.ts",
"./idb": "./src/idb.ts"
},
"dependencies": { "dependencies": {
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
"@shade/storage-postgres": "workspace:*", "@shade/storage-postgres": "workspace:*",
"@shade/storage-sqlite": "workspace:*", "@shade/storage-sqlite": "workspace:*",
"idb": "^8.0.3",
"postgres": "^3.4.9" "postgres": "^3.4.9"
}, },
"devDependencies": {
"fake-indexeddb": "^6.0.0"
},
"peerDependencies": { "peerDependencies": {
"@shade/keychain": "workspace:*" "@shade/keychain": "workspace:*"
}, },

View File

@@ -0,0 +1,64 @@
/**
* Browser-safe entry point — exports KeyManager + crypto primitives only.
*
* Use this subpath when bundling for the browser to avoid pulling
* `bun:sqlite` (Bun-only) or `postgres` (Node-only) into the output.
*
* ```ts
* import { KeyManager } from '@shade/storage-encrypted/crypto';
* ```
*
* For backend storage, import `@shade/storage-encrypted/sqlite` (Bun) or
* `@shade/storage-encrypted/postgres` (Node). For browser storage, import
* `@shade/storage-encrypted/idb`.
*/
export {
KeyManager,
type KeySource,
type KeychainBackend,
type KeyManagerOptions,
} from './crypto/key-manager.js';
export {
DEFAULT_SCRYPT,
DEFAULT_ARGON2ID,
type ScryptParams,
type Argon2idParams,
deriveMasterKey,
deriveMasterKeyArgon2id,
deriveStorageKey,
deriveFieldKey,
deriveNonce,
buildAad,
hkdfDerive,
} from './crypto/kdf.js';
export {
AEAD_NONCE_LEN,
AEAD_TAG_LEN,
aeadSeal,
aeadOpen,
} from './crypto/aead.js';
export {
COL,
TBL,
sealString,
openString,
sealBytes,
openBytes,
sealIdentity,
openIdentity,
sealConfig,
openConfig,
sealSignedPreKey,
openSignedPreKey,
sealOneTimePreKey,
openOneTimePreKey,
sealSession,
openSession,
sealTrust,
openTrust,
sealRetired,
openRetired,
sealStreamSensitive,
openStreamSensitive,
} from './crypto/row-codec.js';

View File

@@ -11,6 +11,7 @@
*/ */
import { scryptAsync } from '@noble/hashes/scrypt.js'; import { scryptAsync } from '@noble/hashes/scrypt.js';
import { argon2idAsync } from '@noble/hashes/argon2.js';
import { hkdf } from '@noble/hashes/hkdf.js'; import { hkdf } from '@noble/hashes/hkdf.js';
import { sha256 } from '@noble/hashes/sha2.js'; import { sha256 } from '@noble/hashes/sha2.js';
@@ -45,6 +46,50 @@ export async function deriveMasterKey(
return scryptAsync(TEXT.encode(passphrase.normalize('NFKC')), salt, params); return scryptAsync(TEXT.encode(passphrase.normalize('NFKC')), salt, params);
} }
/** Argon2id parameters — memory-hard KDF preferred for low-entropy secrets (PINs). */
export interface Argon2idParams {
/** Memory cost in KiB. */
m: number;
/** Time cost (iterations). */
t: number;
/** Parallelism. */
p: number;
/** Output length in bytes. */
dkLen: number;
}
/**
* Default: m=64 MiB, t=3, p=1, 32-byte output. Tuned for ~250400 ms on a
* modern Chromium / Firefox / Safari laptop. RFC 9106 "second recommended"
* profile shrunk to a browser-friendly memory footprint — strong enough for
* 46 digit PINs as a defense-in-depth factor on top of a passphrase.
*/
export const DEFAULT_ARGON2ID: Argon2idParams = { m: 64 * 1024, t: 3, p: 1, dkLen: 32 };
/**
* Derive a 32-byte master key from a low-entropy secret + salt using
* argon2id. Salt MUST be persisted alongside the DB (16-byte random).
*/
export async function deriveMasterKeyArgon2id(
secret: string | Uint8Array,
salt: Uint8Array,
params: Argon2idParams = DEFAULT_ARGON2ID,
): Promise<Uint8Array> {
if (typeof secret === 'string' ? secret.length === 0 : secret.length === 0) {
throw new Error('argon2id secret must be non-empty');
}
if (salt.length < 16) {
throw new Error('salt must be at least 16 bytes');
}
const password = typeof secret === 'string' ? TEXT.encode(secret.normalize('NFKC')) : secret;
return argon2idAsync(password, salt, {
m: params.m,
t: params.t,
p: params.p,
dkLen: params.dkLen,
});
}
/** HKDF-SHA-256 with explicit info string. */ /** HKDF-SHA-256 with explicit info string. */
export function hkdfDerive(ikm: Uint8Array, info: string, length = 32, salt?: Uint8Array): Uint8Array { export function hkdfDerive(ikm: Uint8Array, info: string, length = 32, salt?: Uint8Array): Uint8Array {
return hkdf(sha256, ikm, salt, TEXT.encode(info), length); return hkdf(sha256, ikm, salt, TEXT.encode(info), length);

View File

@@ -1,17 +1,29 @@
/** /**
* KeyManager — owns the masterKey lifecycle for at-rest encryption. * KeyManager — owns the masterKey lifecycle for at-rest encryption.
* *
* Three sources are supported: * Five sources are supported:
* 1. passphrase — scrypt-derived from a developer-supplied secret * 1. passphrase — scrypt-derived from a developer-supplied secret
* 2. keychain — fetched from OS keychain via @shade/keychain (optional dep) * 2. argon2id — memory-hard KDF for low-entropy secrets (PINs)
* 3. injected — caller supplies the 32-byte raw key directly * 3. keychain — fetched from OS keychain via @shade/keychain (optional dep)
* 4. injected — caller supplies the 32-byte raw key directly
* 5. composite — HKDF-combine N sub-sources into one master key
* (multi-factor unlock — every source is required)
* *
* The KeyManager pre-derives storageKey at construction time and caches the * The KeyManager pre-derives storageKey at construction time and caches the
* per-(table, column) field keys. masterKey is zeroized after storageKey * per-(table, column) field keys. masterKey is zeroized after storageKey
* derivation to limit residency. * derivation to limit residency.
*/ */
import { deriveFieldKey, deriveMasterKey, deriveStorageKey, type ScryptParams, DEFAULT_SCRYPT } from './kdf.js'; import {
deriveFieldKey,
deriveMasterKey,
deriveMasterKeyArgon2id,
deriveStorageKey,
hkdfDerive,
DEFAULT_SCRYPT,
type Argon2idParams,
type ScryptParams,
} from './kdf.js';
export type KeySource = export type KeySource =
| { | {
@@ -21,6 +33,14 @@ export type KeySource =
salt: Uint8Array; salt: Uint8Array;
params?: ScryptParams; params?: ScryptParams;
} }
| {
kind: 'argon2id';
/** Low-entropy secret (PIN, short password). */
secret: string | Uint8Array;
/** Stable 16+ byte salt persisted alongside the DB. */
salt: Uint8Array;
params?: Argon2idParams;
}
| { | {
kind: 'keychain'; kind: 'keychain';
/** Service identifier (e.g. "shade.storage"). */ /** Service identifier (e.g. "shade.storage"). */
@@ -34,6 +54,25 @@ export type KeySource =
kind: 'injected'; kind: 'injected';
/** Raw 32-byte master key. */ /** Raw 32-byte master key. */
key: Uint8Array; key: Uint8Array;
}
| {
kind: 'composite';
/**
* Sub-sources HKDF-combined into the master key, in order. Every
* source is mandatory: omitting or substituting any source yields
* a different master key and the storage open() will fail.
*
* Order is significant by design — `[pwd, pin]` and `[pin, pwd]`
* derive different master keys.
*/
sources: KeySource[];
/**
* HKDF info string for domain separation. Defaults to
* `shade-composite-master-v1`. Apps that want their composite key
* to be cryptographically distinct from any other Shade composite
* should override this with an app-specific string.
*/
info?: string;
}; };
/** Pluggable keychain backend. Implementations live in @shade/keychain. */ /** Pluggable keychain backend. Implementations live in @shade/keychain. */
@@ -107,6 +146,9 @@ async function resolveMasterKey(source: KeySource, opts: KeyManagerOptions): Pro
case 'passphrase': case 'passphrase':
return deriveMasterKey(source.passphrase, source.salt, source.params ?? DEFAULT_SCRYPT); return deriveMasterKey(source.passphrase, source.salt, source.params ?? DEFAULT_SCRYPT);
case 'argon2id':
return deriveMasterKeyArgon2id(source.secret, source.salt, source.params);
case 'injected': case 'injected':
if (source.key.length !== 32) throw new Error('injected key must be exactly 32 bytes'); if (source.key.length !== 32) throw new Error('injected key must be exactly 32 bytes');
return new Uint8Array(source.key); // copy, in case caller mutates return new Uint8Array(source.key); // copy, in case caller mutates
@@ -128,5 +170,39 @@ async function resolveMasterKey(source: KeySource, opts: KeyManagerOptions): Pro
await opts.keychain.set(source.service, source.account, fresh); await opts.keychain.set(source.service, source.account, fresh);
return fresh; return fresh;
} }
case 'composite': {
if (source.sources.length === 0) {
throw new Error('composite source requires at least one sub-source');
}
const subKeys: Uint8Array[] = [];
try {
for (const sub of source.sources) {
if (sub.kind === 'composite') {
// Composite-of-composite would silently flatten the unlock
// semantics ("any of N" vs "all of N") if the inner is misread.
// Forbidding nesting keeps the contract clear.
throw new Error('composite sources cannot be nested');
}
subKeys.push(await resolveMasterKey(sub, opts));
}
let total = 0;
for (const k of subKeys) total += k.length;
const ikm = new Uint8Array(total);
let off = 0;
for (const k of subKeys) {
ikm.set(k, off);
off += k.length;
}
const info = source.info ?? 'shade-composite-master-v1';
try {
return hkdfDerive(ikm, info, 32);
} finally {
ikm.fill(0);
}
} finally {
for (const k of subKeys) k.fill(0);
}
}
} }
} }

View File

@@ -39,6 +39,7 @@ export const COL = {
trustedIdentity: 'trusted_identity', trustedIdentity: 'trusted_identity',
retiredIdentity: 'retired_identity', retiredIdentity: 'retired_identity',
streamSensitive: 'stream_sensitive', streamSensitive: 'stream_sensitive',
broadcastChannelSensitive: 'broadcast_channel_sensitive',
} as const; } as const;
/** Logical table identifiers — used for fieldKey + AAD binding. */ /** Logical table identifiers — used for fieldKey + AAD binding. */
@@ -51,6 +52,7 @@ export const TBL = {
trustedIdentities: 'trusted_identities', trustedIdentities: 'trusted_identities',
retiredIdentities: 'retired_identities', retiredIdentities: 'retired_identities',
streamState: 'stream_state', streamState: 'stream_state',
broadcastChannels: 'broadcast_channels',
} as const; } as const;
/** Encrypt an arbitrary string payload bound to (table, column, pk). */ /** Encrypt an arbitrary string payload bound to (table, column, pk). */
@@ -226,3 +228,76 @@ export async function openStreamSensitive(
if (b.overallHashState !== undefined) (out as { overallHashState?: string }).overallHashState = b.overallHashState; if (b.overallHashState !== undefined) (out as { overallHashState?: string }).overallHashState = b.overallHashState;
return out; return out;
} }
/**
* Broadcast-channel sensitive bundle (V4.6). Routing fields (channelId,
* ownerRole, ownerAddress, label, generation, createdAt, updatedAt) live
* in plaintext columns so backends can list/query without unsealing every
* row; the chain key, iteration, and signing keys all live in this sealed
* blob.
*/
interface BroadcastChannelSensitiveBundle {
chainKey: string; // base64(32B)
iteration: number;
signingPublicKey: string; // base64(32B)
signingPrivateKey?: string; // base64; only when ownerRole === 'sender'
}
export async function sealBroadcastChannelSensitive(
km: KeyManager,
channelId: string,
s: {
chainKey: Uint8Array;
iteration: number;
signingPublicKey: Uint8Array;
signingPrivateKey?: Uint8Array;
},
): Promise<Uint8Array> {
const bundle: BroadcastChannelSensitiveBundle = {
chainKey: toBase64(s.chainKey),
iteration: s.iteration,
signingPublicKey: toBase64(s.signingPublicKey),
};
if (s.signingPrivateKey !== undefined) {
bundle.signingPrivateKey = toBase64(s.signingPrivateKey);
}
return sealString(
km,
TBL.broadcastChannels,
COL.broadcastChannelSensitive,
channelId,
JSON.stringify(bundle),
);
}
export async function openBroadcastChannelSensitive(
km: KeyManager,
channelId: string,
blob: Uint8Array,
): Promise<{
chainKey: Uint8Array;
iteration: number;
signingPublicKey: Uint8Array;
signingPrivateKey?: Uint8Array;
}> {
const json = await openString(
km,
TBL.broadcastChannels,
COL.broadcastChannelSensitive,
channelId,
blob,
);
const b = JSON.parse(json) as BroadcastChannelSensitiveBundle;
const out: {
chainKey: Uint8Array;
iteration: number;
signingPublicKey: Uint8Array;
signingPrivateKey?: Uint8Array;
} = {
chainKey: fromBase64(b.chainKey),
iteration: b.iteration,
signingPublicKey: fromBase64(b.signingPublicKey),
};
if (b.signingPrivateKey !== undefined) out.signingPrivateKey = fromBase64(b.signingPrivateKey);
return out;
}

View File

@@ -0,0 +1,13 @@
/**
* Browser entry point — Encrypted IndexedDB storage.
*
* Pulls `idb` (~12 kB) and SubtleCrypto only. Safe to bundle into a
* browser app via Vite/webpack/esbuild.
*
* ```ts
* import { KeyManager } from '@shade/storage-encrypted/crypto';
* import { EncryptedIndexedDBStorage } from '@shade/storage-encrypted/idb';
* ```
*/
export { EncryptedIndexedDBStorage } from './storage/encrypted-indexeddb.js';

View File

@@ -0,0 +1,14 @@
/**
* Default barrel resolved by bundlers under the `browser` condition.
*
* Excludes `EncryptedSQLiteStorage` (pulls `bun:sqlite`) and
* `EncryptedPostgresStorage` (pulls `postgres`). Browser apps can keep using
* `import { KeyManager } from '@shade/storage-encrypted'` — Vite/webpack will
* pick this file via the `browser` condition and produce a clean bundle.
*
* Node/Bun consumers continue to resolve to `./index.js`, which exposes the
* full surface area (back-compat).
*/
export * from './crypto.js';
export { EncryptedIndexedDBStorage } from './storage/encrypted-indexeddb.js';

View File

@@ -6,8 +6,11 @@ export {
} from './crypto/key-manager.js'; } from './crypto/key-manager.js';
export { export {
DEFAULT_SCRYPT, DEFAULT_SCRYPT,
DEFAULT_ARGON2ID,
type ScryptParams, type ScryptParams,
type Argon2idParams,
deriveMasterKey, deriveMasterKey,
deriveMasterKeyArgon2id,
deriveStorageKey, deriveStorageKey,
deriveFieldKey, deriveFieldKey,
deriveNonce, deriveNonce,
@@ -25,6 +28,9 @@ export {
EncryptedPostgresStorage, EncryptedPostgresStorage,
ensureEncryptedClientTables, ensureEncryptedClientTables,
} from './storage/encrypted-postgres.js'; } from './storage/encrypted-postgres.js';
// EncryptedIndexedDBStorage is browser-only; import it from
// `@shade/storage-encrypted/idb` (or rely on the `browser` condition
// resolving to ./index.browser.ts).
export { export {
migrateSqliteToEncrypted, migrateSqliteToEncrypted,
rotateSqliteEncryptionKey, rotateSqliteEncryptionKey,

View File

@@ -0,0 +1,11 @@
/**
* Node-only entry point — Encrypted Postgres storage.
*
* Pulls `postgres`. Do not import from a browser bundle; use
* `@shade/storage-encrypted/idb` instead.
*/
export {
EncryptedPostgresStorage,
ensureEncryptedClientTables,
} from './storage/encrypted-postgres.js';

View File

@@ -0,0 +1,15 @@
/**
* Bun-only entry point — Encrypted SQLite storage + migration utilities.
*
* Pulls `bun:sqlite`. Do not import from a browser bundle; use
* `@shade/storage-encrypted/idb` instead.
*/
export { EncryptedSQLiteStorage } from './storage/encrypted-sqlite.js';
export {
migrateSqliteToEncrypted,
rotateSqliteEncryptionKey,
type MigrateOptions,
type RotateOptions,
type MigrateReport,
} from './migrate/migrate-sqlite.js';

View File

@@ -0,0 +1,548 @@
import { openDB, type IDBPDatabase, type DBSchema } from 'idb';
import type {
BroadcastChannelRecord,
BroadcastMemberRecord,
IdentityKeyPair,
OneTimePreKey,
PeerVerification,
PeerVerificationSource,
PersistedStreamState,
RetiredIdentity,
SessionState,
SignedPreKey,
StorageProvider,
} from '@shade/core';
import { constantTimeEqual, toBase64 } from '@shade/core';
import { KeyManager } from '../crypto/key-manager.js';
import {
openBroadcastChannelSensitive, openConfig, openIdentity, openOneTimePreKey,
openRetired, openSession, openSignedPreKey, openStreamSensitive, openTrust,
sealBroadcastChannelSensitive, sealConfig, sealIdentity, sealOneTimePreKey,
sealRetired, sealSession, sealSignedPreKey, sealStreamSensitive, sealTrust,
} from '../crypto/row-codec.js';
/**
* IndexedDB-backed StorageProvider with at-rest encryption.
*
* Schema is the IndexedDB equivalent of `EncryptedSQLiteStorage`: one object
* store per `_enc` table, sealed payloads stored as `Uint8Array` in a
* `ciphertext` field, routing/timestamp fields kept plaintext where SQLite
* does so for query efficiency. Crypto stack — `KeyManager`, `aeadSeal`,
* `aeadOpen`, row-codec sealers, AAD scheme — is shared verbatim with the
* SQLite/Postgres backends, so a row sealed under one backend decrypts
* under another given the same `KeyManager`.
*
* Browser-safe: imports `idb` (~12 kB, pure JS) and SubtleCrypto only.
*
* Usage:
* ```ts
* import { KeyManager } from '@shade/storage-encrypted/crypto';
* import { EncryptedIndexedDBStorage } from '@shade/storage-encrypted/idb';
*
* const km = await KeyManager.open({
* kind: 'composite',
* sources: [
* { kind: 'passphrase', passphrase, salt: pwSalt },
* { kind: 'argon2id', secret: pin, salt: pinSalt },
* ],
* });
* const storage = await EncryptedIndexedDBStorage.open({
* dbName: 'my-app-shade',
* keyManager: km,
* });
* ```
*/
export class EncryptedIndexedDBStorage implements StorageProvider {
private constructor(
private db: IDBPDatabase<EncryptedShadeSchema>,
private km: KeyManager,
) {}
/**
* Open (or create) the encrypted IndexedDB database. On first open the
* storageKey fingerprint is persisted; subsequent opens with a different
* KeyManager (wrong passphrase / PIN) reject with a clear error rather
* than silently writing data under the wrong key.
*/
static async open(opts: {
dbName?: string;
keyManager: KeyManager;
}): Promise<EncryptedIndexedDBStorage> {
const dbName = opts.dbName ?? 'shade-encrypted';
const db = await openDB<EncryptedShadeSchema>(dbName, SCHEMA_VERSION, {
upgrade(db, oldVersion) {
if (oldVersion < 1) {
db.createObjectStore('meta_enc', { keyPath: 'key' });
db.createObjectStore('identity_enc', { keyPath: 'id' });
db.createObjectStore('config_enc', { keyPath: 'key' });
db.createObjectStore('signed_prekeys_enc', { keyPath: 'keyId' });
db.createObjectStore('one_time_prekeys_enc', { keyPath: 'keyId' });
db.createObjectStore('sessions_enc', { keyPath: 'address' });
db.createObjectStore('trusted_identities_enc', { keyPath: 'address' });
const retired = db.createObjectStore('retired_identities_enc', {
keyPath: 'retiredAt',
});
retired.createIndex('byRetiredAt', 'retiredAt');
const stream = db.createObjectStore('stream_state_enc', { keyPath: 'streamId' });
stream.createIndex('byStatus', 'status');
stream.createIndex('byPeerAddress', 'peerAddress');
stream.createIndex('byUpdatedAt', 'updatedAt');
db.createObjectStore('peer_verifications_enc', { keyPath: 'peerAddress' });
db.createObjectStore('peer_identity_versions_enc', { keyPath: 'peerAddress' });
}
if (oldVersion < 2) {
db.createObjectStore('broadcast_channels_enc', { keyPath: 'channelId' });
const members = db.createObjectStore('broadcast_members_enc', {
keyPath: ['channelId', 'peerAddress'],
});
members.createIndex('byChannelId', 'channelId');
}
},
});
const store = new EncryptedIndexedDBStorage(db, opts.keyManager);
await store.assertKeyMatchesOrPersistFingerprint();
return store;
}
/** Cleanly close the underlying connection. KeyManager is destroyed. */
close(): void {
this.db.close();
this.km.destroy();
}
/**
* On first open, persist a fingerprint of the storageKey. On subsequent
* opens, compare and reject mismatches with a clear error rather than
* silently writing data under the wrong key.
*/
private async assertKeyMatchesOrPersistFingerprint(): Promise<void> {
const expected = toBase64(this.km.storageKeyFingerprint());
const row = await this.db.get('meta_enc', 'storage_key_fingerprint');
if (!row) {
await this.db.put('meta_enc', { key: 'storage_key_fingerprint', value: expected });
return;
}
if (row.value !== expected) {
throw new Error(
'storage key mismatch — the supplied passphrase / keychain entry does not unlock this database',
);
}
}
// ─── Identity ──────────────────────────────────────────────
async getIdentityKeyPair(): Promise<IdentityKeyPair | null> {
const row = await this.db.get('identity_enc', 1);
if (!row) return null;
return openIdentity(this.km, row.ciphertext);
}
async saveIdentityKeyPair(kp: IdentityKeyPair): Promise<void> {
const blob = await sealIdentity(this.km, kp);
await this.db.put('identity_enc', { id: 1, ciphertext: blob });
}
async getLocalRegistrationId(): Promise<number> {
const row = await this.db.get('config_enc', 'registrationId');
if (!row) return 0;
const v = await openConfig(this.km, 'registrationId', row.ciphertext);
return parseInt(v, 10);
}
async saveLocalRegistrationId(id: number): Promise<void> {
const blob = await sealConfig(this.km, 'registrationId', String(id));
await this.db.put('config_enc', { key: 'registrationId', ciphertext: blob });
}
// ─── Signed PreKeys ────────────────────────────────────────
async getSignedPreKey(keyId: number): Promise<SignedPreKey | null> {
const row = await this.db.get('signed_prekeys_enc', keyId);
if (!row) return null;
return openSignedPreKey(this.km, keyId, row.ciphertext);
}
async saveSignedPreKey(key: SignedPreKey): Promise<void> {
const blob = await sealSignedPreKey(this.km, key);
await this.db.put('signed_prekeys_enc', { keyId: key.keyId, ciphertext: blob });
}
async removeSignedPreKey(keyId: number): Promise<void> {
await this.db.delete('signed_prekeys_enc', keyId);
}
// ─── One-Time PreKeys ──────────────────────────────────────
async getOneTimePreKey(keyId: number): Promise<OneTimePreKey | null> {
const row = await this.db.get('one_time_prekeys_enc', keyId);
if (!row) return null;
return openOneTimePreKey(this.km, keyId, row.ciphertext);
}
async saveOneTimePreKey(key: OneTimePreKey): Promise<void> {
const blob = await sealOneTimePreKey(this.km, key);
await this.db.put('one_time_prekeys_enc', { keyId: key.keyId, ciphertext: blob });
}
async removeOneTimePreKey(keyId: number): Promise<void> {
await this.db.delete('one_time_prekeys_enc', keyId);
}
async getOneTimePreKeyCount(): Promise<number> {
return this.db.count('one_time_prekeys_enc');
}
// ─── Sessions ──────────────────────────────────────────────
async getSession(address: string): Promise<SessionState | null> {
const row = await this.db.get('sessions_enc', address);
if (!row) return null;
return openSession(this.km, address, row.ciphertext);
}
async saveSession(address: string, state: SessionState): Promise<void> {
const blob = await sealSession(this.km, address, state);
await this.db.put('sessions_enc', { address, ciphertext: blob });
}
async removeSession(address: string): Promise<void> {
await this.db.delete('sessions_enc', address);
}
// ─── Trust ─────────────────────────────────────────────────
async isTrustedIdentity(address: string, identityKey: Uint8Array): Promise<boolean> {
const row = await this.db.get('trusted_identities_enc', address);
if (!row) return true; // TOFU
const stored = await openTrust(this.km, address, row.ciphertext);
return constantTimeEqual(stored, identityKey);
}
async saveTrustedIdentity(address: string, identityKey: Uint8Array): Promise<void> {
const blob = await sealTrust(this.km, address, identityKey);
await this.db.put('trusted_identities_enc', { address, ciphertext: blob });
}
// ─── Identity History ──────────────────────────────────────
async addRetiredIdentity(identity: RetiredIdentity): Promise<void> {
const blob = await sealRetired(this.km, identity);
await this.db.put('retired_identities_enc', {
retiredAt: identity.retiredAt,
ciphertext: blob,
});
}
async getRetiredIdentities(): Promise<RetiredIdentity[]> {
// Mirror SQLite's `ORDER BY retired_at DESC`
const rows = await this.db.getAllFromIndex('retired_identities_enc', 'byRetiredAt');
rows.reverse();
return Promise.all(
rows.map((r) => openRetired(this.km, r.retiredAt, r.ciphertext)),
);
}
async pruneRetiredIdentities(olderThan: number): Promise<void> {
const tx = this.db.transaction('retired_identities_enc', 'readwrite');
const idx = tx.store.index('byRetiredAt');
const range = IDBKeyRange.upperBound(olderThan, true);
let cursor = await idx.openCursor(range);
while (cursor) {
await cursor.delete();
cursor = await cursor.continue();
}
await tx.done;
}
// ─── Stream-transfer resume state ──────────────────────────
async saveStreamState(state: PersistedStreamState): Promise<void> {
const blob = await sealStreamSensitive(this.km, state);
await this.db.put('stream_state_enc', {
streamId: state.streamId,
direction: state.direction,
peerAddress: state.peerAddress,
status: state.status,
ciphertext: blob,
createdAt: state.createdAt,
updatedAt: state.updatedAt,
});
}
async getStreamState(streamId: string): Promise<PersistedStreamState | null> {
const row = await this.db.get('stream_state_enc', streamId);
if (!row) return null;
return this.rowToStreamState(row);
}
async removeStreamState(streamId: string): Promise<void> {
await this.db.delete('stream_state_enc', streamId);
}
async listActiveStreamStates(direction?: 'send' | 'receive'): Promise<PersistedStreamState[]> {
const tx = this.db.transaction('stream_state_enc');
const idx = tx.store.index('byStatus');
const active = await idx.getAll(IDBKeyRange.only('active'));
const paused = await idx.getAll(IDBKeyRange.only('paused'));
const merged = [...active, ...paused];
const filtered = direction === undefined
? merged
: merged.filter((r) => r.direction === direction);
filtered.sort((a, b) => b.updatedAt - a.updatedAt);
return Promise.all(filtered.map((r) => this.rowToStreamState(r)));
}
async pruneStreamStates(olderThan: number): Promise<void> {
const tx = this.db.transaction('stream_state_enc', 'readwrite');
const idx = tx.store.index('byUpdatedAt');
const range = IDBKeyRange.upperBound(olderThan, true);
let cursor = await idx.openCursor(range);
while (cursor) {
const row = cursor.value;
if (row.status === 'finished' || row.status === 'aborted') {
await cursor.delete();
}
cursor = await cursor.continue();
}
await tx.done;
}
// ─── Peer verifications (V3.3) ────────────────────────────
// Fingerprints are public-by-design; stored in plaintext for symmetry
// with the SQLite/Postgres encrypted backends.
async savePeerVerification(v: PeerVerification): Promise<void> {
await this.db.put('peer_verifications_enc', { ...v });
}
async getPeerVerification(address: string): Promise<PeerVerification | null> {
const row = await this.db.get('peer_verifications_enc', address);
if (!row) return null;
return {
peerAddress: row.peerAddress,
fingerprint: row.fingerprint,
verifiedAt: row.verifiedAt,
verifiedBy: row.verifiedBy as PeerVerificationSource,
identityVersion: row.identityVersion,
};
}
async removePeerVerification(address: string): Promise<void> {
await this.db.delete('peer_verifications_enc', address);
}
async getPeerIdentityVersion(address: string): Promise<number> {
const row = await this.db.get('peer_identity_versions_enc', address);
return row ? row.version : 1;
}
async bumpPeerIdentityVersion(address: string): Promise<number> {
// Atomic read-modify-write under one IDB transaction. Closes the race
// that exists in the SQLite version's non-atomic read-then-upsert.
const tx = this.db.transaction('peer_identity_versions_enc', 'readwrite');
const existing = await tx.store.get(address);
const next = (existing ? existing.version : 1) + 1;
await tx.store.put({ peerAddress: address, version: next });
await tx.done;
return next;
}
// ─── Broadcast channels (V4.6) ────────────────────────────
async saveBroadcastChannel(channel: BroadcastChannelRecord): Promise<void> {
const sealed = await sealBroadcastChannelSensitive(this.km, channel.channelId, {
chainKey: channel.chainKey,
iteration: channel.iteration,
signingPublicKey: channel.signingPublicKey,
...(channel.signingPrivateKey !== undefined ? { signingPrivateKey: channel.signingPrivateKey } : {}),
});
await this.db.put('broadcast_channels_enc', {
channelId: channel.channelId,
ownerRole: channel.ownerRole,
ownerAddress: channel.ownerAddress,
label: channel.label ?? null,
generation: channel.generation,
ciphertext: sealed,
createdAt: channel.createdAt,
updatedAt: channel.updatedAt,
});
}
async getBroadcastChannel(channelId: string): Promise<BroadcastChannelRecord | null> {
const row = await this.db.get('broadcast_channels_enc', channelId);
if (!row) return null;
return this.encRowToChannel(row);
}
async listBroadcastChannels(): Promise<BroadcastChannelRecord[]> {
const rows = await this.db.getAll('broadcast_channels_enc');
rows.sort((a, b) => a.createdAt - b.createdAt);
return Promise.all(rows.map((r) => this.encRowToChannel(r)));
}
async removeBroadcastChannel(channelId: string): Promise<void> {
const tx = this.db.transaction(
['broadcast_channels_enc', 'broadcast_members_enc'],
'readwrite',
);
const memIdx = tx.objectStore('broadcast_members_enc').index('byChannelId');
let cursor = await memIdx.openCursor(IDBKeyRange.only(channelId));
while (cursor) {
await cursor.delete();
cursor = await cursor.continue();
}
await tx.objectStore('broadcast_channels_enc').delete(channelId);
await tx.done;
}
async saveBroadcastMember(member: BroadcastMemberRecord): Promise<void> {
await this.db.put('broadcast_members_enc', { ...member });
}
async getBroadcastMembers(channelId: string): Promise<BroadcastMemberRecord[]> {
const rows = await this.db.getAllFromIndex(
'broadcast_members_enc',
'byChannelId',
IDBKeyRange.only(channelId),
);
rows.sort((a, b) => a.joinedAt - b.joinedAt);
return rows.map((r) => ({
channelId: r.channelId,
peerAddress: r.peerAddress,
joinedAt: r.joinedAt,
removedAt: r.removedAt,
}));
}
async removeBroadcastMember(channelId: string, peerAddress: string): Promise<void> {
await this.db.delete('broadcast_members_enc', [channelId, peerAddress]);
}
private async encRowToChannel(row: BroadcastChannelEncRow): Promise<BroadcastChannelRecord> {
const sensitive = await openBroadcastChannelSensitive(
this.km,
row.channelId,
row.ciphertext,
);
const out: BroadcastChannelRecord = {
channelId: row.channelId,
ownerRole: row.ownerRole,
ownerAddress: row.ownerAddress,
generation: row.generation,
chainKey: sensitive.chainKey,
iteration: sensitive.iteration,
signingPublicKey: sensitive.signingPublicKey,
createdAt: row.createdAt,
updatedAt: row.updatedAt,
};
if (row.label !== null) out.label = row.label;
if (sensitive.signingPrivateKey !== undefined) out.signingPrivateKey = sensitive.signingPrivateKey;
return out;
}
private async rowToStreamState(row: StreamStateEncRow): Promise<PersistedStreamState> {
const sensitive = await openStreamSensitive(this.km, row.streamId, row.ciphertext);
const out: PersistedStreamState = {
streamId: row.streamId,
direction: row.direction,
peerAddress: row.peerAddress,
status: row.status,
metadataJson: sensitive.metadataJson,
partitionJson: sensitive.partitionJson,
laneStateJson: sensitive.laneStateJson,
ioDescriptorJson: sensitive.ioDescriptorJson,
secretEnc: sensitive.secretEnc,
secretNonce: sensitive.secretNonce,
createdAt: row.createdAt,
updatedAt: row.updatedAt,
};
if (sensitive.overallHashState !== undefined) out.overallHashState = sensitive.overallHashState;
return out;
}
}
// ─── Schema ────────────────────────────────────────────────
const SCHEMA_VERSION = 2;
interface MetaRow { key: string; value: string }
interface IdentityRow { id: 1; ciphertext: Uint8Array }
interface ConfigRow { key: string; ciphertext: Uint8Array }
interface SignedPreKeyRow { keyId: number; ciphertext: Uint8Array }
interface OneTimePreKeyRow { keyId: number; ciphertext: Uint8Array }
interface SessionRow { address: string; ciphertext: Uint8Array }
interface TrustedIdentityRow { address: string; ciphertext: Uint8Array }
interface RetiredIdentityRow { retiredAt: number; ciphertext: Uint8Array }
interface StreamStateEncRow {
streamId: string;
direction: 'send' | 'receive';
peerAddress: string;
status: 'active' | 'paused' | 'finished' | 'aborted';
ciphertext: Uint8Array;
createdAt: number;
updatedAt: number;
}
interface PeerVerificationRow {
peerAddress: string;
fingerprint: string;
verifiedAt: number;
verifiedBy: string;
identityVersion: number;
}
interface PeerIdentityVersionRow { peerAddress: string; version: number }
interface BroadcastChannelEncRow {
channelId: string;
ownerRole: 'sender' | 'receiver';
ownerAddress: string;
label: string | null;
generation: number;
ciphertext: Uint8Array;
createdAt: number;
updatedAt: number;
}
interface BroadcastMemberEncRow {
channelId: string;
peerAddress: string;
joinedAt: number;
removedAt: number | null;
}
interface EncryptedShadeSchema extends DBSchema {
meta_enc: { key: string; value: MetaRow };
identity_enc: { key: number; value: IdentityRow };
config_enc: { key: string; value: ConfigRow };
signed_prekeys_enc: { key: number; value: SignedPreKeyRow };
one_time_prekeys_enc: { key: number; value: OneTimePreKeyRow };
sessions_enc: { key: string; value: SessionRow };
trusted_identities_enc: { key: string; value: TrustedIdentityRow };
retired_identities_enc: {
key: number;
value: RetiredIdentityRow;
indexes: { byRetiredAt: number };
};
stream_state_enc: {
key: string;
value: StreamStateEncRow;
indexes: {
byStatus: string;
byPeerAddress: string;
byUpdatedAt: number;
};
};
peer_verifications_enc: { key: string; value: PeerVerificationRow };
peer_identity_versions_enc: { key: string; value: PeerIdentityVersionRow };
broadcast_channels_enc: { key: string; value: BroadcastChannelEncRow };
broadcast_members_enc: {
key: [string, string];
value: BroadcastMemberEncRow;
indexes: { byChannelId: string };
};
}

View File

@@ -1,6 +1,8 @@
import type { Sql } from 'postgres'; import type { Sql } from 'postgres';
import postgres from 'postgres'; import postgres from 'postgres';
import type { import type {
BroadcastChannelRecord,
BroadcastMemberRecord,
IdentityKeyPair, IdentityKeyPair,
OneTimePreKey, OneTimePreKey,
PeerVerification, PeerVerification,
@@ -14,10 +16,10 @@ import type {
import { constantTimeEqual, toBase64 } from '@shade/core'; import { constantTimeEqual, toBase64 } from '@shade/core';
import { KeyManager } from '../crypto/key-manager.js'; import { KeyManager } from '../crypto/key-manager.js';
import { import {
openConfig, openIdentity, openOneTimePreKey, openRetired, openSession, openBroadcastChannelSensitive, openConfig, openIdentity, openOneTimePreKey,
openSignedPreKey, openStreamSensitive, openTrust, sealConfig, sealIdentity, openRetired, openSession, openSignedPreKey, openStreamSensitive, openTrust,
sealOneTimePreKey, sealRetired, sealSession, sealSignedPreKey, sealBroadcastChannelSensitive, sealConfig, sealIdentity, sealOneTimePreKey,
sealStreamSensitive, sealTrust, sealRetired, sealSession, sealSignedPreKey, sealStreamSensitive, sealTrust,
} from '../crypto/row-codec.js'; } from '../crypto/row-codec.js';
/** /**
@@ -332,6 +334,108 @@ export class EncryptedPostgresStorage implements StorageProvider {
return next; return next;
} }
// ─── Broadcast channels (V4.6) ────────────────────────────
async saveBroadcastChannel(channel: BroadcastChannelRecord): Promise<void> {
const sealed = await sealBroadcastChannelSensitive(this.km, channel.channelId, {
chainKey: channel.chainKey,
iteration: channel.iteration,
signingPublicKey: channel.signingPublicKey,
...(channel.signingPrivateKey !== undefined ? { signingPrivateKey: channel.signingPrivateKey } : {}),
});
await this.sql`
INSERT INTO shade_broadcast_channels_enc
(channel_id, owner_role, owner_address, label, generation, ciphertext, created_at, updated_at)
VALUES
(${channel.channelId}, ${channel.ownerRole}, ${channel.ownerAddress},
${channel.label ?? null}, ${channel.generation}, ${sealed},
${channel.createdAt}, ${channel.updatedAt})
ON CONFLICT (channel_id) DO UPDATE SET
owner_role = EXCLUDED.owner_role,
owner_address = EXCLUDED.owner_address,
label = EXCLUDED.label,
generation = EXCLUDED.generation,
ciphertext = EXCLUDED.ciphertext,
updated_at = EXCLUDED.updated_at
`;
}
async getBroadcastChannel(channelId: string): Promise<BroadcastChannelRecord | null> {
const rows = await this.sql<Array<BroadcastChannelEncRow>>`
SELECT * FROM shade_broadcast_channels_enc WHERE channel_id = ${channelId}
`;
if (rows.length === 0) return null;
return this.encRowToChannel(rows[0]!);
}
async listBroadcastChannels(): Promise<BroadcastChannelRecord[]> {
const rows = await this.sql<Array<BroadcastChannelEncRow>>`
SELECT * FROM shade_broadcast_channels_enc ORDER BY created_at ASC
`;
return Promise.all(rows.map((r) => this.encRowToChannel(r)));
}
async removeBroadcastChannel(channelId: string): Promise<void> {
await this.sql`DELETE FROM shade_broadcast_members_enc WHERE channel_id = ${channelId}`;
await this.sql`DELETE FROM shade_broadcast_channels_enc WHERE channel_id = ${channelId}`;
}
async saveBroadcastMember(member: BroadcastMemberRecord): Promise<void> {
await this.sql`
INSERT INTO shade_broadcast_members_enc
(channel_id, peer_address, joined_at, removed_at)
VALUES
(${member.channelId}, ${member.peerAddress}, ${member.joinedAt}, ${member.removedAt})
ON CONFLICT (channel_id, peer_address) DO UPDATE SET
joined_at = EXCLUDED.joined_at,
removed_at = EXCLUDED.removed_at
`;
}
async getBroadcastMembers(channelId: string): Promise<BroadcastMemberRecord[]> {
const rows = await this.sql<Array<{ channel_id: string; peer_address: string; joined_at: string | number; removed_at: string | number | null }>>`
SELECT channel_id, peer_address, joined_at, removed_at
FROM shade_broadcast_members_enc
WHERE channel_id = ${channelId}
ORDER BY joined_at ASC
`;
return rows.map((r) => ({
channelId: r.channel_id,
peerAddress: r.peer_address,
joinedAt: Number(r.joined_at),
removedAt: r.removed_at === null ? null : Number(r.removed_at),
}));
}
async removeBroadcastMember(channelId: string, peerAddress: string): Promise<void> {
await this.sql`
DELETE FROM shade_broadcast_members_enc
WHERE channel_id = ${channelId} AND peer_address = ${peerAddress}
`;
}
private async encRowToChannel(row: BroadcastChannelEncRow): Promise<BroadcastChannelRecord> {
const sensitive = await openBroadcastChannelSensitive(
this.km,
String(row.channel_id),
row.ciphertext,
);
const out: BroadcastChannelRecord = {
channelId: String(row.channel_id),
ownerRole: row.owner_role,
ownerAddress: String(row.owner_address),
generation: Number(row.generation),
chainKey: sensitive.chainKey,
iteration: sensitive.iteration,
signingPublicKey: sensitive.signingPublicKey,
createdAt: Number(row.created_at),
updatedAt: Number(row.updated_at),
};
if (row.label !== null && row.label !== undefined) out.label = row.label;
if (sensitive.signingPrivateKey !== undefined) out.signingPrivateKey = sensitive.signingPrivateKey;
return out;
}
private async rowToStreamState(row: StreamRow): Promise<PersistedStreamState> { private async rowToStreamState(row: StreamRow): Promise<PersistedStreamState> {
const sensitive = await openStreamSensitive(this.km, String(row.stream_id), row.ciphertext); const sensitive = await openStreamSensitive(this.km, String(row.stream_id), row.ciphertext);
const out: PersistedStreamState = { const out: PersistedStreamState = {
@@ -363,6 +467,17 @@ interface StreamRow {
updated_at: string | number; updated_at: string | number;
} }
interface BroadcastChannelEncRow {
channel_id: string;
owner_role: 'sender' | 'receiver';
owner_address: string;
label: string | null;
generation: string | number;
ciphertext: Uint8Array;
created_at: string | number;
updated_at: string | number;
}
export async function ensureEncryptedClientTables(sql: Sql): Promise<void> { export async function ensureEncryptedClientTables(sql: Sql): Promise<void> {
await sql` await sql`
CREATE TABLE IF NOT EXISTS shade_meta_enc ( CREATE TABLE IF NOT EXISTS shade_meta_enc (
@@ -454,4 +569,29 @@ export async function ensureEncryptedClientTables(sql: Sql): Promise<void> {
version BIGINT NOT NULL version BIGINT NOT NULL
) )
`; `;
await sql`
CREATE TABLE IF NOT EXISTS shade_broadcast_channels_enc (
channel_id TEXT PRIMARY KEY,
owner_role TEXT NOT NULL CHECK (owner_role IN ('sender','receiver')),
owner_address TEXT NOT NULL,
label TEXT,
generation BIGINT NOT NULL,
ciphertext BYTEA NOT NULL,
created_at BIGINT NOT NULL,
updated_at BIGINT NOT NULL
)
`;
await sql`
CREATE TABLE IF NOT EXISTS shade_broadcast_members_enc (
channel_id TEXT NOT NULL,
peer_address TEXT NOT NULL,
joined_at BIGINT NOT NULL,
removed_at BIGINT,
PRIMARY KEY (channel_id, peer_address)
)
`;
await sql`
CREATE INDEX IF NOT EXISTS shade_broadcast_members_enc_channel_idx
ON shade_broadcast_members_enc(channel_id)
`;
} }

View File

@@ -1,5 +1,7 @@
import { Database } from 'bun:sqlite'; import { Database } from 'bun:sqlite';
import type { import type {
BroadcastChannelRecord,
BroadcastMemberRecord,
IdentityKeyPair, IdentityKeyPair,
OneTimePreKey, OneTimePreKey,
PeerVerification, PeerVerification,
@@ -13,10 +15,10 @@ import type {
import { constantTimeEqual, toBase64 } from '@shade/core'; import { constantTimeEqual, toBase64 } from '@shade/core';
import { KeyManager } from '../crypto/key-manager.js'; import { KeyManager } from '../crypto/key-manager.js';
import { import {
openConfig, openIdentity, openOneTimePreKey, openRetired, openSession, openBroadcastChannelSensitive, openConfig, openIdentity, openOneTimePreKey, openRetired,
openSignedPreKey, openStreamSensitive, openTrust, sealConfig, sealIdentity, openSession, openSignedPreKey, openStreamSensitive, openTrust,
sealOneTimePreKey, sealRetired, sealSession, sealSignedPreKey, sealBroadcastChannelSensitive, sealConfig, sealIdentity, sealOneTimePreKey,
sealStreamSensitive, sealTrust, sealRetired, sealSession, sealSignedPreKey, sealStreamSensitive, sealTrust,
} from '../crypto/row-codec.js'; } from '../crypto/row-codec.js';
/** /**
@@ -68,6 +70,14 @@ export class EncryptedSQLiteStorage implements StorageProvider {
removePeerVerification: ReturnType<Database['prepare']>; removePeerVerification: ReturnType<Database['prepare']>;
getPeerIdentityVersion: ReturnType<Database['prepare']>; getPeerIdentityVersion: ReturnType<Database['prepare']>;
upsertPeerIdentityVersion: ReturnType<Database['prepare']>; upsertPeerIdentityVersion: ReturnType<Database['prepare']>;
saveBroadcastChannel: ReturnType<Database['prepare']>;
getBroadcastChannel: ReturnType<Database['prepare']>;
listBroadcastChannels: ReturnType<Database['prepare']>;
removeBroadcastChannel: ReturnType<Database['prepare']>;
removeBroadcastChannelMembers: ReturnType<Database['prepare']>;
saveBroadcastMember: ReturnType<Database['prepare']>;
getBroadcastMembers: ReturnType<Database['prepare']>;
removeBroadcastMember: ReturnType<Database['prepare']>;
}; };
private constructor(db: Database, km: KeyManager, ownsDb: boolean) { private constructor(db: Database, km: KeyManager, ownsDb: boolean) {
@@ -156,6 +166,24 @@ export class EncryptedSQLiteStorage implements StorageProvider {
peer_address TEXT PRIMARY KEY, peer_address TEXT PRIMARY KEY,
version INTEGER NOT NULL version INTEGER NOT NULL
); );
CREATE TABLE IF NOT EXISTS broadcast_channels_enc (
channel_id TEXT PRIMARY KEY,
owner_role TEXT NOT NULL,
owner_address TEXT NOT NULL,
label TEXT,
generation INTEGER NOT NULL,
ciphertext BLOB NOT NULL,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS broadcast_members_enc (
channel_id TEXT NOT NULL,
peer_address TEXT NOT NULL,
joined_at INTEGER NOT NULL,
removed_at INTEGER,
PRIMARY KEY (channel_id, peer_address)
);
CREATE INDEX IF NOT EXISTS idx_broadcast_members_enc_channel ON broadcast_members_enc(channel_id);
`); `);
} }
@@ -212,6 +240,35 @@ export class EncryptedSQLiteStorage implements StorageProvider {
`INSERT INTO peer_identity_versions_enc (peer_address, version) VALUES (?, ?) `INSERT INTO peer_identity_versions_enc (peer_address, version) VALUES (?, ?)
ON CONFLICT(peer_address) DO UPDATE SET version = excluded.version`, ON CONFLICT(peer_address) DO UPDATE SET version = excluded.version`,
), ),
saveBroadcastChannel: this.db.prepare(
`INSERT OR REPLACE INTO broadcast_channels_enc
(channel_id, owner_role, owner_address, label, generation,
ciphertext, created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
),
getBroadcastChannel: this.db.prepare(
'SELECT * FROM broadcast_channels_enc WHERE channel_id = ?',
),
listBroadcastChannels: this.db.prepare(
'SELECT * FROM broadcast_channels_enc ORDER BY created_at ASC',
),
removeBroadcastChannel: this.db.prepare(
'DELETE FROM broadcast_channels_enc WHERE channel_id = ?',
),
removeBroadcastChannelMembers: this.db.prepare(
'DELETE FROM broadcast_members_enc WHERE channel_id = ?',
),
saveBroadcastMember: this.db.prepare(
`INSERT OR REPLACE INTO broadcast_members_enc
(channel_id, peer_address, joined_at, removed_at)
VALUES (?, ?, ?, ?)`,
),
getBroadcastMembers: this.db.prepare(
'SELECT channel_id, peer_address, joined_at, removed_at FROM broadcast_members_enc WHERE channel_id = ? ORDER BY joined_at ASC',
),
removeBroadcastMember: this.db.prepare(
'DELETE FROM broadcast_members_enc WHERE channel_id = ? AND peer_address = ?',
),
}; };
} }
@@ -432,6 +489,88 @@ export class EncryptedSQLiteStorage implements StorageProvider {
return next; return next;
} }
// ─── Broadcast channels (V4.6) ────────────────────────────
async saveBroadcastChannel(channel: BroadcastChannelRecord): Promise<void> {
const sealed = await sealBroadcastChannelSensitive(this.km, channel.channelId, {
chainKey: channel.chainKey,
iteration: channel.iteration,
signingPublicKey: channel.signingPublicKey,
...(channel.signingPrivateKey !== undefined ? { signingPrivateKey: channel.signingPrivateKey } : {}),
});
this.stmts.saveBroadcastChannel.run(
channel.channelId,
channel.ownerRole,
channel.ownerAddress,
channel.label ?? null,
channel.generation,
sealed,
channel.createdAt,
channel.updatedAt,
);
}
async getBroadcastChannel(channelId: string): Promise<BroadcastChannelRecord | null> {
const row = this.stmts.getBroadcastChannel.get(channelId) as BroadcastChannelEncRow | undefined;
if (!row) return null;
return this.encRowToChannel(row);
}
async listBroadcastChannels(): Promise<BroadcastChannelRecord[]> {
const rows = this.stmts.listBroadcastChannels.all() as BroadcastChannelEncRow[];
return Promise.all(rows.map((r) => this.encRowToChannel(r)));
}
async removeBroadcastChannel(channelId: string): Promise<void> {
this.stmts.removeBroadcastChannelMembers.run(channelId);
this.stmts.removeBroadcastChannel.run(channelId);
}
async saveBroadcastMember(member: BroadcastMemberRecord): Promise<void> {
this.stmts.saveBroadcastMember.run(
member.channelId,
member.peerAddress,
member.joinedAt,
member.removedAt,
);
}
async getBroadcastMembers(channelId: string): Promise<BroadcastMemberRecord[]> {
const rows = this.stmts.getBroadcastMembers.all(channelId) as BroadcastMemberEncRow[];
return rows.map((r) => ({
channelId: r.channel_id,
peerAddress: r.peer_address,
joinedAt: Number(r.joined_at),
removedAt: r.removed_at === null || r.removed_at === undefined ? null : Number(r.removed_at),
}));
}
async removeBroadcastMember(channelId: string, peerAddress: string): Promise<void> {
this.stmts.removeBroadcastMember.run(channelId, peerAddress);
}
private async encRowToChannel(row: BroadcastChannelEncRow): Promise<BroadcastChannelRecord> {
const sensitive = await openBroadcastChannelSensitive(
this.km,
row.channel_id,
toBytes(row.ciphertext),
);
const out: BroadcastChannelRecord = {
channelId: row.channel_id,
ownerRole: row.owner_role,
ownerAddress: row.owner_address,
generation: Number(row.generation),
chainKey: sensitive.chainKey,
iteration: sensitive.iteration,
signingPublicKey: sensitive.signingPublicKey,
createdAt: Number(row.created_at),
updatedAt: Number(row.updated_at),
};
if (row.label !== null && row.label !== undefined) out.label = row.label;
if (sensitive.signingPrivateKey !== undefined) out.signingPrivateKey = sensitive.signingPrivateKey;
return out;
}
private async rowToStreamState(row: StreamRow): Promise<PersistedStreamState> { private async rowToStreamState(row: StreamRow): Promise<PersistedStreamState> {
const sensitive = await openStreamSensitive(this.km, row.stream_id, toBytes(row.ciphertext)); const sensitive = await openStreamSensitive(this.km, row.stream_id, toBytes(row.ciphertext));
const out: PersistedStreamState = { const out: PersistedStreamState = {
@@ -463,6 +602,24 @@ interface StreamRow {
updated_at: number | bigint; updated_at: number | bigint;
} }
interface BroadcastChannelEncRow {
channel_id: string;
owner_role: 'sender' | 'receiver';
owner_address: string;
label: string | null;
generation: number | bigint;
ciphertext: Uint8Array | ArrayBuffer;
created_at: number | bigint;
updated_at: number | bigint;
}
interface BroadcastMemberEncRow {
channel_id: string;
peer_address: string;
joined_at: number | bigint;
removed_at: number | bigint | null;
}
function toBytes(value: Uint8Array | ArrayBuffer | unknown): Uint8Array { function toBytes(value: Uint8Array | ArrayBuffer | unknown): Uint8Array {
if (value instanceof Uint8Array) return value; if (value instanceof Uint8Array) return value;
if (value instanceof ArrayBuffer) return new Uint8Array(value); if (value instanceof ArrayBuffer) return new Uint8Array(value);

View File

@@ -0,0 +1,292 @@
import 'fake-indexeddb/auto';
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { tmpdir } from 'os';
import { join } from 'path';
import { unlinkSync } from 'fs';
import { EncryptedIndexedDBStorage } from '../src/storage/encrypted-indexeddb.js';
import { EncryptedSQLiteStorage } from '../src/storage/encrypted-sqlite.js';
import { KeyManager } from '../src/crypto/key-manager.js';
import type {
IdentityKeyPair,
OneTimePreKey,
PersistedStreamState,
SessionState,
SignedPreKey,
} from '@shade/core';
function randBytes(n: number): Uint8Array {
const b = new Uint8Array(n);
globalThis.crypto.getRandomValues(b);
return b;
}
function dummyIdentity(): IdentityKeyPair {
return {
signingPublicKey: randBytes(32),
signingPrivateKey: randBytes(32),
dhPublicKey: randBytes(32),
dhPrivateKey: randBytes(32),
};
}
function dummySignedPreKey(id: number): SignedPreKey {
return {
keyId: id,
keyPair: { publicKey: randBytes(32), privateKey: randBytes(32) },
signature: randBytes(64),
timestamp: Date.now(),
};
}
function dummyOTP(id: number): OneTimePreKey {
return {
keyId: id,
keyPair: { publicKey: randBytes(32), privateKey: randBytes(32) },
};
}
function dummySession(): SessionState {
return {
remoteIdentityKey: randBytes(32),
rootKey: randBytes(32),
sendChain: { chainKey: randBytes(32), counter: 0 },
receiveChain: { chainKey: randBytes(32), counter: 0 },
dhSend: { publicKey: randBytes(32), privateKey: randBytes(32) },
dhReceive: randBytes(32),
previousSendCounter: 0,
skippedKeys: new Map(),
};
}
function uniqueDbName(): string {
return `shade-enc-idb-${Date.now()}-${Math.random().toString(36).slice(2)}`;
}
async function deleteDb(name: string): Promise<void> {
await new Promise<void>((resolve) => {
const req = indexedDB.deleteDatabase(name);
req.onsuccess = () => resolve();
req.onerror = () => resolve();
req.onblocked = () => resolve();
});
}
const KEY_BYTES = randBytes(32);
async function freshKM(): Promise<KeyManager> {
return KeyManager.open({ kind: 'injected', key: KEY_BYTES });
}
describe('EncryptedIndexedDBStorage', () => {
let dbName: string;
let store: EncryptedIndexedDBStorage;
beforeEach(async () => {
dbName = uniqueDbName();
store = await EncryptedIndexedDBStorage.open({ dbName, keyManager: await freshKM() });
});
afterEach(async () => {
store.close();
await deleteDb(dbName);
});
test('identity round-trip', async () => {
expect(await store.getIdentityKeyPair()).toBeNull();
const kp = dummyIdentity();
await store.saveIdentityKeyPair(kp);
const got = await store.getIdentityKeyPair();
expect(got).toEqual(kp);
});
test('registrationId round-trip', async () => {
expect(await store.getLocalRegistrationId()).toBe(0);
await store.saveLocalRegistrationId(987);
expect(await store.getLocalRegistrationId()).toBe(987);
});
test('signed prekey round-trip + remove', async () => {
expect(await store.getSignedPreKey(7)).toBeNull();
const k = dummySignedPreKey(7);
await store.saveSignedPreKey(k);
const got = await store.getSignedPreKey(7);
expect(got?.keyId).toBe(7);
expect(got?.keyPair.privateKey).toEqual(k.keyPair.privateKey);
await store.removeSignedPreKey(7);
expect(await store.getSignedPreKey(7)).toBeNull();
});
test('one-time prekey count + remove', async () => {
expect(await store.getOneTimePreKeyCount()).toBe(0);
await store.saveOneTimePreKey(dummyOTP(1));
await store.saveOneTimePreKey(dummyOTP(2));
expect(await store.getOneTimePreKeyCount()).toBe(2);
expect(await store.getOneTimePreKey(1)).not.toBeNull();
await store.removeOneTimePreKey(1);
expect(await store.getOneTimePreKeyCount()).toBe(1);
});
test('session round-trip + remove', async () => {
const s = dummySession();
await store.saveSession('device:abc', s);
const got = await store.getSession('device:abc');
expect(got?.rootKey).toEqual(s.rootKey);
await store.removeSession('device:abc');
expect(await store.getSession('device:abc')).toBeNull();
});
test('TOFU + trust check', async () => {
const ik = randBytes(32);
expect(await store.isTrustedIdentity('peer-1', ik)).toBe(true); // TOFU
await store.saveTrustedIdentity('peer-1', ik);
expect(await store.isTrustedIdentity('peer-1', ik)).toBe(true);
expect(await store.isTrustedIdentity('peer-1', randBytes(32))).toBe(false);
});
test('retired identities are sorted DESC + prune', async () => {
await store.addRetiredIdentity({ keyPair: dummyIdentity(), retiredAt: 100 });
await store.addRetiredIdentity({ keyPair: dummyIdentity(), retiredAt: 200 });
const list = await store.getRetiredIdentities();
expect(list.length).toBe(2);
expect(list[0]!.retiredAt).toBe(200);
await store.pruneRetiredIdentities(150);
const after = await store.getRetiredIdentities();
expect(after.length).toBe(1);
expect(after[0]!.retiredAt).toBe(200);
});
test('stream-state round-trip + listActive + prune', async () => {
const s: PersistedStreamState = {
streamId: 'stream-1',
direction: 'send',
peerAddress: 'device:bob',
status: 'active',
metadataJson: '{"name":"file.bin"}',
partitionJson: '[]',
laneStateJson: '[]',
ioDescriptorJson: '{"path":"/tmp/x"}',
secretEnc: randBytes(32),
secretNonce: randBytes(12),
createdAt: 1,
updatedAt: 2,
};
await store.saveStreamState(s);
const got = await store.getStreamState('stream-1');
expect(got).toEqual(s);
const active = await store.listActiveStreamStates();
expect(active.length).toBe(1);
expect((await store.listActiveStreamStates('receive')).length).toBe(0);
await store.saveStreamState({ ...s, streamId: 'stream-2', status: 'finished', updatedAt: 50 });
expect((await store.listActiveStreamStates()).length).toBe(1);
await store.pruneStreamStates(100);
expect(await store.getStreamState('stream-2')).toBeNull();
expect(await store.getStreamState('stream-1')).not.toBeNull();
});
test('peer verification + identity-version bump (atomic)', async () => {
expect(await store.getPeerVerification('peer-x')).toBeNull();
await store.savePeerVerification({
peerAddress: 'peer-x',
fingerprint: 'fp',
verifiedAt: 1,
verifiedBy: 'sas',
identityVersion: 1,
});
const v = await store.getPeerVerification('peer-x');
expect(v?.fingerprint).toBe('fp');
expect(await store.getPeerIdentityVersion('peer-x')).toBe(1);
expect(await store.bumpPeerIdentityVersion('peer-x')).toBe(2);
expect(await store.bumpPeerIdentityVersion('peer-x')).toBe(3);
expect(await store.getPeerIdentityVersion('peer-x')).toBe(3);
await store.removePeerVerification('peer-x');
expect(await store.getPeerVerification('peer-x')).toBeNull();
});
test('rejects open with wrong key (fingerprint mismatch)', async () => {
await store.saveIdentityKeyPair(dummyIdentity());
store.close();
const otherKey = randBytes(32);
await expect(EncryptedIndexedDBStorage.open({
dbName,
keyManager: await KeyManager.open({ kind: 'injected', key: otherKey }),
})).rejects.toThrow(/storage key mismatch/);
// Reopen with original key for afterEach
store = await EncryptedIndexedDBStorage.open({ dbName, keyManager: await freshKM() });
});
});
describe('EncryptedIndexedDBStorage — cross-impl roundtrip with EncryptedSQLiteStorage', () => {
test('row sealed by SQLite backend decrypts under IDB backend with same KeyManager', async () => {
const sharedKey = randBytes(32);
const dbPath = join(tmpdir(), `shade-cross-${Date.now()}-${Math.random().toString(36).slice(2)}.db`);
const dbName = uniqueDbName();
// Write via SQLite
const km1 = await KeyManager.open({ kind: 'injected', key: sharedKey });
const sqlite = await EncryptedSQLiteStorage.open({ dbPath, keyManager: km1 });
const kp = dummyIdentity();
await sqlite.saveIdentityKeyPair(kp);
await sqlite.saveLocalRegistrationId(424242);
await sqlite.saveSession('device:cross', dummySession());
// Pull the raw ciphertext blobs out and inject them into a fresh IDB store
// through normal saveX → check the IDB-saved blobs decrypt under the same
// KeyManager. Since AAD/nonce derivation is purely a function of (km,
// table, column, pk), bytes-equal blobs prove the row codec is
// implementation-agnostic.
sqlite.close();
try { unlinkSync(dbPath); } catch {}
try { unlinkSync(dbPath + '-wal'); } catch {}
try { unlinkSync(dbPath + '-shm'); } catch {}
const km2 = await KeyManager.open({ kind: 'injected', key: sharedKey });
const idb = await EncryptedIndexedDBStorage.open({ dbName, keyManager: km2 });
await idb.saveIdentityKeyPair(kp);
await idb.saveLocalRegistrationId(424242);
expect(await idb.getIdentityKeyPair()).toEqual(kp);
expect(await idb.getLocalRegistrationId()).toBe(424242);
idb.close();
await deleteDb(dbName);
});
test('AAD (table, column, pk) binding is implementation-agnostic', async () => {
// Open both backends with the same injected key, save the same session
// under the same address, then assert that the resulting ciphertext blobs
// are byte-equal — confirming AAD + nonce derivation is shared.
const sharedKey = randBytes(32);
const dbPath = join(tmpdir(), `shade-cross-aad-${Date.now()}.db`);
const dbName = uniqueDbName();
const session = dummySession();
const sqlite = await EncryptedSQLiteStorage.open({
dbPath,
keyManager: await KeyManager.open({ kind: 'injected', key: sharedKey }),
});
await sqlite.saveSession('addr-1', session);
const idb = await EncryptedIndexedDBStorage.open({
dbName,
keyManager: await KeyManager.open({ kind: 'injected', key: sharedKey }),
});
await idb.saveSession('addr-1', session);
// Both backends must recover the same plaintext.
const fromSqlite = await sqlite.getSession('addr-1');
const fromIdb = await idb.getSession('addr-1');
expect(fromSqlite?.rootKey).toEqual(session.rootKey);
expect(fromIdb?.rootKey).toEqual(session.rootKey);
expect(fromSqlite?.rootKey).toEqual(fromIdb!.rootKey);
sqlite.close();
idb.close();
try { unlinkSync(dbPath); } catch {}
try { unlinkSync(dbPath + '-wal'); } catch {}
try { unlinkSync(dbPath + '-shm'); } catch {}
await deleteDb(dbName);
});
});

View File

@@ -168,6 +168,82 @@ describe('EncryptedSQLiteStorage', () => {
expect(await store.getStreamState('stream-1')).not.toBeNull(); // active rows untouched expect(await store.getStreamState('stream-1')).not.toBeNull(); // active rows untouched
}); });
test('broadcast channel + member round-trip (V4.6)', async () => {
const channel = {
channelId: 'c-1',
ownerRole: 'sender' as const,
ownerAddress: 'pc',
label: 'output',
generation: 0,
chainKey: randBytes(32),
iteration: 0,
signingPublicKey: randBytes(32),
signingPrivateKey: randBytes(32),
createdAt: 1_700_000_000_000,
updatedAt: 1_700_000_000_000,
};
await store.saveBroadcastChannel(channel);
const fetched = await store.getBroadcastChannel('c-1');
expect(fetched).not.toBeNull();
expect(fetched!.ownerRole).toBe('sender');
expect(fetched!.label).toBe('output');
expect(fetched!.chainKey).toEqual(channel.chainKey);
expect(fetched!.signingPrivateKey).toEqual(channel.signingPrivateKey);
// Add two members + verify list + remove.
await store.saveBroadcastMember({
channelId: 'c-1',
peerAddress: 'mobile-a',
joinedAt: 1_700_000_000_001,
removedAt: null,
});
await store.saveBroadcastMember({
channelId: 'c-1',
peerAddress: 'mobile-b',
joinedAt: 1_700_000_000_002,
removedAt: null,
});
let members = await store.getBroadcastMembers('c-1');
expect(members.map((m) => m.peerAddress)).toEqual(['mobile-a', 'mobile-b']);
// Mark one removed.
await store.saveBroadcastMember({
channelId: 'c-1',
peerAddress: 'mobile-a',
joinedAt: 1_700_000_000_001,
removedAt: 1_700_000_000_500,
});
members = await store.getBroadcastMembers('c-1');
expect(members.find((m) => m.peerAddress === 'mobile-a')?.removedAt).toBe(1_700_000_000_500);
// List + delete cascade.
expect((await store.listBroadcastChannels())).toHaveLength(1);
await store.removeBroadcastChannel('c-1');
expect(await store.getBroadcastChannel('c-1')).toBeNull();
expect((await store.getBroadcastMembers('c-1'))).toHaveLength(0);
});
test('broadcast channel sealed: receiver-side row has no private key', async () => {
await store.saveBroadcastChannel({
channelId: 'c-2',
ownerRole: 'receiver',
ownerAddress: 'pc',
generation: 1,
chainKey: randBytes(32),
iteration: 5,
signingPublicKey: randBytes(32),
// no signingPrivateKey
createdAt: 1,
updatedAt: 1,
});
const r = await store.getBroadcastChannel('c-2');
expect(r).not.toBeNull();
expect(r!.ownerRole).toBe('receiver');
expect(r!.signingPrivateKey).toBeUndefined();
expect(r!.iteration).toBe(5);
expect(r!.generation).toBe(1);
});
test('rejects open with wrong key (fingerprint mismatch)', async () => { test('rejects open with wrong key (fingerprint mismatch)', async () => {
await store.saveIdentityKeyPair(dummyIdentity()); await store.saveIdentityKeyPair(dummyIdentity());
store.close(); store.close();

View File

@@ -0,0 +1,183 @@
import { describe, test, expect } from 'bun:test';
import { KeyManager } from '../src/crypto/key-manager.js';
import {
DEFAULT_ARGON2ID,
deriveMasterKeyArgon2id,
type Argon2idParams,
} from '../src/crypto/kdf.js';
const FAST_ARGON: Argon2idParams = { m: 256, t: 1, p: 1, dkLen: 32 };
function randBytes(n: number): Uint8Array {
const b = new Uint8Array(n);
globalThis.crypto.getRandomValues(b);
return b;
}
describe('argon2id source', () => {
const salt = new Uint8Array(16).fill(0x33);
test('deriveMasterKeyArgon2id is deterministic', async () => {
const a = await deriveMasterKeyArgon2id('1234', salt, FAST_ARGON);
const b = await deriveMasterKeyArgon2id('1234', salt, FAST_ARGON);
expect(a).toEqual(b);
expect(a.length).toBe(32);
});
test('different secret → different key', async () => {
const a = await deriveMasterKeyArgon2id('1234', salt, FAST_ARGON);
const b = await deriveMasterKeyArgon2id('1235', salt, FAST_ARGON);
expect(a).not.toEqual(b);
});
test('different salt → different key', async () => {
const a = await deriveMasterKeyArgon2id('1234', salt, FAST_ARGON);
const b = await deriveMasterKeyArgon2id('1234', new Uint8Array(16).fill(0x44), FAST_ARGON);
expect(a).not.toEqual(b);
});
test('rejects empty secret', async () => {
await expect(deriveMasterKeyArgon2id('', salt, FAST_ARGON)).rejects.toThrow(/non-empty/);
});
test('rejects too-short salt', async () => {
await expect(deriveMasterKeyArgon2id('p', new Uint8Array(8), FAST_ARGON))
.rejects.toThrow(/at least 16/);
});
test('KeyManager.open opens with argon2id source', async () => {
const km = await KeyManager.open({
kind: 'argon2id',
secret: '123456',
salt,
params: FAST_ARGON,
});
expect(km.fieldKey('t', 'c').length).toBe(32);
km.destroy();
});
test('DEFAULT_ARGON2ID is exposed and sensible', () => {
expect(DEFAULT_ARGON2ID.dkLen).toBe(32);
expect(DEFAULT_ARGON2ID.m).toBeGreaterThanOrEqual(8 * 1024);
expect(DEFAULT_ARGON2ID.t).toBeGreaterThanOrEqual(1);
});
test('accepts Uint8Array secret', async () => {
const secretBytes = new TextEncoder().encode('1234');
const a = await deriveMasterKeyArgon2id(secretBytes, salt, FAST_ARGON);
const b = await deriveMasterKeyArgon2id('1234', salt, FAST_ARGON);
expect(a).toEqual(b);
});
});
describe('composite source — multi-factor unlock', () => {
const pwSalt = new Uint8Array(16).fill(0x11);
const pinSalt = new Uint8Array(16).fill(0x22);
const FAST_SCRYPT = { N: 1 << 10, r: 8, p: 1, dkLen: 32 };
function pwSource(passphrase: string) {
return { kind: 'passphrase' as const, passphrase, salt: pwSalt, params: FAST_SCRYPT };
}
function pinSource(secret: string) {
return { kind: 'argon2id' as const, secret, salt: pinSalt, params: FAST_ARGON };
}
test('same factors → same masterKey', async () => {
const a = await KeyManager.open({
kind: 'composite',
sources: [pwSource('correct horse'), pinSource('1234')],
});
const b = await KeyManager.open({
kind: 'composite',
sources: [pwSource('correct horse'), pinSource('1234')],
});
expect(a.storageKeyFingerprint()).toEqual(b.storageKeyFingerprint());
a.destroy();
b.destroy();
});
test('wrong PIN → different masterKey (same shape as wrong-passphrase)', async () => {
const right = await KeyManager.open({
kind: 'composite',
sources: [pwSource('correct horse'), pinSource('1234')],
});
const wrongPin = await KeyManager.open({
kind: 'composite',
sources: [pwSource('correct horse'), pinSource('9999')],
});
expect(right.storageKeyFingerprint()).not.toEqual(wrongPin.storageKeyFingerprint());
right.destroy();
wrongPin.destroy();
});
test('wrong passphrase → different masterKey', async () => {
const right = await KeyManager.open({
kind: 'composite',
sources: [pwSource('correct horse'), pinSource('1234')],
});
const wrongPwd = await KeyManager.open({
kind: 'composite',
sources: [pwSource('wrong horse'), pinSource('1234')],
});
expect(right.storageKeyFingerprint()).not.toEqual(wrongPwd.storageKeyFingerprint());
right.destroy();
wrongPwd.destroy();
});
test('order is significant by design', async () => {
const ab = await KeyManager.open({
kind: 'composite',
sources: [pwSource('horse'), pinSource('1234')],
});
const ba = await KeyManager.open({
kind: 'composite',
sources: [pinSource('1234'), pwSource('horse')],
});
expect(ab.storageKeyFingerprint()).not.toEqual(ba.storageKeyFingerprint());
ab.destroy();
ba.destroy();
});
test('explicit info string changes masterKey (domain separation)', async () => {
const a = await KeyManager.open({
kind: 'composite',
sources: [pwSource('horse'), pinSource('1234')],
});
const b = await KeyManager.open({
kind: 'composite',
sources: [pwSource('horse'), pinSource('1234')],
info: 'my-app-v1',
});
expect(a.storageKeyFingerprint()).not.toEqual(b.storageKeyFingerprint());
a.destroy();
b.destroy();
});
test('rejects empty source list', async () => {
await expect(KeyManager.open({ kind: 'composite', sources: [] }))
.rejects.toThrow(/at least one/);
});
test('rejects nested composite', async () => {
await expect(KeyManager.open({
kind: 'composite',
sources: [
{ kind: 'composite', sources: [pwSource('a')] },
pinSource('1234'),
],
})).rejects.toThrow(/cannot be nested/);
});
test('composite of three sources works', async () => {
const km = await KeyManager.open({
kind: 'composite',
sources: [
pwSource('horse'),
pinSource('1234'),
{ kind: 'injected', key: randBytes(32) },
],
});
expect(km.fieldKey('t', 'c').length).toBe(32);
km.destroy();
});
});

View File

@@ -1,5 +1,5 @@
{ {
"extends": "../../tsconfig.json", "extends": "../../tsconfig.json",
"compilerOptions": { "outDir": "dist", "rootDir": "src" }, "compilerOptions": { "outDir": "dist", "rootDir": "src", "lib": ["ES2022", "DOM"] },
"include": ["src"] "include": ["src"]
} }

View File

@@ -0,0 +1,15 @@
{
"name": "@shade/storage-indexeddb",
"version": "4.8.2",
"type": "module",
"main": "src/index.ts",
"types": "src/index.ts",
"dependencies": {
"@shade/core": "workspace:*",
"idb": "^8.0.3"
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
"fake-indexeddb": "^6.0.0"
}
}

View File

@@ -0,0 +1 @@
export { IndexedDBStorage } from './indexeddb-storage.js';

View File

@@ -0,0 +1,555 @@
import { openDB, type IDBPDatabase, type DBSchema } from 'idb';
import type {
StorageProvider,
IdentityKeyPair,
SignedPreKey,
OneTimePreKey,
SessionState,
RetiredIdentity,
PersistedStreamState,
PeerVerification,
PeerVerificationSource,
BroadcastChannelRecord,
BroadcastMemberRecord,
} from '@shade/core';
import {
toBase64, fromBase64,
constantTimeEqual,
serializeSessionState, deserializeSessionState,
serializeSignedPreKey, deserializeSignedPreKey,
serializeOneTimePreKey, deserializeOneTimePreKey,
serializeIdentityKeyPair, deserializeIdentityKeyPair,
} from '@shade/core';
/**
* IndexedDB-backed StorageProvider for browser-side Shade clients.
*
* Persists identity, prekeys, sessions, retired identities, peer
* verifications and stream-resume state across tab refresh and browser
* restart. Same data shapes as `@shade/storage-sqlite` so cross-adapter
* import/export remains feasible.
*
* Usage:
* ```ts
* const storage = await IndexedDBStorage.create({ dbName: 'my-app-shade' });
* const manager = new ShadeSessionManager(crypto, storage);
* ```
*/
export class IndexedDBStorage implements StorageProvider {
private constructor(private db: IDBPDatabase<ShadeSchema>) {}
/**
* Open (or create) the IndexedDB database. Idempotent — repeated calls
* with the same dbName resolve to a fresh connection sharing the same
* underlying object stores.
*/
static async create(opts: { dbName?: string } = {}): Promise<IndexedDBStorage> {
const dbName = opts.dbName ?? 'shade';
const db = await openDB<ShadeSchema>(dbName, SCHEMA_VERSION, {
upgrade(db, oldVersion) {
if (oldVersion < 1) {
db.createObjectStore('identity', { keyPath: 'id' });
db.createObjectStore('config', { keyPath: 'key' });
db.createObjectStore('signedPreKeys', { keyPath: 'keyId' });
db.createObjectStore('oneTimePreKeys', { keyPath: 'keyId' });
db.createObjectStore('sessions', { keyPath: 'address' });
db.createObjectStore('trustedIdentities', { keyPath: 'address' });
const retired = db.createObjectStore('retiredIdentities', {
keyPath: 'id',
autoIncrement: true,
});
retired.createIndex('byRetiredAt', 'retiredAt');
const stream = db.createObjectStore('streamStates', { keyPath: 'streamId' });
stream.createIndex('byStatus', 'status');
stream.createIndex('byPeerAddress', 'peerAddress');
stream.createIndex('byUpdatedAt', 'updatedAt');
db.createObjectStore('peerVerifications', { keyPath: 'peerAddress' });
db.createObjectStore('peerIdentityVersions', { keyPath: 'peerAddress' });
}
if (oldVersion < 2) {
db.createObjectStore('broadcastChannels', { keyPath: 'channelId' });
const members = db.createObjectStore('broadcastMembers', { keyPath: ['channelId', 'peerAddress'] });
members.createIndex('byChannelId', 'channelId');
}
},
});
return new IndexedDBStorage(db);
}
/** Cleanly close the underlying connection. */
async close(): Promise<void> {
this.db.close();
}
// ─── Identity ──────────────────────────────────────────────
async getIdentityKeyPair(): Promise<IdentityKeyPair | null> {
const row = await this.db.get('identity', 1);
if (!row) return null;
return {
signingPublicKey: fromBase64(row.signingPublicKey),
signingPrivateKey: fromBase64(row.signingPrivateKey),
dhPublicKey: fromBase64(row.dhPublicKey),
dhPrivateKey: fromBase64(row.dhPrivateKey),
};
}
async saveIdentityKeyPair(kp: IdentityKeyPair): Promise<void> {
await this.db.put('identity', {
id: 1,
signingPublicKey: toBase64(kp.signingPublicKey),
signingPrivateKey: toBase64(kp.signingPrivateKey),
dhPublicKey: toBase64(kp.dhPublicKey),
dhPrivateKey: toBase64(kp.dhPrivateKey),
});
}
async getLocalRegistrationId(): Promise<number> {
const row = await this.db.get('config', 'registrationId');
return row ? parseInt(row.value, 10) : 0;
}
async saveLocalRegistrationId(id: number): Promise<void> {
await this.db.put('config', { key: 'registrationId', value: String(id) });
}
// ─── Signed PreKeys ───────────────────────────────────────
async getSignedPreKey(keyId: number): Promise<SignedPreKey | null> {
const row = await this.db.get('signedPreKeys', keyId);
if (!row) return null;
return deserializeSignedPreKey(row.dataJson);
}
async saveSignedPreKey(key: SignedPreKey): Promise<void> {
await this.db.put('signedPreKeys', {
keyId: key.keyId,
dataJson: serializeSignedPreKey(key),
});
}
async removeSignedPreKey(keyId: number): Promise<void> {
await this.db.delete('signedPreKeys', keyId);
}
// ─── One-Time PreKeys ─────────────────────────────────────
async getOneTimePreKey(keyId: number): Promise<OneTimePreKey | null> {
const row = await this.db.get('oneTimePreKeys', keyId);
if (!row) return null;
return deserializeOneTimePreKey(row.dataJson);
}
async saveOneTimePreKey(key: OneTimePreKey): Promise<void> {
await this.db.put('oneTimePreKeys', {
keyId: key.keyId,
dataJson: serializeOneTimePreKey(key),
});
}
async removeOneTimePreKey(keyId: number): Promise<void> {
await this.db.delete('oneTimePreKeys', keyId);
}
async getOneTimePreKeyCount(): Promise<number> {
return this.db.count('oneTimePreKeys');
}
// ─── Sessions ─────────────────────────────────────────────
async getSession(address: string): Promise<SessionState | null> {
const row = await this.db.get('sessions', address);
if (!row) return null;
return deserializeSessionState(row.stateJson);
}
async saveSession(address: string, state: SessionState): Promise<void> {
await this.db.put('sessions', { address, stateJson: serializeSessionState(state) });
}
async removeSession(address: string): Promise<void> {
await this.db.delete('sessions', address);
}
// ─── Trust ────────────────────────────────────────────────
async isTrustedIdentity(address: string, identityKey: Uint8Array): Promise<boolean> {
const row = await this.db.get('trustedIdentities', address);
if (!row) return true; // TOFU
const stored = fromBase64(row.identityKey);
return constantTimeEqual(stored, identityKey);
}
async saveTrustedIdentity(address: string, identityKey: Uint8Array): Promise<void> {
await this.db.put('trustedIdentities', { address, identityKey: toBase64(identityKey) });
}
// ─── Identity History ─────────────────────────────────────
async addRetiredIdentity(identity: RetiredIdentity): Promise<void> {
// autoIncrement: omit `id` so IDB assigns one
await this.db.add('retiredIdentities', {
dataJson: serializeIdentityKeyPair(identity.keyPair),
retiredAt: identity.retiredAt,
} as RetiredIdentityRow);
}
async getRetiredIdentities(): Promise<RetiredIdentity[]> {
// Mirror SQLite's `ORDER BY retired_at DESC`
const rows = await this.db.getAllFromIndex('retiredIdentities', 'byRetiredAt');
rows.reverse();
return rows.map((r) => ({
keyPair: deserializeIdentityKeyPair(r.dataJson),
retiredAt: r.retiredAt,
}));
}
async pruneRetiredIdentities(olderThan: number): Promise<void> {
const tx = this.db.transaction('retiredIdentities', 'readwrite');
const idx = tx.store.index('byRetiredAt');
const range = IDBKeyRange.upperBound(olderThan, true);
let cursor = await idx.openCursor(range);
while (cursor) {
await cursor.delete();
cursor = await cursor.continue();
}
await tx.done;
}
// ─── Stream-transfer resume state ─────────────────────────
async saveStreamState(state: PersistedStreamState): Promise<void> {
await this.db.put('streamStates', persistedToRow(state));
}
async getStreamState(streamId: string): Promise<PersistedStreamState | null> {
const row = await this.db.get('streamStates', streamId);
if (!row) return null;
return rowToPersisted(row);
}
async removeStreamState(streamId: string): Promise<void> {
await this.db.delete('streamStates', streamId);
}
async listActiveStreamStates(direction?: 'send' | 'receive'): Promise<PersistedStreamState[]> {
const idx = this.db.transaction('streamStates').store.index('byStatus');
const active = await idx.getAll(IDBKeyRange.only('active'));
const paused = await idx.getAll(IDBKeyRange.only('paused'));
const merged = [...active, ...paused];
const filtered = direction === undefined
? merged
: merged.filter((r) => r.direction === direction);
filtered.sort((a, b) => b.updatedAt - a.updatedAt);
return filtered.map(rowToPersisted);
}
async pruneStreamStates(olderThan: number): Promise<void> {
const tx = this.db.transaction('streamStates', 'readwrite');
const idx = tx.store.index('byUpdatedAt');
const range = IDBKeyRange.upperBound(olderThan, true);
let cursor = await idx.openCursor(range);
while (cursor) {
const row = cursor.value;
if (row.status === 'finished' || row.status === 'aborted') {
await cursor.delete();
}
cursor = await cursor.continue();
}
await tx.done;
}
// ─── Peer verifications (V3.3) ────────────────────────────
async savePeerVerification(v: PeerVerification): Promise<void> {
await this.db.put('peerVerifications', { ...v });
}
async getPeerVerification(address: string): Promise<PeerVerification | null> {
const row = await this.db.get('peerVerifications', address);
if (!row) return null;
return {
peerAddress: row.peerAddress,
fingerprint: row.fingerprint,
verifiedAt: row.verifiedAt,
verifiedBy: row.verifiedBy as PeerVerificationSource,
identityVersion: row.identityVersion,
};
}
async removePeerVerification(address: string): Promise<void> {
await this.db.delete('peerVerifications', address);
}
async getPeerIdentityVersion(address: string): Promise<number> {
const row = await this.db.get('peerIdentityVersions', address);
return row ? row.version : 1;
}
/**
* Atomic read-modify-write under one IDB transaction. SQLite's version
* is a non-atomic read-then-upsert; the IDB version closes that race
* because IDB transactions auto-commit only when control returns to
* the event loop without pending requests.
*/
async bumpPeerIdentityVersion(address: string): Promise<number> {
const tx = this.db.transaction('peerIdentityVersions', 'readwrite');
const existing = await tx.store.get(address);
const next = (existing ? existing.version : 1) + 1;
await tx.store.put({ peerAddress: address, version: next });
await tx.done;
return next;
}
// ─── Broadcast channels (V4.6) ────────────────────────────
async saveBroadcastChannel(channel: BroadcastChannelRecord): Promise<void> {
await this.db.put('broadcastChannels', channelToRow(channel));
}
async getBroadcastChannel(channelId: string): Promise<BroadcastChannelRecord | null> {
const row = await this.db.get('broadcastChannels', channelId);
if (!row) return null;
return rowToChannel(row);
}
async listBroadcastChannels(): Promise<BroadcastChannelRecord[]> {
const rows = await this.db.getAll('broadcastChannels');
rows.sort((a, b) => a.createdAt - b.createdAt);
return rows.map(rowToChannel);
}
async removeBroadcastChannel(channelId: string): Promise<void> {
const tx = this.db.transaction(['broadcastChannels', 'broadcastMembers'], 'readwrite');
const memIdx = tx.objectStore('broadcastMembers').index('byChannelId');
let cursor = await memIdx.openCursor(IDBKeyRange.only(channelId));
while (cursor) {
await cursor.delete();
cursor = await cursor.continue();
}
await tx.objectStore('broadcastChannels').delete(channelId);
await tx.done;
}
async saveBroadcastMember(member: BroadcastMemberRecord): Promise<void> {
await this.db.put('broadcastMembers', { ...member });
}
async getBroadcastMembers(channelId: string): Promise<BroadcastMemberRecord[]> {
const rows = await this.db.getAllFromIndex(
'broadcastMembers',
'byChannelId',
IDBKeyRange.only(channelId),
);
rows.sort((a, b) => a.joinedAt - b.joinedAt);
return rows.map((r) => ({
channelId: r.channelId,
peerAddress: r.peerAddress,
joinedAt: r.joinedAt,
removedAt: r.removedAt,
}));
}
async removeBroadcastMember(channelId: string, peerAddress: string): Promise<void> {
await this.db.delete('broadcastMembers', [channelId, peerAddress]);
}
}
// ─── Schema ────────────────────────────────────────────────
const SCHEMA_VERSION = 2;
interface IdentityRow {
id: 1;
signingPublicKey: string;
signingPrivateKey: string;
dhPublicKey: string;
dhPrivateKey: string;
}
interface ConfigRow {
key: string;
value: string;
}
interface SignedPreKeyRow {
keyId: number;
dataJson: string;
}
interface OneTimePreKeyRow {
keyId: number;
dataJson: string;
}
interface SessionRow {
address: string;
stateJson: string;
}
interface TrustedIdentityRow {
address: string;
identityKey: string;
}
interface RetiredIdentityRow {
id?: number;
dataJson: string;
retiredAt: number;
}
interface StreamStateRow {
streamId: string;
direction: 'send' | 'receive';
peerAddress: string;
status: 'active' | 'paused' | 'finished' | 'aborted';
metadataJson: string;
partitionJson: string;
laneStateJson: string;
ioDescriptorJson: string;
secretEnc: Uint8Array;
secretNonce: Uint8Array;
overallHashState: string | null;
createdAt: number;
updatedAt: number;
}
interface PeerVerificationRow {
peerAddress: string;
fingerprint: string;
verifiedAt: number;
verifiedBy: string;
identityVersion: number;
}
interface PeerIdentityVersionRow {
peerAddress: string;
version: number;
}
interface BroadcastChannelRow {
channelId: string;
ownerRole: 'sender' | 'receiver';
ownerAddress: string;
label: string | null;
generation: number;
chainKey: Uint8Array;
iteration: number;
signingPublicKey: Uint8Array;
signingPrivateKey: Uint8Array | null;
createdAt: number;
updatedAt: number;
}
interface BroadcastMemberRow {
channelId: string;
peerAddress: string;
joinedAt: number;
removedAt: number | null;
}
interface ShadeSchema extends DBSchema {
identity: { key: number; value: IdentityRow };
config: { key: string; value: ConfigRow };
signedPreKeys: { key: number; value: SignedPreKeyRow };
oneTimePreKeys: { key: number; value: OneTimePreKeyRow };
sessions: { key: string; value: SessionRow };
trustedIdentities: { key: string; value: TrustedIdentityRow };
retiredIdentities: {
key: number;
value: RetiredIdentityRow;
indexes: { byRetiredAt: number };
};
streamStates: {
key: string;
value: StreamStateRow;
indexes: {
byStatus: string;
byPeerAddress: string;
byUpdatedAt: number;
};
};
peerVerifications: { key: string; value: PeerVerificationRow };
peerIdentityVersions: { key: string; value: PeerIdentityVersionRow };
broadcastChannels: { key: string; value: BroadcastChannelRow };
broadcastMembers: {
key: [string, string];
value: BroadcastMemberRow;
indexes: { byChannelId: string };
};
}
// ─── Helpers ──────────────────────────────────────────────
function persistedToRow(s: PersistedStreamState): StreamStateRow {
return {
streamId: s.streamId,
direction: s.direction,
peerAddress: s.peerAddress,
status: s.status,
metadataJson: s.metadataJson,
partitionJson: s.partitionJson,
laneStateJson: s.laneStateJson,
ioDescriptorJson: s.ioDescriptorJson,
secretEnc: s.secretEnc,
secretNonce: s.secretNonce,
overallHashState: s.overallHashState ?? null,
createdAt: s.createdAt,
updatedAt: s.updatedAt,
};
}
function channelToRow(c: BroadcastChannelRecord): BroadcastChannelRow {
return {
channelId: c.channelId,
ownerRole: c.ownerRole,
ownerAddress: c.ownerAddress,
label: c.label ?? null,
generation: c.generation,
chainKey: c.chainKey,
iteration: c.iteration,
signingPublicKey: c.signingPublicKey,
signingPrivateKey: c.signingPrivateKey ?? null,
createdAt: c.createdAt,
updatedAt: c.updatedAt,
};
}
function rowToChannel(r: BroadcastChannelRow): BroadcastChannelRecord {
const out: BroadcastChannelRecord = {
channelId: r.channelId,
ownerRole: r.ownerRole,
ownerAddress: r.ownerAddress,
generation: r.generation,
chainKey: r.chainKey,
iteration: r.iteration,
signingPublicKey: r.signingPublicKey,
createdAt: r.createdAt,
updatedAt: r.updatedAt,
};
if (r.label !== null) out.label = r.label;
if (r.signingPrivateKey !== null) out.signingPrivateKey = r.signingPrivateKey;
return out;
}
function rowToPersisted(r: StreamStateRow): PersistedStreamState {
const out: PersistedStreamState = {
streamId: r.streamId,
direction: r.direction,
peerAddress: r.peerAddress,
status: r.status,
metadataJson: r.metadataJson,
partitionJson: r.partitionJson,
laneStateJson: r.laneStateJson,
ioDescriptorJson: r.ioDescriptorJson,
secretEnc: r.secretEnc,
secretNonce: r.secretNonce,
createdAt: r.createdAt,
updatedAt: r.updatedAt,
};
if (r.overallHashState !== null) out.overallHashState = r.overallHashState;
return out;
}

View File

@@ -0,0 +1,272 @@
import 'fake-indexeddb/auto';
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { IndexedDBStorage } from '../src/indexeddb-storage.js';
import { SubtleCryptoProvider, MemoryStorage as _MemoryStorage } from '@shade/crypto-web';
import { ShadeSessionManager } from '@shade/core';
import type { IdentityKeyPair, SignedPreKey, OneTimePreKey } from '@shade/core';
void _MemoryStorage; // keep import side-effect-free reference shape
const crypto = new SubtleCryptoProvider();
function randBytes(n: number): Uint8Array {
const buf = new Uint8Array(n);
globalThis.crypto.getRandomValues(buf);
return buf;
}
function uniqueDbName(): string {
return `shade-test-${Date.now()}-${Math.random().toString(36).slice(2)}`;
}
async function deleteDb(dbName: string): Promise<void> {
await new Promise<void>((resolve, reject) => {
const req = indexedDB.deleteDatabase(dbName);
req.onsuccess = () => resolve();
req.onerror = () => reject(req.error);
req.onblocked = () => resolve();
});
}
describe('IndexedDBStorage', () => {
let dbName: string;
let storage: IndexedDBStorage;
beforeEach(async () => {
dbName = uniqueDbName();
storage = await IndexedDBStorage.create({ dbName });
});
afterEach(async () => {
await storage.close();
await deleteDb(dbName);
});
// ─── Identity ──────────────────────────────────────────────
describe('identity', () => {
test('returns null when no identity stored', async () => {
expect(await storage.getIdentityKeyPair()).toBeNull();
});
test('save and retrieve identity keypair', async () => {
const ikp: IdentityKeyPair = {
signingPublicKey: randBytes(32),
signingPrivateKey: randBytes(32),
dhPublicKey: randBytes(32),
dhPrivateKey: randBytes(32),
};
await storage.saveIdentityKeyPair(ikp);
const restored = await storage.getIdentityKeyPair();
expect(restored).not.toBeNull();
expect(restored!.signingPublicKey).toEqual(ikp.signingPublicKey);
expect(restored!.dhPrivateKey).toEqual(ikp.dhPrivateKey);
});
test('registration ID roundtrip', async () => {
expect(await storage.getLocalRegistrationId()).toBe(0);
await storage.saveLocalRegistrationId(42);
expect(await storage.getLocalRegistrationId()).toBe(42);
});
});
// ─── Signed PreKeys ───────────────────────────────────────
describe('signed prekeys', () => {
test('save, get, remove', async () => {
const spk: SignedPreKey = {
keyId: 1,
keyPair: { publicKey: randBytes(32), privateKey: randBytes(32) },
signature: randBytes(64),
timestamp: Date.now(),
};
await storage.saveSignedPreKey(spk);
const restored = await storage.getSignedPreKey(1);
expect(restored).not.toBeNull();
expect(restored!.keyId).toBe(1);
expect(restored!.keyPair.publicKey).toEqual(spk.keyPair.publicKey);
await storage.removeSignedPreKey(1);
expect(await storage.getSignedPreKey(1)).toBeNull();
});
});
// ─── One-Time PreKeys ─────────────────────────────────────
describe('one-time prekeys', () => {
test('save, get, remove, count', async () => {
const otpk: OneTimePreKey = {
keyId: 100,
keyPair: { publicKey: randBytes(32), privateKey: randBytes(32) },
};
await storage.saveOneTimePreKey(otpk);
expect(await storage.getOneTimePreKeyCount()).toBe(1);
const restored = await storage.getOneTimePreKey(100);
expect(restored!.keyId).toBe(100);
await storage.removeOneTimePreKey(100);
expect(await storage.getOneTimePreKeyCount()).toBe(0);
expect(await storage.getOneTimePreKey(100)).toBeNull();
});
});
// ─── Sessions ─────────────────────────────────────────────
describe('sessions', () => {
test('save and restore session with skipped keys', async () => {
const state = {
remoteIdentityKey: randBytes(32),
rootKey: randBytes(32),
sendChain: { chainKey: randBytes(32), counter: 5 },
receiveChain: { chainKey: randBytes(32), counter: 3 },
dhSend: { publicKey: randBytes(32), privateKey: randBytes(32) },
dhReceive: randBytes(32),
previousSendCounter: 4,
skippedKeys: new Map([['key:1', randBytes(32)]]),
};
await storage.saveSession('bob', state);
const restored = await storage.getSession('bob');
expect(restored).not.toBeNull();
expect(restored!.sendChain.counter).toBe(5);
expect(restored!.skippedKeys.size).toBe(1);
expect(restored!.skippedKeys.get('key:1')).toEqual(state.skippedKeys.get('key:1')!);
});
test('remove session', async () => {
await storage.saveSession('bob', {
remoteIdentityKey: randBytes(32),
rootKey: randBytes(32),
sendChain: { chainKey: randBytes(32), counter: 0 },
receiveChain: null,
dhSend: { publicKey: randBytes(32), privateKey: randBytes(32) },
dhReceive: null,
previousSendCounter: 0,
skippedKeys: new Map(),
});
await storage.removeSession('bob');
expect(await storage.getSession('bob')).toBeNull();
});
});
// ─── Trust ────────────────────────────────────────────────
describe('trust', () => {
test('TOFU: first use is trusted', async () => {
expect(await storage.isTrustedIdentity('bob', randBytes(32))).toBe(true);
});
test('saved identity matches', async () => {
const key = randBytes(32);
await storage.saveTrustedIdentity('bob', key);
expect(await storage.isTrustedIdentity('bob', key)).toBe(true);
expect(await storage.isTrustedIdentity('bob', randBytes(32))).toBe(false);
});
});
// ─── Retired identities ───────────────────────────────────
describe('retired identities', () => {
test('add, list (newest first), prune', async () => {
const mk = (retiredAt: number): { keyPair: IdentityKeyPair; retiredAt: number } => ({
keyPair: {
signingPublicKey: randBytes(32),
signingPrivateKey: randBytes(32),
dhPublicKey: randBytes(32),
dhPrivateKey: randBytes(32),
},
retiredAt,
});
await storage.addRetiredIdentity(mk(100));
await storage.addRetiredIdentity(mk(300));
await storage.addRetiredIdentity(mk(200));
const list = await storage.getRetiredIdentities();
expect(list.map((r) => r.retiredAt)).toEqual([300, 200, 100]);
await storage.pruneRetiredIdentities(250);
const after = await storage.getRetiredIdentities();
expect(after.map((r) => r.retiredAt)).toEqual([300]);
});
});
// ─── Crash Recovery ───────────────────────────────────────
describe('persistence across close/reopen', () => {
test('data survives close and reopen', async () => {
const ikp: IdentityKeyPair = {
signingPublicKey: randBytes(32),
signingPrivateKey: randBytes(32),
dhPublicKey: randBytes(32),
dhPrivateKey: randBytes(32),
};
await storage.saveIdentityKeyPair(ikp);
await storage.saveLocalRegistrationId(99);
await storage.saveSession('alice', {
remoteIdentityKey: randBytes(32),
rootKey: randBytes(32),
sendChain: { chainKey: randBytes(32), counter: 7 },
receiveChain: null,
dhSend: { publicKey: randBytes(32), privateKey: randBytes(32) },
dhReceive: null,
previousSendCounter: 0,
skippedKeys: new Map(),
});
// Close and reopen against the same dbName
await storage.close();
storage = await IndexedDBStorage.create({ dbName });
const restored = await storage.getIdentityKeyPair();
expect(restored!.signingPublicKey).toEqual(ikp.signingPublicKey);
expect(await storage.getLocalRegistrationId()).toBe(99);
const session = await storage.getSession('alice');
expect(session!.sendChain.counter).toBe(7);
});
});
// ─── Full E2EE with IndexedDBStorage ──────────────────────
describe('full E2EE conversation with persistent storage', () => {
test('encrypt, close, reopen, continue conversation', async () => {
const bobDbName = uniqueDbName();
let bobStorage = await IndexedDBStorage.create({ dbName: bobDbName });
try {
const alice = new ShadeSessionManager(crypto, storage);
let bob = new ShadeSessionManager(crypto, bobStorage);
await alice.initialize();
await bob.initialize();
const otpks = await bob.generateOneTimePreKeys(5);
const bundle = await bob.createPreKeyBundle();
bundle.oneTimePreKey = { keyId: otpks[0]!.keyId, publicKey: otpks[0]!.keyPair.publicKey };
await alice.initSessionFromBundle('bob', bundle);
const env1 = await alice.encrypt('bob', 'Hello persistent!');
expect(await bob.decrypt('alice', env1)).toBe('Hello persistent!');
const env2 = await bob.encrypt('alice', 'Got it!');
expect(await alice.decrypt('bob', env2)).toBe('Got it!');
// "Crash" Bob — close the IDB and reopen
await bobStorage.close();
bobStorage = await IndexedDBStorage.create({ dbName: bobDbName });
bob = new ShadeSessionManager(crypto, bobStorage);
await bob.initialize();
const env3 = await alice.encrypt('bob', 'After your restart');
expect(await bob.decrypt('alice', env3)).toBe('After your restart');
const env4 = await bob.encrypt('alice', 'I survived!');
expect(await alice.decrypt('bob', env4)).toBe('I survived!');
} finally {
await bobStorage.close();
await deleteDb(bobDbName);
}
});
});
});

View File

@@ -0,0 +1,99 @@
import 'fake-indexeddb/auto';
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { IndexedDBStorage } from '../src/index.js';
function uniqueDbName(): string {
return `shade-test-${Date.now()}-${Math.random().toString(36).slice(2)}`;
}
async function deleteDb(dbName: string): Promise<void> {
await new Promise<void>((resolve, reject) => {
const req = indexedDB.deleteDatabase(dbName);
req.onsuccess = () => resolve();
req.onerror = () => reject(req.error);
req.onblocked = () => resolve();
});
}
describe('IndexedDBStorage — peer_verifications (V3.3)', () => {
let dbName: string;
let storage: IndexedDBStorage;
beforeEach(async () => {
dbName = uniqueDbName();
storage = await IndexedDBStorage.create({ dbName });
});
afterEach(async () => {
await storage.close();
await deleteDb(dbName);
});
test('round trip: save → get → remove', async () => {
await storage.savePeerVerification({
peerAddress: 'bob',
fingerprint: '12345 67890 12345 67890 12345 67890 12345 67890 12345 67890 12345 67890',
verifiedAt: 1_700_000_000_000,
verifiedBy: 'user',
identityVersion: 1,
});
const v = await storage.getPeerVerification('bob');
expect(v).not.toBeNull();
expect(v!.peerAddress).toBe('bob');
expect(v!.verifiedBy).toBe('user');
expect(v!.identityVersion).toBe(1);
await storage.removePeerVerification('bob');
expect(await storage.getPeerVerification('bob')).toBeNull();
});
test('upsert overwrites on duplicate peer_address', async () => {
await storage.savePeerVerification({
peerAddress: 'bob',
fingerprint: 'fp-1',
verifiedAt: 1,
verifiedBy: 'user',
identityVersion: 1,
});
await storage.savePeerVerification({
peerAddress: 'bob',
fingerprint: 'fp-2',
verifiedAt: 2,
verifiedBy: 'transitive',
identityVersion: 2,
});
const v = await storage.getPeerVerification('bob');
expect(v!.fingerprint).toBe('fp-2');
expect(v!.verifiedBy).toBe('transitive');
expect(v!.identityVersion).toBe(2);
});
test('identity-version starts at 1 and increments via bump', async () => {
expect(await storage.getPeerIdentityVersion('alice')).toBe(1);
expect(await storage.bumpPeerIdentityVersion('alice')).toBe(2);
expect(await storage.bumpPeerIdentityVersion('alice')).toBe(3);
expect(await storage.getPeerIdentityVersion('alice')).toBe(3);
// Independent counter per peer
expect(await storage.getPeerIdentityVersion('bob')).toBe(1);
});
test('survives reopen', async () => {
await storage.savePeerVerification({
peerAddress: 'bob',
fingerprint: 'fp',
verifiedAt: 42,
verifiedBy: 'user',
identityVersion: 5,
});
await storage.bumpPeerIdentityVersion('bob');
await storage.close();
storage = await IndexedDBStorage.create({ dbName });
const v = await storage.getPeerVerification('bob');
expect(v!.fingerprint).toBe('fp');
expect(v!.identityVersion).toBe(5);
expect(await storage.getPeerIdentityVersion('bob')).toBe(2);
});
});

View File

@@ -0,0 +1,9 @@
{
"extends": "../../tsconfig.json",
"compilerOptions": {
"outDir": "dist",
"rootDir": "src",
"lib": ["ES2022", "DOM"]
},
"include": ["src"]
}

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/storage-postgres", "name": "@shade/storage-postgres",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -223,9 +223,16 @@ export async function ensureInboxServerTables(sql: Sql): Promise<void> {
ciphertext TEXT NOT NULL, ciphertext TEXT NOT NULL,
received_at BIGINT NOT NULL, received_at BIGINT NOT NULL,
expires_at BIGINT NOT NULL, expires_at BIGINT NOT NULL,
sender_fp TEXT,
PRIMARY KEY (address, msg_id) PRIMARY KEY (address, msg_id)
) )
`; `;
// V4.8 — sender fingerprint column. Idempotent ADD COLUMN for live
// databases that came up under a 4.7-or-earlier schema.
await sql`
ALTER TABLE shade_inbox_blobs
ADD COLUMN IF NOT EXISTS sender_fp TEXT
`;
await sql` await sql`
CREATE INDEX IF NOT EXISTS shade_inbox_addr_expires_idx CREATE INDEX IF NOT EXISTS shade_inbox_addr_expires_idx
ON shade_inbox_blobs(address, expires_at) ON shade_inbox_blobs(address, expires_at)

View File

@@ -56,18 +56,20 @@ export class PostgresInboxStore implements InboxStore {
msgId: string; msgId: string;
ciphertext: Uint8Array; ciphertext: Uint8Array;
expiresAt: number; expiresAt: number;
senderFp?: string;
}): Promise<{ created: boolean; receivedAt: number }> { }): Promise<{ created: boolean; receivedAt: number }> {
// ON CONFLICT DO NOTHING + RETURNING keeps it idempotent and atomic. // ON CONFLICT DO NOTHING + RETURNING keeps it idempotent and atomic.
// When a row already exists, we look up its received_at in a follow-up // When a row already exists, we look up its received_at in a follow-up
// SELECT. // SELECT.
const inserted = await this.sql<Array<{ received_at: string }>>` const inserted = await this.sql<Array<{ received_at: string }>>`
INSERT INTO shade_inbox_blobs (address, msg_id, ciphertext, received_at, expires_at) INSERT INTO shade_inbox_blobs (address, msg_id, ciphertext, received_at, expires_at, sender_fp)
VALUES ( VALUES (
${args.address}, ${args.address},
${args.msgId}, ${args.msgId},
${toBase64(args.ciphertext)}, ${toBase64(args.ciphertext)},
nextval('shade_inbox_seq'), nextval('shade_inbox_seq'),
${args.expiresAt} ${args.expiresAt},
${args.senderFp ?? null}
) )
ON CONFLICT (address, msg_id) DO NOTHING ON CONFLICT (address, msg_id) DO NOTHING
RETURNING received_at::text RETURNING received_at::text
@@ -90,14 +92,23 @@ export class PostgresInboxStore implements InboxStore {
sinceCursor: number; sinceCursor: number;
now: number; now: number;
limit: number; limit: number;
}): Promise<Array<{ msgId: string; ciphertext: Uint8Array; receivedAt: number; expiresAt: number }>> { }): Promise<
Array<{
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
senderFp?: string;
}>
> {
const rows = await this.sql<Array<{ const rows = await this.sql<Array<{
msg_id: string; msg_id: string;
ciphertext: string; ciphertext: string;
received_at: string; received_at: string;
expires_at: string; expires_at: string;
sender_fp: string | null;
}>>` }>>`
SELECT msg_id, ciphertext, received_at::text, expires_at::text SELECT msg_id, ciphertext, received_at::text, expires_at::text, sender_fp
FROM shade_inbox_blobs FROM shade_inbox_blobs
WHERE address = ${args.address} WHERE address = ${args.address}
AND received_at > ${args.sinceCursor} AND received_at > ${args.sinceCursor}
@@ -105,12 +116,22 @@ export class PostgresInboxStore implements InboxStore {
ORDER BY received_at ASC ORDER BY received_at ASC
LIMIT ${args.limit} LIMIT ${args.limit}
`; `;
return rows.map((r) => ({ return rows.map((r) => {
const out: {
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
senderFp?: string;
} = {
msgId: r.msg_id, msgId: r.msg_id,
ciphertext: fromBase64(r.ciphertext), ciphertext: fromBase64(r.ciphertext),
receivedAt: parseInt(r.received_at, 10), receivedAt: parseInt(r.received_at, 10),
expiresAt: parseInt(r.expires_at, 10), expiresAt: parseInt(r.expires_at, 10),
})); };
if (r.sender_fp) out.senderFp = r.sender_fp;
return out;
});
} }
async deleteBlob(address: string, msgId: string): Promise<boolean> { async deleteBlob(address: string, msgId: string): Promise<boolean> {

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/storage-sqlite", "name": "@shade/storage-sqlite",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -53,6 +53,7 @@ export class SqliteInboxStore implements InboxStore {
ciphertext TEXT NOT NULL, ciphertext TEXT NOT NULL,
received_at INTEGER NOT NULL, received_at INTEGER NOT NULL,
expires_at INTEGER NOT NULL, expires_at INTEGER NOT NULL,
sender_fp TEXT,
PRIMARY KEY (address, msg_id) PRIMARY KEY (address, msg_id)
); );
CREATE INDEX IF NOT EXISTS idx_inbox_addr_expires CREATE INDEX IF NOT EXISTS idx_inbox_addr_expires
@@ -62,6 +63,14 @@ export class SqliteInboxStore implements InboxStore {
CREATE INDEX IF NOT EXISTS idx_inbox_expires CREATE INDEX IF NOT EXISTS idx_inbox_expires
ON inbox_blobs(expires_at); ON inbox_blobs(expires_at);
`); `);
// V4.8 — sender fingerprint column. Idempotent ALTER for live
// databases that came up under a 4.7-or-earlier schema.
const cols = this.db
.prepare(`PRAGMA table_info(inbox_blobs)`)
.all() as Array<{ name: string }>;
if (!cols.some((c) => c.name === 'sender_fp')) {
this.db.exec(`ALTER TABLE inbox_blobs ADD COLUMN sender_fp TEXT`);
}
} }
private prepareStatements() { private prepareStatements() {
@@ -73,13 +82,13 @@ export class SqliteInboxStore implements InboxStore {
deleteOwner: this.db.prepare('DELETE FROM inbox_owners WHERE address = ?'), deleteOwner: this.db.prepare('DELETE FROM inbox_owners WHERE address = ?'),
deleteOwnerBlobs: this.db.prepare('DELETE FROM inbox_blobs WHERE address = ?'), deleteOwnerBlobs: this.db.prepare('DELETE FROM inbox_blobs WHERE address = ?'),
insertBlob: this.db.prepare( insertBlob: this.db.prepare(
'INSERT INTO inbox_blobs (address, msg_id, ciphertext, received_at, expires_at) VALUES (?, ?, ?, ?, ?)', 'INSERT INTO inbox_blobs (address, msg_id, ciphertext, received_at, expires_at, sender_fp) VALUES (?, ?, ?, ?, ?, ?)',
), ),
findBlob: this.db.prepare( findBlob: this.db.prepare(
'SELECT received_at FROM inbox_blobs WHERE address = ? AND msg_id = ?', 'SELECT received_at FROM inbox_blobs WHERE address = ? AND msg_id = ?',
), ),
fetchSince: this.db.prepare( fetchSince: this.db.prepare(
`SELECT msg_id, ciphertext, received_at, expires_at `SELECT msg_id, ciphertext, received_at, expires_at, sender_fp
FROM inbox_blobs FROM inbox_blobs
WHERE address = ? AND received_at > ? AND expires_at > ? WHERE address = ? AND received_at > ? AND expires_at > ?
ORDER BY received_at ASC ORDER BY received_at ASC
@@ -124,6 +133,7 @@ export class SqliteInboxStore implements InboxStore {
msgId: string; msgId: string;
ciphertext: Uint8Array; ciphertext: Uint8Array;
expiresAt: number; expiresAt: number;
senderFp?: string;
}): Promise<{ created: boolean; receivedAt: number }> { }): Promise<{ created: boolean; receivedAt: number }> {
const existing = this.stmts.findBlob.get(args.address, args.msgId) as const existing = this.stmts.findBlob.get(args.address, args.msgId) as
| { received_at: number } | { received_at: number }
@@ -139,6 +149,7 @@ export class SqliteInboxStore implements InboxStore {
toBase64(args.ciphertext), toBase64(args.ciphertext),
receivedAt, receivedAt,
args.expiresAt, args.expiresAt,
args.senderFp ?? null,
); );
return { created: true, receivedAt }; return { created: true, receivedAt };
} }
@@ -148,19 +159,43 @@ export class SqliteInboxStore implements InboxStore {
sinceCursor: number; sinceCursor: number;
now: number; now: number;
limit: number; limit: number;
}): Promise<Array<{ msgId: string; ciphertext: Uint8Array; receivedAt: number; expiresAt: number }>> { }): Promise<
Array<{
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
senderFp?: string;
}>
> {
const rows = this.stmts.fetchSince.all( const rows = this.stmts.fetchSince.all(
args.address, args.address,
args.sinceCursor, args.sinceCursor,
args.now, args.now,
args.limit, args.limit,
) as Array<{ msg_id: string; ciphertext: string; received_at: number; expires_at: number }>; ) as Array<{
return rows.map((r) => ({ msg_id: string;
ciphertext: string;
received_at: number;
expires_at: number;
sender_fp: string | null;
}>;
return rows.map((r) => {
const out: {
msgId: string;
ciphertext: Uint8Array;
receivedAt: number;
expiresAt: number;
senderFp?: string;
} = {
msgId: r.msg_id, msgId: r.msg_id,
ciphertext: fromBase64(r.ciphertext), ciphertext: fromBase64(r.ciphertext),
receivedAt: r.received_at, receivedAt: r.received_at,
expiresAt: r.expires_at, expiresAt: r.expires_at,
})); };
if (r.sender_fp) out.senderFp = r.sender_fp;
return out;
});
} }
async deleteBlob(address: string, msgId: string): Promise<boolean> { async deleteBlob(address: string, msgId: string): Promise<boolean> {

View File

@@ -1,5 +1,9 @@
import { Database } from 'bun:sqlite'; import { Database } from 'bun:sqlite';
import type { StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey, SessionState, RetiredIdentity, PersistedStreamState, PeerVerification, PeerVerificationSource } from '@shade/core'; import type {
StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey,
SessionState, RetiredIdentity, PersistedStreamState, PeerVerification,
PeerVerificationSource, BroadcastChannelRecord, BroadcastMemberRecord,
} from '@shade/core';
import { import {
toBase64, fromBase64, toBase64, fromBase64,
constantTimeEqual, constantTimeEqual,
@@ -53,6 +57,14 @@ export class SQLiteStorage implements StorageProvider {
removePeerVerification: ReturnType<Database['prepare']>; removePeerVerification: ReturnType<Database['prepare']>;
getPeerIdentityVersion: ReturnType<Database['prepare']>; getPeerIdentityVersion: ReturnType<Database['prepare']>;
upsertPeerIdentityVersion: ReturnType<Database['prepare']>; upsertPeerIdentityVersion: ReturnType<Database['prepare']>;
saveBroadcastChannel: ReturnType<Database['prepare']>;
getBroadcastChannel: ReturnType<Database['prepare']>;
listBroadcastChannels: ReturnType<Database['prepare']>;
removeBroadcastChannel: ReturnType<Database['prepare']>;
removeBroadcastChannelMembers: ReturnType<Database['prepare']>;
saveBroadcastMember: ReturnType<Database['prepare']>;
getBroadcastMembers: ReturnType<Database['prepare']>;
removeBroadcastMember: ReturnType<Database['prepare']>;
}; };
constructor(dbPath?: string) { constructor(dbPath?: string) {
@@ -127,6 +139,27 @@ export class SQLiteStorage implements StorageProvider {
peer_address TEXT PRIMARY KEY, peer_address TEXT PRIMARY KEY,
version INTEGER NOT NULL version INTEGER NOT NULL
); );
CREATE TABLE IF NOT EXISTS broadcast_channels (
channel_id TEXT PRIMARY KEY,
owner_role TEXT NOT NULL,
owner_address TEXT NOT NULL,
label TEXT,
generation INTEGER NOT NULL,
chain_key BLOB NOT NULL,
iteration INTEGER NOT NULL,
signing_public_key BLOB NOT NULL,
signing_private_key BLOB,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS broadcast_members (
channel_id TEXT NOT NULL,
peer_address TEXT NOT NULL,
joined_at INTEGER NOT NULL,
removed_at INTEGER,
PRIMARY KEY (channel_id, peer_address)
);
CREATE INDEX IF NOT EXISTS idx_broadcast_members_channel ON broadcast_members(channel_id);
`); `);
} }
@@ -183,6 +216,36 @@ export class SQLiteStorage implements StorageProvider {
`INSERT INTO peer_identity_versions (peer_address, version) VALUES (?, ?) `INSERT INTO peer_identity_versions (peer_address, version) VALUES (?, ?)
ON CONFLICT(peer_address) DO UPDATE SET version = excluded.version`, ON CONFLICT(peer_address) DO UPDATE SET version = excluded.version`,
), ),
saveBroadcastChannel: this.db.prepare(
`INSERT OR REPLACE INTO broadcast_channels
(channel_id, owner_role, owner_address, label, generation,
chain_key, iteration, signing_public_key, signing_private_key,
created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
),
getBroadcastChannel: this.db.prepare(
'SELECT * FROM broadcast_channels WHERE channel_id = ?',
),
listBroadcastChannels: this.db.prepare(
'SELECT * FROM broadcast_channels ORDER BY created_at ASC',
),
removeBroadcastChannel: this.db.prepare(
'DELETE FROM broadcast_channels WHERE channel_id = ?',
),
removeBroadcastChannelMembers: this.db.prepare(
'DELETE FROM broadcast_members WHERE channel_id = ?',
),
saveBroadcastMember: this.db.prepare(
`INSERT OR REPLACE INTO broadcast_members
(channel_id, peer_address, joined_at, removed_at)
VALUES (?, ?, ?, ?)`,
),
getBroadcastMembers: this.db.prepare(
'SELECT channel_id, peer_address, joined_at, removed_at FROM broadcast_members WHERE channel_id = ? ORDER BY joined_at ASC',
),
removeBroadcastMember: this.db.prepare(
'DELETE FROM broadcast_members WHERE channel_id = ? AND peer_address = ?',
),
}; };
} }
@@ -392,6 +455,103 @@ export class SQLiteStorage implements StorageProvider {
this.stmts.upsertPeerIdentityVersion.run(address, next); this.stmts.upsertPeerIdentityVersion.run(address, next);
return next; return next;
} }
// ─── Broadcast channels (V4.6) ────────────────────────────
async saveBroadcastChannel(channel: BroadcastChannelRecord): Promise<void> {
this.stmts.saveBroadcastChannel.run(
channel.channelId,
channel.ownerRole,
channel.ownerAddress,
channel.label ?? null,
channel.generation,
channel.chainKey,
channel.iteration,
channel.signingPublicKey,
channel.signingPrivateKey ?? null,
channel.createdAt,
channel.updatedAt,
);
}
async getBroadcastChannel(channelId: string): Promise<BroadcastChannelRecord | null> {
const row = this.stmts.getBroadcastChannel.get(channelId) as BroadcastChannelRow | undefined;
if (!row) return null;
return rowToBroadcastChannel(row);
}
async listBroadcastChannels(): Promise<BroadcastChannelRecord[]> {
const rows = this.stmts.listBroadcastChannels.all() as BroadcastChannelRow[];
return rows.map(rowToBroadcastChannel);
}
async removeBroadcastChannel(channelId: string): Promise<void> {
this.stmts.removeBroadcastChannelMembers.run(channelId);
this.stmts.removeBroadcastChannel.run(channelId);
}
async saveBroadcastMember(member: BroadcastMemberRecord): Promise<void> {
this.stmts.saveBroadcastMember.run(
member.channelId,
member.peerAddress,
member.joinedAt,
member.removedAt,
);
}
async getBroadcastMembers(channelId: string): Promise<BroadcastMemberRecord[]> {
const rows = this.stmts.getBroadcastMembers.all(channelId) as BroadcastMemberRow[];
return rows.map((r) => ({
channelId: r.channel_id,
peerAddress: r.peer_address,
joinedAt: Number(r.joined_at),
removedAt: r.removed_at === null || r.removed_at === undefined ? null : Number(r.removed_at),
}));
}
async removeBroadcastMember(channelId: string, peerAddress: string): Promise<void> {
this.stmts.removeBroadcastMember.run(channelId, peerAddress);
}
}
interface BroadcastChannelRow {
channel_id: string;
owner_role: 'sender' | 'receiver';
owner_address: string;
label: string | null;
generation: number | bigint;
chain_key: Uint8Array | ArrayBuffer;
iteration: number | bigint;
signing_public_key: Uint8Array | ArrayBuffer;
signing_private_key: Uint8Array | ArrayBuffer | null;
created_at: number | bigint;
updated_at: number | bigint;
}
interface BroadcastMemberRow {
channel_id: string;
peer_address: string;
joined_at: number | bigint;
removed_at: number | bigint | null;
}
function rowToBroadcastChannel(r: BroadcastChannelRow): BroadcastChannelRecord {
const out: BroadcastChannelRecord = {
channelId: r.channel_id,
ownerRole: r.owner_role,
ownerAddress: r.owner_address,
generation: Number(r.generation),
chainKey: toBytes(r.chain_key),
iteration: Number(r.iteration),
signingPublicKey: toBytes(r.signing_public_key),
createdAt: Number(r.created_at),
updatedAt: Number(r.updated_at),
};
if (r.label !== null && r.label !== undefined) out.label = r.label;
if (r.signing_private_key !== null && r.signing_private_key !== undefined) {
out.signingPrivateKey = toBytes(r.signing_private_key);
}
return out;
} }
function rowToStreamState(row: any): PersistedStreamState { function rowToStreamState(row: any): PersistedStreamState {

View File

@@ -178,6 +178,104 @@ describe('SqliteInboxStore', () => {
expect(blobs.length).toBe(0); expect(blobs.length).toBe(0);
}); });
test('senderFp round-trips through put + fetch (V4.8)', async () => {
const ct = randBytes(40);
const fp = '0123456789abcdef';
await store.putBlob({
address: 'bob',
msgId: 'a'.repeat(64),
ciphertext: ct,
expiresAt: Date.now() + 60_000,
senderFp: fp,
});
const blobs = await store.fetchBlobs({
address: 'bob',
sinceCursor: 0,
now: Date.now(),
limit: 10,
});
expect(blobs.length).toBe(1);
expect(blobs[0]!.senderFp).toBe(fp);
});
test('senderFp omitted on put → fetched row has senderFp undefined (V4.8 backward-compat)', async () => {
const ct = randBytes(40);
await store.putBlob({
address: 'bob',
msgId: 'b'.repeat(64),
ciphertext: ct,
expiresAt: Date.now() + 60_000,
});
const blobs = await store.fetchBlobs({
address: 'bob',
sinceCursor: 0,
now: Date.now(),
limit: 10,
});
expect(blobs.length).toBe(1);
expect(blobs[0]!.senderFp).toBeUndefined();
});
test('ALTER TABLE migration adds sender_fp to a pre-4.8 schema (V4.8)', async () => {
// Reproduce a pre-4.8 schema in a fresh DB, then reopen via
// SqliteInboxStore which should run the idempotent ALTER without
// dropping the existing rows.
store.close();
try {
unlinkSync(dbPath);
} catch {}
try {
unlinkSync(dbPath + '-wal');
} catch {}
try {
unlinkSync(dbPath + '-shm');
} catch {}
const { Database } = await import('bun:sqlite');
const legacy = new Database(dbPath, { create: true });
legacy.exec(`
CREATE TABLE inbox_blobs (
address TEXT NOT NULL,
msg_id TEXT NOT NULL,
ciphertext TEXT NOT NULL,
received_at INTEGER NOT NULL,
expires_at INTEGER NOT NULL,
PRIMARY KEY (address, msg_id)
);
`);
legacy.prepare(
'INSERT INTO inbox_blobs (address, msg_id, ciphertext, received_at, expires_at) VALUES (?, ?, ?, ?, ?)',
).run('bob', 'c'.repeat(64), 'AAAA', Date.now(), Date.now() + 60_000);
legacy.close();
store = new SqliteInboxStore(dbPath);
const blobs = await store.fetchBlobs({
address: 'bob',
sinceCursor: 0,
now: Date.now(),
limit: 10,
});
expect(blobs.length).toBe(1);
expect(blobs[0]!.senderFp).toBeUndefined();
// New writes after migration carry senderFp.
await store.putBlob({
address: 'bob',
msgId: 'd'.repeat(64),
ciphertext: randBytes(8),
expiresAt: Date.now() + 60_000,
senderFp: 'feedfacedeadbeef',
});
const after = await store.fetchBlobs({
address: 'bob',
sinceCursor: 0,
now: Date.now(),
limit: 10,
});
const newer = after.find((b) => b.msgId === 'd'.repeat(64));
expect(newer?.senderFp).toBe('feedfacedeadbeef');
});
test('countBlobs ignores expired entries', async () => { test('countBlobs ignores expired entries', async () => {
const now = Date.now(); const now = Date.now();
await store.putBlob({ await store.putBlob({

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/streams", "name": "@shade/streams",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/transfer", "name": "@shade/transfer",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -7,6 +7,7 @@ export * from './transport/memory.js';
export * from './transport/http-transport.js'; export * from './transport/http-transport.js';
export * from './transport/ws-transport.js'; export * from './transport/ws-transport.js';
export * from './transport/multi-fallback.js'; export * from './transport/multi-fallback.js';
export * from './transport/queue-transport.js';
export * from './engine.js'; export * from './engine.js';
export { export {
createTransferRoutes, createTransferRoutes,

View File

@@ -0,0 +1,332 @@
/**
* Per-peer outbound queue + queue-transport for browser-friendly
* pull-mode streams.
*
* The default `ShadeTransferHttpTransport` POSTs each chunk directly
* to the receiver's `/v1/transfer/<streamId>/chunk` route. That
* requires the receiver to host an HTTP server, which a browser tab
* cannot. `QueueTransferTransport` flips the direction: it queues
* chunks per peer and lets the receiver pull them via a long-poll
* endpoint.
*
* The companion `OutboundQueue` data structure is plain server-side
* state — wired up by `@shade/files`'s `transferQueueRoute()` (and any
* future consumer) to expose the long-poll surface and feed envelopes
* + chunks into the browser receiver.
*/
import type {
ChunkAck,
ChunkSendOptions,
ITransferTransport,
TransferResumeState,
} from './transport.js';
import { TransferTransportError } from '../errors.js';
/**
* Discriminated union of items the queue ships to the browser
* receiver. Both kinds carry **wire-encoded bytes** of an envelope —
* the receiver decodes via `decodeEnvelope` (control envelopes) or
* forwards directly to `engine.receiveChunk` (chunk envelopes).
*/
export type QueuedEvent =
| {
id: number;
timestampMs: number;
kind: 'envelope';
/** Wire-encoded `0x02` ratchet envelope (or `0x01` first-message). */
bytes: Uint8Array;
}
| {
id: number;
timestampMs: number;
kind: 'chunk';
/** Wire-encoded `0x11` stream-chunk envelope. */
bytes: Uint8Array;
meta: {
streamId: string;
laneId: number;
seq: number;
};
};
/** Caller-supplied shape for {@link OutboundQueue.enqueue} — the queue assigns `id` + `timestampMs`. */
export type QueuedEventInput =
| { kind: 'envelope'; bytes: Uint8Array }
| {
kind: 'chunk';
bytes: Uint8Array;
meta: { streamId: string; laneId: number; seq: number };
};
export interface OutboundQueueOptions {
/**
* Maximum events held per peer. When the queue is full, the oldest
* unacked event is dropped on next enqueue. Default 1000 — at the
* default chunk size (256 KiB plaintext) this caps a single peer's
* outbound buffer at ~256 MiB. Tune up for fewer/bigger streams,
* down for many concurrent small flows.
*/
maxEventsPerPeer?: number;
/**
* After a peer has not polled for this long, the queue's events are
* dropped and any pending waiters are released. Default 10 minutes.
* Setting to `0` disables idle-eviction.
*/
idleEvictionMs?: number;
}
const DEFAULT_MAX_EVENTS = 1000;
const DEFAULT_IDLE_EVICTION_MS = 10 * 60 * 1000;
interface PendingWaiter {
resolve(events: QueuedEvent[]): void;
reject(err: Error): void;
/**
* The waiter's `since` cursor — only events with `id > since` should
* be delivered when this waiter is resolved. Without this, an
* enqueue that arrives while a poller is waiting would replay
* already-processed events, causing the receiver to double-decrypt
* (and corrupt ratchet state).
*/
since: number;
timer: ReturnType<typeof setTimeout>;
abortHandler?: () => void;
signal?: AbortSignal;
}
interface PerPeerState {
nextId: number;
events: QueuedEvent[];
waiters: PendingWaiter[];
lastTouchedMs: number;
}
/**
* Per-peer monotonic event log with long-poll semantics.
*
* `enqueue` appends; `drain` returns all events with `id > since`,
* blocking up to `blockMs` if there are none. `since`-based pagination
* is the resume mechanism: a client crashing mid-stream restarts with
* its last-processed id and the queue replays everything after it
* (subject to `maxEventsPerPeer` retention).
*/
export class OutboundQueue {
private peers = new Map<string, PerPeerState>();
private readonly maxEvents: number;
private readonly idleEvictionMs: number;
private evictTimer: ReturnType<typeof setTimeout> | null = null;
private destroyed = false;
constructor(opts: OutboundQueueOptions = {}) {
this.maxEvents = opts.maxEventsPerPeer ?? DEFAULT_MAX_EVENTS;
this.idleEvictionMs = opts.idleEvictionMs ?? DEFAULT_IDLE_EVICTION_MS;
if (this.idleEvictionMs > 0) this.scheduleEviction();
}
/** Append an event and wake any waiters for that peer. */
enqueue(peer: string, ev: QueuedEventInput): QueuedEvent {
if (this.destroyed) throw new Error('OutboundQueue: destroyed');
const state = this.getOrCreate(peer);
const event: QueuedEvent =
ev.kind === 'chunk'
? {
id: state.nextId++,
timestampMs: Date.now(),
kind: 'chunk',
bytes: ev.bytes,
meta: ev.meta,
}
: {
id: state.nextId++,
timestampMs: Date.now(),
kind: 'envelope',
bytes: ev.bytes,
};
state.events.push(event);
state.lastTouchedMs = Date.now();
// Cap: drop oldest. Lost events trigger receiver-side resume from
// last polled id; the @shade/transfer engine handles missing seqs
// by re-sending on resume.
while (state.events.length > this.maxEvents) state.events.shift();
// Wake each waiter with events newer than ITS OWN `since`. Using a
// shared snapshot from `since=0` would replay events the waiter has
// already processed once a fresh enqueue arrived mid-poll, which on
// the receiver side double-dispatches an envelope into shade.receive
// → manager.decrypt and consumes the same skipped-key twice (the
// second dispatch corrupts the ratchet chain).
if (state.waiters.length > 0) {
const waiters = state.waiters.splice(0);
for (const w of waiters) {
clearTimeout(w.timer);
if (w.abortHandler !== undefined && w.signal !== undefined) {
w.signal.removeEventListener('abort', w.abortHandler);
}
const wDrained = this.collect(state, w.since);
w.resolve(wDrained);
}
}
return event;
}
/**
* Drain events with `id > since`. If none are available, block up
* to `blockMs` until any arrive. `signal` cancels the wait early.
*/
async drain(
peer: string,
since: number,
blockMs: number,
signal?: AbortSignal,
): Promise<QueuedEvent[]> {
if (this.destroyed) throw new Error('OutboundQueue: destroyed');
const state = this.getOrCreate(peer);
state.lastTouchedMs = Date.now();
const ready = this.collect(state, since);
if (ready.length > 0 || blockMs <= 0) return ready;
if (signal?.aborted) return [];
return await new Promise<QueuedEvent[]>((resolve, reject) => {
const timer = setTimeout(() => {
const idx = state.waiters.indexOf(waiter);
if (idx >= 0) state.waiters.splice(idx, 1);
if (waiter.abortHandler !== undefined && waiter.signal !== undefined) {
waiter.signal.removeEventListener('abort', waiter.abortHandler);
}
// Empty drain on timeout — that's the "no new events" signal.
resolve([]);
}, blockMs);
const waiter: PendingWaiter = { resolve, reject, since, timer };
if (signal !== undefined) {
const handler = () => {
const idx = state.waiters.indexOf(waiter);
if (idx >= 0) state.waiters.splice(idx, 1);
clearTimeout(timer);
resolve([]);
};
signal.addEventListener('abort', handler, { once: true });
waiter.abortHandler = handler;
waiter.signal = signal;
}
state.waiters.push(waiter);
});
}
/** Drop a peer's queue + reject waiters. */
evict(peer: string): void {
const state = this.peers.get(peer);
if (state === undefined) return;
this.peers.delete(peer);
for (const w of state.waiters) {
clearTimeout(w.timer);
if (w.abortHandler !== undefined && w.signal !== undefined) {
w.signal.removeEventListener('abort', w.abortHandler);
}
w.reject(new Error('OutboundQueue: peer evicted'));
}
}
/** Peer-specific snapshot for diagnostics. */
stats(peer: string): { eventCount: number; nextId: number; waiters: number } | null {
const state = this.peers.get(peer);
if (state === undefined) return null;
return {
eventCount: state.events.length,
nextId: state.nextId,
waiters: state.waiters.length,
};
}
/** Tear everything down. Pending waiters are rejected. */
destroy(): void {
if (this.destroyed) return;
this.destroyed = true;
if (this.evictTimer !== null) clearTimeout(this.evictTimer);
for (const peer of [...this.peers.keys()]) this.evict(peer);
}
// ─── internals ──────────────────────────────────────────────
private getOrCreate(peer: string): PerPeerState {
let state = this.peers.get(peer);
if (state === undefined) {
state = {
nextId: 1,
events: [],
waiters: [],
lastTouchedMs: Date.now(),
};
this.peers.set(peer, state);
}
return state;
}
private collect(state: PerPeerState, since: number): QueuedEvent[] {
if (state.events.length === 0) return [];
return state.events.filter((e) => e.id > since);
}
private scheduleEviction(): void {
const interval = Math.max(60_000, Math.floor(this.idleEvictionMs / 4));
this.evictTimer = setTimeout(() => {
if (this.destroyed) return;
const cutoff = Date.now() - this.idleEvictionMs;
for (const [peer, state] of this.peers.entries()) {
if (state.lastTouchedMs < cutoff) this.evict(peer);
}
this.scheduleEviction();
}, interval);
(this.evictTimer as unknown as { unref?: () => void }).unref?.();
}
}
/**
* Chunk transport that enqueues into an {@link OutboundQueue} instead
* of POSTing.
*
* Returns an optimistic `ChunkAck` immediately because the queue *is*
* the delivery — the receiver polls and dispatches. Browser receivers
* cannot synchronously confirm receipt before the next chunk; the
* engine's stream-protocol catches dropped chunks at finish-time
* integrity check, and chunk-resume restarts the lane from the last
* polled `since`.
*/
export class QueueTransferTransport implements ITransferTransport {
constructor(private readonly queue: OutboundQueue) {}
async probe(_peer: string): Promise<void> {
// The queue is local. Reachability is "is there a poller?" which
// is decided by `idleEvictionMs`. We don't synchronously check
// here; the engine retries via `withRetry` on `sendChunk` errors.
}
async sendChunk(
peer: string,
streamId: string,
laneId: number,
seq: number | bigint,
bytes: Uint8Array,
options?: ChunkSendOptions,
): Promise<ChunkAck> {
if (options?.signal?.aborted) {
throw new TransferTransportError('sendChunk aborted by caller');
}
const seqNum = typeof seq === 'bigint' ? Number(seq) : seq;
this.queue.enqueue(peer, {
kind: 'chunk',
bytes,
meta: { streamId, laneId, seq: seqNum },
});
return { lastSeq: seqNum };
}
async fetchResumeState(
_peer: string,
_streamId: string,
): Promise<TransferResumeState | null> {
// Pull-mode receivers report resume state by re-polling with the
// `since` cursor they last successfully processed; the queue does
// not need to query the receiver. Return null so the engine
// restarts from seq 0 (deterministic), and the queue replays from
// `since=0` if the client reconnects fresh.
return null;
}
}

View File

@@ -0,0 +1,60 @@
import { describe, expect, test } from 'bun:test';
import { OutboundQueue } from '../src/index.js';
/**
* Regression coverage for the long-poll waiter `since` cursor.
*
* The bug being guarded against: when `enqueue` woke a pending
* `drain` waiter, it used a `since=0` snapshot and replayed every
* event that had ever been queued — including the ones the waiter
* had already processed in a previous poll. Downstream the queue
* fed `Shade.acceptTransferEnvelope`, so the duplicate replay
* dispatched the same envelope into `manager.decrypt` twice. The
* second decrypt consumed an already-used skipped key, fell into
* the stale-counter branch of `ratchetDecrypt`, and corrupted the
* Double Ratchet receive chain — surfacing as
* `DecryptionError: wrong key or tampered data` on every
* subsequent message.
*/
describe('OutboundQueue — waiter since cursor', () => {
test('mid-poll enqueue must not replay events the waiter already saw', async () => {
const queue = new OutboundQueue({ idleEvictionMs: 0 });
const peer = 'alice';
const e1 = queue.enqueue(peer, { kind: 'envelope', bytes: new Uint8Array([1]) });
const e2 = queue.enqueue(peer, { kind: 'envelope', bytes: new Uint8Array([2]) });
// First poll drains both events (no blocking — they're already there).
const first = await queue.drain(peer, 0, 0);
expect(first.map((e) => e.id)).toEqual([e1.id, e2.id]);
// Now the waiter polls past the last seen id. It blocks because
// there are no events newer than `since`. Concurrently a fresh
// event gets enqueued — that's the path the bug fired on.
const blockMs = 5_000;
const polling = queue.drain(peer, e2.id, blockMs);
// Yield so `drain` actually parks on the waiter list before we
// race the enqueue against it.
await Promise.resolve();
const e3 = queue.enqueue(peer, { kind: 'envelope', bytes: new Uint8Array([3]) });
const woken = await polling;
// Pre-fix: would resolve with [e1, e2, e3] (a `since=0` snapshot
// drained verbatim). Post-fix: only the events newer than the
// waiter's recorded `since` come through.
expect(woken.map((e) => e.id)).toEqual([e3.id]);
});
test('parked waiter at the head still gets the new event when others have polled past it', async () => {
const queue = new OutboundQueue({ idleEvictionMs: 0 });
const peer = 'alice';
const e1 = queue.enqueue(peer, { kind: 'envelope', bytes: new Uint8Array([1]) });
// A waiter that parks past the head — there are no events newer
// than e1.id, so it has to block.
const polling = queue.drain(peer, e1.id, 5_000);
await Promise.resolve();
const e2 = queue.enqueue(peer, { kind: 'envelope', bytes: new Uint8Array([2]) });
const woken = await polling;
expect(woken.map((e) => e.id)).toEqual([e2.id]);
});
});

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/transport-bridge", "name": "@shade/transport-bridge",
"version": "4.0.1", "version": "4.8.2",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -62,3 +62,39 @@ export function bridgeQueryToCanonical(qs: URLSearchParams): {
if (!Number.isFinite(signedAt)) return null; if (!Number.isFinite(signedAt)) return null;
return { address, kind, since, signedAt, signature }; return { address, kind, since, signedAt, signature };
} }
// ─── V4.7 — presence subscription auth ────────────────────────────
export interface PresenceAuthInput {
crypto: CryptoProvider;
signingPrivateKey: Uint8Array;
/** The watcher's own address (signer of the request). */
address: string;
/** Addresses to subscribe presence updates for. May be empty. */
watched: readonly string[];
}
/**
* Build the signed query string for `GET /v1/bridge/presence`. The
* `kind: 'presence'` field is bound into the canonical payload to
* prevent cross-endpoint replay against `/v1/bridge/{stream,poll,ws}`.
*
* The `watched` array is sorted to give the signed bytes a canonical
* order; the wire form encodes it as a single comma-separated
* `watched=` parameter (address syntax disallows `,`).
*/
export async function signPresenceQuery(input: PresenceAuthInput): Promise<URLSearchParams> {
const watched = [...input.watched].sort();
const signed = await signPayload(input.crypto, input.signingPrivateKey, {
address: input.address,
kind: 'presence',
watched,
});
const qs = new URLSearchParams();
qs.set('address', input.address);
qs.set('kind', 'presence');
qs.set('watched', watched.join(','));
qs.set('signedAt', String(signed.signedAt));
qs.set('signature', signed.signature);
return qs;
}

View File

@@ -23,8 +23,17 @@ export { decodeWireMessage } from './types.js';
export { BridgeError } from './errors.js'; export { BridgeError } from './errors.js';
export { signBridgeQuery, bridgeQueryToCanonical } from './auth.js'; export { signBridgeQuery, bridgeQueryToCanonical, signPresenceQuery } from './auth.js';
export type { BridgeKind, BridgeAuthInput } from './auth.js'; export type { BridgeKind, BridgeAuthInput, PresenceAuthInput } from './auth.js';
export { PresenceBridge } from './presence-bridge.js';
export type {
PresenceBridgeOptions,
PresenceSubscribeOptions,
PresenceSubscription,
PresenceChange,
PresenceVia,
} from './presence-bridge.js';
export { SseBridge } from './sse-bridge.js'; export { SseBridge } from './sse-bridge.js';
export type { SseBridgeOptions } from './sse-bridge.js'; export type { SseBridgeOptions } from './sse-bridge.js';

View File

@@ -51,7 +51,10 @@ export class LongPollBridge implements BridgeTransport {
private loopPromise: Promise<void> | null = null; private loopPromise: Promise<void> | null = null;
constructor(private readonly options: LongPollBridgeOptions) { constructor(private readonly options: LongPollBridgeOptions) {
this.fetchFn = options.fetch ?? globalThis.fetch; // Bind to globalThis: browser `fetch` is a WebIDL bound operation
// and throws "Illegal invocation" when called as `this.fetchFn(...)`.
const f = options.fetch ?? globalThis.fetch;
this.fetchFn = f.bind(globalThis);
this.cursor = options.startCursor ?? 0; this.cursor = options.startCursor ?? 0;
} }

View File

@@ -0,0 +1,337 @@
/**
* V4.7 — presence subscription client.
*
* Consumes the SSE feed at `<base>/v1/bridge/presence?…` and fires
* `onPresenceChange` whenever a watched address transitions
* online/offline. Tracking is server-side: the inbox-server emits
* presence events on the 0↔1 boundary across WS + SSE bridge
* connections, and this client filters by the watcher's declared
* address list.
*
* Threat model context: the typical consumer (Prism, password
* managers, anything sender-key-broadcasting) wires this to
* `BroadcastChannel.removeMember` so a clean WS/SSE close on a
* paired-peer device revokes its sender-key membership within
* ~50ms. Long-poll bridges are deliberately NOT tracked on the
* server (see `inbox-server` `events.ts`); presence here is
* push-transport only.
*
* Watched-list mutations (`addPeer` / `removePeer`) trigger a
* reconnect with a fresh signed query so the server-side filter
* reflects the new set. Mutations are expected to be rare (only on
* pair / unpair, not on every message), so the brief reconnect gap
* is acceptable.
*/
import type { CryptoProvider } from '@shade/core';
import { signPresenceQuery } from './auth.js';
import { BridgeError } from './errors.js';
export type PresenceVia = 'ws' | 'sse';
export interface PresenceChange {
address: string;
status: 'online' | 'offline';
/** Server's wall-clock time (ms since epoch) when the change happened. */
at: number;
/** Which transport carried the connection. Absent on the initial snapshot. */
via?: PresenceVia;
}
export interface PresenceBridgeOptions {
/** Bridge base URL — same as `LongPollBridge` / `SseBridge`. */
baseUrl: string;
crypto: CryptoProvider;
/** Watcher's Ed25519 signing key (the address must be a registered inbox). */
signingPrivateKey: Uint8Array;
/** Watcher's address (the registered inbox owner). */
address: string;
/** Override `fetch` (tests). */
fetch?: typeof fetch;
/** Initial reconnect backoff (ms). Default 250. */
initialBackoffMs?: number;
/** Max reconnect backoff (ms). Default 10_000. */
maxBackoffMs?: number;
/** Disable automatic reconnect. Default false. */
disableAutoReconnect?: boolean;
}
export interface PresenceSubscribeOptions {
/** Initial set of addresses to watch. May be empty. */
watch: readonly string[];
/** Fired whenever a watched address transitions, plus once per address on initial open. */
onPresenceChange: (change: PresenceChange) => void | Promise<void>;
/** Optional reconnect / parse error reporter. */
onError?: (err: Error) => void;
}
export interface PresenceSubscription {
/** Add an address to the watched set. Triggers a reconnect. */
addPeer(address: string): Promise<void>;
/** Remove an address from the watched set. Triggers a reconnect. */
removePeer(address: string): Promise<void>;
/** Snapshot of the currently-watched addresses. */
watching(): readonly string[];
/** Tear down. Idempotent. */
unsubscribe(): Promise<void>;
}
const DEFAULT_INITIAL_BACKOFF = 250;
const DEFAULT_MAX_BACKOFF = 10_000;
export class PresenceBridge {
private readonly fetchFn: typeof fetch;
constructor(private readonly options: PresenceBridgeOptions) {
const f = options.fetch ?? globalThis.fetch;
this.fetchFn = f.bind(globalThis);
}
async subscribe(opts: PresenceSubscribeOptions): Promise<PresenceSubscription> {
const session = new PresenceSession(this.options, this.fetchFn, opts);
await session.start();
return session;
}
}
class PresenceSession implements PresenceSubscription {
private watched: string[];
private abortController: AbortController | null = null;
private currentReader: ReadableStreamDefaultReader<Uint8Array> | null = null;
private disposed = false;
private readLoopPromise: Promise<void> | null = null;
private readonly onPresenceChange: PresenceSubscribeOptions['onPresenceChange'];
private readonly onError: NonNullable<PresenceSubscribeOptions['onError']>;
private firstOpenResolve: (() => void) | null = null;
private firstOpenReject: ((err: Error) => void) | null = null;
private firstOpenSettled = false;
constructor(
private readonly options: PresenceBridgeOptions,
private readonly fetchFn: typeof fetch,
opts: PresenceSubscribeOptions,
) {
this.watched = [...opts.watch];
this.onPresenceChange = opts.onPresenceChange;
this.onError =
opts.onError ?? ((err) => console.warn('[shade-bridge:presence]', err.message));
}
watching(): readonly string[] {
return [...this.watched];
}
async start(): Promise<void> {
return this.openAndPump();
}
/**
* Open one SSE connection and resolve once the first response has
* been received (so that callers can `await subscribe()` and know
* the connection is established before the first state change).
* The read loop continues in the background.
*/
private openAndPump(): Promise<void> {
return new Promise<void>((resolve, reject) => {
this.firstOpenSettled = false;
this.firstOpenResolve = () => {
if (this.firstOpenSettled) return;
this.firstOpenSettled = true;
resolve();
};
this.firstOpenReject = (err: Error) => {
if (this.firstOpenSettled) return;
this.firstOpenSettled = true;
reject(err);
};
this.readLoopPromise = this.runLoop();
});
}
private async runLoop(): Promise<void> {
let backoff = this.options.initialBackoffMs ?? DEFAULT_INITIAL_BACKOFF;
const maxBackoff = this.options.maxBackoffMs ?? DEFAULT_MAX_BACKOFF;
let firstAttempt = true;
while (!this.disposed) {
try {
await this.openOnce();
if (firstAttempt) {
firstAttempt = false;
this.firstOpenResolve?.();
}
// Reset backoff on a successful open.
backoff = this.options.initialBackoffMs ?? DEFAULT_INITIAL_BACKOFF;
await this.consume();
} catch (err) {
if (this.disposed) return;
if (firstAttempt) {
// Failed before we ever got a 200 — surface to the caller of subscribe().
this.firstOpenReject?.(err as Error);
return;
}
this.onError(err as Error);
}
this.currentReader = null;
if (this.disposed || this.options.disableAutoReconnect) return;
await sleep(backoff);
backoff = Math.min(backoff * 2, maxBackoff);
}
}
private async openOnce(): Promise<void> {
const qs = await signPresenceQuery({
crypto: this.options.crypto,
signingPrivateKey: this.options.signingPrivateKey,
address: this.options.address,
watched: this.watched,
});
const url = `${stripTrailingSlash(this.options.baseUrl)}/v1/bridge/presence?${qs.toString()}`;
this.abortController = new AbortController();
let res: Response;
try {
res = await this.fetchFn(url, {
method: 'GET',
headers: { accept: 'text/event-stream', 'cache-control': 'no-cache' },
signal: this.abortController.signal,
});
} catch (err) {
throw new BridgeError(`presence connect failed: ${(err as Error).message}`);
}
if (!res.ok) {
throw new BridgeError(`presence connect failed: HTTP ${res.status}`, res.status);
}
if (!res.body) {
throw new BridgeError('presence response has no body');
}
this.currentReader = res.body.getReader() as ReadableStreamDefaultReader<Uint8Array>;
}
private async consume(): Promise<void> {
const reader = this.currentReader;
if (!reader) return;
const decoder = new TextDecoder();
let buf = '';
let dataLines: string[] = [];
let eventName: string | null = null;
while (true) {
let chunk: Awaited<ReturnType<typeof reader.read>>;
try {
chunk = await reader.read();
} catch (err) {
// Reader cancelled (mutation / unsubscribe) — exit cleanly.
if (this.disposed || (err as Error).name === 'AbortError') return;
throw err;
}
if (chunk.done) return;
buf += decoder.decode(chunk.value, { stream: true });
let idx;
while ((idx = buf.indexOf('\n')) !== -1) {
const rawLine = buf.slice(0, idx);
buf = buf.slice(idx + 1);
const line = rawLine.endsWith('\r') ? rawLine.slice(0, -1) : rawLine;
if (line === '') {
if (dataLines.length > 0) {
await this.dispatch(eventName, dataLines.join('\n'));
}
dataLines = [];
eventName = null;
continue;
}
if (line.startsWith(':')) continue;
const colon = line.indexOf(':');
const field = colon === -1 ? line : line.slice(0, colon);
let val = colon === -1 ? '' : line.slice(colon + 1);
if (val.startsWith(' ')) val = val.slice(1);
if (field === 'data') dataLines.push(val);
else if (field === 'event') eventName = val;
}
}
}
private async dispatch(name: string | null, data: string): Promise<void> {
if (name !== null && name !== '' && name !== 'presence') return;
let parsed: unknown;
try {
parsed = JSON.parse(data);
} catch (err) {
this.onError(new BridgeError(`malformed presence data: ${(err as Error).message}`));
return;
}
const change = parsed as PresenceChange;
if (
typeof change.address !== 'string' ||
(change.status !== 'online' && change.status !== 'offline') ||
typeof change.at !== 'number'
) {
this.onError(new BridgeError('presence frame missing required fields'));
return;
}
try {
await this.onPresenceChange(change);
} catch (err) {
this.onError(err as Error);
}
}
async addPeer(address: string): Promise<void> {
if (this.disposed) throw new BridgeError('PresenceBridge subscription disposed');
if (this.watched.includes(address)) return;
this.watched = [...this.watched, address];
await this.reconnect();
}
async removePeer(address: string): Promise<void> {
if (this.disposed) throw new BridgeError('PresenceBridge subscription disposed');
if (!this.watched.includes(address)) return;
this.watched = this.watched.filter((a) => a !== address);
await this.reconnect();
}
/**
* Tear down the current SSE connection so the run loop reopens with
* the new watched list. Cancels via abort + reader.cancel — both are
* tolerated by the consume() catch path.
*/
private async reconnect(): Promise<void> {
const reader = this.currentReader;
this.currentReader = null;
this.abortController?.abort();
if (reader) {
try {
await reader.cancel();
} catch {
/* ignore */
}
}
}
async unsubscribe(): Promise<void> {
if (this.disposed) return;
this.disposed = true;
const reader = this.currentReader;
this.currentReader = null;
this.abortController?.abort();
if (reader) {
try {
await reader.cancel();
} catch {
/* ignore */
}
}
if (this.readLoopPromise) {
try {
await this.readLoopPromise;
} catch {
/* ignore */
}
}
}
}
function stripTrailingSlash(s: string): string {
return s.endsWith('/') ? s.slice(0, -1) : s;
}
function sleep(ms: number): Promise<void> {
return new Promise((r) => setTimeout(r, ms));
}

Some files were not shown because too many files have changed in this diff Show More