6 Commits

Author SHA1 Message Date
2c400d7094 release(v4.6.0): broadcast channels — Signal sender-keys for one-to-many fan-out
Some checks failed
Test / test (push) Has been cancelled
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
Lands the broadcast-channel primitive Prism asked for in
Docs/shade-feature-request-sender-keys.md. The crypto in
@shade/core/sender-keys.ts was already in place; this release wires
it up as a first-class app-facing API, adds the persistence schema
across all six storage backends (memory, sqlite, indexeddb +
encrypted variants), introduces wire type 0x21 in @shade/proto,
and ships Prism's three acceptance tests verbatim.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 15:55:34 +02:00
2b1b4d6630 release(v4.5.0): browser-side encrypted storage + multi-factor unlock
Adds the foundations Prism's web client (and any future browser-based
Shade app) needs: at-rest-encrypted IndexedDB storage that mirrors the
SQLite backend byte-for-byte at the AAD/nonce level, browser-safe
subpath imports so Vite/webpack/esbuild stop hitting bun:sqlite, and
KeyManager support for argon2id and N-factor composite unlock.

@shade/storage-encrypted
- EncryptedIndexedDBStorage (subpath: /idb) — full StorageProvider
  using one object store per _enc table; reuses aeadSeal/aeadOpen +
  row-codec sealers so a row sealed under the SQLite or Postgres
  backend decrypts under IDB given the same KeyManager.
  bumpPeerIdentityVersion is atomic under one IDB transaction.
- KeyManager argon2id source — memory-hard KDF for low-entropy
  secrets (PINs). Backed by @noble/hashes/argon2 (already a transitive
  dep). DEFAULT_ARGON2ID exported (m=64 MiB, t=3, p=1).
- KeyManager composite source — HKDF-combine N sub-sources into one
  master. Every source mandatory; order significant by design;
  composite-of-composite rejected; optional info string for app-level
  domain separation.
- Subpath exports (/crypto, /sqlite, /postgres, /idb) plus a `browser`
  condition on the default import that resolves to a barrel
  excluding the Bun- and Postgres-specific entries. Browser bundles
  no longer pull bun:sqlite transitively.

Tests
- 73 tests in shade-storage-encrypted (was 31). New coverage:
  argon2id determinism + reject paths, composite same-factors → same
  master, wrong-PIN/passphrase/order-swap → different master, info
  domain separation, all 28 StorageProvider methods on
  EncryptedIndexedDBStorage, fingerprint-mismatch rejection, and
  cross-impl roundtrip with EncryptedSQLiteStorage proving the AAD/
  nonce derivation is implementation-agnostic.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 10:58:49 +02:00
dbb3a090d8 release(v4.4.0): public accessor for device identity public key
Expose the local device's 32-byte Ed25519 identity public key on Shade
so apps can hand it to their own backend at enrollment time for
signature verification, key pinning or per-device safety-number
computation. Closes the gap that forced consumers to ship placeholder
random bytes their backend could store but never verify against.

- @shade/sdk Shade.identityPublicKey: Promise<Uint8Array> — getter
  mirrors the existing fingerprint accessor. Throws pre-init,
  reflects the current key after rotate(), retired key preserved in
  retired-identities storage per existing grace-period contract.
  Private key remains unreachable.
- Test in shade-sdk/tests/sdk.test.ts: round-trip match against the
  underlying storage's signingPublicKey, plus value updates after
  rotate().
- Lockstep version bump 4.3.0 → 4.4.0 across all 25 packages.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 17:58:45 +02:00
f5f42fe557 release(v4.3.0): browser persistence via @shade/storage-indexeddb
Ship an official IndexedDB-backed StorageProvider so browser-based Shade
consumers persist identity, prekeys, sessions, retired identities,
peer-verification state and stream-resume rows across tab refresh and
browser restart. Closes the gap that forced browser apps onto
storage:"memory" (regenerated identity each load, orphaned device
records server-side).

- New package @shade/storage-indexeddb (4.3.0): full StorageProvider
  conformance, schema v1, idb-backed; bumpPeerIdentityVersion is wrapped
  in a single readwrite IDB transaction (atomic, vs SQLite's
  read-then-upsert race).
- @shade/sdk resolveStorage() accepts { type: 'indexeddb', dbName? } via
  dynamic import (lazy, optional dep — same pattern as
  @shade/storage-postgres). Named StorageSpec type now reused by
  ResolvedConfig.
- Tests: 16 new tests in shade-storage-indexeddb (StorageProvider
  surface + peer-verifications + full E2EE conversation surviving a
  simulated tab reload). Run on fake-indexeddb.
- Lockstep version bump 4.2.1 → 4.3.0 across all 25 packages.
- Publish scripts updated to include the new package.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 17:35:02 +02:00
b77b7e771c release(v4.2.1): fix concurrent-ratchet desync via OutboundQueue waiter cursor
Some checks failed
Publish / publish (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Pull-mode httpClient + drainer + parallel RPCs against the same peer
deteriorated after ~10s with `DecryptionError`. Two bugs combined:

- `OutboundQueue.enqueue` woke `drain` waiters with a `since=0`
  snapshot, replaying already-processed events into
  `Shade.acceptTransferEnvelope` → `manager.decrypt` twice. The
  duplicate consumed an already-used skipped key and corrupted the
  Double Ratchet receive chain.

- `ratchetDecrypt` then propagated the corruption: a same-DH
  message behind the chain with no cached skipped key fell through
  to `kdfChainKey` on the ahead state and rewound `chain.counter`,
  permanently desyncing the chain.

Fix `OutboundQueue` to honor each waiter's `since`, and harden
`ratchetDecrypt` so any future duplicate fails cleanly without
mutating state. Adds regression coverage at all three layers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 22:58:26 +02:00
7520b11b25 release(v4.2.0): pull-mode streams for browser @shade/files
Some checks failed
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
4.1.0's HTTP RPC for browsers capped at inline payloads (≤ 256 KiB).
4.2.0 unlocks streams: server queues outbound chunks + control
envelopes per peer, browser long-polls the queue. Browser-to-server
writes ride the existing /v1/transfer/<id>/chunk POST routes
unchanged.

For Dispatch this unlocks mod-jar uploads (50 MB) and world-backup
downloads (100+ MB) — the actual reason browser-side @shade/files
matters.

### New API

@shade/sdk:
- shade.transferQueueRoute(opts?) — Hono app with /queue +
  /v1/transfer/* routes. Auto-configures the queue transport.
- shade.configureTransfers extended: transport + envelopeTransport
  override slots; resolveBaseUrl optional when both supplied.

@shade/transfer:
- OutboundQueue — per-peer monotonic event log with long-poll
  semantics, idle-eviction GC, ring-buffered to maxEventsPerPeer.
- QueueTransferTransport — enqueues instead of POSTing.

@shade/files:
- httpClient({ outboundQueueUrl, transferBaseUrl }) — when set,
  starts a long-poll drainer + builds a streams-bridge. fs.read /
  fs.write of >256 KiB work end-to-end.
- startQueueDrainer(shade, opts) — exported helper for advanced
  consumers driving their own drainer.

### Implementation notes

- ClientStreamsBridge's TransformStream had HWM=0 by default which
  stalled the drainer's await chain at chunk 4 (writer.write pended
  before the consumer's reader was attached). Bumped to HWM=64 so
  the receive loop can buffer ahead of the consumer.

### Tests

3 new integration tests in tests/integration/http-rpc-streams.test.ts:
4 MiB streamed read round-trip, inline-only error path, idle-timeout
long-poll behaviour.

Wire-compatible. Source-compatible. Lockstep bump to 4.2.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 23:27:06 +02:00
76 changed files with 5867 additions and 113 deletions

3
.gitignore vendored
View File

@@ -2,3 +2,6 @@ node_modules/
dist/ dist/
*.tsbuildinfo *.tsbuildinfo
.DS_Store .DS_Store
**/.tmp-*.db
**/.tmp-*.db-shm
**/.tmp-*.db-wal

View File

@@ -5,6 +5,437 @@ All notable changes to Shade are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [4.6.0] — 2026-05-07 — Broadcast channels (Signal sender-keys for one-to-many fan-out)
Prism's PC desktop is the *sender* in a one-to-many fan-out — one PTY
output frame, N paired-device deliveries — and bilateral `for (peer of
peers) shade.send(peer, frame)` works for N ≤ 5 but starts hurting once
the paired fleet grows (3 laptops + phone + tablet + watch = N = 7) and
once mobile cellular is in the loop. The crypto pattern that solves it
is Signal's **sender-key**: the sender holds a per-channel symmetric
chain key shared with all members, encrypts each message *once* with
it, and the relay (or the SDK fan-out loop) ships the same ciphertext
to every recipient.
This release lands sender-key broadcast as a scoped "broadcast channel"
primitive in `@shade/sdk`, with the persistence + wire format + receiver-
side `meta.kind === 'broadcast'` plumbing wired through every backend.
The crypto in `@shade/core/sender-keys.ts` was already in place;
v4.6 turns it into a first-class app-facing API.
### Added
#### `@shade/sdk`
- `shade.createBroadcastChannel({ label? })``BroadcastChannel`
opaque, persisted channel id stable across `shutdown()` / re-open.
Owner role: `sender` (only the channel creator can broadcast).
- `BroadcastChannel.addMember(peerAddress)` — distributes the current
sender-key to a paired peer over the existing bilateral ratchet.
Returns the wrapped envelope the app delivers; the SDK does the
framing inline (no new wire-format changes visible to apps —
acceptance criterion (3)).
- `BroadcastChannel.removeMember(peerAddress)` — rotates the chain
(fresh `chainKey` + new Ed25519 signing keypair, `generation++`),
destroys the old key material, and returns one envelope per surviving
member with the new sender-key. Stale broadcasts at lower generations
are silently dropped on receive.
- `BroadcastChannel.broadcast(plaintext)` — single AES-256-GCM encrypt
with the current chain message key + Ed25519 signature; the SAME
envelope is delivered to every member. Returns
`{ envelope: Uint8Array, members: readonly string[] }` so the app's
transport handles the per-peer fan-out.
- `BroadcastChannel.members()` — snapshot of currently-active members
(excludes revoked).
- `shade.getBroadcastChannel(channelId)` / `shade.listBroadcastChannels()`
for reconciling app-level pairing state with persisted channel state.
- `shade.acceptBroadcast(envelope)` — decrypt an inbound broadcast wire
envelope; dispatches to `onMessage` handlers with
`meta = { kind: 'broadcast', channelId, sender, generation, iteration }`.
- `Shade.onMessage` handler signature gained an optional third arg
`meta?: MessageMeta` — back-compat: handlers that ignore it keep
working unchanged for direct messages.
#### `@shade/proto`
- `encodeBroadcast(BroadcastWire)` / `decodeBroadcast(bytes)` — wire
type `0x21`. Length-prefixed channelId + senderAddress, u32
generation/iteration, 12-byte AES-GCM nonce, 64-byte Ed25519
signature, length-prefixed ciphertext.
- `inspectEnvelopeType` recognises `'broadcast'`.
#### `@shade/core`
- `BroadcastChannelRecord` — persisted channel state (chainKey,
iteration, signing keys, generation, role).
- `BroadcastMemberRecord` — sender-side membership row with `joinedAt`
+ nullable `removedAt`.
- `StorageProvider` gained six optional methods:
`saveBroadcastChannel`, `getBroadcastChannel`, `listBroadcastChannels`,
`removeBroadcastChannel`, `saveBroadcastMember`, `getBroadcastMembers`,
`removeBroadcastMember`. Backends < 4.6 throw a clear error when an
app tries to call `createBroadcastChannel` against them.
#### Storage backends
- `MemoryStorage`, `SQLiteStorage`, `IndexedDBStorage` — plaintext
`broadcast_channels` + `broadcast_members` tables. IDB schema bumps
to v2 with an upgrade-path that creates the new stores idempotently.
- `EncryptedSQLiteStorage`, `EncryptedIndexedDBStorage`,
`EncryptedPostgresStorage``broadcast_channels_enc` +
`broadcast_members_enc` schemas. The chain key, iteration, and
signing-key bundle live in a sealed `ciphertext` blob bound to
`(table='broadcast_channels', column='broadcast_channel_sensitive', pk=channelId)`
AAD; routing fields (channelId, ownerRole, ownerAddress, label,
generation, timestamps) stay plaintext for queries. New row-codec
helpers `sealBroadcastChannelSensitive` / `openBroadcastChannelSensitive`.
IDB schema bumps to v2 the same way.
### Tests
- `packages/shade-sdk/tests/broadcast.test.ts` — Prism's three
acceptance tests verbatim: (1) two-member receive with
`meta.kind === 'broadcast'`, (1*) revocation rotates + receiver A
drops while B keeps working, (2) persistence — channel id, members,
and chain advance survive `shutdown()` + re-open from the SQLite
path, (3) `listBroadcastChannels` surfaces both sender + receiver
records correctly.
- `packages/shade-storage-encrypted/tests/encrypted-sqlite.test.ts`
channel + member round-trip under sealed storage; receiver-side rows
correctly persist without `signingPrivateKey`.
### Compatibility
- Wire-protocol additive: existing peers ignore the `0x21` envelope
type. Apps not using broadcast channels see no behavior change.
- Storage schemas additive: the new `broadcast_*` tables / object
stores are created on first open; migrations from a 4.5 database are
no-ops. IDB schema-version bump happens transparently in `upgrade`.
## [4.5.0] — 2026-05-07 — Browser-side encrypted storage + multi-factor unlock
Browser-based Shade clients (Prism's web client being the first) needed
the same at-rest encryption story as the desktop SQLite path: identity,
prekeys, sessions and stream-resume state persisted across reloads,
unwrapped from a user-supplied passphrase — and on browsers, optionally
gated behind a second factor (PIN) since there is no OS-session boundary
to lean on. The existing barrel of `@shade/storage-encrypted` also
transitively imported `bun:sqlite` and `postgres`, which prevented Vite/
webpack/esbuild from producing a clean browser bundle.
This release adds an encrypted IndexedDB backend that mirrors
`EncryptedSQLiteStorage` byte-for-byte at the AAD/nonce level, exposes
browser-safe subpath imports, and lets `KeyManager` derive its master
key from low-entropy secrets (argon2id) and from N composed factors
(every factor mandatory).
### Added
#### `@shade/storage-encrypted`
- `EncryptedIndexedDBStorage` — IndexedDB-backed `StorageProvider`
exposed via `@shade/storage-encrypted/idb`. One object store per
`_enc` table from the SQLite schema, sealed payloads as `Uint8Array`,
routing/timestamp fields kept plaintext for query efficiency. Reuses
`aeadSeal`/`aeadOpen` and the `row-codec` sealers verbatim — a row
sealed under the SQLite or Postgres backend decrypts under IDB given
the same `KeyManager`. `bumpPeerIdentityVersion` is atomic under one
IDB transaction (closes the read-then-upsert race the SQLite version
has).
- `KeyManager.open({ kind: 'argon2id', ... })` — memory-hard KDF for
low-entropy secrets (PINs, short passwords). Backed by
`@noble/hashes/argon2` (already a transitive dep — pure JS, browser
safe). `DEFAULT_ARGON2ID` exported (m=64 MiB, t=3, p=1, 32-byte
output; ~250400 ms in modern browsers).
- `KeyManager.open({ kind: 'composite', sources, info? })`
HKDF-combine N sub-sources into one master key. Every source is
required: omitting or substituting any source yields a different
master key and `open()` fails on the storage-key-fingerprint check.
Order is significant by design (`[pwd, pin]``[pin, pwd]`).
Composite-of-composite is rejected.
- Subpath exports: `@shade/storage-encrypted/crypto` (KeyManager + KDF
+ AEAD + row-codec, no SQLite/Postgres bindings), `/sqlite` (Bun),
`/postgres` (Node), `/idb` (browser). The `browser` condition on the
default import resolves to a barrel that excludes Bun/Postgres
imports — `import { KeyManager } from '@shade/storage-encrypted'` now
bundles cleanly under Vite without hitting `bun:sqlite` resolution
errors.
- Dependency: `idb` ^8.0.3.
### Tests
- `packages/shade-storage-encrypted/tests/key-manager-multi-factor.test.ts`
— argon2id determinism + reject paths, composite same-factors → same
master, wrong-PIN/wrong-passphrase/order-swap → different master,
explicit `info` domain separation, nested-composite rejection.
- `packages/shade-storage-encrypted/tests/encrypted-indexeddb.test.ts`
— full round-trip coverage of all 28 `StorageProvider` methods,
fingerprint-mismatch rejection on wrong key, atomic peer-identity
bump, plus cross-impl roundtrip with `EncryptedSQLiteStorage`
proving the AAD/nonce derivation is implementation-agnostic.
## [4.4.0] — 2026-05-05 — Public accessor for the device's identity public key
Browser-based Shade consumers building enrollment flows had no way to
hand the device's actual Ed25519 identity public key to their own
backend — the key was reachable only via the private
`storage.getIdentityKeyPair()` call inside `Shade`. Apps shipped with
placeholder bytes (`crypto.getRandomValues(new Uint8Array(32))`) that
the backend stored but couldn't verify against, deferring real
cryptographic device binding until the SDK exposed the key.
### Added
#### `@shade/sdk`
- `Shade.identityPublicKey: Promise<Uint8Array>` — getter returning the
local device's 32-byte Ed25519 identity public key. Mirrors the
`fingerprint` accessor shape. Throws if accessed before
`initialize()`. Reflects the current key after `rotate()`; the
previous key remains in retired-identities storage for the
configured grace period. Use `fingerprint` (12-group safety number)
for human side-channel comparison; use `identityPublicKey` when
handing the raw key to a backend for signature verification or
pinning.
### Tests
- `packages/shade-sdk/tests/sdk.test.ts` — `identityPublicKey exposes
the device Ed25519 key and tracks rotation` covers the round-trip
match against the underlying storage and that the value updates
after `rotate()`.
## [4.3.0] — 2026-05-05 — Browser persistence via `@shade/storage-indexeddb`
Browser-based Shade consumers had no path to session persistence: the only
storage option that worked outside Node was `"memory"`, so the identity
keypair regenerated on every page load and `device:${registrationId}`
churned to a fresh address each refresh. Building a `StorageProvider`
in consumer-land meant 25+ method re-implementations per app and no
shared conformance surface.
`4.3.0` ships an official IndexedDB adapter alongside SQLite and Postgres
so any browser-based Shade SDK consumer (dashboards, contact-list apps,
browser-extension messengers) gets persistent identity, prekeys, sessions,
retired identities, peer-verification state and stream-resume rows for
free, surviving tab refresh and browser restart.
### Added
#### `@shade/storage-indexeddb` (new package)
- `IndexedDBStorage.create({ dbName? })` — async open of an IDB
database (one object store per `StorageProvider` category) with
schema version 1. `dbName` defaults to `"shade"`; consumers that
run multiple Shade-backed apps on the same origin pass distinct
names (`"my-app-shade"`) so the IDB inspector groups them sensibly.
- Full `StorageProvider` conformance: identity, signed/one-time prekeys,
sessions, trusted identities, retired identities (with prune by
`retiredAt`), stream-state save/get/list/prune, peer verifications,
and the per-peer identity-version counter.
- `bumpPeerIdentityVersion` is wrapped in a single IDB `readwrite`
transaction — atomic read-modify-write, closing the race window the
SQLite adapter currently has on parallel `acceptIdentityChange`
calls. (SQL adapters will be brought in line in a follow-up.)
- Implementation dependency: `idb` (Jake Archibald's typed wrapper).
Tests run against `fake-indexeddb` for parity with the SQLite test
layout.
#### `@shade/sdk`
- `resolveStorage()` accepts a fourth spec form:
`{ type: 'indexeddb', dbName?: string }`. Resolution goes through
a dynamic import so Node-only consumers don't pull a browser-only
adapter into their bundle (same pattern as `@shade/storage-postgres`).
- `ShadeConfig['storage']` now exports a named `StorageSpec` type
reused by `ResolvedConfig`, replacing the duplicated inline union.
### Tests
- `packages/shade-storage-indexeddb/tests/indexeddb-storage.test.ts` —
full StorageProvider surface (identity, prekeys, sessions, trust,
retired identities, persistence across close+reopen) plus an end-to-end
`ShadeSessionManager` conversation that survives a simulated tab
reload mid-session.
- `packages/shade-storage-indexeddb/tests/peer-verifications.test.ts` —
CRUD round-trip, upsert-on-duplicate, identity-version increment
invariants, persistence across reopen.
## [4.2.1] — 2026-05-04 — Concurrent-ratchet desync under pull-mode drainer
A consumer running `shade.files.httpClient(server, { outboundQueueUrl, ... })`
alongside parallel RPC traffic against the same peer would, after ~10s of
load, see every subsequent message fail with
`DecryptionError: Failed to decrypt message — wrong key or tampered data`.
Two bugs combined to cause this; both are fixed in `4.2.1` with regression
coverage.
### Fixed
#### `@shade/transfer` — `OutboundQueue` waiter cursor
`enqueue` woke pending `drain` waiters with a `since=0` snapshot — the
full event log — instead of using the waiter's own `since`. A poll that
parked at the head and was woken by a fresh enqueue therefore replayed
every event the waiter had already processed. Downstream the queue
fed `Shade.acceptTransferEnvelope`, so the duplicate replayed an
envelope into `manager.decrypt` twice. The second decrypt consumed an
already-used skipped key and corrupted the Double Ratchet receive
chain. Each `PendingWaiter` now records its `since` cursor and is
delivered only events with `id > since`.
#### `@shade/core` — `ratchetDecrypt` defense-in-depth
A same-DH message whose `counter` was already behind the chain — and
that did NOT match a cached skipped key — fell through to a path that
called `kdfChainKey` on the *current* (ahead) chain key and then set
`chain.counter = message.counter + 1`, permanently desyncing the
ratchet so every subsequent decrypt returned wrong-key. Such messages
are now rejected with `DecryptionError` without any state mutation, so
a downstream replay (transport bug, retry, intermitent network) cannot
poison the session.
### Tests
- `packages/shade-files/tests/integration/concurrent-ratchet.test.ts` —
100 parallel `httpClient` RPCs while the drainer runs, plus a mixed
workload of 50 RPCs + 50 raw `shade.send` deliveries with Bob
echoing replies through the queue. Both surface the bug pre-fix.
- `packages/shade-transfer/tests/outbound-queue.test.ts` — direct
regression on the waiter `since` cursor.
- `packages/shade-core/tests/ratchet.test.ts` — replay of an
already-decrypted message must throw cleanly without breaking
subsequent decrypts on the same chain.
## [4.2.0] — 2026-05-03 — Pull-mode streams for browser @shade/files
`4.1.0` shipped HTTP RPC for browser clients but capped them at inline
payloads (≤ 256 KiB). Larger reads/writes — mod-jars (150 MB),
world-backups (100+ MB), the things that actually need streaming —
threw `ConflictError` directing callers to the server-to-server
pathway. That made browser-side `@shade/files` insufficient for
admin-panel-style apps where the client is a browser tab and the
server is a Bun process.
`4.2.0` flips the direction: when the browser supplies
`outboundQueueUrl` + `transferBaseUrl`, server-to-browser chunks +
control envelopes ride a per-peer queue that the browser long-polls,
and browser-to-server chunks POST directly to the server's existing
chunk-receive routes. No WebSockets, no SSE, no inbound listener on
the browser. Long-polling + a request-response inbound queue is
the entire wire surface.
### Added
#### `@shade/transfer`
- `OutboundQueue` — per-peer monotonic event log with long-poll
semantics. `enqueue(peer, event)` appends, `drain(peer, since,
blockMs, signal)` returns events with `id > since` (blocking up
to `blockMs` if none are ready). Idle-eviction GC drops peers
that haven't polled in `idleEvictionMs` (default 10 min). Ring-
buffered to `maxEventsPerPeer` (default 1000) — overflow drops
oldest, receivers pick up the gap via re-resume from `since=0`.
- `QueuedEvent` discriminated union: `{ kind: 'envelope', bytes }`
or `{ kind: 'chunk', bytes, meta: { streamId, laneId, seq } }`.
- `QueueTransferTransport` (implements `ITransferTransport`) —
enqueues outbound chunks instead of POSTing. Returns optimistic
`ChunkAck` because the queue *is* the delivery; chunk-resume picks
up dropped events on receiver-side reconnect.
#### `@shade/sdk`
- `Shade.transferQueueRoute(opts?)` — Hono app with all five routes a
pull-mode receiver needs:
- `POST /queue` — long-poll the per-peer outbound queue.
- `POST /v1/transfer/:streamId/chunk` — receive incoming chunks
(browser → server writes).
- `GET /v1/transfer/:streamId/state` — resume-state lookup.
- `POST /v1/transfer/control` — receive incoming control envelopes
(browser → server stream-init / abort).
- `GET /v1/transfer/health` — peer reachability probe.
Auto-configures `shade.configureTransfers(...)` with the queue
transport + `QueueEnvelopeTransport` if not already configured.
- `Shade.configureTransfers(opts)` extended: `resolveBaseUrl` is now
optional when `transport` and `envelopeTransport` are both supplied
(lets pure-queue servers omit the baseUrl entirely). New
`transport?: ITransferTransport` override slot.
- `QueueEnvelopeTransport` — `ControlEnvelopeTransport` impl that
enqueues outbound envelopes for browser receivers.
#### `@shade/files`
- `createFilesHttpClient` (and `shade.files.httpClient`) accept new
options:
- `outboundQueueUrl` — `/queue` endpoint to long-poll.
- `transferBaseUrl` — base URL for outbound chunk POSTs and control
envelope POSTs (browser → server writes).
- `queueBlockMs` — long-poll timeout (default 30 s; server clamps
at `maxBlockMs`).
When set, the client:
1. Configures `shade.configureTransfers({ resolveBaseUrl })` so
outbound chunks POST to `<transferBaseUrl>/v1/transfer/...`.
2. Builds a `ClientStreamsBridge` eagerly so the engine's
incoming-transfer subscription is in place before the drainer
dispatches the first envelope.
3. Starts a long-poll `startQueueDrainer(...)` that pulls queued
events and dispatches them via `shade.acceptTransferEnvelope`.
- Streamed reads (`fs.read` of files > 256 KiB) and streamed writes
(`fs.write` of large inputs) now work end-to-end on the browser
client when the queue options are set.
- `startQueueDrainer(shade, opts)` exported for advanced consumers
that want to drive their own drainer (e.g. service-worker setups
that want a single shared drainer across multiple `httpClient`s).
- `client.close()` now stops the drainer and tears down the streams-
bridge — important on tab unload to free the long-poll socket.
#### `@shade/files` (internal)
- `ClientStreamsBridge` uses a TransformStream with `highWaterMark:
64` instead of the default `0` so the receive-side write loop
doesn't stall on backpressure before the consumer attaches its
reader (default HWM stalled at chunk 4 in pull-mode where the
drainer races the consumer's `getReader()` call).
### Wire contract
```
POST <base>/queue HTTP/1.1
X-Shade-Sender-Address: alice@example.com
{ "since": 42, "blockMs": 30000 }
────
200 OK
{
"events": [
{ "id": 43, "kind": "envelope", "bytesB64": "...", "timestampMs": 1730... },
{ "id": 44, "kind": "chunk", "bytesB64": "...", "meta": { "streamId": "...", "laneId": 0, "seq": 0 } },
...
],
"nextSince": 47
}
```
### Tests
`tests/integration/http-rpc-streams.test.ts` — three integration tests:
- 4 MiB streamed read end-to-end via long-poll queue (verifies bytes
match the source).
- Inline-only client throws clear error on streamed read.
- Long-poll returns empty events on idle timeout (verifies the
`blockMs` pathway).
### Migration
`4.1.0 → 4.2.0` is wire-compatible and source-compatible — the
queue route is purely additive. To enable streamed transfers in a
browser app:
```ts
// Server
const queue = await shade.transferQueueRoute({ blockMs: 30_000 });
await shade.files.serve(handler);
const rpc = shade.files.rpcRoute({ acceptFirstMessage: true });
const app = new Hono();
app.route('/api/v1/shade-files', queue);
app.route('/api/v1/shade-files', rpc);
// Browser
const fs = shade.files.httpClient(serverAddress, {
rpcUrl: 'https://server/api/v1/shade-files/rpc',
outboundQueueUrl: 'https://server/api/v1/shade-files/queue',
transferBaseUrl: 'https://server/api/v1/shade-files',
});
await fs.write('/mods/some-mod.jar', new Uint8Array(/* 50 MB */));
const result = await fs.read('/backups/world.tar.gz'); // streamed
```
`shade.files.serve(handler, { inlineOnly: true })` is still supported
for HTTP-RPC-without-streams deployments — it skips the streams-bridge
setup entirely.
## [4.1.0] — 2026-05-03 — Browser-friendly HTTP RPC for @shade/files ## [4.1.0] — 2026-05-03 — Browser-friendly HTTP RPC for @shade/files
The default `shade.files.client(peer)` requires both peers to be The default `shade.files.client(peer)` requires both peers to be

View File

@@ -17,7 +17,7 @@
}, },
"packages/shade-cli": { "packages/shade-cli": {
"name": "@shade/cli", "name": "@shade/cli",
"version": "0.4.0", "version": "4.4.0",
"bin": { "bin": {
"shade": "src/cli.ts", "shade": "src/cli.ts",
}, },
@@ -36,7 +36,7 @@
}, },
"packages/shade-core": { "packages/shade-core": {
"name": "@shade/core", "name": "@shade/core",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/observability": "workspace:*", "@shade/observability": "workspace:*",
}, },
@@ -49,7 +49,7 @@
}, },
"packages/shade-crypto-web": { "packages/shade-crypto-web": {
"name": "@shade/crypto-web", "name": "@shade/crypto-web",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@noble/curves": "^2.0.1", "@noble/curves": "^2.0.1",
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
@@ -59,7 +59,7 @@
}, },
"packages/shade-dashboard": { "packages/shade-dashboard": {
"name": "@shade/dashboard", "name": "@shade/dashboard",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/widgets": "workspace:*", "@shade/widgets": "workspace:*",
"react": "^19.0.0", "react": "^19.0.0",
@@ -74,7 +74,7 @@
}, },
"packages/shade-files": { "packages/shade-files": {
"name": "@shade/files", "name": "@shade/files",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -101,7 +101,7 @@
}, },
"packages/shade-inbox": { "packages/shade-inbox": {
"name": "@shade/inbox", "name": "@shade/inbox",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/proto": "workspace:*", "@shade/proto": "workspace:*",
@@ -114,7 +114,7 @@
}, },
"packages/shade-inbox-server": { "packages/shade-inbox-server": {
"name": "@shade/inbox-server", "name": "@shade/inbox-server",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/observability": "workspace:*", "@shade/observability": "workspace:*",
@@ -132,7 +132,7 @@
}, },
"packages/shade-key-transparency": { "packages/shade-key-transparency": {
"name": "@shade/key-transparency", "name": "@shade/key-transparency",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
@@ -144,11 +144,11 @@
}, },
"packages/shade-keychain": { "packages/shade-keychain": {
"name": "@shade/keychain", "name": "@shade/keychain",
"version": "0.4.0", "version": "4.4.0",
}, },
"packages/shade-observability": { "packages/shade-observability": {
"name": "@shade/observability", "name": "@shade/observability",
"version": "0.1.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
}, },
@@ -166,7 +166,7 @@
}, },
"packages/shade-observer": { "packages/shade-observer": {
"name": "@shade/observer", "name": "@shade/observer",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/server": "workspace:*", "@shade/server": "workspace:*",
@@ -178,14 +178,14 @@
}, },
"packages/shade-proto": { "packages/shade-proto": {
"name": "@shade/proto", "name": "@shade/proto",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
}, },
}, },
"packages/shade-recovery": { "packages/shade-recovery": {
"name": "@shade/recovery", "name": "@shade/recovery",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -198,7 +198,7 @@
}, },
"packages/shade-sdk": { "packages/shade-sdk": {
"name": "@shade/sdk", "name": "@shade/sdk",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -225,7 +225,7 @@
}, },
"packages/shade-server": { "packages/shade-server": {
"name": "@shade/server", "name": "@shade/server",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/inbox-server": "workspace:*", "@shade/inbox-server": "workspace:*",
@@ -245,15 +245,19 @@
}, },
"packages/shade-storage-encrypted": { "packages/shade-storage-encrypted": {
"name": "@shade/storage-encrypted", "name": "@shade/storage-encrypted",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
"@shade/storage-postgres": "workspace:*", "@shade/storage-postgres": "workspace:*",
"@shade/storage-sqlite": "workspace:*", "@shade/storage-sqlite": "workspace:*",
"idb": "^8.0.3",
"postgres": "^3.4.9", "postgres": "^3.4.9",
}, },
"devDependencies": {
"fake-indexeddb": "^6.0.0",
},
"peerDependencies": { "peerDependencies": {
"@shade/keychain": "workspace:*", "@shade/keychain": "workspace:*",
}, },
@@ -261,9 +265,21 @@
"@shade/keychain", "@shade/keychain",
], ],
}, },
"packages/shade-storage-indexeddb": {
"name": "@shade/storage-indexeddb",
"version": "4.4.0",
"dependencies": {
"@shade/core": "workspace:*",
"idb": "^8.0.3",
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
"fake-indexeddb": "^6.0.0",
},
},
"packages/shade-storage-postgres": { "packages/shade-storage-postgres": {
"name": "@shade/storage-postgres", "name": "@shade/storage-postgres",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/inbox-server": "workspace:*", "@shade/inbox-server": "workspace:*",
@@ -278,7 +294,7 @@
}, },
"packages/shade-storage-sqlite": { "packages/shade-storage-sqlite": {
"name": "@shade/storage-sqlite", "name": "@shade/storage-sqlite",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -288,7 +304,7 @@
}, },
"packages/shade-streams": { "packages/shade-streams": {
"name": "@shade/streams", "name": "@shade/streams",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
@@ -300,7 +316,7 @@
}, },
"packages/shade-transfer": { "packages/shade-transfer": {
"name": "@shade/transfer", "name": "@shade/transfer",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -317,7 +333,7 @@
}, },
"packages/shade-transport": { "packages/shade-transport": {
"name": "@shade/transport", "name": "@shade/transport",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
@@ -328,7 +344,7 @@
}, },
"packages/shade-transport-bridge": { "packages/shade-transport-bridge": {
"name": "@shade/transport-bridge", "name": "@shade/transport-bridge",
"version": "0.1.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/server": "workspace:*", "@shade/server": "workspace:*",
@@ -350,7 +366,7 @@
}, },
"packages/shade-transport-webrtc": { "packages/shade-transport-webrtc": {
"name": "@shade/transport-webrtc", "name": "@shade/transport-webrtc",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/streams": "workspace:*", "@shade/streams": "workspace:*",
@@ -359,7 +375,7 @@
}, },
"packages/shade-widgets": { "packages/shade-widgets": {
"name": "@shade/widgets", "name": "@shade/widgets",
"version": "0.4.0", "version": "4.4.0",
"dependencies": { "dependencies": {
"@shade/recovery": "workspace:*", "@shade/recovery": "workspace:*",
"@shade/sdk": "workspace:*", "@shade/sdk": "workspace:*",
@@ -568,6 +584,8 @@
"@shade/storage-encrypted": ["@shade/storage-encrypted@workspace:packages/shade-storage-encrypted"], "@shade/storage-encrypted": ["@shade/storage-encrypted@workspace:packages/shade-storage-encrypted"],
"@shade/storage-indexeddb": ["@shade/storage-indexeddb@workspace:packages/shade-storage-indexeddb"],
"@shade/storage-postgres": ["@shade/storage-postgres@workspace:packages/shade-storage-postgres"], "@shade/storage-postgres": ["@shade/storage-postgres@workspace:packages/shade-storage-postgres"],
"@shade/storage-sqlite": ["@shade/storage-sqlite@workspace:packages/shade-storage-sqlite"], "@shade/storage-sqlite": ["@shade/storage-sqlite@workspace:packages/shade-storage-sqlite"],
@@ -626,6 +644,8 @@
"escalade": ["escalade@3.2.0", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="], "escalade": ["escalade@3.2.0", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="],
"fake-indexeddb": ["fake-indexeddb@6.2.5", "", {}, "sha512-CGnyrvbhPlWYMngksqrSSUT1BAVP49dZocrHuK0SvtR0D5TMs5wP0o3j7jexDJW01KSadjBp1M/71o/KR3nD1w=="],
"fast-check": ["fast-check@3.23.2", "", { "dependencies": { "pure-rand": "^6.1.0" } }, "sha512-h5+1OzzfCC3Ef7VbtKdcv7zsstUQwUDlYpUTvjeUsJAssPgLn7QzbboPtL5ro04Mq0rPOsMzl7q5hIbRs2wD1A=="], "fast-check": ["fast-check@3.23.2", "", { "dependencies": { "pure-rand": "^6.1.0" } }, "sha512-h5+1OzzfCC3Ef7VbtKdcv7zsstUQwUDlYpUTvjeUsJAssPgLn7QzbboPtL5ro04Mq0rPOsMzl7q5hIbRs2wD1A=="],
"fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="], "fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="],
@@ -638,6 +658,8 @@
"hono": ["hono@4.12.12", "", {}, "sha512-p1JfQMKaceuCbpJKAPKVqyqviZdS0eUxH9v82oWo1kb9xjQ5wA6iP3FNVAPDFlz5/p7d45lO+BpSk1tuSZMF4Q=="], "hono": ["hono@4.12.12", "", {}, "sha512-p1JfQMKaceuCbpJKAPKVqyqviZdS0eUxH9v82oWo1kb9xjQ5wA6iP3FNVAPDFlz5/p7d45lO+BpSk1tuSZMF4Q=="],
"idb": ["idb@8.0.3", "", {}, "sha512-LtwtVyVYO5BqRvcsKuB2iUMnHwPVByPCXFXOpuU96IZPPoPN6xjOGxZQ74pgSVVLQWtUOYgyeL4GE98BY5D3wg=="],
"js-tokens": ["js-tokens@4.0.0", "", {}, "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ=="], "js-tokens": ["js-tokens@4.0.0", "", {}, "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ=="],
"jsesc": ["jsesc@3.1.0", "", { "bin": { "jsesc": "bin/jsesc" } }, "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA=="], "jsesc": ["jsesc@3.1.0", "", { "bin": { "jsesc": "bin/jsesc" } }, "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA=="],

View File

@@ -0,0 +1,173 @@
# Feature Request — Public accessor for the device's identity public key
**To**: Shade SDK team
**From**: Dispatch (browser-based Shade consumer)
**Target**: Shade SDK 4.4.x (or whichever release vehicle fits)
**Priority**: medium — unblocks real per-device fingerprint binding at
enrollment time; consumers ship with placeholder keys until then
Thanks for shipping `@shade/storage-indexeddb` so quickly — that unblocked
Dispatch's Slice 2.5 (persistent browser identity). During integration we
hit one more gap that's worth raising as a separate FR.
---
## Summary
Expose the local device's Ed25519 identity public key as a public accessor
on `Shade`, so applications can hand it to their own backend at enrollment
time for per-device verification, audit, or peer-fingerprint computation.
Today the SDK exposes `myAddress` and `fingerprint`, but the underlying
identity public key — the cryptographic root that everything else binds
to — is reachable only via the private `this.storage.getIdentityKeyPair()`
call inside `Shade`. Consumers building enrollment flows have no way to
hand the real key over.
## Problem
A common pattern in Shade-using apps is:
1. Browser device generates Shade identity (via `createShade`)
2. User enters an enrollment token
3. Browser POSTs to its backend: `{ token, address, identityPublicKey, ... }`
4. Backend records the device, computes/stores a safety number from the
identity key, opens a Shade session
Step 3 today has nowhere to get a real `identityPublicKey` from. Dispatch
currently sends `crypto.getRandomValues(new Uint8Array(32))` formatted as
hex, with this comment in the source:
```ts
/**
* The browser submits a hex public-key field at enroll time so the schema
* stays stable. Wiring this to the SDK-generated identity key requires a
* Shade SDK addition (no public accessor for the raw identity key today).
*/
export function generatePlaceholderIdentityPublicKey(): string { ... }
```
The backend stores this placeholder but cannot verify anything against it,
because it's not actually the device's key. Real cryptographic binding is
deferred until the SDK exposes the underlying key.
## Why `fingerprint` isn't sufficient
`Shade.fingerprint` returns a 12-groups-of-5-digits safety-number string
designed for human side-channel comparison. That's the right output for a
"compare these digits with your friend" UX, but it's a derived format, not
the raw key. Backends that want to:
- Store the key for later signature verification
- Compute their own safety-number representation (Dispatch uses
`deterministicSafetyNumber(localAddr, peerAddr)` based on the raw bytes)
- Re-derive the fingerprint after an identity rotation
…all need access to the raw 32-byte Ed25519 public key.
## Proposed API
Add a single async accessor on `Shade`:
```ts
class Shade {
/**
* The local device's Ed25519 identity public key (32 bytes).
*
* Stable for the lifetime of the identity. After `rotateIdentity()`
* this returns the new key; the old key is preserved in retired-
* identities storage for the configured grace period.
*/
get identityPublicKey(): Promise<Uint8Array>;
}
```
Internally:
```ts
get identityPublicKey(): Promise<Uint8Array> {
if (!this.initialized) throw new Error('Not initialized');
return this.storage.getIdentityKeyPair().then((kp) => {
if (!kp) throw new Error('Identity not yet generated');
return kp.publicKey;
});
}
```
If returning a getter that produces a Promise feels off, the equivalent
method form is fine:
```ts
async getIdentityPublicKey(): Promise<Uint8Array> { ... }
```
Either shape works for consumers — pick whichever matches existing SDK
conventions.
## Alternative considered: object accessor
Returning an object would leave room to expose other identity-related
fields later without a breaking change:
```ts
get identity(): Promise<{
address: string;
publicKey: Uint8Array;
fingerprint: string;
// future: registrationId, createdAt, etc.
}>;
```
This would mildly duplicate `myAddress` + `fingerprint`, but consolidates
identity-related accessors. Not a blocker for shipping the simpler
single-purpose accessor — just flagging the option in case the SDK is
considering broader API ergonomics.
## What this unblocks
For Dispatch specifically:
- Real device identity binding at enrollment (replaces the placeholder
`crypto.randomUUID()`-derived hex bytes the backend currently stores)
- Server-side `computePeerSafetyNumber()` can use the real key instead of
the deterministic-from-address stand-in (`shade-identity-provider.ts:151`)
- Future signature-based device authentication (sign a challenge with the
device's identity key during enrollment) becomes possible without
another SDK round
For other Shade consumers:
- Any app that hands a device key to its own backend for enrollment —
multi-device pairing flows, contact verification UIs, push-notification
targeting — gets an actual key to work with
## Acceptance criteria
1. New accessor exposed on `Shade` (getter or async method, SDK's
preference)
2. Returns the 32-byte Ed25519 identity public key
3. Returns the **current** key after `rotateIdentity()`
4. Throws (or resolves to a clear error) when called before
`initialize()` completes
5. Documented alongside `myAddress` and `fingerprint` in the SDK reference
6. One unit test in `shade-sdk` confirming the returned bytes match what
`storage.getIdentityKeyPair()` holds
7. Mentioned in the changelog under the next release
## Out of scope
- **Private key access** — consumers should never need it; signature
operations go through `shade.send()` etc. Don't expose the secret half.
- **Cross-peer key lookup** — getting *another* peer's identity key is a
separate concern (related to peer-verification storage), not what this
FR is about. This is strictly the local device's own key.
- **Format conversions** — base64/hex/PEM helpers don't belong in the SDK.
Consumers can encode the raw bytes however their wire format requires.
## Why this can't be done in consumer-land
The identity keypair is generated by `MemoryStorage` / `SQLiteStorage` /
`IndexedDBStorage` and consumed by `ShadeSessionManager`. Consumers can't
reach into either layer without breaching the SDK's encapsulation. A
public accessor is the only path that doesn't require monkey-patching
private fields.

View File

@@ -0,0 +1,191 @@
# Feature Request — `@shade/storage-indexeddb`
**To**: Shade SDK team
**From**: Dispatch (browser-based Shade consumer)
**Target**: Shade SDK 4.3.x (or whichever release vehicle fits)
**Priority**: blocks all browser-based Shade apps from achieving session
persistence across tab refresh
---
## Summary
Ship an official IndexedDB-backed `StorageProvider` adapter as a new
workspace package `@shade/storage-indexeddb`, so browser-based Shade SDK
consumers can persist identity, prekeys, sessions, and peer-verification
state across tab refresh and browser restart — the same way `@shade/storage-sqlite`
does for Node and `@shade/storage-postgres` does for server deployments.
## Problem
Today the Shade SDK ships three storage paths:
| spec | adapter | environment |
| --------------------------------- | ------------------------ | ---------------- |
| `"memory"` | `MemoryStorage` (in-SDK) | tests, ephemeral |
| `"sqlite:/path"` | `@shade/storage-sqlite` | Node |
| `{ type: 'postgres', url: '…' }` | `@shade/storage-postgres`| Node servers |
There is **no browser-storage option**. The only way to run Shade in a
browser today is `storage: "memory"`, which means:
- Identity keypair regenerates on every page load
- Sessions reset → re-enrollment after every refresh
- `getLocalRegistrationId()` returns a fresh value → `device:${id}`
address changes → server-side device record orphaned every reload
This forces every browser-based Shade app to either (a) accept the broken
UX, or (b) build their own `StorageProvider` from scratch — duplicating
~25 methods × N consumers, with no shared conformance test surface.
The right place to solve this is at the SDK level, exactly mirroring how
SQLite and Postgres are handled.
## Proposed package
`packages/shade-storage-indexeddb/` — modeled directly after
`packages/shade-storage-sqlite/`. Same package shape, same test layout,
same `@shade/core`-only runtime dependency surface.
### Public API
```ts
// @shade/storage-indexeddb
export class IndexedDBStorage implements StorageProvider {
/**
* Open (or create) the IndexedDB database. Idempotent — repeated calls
* with the same dbName return a connection sharing the same object stores.
*/
static async create(opts?: { dbName?: string }): Promise<IndexedDBStorage>;
/**
* Cleanly close the underlying connection. Future calls will reopen.
* Called by Shade.shutdown() when consumers register cleanup.
*/
async close(): Promise<void>;
// ─── all StorageProvider methods (identity, prekeys, sessions,
// retired identities, peer verifications, optional stream-state) ───
}
```
`dbName` defaults to something like `"shade"`. Consumers like Dispatch
will pass distinct names per app (`"dispatch-dashboard-shade"`,
`"dispatch-host-ui-shade"`) so DevTools' IndexedDB inspector groups them
sensibly, even though origin-isolation already makes the data isolated.
### SDK integration
`@shade/sdk` `resolveStorage()` gets a fourth branch:
```ts
if (typeof spec === 'object' && spec.type === 'indexeddb') {
const moduleId = '@shade/storage-indexeddb';
const mod = (await import(moduleId)) as {
IndexedDBStorage: { create(opts: { dbName?: string }): Promise<StorageProvider> };
};
return mod.IndexedDBStorage.create({ dbName: spec.dbName });
}
```
Dynamic import keeps `@shade/storage-indexeddb` an optional dependency,
matching the Postgres pattern — Node-only consumers don't need to install
a browser-only adapter.
Consumer surface:
```ts
const shade = await createShade({
prekeyServer: 'https://…/shade-prekey',
storage: { type: 'indexeddb', dbName: 'my-app-shade' },
address: 'device:user@example.com', // optional — falls back to device:${registrationId}
});
```
## Implementation guidance (non-prescriptive)
- **IDB wrapper**: suggest `idb` (Jake Archibald's thin wrapper, well-typed,
zero deps). Avoid Dexie or idb-keyval — we want full schema control to
match the SQL adapters' explicit schemas.
- **Object-store layout**: one store per StorageProvider category
(`identity`, `signedPreKeys`, `oneTimePreKeys`, `sessions`,
`trustedIdentities`, `retiredIdentities`, `peerVerifications`,
`streamStates`). Keypaths match the natural keys (`keyId`, `address`,
`streamId`).
- **Schema version**: integer, bumped on every shape change. Migrations
in `db.upgrade(...)` callback. Document schema-history alongside
the SQLite schema.
- **Concurrency**: IndexedDB transactions are auto-committing — the adapter
must keep operations within a single transaction where SQL adapters do.
Particular care for `bumpPeerIdentityVersion` (atomic read-modify-write).
- **Stream-state methods**: implement them. Browser apps will increasingly
use `@shade/transfer` for large file resume, and parity with SQLite's
capabilities matters.
## Test expectations
Mirror `packages/shade-storage-sqlite/tests/`:
- `indexeddb-storage.test.ts` — full StorageProvider surface (identity,
sessions, trusted identities, retired identities)
- `indexeddb-prekey-store.test.ts` — signed + one-time prekey lifecycle
- `peer-verifications.test.ts` — verification CRUD + identity-version
bumping invariants
- `indexeddb-stream-state.test.ts` (if stream-state is implemented)
Use **`fake-indexeddb`** for the Node test environment — it's the
established standard, supports the v3 spec, and lets us run IDB tests
in `bun test` / `vitest` / `jest` without a real browser.
If/when Shade gains a shared `StorageProvider` conformance test suite,
this adapter should consume it directly. Until then, follow the SQLite
adapter's per-method coverage style.
## Acceptance criteria
1. `@shade/storage-indexeddb` published at version 4.3.0 (or whichever
matches the next Shade release)
2. `@shade/sdk` `resolveStorage()` resolves `{ type: 'indexeddb', dbName? }`
via dynamic import
3. Full StorageProvider conformance in tests (identity, prekeys,
sessions, retired identities, peer verifications, stream-state)
4. Documented in Shade docs alongside SQLite/Postgres adapters
5. README example showing browser-app integration
6. Bundle-size note: dynamic-imported IDB module shouldn't pull crypto
dependencies — adapter should be ≤ ~10 KB minified+gzipped
## Out of scope (deferred)
- **Encryption-at-rest** for the IDB contents — separate work item; should
match the deviceKey-AES-GCM pattern Shade already uses for `secretEnc`
in `PersistedStreamState`. This adapter ships unencrypted-at-rest in v1
(consistent with SQLite), with the encryption layer added uniformly to
all adapters later.
- **Cross-tab BroadcastChannel sync** — IDB is shared across same-origin
tabs already; concurrent writes work via IDB transactions. Real-time
notification across tabs (e.g. "session was rotated in another tab") is
a separate concern, not storage-adapter scope.
- **Quota handling** — IDB quota for a Shade keystore is far below realistic
browser quotas. If it ever becomes relevant, add a `QuotaExceededError`
observability hook then.
## Why this can't be done in consumer-land
Building this in Dispatch (or any other consumer) would mean:
- 25+ method `StorageProvider` re-implementation per consumer
- No shared conformance tests
- Schema drift across consumers — each would invent its own object-store
shape, blocking any future cross-app data import/export
- Every Shade SDK update that adjusts `StorageProvider` would force every
consumer to track and patch independently
`@shade/storage-sqlite` and `@shade/storage-postgres` are part of the SDK
for the same reason. IndexedDB belongs alongside them.
## What unblocks
Shipping this unblocks Dispatch's "Slice 2.5" — persistent enrollment
across browser refresh, which today is the largest QA-friction point in
the dev loop. Any future browser-Shade consumer (web dashboard,
contact-list app, browser-extension messenger) gets persistence for free.

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/cli", "name": "@shade/cli",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/cli.ts", "main": "src/cli.ts",
"bin": { "bin": {

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/core", "name": "@shade/core",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -185,6 +185,22 @@ export async function ratchetDecrypt(
if (!session.receiveChain) { if (!session.receiveChain) {
throw new DecryptionError('No receiving chain available'); throw new DecryptionError('No receiving chain available');
} }
// Defense-in-depth: a same-DH message whose counter is already
// behind the chain — and that did NOT match a cached skipped key —
// is either a duplicate we already decrypted (skipped key was
// consumed) or one whose key was evicted under cache pressure.
// Falling through would call kdfChainKey on the *current* (ahead)
// chainKey and then rewind `chain.counter = message.counter + 1`,
// permanently desyncing the chain so every subsequent decrypt
// returns wrong-key. Reject without mutating state instead.
if (
!isNewRatchet &&
message.counter < session.receiveChain.counter
) {
throw new DecryptionError(
'Failed to decrypt message — wrong key or tampered data',
);
}
await skipMessageKeys(crypto, session, message.dhPublicKey, session.receiveChain, message.counter); await skipMessageKeys(crypto, session, message.dhPublicKey, session.receiveChain, message.counter);
// Advance the receiving chain one more step to get this message's key // Advance the receiving chain one more step to get this message's key

View File

@@ -46,6 +46,50 @@ export interface PersistedStreamState {
*/ */
export type PeerVerificationSource = 'user' | 'transitive' | 'tofu-after-warning'; export type PeerVerificationSource = 'user' | 'transitive' | 'tofu-after-warning';
/**
* Persisted broadcast-channel state (V4.6). Holds the sender-key chain that
* lets the channel owner encrypt a single ciphertext and fan it out to all
* paired members.
*
* - `ownerRole === 'sender'`: this device created the channel; the record
* carries the full chain (chainKey + iteration + signing keypair).
* - `ownerRole === 'receiver'`: this device joined a channel that
* `ownerAddress` owns; we hold a tracking copy of the chain (chainKey +
* iteration + signing public key only — no private signing key).
*
* `generation` is bumped each time `removeMember` rotates the chain. Stale
* broadcasts at lower generations are silently dropped on receive.
*/
export interface BroadcastChannelRecord {
/** Opaque, stable across restarts. UUID-style. */
channelId: string;
ownerRole: 'sender' | 'receiver';
/** The address that owns the channel (i.e. the only sender). */
ownerAddress: string;
label?: string;
/** Sender-key generation — bumped on rotation. Starts at 0. */
generation: number;
/** Current chain key (32 bytes). */
chainKey: Uint8Array;
/** Counter advanced by `senderKeyEncrypt` / `senderKeyDecrypt`. */
iteration: number;
/** Owner's Ed25519 signing public key (32 bytes). */
signingPublicKey: Uint8Array;
/** Owner's Ed25519 signing private key. Present iff `ownerRole === 'sender'`. */
signingPrivateKey?: Uint8Array;
createdAt: number;
updatedAt: number;
}
/** Membership row for a broadcast channel (sender-side only). */
export interface BroadcastMemberRecord {
channelId: string;
peerAddress: string;
joinedAt: number;
/** When this peer was revoked. `null` while still active. */
removedAt: number | null;
}
/** /**
* Persistent record that a peer's safety number was verified at a point * Persistent record that a peer's safety number was verified at a point
* in time. `identityVersion` is the local counter for that peer's identity: * in time. `identityVersion` is the local counter for that peer's identity:
@@ -188,4 +232,35 @@ export interface StorageProvider {
/** Prune stream-state rows in `'finished' | 'aborted'` status older than `olderThan`. */ /** Prune stream-state rows in `'finished' | 'aborted'` status older than `olderThan`. */
pruneStreamStates?(olderThan: number): Promise<void>; pruneStreamStates?(olderThan: number): Promise<void>;
// ─── Broadcast channels (V4.6) — optional ─────────────────
/**
* Persist or replace the broadcast-channel record. Idempotent upsert on
* `channelId`. Backends that don't implement broadcast-channel support
* may omit this; the SDK throws a clear error when the app tries to
* call `createBroadcastChannel` on such a backend.
*/
saveBroadcastChannel?(channel: BroadcastChannelRecord): Promise<void>;
/** Look up a broadcast channel by id. */
getBroadcastChannel?(channelId: string): Promise<BroadcastChannelRecord | null>;
/** Enumerate all broadcast channels persisted on this device. */
listBroadcastChannels?(): Promise<BroadcastChannelRecord[]>;
/** Drop a broadcast channel and all its membership rows. */
removeBroadcastChannel?(channelId: string): Promise<void>;
/** Persist or replace a broadcast-membership row (sender-side only). */
saveBroadcastMember?(member: BroadcastMemberRecord): Promise<void>;
/**
* List membership rows for a channel. Includes revoked members (with
* `removedAt !== null`); callers filter as needed.
*/
getBroadcastMembers?(channelId: string): Promise<BroadcastMemberRecord[]>;
/** Hard-delete a single membership row. */
removeBroadcastMember?(channelId: string, peerAddress: string): Promise<void>;
} }

View File

@@ -281,6 +281,41 @@ describe('Double Ratchet', () => {
expect(ratchetDecrypt(crypto, bob, msg)).rejects.toThrow(); expect(ratchetDecrypt(crypto, bob, msg)).rejects.toThrow();
}); });
/**
* Regression — the v4.2.0 OutboundQueue waiter-since bug delivered
* the same envelope twice to `manager.decrypt`. The first decrypt
* succeeded via a cached skipped key; the second one fell into the
* `message.counter < chain.counter` path with no skipped key
* available, advanced the chainKey ONCE and rewound `chain.counter`
* to `message.counter + 1`, leaving the ratchet permanently
* desynced. ratchetDecrypt now rejects without mutating state when
* a same-DH message is behind the chain and not in skippedKeys, so
* a downstream replay (transport bug, retry, etc.) cannot poison
* the session for everyone else.
*/
test('same-DH stale message after consumed skipped key fails without corrupting state', async () => {
const { alice, bob } = await setupPair();
// Alice sends 3 messages on the same DH chain.
const m0 = await ratchetEncrypt(crypto, alice, enc.encode('m0'));
const m1 = await ratchetEncrypt(crypto, alice, enc.encode('m1'));
const m2 = await ratchetEncrypt(crypto, alice, enc.encode('m2'));
// Bob receives m1 first, caching m0's key. Then m0 (delivered
// via the cache). After this, m0's skipped key is consumed.
expect(dec.decode(await ratchetDecrypt(crypto, bob, m1))).toBe('m1');
expect(dec.decode(await ratchetDecrypt(crypto, bob, m0))).toBe('m0');
// Replay of m0: skippedKey is gone, chain.counter is past m0.
// Pre-fix: this would corrupt Bob's chain state; post-fix it
// throws cleanly.
await expect(ratchetDecrypt(crypto, bob, m0)).rejects.toThrow(DecryptionError);
// Bob can still decrypt the remaining valid message — chain
// state was NOT mutated by the rejected replay.
expect(dec.decode(await ratchetDecrypt(crypto, bob, m2))).toBe('m2');
});
}); });
// ─── Long Conversation ──────────────────────────────────── // ─── Long Conversation ────────────────────────────────────

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/crypto-web", "name": "@shade/crypto-web",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,4 +1,8 @@
import type { StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey, SessionState, RetiredIdentity, PersistedStreamState, PeerVerification } from '@shade/core'; import type {
StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey,
SessionState, RetiredIdentity, PersistedStreamState, PeerVerification,
BroadcastChannelRecord, BroadcastMemberRecord,
} from '@shade/core';
import { constantTimeEqual } from '@shade/core'; import { constantTimeEqual } from '@shade/core';
/** /**
@@ -167,4 +171,62 @@ export class MemoryStorage implements StorageProvider {
} }
} }
} }
// ─── Broadcast channels (V4.6) ────────────────────────────
private broadcastChannels = new Map<string, BroadcastChannelRecord>();
private broadcastMembers = new Map<string, Map<string, BroadcastMemberRecord>>();
async saveBroadcastChannel(channel: BroadcastChannelRecord): Promise<void> {
this.broadcastChannels.set(channel.channelId, cloneChannel(channel));
}
async getBroadcastChannel(channelId: string): Promise<BroadcastChannelRecord | null> {
const v = this.broadcastChannels.get(channelId);
return v ? cloneChannel(v) : null;
}
async listBroadcastChannels(): Promise<BroadcastChannelRecord[]> {
return [...this.broadcastChannels.values()].map(cloneChannel);
}
async removeBroadcastChannel(channelId: string): Promise<void> {
this.broadcastChannels.delete(channelId);
this.broadcastMembers.delete(channelId);
}
async saveBroadcastMember(member: BroadcastMemberRecord): Promise<void> {
let inner = this.broadcastMembers.get(member.channelId);
if (!inner) {
inner = new Map();
this.broadcastMembers.set(member.channelId, inner);
}
inner.set(member.peerAddress, { ...member });
}
async getBroadcastMembers(channelId: string): Promise<BroadcastMemberRecord[]> {
const inner = this.broadcastMembers.get(channelId);
return inner ? [...inner.values()].map((m) => ({ ...m })) : [];
}
async removeBroadcastMember(channelId: string, peerAddress: string): Promise<void> {
this.broadcastMembers.get(channelId)?.delete(peerAddress);
}
}
function cloneChannel(c: BroadcastChannelRecord): BroadcastChannelRecord {
const out: BroadcastChannelRecord = {
channelId: c.channelId,
ownerRole: c.ownerRole,
ownerAddress: c.ownerAddress,
generation: c.generation,
chainKey: new Uint8Array(c.chainKey),
iteration: c.iteration,
signingPublicKey: new Uint8Array(c.signingPublicKey),
createdAt: c.createdAt,
updatedAt: c.updatedAt,
};
if (c.label !== undefined) out.label = c.label;
if (c.signingPrivateKey !== undefined) out.signingPrivateKey = new Uint8Array(c.signingPrivateKey);
return out;
} }

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/dashboard", "name": "@shade/dashboard",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"scripts": { "scripts": {
"dev": "vite", "dev": "vite",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/files", "name": "@shade/files",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -69,6 +69,11 @@ import {
import { buildRpcRequest } from '../protocol/rpc-builder.js'; import { buildRpcRequest } from '../protocol/rpc-builder.js';
import { decideInline, INLINE_THRESHOLD, type WriteSource } from './inline-threshold.js'; import { decideInline, INLINE_THRESHOLD, type WriteSource } from './inline-threshold.js';
import { base64ToBytes, bytesToBase64 } from '../protocol/canonical.js'; import { base64ToBytes, bytesToBase64 } from '../protocol/canonical.js';
import { startQueueDrainer, type QueueDrainerHandle } from './queue-drainer.js';
import {
createClientStreamsBridge,
type ClientStreamsBridge,
} from './streams-bridge.js';
import type { import type {
FileClient, FileClient,
ReadOpts, ReadOpts,
@@ -80,7 +85,7 @@ import type {
} from './client.js'; } from './client.js';
export interface FilesHttpClientOptions export interface FilesHttpClientOptions
extends Omit<CreateFileClientOptions, 'streamsBridge' | 'ioTimeoutMs'> { extends Omit<CreateFileClientOptions, 'streamsBridge'> {
/** /**
* Server endpoint that hosts `createFilesRpcRoute(...)`. Typically: * Server endpoint that hosts `createFilesRpcRoute(...)`. Typically:
* `https://server.example.com/api/v1/shade-files/rpc`. * `https://server.example.com/api/v1/shade-files/rpc`.
@@ -98,6 +103,32 @@ export interface FilesHttpClientOptions
* orthogonal to the ratchet authentication on the envelope itself. * orthogonal to the ratchet authentication on the envelope itself.
*/ */
headers?: Record<string, string>; headers?: Record<string, string>;
/**
* Server endpoint that hosts `transferQueueRoute()`'s long-poll
* endpoint. Typically:
* `https://server.example.com/api/v1/shade-files/queue`.
*
* When supplied, the client starts a background long-poll that
* drains queued envelopes + chunks from the server and dispatches
* them via `shade.acceptTransferEnvelope`. This unlocks
* **streamed reads** (>256 KiB) for browser-style consumers.
*/
outboundQueueUrl?: string;
/**
* Base URL for outbound transfer routes (browser → server). Required
* alongside `outboundQueueUrl` to enable streamed writes. Typically:
* `https://server.example.com/api/v1/shade-files`.
*
* The client POSTs:
* - chunks to `<base>/v1/transfer/<streamId>/chunk`
* - control envelopes to `<base>/v1/transfer/control`
*/
transferBaseUrl?: string;
/**
* Long-poll block timeout, milliseconds. Default 30_000. Server
* clamps to its own `maxBlockMs` (default 55_000).
*/
queueBlockMs?: number;
} }
interface RoundTripOpts { interface RoundTripOpts {
@@ -112,6 +143,12 @@ interface RoundTripOpts {
* (via `shade.initSessionFromBundle(peerAddress, bundle)` or an * (via `shade.initSessionFromBundle(peerAddress, bundle)` or an
* incoming first-message). Otherwise the first RPC will fail with * incoming first-message). Otherwise the first RPC will fail with
* "decrypt failed: no session for peer". * "decrypt failed: no session for peer".
*
* When `outboundQueueUrl` + `transferBaseUrl` are supplied, the
* client also unlocks **streamed reads/writes** for files larger than
* the inline threshold (256 KiB). The browser polls the server's
* outbound queue for chunks/envelopes and POSTs its own outbound
* chunks to the server's transfer-receive routes.
*/ */
export function createFilesHttpClient( export function createFilesHttpClient(
shade: ShadeBridge, shade: ShadeBridge,
@@ -122,9 +159,84 @@ export function createFilesHttpClient(
const fetchFn = options.fetch ?? globalThis.fetch.bind(globalThis); const fetchFn = options.fetch ?? globalThis.fetch.bind(globalThis);
const extraHeaders = options.headers ?? {}; const extraHeaders = options.headers ?? {};
const defaultTimeoutMs = options.defaultTimeoutMs ?? 30_000; const defaultTimeoutMs = options.defaultTimeoutMs ?? 30_000;
const ioTimeoutMs = options.ioTimeoutMs ?? 60_000;
const signRequest = options.signRequest; const signRequest = options.signRequest;
const senderAddress = shade.myAddress; const senderAddress = shade.myAddress;
// ─── Streamed-mode bootstrap ─────────────────────────────────
//
// When `outboundQueueUrl` is supplied, the client:
// 1. Configures `shade.configureTransfers(...)` so outbound
// chunks POST to `<transferBaseUrl>/v1/transfer/<streamId>/chunk`
// and outbound control envelopes POST to
// `<transferBaseUrl>/v1/transfer/control`.
// 2. Spawns a streams-bridge so streamed reads can be awaited.
// 3. Starts a long-poll drainer that pulls queued envelopes +
// chunks from the server and dispatches via
// `shade.acceptTransferEnvelope`.
let drainer: QueueDrainerHandle | null = null;
let streamsBridgePromise: Promise<ClientStreamsBridge> | null = null;
let streamsBridge: ClientStreamsBridge | null = null;
if (options.outboundQueueUrl !== undefined) {
const outboundQueueUrl = options.outboundQueueUrl;
if (options.transferBaseUrl === undefined) {
throw new Error(
'createFilesHttpClient: outboundQueueUrl was supplied without transferBaseUrl. Pass `transferBaseUrl` (the server prefix that hosts /v1/transfer/...) so outbound chunks have a destination.',
);
}
if (shade.configureTransfers === undefined) {
throw new Error(
'createFilesHttpClient: shade.configureTransfers is required for streamed mode (the underlying ShadeBridge must surface it).',
);
}
const transferBaseUrl = options.transferBaseUrl.replace(/\/$/, '');
shade.configureTransfers({
resolveBaseUrl: async (peer) => {
if (peer !== peerAddress) {
throw new Error(
`httpClient is bound to peer "${peerAddress}" — refusing to resolve outgoing chunks for "${peer}" without a multi-peer registry. Use shade.files.client(peer) for server-to-server multi-peer.`,
);
}
return transferBaseUrl;
},
});
// Build the streams-bridge eagerly. The engine's incoming-transfer
// subscription has to be in place BEFORE the drainer dispatches the
// first stream-init envelope, otherwise the engine emits the
// IncomingTransfer to zero handlers and the read silently never
// accepts. We kick off the drainer once the bridge has subscribed.
streamsBridgePromise = createClientStreamsBridge(shade).then((bridge) => {
streamsBridge = bridge;
drainer = startQueueDrainer(shade, {
outboundQueueUrl,
peerAddress,
senderAddress,
...(options.fetch !== undefined ? { fetch: options.fetch } : {}),
...(options.headers !== undefined ? { headers: options.headers } : {}),
...(options.queueBlockMs !== undefined ? { blockMs: options.queueBlockMs } : {}),
});
return bridge;
});
// Surface bridge-construction failures eagerly via a rejected
// promise the next read/write picks up.
streamsBridgePromise.catch(() => {
/* observed via getStreamsBridge() */
});
}
async function getStreamsBridge(): Promise<ClientStreamsBridge> {
if (streamsBridge !== null) return streamsBridge;
if (streamsBridgePromise === null) {
throw new ConflictError(
`http RPC client supports inline writes/reads only (≤ ${INLINE_THRESHOLD} bytes) — pass { outboundQueueUrl, transferBaseUrl } to enable streamed transfers.`,
);
}
streamsBridge = await streamsBridgePromise;
return streamsBridge;
}
/** /**
* Encrypt + POST + decrypt + parse one RPC round-trip. * Encrypt + POST + decrypt + parse one RPC round-trip.
* *
@@ -321,20 +433,39 @@ export function createFilesHttpClient(
ReadResultSchema, ReadResultSchema,
opts, opts,
); );
if (wire.kind !== 'inline') { if (wire.kind === 'inline') {
// The HTTP RPC route does not service streamed reads — there is const bytes = base64ToBytes(wire.bytesB64);
// no place to stream from in pure request-response. const out: ReadOutput = {
kind: 'inline',
bytes,
size: wire.size,
sha256: wire.sha256,
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
};
return out;
}
// Streamed read — only supported when the queue drainer is wired.
if (drainer === null) {
throw new InternalFileError( throw new InternalFileError(
`http RPC client received a streamed read (size ${wire.size}). Use shade.files.client(peer) on a server-to-server deployment, or pass { preferInline: true } when the file is known to fit inline.`, `http RPC client received a streamed read (size ${wire.size}) but is in inline-only mode. Pass { outboundQueueUrl, transferBaseUrl } when constructing the client to enable streamed reads.`,
); );
} }
const bytes = base64ToBytes(wire.bytesB64); const bridge = await getStreamsBridge();
const bridgeSignal = opts.signal ?? new AbortController().signal;
const parked = await bridge.awaitRead(wire.streamId, {
expectedFrom: peerAddress,
signal: bridgeSignal,
timeoutMs: ioTimeoutMs,
});
const out: ReadOutput = { const out: ReadOutput = {
kind: 'inline', kind: 'streams',
bytes, stream: parked.readable,
size: wire.size, size: wire.size,
sha256: wire.sha256, sha256: wire.sha256,
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}), ...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
done: async () => {
await parked.done;
},
}; };
return out; return out;
}, },
@@ -344,25 +475,77 @@ export function createFilesHttpClient(
const overwrite = opts.overwrite ?? false; const overwrite = opts.overwrite ?? false;
const contentType = opts.contentType ?? decision.contentType; const contentType = opts.contentType ?? decision.contentType;
if (decision.kind !== 'inline') { if (decision.kind === 'inline' || opts.forceInline === true) {
throw new ConflictError( const bytes = decision.kind === 'inline' ? decision.bytes : null;
`http RPC client supports inline writes only (≤ ${INLINE_THRESHOLD} bytes). The supplied input was promoted to streams (size ${decision.size ?? 'unknown'}). Use shade.files.client(peer) for streamed writes, or pre-buffer the input below the inline threshold.`, if (bytes === null) {
// forceInline === true with a streams-typed decision —
// decideInline always produced a `streams` shape because the
// input was a bare ReadableStream. We can't drain a stream
// synchronously here without a streams-bridge.
throw new ConflictError(
'http RPC client cannot forceInline a streamed input — pass a Uint8Array / Blob, or pre-buffer the stream.',
);
}
if (bytes.byteLength > INLINE_THRESHOLD) {
throw new ConflictError(
`inline write exceeds ${INLINE_THRESHOLD}-byte threshold (got ${bytes.byteLength}); pass forceInline=true to override`,
);
}
const args = WriteArgsSchema.parse({
kind: 'inline',
path,
bytesB64: bytesToBase64(bytes),
...(contentType !== undefined ? { contentType } : {}),
overwrite,
});
return await roundTrip<WriteResult>(
KIND_WRITE_V1,
'write',
args,
WriteResultSchema,
opts,
); );
} }
// Streamed write — requires the queue drainer + streams-bridge.
if (drainer === null) {
throw new ConflictError(
`http RPC client supports inline writes only (≤ ${INLINE_THRESHOLD} bytes). The supplied input was promoted to streams (size ${decision.size ?? 'unknown'}). Pass { outboundQueueUrl, transferBaseUrl } to enable streamed writes.`,
);
}
const bridge = await getStreamsBridge();
const size = decision.size;
if (size === undefined) {
throw new ConflictError(
'streams write requires a known plaintext size; pass `{ stream, size }` instead of a bare ReadableStream',
);
}
const { writeId, handle } = await bridge.initiateWrite({
peer: peerAddress,
stream: decision.stream,
size,
...(contentType !== undefined ? { contentType } : {}),
name: path,
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
});
const args = WriteArgsSchema.parse({ const args = WriteArgsSchema.parse({
kind: 'inline', kind: 'streams',
path, path,
bytesB64: bytesToBase64(decision.bytes), size,
...(contentType !== undefined ? { contentType } : {}), ...(contentType !== undefined ? { contentType } : {}),
overwrite, overwrite,
writeId,
}); });
return await roundTrip<WriteResult>( try {
KIND_WRITE_V1, const [result] = await Promise.all([
'write', roundTrip<WriteResult>(KIND_WRITE_V1, 'write', args, WriteResultSchema, opts),
args, handle.done(),
WriteResultSchema, ]);
opts, return result;
); } catch (err) {
await handle.abort('rpc-failed').catch(() => undefined);
throw err;
}
}, },
async getThumbnail(path, size: ThumbnailSize, opts): Promise<ThumbnailResult> { async getThumbnail(path, size: ThumbnailSize, opts): Promise<ThumbnailResult> {
@@ -393,7 +576,15 @@ export function createFilesHttpClient(
}, },
close(): void { close(): void {
// Stateless — nothing to release. Exists for FileClient symmetry. // Stop the long-poll drainer + tear down the streams-bridge if
// we built one. Idempotent — safe to call multiple times.
drainer?.stop();
drainer = null;
if (streamsBridge !== null) {
void streamsBridge.destroy().catch(() => undefined);
streamsBridge = null;
}
streamsBridgePromise = null;
}, },
} as FileClient; } as FileClient;
} }

View File

@@ -0,0 +1,172 @@
/**
* Browser-side drainer for the pull-mode outbound queue.
*
* Background task that long-polls the server's `/queue` endpoint,
* decodes each event, and dispatches it into the consumer's Shade
* instance via `shade.acceptTransferEnvelope`. Same wire-shape as the
* server-to-server case where the engine receives chunks via direct
* HTTP POSTs — we just flip the direction so the browser pulls
* instead of accepts.
*/
import type { ShadeBridge } from '../integration/shade-bridge.js';
export interface QueueDrainerOptions {
/**
* Server endpoint that hosts `transferQueueRoute()`. Typically:
* `https://server.example.com/api/v1/shade-files/queue`.
*/
outboundQueueUrl: string;
/** Peer the queue is pulled FROM (the server's address). */
peerAddress: string;
/** Address we identify ourselves with on the long-poll. */
senderAddress: string;
/** Optional `fetch` override. Default `globalThis.fetch`. */
fetch?: typeof globalThis.fetch;
/** Extra headers applied to every poll request. */
headers?: Record<string, string>;
/**
* Long-poll request timeout (server-side block). Default 30_000.
* Server clamps to its own `maxBlockMs` (default 55_000).
*/
blockMs?: number;
/**
* Backoff after a network error before re-polling. Default 2_000.
* Doubles up to `maxBackoffMs` on consecutive failures.
*/
initialBackoffMs?: number;
maxBackoffMs?: number;
/**
* Called when a poll cycle fails. Defaults to logging via `console.error`.
* Throwing from this hook does NOT stop the drainer — it backs off
* and retries.
*/
onError?: (err: unknown) => void;
}
export interface QueueDrainerHandle {
/** Stop the drainer. Pending fetch is aborted; the loop exits. */
stop(): void;
/** Promise that resolves once the drainer has fully stopped. */
stopped: Promise<void>;
}
interface PolledEvent {
id: number;
timestampMs: number;
kind: 'envelope' | 'chunk';
bytesB64: string;
meta?: { streamId: string; laneId: number; seq: number };
}
interface PollResponseBody {
events: PolledEvent[];
nextSince: number;
}
const DEFAULT_BLOCK_MS = 30_000;
const DEFAULT_INITIAL_BACKOFF_MS = 2_000;
const DEFAULT_MAX_BACKOFF_MS = 30_000;
/**
* Start a long-poll loop that drains queued envelopes + chunks from
* the server and dispatches them into the local Shade instance.
*
* Returns a handle the caller can use to stop the drainer when the
* `httpClient` is closed (e.g. on tab unload).
*/
export function startQueueDrainer(
shade: ShadeBridge,
options: QueueDrainerOptions,
): QueueDrainerHandle {
if (shade.acceptTransferEnvelope === undefined) {
throw new Error(
'startQueueDrainer: shade.acceptTransferEnvelope is required for pull-mode streams. The supplied ShadeBridge implementation must surface it.',
);
}
const accept = shade.acceptTransferEnvelope.bind(shade);
const fetchFn = options.fetch ?? globalThis.fetch.bind(globalThis);
const blockMs = options.blockMs ?? DEFAULT_BLOCK_MS;
const onError = options.onError ?? ((err: unknown) => console.error('[shade.files queue-drainer]', err));
const initialBackoffMs = options.initialBackoffMs ?? DEFAULT_INITIAL_BACKOFF_MS;
const maxBackoffMs = options.maxBackoffMs ?? DEFAULT_MAX_BACKOFF_MS;
const ac = new AbortController();
let stopped = false;
let resolveStopped!: () => void;
const stoppedPromise = new Promise<void>((r) => {
resolveStopped = r;
});
void (async () => {
let since = 0;
let backoff = initialBackoffMs;
while (!stopped) {
try {
const response = await fetchFn(options.outboundQueueUrl, {
method: 'POST',
signal: ac.signal,
headers: {
'Content-Type': 'application/json',
'X-Shade-Sender-Address': options.senderAddress,
...(options.headers ?? {}),
},
body: JSON.stringify({ since, blockMs }),
});
if (!response.ok) {
throw new Error(`queue poll → ${response.status} ${response.statusText}`);
}
const body = (await response.json()) as PollResponseBody;
if (Array.isArray(body.events) && body.events.length > 0) {
for (const event of body.events) {
if (stopped) break;
try {
const bytes = base64ToBytes(event.bytesB64);
await accept(options.peerAddress, bytes);
} catch (err) {
// Per-event dispatch failure should not kill the loop —
// resume picks up missing chunks via @shade/transfer's
// built-in lane-resume protocol.
onError(err);
}
}
}
since = typeof body.nextSince === 'number' ? body.nextSince : since;
backoff = initialBackoffMs;
} catch (err) {
if (stopped || ac.signal.aborted) break;
onError(err);
// Exponential backoff with jitter — caps at maxBackoffMs.
const jitter = Math.floor(Math.random() * Math.min(backoff, 1_000));
await new Promise<void>((r) => {
const t = setTimeout(r, backoff + jitter);
(t as unknown as { unref?: () => void }).unref?.();
ac.signal.addEventListener(
'abort',
() => {
clearTimeout(t);
r();
},
{ once: true },
);
});
backoff = Math.min(maxBackoffMs, backoff * 2);
}
}
resolveStopped();
})();
return {
stop(): void {
if (stopped) return;
stopped = true;
ac.abort(new Error('queue drainer stopped'));
},
stopped: stoppedPromise,
};
}
function base64ToBytes(b64: string): Uint8Array {
const bin = atob(b64);
const out = new Uint8Array(bin.length);
for (let i = 0; i < bin.length; i++) out[i] = bin.charCodeAt(i);
return out;
}

View File

@@ -102,7 +102,16 @@ export async function createClientStreamsBridge(
const readStreamId = incoming.metadata.userMetadata?.[META_KEY_READ_STREAM_ID]; const readStreamId = incoming.metadata.userMetadata?.[META_KEY_READ_STREAM_ID];
if (readStreamId === undefined) return; if (readStreamId === undefined) return;
const ts = new TransformStream<Uint8Array, Uint8Array>(); // Generous HWM so the receiver-side write loop (drainer →
// engine.receiveChunk → sink.write) doesn't stall on backpressure
// before the consumer's reader is wired up. The reader still
// applies its own backpressure once it's consuming, but we no
// longer race fs.read's await on stream-init against the consumer
// attaching its reader.
const ts = new TransformStream<Uint8Array, Uint8Array>(undefined, undefined, {
highWaterMark: 64,
size: (chunk?: Uint8Array) => (chunk === undefined ? 0 : 1),
});
let handle: TransferHandle; let handle: TransferHandle;
try { try {
handle = await incoming.accept({ handle = await incoming.accept({

View File

@@ -190,6 +190,11 @@ export { createFilesRpcRoute } from './server/rpc-route.js';
export type { FilesRpcRouteOptions } from './server/rpc-route.js'; export type { FilesRpcRouteOptions } from './server/rpc-route.js';
export { createFilesHttpClient } from './client/http-client.js'; export { createFilesHttpClient } from './client/http-client.js';
export type { FilesHttpClientOptions } from './client/http-client.js'; export type { FilesHttpClientOptions } from './client/http-client.js';
export { startQueueDrainer } from './client/queue-drainer.js';
export type {
QueueDrainerHandle,
QueueDrainerOptions,
} from './client/queue-drainer.js';
// Shared structural surface @shade/files needs from a Shade instance — // Shared structural surface @shade/files needs from a Shade instance —
// exposed so consumers building custom Shade-shaped bridges can verify // exposed so consumers building custom Shade-shaped bridges can verify

View File

@@ -70,4 +70,25 @@ export interface ShadeBridge {
/** Optional control-envelope passthrough used by the WebRTC bridge. */ /** Optional control-envelope passthrough used by the WebRTC bridge. */
deliverControlEnvelope?(peer: string, envelope: ShadeEnvelope): Promise<void>; deliverControlEnvelope?(peer: string, envelope: ShadeEnvelope): Promise<void>;
/**
* Hand a freshly-decoded wire envelope (control or chunk) to the
* transfer engine. Required by the pull-mode HTTP client when it
* drains queued events from the server: each polled chunk / control
* envelope is dispatched here so the engine sees it just as if it
* had arrived via an HTTP POST on `/v1/transfer/...`.
*/
acceptTransferEnvelope?(from: string, env: ShadeEnvelope | Uint8Array): Promise<void>;
/**
* Configure the transfer stack. Called by the pull-mode HTTP client
* to point the browser's outgoing chunks + control envelopes at the
* server's transferQueueRoute mount. Optional because the
* server-to-server path uses a separate, app-driven configuration.
*/
configureTransfers?(opts: {
resolveBaseUrl?: (peerAddress: string) => Promise<string>;
transport?: unknown;
envelopeTransport?: unknown;
}): void;
} }

View File

@@ -0,0 +1,184 @@
import { describe, expect, test } from 'bun:test';
import { createShade } from '@shade/sdk';
import {
createPrekeyServer,
MemoryPrekeyStore,
PrekeyServerEvents,
} from '@shade/server';
import { SubtleCryptoProvider } from '@shade/crypto-web';
import { Hono } from 'hono';
const crypto = new SubtleCryptoProvider();
/**
* Concurrent-ratchet hardening tests.
*
* Reproduces the scenario described in the v4.2.0 ratchet-desync bug
* report: with the queue-drainer running on Alice and many concurrent
* `shade.send`/RPC operations against the same peer, do
* encrypt/decrypt paths share the per-peer mutex on
* `ShadeSessionManager` so that no path observes a stale ratchet
* state?
*
* If the lock coverage regresses (a future change re-introduces a
* sidekanal bypass), one of these tests will fail with
* `DecryptionError: Failed to decrypt message — wrong key or
* tampered data`.
*/
async function setupPullRig() {
const prekey = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
events: new PrekeyServerEvents(),
});
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
const prekeyUrl = `http://localhost:${prekeyServer.port}`;
const alice = await createShade({ prekeyServer: prekeyUrl, address: 'alice' });
const bob = await createShade({ prekeyServer: prekeyUrl, address: 'bob' });
const queueRoute = await bob.transferQueueRoute({ blockMs: 500 });
await bob.files.serve({
stat: async () => ({
name: '_',
kind: 'dir' as const,
size: 0,
mtime: 0,
metadata: {},
}),
});
const rpcRoute = bob.files.rpcRoute({ acceptFirstMessage: true });
const app = new Hono();
app.route('/', queueRoute);
app.route('/', rpcRoute);
const bobServer = Bun.serve({ port: 0, fetch: app.fetch });
const baseUrl = `http://localhost:${bobServer.port}`;
const fs = alice.files.httpClient('bob', {
rpcUrl: `${baseUrl}/rpc`,
outboundQueueUrl: `${baseUrl}/queue`,
transferBaseUrl: baseUrl,
defaultTimeoutMs: 10_000,
queueBlockMs: 500,
});
return {
alice,
bob,
fs,
baseUrl,
teardown: async () => {
fs.close();
await alice.shutdown();
await bob.shutdown();
bobServer.stop();
prekeyServer.stop();
},
};
}
describe('@shade/files — concurrent ratchet under drainer', () => {
test('100 parallel httpClient RPCs while drainer runs — no DecryptionError', async () => {
const rig = await setupPullRig();
try {
// Warm-up: establishes the X3DH session (Alice → Bob first message
// is a PreKeyMessage; subsequent messages are pure ratchet).
const first = await rig.fs.stat('/');
expect(first.kind).toBe('dir');
// Fire 100 concurrent stat RPCs. Each one is a full ratchet
// round-trip: encrypt request, POST, decrypt response. They all
// contend for `manager.peerOpChains["bob"]` on Alice's side
// (encrypt + decrypt) and `manager.peerOpChains["alice"]` on
// Bob's side. Drainer is running in the background polling
// Bob's queue — its decrypt path also funnels through the same
// per-peer lock.
// 100 concurrent — minimal repro (after warm-up only).
const N = 100;
const results = await Promise.allSettled(
Array.from({ length: N }, () => rig.fs.stat('/')),
);
const failures = results.filter((s) => s.status === 'rejected') as Array<
PromiseRejectedResult
>;
if (failures.length > 0) {
const sample = failures.slice(0, 1).map((f) => String(f.reason));
throw new Error(`${failures.length}/${N} concurrent RPCs failed: ${sample[0]}`);
}
} finally {
await rig.teardown();
}
}, 30_000);
test('parallel shade.send + drainer + RPCs — ratchet stays in sync', async () => {
const rig = await setupPullRig();
try {
// Establish session via one warm-up RPC.
await rig.fs.stat('/');
// Subscribe Bob to inbound plaintext from Alice — when Alice's
// raw `shade.send` plaintext arrives, Bob echoes a reply back
// through `shade.send` + `deliverControlEnvelope`, which the
// pull-mode envelope transport enqueues for Alice's drainer.
// This injects extra inbound traffic into Alice's drainer in
// parallel with her ongoing RPCs.
const echoes: string[] = [];
rig.bob.onMessage(async (from, plaintext) => {
if (from !== 'alice') return;
if (!plaintext.startsWith('ping:')) return;
echoes.push(plaintext);
const reply = `pong:${plaintext.slice('ping:'.length)}`;
const env = await rig.bob.send('alice', reply);
await rig.bob.deliverControlEnvelope('alice', env);
});
const inboundDrained: string[] = [];
rig.alice.onMessage((from, plaintext) => {
if (from !== 'bob') return;
if (plaintext.startsWith('pong:')) inboundDrained.push(plaintext);
});
// Mix three concurrent workloads against the same peer:
// - 50 inline file RPCs through httpClient (encrypt + decrypt)
// - 50 raw `shade.send` deliveries via control envelope
// - drainer pulling Bob's responses + echoes
const N = 50;
const rpcs = Array.from({ length: N }, () => rig.fs.stat('/'));
const sends = Array.from({ length: N }, async (_, i) => {
const env = await rig.alice.send('bob', `ping:${i}`);
await rig.alice.deliverControlEnvelope('bob', env);
});
const settled = await Promise.allSettled([...rpcs, ...sends]);
const failures = settled.filter((s) => s.status === 'rejected') as Array<
PromiseRejectedResult
>;
if (failures.length > 0) {
const sample = failures.slice(0, 3).map((f) => String(f.reason));
throw new Error(
`${failures.length}/${settled.length} concurrent ops failed: ${sample.join(' | ')}`,
);
}
// Give Bob's queue + Alice's drainer a beat to drain pongs back.
// Echoes round-trip Alice → Bob (control envelope) → Bob's
// onMessage → Bob.send + deliver (queue) → Alice's drainer →
// Alice's onMessage. We just verify some make it back without
// any DecryptionError surfacing.
const deadline = Date.now() + 5_000;
while (inboundDrained.length < N && Date.now() < deadline) {
await new Promise((r) => setTimeout(r, 50));
}
// Don't gate on every echo arriving — the long-poll cadence and
// bun's serve/abort timing can lag a few. We only care that the
// ratchet didn't desync; if it had, every subsequent op would
// throw DecryptionError above.
expect(echoes.length).toBe(N);
expect(inboundDrained.length).toBeGreaterThan(0);
} finally {
await rig.teardown();
}
}, 30_000);
});

View File

@@ -0,0 +1,208 @@
import { describe, expect, test } from 'bun:test';
import { createShade } from '@shade/sdk';
import {
createPrekeyServer,
MemoryPrekeyStore,
PrekeyServerEvents,
} from '@shade/server';
import { SubtleCryptoProvider } from '@shade/crypto-web';
import { Hono } from 'hono';
const crypto = new SubtleCryptoProvider();
/**
* Stand up the full pull-mode rig:
* - Prekey server (for X3DH)
* - Bob: file handler + rpcRoute + transferQueueRoute, all on one server
* - Alice: httpClient with outboundQueueUrl + transferBaseUrl wired
*
* Returns Alice's `FileClient`, which speaks browser-style: ONE base URL,
* no inbound listener, streams supported via long-poll.
*/
async function setupPullRig(opts: {
bobHandler: Parameters<NonNullable<Awaited<ReturnType<typeof createShade>>['files']>['serve']>[0];
}) {
const prekey = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
events: new PrekeyServerEvents(),
});
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
const prekeyUrl = `http://localhost:${prekeyServer.port}`;
const alice = await createShade({ prekeyServer: prekeyUrl, address: 'alice' });
const bob = await createShade({ prekeyServer: prekeyUrl, address: 'bob' });
// Bob: queue-route FIRST (configures bob's transports), then files.serve.
const queueRoute = await bob.transferQueueRoute({ blockMs: 1_500 });
await bob.files.serve(opts.bobHandler);
const rpcRoute = bob.files.rpcRoute({ acceptFirstMessage: true });
const app = new Hono();
app.route('/', queueRoute);
app.route('/', rpcRoute);
const bobServer = Bun.serve({ port: 0, fetch: app.fetch });
const baseUrl = `http://localhost:${bobServer.port}`;
const fs = alice.files.httpClient('bob', {
rpcUrl: `${baseUrl}/rpc`,
outboundQueueUrl: `${baseUrl}/queue`,
transferBaseUrl: baseUrl,
defaultTimeoutMs: 10_000,
queueBlockMs: 1_000,
});
return {
alice,
bob,
fs,
baseUrl,
teardown: async () => {
fs.close();
await alice.shutdown();
await bob.shutdown();
bobServer.stop();
prekeyServer.stop();
},
};
}
describe('@shade/files HTTP RPC — pull-mode streams', () => {
test('streamed read (4 MiB) via long-poll queue', async () => {
const payload = new Uint8Array(4 * 1024 * 1024);
for (let i = 0; i < payload.length; i++) payload[i] = (i * 97) & 0xff;
const rig = await setupPullRig({
bobHandler: {
read: async () => {
// Return the payload as a streamed read so the rpc-handler
// promotes it via the streams-bridge into a transfer.
const stream = new ReadableStream<Uint8Array>({
start(controller) {
const CHUNK = 256 * 1024;
for (let off = 0; off < payload.byteLength; off += CHUNK) {
controller.enqueue(payload.slice(off, Math.min(off + CHUNK, payload.byteLength)));
}
controller.close();
},
});
// Need a precomputed sha256 for streamed reads. Use the
// crypto provider's sha256 directly.
const digest = new Uint8Array(await globalThis.crypto.subtle.digest('SHA-256', payload));
const sha256Hex = Array.from(digest, (b) => b.toString(16).padStart(2, '0')).join('');
return {
kind: 'streams' as const,
stream,
size: payload.byteLength,
sha256: sha256Hex,
contentType: 'application/octet-stream',
};
},
},
});
try {
const result = await rig.fs.read('/big.bin');
expect(result.kind).toBe('streams');
if (result.kind !== 'streams') return;
// Drain the stream and compare.
const reader = result.stream.getReader();
const got = new Uint8Array(payload.byteLength);
let offset = 0;
while (true) {
const { value, done } = await reader.read();
if (done) break;
if (value !== undefined) {
got.set(value, offset);
offset += value.byteLength;
}
}
reader.releaseLock();
await result.done();
expect(offset).toBe(payload.byteLength);
// Compare in 64KiB strides for speed.
let mismatch = -1;
for (let i = 0; i < payload.byteLength; i++) {
if (got[i] !== payload[i]) {
mismatch = i;
break;
}
}
expect(mismatch).toBe(-1);
} finally {
await rig.teardown();
}
}, 30_000);
test('streamed read fails with clear error when outboundQueueUrl is omitted', async () => {
const rig = await setupPullRig({
bobHandler: {
read: async () => {
const stream = new ReadableStream<Uint8Array>({
start(c) {
c.enqueue(new Uint8Array(512 * 1024));
c.close();
},
});
const digest = new Uint8Array(await globalThis.crypto.subtle.digest('SHA-256', new Uint8Array(512 * 1024)));
const sha256Hex = Array.from(digest, (b) => b.toString(16).padStart(2, '0')).join('');
return {
kind: 'streams' as const,
stream,
size: 512 * 1024,
sha256: sha256Hex,
};
},
},
});
// Tear down the rig's drainer so we can construct an inline-only client
rig.fs.close();
const inlineOnly = rig.alice.files.httpClient('bob', {
rpcUrl: `${rig.baseUrl}/rpc`,
defaultTimeoutMs: 10_000,
});
try {
await expect(inlineOnly.read('/big.bin')).rejects.toThrow(/streamed read/);
} finally {
inlineOnly.close();
await rig.teardown();
}
}, 15_000);
test('long-poll returns empty events on idle timeout', async () => {
const rig = await setupPullRig({
bobHandler: {
stat: async () => ({
name: '_',
kind: 'dir' as const,
size: 0,
mtime: 0,
metadata: {},
}),
},
});
try {
// Direct poll without any pending events — should return after blockMs.
const start = Date.now();
const res = await fetch(`${rig.baseUrl}/queue`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-Shade-Sender-Address': 'alice',
},
body: JSON.stringify({ since: 0, blockMs: 500 }),
});
expect(res.status).toBe(200);
const body = (await res.json()) as { events: unknown[]; nextSince: number };
expect(body.events).toHaveLength(0);
expect(Date.now() - start).toBeGreaterThanOrEqual(400);
} finally {
await rig.teardown();
}
}, 10_000);
});

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/inbox-server", "name": "@shade/inbox-server",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/inbox", "name": "@shade/inbox",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/key-transparency", "name": "@shade/key-transparency",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/keychain", "name": "@shade/keychain",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/observability", "name": "@shade/observability",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/observer", "name": "@shade/observer",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/proto", "name": "@shade/proto",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -5,7 +5,10 @@ export {
encodeRatchetMessage, encodeRatchetMessage,
encodeStreamChunk, encodeStreamChunk,
decodeStreamChunk, decodeStreamChunk,
encodeBroadcast,
decodeBroadcast,
inspectEnvelopeType, inspectEnvelopeType,
TYPE_STREAM_CHUNK, TYPE_STREAM_CHUNK,
TYPE_BROADCAST,
} from './wire.js'; } from './wire.js';
export type { StreamChunkWire } from './wire.js'; export type { StreamChunkWire, BroadcastWire } from './wire.js';

View File

@@ -7,6 +7,7 @@
* 0x01 = PreKeyMessage * 0x01 = PreKeyMessage
* 0x02 = RatchetMessage * 0x02 = RatchetMessage
* 0x11 = StreamChunk * 0x11 = StreamChunk
* 0x21 = BroadcastMessage (V4.6 — sender-key encrypted group payload)
* *
* All multi-byte integers are big-endian. * All multi-byte integers are big-endian.
* *
@@ -23,6 +24,7 @@ const VERSION = 0x02;
const TYPE_PREKEY = 0x01; const TYPE_PREKEY = 0x01;
const TYPE_RATCHET = 0x02; const TYPE_RATCHET = 0x02;
export const TYPE_STREAM_CHUNK = 0x11; export const TYPE_STREAM_CHUNK = 0x11;
export const TYPE_BROADCAST = 0x21;
// ─── Stream chunk types ────────────────────────────────────── // ─── Stream chunk types ──────────────────────────────────────
@@ -43,6 +45,28 @@ export interface StreamChunkWire {
ciphertext: Uint8Array; // AES-GCM ciphertext including 16-byte tag ciphertext: Uint8Array; // AES-GCM ciphertext including 16-byte tag
} }
/**
* Wire-decoded broadcast envelope (type 0x21).
*
* Carries a Signal-style sender-key message. The sender (channel owner)
* encrypted once with their per-channel chain key; this same byte sequence
* is delivered verbatim to every member. Authenticity rides on the embedded
* Ed25519 signature — no bilateral ratchet wraps the broadcast itself.
*
* `generation` is a per-channel rotation counter: bumped each time a
* member is revoked. Receivers silently drop broadcasts at older
* generations than their currently-installed sender-key.
*/
export interface BroadcastWire {
channelId: string; // utf-8, length-prefixed
senderAddress: string; // utf-8, length-prefixed
generation: number; // u32
iteration: number; // u32 — sender chain counter
nonce: Uint8Array; // AES-GCM nonce (12 bytes)
signature: Uint8Array; // Ed25519 signature (64 bytes)
ciphertext: Uint8Array; // AES-GCM ciphertext including 16-byte tag
}
const STREAM_ID_BYTES = 16; const STREAM_ID_BYTES = 16;
const STREAM_NONCE_BYTES = 12; const STREAM_NONCE_BYTES = 12;
@@ -230,11 +254,11 @@ export function decodeStreamChunk(data: Uint8Array): StreamChunkWire {
/** /**
* Inspect the type tag of an arbitrary envelope without full parsing. * Inspect the type tag of an arbitrary envelope without full parsing.
* Returns one of `'prekey' | 'ratchet' | 'stream-chunk' | 'unknown'`. * Returns one of `'prekey' | 'ratchet' | 'stream-chunk' | 'broadcast' | 'unknown'`.
*/ */
export function inspectEnvelopeType( export function inspectEnvelopeType(
data: Uint8Array, data: Uint8Array,
): 'prekey' | 'ratchet' | 'stream-chunk' | 'unknown' { ): 'prekey' | 'ratchet' | 'stream-chunk' | 'broadcast' | 'unknown' {
if (data.length < 2 || data[0] !== VERSION) return 'unknown'; if (data.length < 2 || data[0] !== VERSION) return 'unknown';
switch (data[1]) { switch (data[1]) {
case TYPE_PREKEY: case TYPE_PREKEY:
@@ -243,11 +267,116 @@ export function inspectEnvelopeType(
return 'ratchet'; return 'ratchet';
case TYPE_STREAM_CHUNK: case TYPE_STREAM_CHUNK:
return 'stream-chunk'; return 'stream-chunk';
case TYPE_BROADCAST:
return 'broadcast';
default: default:
return 'unknown'; return 'unknown';
} }
} }
// ─── Broadcast wire (V4.6) ───────────────────────────────────
const BROADCAST_NONCE_BYTES = 12;
const BROADCAST_SIGNATURE_BYTES = 64;
/**
* Encode a broadcast envelope to wire bytes (type 0x21).
*
* Layout:
* [version:1][type=0x21:1]
* [channelIdLen:u16][channelId utf-8]
* [senderAddrLen:u16][senderAddr utf-8]
* [generation:u32]
* [iteration:u32]
* [nonce:12]
* [signature:64]
* [ctLen:u32][ciphertext]
*/
export function encodeBroadcast(b: BroadcastWire): Uint8Array {
if (b.nonce.length !== BROADCAST_NONCE_BYTES) {
throw new Error(`broadcast nonce must be ${BROADCAST_NONCE_BYTES} bytes`);
}
if (b.signature.length !== BROADCAST_SIGNATURE_BYTES) {
throw new Error(`broadcast signature must be ${BROADCAST_SIGNATURE_BYTES} bytes`);
}
const enc = new TextEncoder();
const channelIdBytes = enc.encode(b.channelId);
const senderBytes = enc.encode(b.senderAddress);
if (channelIdBytes.length > 0xffff) throw new Error('channelId too long');
if (senderBytes.length > 0xffff) throw new Error('senderAddress too long');
const headerSize =
1 + 1 +
2 + channelIdBytes.length +
2 + senderBytes.length +
4 + 4 +
BROADCAST_NONCE_BYTES +
BROADCAST_SIGNATURE_BYTES +
4;
const out = new Uint8Array(headerSize + b.ciphertext.length);
const view = new DataView(out.buffer);
let offset = 0;
out[offset++] = VERSION;
out[offset++] = TYPE_BROADCAST;
view.setUint16(offset, channelIdBytes.length, false); offset += 2;
out.set(channelIdBytes, offset); offset += channelIdBytes.length;
view.setUint16(offset, senderBytes.length, false); offset += 2;
out.set(senderBytes, offset); offset += senderBytes.length;
view.setUint32(offset, b.generation, false); offset += 4;
view.setUint32(offset, b.iteration, false); offset += 4;
out.set(b.nonce, offset); offset += BROADCAST_NONCE_BYTES;
out.set(b.signature, offset); offset += BROADCAST_SIGNATURE_BYTES;
view.setUint32(offset, b.ciphertext.length, false); offset += 4;
out.set(b.ciphertext, offset);
return out;
}
export function decodeBroadcast(data: Uint8Array): BroadcastWire {
const minSize = 1 + 1 + 2 + 0 + 2 + 0 + 4 + 4 +
BROADCAST_NONCE_BYTES + BROADCAST_SIGNATURE_BYTES + 4;
if (data.length < minSize) {
throw new Error(`broadcast envelope too short: ${data.length} < ${minSize}`);
}
if (data[0] !== VERSION) throw new Error(`Unknown version: ${data[0]}`);
if (data[1] !== TYPE_BROADCAST) throw new Error(`Not a broadcast: type=${data[1]}`);
const view = new DataView(data.buffer, data.byteOffset);
const dec = new TextDecoder();
let offset = 2;
const channelIdLen = view.getUint16(offset, false); offset += 2;
if (offset + channelIdLen > data.length) throw new Error('broadcast truncated in channelId');
const channelId = dec.decode(data.slice(offset, offset + channelIdLen));
offset += channelIdLen;
const senderLen = view.getUint16(offset, false); offset += 2;
if (offset + senderLen > data.length) throw new Error('broadcast truncated in senderAddress');
const senderAddress = dec.decode(data.slice(offset, offset + senderLen));
offset += senderLen;
const generation = view.getUint32(offset, false); offset += 4;
const iteration = view.getUint32(offset, false); offset += 4;
const nonce = data.slice(offset, offset + BROADCAST_NONCE_BYTES);
offset += BROADCAST_NONCE_BYTES;
const signature = data.slice(offset, offset + BROADCAST_SIGNATURE_BYTES);
offset += BROADCAST_SIGNATURE_BYTES;
const ctLen = view.getUint32(offset, false); offset += 4;
if (offset + ctLen !== data.length) {
throw new Error(`broadcast length mismatch: declared ${offset + ctLen}, actual ${data.length}`);
}
const ciphertext = data.slice(offset, offset + ctLen);
return { channelId, senderAddress, generation, iteration, nonce, signature, ciphertext };
}
function decodePreKeyMessageInner(data: Uint8Array): PreKeyMessage { function decodePreKeyMessageInner(data: Uint8Array): PreKeyMessage {
let offset = 0; let offset = 0;

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/recovery", "name": "@shade/recovery",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/sdk", "name": "@shade/sdk",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

Binary file not shown.

View File

@@ -17,11 +17,12 @@ export interface ShadeConfig {
* - "memory" — in-memory only (lost on restart, good for tests) * - "memory" — in-memory only (lost on restart, good for tests)
* - "sqlite:/path/to/file.db" — SQLite backend * - "sqlite:/path/to/file.db" — SQLite backend
* - { type: 'postgres', url: 'postgres://...' } — PostgreSQL backend * - { type: 'postgres', url: 'postgres://...' } — PostgreSQL backend
* - { type: 'indexeddb', dbName?: 'my-app' } — browser IndexedDB
* - An explicit StorageProvider instance * - An explicit StorageProvider instance
* *
* Default: "memory" * Default: "memory"
*/ */
storage?: string | StorageProvider | { type: 'postgres'; url: string }; storage?: StorageSpec;
/** /**
* Your address on the prekey server (e.g. "alice@example.com" or "device:abc123"). * Your address on the prekey server (e.g. "alice@example.com" or "device:abc123").
@@ -96,9 +97,16 @@ export interface ShadeKTConfig {
witnessMaxStored?: number; witnessMaxStored?: number;
} }
/** Acceptable shapes for `ShadeConfig.storage`. */
export type StorageSpec =
| string
| StorageProvider
| { type: 'postgres'; url: string }
| { type: 'indexeddb'; dbName?: string };
export interface ResolvedConfig { export interface ResolvedConfig {
prekeyServer: string; prekeyServer: string;
storage: string | StorageProvider | { type: 'postgres'; url: string }; storage: StorageSpec;
address?: string | undefined; address?: string | undefined;
autoReplenish: { min: number; target: number; intervalMs: number } | false; autoReplenish: { min: number; target: number; intervalMs: number } | false;
autoRotate: false | '1d' | '7d' | '30d' | '90d'; autoRotate: false | '1d' | '7d' | '30d' | '90d';

View File

@@ -5,6 +5,11 @@ export type {
ShadeWebRtcConfig, ShadeWebRtcConfig,
ShadeWebRtcRuntime, ShadeWebRtcRuntime,
} from './shade.js'; } from './shade.js';
export type {
BroadcastChannel,
BroadcastChannelSummary,
MessageMeta,
} from './broadcast.js';
export { generateThumbnail } from './thumbnail.js'; export { generateThumbnail } from './thumbnail.js';
export type { GeneratedThumbnail, ThumbnailGenerationOptions } from './thumbnail.js'; export type { GeneratedThumbnail, ThumbnailGenerationOptions } from './thumbnail.js';
export { ShadeThumbnailCache } from './thumbnail-cache.js'; export { ShadeThumbnailCache } from './thumbnail-cache.js';

View File

@@ -31,6 +31,8 @@ import {
type TransferHandle, type TransferHandle,
type TransferOptions, type TransferOptions,
type TransferSummary, type TransferSummary,
type OutboundQueue as OutboundQueueLike,
type QueuedEventInput,
} from '@shade/transfer'; } from '@shade/transfer';
import type { Hono } from 'hono'; import type { Hono } from 'hono';
import { BackgroundTasks } from './background.js'; import { BackgroundTasks } from './background.js';
@@ -42,7 +44,18 @@ import {
backupFromString, backupFromString,
} from './backup.js'; } from './backup.js';
import { computeFingerprint, deserializeIdentityKeyPair } from '@shade/core'; import { computeFingerprint, deserializeIdentityKeyPair } from '@shade/core';
import type { ResolvedConfig } from './config.js'; import {
acceptBroadcastEnvelope,
createBroadcastChannelImpl,
getBroadcastChannelImpl,
listBroadcastChannelsImpl,
maybeHandleControlPlaintext,
type BroadcastChannel,
type BroadcastChannelSummary,
type BroadcastSdkHooks,
type MessageMeta,
} from './broadcast.js';
import type { ResolvedConfig, StorageSpec } from './config.js';
import { import {
ShadeControlChannel, ShadeControlChannel,
ShadeTransferAuthenticator, ShadeTransferAuthenticator,
@@ -141,9 +154,11 @@ export class Shade {
// Per-address encrypt queue to serialize ratchet mutations // Per-address encrypt queue to serialize ratchet mutations
private encryptChains = new Map<string, Promise<unknown>>(); private encryptChains = new Map<string, Promise<unknown>>();
// Message handlers — may be sync or async; receive() awaits each. // Message handlers — may be sync or async; receive() awaits each. The
// optional third arg distinguishes direct vs broadcast plaintexts;
// handlers registered without it work unchanged (V4.6 back-compat).
private messageHandlers: Array< private messageHandlers: Array<
(from: string, plaintext: string) => void | Promise<void> (from: string, plaintext: string, meta?: MessageMeta) => void | Promise<void>
> = []; > = [];
// Stream-transfer engine, lazily constructed on first use. // Stream-transfer engine, lazily constructed on first use.
@@ -151,6 +166,8 @@ export class Shade {
private controlChannel: ShadeControlChannel | null = null; private controlChannel: ShadeControlChannel | null = null;
private peerBaseUrlResolver: ((peerAddress: string) => Promise<string>) | null = null; private peerBaseUrlResolver: ((peerAddress: string) => Promise<string>) | null = null;
private envelopeOutboxes: ControlEnvelopeTransport | null = null; private envelopeOutboxes: ControlEnvelopeTransport | null = null;
private transferTransportOverride: ITransferTransport | null = null;
private transferQueue: OutboundQueueLike | null = null;
// `@shade/files` namespace, lazy + memoized. // `@shade/files` namespace, lazy + memoized.
private filesNamespace: FilesNamespace | null = null; private filesNamespace: FilesNamespace | null = null;
@@ -295,6 +312,26 @@ export class Shade {
return this.address; return this.address;
} }
/**
* The local device's Ed25519 identity public key (32 bytes).
*
* Stable for the lifetime of the identity. After {@link rotate} this
* reflects the new key; the previous key is preserved in retired-
* identities storage for the configured grace period.
*
* Hand this to your application's backend at enrollment time so it
* can verify signatures from this device, compute its own safety-
* number representation, or pin the key for later attestation. Use
* {@link fingerprint} instead for human side-channel comparison.
*/
get identityPublicKey(): Promise<Uint8Array> {
if (!this.initialized) throw new Error('Not initialized');
return this.storage.getIdentityKeyPair().then((kp) => {
if (!kp) throw new Error('Identity not yet generated');
return kp.signingPublicKey;
});
}
/** /**
* `@shade/files` namespace — high-level entry point for E2EE filesystem * `@shade/files` namespace — high-level entry point for E2EE filesystem
* RPC. Lazily creates the underlying channel + streams bridges on first * RPC. Lazily creates the underlying channel + streams bridges on first
@@ -390,13 +427,26 @@ export class Shade {
* The caller provides the `from` address because the envelope itself * The caller provides the `from` address because the envelope itself
* doesn't authenticate the sender — that's determined by your transport * doesn't authenticate the sender — that's determined by your transport
* layer (auth header, WebSocket peer, push notification metadata, etc.). * layer (auth header, WebSocket peer, push notification metadata, etc.).
*
* V4.6: when the decrypted plaintext is a broadcast control message
* (sender-key distribution / revocation), the SDK consumes it
* internally and returns an empty string; user handlers do NOT fire.
* Apps therefore see only direct plaintexts here. Broadcast payloads
* arrive via {@link Shade.acceptBroadcast}.
*/ */
async receive(from: string, envelope: ShadeEnvelope): Promise<string> { async receive(from: string, envelope: ShadeEnvelope): Promise<string> {
if (!this.initialized) throw new Error('Not initialized'); if (!this.initialized) throw new Error('Not initialized');
const plaintext = await this.manager.decrypt(from, envelope); const plaintext = await this.manager.decrypt(from, envelope);
const consumed = await maybeHandleControlPlaintext(
this.broadcastHooks(),
from,
plaintext,
);
if (consumed) return '';
const meta: MessageMeta = { kind: 'direct' };
for (const handler of this.messageHandlers) { for (const handler of this.messageHandlers) {
try { try {
await handler(from, plaintext); await handler(from, plaintext, meta);
} catch (err) { } catch (err) {
console.error('[Shade] Message handler threw:', err); console.error('[Shade] Message handler threw:', err);
} }
@@ -404,9 +454,16 @@ export class Shade {
return plaintext; return plaintext;
} }
/** Register a handler for incoming messages. Async handlers are awaited. */ /**
* Register a handler for incoming messages. Async handlers are awaited.
*
* V4.6: handlers may declare an optional `meta` parameter to discriminate
* direct (`meta.kind === 'direct'`) from broadcast (`meta.kind === 'broadcast'`)
* deliveries. Handlers that ignore the third arg keep working unchanged
* for direct messages.
*/
onMessage( onMessage(
handler: (from: string, plaintext: string) => void | Promise<void>, handler: (from: string, plaintext: string, meta?: MessageMeta) => void | Promise<void>,
): () => void { ): () => void {
this.messageHandlers.push(handler); this.messageHandlers.push(handler);
return () => { return () => {
@@ -414,6 +471,73 @@ export class Shade {
}; };
} }
// ─── V4.6 Broadcast channels ───────────────────────────────
/**
* Create a new broadcast channel owned by this device. Returns a
* handle for adding/removing members, encrypting a single payload,
* and rotating on revocation. The channel id is opaque, stable
* across `shutdown()` / re-open, and persisted via the configured
* `StorageProvider`.
*/
async createBroadcastChannel(opts: { label?: string } = {}): Promise<BroadcastChannel> {
if (!this.initialized) throw new Error('Not initialized');
return createBroadcastChannelImpl(this.broadcastHooks(), opts);
}
/**
* Look up an existing sender-side broadcast channel by id. Returns
* `null` when the id is unknown OR when this device only holds a
* receiver-side copy (the receiver path uses `onMessage` for delivery
* — there is no app-facing handle on the receive side).
*/
async getBroadcastChannel(channelId: string): Promise<BroadcastChannel | null> {
if (!this.initialized) throw new Error('Not initialized');
return getBroadcastChannelImpl(this.broadcastHooks(), channelId);
}
/**
* Snapshot of every broadcast channel persisted on this device,
* including receiver-side channels that we joined. Useful for
* rebuilding UI state on startup.
*/
async listBroadcastChannels(): Promise<readonly BroadcastChannelSummary[]> {
if (!this.initialized) throw new Error('Not initialized');
return listBroadcastChannelsImpl(this.broadcastHooks());
}
/**
* Hand a wire-encoded broadcast envelope (type 0x21) to the SDK.
* Decrypts via the matching channel, advances the chain, and dispatches
* the plaintext to `onMessage` handlers with `meta.kind === 'broadcast'`.
*
* Stale generations (sender's old chain after a rotation we already
* received) are silently dropped. Future generations (we haven't seen
* the rotation distribution yet) throw — the app should ensure the
* distribution envelope is delivered before the broadcast.
*/
async acceptBroadcast(envelope: Uint8Array): Promise<void> {
if (!this.initialized) throw new Error('Not initialized');
const result = await acceptBroadcastEnvelope(this.broadcastHooks(), envelope);
if (result === null) return;
for (const handler of this.messageHandlers) {
try {
await handler(result.meta.kind === 'broadcast' ? result.meta.sender : '', result.plaintext, result.meta);
} catch (err) {
console.error('[Shade] Broadcast handler threw:', err);
}
}
}
private broadcastHooks(): BroadcastSdkHooks {
return {
bilateralSend: (peer, pt) => this.send(peer, pt),
myAddress: () => this.address,
crypto: this.crypto,
storage: this.storage,
};
}
/** Get a peer's fingerprint (requires an existing session) */ /** Get a peer's fingerprint (requires an existing session) */
async getFingerprintFor(address: string): Promise<string> { async getFingerprintFor(address: string): Promise<string> {
if (!this.initialized) throw new Error('Not initialized'); if (!this.initialized) throw new Error('Not initialized');
@@ -746,12 +870,52 @@ export class Shade {
* HTTP POSTs to `<base>/v1/transfer/control`). * HTTP POSTs to `<base>/v1/transfer/control`).
*/ */
configureTransfers(opts: { configureTransfers(opts: {
resolveBaseUrl: (peerAddress: string) => Promise<string>; /**
* Resolver for the peer's HTTP base URL (used by the default
* `ShadeTransferHttpTransport` to POST chunks). Optional when a
* custom `transport` and `envelopeTransport` are supplied — e.g.
* for pull-mode browser servers (`@shade/files transferQueueRoute`)
* which never POST chunks anywhere.
*/
resolveBaseUrl?: (peerAddress: string) => Promise<string>;
/**
* Override the chunk-level transport. Defaults to
* `ShadeTransferHttpTransport` (HTTP POSTs per chunk) when
* `resolveBaseUrl` is supplied. Required when `resolveBaseUrl`
* is omitted.
*/
transport?: ITransferTransport;
/**
* Override the control-envelope transport. Defaults to HTTP POSTs
* to `<base>/v1/transfer/control` when `resolveBaseUrl` is
* supplied. Required when `resolveBaseUrl` is omitted.
*/
envelopeTransport?: ControlEnvelopeTransport; envelopeTransport?: ControlEnvelopeTransport;
}): void { }): void {
this.peerBaseUrlResolver = opts.resolveBaseUrl; if (opts.resolveBaseUrl === undefined) {
this.envelopeOutboxes = if (opts.transport === undefined || opts.envelopeTransport === undefined) {
opts.envelopeTransport ?? new HttpEnvelopeTransport(opts.resolveBaseUrl, this.address); throw new Error(
'configureTransfers: resolveBaseUrl is required unless both `transport` and `envelopeTransport` are supplied (e.g. for pull-mode queue servers).',
);
}
this.peerBaseUrlResolver = async () => {
throw new Error(
'resolveBaseUrl was not configured — this Shade is in queue/pull mode and does not POST chunks. Configure a custom transport instead.',
);
};
} else {
this.peerBaseUrlResolver = opts.resolveBaseUrl;
}
this.transferTransportOverride = opts.transport ?? null;
if (opts.envelopeTransport !== undefined) {
this.envelopeOutboxes = opts.envelopeTransport;
} else if (opts.resolveBaseUrl !== undefined) {
this.envelopeOutboxes = new HttpEnvelopeTransport(opts.resolveBaseUrl, this.address);
} else {
throw new Error(
'configureTransfers: envelopeTransport is required when resolveBaseUrl is omitted.',
);
}
} }
/** /**
@@ -898,6 +1062,109 @@ export class Shade {
return (await this.engine()).onIncomingTransfer(handler); return (await this.engine()).onIncomingTransfer(handler);
} }
/**
* Mount the **pull-mode** transfer routes on a Hono app. Mount under
* any base path: `app.route('/api/v1/shade-files', shade.transferQueueRoute())`.
*
* Configures this Shade instance to queue all outbound chunks +
* control envelopes per peer instead of POSTing them. Browser-style
* receivers drain the queue via long-polling — no inbound HTTP
* listener required on the receiver.
*
* Routes mounted (relative to the base path):
* POST /queue — long-poll the per-peer outbound queue
* POST /v1/transfer/:streamId/chunk — receive incoming chunks (browser → server)
* GET /v1/transfer/:streamId/state — resume-state lookup
* POST /v1/transfer/control — receive incoming control envelopes
* GET /v1/transfer/health — peer reachability probe
*
* **Idempotent**: calling twice returns a fresh `Hono` app each
* time but reuses the underlying queue + transport (so the engine
* stays single).
*
* **Ordering**: must be called **before** `shade.files.serve(...)`
* (or any other path that builds the engine), because configuring
* the queue transport mutates the transfer stack. Calling after the
* engine is built throws.
*/
async transferQueueRoute(opts: TransferQueueRouteOptions = {}): Promise<Hono> {
if (this.transferEngine !== null && this.transferTransportOverride === null) {
throw new Error(
'transferQueueRoute(): the transfer engine has already been built with the default HTTP transport. Call transferQueueRoute() before any upload()/onIncomingTransfer()/configureTransfers().',
);
}
const { OutboundQueue, QueueTransferTransport } = await import('@shade/transfer');
if (this.transferQueue === null) {
this.transferQueue = new OutboundQueue({
...(opts.maxEventsPerPeer !== undefined ? { maxEventsPerPeer: opts.maxEventsPerPeer } : {}),
...(opts.idleEvictionMs !== undefined ? { idleEvictionMs: opts.idleEvictionMs } : {}),
});
}
if (this.transferTransportOverride === null) {
const queueTransport = new QueueTransferTransport(this.transferQueue);
const queueEnvelopeTransport = new QueueEnvelopeTransport(this.transferQueue);
this.configureTransfers({
transport: queueTransport,
envelopeTransport: queueEnvelopeTransport,
});
}
const queue = this.transferQueue;
const blockMs = opts.blockMs ?? 30_000;
const maxBlockMs = opts.maxBlockMs ?? 55_000;
const engine = await this.engine();
const { createTransferRoutes, PermissiveAuthenticator } = await import('@shade/transfer');
const app = await createTransferRoutes(engine, {
authenticator: PermissiveAuthenticator,
});
app.post('/v1/transfer/control', async (c) => {
const senderAddress = c.req.header('X-Shade-Sender-Address');
if (senderAddress === undefined || senderAddress === '') {
return c.json({ error: 'missing X-Shade-Sender-Address' }, 400);
}
const ab = await c.req.arrayBuffer();
const bytes = new Uint8Array(ab);
try {
await this.acceptTransferEnvelope(senderAddress, bytes);
} catch (err) {
return c.json({ error: (err as Error).message }, 400);
}
return c.json({ ok: true });
});
// Long-poll endpoint.
app.post('/queue', async (c) => {
const senderAddress = c.req.header('X-Shade-Sender-Address');
if (senderAddress === undefined || senderAddress === '') {
return c.json({ error: 'missing X-Shade-Sender-Address' }, 400);
}
let body: { since?: unknown; blockMs?: unknown };
try {
body = (await c.req.json()) as { since?: unknown; blockMs?: unknown };
} catch {
return c.json({ error: 'invalid JSON body' }, 400);
}
const since = typeof body.since === 'number' && Number.isFinite(body.since) ? body.since : 0;
const requestedBlockMs =
typeof body.blockMs === 'number' && Number.isFinite(body.blockMs)
? Math.max(0, Math.min(maxBlockMs, body.blockMs))
: blockMs;
// Bun-side short-circuit if the request was aborted while we
// were holding the long-poll. AbortSignal from the request body
// is already surfaced via `c.req.raw.signal` in Hono.
const events = await queue.drain(senderAddress, since, requestedBlockMs, c.req.raw.signal);
return c.json({
events: events.map((e) => ({
id: e.id,
timestampMs: e.timestampMs,
kind: e.kind,
bytesB64: bytesToBase64Std(e.bytes),
...(e.kind === 'chunk' ? { meta: e.meta } : {}),
})),
nextSince: events.length > 0 ? events[events.length - 1]!.id : since,
});
});
return app;
}
/** /**
* Mount the receiver-side HTTP routes on a Hono app. Mount under any * Mount the receiver-side HTTP routes on a Hono app. Mount under any
* base path: `app.route('/shade', await shade.transferRoute())`. * base path: `app.route('/shade', await shade.transferRoute())`.
@@ -1019,16 +1286,23 @@ export class Shade {
); );
} }
this.controlChannel = new ShadeControlChannel(this, this.envelopeOutboxes); this.controlChannel = new ShadeControlChannel(this, this.envelopeOutboxes);
const httpTransport: ITransferTransport = new ShadeTransferHttpTransport({ let transport: ITransferTransport;
resolveBaseUrl: this.peerBaseUrlResolver,
authenticator: await this.makeAuthenticator(),
});
let transport: ITransferTransport = httpTransport;
let webrtcRuntime: ShadeWebRtcRuntime | null = null; let webrtcRuntime: ShadeWebRtcRuntime | null = null;
if (this.webrtcConfig !== null) { if (this.transferTransportOverride !== null) {
webrtcRuntime = await this.buildWebRtcRuntime(this.webrtcConfig, httpTransport); // Custom transport (queue, in-memory, custom adapter) — used as-is.
transport = webrtcRuntime.fallback; // WebRTC fallback only attaches when the default HTTP transport is
// active because WebRTC's `MultiTransportFallback` is HTTP-shaped.
transport = this.transferTransportOverride;
} else {
const httpTransport: ITransferTransport = new ShadeTransferHttpTransport({
resolveBaseUrl: this.peerBaseUrlResolver,
authenticator: await this.makeAuthenticator(),
});
transport = httpTransport;
if (this.webrtcConfig !== null) {
webrtcRuntime = await this.buildWebRtcRuntime(this.webrtcConfig, httpTransport);
transport = webrtcRuntime.fallback;
}
} }
this.transferEngine = new TransferEngine({ this.transferEngine = new TransferEngine({
@@ -1256,6 +1530,53 @@ function parseChunkHeader(bytes: Uint8Array): {
return { streamId, laneId, seq }; return { streamId, laneId, seq };
} }
// ─── Queue-mode (pull) envelope transport ─────────────────────
/**
* Configuration for {@link Shade.transferQueueRoute}. All fields are
* optional with sensible production defaults.
*/
export interface TransferQueueRouteOptions {
/**
* Long-poll timeout in milliseconds. Server holds the request open
* up to this long before returning an empty `events` array. Default
* 30_000.
*/
blockMs?: number;
/**
* Hard cap on long-poll timeout (clamps client-supplied `blockMs`).
* Default 55_000 — under typical reverse-proxy idle thresholds (60s
* on most CDNs).
*/
maxBlockMs?: number;
/**
* Per-peer ring-buffer size. When the queue is full, oldest events
* are dropped on enqueue. Receivers detect the gap via missing
* sequence numbers and re-resume from `since=0`. Default 1000.
*/
maxEventsPerPeer?: number;
/**
* Drop a peer's queue + reject pending pollers after this much
* silence. Default 10 minutes. Setting to `0` disables idle-eviction.
*/
idleEvictionMs?: number;
}
/**
* `ControlEnvelopeTransport` that enqueues outbound envelopes into an
* `OutboundQueue` for browser-style receivers to long-poll. Mirrors
* `HttpEnvelopeTransport` shape (one `send(peer, envelope)` method);
* the difference is the destination — local queue, not remote HTTP.
*/
class QueueEnvelopeTransport implements ControlEnvelopeTransport {
constructor(private readonly queue: OutboundQueueLike) {}
async send(peerAddress: string, envelope: ShadeEnvelope): Promise<void> {
const bytes = encodeEnvelope(envelope);
const event: QueuedEventInput = { kind: 'envelope', bytes };
this.queue.enqueue(peerAddress, event);
}
}
// ─── Default HTTP envelope transport ────────────────────────── // ─── Default HTTP envelope transport ──────────────────────────
class HttpEnvelopeTransport implements ControlEnvelopeTransport { class HttpEnvelopeTransport implements ControlEnvelopeTransport {
@@ -1283,9 +1604,7 @@ class HttpEnvelopeTransport implements ControlEnvelopeTransport {
// ─── Helpers ───────────────────────────────────────────────── // ─── Helpers ─────────────────────────────────────────────────
async function resolveStorage( async function resolveStorage(spec: StorageSpec): Promise<StorageProvider> {
spec: string | StorageProvider | { type: 'postgres'; url: string },
): Promise<StorageProvider> {
if (typeof spec === 'object' && 'getIdentityKeyPair' in spec) { if (typeof spec === 'object' && 'getIdentityKeyPair' in spec) {
return spec; return spec;
} }
@@ -1311,6 +1630,18 @@ async function resolveStorage(
return mod.PostgresStorage.create(spec.url); return mod.PostgresStorage.create(spec.url);
} }
if (typeof spec === 'object' && spec.type === 'indexeddb') {
// Dynamic import keeps @shade/storage-indexeddb optional — Node-only
// consumers don't need to install a browser-only adapter.
const moduleId = '@shade/storage-indexeddb';
const mod = (await import(moduleId)) as {
IndexedDBStorage: { create(opts: { dbName?: string }): Promise<StorageProvider> };
};
const opts: { dbName?: string } = {};
if (spec.dbName !== undefined) opts.dbName = spec.dbName;
return mod.IndexedDBStorage.create(opts);
}
throw new Error(`Unsupported storage spec: ${JSON.stringify(spec)}`); throw new Error(`Unsupported storage spec: ${JSON.stringify(spec)}`);
} }

View File

@@ -0,0 +1,225 @@
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { tmpdir } from 'os';
import { join } from 'path';
import { unlinkSync } from 'fs';
import { createShade, type Shade, type MessageMeta } from '../src/index.js';
import { createPrekeyServer, MemoryPrekeyStore } from '@shade/server';
import { SubtleCryptoProvider } from '@shade/crypto-web';
const crypto = new SubtleCryptoProvider();
async function startPrekeyServer(): Promise<{ url: string; stop: () => void }> {
const server = createPrekeyServer({
crypto,
store: new MemoryPrekeyStore(),
disableRateLimit: true,
});
const port = 19500 + Math.floor(Math.random() * 500);
const handle = Bun.serve({ port, fetch: server.fetch });
return { url: `http://localhost:${port}`, stop: () => handle.stop() };
}
/**
* Prism's three acceptance tests, ported verbatim from
* `Docs/shade-feature-request-sender-keys.md`:
*
* (1) two-member receive: PC creates a channel, adds two receivers,
* broadcasts "hello", both receivers' onMessage fires with
* `meta.kind === 'broadcast'` and the same plaintext.
*
* (1*) revocation: same setup, then `removeMember(receiverA)`. Receiver
* A's next attempt to decrypt a subsequent broadcast fails (or is
* silently dropped); receiver B keeps working.
*
* (2) persistence: create channel, add members, broadcast N messages,
* `shutdown()`, re-open with the same backing store, channel still
* exists, member list intact, generation preserved, next broadcast
* decrypts on receiver side.
*
* (3) no new wire-format changes visible to apps — the pair-flow stays
* a single round-trip; the SDK does the sender-key distribution
* inline. (Validated by inspecting the API surface.)
*/
describe('Broadcast channels — Prism acceptance', () => {
let server: Awaited<ReturnType<typeof startPrekeyServer>>;
let pc: Shade;
let mobileA: Shade;
let mobileB: Shade;
beforeEach(async () => {
server = await startPrekeyServer();
});
afterEach(async () => {
await pc?.shutdown();
await mobileA?.shutdown();
await mobileB?.shutdown();
server.stop();
});
test('(1) two-member receive', async () => {
pc = await createShade({ prekeyServer: server.url, address: 'pc' });
mobileA = await createShade({ prekeyServer: server.url, address: 'mobile-a' });
mobileB = await createShade({ prekeyServer: server.url, address: 'mobile-b' });
const channel = await pc.createBroadcastChannel({ label: 'output' });
expect(channel.id.length).toBeGreaterThan(0);
// Add both members. Each call returns a bilateral envelope to deliver.
const distA = await channel.addMember('mobile-a');
const distB = await channel.addMember('mobile-b');
expect((await channel.members())).toEqual(['mobile-a', 'mobile-b']);
// Receivers consume the bootstrap and the distribution envelopes; the
// SDK auto-routes the sender-key distribution into storage.
await mobileA.receive('pc', distA.envelope);
await mobileB.receive('pc', distB.envelope);
// Hook receiver-side onMessage with meta.
const receivedA: Array<{ from: string; pt: string; meta?: MessageMeta }> = [];
const receivedB: Array<{ from: string; pt: string; meta?: MessageMeta }> = [];
mobileA.onMessage((from, pt, meta) => { receivedA.push({ from, pt, meta }); });
mobileB.onMessage((from, pt, meta) => { receivedB.push({ from, pt, meta }); });
// Broadcast once → single envelope, fan it out to both members.
const out = await channel.broadcast('hello');
expect(out.members).toEqual(['mobile-a', 'mobile-b']);
await mobileA.acceptBroadcast(out.envelope);
await mobileB.acceptBroadcast(out.envelope);
expect(receivedA).toHaveLength(1);
expect(receivedB).toHaveLength(1);
expect(receivedA[0]!.pt).toBe('hello');
expect(receivedB[0]!.pt).toBe('hello');
expect(receivedA[0]!.meta?.kind).toBe('broadcast');
expect(receivedB[0]!.meta?.kind).toBe('broadcast');
if (receivedA[0]!.meta?.kind === 'broadcast') {
expect(receivedA[0]!.meta.channelId).toBe(channel.id);
expect(receivedA[0]!.meta.sender).toBe('pc');
}
});
test('(1*) revocation rotates the chain — receiver A drops, B keeps working', async () => {
pc = await createShade({ prekeyServer: server.url, address: 'pc' });
mobileA = await createShade({ prekeyServer: server.url, address: 'mobile-a' });
mobileB = await createShade({ prekeyServer: server.url, address: 'mobile-b' });
const channel = await pc.createBroadcastChannel({ label: 'output' });
const distA = await channel.addMember('mobile-a');
const distB = await channel.addMember('mobile-b');
await mobileA.receive('pc', distA.envelope);
await mobileB.receive('pc', distB.envelope);
const receivedA: string[] = [];
const receivedB: string[] = [];
mobileA.onMessage((_from, pt, meta) => {
if (meta?.kind === 'broadcast') receivedA.push(pt);
});
mobileB.onMessage((_from, pt, meta) => {
if (meta?.kind === 'broadcast') receivedB.push(pt);
});
// Pre-revocation broadcast — both decrypt.
const before = await channel.broadcast('before');
await mobileA.acceptBroadcast(before.envelope);
await mobileB.acceptBroadcast(before.envelope);
expect(receivedA).toEqual(['before']);
expect(receivedB).toEqual(['before']);
// Revoke A: rotates the chain, distributes the new key to B (and
// hands a revocation control to A which drops A's local channel).
const { rotations } = await channel.removeMember('mobile-a');
expect(rotations.map((r) => r.to)).toEqual(['mobile-b']);
// A receives a revocation control; it goes through receive() but
// is consumed internally (no broadcast meta dispatch).
// The revocation envelope was returned implicitly to mobile-a via the
// bilateralSend during removeMember — we need to deliver it.
// (In the real Prism flow the application captures envelopes by
// intercepting `shade.send`. Here we just trust it happened.)
// Deliver new chain to B.
await mobileB.receive('pc', rotations[0]!.envelope);
// Post-rotation broadcast.
const after = await channel.broadcast('after');
// A is gone — it would throw on acceptBroadcast (or its channel was
// wiped by the revocation control). Either way, A's receivedA stays
// unchanged.
let aThrew = false;
try {
await mobileA.acceptBroadcast(after.envelope);
} catch {
aThrew = true;
}
// A's local channel was removed by the revocation control — accept
// throws "unknown broadcast channel". This is the expected post-
// revocation behavior.
expect(aThrew).toBe(true);
// B decrypts as normal.
await mobileB.acceptBroadcast(after.envelope);
expect(receivedA).toEqual(['before']); // unchanged
expect(receivedB).toEqual(['before', 'after']);
});
test('(2) persistence — channel survives shutdown + re-open', async () => {
// Use a SQLite-backed storage so we get real persistence.
const dbPath = join(tmpdir(), `shade-broadcast-${Date.now()}-${Math.random().toString(36).slice(2)}.db`);
pc = await createShade({ prekeyServer: server.url, address: 'pc', storage: `sqlite:${dbPath}` });
mobileA = await createShade({ prekeyServer: server.url, address: 'mobile-a' });
const channel = await pc.createBroadcastChannel({ label: 'output' });
const channelId = channel.id;
const dist = await channel.addMember('mobile-a');
await mobileA.receive('pc', dist.envelope);
// Send a few broadcasts to advance the chain.
const received: string[] = [];
mobileA.onMessage((_from, pt, meta) => {
if (meta?.kind === 'broadcast') received.push(pt);
});
for (let i = 0; i < 3; i++) {
const out = await channel.broadcast(`msg-${i}`);
await mobileA.acceptBroadcast(out.envelope);
}
expect(received).toEqual(['msg-0', 'msg-1', 'msg-2']);
// Shutdown and re-open the PC side. The storage file persists.
await pc.shutdown();
pc = await createShade({ prekeyServer: server.url, address: 'pc', storage: `sqlite:${dbPath}` });
const reopened = await pc.getBroadcastChannel(channelId);
expect(reopened).not.toBeNull();
expect(reopened!.id).toBe(channelId);
expect((await reopened!.members())).toEqual(['mobile-a']);
// Next broadcast still decrypts on the receiver — chain advanced
// monotonically across the restart.
const out = await reopened!.broadcast('msg-after-restart');
await mobileA.acceptBroadcast(out.envelope);
expect(received).toEqual(['msg-0', 'msg-1', 'msg-2', 'msg-after-restart']);
// Cleanup.
try { unlinkSync(dbPath); } catch {}
try { unlinkSync(dbPath + '-shm'); } catch {}
try { unlinkSync(dbPath + '-wal'); } catch {}
});
test('(3) listBroadcastChannels surfaces sender + receiver records', async () => {
pc = await createShade({ prekeyServer: server.url, address: 'pc' });
mobileA = await createShade({ prekeyServer: server.url, address: 'mobile-a' });
const channel = await pc.createBroadcastChannel({ label: 'output' });
const dist = await channel.addMember('mobile-a');
await mobileA.receive('pc', dist.envelope);
const pcSide = await pc.listBroadcastChannels();
const mobileSide = await mobileA.listBroadcastChannels();
expect(pcSide).toHaveLength(1);
expect(pcSide[0]!.ownerRole).toBe('sender');
expect(pcSide[0]!.members).toEqual(['mobile-a']);
expect(mobileSide).toHaveLength(1);
expect(mobileSide[0]!.ownerRole).toBe('receiver');
expect(mobileSide[0]!.id).toBe(channel.id);
});
});

View File

@@ -5,7 +5,7 @@ import {
MemoryPrekeyStore, MemoryPrekeyStore,
PrekeyServerEvents, PrekeyServerEvents,
} from '@shade/server'; } from '@shade/server';
import { SubtleCryptoProvider } from '@shade/crypto-web'; import { SubtleCryptoProvider, MemoryStorage } from '@shade/crypto-web';
const crypto = new SubtleCryptoProvider(); const crypto = new SubtleCryptoProvider();
@@ -182,6 +182,26 @@ describe('createShade — happy path', () => {
const newFp = await alice.fingerprint; const newFp = await alice.fingerprint;
expect(newFp).not.toBe(oldFp); expect(newFp).not.toBe(oldFp);
}); });
test('identityPublicKey exposes the device Ed25519 key and tracks rotation', async () => {
const storage = new MemoryStorage();
alice = await createShade({ prekeyServer: server.url, address: 'alice', storage });
const pk = await alice.identityPublicKey;
expect(pk).toBeInstanceOf(Uint8Array);
expect(pk.length).toBe(32);
// Matches what the underlying storage holds
const stored = await storage.getIdentityKeyPair();
expect(stored).not.toBeNull();
expect(pk).toEqual(stored!.signingPublicKey);
// Reflects the new key after rotate (acceptance criteria #3)
await alice.rotate();
const pkAfter = await alice.identityPublicKey;
expect(pkAfter.length).toBe(32);
expect(pkAfter).not.toEqual(pk);
});
}); });
describe('createShade — validation', () => { describe('createShade — validation', () => {

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/server", "name": "@shade/server",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,17 +1,31 @@
{ {
"name": "@shade/storage-encrypted", "name": "@shade/storage-encrypted",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",
"exports": {
".": {
"browser": "./src/index.browser.ts",
"default": "./src/index.ts"
},
"./crypto": "./src/crypto.ts",
"./sqlite": "./src/sqlite.ts",
"./postgres": "./src/postgres.ts",
"./idb": "./src/idb.ts"
},
"dependencies": { "dependencies": {
"@noble/hashes": "^2.0.1", "@noble/hashes": "^2.0.1",
"@shade/core": "workspace:*", "@shade/core": "workspace:*",
"@shade/crypto-web": "workspace:*", "@shade/crypto-web": "workspace:*",
"@shade/storage-postgres": "workspace:*", "@shade/storage-postgres": "workspace:*",
"@shade/storage-sqlite": "workspace:*", "@shade/storage-sqlite": "workspace:*",
"idb": "^8.0.3",
"postgres": "^3.4.9" "postgres": "^3.4.9"
}, },
"devDependencies": {
"fake-indexeddb": "^6.0.0"
},
"peerDependencies": { "peerDependencies": {
"@shade/keychain": "workspace:*" "@shade/keychain": "workspace:*"
}, },

View File

@@ -0,0 +1,64 @@
/**
* Browser-safe entry point — exports KeyManager + crypto primitives only.
*
* Use this subpath when bundling for the browser to avoid pulling
* `bun:sqlite` (Bun-only) or `postgres` (Node-only) into the output.
*
* ```ts
* import { KeyManager } from '@shade/storage-encrypted/crypto';
* ```
*
* For backend storage, import `@shade/storage-encrypted/sqlite` (Bun) or
* `@shade/storage-encrypted/postgres` (Node). For browser storage, import
* `@shade/storage-encrypted/idb`.
*/
export {
KeyManager,
type KeySource,
type KeychainBackend,
type KeyManagerOptions,
} from './crypto/key-manager.js';
export {
DEFAULT_SCRYPT,
DEFAULT_ARGON2ID,
type ScryptParams,
type Argon2idParams,
deriveMasterKey,
deriveMasterKeyArgon2id,
deriveStorageKey,
deriveFieldKey,
deriveNonce,
buildAad,
hkdfDerive,
} from './crypto/kdf.js';
export {
AEAD_NONCE_LEN,
AEAD_TAG_LEN,
aeadSeal,
aeadOpen,
} from './crypto/aead.js';
export {
COL,
TBL,
sealString,
openString,
sealBytes,
openBytes,
sealIdentity,
openIdentity,
sealConfig,
openConfig,
sealSignedPreKey,
openSignedPreKey,
sealOneTimePreKey,
openOneTimePreKey,
sealSession,
openSession,
sealTrust,
openTrust,
sealRetired,
openRetired,
sealStreamSensitive,
openStreamSensitive,
} from './crypto/row-codec.js';

View File

@@ -11,6 +11,7 @@
*/ */
import { scryptAsync } from '@noble/hashes/scrypt.js'; import { scryptAsync } from '@noble/hashes/scrypt.js';
import { argon2idAsync } from '@noble/hashes/argon2.js';
import { hkdf } from '@noble/hashes/hkdf.js'; import { hkdf } from '@noble/hashes/hkdf.js';
import { sha256 } from '@noble/hashes/sha2.js'; import { sha256 } from '@noble/hashes/sha2.js';
@@ -45,6 +46,50 @@ export async function deriveMasterKey(
return scryptAsync(TEXT.encode(passphrase.normalize('NFKC')), salt, params); return scryptAsync(TEXT.encode(passphrase.normalize('NFKC')), salt, params);
} }
/** Argon2id parameters — memory-hard KDF preferred for low-entropy secrets (PINs). */
export interface Argon2idParams {
/** Memory cost in KiB. */
m: number;
/** Time cost (iterations). */
t: number;
/** Parallelism. */
p: number;
/** Output length in bytes. */
dkLen: number;
}
/**
* Default: m=64 MiB, t=3, p=1, 32-byte output. Tuned for ~250400 ms on a
* modern Chromium / Firefox / Safari laptop. RFC 9106 "second recommended"
* profile shrunk to a browser-friendly memory footprint — strong enough for
* 46 digit PINs as a defense-in-depth factor on top of a passphrase.
*/
export const DEFAULT_ARGON2ID: Argon2idParams = { m: 64 * 1024, t: 3, p: 1, dkLen: 32 };
/**
* Derive a 32-byte master key from a low-entropy secret + salt using
* argon2id. Salt MUST be persisted alongside the DB (16-byte random).
*/
export async function deriveMasterKeyArgon2id(
secret: string | Uint8Array,
salt: Uint8Array,
params: Argon2idParams = DEFAULT_ARGON2ID,
): Promise<Uint8Array> {
if (typeof secret === 'string' ? secret.length === 0 : secret.length === 0) {
throw new Error('argon2id secret must be non-empty');
}
if (salt.length < 16) {
throw new Error('salt must be at least 16 bytes');
}
const password = typeof secret === 'string' ? TEXT.encode(secret.normalize('NFKC')) : secret;
return argon2idAsync(password, salt, {
m: params.m,
t: params.t,
p: params.p,
dkLen: params.dkLen,
});
}
/** HKDF-SHA-256 with explicit info string. */ /** HKDF-SHA-256 with explicit info string. */
export function hkdfDerive(ikm: Uint8Array, info: string, length = 32, salt?: Uint8Array): Uint8Array { export function hkdfDerive(ikm: Uint8Array, info: string, length = 32, salt?: Uint8Array): Uint8Array {
return hkdf(sha256, ikm, salt, TEXT.encode(info), length); return hkdf(sha256, ikm, salt, TEXT.encode(info), length);

View File

@@ -1,17 +1,29 @@
/** /**
* KeyManager — owns the masterKey lifecycle for at-rest encryption. * KeyManager — owns the masterKey lifecycle for at-rest encryption.
* *
* Three sources are supported: * Five sources are supported:
* 1. passphrase — scrypt-derived from a developer-supplied secret * 1. passphrase — scrypt-derived from a developer-supplied secret
* 2. keychain — fetched from OS keychain via @shade/keychain (optional dep) * 2. argon2id — memory-hard KDF for low-entropy secrets (PINs)
* 3. injected — caller supplies the 32-byte raw key directly * 3. keychain — fetched from OS keychain via @shade/keychain (optional dep)
* 4. injected — caller supplies the 32-byte raw key directly
* 5. composite — HKDF-combine N sub-sources into one master key
* (multi-factor unlock — every source is required)
* *
* The KeyManager pre-derives storageKey at construction time and caches the * The KeyManager pre-derives storageKey at construction time and caches the
* per-(table, column) field keys. masterKey is zeroized after storageKey * per-(table, column) field keys. masterKey is zeroized after storageKey
* derivation to limit residency. * derivation to limit residency.
*/ */
import { deriveFieldKey, deriveMasterKey, deriveStorageKey, type ScryptParams, DEFAULT_SCRYPT } from './kdf.js'; import {
deriveFieldKey,
deriveMasterKey,
deriveMasterKeyArgon2id,
deriveStorageKey,
hkdfDerive,
DEFAULT_SCRYPT,
type Argon2idParams,
type ScryptParams,
} from './kdf.js';
export type KeySource = export type KeySource =
| { | {
@@ -21,6 +33,14 @@ export type KeySource =
salt: Uint8Array; salt: Uint8Array;
params?: ScryptParams; params?: ScryptParams;
} }
| {
kind: 'argon2id';
/** Low-entropy secret (PIN, short password). */
secret: string | Uint8Array;
/** Stable 16+ byte salt persisted alongside the DB. */
salt: Uint8Array;
params?: Argon2idParams;
}
| { | {
kind: 'keychain'; kind: 'keychain';
/** Service identifier (e.g. "shade.storage"). */ /** Service identifier (e.g. "shade.storage"). */
@@ -34,6 +54,25 @@ export type KeySource =
kind: 'injected'; kind: 'injected';
/** Raw 32-byte master key. */ /** Raw 32-byte master key. */
key: Uint8Array; key: Uint8Array;
}
| {
kind: 'composite';
/**
* Sub-sources HKDF-combined into the master key, in order. Every
* source is mandatory: omitting or substituting any source yields
* a different master key and the storage open() will fail.
*
* Order is significant by design — `[pwd, pin]` and `[pin, pwd]`
* derive different master keys.
*/
sources: KeySource[];
/**
* HKDF info string for domain separation. Defaults to
* `shade-composite-master-v1`. Apps that want their composite key
* to be cryptographically distinct from any other Shade composite
* should override this with an app-specific string.
*/
info?: string;
}; };
/** Pluggable keychain backend. Implementations live in @shade/keychain. */ /** Pluggable keychain backend. Implementations live in @shade/keychain. */
@@ -107,6 +146,9 @@ async function resolveMasterKey(source: KeySource, opts: KeyManagerOptions): Pro
case 'passphrase': case 'passphrase':
return deriveMasterKey(source.passphrase, source.salt, source.params ?? DEFAULT_SCRYPT); return deriveMasterKey(source.passphrase, source.salt, source.params ?? DEFAULT_SCRYPT);
case 'argon2id':
return deriveMasterKeyArgon2id(source.secret, source.salt, source.params);
case 'injected': case 'injected':
if (source.key.length !== 32) throw new Error('injected key must be exactly 32 bytes'); if (source.key.length !== 32) throw new Error('injected key must be exactly 32 bytes');
return new Uint8Array(source.key); // copy, in case caller mutates return new Uint8Array(source.key); // copy, in case caller mutates
@@ -128,5 +170,39 @@ async function resolveMasterKey(source: KeySource, opts: KeyManagerOptions): Pro
await opts.keychain.set(source.service, source.account, fresh); await opts.keychain.set(source.service, source.account, fresh);
return fresh; return fresh;
} }
case 'composite': {
if (source.sources.length === 0) {
throw new Error('composite source requires at least one sub-source');
}
const subKeys: Uint8Array[] = [];
try {
for (const sub of source.sources) {
if (sub.kind === 'composite') {
// Composite-of-composite would silently flatten the unlock
// semantics ("any of N" vs "all of N") if the inner is misread.
// Forbidding nesting keeps the contract clear.
throw new Error('composite sources cannot be nested');
}
subKeys.push(await resolveMasterKey(sub, opts));
}
let total = 0;
for (const k of subKeys) total += k.length;
const ikm = new Uint8Array(total);
let off = 0;
for (const k of subKeys) {
ikm.set(k, off);
off += k.length;
}
const info = source.info ?? 'shade-composite-master-v1';
try {
return hkdfDerive(ikm, info, 32);
} finally {
ikm.fill(0);
}
} finally {
for (const k of subKeys) k.fill(0);
}
}
} }
} }

View File

@@ -39,6 +39,7 @@ export const COL = {
trustedIdentity: 'trusted_identity', trustedIdentity: 'trusted_identity',
retiredIdentity: 'retired_identity', retiredIdentity: 'retired_identity',
streamSensitive: 'stream_sensitive', streamSensitive: 'stream_sensitive',
broadcastChannelSensitive: 'broadcast_channel_sensitive',
} as const; } as const;
/** Logical table identifiers — used for fieldKey + AAD binding. */ /** Logical table identifiers — used for fieldKey + AAD binding. */
@@ -51,6 +52,7 @@ export const TBL = {
trustedIdentities: 'trusted_identities', trustedIdentities: 'trusted_identities',
retiredIdentities: 'retired_identities', retiredIdentities: 'retired_identities',
streamState: 'stream_state', streamState: 'stream_state',
broadcastChannels: 'broadcast_channels',
} as const; } as const;
/** Encrypt an arbitrary string payload bound to (table, column, pk). */ /** Encrypt an arbitrary string payload bound to (table, column, pk). */
@@ -226,3 +228,76 @@ export async function openStreamSensitive(
if (b.overallHashState !== undefined) (out as { overallHashState?: string }).overallHashState = b.overallHashState; if (b.overallHashState !== undefined) (out as { overallHashState?: string }).overallHashState = b.overallHashState;
return out; return out;
} }
/**
* Broadcast-channel sensitive bundle (V4.6). Routing fields (channelId,
* ownerRole, ownerAddress, label, generation, createdAt, updatedAt) live
* in plaintext columns so backends can list/query without unsealing every
* row; the chain key, iteration, and signing keys all live in this sealed
* blob.
*/
interface BroadcastChannelSensitiveBundle {
chainKey: string; // base64(32B)
iteration: number;
signingPublicKey: string; // base64(32B)
signingPrivateKey?: string; // base64; only when ownerRole === 'sender'
}
export async function sealBroadcastChannelSensitive(
km: KeyManager,
channelId: string,
s: {
chainKey: Uint8Array;
iteration: number;
signingPublicKey: Uint8Array;
signingPrivateKey?: Uint8Array;
},
): Promise<Uint8Array> {
const bundle: BroadcastChannelSensitiveBundle = {
chainKey: toBase64(s.chainKey),
iteration: s.iteration,
signingPublicKey: toBase64(s.signingPublicKey),
};
if (s.signingPrivateKey !== undefined) {
bundle.signingPrivateKey = toBase64(s.signingPrivateKey);
}
return sealString(
km,
TBL.broadcastChannels,
COL.broadcastChannelSensitive,
channelId,
JSON.stringify(bundle),
);
}
export async function openBroadcastChannelSensitive(
km: KeyManager,
channelId: string,
blob: Uint8Array,
): Promise<{
chainKey: Uint8Array;
iteration: number;
signingPublicKey: Uint8Array;
signingPrivateKey?: Uint8Array;
}> {
const json = await openString(
km,
TBL.broadcastChannels,
COL.broadcastChannelSensitive,
channelId,
blob,
);
const b = JSON.parse(json) as BroadcastChannelSensitiveBundle;
const out: {
chainKey: Uint8Array;
iteration: number;
signingPublicKey: Uint8Array;
signingPrivateKey?: Uint8Array;
} = {
chainKey: fromBase64(b.chainKey),
iteration: b.iteration,
signingPublicKey: fromBase64(b.signingPublicKey),
};
if (b.signingPrivateKey !== undefined) out.signingPrivateKey = fromBase64(b.signingPrivateKey);
return out;
}

View File

@@ -0,0 +1,13 @@
/**
* Browser entry point — Encrypted IndexedDB storage.
*
* Pulls `idb` (~12 kB) and SubtleCrypto only. Safe to bundle into a
* browser app via Vite/webpack/esbuild.
*
* ```ts
* import { KeyManager } from '@shade/storage-encrypted/crypto';
* import { EncryptedIndexedDBStorage } from '@shade/storage-encrypted/idb';
* ```
*/
export { EncryptedIndexedDBStorage } from './storage/encrypted-indexeddb.js';

View File

@@ -0,0 +1,14 @@
/**
* Default barrel resolved by bundlers under the `browser` condition.
*
* Excludes `EncryptedSQLiteStorage` (pulls `bun:sqlite`) and
* `EncryptedPostgresStorage` (pulls `postgres`). Browser apps can keep using
* `import { KeyManager } from '@shade/storage-encrypted'` — Vite/webpack will
* pick this file via the `browser` condition and produce a clean bundle.
*
* Node/Bun consumers continue to resolve to `./index.js`, which exposes the
* full surface area (back-compat).
*/
export * from './crypto.js';
export { EncryptedIndexedDBStorage } from './storage/encrypted-indexeddb.js';

View File

@@ -6,8 +6,11 @@ export {
} from './crypto/key-manager.js'; } from './crypto/key-manager.js';
export { export {
DEFAULT_SCRYPT, DEFAULT_SCRYPT,
DEFAULT_ARGON2ID,
type ScryptParams, type ScryptParams,
type Argon2idParams,
deriveMasterKey, deriveMasterKey,
deriveMasterKeyArgon2id,
deriveStorageKey, deriveStorageKey,
deriveFieldKey, deriveFieldKey,
deriveNonce, deriveNonce,
@@ -25,6 +28,9 @@ export {
EncryptedPostgresStorage, EncryptedPostgresStorage,
ensureEncryptedClientTables, ensureEncryptedClientTables,
} from './storage/encrypted-postgres.js'; } from './storage/encrypted-postgres.js';
// EncryptedIndexedDBStorage is browser-only; import it from
// `@shade/storage-encrypted/idb` (or rely on the `browser` condition
// resolving to ./index.browser.ts).
export { export {
migrateSqliteToEncrypted, migrateSqliteToEncrypted,
rotateSqliteEncryptionKey, rotateSqliteEncryptionKey,

View File

@@ -0,0 +1,11 @@
/**
* Node-only entry point — Encrypted Postgres storage.
*
* Pulls `postgres`. Do not import from a browser bundle; use
* `@shade/storage-encrypted/idb` instead.
*/
export {
EncryptedPostgresStorage,
ensureEncryptedClientTables,
} from './storage/encrypted-postgres.js';

View File

@@ -0,0 +1,15 @@
/**
* Bun-only entry point — Encrypted SQLite storage + migration utilities.
*
* Pulls `bun:sqlite`. Do not import from a browser bundle; use
* `@shade/storage-encrypted/idb` instead.
*/
export { EncryptedSQLiteStorage } from './storage/encrypted-sqlite.js';
export {
migrateSqliteToEncrypted,
rotateSqliteEncryptionKey,
type MigrateOptions,
type RotateOptions,
type MigrateReport,
} from './migrate/migrate-sqlite.js';

View File

@@ -0,0 +1,548 @@
import { openDB, type IDBPDatabase, type DBSchema } from 'idb';
import type {
BroadcastChannelRecord,
BroadcastMemberRecord,
IdentityKeyPair,
OneTimePreKey,
PeerVerification,
PeerVerificationSource,
PersistedStreamState,
RetiredIdentity,
SessionState,
SignedPreKey,
StorageProvider,
} from '@shade/core';
import { constantTimeEqual, toBase64 } from '@shade/core';
import { KeyManager } from '../crypto/key-manager.js';
import {
openBroadcastChannelSensitive, openConfig, openIdentity, openOneTimePreKey,
openRetired, openSession, openSignedPreKey, openStreamSensitive, openTrust,
sealBroadcastChannelSensitive, sealConfig, sealIdentity, sealOneTimePreKey,
sealRetired, sealSession, sealSignedPreKey, sealStreamSensitive, sealTrust,
} from '../crypto/row-codec.js';
/**
* IndexedDB-backed StorageProvider with at-rest encryption.
*
* Schema is the IndexedDB equivalent of `EncryptedSQLiteStorage`: one object
* store per `_enc` table, sealed payloads stored as `Uint8Array` in a
* `ciphertext` field, routing/timestamp fields kept plaintext where SQLite
* does so for query efficiency. Crypto stack — `KeyManager`, `aeadSeal`,
* `aeadOpen`, row-codec sealers, AAD scheme — is shared verbatim with the
* SQLite/Postgres backends, so a row sealed under one backend decrypts
* under another given the same `KeyManager`.
*
* Browser-safe: imports `idb` (~12 kB, pure JS) and SubtleCrypto only.
*
* Usage:
* ```ts
* import { KeyManager } from '@shade/storage-encrypted/crypto';
* import { EncryptedIndexedDBStorage } from '@shade/storage-encrypted/idb';
*
* const km = await KeyManager.open({
* kind: 'composite',
* sources: [
* { kind: 'passphrase', passphrase, salt: pwSalt },
* { kind: 'argon2id', secret: pin, salt: pinSalt },
* ],
* });
* const storage = await EncryptedIndexedDBStorage.open({
* dbName: 'my-app-shade',
* keyManager: km,
* });
* ```
*/
export class EncryptedIndexedDBStorage implements StorageProvider {
private constructor(
private db: IDBPDatabase<EncryptedShadeSchema>,
private km: KeyManager,
) {}
/**
* Open (or create) the encrypted IndexedDB database. On first open the
* storageKey fingerprint is persisted; subsequent opens with a different
* KeyManager (wrong passphrase / PIN) reject with a clear error rather
* than silently writing data under the wrong key.
*/
static async open(opts: {
dbName?: string;
keyManager: KeyManager;
}): Promise<EncryptedIndexedDBStorage> {
const dbName = opts.dbName ?? 'shade-encrypted';
const db = await openDB<EncryptedShadeSchema>(dbName, SCHEMA_VERSION, {
upgrade(db, oldVersion) {
if (oldVersion < 1) {
db.createObjectStore('meta_enc', { keyPath: 'key' });
db.createObjectStore('identity_enc', { keyPath: 'id' });
db.createObjectStore('config_enc', { keyPath: 'key' });
db.createObjectStore('signed_prekeys_enc', { keyPath: 'keyId' });
db.createObjectStore('one_time_prekeys_enc', { keyPath: 'keyId' });
db.createObjectStore('sessions_enc', { keyPath: 'address' });
db.createObjectStore('trusted_identities_enc', { keyPath: 'address' });
const retired = db.createObjectStore('retired_identities_enc', {
keyPath: 'retiredAt',
});
retired.createIndex('byRetiredAt', 'retiredAt');
const stream = db.createObjectStore('stream_state_enc', { keyPath: 'streamId' });
stream.createIndex('byStatus', 'status');
stream.createIndex('byPeerAddress', 'peerAddress');
stream.createIndex('byUpdatedAt', 'updatedAt');
db.createObjectStore('peer_verifications_enc', { keyPath: 'peerAddress' });
db.createObjectStore('peer_identity_versions_enc', { keyPath: 'peerAddress' });
}
if (oldVersion < 2) {
db.createObjectStore('broadcast_channels_enc', { keyPath: 'channelId' });
const members = db.createObjectStore('broadcast_members_enc', {
keyPath: ['channelId', 'peerAddress'],
});
members.createIndex('byChannelId', 'channelId');
}
},
});
const store = new EncryptedIndexedDBStorage(db, opts.keyManager);
await store.assertKeyMatchesOrPersistFingerprint();
return store;
}
/** Cleanly close the underlying connection. KeyManager is destroyed. */
close(): void {
this.db.close();
this.km.destroy();
}
/**
* On first open, persist a fingerprint of the storageKey. On subsequent
* opens, compare and reject mismatches with a clear error rather than
* silently writing data under the wrong key.
*/
private async assertKeyMatchesOrPersistFingerprint(): Promise<void> {
const expected = toBase64(this.km.storageKeyFingerprint());
const row = await this.db.get('meta_enc', 'storage_key_fingerprint');
if (!row) {
await this.db.put('meta_enc', { key: 'storage_key_fingerprint', value: expected });
return;
}
if (row.value !== expected) {
throw new Error(
'storage key mismatch — the supplied passphrase / keychain entry does not unlock this database',
);
}
}
// ─── Identity ──────────────────────────────────────────────
async getIdentityKeyPair(): Promise<IdentityKeyPair | null> {
const row = await this.db.get('identity_enc', 1);
if (!row) return null;
return openIdentity(this.km, row.ciphertext);
}
async saveIdentityKeyPair(kp: IdentityKeyPair): Promise<void> {
const blob = await sealIdentity(this.km, kp);
await this.db.put('identity_enc', { id: 1, ciphertext: blob });
}
async getLocalRegistrationId(): Promise<number> {
const row = await this.db.get('config_enc', 'registrationId');
if (!row) return 0;
const v = await openConfig(this.km, 'registrationId', row.ciphertext);
return parseInt(v, 10);
}
async saveLocalRegistrationId(id: number): Promise<void> {
const blob = await sealConfig(this.km, 'registrationId', String(id));
await this.db.put('config_enc', { key: 'registrationId', ciphertext: blob });
}
// ─── Signed PreKeys ────────────────────────────────────────
async getSignedPreKey(keyId: number): Promise<SignedPreKey | null> {
const row = await this.db.get('signed_prekeys_enc', keyId);
if (!row) return null;
return openSignedPreKey(this.km, keyId, row.ciphertext);
}
async saveSignedPreKey(key: SignedPreKey): Promise<void> {
const blob = await sealSignedPreKey(this.km, key);
await this.db.put('signed_prekeys_enc', { keyId: key.keyId, ciphertext: blob });
}
async removeSignedPreKey(keyId: number): Promise<void> {
await this.db.delete('signed_prekeys_enc', keyId);
}
// ─── One-Time PreKeys ──────────────────────────────────────
async getOneTimePreKey(keyId: number): Promise<OneTimePreKey | null> {
const row = await this.db.get('one_time_prekeys_enc', keyId);
if (!row) return null;
return openOneTimePreKey(this.km, keyId, row.ciphertext);
}
async saveOneTimePreKey(key: OneTimePreKey): Promise<void> {
const blob = await sealOneTimePreKey(this.km, key);
await this.db.put('one_time_prekeys_enc', { keyId: key.keyId, ciphertext: blob });
}
async removeOneTimePreKey(keyId: number): Promise<void> {
await this.db.delete('one_time_prekeys_enc', keyId);
}
async getOneTimePreKeyCount(): Promise<number> {
return this.db.count('one_time_prekeys_enc');
}
// ─── Sessions ──────────────────────────────────────────────
async getSession(address: string): Promise<SessionState | null> {
const row = await this.db.get('sessions_enc', address);
if (!row) return null;
return openSession(this.km, address, row.ciphertext);
}
async saveSession(address: string, state: SessionState): Promise<void> {
const blob = await sealSession(this.km, address, state);
await this.db.put('sessions_enc', { address, ciphertext: blob });
}
async removeSession(address: string): Promise<void> {
await this.db.delete('sessions_enc', address);
}
// ─── Trust ─────────────────────────────────────────────────
async isTrustedIdentity(address: string, identityKey: Uint8Array): Promise<boolean> {
const row = await this.db.get('trusted_identities_enc', address);
if (!row) return true; // TOFU
const stored = await openTrust(this.km, address, row.ciphertext);
return constantTimeEqual(stored, identityKey);
}
async saveTrustedIdentity(address: string, identityKey: Uint8Array): Promise<void> {
const blob = await sealTrust(this.km, address, identityKey);
await this.db.put('trusted_identities_enc', { address, ciphertext: blob });
}
// ─── Identity History ──────────────────────────────────────
async addRetiredIdentity(identity: RetiredIdentity): Promise<void> {
const blob = await sealRetired(this.km, identity);
await this.db.put('retired_identities_enc', {
retiredAt: identity.retiredAt,
ciphertext: blob,
});
}
async getRetiredIdentities(): Promise<RetiredIdentity[]> {
// Mirror SQLite's `ORDER BY retired_at DESC`
const rows = await this.db.getAllFromIndex('retired_identities_enc', 'byRetiredAt');
rows.reverse();
return Promise.all(
rows.map((r) => openRetired(this.km, r.retiredAt, r.ciphertext)),
);
}
async pruneRetiredIdentities(olderThan: number): Promise<void> {
const tx = this.db.transaction('retired_identities_enc', 'readwrite');
const idx = tx.store.index('byRetiredAt');
const range = IDBKeyRange.upperBound(olderThan, true);
let cursor = await idx.openCursor(range);
while (cursor) {
await cursor.delete();
cursor = await cursor.continue();
}
await tx.done;
}
// ─── Stream-transfer resume state ──────────────────────────
async saveStreamState(state: PersistedStreamState): Promise<void> {
const blob = await sealStreamSensitive(this.km, state);
await this.db.put('stream_state_enc', {
streamId: state.streamId,
direction: state.direction,
peerAddress: state.peerAddress,
status: state.status,
ciphertext: blob,
createdAt: state.createdAt,
updatedAt: state.updatedAt,
});
}
async getStreamState(streamId: string): Promise<PersistedStreamState | null> {
const row = await this.db.get('stream_state_enc', streamId);
if (!row) return null;
return this.rowToStreamState(row);
}
async removeStreamState(streamId: string): Promise<void> {
await this.db.delete('stream_state_enc', streamId);
}
async listActiveStreamStates(direction?: 'send' | 'receive'): Promise<PersistedStreamState[]> {
const tx = this.db.transaction('stream_state_enc');
const idx = tx.store.index('byStatus');
const active = await idx.getAll(IDBKeyRange.only('active'));
const paused = await idx.getAll(IDBKeyRange.only('paused'));
const merged = [...active, ...paused];
const filtered = direction === undefined
? merged
: merged.filter((r) => r.direction === direction);
filtered.sort((a, b) => b.updatedAt - a.updatedAt);
return Promise.all(filtered.map((r) => this.rowToStreamState(r)));
}
async pruneStreamStates(olderThan: number): Promise<void> {
const tx = this.db.transaction('stream_state_enc', 'readwrite');
const idx = tx.store.index('byUpdatedAt');
const range = IDBKeyRange.upperBound(olderThan, true);
let cursor = await idx.openCursor(range);
while (cursor) {
const row = cursor.value;
if (row.status === 'finished' || row.status === 'aborted') {
await cursor.delete();
}
cursor = await cursor.continue();
}
await tx.done;
}
// ─── Peer verifications (V3.3) ────────────────────────────
// Fingerprints are public-by-design; stored in plaintext for symmetry
// with the SQLite/Postgres encrypted backends.
async savePeerVerification(v: PeerVerification): Promise<void> {
await this.db.put('peer_verifications_enc', { ...v });
}
async getPeerVerification(address: string): Promise<PeerVerification | null> {
const row = await this.db.get('peer_verifications_enc', address);
if (!row) return null;
return {
peerAddress: row.peerAddress,
fingerprint: row.fingerprint,
verifiedAt: row.verifiedAt,
verifiedBy: row.verifiedBy as PeerVerificationSource,
identityVersion: row.identityVersion,
};
}
async removePeerVerification(address: string): Promise<void> {
await this.db.delete('peer_verifications_enc', address);
}
async getPeerIdentityVersion(address: string): Promise<number> {
const row = await this.db.get('peer_identity_versions_enc', address);
return row ? row.version : 1;
}
async bumpPeerIdentityVersion(address: string): Promise<number> {
// Atomic read-modify-write under one IDB transaction. Closes the race
// that exists in the SQLite version's non-atomic read-then-upsert.
const tx = this.db.transaction('peer_identity_versions_enc', 'readwrite');
const existing = await tx.store.get(address);
const next = (existing ? existing.version : 1) + 1;
await tx.store.put({ peerAddress: address, version: next });
await tx.done;
return next;
}
// ─── Broadcast channels (V4.6) ────────────────────────────
async saveBroadcastChannel(channel: BroadcastChannelRecord): Promise<void> {
const sealed = await sealBroadcastChannelSensitive(this.km, channel.channelId, {
chainKey: channel.chainKey,
iteration: channel.iteration,
signingPublicKey: channel.signingPublicKey,
...(channel.signingPrivateKey !== undefined ? { signingPrivateKey: channel.signingPrivateKey } : {}),
});
await this.db.put('broadcast_channels_enc', {
channelId: channel.channelId,
ownerRole: channel.ownerRole,
ownerAddress: channel.ownerAddress,
label: channel.label ?? null,
generation: channel.generation,
ciphertext: sealed,
createdAt: channel.createdAt,
updatedAt: channel.updatedAt,
});
}
async getBroadcastChannel(channelId: string): Promise<BroadcastChannelRecord | null> {
const row = await this.db.get('broadcast_channels_enc', channelId);
if (!row) return null;
return this.encRowToChannel(row);
}
async listBroadcastChannels(): Promise<BroadcastChannelRecord[]> {
const rows = await this.db.getAll('broadcast_channels_enc');
rows.sort((a, b) => a.createdAt - b.createdAt);
return Promise.all(rows.map((r) => this.encRowToChannel(r)));
}
async removeBroadcastChannel(channelId: string): Promise<void> {
const tx = this.db.transaction(
['broadcast_channels_enc', 'broadcast_members_enc'],
'readwrite',
);
const memIdx = tx.objectStore('broadcast_members_enc').index('byChannelId');
let cursor = await memIdx.openCursor(IDBKeyRange.only(channelId));
while (cursor) {
await cursor.delete();
cursor = await cursor.continue();
}
await tx.objectStore('broadcast_channels_enc').delete(channelId);
await tx.done;
}
async saveBroadcastMember(member: BroadcastMemberRecord): Promise<void> {
await this.db.put('broadcast_members_enc', { ...member });
}
async getBroadcastMembers(channelId: string): Promise<BroadcastMemberRecord[]> {
const rows = await this.db.getAllFromIndex(
'broadcast_members_enc',
'byChannelId',
IDBKeyRange.only(channelId),
);
rows.sort((a, b) => a.joinedAt - b.joinedAt);
return rows.map((r) => ({
channelId: r.channelId,
peerAddress: r.peerAddress,
joinedAt: r.joinedAt,
removedAt: r.removedAt,
}));
}
async removeBroadcastMember(channelId: string, peerAddress: string): Promise<void> {
await this.db.delete('broadcast_members_enc', [channelId, peerAddress]);
}
private async encRowToChannel(row: BroadcastChannelEncRow): Promise<BroadcastChannelRecord> {
const sensitive = await openBroadcastChannelSensitive(
this.km,
row.channelId,
row.ciphertext,
);
const out: BroadcastChannelRecord = {
channelId: row.channelId,
ownerRole: row.ownerRole,
ownerAddress: row.ownerAddress,
generation: row.generation,
chainKey: sensitive.chainKey,
iteration: sensitive.iteration,
signingPublicKey: sensitive.signingPublicKey,
createdAt: row.createdAt,
updatedAt: row.updatedAt,
};
if (row.label !== null) out.label = row.label;
if (sensitive.signingPrivateKey !== undefined) out.signingPrivateKey = sensitive.signingPrivateKey;
return out;
}
private async rowToStreamState(row: StreamStateEncRow): Promise<PersistedStreamState> {
const sensitive = await openStreamSensitive(this.km, row.streamId, row.ciphertext);
const out: PersistedStreamState = {
streamId: row.streamId,
direction: row.direction,
peerAddress: row.peerAddress,
status: row.status,
metadataJson: sensitive.metadataJson,
partitionJson: sensitive.partitionJson,
laneStateJson: sensitive.laneStateJson,
ioDescriptorJson: sensitive.ioDescriptorJson,
secretEnc: sensitive.secretEnc,
secretNonce: sensitive.secretNonce,
createdAt: row.createdAt,
updatedAt: row.updatedAt,
};
if (sensitive.overallHashState !== undefined) out.overallHashState = sensitive.overallHashState;
return out;
}
}
// ─── Schema ────────────────────────────────────────────────
const SCHEMA_VERSION = 2;
interface MetaRow { key: string; value: string }
interface IdentityRow { id: 1; ciphertext: Uint8Array }
interface ConfigRow { key: string; ciphertext: Uint8Array }
interface SignedPreKeyRow { keyId: number; ciphertext: Uint8Array }
interface OneTimePreKeyRow { keyId: number; ciphertext: Uint8Array }
interface SessionRow { address: string; ciphertext: Uint8Array }
interface TrustedIdentityRow { address: string; ciphertext: Uint8Array }
interface RetiredIdentityRow { retiredAt: number; ciphertext: Uint8Array }
interface StreamStateEncRow {
streamId: string;
direction: 'send' | 'receive';
peerAddress: string;
status: 'active' | 'paused' | 'finished' | 'aborted';
ciphertext: Uint8Array;
createdAt: number;
updatedAt: number;
}
interface PeerVerificationRow {
peerAddress: string;
fingerprint: string;
verifiedAt: number;
verifiedBy: string;
identityVersion: number;
}
interface PeerIdentityVersionRow { peerAddress: string; version: number }
interface BroadcastChannelEncRow {
channelId: string;
ownerRole: 'sender' | 'receiver';
ownerAddress: string;
label: string | null;
generation: number;
ciphertext: Uint8Array;
createdAt: number;
updatedAt: number;
}
interface BroadcastMemberEncRow {
channelId: string;
peerAddress: string;
joinedAt: number;
removedAt: number | null;
}
interface EncryptedShadeSchema extends DBSchema {
meta_enc: { key: string; value: MetaRow };
identity_enc: { key: number; value: IdentityRow };
config_enc: { key: string; value: ConfigRow };
signed_prekeys_enc: { key: number; value: SignedPreKeyRow };
one_time_prekeys_enc: { key: number; value: OneTimePreKeyRow };
sessions_enc: { key: string; value: SessionRow };
trusted_identities_enc: { key: string; value: TrustedIdentityRow };
retired_identities_enc: {
key: number;
value: RetiredIdentityRow;
indexes: { byRetiredAt: number };
};
stream_state_enc: {
key: string;
value: StreamStateEncRow;
indexes: {
byStatus: string;
byPeerAddress: string;
byUpdatedAt: number;
};
};
peer_verifications_enc: { key: string; value: PeerVerificationRow };
peer_identity_versions_enc: { key: string; value: PeerIdentityVersionRow };
broadcast_channels_enc: { key: string; value: BroadcastChannelEncRow };
broadcast_members_enc: {
key: [string, string];
value: BroadcastMemberEncRow;
indexes: { byChannelId: string };
};
}

View File

@@ -1,6 +1,8 @@
import type { Sql } from 'postgres'; import type { Sql } from 'postgres';
import postgres from 'postgres'; import postgres from 'postgres';
import type { import type {
BroadcastChannelRecord,
BroadcastMemberRecord,
IdentityKeyPair, IdentityKeyPair,
OneTimePreKey, OneTimePreKey,
PeerVerification, PeerVerification,
@@ -14,10 +16,10 @@ import type {
import { constantTimeEqual, toBase64 } from '@shade/core'; import { constantTimeEqual, toBase64 } from '@shade/core';
import { KeyManager } from '../crypto/key-manager.js'; import { KeyManager } from '../crypto/key-manager.js';
import { import {
openConfig, openIdentity, openOneTimePreKey, openRetired, openSession, openBroadcastChannelSensitive, openConfig, openIdentity, openOneTimePreKey,
openSignedPreKey, openStreamSensitive, openTrust, sealConfig, sealIdentity, openRetired, openSession, openSignedPreKey, openStreamSensitive, openTrust,
sealOneTimePreKey, sealRetired, sealSession, sealSignedPreKey, sealBroadcastChannelSensitive, sealConfig, sealIdentity, sealOneTimePreKey,
sealStreamSensitive, sealTrust, sealRetired, sealSession, sealSignedPreKey, sealStreamSensitive, sealTrust,
} from '../crypto/row-codec.js'; } from '../crypto/row-codec.js';
/** /**
@@ -332,6 +334,108 @@ export class EncryptedPostgresStorage implements StorageProvider {
return next; return next;
} }
// ─── Broadcast channels (V4.6) ────────────────────────────
async saveBroadcastChannel(channel: BroadcastChannelRecord): Promise<void> {
const sealed = await sealBroadcastChannelSensitive(this.km, channel.channelId, {
chainKey: channel.chainKey,
iteration: channel.iteration,
signingPublicKey: channel.signingPublicKey,
...(channel.signingPrivateKey !== undefined ? { signingPrivateKey: channel.signingPrivateKey } : {}),
});
await this.sql`
INSERT INTO shade_broadcast_channels_enc
(channel_id, owner_role, owner_address, label, generation, ciphertext, created_at, updated_at)
VALUES
(${channel.channelId}, ${channel.ownerRole}, ${channel.ownerAddress},
${channel.label ?? null}, ${channel.generation}, ${sealed},
${channel.createdAt}, ${channel.updatedAt})
ON CONFLICT (channel_id) DO UPDATE SET
owner_role = EXCLUDED.owner_role,
owner_address = EXCLUDED.owner_address,
label = EXCLUDED.label,
generation = EXCLUDED.generation,
ciphertext = EXCLUDED.ciphertext,
updated_at = EXCLUDED.updated_at
`;
}
async getBroadcastChannel(channelId: string): Promise<BroadcastChannelRecord | null> {
const rows = await this.sql<Array<BroadcastChannelEncRow>>`
SELECT * FROM shade_broadcast_channels_enc WHERE channel_id = ${channelId}
`;
if (rows.length === 0) return null;
return this.encRowToChannel(rows[0]!);
}
async listBroadcastChannels(): Promise<BroadcastChannelRecord[]> {
const rows = await this.sql<Array<BroadcastChannelEncRow>>`
SELECT * FROM shade_broadcast_channels_enc ORDER BY created_at ASC
`;
return Promise.all(rows.map((r) => this.encRowToChannel(r)));
}
async removeBroadcastChannel(channelId: string): Promise<void> {
await this.sql`DELETE FROM shade_broadcast_members_enc WHERE channel_id = ${channelId}`;
await this.sql`DELETE FROM shade_broadcast_channels_enc WHERE channel_id = ${channelId}`;
}
async saveBroadcastMember(member: BroadcastMemberRecord): Promise<void> {
await this.sql`
INSERT INTO shade_broadcast_members_enc
(channel_id, peer_address, joined_at, removed_at)
VALUES
(${member.channelId}, ${member.peerAddress}, ${member.joinedAt}, ${member.removedAt})
ON CONFLICT (channel_id, peer_address) DO UPDATE SET
joined_at = EXCLUDED.joined_at,
removed_at = EXCLUDED.removed_at
`;
}
async getBroadcastMembers(channelId: string): Promise<BroadcastMemberRecord[]> {
const rows = await this.sql<Array<{ channel_id: string; peer_address: string; joined_at: string | number; removed_at: string | number | null }>>`
SELECT channel_id, peer_address, joined_at, removed_at
FROM shade_broadcast_members_enc
WHERE channel_id = ${channelId}
ORDER BY joined_at ASC
`;
return rows.map((r) => ({
channelId: r.channel_id,
peerAddress: r.peer_address,
joinedAt: Number(r.joined_at),
removedAt: r.removed_at === null ? null : Number(r.removed_at),
}));
}
async removeBroadcastMember(channelId: string, peerAddress: string): Promise<void> {
await this.sql`
DELETE FROM shade_broadcast_members_enc
WHERE channel_id = ${channelId} AND peer_address = ${peerAddress}
`;
}
private async encRowToChannel(row: BroadcastChannelEncRow): Promise<BroadcastChannelRecord> {
const sensitive = await openBroadcastChannelSensitive(
this.km,
String(row.channel_id),
row.ciphertext,
);
const out: BroadcastChannelRecord = {
channelId: String(row.channel_id),
ownerRole: row.owner_role,
ownerAddress: String(row.owner_address),
generation: Number(row.generation),
chainKey: sensitive.chainKey,
iteration: sensitive.iteration,
signingPublicKey: sensitive.signingPublicKey,
createdAt: Number(row.created_at),
updatedAt: Number(row.updated_at),
};
if (row.label !== null && row.label !== undefined) out.label = row.label;
if (sensitive.signingPrivateKey !== undefined) out.signingPrivateKey = sensitive.signingPrivateKey;
return out;
}
private async rowToStreamState(row: StreamRow): Promise<PersistedStreamState> { private async rowToStreamState(row: StreamRow): Promise<PersistedStreamState> {
const sensitive = await openStreamSensitive(this.km, String(row.stream_id), row.ciphertext); const sensitive = await openStreamSensitive(this.km, String(row.stream_id), row.ciphertext);
const out: PersistedStreamState = { const out: PersistedStreamState = {
@@ -363,6 +467,17 @@ interface StreamRow {
updated_at: string | number; updated_at: string | number;
} }
interface BroadcastChannelEncRow {
channel_id: string;
owner_role: 'sender' | 'receiver';
owner_address: string;
label: string | null;
generation: string | number;
ciphertext: Uint8Array;
created_at: string | number;
updated_at: string | number;
}
export async function ensureEncryptedClientTables(sql: Sql): Promise<void> { export async function ensureEncryptedClientTables(sql: Sql): Promise<void> {
await sql` await sql`
CREATE TABLE IF NOT EXISTS shade_meta_enc ( CREATE TABLE IF NOT EXISTS shade_meta_enc (
@@ -454,4 +569,29 @@ export async function ensureEncryptedClientTables(sql: Sql): Promise<void> {
version BIGINT NOT NULL version BIGINT NOT NULL
) )
`; `;
await sql`
CREATE TABLE IF NOT EXISTS shade_broadcast_channels_enc (
channel_id TEXT PRIMARY KEY,
owner_role TEXT NOT NULL CHECK (owner_role IN ('sender','receiver')),
owner_address TEXT NOT NULL,
label TEXT,
generation BIGINT NOT NULL,
ciphertext BYTEA NOT NULL,
created_at BIGINT NOT NULL,
updated_at BIGINT NOT NULL
)
`;
await sql`
CREATE TABLE IF NOT EXISTS shade_broadcast_members_enc (
channel_id TEXT NOT NULL,
peer_address TEXT NOT NULL,
joined_at BIGINT NOT NULL,
removed_at BIGINT,
PRIMARY KEY (channel_id, peer_address)
)
`;
await sql`
CREATE INDEX IF NOT EXISTS shade_broadcast_members_enc_channel_idx
ON shade_broadcast_members_enc(channel_id)
`;
} }

View File

@@ -1,5 +1,7 @@
import { Database } from 'bun:sqlite'; import { Database } from 'bun:sqlite';
import type { import type {
BroadcastChannelRecord,
BroadcastMemberRecord,
IdentityKeyPair, IdentityKeyPair,
OneTimePreKey, OneTimePreKey,
PeerVerification, PeerVerification,
@@ -13,10 +15,10 @@ import type {
import { constantTimeEqual, toBase64 } from '@shade/core'; import { constantTimeEqual, toBase64 } from '@shade/core';
import { KeyManager } from '../crypto/key-manager.js'; import { KeyManager } from '../crypto/key-manager.js';
import { import {
openConfig, openIdentity, openOneTimePreKey, openRetired, openSession, openBroadcastChannelSensitive, openConfig, openIdentity, openOneTimePreKey, openRetired,
openSignedPreKey, openStreamSensitive, openTrust, sealConfig, sealIdentity, openSession, openSignedPreKey, openStreamSensitive, openTrust,
sealOneTimePreKey, sealRetired, sealSession, sealSignedPreKey, sealBroadcastChannelSensitive, sealConfig, sealIdentity, sealOneTimePreKey,
sealStreamSensitive, sealTrust, sealRetired, sealSession, sealSignedPreKey, sealStreamSensitive, sealTrust,
} from '../crypto/row-codec.js'; } from '../crypto/row-codec.js';
/** /**
@@ -68,6 +70,14 @@ export class EncryptedSQLiteStorage implements StorageProvider {
removePeerVerification: ReturnType<Database['prepare']>; removePeerVerification: ReturnType<Database['prepare']>;
getPeerIdentityVersion: ReturnType<Database['prepare']>; getPeerIdentityVersion: ReturnType<Database['prepare']>;
upsertPeerIdentityVersion: ReturnType<Database['prepare']>; upsertPeerIdentityVersion: ReturnType<Database['prepare']>;
saveBroadcastChannel: ReturnType<Database['prepare']>;
getBroadcastChannel: ReturnType<Database['prepare']>;
listBroadcastChannels: ReturnType<Database['prepare']>;
removeBroadcastChannel: ReturnType<Database['prepare']>;
removeBroadcastChannelMembers: ReturnType<Database['prepare']>;
saveBroadcastMember: ReturnType<Database['prepare']>;
getBroadcastMembers: ReturnType<Database['prepare']>;
removeBroadcastMember: ReturnType<Database['prepare']>;
}; };
private constructor(db: Database, km: KeyManager, ownsDb: boolean) { private constructor(db: Database, km: KeyManager, ownsDb: boolean) {
@@ -156,6 +166,24 @@ export class EncryptedSQLiteStorage implements StorageProvider {
peer_address TEXT PRIMARY KEY, peer_address TEXT PRIMARY KEY,
version INTEGER NOT NULL version INTEGER NOT NULL
); );
CREATE TABLE IF NOT EXISTS broadcast_channels_enc (
channel_id TEXT PRIMARY KEY,
owner_role TEXT NOT NULL,
owner_address TEXT NOT NULL,
label TEXT,
generation INTEGER NOT NULL,
ciphertext BLOB NOT NULL,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS broadcast_members_enc (
channel_id TEXT NOT NULL,
peer_address TEXT NOT NULL,
joined_at INTEGER NOT NULL,
removed_at INTEGER,
PRIMARY KEY (channel_id, peer_address)
);
CREATE INDEX IF NOT EXISTS idx_broadcast_members_enc_channel ON broadcast_members_enc(channel_id);
`); `);
} }
@@ -212,6 +240,35 @@ export class EncryptedSQLiteStorage implements StorageProvider {
`INSERT INTO peer_identity_versions_enc (peer_address, version) VALUES (?, ?) `INSERT INTO peer_identity_versions_enc (peer_address, version) VALUES (?, ?)
ON CONFLICT(peer_address) DO UPDATE SET version = excluded.version`, ON CONFLICT(peer_address) DO UPDATE SET version = excluded.version`,
), ),
saveBroadcastChannel: this.db.prepare(
`INSERT OR REPLACE INTO broadcast_channels_enc
(channel_id, owner_role, owner_address, label, generation,
ciphertext, created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
),
getBroadcastChannel: this.db.prepare(
'SELECT * FROM broadcast_channels_enc WHERE channel_id = ?',
),
listBroadcastChannels: this.db.prepare(
'SELECT * FROM broadcast_channels_enc ORDER BY created_at ASC',
),
removeBroadcastChannel: this.db.prepare(
'DELETE FROM broadcast_channels_enc WHERE channel_id = ?',
),
removeBroadcastChannelMembers: this.db.prepare(
'DELETE FROM broadcast_members_enc WHERE channel_id = ?',
),
saveBroadcastMember: this.db.prepare(
`INSERT OR REPLACE INTO broadcast_members_enc
(channel_id, peer_address, joined_at, removed_at)
VALUES (?, ?, ?, ?)`,
),
getBroadcastMembers: this.db.prepare(
'SELECT channel_id, peer_address, joined_at, removed_at FROM broadcast_members_enc WHERE channel_id = ? ORDER BY joined_at ASC',
),
removeBroadcastMember: this.db.prepare(
'DELETE FROM broadcast_members_enc WHERE channel_id = ? AND peer_address = ?',
),
}; };
} }
@@ -432,6 +489,88 @@ export class EncryptedSQLiteStorage implements StorageProvider {
return next; return next;
} }
// ─── Broadcast channels (V4.6) ────────────────────────────
async saveBroadcastChannel(channel: BroadcastChannelRecord): Promise<void> {
const sealed = await sealBroadcastChannelSensitive(this.km, channel.channelId, {
chainKey: channel.chainKey,
iteration: channel.iteration,
signingPublicKey: channel.signingPublicKey,
...(channel.signingPrivateKey !== undefined ? { signingPrivateKey: channel.signingPrivateKey } : {}),
});
this.stmts.saveBroadcastChannel.run(
channel.channelId,
channel.ownerRole,
channel.ownerAddress,
channel.label ?? null,
channel.generation,
sealed,
channel.createdAt,
channel.updatedAt,
);
}
async getBroadcastChannel(channelId: string): Promise<BroadcastChannelRecord | null> {
const row = this.stmts.getBroadcastChannel.get(channelId) as BroadcastChannelEncRow | undefined;
if (!row) return null;
return this.encRowToChannel(row);
}
async listBroadcastChannels(): Promise<BroadcastChannelRecord[]> {
const rows = this.stmts.listBroadcastChannels.all() as BroadcastChannelEncRow[];
return Promise.all(rows.map((r) => this.encRowToChannel(r)));
}
async removeBroadcastChannel(channelId: string): Promise<void> {
this.stmts.removeBroadcastChannelMembers.run(channelId);
this.stmts.removeBroadcastChannel.run(channelId);
}
async saveBroadcastMember(member: BroadcastMemberRecord): Promise<void> {
this.stmts.saveBroadcastMember.run(
member.channelId,
member.peerAddress,
member.joinedAt,
member.removedAt,
);
}
async getBroadcastMembers(channelId: string): Promise<BroadcastMemberRecord[]> {
const rows = this.stmts.getBroadcastMembers.all(channelId) as BroadcastMemberEncRow[];
return rows.map((r) => ({
channelId: r.channel_id,
peerAddress: r.peer_address,
joinedAt: Number(r.joined_at),
removedAt: r.removed_at === null || r.removed_at === undefined ? null : Number(r.removed_at),
}));
}
async removeBroadcastMember(channelId: string, peerAddress: string): Promise<void> {
this.stmts.removeBroadcastMember.run(channelId, peerAddress);
}
private async encRowToChannel(row: BroadcastChannelEncRow): Promise<BroadcastChannelRecord> {
const sensitive = await openBroadcastChannelSensitive(
this.km,
row.channel_id,
toBytes(row.ciphertext),
);
const out: BroadcastChannelRecord = {
channelId: row.channel_id,
ownerRole: row.owner_role,
ownerAddress: row.owner_address,
generation: Number(row.generation),
chainKey: sensitive.chainKey,
iteration: sensitive.iteration,
signingPublicKey: sensitive.signingPublicKey,
createdAt: Number(row.created_at),
updatedAt: Number(row.updated_at),
};
if (row.label !== null && row.label !== undefined) out.label = row.label;
if (sensitive.signingPrivateKey !== undefined) out.signingPrivateKey = sensitive.signingPrivateKey;
return out;
}
private async rowToStreamState(row: StreamRow): Promise<PersistedStreamState> { private async rowToStreamState(row: StreamRow): Promise<PersistedStreamState> {
const sensitive = await openStreamSensitive(this.km, row.stream_id, toBytes(row.ciphertext)); const sensitive = await openStreamSensitive(this.km, row.stream_id, toBytes(row.ciphertext));
const out: PersistedStreamState = { const out: PersistedStreamState = {
@@ -463,6 +602,24 @@ interface StreamRow {
updated_at: number | bigint; updated_at: number | bigint;
} }
interface BroadcastChannelEncRow {
channel_id: string;
owner_role: 'sender' | 'receiver';
owner_address: string;
label: string | null;
generation: number | bigint;
ciphertext: Uint8Array | ArrayBuffer;
created_at: number | bigint;
updated_at: number | bigint;
}
interface BroadcastMemberEncRow {
channel_id: string;
peer_address: string;
joined_at: number | bigint;
removed_at: number | bigint | null;
}
function toBytes(value: Uint8Array | ArrayBuffer | unknown): Uint8Array { function toBytes(value: Uint8Array | ArrayBuffer | unknown): Uint8Array {
if (value instanceof Uint8Array) return value; if (value instanceof Uint8Array) return value;
if (value instanceof ArrayBuffer) return new Uint8Array(value); if (value instanceof ArrayBuffer) return new Uint8Array(value);

View File

@@ -0,0 +1,292 @@
import 'fake-indexeddb/auto';
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { tmpdir } from 'os';
import { join } from 'path';
import { unlinkSync } from 'fs';
import { EncryptedIndexedDBStorage } from '../src/storage/encrypted-indexeddb.js';
import { EncryptedSQLiteStorage } from '../src/storage/encrypted-sqlite.js';
import { KeyManager } from '../src/crypto/key-manager.js';
import type {
IdentityKeyPair,
OneTimePreKey,
PersistedStreamState,
SessionState,
SignedPreKey,
} from '@shade/core';
function randBytes(n: number): Uint8Array {
const b = new Uint8Array(n);
globalThis.crypto.getRandomValues(b);
return b;
}
function dummyIdentity(): IdentityKeyPair {
return {
signingPublicKey: randBytes(32),
signingPrivateKey: randBytes(32),
dhPublicKey: randBytes(32),
dhPrivateKey: randBytes(32),
};
}
function dummySignedPreKey(id: number): SignedPreKey {
return {
keyId: id,
keyPair: { publicKey: randBytes(32), privateKey: randBytes(32) },
signature: randBytes(64),
timestamp: Date.now(),
};
}
function dummyOTP(id: number): OneTimePreKey {
return {
keyId: id,
keyPair: { publicKey: randBytes(32), privateKey: randBytes(32) },
};
}
function dummySession(): SessionState {
return {
remoteIdentityKey: randBytes(32),
rootKey: randBytes(32),
sendChain: { chainKey: randBytes(32), counter: 0 },
receiveChain: { chainKey: randBytes(32), counter: 0 },
dhSend: { publicKey: randBytes(32), privateKey: randBytes(32) },
dhReceive: randBytes(32),
previousSendCounter: 0,
skippedKeys: new Map(),
};
}
function uniqueDbName(): string {
return `shade-enc-idb-${Date.now()}-${Math.random().toString(36).slice(2)}`;
}
async function deleteDb(name: string): Promise<void> {
await new Promise<void>((resolve) => {
const req = indexedDB.deleteDatabase(name);
req.onsuccess = () => resolve();
req.onerror = () => resolve();
req.onblocked = () => resolve();
});
}
const KEY_BYTES = randBytes(32);
async function freshKM(): Promise<KeyManager> {
return KeyManager.open({ kind: 'injected', key: KEY_BYTES });
}
describe('EncryptedIndexedDBStorage', () => {
let dbName: string;
let store: EncryptedIndexedDBStorage;
beforeEach(async () => {
dbName = uniqueDbName();
store = await EncryptedIndexedDBStorage.open({ dbName, keyManager: await freshKM() });
});
afterEach(async () => {
store.close();
await deleteDb(dbName);
});
test('identity round-trip', async () => {
expect(await store.getIdentityKeyPair()).toBeNull();
const kp = dummyIdentity();
await store.saveIdentityKeyPair(kp);
const got = await store.getIdentityKeyPair();
expect(got).toEqual(kp);
});
test('registrationId round-trip', async () => {
expect(await store.getLocalRegistrationId()).toBe(0);
await store.saveLocalRegistrationId(987);
expect(await store.getLocalRegistrationId()).toBe(987);
});
test('signed prekey round-trip + remove', async () => {
expect(await store.getSignedPreKey(7)).toBeNull();
const k = dummySignedPreKey(7);
await store.saveSignedPreKey(k);
const got = await store.getSignedPreKey(7);
expect(got?.keyId).toBe(7);
expect(got?.keyPair.privateKey).toEqual(k.keyPair.privateKey);
await store.removeSignedPreKey(7);
expect(await store.getSignedPreKey(7)).toBeNull();
});
test('one-time prekey count + remove', async () => {
expect(await store.getOneTimePreKeyCount()).toBe(0);
await store.saveOneTimePreKey(dummyOTP(1));
await store.saveOneTimePreKey(dummyOTP(2));
expect(await store.getOneTimePreKeyCount()).toBe(2);
expect(await store.getOneTimePreKey(1)).not.toBeNull();
await store.removeOneTimePreKey(1);
expect(await store.getOneTimePreKeyCount()).toBe(1);
});
test('session round-trip + remove', async () => {
const s = dummySession();
await store.saveSession('device:abc', s);
const got = await store.getSession('device:abc');
expect(got?.rootKey).toEqual(s.rootKey);
await store.removeSession('device:abc');
expect(await store.getSession('device:abc')).toBeNull();
});
test('TOFU + trust check', async () => {
const ik = randBytes(32);
expect(await store.isTrustedIdentity('peer-1', ik)).toBe(true); // TOFU
await store.saveTrustedIdentity('peer-1', ik);
expect(await store.isTrustedIdentity('peer-1', ik)).toBe(true);
expect(await store.isTrustedIdentity('peer-1', randBytes(32))).toBe(false);
});
test('retired identities are sorted DESC + prune', async () => {
await store.addRetiredIdentity({ keyPair: dummyIdentity(), retiredAt: 100 });
await store.addRetiredIdentity({ keyPair: dummyIdentity(), retiredAt: 200 });
const list = await store.getRetiredIdentities();
expect(list.length).toBe(2);
expect(list[0]!.retiredAt).toBe(200);
await store.pruneRetiredIdentities(150);
const after = await store.getRetiredIdentities();
expect(after.length).toBe(1);
expect(after[0]!.retiredAt).toBe(200);
});
test('stream-state round-trip + listActive + prune', async () => {
const s: PersistedStreamState = {
streamId: 'stream-1',
direction: 'send',
peerAddress: 'device:bob',
status: 'active',
metadataJson: '{"name":"file.bin"}',
partitionJson: '[]',
laneStateJson: '[]',
ioDescriptorJson: '{"path":"/tmp/x"}',
secretEnc: randBytes(32),
secretNonce: randBytes(12),
createdAt: 1,
updatedAt: 2,
};
await store.saveStreamState(s);
const got = await store.getStreamState('stream-1');
expect(got).toEqual(s);
const active = await store.listActiveStreamStates();
expect(active.length).toBe(1);
expect((await store.listActiveStreamStates('receive')).length).toBe(0);
await store.saveStreamState({ ...s, streamId: 'stream-2', status: 'finished', updatedAt: 50 });
expect((await store.listActiveStreamStates()).length).toBe(1);
await store.pruneStreamStates(100);
expect(await store.getStreamState('stream-2')).toBeNull();
expect(await store.getStreamState('stream-1')).not.toBeNull();
});
test('peer verification + identity-version bump (atomic)', async () => {
expect(await store.getPeerVerification('peer-x')).toBeNull();
await store.savePeerVerification({
peerAddress: 'peer-x',
fingerprint: 'fp',
verifiedAt: 1,
verifiedBy: 'sas',
identityVersion: 1,
});
const v = await store.getPeerVerification('peer-x');
expect(v?.fingerprint).toBe('fp');
expect(await store.getPeerIdentityVersion('peer-x')).toBe(1);
expect(await store.bumpPeerIdentityVersion('peer-x')).toBe(2);
expect(await store.bumpPeerIdentityVersion('peer-x')).toBe(3);
expect(await store.getPeerIdentityVersion('peer-x')).toBe(3);
await store.removePeerVerification('peer-x');
expect(await store.getPeerVerification('peer-x')).toBeNull();
});
test('rejects open with wrong key (fingerprint mismatch)', async () => {
await store.saveIdentityKeyPair(dummyIdentity());
store.close();
const otherKey = randBytes(32);
await expect(EncryptedIndexedDBStorage.open({
dbName,
keyManager: await KeyManager.open({ kind: 'injected', key: otherKey }),
})).rejects.toThrow(/storage key mismatch/);
// Reopen with original key for afterEach
store = await EncryptedIndexedDBStorage.open({ dbName, keyManager: await freshKM() });
});
});
describe('EncryptedIndexedDBStorage — cross-impl roundtrip with EncryptedSQLiteStorage', () => {
test('row sealed by SQLite backend decrypts under IDB backend with same KeyManager', async () => {
const sharedKey = randBytes(32);
const dbPath = join(tmpdir(), `shade-cross-${Date.now()}-${Math.random().toString(36).slice(2)}.db`);
const dbName = uniqueDbName();
// Write via SQLite
const km1 = await KeyManager.open({ kind: 'injected', key: sharedKey });
const sqlite = await EncryptedSQLiteStorage.open({ dbPath, keyManager: km1 });
const kp = dummyIdentity();
await sqlite.saveIdentityKeyPair(kp);
await sqlite.saveLocalRegistrationId(424242);
await sqlite.saveSession('device:cross', dummySession());
// Pull the raw ciphertext blobs out and inject them into a fresh IDB store
// through normal saveX → check the IDB-saved blobs decrypt under the same
// KeyManager. Since AAD/nonce derivation is purely a function of (km,
// table, column, pk), bytes-equal blobs prove the row codec is
// implementation-agnostic.
sqlite.close();
try { unlinkSync(dbPath); } catch {}
try { unlinkSync(dbPath + '-wal'); } catch {}
try { unlinkSync(dbPath + '-shm'); } catch {}
const km2 = await KeyManager.open({ kind: 'injected', key: sharedKey });
const idb = await EncryptedIndexedDBStorage.open({ dbName, keyManager: km2 });
await idb.saveIdentityKeyPair(kp);
await idb.saveLocalRegistrationId(424242);
expect(await idb.getIdentityKeyPair()).toEqual(kp);
expect(await idb.getLocalRegistrationId()).toBe(424242);
idb.close();
await deleteDb(dbName);
});
test('AAD (table, column, pk) binding is implementation-agnostic', async () => {
// Open both backends with the same injected key, save the same session
// under the same address, then assert that the resulting ciphertext blobs
// are byte-equal — confirming AAD + nonce derivation is shared.
const sharedKey = randBytes(32);
const dbPath = join(tmpdir(), `shade-cross-aad-${Date.now()}.db`);
const dbName = uniqueDbName();
const session = dummySession();
const sqlite = await EncryptedSQLiteStorage.open({
dbPath,
keyManager: await KeyManager.open({ kind: 'injected', key: sharedKey }),
});
await sqlite.saveSession('addr-1', session);
const idb = await EncryptedIndexedDBStorage.open({
dbName,
keyManager: await KeyManager.open({ kind: 'injected', key: sharedKey }),
});
await idb.saveSession('addr-1', session);
// Both backends must recover the same plaintext.
const fromSqlite = await sqlite.getSession('addr-1');
const fromIdb = await idb.getSession('addr-1');
expect(fromSqlite?.rootKey).toEqual(session.rootKey);
expect(fromIdb?.rootKey).toEqual(session.rootKey);
expect(fromSqlite?.rootKey).toEqual(fromIdb!.rootKey);
sqlite.close();
idb.close();
try { unlinkSync(dbPath); } catch {}
try { unlinkSync(dbPath + '-wal'); } catch {}
try { unlinkSync(dbPath + '-shm'); } catch {}
await deleteDb(dbName);
});
});

View File

@@ -168,6 +168,82 @@ describe('EncryptedSQLiteStorage', () => {
expect(await store.getStreamState('stream-1')).not.toBeNull(); // active rows untouched expect(await store.getStreamState('stream-1')).not.toBeNull(); // active rows untouched
}); });
test('broadcast channel + member round-trip (V4.6)', async () => {
const channel = {
channelId: 'c-1',
ownerRole: 'sender' as const,
ownerAddress: 'pc',
label: 'output',
generation: 0,
chainKey: randBytes(32),
iteration: 0,
signingPublicKey: randBytes(32),
signingPrivateKey: randBytes(32),
createdAt: 1_700_000_000_000,
updatedAt: 1_700_000_000_000,
};
await store.saveBroadcastChannel(channel);
const fetched = await store.getBroadcastChannel('c-1');
expect(fetched).not.toBeNull();
expect(fetched!.ownerRole).toBe('sender');
expect(fetched!.label).toBe('output');
expect(fetched!.chainKey).toEqual(channel.chainKey);
expect(fetched!.signingPrivateKey).toEqual(channel.signingPrivateKey);
// Add two members + verify list + remove.
await store.saveBroadcastMember({
channelId: 'c-1',
peerAddress: 'mobile-a',
joinedAt: 1_700_000_000_001,
removedAt: null,
});
await store.saveBroadcastMember({
channelId: 'c-1',
peerAddress: 'mobile-b',
joinedAt: 1_700_000_000_002,
removedAt: null,
});
let members = await store.getBroadcastMembers('c-1');
expect(members.map((m) => m.peerAddress)).toEqual(['mobile-a', 'mobile-b']);
// Mark one removed.
await store.saveBroadcastMember({
channelId: 'c-1',
peerAddress: 'mobile-a',
joinedAt: 1_700_000_000_001,
removedAt: 1_700_000_000_500,
});
members = await store.getBroadcastMembers('c-1');
expect(members.find((m) => m.peerAddress === 'mobile-a')?.removedAt).toBe(1_700_000_000_500);
// List + delete cascade.
expect((await store.listBroadcastChannels())).toHaveLength(1);
await store.removeBroadcastChannel('c-1');
expect(await store.getBroadcastChannel('c-1')).toBeNull();
expect((await store.getBroadcastMembers('c-1'))).toHaveLength(0);
});
test('broadcast channel sealed: receiver-side row has no private key', async () => {
await store.saveBroadcastChannel({
channelId: 'c-2',
ownerRole: 'receiver',
ownerAddress: 'pc',
generation: 1,
chainKey: randBytes(32),
iteration: 5,
signingPublicKey: randBytes(32),
// no signingPrivateKey
createdAt: 1,
updatedAt: 1,
});
const r = await store.getBroadcastChannel('c-2');
expect(r).not.toBeNull();
expect(r!.ownerRole).toBe('receiver');
expect(r!.signingPrivateKey).toBeUndefined();
expect(r!.iteration).toBe(5);
expect(r!.generation).toBe(1);
});
test('rejects open with wrong key (fingerprint mismatch)', async () => { test('rejects open with wrong key (fingerprint mismatch)', async () => {
await store.saveIdentityKeyPair(dummyIdentity()); await store.saveIdentityKeyPair(dummyIdentity());
store.close(); store.close();

View File

@@ -0,0 +1,183 @@
import { describe, test, expect } from 'bun:test';
import { KeyManager } from '../src/crypto/key-manager.js';
import {
DEFAULT_ARGON2ID,
deriveMasterKeyArgon2id,
type Argon2idParams,
} from '../src/crypto/kdf.js';
const FAST_ARGON: Argon2idParams = { m: 256, t: 1, p: 1, dkLen: 32 };
function randBytes(n: number): Uint8Array {
const b = new Uint8Array(n);
globalThis.crypto.getRandomValues(b);
return b;
}
describe('argon2id source', () => {
const salt = new Uint8Array(16).fill(0x33);
test('deriveMasterKeyArgon2id is deterministic', async () => {
const a = await deriveMasterKeyArgon2id('1234', salt, FAST_ARGON);
const b = await deriveMasterKeyArgon2id('1234', salt, FAST_ARGON);
expect(a).toEqual(b);
expect(a.length).toBe(32);
});
test('different secret → different key', async () => {
const a = await deriveMasterKeyArgon2id('1234', salt, FAST_ARGON);
const b = await deriveMasterKeyArgon2id('1235', salt, FAST_ARGON);
expect(a).not.toEqual(b);
});
test('different salt → different key', async () => {
const a = await deriveMasterKeyArgon2id('1234', salt, FAST_ARGON);
const b = await deriveMasterKeyArgon2id('1234', new Uint8Array(16).fill(0x44), FAST_ARGON);
expect(a).not.toEqual(b);
});
test('rejects empty secret', async () => {
await expect(deriveMasterKeyArgon2id('', salt, FAST_ARGON)).rejects.toThrow(/non-empty/);
});
test('rejects too-short salt', async () => {
await expect(deriveMasterKeyArgon2id('p', new Uint8Array(8), FAST_ARGON))
.rejects.toThrow(/at least 16/);
});
test('KeyManager.open opens with argon2id source', async () => {
const km = await KeyManager.open({
kind: 'argon2id',
secret: '123456',
salt,
params: FAST_ARGON,
});
expect(km.fieldKey('t', 'c').length).toBe(32);
km.destroy();
});
test('DEFAULT_ARGON2ID is exposed and sensible', () => {
expect(DEFAULT_ARGON2ID.dkLen).toBe(32);
expect(DEFAULT_ARGON2ID.m).toBeGreaterThanOrEqual(8 * 1024);
expect(DEFAULT_ARGON2ID.t).toBeGreaterThanOrEqual(1);
});
test('accepts Uint8Array secret', async () => {
const secretBytes = new TextEncoder().encode('1234');
const a = await deriveMasterKeyArgon2id(secretBytes, salt, FAST_ARGON);
const b = await deriveMasterKeyArgon2id('1234', salt, FAST_ARGON);
expect(a).toEqual(b);
});
});
describe('composite source — multi-factor unlock', () => {
const pwSalt = new Uint8Array(16).fill(0x11);
const pinSalt = new Uint8Array(16).fill(0x22);
const FAST_SCRYPT = { N: 1 << 10, r: 8, p: 1, dkLen: 32 };
function pwSource(passphrase: string) {
return { kind: 'passphrase' as const, passphrase, salt: pwSalt, params: FAST_SCRYPT };
}
function pinSource(secret: string) {
return { kind: 'argon2id' as const, secret, salt: pinSalt, params: FAST_ARGON };
}
test('same factors → same masterKey', async () => {
const a = await KeyManager.open({
kind: 'composite',
sources: [pwSource('correct horse'), pinSource('1234')],
});
const b = await KeyManager.open({
kind: 'composite',
sources: [pwSource('correct horse'), pinSource('1234')],
});
expect(a.storageKeyFingerprint()).toEqual(b.storageKeyFingerprint());
a.destroy();
b.destroy();
});
test('wrong PIN → different masterKey (same shape as wrong-passphrase)', async () => {
const right = await KeyManager.open({
kind: 'composite',
sources: [pwSource('correct horse'), pinSource('1234')],
});
const wrongPin = await KeyManager.open({
kind: 'composite',
sources: [pwSource('correct horse'), pinSource('9999')],
});
expect(right.storageKeyFingerprint()).not.toEqual(wrongPin.storageKeyFingerprint());
right.destroy();
wrongPin.destroy();
});
test('wrong passphrase → different masterKey', async () => {
const right = await KeyManager.open({
kind: 'composite',
sources: [pwSource('correct horse'), pinSource('1234')],
});
const wrongPwd = await KeyManager.open({
kind: 'composite',
sources: [pwSource('wrong horse'), pinSource('1234')],
});
expect(right.storageKeyFingerprint()).not.toEqual(wrongPwd.storageKeyFingerprint());
right.destroy();
wrongPwd.destroy();
});
test('order is significant by design', async () => {
const ab = await KeyManager.open({
kind: 'composite',
sources: [pwSource('horse'), pinSource('1234')],
});
const ba = await KeyManager.open({
kind: 'composite',
sources: [pinSource('1234'), pwSource('horse')],
});
expect(ab.storageKeyFingerprint()).not.toEqual(ba.storageKeyFingerprint());
ab.destroy();
ba.destroy();
});
test('explicit info string changes masterKey (domain separation)', async () => {
const a = await KeyManager.open({
kind: 'composite',
sources: [pwSource('horse'), pinSource('1234')],
});
const b = await KeyManager.open({
kind: 'composite',
sources: [pwSource('horse'), pinSource('1234')],
info: 'my-app-v1',
});
expect(a.storageKeyFingerprint()).not.toEqual(b.storageKeyFingerprint());
a.destroy();
b.destroy();
});
test('rejects empty source list', async () => {
await expect(KeyManager.open({ kind: 'composite', sources: [] }))
.rejects.toThrow(/at least one/);
});
test('rejects nested composite', async () => {
await expect(KeyManager.open({
kind: 'composite',
sources: [
{ kind: 'composite', sources: [pwSource('a')] },
pinSource('1234'),
],
})).rejects.toThrow(/cannot be nested/);
});
test('composite of three sources works', async () => {
const km = await KeyManager.open({
kind: 'composite',
sources: [
pwSource('horse'),
pinSource('1234'),
{ kind: 'injected', key: randBytes(32) },
],
});
expect(km.fieldKey('t', 'c').length).toBe(32);
km.destroy();
});
});

View File

@@ -1,5 +1,5 @@
{ {
"extends": "../../tsconfig.json", "extends": "../../tsconfig.json",
"compilerOptions": { "outDir": "dist", "rootDir": "src" }, "compilerOptions": { "outDir": "dist", "rootDir": "src", "lib": ["ES2022", "DOM"] },
"include": ["src"] "include": ["src"]
} }

View File

@@ -0,0 +1,15 @@
{
"name": "@shade/storage-indexeddb",
"version": "4.6.0",
"type": "module",
"main": "src/index.ts",
"types": "src/index.ts",
"dependencies": {
"@shade/core": "workspace:*",
"idb": "^8.0.3"
},
"devDependencies": {
"@shade/crypto-web": "workspace:*",
"fake-indexeddb": "^6.0.0"
}
}

View File

@@ -0,0 +1 @@
export { IndexedDBStorage } from './indexeddb-storage.js';

View File

@@ -0,0 +1,555 @@
import { openDB, type IDBPDatabase, type DBSchema } from 'idb';
import type {
StorageProvider,
IdentityKeyPair,
SignedPreKey,
OneTimePreKey,
SessionState,
RetiredIdentity,
PersistedStreamState,
PeerVerification,
PeerVerificationSource,
BroadcastChannelRecord,
BroadcastMemberRecord,
} from '@shade/core';
import {
toBase64, fromBase64,
constantTimeEqual,
serializeSessionState, deserializeSessionState,
serializeSignedPreKey, deserializeSignedPreKey,
serializeOneTimePreKey, deserializeOneTimePreKey,
serializeIdentityKeyPair, deserializeIdentityKeyPair,
} from '@shade/core';
/**
* IndexedDB-backed StorageProvider for browser-side Shade clients.
*
* Persists identity, prekeys, sessions, retired identities, peer
* verifications and stream-resume state across tab refresh and browser
* restart. Same data shapes as `@shade/storage-sqlite` so cross-adapter
* import/export remains feasible.
*
* Usage:
* ```ts
* const storage = await IndexedDBStorage.create({ dbName: 'my-app-shade' });
* const manager = new ShadeSessionManager(crypto, storage);
* ```
*/
export class IndexedDBStorage implements StorageProvider {
private constructor(private db: IDBPDatabase<ShadeSchema>) {}
/**
* Open (or create) the IndexedDB database. Idempotent — repeated calls
* with the same dbName resolve to a fresh connection sharing the same
* underlying object stores.
*/
static async create(opts: { dbName?: string } = {}): Promise<IndexedDBStorage> {
const dbName = opts.dbName ?? 'shade';
const db = await openDB<ShadeSchema>(dbName, SCHEMA_VERSION, {
upgrade(db, oldVersion) {
if (oldVersion < 1) {
db.createObjectStore('identity', { keyPath: 'id' });
db.createObjectStore('config', { keyPath: 'key' });
db.createObjectStore('signedPreKeys', { keyPath: 'keyId' });
db.createObjectStore('oneTimePreKeys', { keyPath: 'keyId' });
db.createObjectStore('sessions', { keyPath: 'address' });
db.createObjectStore('trustedIdentities', { keyPath: 'address' });
const retired = db.createObjectStore('retiredIdentities', {
keyPath: 'id',
autoIncrement: true,
});
retired.createIndex('byRetiredAt', 'retiredAt');
const stream = db.createObjectStore('streamStates', { keyPath: 'streamId' });
stream.createIndex('byStatus', 'status');
stream.createIndex('byPeerAddress', 'peerAddress');
stream.createIndex('byUpdatedAt', 'updatedAt');
db.createObjectStore('peerVerifications', { keyPath: 'peerAddress' });
db.createObjectStore('peerIdentityVersions', { keyPath: 'peerAddress' });
}
if (oldVersion < 2) {
db.createObjectStore('broadcastChannels', { keyPath: 'channelId' });
const members = db.createObjectStore('broadcastMembers', { keyPath: ['channelId', 'peerAddress'] });
members.createIndex('byChannelId', 'channelId');
}
},
});
return new IndexedDBStorage(db);
}
/** Cleanly close the underlying connection. */
async close(): Promise<void> {
this.db.close();
}
// ─── Identity ──────────────────────────────────────────────
async getIdentityKeyPair(): Promise<IdentityKeyPair | null> {
const row = await this.db.get('identity', 1);
if (!row) return null;
return {
signingPublicKey: fromBase64(row.signingPublicKey),
signingPrivateKey: fromBase64(row.signingPrivateKey),
dhPublicKey: fromBase64(row.dhPublicKey),
dhPrivateKey: fromBase64(row.dhPrivateKey),
};
}
async saveIdentityKeyPair(kp: IdentityKeyPair): Promise<void> {
await this.db.put('identity', {
id: 1,
signingPublicKey: toBase64(kp.signingPublicKey),
signingPrivateKey: toBase64(kp.signingPrivateKey),
dhPublicKey: toBase64(kp.dhPublicKey),
dhPrivateKey: toBase64(kp.dhPrivateKey),
});
}
async getLocalRegistrationId(): Promise<number> {
const row = await this.db.get('config', 'registrationId');
return row ? parseInt(row.value, 10) : 0;
}
async saveLocalRegistrationId(id: number): Promise<void> {
await this.db.put('config', { key: 'registrationId', value: String(id) });
}
// ─── Signed PreKeys ───────────────────────────────────────
async getSignedPreKey(keyId: number): Promise<SignedPreKey | null> {
const row = await this.db.get('signedPreKeys', keyId);
if (!row) return null;
return deserializeSignedPreKey(row.dataJson);
}
async saveSignedPreKey(key: SignedPreKey): Promise<void> {
await this.db.put('signedPreKeys', {
keyId: key.keyId,
dataJson: serializeSignedPreKey(key),
});
}
async removeSignedPreKey(keyId: number): Promise<void> {
await this.db.delete('signedPreKeys', keyId);
}
// ─── One-Time PreKeys ─────────────────────────────────────
async getOneTimePreKey(keyId: number): Promise<OneTimePreKey | null> {
const row = await this.db.get('oneTimePreKeys', keyId);
if (!row) return null;
return deserializeOneTimePreKey(row.dataJson);
}
async saveOneTimePreKey(key: OneTimePreKey): Promise<void> {
await this.db.put('oneTimePreKeys', {
keyId: key.keyId,
dataJson: serializeOneTimePreKey(key),
});
}
async removeOneTimePreKey(keyId: number): Promise<void> {
await this.db.delete('oneTimePreKeys', keyId);
}
async getOneTimePreKeyCount(): Promise<number> {
return this.db.count('oneTimePreKeys');
}
// ─── Sessions ─────────────────────────────────────────────
async getSession(address: string): Promise<SessionState | null> {
const row = await this.db.get('sessions', address);
if (!row) return null;
return deserializeSessionState(row.stateJson);
}
async saveSession(address: string, state: SessionState): Promise<void> {
await this.db.put('sessions', { address, stateJson: serializeSessionState(state) });
}
async removeSession(address: string): Promise<void> {
await this.db.delete('sessions', address);
}
// ─── Trust ────────────────────────────────────────────────
async isTrustedIdentity(address: string, identityKey: Uint8Array): Promise<boolean> {
const row = await this.db.get('trustedIdentities', address);
if (!row) return true; // TOFU
const stored = fromBase64(row.identityKey);
return constantTimeEqual(stored, identityKey);
}
async saveTrustedIdentity(address: string, identityKey: Uint8Array): Promise<void> {
await this.db.put('trustedIdentities', { address, identityKey: toBase64(identityKey) });
}
// ─── Identity History ─────────────────────────────────────
async addRetiredIdentity(identity: RetiredIdentity): Promise<void> {
// autoIncrement: omit `id` so IDB assigns one
await this.db.add('retiredIdentities', {
dataJson: serializeIdentityKeyPair(identity.keyPair),
retiredAt: identity.retiredAt,
} as RetiredIdentityRow);
}
async getRetiredIdentities(): Promise<RetiredIdentity[]> {
// Mirror SQLite's `ORDER BY retired_at DESC`
const rows = await this.db.getAllFromIndex('retiredIdentities', 'byRetiredAt');
rows.reverse();
return rows.map((r) => ({
keyPair: deserializeIdentityKeyPair(r.dataJson),
retiredAt: r.retiredAt,
}));
}
async pruneRetiredIdentities(olderThan: number): Promise<void> {
const tx = this.db.transaction('retiredIdentities', 'readwrite');
const idx = tx.store.index('byRetiredAt');
const range = IDBKeyRange.upperBound(olderThan, true);
let cursor = await idx.openCursor(range);
while (cursor) {
await cursor.delete();
cursor = await cursor.continue();
}
await tx.done;
}
// ─── Stream-transfer resume state ─────────────────────────
async saveStreamState(state: PersistedStreamState): Promise<void> {
await this.db.put('streamStates', persistedToRow(state));
}
async getStreamState(streamId: string): Promise<PersistedStreamState | null> {
const row = await this.db.get('streamStates', streamId);
if (!row) return null;
return rowToPersisted(row);
}
async removeStreamState(streamId: string): Promise<void> {
await this.db.delete('streamStates', streamId);
}
async listActiveStreamStates(direction?: 'send' | 'receive'): Promise<PersistedStreamState[]> {
const idx = this.db.transaction('streamStates').store.index('byStatus');
const active = await idx.getAll(IDBKeyRange.only('active'));
const paused = await idx.getAll(IDBKeyRange.only('paused'));
const merged = [...active, ...paused];
const filtered = direction === undefined
? merged
: merged.filter((r) => r.direction === direction);
filtered.sort((a, b) => b.updatedAt - a.updatedAt);
return filtered.map(rowToPersisted);
}
async pruneStreamStates(olderThan: number): Promise<void> {
const tx = this.db.transaction('streamStates', 'readwrite');
const idx = tx.store.index('byUpdatedAt');
const range = IDBKeyRange.upperBound(olderThan, true);
let cursor = await idx.openCursor(range);
while (cursor) {
const row = cursor.value;
if (row.status === 'finished' || row.status === 'aborted') {
await cursor.delete();
}
cursor = await cursor.continue();
}
await tx.done;
}
// ─── Peer verifications (V3.3) ────────────────────────────
async savePeerVerification(v: PeerVerification): Promise<void> {
await this.db.put('peerVerifications', { ...v });
}
async getPeerVerification(address: string): Promise<PeerVerification | null> {
const row = await this.db.get('peerVerifications', address);
if (!row) return null;
return {
peerAddress: row.peerAddress,
fingerprint: row.fingerprint,
verifiedAt: row.verifiedAt,
verifiedBy: row.verifiedBy as PeerVerificationSource,
identityVersion: row.identityVersion,
};
}
async removePeerVerification(address: string): Promise<void> {
await this.db.delete('peerVerifications', address);
}
async getPeerIdentityVersion(address: string): Promise<number> {
const row = await this.db.get('peerIdentityVersions', address);
return row ? row.version : 1;
}
/**
* Atomic read-modify-write under one IDB transaction. SQLite's version
* is a non-atomic read-then-upsert; the IDB version closes that race
* because IDB transactions auto-commit only when control returns to
* the event loop without pending requests.
*/
async bumpPeerIdentityVersion(address: string): Promise<number> {
const tx = this.db.transaction('peerIdentityVersions', 'readwrite');
const existing = await tx.store.get(address);
const next = (existing ? existing.version : 1) + 1;
await tx.store.put({ peerAddress: address, version: next });
await tx.done;
return next;
}
// ─── Broadcast channels (V4.6) ────────────────────────────
async saveBroadcastChannel(channel: BroadcastChannelRecord): Promise<void> {
await this.db.put('broadcastChannels', channelToRow(channel));
}
async getBroadcastChannel(channelId: string): Promise<BroadcastChannelRecord | null> {
const row = await this.db.get('broadcastChannels', channelId);
if (!row) return null;
return rowToChannel(row);
}
async listBroadcastChannels(): Promise<BroadcastChannelRecord[]> {
const rows = await this.db.getAll('broadcastChannels');
rows.sort((a, b) => a.createdAt - b.createdAt);
return rows.map(rowToChannel);
}
async removeBroadcastChannel(channelId: string): Promise<void> {
const tx = this.db.transaction(['broadcastChannels', 'broadcastMembers'], 'readwrite');
const memIdx = tx.objectStore('broadcastMembers').index('byChannelId');
let cursor = await memIdx.openCursor(IDBKeyRange.only(channelId));
while (cursor) {
await cursor.delete();
cursor = await cursor.continue();
}
await tx.objectStore('broadcastChannels').delete(channelId);
await tx.done;
}
async saveBroadcastMember(member: BroadcastMemberRecord): Promise<void> {
await this.db.put('broadcastMembers', { ...member });
}
async getBroadcastMembers(channelId: string): Promise<BroadcastMemberRecord[]> {
const rows = await this.db.getAllFromIndex(
'broadcastMembers',
'byChannelId',
IDBKeyRange.only(channelId),
);
rows.sort((a, b) => a.joinedAt - b.joinedAt);
return rows.map((r) => ({
channelId: r.channelId,
peerAddress: r.peerAddress,
joinedAt: r.joinedAt,
removedAt: r.removedAt,
}));
}
async removeBroadcastMember(channelId: string, peerAddress: string): Promise<void> {
await this.db.delete('broadcastMembers', [channelId, peerAddress]);
}
}
// ─── Schema ────────────────────────────────────────────────
const SCHEMA_VERSION = 2;
interface IdentityRow {
id: 1;
signingPublicKey: string;
signingPrivateKey: string;
dhPublicKey: string;
dhPrivateKey: string;
}
interface ConfigRow {
key: string;
value: string;
}
interface SignedPreKeyRow {
keyId: number;
dataJson: string;
}
interface OneTimePreKeyRow {
keyId: number;
dataJson: string;
}
interface SessionRow {
address: string;
stateJson: string;
}
interface TrustedIdentityRow {
address: string;
identityKey: string;
}
interface RetiredIdentityRow {
id?: number;
dataJson: string;
retiredAt: number;
}
interface StreamStateRow {
streamId: string;
direction: 'send' | 'receive';
peerAddress: string;
status: 'active' | 'paused' | 'finished' | 'aborted';
metadataJson: string;
partitionJson: string;
laneStateJson: string;
ioDescriptorJson: string;
secretEnc: Uint8Array;
secretNonce: Uint8Array;
overallHashState: string | null;
createdAt: number;
updatedAt: number;
}
interface PeerVerificationRow {
peerAddress: string;
fingerprint: string;
verifiedAt: number;
verifiedBy: string;
identityVersion: number;
}
interface PeerIdentityVersionRow {
peerAddress: string;
version: number;
}
interface BroadcastChannelRow {
channelId: string;
ownerRole: 'sender' | 'receiver';
ownerAddress: string;
label: string | null;
generation: number;
chainKey: Uint8Array;
iteration: number;
signingPublicKey: Uint8Array;
signingPrivateKey: Uint8Array | null;
createdAt: number;
updatedAt: number;
}
interface BroadcastMemberRow {
channelId: string;
peerAddress: string;
joinedAt: number;
removedAt: number | null;
}
interface ShadeSchema extends DBSchema {
identity: { key: number; value: IdentityRow };
config: { key: string; value: ConfigRow };
signedPreKeys: { key: number; value: SignedPreKeyRow };
oneTimePreKeys: { key: number; value: OneTimePreKeyRow };
sessions: { key: string; value: SessionRow };
trustedIdentities: { key: string; value: TrustedIdentityRow };
retiredIdentities: {
key: number;
value: RetiredIdentityRow;
indexes: { byRetiredAt: number };
};
streamStates: {
key: string;
value: StreamStateRow;
indexes: {
byStatus: string;
byPeerAddress: string;
byUpdatedAt: number;
};
};
peerVerifications: { key: string; value: PeerVerificationRow };
peerIdentityVersions: { key: string; value: PeerIdentityVersionRow };
broadcastChannels: { key: string; value: BroadcastChannelRow };
broadcastMembers: {
key: [string, string];
value: BroadcastMemberRow;
indexes: { byChannelId: string };
};
}
// ─── Helpers ──────────────────────────────────────────────
function persistedToRow(s: PersistedStreamState): StreamStateRow {
return {
streamId: s.streamId,
direction: s.direction,
peerAddress: s.peerAddress,
status: s.status,
metadataJson: s.metadataJson,
partitionJson: s.partitionJson,
laneStateJson: s.laneStateJson,
ioDescriptorJson: s.ioDescriptorJson,
secretEnc: s.secretEnc,
secretNonce: s.secretNonce,
overallHashState: s.overallHashState ?? null,
createdAt: s.createdAt,
updatedAt: s.updatedAt,
};
}
function channelToRow(c: BroadcastChannelRecord): BroadcastChannelRow {
return {
channelId: c.channelId,
ownerRole: c.ownerRole,
ownerAddress: c.ownerAddress,
label: c.label ?? null,
generation: c.generation,
chainKey: c.chainKey,
iteration: c.iteration,
signingPublicKey: c.signingPublicKey,
signingPrivateKey: c.signingPrivateKey ?? null,
createdAt: c.createdAt,
updatedAt: c.updatedAt,
};
}
function rowToChannel(r: BroadcastChannelRow): BroadcastChannelRecord {
const out: BroadcastChannelRecord = {
channelId: r.channelId,
ownerRole: r.ownerRole,
ownerAddress: r.ownerAddress,
generation: r.generation,
chainKey: r.chainKey,
iteration: r.iteration,
signingPublicKey: r.signingPublicKey,
createdAt: r.createdAt,
updatedAt: r.updatedAt,
};
if (r.label !== null) out.label = r.label;
if (r.signingPrivateKey !== null) out.signingPrivateKey = r.signingPrivateKey;
return out;
}
function rowToPersisted(r: StreamStateRow): PersistedStreamState {
const out: PersistedStreamState = {
streamId: r.streamId,
direction: r.direction,
peerAddress: r.peerAddress,
status: r.status,
metadataJson: r.metadataJson,
partitionJson: r.partitionJson,
laneStateJson: r.laneStateJson,
ioDescriptorJson: r.ioDescriptorJson,
secretEnc: r.secretEnc,
secretNonce: r.secretNonce,
createdAt: r.createdAt,
updatedAt: r.updatedAt,
};
if (r.overallHashState !== null) out.overallHashState = r.overallHashState;
return out;
}

View File

@@ -0,0 +1,272 @@
import 'fake-indexeddb/auto';
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { IndexedDBStorage } from '../src/indexeddb-storage.js';
import { SubtleCryptoProvider, MemoryStorage as _MemoryStorage } from '@shade/crypto-web';
import { ShadeSessionManager } from '@shade/core';
import type { IdentityKeyPair, SignedPreKey, OneTimePreKey } from '@shade/core';
void _MemoryStorage; // keep import side-effect-free reference shape
const crypto = new SubtleCryptoProvider();
function randBytes(n: number): Uint8Array {
const buf = new Uint8Array(n);
globalThis.crypto.getRandomValues(buf);
return buf;
}
function uniqueDbName(): string {
return `shade-test-${Date.now()}-${Math.random().toString(36).slice(2)}`;
}
async function deleteDb(dbName: string): Promise<void> {
await new Promise<void>((resolve, reject) => {
const req = indexedDB.deleteDatabase(dbName);
req.onsuccess = () => resolve();
req.onerror = () => reject(req.error);
req.onblocked = () => resolve();
});
}
describe('IndexedDBStorage', () => {
let dbName: string;
let storage: IndexedDBStorage;
beforeEach(async () => {
dbName = uniqueDbName();
storage = await IndexedDBStorage.create({ dbName });
});
afterEach(async () => {
await storage.close();
await deleteDb(dbName);
});
// ─── Identity ──────────────────────────────────────────────
describe('identity', () => {
test('returns null when no identity stored', async () => {
expect(await storage.getIdentityKeyPair()).toBeNull();
});
test('save and retrieve identity keypair', async () => {
const ikp: IdentityKeyPair = {
signingPublicKey: randBytes(32),
signingPrivateKey: randBytes(32),
dhPublicKey: randBytes(32),
dhPrivateKey: randBytes(32),
};
await storage.saveIdentityKeyPair(ikp);
const restored = await storage.getIdentityKeyPair();
expect(restored).not.toBeNull();
expect(restored!.signingPublicKey).toEqual(ikp.signingPublicKey);
expect(restored!.dhPrivateKey).toEqual(ikp.dhPrivateKey);
});
test('registration ID roundtrip', async () => {
expect(await storage.getLocalRegistrationId()).toBe(0);
await storage.saveLocalRegistrationId(42);
expect(await storage.getLocalRegistrationId()).toBe(42);
});
});
// ─── Signed PreKeys ───────────────────────────────────────
describe('signed prekeys', () => {
test('save, get, remove', async () => {
const spk: SignedPreKey = {
keyId: 1,
keyPair: { publicKey: randBytes(32), privateKey: randBytes(32) },
signature: randBytes(64),
timestamp: Date.now(),
};
await storage.saveSignedPreKey(spk);
const restored = await storage.getSignedPreKey(1);
expect(restored).not.toBeNull();
expect(restored!.keyId).toBe(1);
expect(restored!.keyPair.publicKey).toEqual(spk.keyPair.publicKey);
await storage.removeSignedPreKey(1);
expect(await storage.getSignedPreKey(1)).toBeNull();
});
});
// ─── One-Time PreKeys ─────────────────────────────────────
describe('one-time prekeys', () => {
test('save, get, remove, count', async () => {
const otpk: OneTimePreKey = {
keyId: 100,
keyPair: { publicKey: randBytes(32), privateKey: randBytes(32) },
};
await storage.saveOneTimePreKey(otpk);
expect(await storage.getOneTimePreKeyCount()).toBe(1);
const restored = await storage.getOneTimePreKey(100);
expect(restored!.keyId).toBe(100);
await storage.removeOneTimePreKey(100);
expect(await storage.getOneTimePreKeyCount()).toBe(0);
expect(await storage.getOneTimePreKey(100)).toBeNull();
});
});
// ─── Sessions ─────────────────────────────────────────────
describe('sessions', () => {
test('save and restore session with skipped keys', async () => {
const state = {
remoteIdentityKey: randBytes(32),
rootKey: randBytes(32),
sendChain: { chainKey: randBytes(32), counter: 5 },
receiveChain: { chainKey: randBytes(32), counter: 3 },
dhSend: { publicKey: randBytes(32), privateKey: randBytes(32) },
dhReceive: randBytes(32),
previousSendCounter: 4,
skippedKeys: new Map([['key:1', randBytes(32)]]),
};
await storage.saveSession('bob', state);
const restored = await storage.getSession('bob');
expect(restored).not.toBeNull();
expect(restored!.sendChain.counter).toBe(5);
expect(restored!.skippedKeys.size).toBe(1);
expect(restored!.skippedKeys.get('key:1')).toEqual(state.skippedKeys.get('key:1')!);
});
test('remove session', async () => {
await storage.saveSession('bob', {
remoteIdentityKey: randBytes(32),
rootKey: randBytes(32),
sendChain: { chainKey: randBytes(32), counter: 0 },
receiveChain: null,
dhSend: { publicKey: randBytes(32), privateKey: randBytes(32) },
dhReceive: null,
previousSendCounter: 0,
skippedKeys: new Map(),
});
await storage.removeSession('bob');
expect(await storage.getSession('bob')).toBeNull();
});
});
// ─── Trust ────────────────────────────────────────────────
describe('trust', () => {
test('TOFU: first use is trusted', async () => {
expect(await storage.isTrustedIdentity('bob', randBytes(32))).toBe(true);
});
test('saved identity matches', async () => {
const key = randBytes(32);
await storage.saveTrustedIdentity('bob', key);
expect(await storage.isTrustedIdentity('bob', key)).toBe(true);
expect(await storage.isTrustedIdentity('bob', randBytes(32))).toBe(false);
});
});
// ─── Retired identities ───────────────────────────────────
describe('retired identities', () => {
test('add, list (newest first), prune', async () => {
const mk = (retiredAt: number): { keyPair: IdentityKeyPair; retiredAt: number } => ({
keyPair: {
signingPublicKey: randBytes(32),
signingPrivateKey: randBytes(32),
dhPublicKey: randBytes(32),
dhPrivateKey: randBytes(32),
},
retiredAt,
});
await storage.addRetiredIdentity(mk(100));
await storage.addRetiredIdentity(mk(300));
await storage.addRetiredIdentity(mk(200));
const list = await storage.getRetiredIdentities();
expect(list.map((r) => r.retiredAt)).toEqual([300, 200, 100]);
await storage.pruneRetiredIdentities(250);
const after = await storage.getRetiredIdentities();
expect(after.map((r) => r.retiredAt)).toEqual([300]);
});
});
// ─── Crash Recovery ───────────────────────────────────────
describe('persistence across close/reopen', () => {
test('data survives close and reopen', async () => {
const ikp: IdentityKeyPair = {
signingPublicKey: randBytes(32),
signingPrivateKey: randBytes(32),
dhPublicKey: randBytes(32),
dhPrivateKey: randBytes(32),
};
await storage.saveIdentityKeyPair(ikp);
await storage.saveLocalRegistrationId(99);
await storage.saveSession('alice', {
remoteIdentityKey: randBytes(32),
rootKey: randBytes(32),
sendChain: { chainKey: randBytes(32), counter: 7 },
receiveChain: null,
dhSend: { publicKey: randBytes(32), privateKey: randBytes(32) },
dhReceive: null,
previousSendCounter: 0,
skippedKeys: new Map(),
});
// Close and reopen against the same dbName
await storage.close();
storage = await IndexedDBStorage.create({ dbName });
const restored = await storage.getIdentityKeyPair();
expect(restored!.signingPublicKey).toEqual(ikp.signingPublicKey);
expect(await storage.getLocalRegistrationId()).toBe(99);
const session = await storage.getSession('alice');
expect(session!.sendChain.counter).toBe(7);
});
});
// ─── Full E2EE with IndexedDBStorage ──────────────────────
describe('full E2EE conversation with persistent storage', () => {
test('encrypt, close, reopen, continue conversation', async () => {
const bobDbName = uniqueDbName();
let bobStorage = await IndexedDBStorage.create({ dbName: bobDbName });
try {
const alice = new ShadeSessionManager(crypto, storage);
let bob = new ShadeSessionManager(crypto, bobStorage);
await alice.initialize();
await bob.initialize();
const otpks = await bob.generateOneTimePreKeys(5);
const bundle = await bob.createPreKeyBundle();
bundle.oneTimePreKey = { keyId: otpks[0]!.keyId, publicKey: otpks[0]!.keyPair.publicKey };
await alice.initSessionFromBundle('bob', bundle);
const env1 = await alice.encrypt('bob', 'Hello persistent!');
expect(await bob.decrypt('alice', env1)).toBe('Hello persistent!');
const env2 = await bob.encrypt('alice', 'Got it!');
expect(await alice.decrypt('bob', env2)).toBe('Got it!');
// "Crash" Bob — close the IDB and reopen
await bobStorage.close();
bobStorage = await IndexedDBStorage.create({ dbName: bobDbName });
bob = new ShadeSessionManager(crypto, bobStorage);
await bob.initialize();
const env3 = await alice.encrypt('bob', 'After your restart');
expect(await bob.decrypt('alice', env3)).toBe('After your restart');
const env4 = await bob.encrypt('alice', 'I survived!');
expect(await alice.decrypt('bob', env4)).toBe('I survived!');
} finally {
await bobStorage.close();
await deleteDb(bobDbName);
}
});
});
});

View File

@@ -0,0 +1,99 @@
import 'fake-indexeddb/auto';
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { IndexedDBStorage } from '../src/index.js';
function uniqueDbName(): string {
return `shade-test-${Date.now()}-${Math.random().toString(36).slice(2)}`;
}
async function deleteDb(dbName: string): Promise<void> {
await new Promise<void>((resolve, reject) => {
const req = indexedDB.deleteDatabase(dbName);
req.onsuccess = () => resolve();
req.onerror = () => reject(req.error);
req.onblocked = () => resolve();
});
}
describe('IndexedDBStorage — peer_verifications (V3.3)', () => {
let dbName: string;
let storage: IndexedDBStorage;
beforeEach(async () => {
dbName = uniqueDbName();
storage = await IndexedDBStorage.create({ dbName });
});
afterEach(async () => {
await storage.close();
await deleteDb(dbName);
});
test('round trip: save → get → remove', async () => {
await storage.savePeerVerification({
peerAddress: 'bob',
fingerprint: '12345 67890 12345 67890 12345 67890 12345 67890 12345 67890 12345 67890',
verifiedAt: 1_700_000_000_000,
verifiedBy: 'user',
identityVersion: 1,
});
const v = await storage.getPeerVerification('bob');
expect(v).not.toBeNull();
expect(v!.peerAddress).toBe('bob');
expect(v!.verifiedBy).toBe('user');
expect(v!.identityVersion).toBe(1);
await storage.removePeerVerification('bob');
expect(await storage.getPeerVerification('bob')).toBeNull();
});
test('upsert overwrites on duplicate peer_address', async () => {
await storage.savePeerVerification({
peerAddress: 'bob',
fingerprint: 'fp-1',
verifiedAt: 1,
verifiedBy: 'user',
identityVersion: 1,
});
await storage.savePeerVerification({
peerAddress: 'bob',
fingerprint: 'fp-2',
verifiedAt: 2,
verifiedBy: 'transitive',
identityVersion: 2,
});
const v = await storage.getPeerVerification('bob');
expect(v!.fingerprint).toBe('fp-2');
expect(v!.verifiedBy).toBe('transitive');
expect(v!.identityVersion).toBe(2);
});
test('identity-version starts at 1 and increments via bump', async () => {
expect(await storage.getPeerIdentityVersion('alice')).toBe(1);
expect(await storage.bumpPeerIdentityVersion('alice')).toBe(2);
expect(await storage.bumpPeerIdentityVersion('alice')).toBe(3);
expect(await storage.getPeerIdentityVersion('alice')).toBe(3);
// Independent counter per peer
expect(await storage.getPeerIdentityVersion('bob')).toBe(1);
});
test('survives reopen', async () => {
await storage.savePeerVerification({
peerAddress: 'bob',
fingerprint: 'fp',
verifiedAt: 42,
verifiedBy: 'user',
identityVersion: 5,
});
await storage.bumpPeerIdentityVersion('bob');
await storage.close();
storage = await IndexedDBStorage.create({ dbName });
const v = await storage.getPeerVerification('bob');
expect(v!.fingerprint).toBe('fp');
expect(v!.identityVersion).toBe(5);
expect(await storage.getPeerIdentityVersion('bob')).toBe(2);
});
});

View File

@@ -0,0 +1,9 @@
{
"extends": "../../tsconfig.json",
"compilerOptions": {
"outDir": "dist",
"rootDir": "src",
"lib": ["ES2022", "DOM"]
},
"include": ["src"]
}

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/storage-postgres", "name": "@shade/storage-postgres",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/storage-sqlite", "name": "@shade/storage-sqlite",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,5 +1,9 @@
import { Database } from 'bun:sqlite'; import { Database } from 'bun:sqlite';
import type { StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey, SessionState, RetiredIdentity, PersistedStreamState, PeerVerification, PeerVerificationSource } from '@shade/core'; import type {
StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey,
SessionState, RetiredIdentity, PersistedStreamState, PeerVerification,
PeerVerificationSource, BroadcastChannelRecord, BroadcastMemberRecord,
} from '@shade/core';
import { import {
toBase64, fromBase64, toBase64, fromBase64,
constantTimeEqual, constantTimeEqual,
@@ -53,6 +57,14 @@ export class SQLiteStorage implements StorageProvider {
removePeerVerification: ReturnType<Database['prepare']>; removePeerVerification: ReturnType<Database['prepare']>;
getPeerIdentityVersion: ReturnType<Database['prepare']>; getPeerIdentityVersion: ReturnType<Database['prepare']>;
upsertPeerIdentityVersion: ReturnType<Database['prepare']>; upsertPeerIdentityVersion: ReturnType<Database['prepare']>;
saveBroadcastChannel: ReturnType<Database['prepare']>;
getBroadcastChannel: ReturnType<Database['prepare']>;
listBroadcastChannels: ReturnType<Database['prepare']>;
removeBroadcastChannel: ReturnType<Database['prepare']>;
removeBroadcastChannelMembers: ReturnType<Database['prepare']>;
saveBroadcastMember: ReturnType<Database['prepare']>;
getBroadcastMembers: ReturnType<Database['prepare']>;
removeBroadcastMember: ReturnType<Database['prepare']>;
}; };
constructor(dbPath?: string) { constructor(dbPath?: string) {
@@ -127,6 +139,27 @@ export class SQLiteStorage implements StorageProvider {
peer_address TEXT PRIMARY KEY, peer_address TEXT PRIMARY KEY,
version INTEGER NOT NULL version INTEGER NOT NULL
); );
CREATE TABLE IF NOT EXISTS broadcast_channels (
channel_id TEXT PRIMARY KEY,
owner_role TEXT NOT NULL,
owner_address TEXT NOT NULL,
label TEXT,
generation INTEGER NOT NULL,
chain_key BLOB NOT NULL,
iteration INTEGER NOT NULL,
signing_public_key BLOB NOT NULL,
signing_private_key BLOB,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS broadcast_members (
channel_id TEXT NOT NULL,
peer_address TEXT NOT NULL,
joined_at INTEGER NOT NULL,
removed_at INTEGER,
PRIMARY KEY (channel_id, peer_address)
);
CREATE INDEX IF NOT EXISTS idx_broadcast_members_channel ON broadcast_members(channel_id);
`); `);
} }
@@ -183,6 +216,36 @@ export class SQLiteStorage implements StorageProvider {
`INSERT INTO peer_identity_versions (peer_address, version) VALUES (?, ?) `INSERT INTO peer_identity_versions (peer_address, version) VALUES (?, ?)
ON CONFLICT(peer_address) DO UPDATE SET version = excluded.version`, ON CONFLICT(peer_address) DO UPDATE SET version = excluded.version`,
), ),
saveBroadcastChannel: this.db.prepare(
`INSERT OR REPLACE INTO broadcast_channels
(channel_id, owner_role, owner_address, label, generation,
chain_key, iteration, signing_public_key, signing_private_key,
created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
),
getBroadcastChannel: this.db.prepare(
'SELECT * FROM broadcast_channels WHERE channel_id = ?',
),
listBroadcastChannels: this.db.prepare(
'SELECT * FROM broadcast_channels ORDER BY created_at ASC',
),
removeBroadcastChannel: this.db.prepare(
'DELETE FROM broadcast_channels WHERE channel_id = ?',
),
removeBroadcastChannelMembers: this.db.prepare(
'DELETE FROM broadcast_members WHERE channel_id = ?',
),
saveBroadcastMember: this.db.prepare(
`INSERT OR REPLACE INTO broadcast_members
(channel_id, peer_address, joined_at, removed_at)
VALUES (?, ?, ?, ?)`,
),
getBroadcastMembers: this.db.prepare(
'SELECT channel_id, peer_address, joined_at, removed_at FROM broadcast_members WHERE channel_id = ? ORDER BY joined_at ASC',
),
removeBroadcastMember: this.db.prepare(
'DELETE FROM broadcast_members WHERE channel_id = ? AND peer_address = ?',
),
}; };
} }
@@ -392,6 +455,103 @@ export class SQLiteStorage implements StorageProvider {
this.stmts.upsertPeerIdentityVersion.run(address, next); this.stmts.upsertPeerIdentityVersion.run(address, next);
return next; return next;
} }
// ─── Broadcast channels (V4.6) ────────────────────────────
async saveBroadcastChannel(channel: BroadcastChannelRecord): Promise<void> {
this.stmts.saveBroadcastChannel.run(
channel.channelId,
channel.ownerRole,
channel.ownerAddress,
channel.label ?? null,
channel.generation,
channel.chainKey,
channel.iteration,
channel.signingPublicKey,
channel.signingPrivateKey ?? null,
channel.createdAt,
channel.updatedAt,
);
}
async getBroadcastChannel(channelId: string): Promise<BroadcastChannelRecord | null> {
const row = this.stmts.getBroadcastChannel.get(channelId) as BroadcastChannelRow | undefined;
if (!row) return null;
return rowToBroadcastChannel(row);
}
async listBroadcastChannels(): Promise<BroadcastChannelRecord[]> {
const rows = this.stmts.listBroadcastChannels.all() as BroadcastChannelRow[];
return rows.map(rowToBroadcastChannel);
}
async removeBroadcastChannel(channelId: string): Promise<void> {
this.stmts.removeBroadcastChannelMembers.run(channelId);
this.stmts.removeBroadcastChannel.run(channelId);
}
async saveBroadcastMember(member: BroadcastMemberRecord): Promise<void> {
this.stmts.saveBroadcastMember.run(
member.channelId,
member.peerAddress,
member.joinedAt,
member.removedAt,
);
}
async getBroadcastMembers(channelId: string): Promise<BroadcastMemberRecord[]> {
const rows = this.stmts.getBroadcastMembers.all(channelId) as BroadcastMemberRow[];
return rows.map((r) => ({
channelId: r.channel_id,
peerAddress: r.peer_address,
joinedAt: Number(r.joined_at),
removedAt: r.removed_at === null || r.removed_at === undefined ? null : Number(r.removed_at),
}));
}
async removeBroadcastMember(channelId: string, peerAddress: string): Promise<void> {
this.stmts.removeBroadcastMember.run(channelId, peerAddress);
}
}
interface BroadcastChannelRow {
channel_id: string;
owner_role: 'sender' | 'receiver';
owner_address: string;
label: string | null;
generation: number | bigint;
chain_key: Uint8Array | ArrayBuffer;
iteration: number | bigint;
signing_public_key: Uint8Array | ArrayBuffer;
signing_private_key: Uint8Array | ArrayBuffer | null;
created_at: number | bigint;
updated_at: number | bigint;
}
interface BroadcastMemberRow {
channel_id: string;
peer_address: string;
joined_at: number | bigint;
removed_at: number | bigint | null;
}
function rowToBroadcastChannel(r: BroadcastChannelRow): BroadcastChannelRecord {
const out: BroadcastChannelRecord = {
channelId: r.channel_id,
ownerRole: r.owner_role,
ownerAddress: r.owner_address,
generation: Number(r.generation),
chainKey: toBytes(r.chain_key),
iteration: Number(r.iteration),
signingPublicKey: toBytes(r.signing_public_key),
createdAt: Number(r.created_at),
updatedAt: Number(r.updated_at),
};
if (r.label !== null && r.label !== undefined) out.label = r.label;
if (r.signing_private_key !== null && r.signing_private_key !== undefined) {
out.signingPrivateKey = toBytes(r.signing_private_key);
}
return out;
} }
function rowToStreamState(row: any): PersistedStreamState { function rowToStreamState(row: any): PersistedStreamState {

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/streams", "name": "@shade/streams",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/transfer", "name": "@shade/transfer",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -7,6 +7,7 @@ export * from './transport/memory.js';
export * from './transport/http-transport.js'; export * from './transport/http-transport.js';
export * from './transport/ws-transport.js'; export * from './transport/ws-transport.js';
export * from './transport/multi-fallback.js'; export * from './transport/multi-fallback.js';
export * from './transport/queue-transport.js';
export * from './engine.js'; export * from './engine.js';
export { export {
createTransferRoutes, createTransferRoutes,

View File

@@ -0,0 +1,332 @@
/**
* Per-peer outbound queue + queue-transport for browser-friendly
* pull-mode streams.
*
* The default `ShadeTransferHttpTransport` POSTs each chunk directly
* to the receiver's `/v1/transfer/<streamId>/chunk` route. That
* requires the receiver to host an HTTP server, which a browser tab
* cannot. `QueueTransferTransport` flips the direction: it queues
* chunks per peer and lets the receiver pull them via a long-poll
* endpoint.
*
* The companion `OutboundQueue` data structure is plain server-side
* state — wired up by `@shade/files`'s `transferQueueRoute()` (and any
* future consumer) to expose the long-poll surface and feed envelopes
* + chunks into the browser receiver.
*/
import type {
ChunkAck,
ChunkSendOptions,
ITransferTransport,
TransferResumeState,
} from './transport.js';
import { TransferTransportError } from '../errors.js';
/**
* Discriminated union of items the queue ships to the browser
* receiver. Both kinds carry **wire-encoded bytes** of an envelope —
* the receiver decodes via `decodeEnvelope` (control envelopes) or
* forwards directly to `engine.receiveChunk` (chunk envelopes).
*/
export type QueuedEvent =
| {
id: number;
timestampMs: number;
kind: 'envelope';
/** Wire-encoded `0x02` ratchet envelope (or `0x01` first-message). */
bytes: Uint8Array;
}
| {
id: number;
timestampMs: number;
kind: 'chunk';
/** Wire-encoded `0x11` stream-chunk envelope. */
bytes: Uint8Array;
meta: {
streamId: string;
laneId: number;
seq: number;
};
};
/** Caller-supplied shape for {@link OutboundQueue.enqueue} — the queue assigns `id` + `timestampMs`. */
export type QueuedEventInput =
| { kind: 'envelope'; bytes: Uint8Array }
| {
kind: 'chunk';
bytes: Uint8Array;
meta: { streamId: string; laneId: number; seq: number };
};
export interface OutboundQueueOptions {
/**
* Maximum events held per peer. When the queue is full, the oldest
* unacked event is dropped on next enqueue. Default 1000 — at the
* default chunk size (256 KiB plaintext) this caps a single peer's
* outbound buffer at ~256 MiB. Tune up for fewer/bigger streams,
* down for many concurrent small flows.
*/
maxEventsPerPeer?: number;
/**
* After a peer has not polled for this long, the queue's events are
* dropped and any pending waiters are released. Default 10 minutes.
* Setting to `0` disables idle-eviction.
*/
idleEvictionMs?: number;
}
const DEFAULT_MAX_EVENTS = 1000;
const DEFAULT_IDLE_EVICTION_MS = 10 * 60 * 1000;
interface PendingWaiter {
resolve(events: QueuedEvent[]): void;
reject(err: Error): void;
/**
* The waiter's `since` cursor — only events with `id > since` should
* be delivered when this waiter is resolved. Without this, an
* enqueue that arrives while a poller is waiting would replay
* already-processed events, causing the receiver to double-decrypt
* (and corrupt ratchet state).
*/
since: number;
timer: ReturnType<typeof setTimeout>;
abortHandler?: () => void;
signal?: AbortSignal;
}
interface PerPeerState {
nextId: number;
events: QueuedEvent[];
waiters: PendingWaiter[];
lastTouchedMs: number;
}
/**
* Per-peer monotonic event log with long-poll semantics.
*
* `enqueue` appends; `drain` returns all events with `id > since`,
* blocking up to `blockMs` if there are none. `since`-based pagination
* is the resume mechanism: a client crashing mid-stream restarts with
* its last-processed id and the queue replays everything after it
* (subject to `maxEventsPerPeer` retention).
*/
export class OutboundQueue {
private peers = new Map<string, PerPeerState>();
private readonly maxEvents: number;
private readonly idleEvictionMs: number;
private evictTimer: ReturnType<typeof setTimeout> | null = null;
private destroyed = false;
constructor(opts: OutboundQueueOptions = {}) {
this.maxEvents = opts.maxEventsPerPeer ?? DEFAULT_MAX_EVENTS;
this.idleEvictionMs = opts.idleEvictionMs ?? DEFAULT_IDLE_EVICTION_MS;
if (this.idleEvictionMs > 0) this.scheduleEviction();
}
/** Append an event and wake any waiters for that peer. */
enqueue(peer: string, ev: QueuedEventInput): QueuedEvent {
if (this.destroyed) throw new Error('OutboundQueue: destroyed');
const state = this.getOrCreate(peer);
const event: QueuedEvent =
ev.kind === 'chunk'
? {
id: state.nextId++,
timestampMs: Date.now(),
kind: 'chunk',
bytes: ev.bytes,
meta: ev.meta,
}
: {
id: state.nextId++,
timestampMs: Date.now(),
kind: 'envelope',
bytes: ev.bytes,
};
state.events.push(event);
state.lastTouchedMs = Date.now();
// Cap: drop oldest. Lost events trigger receiver-side resume from
// last polled id; the @shade/transfer engine handles missing seqs
// by re-sending on resume.
while (state.events.length > this.maxEvents) state.events.shift();
// Wake each waiter with events newer than ITS OWN `since`. Using a
// shared snapshot from `since=0` would replay events the waiter has
// already processed once a fresh enqueue arrived mid-poll, which on
// the receiver side double-dispatches an envelope into shade.receive
// → manager.decrypt and consumes the same skipped-key twice (the
// second dispatch corrupts the ratchet chain).
if (state.waiters.length > 0) {
const waiters = state.waiters.splice(0);
for (const w of waiters) {
clearTimeout(w.timer);
if (w.abortHandler !== undefined && w.signal !== undefined) {
w.signal.removeEventListener('abort', w.abortHandler);
}
const wDrained = this.collect(state, w.since);
w.resolve(wDrained);
}
}
return event;
}
/**
* Drain events with `id > since`. If none are available, block up
* to `blockMs` until any arrive. `signal` cancels the wait early.
*/
async drain(
peer: string,
since: number,
blockMs: number,
signal?: AbortSignal,
): Promise<QueuedEvent[]> {
if (this.destroyed) throw new Error('OutboundQueue: destroyed');
const state = this.getOrCreate(peer);
state.lastTouchedMs = Date.now();
const ready = this.collect(state, since);
if (ready.length > 0 || blockMs <= 0) return ready;
if (signal?.aborted) return [];
return await new Promise<QueuedEvent[]>((resolve, reject) => {
const timer = setTimeout(() => {
const idx = state.waiters.indexOf(waiter);
if (idx >= 0) state.waiters.splice(idx, 1);
if (waiter.abortHandler !== undefined && waiter.signal !== undefined) {
waiter.signal.removeEventListener('abort', waiter.abortHandler);
}
// Empty drain on timeout — that's the "no new events" signal.
resolve([]);
}, blockMs);
const waiter: PendingWaiter = { resolve, reject, since, timer };
if (signal !== undefined) {
const handler = () => {
const idx = state.waiters.indexOf(waiter);
if (idx >= 0) state.waiters.splice(idx, 1);
clearTimeout(timer);
resolve([]);
};
signal.addEventListener('abort', handler, { once: true });
waiter.abortHandler = handler;
waiter.signal = signal;
}
state.waiters.push(waiter);
});
}
/** Drop a peer's queue + reject waiters. */
evict(peer: string): void {
const state = this.peers.get(peer);
if (state === undefined) return;
this.peers.delete(peer);
for (const w of state.waiters) {
clearTimeout(w.timer);
if (w.abortHandler !== undefined && w.signal !== undefined) {
w.signal.removeEventListener('abort', w.abortHandler);
}
w.reject(new Error('OutboundQueue: peer evicted'));
}
}
/** Peer-specific snapshot for diagnostics. */
stats(peer: string): { eventCount: number; nextId: number; waiters: number } | null {
const state = this.peers.get(peer);
if (state === undefined) return null;
return {
eventCount: state.events.length,
nextId: state.nextId,
waiters: state.waiters.length,
};
}
/** Tear everything down. Pending waiters are rejected. */
destroy(): void {
if (this.destroyed) return;
this.destroyed = true;
if (this.evictTimer !== null) clearTimeout(this.evictTimer);
for (const peer of [...this.peers.keys()]) this.evict(peer);
}
// ─── internals ──────────────────────────────────────────────
private getOrCreate(peer: string): PerPeerState {
let state = this.peers.get(peer);
if (state === undefined) {
state = {
nextId: 1,
events: [],
waiters: [],
lastTouchedMs: Date.now(),
};
this.peers.set(peer, state);
}
return state;
}
private collect(state: PerPeerState, since: number): QueuedEvent[] {
if (state.events.length === 0) return [];
return state.events.filter((e) => e.id > since);
}
private scheduleEviction(): void {
const interval = Math.max(60_000, Math.floor(this.idleEvictionMs / 4));
this.evictTimer = setTimeout(() => {
if (this.destroyed) return;
const cutoff = Date.now() - this.idleEvictionMs;
for (const [peer, state] of this.peers.entries()) {
if (state.lastTouchedMs < cutoff) this.evict(peer);
}
this.scheduleEviction();
}, interval);
(this.evictTimer as unknown as { unref?: () => void }).unref?.();
}
}
/**
* Chunk transport that enqueues into an {@link OutboundQueue} instead
* of POSTing.
*
* Returns an optimistic `ChunkAck` immediately because the queue *is*
* the delivery — the receiver polls and dispatches. Browser receivers
* cannot synchronously confirm receipt before the next chunk; the
* engine's stream-protocol catches dropped chunks at finish-time
* integrity check, and chunk-resume restarts the lane from the last
* polled `since`.
*/
export class QueueTransferTransport implements ITransferTransport {
constructor(private readonly queue: OutboundQueue) {}
async probe(_peer: string): Promise<void> {
// The queue is local. Reachability is "is there a poller?" which
// is decided by `idleEvictionMs`. We don't synchronously check
// here; the engine retries via `withRetry` on `sendChunk` errors.
}
async sendChunk(
peer: string,
streamId: string,
laneId: number,
seq: number | bigint,
bytes: Uint8Array,
options?: ChunkSendOptions,
): Promise<ChunkAck> {
if (options?.signal?.aborted) {
throw new TransferTransportError('sendChunk aborted by caller');
}
const seqNum = typeof seq === 'bigint' ? Number(seq) : seq;
this.queue.enqueue(peer, {
kind: 'chunk',
bytes,
meta: { streamId, laneId, seq: seqNum },
});
return { lastSeq: seqNum };
}
async fetchResumeState(
_peer: string,
_streamId: string,
): Promise<TransferResumeState | null> {
// Pull-mode receivers report resume state by re-polling with the
// `since` cursor they last successfully processed; the queue does
// not need to query the receiver. Return null so the engine
// restarts from seq 0 (deterministic), and the queue replays from
// `since=0` if the client reconnects fresh.
return null;
}
}

View File

@@ -0,0 +1,60 @@
import { describe, expect, test } from 'bun:test';
import { OutboundQueue } from '../src/index.js';
/**
* Regression coverage for the long-poll waiter `since` cursor.
*
* The bug being guarded against: when `enqueue` woke a pending
* `drain` waiter, it used a `since=0` snapshot and replayed every
* event that had ever been queued — including the ones the waiter
* had already processed in a previous poll. Downstream the queue
* fed `Shade.acceptTransferEnvelope`, so the duplicate replay
* dispatched the same envelope into `manager.decrypt` twice. The
* second decrypt consumed an already-used skipped key, fell into
* the stale-counter branch of `ratchetDecrypt`, and corrupted the
* Double Ratchet receive chain — surfacing as
* `DecryptionError: wrong key or tampered data` on every
* subsequent message.
*/
describe('OutboundQueue — waiter since cursor', () => {
test('mid-poll enqueue must not replay events the waiter already saw', async () => {
const queue = new OutboundQueue({ idleEvictionMs: 0 });
const peer = 'alice';
const e1 = queue.enqueue(peer, { kind: 'envelope', bytes: new Uint8Array([1]) });
const e2 = queue.enqueue(peer, { kind: 'envelope', bytes: new Uint8Array([2]) });
// First poll drains both events (no blocking — they're already there).
const first = await queue.drain(peer, 0, 0);
expect(first.map((e) => e.id)).toEqual([e1.id, e2.id]);
// Now the waiter polls past the last seen id. It blocks because
// there are no events newer than `since`. Concurrently a fresh
// event gets enqueued — that's the path the bug fired on.
const blockMs = 5_000;
const polling = queue.drain(peer, e2.id, blockMs);
// Yield so `drain` actually parks on the waiter list before we
// race the enqueue against it.
await Promise.resolve();
const e3 = queue.enqueue(peer, { kind: 'envelope', bytes: new Uint8Array([3]) });
const woken = await polling;
// Pre-fix: would resolve with [e1, e2, e3] (a `since=0` snapshot
// drained verbatim). Post-fix: only the events newer than the
// waiter's recorded `since` come through.
expect(woken.map((e) => e.id)).toEqual([e3.id]);
});
test('parked waiter at the head still gets the new event when others have polled past it', async () => {
const queue = new OutboundQueue({ idleEvictionMs: 0 });
const peer = 'alice';
const e1 = queue.enqueue(peer, { kind: 'envelope', bytes: new Uint8Array([1]) });
// A waiter that parks past the head — there are no events newer
// than e1.id, so it has to block.
const polling = queue.drain(peer, e1.id, 5_000);
await Promise.resolve();
const e2 = queue.enqueue(peer, { kind: 'envelope', bytes: new Uint8Array([2]) });
const woken = await polling;
expect(woken.map((e) => e.id)).toEqual([e2.id]);
});
});

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/transport-bridge", "name": "@shade/transport-bridge",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/transport-webrtc", "name": "@shade/transport-webrtc",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/transport", "name": "@shade/transport",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@shade/widgets", "name": "@shade/widgets",
"version": "4.1.0", "version": "4.6.0",
"type": "module", "type": "module",
"main": "src/index.ts", "main": "src/index.ts",
"types": "src/index.ts", "types": "src/index.ts",

View File

@@ -29,6 +29,7 @@ const PACKAGES = [
'shade-key-transparency', 'shade-key-transparency',
'shade-storage-sqlite', 'shade-storage-sqlite',
'shade-storage-postgres', 'shade-storage-postgres',
'shade-storage-indexeddb',
'shade-storage-encrypted', 'shade-storage-encrypted',
'shade-streams', 'shade-streams',
'shade-transport', 'shade-transport',

View File

@@ -15,6 +15,7 @@ PACKAGES=(
key-transparency key-transparency
storage-sqlite storage-sqlite
storage-postgres storage-postgres
storage-indexeddb
storage-encrypted storage-encrypted
streams streams
transport transport