Compare commits
24 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 037f994572 | |||
| 188c3db56a | |||
| 1bd7037a6d | |||
| 80c410f518 | |||
| 3c0db14904 | |||
| a98ea8a1bd | |||
| d47774ef1c | |||
| 8c606ad498 | |||
| 680d6386f3 | |||
| 1fb59a7076 | |||
| 594992a183 | |||
| 8746571d2a | |||
| 2c400d7094 | |||
| 2b1b4d6630 | |||
| dbb3a090d8 | |||
| f5f42fe557 | |||
| b77b7e771c | |||
| 7520b11b25 | |||
| da93b97cce | |||
| 0bdf9e859c | |||
| 70e319fef8 | |||
| f301b391a5 | |||
| 40766c60f4 | |||
| de25b19033 |
3
.gitignore
vendored
3
.gitignore
vendored
@@ -2,3 +2,6 @@ node_modules/
|
||||
dist/
|
||||
*.tsbuildinfo
|
||||
.DS_Store
|
||||
**/.tmp-*.db
|
||||
**/.tmp-*.db-shm
|
||||
**/.tmp-*.db-wal
|
||||
|
||||
1663
CHANGELOG.md
1663
CHANGELOG.md
File diff suppressed because it is too large
Load Diff
322
README.md
322
README.md
@@ -3,37 +3,82 @@
|
||||
End-to-end encryption library implementing the Signal Protocol (X3DH + Double Ratchet) for TypeScript/Bun. Drop into any project — frontend, backend, mobile — to get forward secrecy, post-compromise recovery, and self-healing security.
|
||||
|
||||
> **4.0.0 — General Availability.** All V3.1 → V3.12 work is merged,
|
||||
> the cross-platform vector suite is green on TS + Kotlin, the threat
|
||||
> model has been refreshed for every new surface, and the core stack
|
||||
> (X3DH, ratchet, storage encryption, recovery, WebRTC P2P, Key
|
||||
> Transparency) has been packaged for external review. The wire
|
||||
> format is **unchanged from 0.4.x** — 4.0 peers interoperate with
|
||||
> 0.4.x peers byte-for-byte. See
|
||||
> the cross-platform vector suite is green on TS + Kotlin (1000 / 1000
|
||||
> + 11 / 11), the threat model has been refreshed for every new
|
||||
> surface, and the core stack (X3DH, ratchet, storage encryption,
|
||||
> recovery, WebRTC P2P, Key Transparency) has been packaged for
|
||||
> external review. The wire format is **unchanged from 0.4.x** — 4.0
|
||||
> peers interoperate with 0.4.x peers byte-for-byte. See
|
||||
> [MIGRATION.md § 0.3.x → 4.0](./MIGRATION.md#migrating-from-03x-to-40-ga)
|
||||
> for the upgrade path and [CHANGELOG.md § 4.0.0](./CHANGELOG.md) for
|
||||
> the consolidated release notes. Voice / Video have been moved to
|
||||
> [V5.0](./docs/V5.0.md), to be built on top of the frozen 4.0
|
||||
> baseline.
|
||||
|
||||
## Status
|
||||
|
||||
| Area | 4.0 status | Pointers |
|
||||
|------|-----------|----------|
|
||||
| Protocol core (X3DH + ratchet + sender keys) | ✅ Done — frozen | [`packages/shade-core`](./packages/shade-core) |
|
||||
| Storage encryption (V3.2) | ✅ Done — opt-in `EncryptedSQLiteStorage` / `EncryptedPostgresStorage`, key sources: passphrase / OS keychain / app-injected | [`docs/storage-encryption.md`](./docs/storage-encryption.md) |
|
||||
| Fingerprint gates & trust UX (V3.3) | ✅ Done — `Shade.beforeFirstLargeFile / beforeBackupImport / beforeNewDeviceTrust` | [`docs/trust-ux.md`](./docs/trust-ux.md) |
|
||||
| Observability v2 (V3.4) | ✅ Done — OpenTelemetry-shaped events, `/metrics`, observer dashboard | [`docs/observability.md`](./docs/observability.md) |
|
||||
| Android parity & cross-platform CI (V3.5) | ✅ Done — TS + Kotlin vector-gate live; Android `KeystoreStorage` is post-GA | [`android/shade-android/README.md`](./android/shade-android/README.md), [`docs/cross-platform.md`](./docs/cross-platform.md) |
|
||||
| Async store-and-forward (V3.6) | ✅ Done — `@shade/inbox` + `@shade/inbox-server` | [`docs/inbox.md`](./docs/inbox.md) |
|
||||
| Transport bridge (V3.7) | ✅ Done — SSE / long-poll / WS adapters | [`docs/transport.md`](./docs/transport.md) |
|
||||
| Web Workers crypto (V3.8) | ✅ Done — lane keys never cross the thread boundary | [`docs/web-workers.md`](./docs/web-workers.md) |
|
||||
| Rich file metadata + thumbnails (V3.9) | ✅ Done — in `@shade/files` | [`docs/files.md`](./docs/files.md) |
|
||||
| Social key recovery (V3.10) | ✅ Done — Shamir + AEAD-gated reconstruction + guardian widgets | [`docs/recovery.md`](./docs/recovery.md) |
|
||||
| WebRTC P2P transport (V3.11) | ✅ Done — `RTCDataChannel` with `MultiTransportFallback([webrtc, http])` | [`docs/webrtc.md`](./docs/webrtc.md) |
|
||||
| Key Transparency (V3.12) | ✅ Done — opt-in Merkle log, signed STH, witness gossip | [`docs/key-transparency.md`](./docs/key-transparency.md) |
|
||||
| External crypto review | 🟡 Bundle ready — review window open after tag | [`docs/audit/REVIEW-BUNDLE.md`](./docs/audit/REVIEW-BUNDLE.md) |
|
||||
| Soak (≥ 2 weeks under load) | 🟡 Harness shipped — operator runs it | [`scripts/soak.ts`](./scripts/soak.ts) (`bun run soak --hours 336`) |
|
||||
| Voice / Video / Broadcast | 🔜 V5.0 — built on top of frozen 4.0 stack | [`docs/V5.0.md`](./docs/V5.0.md) |
|
||||
|
||||
## What you get
|
||||
|
||||
**Protocol core**
|
||||
- **X3DH** initial key agreement (works asynchronously via prekey bundles)
|
||||
- **Double Ratchet** for per-message forward secrecy and post-compromise security
|
||||
- **Self-authenticated prekey server** (Hono, Docker-ready) with rate limiting, metrics, health checks
|
||||
- **Persistent storage backends**: SQLite (zero-config) and PostgreSQL (Drizzle)
|
||||
- **Sender keys** for group ratchet (1:N broadcast key derivation)
|
||||
- **Identity rotation** with grace period for old sessions
|
||||
- **Safety numbers** (Signal-style fingerprints) for out-of-band verification
|
||||
- **Constant-time comparisons** and **memory zeroization** for hardened operation
|
||||
- **Binary wire format** that's significantly smaller than JSON
|
||||
- **Binary wire format** (`@shade/proto`) — significantly smaller than JSON
|
||||
|
||||
**Storage**
|
||||
- **Persistent backends**: SQLite (zero-config, `bun:sqlite`) and PostgreSQL (Drizzle, FOR UPDATE SKIP LOCKED)
|
||||
- **Crash-safe** — sessions survive container restarts, power outages, SIGKILL
|
||||
- **Live observability** — bundled dashboard SPA + embeddable React widgets to see what's happening between every step
|
||||
- **E2EE file transfers** — multi-lane chunked uploads/downloads with resume, integrity checks, and HTTP/WS fallback (`@shade/streams` + `@shade/transfer`)
|
||||
- **WebRTC P2P transport (V3.11)** — opt-in `RTCDataChannel` upload path with public-STUN defaults, TURN-relay support, glare-safe peer pool, and automatic `MultiTransportFallback` back to HTTP when NAT traversal fails (`@shade/transport-webrtc`, [docs/webrtc.md](./docs/webrtc.md))
|
||||
- **Web Workers crypto** — AEAD, HKDF, HMAC, X25519, Ed25519 and per-lane stream state run in a dedicated worker. 100 MB+ uploads stay smooth without frame drops, lane keys never cross the thread boundary (`@shade/crypto-web/worker`, [docs/web-workers.md](./docs/web-workers.md))
|
||||
- **E2EE filesystem RPC** — typed `list/stat/mkdir/delete/move/read/write/getThumbnail` + custom ops between peers, with rate-limit, retention, and fingerprint-gate hooks (`@shade/files`)
|
||||
- **Async store-and-forward** — deliver to offline recipients via a relay that holds ciphertext-only blobs with TTL, idempotent PUT, signed fetch/ack, and an `onMessageQueued` push-trigger hook (`@shade/inbox` + `@shade/inbox-server`)
|
||||
- **Social key recovery** — Shamir-split your identity to `n` guardians; any threshold-many `k` together restore it on a new device. No centralized recovery agent; OOB-fingerprint gate on every guardian release; AES-GCM authenticates the reconstruction (`@shade/recovery` + `<RecoverySetup />` / `<RecoveryRequest />` / `<RecoveryApprove />` widgets, [docs/recovery.md](./docs/recovery.md))
|
||||
- **Key Transparency (V3.12)** — opt-in append-only Merkle log over the prekey server. Every `register` / `delete` becomes a signed leaf; every bundle-fetch carries an inclusion proof; an Ed25519-signed Tree Head ties roots to a fixed `log_id`. A `LightWitness` cross-checks STHs across clients so a malicious server that splits its view or rewrites history is caught (`@shade/key-transparency`, [docs/key-transparency.md](./docs/key-transparency.md))
|
||||
- **At-rest encryption (V3.2, opt-in)** — AES-256-GCM under per-(table,column) DEKs; key sources: passphrase (scrypt), OS keychain (`@shade/keychain`), or app-injected. Online re-key, no downtime.
|
||||
|
||||
**Servers**
|
||||
- **Self-authenticated prekey server** (`@shade/server`, Hono, Docker-ready) with rate limiting, metrics, health checks
|
||||
- **Async store-and-forward relay** (`@shade/inbox-server`) — TTL-bound ciphertext blobs, signed PUT/FETCH/ACK, idempotent on `(address, msgId)`, per-recipient quota
|
||||
- **Bridge transports** (`@shade/transport-bridge`) — WS → SSE → long-poll fallback chain for clients that can't keep a WebSocket open. Same `IncomingMessage` shape across all three.
|
||||
- **Standalone container** — one image bundles prekey + inbox + bridge + transfer + KT + observer
|
||||
|
||||
**Trust UX**
|
||||
- **Fingerprint gates (V3.3)** — `Shade.beforeFirstLargeFile(threshold, handler)`, `beforeBackupImport`, `beforeNewDeviceTrust`. Gates raise `FingerprintNotVerifiedError` on the operations that matter, default-warn TOFU otherwise.
|
||||
- **`<FingerprintCompare />` / `<FingerprintGate />`** widgets for the matching UI side.
|
||||
|
||||
**File transfer & filesystem**
|
||||
- **E2EE file transfers** (`@shade/streams` + `@shade/transfer`) — multi-lane chunked uploads/downloads with resume, integrity checks (per-lane sha256 + overall sha256), HTTP/WS fallback, `MultiTransportFallback` for N-ary demotion
|
||||
- **WebRTC P2P transport (V3.11, opt-in)** — `RTCDataChannel` chunk path with public-STUN defaults, TURN-relay support, glare-safe peer pool, automatic fallback to HTTP when NAT traversal fails (`@shade/transport-webrtc`)
|
||||
- **Web Workers crypto (V3.8, opt-in)** — AEAD, HKDF, HMAC, X25519, Ed25519 and per-lane stream state run in a dedicated worker. 100 MB+ uploads stay smooth without frame drops; lane keys never cross the thread boundary (`@shade/crypto-web/worker`)
|
||||
- **E2EE filesystem RPC** (`@shade/files`) — typed `list/stat/mkdir/delete/move/read/write/getThumbnail` + custom ops, with rate-limit, retention, fingerprint-gate, and metrics hooks. React hooks under `@shade/files/react`.
|
||||
|
||||
**Recovery**
|
||||
- **Social key recovery (V3.10)** — Shamir-split your identity to `n` guardians; any threshold-many `k` together restore it on a new device. No centralized recovery agent; OOB-fingerprint gate per guardian release; AES-GCM-authenticated reconstruction (`@shade/recovery` + `<RecoverySetup />` / `<RecoveryRequest />` / `<RecoveryApprove />`)
|
||||
|
||||
**Verifiable distribution**
|
||||
- **Key Transparency (V3.12, opt-in)** — append-only Merkle log over the prekey server. Every `register` / `delete` becomes a signed leaf; every bundle-fetch carries an inclusion proof; an Ed25519-signed Tree Head ties roots to a fixed `log_id`. A `LightWitness` cross-checks STHs across clients so a malicious server that splits its view or rewrites history is caught (`@shade/key-transparency`).
|
||||
|
||||
**Observability**
|
||||
- **Live observability** — OpenTelemetry-shaped events, bundled dashboard SPA + embeddable React widgets to see what's happening between every step (`@shade/observability` + `@shade/observer` + `@shade/dashboard`)
|
||||
|
||||
**Tooling**
|
||||
- **CLI** — `shade init` scaffolder, `shade migrate-storage` (V3.2), `shade rotate-storage-key`, `shade fingerprint`, `shade rotate`, `shade peer`, `shade dashboard`, `shade doctor`, `shade backup` (`@shade/cli`)
|
||||
- **Soak harness** — `bun run soak --hours 336` for the 2-week GA-stable window (`scripts/soak.ts`)
|
||||
|
||||
## Quick start
|
||||
|
||||
@@ -84,7 +129,38 @@ const plaintext = await shade.receive('alice@example.com', incomingEnvelope);
|
||||
console.log(await shade.fingerprint);
|
||||
```
|
||||
|
||||
Need to ship a file or expose a filesystem to a peer? `Shade.files` is the high-level entrypoint:
|
||||
### Opt-in surfaces (V3.x → 4.0 GA)
|
||||
|
||||
All of these are off by default. Wire them only where you need them.
|
||||
|
||||
```ts
|
||||
// V3.3 — Fingerprint gates: enforce verification on the operations that matter
|
||||
shade.beforeFirstLargeFile(10 * 1024 * 1024, async ({ peer, fingerprint }) => {
|
||||
return await ui.confirmSafetyNumberMatches(peer, fingerprint);
|
||||
});
|
||||
shade.beforeBackupImport(async ({ embeddedFingerprint }) => { /* ... */ });
|
||||
shade.beforeNewDeviceTrust(async ({ peer, oldFp, newFp }) => { /* ... */ });
|
||||
|
||||
// V3.8 — Web Workers crypto: opt-in, lane keys stay off the main thread
|
||||
shade.configureWorkerCrypto({
|
||||
workerUrl: new URL('@shade/crypto-web/worker', import.meta.url),
|
||||
});
|
||||
|
||||
// V3.11 — WebRTC P2P transport: file transfers ride RTCDataChannel where NAT allows
|
||||
import { nativeRtcFactory } from '@shade/transport-webrtc';
|
||||
shade.configureWebRTC({
|
||||
factory: nativeRtcFactory(),
|
||||
iceServers: [{ urls: 'stun:stun.l.google.com:19302' }],
|
||||
});
|
||||
|
||||
// V3.12 — Key Transparency: detect server-side bundle swaps
|
||||
const shade = await createShade({
|
||||
prekeyServer: 'https://shade.example.com',
|
||||
keyTransparency: { mode: 'observe-strict', logPublicKey: PINNED_KEY_BYTES_32 },
|
||||
});
|
||||
```
|
||||
|
||||
### Files RPC (`@shade/files`)
|
||||
|
||||
```ts
|
||||
// Server side — Bob exposes a virtual filesystem
|
||||
@@ -97,13 +173,13 @@ const stop = await shade.files.serve({
|
||||
|
||||
// Client side — Alice consumes Bob's filesystem
|
||||
const fs = await shade.files.client('bob');
|
||||
await fs.write('/photos/cover.png', new Uint8Array(...)); // auto inline/streams
|
||||
await fs.write('/photos/cover.png', new Uint8Array([/* ... */])); // auto inline/streams
|
||||
const result = await fs.read('/photos/cover.png');
|
||||
```
|
||||
|
||||
Files ≤ 256 KiB ride inline in the RPC envelope; larger files automatically promote to multi-lane `@shade/transfer` streams with sha256 integrity. See [`docs/files.md`](./docs/files.md) for the full API.
|
||||
|
||||
Or use the lower-level packages directly if you need full control:
|
||||
### Lower-level access
|
||||
|
||||
```ts
|
||||
import { ShadeSessionManager } from '@shade/core';
|
||||
@@ -117,20 +193,43 @@ const manager = new ShadeSessionManager(
|
||||
await manager.initialize();
|
||||
```
|
||||
|
||||
### At-rest encryption (V3.2)
|
||||
|
||||
```ts
|
||||
import { KeyManager, EncryptedSQLiteStorage } from '@shade/storage-encrypted';
|
||||
|
||||
const km = await KeyManager.open({
|
||||
kind: 'passphrase',
|
||||
passphrase: process.env.SHADE_STORAGE_PASSPHRASE!,
|
||||
salt: loadSaltFromDisk(),
|
||||
});
|
||||
const storage = await EncryptedSQLiteStorage.open({
|
||||
dbPath: '/data/shade-client.db',
|
||||
keyManager: km,
|
||||
});
|
||||
```
|
||||
|
||||
To migrate an existing 0.3.x SQLite DB in place:
|
||||
|
||||
```bash
|
||||
shade migrate-storage \
|
||||
--key-source passphrase \
|
||||
--passphrase "$SHADE_STORAGE_PASSPHRASE" \
|
||||
--salt-file /data/shade-client.db.salt
|
||||
```
|
||||
|
||||
## Architecture — keys vs. payloads
|
||||
|
||||
Shade splits the network into two planes. The **prekey server** only
|
||||
sees public keys; **everything else** rides the encrypted Double
|
||||
Ratchet between peers. If you remember nothing else from this README,
|
||||
remember this picture:
|
||||
Shade splits the network into a **public-key plane** (the prekey
|
||||
server) and an **encrypted plane** (everything else). The prekey
|
||||
server only sees public key material. If you remember nothing else
|
||||
from this README, remember this picture:
|
||||
|
||||
```
|
||||
Shade Prekey Server (Hono, public keys only)
|
||||
Shade Prekey Container (Hono — public keys only)
|
||||
│
|
||||
POST /v1/keys/register (signed)
|
||||
GET /v1/keys/bundle/:address
|
||||
POST /v1/keys/replenish (signed)
|
||||
DELETE /v1/keys/:address (signed)
|
||||
/v1/keys/* /v1/inbox/* /v1/bridge/* /v1/transfer/*
|
||||
/v1/kt/* /metrics /healthz /ready
|
||||
│
|
||||
┌──────────────────┴──────────────────┐
|
||||
│ │
|
||||
@@ -142,11 +241,20 @@ remember this picture:
|
||||
│◄── Double Ratchet messages ────────►│ ← end-to-end,
|
||||
│ (ratchet 0x02 / chunks 0x11) │ never on the
|
||||
│ │ prekey server
|
||||
│◄── @shade/transfer chunks ─────────►│
|
||||
│ POST /v1/transfer/:id/chunk │ ← peer-to-peer
|
||||
│ GET /v1/transfer/:id/state │ HTTP, opaque
|
||||
│ │ ciphertext
|
||||
│
|
||||
│◄── @shade/transfer chunks ─────────►│ ← peer-to-peer
|
||||
│ POST /v1/transfer/:id/chunk │ HTTP, opaque
|
||||
│ GET /v1/transfer/:id/state │ ciphertext
|
||||
│
|
||||
│◄── @shade/inbox blobs (offline) ───►│ ← TTL-bound
|
||||
│ POST /v1/inbox/:address │ ciphertext-only
|
||||
│ POST /v1/inbox/:address/fetch │ relay
|
||||
│
|
||||
│◄── @shade/transport-webrtc ────────►│ ← optional P2P
|
||||
│ RTCDataChannel `shade-transfer/v1` │ `MultiTransportFallback`
|
||||
│ │ auto-demotes to HTTP
|
||||
SQLiteStorage / PostgresStorage SQLiteStorage / PostgresStorage
|
||||
+ EncryptedSQLiteStorage (V3.2) + EncryptedSQLiteStorage (V3.2)
|
||||
(private keys + sessions) (private keys + sessions)
|
||||
```
|
||||
|
||||
@@ -155,17 +263,23 @@ remember this picture:
|
||||
- Identity public keys (Ed25519 + X25519)
|
||||
- Signed prekeys + one-time prekey bundles
|
||||
- Registration / replenish / delete writes, all Ed25519-signed
|
||||
- Operator-only metrics, `/health`, and the optional observer
|
||||
dashboard
|
||||
- (V3.6) Inbox ciphertext blobs with TTL — same container, separate
|
||||
routes; the relay only sees `address || msgId || ciphertext-bytes`
|
||||
- (V3.7) Bridge transports (SSE / long-poll / WS) — also delivered by
|
||||
the same Hono app for clients that can't hold a WebSocket
|
||||
- (V3.12, opt-in) KT inclusion proofs + signed tree heads on
|
||||
`/v1/kt/*` — verifiable distribution
|
||||
- Operator-only metrics and the optional observer dashboard
|
||||
|
||||
### What does **not** go via the prekey server
|
||||
|
||||
- **Message plaintext, ever.** Encrypted ratchet envelopes flow peer-
|
||||
to-peer over whatever transport you choose (HTTP, WebSocket, your
|
||||
own broker).
|
||||
own broker, or the inbox relay above — which carries ciphertext only).
|
||||
- **File chunks.** `@shade/transfer` POSTs ciphertext directly to the
|
||||
receiver's `/v1/transfer/:streamId/chunk` route — the prekey server
|
||||
is not involved.
|
||||
is not involved. With V3.11 + `configureWebRTC()`, chunks ride
|
||||
`RTCDataChannel` peer-to-peer; the relay is bypassed entirely.
|
||||
- **Identity private keys.** They never leave the device's storage.
|
||||
- **Filesystem RPC.** `@shade/files` rides the Double Ratchet for
|
||||
control + small payloads, then promotes to direct `@shade/transfer`
|
||||
@@ -176,6 +290,8 @@ remember this picture:
|
||||
The prekey server is metadata-bearing (see `THREAT-MODEL.md § 2`):
|
||||
it sees who registers, who fetches whose bundle, and when. It does
|
||||
**not** see message contents, transfer contents, or session state.
|
||||
**V3.12 Key Transparency** (opt-in) makes its bundle distribution
|
||||
*verifiable* so a malicious server that swaps a bundle is caught.
|
||||
|
||||
For the full threat model and mitigations, read
|
||||
[THREAT-MODEL.md](./THREAT-MODEL.md). For deployment-time guarantees,
|
||||
@@ -183,53 +299,84 @@ read [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md).
|
||||
|
||||
## Packages
|
||||
|
||||
All packages publish in lockstep at `4.0.0`.
|
||||
|
||||
| Package | Purpose |
|
||||
|---------|---------|
|
||||
| `@shade/core` | Protocol logic (X3DH, Double Ratchet, session manager, errors, events) |
|
||||
| `@shade/core` | Protocol logic (X3DH, Double Ratchet, sender keys, session manager, errors, events) |
|
||||
| `@shade/proto` | Compact binary wire format (`0x01` PreKeyMessage, `0x02` RatchetMessage, `0x11` StreamChunk) |
|
||||
| `@shade/crypto-web` | SubtleCrypto + @noble/curves provider, in-memory storage. Includes the V3.8 Web Workers entrypoint (`@shade/crypto-web/worker`) — drop-in `WorkerCryptoProvider` plus `createEncryptStream` / `createDecryptStream` TransformStream factories |
|
||||
| `@shade/storage-sqlite` | Persistent SQLite storage (zero-config, bun:sqlite) |
|
||||
| `@shade/storage-postgres` | PostgreSQL storage with Drizzle for shared databases |
|
||||
| `@shade/server` | Prekey server (Hono routes, auth, rate limit, health, metrics) |
|
||||
| `@shade/transport` | HTTP + WebSocket transport wrappers with auto-encryption |
|
||||
| `@shade/observability` | OpenTelemetry-shaped event bus consumed by `@shade/observer`, server hooks, and the dashboard |
|
||||
| `@shade/keychain` | OS keychain bindings (libsecret / Keychain / Credential Manager) used by `@shade/storage-encrypted` and the CLI |
|
||||
| `@shade/key-transparency` | Key Transparency (V3.12) — RFC 6962-style append-only Merkle log, address-index commitment, signed tree heads, and a `LightWitness` for split-view detection. Opt-in on both server and client. |
|
||||
| `@shade/storage-sqlite` | Persistent SQLite storage (zero-config, `bun:sqlite`); also ships `SqliteInboxStore` |
|
||||
| `@shade/storage-postgres` | PostgreSQL storage with Drizzle for shared databases; also ships `PostgresInboxStore` + `PostgresKTLogStore` |
|
||||
| `@shade/storage-encrypted` | At-rest encryption (V3.2) — `EncryptedSQLiteStorage` / `EncryptedPostgresStorage`, `KeyManager`, online re-key |
|
||||
| `@shade/streams` | Multi-lane chunk encryption — HKDF-derived per-lane keys, deterministic AES-GCM nonces, streaming SHA-256, file metadata + thumbnails (V3.9) |
|
||||
| `@shade/transport` | HTTP + WebSocket transport wrappers with auto-encryption; KT-verifying `fetchBundleVerified` |
|
||||
| `@shade/transport-bridge` | WS → SSE → long-poll fallback chain (V3.7) — single `IncomingMessage` shape across transports for clients that can't keep a WebSocket open |
|
||||
| `@shade/transport-webrtc` | V3.11 P2P chunk transport via `RTCDataChannel`. Plugs into `@shade/transfer` as an `ITransferTransport`; signaling rides Shade's own ratchet. Memory factory + native (`globalThis.RTCPeerConnection`) factory included; `MultiTransportFallback([webrtc, http])` wired automatically when `shade.configureWebRTC()` is called. |
|
||||
| `@shade/proto` | Compact binary wire format (smaller than JSON) |
|
||||
| `@shade/streams` | Multi-lane chunk encryption — HKDF-derived per-lane keys, deterministic AES-GCM nonces, streaming SHA-256 |
|
||||
| `@shade/transfer` | Transfer engine on top of streams: parallel lanes, resume, HTTP + WS transport with auto-fallback, integrity verification |
|
||||
| `@shade/files` | Typed E2EE filesystem RPC — list/stat/mkdir/delete/move/read/write/getThumbnail + custom ops, auto inline/streams routing, production hooks (rate limit, retention, fingerprint gate, metrics), React hooks |
|
||||
| `@shade/recovery` | Social key recovery (V3.10) — Shamir-split identity to `n` guardians; threshold-many `k` reconstruct on a new device. AES-GCM-authenticated reconstruction; OOB-fingerprint gate per guardian release |
|
||||
| `@shade/key-transparency` | Key Transparency (V3.12) — RFC 6962-style append-only Merkle log, address-index commitment, signed tree heads, and a `LightWitness` for split-view detection. Opt-in on both server and client. See [docs/key-transparency.md](./docs/key-transparency.md) |
|
||||
| `@shade/server` | Prekey server (Hono routes, auth, rate limit, health, metrics). `createPrekeyServerWithKT(...)` opts into V3.12 KT mode |
|
||||
| `@shade/inbox-server` | Async store-and-forward relay (V3.6) — Hono routes, signed PUT/FETCH/DELETE, per-recipient TTL + quota, idempotent on `(address, msgId)`. Bundles into the same standalone container as the prekey server |
|
||||
| `@shade/inbox` | Inbox client + durable outgoing queue + receive cursor + push-trigger hook (`onMessageQueued`); composes on top of `Shade.send`/`Shade.receive` for offline-recipient delivery |
|
||||
| `@shade/transfer` | Transfer engine on top of streams: parallel lanes, resume, HTTP + WS transport with auto-fallback, `MultiTransportFallback` (N-ary demotion), integrity verification |
|
||||
| `@shade/files` | Typed E2EE filesystem RPC — list/stat/mkdir/delete/move/read/write/getThumbnail + custom ops, auto inline/streams routing, production hooks (rate limit, retention, fingerprint gate, metrics), React hooks under `@shade/files/react` |
|
||||
| `@shade/recovery` | Social key recovery (V3.10) — Shamir-split identity to `n` guardians; threshold-many `k` reconstruct on a new device. AES-GCM-authenticated reconstruction; OOB-fingerprint gate per guardian release |
|
||||
| `@shade/observer` | Live debugger backend (snapshot, SSE, dashboard) — see [README](./packages/shade-observer/README.md) |
|
||||
| `@shade/widgets` | Embeddable React widgets including transfer uploader/downloader — see [README](./packages/shade-widgets/README.md) |
|
||||
| `@shade/dashboard` | Standalone dashboard SPA bundled into the observer |
|
||||
| `@shade/sdk` | High-level wrapper with `createShade()` one-liner, auto-publish, auto-establish, auto-replenish, `Shade.files` namespace |
|
||||
| `@shade/cli` | `shade init` scaffolder + utilities (fingerprint, rotate, peer, dashboard, doctor) |
|
||||
| `@shade/sdk` | High-level wrapper with `createShade()` one-liner, auto-publish, auto-establish, auto-replenish, `Shade.files` namespace, fingerprint gates, KT integration, WebRTC opt-in |
|
||||
| `@shade/widgets` | Embeddable React widgets — fingerprint compare/gate, recovery setup/request/approve, transfer uploader/downloader, observer panels |
|
||||
| `@shade/cli` | `shade init` scaffolder + utilities (fingerprint, rotate, peer, dashboard, doctor, backup, migrate-storage, rotate-storage-key) |
|
||||
|
||||
## Shade as a modular toolkit
|
||||
|
||||
Shade is split into packages so each project can depend on **only what it needs**—encrypted messaging, file transfer, prekey hosting, or lower-level building blocks. You do not need one giant stack for every use case.
|
||||
Shade is split into packages so each project can depend on **only what it needs**—encrypted messaging, file transfer, prekey hosting, social recovery, KT verification, or lower-level building blocks. You do not need one giant stack for every use case.
|
||||
|
||||
For a **plain-language map** (which packages to add, what the prekey server does vs your own wiring, and where to start in code), see **[docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md)**.
|
||||
|
||||
## Publishing
|
||||
|
||||
All packages publish to a self-hosted Gitea npm registry on `gt.zyon.no`.
|
||||
All packages publish to a self-hosted Gitea npm registry on `gt.zyon.no`. The Docker image of the standalone container ships at `gt.zyon.no/stian/shade-prekey:<tag>`.
|
||||
|
||||
```bash
|
||||
# Bump all packages in lockstep
|
||||
bun run version 1.1.0
|
||||
bun run version 4.0.1
|
||||
|
||||
# Dry-run (pack all tarballs without publishing)
|
||||
# Dry-run (pack all tarballs without publishing) — no token required
|
||||
bun run publish:dry
|
||||
|
||||
# Real publish (requires GITEA_TOKEN env var)
|
||||
# Real publish — interactive (prompts for GITEA_TOKEN, checks
|
||||
# registry for conflicts, publishes via scripts/publish-all.ts)
|
||||
bun run publish:all
|
||||
|
||||
# Or via CI: push a git tag v1.1.0 and .gitea/workflows/publish.yml runs
|
||||
# Build + push the standalone Docker image
|
||||
bun run scripts/build-docker.ts -- --tag 4.0.1 --push
|
||||
```
|
||||
|
||||
The interactive `scripts/publish-shade.sh` is the human entrypoint;
|
||||
`scripts/publish-all.ts` is the headless variant used by CI and
|
||||
`publish:dry`. They share a single `PACKAGES` list (24 entries at
|
||||
4.0.0) so the two flows can never drift.
|
||||
|
||||
## Soak / GA-stable
|
||||
|
||||
Before tagging `4.0.0` as `latest` and recommending production
|
||||
upgrades, run the combined soak harness for ≥ 2 weeks:
|
||||
|
||||
```bash
|
||||
# Full GA window (V4.0 §Soak): 14 days × 24 hours
|
||||
bun run soak --hours 336
|
||||
|
||||
# Smoke (~3 minutes — ratchet ping-pong, integrity check)
|
||||
bun run soak:smoke
|
||||
```
|
||||
|
||||
The harness fans out N concurrent ratchet pairs, ping-pongs at
|
||||
~400 ops/sec/pair, and reports cumulative counters every minute.
|
||||
Any exception in any pair is captured and re-raised at shutdown so
|
||||
silent failures cannot hide. Wrap it in `systemd-run --user`,
|
||||
`nohup`, or a Gitea scheduled job for the actual 2-week window.
|
||||
|
||||
## Security properties
|
||||
|
||||
| Property | Description |
|
||||
@@ -242,27 +389,57 @@ bun run publish:all
|
||||
| **Memory zeroization** | Key material is zeroed after use (best-effort in JS) |
|
||||
| **Identity verification** | Safety numbers (60 digits) for out-of-band comparison |
|
||||
| **Identity rotation** | 7-day grace period for old sessions during rotation |
|
||||
| **At-rest encryption** *(V3.2, opt-in)* | AES-256-GCM under per-(table, column) DEKs; AAD binds `(table, column, pk)`; passphrase / OS keychain / app-injected master key; online re-key |
|
||||
| **Fingerprint gates** *(V3.3)* | `Shade.beforeFirstLargeFile / beforeBackupImport / beforeNewDeviceTrust` raise `FingerprintNotVerifiedError` on the operations that matter; defaults to TOFU + warning when no gate is registered |
|
||||
| **Async store-and-forward** *(V3.6)* | Relay only sees `address || msgId || ciphertext`; idempotent PUT; signed FETCH/ACK; TTL-bounded |
|
||||
| **Web-Worker key isolation** *(V3.8)* | Lane keys, identity keys, and ratchet chain keys live inside a dedicated worker; main thread only ferries plaintext via transferable buffers; idle terminate releases worker memory |
|
||||
| **Social key recovery** *(V3.10)* | Shamir over GF(2^8); AEAD-authenticated reconstruction (forged shares fail); guardian-side fingerprint gate before share release |
|
||||
| **WebRTC P2P transport** *(V3.11)* | Same Double Ratchet authenticates SDP/ICE signaling; chunk frames AEAD-bound to `streamId/laneId/seq`; deterministic glare resolution; `MultiTransportFallback` auto-demotes to HTTP |
|
||||
| **Key Transparency** *(V3.12, opt-in)* | Append-only Merkle log + signed tree heads + witness gossip — split-view and history-rewrite are detected by clients |
|
||||
|
||||
## Documentation
|
||||
|
||||
**Operator + integrator**
|
||||
|
||||
- [docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md) — **Modular toolkit**: pick packages by scenario (messages, files, browser, ops)
|
||||
- [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md) — Pre-flight gates for going to production
|
||||
- [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md) — Full deployment guide (Docker, env vars, PostgreSQL, backup, Dokploy)
|
||||
- [docs/ROADMAP.md](./docs/ROADMAP.md) — V3.x → 4.0 GA → V5.0 trajectory
|
||||
|
||||
**Per-surface deep-dives**
|
||||
|
||||
- [docs/files.md](./docs/files.md) — `@shade/files` API + design (filesystem RPC, custom ops, hooks, React)
|
||||
- [docs/recovery.md](./docs/recovery.md) — `@shade/recovery` social key recovery (V3.10): Shamir setup, guardian-side gates, threshold tuning
|
||||
- [docs/streams.md](./docs/streams.md) — `@shade/streams` + `@shade/transfer` deep dive (incl. hardening + retention)
|
||||
- [docs/inbox.md](./docs/inbox.md) — `@shade/inbox` + `@shade/inbox-server` async store-and-forward relay (V3.6)
|
||||
- [docs/transport.md](./docs/transport.md) — `@shade/transport-bridge` SSE / long-poll / WS bridge layer (V3.7)
|
||||
- [docs/web-workers.md](./docs/web-workers.md) — V3.8 Web Workers crypto: setup, bundler recipes (Vite/Webpack/Rollup), Safari notes, lifecycle, threat-model
|
||||
- [docs/recovery.md](./docs/recovery.md) — `@shade/recovery` social key recovery (V3.10): Shamir setup, guardian-side gates, threshold tuning
|
||||
- [docs/webrtc.md](./docs/webrtc.md) — `@shade/transport-webrtc` P2P transport (V3.11): NAT-traversal, TURN config, glare resolution, wire format, multi-fallback wiring
|
||||
- [docs/key-transparency.md](./docs/key-transparency.md) — `@shade/key-transparency` (V3.12): operator + client onboarding, witness role, recovery procedures
|
||||
- [docs/V3.12-DESIGN.md](./docs/V3.12-DESIGN.md) — V3.12 design notat (threat model, RFC 6962 vs CONIKS choice, freshness model, open-questions resolution)
|
||||
- [docs/web-workers.md](./docs/web-workers.md) — V3.8 Web Workers crypto: setup, bundler recipes (Vite/Webpack/Rollup), Safari notes, lifecycle, threat-model
|
||||
- [docs/storage-encryption.md](./docs/storage-encryption.md) — V3.2 at-rest encryption: design, key sources, rotation
|
||||
- [docs/trust-ux.md](./docs/trust-ux.md) — V3.3 fingerprint gates: when each fires, handler patterns, widget integration
|
||||
- [docs/observability.md](./docs/observability.md) — V3.4 event bus + dashboard
|
||||
- [docs/cross-platform.md](./docs/cross-platform.md) — V3.5 Android parity + cross-platform vector regime
|
||||
|
||||
**Threat model + audit**
|
||||
|
||||
- [SECURITY.md](./SECURITY.md) — Reporting vulnerabilities, security policy, threat-/test-matrix
|
||||
- [THREAT-MODEL.md](./THREAT-MODEL.md) — Honest threat model and assumptions
|
||||
- [THREAT-MODEL.md](./THREAT-MODEL.md) — Honest threat model and assumptions (12 numbered sections + residual-risks table)
|
||||
- [docs/audit/REVIEW-BUNDLE.md](./docs/audit/REVIEW-BUNDLE.md) — External crypto-review entrypoint (scope, build instructions, reporting)
|
||||
- [docs/audit/SCOPE.md](./docs/audit/SCOPE.md) — One-page audit-scope summary
|
||||
|
||||
**Migration + history**
|
||||
|
||||
- [MIGRATION.md](./MIGRATION.md) — How to replace existing crypto with Shade + the [0.3.x → 4.0 upgrade path](./MIGRATION.md#migrating-from-03x-to-40-ga)
|
||||
- [CHANGELOG.md](./CHANGELOG.md) — `4.0.0` GA section + every prior release
|
||||
- [docs/archive/](./docs/archive/) — V2.1 / V2.2 / V2.3 backlog and V3.1 → V3.12 implementation plans (all `Status: Done`)
|
||||
- [docs/V5.0.md](./docs/V5.0.md) — Voice / Video / Broadcast (post-GA, built on the frozen 4.0 stack)
|
||||
|
||||
**Examples**
|
||||
|
||||
- [examples/](./examples/) — Runnable example applications, including
|
||||
[`07-streams-upload`](./examples/07-streams-upload) (multi-lane file transfer)
|
||||
and [`08-files-browser`](./examples/08-files-browser) (filesystem RPC)
|
||||
- [MIGRATION.md](./MIGRATION.md) — How to replace existing crypto with Shade
|
||||
|
||||
## Deployment — one container per project
|
||||
|
||||
@@ -274,15 +451,30 @@ docker run -d \
|
||||
-v my-project-shade:/data \
|
||||
-p 3900:3900 \
|
||||
-e SHADE_OBSERVER_TOKEN=change-me-to-at-least-16-chars \
|
||||
gt.zyon.no/stian/shade-prekey:latest
|
||||
gt.zyon.no/stian/shade-prekey:4.0.0
|
||||
```
|
||||
|
||||
The container includes:
|
||||
|
||||
- **Prekey server** — `/v1/keys/*` REST API
|
||||
- **Observer dashboard** — `/shade-observer/dashboard/` (off unless token is set)
|
||||
- **Inbox relay (V3.6)** — `/v1/inbox/*` async store-and-forward; enable
|
||||
with `SHADE_INBOX_DB_PATH=/data/inbox.db` (or `SHADE_INBOX_PG_URL`).
|
||||
`SHADE_INBOX_PRUNE_INTERVAL_MINUTES` controls TTL prune cadence.
|
||||
- **Bridge transports (V3.7)** — `/v1/bridge/{stream,poll,ws}` SSE +
|
||||
long-poll + WS adapters for clients that can't keep a WebSocket open.
|
||||
- **Transfer routes** — `/v1/transfer/*` chunk + state + control routes
|
||||
for `@shade/transfer`.
|
||||
- **Key Transparency (V3.12)** — `/v1/kt/*` exposes `log_id`, latest +
|
||||
historical STH, and consistency proofs. Enable with
|
||||
`SHADE_KT_*` env vars; off by default.
|
||||
- **Observer dashboard** — `/shade-observer/dashboard/` (off unless
|
||||
`SHADE_OBSERVER_TOKEN` is set)
|
||||
- **OpenAPI spec** — `/openapi.yaml` and interactive `/docs` viewer
|
||||
(covers all 27 routes — prekey, inbox, bridge, transfer, KT,
|
||||
observer, `/metrics`, `/healthz`, `/ready`)
|
||||
- **Prometheus metrics** — `/metrics`
|
||||
- **Health check** — `/health`
|
||||
- **Health probes** — `/health` (full), `/healthz` (liveness),
|
||||
`/ready` (readiness)
|
||||
- **Stale cleanup** — purges inactive identities automatically
|
||||
|
||||
See [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md) for the full deployment guide, environment variables, PostgreSQL config, backup strategy, and Dokploy instructions.
|
||||
|
||||
@@ -100,6 +100,9 @@ matrix row in the same change.
|
||||
| § 1 Network attacker — replay window | ±5 min `signedAt` enforcement | `packages/shade-server/tests/server.test.ts` (`"rejects registration with stale signedAt"`) |
|
||||
| § 1 Network attacker — header AAD | Ratchet headers bound to ciphertext | `packages/shade-core/tests/ratchet.test.ts`, `packages/shade-streams/tests/tamper.test.ts`, `packages/shade-streams/tests/aead.test.ts` |
|
||||
| § 1 Network attacker — forward secrecy | DH ratchet step + chain-key zeroize | `packages/shade-core/tests/ratchet.test.ts`, `packages/shade-crypto-web/tests/hardening.test.ts` |
|
||||
| § 1 Network attacker — streaming sub-session FS/replay (V4.11) | Per-frame Double-Ratchet `seal`/`open`; counter-rewind & replay rejected; in-memory-only (never persisted) | `packages/shade-core/tests/stream.test.ts` (`"R1: replayed / rewound frame is rejected"`, `"R2/R3: long one-directional burst stays correct and memory-bounded"`) |
|
||||
| § 1 Network attacker — streaming handshake auth (V4.11) | Identity-bound 3-DH against parent-session-pinned identities | `packages/shade-core/tests/stream.test.ts` (`"handshake is mutually authenticated against pinned identities"`) |
|
||||
| § 3 Endpoint compromise — streaming sub-session isolation (V4.11) | Stream ratchet derived without touching the stored parent session; zeroized on close | `packages/shade-core/tests/stream.test.ts` (`"R5: opening/using/closing a stream never touches the parent session"`, `"close() zeroizes and blocks further use; idempotent"`) |
|
||||
| § 2 Compromised prekey server — public-only storage | Prekey store never accepts a private key | `packages/shade-server/tests/server.test.ts`, `packages/shade-storage-sqlite/tests/sqlite-prekey-store.test.ts` |
|
||||
| § 2 Compromised prekey server — signed replenish/delete | Per-identity Ed25519 signature | `packages/shade-server/tests/server.test.ts` |
|
||||
| § 2 Compromised prekey server — fake-bundle detection | Out-of-band fingerprint comparison | `packages/shade-core/tests/fingerprint-session.test.ts` |
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
plugins {
|
||||
kotlin("jvm") version "2.0.20" apply false
|
||||
kotlin("android") version "2.0.20" apply false
|
||||
id("com.android.library") version "8.7.3" apply false
|
||||
}
|
||||
|
||||
@@ -2,3 +2,5 @@ org.gradle.jvmargs=-Xmx2g -Dfile.encoding=UTF-8
|
||||
org.gradle.parallel=true
|
||||
org.gradle.caching=true
|
||||
kotlin.code.style=official
|
||||
android.useAndroidX=true
|
||||
android.nonTransitiveRClass=true
|
||||
|
||||
@@ -4,6 +4,7 @@ pluginManagement {
|
||||
repositories {
|
||||
gradlePluginPortal()
|
||||
mavenCentral()
|
||||
google()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -17,3 +18,6 @@ dependencyResolutionManagement {
|
||||
|
||||
include(":shade-android")
|
||||
project(":shade-android").projectDir = file("shade-android")
|
||||
|
||||
include(":shade-android-keystore")
|
||||
project(":shade-android-keystore").projectDir = file("shade-android-keystore")
|
||||
|
||||
68
android/shade-android-keystore/README.md
Normal file
68
android/shade-android-keystore/README.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# shade-android-keystore
|
||||
|
||||
Android-specific bindings for `shade-android`. Lives as a sibling Gradle module so the JVM-only protocol code can keep running in CI without an Android SDK install.
|
||||
|
||||
Provides:
|
||||
|
||||
- **`KeystoreMasterKey`** — hardware-backed AES-256-GCM master key in the Android Keystore. Optionally biometric-gated (BIOMETRIC_STRONG only — Class 3 assurance), StrongBox-backed when available, invalidated on new biometric enrollment.
|
||||
- **`BiometricUnlock`** — coroutine wrapper around `BiometricPrompt` for unlocking a `Cipher` instance bound to the keystore key. Throws `BiometricCancelledException` / `BiometricFailedException` so callers can handle the auth flow without writing custom callbacks.
|
||||
- **`KeystoreStorage`** — `StorageProvider` implementation that persists session/identity/prekey state to `SharedPreferences`, each row encrypted under the keystore key with the row's preference key bound as AAD.
|
||||
|
||||
## Usage
|
||||
|
||||
```kotlin
|
||||
import androidx.fragment.app.FragmentActivity
|
||||
import no.zyon.shade.ShadeSessionManager
|
||||
import no.zyon.shade.crypto.TinkProvider
|
||||
import no.zyon.shade.keystore.BiometricUnlock
|
||||
import no.zyon.shade.keystore.KeystoreStorage
|
||||
|
||||
class MyActivity : FragmentActivity() {
|
||||
private val crypto = TinkProvider()
|
||||
private lateinit var storage: KeystoreStorage
|
||||
private lateinit var manager: ShadeSessionManager
|
||||
|
||||
override fun onCreate(savedInstanceState: Bundle?) {
|
||||
super.onCreate(savedInstanceState)
|
||||
|
||||
storage = KeystoreStorage(this, crypto)
|
||||
|
||||
lifecycleScope.launch {
|
||||
val unlock = BiometricUnlock(
|
||||
activity = this@MyActivity,
|
||||
title = "Unlock Shade",
|
||||
subtitle = "Tap your fingerprint to access your messages",
|
||||
)
|
||||
try {
|
||||
storage.unlock(unlock)
|
||||
} catch (e: BiometricCancelledException) {
|
||||
// user backed out — show a "tap to retry" UI
|
||||
return@launch
|
||||
}
|
||||
|
||||
manager = ShadeSessionManager(crypto, storage)
|
||||
manager.initialize()
|
||||
// ... use manager normally
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For credential-driven bootstrap (V4.9 profile + V4.10 approval), pair this with `no.zyon.shade.blob.createProfileNamespace` and `no.zyon.shade.approval.signProxyApproval` — both pure-JVM (in `:shade-android`).
|
||||
|
||||
## Threat model
|
||||
|
||||
- **Compromised app process**: cannot read the AES key (it's in the secure environment). Can attempt to use the cipher only after the user has authenticated; biometric re-prompts are required after each biometric event.
|
||||
- **Stolen device with known PIN**: cannot unlock — `setUserAuthenticationParameters(0, BIOMETRIC_STRONG)` excludes `DEVICE_CREDENTIAL`.
|
||||
- **Attacker enrolls own biometric**: `setInvalidatedByBiometricEnrollment(true)` invalidates the key on enrollment, forcing a credential rebootstrap (which would need username + password + PIN).
|
||||
- **Catastrophic recovery**: `forgetEverything()` deletes the master key and clears the SharedPreferences. Pair with `Profile.delete()` for full account erasure.
|
||||
|
||||
## Build
|
||||
|
||||
Requires an Android SDK. The Gradle build uses Android Gradle Plugin 8.7+, AGP minSdk 28 (Pie+ for BiometricPrompt baseline), targetSdk 35.
|
||||
|
||||
```bash
|
||||
JAVA_HOME=/path/to/jdk-21 ./gradlew :shade-android-keystore:assembleDebug
|
||||
```
|
||||
|
||||
Unit tests: none yet — `KeystoreStorage` requires Android runtime. Robolectric or instrumented tests against an emulator are tracked as a follow-up. The pure-JVM `SessionStateJson` round-trip serializer is tested in `:shade-android` (`SessionStateJsonTest`).
|
||||
59
android/shade-android-keystore/build.gradle.kts
Normal file
59
android/shade-android-keystore/build.gradle.kts
Normal file
@@ -0,0 +1,59 @@
|
||||
plugins {
|
||||
id("com.android.library")
|
||||
kotlin("android")
|
||||
}
|
||||
|
||||
// V4.10 — Android-specific KeystoreStorage adapter.
|
||||
//
|
||||
// Lives as a sibling module to `:shade-android` so the JVM-only
|
||||
// protocol code can keep running in CI without an Android SDK.
|
||||
// This module pulls in `:shade-android` for `StorageProvider`,
|
||||
// `IdentityKeyPair`, etc., and binds those types to a hardware-
|
||||
// backed Android Keystore master key with biometric gating.
|
||||
|
||||
android {
|
||||
namespace = "no.zyon.shade.keystore"
|
||||
compileSdk = 35
|
||||
|
||||
defaultConfig {
|
||||
minSdk = 28 // BiometricPrompt + StrongBox baseline
|
||||
}
|
||||
|
||||
compileOptions {
|
||||
sourceCompatibility = JavaVersion.VERSION_17
|
||||
targetCompatibility = JavaVersion.VERSION_17
|
||||
}
|
||||
|
||||
kotlinOptions {
|
||||
jvmTarget = "17"
|
||||
}
|
||||
|
||||
buildTypes {
|
||||
release {
|
||||
isMinifyEnabled = false
|
||||
}
|
||||
}
|
||||
|
||||
testOptions {
|
||||
unitTests.isReturnDefaultValues = true
|
||||
}
|
||||
}
|
||||
|
||||
dependencies {
|
||||
// Sibling: protocol types + StorageProvider interface.
|
||||
api(project(":shade-android"))
|
||||
|
||||
// androidx.biometric — fragment-safe BiometricPrompt wrapper.
|
||||
// 1.2.0-alpha05 is the latest with stable BiometricPrompt API.
|
||||
implementation("androidx.biometric:biometric:1.2.0-alpha05")
|
||||
|
||||
// androidx.fragment — BiometricPrompt requires FragmentActivity.
|
||||
implementation("androidx.fragment:fragment-ktx:1.8.5")
|
||||
|
||||
// Coroutines for the suspend-function StorageProvider implementation.
|
||||
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.9.0")
|
||||
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.9.0")
|
||||
|
||||
testImplementation("junit:junit:4.13.2")
|
||||
testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.9.0")
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<manifest xmlns:android="http://schemas.android.com/apk/res/android" />
|
||||
@@ -0,0 +1,112 @@
|
||||
package no.zyon.shade.keystore
|
||||
|
||||
import androidx.biometric.BiometricManager
|
||||
import androidx.biometric.BiometricPrompt
|
||||
import androidx.fragment.app.FragmentActivity
|
||||
import javax.crypto.Cipher
|
||||
import kotlin.coroutines.resume
|
||||
import kotlin.coroutines.resumeWithException
|
||||
import kotlinx.coroutines.suspendCancellableCoroutine
|
||||
|
||||
/**
|
||||
* Biometric unlock for a `KeystoreMasterKey`-bound `Cipher`.
|
||||
*
|
||||
* The Android keystore enforces that any operation on a
|
||||
* user-authentication-required key must happen via a
|
||||
* `BiometricPrompt.CryptoObject`-wrapped `Cipher`. The user sees a
|
||||
* system biometric prompt; on success the same `Cipher` instance is
|
||||
* usable for one operation (or one streaming session) before
|
||||
* needing to re-prompt.
|
||||
*
|
||||
* This is a thin coroutine wrapper around `BiometricPrompt` that
|
||||
* resolves to the authenticated cipher or throws on user
|
||||
* cancellation. Callers typically run it once at app start to
|
||||
* unlock the master key for the lifetime of the foreground session.
|
||||
*/
|
||||
class BiometricUnlock(
|
||||
private val activity: FragmentActivity,
|
||||
private val title: String,
|
||||
private val subtitle: String? = null,
|
||||
private val negativeButton: String = "Cancel",
|
||||
) {
|
||||
|
||||
/**
|
||||
* True if BIOMETRIC_STRONG is currently usable on this device.
|
||||
* False means the user has no enrolled fingerprint/face that
|
||||
* meets the class-3 assurance level — fall back to a credential
|
||||
* recovery flow rather than crashing.
|
||||
*/
|
||||
fun canAuthenticate(): Boolean {
|
||||
val mgr = BiometricManager.from(activity)
|
||||
return mgr.canAuthenticate(BiometricManager.Authenticators.BIOMETRIC_STRONG) ==
|
||||
BiometricManager.BIOMETRIC_SUCCESS
|
||||
}
|
||||
|
||||
/**
|
||||
* Show the biometric prompt and return the authenticated cipher.
|
||||
*
|
||||
* Cancellation paths:
|
||||
* - User taps the negative button → throws `BiometricCancelledException`.
|
||||
* - System errors out (e.g. too many failures) → throws
|
||||
* `BiometricFailedException` with the system error code.
|
||||
*/
|
||||
suspend fun unlock(cipher: Cipher): Cipher = suspendCancellableCoroutine { cont ->
|
||||
val callback = object : BiometricPrompt.AuthenticationCallback() {
|
||||
override fun onAuthenticationSucceeded(result: BiometricPrompt.AuthenticationResult) {
|
||||
val authedCipher = result.cryptoObject?.cipher
|
||||
if (authedCipher == null) {
|
||||
cont.resumeWithException(
|
||||
BiometricFailedException(-1, "BiometricPrompt returned no cipher"),
|
||||
)
|
||||
} else {
|
||||
cont.resume(authedCipher)
|
||||
}
|
||||
}
|
||||
|
||||
override fun onAuthenticationError(errorCode: Int, errString: CharSequence) {
|
||||
if (errorCode == BiometricPrompt.ERROR_USER_CANCELED ||
|
||||
errorCode == BiometricPrompt.ERROR_NEGATIVE_BUTTON ||
|
||||
errorCode == BiometricPrompt.ERROR_CANCELED
|
||||
) {
|
||||
cont.resumeWithException(BiometricCancelledException(errString.toString()))
|
||||
} else {
|
||||
cont.resumeWithException(
|
||||
BiometricFailedException(errorCode, errString.toString()),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
override fun onAuthenticationFailed() {
|
||||
// A single failed attempt — the prompt stays open and
|
||||
// gives the user another try. Don't resume the
|
||||
// continuation; let the system flow continue.
|
||||
}
|
||||
}
|
||||
|
||||
val prompt = BiometricPrompt(
|
||||
activity,
|
||||
activity.mainExecutor,
|
||||
callback,
|
||||
)
|
||||
val info = BiometricPrompt.PromptInfo.Builder()
|
||||
.setTitle(title)
|
||||
.apply { if (subtitle != null) setSubtitle(subtitle) }
|
||||
.setNegativeButtonText(negativeButton)
|
||||
.setAllowedAuthenticators(BiometricManager.Authenticators.BIOMETRIC_STRONG)
|
||||
.build()
|
||||
|
||||
cont.invokeOnCancellation { prompt.cancelAuthentication() }
|
||||
prompt.authenticate(info, BiometricPrompt.CryptoObject(cipher))
|
||||
}
|
||||
}
|
||||
|
||||
/** User cancelled the biometric prompt. */
|
||||
class BiometricCancelledException(message: String) : RuntimeException(message)
|
||||
|
||||
/**
|
||||
* BiometricPrompt returned a non-cancellation error (lockout, hardware
|
||||
* unavailable, no enrolled biometrics, etc.). Inspect `errorCode`
|
||||
* against `BiometricPrompt.ERROR_*` constants to decide UX response.
|
||||
*/
|
||||
class BiometricFailedException(val errorCode: Int, message: String) :
|
||||
RuntimeException("[$errorCode] $message")
|
||||
@@ -0,0 +1,172 @@
|
||||
package no.zyon.shade.keystore
|
||||
|
||||
import android.os.Build
|
||||
import android.security.keystore.KeyGenParameterSpec
|
||||
import android.security.keystore.KeyProperties
|
||||
import javax.crypto.Cipher
|
||||
import javax.crypto.KeyGenerator
|
||||
import javax.crypto.SecretKey
|
||||
import javax.crypto.spec.GCMParameterSpec
|
||||
import java.security.KeyStore
|
||||
|
||||
/**
|
||||
* Hardware-backed AES-256-GCM master key in the Android Keystore.
|
||||
*
|
||||
* The key never leaves the secure environment — Android's keystore
|
||||
* implementation enforces that all encrypt/decrypt operations
|
||||
* happen inside the TEE (or StrongBox if present), and the raw
|
||||
* key bytes are never returned to userspace.
|
||||
*
|
||||
* The key is created on first use with these properties:
|
||||
*
|
||||
* - AES-256-GCM, no padding
|
||||
* - User authentication required: opt-in via the `requireBiometric`
|
||||
* flag. When true, every encrypt/decrypt operation must be wrapped
|
||||
* in a `BiometricPrompt.authenticate(CryptoObject(cipher))` call
|
||||
* that succeeds within the same `Cipher` instance.
|
||||
* - StrongBox-backed if available (Pixel 3+, most Samsung flagships).
|
||||
* Falls back to TEE on devices without StrongBox.
|
||||
* - InvalidatedByBiometricEnrollment(true): a newly enrolled
|
||||
* fingerprint/face invalidates the key, forcing the user to
|
||||
* re-bootstrap from credentials. Defends against a thief who
|
||||
* enrolls their own biometric.
|
||||
*
|
||||
* Mirrors the role `KeyManager` plays in `@shade/storage-encrypted`'s
|
||||
* V4.5 KDF chain: this is the *encryption-at-rest* master key, not
|
||||
* the X3DH identity key. The Shade protocol's identity keys are
|
||||
* stored encrypted under THIS key.
|
||||
*/
|
||||
class KeystoreMasterKey(
|
||||
private val alias: String,
|
||||
private val requireBiometric: Boolean = true,
|
||||
) {
|
||||
|
||||
init {
|
||||
require(alias.isNotEmpty()) { "alias must be non-empty" }
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a `Cipher` initialized for encryption with the master key.
|
||||
*
|
||||
* If the key requires user auth, the returned cipher is *not yet
|
||||
* usable* — the caller MUST wrap it in a
|
||||
* `BiometricPrompt.authenticate(CryptoObject(cipher))` and use
|
||||
* the cipher exposed by the auth-success callback. Calling
|
||||
* `cipher.doFinal(...)` before authentication throws
|
||||
* `UserNotAuthenticatedException`.
|
||||
*/
|
||||
fun cipherForEncrypt(): Cipher {
|
||||
val key = getOrCreateKey()
|
||||
val cipher = Cipher.getInstance(TRANSFORMATION)
|
||||
cipher.init(Cipher.ENCRYPT_MODE, key)
|
||||
return cipher
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a `Cipher` initialized for decryption with the master key
|
||||
* and a previously-stored 12-byte nonce. Same authentication
|
||||
* requirement as `cipherForEncrypt`.
|
||||
*/
|
||||
fun cipherForDecrypt(nonce: ByteArray): Cipher {
|
||||
require(nonce.size == 12) { "GCM nonce must be 12 bytes" }
|
||||
val key = getOrCreateKey()
|
||||
val cipher = Cipher.getInstance(TRANSFORMATION)
|
||||
cipher.init(Cipher.DECRYPT_MODE, key, GCMParameterSpec(128, nonce))
|
||||
return cipher
|
||||
}
|
||||
|
||||
/** True if the key already exists in the Android Keystore. */
|
||||
fun exists(): Boolean {
|
||||
val ks = KeyStore.getInstance(ANDROID_KEYSTORE).apply { load(null) }
|
||||
return ks.containsAlias(alias)
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete the master key. Catastrophic — all data encrypted under
|
||||
* it becomes unrecoverable. Used by the "forget everything" flow
|
||||
* (paired with `Profile.delete()` in the V4.9 namespace).
|
||||
*/
|
||||
fun deleteKey() {
|
||||
val ks = KeyStore.getInstance(ANDROID_KEYSTORE).apply { load(null) }
|
||||
if (ks.containsAlias(alias)) ks.deleteEntry(alias)
|
||||
}
|
||||
|
||||
private fun getOrCreateKey(): SecretKey {
|
||||
val ks = KeyStore.getInstance(ANDROID_KEYSTORE).apply { load(null) }
|
||||
ks.getEntry(alias, null)?.let { entry ->
|
||||
return (entry as KeyStore.SecretKeyEntry).secretKey
|
||||
}
|
||||
return generateKey()
|
||||
}
|
||||
|
||||
private fun generateKey(): SecretKey {
|
||||
val builder = KeyGenParameterSpec.Builder(
|
||||
alias,
|
||||
KeyProperties.PURPOSE_ENCRYPT or KeyProperties.PURPOSE_DECRYPT,
|
||||
)
|
||||
.setBlockModes(KeyProperties.BLOCK_MODE_GCM)
|
||||
.setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_NONE)
|
||||
.setKeySize(256)
|
||||
// Each encrypt operation generates a fresh IV in the secure
|
||||
// env; we read it back via `cipher.iv` after init.
|
||||
.setRandomizedEncryptionRequired(true)
|
||||
|
||||
if (requireBiometric) {
|
||||
builder.setUserAuthenticationRequired(true)
|
||||
// BIOMETRIC_STRONG only — class 3, the highest assurance
|
||||
// level (Class 3 = false-accept rate < 1/50 000 per BiometricPrompt).
|
||||
// DEVICE_CREDENTIAL is intentionally NOT included: a stolen
|
||||
// device with a known PIN should not unlock Shade.
|
||||
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.R) {
|
||||
builder.setUserAuthenticationParameters(
|
||||
/* timeout = */ 0,
|
||||
KeyProperties.AUTH_BIOMETRIC_STRONG,
|
||||
)
|
||||
} else {
|
||||
@Suppress("DEPRECATION")
|
||||
builder.setUserAuthenticationValidityDurationSeconds(-1)
|
||||
}
|
||||
builder.setInvalidatedByBiometricEnrollment(true)
|
||||
}
|
||||
|
||||
// StrongBox if available — bumps key storage to a dedicated
|
||||
// tamper-resistant chip on Pixel 3+ / most Samsung flagships.
|
||||
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) {
|
||||
builder.setIsStrongBoxBacked(true)
|
||||
}
|
||||
|
||||
val gen = KeyGenerator.getInstance(KeyProperties.KEY_ALGORITHM_AES, ANDROID_KEYSTORE)
|
||||
return try {
|
||||
gen.init(builder.build())
|
||||
gen.generateKey()
|
||||
} catch (_: Exception) {
|
||||
// StrongBox not present or full → retry without StrongBox.
|
||||
// Same for older devices that don't honor
|
||||
// setUserAuthenticationParameters.
|
||||
val fallback = KeyGenParameterSpec.Builder(
|
||||
alias,
|
||||
KeyProperties.PURPOSE_ENCRYPT or KeyProperties.PURPOSE_DECRYPT,
|
||||
)
|
||||
.setBlockModes(KeyProperties.BLOCK_MODE_GCM)
|
||||
.setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_NONE)
|
||||
.setKeySize(256)
|
||||
.setRandomizedEncryptionRequired(true)
|
||||
.apply {
|
||||
if (requireBiometric) {
|
||||
setUserAuthenticationRequired(true)
|
||||
@Suppress("DEPRECATION")
|
||||
setUserAuthenticationValidityDurationSeconds(-1)
|
||||
setInvalidatedByBiometricEnrollment(true)
|
||||
}
|
||||
}
|
||||
.build()
|
||||
gen.init(fallback)
|
||||
gen.generateKey()
|
||||
}
|
||||
}
|
||||
|
||||
companion object {
|
||||
private const val ANDROID_KEYSTORE = "AndroidKeyStore"
|
||||
private const val TRANSFORMATION = "AES/GCM/NoPadding"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
package no.zyon.shade.keystore
|
||||
|
||||
import android.content.Context
|
||||
import android.content.SharedPreferences
|
||||
import android.util.Base64
|
||||
import kotlinx.coroutines.sync.Mutex
|
||||
import kotlinx.coroutines.sync.withLock
|
||||
import no.zyon.shade.crypto.CryptoProvider
|
||||
import no.zyon.shade.serialization.SessionStateJson
|
||||
import no.zyon.shade.storage.StorageProvider
|
||||
import no.zyon.shade.types.IdentityKeyPair
|
||||
import no.zyon.shade.types.OneTimePreKey
|
||||
import no.zyon.shade.types.SessionState
|
||||
import no.zyon.shade.types.SignedPreKey
|
||||
|
||||
/**
|
||||
* `StorageProvider` implementation that gates all reads/writes through
|
||||
* a biometric-locked `KeystoreMasterKey`. Mirrors `MemoryStorage` for
|
||||
* the API surface but persists state to `SharedPreferences` with each
|
||||
* row encrypted under the keystore key.
|
||||
*
|
||||
* Lifecycle:
|
||||
*
|
||||
* 1. App start → construct `KeystoreStorage(context, alias)`.
|
||||
* 2. `unlock(BiometricUnlock)` runs the system biometric prompt.
|
||||
* The Android keystore caches the auth state under the key's
|
||||
* `setUserAuthenticationParameters(0, BIOMETRIC_STRONG)` policy
|
||||
* until the next biometric event (re-enrollment, etc.).
|
||||
* 3. While unlocked, `getSession`/`saveSession` etc. work normally.
|
||||
* 4. `lock()` clears the in-memory unlocked flag so a future
|
||||
* operation triggers another biometric prompt.
|
||||
*
|
||||
* Wire layout per row:
|
||||
* `<base64(nonce(12))>:<base64(ct||tag)>`
|
||||
*
|
||||
* Stored as `String` SharedPreferences entries. AAD = the row's
|
||||
* preference key (`session:<address>`, `signedPreKey:<id>`, etc.) so
|
||||
* a substituted-prefs swap fails to open.
|
||||
*/
|
||||
class KeystoreStorage(
|
||||
context: Context,
|
||||
private val crypto: CryptoProvider,
|
||||
keyAlias: String = DEFAULT_KEY_ALIAS,
|
||||
prefsName: String = DEFAULT_PREFS_NAME,
|
||||
requireBiometric: Boolean = true,
|
||||
) : StorageProvider {
|
||||
|
||||
private val prefs: SharedPreferences =
|
||||
context.applicationContext.getSharedPreferences(prefsName, Context.MODE_PRIVATE)
|
||||
private val masterKey = KeystoreMasterKey(keyAlias, requireBiometric = requireBiometric)
|
||||
private val writeMutex = Mutex()
|
||||
|
||||
@Volatile
|
||||
private var unlocked: Boolean = false
|
||||
|
||||
/**
|
||||
* Unlock the keystore via biometric prompt. Idempotent — calling
|
||||
* twice without a `lock()` between is a no-op.
|
||||
*/
|
||||
suspend fun unlock(unlock: BiometricUnlock) {
|
||||
if (unlocked) return
|
||||
// The biometric flow returns an authenticated *encrypt*
|
||||
// cipher; we discard it after a one-shot probe to confirm
|
||||
// the master key is reachable. The actual encrypt/decrypt
|
||||
// ciphers in the I/O path use the authentication state
|
||||
// established here (Android Keystore caches the auth for
|
||||
// the user-authentication-required key under the
|
||||
// `setUserAuthenticationParameters(0, BIOMETRIC_STRONG)`
|
||||
// policy until the next biometric event).
|
||||
val probe = masterKey.cipherForEncrypt()
|
||||
unlock.unlock(probe)
|
||||
unlocked = true
|
||||
}
|
||||
|
||||
/** Unlock without biometric — only valid for keys constructed with `requireBiometric=false`. */
|
||||
fun unlockNoBiometric() {
|
||||
unlocked = true
|
||||
}
|
||||
|
||||
/** Wipe in-memory unlock state. The key itself stays in the keystore. */
|
||||
fun lock() {
|
||||
unlocked = false
|
||||
}
|
||||
|
||||
/**
|
||||
* Catastrophic reset: deletes the master key + all encrypted
|
||||
* preferences. Used by the "forget everything" / 3-strikes-wipe
|
||||
* path. The next bootstrap rebuilds from credentials.
|
||||
*/
|
||||
fun forgetEverything() {
|
||||
masterKey.deleteKey()
|
||||
prefs.edit().clear().apply()
|
||||
unlocked = false
|
||||
}
|
||||
|
||||
// ─── Identity ──────────────────────────────────────────────
|
||||
|
||||
override suspend fun getIdentityKeyPair(): IdentityKeyPair? {
|
||||
val json = readDecrypted(KEY_IDENTITY) ?: return null
|
||||
return SessionStateJson.deserializeIdentityKeyPair(json)
|
||||
}
|
||||
|
||||
override suspend fun saveIdentityKeyPair(keyPair: IdentityKeyPair) {
|
||||
writeEncrypted(KEY_IDENTITY, SessionStateJson.serializeIdentityKeyPair(keyPair))
|
||||
}
|
||||
|
||||
override suspend fun getLocalRegistrationId(): Int {
|
||||
return readDecrypted(KEY_REGISTRATION_ID)?.toIntOrNull() ?: 0
|
||||
}
|
||||
|
||||
override suspend fun saveLocalRegistrationId(id: Int) {
|
||||
writeEncrypted(KEY_REGISTRATION_ID, id.toString())
|
||||
}
|
||||
|
||||
// ─── Signed prekeys ────────────────────────────────────────
|
||||
|
||||
override suspend fun getSignedPreKey(keyId: Int): SignedPreKey? {
|
||||
val json = readDecrypted("$KEY_SIGNED_PREKEY:$keyId") ?: return null
|
||||
return SessionStateJson.deserializeSignedPreKey(json)
|
||||
}
|
||||
|
||||
override suspend fun saveSignedPreKey(key: SignedPreKey) {
|
||||
writeEncrypted(
|
||||
"$KEY_SIGNED_PREKEY:${key.keyId}",
|
||||
SessionStateJson.serializeSignedPreKey(key),
|
||||
)
|
||||
}
|
||||
|
||||
override suspend fun removeSignedPreKey(keyId: Int) {
|
||||
writeMutex.withLock { prefs.edit().remove("$KEY_SIGNED_PREKEY:$keyId").apply() }
|
||||
}
|
||||
|
||||
// ─── One-time prekeys ──────────────────────────────────────
|
||||
|
||||
override suspend fun getOneTimePreKey(keyId: Int): OneTimePreKey? {
|
||||
val json = readDecrypted("$KEY_ONETIME_PREKEY:$keyId") ?: return null
|
||||
return SessionStateJson.deserializeOneTimePreKey(json)
|
||||
}
|
||||
|
||||
override suspend fun saveOneTimePreKey(key: OneTimePreKey) {
|
||||
writeEncrypted(
|
||||
"$KEY_ONETIME_PREKEY:${key.keyId}",
|
||||
SessionStateJson.serializeOneTimePreKey(key),
|
||||
)
|
||||
}
|
||||
|
||||
override suspend fun removeOneTimePreKey(keyId: Int) {
|
||||
writeMutex.withLock { prefs.edit().remove("$KEY_ONETIME_PREKEY:$keyId").apply() }
|
||||
}
|
||||
|
||||
override suspend fun getOneTimePreKeyCount(): Int {
|
||||
return prefs.all.keys.count { it.startsWith("$KEY_ONETIME_PREKEY:") }
|
||||
}
|
||||
|
||||
// ─── Sessions ──────────────────────────────────────────────
|
||||
|
||||
override suspend fun getSession(address: String): SessionState? {
|
||||
val json = readDecrypted("$KEY_SESSION:$address") ?: return null
|
||||
return SessionStateJson.deserialize(json)
|
||||
}
|
||||
|
||||
override suspend fun saveSession(address: String, state: SessionState) {
|
||||
writeEncrypted("$KEY_SESSION:$address", SessionStateJson.serialize(state))
|
||||
}
|
||||
|
||||
override suspend fun removeSession(address: String) {
|
||||
writeMutex.withLock { prefs.edit().remove("$KEY_SESSION:$address").apply() }
|
||||
}
|
||||
|
||||
// ─── Trust ─────────────────────────────────────────────────
|
||||
|
||||
override suspend fun isTrustedIdentity(address: String, identityKey: ByteArray): Boolean {
|
||||
val stored = readDecrypted("$KEY_TRUSTED:$address") ?: return true // TOFU
|
||||
val storedBytes = Base64.decode(stored, Base64.NO_WRAP)
|
||||
return crypto.constantTimeEqual(storedBytes, identityKey)
|
||||
}
|
||||
|
||||
override suspend fun saveTrustedIdentity(address: String, identityKey: ByteArray) {
|
||||
writeEncrypted(
|
||||
"$KEY_TRUSTED:$address",
|
||||
Base64.encodeToString(identityKey, Base64.NO_WRAP),
|
||||
)
|
||||
}
|
||||
|
||||
// ─── Encrypted-row plumbing ────────────────────────────────
|
||||
|
||||
private fun ensureUnlocked() {
|
||||
check(unlocked) {
|
||||
"KeystoreStorage is locked — call unlock(BiometricUnlock) first"
|
||||
}
|
||||
}
|
||||
|
||||
private suspend fun readDecrypted(prefKey: String): String? {
|
||||
ensureUnlocked()
|
||||
val raw = prefs.getString(prefKey, null) ?: return null
|
||||
val parts = raw.split(":", limit = 2)
|
||||
require(parts.size == 2) { "malformed encrypted row at $prefKey" }
|
||||
val nonce = Base64.decode(parts[0], Base64.NO_WRAP)
|
||||
val ct = Base64.decode(parts[1], Base64.NO_WRAP)
|
||||
val cipher = masterKey.cipherForDecrypt(nonce)
|
||||
cipher.updateAAD(prefKey.toByteArray(Charsets.UTF_8))
|
||||
return cipher.doFinal(ct).toString(Charsets.UTF_8)
|
||||
}
|
||||
|
||||
private suspend fun writeEncrypted(prefKey: String, plaintext: String) {
|
||||
ensureUnlocked()
|
||||
writeMutex.withLock {
|
||||
val cipher = masterKey.cipherForEncrypt()
|
||||
cipher.updateAAD(prefKey.toByteArray(Charsets.UTF_8))
|
||||
val ct = cipher.doFinal(plaintext.toByteArray(Charsets.UTF_8))
|
||||
val nonce = cipher.iv
|
||||
val nonceB64 = Base64.encodeToString(nonce, Base64.NO_WRAP)
|
||||
val ctB64 = Base64.encodeToString(ct, Base64.NO_WRAP)
|
||||
prefs.edit().putString(prefKey, "$nonceB64:$ctB64").apply()
|
||||
}
|
||||
}
|
||||
|
||||
companion object {
|
||||
const val DEFAULT_KEY_ALIAS = "shade-master-v1"
|
||||
const val DEFAULT_PREFS_NAME = "shade-keystore-storage-v1"
|
||||
|
||||
private const val KEY_IDENTITY = "identity"
|
||||
private const val KEY_REGISTRATION_ID = "registrationId"
|
||||
private const val KEY_SIGNED_PREKEY = "signedPreKey"
|
||||
private const val KEY_ONETIME_PREKEY = "oneTimePreKey"
|
||||
private const val KEY_SESSION = "session"
|
||||
private const val KEY_TRUSTED = "trusted"
|
||||
}
|
||||
}
|
||||
@@ -7,7 +7,11 @@ Kotlin implementation of the Shade E2EE protocol for Android apps. Byte-for-byte
|
||||
**M-Cross 1 ✅** — keys + HKDF + X3DH + fingerprint.
|
||||
**M-Cross 2 ✅** — full ratchet step (encrypt + decrypt roundtrip) + wire 0x02 (RatchetMessage and PreKeyMessage with/without OTPK).
|
||||
**M-Cross 3 ✅** — streams 0x11 (KDF labels with embedded NULs, deterministic chunk nonce/AAD, wire 0x11 encode/decode).
|
||||
**M-Cross 4 ✅** — backup-key HKDF + AES-GCM, group sender-key step (`kdfChainKey` + Ed25519 sign over `aad ‖ ct`), storage HKDF (storageKey/fieldKey/rowNonce). Pending: scrypt master-key, argon2id swap, Android KeystoreStorage (sibling module).
|
||||
**M-Cross 4 ✅** — backup-key HKDF + AES-GCM, group sender-key step (`kdfChainKey` + Ed25519 sign over `aad ‖ ct`), storage HKDF (storageKey/fieldKey/rowNonce).
|
||||
**M-Cross 5 ✅** — V4.9 blob KDF + AEAD (`deriveBlobSlotId / deriveBlobKey / deriveBlobSigningSeed`, AAD-bound seal/open), `BlobClient` HTTP, `Profile` namespace. Cross-platform vectors in `blob.json`.
|
||||
**M-Cross 6 ✅** — V4.10 cross-host approval routing: canonical profile-blob schema (`hosts[]` / `clients[]` / `trustedApproverFingerprints[]`), build/sign/verify proxy approvals via `canonicalApprovalSigningBytes` (length-prefixed u16 BE UTF-8). Cross-platform vectors in `approval.json`, including a TS-signed Ed25519 signature that the Kotlin port verifies.
|
||||
**M-Cross 7 ✅** — scrypt + argon2id password-KDF wrappers (Bouncy Castle), NFKC-normalized inputs.
|
||||
**M-Cross 8 ✅** — `:shade-android-keystore` sibling module: `KeystoreMasterKey` (StrongBox-backed AES-256-GCM, BIOMETRIC_STRONG-gated, invalidated on biometric enrollment), `BiometricUnlock`, `KeystoreStorage` (`StorageProvider` over biometric-gated AES-encrypted SharedPreferences).
|
||||
|
||||
Cross-platform test vectors in `/test-vectors/` are loaded by both the TS
|
||||
and Kotlin test suites; any byte-divergence fails CI within 60 s. See
|
||||
|
||||
@@ -29,6 +29,11 @@ dependencies {
|
||||
// The same `subtle.*` API as `tink-android` so the source compiles unchanged.
|
||||
implementation("com.google.crypto.tink:tink:1.15.0")
|
||||
|
||||
// Bouncy Castle for scrypt + argon2id. Tink doesn't ship password
|
||||
// KDFs; @shade/storage-encrypted uses @noble/hashes for both. We
|
||||
// pin to the JDK18-on artifact so it works on JVM 17 + Android.
|
||||
implementation("org.bouncycastle:bcprov-jdk18on:1.78.1")
|
||||
|
||||
// JSON serialization (session state + test-vector loader on JVM).
|
||||
implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.7.3")
|
||||
|
||||
|
||||
@@ -0,0 +1,273 @@
|
||||
package no.zyon.shade.approval
|
||||
|
||||
import no.zyon.shade.crypto.CryptoProvider
|
||||
import java.nio.ByteBuffer
|
||||
import java.nio.ByteOrder
|
||||
|
||||
/**
|
||||
* V4.10 — cross-host approval routing helpers. Mirror
|
||||
* `@shade/sdk/approval.ts` byte-for-byte.
|
||||
*
|
||||
* The frames themselves (`approvalNeeded` / `linkApproveByProxy`) are
|
||||
* app-defined payloads sent over the existing Shade bilateral E2EE
|
||||
* channel. This file ships the canonical signing-payload layout, the
|
||||
* Ed25519 sign step a phone runs after biometric unlock, and the
|
||||
* verify step a host runs against the freshest profile blob.
|
||||
*
|
||||
* The signing payload is length-prefixed binary (u16 BE) so any
|
||||
* platform — Kotlin, Swift, Go — can produce byte-identical input
|
||||
* without needing a JSON canonicalizer. Cross-platform parity is
|
||||
* gated by `test-vectors/blob-storage.json` (signing payload
|
||||
* fixtures) plus a Kotlin↔TS round-trip in `CrossPlatformVectorTest`.
|
||||
*/
|
||||
|
||||
/** Default domain separator. Apps with their own canonical name (e.g. Prism) override. */
|
||||
const val DEFAULT_APPROVAL_DOMAIN = "shade-link-approve-v1"
|
||||
|
||||
/** Default expiry: 5 minutes after the host issues the request. */
|
||||
const val DEFAULT_APPROVAL_EXPIRES_IN_MS = 5L * 60 * 1000
|
||||
|
||||
/** Information about the device the host received a `linkRequest` from. */
|
||||
data class ApprovalRequestingDevice(
|
||||
val fingerprint: String,
|
||||
val deviceName: String? = null,
|
||||
val userAgent: String? = null,
|
||||
val ipHint: String? = null,
|
||||
val receivedAt: Long,
|
||||
)
|
||||
|
||||
data class ApprovalRequestFrame(
|
||||
val kind: String = "approvalNeeded",
|
||||
/** 128-bit hex (32 chars) random idempotency key. */
|
||||
val requestId: String,
|
||||
val hostAddress: String,
|
||||
val hostFingerprint: String,
|
||||
val requestingDevice: ApprovalRequestingDevice,
|
||||
val expiresAt: Long,
|
||||
val domain: String,
|
||||
)
|
||||
|
||||
data class ProxyApprovalFrame(
|
||||
val kind: String = "linkApproveByProxy",
|
||||
val requestId: String,
|
||||
val decision: String,
|
||||
val approverFingerprint: String,
|
||||
/** 64-byte Ed25519 signature, lowercase hex (128 chars). */
|
||||
val signature: String,
|
||||
val domain: String,
|
||||
)
|
||||
|
||||
/**
|
||||
* Build a fresh `approvalNeeded` frame with a 128-bit random
|
||||
* `requestId`. Hosts SHOULD persist the requestId in a pending-set
|
||||
* keyed by `expiresAt` so a returning `linkApproveByProxy` can be
|
||||
* matched up — that's app state, the SDK doesn't track it.
|
||||
*/
|
||||
fun buildApprovalRequest(
|
||||
crypto: CryptoProvider,
|
||||
hostAddress: String,
|
||||
hostFingerprint: String,
|
||||
requestingDeviceFingerprint: String,
|
||||
deviceName: String? = null,
|
||||
userAgent: String? = null,
|
||||
ipHint: String? = null,
|
||||
expiresInMs: Long = DEFAULT_APPROVAL_EXPIRES_IN_MS,
|
||||
domain: String = DEFAULT_APPROVAL_DOMAIN,
|
||||
now: Long = System.currentTimeMillis(),
|
||||
): ApprovalRequestFrame {
|
||||
val requestId = crypto.randomBytes(16).joinToString("") { "%02x".format(it) }
|
||||
return ApprovalRequestFrame(
|
||||
requestId = requestId,
|
||||
hostAddress = hostAddress,
|
||||
hostFingerprint = hostFingerprint,
|
||||
requestingDevice = ApprovalRequestingDevice(
|
||||
fingerprint = requestingDeviceFingerprint,
|
||||
deviceName = deviceName,
|
||||
userAgent = userAgent,
|
||||
ipHint = ipHint,
|
||||
receivedAt = now,
|
||||
),
|
||||
expiresAt = now + expiresInMs,
|
||||
domain = domain,
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Sign a `linkApproveByProxy` frame with the approver's long-term
|
||||
* Ed25519 identity key. The seed is the 32-byte Ed25519 private key
|
||||
* (Tink's `Ed25519Sign(seed)` consumes it directly).
|
||||
*/
|
||||
fun signProxyApproval(
|
||||
crypto: CryptoProvider,
|
||||
request: ApprovalRequestFrame,
|
||||
decision: String,
|
||||
approverFingerprint: String,
|
||||
approverSigningKey: ByteArray,
|
||||
): ProxyApprovalFrame {
|
||||
require(decision == "approve" || decision == "reject") {
|
||||
"decision must be 'approve' or 'reject'"
|
||||
}
|
||||
require(approverSigningKey.size == 32) {
|
||||
"approverSigningKey must be 32 bytes (Ed25519 seed)"
|
||||
}
|
||||
val payload = canonicalApprovalSigningBytes(
|
||||
domain = request.domain,
|
||||
requestId = request.requestId,
|
||||
hostFingerprint = request.hostFingerprint,
|
||||
requestingDeviceFingerprint = request.requestingDevice.fingerprint,
|
||||
decision = decision,
|
||||
)
|
||||
val sig = crypto.sign(approverSigningKey, payload)
|
||||
return ProxyApprovalFrame(
|
||||
requestId = request.requestId,
|
||||
decision = decision,
|
||||
approverFingerprint = approverFingerprint,
|
||||
signature = sig.joinToString("") { "%02x".format(it) },
|
||||
domain = request.domain,
|
||||
)
|
||||
}
|
||||
|
||||
/** Tagged result of `verifyProxyApproval`. */
|
||||
sealed class VerifyProxyApprovalResult {
|
||||
data class Ok(val approver: ProfileClientEntry) : VerifyProxyApprovalResult()
|
||||
data class Failed(val reason: Reason) : VerifyProxyApprovalResult()
|
||||
|
||||
enum class Reason {
|
||||
REQUEST_ID_MISMATCH,
|
||||
DOMAIN_MISMATCH,
|
||||
UNKNOWN_APPROVER,
|
||||
NOT_TRUSTED,
|
||||
BAD_SIGNATURE,
|
||||
EXPIRED,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Verify a `linkApproveByProxy` against the originating
|
||||
* `approvalNeeded` and the host's freshest profile blob. Returns a
|
||||
* tagged result rather than throwing — callers usually want to log
|
||||
* the reason before deciding what to surface to the user.
|
||||
*
|
||||
* Order of checks:
|
||||
*
|
||||
* 1. requestId match (replay defense)
|
||||
* 2. domain match (cross-app confusion defense)
|
||||
* 3. approver resolves to a `clients[]` entry
|
||||
* 4. approver is in `trustedApproverFingerprints[]` AND has the
|
||||
* `trustedApprover` flag (cross-checked via `isTrustedApprover`)
|
||||
* 5. expiresAt in the future
|
||||
* 6. Ed25519 signature verifies against `clients[].identityPublicKey`
|
||||
*
|
||||
* Hosts MUST refetch the profile blob fresh before calling this — see
|
||||
* the FR §5 "approver-revocation propagation" rationale.
|
||||
*/
|
||||
fun verifyProxyApproval(
|
||||
crypto: CryptoProvider,
|
||||
request: ApprovalRequestFrame,
|
||||
approval: ProxyApprovalFrame,
|
||||
profile: CanonicalProfileBlob,
|
||||
now: Long = System.currentTimeMillis(),
|
||||
): VerifyProxyApprovalResult {
|
||||
if (approval.requestId != request.requestId) {
|
||||
return VerifyProxyApprovalResult.Failed(
|
||||
VerifyProxyApprovalResult.Reason.REQUEST_ID_MISMATCH,
|
||||
)
|
||||
}
|
||||
if (approval.domain != request.domain) {
|
||||
return VerifyProxyApprovalResult.Failed(
|
||||
VerifyProxyApprovalResult.Reason.DOMAIN_MISMATCH,
|
||||
)
|
||||
}
|
||||
|
||||
val approver = findClientByFingerprint(profile, approval.approverFingerprint)
|
||||
?: return VerifyProxyApprovalResult.Failed(
|
||||
VerifyProxyApprovalResult.Reason.UNKNOWN_APPROVER,
|
||||
)
|
||||
|
||||
if (!isTrustedApprover(profile, approval.approverFingerprint)) {
|
||||
return VerifyProxyApprovalResult.Failed(
|
||||
VerifyProxyApprovalResult.Reason.NOT_TRUSTED,
|
||||
)
|
||||
}
|
||||
|
||||
if (now > request.expiresAt) {
|
||||
return VerifyProxyApprovalResult.Failed(VerifyProxyApprovalResult.Reason.EXPIRED)
|
||||
}
|
||||
|
||||
val pubkey = try {
|
||||
hexToBytes(approver.identityPublicKey)
|
||||
} catch (_: Exception) {
|
||||
return VerifyProxyApprovalResult.Failed(VerifyProxyApprovalResult.Reason.BAD_SIGNATURE)
|
||||
}
|
||||
val sig = try {
|
||||
hexToBytes(approval.signature)
|
||||
} catch (_: Exception) {
|
||||
return VerifyProxyApprovalResult.Failed(VerifyProxyApprovalResult.Reason.BAD_SIGNATURE)
|
||||
}
|
||||
if (pubkey.size != 32 || sig.size != 64) {
|
||||
return VerifyProxyApprovalResult.Failed(VerifyProxyApprovalResult.Reason.BAD_SIGNATURE)
|
||||
}
|
||||
|
||||
val payload = canonicalApprovalSigningBytes(
|
||||
domain = approval.domain,
|
||||
requestId = approval.requestId,
|
||||
hostFingerprint = request.hostFingerprint,
|
||||
requestingDeviceFingerprint = request.requestingDevice.fingerprint,
|
||||
decision = approval.decision,
|
||||
)
|
||||
val ok = crypto.verify(pubkey, payload, sig)
|
||||
return if (ok) VerifyProxyApprovalResult.Ok(approver)
|
||||
else VerifyProxyApprovalResult.Failed(VerifyProxyApprovalResult.Reason.BAD_SIGNATURE)
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the canonical signing payload bytes for a proxy approval.
|
||||
*
|
||||
* Format (length-prefixed UTF-8, big-endian u16 lengths):
|
||||
*
|
||||
* u16(len(domain)) || domain
|
||||
* u16(len(requestId)) || requestId
|
||||
* u16(len(hostFp)) || hostFingerprint
|
||||
* u16(len(requestFp)) || requestingDeviceFingerprint
|
||||
* u16(len(decision)) || decision
|
||||
*
|
||||
* This is the EXACT byte layout `@shade/sdk`'s
|
||||
* `canonicalApprovalSigningBytes` produces, ensuring an Android-signed
|
||||
* approval verifies on a TS host and vice versa.
|
||||
*/
|
||||
fun canonicalApprovalSigningBytes(
|
||||
domain: String,
|
||||
requestId: String,
|
||||
hostFingerprint: String,
|
||||
requestingDeviceFingerprint: String,
|
||||
decision: String,
|
||||
): ByteArray {
|
||||
val fields = listOf(
|
||||
domain.toByteArray(Charsets.UTF_8),
|
||||
requestId.toByteArray(Charsets.UTF_8),
|
||||
hostFingerprint.toByteArray(Charsets.UTF_8),
|
||||
requestingDeviceFingerprint.toByteArray(Charsets.UTF_8),
|
||||
decision.toByteArray(Charsets.UTF_8),
|
||||
)
|
||||
for (f in fields) {
|
||||
require(f.size <= 0xFFFF) { "signing field too long: ${f.size} bytes (max 65535)" }
|
||||
}
|
||||
val total = fields.sumOf { 2 + it.size }
|
||||
val buf = ByteBuffer.allocate(total).order(ByteOrder.BIG_ENDIAN)
|
||||
for (f in fields) {
|
||||
buf.putShort(f.size.toShort())
|
||||
buf.put(f)
|
||||
}
|
||||
return buf.array()
|
||||
}
|
||||
|
||||
private fun hexToBytes(hex: String): ByteArray {
|
||||
require(hex.length % 2 == 0) { "hex length must be even" }
|
||||
require(hex.all { it.isDigit() || it in 'a'..'f' }) { "hex must be lowercase 0-9a-f" }
|
||||
val out = ByteArray(hex.length / 2)
|
||||
for (i in out.indices) {
|
||||
out[i] = ((Character.digit(hex[i * 2], 16) shl 4) +
|
||||
Character.digit(hex[i * 2 + 1], 16)).toByte()
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,307 @@
|
||||
package no.zyon.shade.approval
|
||||
|
||||
import org.json.JSONArray
|
||||
import org.json.JSONObject
|
||||
|
||||
/**
|
||||
* V4.10 — canonical profile-blob schema. Mirror
|
||||
* `@shade/sdk/approval.ts` byte-for-byte: same field names, same
|
||||
* JSON shape, same denormalization invariants.
|
||||
*
|
||||
* The blob is the AEAD plaintext stored in the V4.9 profile slot. It
|
||||
* holds the user's list of paired hosts + clients; cross-host
|
||||
* approval routing reads `clients[]` to find trusted approvers when
|
||||
* a headless host needs to dispatch a `linkRequest` to a phone.
|
||||
*
|
||||
* Mutators (`upsertHost`, `setTrustedApprover`, ...) are immutable —
|
||||
* they return a new blob and never modify the input. The denormalized
|
||||
* `trustedApproverFingerprints[]` is rederived on every mutation so it
|
||||
* can never drift from the per-client `trustedApprover` flag.
|
||||
*/
|
||||
|
||||
/** A host: a device that receives `linkRequest` frames and runs pairing. */
|
||||
data class ProfileHostEntry(
|
||||
val address: String,
|
||||
val name: String,
|
||||
/** Open enum: `"desktop" | "server" | "laptop" | ...`. */
|
||||
val kind: String,
|
||||
val addedAt: Long,
|
||||
)
|
||||
|
||||
/**
|
||||
* A client: a device that initiates link/approval flows and may
|
||||
* proxy-approve when `trustedApprover == true`. Stores both the
|
||||
* 32-byte Ed25519 identity public key (hex) and the safety-number
|
||||
* fingerprint — the public key is what `verifyProxyApproval` checks
|
||||
* signatures against; the fingerprint is what UIs display.
|
||||
*/
|
||||
data class ProfileClientEntry(
|
||||
val address: String,
|
||||
/** 32-byte Ed25519 long-term identity public key, lowercase hex (64 chars). */
|
||||
val identityPublicKey: String,
|
||||
/** Safety-number fingerprint of the identity key (computeFingerprint output). */
|
||||
val identityFingerprint: String,
|
||||
val name: String,
|
||||
/** Open enum: `"mobile" | "tablet" | "browser" | ...`. */
|
||||
val kind: String,
|
||||
val addedAt: Long,
|
||||
val trustedApprover: Boolean = false,
|
||||
)
|
||||
|
||||
/**
|
||||
* Canonical profile blob. `version=1` is the only currently-supported
|
||||
* shape; bump when an incompatible field is added. Unknown top-level
|
||||
* fields are dropped on parse — additive changes need a coordinated
|
||||
* schema bump on both platforms.
|
||||
*/
|
||||
data class CanonicalProfileBlob(
|
||||
val version: Int = 1,
|
||||
val hosts: List<ProfileHostEntry> = emptyList(),
|
||||
val clients: List<ProfileClientEntry> = emptyList(),
|
||||
/** Denormalized list of trusted-approver fingerprints. Rederived on mutate. */
|
||||
val trustedApproverFingerprints: List<String> = emptyList(),
|
||||
val updatedAt: Long = 0,
|
||||
/** Optional hex-encoded pubkey of the writer; informational only. */
|
||||
val signedBy: String? = null,
|
||||
)
|
||||
|
||||
/** Build a fresh empty profile blob with `updatedAt = now ?? System.currentTimeMillis()`. */
|
||||
fun emptyCanonicalProfile(now: Long? = null): CanonicalProfileBlob =
|
||||
CanonicalProfileBlob(updatedAt = now ?: System.currentTimeMillis())
|
||||
|
||||
/**
|
||||
* Decode a profile-blob plaintext (the AEAD-opened bytes) into the
|
||||
* canonical shape. Throws `IllegalArgumentException` on malformed JSON
|
||||
* or wrong shape.
|
||||
*/
|
||||
fun parseCanonicalProfile(plaintext: ByteArray): CanonicalProfileBlob =
|
||||
parseCanonicalProfile(plaintext.toString(Charsets.UTF_8))
|
||||
|
||||
fun parseCanonicalProfile(plaintext: String): CanonicalProfileBlob {
|
||||
val obj = try {
|
||||
JSONObject(plaintext)
|
||||
} catch (e: Exception) {
|
||||
throw IllegalArgumentException("profile blob is not valid JSON: ${e.message}")
|
||||
}
|
||||
val version = obj.optInt("version", -1)
|
||||
require(version == 1) { "unsupported profile blob version: $version" }
|
||||
|
||||
val hosts = parseArray(obj.optJSONArray("hosts"), "hosts", ::parseHostEntry)
|
||||
val clients = parseArray(obj.optJSONArray("clients"), "clients", ::parseClientEntry)
|
||||
val trustedApproverFingerprints = parseStringArray(
|
||||
obj.optJSONArray("trustedApproverFingerprints"),
|
||||
"trustedApproverFingerprints",
|
||||
)
|
||||
val updatedAt = if (obj.has("updatedAt") && !obj.isNull("updatedAt"))
|
||||
obj.getLong("updatedAt") else 0L
|
||||
val signedBy = obj.optString("signedBy", "").takeIf { it.isNotEmpty() }
|
||||
|
||||
return CanonicalProfileBlob(
|
||||
version = 1,
|
||||
hosts = hosts,
|
||||
clients = clients,
|
||||
trustedApproverFingerprints = trustedApproverFingerprints,
|
||||
updatedAt = updatedAt,
|
||||
signedBy = signedBy,
|
||||
)
|
||||
}
|
||||
|
||||
/** Serialize a profile blob to UTF-8 JSON ready for `Profile.put`. */
|
||||
fun serializeCanonicalProfile(blob: CanonicalProfileBlob): ByteArray {
|
||||
val json = JSONObject()
|
||||
json.put("version", blob.version)
|
||||
json.put("hosts", JSONArray().apply {
|
||||
blob.hosts.forEach { put(hostEntryToJson(it)) }
|
||||
})
|
||||
json.put("clients", JSONArray().apply {
|
||||
blob.clients.forEach { put(clientEntryToJson(it)) }
|
||||
})
|
||||
json.put("trustedApproverFingerprints", JSONArray(blob.trustedApproverFingerprints))
|
||||
json.put("updatedAt", blob.updatedAt)
|
||||
if (blob.signedBy != null) json.put("signedBy", blob.signedBy)
|
||||
return json.toString().toByteArray(Charsets.UTF_8)
|
||||
}
|
||||
|
||||
private fun parseHostEntry(o: JSONObject): ProfileHostEntry =
|
||||
ProfileHostEntry(
|
||||
address = o.requireString("address", "hosts"),
|
||||
name = o.requireString("name", "hosts"),
|
||||
kind = o.requireString("kind", "hosts"),
|
||||
addedAt = o.requireLong("addedAt", "hosts"),
|
||||
)
|
||||
|
||||
private fun parseClientEntry(o: JSONObject): ProfileClientEntry {
|
||||
val identityPublicKey = o.requireString("identityPublicKey", "clients")
|
||||
require(identityPublicKey.matches(Regex("^[0-9a-f]{64}$"))) {
|
||||
"clients[].identityPublicKey must be 64 lowercase hex chars"
|
||||
}
|
||||
return ProfileClientEntry(
|
||||
address = o.requireString("address", "clients"),
|
||||
identityPublicKey = identityPublicKey,
|
||||
identityFingerprint = o.requireString("identityFingerprint", "clients"),
|
||||
name = o.requireString("name", "clients"),
|
||||
kind = o.requireString("kind", "clients"),
|
||||
addedAt = o.requireLong("addedAt", "clients"),
|
||||
trustedApprover = o.optBoolean("trustedApprover", false),
|
||||
)
|
||||
}
|
||||
|
||||
private fun hostEntryToJson(e: ProfileHostEntry): JSONObject = JSONObject().apply {
|
||||
put("address", e.address)
|
||||
put("name", e.name)
|
||||
put("kind", e.kind)
|
||||
put("addedAt", e.addedAt)
|
||||
}
|
||||
|
||||
private fun clientEntryToJson(e: ProfileClientEntry): JSONObject = JSONObject().apply {
|
||||
put("address", e.address)
|
||||
put("identityPublicKey", e.identityPublicKey)
|
||||
put("identityFingerprint", e.identityFingerprint)
|
||||
put("name", e.name)
|
||||
put("kind", e.kind)
|
||||
put("addedAt", e.addedAt)
|
||||
if (e.trustedApprover) put("trustedApprover", true)
|
||||
}
|
||||
|
||||
private fun <T> parseArray(
|
||||
arr: JSONArray?,
|
||||
field: String,
|
||||
parse: (JSONObject) -> T,
|
||||
): List<T> {
|
||||
if (arr == null) return emptyList()
|
||||
return (0 until arr.length()).map { i ->
|
||||
val item = arr.opt(i)
|
||||
require(item is JSONObject) { "$field[$i] must be an object" }
|
||||
parse(item)
|
||||
}
|
||||
}
|
||||
|
||||
private fun parseStringArray(arr: JSONArray?, field: String): List<String> {
|
||||
if (arr == null) return emptyList()
|
||||
return (0 until arr.length()).map { i ->
|
||||
val item = arr.opt(i)
|
||||
require(item is String) { "$field[$i] must be a string" }
|
||||
item
|
||||
}
|
||||
}
|
||||
|
||||
private fun JSONObject.requireString(key: String, ctx: String): String {
|
||||
require(has(key) && !isNull(key)) { "$ctx[].$key is required" }
|
||||
val v = get(key)
|
||||
require(v is String) { "$ctx[].$key must be a string" }
|
||||
return v
|
||||
}
|
||||
|
||||
private fun JSONObject.requireLong(key: String, ctx: String): Long {
|
||||
require(has(key) && !isNull(key)) { "$ctx[].$key is required" }
|
||||
return when (val v = get(key)) {
|
||||
is Number -> v.toLong()
|
||||
else -> throw IllegalArgumentException("$ctx[].$key must be a number")
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Mutators (immutable; return new blob, never mutate input) ──
|
||||
|
||||
/** Insert or replace a host entry by address. Bumps `updatedAt`. */
|
||||
fun upsertHost(
|
||||
blob: CanonicalProfileBlob,
|
||||
host: ProfileHostEntry,
|
||||
now: Long? = null,
|
||||
): CanonicalProfileBlob {
|
||||
val hosts = blob.hosts.filter { it.address != host.address } + host
|
||||
return blob.copy(hosts = hosts, updatedAt = now ?: System.currentTimeMillis())
|
||||
}
|
||||
|
||||
/** Remove the host with the given address, if any. */
|
||||
fun removeHost(
|
||||
blob: CanonicalProfileBlob,
|
||||
address: String,
|
||||
now: Long? = null,
|
||||
): CanonicalProfileBlob {
|
||||
val hosts = blob.hosts.filter { it.address != address }
|
||||
if (hosts.size == blob.hosts.size) return blob
|
||||
return blob.copy(hosts = hosts, updatedAt = now ?: System.currentTimeMillis())
|
||||
}
|
||||
|
||||
/**
|
||||
* Insert or replace a client entry by `identityFingerprint`. Re-derives
|
||||
* `trustedApproverFingerprints` from the resulting `clients[]` so the
|
||||
* denormalized list never drifts.
|
||||
*/
|
||||
fun upsertClient(
|
||||
blob: CanonicalProfileBlob,
|
||||
client: ProfileClientEntry,
|
||||
now: Long? = null,
|
||||
): CanonicalProfileBlob {
|
||||
val clients = blob.clients
|
||||
.filter { it.identityFingerprint != client.identityFingerprint } + client
|
||||
return blob.copy(
|
||||
clients = clients,
|
||||
trustedApproverFingerprints = deriveTrustedApprovers(clients),
|
||||
updatedAt = now ?: System.currentTimeMillis(),
|
||||
)
|
||||
}
|
||||
|
||||
/** Remove the client with the given identityFingerprint, if any. */
|
||||
fun removeClient(
|
||||
blob: CanonicalProfileBlob,
|
||||
identityFingerprint: String,
|
||||
now: Long? = null,
|
||||
): CanonicalProfileBlob {
|
||||
val clients = blob.clients.filter { it.identityFingerprint != identityFingerprint }
|
||||
if (clients.size == blob.clients.size) return blob
|
||||
return blob.copy(
|
||||
clients = clients,
|
||||
trustedApproverFingerprints = deriveTrustedApprovers(clients),
|
||||
updatedAt = now ?: System.currentTimeMillis(),
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Toggle the `trustedApprover` flag on a client by fingerprint.
|
||||
* Returns the input unchanged if fingerprint isn't found OR the
|
||||
* desired state already matches (no spurious updatedAt bump).
|
||||
*/
|
||||
fun setTrustedApprover(
|
||||
blob: CanonicalProfileBlob,
|
||||
identityFingerprint: String,
|
||||
trusted: Boolean,
|
||||
now: Long? = null,
|
||||
): CanonicalProfileBlob {
|
||||
var touched = false
|
||||
val clients = blob.clients.map { c ->
|
||||
if (c.identityFingerprint != identityFingerprint) c
|
||||
else if (c.trustedApprover == trusted) c
|
||||
else {
|
||||
touched = true
|
||||
c.copy(trustedApprover = trusted)
|
||||
}
|
||||
}
|
||||
if (!touched) return blob
|
||||
return blob.copy(
|
||||
clients = clients,
|
||||
trustedApproverFingerprints = deriveTrustedApprovers(clients),
|
||||
updatedAt = now ?: System.currentTimeMillis(),
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* True iff the given fingerprint resolves to a client with both
|
||||
* `trustedApprover == true` AND an entry in `trustedApproverFingerprints[]`.
|
||||
*/
|
||||
fun isTrustedApprover(blob: CanonicalProfileBlob, identityFingerprint: String): Boolean {
|
||||
if (!blob.trustedApproverFingerprints.contains(identityFingerprint)) return false
|
||||
val c = findClientByFingerprint(blob, identityFingerprint) ?: return false
|
||||
return c.trustedApprover
|
||||
}
|
||||
|
||||
fun findClientByFingerprint(
|
||||
blob: CanonicalProfileBlob,
|
||||
identityFingerprint: String,
|
||||
): ProfileClientEntry? = blob.clients.firstOrNull { it.identityFingerprint == identityFingerprint }
|
||||
|
||||
fun findClientByAddress(blob: CanonicalProfileBlob, address: String): ProfileClientEntry? =
|
||||
blob.clients.firstOrNull { it.address == address }
|
||||
|
||||
private fun deriveTrustedApprovers(clients: List<ProfileClientEntry>): List<String> =
|
||||
clients.filter { it.trustedApprover }.map { it.identityFingerprint }
|
||||
@@ -0,0 +1,94 @@
|
||||
package no.zyon.shade.blob
|
||||
|
||||
import javax.crypto.Cipher
|
||||
import javax.crypto.spec.GCMParameterSpec
|
||||
import javax.crypto.spec.SecretKeySpec
|
||||
|
||||
/**
|
||||
* AEAD wrapper for the V4.9 profile blob.
|
||||
*
|
||||
* Wire format for one ciphertext blob:
|
||||
* `nonce(12) || ciphertext(N) || tag(16)`
|
||||
*
|
||||
* Mirror `@shade/storage-encrypted/crypto/aead.ts` byte-for-byte. The
|
||||
* relay stores this as a single opaque BLOB column; AAD is reconstructed
|
||||
* at read-time as `"shade-profile-aad-v1:" + slotIdHex` and is NOT
|
||||
* stored on the relay.
|
||||
*/
|
||||
|
||||
const val BLOB_AEAD_NONCE_LEN = 12
|
||||
const val BLOB_AEAD_TAG_LEN = 16
|
||||
|
||||
private const val MIN_CIPHERTEXT_LEN = BLOB_AEAD_NONCE_LEN + BLOB_AEAD_TAG_LEN
|
||||
|
||||
/**
|
||||
* Seal a plaintext blob. Returns `nonce || ct||tag` ready for direct
|
||||
* blob storage. The caller supplies the nonce so this function is
|
||||
* deterministic — the high-level Profile namespace generates a fresh
|
||||
* 12-byte random nonce per write to keep (key, nonce, plaintext)
|
||||
* unique across re-uploads.
|
||||
*/
|
||||
fun aeadSeal(
|
||||
key: ByteArray,
|
||||
nonce: ByteArray,
|
||||
plaintext: ByteArray,
|
||||
aad: ByteArray,
|
||||
): ByteArray {
|
||||
require(key.size == 32) { "AES-256-GCM key must be 32 bytes" }
|
||||
require(nonce.size == BLOB_AEAD_NONCE_LEN) {
|
||||
"nonce must be $BLOB_AEAD_NONCE_LEN bytes"
|
||||
}
|
||||
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
|
||||
val spec = GCMParameterSpec(128, nonce)
|
||||
cipher.init(Cipher.ENCRYPT_MODE, SecretKeySpec(key, "AES"), spec)
|
||||
cipher.updateAAD(aad)
|
||||
val ctTag = cipher.doFinal(plaintext)
|
||||
val out = ByteArray(BLOB_AEAD_NONCE_LEN + ctTag.size)
|
||||
System.arraycopy(nonce, 0, out, 0, BLOB_AEAD_NONCE_LEN)
|
||||
System.arraycopy(ctTag, 0, out, BLOB_AEAD_NONCE_LEN, ctTag.size)
|
||||
return out
|
||||
}
|
||||
|
||||
/**
|
||||
* Open a `nonce || ct||tag` blob and return the plaintext. Throws on
|
||||
* tamper (AEAD tag mismatch) or short input. The caller may pass an
|
||||
* `expectedNonce` to enforce a deterministic nonce — mismatch throws
|
||||
* before the AEAD even runs (defense-in-depth against a relay returning
|
||||
* the wrong slot's blob).
|
||||
*/
|
||||
fun aeadOpen(
|
||||
key: ByteArray,
|
||||
blob: ByteArray,
|
||||
aad: ByteArray,
|
||||
expectedNonce: ByteArray? = null,
|
||||
): ByteArray {
|
||||
require(key.size == 32) { "AES-256-GCM key must be 32 bytes" }
|
||||
require(blob.size >= MIN_CIPHERTEXT_LEN) { "ciphertext blob too short" }
|
||||
val nonce = blob.copyOfRange(0, BLOB_AEAD_NONCE_LEN)
|
||||
if (expectedNonce != null && !ctEqual(nonce, expectedNonce)) {
|
||||
throw IllegalArgumentException(
|
||||
"nonce mismatch — ciphertext blob has been tampered or row identity changed",
|
||||
)
|
||||
}
|
||||
val ctTag = blob.copyOfRange(BLOB_AEAD_NONCE_LEN, blob.size)
|
||||
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
|
||||
val spec = GCMParameterSpec(128, nonce)
|
||||
cipher.init(Cipher.DECRYPT_MODE, SecretKeySpec(key, "AES"), spec)
|
||||
cipher.updateAAD(aad)
|
||||
return cipher.doFinal(ctTag)
|
||||
}
|
||||
|
||||
private fun ctEqual(a: ByteArray, b: ByteArray): Boolean {
|
||||
if (a.size != b.size) return false
|
||||
var diff = 0
|
||||
for (i in a.indices) {
|
||||
diff = diff or (a[i].toInt() xor b[i].toInt())
|
||||
}
|
||||
return diff == 0
|
||||
}
|
||||
|
||||
/** Build the AAD for a given slotId hex string. */
|
||||
fun blobAadForSlot(slotIdHex: String): ByteArray {
|
||||
require(slotIdHex.length == 64) { "slotIdHex must be 64 hex chars" }
|
||||
return "shade-profile-aad-v1:$slotIdHex".toByteArray(Charsets.UTF_8)
|
||||
}
|
||||
@@ -0,0 +1,250 @@
|
||||
package no.zyon.shade.blob
|
||||
|
||||
import no.zyon.shade.crypto.CryptoProvider
|
||||
import org.json.JSONObject
|
||||
import java.net.URI
|
||||
import java.net.http.HttpClient
|
||||
import java.net.http.HttpRequest
|
||||
import java.net.http.HttpResponse
|
||||
import java.time.Duration
|
||||
import java.util.Base64
|
||||
|
||||
/**
|
||||
* Low-level HTTP client for the V4.9 encrypted-blob primitive
|
||||
* (`/v1/blob/<slotId>`). Mirror `@shade/inbox`'s `BlobClient` —
|
||||
* stateless, reusable, and protocol-compatible with the TypeScript
|
||||
* relay endpoints.
|
||||
*
|
||||
* The client doesn't care what the blob bytes mean — it just
|
||||
* transports them. Higher-level wrappers (e.g. `Profile`) compose
|
||||
* this client with AEAD-sealing of the actual payload.
|
||||
*
|
||||
* Auth model: every PUT/DELETE carries a detached Ed25519 signature
|
||||
* (base64) over a canonical-JSON form of the request body. The
|
||||
* canonicalization is deterministic — sorted keys, compact JSON, no
|
||||
* trailing whitespace — so signatures generated on Kotlin verify on
|
||||
* the TS server.
|
||||
*/
|
||||
class BlobClient(
|
||||
private val baseUrl: String,
|
||||
private val crypto: CryptoProvider,
|
||||
private val httpClient: HttpClient = HttpClient.newBuilder()
|
||||
.connectTimeout(Duration.ofSeconds(15))
|
||||
.build(),
|
||||
) {
|
||||
|
||||
data class GetResult(
|
||||
val blob: ByteArray,
|
||||
val etag: String,
|
||||
val updatedAt: Long,
|
||||
)
|
||||
|
||||
data class PutResult(
|
||||
val created: Boolean,
|
||||
val etag: String,
|
||||
val updatedAt: Long,
|
||||
)
|
||||
|
||||
/**
|
||||
* Read a slot. Returns null if no blob has ever been written there
|
||||
* (or if it was DELETE'd). GET is unauthenticated by design — the
|
||||
* slotId is itself a 256-bit secret derived from the master key.
|
||||
*/
|
||||
fun get(slotIdHex: String): GetResult? {
|
||||
validateSlotIdHex(slotIdHex)
|
||||
val req = HttpRequest.newBuilder(URI.create(joinUrl(baseUrl, "/v1/blob/$slotIdHex")))
|
||||
.GET()
|
||||
.build()
|
||||
val res = httpClient.send(req, HttpResponse.BodyHandlers.ofString())
|
||||
if (res.statusCode() == 404) return null
|
||||
val json = parseJson(res, "GET")
|
||||
val blob = Base64.getDecoder().decode(json.getString("blob"))
|
||||
return GetResult(
|
||||
blob = blob,
|
||||
etag = json.getString("etag"),
|
||||
updatedAt = json.getLong("updatedAt"),
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Create or update a slot.
|
||||
*
|
||||
* `ifMatch` semantics:
|
||||
* - `null`: create-only. Slot must be empty (else 409).
|
||||
* - `<etag-string>`: compare-and-swap. Must match (else 412).
|
||||
* - `"*"`: unconditional overwrite. Slot must already exist (else 412).
|
||||
*/
|
||||
fun put(
|
||||
slotIdHex: String,
|
||||
blob: ByteArray,
|
||||
signingSeed: ByteArray,
|
||||
ownerPubkey: ByteArray,
|
||||
ifMatch: String? = null,
|
||||
): PutResult {
|
||||
validateSlotIdHex(slotIdHex)
|
||||
require(blob.isNotEmpty()) { "Empty blob" }
|
||||
require(ownerPubkey.size == 32) { "ownerPubkey must be 32 bytes (Ed25519)" }
|
||||
require(signingSeed.size == 32) { "signingSeed must be 32 bytes (Ed25519)" }
|
||||
|
||||
// Canonical form for signing: sorted keys, slotId included,
|
||||
// signature field absent. The wire body strips slotId (it's
|
||||
// in the URL) but the signature is computed over the
|
||||
// slotId-bearing form.
|
||||
val signedAt = System.currentTimeMillis()
|
||||
val canonical = sortedMapOf<String, Any>().apply {
|
||||
put("blob", Base64.getEncoder().encodeToString(blob))
|
||||
if (ifMatch != null) put("ifMatch", ifMatch)
|
||||
put("ownerPubkey", Base64.getEncoder().encodeToString(ownerPubkey))
|
||||
put("signedAt", signedAt)
|
||||
put("slotId", slotIdHex)
|
||||
}
|
||||
val canonicalBytes = canonicalJson(canonical).toByteArray(Charsets.UTF_8)
|
||||
val sig = crypto.sign(signingSeed, canonicalBytes)
|
||||
|
||||
// Wire body: same as canonical minus slotId, plus signature.
|
||||
val wire = JSONObject()
|
||||
wire.put("blob", Base64.getEncoder().encodeToString(blob))
|
||||
if (ifMatch != null) wire.put("ifMatch", ifMatch)
|
||||
wire.put("ownerPubkey", Base64.getEncoder().encodeToString(ownerPubkey))
|
||||
wire.put("signedAt", signedAt)
|
||||
wire.put("signature", Base64.getEncoder().encodeToString(sig))
|
||||
|
||||
val req = HttpRequest.newBuilder(URI.create(joinUrl(baseUrl, "/v1/blob/$slotIdHex")))
|
||||
.header("content-type", "application/json")
|
||||
.PUT(HttpRequest.BodyPublishers.ofString(wire.toString()))
|
||||
.build()
|
||||
val res = httpClient.send(req, HttpResponse.BodyHandlers.ofString())
|
||||
val json = parseJson(res, "PUT")
|
||||
return PutResult(
|
||||
created = json.optBoolean("created"),
|
||||
etag = json.getString("etag"),
|
||||
updatedAt = json.getLong("updatedAt"),
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete a slot. The next PUT TOFU-claims it again, possibly under
|
||||
* a fresh signing key (e.g. after rotation). Used by "forget
|
||||
* everything" flows.
|
||||
*/
|
||||
fun delete(slotIdHex: String, signingSeed: ByteArray): Boolean {
|
||||
validateSlotIdHex(slotIdHex)
|
||||
require(signingSeed.size == 32) { "signingSeed must be 32 bytes (Ed25519)" }
|
||||
|
||||
val signedAt = System.currentTimeMillis()
|
||||
val canonical = sortedMapOf<String, Any>().apply {
|
||||
put("signedAt", signedAt)
|
||||
put("slotId", slotIdHex)
|
||||
}
|
||||
val canonicalBytes = canonicalJson(canonical).toByteArray(Charsets.UTF_8)
|
||||
val sig = crypto.sign(signingSeed, canonicalBytes)
|
||||
|
||||
val wire = JSONObject()
|
||||
wire.put("signedAt", signedAt)
|
||||
wire.put("signature", Base64.getEncoder().encodeToString(sig))
|
||||
|
||||
val req = HttpRequest.newBuilder(URI.create(joinUrl(baseUrl, "/v1/blob/$slotIdHex")))
|
||||
.header("content-type", "application/json")
|
||||
.method("DELETE", HttpRequest.BodyPublishers.ofString(wire.toString()))
|
||||
.build()
|
||||
val res = httpClient.send(req, HttpResponse.BodyHandlers.ofString())
|
||||
val json = parseJson(res, "DELETE")
|
||||
return json.optBoolean("ok", false)
|
||||
}
|
||||
|
||||
private fun parseJson(res: HttpResponse<String>, op: String): JSONObject {
|
||||
val text = res.body() ?: ""
|
||||
val json = if (text.isEmpty()) JSONObject() else try {
|
||||
JSONObject(text)
|
||||
} catch (e: Exception) {
|
||||
throw BlobClientException(
|
||||
code = "SHADE_NETWORK",
|
||||
statusCode = res.statusCode(),
|
||||
message = "Blob $op response not JSON: ${text.take(200)}",
|
||||
)
|
||||
}
|
||||
if (res.statusCode() !in 200..299) {
|
||||
throw BlobClientException(
|
||||
code = json.optString("code", "SHADE_NETWORK"),
|
||||
statusCode = res.statusCode(),
|
||||
message = json.optString("message", text),
|
||||
)
|
||||
}
|
||||
return json
|
||||
}
|
||||
|
||||
private fun validateSlotIdHex(s: String) {
|
||||
require(s.matches(Regex("^[0-9a-f]{64}$"))) {
|
||||
"slotIdHex must be 64 lowercase hex chars (32 bytes)"
|
||||
}
|
||||
}
|
||||
|
||||
private fun joinUrl(base: String, path: String): String =
|
||||
when {
|
||||
base.endsWith("/") && path.startsWith("/") -> base + path.substring(1)
|
||||
!base.endsWith("/") && !path.startsWith("/") -> "$base/$path"
|
||||
else -> base + path
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Mirror of TS `signPayload`'s canonicalization: sorted keys, compact
|
||||
* JSON, signature field absent. Only handles the subset of types we
|
||||
* need (strings + longs + base64 strings) — keeping the implementation
|
||||
* narrow so it can't accidentally diverge from `JSON.stringify` on
|
||||
* structurally-different inputs.
|
||||
*/
|
||||
internal fun canonicalJson(map: Map<String, Any>): String {
|
||||
val sb = StringBuilder()
|
||||
sb.append('{')
|
||||
var first = true
|
||||
for ((key, value) in map.toSortedMap()) {
|
||||
if (!first) sb.append(',')
|
||||
first = false
|
||||
appendJsonString(sb, key)
|
||||
sb.append(':')
|
||||
appendJsonValue(sb, value)
|
||||
}
|
||||
sb.append('}')
|
||||
return sb.toString()
|
||||
}
|
||||
|
||||
private fun appendJsonValue(sb: StringBuilder, value: Any) {
|
||||
when (value) {
|
||||
is String -> appendJsonString(sb, value)
|
||||
is Long, is Int -> sb.append(value.toString())
|
||||
is Boolean -> sb.append(value.toString())
|
||||
else -> throw IllegalArgumentException(
|
||||
"canonicalJson: unsupported value type ${value::class.java}",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
private fun appendJsonString(sb: StringBuilder, s: String) {
|
||||
sb.append('"')
|
||||
for (c in s) {
|
||||
when (c) {
|
||||
'\\' -> sb.append("\\\\")
|
||||
'"' -> sb.append("\\\"")
|
||||
'\b' -> sb.append("\\b")
|
||||
'\u000C' -> sb.append("\\f")
|
||||
'\n' -> sb.append("\\n")
|
||||
'\r' -> sb.append("\\r")
|
||||
'\t' -> sb.append("\\t")
|
||||
else -> {
|
||||
if (c.code < 0x20) {
|
||||
sb.append("\\u").append("%04x".format(c.code))
|
||||
} else {
|
||||
sb.append(c)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
sb.append('"')
|
||||
}
|
||||
|
||||
class BlobClientException(
|
||||
val code: String,
|
||||
val statusCode: Int,
|
||||
message: String,
|
||||
) : RuntimeException("[$code @ $statusCode] $message")
|
||||
@@ -0,0 +1,88 @@
|
||||
package no.zyon.shade.blob
|
||||
|
||||
import com.google.crypto.tink.subtle.Ed25519Sign
|
||||
import no.zyon.shade.crypto.CryptoProvider
|
||||
|
||||
/**
|
||||
* V4.9 — relay-side encrypted blob primitive: deterministic
|
||||
* derivations from a 32-byte master key + per-app namespace string.
|
||||
*
|
||||
* Mirror `@shade/storage-encrypted/crypto/kdf.ts` byte-for-byte.
|
||||
* Reference vectors: `test-vectors/blob-storage.json`.
|
||||
*
|
||||
* Three independent 32-byte derivations:
|
||||
*
|
||||
* slotId = HKDF(masterKey, info=`shade-blob-slot-v1:<app>`) // relay-visible opaque ID
|
||||
* blobKey = HKDF(masterKey, info=`shade-blob-key-v1:<app>`) // AEAD key for the blob
|
||||
* sigSeed = HKDF(masterKey, info=`shade-blob-sig-v1:<app>`) // Ed25519 owner signing seed
|
||||
*
|
||||
* `app` is a caller-supplied namespace string — distinct apps under
|
||||
* the same master MUST pass different values (e.g. `prism-profile-v1`)
|
||||
* so they don't collide on the same slot.
|
||||
*
|
||||
* The signing seed is an Ed25519 *seed* in the @noble/curves convention:
|
||||
* `pubkey = Ed25519.publicFromSeed(seed)` is what the relay TOFU-stores
|
||||
* on the first PUT and verifies subsequent writes against.
|
||||
*/
|
||||
|
||||
private const val SLOT_INFO_PREFIX = "shade-blob-slot-v1:"
|
||||
private const val BLOB_KEY_INFO_PREFIX = "shade-blob-key-v1:"
|
||||
private const val SIG_SEED_INFO_PREFIX = "shade-blob-sig-v1:"
|
||||
|
||||
private const val DERIVED_LEN = 32
|
||||
|
||||
/** Lower-hex 64-char slotId derived deterministically from the master key. */
|
||||
fun deriveBlobSlotId(crypto: CryptoProvider, masterKey: ByteArray, app: String): ByteArray {
|
||||
require(masterKey.size == 32) { "masterKey must be 32 bytes" }
|
||||
require(app.isNotEmpty()) { "app namespace must be non-empty" }
|
||||
return crypto.hkdf(
|
||||
masterKey,
|
||||
ByteArray(0),
|
||||
(SLOT_INFO_PREFIX + app).toByteArray(Charsets.UTF_8),
|
||||
DERIVED_LEN,
|
||||
)
|
||||
}
|
||||
|
||||
/** AEAD key for sealing/opening the blob. The slotId hex is bound as AAD. */
|
||||
fun deriveBlobKey(crypto: CryptoProvider, masterKey: ByteArray, app: String): ByteArray {
|
||||
require(masterKey.size == 32) { "masterKey must be 32 bytes" }
|
||||
require(app.isNotEmpty()) { "app namespace must be non-empty" }
|
||||
return crypto.hkdf(
|
||||
masterKey,
|
||||
ByteArray(0),
|
||||
(BLOB_KEY_INFO_PREFIX + app).toByteArray(Charsets.UTF_8),
|
||||
DERIVED_LEN,
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* 32-byte Ed25519 signing seed. The pubkey, derived deterministically
|
||||
* from the seed, is what the relay TOFU-stores on the first PUT.
|
||||
*/
|
||||
fun deriveBlobSigningSeed(crypto: CryptoProvider, masterKey: ByteArray, app: String): ByteArray {
|
||||
require(masterKey.size == 32) { "masterKey must be 32 bytes" }
|
||||
require(app.isNotEmpty()) { "app namespace must be non-empty" }
|
||||
return crypto.hkdf(
|
||||
masterKey,
|
||||
ByteArray(0),
|
||||
(SIG_SEED_INFO_PREFIX + app).toByteArray(Charsets.UTF_8),
|
||||
DERIVED_LEN,
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Recover the Ed25519 public key from a 32-byte seed. Mirrors
|
||||
* `@shade/crypto-web`'s `ed25519PublicKeyFromSeed`. Tink's
|
||||
* `Ed25519Sign.KeyPair.newKeyPairFromSeed(seed)` exposes both halves;
|
||||
* we discard the private half here.
|
||||
*/
|
||||
fun ed25519PublicKeyFromSeed(seed: ByteArray): ByteArray {
|
||||
require(seed.size == 32) { "Ed25519 seed must be 32 bytes" }
|
||||
return Ed25519Sign.KeyPair.newKeyPairFromSeed(seed).publicKey
|
||||
}
|
||||
|
||||
/** Convert a 32-byte slotId into the lowercase-hex wire form (64 chars). */
|
||||
fun slotIdToHex(slotId: ByteArray): String {
|
||||
require(slotId.size == 32) { "slotId must be 32 bytes" }
|
||||
return slotId.joinToString("") { "%02x".format(it) }
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
package no.zyon.shade.blob
|
||||
|
||||
import no.zyon.shade.crypto.CryptoProvider
|
||||
|
||||
/**
|
||||
* V4.9 — high-level profile namespace. Mirror
|
||||
* `@shade/sdk`'s `createProfileNamespace`. The relay never sees
|
||||
* plaintext; AAD binds the slotId so a relay returning the wrong
|
||||
* slot's blob fails to open.
|
||||
*
|
||||
* Usage:
|
||||
*
|
||||
* val crypto = TinkProvider()
|
||||
* val masterKey = deriveMasterKey("password", salt) // V4.5 KDF chain
|
||||
* val profile = createProfileNamespace(
|
||||
* baseUrl = "https://shade.example/",
|
||||
* crypto = crypto,
|
||||
* masterKey = masterKey,
|
||||
* app = "prism-profile-v1",
|
||||
* )
|
||||
*
|
||||
* val current = profile.get() // null if no blob yet
|
||||
* profile.put(serializeCanonicalProfile(...), ifMatch = current?.etag)
|
||||
* profile.delete()
|
||||
*
|
||||
* Apps with the same master key + app namespace converge on the same
|
||||
* slot — that's the whole point: a brand new device with the right
|
||||
* credentials can locate, decrypt, and update the blob.
|
||||
*/
|
||||
class ProfileNamespace internal constructor(
|
||||
/** Lower-hex 64-char slotId. Stable per (master, app). */
|
||||
val slotIdHex: String,
|
||||
private val blobKey: ByteArray,
|
||||
private val signingSeed: ByteArray,
|
||||
private val ownerPubkey: ByteArray,
|
||||
private val aad: ByteArray,
|
||||
private val client: BlobClient,
|
||||
private val crypto: CryptoProvider,
|
||||
) {
|
||||
|
||||
data class GetResult(
|
||||
val plaintext: ByteArray,
|
||||
val etag: String,
|
||||
val updatedAt: Long,
|
||||
)
|
||||
|
||||
data class PutResult(
|
||||
val created: Boolean,
|
||||
val etag: String,
|
||||
val updatedAt: Long,
|
||||
)
|
||||
|
||||
/** Returns null when the slot has never been written (or was deleted). */
|
||||
fun get(): GetResult? {
|
||||
val raw = client.get(slotIdHex) ?: return null
|
||||
val plaintext = aeadOpen(blobKey, raw.blob, aad)
|
||||
return GetResult(plaintext = plaintext, etag = raw.etag, updatedAt = raw.updatedAt)
|
||||
}
|
||||
|
||||
/**
|
||||
* Create or update. `ifMatch`:
|
||||
* - null: create-only (fails with 409 if slot populated).
|
||||
* - "<etag>": CAS (fails with 412 on stale).
|
||||
* - "*": unconditional overwrite.
|
||||
*/
|
||||
fun put(plaintext: ByteArray, ifMatch: String? = null): PutResult {
|
||||
// Fresh random nonce per write — see `BlobAead`. Re-uploading
|
||||
// the same plaintext after a transient error reuses neither
|
||||
// (key, nonce, plaintext) nor (key, nonce).
|
||||
val nonce = crypto.randomBytes(BLOB_AEAD_NONCE_LEN)
|
||||
val sealed = aeadSeal(blobKey, nonce, plaintext, aad)
|
||||
val r = client.put(
|
||||
slotIdHex = slotIdHex,
|
||||
blob = sealed,
|
||||
signingSeed = signingSeed,
|
||||
ownerPubkey = ownerPubkey,
|
||||
ifMatch = ifMatch,
|
||||
)
|
||||
return PutResult(created = r.created, etag = r.etag, updatedAt = r.updatedAt)
|
||||
}
|
||||
|
||||
/** "Forget everything" path — the next PUT TOFU-claims it again. */
|
||||
fun delete(): Boolean = client.delete(slotIdHex, signingSeed)
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a Profile namespace bound to a (master key, app) pair. The
|
||||
* derivations are deterministic: any device with the same master
|
||||
* key + app namespace produces the same slot, so a fresh device
|
||||
* after credential entry can locate the existing profile blob.
|
||||
*/
|
||||
fun createProfileNamespace(
|
||||
baseUrl: String,
|
||||
crypto: CryptoProvider,
|
||||
masterKey: ByteArray,
|
||||
app: String,
|
||||
): ProfileNamespace {
|
||||
require(masterKey.size == 32) { "masterKey must be 32 bytes" }
|
||||
require(app.isNotEmpty()) { "app namespace must be non-empty" }
|
||||
|
||||
val slotIdBytes = deriveBlobSlotId(crypto, masterKey, app)
|
||||
val slotIdHex = slotIdToHex(slotIdBytes)
|
||||
val blobKey = deriveBlobKey(crypto, masterKey, app)
|
||||
val signingSeed = deriveBlobSigningSeed(crypto, masterKey, app)
|
||||
val ownerPubkey = ed25519PublicKeyFromSeed(signingSeed)
|
||||
val aad = blobAadForSlot(slotIdHex)
|
||||
|
||||
val client = BlobClient(baseUrl = baseUrl, crypto = crypto)
|
||||
return ProfileNamespace(
|
||||
slotIdHex = slotIdHex,
|
||||
blobKey = blobKey,
|
||||
signingSeed = signingSeed,
|
||||
ownerPubkey = ownerPubkey,
|
||||
aad = aad,
|
||||
client = client,
|
||||
crypto = crypto,
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
package no.zyon.shade.serialization
|
||||
|
||||
import no.zyon.shade.types.ChainState
|
||||
import no.zyon.shade.types.IdentityKeyPair
|
||||
import no.zyon.shade.types.KeyPair
|
||||
import no.zyon.shade.types.OneTimePreKey
|
||||
import no.zyon.shade.types.SessionState
|
||||
import no.zyon.shade.types.SignedPreKey
|
||||
import org.json.JSONObject
|
||||
import java.util.Base64 as JdkBase64
|
||||
|
||||
/**
|
||||
* Plain-JSON serialization for the persisted protocol state types
|
||||
* (`IdentityKeyPair`, `SignedPreKey`, `OneTimePreKey`, `SessionState`).
|
||||
*
|
||||
* The on-disk shape is for at-rest storage only — it does NOT need
|
||||
* to round-trip across platforms (TS uses its own JSON shape via
|
||||
* `@shade/core/serialization`). What matters is that the Kotlin
|
||||
* round-trip (`serialize` then `deserialize`) preserves every byte.
|
||||
*
|
||||
* Both Android-targeted (`shade-android-keystore`) and pure-JVM
|
||||
* (`shade-android` tests) callers use this — the function works
|
||||
* without any `android.*` imports so it compiles in both.
|
||||
*/
|
||||
object SessionStateJson {
|
||||
|
||||
fun serialize(state: SessionState): String {
|
||||
val o = JSONObject()
|
||||
o.put("remoteIdentityKey", b64(state.remoteIdentityKey))
|
||||
o.put("rootKey", b64(state.rootKey))
|
||||
o.put("sendChain", chainToJson(state.sendChain))
|
||||
if (state.receiveChain != null) o.put("receiveChain", chainToJson(state.receiveChain!!))
|
||||
o.put("dhSend", keyPairToJson(state.dhSend))
|
||||
if (state.dhReceive != null) o.put("dhReceive", b64(state.dhReceive!!))
|
||||
o.put("previousSendCounter", state.previousSendCounter)
|
||||
val skipped = JSONObject()
|
||||
for ((k, v) in state.skippedKeys) skipped.put(k, b64(v))
|
||||
o.put("skippedKeys", skipped)
|
||||
return o.toString()
|
||||
}
|
||||
|
||||
fun deserialize(s: String): SessionState {
|
||||
val o = JSONObject(s)
|
||||
val skipped = mutableMapOf<String, ByteArray>()
|
||||
val skJson = o.optJSONObject("skippedKeys")
|
||||
if (skJson != null) {
|
||||
val it = skJson.keys()
|
||||
while (it.hasNext()) {
|
||||
val k = it.next()
|
||||
skipped[k] = fb64(skJson.getString(k))
|
||||
}
|
||||
}
|
||||
return SessionState(
|
||||
remoteIdentityKey = fb64(o.getString("remoteIdentityKey")),
|
||||
rootKey = fb64(o.getString("rootKey")),
|
||||
sendChain = chainFromJson(o.getJSONObject("sendChain")),
|
||||
receiveChain = if (o.has("receiveChain"))
|
||||
chainFromJson(o.getJSONObject("receiveChain")) else null,
|
||||
dhSend = keyPairFromJson(o.getJSONObject("dhSend")),
|
||||
dhReceive = if (o.has("dhReceive")) fb64(o.getString("dhReceive")) else null,
|
||||
previousSendCounter = o.getInt("previousSendCounter"),
|
||||
skippedKeys = skipped,
|
||||
)
|
||||
}
|
||||
|
||||
fun serializeIdentityKeyPair(k: IdentityKeyPair): String = JSONObject().apply {
|
||||
put("signingPublicKey", b64(k.signingPublicKey))
|
||||
put("signingPrivateKey", b64(k.signingPrivateKey))
|
||||
put("dhPublicKey", b64(k.dhPublicKey))
|
||||
put("dhPrivateKey", b64(k.dhPrivateKey))
|
||||
}.toString()
|
||||
|
||||
fun deserializeIdentityKeyPair(s: String): IdentityKeyPair = JSONObject(s).run {
|
||||
IdentityKeyPair(
|
||||
signingPublicKey = fb64(getString("signingPublicKey")),
|
||||
signingPrivateKey = fb64(getString("signingPrivateKey")),
|
||||
dhPublicKey = fb64(getString("dhPublicKey")),
|
||||
dhPrivateKey = fb64(getString("dhPrivateKey")),
|
||||
)
|
||||
}
|
||||
|
||||
fun serializeSignedPreKey(k: SignedPreKey): String = JSONObject().apply {
|
||||
put("keyId", k.keyId)
|
||||
put("keyPair", keyPairToJson(k.keyPair))
|
||||
put("signature", b64(k.signature))
|
||||
put("timestamp", k.timestamp)
|
||||
}.toString()
|
||||
|
||||
fun deserializeSignedPreKey(s: String): SignedPreKey = JSONObject(s).run {
|
||||
SignedPreKey(
|
||||
keyId = getInt("keyId"),
|
||||
keyPair = keyPairFromJson(getJSONObject("keyPair")),
|
||||
signature = fb64(getString("signature")),
|
||||
timestamp = getLong("timestamp"),
|
||||
)
|
||||
}
|
||||
|
||||
fun serializeOneTimePreKey(k: OneTimePreKey): String = JSONObject().apply {
|
||||
put("keyId", k.keyId)
|
||||
put("keyPair", keyPairToJson(k.keyPair))
|
||||
}.toString()
|
||||
|
||||
fun deserializeOneTimePreKey(s: String): OneTimePreKey = JSONObject(s).run {
|
||||
OneTimePreKey(
|
||||
keyId = getInt("keyId"),
|
||||
keyPair = keyPairFromJson(getJSONObject("keyPair")),
|
||||
)
|
||||
}
|
||||
|
||||
private fun chainToJson(c: ChainState): JSONObject = JSONObject().apply {
|
||||
put("chainKey", b64(c.chainKey))
|
||||
put("counter", c.counter)
|
||||
}
|
||||
|
||||
private fun chainFromJson(o: JSONObject): ChainState =
|
||||
ChainState(chainKey = fb64(o.getString("chainKey")), counter = o.getInt("counter"))
|
||||
|
||||
private fun keyPairToJson(k: KeyPair): JSONObject = JSONObject().apply {
|
||||
put("publicKey", b64(k.publicKey))
|
||||
put("privateKey", b64(k.privateKey))
|
||||
}
|
||||
|
||||
private fun keyPairFromJson(o: JSONObject): KeyPair = KeyPair(
|
||||
publicKey = fb64(o.getString("publicKey")),
|
||||
privateKey = fb64(o.getString("privateKey")),
|
||||
)
|
||||
|
||||
// android.util.Base64 isn't on the JVM classpath; java.util.Base64
|
||||
// is available on both modern JVM and Android API 26+. Use JDK
|
||||
// Base64 throughout — it's present on both targets.
|
||||
private fun b64(b: ByteArray): String = JdkBase64.getEncoder().encodeToString(b)
|
||||
private fun fb64(s: String): ByteArray = JdkBase64.getDecoder().decode(s)
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
package no.zyon.shade.storage
|
||||
|
||||
import org.bouncycastle.crypto.generators.Argon2BytesGenerator
|
||||
import org.bouncycastle.crypto.generators.SCrypt
|
||||
import org.bouncycastle.crypto.params.Argon2Parameters
|
||||
import java.text.Normalizer
|
||||
|
||||
/**
|
||||
* Password / PIN key-derivation primitives. Mirror
|
||||
* `@shade/storage-encrypted/crypto/kdf` (`deriveMasterKey` /
|
||||
* `deriveMasterKeyArgon2id`) byte-for-byte — Tink doesn't ship password
|
||||
* KDFs so we wrap Bouncy Castle.
|
||||
*
|
||||
* Both functions normalize string passphrases to NFKC before hashing,
|
||||
* matching the TS implementation's `passphrase.normalize('NFKC')`.
|
||||
* This ensures the same password typed on different OSes/keyboards
|
||||
* produces the same master key regardless of which compatibility-form
|
||||
* the input arrived in.
|
||||
*
|
||||
* The reference test-vector lives in `test-vectors/storage-encryption.json`
|
||||
* and `test-vectors/blob-storage.json`. Cross-platform parity is gated
|
||||
* by `CrossPlatformVectorTest`.
|
||||
*/
|
||||
|
||||
/** scrypt parameters. Defaults match `DEFAULT_SCRYPT` in TS. */
|
||||
data class ScryptParams(
|
||||
/** CPU/memory cost. Must be a power of 2. */
|
||||
val n: Int = 1 shl 17,
|
||||
/** Block size. */
|
||||
val r: Int = 8,
|
||||
/** Parallelization. */
|
||||
val p: Int = 1,
|
||||
/** Output length in bytes. */
|
||||
val dkLen: Int = 32,
|
||||
)
|
||||
|
||||
/** Argon2id parameters. Defaults match `DEFAULT_ARGON2ID` in TS. */
|
||||
data class Argon2idParams(
|
||||
/** Memory cost in KiB. Default 64 MiB. */
|
||||
val m: Int = 64 * 1024,
|
||||
/** Time cost (iterations). Default 3. */
|
||||
val t: Int = 3,
|
||||
/** Parallelism. Default 1. */
|
||||
val p: Int = 1,
|
||||
/** Output length in bytes. Default 32. */
|
||||
val dkLen: Int = 32,
|
||||
)
|
||||
|
||||
/**
|
||||
* Derive a 32-byte master key from a passphrase + salt via scrypt.
|
||||
* Salt MUST be at least 16 bytes and persisted alongside the
|
||||
* encrypted database. Throws on empty passphrase.
|
||||
*/
|
||||
fun deriveMasterKey(
|
||||
passphrase: String,
|
||||
salt: ByteArray,
|
||||
params: ScryptParams = ScryptParams(),
|
||||
): ByteArray {
|
||||
require(passphrase.isNotEmpty()) { "passphrase must be non-empty" }
|
||||
require(salt.size >= 16) { "salt must be at least 16 bytes" }
|
||||
val nfkc = Normalizer.normalize(passphrase, Normalizer.Form.NFKC)
|
||||
val pwBytes = nfkc.toByteArray(Charsets.UTF_8)
|
||||
return SCrypt.generate(pwBytes, salt, params.n, params.r, params.p, params.dkLen)
|
||||
}
|
||||
|
||||
/**
|
||||
* Derive a 32-byte master key from a low-entropy secret (PIN) + salt
|
||||
* via argon2id. Salt MUST be at least 16 bytes. The Bouncy Castle
|
||||
* `Argon2BytesGenerator` parameters mirror RFC 9106's argon2id mode
|
||||
* with version 1.3 (`Argon2Parameters.ARGON2_VERSION_13`), which is
|
||||
* what `@noble/hashes/argon2` produces — keeping cross-platform parity.
|
||||
*/
|
||||
fun deriveMasterKeyArgon2id(
|
||||
secret: String,
|
||||
salt: ByteArray,
|
||||
params: Argon2idParams = Argon2idParams(),
|
||||
): ByteArray {
|
||||
require(secret.isNotEmpty()) { "argon2id secret must be non-empty" }
|
||||
require(salt.size >= 16) { "salt must be at least 16 bytes" }
|
||||
val nfkc = Normalizer.normalize(secret, Normalizer.Form.NFKC)
|
||||
return deriveMasterKeyArgon2id(nfkc.toByteArray(Charsets.UTF_8), salt, params)
|
||||
}
|
||||
|
||||
/**
|
||||
* Byte-array overload — useful when the secret is already binary
|
||||
* (e.g. derived from a hardware token rather than typed) and
|
||||
* shouldn't be NFKC-normalized as text.
|
||||
*/
|
||||
fun deriveMasterKeyArgon2id(
|
||||
secret: ByteArray,
|
||||
salt: ByteArray,
|
||||
params: Argon2idParams = Argon2idParams(),
|
||||
): ByteArray {
|
||||
require(secret.isNotEmpty()) { "argon2id secret must be non-empty" }
|
||||
require(salt.size >= 16) { "salt must be at least 16 bytes" }
|
||||
val builder = Argon2Parameters.Builder(Argon2Parameters.ARGON2_id)
|
||||
.withVersion(Argon2Parameters.ARGON2_VERSION_13)
|
||||
.withIterations(params.t)
|
||||
.withMemoryAsKB(params.m)
|
||||
.withParallelism(params.p)
|
||||
.withSalt(salt)
|
||||
val gen = Argon2BytesGenerator()
|
||||
gen.init(builder.build())
|
||||
val out = ByteArray(params.dkLen)
|
||||
gen.generateBytes(secret, out)
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,593 @@
|
||||
package no.zyon.shade
|
||||
|
||||
import no.zyon.shade.approval.ApprovalRequestFrame
|
||||
import no.zyon.shade.approval.ApprovalRequestingDevice
|
||||
import no.zyon.shade.approval.CanonicalProfileBlob
|
||||
import no.zyon.shade.approval.DEFAULT_APPROVAL_DOMAIN
|
||||
import no.zyon.shade.approval.ProfileClientEntry
|
||||
import no.zyon.shade.approval.ProfileHostEntry
|
||||
import no.zyon.shade.approval.ProxyApprovalFrame
|
||||
import no.zyon.shade.approval.VerifyProxyApprovalResult
|
||||
import no.zyon.shade.approval.buildApprovalRequest
|
||||
import no.zyon.shade.approval.canonicalApprovalSigningBytes
|
||||
import no.zyon.shade.approval.emptyCanonicalProfile
|
||||
import no.zyon.shade.approval.findClientByAddress
|
||||
import no.zyon.shade.approval.findClientByFingerprint
|
||||
import no.zyon.shade.approval.isTrustedApprover
|
||||
import no.zyon.shade.approval.parseCanonicalProfile
|
||||
import no.zyon.shade.approval.removeClient
|
||||
import no.zyon.shade.approval.serializeCanonicalProfile
|
||||
import no.zyon.shade.approval.setTrustedApprover
|
||||
import no.zyon.shade.approval.signProxyApproval
|
||||
import no.zyon.shade.approval.upsertClient
|
||||
import no.zyon.shade.approval.upsertHost
|
||||
import no.zyon.shade.approval.verifyProxyApproval
|
||||
import no.zyon.shade.blob.aeadOpen
|
||||
import no.zyon.shade.blob.aeadSeal
|
||||
import no.zyon.shade.blob.blobAadForSlot
|
||||
import no.zyon.shade.blob.deriveBlobKey
|
||||
import no.zyon.shade.blob.deriveBlobSigningSeed
|
||||
import no.zyon.shade.blob.deriveBlobSlotId
|
||||
import no.zyon.shade.blob.ed25519PublicKeyFromSeed
|
||||
import no.zyon.shade.blob.slotIdToHex
|
||||
import no.zyon.shade.crypto.TinkProvider
|
||||
import no.zyon.shade.storage.Argon2idParams
|
||||
import no.zyon.shade.storage.ScryptParams
|
||||
import no.zyon.shade.storage.deriveMasterKey
|
||||
import no.zyon.shade.storage.deriveMasterKeyArgon2id
|
||||
import org.junit.Assert.assertArrayEquals
|
||||
import org.junit.Assert.assertEquals
|
||||
import org.junit.Assert.assertFalse
|
||||
import org.junit.Assert.assertNotNull
|
||||
import org.junit.Assert.assertNull
|
||||
import org.junit.Assert.assertNotEquals
|
||||
import org.junit.Assert.assertTrue
|
||||
import org.junit.Test
|
||||
|
||||
/**
|
||||
* Unit tests for the V4.9 blob primitive ports + V4.10 approval
|
||||
* helpers + scrypt/argon2id wrappers. Cross-platform vector parity
|
||||
* lives in `CrossPlatformVectorTest`; this file tests Kotlin-side
|
||||
* round-trip behavior independent of the TS reference.
|
||||
*/
|
||||
class BlobAndApprovalTest {
|
||||
|
||||
private val crypto = TinkProvider()
|
||||
|
||||
private fun hex(bytes: ByteArray): String =
|
||||
bytes.joinToString("") { "%02x".format(it) }
|
||||
|
||||
// ─── V4.9 blob KDF ─────────────────────────────────────────
|
||||
|
||||
@Test
|
||||
fun deriveBlobSlotIdIsDeterministicPerMasterAndApp() {
|
||||
val km = ByteArray(32) { it.toByte() }
|
||||
val a1 = deriveBlobSlotId(crypto, km, "foo")
|
||||
val a2 = deriveBlobSlotId(crypto, km, "foo")
|
||||
assertArrayEquals(a1, a2)
|
||||
|
||||
val b = deriveBlobSlotId(crypto, km, "bar")
|
||||
assertFalse(a1.contentEquals(b))
|
||||
|
||||
val km2 = ByteArray(32) { (it + 1).toByte() }
|
||||
val c = deriveBlobSlotId(crypto, km2, "foo")
|
||||
assertFalse(a1.contentEquals(c))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun blobKdfHelpersAreIndependent() {
|
||||
val km = ByteArray(32) { it.toByte() }
|
||||
val slot = deriveBlobSlotId(crypto, km, "x")
|
||||
val key = deriveBlobKey(crypto, km, "x")
|
||||
val seed = deriveBlobSigningSeed(crypto, km, "x")
|
||||
assertFalse(slot.contentEquals(key))
|
||||
assertFalse(slot.contentEquals(seed))
|
||||
assertFalse(key.contentEquals(seed))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun ed25519PublicKeyFromSeedIsDeterministic() {
|
||||
val seed = ByteArray(32) { it.toByte() }
|
||||
val pk1 = ed25519PublicKeyFromSeed(seed)
|
||||
val pk2 = ed25519PublicKeyFromSeed(seed)
|
||||
assertArrayEquals(pk1, pk2)
|
||||
assertEquals(32, pk1.size)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun slotIdToHexProducesLowercase64Chars() {
|
||||
val s = ByteArray(32) { 0xab.toByte() }
|
||||
val hex = slotIdToHex(s)
|
||||
assertEquals(64, hex.length)
|
||||
assertEquals("a".repeat(0) + "ab".repeat(32), hex)
|
||||
}
|
||||
|
||||
// ─── V4.9 AEAD round-trip ──────────────────────────────────
|
||||
|
||||
@Test
|
||||
fun aeadSealOpenRoundTrip() {
|
||||
val key = crypto.randomBytes(32)
|
||||
val nonce = crypto.randomBytes(12)
|
||||
val pt = "hello".toByteArray()
|
||||
val aad = blobAadForSlot("00".repeat(32))
|
||||
val sealed = aeadSeal(key, nonce, pt, aad)
|
||||
val opened = aeadOpen(key, sealed, aad)
|
||||
assertArrayEquals(pt, opened)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun aeadOpenWithWrongAadFails() {
|
||||
val key = crypto.randomBytes(32)
|
||||
val nonce = crypto.randomBytes(12)
|
||||
val pt = "hello".toByteArray()
|
||||
val aad1 = blobAadForSlot("00".repeat(32))
|
||||
val aad2 = blobAadForSlot("ff".repeat(32))
|
||||
val sealed = aeadSeal(key, nonce, pt, aad1)
|
||||
try {
|
||||
aeadOpen(key, sealed, aad2)
|
||||
org.junit.Assert.fail("expected AEAD to reject wrong AAD")
|
||||
} catch (_: Exception) {
|
||||
// expected
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun aeadOpenWithExpectedNonceMismatchFails() {
|
||||
val key = crypto.randomBytes(32)
|
||||
val nonce = crypto.randomBytes(12)
|
||||
val wrongNonce = crypto.randomBytes(12)
|
||||
val pt = "hello".toByteArray()
|
||||
val aad = blobAadForSlot("00".repeat(32))
|
||||
val sealed = aeadSeal(key, nonce, pt, aad)
|
||||
try {
|
||||
aeadOpen(key, sealed, aad, expectedNonce = wrongNonce)
|
||||
org.junit.Assert.fail("expected expectedNonce check to reject")
|
||||
} catch (_: IllegalArgumentException) {
|
||||
// expected
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Password KDFs ─────────────────────────────────────────
|
||||
|
||||
@Test
|
||||
fun scryptDerivesDeterministically() {
|
||||
val pw = "correct-horse-battery-staple"
|
||||
val salt = ByteArray(16) { 0x42.toByte() }
|
||||
val k1 = deriveMasterKey(pw, salt, ScryptParams(n = 1024))
|
||||
val k2 = deriveMasterKey(pw, salt, ScryptParams(n = 1024))
|
||||
assertArrayEquals(k1, k2)
|
||||
assertEquals(32, k1.size)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun argon2idDerivesDeterministically() {
|
||||
val pw = "1234"
|
||||
val salt = ByteArray(16) { 0x55.toByte() }
|
||||
val k1 = deriveMasterKeyArgon2id(pw, salt, Argon2idParams(m = 256, t = 2))
|
||||
val k2 = deriveMasterKeyArgon2id(pw, salt, Argon2idParams(m = 256, t = 2))
|
||||
assertArrayEquals(k1, k2)
|
||||
assertEquals(32, k1.size)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun nfkcNormalizationMakesEquivalentInputsConverge() {
|
||||
// "café" can be encoded either as 'c','a','f','é' (NFC) or
|
||||
// 'c','a','f','e','́' (NFD). NFKC normalization on both
|
||||
// should converge to the same bytes.
|
||||
val nfc = "café"
|
||||
val nfd = "café"
|
||||
assertNotEquals(nfc, nfd)
|
||||
val salt = ByteArray(16) { 1.toByte() }
|
||||
val k1 = deriveMasterKey(nfc, salt, ScryptParams(n = 1024))
|
||||
val k2 = deriveMasterKey(nfd, salt, ScryptParams(n = 1024))
|
||||
assertArrayEquals(k1, k2)
|
||||
}
|
||||
|
||||
// ─── Canonical profile schema ──────────────────────────────
|
||||
|
||||
private fun makeClient(name: String, trusted: Boolean = false): Pair<ProfileClientEntry, ByteArray> {
|
||||
val seed = crypto.randomBytes(32)
|
||||
val pubkey = ed25519PublicKeyFromSeed(seed)
|
||||
val fp = "fp-$name-${hex(pubkey).take(8)}"
|
||||
return ProfileClientEntry(
|
||||
address = "device:$name",
|
||||
identityPublicKey = hex(pubkey),
|
||||
identityFingerprint = fp,
|
||||
name = name,
|
||||
kind = "mobile",
|
||||
addedAt = 1_700_000_000_000L,
|
||||
trustedApprover = trusted,
|
||||
) to seed
|
||||
}
|
||||
|
||||
private fun makeHost(): ProfileHostEntry = ProfileHostEntry(
|
||||
address = "device:host-server",
|
||||
name = "Server",
|
||||
kind = "server",
|
||||
addedAt = 1_700_000_000_000L,
|
||||
)
|
||||
|
||||
@Test
|
||||
fun emptyCanonicalProfileRoundTrips() {
|
||||
val blob = emptyCanonicalProfile(now = 123L)
|
||||
val bytes = serializeCanonicalProfile(blob)
|
||||
val parsed = parseCanonicalProfile(bytes)
|
||||
assertEquals(1, parsed.version)
|
||||
assertTrue(parsed.hosts.isEmpty())
|
||||
assertTrue(parsed.clients.isEmpty())
|
||||
assertTrue(parsed.trustedApproverFingerprints.isEmpty())
|
||||
assertEquals(123L, parsed.updatedAt)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun upsertClientDenormalizesTrustedApprovers() {
|
||||
var blob = emptyCanonicalProfile(0)
|
||||
val (a, _) = makeClient("phone-a", trusted = true)
|
||||
val (b, _) = makeClient("phone-b", trusted = false)
|
||||
blob = upsertClient(blob, a)
|
||||
blob = upsertClient(blob, b)
|
||||
assertEquals(2, blob.clients.size)
|
||||
assertEquals(listOf(a.identityFingerprint), blob.trustedApproverFingerprints)
|
||||
assertTrue(isTrustedApprover(blob, a.identityFingerprint))
|
||||
assertFalse(isTrustedApprover(blob, b.identityFingerprint))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun setTrustedApproverIsIdempotentNoOpReturnsSameInstance() {
|
||||
var blob = emptyCanonicalProfile(0)
|
||||
val (c, _) = makeClient("phone", trusted = false)
|
||||
blob = upsertClient(blob, c)
|
||||
val before = blob
|
||||
blob = setTrustedApprover(blob, c.identityFingerprint, false, now = 999L)
|
||||
assertTrue(blob === before)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun setTrustedApproverFlipsFlagAndDenormalizedList() {
|
||||
var blob = emptyCanonicalProfile(0)
|
||||
val (c, _) = makeClient("phone", trusted = false)
|
||||
blob = upsertClient(blob, c)
|
||||
blob = setTrustedApprover(blob, c.identityFingerprint, true, now = 100L)
|
||||
assertEquals(listOf(c.identityFingerprint), blob.trustedApproverFingerprints)
|
||||
assertEquals(true, blob.clients[0].trustedApprover)
|
||||
|
||||
blob = setTrustedApprover(blob, c.identityFingerprint, false, now = 200L)
|
||||
assertTrue(blob.trustedApproverFingerprints.isEmpty())
|
||||
assertEquals(false, blob.clients[0].trustedApprover)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun removeClientCleansUpDenormalizedList() {
|
||||
var blob = emptyCanonicalProfile(0)
|
||||
val (c, _) = makeClient("phone", trusted = true)
|
||||
blob = upsertClient(blob, c)
|
||||
blob = removeClient(blob, c.identityFingerprint)
|
||||
assertTrue(blob.clients.isEmpty())
|
||||
assertTrue(blob.trustedApproverFingerprints.isEmpty())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun findClientByFingerprintAndAddress() {
|
||||
var blob = emptyCanonicalProfile(0)
|
||||
val (c, _) = makeClient("phone")
|
||||
blob = upsertClient(blob, c)
|
||||
assertEquals(c.address, findClientByFingerprint(blob, c.identityFingerprint)?.address)
|
||||
assertEquals(
|
||||
c.identityFingerprint,
|
||||
findClientByAddress(blob, c.address)?.identityFingerprint,
|
||||
)
|
||||
assertNull(findClientByFingerprint(blob, "unknown"))
|
||||
assertNull(findClientByAddress(blob, "unknown"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun parseRejectsMalformedProfile() {
|
||||
try {
|
||||
parseCanonicalProfile("not json")
|
||||
org.junit.Assert.fail("expected throw")
|
||||
} catch (_: IllegalArgumentException) {}
|
||||
try {
|
||||
parseCanonicalProfile("""{"version":2}""")
|
||||
org.junit.Assert.fail("expected throw")
|
||||
} catch (_: IllegalArgumentException) {}
|
||||
try {
|
||||
parseCanonicalProfile(
|
||||
"""{"version":1,"clients":[{"address":"x","name":"x","kind":"m","addedAt":0}]}""",
|
||||
)
|
||||
org.junit.Assert.fail("expected throw — missing identityPublicKey")
|
||||
} catch (_: IllegalArgumentException) {}
|
||||
try {
|
||||
parseCanonicalProfile(
|
||||
"""{"version":1,"clients":[{"address":"x","identityPublicKey":"NOTHEX","identityFingerprint":"x","name":"x","kind":"m","addedAt":0}]}""",
|
||||
)
|
||||
org.junit.Assert.fail("expected throw — bad pubkey hex")
|
||||
} catch (_: IllegalArgumentException) {}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun fullProfileSerializeParsePreservesAllFields() {
|
||||
var blob = emptyCanonicalProfile(1L)
|
||||
blob = upsertHost(blob, makeHost(), now = 2L)
|
||||
val (c, _) = makeClient("phone", trusted = true)
|
||||
blob = upsertClient(blob, c, now = 3L)
|
||||
blob = blob.copy(signedBy = "aabbccdd")
|
||||
|
||||
val bytes = serializeCanonicalProfile(blob)
|
||||
val parsed = parseCanonicalProfile(bytes)
|
||||
assertEquals(blob, parsed)
|
||||
}
|
||||
|
||||
// ─── Approval signing payload ──────────────────────────────
|
||||
|
||||
@Test
|
||||
fun canonicalApprovalSigningBytesIsDeterministic() {
|
||||
val a = canonicalApprovalSigningBytes(
|
||||
domain = DEFAULT_APPROVAL_DOMAIN,
|
||||
requestId = "aabbccddeeff00112233445566778899",
|
||||
hostFingerprint = "11111 22222 33333 44444",
|
||||
requestingDeviceFingerprint = "55555 66666 77777 88888",
|
||||
decision = "approve",
|
||||
)
|
||||
val b = canonicalApprovalSigningBytes(
|
||||
domain = DEFAULT_APPROVAL_DOMAIN,
|
||||
requestId = "aabbccddeeff00112233445566778899",
|
||||
hostFingerprint = "11111 22222 33333 44444",
|
||||
requestingDeviceFingerprint = "55555 66666 77777 88888",
|
||||
decision = "approve",
|
||||
)
|
||||
assertArrayEquals(a, b)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun differentDecisionProducesDifferentSigningBytes() {
|
||||
val approve = canonicalApprovalSigningBytes(
|
||||
DEFAULT_APPROVAL_DOMAIN, "r", "h", "d", "approve",
|
||||
)
|
||||
val reject = canonicalApprovalSigningBytes(
|
||||
DEFAULT_APPROVAL_DOMAIN, "r", "h", "d", "reject",
|
||||
)
|
||||
assertFalse(approve.contentEquals(reject))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun differentDomainProducesDifferentSigningBytes() {
|
||||
val a = canonicalApprovalSigningBytes("shade-link-approve-v1", "r", "h", "d", "approve")
|
||||
val b = canonicalApprovalSigningBytes("prism-link-approve-v1", "r", "h", "d", "approve")
|
||||
assertFalse(a.contentEquals(b))
|
||||
}
|
||||
|
||||
// ─── Build / sign / verify ─────────────────────────────────
|
||||
|
||||
private data class Scenario(
|
||||
val phone: ProfileClientEntry,
|
||||
val phoneSeed: ByteArray,
|
||||
val profile: CanonicalProfileBlob,
|
||||
val request: ApprovalRequestFrame,
|
||||
)
|
||||
|
||||
private fun buildScenario(): Scenario {
|
||||
val (phone, seed) = makeClient("phone", trusted = true)
|
||||
var profile = emptyCanonicalProfile(0)
|
||||
profile = upsertHost(profile, makeHost())
|
||||
profile = upsertClient(profile, phone)
|
||||
|
||||
val request = buildApprovalRequest(
|
||||
crypto = crypto,
|
||||
hostAddress = "device:host-server",
|
||||
hostFingerprint = "host-fp-12345",
|
||||
requestingDeviceFingerprint = "cafe-laptop-fp-67890",
|
||||
deviceName = "cafe-laptop",
|
||||
userAgent = "Mozilla/5.0",
|
||||
ipHint = "203.0.113.7",
|
||||
)
|
||||
return Scenario(phone, seed, profile, request)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun happyPathApproveVerifies() {
|
||||
val s = buildScenario()
|
||||
val approval = signProxyApproval(
|
||||
crypto, s.request,
|
||||
decision = "approve",
|
||||
approverFingerprint = s.phone.identityFingerprint,
|
||||
approverSigningKey = s.phoneSeed,
|
||||
)
|
||||
assertEquals("linkApproveByProxy", approval.kind)
|
||||
assertEquals(s.request.requestId, approval.requestId)
|
||||
assertEquals(128, approval.signature.length)
|
||||
|
||||
val r = verifyProxyApproval(crypto, s.request, approval, s.profile)
|
||||
assertTrue(r is VerifyProxyApprovalResult.Ok)
|
||||
assertEquals(s.phone.address, (r as VerifyProxyApprovalResult.Ok).approver.address)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun happyPathRejectVerifies() {
|
||||
val s = buildScenario()
|
||||
val approval = signProxyApproval(
|
||||
crypto, s.request, "reject",
|
||||
s.phone.identityFingerprint, s.phoneSeed,
|
||||
)
|
||||
val r = verifyProxyApproval(crypto, s.request, approval, s.profile)
|
||||
assertTrue(r is VerifyProxyApprovalResult.Ok)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun replayAgainstDifferentRequestFails() {
|
||||
val s = buildScenario()
|
||||
val approval = signProxyApproval(
|
||||
crypto, s.request, "approve",
|
||||
s.phone.identityFingerprint, s.phoneSeed,
|
||||
)
|
||||
val other = s.request.copy(requestId = "f".repeat(32))
|
||||
val r = verifyProxyApproval(crypto, other, approval, s.profile)
|
||||
assertEquals(
|
||||
VerifyProxyApprovalResult.Reason.REQUEST_ID_MISMATCH,
|
||||
(r as VerifyProxyApprovalResult.Failed).reason,
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun decisionTamperingFails() {
|
||||
val s = buildScenario()
|
||||
val approval = signProxyApproval(
|
||||
crypto, s.request, "approve",
|
||||
s.phone.identityFingerprint, s.phoneSeed,
|
||||
)
|
||||
val tampered = approval.copy(decision = "reject")
|
||||
val r = verifyProxyApproval(crypto, s.request, tampered, s.profile)
|
||||
assertEquals(
|
||||
VerifyProxyApprovalResult.Reason.BAD_SIGNATURE,
|
||||
(r as VerifyProxyApprovalResult.Failed).reason,
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun hostFingerprintSwapFails() {
|
||||
val s = buildScenario()
|
||||
val approval = signProxyApproval(
|
||||
crypto, s.request, "approve",
|
||||
s.phone.identityFingerprint, s.phoneSeed,
|
||||
)
|
||||
val swapped = s.request.copy(hostFingerprint = "evil-host-fp")
|
||||
val r = verifyProxyApproval(crypto, swapped, approval, s.profile)
|
||||
assertEquals(
|
||||
VerifyProxyApprovalResult.Reason.BAD_SIGNATURE,
|
||||
(r as VerifyProxyApprovalResult.Failed).reason,
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun domainMismatchIsRejectedBeforeSignature() {
|
||||
val s = buildScenario()
|
||||
val approval = signProxyApproval(
|
||||
crypto, s.request, "approve",
|
||||
s.phone.identityFingerprint, s.phoneSeed,
|
||||
)
|
||||
val r = verifyProxyApproval(
|
||||
crypto, s.request, approval.copy(domain = "prism-link-approve-v1"), s.profile,
|
||||
)
|
||||
assertEquals(
|
||||
VerifyProxyApprovalResult.Reason.DOMAIN_MISMATCH,
|
||||
(r as VerifyProxyApprovalResult.Failed).reason,
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun unknownApproverFails() {
|
||||
val s = buildScenario()
|
||||
val approval = signProxyApproval(
|
||||
crypto, s.request, "approve",
|
||||
s.phone.identityFingerprint, s.phoneSeed,
|
||||
)
|
||||
val lying = approval.copy(approverFingerprint = "no-such-fingerprint")
|
||||
val r = verifyProxyApproval(crypto, s.request, lying, s.profile)
|
||||
assertEquals(
|
||||
VerifyProxyApprovalResult.Reason.UNKNOWN_APPROVER,
|
||||
(r as VerifyProxyApprovalResult.Failed).reason,
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun revokedApproverFailsWithNotTrusted() {
|
||||
val s = buildScenario()
|
||||
val approval = signProxyApproval(
|
||||
crypto, s.request, "approve",
|
||||
s.phone.identityFingerprint, s.phoneSeed,
|
||||
)
|
||||
val revoked = setTrustedApprover(s.profile, s.phone.identityFingerprint, false)
|
||||
val r = verifyProxyApproval(crypto, s.request, approval, revoked)
|
||||
assertEquals(
|
||||
VerifyProxyApprovalResult.Reason.NOT_TRUSTED,
|
||||
(r as VerifyProxyApprovalResult.Failed).reason,
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun expiredRequestIsRejected() {
|
||||
val s = buildScenario()
|
||||
val approval = signProxyApproval(
|
||||
crypto, s.request, "approve",
|
||||
s.phone.identityFingerprint, s.phoneSeed,
|
||||
)
|
||||
val r = verifyProxyApproval(crypto, s.request, approval, s.profile, now = s.request.expiresAt + 1)
|
||||
assertEquals(
|
||||
VerifyProxyApprovalResult.Reason.EXPIRED,
|
||||
(r as VerifyProxyApprovalResult.Failed).reason,
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun signatureWithWrongKeyFails() {
|
||||
val s = buildScenario()
|
||||
val wrongSeed = crypto.randomBytes(32)
|
||||
val approval = signProxyApproval(
|
||||
crypto, s.request, "approve",
|
||||
approverFingerprint = s.phone.identityFingerprint, // claim phone
|
||||
approverSigningKey = wrongSeed, // sign with different key
|
||||
)
|
||||
val r = verifyProxyApproval(crypto, s.request, approval, s.profile)
|
||||
assertEquals(
|
||||
VerifyProxyApprovalResult.Reason.BAD_SIGNATURE,
|
||||
(r as VerifyProxyApprovalResult.Failed).reason,
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun customDomainSurvivesRoundTrip() {
|
||||
val s = buildScenario()
|
||||
val request = s.request.copy(domain = "prism-link-approve-v1")
|
||||
val approval = signProxyApproval(
|
||||
crypto, request, "approve",
|
||||
s.phone.identityFingerprint, s.phoneSeed,
|
||||
)
|
||||
assertEquals("prism-link-approve-v1", approval.domain)
|
||||
val r = verifyProxyApproval(crypto, request, approval, s.profile)
|
||||
assertTrue(r is VerifyProxyApprovalResult.Ok)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun requestIdIs32LowercaseHexChars() {
|
||||
val r = buildApprovalRequest(
|
||||
crypto, "device:h", "h", "r",
|
||||
)
|
||||
assertTrue(r.requestId.matches(Regex("^[0-9a-f]{32}$")))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun consecutiveBuildsProduceDistinctRequestIds() {
|
||||
val a = buildApprovalRequest(crypto, "device:h", "h", "r")
|
||||
val b = buildApprovalRequest(crypto, "device:h", "h", "r")
|
||||
assertNotEquals(a.requestId, b.requestId)
|
||||
}
|
||||
|
||||
// ─── Sanity glue: TS-side reference frame parses on Kotlin ──
|
||||
|
||||
@Test
|
||||
fun tsStyleProxyApprovalFrameParsesAndStructurallyMatches() {
|
||||
// Construct a frame the way @shade/sdk would emit it via JSON,
|
||||
// and check our Kotlin types accept the same field names.
|
||||
val expected = ProxyApprovalFrame(
|
||||
requestId = "00112233445566778899aabbccddeeff",
|
||||
decision = "approve",
|
||||
approverFingerprint = "fp",
|
||||
signature = "ab".repeat(64),
|
||||
domain = DEFAULT_APPROVAL_DOMAIN,
|
||||
)
|
||||
assertEquals("linkApproveByProxy", expected.kind)
|
||||
assertEquals("approve", expected.decision)
|
||||
|
||||
val req = ApprovalRequestFrame(
|
||||
requestId = "00112233445566778899aabbccddeeff",
|
||||
hostAddress = "device:h",
|
||||
hostFingerprint = "host-fp",
|
||||
requestingDevice = ApprovalRequestingDevice(
|
||||
fingerprint = "req-fp",
|
||||
receivedAt = 1L,
|
||||
),
|
||||
expiresAt = 2L,
|
||||
domain = DEFAULT_APPROVAL_DOMAIN,
|
||||
)
|
||||
assertNotNull(req)
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,13 @@
|
||||
package no.zyon.shade
|
||||
|
||||
import no.zyon.shade.approval.canonicalApprovalSigningBytes
|
||||
import no.zyon.shade.backup.deriveBackupKey
|
||||
import no.zyon.shade.blob.aeadOpen
|
||||
import no.zyon.shade.blob.blobAadForSlot
|
||||
import no.zyon.shade.blob.deriveBlobKey
|
||||
import no.zyon.shade.blob.deriveBlobSigningSeed
|
||||
import no.zyon.shade.blob.deriveBlobSlotId
|
||||
import no.zyon.shade.blob.ed25519PublicKeyFromSeed
|
||||
import no.zyon.shade.crypto.TinkProvider
|
||||
import no.zyon.shade.fingerprint.computeFingerprint
|
||||
import no.zyon.shade.group.encodeSenderHeader
|
||||
@@ -447,6 +454,92 @@ class CrossPlatformVectorTest {
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun blobKdfAndAeadVectorsMatch() {
|
||||
val vectors = loadVectors("blob.json")
|
||||
var kdfMatched = 0
|
||||
var aeadMatched = 0
|
||||
for (i in 0 until vectors.length()) {
|
||||
val v = vectors.getJSONObject(i)
|
||||
val desc = v.getString("description")
|
||||
if (desc.startsWith("V4.9 blob KDF")) {
|
||||
kdfMatched++
|
||||
val masterKey = fromHex(v.getString("masterKey"))
|
||||
val app = v.getString("app")
|
||||
val slotId = deriveBlobSlotId(crypto, masterKey, app)
|
||||
assertEquals(v.getString("slotId"), hex(slotId))
|
||||
assertEquals(v.getString("blobKey"), hex(deriveBlobKey(crypto, masterKey, app)))
|
||||
val seed = deriveBlobSigningSeed(crypto, masterKey, app)
|
||||
assertEquals(v.getString("signingSeed"), hex(seed))
|
||||
assertEquals(v.getString("ownerPubkey"), hex(ed25519PublicKeyFromSeed(seed)))
|
||||
} else if (desc.startsWith("V4.9 blob AEAD")) {
|
||||
aeadMatched++
|
||||
val key = fromHex(v.getString("key"))
|
||||
val slotIdHex = v.getString("slotIdHex")
|
||||
val expectedPlaintext = fromHex(v.getString("plaintext"))
|
||||
val wire = fromHex(v.getString("wire"))
|
||||
val aad = blobAadForSlot(slotIdHex)
|
||||
val opened = aeadOpen(key, wire, aad)
|
||||
assertEquals(hex(expectedPlaintext), hex(opened))
|
||||
}
|
||||
}
|
||||
assertTrue("KDF vectors expected", kdfMatched >= 3)
|
||||
assertTrue("AEAD vectors expected", aeadMatched >= 2)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun approvalSigningPayloadVectorsMatch() {
|
||||
val vectors = loadVectors("approval.json")
|
||||
var payloadMatched = 0
|
||||
var e2eMatched = 0
|
||||
for (i in 0 until vectors.length()) {
|
||||
val v = vectors.getJSONObject(i)
|
||||
val desc = v.getString("description")
|
||||
if (desc.startsWith("V4.10 approval signing payload")) {
|
||||
payloadMatched++
|
||||
val out = canonicalApprovalSigningBytes(
|
||||
domain = v.getString("domain"),
|
||||
requestId = v.getString("requestId"),
|
||||
hostFingerprint = v.getString("hostFingerprint"),
|
||||
requestingDeviceFingerprint = v.getString("requestingDeviceFingerprint"),
|
||||
decision = v.getString("decision"),
|
||||
)
|
||||
assertEquals(v.getString("signingPayload"), hex(out))
|
||||
} else if (desc.startsWith("V4.10 approval Ed25519 sign/verify")) {
|
||||
e2eMatched++
|
||||
val seed = fromHex(v.getString("seed"))
|
||||
val pubkey = fromHex(v.getString("publicKey"))
|
||||
assertEquals(hex(pubkey), hex(ed25519PublicKeyFromSeed(seed)))
|
||||
|
||||
val req = v.getJSONObject("request")
|
||||
val payload = canonicalApprovalSigningBytes(
|
||||
domain = req.getString("domain"),
|
||||
requestId = req.getString("requestId"),
|
||||
hostFingerprint = req.getString("hostFingerprint"),
|
||||
requestingDeviceFingerprint = req.getString("requestingDeviceFingerprint"),
|
||||
decision = req.getString("decision"),
|
||||
)
|
||||
assertEquals(v.getString("signingPayload"), hex(payload))
|
||||
|
||||
// Verify the TS-generated signature against our pubkey + payload.
|
||||
// This is the load-bearing parity check: a Kotlin-implemented
|
||||
// verifyProxyApproval running against a TS-signed approval
|
||||
// succeeds.
|
||||
val sig = fromHex(v.getString("signature"))
|
||||
val ok = crypto.verify(pubkey, payload, sig)
|
||||
assertTrue("Ed25519 verify of TS-signed approval failed", ok)
|
||||
|
||||
// And: Kotlin signs the same payload with the same seed and
|
||||
// produces a sig the TS pubkey verifies. Ed25519 is
|
||||
// deterministic, so the sig bytes also match exactly.
|
||||
val mySig = crypto.sign(seed, payload)
|
||||
assertEquals(v.getString("signature"), hex(mySig))
|
||||
}
|
||||
}
|
||||
assertTrue("payload vectors expected", payloadMatched >= 3)
|
||||
assertTrue("e2e sign/verify vector expected", e2eMatched >= 1)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun ratchetStepRoundtripMatches() {
|
||||
val vectors = loadVectors("ratchet-step.json")
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
package no.zyon.shade
|
||||
|
||||
import no.zyon.shade.serialization.SessionStateJson
|
||||
import no.zyon.shade.types.ChainState
|
||||
import no.zyon.shade.types.IdentityKeyPair
|
||||
import no.zyon.shade.types.KeyPair
|
||||
import no.zyon.shade.types.OneTimePreKey
|
||||
import no.zyon.shade.types.SessionState
|
||||
import no.zyon.shade.types.SignedPreKey
|
||||
import org.junit.Assert.assertArrayEquals
|
||||
import org.junit.Assert.assertEquals
|
||||
import org.junit.Assert.assertNotNull
|
||||
import org.junit.Assert.assertNull
|
||||
import org.junit.Test
|
||||
|
||||
/**
|
||||
* Round-trip tests for the at-rest JSON serialization used by
|
||||
* `KeystoreStorage`. The format isn't cross-platform (TS uses its
|
||||
* own shape) — what matters is `serialize → deserialize` preserves
|
||||
* every byte of every key.
|
||||
*/
|
||||
class SessionStateJsonTest {
|
||||
|
||||
private fun bytes(n: Int, fill: Byte): ByteArray = ByteArray(n) { fill }
|
||||
|
||||
@Test
|
||||
fun identityKeyPairRoundTrip() {
|
||||
val k = IdentityKeyPair(
|
||||
signingPublicKey = bytes(32, 0x11),
|
||||
signingPrivateKey = bytes(32, 0x22),
|
||||
dhPublicKey = bytes(32, 0x33),
|
||||
dhPrivateKey = bytes(32, 0x44),
|
||||
)
|
||||
val s = SessionStateJson.serializeIdentityKeyPair(k)
|
||||
val d = SessionStateJson.deserializeIdentityKeyPair(s)
|
||||
assertArrayEquals(k.signingPublicKey, d.signingPublicKey)
|
||||
assertArrayEquals(k.signingPrivateKey, d.signingPrivateKey)
|
||||
assertArrayEquals(k.dhPublicKey, d.dhPublicKey)
|
||||
assertArrayEquals(k.dhPrivateKey, d.dhPrivateKey)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun signedPreKeyRoundTrip() {
|
||||
val k = SignedPreKey(
|
||||
keyId = 42,
|
||||
keyPair = KeyPair(publicKey = bytes(32, 0x55), privateKey = bytes(32, 0x66)),
|
||||
signature = bytes(64, 0x77),
|
||||
timestamp = 1_700_000_000_000L,
|
||||
)
|
||||
val s = SessionStateJson.serializeSignedPreKey(k)
|
||||
val d = SessionStateJson.deserializeSignedPreKey(s)
|
||||
assertEquals(k.keyId, d.keyId)
|
||||
assertArrayEquals(k.keyPair.publicKey, d.keyPair.publicKey)
|
||||
assertArrayEquals(k.keyPair.privateKey, d.keyPair.privateKey)
|
||||
assertArrayEquals(k.signature, d.signature)
|
||||
assertEquals(k.timestamp, d.timestamp)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun oneTimePreKeyRoundTrip() {
|
||||
val k = OneTimePreKey(
|
||||
keyId = 7,
|
||||
keyPair = KeyPair(publicKey = bytes(32, 0x88.toByte()), privateKey = bytes(32, 0x99.toByte())),
|
||||
)
|
||||
val s = SessionStateJson.serializeOneTimePreKey(k)
|
||||
val d = SessionStateJson.deserializeOneTimePreKey(s)
|
||||
assertEquals(k.keyId, d.keyId)
|
||||
assertArrayEquals(k.keyPair.publicKey, d.keyPair.publicKey)
|
||||
assertArrayEquals(k.keyPair.privateKey, d.keyPair.privateKey)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun sessionStateRoundTripFullPopulated() {
|
||||
val state = SessionState(
|
||||
remoteIdentityKey = bytes(32, 0x01),
|
||||
rootKey = bytes(32, 0x02),
|
||||
sendChain = ChainState(chainKey = bytes(32, 0x03), counter = 5),
|
||||
receiveChain = ChainState(chainKey = bytes(32, 0x04), counter = 3),
|
||||
dhSend = KeyPair(publicKey = bytes(32, 0x05), privateKey = bytes(32, 0x06)),
|
||||
dhReceive = bytes(32, 0x07),
|
||||
previousSendCounter = 9,
|
||||
skippedKeys = mutableMapOf(
|
||||
"remote:1" to bytes(32, 0x0A),
|
||||
"remote:2" to bytes(32, 0x0B),
|
||||
),
|
||||
)
|
||||
val s = SessionStateJson.serialize(state)
|
||||
val d = SessionStateJson.deserialize(s)
|
||||
assertArrayEquals(state.remoteIdentityKey, d.remoteIdentityKey)
|
||||
assertArrayEquals(state.rootKey, d.rootKey)
|
||||
assertArrayEquals(state.sendChain.chainKey, d.sendChain.chainKey)
|
||||
assertEquals(state.sendChain.counter, d.sendChain.counter)
|
||||
assertNotNull(d.receiveChain)
|
||||
assertArrayEquals(state.receiveChain!!.chainKey, d.receiveChain!!.chainKey)
|
||||
assertArrayEquals(state.dhSend.publicKey, d.dhSend.publicKey)
|
||||
assertArrayEquals(state.dhSend.privateKey, d.dhSend.privateKey)
|
||||
assertArrayEquals(state.dhReceive, d.dhReceive)
|
||||
assertEquals(state.previousSendCounter, d.previousSendCounter)
|
||||
assertEquals(state.skippedKeys.size, d.skippedKeys.size)
|
||||
for ((k, v) in state.skippedKeys) {
|
||||
assertArrayEquals(v, d.skippedKeys[k])
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun sessionStateRoundTripWithNullableFields() {
|
||||
val state = SessionState(
|
||||
remoteIdentityKey = bytes(32, 0x01),
|
||||
rootKey = bytes(32, 0x02),
|
||||
sendChain = ChainState(chainKey = bytes(32, 0x03), counter = 0),
|
||||
receiveChain = null,
|
||||
dhSend = KeyPair(publicKey = bytes(32, 0x05), privateKey = bytes(32, 0x06)),
|
||||
dhReceive = null,
|
||||
previousSendCounter = 0,
|
||||
skippedKeys = mutableMapOf(),
|
||||
)
|
||||
val s = SessionStateJson.serialize(state)
|
||||
val d = SessionStateJson.deserialize(s)
|
||||
assertNull(d.receiveChain)
|
||||
assertNull(d.dhReceive)
|
||||
assertEquals(0, d.skippedKeys.size)
|
||||
}
|
||||
}
|
||||
73
bun.lock
73
bun.lock
@@ -17,7 +17,7 @@
|
||||
},
|
||||
"packages/shade-cli": {
|
||||
"name": "@shade/cli",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"bin": {
|
||||
"shade": "src/cli.ts",
|
||||
},
|
||||
@@ -36,7 +36,7 @@
|
||||
},
|
||||
"packages/shade-core": {
|
||||
"name": "@shade/core",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/observability": "workspace:*",
|
||||
},
|
||||
@@ -49,7 +49,7 @@
|
||||
},
|
||||
"packages/shade-crypto-web": {
|
||||
"name": "@shade/crypto-web",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@noble/curves": "^2.0.1",
|
||||
"@noble/hashes": "^2.0.1",
|
||||
@@ -59,7 +59,7 @@
|
||||
},
|
||||
"packages/shade-dashboard": {
|
||||
"name": "@shade/dashboard",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/widgets": "workspace:*",
|
||||
"react": "^19.0.0",
|
||||
@@ -74,7 +74,7 @@
|
||||
},
|
||||
"packages/shade-files": {
|
||||
"name": "@shade/files",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
@@ -101,7 +101,7 @@
|
||||
},
|
||||
"packages/shade-inbox": {
|
||||
"name": "@shade/inbox",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/proto": "workspace:*",
|
||||
@@ -114,7 +114,7 @@
|
||||
},
|
||||
"packages/shade-inbox-server": {
|
||||
"name": "@shade/inbox-server",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/observability": "workspace:*",
|
||||
@@ -132,7 +132,7 @@
|
||||
},
|
||||
"packages/shade-key-transparency": {
|
||||
"name": "@shade/key-transparency",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@noble/hashes": "^2.0.1",
|
||||
"@shade/core": "workspace:*",
|
||||
@@ -144,11 +144,11 @@
|
||||
},
|
||||
"packages/shade-keychain": {
|
||||
"name": "@shade/keychain",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
},
|
||||
"packages/shade-observability": {
|
||||
"name": "@shade/observability",
|
||||
"version": "0.1.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@noble/hashes": "^2.0.1",
|
||||
},
|
||||
@@ -166,7 +166,7 @@
|
||||
},
|
||||
"packages/shade-observer": {
|
||||
"name": "@shade/observer",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/server": "workspace:*",
|
||||
@@ -178,14 +178,14 @@
|
||||
},
|
||||
"packages/shade-proto": {
|
||||
"name": "@shade/proto",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
},
|
||||
},
|
||||
"packages/shade-recovery": {
|
||||
"name": "@shade/recovery",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
@@ -198,22 +198,25 @@
|
||||
},
|
||||
"packages/shade-sdk": {
|
||||
"name": "@shade/sdk",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
"@shade/files": "workspace:*",
|
||||
"@shade/inbox": "workspace:*",
|
||||
"@shade/key-transparency": "workspace:*",
|
||||
"@shade/observability": "workspace:*",
|
||||
"@shade/observer": "workspace:*",
|
||||
"@shade/proto": "workspace:*",
|
||||
"@shade/server": "workspace:*",
|
||||
"@shade/storage-encrypted": "workspace:*",
|
||||
"@shade/storage-sqlite": "workspace:*",
|
||||
"@shade/streams": "workspace:*",
|
||||
"@shade/transfer": "workspace:*",
|
||||
"@shade/transport": "workspace:*",
|
||||
},
|
||||
"devDependencies": {
|
||||
"@shade/inbox-server": "workspace:*",
|
||||
"@shade/transport-webrtc": "workspace:*",
|
||||
},
|
||||
"peerDependencies": {
|
||||
@@ -225,7 +228,7 @@
|
||||
},
|
||||
"packages/shade-server": {
|
||||
"name": "@shade/server",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/inbox-server": "workspace:*",
|
||||
@@ -245,15 +248,19 @@
|
||||
},
|
||||
"packages/shade-storage-encrypted": {
|
||||
"name": "@shade/storage-encrypted",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@noble/hashes": "^2.0.1",
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
"@shade/storage-postgres": "workspace:*",
|
||||
"@shade/storage-sqlite": "workspace:*",
|
||||
"idb": "^8.0.3",
|
||||
"postgres": "^3.4.9",
|
||||
},
|
||||
"devDependencies": {
|
||||
"fake-indexeddb": "^6.0.0",
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@shade/keychain": "workspace:*",
|
||||
},
|
||||
@@ -261,9 +268,21 @@
|
||||
"@shade/keychain",
|
||||
],
|
||||
},
|
||||
"packages/shade-storage-indexeddb": {
|
||||
"name": "@shade/storage-indexeddb",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"idb": "^8.0.3",
|
||||
},
|
||||
"devDependencies": {
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
"fake-indexeddb": "^6.0.0",
|
||||
},
|
||||
},
|
||||
"packages/shade-storage-postgres": {
|
||||
"name": "@shade/storage-postgres",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/inbox-server": "workspace:*",
|
||||
@@ -278,7 +297,7 @@
|
||||
},
|
||||
"packages/shade-storage-sqlite": {
|
||||
"name": "@shade/storage-sqlite",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
@@ -288,7 +307,7 @@
|
||||
},
|
||||
"packages/shade-streams": {
|
||||
"name": "@shade/streams",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@noble/hashes": "^2.0.1",
|
||||
"@shade/core": "workspace:*",
|
||||
@@ -300,7 +319,7 @@
|
||||
},
|
||||
"packages/shade-transfer": {
|
||||
"name": "@shade/transfer",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
@@ -317,7 +336,7 @@
|
||||
},
|
||||
"packages/shade-transport": {
|
||||
"name": "@shade/transport",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
@@ -328,7 +347,7 @@
|
||||
},
|
||||
"packages/shade-transport-bridge": {
|
||||
"name": "@shade/transport-bridge",
|
||||
"version": "0.1.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/server": "workspace:*",
|
||||
@@ -350,7 +369,7 @@
|
||||
},
|
||||
"packages/shade-transport-webrtc": {
|
||||
"name": "@shade/transport-webrtc",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/streams": "workspace:*",
|
||||
@@ -359,7 +378,7 @@
|
||||
},
|
||||
"packages/shade-widgets": {
|
||||
"name": "@shade/widgets",
|
||||
"version": "0.4.0",
|
||||
"version": "4.8.5",
|
||||
"dependencies": {
|
||||
"@shade/recovery": "workspace:*",
|
||||
"@shade/sdk": "workspace:*",
|
||||
@@ -568,6 +587,8 @@
|
||||
|
||||
"@shade/storage-encrypted": ["@shade/storage-encrypted@workspace:packages/shade-storage-encrypted"],
|
||||
|
||||
"@shade/storage-indexeddb": ["@shade/storage-indexeddb@workspace:packages/shade-storage-indexeddb"],
|
||||
|
||||
"@shade/storage-postgres": ["@shade/storage-postgres@workspace:packages/shade-storage-postgres"],
|
||||
|
||||
"@shade/storage-sqlite": ["@shade/storage-sqlite@workspace:packages/shade-storage-sqlite"],
|
||||
@@ -626,6 +647,8 @@
|
||||
|
||||
"escalade": ["escalade@3.2.0", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="],
|
||||
|
||||
"fake-indexeddb": ["fake-indexeddb@6.2.5", "", {}, "sha512-CGnyrvbhPlWYMngksqrSSUT1BAVP49dZocrHuK0SvtR0D5TMs5wP0o3j7jexDJW01KSadjBp1M/71o/KR3nD1w=="],
|
||||
|
||||
"fast-check": ["fast-check@3.23.2", "", { "dependencies": { "pure-rand": "^6.1.0" } }, "sha512-h5+1OzzfCC3Ef7VbtKdcv7zsstUQwUDlYpUTvjeUsJAssPgLn7QzbboPtL5ro04Mq0rPOsMzl7q5hIbRs2wD1A=="],
|
||||
|
||||
"fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="],
|
||||
@@ -638,6 +661,8 @@
|
||||
|
||||
"hono": ["hono@4.12.12", "", {}, "sha512-p1JfQMKaceuCbpJKAPKVqyqviZdS0eUxH9v82oWo1kb9xjQ5wA6iP3FNVAPDFlz5/p7d45lO+BpSk1tuSZMF4Q=="],
|
||||
|
||||
"idb": ["idb@8.0.3", "", {}, "sha512-LtwtVyVYO5BqRvcsKuB2iUMnHwPVByPCXFXOpuU96IZPPoPN6xjOGxZQ74pgSVVLQWtUOYgyeL4GE98BY5D3wg=="],
|
||||
|
||||
"js-tokens": ["js-tokens@4.0.0", "", {}, "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ=="],
|
||||
|
||||
"jsesc": ["jsesc@3.1.0", "", { "bin": { "jsesc": "bin/jsesc" } }, "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA=="],
|
||||
|
||||
@@ -81,6 +81,7 @@ Tables will be created automatically with the `shade_server_*` prefix, so they c
|
||||
| `SHADE_CLEANUP_INTERVAL_HOURS` | `24` | Cleanup cycle interval |
|
||||
| `SHADE_LOG_LEVEL` | `info` | `debug` / `info` / `warn` / `error` |
|
||||
| `SHADE_OTEL_ENABLED` | unset | Set to `1`/`true` to enable OpenTelemetry tracing on `withTracer()`-configured deployments. See [`observability.md`](./observability.md). |
|
||||
| `SHADE_DISABLE_RATE_LIMIT` | unset | Set to `1` to disable IP rate-limits on every prekey + inbox route. **Single-tenant deployments only** — multi-tenant relays must leave this unset to keep the abuse defenses on. |
|
||||
|
||||
## Health and observability
|
||||
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
# Shade V2.1 — Improvements (infrastructure, storage, operations, security)
|
||||
|
||||
This document describes **improvements** agreed for next-generation work on Shade: clearer product story, stronger storage, mobile parity, operational hardening, transfer abuse, and a formal security narrative.
|
||||
**Status:** Done — superseded by the V3.1 → V3.12 plans, all of which
|
||||
landed in the 4.0 GA release. This document is preserved as historical
|
||||
context for the original V2.1 backlog; the concrete deliverables live
|
||||
under [`docs/archive/V3.*.md`](./).
|
||||
|
||||
**Audience:** **Maintainers and contributors** implementing the changes. Add status fields as items land in code/docs.
|
||||
This document describes **improvements** agreed for next-generation work on Shade: clearer product story, stronger storage, mobile parity, operational hardening, transfer abuse, and a formal security narrative.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
# Shade V2.2 — Feature plan: product, platform, and developer experience
|
||||
|
||||
This document gathers **planned features** that extend Shade beyond today’s core (X3DH + Double Ratchet + Streams/transfer): groups, asynchronous delivery, richer file UX, web workers, CLI, API docs, and scaffolding.
|
||||
**Status:** Done — superseded by V3.6 (inbox), V3.7 (bridge), V3.8
|
||||
(workers), V3.9 (file metadata), V3.10 (recovery), V3.12 (KT). All
|
||||
landed in the 4.0 GA release; see [`docs/ROADMAP.md`](../ROADMAP.md).
|
||||
|
||||
Add optional per-feature status (Idea / Design / IMP / Done).
|
||||
This document gathers **planned features** that extend Shade beyond today’s core (X3DH + Double Ratchet + Streams/transfer): groups, asynchronous delivery, richer file UX, web workers, CLI, API docs, and scaffolding.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# Shade V2.3 — Tillit, retention, integrasjon og observability
|
||||
|
||||
**Status:** Done — superseded by V3.3 (trust UX), V3.4 (observability),
|
||||
V3.10 (recovery), V3.12 (KT). Alt levert i 4.0 GA — se
|
||||
[`docs/ROADMAP.md`](../ROADMAP.md).
|
||||
|
||||
Dette dokumentet beskriver **høyere ambisjonsnivå** og **plattformkryssende** arbeid: brukertilfeller der tillit må være **eksplisitt**, data må **utkrympes**/ryddes automatisk, apper må **enkelt koble seg på** kjente transportmønstre, og drift må **observere** uten å lekke innhold.
|
||||
|
||||
---
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# V3.12 — Key Transparency: Designnotat
|
||||
|
||||
**Status:** Approved (in-tree review — markeres `Design` i ROADMAP)
|
||||
**Status:** Done — implementert i `@shade/key-transparency` 0.4.0, frosset i 4.0 GA.
|
||||
**Forfatter:** Shade-teamet
|
||||
**Reviewer-mål:** ekstern crypto-orientert reviewer før produksjons-deploy.
|
||||
**Implementasjons-target:** `@shade/key-transparency` + utvidelser i
|
||||
|
||||
105
docs/files.md
105
docs/files.md
@@ -53,6 +53,111 @@ if (result.kind === 'inline') console.log(result.bytes.byteLength);
|
||||
else for await (const _chunk of /* result.stream */) { /* ... */ }
|
||||
```
|
||||
|
||||
## HTTP RPC — browser-friendly request-response (4.1+)
|
||||
|
||||
The default `shade.files.client(peer)` requires both peers to be mutually
|
||||
addressable over HTTP — the response to a `list`/`read` etc. round-trips
|
||||
through `Shade.deliverControlEnvelope`, which POSTs to the peer's
|
||||
`/v1/transfer/control` endpoint. **That doesn't work for browsers** —
|
||||
a tab can't host an HTTP server, so the server cannot call back outbound.
|
||||
|
||||
`@shade/files` 4.1 ships a parallel **request-response** transport that
|
||||
lets browser-style clients fully consume the file-RPC surface without
|
||||
any inbound channel. It mirrors the way `@shade/server`'s
|
||||
`shade-auth-middleware` works: one POST per RPC, encrypted envelope in
|
||||
the request body, encrypted response in the same HTTP response.
|
||||
|
||||
### Server side — mount the RPC route
|
||||
|
||||
```ts
|
||||
// 1. Register the file handler. `inlineOnly: true` skips the
|
||||
// streams-bridge (which would require @shade/transfer).
|
||||
await shade.files.serve(handlerConfig, { inlineOnly: true });
|
||||
|
||||
// 2. Mount the route on your Hono app under any base path.
|
||||
app.route('/api/v1/shade-files', shade.files.rpcRoute());
|
||||
// ^^^^^^^^^^^^^^
|
||||
// POST <base>/rpc
|
||||
```
|
||||
|
||||
`rpcRoute()` accepts:
|
||||
|
||||
| Option | Default | Purpose |
|
||||
|---------------------|---------|----------------------------------------------------------------------------------------------------|
|
||||
| `maxBodyBytes` | 1 MiB | Max request body. The protocol caps inline payloads at 256 KiB; the headroom is for base64 inflation + custom-op envelopes. |
|
||||
| `acceptFirstMessage`| `false` | Accept `0x01` PreKeyMessage envelopes — required when the RPC route also doubles as the X3DH handshake (browser's first-ever request). |
|
||||
|
||||
### Browser client
|
||||
|
||||
```ts
|
||||
import { createShade } from '@shade/sdk';
|
||||
|
||||
const shade = await createShade({
|
||||
prekeyServer: 'https://shade.example.com',
|
||||
storage: 'memory',
|
||||
address: 'alice@example.com',
|
||||
});
|
||||
|
||||
const fs = shade.files.httpClient('bob@example.com', {
|
||||
rpcUrl: 'https://dispatch.example.com/api/v1/shade-files/rpc',
|
||||
// Optional: thread CSRF / auth tokens, override fetch, etc.
|
||||
headers: { 'X-CSRF-Token': csrfToken },
|
||||
});
|
||||
|
||||
await fs.mkdir('/photos');
|
||||
await fs.write('/photos/cover.png', new Uint8Array([/* ... */]), {
|
||||
contentType: 'image/png',
|
||||
});
|
||||
const result = await fs.read('/photos/cover.png');
|
||||
```
|
||||
|
||||
### What works in HTTP-RPC mode
|
||||
|
||||
- `list`, `stat`, `mkdir`, `delete`, `move`, `getThumbnail`, `custom<K>` — full parity.
|
||||
- `write` — **inline only** (≤ 256 KiB plaintext). Larger inputs throw `ConflictError`.
|
||||
- `read` — **inline only**. If the server returns a streamed `read` result, the client throws `InternalFileError` directing callers to the stateful pathway.
|
||||
|
||||
### Wire contract
|
||||
|
||||
```
|
||||
POST /rpc HTTP/1.1
|
||||
Content-Type: application/octet-stream
|
||||
X-Shade-Sender-Address: alice@example.com
|
||||
|
||||
<wire-encoded ShadeEnvelope (0x01 first-time, 0x02 after) wrapping
|
||||
JSON-encoded RpcRequest>
|
||||
|
||||
────
|
||||
|
||||
200 OK
|
||||
Content-Type: application/octet-stream
|
||||
|
||||
<wire-encoded ShadeEnvelope (0x02) wrapping JSON-encoded RpcResponse | RpcError>
|
||||
```
|
||||
|
||||
Transport-level failures (no session, undecryptable envelope, body too
|
||||
big) return JSON `{ "error": "..." }` with appropriate 4xx status.
|
||||
Application-level failures (file not found, permission denied) ship
|
||||
encrypted `RpcError` envelopes — the client maps them back to typed
|
||||
`FileError` subclasses (`NotFoundError`, `ConflictError`, etc.).
|
||||
|
||||
### Symmetry with `@shade/server`
|
||||
|
||||
The shape mirrors `@shade/server`'s shade-auth-middleware: encrypted
|
||||
envelope rides the request body, server decrypts via the existing
|
||||
ratchet session, performs the protected operation, returns an encrypted
|
||||
envelope in the response. No bidirectional channel required, no
|
||||
WebSocket, no SSE.
|
||||
|
||||
### When to use which
|
||||
|
||||
| Setup | Use |
|
||||
|-----------------------------------------------|-----------------------------------------------|
|
||||
| Browser client ↔ Bun/Hono server | `httpClient()` + `rpcRoute()` |
|
||||
| Server ↔ server (both can host HTTP) | `client()` (default) — supports streams |
|
||||
| Service-worker / extension ↔ server | `httpClient()` (no inbound listener) |
|
||||
| CLI / daemon ↔ daemon | Either; `client()` if you need streams |
|
||||
|
||||
## Op surface
|
||||
|
||||
| Op | Args | Result |
|
||||
|
||||
173
docs/shade-identity-public-key-accessor.md
Normal file
173
docs/shade-identity-public-key-accessor.md
Normal file
@@ -0,0 +1,173 @@
|
||||
# Feature Request — Public accessor for the device's identity public key
|
||||
|
||||
**To**: Shade SDK team
|
||||
**From**: Dispatch (browser-based Shade consumer)
|
||||
**Target**: Shade SDK 4.4.x (or whichever release vehicle fits)
|
||||
**Priority**: medium — unblocks real per-device fingerprint binding at
|
||||
enrollment time; consumers ship with placeholder keys until then
|
||||
|
||||
Thanks for shipping `@shade/storage-indexeddb` so quickly — that unblocked
|
||||
Dispatch's Slice 2.5 (persistent browser identity). During integration we
|
||||
hit one more gap that's worth raising as a separate FR.
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Expose the local device's Ed25519 identity public key as a public accessor
|
||||
on `Shade`, so applications can hand it to their own backend at enrollment
|
||||
time for per-device verification, audit, or peer-fingerprint computation.
|
||||
|
||||
Today the SDK exposes `myAddress` and `fingerprint`, but the underlying
|
||||
identity public key — the cryptographic root that everything else binds
|
||||
to — is reachable only via the private `this.storage.getIdentityKeyPair()`
|
||||
call inside `Shade`. Consumers building enrollment flows have no way to
|
||||
hand the real key over.
|
||||
|
||||
## Problem
|
||||
|
||||
A common pattern in Shade-using apps is:
|
||||
|
||||
1. Browser device generates Shade identity (via `createShade`)
|
||||
2. User enters an enrollment token
|
||||
3. Browser POSTs to its backend: `{ token, address, identityPublicKey, ... }`
|
||||
4. Backend records the device, computes/stores a safety number from the
|
||||
identity key, opens a Shade session
|
||||
|
||||
Step 3 today has nowhere to get a real `identityPublicKey` from. Dispatch
|
||||
currently sends `crypto.getRandomValues(new Uint8Array(32))` formatted as
|
||||
hex, with this comment in the source:
|
||||
|
||||
```ts
|
||||
/**
|
||||
* The browser submits a hex public-key field at enroll time so the schema
|
||||
* stays stable. Wiring this to the SDK-generated identity key requires a
|
||||
* Shade SDK addition (no public accessor for the raw identity key today).
|
||||
*/
|
||||
export function generatePlaceholderIdentityPublicKey(): string { ... }
|
||||
```
|
||||
|
||||
The backend stores this placeholder but cannot verify anything against it,
|
||||
because it's not actually the device's key. Real cryptographic binding is
|
||||
deferred until the SDK exposes the underlying key.
|
||||
|
||||
## Why `fingerprint` isn't sufficient
|
||||
|
||||
`Shade.fingerprint` returns a 12-groups-of-5-digits safety-number string
|
||||
designed for human side-channel comparison. That's the right output for a
|
||||
"compare these digits with your friend" UX, but it's a derived format, not
|
||||
the raw key. Backends that want to:
|
||||
|
||||
- Store the key for later signature verification
|
||||
- Compute their own safety-number representation (Dispatch uses
|
||||
`deterministicSafetyNumber(localAddr, peerAddr)` based on the raw bytes)
|
||||
- Re-derive the fingerprint after an identity rotation
|
||||
|
||||
…all need access to the raw 32-byte Ed25519 public key.
|
||||
|
||||
## Proposed API
|
||||
|
||||
Add a single async accessor on `Shade`:
|
||||
|
||||
```ts
|
||||
class Shade {
|
||||
/**
|
||||
* The local device's Ed25519 identity public key (32 bytes).
|
||||
*
|
||||
* Stable for the lifetime of the identity. After `rotateIdentity()`
|
||||
* this returns the new key; the old key is preserved in retired-
|
||||
* identities storage for the configured grace period.
|
||||
*/
|
||||
get identityPublicKey(): Promise<Uint8Array>;
|
||||
}
|
||||
```
|
||||
|
||||
Internally:
|
||||
|
||||
```ts
|
||||
get identityPublicKey(): Promise<Uint8Array> {
|
||||
if (!this.initialized) throw new Error('Not initialized');
|
||||
return this.storage.getIdentityKeyPair().then((kp) => {
|
||||
if (!kp) throw new Error('Identity not yet generated');
|
||||
return kp.publicKey;
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
If returning a getter that produces a Promise feels off, the equivalent
|
||||
method form is fine:
|
||||
|
||||
```ts
|
||||
async getIdentityPublicKey(): Promise<Uint8Array> { ... }
|
||||
```
|
||||
|
||||
Either shape works for consumers — pick whichever matches existing SDK
|
||||
conventions.
|
||||
|
||||
## Alternative considered: object accessor
|
||||
|
||||
Returning an object would leave room to expose other identity-related
|
||||
fields later without a breaking change:
|
||||
|
||||
```ts
|
||||
get identity(): Promise<{
|
||||
address: string;
|
||||
publicKey: Uint8Array;
|
||||
fingerprint: string;
|
||||
// future: registrationId, createdAt, etc.
|
||||
}>;
|
||||
```
|
||||
|
||||
This would mildly duplicate `myAddress` + `fingerprint`, but consolidates
|
||||
identity-related accessors. Not a blocker for shipping the simpler
|
||||
single-purpose accessor — just flagging the option in case the SDK is
|
||||
considering broader API ergonomics.
|
||||
|
||||
## What this unblocks
|
||||
|
||||
For Dispatch specifically:
|
||||
|
||||
- Real device identity binding at enrollment (replaces the placeholder
|
||||
`crypto.randomUUID()`-derived hex bytes the backend currently stores)
|
||||
- Server-side `computePeerSafetyNumber()` can use the real key instead of
|
||||
the deterministic-from-address stand-in (`shade-identity-provider.ts:151`)
|
||||
- Future signature-based device authentication (sign a challenge with the
|
||||
device's identity key during enrollment) becomes possible without
|
||||
another SDK round
|
||||
|
||||
For other Shade consumers:
|
||||
|
||||
- Any app that hands a device key to its own backend for enrollment —
|
||||
multi-device pairing flows, contact verification UIs, push-notification
|
||||
targeting — gets an actual key to work with
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
1. New accessor exposed on `Shade` (getter or async method, SDK's
|
||||
preference)
|
||||
2. Returns the 32-byte Ed25519 identity public key
|
||||
3. Returns the **current** key after `rotateIdentity()`
|
||||
4. Throws (or resolves to a clear error) when called before
|
||||
`initialize()` completes
|
||||
5. Documented alongside `myAddress` and `fingerprint` in the SDK reference
|
||||
6. One unit test in `shade-sdk` confirming the returned bytes match what
|
||||
`storage.getIdentityKeyPair()` holds
|
||||
7. Mentioned in the changelog under the next release
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **Private key access** — consumers should never need it; signature
|
||||
operations go through `shade.send()` etc. Don't expose the secret half.
|
||||
- **Cross-peer key lookup** — getting *another* peer's identity key is a
|
||||
separate concern (related to peer-verification storage), not what this
|
||||
FR is about. This is strictly the local device's own key.
|
||||
- **Format conversions** — base64/hex/PEM helpers don't belong in the SDK.
|
||||
Consumers can encode the raw bytes however their wire format requires.
|
||||
|
||||
## Why this can't be done in consumer-land
|
||||
|
||||
The identity keypair is generated by `MemoryStorage` / `SQLiteStorage` /
|
||||
`IndexedDBStorage` and consumed by `ShadeSessionManager`. Consumers can't
|
||||
reach into either layer without breaching the SDK's encapsulation. A
|
||||
public accessor is the only path that doesn't require monkey-patching
|
||||
private fields.
|
||||
191
docs/shade-storage-indexeddb.md
Normal file
191
docs/shade-storage-indexeddb.md
Normal file
@@ -0,0 +1,191 @@
|
||||
# Feature Request — `@shade/storage-indexeddb`
|
||||
|
||||
**To**: Shade SDK team
|
||||
**From**: Dispatch (browser-based Shade consumer)
|
||||
**Target**: Shade SDK 4.3.x (or whichever release vehicle fits)
|
||||
**Priority**: blocks all browser-based Shade apps from achieving session
|
||||
persistence across tab refresh
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Ship an official IndexedDB-backed `StorageProvider` adapter as a new
|
||||
workspace package `@shade/storage-indexeddb`, so browser-based Shade SDK
|
||||
consumers can persist identity, prekeys, sessions, and peer-verification
|
||||
state across tab refresh and browser restart — the same way `@shade/storage-sqlite`
|
||||
does for Node and `@shade/storage-postgres` does for server deployments.
|
||||
|
||||
## Problem
|
||||
|
||||
Today the Shade SDK ships three storage paths:
|
||||
|
||||
| spec | adapter | environment |
|
||||
| --------------------------------- | ------------------------ | ---------------- |
|
||||
| `"memory"` | `MemoryStorage` (in-SDK) | tests, ephemeral |
|
||||
| `"sqlite:/path"` | `@shade/storage-sqlite` | Node |
|
||||
| `{ type: 'postgres', url: '…' }` | `@shade/storage-postgres`| Node servers |
|
||||
|
||||
There is **no browser-storage option**. The only way to run Shade in a
|
||||
browser today is `storage: "memory"`, which means:
|
||||
|
||||
- Identity keypair regenerates on every page load
|
||||
- Sessions reset → re-enrollment after every refresh
|
||||
- `getLocalRegistrationId()` returns a fresh value → `device:${id}`
|
||||
address changes → server-side device record orphaned every reload
|
||||
|
||||
This forces every browser-based Shade app to either (a) accept the broken
|
||||
UX, or (b) build their own `StorageProvider` from scratch — duplicating
|
||||
~25 methods × N consumers, with no shared conformance test surface.
|
||||
|
||||
The right place to solve this is at the SDK level, exactly mirroring how
|
||||
SQLite and Postgres are handled.
|
||||
|
||||
## Proposed package
|
||||
|
||||
`packages/shade-storage-indexeddb/` — modeled directly after
|
||||
`packages/shade-storage-sqlite/`. Same package shape, same test layout,
|
||||
same `@shade/core`-only runtime dependency surface.
|
||||
|
||||
### Public API
|
||||
|
||||
```ts
|
||||
// @shade/storage-indexeddb
|
||||
export class IndexedDBStorage implements StorageProvider {
|
||||
/**
|
||||
* Open (or create) the IndexedDB database. Idempotent — repeated calls
|
||||
* with the same dbName return a connection sharing the same object stores.
|
||||
*/
|
||||
static async create(opts?: { dbName?: string }): Promise<IndexedDBStorage>;
|
||||
|
||||
/**
|
||||
* Cleanly close the underlying connection. Future calls will reopen.
|
||||
* Called by Shade.shutdown() when consumers register cleanup.
|
||||
*/
|
||||
async close(): Promise<void>;
|
||||
|
||||
// ─── all StorageProvider methods (identity, prekeys, sessions,
|
||||
// retired identities, peer verifications, optional stream-state) ───
|
||||
}
|
||||
```
|
||||
|
||||
`dbName` defaults to something like `"shade"`. Consumers like Dispatch
|
||||
will pass distinct names per app (`"dispatch-dashboard-shade"`,
|
||||
`"dispatch-host-ui-shade"`) so DevTools' IndexedDB inspector groups them
|
||||
sensibly, even though origin-isolation already makes the data isolated.
|
||||
|
||||
### SDK integration
|
||||
|
||||
`@shade/sdk` `resolveStorage()` gets a fourth branch:
|
||||
|
||||
```ts
|
||||
if (typeof spec === 'object' && spec.type === 'indexeddb') {
|
||||
const moduleId = '@shade/storage-indexeddb';
|
||||
const mod = (await import(moduleId)) as {
|
||||
IndexedDBStorage: { create(opts: { dbName?: string }): Promise<StorageProvider> };
|
||||
};
|
||||
return mod.IndexedDBStorage.create({ dbName: spec.dbName });
|
||||
}
|
||||
```
|
||||
|
||||
Dynamic import keeps `@shade/storage-indexeddb` an optional dependency,
|
||||
matching the Postgres pattern — Node-only consumers don't need to install
|
||||
a browser-only adapter.
|
||||
|
||||
Consumer surface:
|
||||
|
||||
```ts
|
||||
const shade = await createShade({
|
||||
prekeyServer: 'https://…/shade-prekey',
|
||||
storage: { type: 'indexeddb', dbName: 'my-app-shade' },
|
||||
address: 'device:user@example.com', // optional — falls back to device:${registrationId}
|
||||
});
|
||||
```
|
||||
|
||||
## Implementation guidance (non-prescriptive)
|
||||
|
||||
- **IDB wrapper**: suggest `idb` (Jake Archibald's thin wrapper, well-typed,
|
||||
zero deps). Avoid Dexie or idb-keyval — we want full schema control to
|
||||
match the SQL adapters' explicit schemas.
|
||||
- **Object-store layout**: one store per StorageProvider category
|
||||
(`identity`, `signedPreKeys`, `oneTimePreKeys`, `sessions`,
|
||||
`trustedIdentities`, `retiredIdentities`, `peerVerifications`,
|
||||
`streamStates`). Keypaths match the natural keys (`keyId`, `address`,
|
||||
`streamId`).
|
||||
- **Schema version**: integer, bumped on every shape change. Migrations
|
||||
in `db.upgrade(...)` callback. Document schema-history alongside
|
||||
the SQLite schema.
|
||||
- **Concurrency**: IndexedDB transactions are auto-committing — the adapter
|
||||
must keep operations within a single transaction where SQL adapters do.
|
||||
Particular care for `bumpPeerIdentityVersion` (atomic read-modify-write).
|
||||
- **Stream-state methods**: implement them. Browser apps will increasingly
|
||||
use `@shade/transfer` for large file resume, and parity with SQLite's
|
||||
capabilities matters.
|
||||
|
||||
## Test expectations
|
||||
|
||||
Mirror `packages/shade-storage-sqlite/tests/`:
|
||||
|
||||
- `indexeddb-storage.test.ts` — full StorageProvider surface (identity,
|
||||
sessions, trusted identities, retired identities)
|
||||
- `indexeddb-prekey-store.test.ts` — signed + one-time prekey lifecycle
|
||||
- `peer-verifications.test.ts` — verification CRUD + identity-version
|
||||
bumping invariants
|
||||
- `indexeddb-stream-state.test.ts` (if stream-state is implemented)
|
||||
|
||||
Use **`fake-indexeddb`** for the Node test environment — it's the
|
||||
established standard, supports the v3 spec, and lets us run IDB tests
|
||||
in `bun test` / `vitest` / `jest` without a real browser.
|
||||
|
||||
If/when Shade gains a shared `StorageProvider` conformance test suite,
|
||||
this adapter should consume it directly. Until then, follow the SQLite
|
||||
adapter's per-method coverage style.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
1. `@shade/storage-indexeddb` published at version 4.3.0 (or whichever
|
||||
matches the next Shade release)
|
||||
2. `@shade/sdk` `resolveStorage()` resolves `{ type: 'indexeddb', dbName? }`
|
||||
via dynamic import
|
||||
3. Full StorageProvider conformance in tests (identity, prekeys,
|
||||
sessions, retired identities, peer verifications, stream-state)
|
||||
4. Documented in Shade docs alongside SQLite/Postgres adapters
|
||||
5. README example showing browser-app integration
|
||||
6. Bundle-size note: dynamic-imported IDB module shouldn't pull crypto
|
||||
dependencies — adapter should be ≤ ~10 KB minified+gzipped
|
||||
|
||||
## Out of scope (deferred)
|
||||
|
||||
- **Encryption-at-rest** for the IDB contents — separate work item; should
|
||||
match the deviceKey-AES-GCM pattern Shade already uses for `secretEnc`
|
||||
in `PersistedStreamState`. This adapter ships unencrypted-at-rest in v1
|
||||
(consistent with SQLite), with the encryption layer added uniformly to
|
||||
all adapters later.
|
||||
- **Cross-tab BroadcastChannel sync** — IDB is shared across same-origin
|
||||
tabs already; concurrent writes work via IDB transactions. Real-time
|
||||
notification across tabs (e.g. "session was rotated in another tab") is
|
||||
a separate concern, not storage-adapter scope.
|
||||
- **Quota handling** — IDB quota for a Shade keystore is far below realistic
|
||||
browser quotas. If it ever becomes relevant, add a `QuotaExceededError`
|
||||
observability hook then.
|
||||
|
||||
## Why this can't be done in consumer-land
|
||||
|
||||
Building this in Dispatch (or any other consumer) would mean:
|
||||
|
||||
- 25+ method `StorageProvider` re-implementation per consumer
|
||||
- No shared conformance tests
|
||||
- Schema drift across consumers — each would invent its own object-store
|
||||
shape, blocking any future cross-app data import/export
|
||||
- Every Shade SDK update that adjusts `StorageProvider` would force every
|
||||
consumer to track and patch independently
|
||||
|
||||
`@shade/storage-sqlite` and `@shade/storage-postgres` are part of the SDK
|
||||
for the same reason. IndexedDB belongs alongside them.
|
||||
|
||||
## What unblocks
|
||||
|
||||
Shipping this unblocks Dispatch's "Slice 2.5" — persistent enrollment
|
||||
across browser refresh, which today is the largest QA-friction point in
|
||||
the dev loop. Any future browser-Shade consumer (web dashboard,
|
||||
contact-list app, browser-extension messenger) gets persistence for free.
|
||||
128
docs/streaming-sessions.md
Normal file
128
docs/streaming-sessions.md
Normal file
@@ -0,0 +1,128 @@
|
||||
# Streaming Double-Ratchet sub-sessions (V4.11)
|
||||
|
||||
`ShadeStream` wraps individual frames on a long-lived, high-frequency,
|
||||
often one-directional channel (e.g. a server→client console-log
|
||||
WebSocket) in an **independent** Double Ratchet derived from — but never
|
||||
mutating — an already-established parent Shade session.
|
||||
|
||||
This is the answer to Vyvern FR `shade-ws-streaming-ratchet.md`. It is a
|
||||
first-class API, *not* the "documented contract that `send`/`receive` is
|
||||
safe per-frame" fallback: the Double-Ratchet crypto was already safe for
|
||||
that access pattern, but the `send`/`receive` wrapper layer was not
|
||||
(per-frame keystore writes; a shared per-peer mutex and a single stored
|
||||
session row coupling the stream to the HTTP path). `ShadeStream` keeps
|
||||
the proven ratchet and fixes the wrapper.
|
||||
|
||||
## API
|
||||
|
||||
Transport-agnostic, exactly like `send`/`receive`: it emits/consumes
|
||||
wire bytes; you own the WebSocket.
|
||||
|
||||
```ts
|
||||
// Initiator (the side that calls openStream)
|
||||
const stream = await shade.openStream(peerAddr);
|
||||
ws.send(stream.handshakeFrame()); // → STREAM_OPEN
|
||||
// first inbound WS frame is the peer's STREAM_OPEN_ACK:
|
||||
await stream.handleHandshake(ackBytes); // stream now usable
|
||||
ws.send(await stream.seal(utf8(logLine))); // outbound frame
|
||||
onLog(await stream.open(inboundBytes)); // inbound frame
|
||||
await stream.close(); // on ws close/error
|
||||
|
||||
// Responder
|
||||
const stream = await shade.acceptStream(peerAddr, openBytes); // usable now
|
||||
ws.send(stream.handshakeFrame()); // → STREAM_OPEN_ACK
|
||||
// open()/seal() as above
|
||||
```
|
||||
|
||||
Route inbound bytes with `inspectEnvelopeType()`:
|
||||
`'stream-open' | 'stream-open-ack' | 'stream-frame'`.
|
||||
|
||||
## Seeding (no prekey-server round trip)
|
||||
|
||||
The stream root key is derived from an identity-bound **3-DH** exchange
|
||||
— the X3DH pattern minus signed/one-time prekeys, because the peer's
|
||||
identity is *already* mutually pinned by the parent session's TOFU. Two
|
||||
ephemerals are exchanged inside the transport (`STREAM_OPEN` /
|
||||
`STREAM_OPEN_ACK`); no prekey server is involved.
|
||||
|
||||
```
|
||||
slotA = DH(initiatorEphemeral, responderIdentity) — authenticates responder
|
||||
slotB = DH(initiatorIdentity, responderEphemeral) — authenticates initiator
|
||||
slotC = DH(initiatorEphemeral, responderEphemeral) — ephemeral forward secrecy
|
||||
SK = HKDF(ikm = slotA‖slotB‖slotC, salt = streamId, info = "ShadeStream/v1")
|
||||
```
|
||||
|
||||
Both peers compute the identical three scalars regardless of role.
|
||||
`SK` then bootstraps a textbook Double Ratchet by handing the
|
||||
responder's ephemeral to `initSenderSession`/`initReceiverSession`
|
||||
exactly the way X3DH hands its signed prekey to the ratchet — so
|
||||
`ratchetEncrypt`/`ratchetDecrypt` and every guarantee they carry apply
|
||||
unchanged.
|
||||
|
||||
## Security contract (answers FR R1–R7)
|
||||
|
||||
- **R1 — same properties as `send`/`receive`.** Each frame is one
|
||||
`ratchetEncrypt`/`ratchetDecrypt` over the *same* crypto as the HTTP
|
||||
path: AES-256-GCM confidentiality, per-frame forward secrecy via the
|
||||
one-way HMAC chain-key KDF with in-place zeroize of the spent chain
|
||||
key, and replay/rewind rejection (a re-delivered or counter-rewound
|
||||
frame fails closed). The handshake is mutually authenticated against
|
||||
the identities the parent session already pinned.
|
||||
- **R2 — one-directional resilience.** A long server→client burst with
|
||||
no client traffic only advances the symmetric sending chain (no DH
|
||||
step until the peer replies — standard Double Ratchet). Forward
|
||||
secrecy holds per frame in this regime. Over an ordered transport
|
||||
(WebSocket/TCP) zero keys are skipped per frame.
|
||||
- **R3 — bounded memory.** Out-of-order arrivals are capped by the
|
||||
ratchet's `MAX_SKIP` (1000) and `MAX_CACHED_SKIPPED_KEYS` (2000)
|
||||
with oldest-key eviction. In-order delivery retains nothing. Verified
|
||||
to stay at zero retained keys across a 5000-frame burst.
|
||||
- **R4 — browser parity.** Identical API and guarantees in the browser
|
||||
SDK: `ShadeStream` is on the same `Shade` class over the same
|
||||
`CryptoProvider` (`SubtleCryptoProvider`), so the IndexedDB-backed
|
||||
build behaves identically to the `sqlite:` server build. No storage
|
||||
is touched at all (see R7), so the keystore backend is irrelevant.
|
||||
- **R5 — independent lifecycle.** The stream ratchet is derived without
|
||||
reading or writing the stored parent `SessionState`, runs on its own
|
||||
private op-mutex (not the per-peer `send`/`receive` queues), and is
|
||||
zeroized on `close()`. Opening, using for thousands of frames, and
|
||||
closing a stream leaves the parent session byte-identical; the HTTP
|
||||
path keeps working concurrently against the same peer. Each
|
||||
`openStream` gets a fresh `streamId` and an independent root, so
|
||||
concurrent streams to one peer never share key material.
|
||||
- **R6 — wire framing.** `@shade/proto` defines `STREAM_OPEN` (0x31),
|
||||
`STREAM_OPEN_ACK` (0x32), `STREAM_FRAME` (0x33). A `STREAM_FRAME`
|
||||
carries one Double-Ratchet message via the exact ratchet inner codec
|
||||
the HTTP path uses. One sealed logical frame ⇒ one self-delimiting
|
||||
wire frame ⇒ one WS text/binary frame.
|
||||
- **R7 — performance.** The ratchet lives **only in memory and is never
|
||||
persisted**. There is therefore *zero* per-frame storage I/O — the
|
||||
per-frame cost is exactly the symmetric KDF + one AES-GCM, the same
|
||||
primitives the HTTP path runs. This is strictly better than the
|
||||
"doubled CPU" the Vyvern roadmap budgeted, because the dominant cost
|
||||
the naive `send`/`receive`-per-frame approach would have paid (a
|
||||
`saveSession` keystore write per frame) is eliminated, not doubled.
|
||||
Not persisting is also a *security* property, not a shortcut: writing
|
||||
evolving per-frame ratchet secrets to disk would defeat forward
|
||||
secrecy. A dropped/reconnected stream is re-opened with a fresh
|
||||
handshake, never resumed.
|
||||
|
||||
## Double-Ratchet ordering note
|
||||
|
||||
A responder cannot `seal()` until it has `open()`ed at least one frame
|
||||
from the initiator (standard Signal behaviour — the responder has no
|
||||
sending chain until the first DH step). For a server-heavy stream
|
||||
either make the bursty data sender the **initiator**, or have the
|
||||
initiator send one priming frame immediately after the handshake.
|
||||
|
||||
## Tests
|
||||
|
||||
- `packages/shade-core/tests/stream.test.ts` — handshake agreement,
|
||||
frame round-trips, 5000-frame one-directional burst (bounded skipped
|
||||
keys + forward-secrecy zeroize), parent-session independence (R5),
|
||||
replay/rewind rejection, mutual authentication against pinned
|
||||
identities, `close()` zeroize/idempotence.
|
||||
- `packages/shade-proto/tests/stream-wire.test.ts` — wire round-trips
|
||||
and type-tag/length rejection for all three stream frame kinds.
|
||||
</content>
|
||||
</invoke>
|
||||
@@ -16,8 +16,10 @@
|
||||
"version": "bun run scripts/bump-version.ts",
|
||||
"soak": "bun run scripts/soak.ts",
|
||||
"soak:smoke": "bun run scripts/soak.ts --hours 0.05 --pairs 4",
|
||||
"publish:dry": "DRY_RUN=1 bun run scripts/publish-all.ts",
|
||||
"publish:all": "bash scripts/publish-shade.sh",
|
||||
"typecheck": "bun run scripts/typecheck-all.ts",
|
||||
"prepublish:check": "bun run typecheck",
|
||||
"publish:dry": "bun run prepublish:check && DRY_RUN=1 bun run scripts/publish-all.ts",
|
||||
"publish:all": "bun run prepublish:check && bash scripts/publish-shade.sh",
|
||||
"build:docker": "bun run scripts/build-docker.ts",
|
||||
"publish:docker": "bun run scripts/build-docker.ts -- --push"
|
||||
},
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shade/cli",
|
||||
"version": "4.0.0",
|
||||
"version": "4.11.0",
|
||||
"type": "module",
|
||||
"main": "src/cli.ts",
|
||||
"bin": {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shade/core",
|
||||
"version": "4.0.0",
|
||||
"version": "4.11.0",
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
"types": "src/index.ts",
|
||||
|
||||
@@ -107,6 +107,31 @@ export class FingerprintNotVerifiedError extends ShadeError {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Thrown when `seal()` / `open()` is called on a {@link StreamRatchet}
|
||||
* that has already been torn down via `close()`. The stream's ratchet
|
||||
* secrets have been zeroized and cannot be revived — open a fresh
|
||||
* stream instead.
|
||||
*/
|
||||
export class StreamClosedError extends ShadeError {
|
||||
constructor(message = 'Stream is closed') {
|
||||
super('SHADE_STREAM_CLOSED', message);
|
||||
this.name = 'StreamClosedError';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Thrown when a stream handshake frame is malformed, arrives in the
|
||||
* wrong order, or references a streamId that does not match the stream
|
||||
* it was fed to.
|
||||
*/
|
||||
export class StreamHandshakeError extends ShadeError {
|
||||
constructor(message = 'Stream handshake failed') {
|
||||
super('SHADE_STREAM_HANDSHAKE', message);
|
||||
this.name = 'StreamHandshakeError';
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Infrastructure Errors ───────────────────────────────────
|
||||
|
||||
export class NetworkError extends ShadeError {
|
||||
@@ -158,6 +183,30 @@ export class UnauthorizedError extends ShadeError {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 409 Conflict — caller wrote to a resource that already exists without
|
||||
* supplying an If-Match precondition. V4.9: the encrypted blob primitive
|
||||
* uses this to force read-then-write on already-occupied slots.
|
||||
*/
|
||||
export class ConflictError extends ShadeError {
|
||||
constructor(message = 'Conflict') {
|
||||
super('SHADE_CONFLICT', message);
|
||||
this.name = 'ConflictError';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 412 Precondition Failed — caller supplied an If-Match etag that does
|
||||
* not match the current state. V4.9: the encrypted blob primitive uses
|
||||
* this to surface stale-CAS so clients can re-read, merge, and retry.
|
||||
*/
|
||||
export class PreconditionFailedError extends ShadeError {
|
||||
constructor(message = 'Precondition failed') {
|
||||
super('SHADE_PRECONDITION_FAILED', message);
|
||||
this.name = 'PreconditionFailedError';
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Error → HTTP Status Mapping ────────────────────────────
|
||||
|
||||
/**
|
||||
@@ -180,7 +229,10 @@ export function errorToHttpStatus(error: unknown): number {
|
||||
return 400;
|
||||
case 'SHADE_REPLAY':
|
||||
case 'SHADE_DUPLICATE_MESSAGE':
|
||||
case 'SHADE_CONFLICT':
|
||||
return 409;
|
||||
case 'SHADE_PRECONDITION_FAILED':
|
||||
return 412;
|
||||
case 'SHADE_RATE_LIMIT':
|
||||
return 429;
|
||||
case 'SHADE_TIMEOUT':
|
||||
|
||||
@@ -26,6 +26,8 @@ export interface ShadeEventMap {
|
||||
'identity.rotated': { newFingerprint: string };
|
||||
'session.created': { address: string; remoteIdentityKeyHash: string };
|
||||
'session.removed': { address: string };
|
||||
/** V4.8.3 — emitted when `aliasSession` moves a peer's per-peer state. */
|
||||
'session.aliased': { oldLabel: string; newLabel: string };
|
||||
'message.encrypted': { address: string; counter: number; ciphertextSize: number };
|
||||
'message.decrypted': { address: string; counter: number; plaintextSize: number };
|
||||
'ratchet.dh_step': { address: string };
|
||||
@@ -34,6 +36,10 @@ export interface ShadeEventMap {
|
||||
'signed_prekey.rotated': { oldKeyId: number; newKeyId: number };
|
||||
'trust.pinned': { address: string; identityKeyHash: string };
|
||||
'trust.changed': { address: string; oldKeyHash: string; newKeyHash: string };
|
||||
/** V4.11 — a streaming sub-ratchet handshake completed. */
|
||||
'stream.opened': { address: string; role: 'initiator' | 'responder' };
|
||||
/** V4.11 — a streaming sub-ratchet was torn down and zeroized. */
|
||||
'stream.closed': { address: string };
|
||||
}
|
||||
|
||||
export type ShadeEventName = keyof ShadeEventMap;
|
||||
|
||||
@@ -5,6 +5,7 @@ export * from './keys.js';
|
||||
export * from './errors.js';
|
||||
export * from './x3dh.js';
|
||||
export * from './ratchet.js';
|
||||
export * from './stream.js';
|
||||
export { ShadeSessionManager, GRACE_PERIOD_MS } from './session.js';
|
||||
export * from './serialization.js';
|
||||
export * from './fingerprint.js';
|
||||
|
||||
@@ -185,6 +185,22 @@ export async function ratchetDecrypt(
|
||||
if (!session.receiveChain) {
|
||||
throw new DecryptionError('No receiving chain available');
|
||||
}
|
||||
// Defense-in-depth: a same-DH message whose counter is already
|
||||
// behind the chain — and that did NOT match a cached skipped key —
|
||||
// is either a duplicate we already decrypted (skipped key was
|
||||
// consumed) or one whose key was evicted under cache pressure.
|
||||
// Falling through would call kdfChainKey on the *current* (ahead)
|
||||
// chainKey and then rewind `chain.counter = message.counter + 1`,
|
||||
// permanently desyncing the chain so every subsequent decrypt
|
||||
// returns wrong-key. Reject without mutating state instead.
|
||||
if (
|
||||
!isNewRatchet &&
|
||||
message.counter < session.receiveChain.counter
|
||||
) {
|
||||
throw new DecryptionError(
|
||||
'Failed to decrypt message — wrong key or tampered data',
|
||||
);
|
||||
}
|
||||
await skipMessageKeys(crypto, session, message.dhPublicKey, session.receiveChain, message.counter);
|
||||
|
||||
// Advance the receiving chain one more step to get this message's key
|
||||
|
||||
@@ -23,7 +23,14 @@ import {
|
||||
ratchetEncrypt,
|
||||
ratchetDecrypt,
|
||||
} from './ratchet.js';
|
||||
import { NoSessionError } from './errors.js';
|
||||
import {
|
||||
deriveStreamRootKey,
|
||||
bootstrapStreamSession,
|
||||
StreamRatchet,
|
||||
STREAM_ID_BYTES,
|
||||
STREAM_EPHEMERAL_BYTES,
|
||||
} from './stream.js';
|
||||
import { NoSessionError, StreamHandshakeError } from './errors.js';
|
||||
import { computeFingerprint, shortFingerprint } from './fingerprint.js';
|
||||
import { ShadeEventEmitter, shortHash } from './events.js';
|
||||
import {
|
||||
@@ -265,6 +272,73 @@ export class ShadeSessionManager {
|
||||
// Note: we keep the trusted identity; new session will verify against it.
|
||||
}
|
||||
|
||||
/**
|
||||
* Move every per-peer storage row for `oldLabel` (session, trusted
|
||||
* identity, peer-verification, identity-version counter) to
|
||||
* `newLabel`. Used to canonicalize sessions when first-contact
|
||||
* forces the receiver to label by sender-fingerprint hint
|
||||
* (`fp:<hex>`) and a later in-band announcement reveals the peer's
|
||||
* canonical address.
|
||||
*
|
||||
* Holds the per-peer mutex on **both** labels for the whole
|
||||
* migration so concurrent encrypt/decrypt for either label can't
|
||||
* observe a half-moved state. Locks are taken in lexicographic
|
||||
* order to avoid deadlocks if two callers alias in opposite
|
||||
* directions.
|
||||
*
|
||||
* Throws if no session exists for `oldLabel`. Throws (refuses to
|
||||
* overwrite) if a session already exists for `newLabel`. No-ops
|
||||
* when `oldLabel === newLabel`.
|
||||
*
|
||||
* V4.8.3 — Prism FR `session-label-asymmetry-v4.8.2.md`.
|
||||
*/
|
||||
async aliasSession(oldLabel: string, newLabel: string): Promise<void> {
|
||||
if (oldLabel === newLabel) return;
|
||||
const [first, second] = oldLabel < newLabel ? [oldLabel, newLabel] : [newLabel, oldLabel];
|
||||
await this.runUnderPeerLock(first, () =>
|
||||
this.runUnderPeerLock(second, () => this.aliasUnderLock(oldLabel, newLabel)),
|
||||
);
|
||||
}
|
||||
|
||||
private async aliasUnderLock(oldLabel: string, newLabel: string): Promise<void> {
|
||||
const session = await this.storage.getSession(oldLabel);
|
||||
if (!session) throw new NoSessionError(oldLabel);
|
||||
const collision = await this.storage.getSession(newLabel);
|
||||
if (collision) {
|
||||
throw new Error(
|
||||
`aliasSession: refusing to overwrite an existing session for "${newLabel}". ` +
|
||||
`If you want to replace it, call resetSession("${newLabel}") first.`,
|
||||
);
|
||||
}
|
||||
// Move the session.
|
||||
await this.storage.saveSession(newLabel, session);
|
||||
// Re-pin trust under the new label using the session's stored DH
|
||||
// identity key — `saveTrustedIdentity` is the same primitive that
|
||||
// the X3DH initiator/responder uses, and the DH key in `session`
|
||||
// is the value that was pinned at session-establish time. The old
|
||||
// pin under `oldLabel` is harmless leftover (the storage interface
|
||||
// has no remove for trust pins) and would only be re-checked if a
|
||||
// fresh X3DH against `oldLabel` somehow happened later.
|
||||
await this.storage.saveTrustedIdentity(newLabel, session.remoteIdentityKey);
|
||||
// Migrate the peer-verification record if present.
|
||||
const verification = await this.storage.getPeerVerification(oldLabel);
|
||||
if (verification) {
|
||||
await this.storage.savePeerVerification({
|
||||
...verification,
|
||||
peerAddress: newLabel,
|
||||
});
|
||||
await this.storage.removePeerVerification(oldLabel);
|
||||
}
|
||||
// Carry the identity-version counter forward so peer rotation
|
||||
// history is preserved.
|
||||
const oldVersion = await this.storage.getPeerIdentityVersion(oldLabel);
|
||||
for (let i = 1; i < oldVersion; i++) {
|
||||
await this.storage.bumpPeerIdentityVersion(newLabel);
|
||||
}
|
||||
await this.storage.removeSession(oldLabel);
|
||||
this.events?.emit('session.aliased', { oldLabel, newLabel });
|
||||
}
|
||||
|
||||
/**
|
||||
* Accept a changed remote identity. This should only be called after
|
||||
* verifying the new identity out-of-band (e.g., comparing fingerprints).
|
||||
@@ -559,6 +633,121 @@ export class ShadeSessionManager {
|
||||
|
||||
return dec.decode(plaintext);
|
||||
}
|
||||
|
||||
// ─── Streaming sub-sessions (V4.11) ────────────────────────
|
||||
|
||||
/**
|
||||
* Resolve the peer's pinned identity X25519 key for a stream
|
||||
* handshake. Requires an *already established* parent session — the
|
||||
* stream is explicitly a "second channel on a known peer", never a
|
||||
* first contact (so it needs no prekey-server round trip and inherits
|
||||
* the parent's TOFU pin).
|
||||
*/
|
||||
private async streamIdentityMaterial(
|
||||
address: string,
|
||||
): Promise<{ selfIdentityDHPriv: Uint8Array; peerIdentityDHPub: Uint8Array }> {
|
||||
if (!this.identity) throw new Error('Not initialized');
|
||||
const session = await this.storage.getSession(address);
|
||||
if (!session) throw new NoSessionError(address);
|
||||
return {
|
||||
selfIdentityDHPriv: this.identity.dhPrivateKey,
|
||||
peerIdentityDHPub: session.remoteIdentityKey,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Initiator side of a stream handshake. Generates the streamId and
|
||||
* this side's ephemeral, and returns a `complete` continuation that
|
||||
* derives the sub-ratchet once the responder's ephemeral arrives in
|
||||
* the `STREAM_OPEN_ACK`.
|
||||
*
|
||||
* Touches neither the stored parent session nor the per-peer op
|
||||
* queues (R5).
|
||||
*/
|
||||
async beginStream(address: string): Promise<{
|
||||
streamId: Uint8Array;
|
||||
ephemeralPublicKey: Uint8Array;
|
||||
complete: (peerEphemeralPub: Uint8Array) => Promise<StreamRatchet>;
|
||||
}> {
|
||||
const { selfIdentityDHPriv, peerIdentityDHPub } =
|
||||
await this.streamIdentityMaterial(address);
|
||||
const streamId = this.crypto.randomBytes(STREAM_ID_BYTES);
|
||||
const ephemeral = await this.crypto.generateX25519KeyPair();
|
||||
|
||||
const complete = async (peerEphemeralPub: Uint8Array): Promise<StreamRatchet> => {
|
||||
if (peerEphemeralPub.length !== STREAM_EPHEMERAL_BYTES) {
|
||||
throw new StreamHandshakeError(
|
||||
`responder ephemeral must be ${STREAM_EPHEMERAL_BYTES} bytes`,
|
||||
);
|
||||
}
|
||||
const sk = await deriveStreamRootKey(
|
||||
this.crypto,
|
||||
'initiator',
|
||||
streamId,
|
||||
selfIdentityDHPriv,
|
||||
peerIdentityDHPub,
|
||||
ephemeral.privateKey,
|
||||
peerEphemeralPub,
|
||||
);
|
||||
const session = await bootstrapStreamSession(this.crypto, 'initiator', sk, peerIdentityDHPub, {
|
||||
publicKey: peerEphemeralPub,
|
||||
privateKey: new Uint8Array(0),
|
||||
});
|
||||
this.crypto.zeroize(sk);
|
||||
this.crypto.zeroize(ephemeral.privateKey);
|
||||
this.events?.emit('stream.opened', { address, role: 'initiator' });
|
||||
return new StreamRatchet(this.crypto, session, streamId);
|
||||
};
|
||||
|
||||
return { streamId, ephemeralPublicKey: ephemeral.publicKey, complete };
|
||||
}
|
||||
|
||||
/**
|
||||
* Responder side of a stream handshake. Given the initiator's
|
||||
* `STREAM_OPEN` (its streamId + ephemeral), derives the sub-ratchet
|
||||
* immediately and returns this side's ephemeral for the
|
||||
* `STREAM_OPEN_ACK`.
|
||||
*/
|
||||
async acceptStream(
|
||||
address: string,
|
||||
streamId: Uint8Array,
|
||||
initiatorEphemeralPub: Uint8Array,
|
||||
): Promise<{ ephemeralPublicKey: Uint8Array; stream: StreamRatchet }> {
|
||||
if (streamId.length !== STREAM_ID_BYTES) {
|
||||
throw new StreamHandshakeError(`streamId must be ${STREAM_ID_BYTES} bytes`);
|
||||
}
|
||||
if (initiatorEphemeralPub.length !== STREAM_EPHEMERAL_BYTES) {
|
||||
throw new StreamHandshakeError(
|
||||
`initiator ephemeral must be ${STREAM_EPHEMERAL_BYTES} bytes`,
|
||||
);
|
||||
}
|
||||
const { selfIdentityDHPriv, peerIdentityDHPub } =
|
||||
await this.streamIdentityMaterial(address);
|
||||
const ephemeral = await this.crypto.generateX25519KeyPair();
|
||||
|
||||
const sk = await deriveStreamRootKey(
|
||||
this.crypto,
|
||||
'responder',
|
||||
streamId,
|
||||
selfIdentityDHPriv,
|
||||
peerIdentityDHPub,
|
||||
ephemeral.privateKey,
|
||||
initiatorEphemeralPub,
|
||||
);
|
||||
const session = await bootstrapStreamSession(
|
||||
this.crypto,
|
||||
'responder',
|
||||
sk,
|
||||
peerIdentityDHPub,
|
||||
ephemeral,
|
||||
);
|
||||
this.crypto.zeroize(sk);
|
||||
this.events?.emit('stream.opened', { address, role: 'responder' });
|
||||
return {
|
||||
ephemeralPublicKey: ephemeral.publicKey,
|
||||
stream: new StreamRatchet(this.crypto, session, streamId),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function arraysEqual(a: Uint8Array, b: Uint8Array): boolean {
|
||||
|
||||
@@ -46,6 +46,50 @@ export interface PersistedStreamState {
|
||||
*/
|
||||
export type PeerVerificationSource = 'user' | 'transitive' | 'tofu-after-warning';
|
||||
|
||||
/**
|
||||
* Persisted broadcast-channel state (V4.6). Holds the sender-key chain that
|
||||
* lets the channel owner encrypt a single ciphertext and fan it out to all
|
||||
* paired members.
|
||||
*
|
||||
* - `ownerRole === 'sender'`: this device created the channel; the record
|
||||
* carries the full chain (chainKey + iteration + signing keypair).
|
||||
* - `ownerRole === 'receiver'`: this device joined a channel that
|
||||
* `ownerAddress` owns; we hold a tracking copy of the chain (chainKey +
|
||||
* iteration + signing public key only — no private signing key).
|
||||
*
|
||||
* `generation` is bumped each time `removeMember` rotates the chain. Stale
|
||||
* broadcasts at lower generations are silently dropped on receive.
|
||||
*/
|
||||
export interface BroadcastChannelRecord {
|
||||
/** Opaque, stable across restarts. UUID-style. */
|
||||
channelId: string;
|
||||
ownerRole: 'sender' | 'receiver';
|
||||
/** The address that owns the channel (i.e. the only sender). */
|
||||
ownerAddress: string;
|
||||
label?: string;
|
||||
/** Sender-key generation — bumped on rotation. Starts at 0. */
|
||||
generation: number;
|
||||
/** Current chain key (32 bytes). */
|
||||
chainKey: Uint8Array;
|
||||
/** Counter advanced by `senderKeyEncrypt` / `senderKeyDecrypt`. */
|
||||
iteration: number;
|
||||
/** Owner's Ed25519 signing public key (32 bytes). */
|
||||
signingPublicKey: Uint8Array;
|
||||
/** Owner's Ed25519 signing private key. Present iff `ownerRole === 'sender'`. */
|
||||
signingPrivateKey?: Uint8Array;
|
||||
createdAt: number;
|
||||
updatedAt: number;
|
||||
}
|
||||
|
||||
/** Membership row for a broadcast channel (sender-side only). */
|
||||
export interface BroadcastMemberRecord {
|
||||
channelId: string;
|
||||
peerAddress: string;
|
||||
joinedAt: number;
|
||||
/** When this peer was revoked. `null` while still active. */
|
||||
removedAt: number | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Persistent record that a peer's safety number was verified at a point
|
||||
* in time. `identityVersion` is the local counter for that peer's identity:
|
||||
@@ -188,4 +232,35 @@ export interface StorageProvider {
|
||||
|
||||
/** Prune stream-state rows in `'finished' | 'aborted'` status older than `olderThan`. */
|
||||
pruneStreamStates?(olderThan: number): Promise<void>;
|
||||
|
||||
// ─── Broadcast channels (V4.6) — optional ─────────────────
|
||||
|
||||
/**
|
||||
* Persist or replace the broadcast-channel record. Idempotent upsert on
|
||||
* `channelId`. Backends that don't implement broadcast-channel support
|
||||
* may omit this; the SDK throws a clear error when the app tries to
|
||||
* call `createBroadcastChannel` on such a backend.
|
||||
*/
|
||||
saveBroadcastChannel?(channel: BroadcastChannelRecord): Promise<void>;
|
||||
|
||||
/** Look up a broadcast channel by id. */
|
||||
getBroadcastChannel?(channelId: string): Promise<BroadcastChannelRecord | null>;
|
||||
|
||||
/** Enumerate all broadcast channels persisted on this device. */
|
||||
listBroadcastChannels?(): Promise<BroadcastChannelRecord[]>;
|
||||
|
||||
/** Drop a broadcast channel and all its membership rows. */
|
||||
removeBroadcastChannel?(channelId: string): Promise<void>;
|
||||
|
||||
/** Persist or replace a broadcast-membership row (sender-side only). */
|
||||
saveBroadcastMember?(member: BroadcastMemberRecord): Promise<void>;
|
||||
|
||||
/**
|
||||
* List membership rows for a channel. Includes revoked members (with
|
||||
* `removedAt !== null`); callers filter as needed.
|
||||
*/
|
||||
getBroadcastMembers?(channelId: string): Promise<BroadcastMemberRecord[]>;
|
||||
|
||||
/** Hard-delete a single membership row. */
|
||||
removeBroadcastMember?(channelId: string, peerAddress: string): Promise<void>;
|
||||
}
|
||||
|
||||
233
packages/shade-core/src/stream.ts
Normal file
233
packages/shade-core/src/stream.ts
Normal file
@@ -0,0 +1,233 @@
|
||||
import type { CryptoProvider } from './crypto.js';
|
||||
import type { KeyPair, RatchetMessage, SessionState } from './types.js';
|
||||
import {
|
||||
initSenderSession,
|
||||
initReceiverSession,
|
||||
ratchetEncrypt,
|
||||
ratchetDecrypt,
|
||||
} from './ratchet.js';
|
||||
import { StreamClosedError } from './errors.js';
|
||||
|
||||
/**
|
||||
* Streaming Double-Ratchet sub-sessions (V4.11).
|
||||
*
|
||||
* Wraps a long-lived, high-frequency, often-one-directional channel
|
||||
* (e.g. a server→client WebSocket log burst) in an *independent* Double
|
||||
* Ratchet that is derived from — but never mutates — an already
|
||||
* established parent Shade session.
|
||||
*
|
||||
* Why a sub-ratchet rather than reusing `ShadeSessionManager`:
|
||||
*
|
||||
* - **Independence (R5).** A stream gets its own root key, chains, DH
|
||||
* ratchet and op-mutex. Opening/closing it never touches the stored
|
||||
* parent `SessionState` nor serialises against the HTTP send/receive
|
||||
* queue.
|
||||
* - **Performance (R7).** The stream ratchet lives only in memory and
|
||||
* is *never* written to the keystore. There is therefore zero
|
||||
* per-frame storage I/O — the cost is purely the symmetric KDF +
|
||||
* AES-GCM, the same primitives the HTTP path uses.
|
||||
* - **Forward secrecy.** Not persisting the evolving ratchet state is
|
||||
* a feature, not a shortcut: writing per-frame secrets to disk would
|
||||
* actively defeat the forward-secrecy guarantee. A dropped/reconnected
|
||||
* stream is re-opened with a fresh handshake, not resumed.
|
||||
*
|
||||
* ## Seeding (no prekey-server round trip)
|
||||
*
|
||||
* The stream root key is derived from an identity-bound 3-DH exchange —
|
||||
* the X3DH pattern minus the signed / one-time prekeys, because the
|
||||
* peer's identity is *already* mutually pinned by the parent session's
|
||||
* TOFU. Two ephemeral keys are exchanged inside the transport itself
|
||||
* (`STREAM_OPEN` / `STREAM_OPEN_ACK`); no prekey server is involved.
|
||||
*
|
||||
* slotA = DH(initiatorEphemeral, responderIdentity) — auth of responder
|
||||
* slotB = DH(initiatorIdentity, responderEphemeral) — auth of initiator
|
||||
* slotC = DH(initiatorEphemeral, responderEphemeral) — ephemeral FS
|
||||
*
|
||||
* SK = HKDF(ikm = slotA‖slotB‖slotC, salt = streamId, info = "ShadeStream/v1")
|
||||
*
|
||||
* Both peers compute the identical three scalars regardless of role, so
|
||||
* `SK` agrees. An attacker lacking the responder's identity private key
|
||||
* cannot form slotA; one lacking the initiator's cannot form slotB —
|
||||
* the handshake is therefore mutually authenticated against the same
|
||||
* identities the parent session already trusts.
|
||||
*
|
||||
* `SK` then bootstraps a textbook Double Ratchet by handing the
|
||||
* responder's ephemeral to {@link initSenderSession} /
|
||||
* {@link initReceiverSession} exactly the way X3DH hands its signed
|
||||
* prekey to the ratchet — so `ratchetEncrypt` / `ratchetDecrypt` (and
|
||||
* thus every R1–R3 guarantee they already carry) apply unchanged.
|
||||
*/
|
||||
|
||||
export type StreamRole = 'initiator' | 'responder';
|
||||
|
||||
/** Stream identifier length (bytes). 128 bits of collision resistance. */
|
||||
export const STREAM_ID_BYTES = 16;
|
||||
|
||||
/** Ephemeral X25519 public-key length carried in the handshake. */
|
||||
export const STREAM_EPHEMERAL_BYTES = 32;
|
||||
|
||||
const STREAM_KDF_INFO = new TextEncoder().encode('ShadeStream/v1');
|
||||
|
||||
/**
|
||||
* Derive the stream's independent root key from the identity-bound 3-DH
|
||||
* exchange. Pure: never reads or mutates any `SessionState`.
|
||||
*
|
||||
* @param role which end of the handshake we are
|
||||
* @param streamId 16-byte stream id (HKDF salt; binds the
|
||||
* derivation so two concurrent streams to the
|
||||
* same peer never share a root key)
|
||||
* @param selfIdentityDHPriv our long-term identity X25519 private key
|
||||
* @param peerIdentityDHPub peer's pinned identity X25519 public key
|
||||
* (the value the parent session pinned)
|
||||
* @param selfEphemeralPriv our per-stream ephemeral X25519 private key
|
||||
* @param peerEphemeralPub peer's per-stream ephemeral X25519 public key
|
||||
*/
|
||||
export async function deriveStreamRootKey(
|
||||
crypto: CryptoProvider,
|
||||
role: StreamRole,
|
||||
streamId: Uint8Array,
|
||||
selfIdentityDHPriv: Uint8Array,
|
||||
peerIdentityDHPub: Uint8Array,
|
||||
selfEphemeralPriv: Uint8Array,
|
||||
peerEphemeralPub: Uint8Array,
|
||||
): Promise<Uint8Array> {
|
||||
// Each slot is pinned to a fixed semantic (not to local role) so both
|
||||
// sides feed HKDF the identical ikm:
|
||||
// slotA = DH(initiatorEphemeral, responderIdentity)
|
||||
// slotB = DH(initiatorIdentity, responderEphemeral)
|
||||
// slotC = DH(initiatorEphemeral, responderEphemeral)
|
||||
let slotA: Uint8Array;
|
||||
let slotB: Uint8Array;
|
||||
let slotC: Uint8Array;
|
||||
if (role === 'initiator') {
|
||||
slotA = await crypto.x25519(selfEphemeralPriv, peerIdentityDHPub);
|
||||
slotB = await crypto.x25519(selfIdentityDHPriv, peerEphemeralPub);
|
||||
slotC = await crypto.x25519(selfEphemeralPriv, peerEphemeralPub);
|
||||
} else {
|
||||
slotA = await crypto.x25519(selfIdentityDHPriv, peerEphemeralPub);
|
||||
slotB = await crypto.x25519(selfEphemeralPriv, peerIdentityDHPub);
|
||||
slotC = await crypto.x25519(selfEphemeralPriv, peerEphemeralPub);
|
||||
}
|
||||
|
||||
const ikm = new Uint8Array(96);
|
||||
ikm.set(slotA, 0);
|
||||
ikm.set(slotB, 32);
|
||||
ikm.set(slotC, 64);
|
||||
const sk = await crypto.hkdf(ikm, streamId, STREAM_KDF_INFO, 32);
|
||||
|
||||
crypto.zeroize(slotA);
|
||||
crypto.zeroize(slotB);
|
||||
crypto.zeroize(slotC);
|
||||
crypto.zeroize(ikm);
|
||||
return sk;
|
||||
}
|
||||
|
||||
/**
|
||||
* Bootstrap a fresh Double Ratchet `SessionState` from the derived
|
||||
* stream root key. The responder's ephemeral plays exactly the role
|
||||
* X3DH's signed prekey plays in {@link initSenderSession} /
|
||||
* {@link initReceiverSession}, so the ratchet handoff is identical to
|
||||
* the proven HTTP path.
|
||||
*
|
||||
* On the initiator only `responderEphemeral.publicKey` is needed; the
|
||||
* responder must pass its full ephemeral keypair.
|
||||
*
|
||||
* `peerIdentityDHPub` is recorded as the session's `remoteIdentityKey`
|
||||
* so stream fingerprints stay meaningful and consistent with the parent.
|
||||
*/
|
||||
export async function bootstrapStreamSession(
|
||||
crypto: CryptoProvider,
|
||||
role: StreamRole,
|
||||
sk: Uint8Array,
|
||||
peerIdentityDHPub: Uint8Array,
|
||||
responderEphemeral: KeyPair,
|
||||
): Promise<SessionState> {
|
||||
if (role === 'initiator') {
|
||||
// initSenderSession derives a fresh root via kdfRootKey and does not
|
||||
// retain `sk`, so the caller may safely zeroize it afterwards.
|
||||
return initSenderSession(crypto, sk, peerIdentityDHPub, responderEphemeral.publicKey);
|
||||
}
|
||||
// initReceiverSession stores the root key BY REFERENCE. Hand it an
|
||||
// independent copy so the caller zeroizing its `sk` scratch buffer
|
||||
// can't wipe the live session root.
|
||||
return initReceiverSession(new Uint8Array(sk), peerIdentityDHPub, responderEphemeral);
|
||||
}
|
||||
|
||||
/** Zeroize every secret a stream session holds, then drop the chains. */
|
||||
function zeroizeSession(crypto: CryptoProvider, s: SessionState): void {
|
||||
crypto.zeroize(s.rootKey);
|
||||
if (s.sendChain.chainKey.length > 0) crypto.zeroize(s.sendChain.chainKey);
|
||||
if (s.receiveChain && s.receiveChain.chainKey.length > 0) {
|
||||
crypto.zeroize(s.receiveChain.chainKey);
|
||||
}
|
||||
if (s.dhSend.privateKey.length > 0) crypto.zeroize(s.dhSend.privateKey);
|
||||
for (const mk of s.skippedKeys.values()) crypto.zeroize(mk);
|
||||
s.skippedKeys.clear();
|
||||
}
|
||||
|
||||
/**
|
||||
* In-memory holder for a stream's Double Ratchet. Serialises its own
|
||||
* `seal`/`open`/`close` on a private promise chain (independent of the
|
||||
* SDK's per-peer encrypt/decrypt queues — R5) so per-frame ratchet
|
||||
* mutations never interleave, while staying fully concurrent with the
|
||||
* parent session and with other streams.
|
||||
*
|
||||
* Never persisted: the ratchet exists only for the lifetime of the
|
||||
* stream and is zeroized on `close()`.
|
||||
*/
|
||||
export class StreamRatchet {
|
||||
private session: SessionState | null;
|
||||
private opChain: Promise<unknown> = Promise.resolve();
|
||||
|
||||
constructor(
|
||||
private readonly crypto: CryptoProvider,
|
||||
session: SessionState,
|
||||
/** 16-byte stream id this ratchet is bound to. */
|
||||
public readonly streamId: Uint8Array,
|
||||
) {
|
||||
this.session = session;
|
||||
}
|
||||
|
||||
/** True once {@link close} has run; `seal`/`open` will throw. */
|
||||
get closed(): boolean {
|
||||
return this.session === null;
|
||||
}
|
||||
|
||||
private run<T>(fn: (s: SessionState) => Promise<T>): Promise<T> {
|
||||
const next = this.opChain.catch(() => undefined).then(() => {
|
||||
if (!this.session) throw new StreamClosedError();
|
||||
return fn(this.session);
|
||||
});
|
||||
// Keep a never-rejecting tail so a failed frame doesn't poison the
|
||||
// next one (a single bad inbound frame must not wedge the stream).
|
||||
this.opChain = next.catch(() => undefined);
|
||||
return next;
|
||||
}
|
||||
|
||||
/** Wrap one logical frame. Advances the sending chain by one step. */
|
||||
seal(plaintext: Uint8Array): Promise<RatchetMessage> {
|
||||
return this.run((s) => ratchetEncrypt(this.crypto, s, plaintext));
|
||||
}
|
||||
|
||||
/**
|
||||
* Unwrap one inbound frame. Correct and memory-bounded across long
|
||||
* one-directional runs from the peer: ordered transport delivery
|
||||
* skips zero keys per frame, and out-of-order arrivals are still
|
||||
* capped by the ratchet's `MAX_SKIP` / `MAX_CACHED_SKIPPED_KEYS`.
|
||||
*/
|
||||
open(message: RatchetMessage): Promise<Uint8Array> {
|
||||
return this.run((s) => ratchetDecrypt(this.crypto, s, message));
|
||||
}
|
||||
|
||||
/** Zeroize and drop the ratchet. Idempotent. */
|
||||
close(): Promise<void> {
|
||||
return this.opChain
|
||||
.catch(() => undefined)
|
||||
.then(() => {
|
||||
if (this.session) {
|
||||
zeroizeSession(this.crypto, this.session);
|
||||
this.session = null;
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -281,6 +281,41 @@ describe('Double Ratchet', () => {
|
||||
|
||||
expect(ratchetDecrypt(crypto, bob, msg)).rejects.toThrow();
|
||||
});
|
||||
|
||||
/**
|
||||
* Regression — the v4.2.0 OutboundQueue waiter-since bug delivered
|
||||
* the same envelope twice to `manager.decrypt`. The first decrypt
|
||||
* succeeded via a cached skipped key; the second one fell into the
|
||||
* `message.counter < chain.counter` path with no skipped key
|
||||
* available, advanced the chainKey ONCE and rewound `chain.counter`
|
||||
* to `message.counter + 1`, leaving the ratchet permanently
|
||||
* desynced. ratchetDecrypt now rejects without mutating state when
|
||||
* a same-DH message is behind the chain and not in skippedKeys, so
|
||||
* a downstream replay (transport bug, retry, etc.) cannot poison
|
||||
* the session for everyone else.
|
||||
*/
|
||||
test('same-DH stale message after consumed skipped key fails without corrupting state', async () => {
|
||||
const { alice, bob } = await setupPair();
|
||||
|
||||
// Alice sends 3 messages on the same DH chain.
|
||||
const m0 = await ratchetEncrypt(crypto, alice, enc.encode('m0'));
|
||||
const m1 = await ratchetEncrypt(crypto, alice, enc.encode('m1'));
|
||||
const m2 = await ratchetEncrypt(crypto, alice, enc.encode('m2'));
|
||||
|
||||
// Bob receives m1 first, caching m0's key. Then m0 (delivered
|
||||
// via the cache). After this, m0's skipped key is consumed.
|
||||
expect(dec.decode(await ratchetDecrypt(crypto, bob, m1))).toBe('m1');
|
||||
expect(dec.decode(await ratchetDecrypt(crypto, bob, m0))).toBe('m0');
|
||||
|
||||
// Replay of m0: skippedKey is gone, chain.counter is past m0.
|
||||
// Pre-fix: this would corrupt Bob's chain state; post-fix it
|
||||
// throws cleanly.
|
||||
await expect(ratchetDecrypt(crypto, bob, m0)).rejects.toThrow(DecryptionError);
|
||||
|
||||
// Bob can still decrypt the remaining valid message — chain
|
||||
// state was NOT mutated by the rejected replay.
|
||||
expect(dec.decode(await ratchetDecrypt(crypto, bob, m2))).toBe('m2');
|
||||
});
|
||||
});
|
||||
|
||||
// ─── Long Conversation ────────────────────────────────────
|
||||
|
||||
176
packages/shade-core/tests/stream.test.ts
Normal file
176
packages/shade-core/tests/stream.test.ts
Normal file
@@ -0,0 +1,176 @@
|
||||
import { describe, test, expect, beforeEach } from 'bun:test';
|
||||
import { SubtleCryptoProvider, MemoryStorage } from '@shade/crypto-web';
|
||||
import {
|
||||
ShadeSessionManager,
|
||||
StreamRatchet,
|
||||
StreamClosedError,
|
||||
DecryptionError,
|
||||
} from '../src/index.js';
|
||||
|
||||
const crypto = new SubtleCryptoProvider();
|
||||
const enc = new TextEncoder();
|
||||
const dec = new TextDecoder();
|
||||
|
||||
/**
|
||||
* Establish a *bidirectional* parent session: Alice→Bob X3DH, then one
|
||||
* Alice→Bob message Bob decrypts so Bob also has a session for 'alice'.
|
||||
* Both sides then hold the peer's pinned identity DH key — the input the
|
||||
* stream handshake derives from.
|
||||
*/
|
||||
async function bidirectionalPair() {
|
||||
const aliceStorage = new MemoryStorage();
|
||||
const bobStorage = new MemoryStorage();
|
||||
const alice = new ShadeSessionManager(crypto, aliceStorage);
|
||||
const bob = new ShadeSessionManager(crypto, bobStorage);
|
||||
await alice.initialize();
|
||||
await bob.initialize();
|
||||
|
||||
const otpks = await bob.generateOneTimePreKeys(4);
|
||||
const bundle = await bob.createPreKeyBundle();
|
||||
bundle.oneTimePreKey = { keyId: otpks[0]!.keyId, publicKey: otpks[0]!.keyPair.publicKey };
|
||||
await alice.initSessionFromBundle('bob', bundle);
|
||||
|
||||
const hello = await alice.encrypt('bob', 'parent-hello');
|
||||
expect(await bob.decrypt('alice', hello)).toBe('parent-hello');
|
||||
|
||||
return { alice, bob, aliceStorage, bobStorage };
|
||||
}
|
||||
|
||||
/** Run the full STREAM_OPEN / STREAM_OPEN_ACK handshake between managers. */
|
||||
async function openStreamPair(alice: ShadeSessionManager, bob: ShadeSessionManager) {
|
||||
const begun = await alice.beginStream('bob'); // initiator
|
||||
const accepted = await bob.acceptStream('alice', begun.streamId, begun.ephemeralPublicKey);
|
||||
const aliceStream = await begun.complete(accepted.ephemeralPublicKey);
|
||||
return { aliceStream, bobStream: accepted.stream, streamId: begun.streamId };
|
||||
}
|
||||
|
||||
describe('streaming sub-ratchet (V4.11)', () => {
|
||||
let alice: ShadeSessionManager;
|
||||
let bob: ShadeSessionManager;
|
||||
let aliceStorage: MemoryStorage;
|
||||
|
||||
beforeEach(async () => {
|
||||
({ alice, bob, aliceStorage } = await bidirectionalPair());
|
||||
});
|
||||
|
||||
test('both sides derive the same stream root (round-trips frames)', async () => {
|
||||
const { aliceStream, bobStream } = await openStreamPair(alice, bob);
|
||||
|
||||
// Initiator → responder (first frame triggers responder DH step).
|
||||
const f1 = await aliceStream.seal(enc.encode('log line 1'));
|
||||
expect(dec.decode(await bobStream.open(f1))).toBe('log line 1');
|
||||
|
||||
// Responder → initiator (now responder may seal).
|
||||
const r1 = await bobStream.seal(enc.encode('command-response 1'));
|
||||
expect(dec.decode(await aliceStream.open(r1))).toBe('command-response 1');
|
||||
});
|
||||
|
||||
test('two streams to the same peer get independent roots', async () => {
|
||||
const s1 = await openStreamPair(alice, bob);
|
||||
const s2 = await openStreamPair(alice, bob);
|
||||
expect(s1.streamId).not.toEqual(s2.streamId);
|
||||
|
||||
const a = await s1.aliceStream.seal(enc.encode('on stream 1'));
|
||||
// A frame from stream 1 must not decrypt on stream 2's ratchet.
|
||||
await expect(s2.bobStream.open(a)).rejects.toBeInstanceOf(DecryptionError);
|
||||
// …but does on its own.
|
||||
expect(dec.decode(await s1.bobStream.open(a))).toBe('on stream 1');
|
||||
});
|
||||
|
||||
test('R2/R3: long one-directional burst stays correct and memory-bounded', async () => {
|
||||
const { aliceStream, bobStream } = await openStreamPair(alice, bob);
|
||||
const N = 5000;
|
||||
|
||||
// Capture a live receive-chain key buffer to prove forward secrecy:
|
||||
// ratchetDecrypt zeroizes the previous chain key in place.
|
||||
await bobStream.open(await aliceStream.seal(enc.encode('frame-0')));
|
||||
const bobSession = (bobStream as unknown as { session: { receiveChain: { chainKey: Uint8Array }; skippedKeys: Map<string, Uint8Array> } }).session;
|
||||
const staleChainKey = bobSession.receiveChain.chainKey;
|
||||
const staleCopy = staleChainKey.slice();
|
||||
expect(staleCopy.some((b) => b !== 0)).toBe(true);
|
||||
|
||||
for (let i = 1; i < N; i++) {
|
||||
const wire = await aliceStream.seal(enc.encode(`frame-${i}`));
|
||||
expect(dec.decode(await bobStream.open(wire))).toBe(`frame-${i}`);
|
||||
}
|
||||
|
||||
// In-order delivery ⇒ zero skipped keys retained across 5k frames.
|
||||
expect(bobSession.skippedKeys.size).toBe(0);
|
||||
// The chain key in use at frame 0 was overwritten (forward secrecy).
|
||||
expect(staleChainKey.every((b) => b === 0)).toBe(true);
|
||||
});
|
||||
|
||||
test('R5: opening/using/closing a stream never touches the parent session', async () => {
|
||||
const before = await aliceStorage.getSession('bob');
|
||||
const snapshot = JSON.stringify({
|
||||
root: Array.from(before!.rootKey),
|
||||
sendCtr: before!.sendChain.counter,
|
||||
prevCtr: before!.previousSendCounter,
|
||||
});
|
||||
|
||||
const { aliceStream, bobStream } = await openStreamPair(alice, bob);
|
||||
for (let i = 0; i < 200; i++) {
|
||||
await bobStream.open(await aliceStream.seal(enc.encode(`x${i}`)));
|
||||
}
|
||||
await aliceStream.close();
|
||||
await bobStream.close();
|
||||
|
||||
const after = await aliceStorage.getSession('bob');
|
||||
expect(
|
||||
JSON.stringify({
|
||||
root: Array.from(after!.rootKey),
|
||||
sendCtr: after!.sendChain.counter,
|
||||
prevCtr: after!.previousSendCounter,
|
||||
}),
|
||||
).toBe(snapshot);
|
||||
|
||||
// Parent HTTP path still works after the stream lifecycle.
|
||||
const env = await alice.encrypt('bob', 'after-stream');
|
||||
expect(await bob.decrypt('alice', env)).toBe('after-stream');
|
||||
});
|
||||
|
||||
test('R1: replayed / rewound frame is rejected', async () => {
|
||||
const { aliceStream, bobStream } = await openStreamPair(alice, bob);
|
||||
const f1 = await aliceStream.seal(enc.encode('once'));
|
||||
expect(dec.decode(await bobStream.open(f1))).toBe('once');
|
||||
// Re-delivering the exact same sealed frame must fail.
|
||||
await expect(bobStream.open(f1)).rejects.toBeInstanceOf(DecryptionError);
|
||||
});
|
||||
|
||||
test('close() zeroizes and blocks further use; idempotent', async () => {
|
||||
const { aliceStream, bobStream } = await openStreamPair(alice, bob);
|
||||
await aliceStream.close();
|
||||
await aliceStream.close(); // idempotent
|
||||
expect(aliceStream.closed).toBe(true);
|
||||
await expect(aliceStream.seal(enc.encode('nope'))).rejects.toBeInstanceOf(
|
||||
StreamClosedError,
|
||||
);
|
||||
// The peer end is unaffected by our local close.
|
||||
expect(bobStream.closed).toBe(false);
|
||||
});
|
||||
|
||||
test('handshake is mutually authenticated against pinned identities', async () => {
|
||||
// A third party (mallory) with its own identity cannot stand in for
|
||||
// bob: alice derives against bob's pinned identity key, so a
|
||||
// handshake completed with mallory's ephemeral yields a different
|
||||
// root and frames fail to open.
|
||||
const mStorage = new MemoryStorage();
|
||||
const mallory = new ShadeSessionManager(crypto, mStorage);
|
||||
await mallory.initialize();
|
||||
// Give mallory a parent session label so acceptStream has identity
|
||||
// material, but pinned to the WRONG (alice) identity vs what alice
|
||||
// pinned for 'bob'.
|
||||
const otpks = await mallory.generateOneTimePreKeys(2);
|
||||
const mb = await mallory.createPreKeyBundle();
|
||||
mb.oneTimePreKey = { keyId: otpks[0]!.keyId, publicKey: otpks[0]!.keyPair.publicKey };
|
||||
await alice.initSessionFromBundle('mallory', mb);
|
||||
const helo = await alice.encrypt('mallory', 'hi');
|
||||
await mallory.decrypt('alice', helo);
|
||||
|
||||
const begun = await alice.beginStream('bob');
|
||||
const mAccept = await mallory.acceptStream('alice', begun.streamId, begun.ephemeralPublicKey);
|
||||
const aliceStream = await begun.complete(mAccept.ephemeralPublicKey);
|
||||
const frame = await aliceStream.seal(enc.encode('secret'));
|
||||
await expect(mAccept.stream.open(frame)).rejects.toBeInstanceOf(DecryptionError);
|
||||
});
|
||||
});
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shade/crypto-web",
|
||||
"version": "4.0.0",
|
||||
"version": "4.11.0",
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
"types": "src/index.ts",
|
||||
|
||||
18
packages/shade-crypto-web/src/ed25519-derive.ts
Normal file
18
packages/shade-crypto-web/src/ed25519-derive.ts
Normal file
@@ -0,0 +1,18 @@
|
||||
import { ed25519 } from '@noble/curves/ed25519.js';
|
||||
|
||||
/**
|
||||
* Deterministically derive an Ed25519 public key from a 32-byte seed.
|
||||
*
|
||||
* In the @noble/curves convention the "private key" *is* the seed —
|
||||
* `sign(seed, msg)` works directly, and `getPublicKey(seed)` recovers
|
||||
* the matching public key. V4.9's encrypted-blob primitive uses this
|
||||
* to mint a per-slot signing keypair from an HKDF output rooted at the
|
||||
* user's master key, so the same credentials always reproduce the same
|
||||
* keypair.
|
||||
*/
|
||||
export function ed25519PublicKeyFromSeed(seed: Uint8Array): Uint8Array {
|
||||
if (seed.length !== 32) {
|
||||
throw new Error(`Ed25519 seed must be 32 bytes, got ${seed.length}`);
|
||||
}
|
||||
return ed25519.getPublicKey(seed);
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
export { SubtleCryptoProvider } from './provider.js';
|
||||
export { MemoryStorage } from './memory-storage.js';
|
||||
export { ed25519PublicKeyFromSeed } from './ed25519-derive.js';
|
||||
|
||||
// ─── Web Workers crypto (V3.8) ────────────────────────────
|
||||
export {
|
||||
|
||||
@@ -1,4 +1,8 @@
|
||||
import type { StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey, SessionState, RetiredIdentity, PersistedStreamState, PeerVerification } from '@shade/core';
|
||||
import type {
|
||||
StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey,
|
||||
SessionState, RetiredIdentity, PersistedStreamState, PeerVerification,
|
||||
BroadcastChannelRecord, BroadcastMemberRecord,
|
||||
} from '@shade/core';
|
||||
import { constantTimeEqual } from '@shade/core';
|
||||
|
||||
/**
|
||||
@@ -167,4 +171,62 @@ export class MemoryStorage implements StorageProvider {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Broadcast channels (V4.6) ────────────────────────────
|
||||
|
||||
private broadcastChannels = new Map<string, BroadcastChannelRecord>();
|
||||
private broadcastMembers = new Map<string, Map<string, BroadcastMemberRecord>>();
|
||||
|
||||
async saveBroadcastChannel(channel: BroadcastChannelRecord): Promise<void> {
|
||||
this.broadcastChannels.set(channel.channelId, cloneChannel(channel));
|
||||
}
|
||||
|
||||
async getBroadcastChannel(channelId: string): Promise<BroadcastChannelRecord | null> {
|
||||
const v = this.broadcastChannels.get(channelId);
|
||||
return v ? cloneChannel(v) : null;
|
||||
}
|
||||
|
||||
async listBroadcastChannels(): Promise<BroadcastChannelRecord[]> {
|
||||
return [...this.broadcastChannels.values()].map(cloneChannel);
|
||||
}
|
||||
|
||||
async removeBroadcastChannel(channelId: string): Promise<void> {
|
||||
this.broadcastChannels.delete(channelId);
|
||||
this.broadcastMembers.delete(channelId);
|
||||
}
|
||||
|
||||
async saveBroadcastMember(member: BroadcastMemberRecord): Promise<void> {
|
||||
let inner = this.broadcastMembers.get(member.channelId);
|
||||
if (!inner) {
|
||||
inner = new Map();
|
||||
this.broadcastMembers.set(member.channelId, inner);
|
||||
}
|
||||
inner.set(member.peerAddress, { ...member });
|
||||
}
|
||||
|
||||
async getBroadcastMembers(channelId: string): Promise<BroadcastMemberRecord[]> {
|
||||
const inner = this.broadcastMembers.get(channelId);
|
||||
return inner ? [...inner.values()].map((m) => ({ ...m })) : [];
|
||||
}
|
||||
|
||||
async removeBroadcastMember(channelId: string, peerAddress: string): Promise<void> {
|
||||
this.broadcastMembers.get(channelId)?.delete(peerAddress);
|
||||
}
|
||||
}
|
||||
|
||||
function cloneChannel(c: BroadcastChannelRecord): BroadcastChannelRecord {
|
||||
const out: BroadcastChannelRecord = {
|
||||
channelId: c.channelId,
|
||||
ownerRole: c.ownerRole,
|
||||
ownerAddress: c.ownerAddress,
|
||||
generation: c.generation,
|
||||
chainKey: new Uint8Array(c.chainKey),
|
||||
iteration: c.iteration,
|
||||
signingPublicKey: new Uint8Array(c.signingPublicKey),
|
||||
createdAt: c.createdAt,
|
||||
updatedAt: c.updatedAt,
|
||||
};
|
||||
if (c.label !== undefined) out.label = c.label;
|
||||
if (c.signingPrivateKey !== undefined) out.signingPrivateKey = new Uint8Array(c.signingPrivateKey);
|
||||
return out;
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shade/dashboard",
|
||||
"version": "4.0.0",
|
||||
"version": "4.11.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
|
||||
@@ -8,5 +8,5 @@
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "bundler"
|
||||
},
|
||||
"include": ["src", "vite.config.ts"]
|
||||
"include": ["src"]
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shade/files",
|
||||
"version": "4.0.0",
|
||||
"version": "4.11.0",
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
"types": "src/index.ts",
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import type { Shade } from '@shade/sdk';
|
||||
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||
import {
|
||||
KIND_CUSTOM_V1,
|
||||
KIND_DELETE_V1,
|
||||
@@ -184,7 +184,7 @@ export interface CreateFileClientOptions {
|
||||
* transfers that carry the actual bytes.
|
||||
*/
|
||||
export function createFileClient(
|
||||
shade: Shade,
|
||||
shade: ShadeBridge,
|
||||
channel: ShadeFileRpcChannel,
|
||||
pending: PendingRpcRegistry,
|
||||
peerAddress: string,
|
||||
|
||||
590
packages/shade-files/src/client/http-client.ts
Normal file
590
packages/shade-files/src/client/http-client.ts
Normal file
@@ -0,0 +1,590 @@
|
||||
/**
|
||||
* Browser-friendly request-response `FileClient` for `@shade/files`.
|
||||
*
|
||||
* The default `shade.files.client(peer)` ships RPC envelopes via
|
||||
* `Shade.send` + `Shade.deliverControlEnvelope`, which means the
|
||||
* server has to be able to call back outbound to the client. That
|
||||
* doesn't work for browser tabs (no inbound HTTP listener). This
|
||||
* client posts each RPC envelope to a single server endpoint and
|
||||
* reads the encrypted response from the same HTTP response — pure
|
||||
* request-response, no inbound channel required.
|
||||
*
|
||||
* Inline payloads only (≤ 256 KiB). For larger reads/writes, use the
|
||||
* stateful path: `shade.files.client(peer)` server-to-server, with
|
||||
* `@shade/transfer` chunk routes for content I/O.
|
||||
*
|
||||
* @see {@link createFilesRpcRoute} for the matching server-side route.
|
||||
*/
|
||||
import type { ZodTypeAny } from 'zod';
|
||||
import { decodeEnvelope, encodeEnvelope as encodeWireEnvelope } from '@shade/proto';
|
||||
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||
import {
|
||||
encodeEnvelope as encodeRpcEnvelope,
|
||||
tryParseEnvelope,
|
||||
} from '../protocol/envelope-codec.js';
|
||||
import {
|
||||
KIND_CUSTOM_V1,
|
||||
KIND_DELETE_V1,
|
||||
KIND_GET_THUMBNAIL_V1,
|
||||
KIND_LIST_V1,
|
||||
KIND_MKDIR_V1,
|
||||
KIND_MOVE_V1,
|
||||
KIND_READ_V1,
|
||||
KIND_STAT_V1,
|
||||
KIND_WRITE_V1,
|
||||
} from '../protocol/kinds.js';
|
||||
import {
|
||||
CustomArgsSchema,
|
||||
CustomResultSchema,
|
||||
DeleteArgsSchema,
|
||||
DeleteResultSchema,
|
||||
GetThumbnailArgsSchema,
|
||||
GetThumbnailResultSchema,
|
||||
ListArgsSchema,
|
||||
ListResultSchema,
|
||||
MkdirArgsSchema,
|
||||
MkdirResultSchema,
|
||||
MoveArgsSchema,
|
||||
MoveResultSchema,
|
||||
ReadArgsSchema,
|
||||
ReadResultSchema,
|
||||
StatArgsSchema,
|
||||
StatResultSchema,
|
||||
WriteArgsSchema,
|
||||
WriteResultSchema,
|
||||
type ListResult,
|
||||
type MkdirResult,
|
||||
type DeleteResult,
|
||||
type MoveResult,
|
||||
type StatResult,
|
||||
type ThumbnailSize,
|
||||
type WriteResult,
|
||||
} from '../schemas/ops.js';
|
||||
import {
|
||||
fileErrorFromPayload,
|
||||
CancelledError,
|
||||
InternalFileError,
|
||||
ConflictError,
|
||||
} from '../schemas/errors.js';
|
||||
import { buildRpcRequest } from '../protocol/rpc-builder.js';
|
||||
import { decideInline, INLINE_THRESHOLD, type WriteSource } from './inline-threshold.js';
|
||||
import { base64ToBytes, bytesToBase64 } from '../protocol/canonical.js';
|
||||
import { startQueueDrainer, type QueueDrainerHandle } from './queue-drainer.js';
|
||||
import {
|
||||
createClientStreamsBridge,
|
||||
type ClientStreamsBridge,
|
||||
} from './streams-bridge.js';
|
||||
import type {
|
||||
FileClient,
|
||||
ReadOpts,
|
||||
ReadOutput,
|
||||
ThumbnailResult,
|
||||
WriteOpts,
|
||||
CreateFileClientOptions,
|
||||
BaseOpts,
|
||||
} from './client.js';
|
||||
|
||||
export interface FilesHttpClientOptions
|
||||
extends Omit<CreateFileClientOptions, 'streamsBridge'> {
|
||||
/**
|
||||
* Server endpoint that hosts `createFilesRpcRoute(...)`. Typically:
|
||||
* `https://server.example.com/api/v1/shade-files/rpc`.
|
||||
*/
|
||||
rpcUrl: string;
|
||||
/**
|
||||
* Optional `fetch` override. Defaults to `globalThis.fetch`. Wire a
|
||||
* custom `fetch` to thread auth-cookies, CSRF tokens, or
|
||||
* service-worker interception.
|
||||
*/
|
||||
fetch?: typeof globalThis.fetch;
|
||||
/**
|
||||
* Extra HTTP headers applied to every RPC POST. Useful for app-level
|
||||
* auth (CSRF, session cookies via custom header, etc.) — these are
|
||||
* orthogonal to the ratchet authentication on the envelope itself.
|
||||
*/
|
||||
headers?: Record<string, string>;
|
||||
/**
|
||||
* Server endpoint that hosts `transferQueueRoute()`'s long-poll
|
||||
* endpoint. Typically:
|
||||
* `https://server.example.com/api/v1/shade-files/queue`.
|
||||
*
|
||||
* When supplied, the client starts a background long-poll that
|
||||
* drains queued envelopes + chunks from the server and dispatches
|
||||
* them via `shade.acceptTransferEnvelope`. This unlocks
|
||||
* **streamed reads** (>256 KiB) for browser-style consumers.
|
||||
*/
|
||||
outboundQueueUrl?: string;
|
||||
/**
|
||||
* Base URL for outbound transfer routes (browser → server). Required
|
||||
* alongside `outboundQueueUrl` to enable streamed writes. Typically:
|
||||
* `https://server.example.com/api/v1/shade-files`.
|
||||
*
|
||||
* The client POSTs:
|
||||
* - chunks to `<base>/v1/transfer/<streamId>/chunk`
|
||||
* - control envelopes to `<base>/v1/transfer/control`
|
||||
*/
|
||||
transferBaseUrl?: string;
|
||||
/**
|
||||
* Long-poll block timeout, milliseconds. Default 30_000. Server
|
||||
* clamps to its own `maxBlockMs` (default 55_000).
|
||||
*/
|
||||
queueBlockMs?: number;
|
||||
}
|
||||
|
||||
interface RoundTripOpts {
|
||||
signal?: AbortSignal;
|
||||
timeoutMs?: number;
|
||||
idempotencyKey?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a request-response `FileClient` bound to `peerAddress` and a
|
||||
* server-side RPC URL. The session must already be established
|
||||
* (via `shade.initSessionFromBundle(peerAddress, bundle)` or an
|
||||
* incoming first-message). Otherwise the first RPC will fail with
|
||||
* "decrypt failed: no session for peer".
|
||||
*
|
||||
* When `outboundQueueUrl` + `transferBaseUrl` are supplied, the
|
||||
* client also unlocks **streamed reads/writes** for files larger than
|
||||
* the inline threshold (256 KiB). The browser polls the server's
|
||||
* outbound queue for chunks/envelopes and POSTs its own outbound
|
||||
* chunks to the server's transfer-receive routes.
|
||||
*/
|
||||
export function createFilesHttpClient(
|
||||
shade: ShadeBridge,
|
||||
peerAddress: string,
|
||||
options: FilesHttpClientOptions,
|
||||
): FileClient {
|
||||
const rpcUrl = options.rpcUrl;
|
||||
const fetchFn = options.fetch ?? globalThis.fetch.bind(globalThis);
|
||||
const extraHeaders = options.headers ?? {};
|
||||
const defaultTimeoutMs = options.defaultTimeoutMs ?? 30_000;
|
||||
const ioTimeoutMs = options.ioTimeoutMs ?? 60_000;
|
||||
const signRequest = options.signRequest;
|
||||
const senderAddress = shade.myAddress;
|
||||
|
||||
// ─── Streamed-mode bootstrap ─────────────────────────────────
|
||||
//
|
||||
// When `outboundQueueUrl` is supplied, the client:
|
||||
// 1. Configures `shade.configureTransfers(...)` so outbound
|
||||
// chunks POST to `<transferBaseUrl>/v1/transfer/<streamId>/chunk`
|
||||
// and outbound control envelopes POST to
|
||||
// `<transferBaseUrl>/v1/transfer/control`.
|
||||
// 2. Spawns a streams-bridge so streamed reads can be awaited.
|
||||
// 3. Starts a long-poll drainer that pulls queued envelopes +
|
||||
// chunks from the server and dispatches via
|
||||
// `shade.acceptTransferEnvelope`.
|
||||
|
||||
let drainer: QueueDrainerHandle | null = null;
|
||||
let streamsBridgePromise: Promise<ClientStreamsBridge> | null = null;
|
||||
let streamsBridge: ClientStreamsBridge | null = null;
|
||||
|
||||
if (options.outboundQueueUrl !== undefined) {
|
||||
const outboundQueueUrl = options.outboundQueueUrl;
|
||||
if (options.transferBaseUrl === undefined) {
|
||||
throw new Error(
|
||||
'createFilesHttpClient: outboundQueueUrl was supplied without transferBaseUrl. Pass `transferBaseUrl` (the server prefix that hosts /v1/transfer/...) so outbound chunks have a destination.',
|
||||
);
|
||||
}
|
||||
if (shade.configureTransfers === undefined) {
|
||||
throw new Error(
|
||||
'createFilesHttpClient: shade.configureTransfers is required for streamed mode (the underlying ShadeBridge must surface it).',
|
||||
);
|
||||
}
|
||||
const transferBaseUrl = options.transferBaseUrl.replace(/\/$/, '');
|
||||
shade.configureTransfers({
|
||||
resolveBaseUrl: async (peer) => {
|
||||
if (peer !== peerAddress) {
|
||||
throw new Error(
|
||||
`httpClient is bound to peer "${peerAddress}" — refusing to resolve outgoing chunks for "${peer}" without a multi-peer registry. Use shade.files.client(peer) for server-to-server multi-peer.`,
|
||||
);
|
||||
}
|
||||
return transferBaseUrl;
|
||||
},
|
||||
});
|
||||
// Build the streams-bridge eagerly. The engine's incoming-transfer
|
||||
// subscription has to be in place BEFORE the drainer dispatches the
|
||||
// first stream-init envelope, otherwise the engine emits the
|
||||
// IncomingTransfer to zero handlers and the read silently never
|
||||
// accepts. We kick off the drainer once the bridge has subscribed.
|
||||
streamsBridgePromise = createClientStreamsBridge(shade).then((bridge) => {
|
||||
streamsBridge = bridge;
|
||||
drainer = startQueueDrainer(shade, {
|
||||
outboundQueueUrl,
|
||||
peerAddress,
|
||||
senderAddress,
|
||||
...(options.fetch !== undefined ? { fetch: options.fetch } : {}),
|
||||
...(options.headers !== undefined ? { headers: options.headers } : {}),
|
||||
...(options.queueBlockMs !== undefined ? { blockMs: options.queueBlockMs } : {}),
|
||||
});
|
||||
return bridge;
|
||||
});
|
||||
// Surface bridge-construction failures eagerly via a rejected
|
||||
// promise the next read/write picks up.
|
||||
streamsBridgePromise.catch(() => {
|
||||
/* observed via getStreamsBridge() */
|
||||
});
|
||||
}
|
||||
|
||||
async function getStreamsBridge(): Promise<ClientStreamsBridge> {
|
||||
if (streamsBridge !== null) return streamsBridge;
|
||||
if (streamsBridgePromise === null) {
|
||||
throw new ConflictError(
|
||||
`http RPC client supports inline writes/reads only (≤ ${INLINE_THRESHOLD} bytes) — pass { outboundQueueUrl, transferBaseUrl } to enable streamed transfers.`,
|
||||
);
|
||||
}
|
||||
streamsBridge = await streamsBridgePromise;
|
||||
return streamsBridge;
|
||||
}
|
||||
|
||||
/**
|
||||
* Encrypt + POST + decrypt + parse one RPC round-trip.
|
||||
*
|
||||
* Throws a typed `FileError` subclass when the server returns an
|
||||
* encrypted `RpcError`, or `InternalFileError` for transport-level
|
||||
* failures (network, 4xx/5xx, malformed body).
|
||||
*/
|
||||
async function roundTrip<TResult>(
|
||||
kind: string,
|
||||
op: 'list' | 'stat' | 'mkdir' | 'delete' | 'move' | 'read' | 'write' | 'getThumbnail' | 'custom',
|
||||
args: unknown,
|
||||
resultSchema: ZodTypeAny,
|
||||
opts: RoundTripOpts | undefined,
|
||||
): Promise<TResult> {
|
||||
const requestEnv = await buildRpcRequest({
|
||||
senderAddress,
|
||||
kind,
|
||||
op,
|
||||
args,
|
||||
...(opts?.idempotencyKey !== undefined ? { idempotencyKey: opts.idempotencyKey } : {}),
|
||||
...(signRequest !== undefined ? { signRequest } : {}),
|
||||
});
|
||||
|
||||
const plaintext = encodeRpcEnvelope(requestEnv);
|
||||
const ratchetEnvelope = await shade.send(peerAddress, plaintext);
|
||||
const wireBytes = encodeWireEnvelope(ratchetEnvelope);
|
||||
|
||||
const ac = new AbortController();
|
||||
const timeoutMs = opts?.timeoutMs ?? defaultTimeoutMs;
|
||||
const timer = setTimeout(
|
||||
() => ac.abort(new Error(`RPC timeout after ${timeoutMs}ms`)),
|
||||
timeoutMs,
|
||||
);
|
||||
(timer as unknown as { unref?: () => void }).unref?.();
|
||||
if (opts?.signal !== undefined) {
|
||||
const userSignal = opts.signal;
|
||||
if (userSignal.aborted) ac.abort(userSignal.reason);
|
||||
else userSignal.addEventListener('abort', () => ac.abort(userSignal.reason), { once: true });
|
||||
}
|
||||
|
||||
let response: Response;
|
||||
try {
|
||||
// Wrap the wire bytes in a Blob so the body type satisfies the
|
||||
// common-denominator `BodyInit` across DOM, Bun, and node-fetch
|
||||
// (some runtimes accept `Uint8Array` directly, others don't).
|
||||
// Cast through `unknown` because TS's `bun-types` and `lib.dom`
|
||||
// disagree about whether `Uint8Array<ArrayBufferLike>` is itself
|
||||
// a `BlobPart`; the runtime accepts it on every platform.
|
||||
response = await fetchFn(rpcUrl, {
|
||||
method: 'POST',
|
||||
body: new Blob([wireBytes as unknown as ArrayBuffer]),
|
||||
signal: ac.signal,
|
||||
headers: {
|
||||
'Content-Type': 'application/octet-stream',
|
||||
'X-Shade-Sender-Address': senderAddress,
|
||||
...extraHeaders,
|
||||
},
|
||||
});
|
||||
} catch (err) {
|
||||
clearTimeout(timer);
|
||||
if ((err as Error).name === 'AbortError') {
|
||||
throw new CancelledError(`RPC ${kind} aborted: ${(err as Error).message}`);
|
||||
}
|
||||
throw new InternalFileError(`RPC ${kind} fetch failed: ${(err as Error).message}`);
|
||||
}
|
||||
clearTimeout(timer);
|
||||
|
||||
if (!response.ok) {
|
||||
let body: { error?: string } | null = null;
|
||||
try {
|
||||
body = (await response.json()) as { error?: string };
|
||||
} catch {
|
||||
/* server emitted non-JSON body */
|
||||
}
|
||||
throw new InternalFileError(
|
||||
`RPC ${kind} → ${response.status} ${response.statusText}: ${
|
||||
body?.error ?? '(no error body)'
|
||||
}`,
|
||||
);
|
||||
}
|
||||
|
||||
const ab = await response.arrayBuffer();
|
||||
if (ab.byteLength === 0) {
|
||||
throw new InternalFileError(`RPC ${kind}: empty response body`);
|
||||
}
|
||||
|
||||
let responseRatchet;
|
||||
try {
|
||||
responseRatchet = decodeEnvelope(new Uint8Array(ab));
|
||||
} catch (err) {
|
||||
throw new InternalFileError(
|
||||
`RPC ${kind}: response body is not a valid wire envelope: ${(err as Error).message}`,
|
||||
);
|
||||
}
|
||||
let responsePlaintext: string;
|
||||
try {
|
||||
responsePlaintext = await shade.receive(peerAddress, responseRatchet);
|
||||
} catch (err) {
|
||||
throw new InternalFileError(
|
||||
`RPC ${kind}: response decrypt failed: ${(err as Error).message}`,
|
||||
);
|
||||
}
|
||||
const classified = tryParseEnvelope(responsePlaintext);
|
||||
if (classified === null) {
|
||||
throw new InternalFileError(
|
||||
`RPC ${kind}: response plaintext is not a valid @shade/files envelope`,
|
||||
);
|
||||
}
|
||||
if (classified.kind === 'error') {
|
||||
throw fileErrorFromPayload(classified.envelope.error);
|
||||
}
|
||||
if (classified.kind !== 'response') {
|
||||
throw new InternalFileError(
|
||||
`RPC ${kind}: unexpected response envelope kind: ${classified.kind}`,
|
||||
);
|
||||
}
|
||||
if (classified.envelope.id !== requestEnv.id) {
|
||||
throw new InternalFileError(
|
||||
`RPC ${kind}: response correlation id mismatch (got ${classified.envelope.id}, expected ${requestEnv.id})`,
|
||||
);
|
||||
}
|
||||
return resultSchema.parse(classified.envelope.result) as TResult;
|
||||
}
|
||||
|
||||
return {
|
||||
async list(path, opts): Promise<ListResult> {
|
||||
const args = ListArgsSchema.parse({
|
||||
path,
|
||||
...(opts?.cursor !== undefined ? { cursor: opts.cursor } : {}),
|
||||
...(opts?.pageSize !== undefined ? { pageSize: opts.pageSize } : {}),
|
||||
...(opts?.filter !== undefined ? { filter: opts.filter } : {}),
|
||||
});
|
||||
return await roundTrip<ListResult>(
|
||||
KIND_LIST_V1,
|
||||
'list',
|
||||
args,
|
||||
ListResultSchema,
|
||||
opts,
|
||||
);
|
||||
},
|
||||
|
||||
async stat(path, opts): Promise<StatResult> {
|
||||
const args = StatArgsSchema.parse({ path });
|
||||
return await roundTrip<StatResult>(KIND_STAT_V1, 'stat', args, StatResultSchema, opts);
|
||||
},
|
||||
|
||||
async mkdir(path, opts): Promise<MkdirResult> {
|
||||
const args = MkdirArgsSchema.parse({
|
||||
path,
|
||||
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
|
||||
});
|
||||
return await roundTrip<MkdirResult>(
|
||||
KIND_MKDIR_V1,
|
||||
'mkdir',
|
||||
args,
|
||||
MkdirResultSchema,
|
||||
opts,
|
||||
);
|
||||
},
|
||||
|
||||
async delete(path, opts): Promise<DeleteResult> {
|
||||
const args = DeleteArgsSchema.parse({
|
||||
path,
|
||||
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
|
||||
});
|
||||
return await roundTrip<DeleteResult>(
|
||||
KIND_DELETE_V1,
|
||||
'delete',
|
||||
args,
|
||||
DeleteResultSchema,
|
||||
opts,
|
||||
);
|
||||
},
|
||||
|
||||
async move(src, dst, opts): Promise<MoveResult> {
|
||||
const args = MoveArgsSchema.parse({
|
||||
src,
|
||||
dst,
|
||||
...(opts?.overwrite !== undefined ? { overwrite: opts.overwrite } : {}),
|
||||
});
|
||||
return await roundTrip<MoveResult>(KIND_MOVE_V1, 'move', args, MoveResultSchema, opts);
|
||||
},
|
||||
|
||||
async read(path, opts: ReadOpts = {}): Promise<ReadOutput> {
|
||||
const args = ReadArgsSchema.parse({
|
||||
path,
|
||||
...(opts.range !== undefined ? { range: opts.range } : {}),
|
||||
...(opts.preferInline !== undefined ? { preferInline: opts.preferInline } : {}),
|
||||
});
|
||||
const wire = await roundTrip<import('../schemas/ops.js').ReadResult>(
|
||||
KIND_READ_V1,
|
||||
'read',
|
||||
args,
|
||||
ReadResultSchema,
|
||||
opts,
|
||||
);
|
||||
if (wire.kind === 'inline') {
|
||||
const bytes = base64ToBytes(wire.bytesB64);
|
||||
const out: ReadOutput = {
|
||||
kind: 'inline',
|
||||
bytes,
|
||||
size: wire.size,
|
||||
sha256: wire.sha256,
|
||||
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
|
||||
};
|
||||
return out;
|
||||
}
|
||||
// Streamed read — only supported when the queue drainer is wired.
|
||||
if (drainer === null) {
|
||||
throw new InternalFileError(
|
||||
`http RPC client received a streamed read (size ${wire.size}) but is in inline-only mode. Pass { outboundQueueUrl, transferBaseUrl } when constructing the client to enable streamed reads.`,
|
||||
);
|
||||
}
|
||||
const bridge = await getStreamsBridge();
|
||||
const bridgeSignal = opts.signal ?? new AbortController().signal;
|
||||
const parked = await bridge.awaitRead(wire.streamId, {
|
||||
expectedFrom: peerAddress,
|
||||
signal: bridgeSignal,
|
||||
timeoutMs: ioTimeoutMs,
|
||||
});
|
||||
const out: ReadOutput = {
|
||||
kind: 'streams',
|
||||
stream: parked.readable,
|
||||
size: wire.size,
|
||||
sha256: wire.sha256,
|
||||
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
|
||||
done: async () => {
|
||||
await parked.done;
|
||||
},
|
||||
};
|
||||
return out;
|
||||
},
|
||||
|
||||
async write(path, input: WriteSource, opts: WriteOpts = {}): Promise<WriteResult> {
|
||||
const decision = await decideInline(input);
|
||||
const overwrite = opts.overwrite ?? false;
|
||||
const contentType = opts.contentType ?? decision.contentType;
|
||||
|
||||
if (decision.kind === 'inline' || opts.forceInline === true) {
|
||||
const bytes = decision.kind === 'inline' ? decision.bytes : null;
|
||||
if (bytes === null) {
|
||||
// forceInline === true with a streams-typed decision —
|
||||
// decideInline always produced a `streams` shape because the
|
||||
// input was a bare ReadableStream. We can't drain a stream
|
||||
// synchronously here without a streams-bridge.
|
||||
throw new ConflictError(
|
||||
'http RPC client cannot forceInline a streamed input — pass a Uint8Array / Blob, or pre-buffer the stream.',
|
||||
);
|
||||
}
|
||||
if (bytes.byteLength > INLINE_THRESHOLD) {
|
||||
throw new ConflictError(
|
||||
`inline write exceeds ${INLINE_THRESHOLD}-byte threshold (got ${bytes.byteLength}); pass forceInline=true to override`,
|
||||
);
|
||||
}
|
||||
const args = WriteArgsSchema.parse({
|
||||
kind: 'inline',
|
||||
path,
|
||||
bytesB64: bytesToBase64(bytes),
|
||||
...(contentType !== undefined ? { contentType } : {}),
|
||||
overwrite,
|
||||
});
|
||||
return await roundTrip<WriteResult>(
|
||||
KIND_WRITE_V1,
|
||||
'write',
|
||||
args,
|
||||
WriteResultSchema,
|
||||
opts,
|
||||
);
|
||||
}
|
||||
|
||||
// Streamed write — requires the queue drainer + streams-bridge.
|
||||
if (drainer === null) {
|
||||
throw new ConflictError(
|
||||
`http RPC client supports inline writes only (≤ ${INLINE_THRESHOLD} bytes). The supplied input was promoted to streams (size ${decision.size ?? 'unknown'}). Pass { outboundQueueUrl, transferBaseUrl } to enable streamed writes.`,
|
||||
);
|
||||
}
|
||||
const bridge = await getStreamsBridge();
|
||||
const size = decision.size;
|
||||
if (size === undefined) {
|
||||
throw new ConflictError(
|
||||
'streams write requires a known plaintext size; pass `{ stream, size }` instead of a bare ReadableStream',
|
||||
);
|
||||
}
|
||||
const { writeId, handle } = await bridge.initiateWrite({
|
||||
peer: peerAddress,
|
||||
stream: decision.stream,
|
||||
size,
|
||||
...(contentType !== undefined ? { contentType } : {}),
|
||||
name: path,
|
||||
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
|
||||
});
|
||||
const args = WriteArgsSchema.parse({
|
||||
kind: 'streams',
|
||||
path,
|
||||
size,
|
||||
...(contentType !== undefined ? { contentType } : {}),
|
||||
overwrite,
|
||||
writeId,
|
||||
});
|
||||
try {
|
||||
const [result] = await Promise.all([
|
||||
roundTrip<WriteResult>(KIND_WRITE_V1, 'write', args, WriteResultSchema, opts),
|
||||
handle.done(),
|
||||
]);
|
||||
return result;
|
||||
} catch (err) {
|
||||
await handle.abort('rpc-failed').catch(() => undefined);
|
||||
throw err;
|
||||
}
|
||||
},
|
||||
|
||||
async getThumbnail(path, size: ThumbnailSize, opts): Promise<ThumbnailResult> {
|
||||
const args = GetThumbnailArgsSchema.parse({
|
||||
path,
|
||||
size,
|
||||
...(opts?.format !== undefined ? { format: opts.format } : {}),
|
||||
});
|
||||
const raw = await roundTrip<import('../schemas/ops.js').GetThumbnailResult>(
|
||||
KIND_GET_THUMBNAIL_V1,
|
||||
'getThumbnail',
|
||||
args,
|
||||
GetThumbnailResultSchema,
|
||||
opts,
|
||||
);
|
||||
return {
|
||||
bytes: base64ToBytes(raw.bytesB64),
|
||||
format: raw.format,
|
||||
width: raw.width,
|
||||
height: raw.height,
|
||||
sha256: raw.sha256,
|
||||
};
|
||||
},
|
||||
|
||||
async custom(name, args, opts?: BaseOpts): Promise<unknown> {
|
||||
const wireArgs = CustomArgsSchema.parse({ name, args });
|
||||
return await roundTrip(KIND_CUSTOM_V1, 'custom', wireArgs, CustomResultSchema, opts);
|
||||
},
|
||||
|
||||
close(): void {
|
||||
// Stop the long-poll drainer + tear down the streams-bridge if
|
||||
// we built one. Idempotent — safe to call multiple times.
|
||||
drainer?.stop();
|
||||
drainer = null;
|
||||
if (streamsBridge !== null) {
|
||||
void streamsBridge.destroy().catch(() => undefined);
|
||||
streamsBridge = null;
|
||||
}
|
||||
streamsBridgePromise = null;
|
||||
},
|
||||
} as FileClient;
|
||||
}
|
||||
@@ -161,15 +161,33 @@ async function peekStream(stream: ReadableStream<Uint8Array>): Promise<InlineDec
|
||||
}
|
||||
}
|
||||
|
||||
interface MinimalReader {
|
||||
read(): Promise<{ value: Uint8Array | undefined; done: boolean }>;
|
||||
/**
|
||||
* Structural mirror of WHATWG `ReadableStreamDefaultReader<Uint8Array>`.
|
||||
*
|
||||
* The disjoint union shape with `value?: T | undefined` is the lowest
|
||||
* common denominator across every lib environment we care about:
|
||||
* - `bun-types` emits `{ done: true; value?: undefined }`
|
||||
* - `lib.dom` emits `{ done: true; value?: T }`
|
||||
* - `node:stream/web` emits the union form
|
||||
*
|
||||
* `value?: T | undefined` is assignable from all three. A flat
|
||||
* `{ value?: T; done: boolean }` is rejected by
|
||||
* `exactOptionalPropertyTypes` because the present branches require
|
||||
* `value: T`. Defining it as an explicit union avoids the trap.
|
||||
*/
|
||||
type MinimalReadResult<T> =
|
||||
| { done: false; value: T }
|
||||
| { done: true; value?: T | undefined };
|
||||
|
||||
interface MinimalReader<T> {
|
||||
read(): Promise<MinimalReadResult<T>>;
|
||||
cancel(reason?: unknown): Promise<void>;
|
||||
releaseLock(): void;
|
||||
}
|
||||
|
||||
function reconstructStream(
|
||||
prefix: Uint8Array[],
|
||||
reader: MinimalReader,
|
||||
reader: MinimalReader<Uint8Array>,
|
||||
): ReadableStream<Uint8Array> {
|
||||
let prefixIdx = 0;
|
||||
return new ReadableStream<Uint8Array>({
|
||||
|
||||
172
packages/shade-files/src/client/queue-drainer.ts
Normal file
172
packages/shade-files/src/client/queue-drainer.ts
Normal file
@@ -0,0 +1,172 @@
|
||||
/**
|
||||
* Browser-side drainer for the pull-mode outbound queue.
|
||||
*
|
||||
* Background task that long-polls the server's `/queue` endpoint,
|
||||
* decodes each event, and dispatches it into the consumer's Shade
|
||||
* instance via `shade.acceptTransferEnvelope`. Same wire-shape as the
|
||||
* server-to-server case where the engine receives chunks via direct
|
||||
* HTTP POSTs — we just flip the direction so the browser pulls
|
||||
* instead of accepts.
|
||||
*/
|
||||
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||
|
||||
export interface QueueDrainerOptions {
|
||||
/**
|
||||
* Server endpoint that hosts `transferQueueRoute()`. Typically:
|
||||
* `https://server.example.com/api/v1/shade-files/queue`.
|
||||
*/
|
||||
outboundQueueUrl: string;
|
||||
/** Peer the queue is pulled FROM (the server's address). */
|
||||
peerAddress: string;
|
||||
/** Address we identify ourselves with on the long-poll. */
|
||||
senderAddress: string;
|
||||
/** Optional `fetch` override. Default `globalThis.fetch`. */
|
||||
fetch?: typeof globalThis.fetch;
|
||||
/** Extra headers applied to every poll request. */
|
||||
headers?: Record<string, string>;
|
||||
/**
|
||||
* Long-poll request timeout (server-side block). Default 30_000.
|
||||
* Server clamps to its own `maxBlockMs` (default 55_000).
|
||||
*/
|
||||
blockMs?: number;
|
||||
/**
|
||||
* Backoff after a network error before re-polling. Default 2_000.
|
||||
* Doubles up to `maxBackoffMs` on consecutive failures.
|
||||
*/
|
||||
initialBackoffMs?: number;
|
||||
maxBackoffMs?: number;
|
||||
/**
|
||||
* Called when a poll cycle fails. Defaults to logging via `console.error`.
|
||||
* Throwing from this hook does NOT stop the drainer — it backs off
|
||||
* and retries.
|
||||
*/
|
||||
onError?: (err: unknown) => void;
|
||||
}
|
||||
|
||||
export interface QueueDrainerHandle {
|
||||
/** Stop the drainer. Pending fetch is aborted; the loop exits. */
|
||||
stop(): void;
|
||||
/** Promise that resolves once the drainer has fully stopped. */
|
||||
stopped: Promise<void>;
|
||||
}
|
||||
|
||||
interface PolledEvent {
|
||||
id: number;
|
||||
timestampMs: number;
|
||||
kind: 'envelope' | 'chunk';
|
||||
bytesB64: string;
|
||||
meta?: { streamId: string; laneId: number; seq: number };
|
||||
}
|
||||
|
||||
interface PollResponseBody {
|
||||
events: PolledEvent[];
|
||||
nextSince: number;
|
||||
}
|
||||
|
||||
const DEFAULT_BLOCK_MS = 30_000;
|
||||
const DEFAULT_INITIAL_BACKOFF_MS = 2_000;
|
||||
const DEFAULT_MAX_BACKOFF_MS = 30_000;
|
||||
|
||||
/**
|
||||
* Start a long-poll loop that drains queued envelopes + chunks from
|
||||
* the server and dispatches them into the local Shade instance.
|
||||
*
|
||||
* Returns a handle the caller can use to stop the drainer when the
|
||||
* `httpClient` is closed (e.g. on tab unload).
|
||||
*/
|
||||
export function startQueueDrainer(
|
||||
shade: ShadeBridge,
|
||||
options: QueueDrainerOptions,
|
||||
): QueueDrainerHandle {
|
||||
if (shade.acceptTransferEnvelope === undefined) {
|
||||
throw new Error(
|
||||
'startQueueDrainer: shade.acceptTransferEnvelope is required for pull-mode streams. The supplied ShadeBridge implementation must surface it.',
|
||||
);
|
||||
}
|
||||
const accept = shade.acceptTransferEnvelope.bind(shade);
|
||||
const fetchFn = options.fetch ?? globalThis.fetch.bind(globalThis);
|
||||
const blockMs = options.blockMs ?? DEFAULT_BLOCK_MS;
|
||||
const onError = options.onError ?? ((err: unknown) => console.error('[shade.files queue-drainer]', err));
|
||||
const initialBackoffMs = options.initialBackoffMs ?? DEFAULT_INITIAL_BACKOFF_MS;
|
||||
const maxBackoffMs = options.maxBackoffMs ?? DEFAULT_MAX_BACKOFF_MS;
|
||||
const ac = new AbortController();
|
||||
let stopped = false;
|
||||
let resolveStopped!: () => void;
|
||||
const stoppedPromise = new Promise<void>((r) => {
|
||||
resolveStopped = r;
|
||||
});
|
||||
|
||||
void (async () => {
|
||||
let since = 0;
|
||||
let backoff = initialBackoffMs;
|
||||
while (!stopped) {
|
||||
try {
|
||||
const response = await fetchFn(options.outboundQueueUrl, {
|
||||
method: 'POST',
|
||||
signal: ac.signal,
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'X-Shade-Sender-Address': options.senderAddress,
|
||||
...(options.headers ?? {}),
|
||||
},
|
||||
body: JSON.stringify({ since, blockMs }),
|
||||
});
|
||||
if (!response.ok) {
|
||||
throw new Error(`queue poll → ${response.status} ${response.statusText}`);
|
||||
}
|
||||
const body = (await response.json()) as PollResponseBody;
|
||||
if (Array.isArray(body.events) && body.events.length > 0) {
|
||||
for (const event of body.events) {
|
||||
if (stopped) break;
|
||||
try {
|
||||
const bytes = base64ToBytes(event.bytesB64);
|
||||
await accept(options.peerAddress, bytes);
|
||||
} catch (err) {
|
||||
// Per-event dispatch failure should not kill the loop —
|
||||
// resume picks up missing chunks via @shade/transfer's
|
||||
// built-in lane-resume protocol.
|
||||
onError(err);
|
||||
}
|
||||
}
|
||||
}
|
||||
since = typeof body.nextSince === 'number' ? body.nextSince : since;
|
||||
backoff = initialBackoffMs;
|
||||
} catch (err) {
|
||||
if (stopped || ac.signal.aborted) break;
|
||||
onError(err);
|
||||
// Exponential backoff with jitter — caps at maxBackoffMs.
|
||||
const jitter = Math.floor(Math.random() * Math.min(backoff, 1_000));
|
||||
await new Promise<void>((r) => {
|
||||
const t = setTimeout(r, backoff + jitter);
|
||||
(t as unknown as { unref?: () => void }).unref?.();
|
||||
ac.signal.addEventListener(
|
||||
'abort',
|
||||
() => {
|
||||
clearTimeout(t);
|
||||
r();
|
||||
},
|
||||
{ once: true },
|
||||
);
|
||||
});
|
||||
backoff = Math.min(maxBackoffMs, backoff * 2);
|
||||
}
|
||||
}
|
||||
resolveStopped();
|
||||
})();
|
||||
|
||||
return {
|
||||
stop(): void {
|
||||
if (stopped) return;
|
||||
stopped = true;
|
||||
ac.abort(new Error('queue drainer stopped'));
|
||||
},
|
||||
stopped: stoppedPromise,
|
||||
};
|
||||
}
|
||||
|
||||
function base64ToBytes(b64: string): Uint8Array {
|
||||
const bin = atob(b64);
|
||||
const out = new Uint8Array(bin.length);
|
||||
for (let i = 0; i < bin.length; i++) out[i] = bin.charCodeAt(i);
|
||||
return out;
|
||||
}
|
||||
@@ -102,7 +102,16 @@ export async function createClientStreamsBridge(
|
||||
const readStreamId = incoming.metadata.userMetadata?.[META_KEY_READ_STREAM_ID];
|
||||
if (readStreamId === undefined) return;
|
||||
|
||||
const ts = new TransformStream<Uint8Array, Uint8Array>();
|
||||
// Generous HWM so the receiver-side write loop (drainer →
|
||||
// engine.receiveChunk → sink.write) doesn't stall on backpressure
|
||||
// before the consumer's reader is wired up. The reader still
|
||||
// applies its own backpressure once it's consuming, but we no
|
||||
// longer race fs.read's await on stream-init against the consumer
|
||||
// attaching its reader.
|
||||
const ts = new TransformStream<Uint8Array, Uint8Array>(undefined, undefined, {
|
||||
highWaterMark: 64,
|
||||
size: (chunk?: Uint8Array) => (chunk === undefined ? 0 : 1),
|
||||
});
|
||||
let handle: TransferHandle;
|
||||
try {
|
||||
handle = await incoming.accept({
|
||||
@@ -142,13 +151,14 @@ export async function createClientStreamsBridge(
|
||||
}
|
||||
|
||||
parked.set(readStreamId, arrival);
|
||||
setTimeout(() => {
|
||||
const t = setTimeout(() => {
|
||||
const stale = parked.get(readStreamId);
|
||||
if (stale === arrival) {
|
||||
parked.delete(readStreamId);
|
||||
void handle.abort('rpc-timeout').catch(() => undefined);
|
||||
}
|
||||
}, parkedReadTtlMs).unref?.();
|
||||
}, parkedReadTtlMs);
|
||||
(t as unknown as { unref?: () => void }).unref?.();
|
||||
});
|
||||
|
||||
function cleanupWaiter(w: PendingReadWaiter): void {
|
||||
|
||||
@@ -183,6 +183,24 @@ export { MAX_SIGNATURE_AGE_MS } from './server/handler.js';
|
||||
export { createFilesNamespace } from './integration/files-namespace.js';
|
||||
export type { FilesNamespace } from './integration/files-namespace.js';
|
||||
|
||||
// Request-response HTTP transport — for browser-style consumers
|
||||
// (one HTTP POST per RPC, no inbound channel needed). See
|
||||
// `docs/files.md § HTTP RPC`.
|
||||
export { createFilesRpcRoute } from './server/rpc-route.js';
|
||||
export type { FilesRpcRouteOptions } from './server/rpc-route.js';
|
||||
export { createFilesHttpClient } from './client/http-client.js';
|
||||
export type { FilesHttpClientOptions } from './client/http-client.js';
|
||||
export { startQueueDrainer } from './client/queue-drainer.js';
|
||||
export type {
|
||||
QueueDrainerHandle,
|
||||
QueueDrainerOptions,
|
||||
} from './client/queue-drainer.js';
|
||||
|
||||
// Shared structural surface @shade/files needs from a Shade instance —
|
||||
// exposed so consumers building custom Shade-shaped bridges can verify
|
||||
// they implement every required member.
|
||||
export type { ShadeBridge } from './integration/shade-bridge.js';
|
||||
|
||||
// Integration helpers — wire handler + pending registry onto a channel
|
||||
export { attachFileHandler } from './integration/wire-server.js';
|
||||
export { attachClientRouting } from './integration/wire-client.js';
|
||||
|
||||
@@ -4,13 +4,16 @@
|
||||
* so a single Shade can simultaneously serve files AND consume them from
|
||||
* peers without paying the setup cost twice.
|
||||
*/
|
||||
import type { Shade } from '@shade/sdk';
|
||||
import type { Hono } from 'hono';
|
||||
import type { ShadeBridge } from './shade-bridge.js';
|
||||
import {
|
||||
attachClientRouting,
|
||||
attachFileHandler,
|
||||
createClientStreamsBridge,
|
||||
createFileClient,
|
||||
createFileHandler,
|
||||
createFilesHttpClient,
|
||||
createFilesRpcRoute,
|
||||
createServerStreamsBridge,
|
||||
PendingRpcRegistry,
|
||||
ShadeFileRpcChannel,
|
||||
@@ -19,22 +22,79 @@ import {
|
||||
type FileClient,
|
||||
type FileHandler,
|
||||
type FileHandlerConfig,
|
||||
type FilesHttpClientOptions,
|
||||
type FilesRpcRouteOptions,
|
||||
type ServerStreamsBridge,
|
||||
} from '../index.js';
|
||||
import { IdempotencyCache } from '../server/idempotency-cache.js';
|
||||
|
||||
export interface ServeOptions {
|
||||
/**
|
||||
* Skip the streams bridge setup. Required for deployments that only
|
||||
* use the HTTP RPC route ({@link FilesNamespace.rpcRoute}) — those
|
||||
* deployments don't need to configure `@shade/transfer` because the
|
||||
* RPC route only services inline payloads (≤ 256 KiB). Without this
|
||||
* flag, `serve()` calls `createServerStreamsBridge(shade)` which
|
||||
* eagerly instantiates the transfer engine and fails when
|
||||
* `configureTransfers({ resolveBaseUrl })` has not been called.
|
||||
*
|
||||
* Default: `false` (build the streams bridge — full server-to-server
|
||||
* stack with streamed reads/writes).
|
||||
*/
|
||||
inlineOnly?: boolean;
|
||||
}
|
||||
|
||||
export interface FilesNamespace {
|
||||
/**
|
||||
* Register a file handler. Throws if a handler is already attached on
|
||||
* this Shade — only one server per Shade. The returned function detaches
|
||||
* the handler and tears down its idempotency / retention timers.
|
||||
*
|
||||
* Pass `{ inlineOnly: true }` for HTTP-RPC-only deployments to skip
|
||||
* the streams-bridge setup (and the implied `configureTransfers`
|
||||
* pre-condition).
|
||||
*/
|
||||
serve(handler: FileHandlerConfig): Promise<() => Promise<void>>;
|
||||
serve(
|
||||
handler: FileHandlerConfig,
|
||||
options?: ServeOptions,
|
||||
): Promise<() => Promise<void>>;
|
||||
/**
|
||||
* Build a typed file client for `peer`. Multiple concurrent clients to
|
||||
* different peers share the same channel + streams bridge.
|
||||
*
|
||||
* Use this for **server-to-server** deployments where both peers can
|
||||
* receive inbound HTTP. For browser clients (no inbound listener),
|
||||
* use {@link httpClient} instead.
|
||||
*/
|
||||
client(peer: string, opts?: Omit<CreateFileClientOptions, 'streamsBridge'>): Promise<FileClient>;
|
||||
/**
|
||||
* Build a request-response `FileClient` for browser-style consumers.
|
||||
* Each RPC is one HTTP POST to the supplied `rpcUrl`; the encrypted
|
||||
* response rides back in the same response body. No inbound channel
|
||||
* required on the client side.
|
||||
*
|
||||
* Inline payloads only (≤ 256 KiB). Streamed reads/writes throw a
|
||||
* clear error directing callers to {@link client} instead.
|
||||
*
|
||||
* Pre-condition: the session for `peer` must already be established
|
||||
* (typically via `shade.initSessionFromBundle(peer, bundle)`).
|
||||
*/
|
||||
httpClient(peer: string, opts: FilesHttpClientOptions): FileClient;
|
||||
/**
|
||||
* Mount the server-side request-response RPC route. Returns a Hono
|
||||
* app exposing `POST /rpc` that accepts encrypted file-RPC envelopes
|
||||
* and returns encrypted responses in the same HTTP roundtrip.
|
||||
*
|
||||
* Mount under any base path:
|
||||
* ```ts
|
||||
* app.route('/api/v1/shade-files', shade.files.rpcRoute());
|
||||
* ```
|
||||
*
|
||||
* Requires `shade.files.serve(...)` to have been called first —
|
||||
* the route dispatches incoming requests through the attached
|
||||
* handler.
|
||||
*/
|
||||
rpcRoute(opts?: FilesRpcRouteOptions): Hono;
|
||||
/** Tear down channel + bridges. After destroy(), serve()/client() throw. */
|
||||
destroy(): Promise<void>;
|
||||
}
|
||||
@@ -54,7 +114,7 @@ interface NamespaceState {
|
||||
* Construct a `FilesNamespace` bound to a Shade instance. The SDK's
|
||||
* `Shade.files` getter calls this lazily and memoizes the result.
|
||||
*/
|
||||
export function createFilesNamespace(shade: Shade): FilesNamespace {
|
||||
export function createFilesNamespace(shade: ShadeBridge): FilesNamespace {
|
||||
const state: NamespaceState = {
|
||||
channel: new ShadeFileRpcChannel(shade),
|
||||
pending: new PendingRpcRegistry(),
|
||||
@@ -71,24 +131,39 @@ export function createFilesNamespace(shade: Shade): FilesNamespace {
|
||||
}
|
||||
|
||||
return {
|
||||
async serve(handlerConfig) {
|
||||
async serve(handlerConfig, options = {}) {
|
||||
ensureAlive();
|
||||
if (state.serverHandler !== null) {
|
||||
throw new Error('FilesNamespace: a handler is already registered (one per Shade)');
|
||||
}
|
||||
// Lazy server-side streams bridge.
|
||||
if (state.serverBridge === null) {
|
||||
// Lazy server-side streams bridge — skip when the deployment is
|
||||
// HTTP-RPC-only and does not need `@shade/transfer` wired up.
|
||||
if (!options.inlineOnly && state.serverBridge === null) {
|
||||
state.serverBridge = await createServerStreamsBridge(shade);
|
||||
}
|
||||
const inheritedObservability = shade.getObservability?.();
|
||||
const handler = createFileHandler(shade, {
|
||||
...handlerConfig,
|
||||
streamsBridge: state.serverBridge,
|
||||
...(state.serverBridge !== null
|
||||
? { streamsBridge: state.serverBridge }
|
||||
: {}),
|
||||
...(handlerConfig.observability === undefined && inheritedObservability !== undefined
|
||||
? { observability: inheritedObservability }
|
||||
: {}),
|
||||
});
|
||||
const detach = attachFileHandler(state.channel, handler);
|
||||
// In inlineOnly mode, the rpc-route is the sole inbound path —
|
||||
// do NOT also subscribe the channel's onMessage handler to this
|
||||
// file handler, because that would cause every incoming request
|
||||
// to be dispatched twice (once by the rpc-route's direct call,
|
||||
// once by the channel's onMessage handler) and the channel-side
|
||||
// response would attempt an outbound POST via
|
||||
// `deliverControlEnvelope`, which is exactly the path that fails
|
||||
// for browser clients.
|
||||
const detach: () => void = options.inlineOnly
|
||||
? () => {
|
||||
/* no channel subscription to detach */
|
||||
}
|
||||
: attachFileHandler(state.channel, handler);
|
||||
state.serverHandler = handler;
|
||||
state.serverDetach = detach;
|
||||
|
||||
@@ -131,6 +206,21 @@ export function createFilesNamespace(shade: Shade): FilesNamespace {
|
||||
});
|
||||
},
|
||||
|
||||
httpClient(peer, opts) {
|
||||
ensureAlive();
|
||||
return createFilesHttpClient(shade, peer, opts);
|
||||
},
|
||||
|
||||
rpcRoute(opts = {}) {
|
||||
ensureAlive();
|
||||
if (state.serverHandler === null) {
|
||||
throw new Error(
|
||||
'FilesNamespace.rpcRoute(): no handler attached. Call shade.files.serve(...) before mounting the RPC route.',
|
||||
);
|
||||
}
|
||||
return createFilesRpcRoute(shade, state.serverHandler, opts);
|
||||
},
|
||||
|
||||
async destroy() {
|
||||
if (state.destroyed) return;
|
||||
state.destroyed = true;
|
||||
|
||||
94
packages/shade-files/src/integration/shade-bridge.ts
Normal file
94
packages/shade-files/src/integration/shade-bridge.ts
Normal file
@@ -0,0 +1,94 @@
|
||||
/**
|
||||
* Structural surface @shade/files needs from a Shade instance.
|
||||
*
|
||||
* Defining this locally — instead of `import type { Shade } from '@shade/sdk'`
|
||||
* — breaks the @shade/sdk ↔ @shade/files dependency cycle. Without this
|
||||
* break, a consumer that installs @shade/sdk from a registry ends up with
|
||||
* two distinct `Shade` classes in `node_modules` (one from
|
||||
* `@shade/sdk/node_modules/@shade/files/.../Shade`, one from
|
||||
* `@shade/sdk/Shade`). TypeScript treats them as nominally different types,
|
||||
* raising `this is not assignable to Shade` from inside SDK methods that
|
||||
* pass `this` into `createFilesNamespace`.
|
||||
*
|
||||
* The Shade class structurally implements every member listed below, so
|
||||
* `createFilesNamespace(this)` from the SDK side compiles regardless of
|
||||
* how many copies of @shade/sdk a consumer's package manager installs.
|
||||
*
|
||||
* Member signatures match Shade's exactly so this is a structural
|
||||
* subtype, not a parallel API.
|
||||
*/
|
||||
import type { ShadeEnvelope } from '@shade/core';
|
||||
import type {
|
||||
IncomingTransfer,
|
||||
TransferHandle,
|
||||
TransferOptions,
|
||||
} from '@shade/transfer';
|
||||
import type { ObservabilityHook } from '@shade/observability';
|
||||
|
||||
export interface ShadeBridge {
|
||||
/** Address that names this Shade instance to peers. */
|
||||
readonly myAddress: string;
|
||||
|
||||
/** Encrypt + send `plaintext` to `peer`; returns the wire envelope. */
|
||||
send(peer: string, plaintext: string): Promise<ShadeEnvelope>;
|
||||
|
||||
/**
|
||||
* Decrypt an inbound envelope from `peer` and return the plaintext.
|
||||
* Used by the request-response RPC route on the server side.
|
||||
*/
|
||||
receive(peer: string, envelope: ShadeEnvelope): Promise<string>;
|
||||
|
||||
/**
|
||||
* Subscribe to incoming ratchet plaintext. Returns an unsubscribe.
|
||||
* Handlers may be sync or async; async handlers are awaited in
|
||||
* registration order.
|
||||
*/
|
||||
onMessage(
|
||||
handler: (from: string, plaintext: string) => void | Promise<void>,
|
||||
): () => void;
|
||||
|
||||
/**
|
||||
* Upload bytes via the SDK's transfer engine. Required when the bridge
|
||||
* is used with `streams` content I/O (read/write > 256 KiB).
|
||||
*/
|
||||
upload(opts: TransferOptions): Promise<TransferHandle>;
|
||||
|
||||
/** Subscribe to incoming transfers initiated by a peer. */
|
||||
onIncomingTransfer(
|
||||
handler: (incoming: IncomingTransfer) => void | Promise<void>,
|
||||
): Promise<() => void>;
|
||||
|
||||
/** Fingerprint accessor for the trust-gate hooks. */
|
||||
getFingerprintFor(peer: string): Promise<string>;
|
||||
|
||||
/**
|
||||
* Optional inheritable observability bus. Files inherits the bus when
|
||||
* the SDK passes one in via the namespace; otherwise files runs without
|
||||
* observability hooks.
|
||||
*/
|
||||
getObservability?(): ObservabilityHook | undefined;
|
||||
|
||||
/** Optional control-envelope passthrough used by the WebRTC bridge. */
|
||||
deliverControlEnvelope?(peer: string, envelope: ShadeEnvelope): Promise<void>;
|
||||
|
||||
/**
|
||||
* Hand a freshly-decoded wire envelope (control or chunk) to the
|
||||
* transfer engine. Required by the pull-mode HTTP client when it
|
||||
* drains queued events from the server: each polled chunk / control
|
||||
* envelope is dispatched here so the engine sees it just as if it
|
||||
* had arrived via an HTTP POST on `/v1/transfer/...`.
|
||||
*/
|
||||
acceptTransferEnvelope?(from: string, env: ShadeEnvelope | Uint8Array): Promise<void>;
|
||||
|
||||
/**
|
||||
* Configure the transfer stack. Called by the pull-mode HTTP client
|
||||
* to point the browser's outgoing chunks + control envelopes at the
|
||||
* server's transferQueueRoute mount. Optional because the
|
||||
* server-to-server path uses a separate, app-driven configuration.
|
||||
*/
|
||||
configureTransfers?(opts: {
|
||||
resolveBaseUrl?: (peerAddress: string) => Promise<string>;
|
||||
transport?: unknown;
|
||||
envelopeTransport?: unknown;
|
||||
}): void;
|
||||
}
|
||||
126
packages/shade-files/src/protocol/rpc-builder.ts
Normal file
126
packages/shade-files/src/protocol/rpc-builder.ts
Normal file
@@ -0,0 +1,126 @@
|
||||
/**
|
||||
* Shared RPC-request construction.
|
||||
*
|
||||
* Both the channel-based `FileClient` (`createFileClient`) and the
|
||||
* HTTP-based `FilesHttpClient` build identical `RpcRequest` envelopes
|
||||
* — they differ only in *transport* (channel.send → ratchet via
|
||||
* Shade.send/onMessage vs HTTP POST → ratchet via single
|
||||
* request-response). This module is the single source of truth for
|
||||
* the wire shape so the two clients can never drift.
|
||||
*/
|
||||
import type { ZodTypeAny } from 'zod';
|
||||
import {
|
||||
KIND_DELETE_V1,
|
||||
KIND_GET_THUMBNAIL_V1,
|
||||
KIND_LIST_V1,
|
||||
KIND_MKDIR_V1,
|
||||
KIND_MOVE_V1,
|
||||
KIND_READ_V1,
|
||||
KIND_STAT_V1,
|
||||
KIND_WRITE_V1,
|
||||
MUTATION_OPS,
|
||||
type StandardOp,
|
||||
} from './kinds.js';
|
||||
import { generateIdempotencyKey, generateRequestId } from './correlate.js';
|
||||
import { canonicalRpcBytes, hashArgs } from './canonical.js';
|
||||
import type { RpcRequest } from '../schemas/envelope.js';
|
||||
|
||||
export const KIND_BY_OP: Record<StandardOp, string> = {
|
||||
list: KIND_LIST_V1,
|
||||
stat: KIND_STAT_V1,
|
||||
mkdir: KIND_MKDIR_V1,
|
||||
delete: KIND_DELETE_V1,
|
||||
move: KIND_MOVE_V1,
|
||||
read: KIND_READ_V1,
|
||||
write: KIND_WRITE_V1,
|
||||
getThumbnail: KIND_GET_THUMBNAIL_V1,
|
||||
};
|
||||
|
||||
export type SignRequest = (canonicalBytes: Uint8Array) => Promise<string> | string;
|
||||
|
||||
export interface BuildRpcRequestOptions {
|
||||
/** Address that this RPC call originates from. */
|
||||
senderAddress: string;
|
||||
/** RPC kind — `KIND_*_V1` constants for standard ops, `KIND_CUSTOM_V1` otherwise. */
|
||||
kind: string;
|
||||
/** Op classifier so mutations get an auto-generated idempotency key. */
|
||||
op: StandardOp | 'custom';
|
||||
/** Validated args object. The caller is responsible for `Zod.parse(args)`. */
|
||||
args: unknown;
|
||||
/** Caller-supplied idempotency key. Mutations get one auto-generated. */
|
||||
idempotencyKey?: string;
|
||||
/** Optional Ed25519-style signer over the canonical bytes. */
|
||||
signRequest?: SignRequest;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a single `RpcRequest` envelope. Generates `id`, `signedAt`,
|
||||
* idempotency key (mutations only), and the canonical signature.
|
||||
*/
|
||||
export async function buildRpcRequest(
|
||||
options: BuildRpcRequestOptions,
|
||||
): Promise<RpcRequest> {
|
||||
const { senderAddress, kind, op, args, signRequest } = options;
|
||||
const requestId = generateRequestId();
|
||||
const isMutation = MUTATION_OPS.has(op);
|
||||
const idempotencyKey =
|
||||
options.idempotencyKey ?? (isMutation ? generateIdempotencyKey() : undefined);
|
||||
const signedAt = Date.now();
|
||||
|
||||
let sig = 'unsigned';
|
||||
if (signRequest !== undefined) {
|
||||
const canonical = canonicalRpcBytes({
|
||||
address: senderAddress,
|
||||
signedAt,
|
||||
kind,
|
||||
id: requestId,
|
||||
argsHash: hashArgs(args),
|
||||
});
|
||||
sig = await signRequest(canonical);
|
||||
}
|
||||
|
||||
const env: RpcRequest = {
|
||||
kind,
|
||||
id: requestId,
|
||||
args,
|
||||
...(idempotencyKey !== undefined ? { idempotencyKey } : {}),
|
||||
sig,
|
||||
signedAt,
|
||||
};
|
||||
return env;
|
||||
}
|
||||
|
||||
/**
|
||||
* Helper for the typed standard ops: validates args via the supplied Zod
|
||||
* schema, calls {@link buildRpcRequest}, and forwards the result.
|
||||
*
|
||||
* The HTTP and channel clients both call this from each op-method to
|
||||
* guarantee identical wire-shape. Custom ops use {@link buildRpcRequest}
|
||||
* directly because they ship `KIND_CUSTOM_V1` plus runtime-validated args.
|
||||
*/
|
||||
export async function buildStandardRpcRequest<TArgs>(
|
||||
schema: { parse(input: unknown): TArgs },
|
||||
rawArgs: unknown,
|
||||
options: Omit<BuildRpcRequestOptions, 'kind' | 'op' | 'args'> & {
|
||||
op: StandardOp;
|
||||
},
|
||||
): Promise<{ request: RpcRequest; args: TArgs }> {
|
||||
const args = schema.parse(rawArgs) as TArgs;
|
||||
const kind = KIND_BY_OP[options.op];
|
||||
const request = await buildRpcRequest({
|
||||
...options,
|
||||
kind,
|
||||
op: options.op,
|
||||
args,
|
||||
});
|
||||
return { request, args };
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate a returned RpcResponse `result` against the supplied schema.
|
||||
* Centralised so the HTTP and channel clients fail-loudly with the same
|
||||
* error type when a server returns an off-spec response.
|
||||
*/
|
||||
export function parseResult<T>(schema: ZodTypeAny, raw: unknown): T {
|
||||
return schema.parse(raw) as T;
|
||||
}
|
||||
@@ -1,17 +1,23 @@
|
||||
import React, { createContext, useContext, useMemo } from 'react';
|
||||
import type { Shade } from '@shade/sdk';
|
||||
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||
import type { FilesNamespace } from '../integration/files-namespace.js';
|
||||
|
||||
export interface ShadeFilesContextValue {
|
||||
shade: Shade;
|
||||
shade: ShadeBridge;
|
||||
files: FilesNamespace;
|
||||
}
|
||||
|
||||
const ShadeFilesContext = createContext<ShadeFilesContextValue | null>(null);
|
||||
|
||||
export interface ShadeFilesProviderProps {
|
||||
/** Initialized `Shade` instance. `files` namespace is read off it lazily. */
|
||||
shade: Shade;
|
||||
/** Initialized `Shade` instance (or any `ShadeBridge`-shaped object). */
|
||||
shade: ShadeBridge;
|
||||
/**
|
||||
* The `FilesNamespace` to expose to children. Pass `shade.files` from
|
||||
* `@shade/sdk`, or a `createFilesNamespace(...)` result for tests /
|
||||
* custom bridges.
|
||||
*/
|
||||
files: FilesNamespace;
|
||||
children: React.ReactNode;
|
||||
}
|
||||
|
||||
@@ -20,8 +26,8 @@ export interface ShadeFilesProviderProps {
|
||||
* `<ShadeRuntimeProvider>` in `@shade/widgets` so file-RPC consumers
|
||||
* don't pull in the widget tree.
|
||||
*/
|
||||
export function ShadeFilesProvider({ shade, children }: ShadeFilesProviderProps): React.ReactElement {
|
||||
const value = useMemo<ShadeFilesContextValue>(() => ({ shade, files: shade.files }), [shade]);
|
||||
export function ShadeFilesProvider({ shade, files, children }: ShadeFilesProviderProps): React.ReactElement {
|
||||
const value = useMemo<ShadeFilesContextValue>(() => ({ shade, files }), [shade, files]);
|
||||
return React.createElement(ShadeFilesContext.Provider, { value }, children);
|
||||
}
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import type { Shade } from '@shade/sdk';
|
||||
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||
import {
|
||||
encodeEnvelope,
|
||||
looksLikeFileEnvelope,
|
||||
@@ -35,7 +35,7 @@ export class ShadeFileRpcChannel {
|
||||
private readonly unsubscribe: () => void;
|
||||
private destroyed = false;
|
||||
|
||||
constructor(private readonly shade: Shade) {
|
||||
constructor(private readonly shade: ShadeBridge) {
|
||||
this.unsubscribe = shade.onMessage(async (from, plaintext) => {
|
||||
if (!looksLikeFileEnvelope(plaintext)) return;
|
||||
const classified = tryParseEnvelope(plaintext);
|
||||
@@ -72,6 +72,11 @@ export class ShadeFileRpcChannel {
|
||||
if (this.destroyed) throw new Error('ShadeFileRpcChannel: destroyed');
|
||||
const plaintext = encodeEnvelope(envelope);
|
||||
const ratchetEnvelope = await this.shade.send(peerAddress, plaintext);
|
||||
if (this.shade.deliverControlEnvelope === undefined) {
|
||||
throw new Error(
|
||||
'ShadeFileRpcChannel: shade.deliverControlEnvelope is required — call shade.configureTransfers({ resolveBaseUrl }) before using the files namespace.',
|
||||
);
|
||||
}
|
||||
await this.shade.deliverControlEnvelope(peerAddress, ratchetEnvelope);
|
||||
}
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import type { Shade } from '@shade/sdk';
|
||||
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||
import type { StandardOp } from '../protocol/kinds.js';
|
||||
|
||||
export type OpKind = StandardOp | `custom:${string}`;
|
||||
@@ -42,7 +42,7 @@ export function buildOpContext<TArgs>(args: {
|
||||
signal: AbortSignal;
|
||||
idempotencyKey: string | undefined;
|
||||
attemptNumber: number;
|
||||
shade: Shade;
|
||||
shade: ShadeBridge;
|
||||
}): OpContext<TArgs> {
|
||||
return {
|
||||
op: args.op,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import type { Shade } from '@shade/sdk';
|
||||
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||
import type { ZodTypeAny } from 'zod';
|
||||
import {
|
||||
MUTATION_OPS,
|
||||
@@ -215,7 +215,7 @@ const OP_SCHEMAS: Record<StandardOp, OpSchemaPair> = {
|
||||
* via `Shade.files.serve(...)` in the SDK).
|
||||
*/
|
||||
export function createFileHandler(
|
||||
shade: Shade,
|
||||
shade: ShadeBridge,
|
||||
config: FileHandlerConfig,
|
||||
): FileHandler {
|
||||
const idempotency = new IdempotencyCache(config.idempotency);
|
||||
|
||||
218
packages/shade-files/src/server/rpc-route.ts
Normal file
218
packages/shade-files/src/server/rpc-route.ts
Normal file
@@ -0,0 +1,218 @@
|
||||
/**
|
||||
* Request-response RPC route for `@shade/files`.
|
||||
*
|
||||
* Mounts a single `POST /rpc` Hono endpoint that accepts an encrypted
|
||||
* `RpcRequest` envelope, dispatches it through the file handler, and
|
||||
* returns the encrypted `RpcResponse` (or `RpcError`) envelope in the
|
||||
* SAME HTTP response.
|
||||
*
|
||||
* This is the browser-friendly transport: the server never needs to
|
||||
* make outbound calls back to the client, so a browser tab — which
|
||||
* cannot host an HTTP server — can fully consume `@shade/files`.
|
||||
*
|
||||
* ### Wire contract
|
||||
*
|
||||
* Request:
|
||||
* ```
|
||||
* POST <mount>/rpc HTTP/1.1
|
||||
* Content-Type: application/octet-stream
|
||||
* X-Shade-Sender-Address: <peer address>
|
||||
*
|
||||
* <wire-encoded ShadeEnvelope (0x01 PreKeyMessage or 0x02 RatchetMessage)
|
||||
* containing JSON-encoded RpcRequest>
|
||||
* ```
|
||||
*
|
||||
* Response (success):
|
||||
* ```
|
||||
* 200 OK
|
||||
* Content-Type: application/octet-stream
|
||||
*
|
||||
* <wire-encoded ShadeEnvelope (0x02 RatchetMessage)
|
||||
* containing JSON-encoded RpcResponse | RpcError>
|
||||
* ```
|
||||
*
|
||||
* Response (transport-level failure — no session, undecryptable, etc.):
|
||||
* ```
|
||||
* 4xx
|
||||
* Content-Type: application/json
|
||||
*
|
||||
* { "error": "..." }
|
||||
* ```
|
||||
*
|
||||
* ### Symmetry with shade-auth-middleware
|
||||
*
|
||||
* The shape mirrors `@shade/server`'s shade-auth-middleware: an
|
||||
* encrypted envelope rides the request body, the server decrypts via
|
||||
* the existing ratchet session, performs the protected operation,
|
||||
* and returns an encrypted envelope in the response. No bidirectional
|
||||
* channel required.
|
||||
*
|
||||
* @see {@link createFilesHttpClient} for the matching browser client.
|
||||
*/
|
||||
import { Hono } from 'hono';
|
||||
import { decodeEnvelope, encodeEnvelope } from '@shade/proto';
|
||||
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||
import {
|
||||
encodeEnvelope as encodeRpcEnvelope,
|
||||
tryParseEnvelope,
|
||||
} from '../protocol/envelope-codec.js';
|
||||
import type { FileHandler } from './handler.js';
|
||||
import type { RpcError, RpcRequest, RpcResponse } from '../schemas/envelope.js';
|
||||
import { KIND_ERROR_V1 } from '../protocol/kinds.js';
|
||||
|
||||
export interface FilesRpcRouteOptions {
|
||||
/**
|
||||
* Maximum request body size in bytes. Default 1 MiB. Inline payloads
|
||||
* are capped at 256 KiB by the protocol; the headroom is for
|
||||
* custom-op payloads and base64 inflation.
|
||||
*/
|
||||
maxBodyBytes?: number;
|
||||
/**
|
||||
* Allow this server to accept the very first message (PreKeyMessage,
|
||||
* `0x01`) over the RPC route. Disabled by default — most browser
|
||||
* clients establish a session via `shade.initSessionFromBundle`
|
||||
* before the first RPC. Enable when you want the RPC route to also
|
||||
* be the X3DH carrier (uncommon but supported).
|
||||
*/
|
||||
acceptFirstMessage?: boolean;
|
||||
}
|
||||
|
||||
const DEFAULT_MAX_BODY_BYTES = 1 * 1024 * 1024;
|
||||
|
||||
/**
|
||||
* Build a Hono app with a single `POST /rpc` route. Mount under any
|
||||
* base path: `app.route('/api/v1/shade-files', shade.files.rpcRoute())`.
|
||||
*
|
||||
* The `handler` must already be attached (typically via
|
||||
* `shade.files.serve(handlerConfig)`); this route only ships the
|
||||
* transport — it does not register a new file handler.
|
||||
*/
|
||||
export function createFilesRpcRoute(
|
||||
shade: ShadeBridge,
|
||||
handler: FileHandler,
|
||||
options: FilesRpcRouteOptions = {},
|
||||
): Hono {
|
||||
const maxBodyBytes = options.maxBodyBytes ?? DEFAULT_MAX_BODY_BYTES;
|
||||
const app = new Hono();
|
||||
|
||||
app.post('/rpc', async (c) => {
|
||||
const senderAddress = c.req.header('X-Shade-Sender-Address');
|
||||
if (senderAddress === undefined || senderAddress === '') {
|
||||
return c.json({ error: 'missing X-Shade-Sender-Address header' }, 400);
|
||||
}
|
||||
|
||||
const contentLengthHeader = c.req.header('Content-Length');
|
||||
if (contentLengthHeader !== undefined) {
|
||||
const contentLength = Number.parseInt(contentLengthHeader, 10);
|
||||
if (Number.isFinite(contentLength) && contentLength > maxBodyBytes) {
|
||||
return c.json(
|
||||
{ error: `body exceeds maxBodyBytes (${contentLength} > ${maxBodyBytes})` },
|
||||
413,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
let bodyBytes: Uint8Array;
|
||||
try {
|
||||
const ab = await c.req.arrayBuffer();
|
||||
if (ab.byteLength > maxBodyBytes) {
|
||||
return c.json(
|
||||
{ error: `body exceeds maxBodyBytes (${ab.byteLength} > ${maxBodyBytes})` },
|
||||
413,
|
||||
);
|
||||
}
|
||||
bodyBytes = new Uint8Array(ab);
|
||||
} catch (err) {
|
||||
return c.json({ error: `failed to read request body: ${(err as Error).message}` }, 400);
|
||||
}
|
||||
|
||||
if (bodyBytes.byteLength === 0) {
|
||||
return c.json({ error: 'empty request body' }, 400);
|
||||
}
|
||||
|
||||
// Decode the wire envelope. `decodeEnvelope` handles both `0x01`
|
||||
// PreKeyMessage and `0x02` RatchetMessage shapes.
|
||||
let plaintext: string;
|
||||
try {
|
||||
const envelope = decodeEnvelope(bodyBytes);
|
||||
// First-message gate: only allow `prekey` envelopes when the
|
||||
// operator has explicitly opted in.
|
||||
if (options.acceptFirstMessage !== true && envelope.type === 'prekey') {
|
||||
return c.json(
|
||||
{
|
||||
error:
|
||||
'PreKeyMessage envelopes are not accepted on this RPC route — establish the session first via shade.initSessionFromBundle, or set acceptFirstMessage: true',
|
||||
},
|
||||
400,
|
||||
);
|
||||
}
|
||||
plaintext = await shade.receive(senderAddress, envelope);
|
||||
} catch (err) {
|
||||
// Decryption failure — could be no session, corrupted envelope,
|
||||
// or sender address mismatch. Treat as 401 since the envelope is
|
||||
// self-authenticating: a valid sender would decrypt cleanly.
|
||||
return c.json({ error: `decrypt failed: ${(err as Error).message}` }, 401);
|
||||
}
|
||||
|
||||
// Parse the plaintext as an RpcRequest.
|
||||
const classified = tryParseEnvelope(plaintext);
|
||||
if (classified === null) {
|
||||
return c.json({ error: 'plaintext is not a valid @shade/files envelope' }, 400);
|
||||
}
|
||||
if (classified.kind !== 'request') {
|
||||
// Cancel envelopes are silently dropped — RPC route is request/
|
||||
// response only. Cancellation across HTTP is achieved via
|
||||
// AbortController on the client side, not protocol-level.
|
||||
if (classified.kind === 'cancel') {
|
||||
handler.handleCancel(senderAddress, classified.envelope);
|
||||
// No response body — the cancel was best-effort.
|
||||
return new Response(null, { status: 204 });
|
||||
}
|
||||
return c.json(
|
||||
{ error: `unexpected envelope kind on RPC route: ${classified.kind}` },
|
||||
400,
|
||||
);
|
||||
}
|
||||
|
||||
const request: RpcRequest = classified.envelope;
|
||||
|
||||
// Dispatch through the file handler.
|
||||
let result: RpcResponse | RpcError;
|
||||
try {
|
||||
result = await handler.handleRequest(senderAddress, request);
|
||||
} catch (err) {
|
||||
// Should never happen — handler.handleRequest catches its own
|
||||
// errors and returns RpcError. If it didn't, that's a bug; emit
|
||||
// a generic transport-level RpcError so the client can surface
|
||||
// it deterministically.
|
||||
result = {
|
||||
kind: KIND_ERROR_V1,
|
||||
id: request.id,
|
||||
error: {
|
||||
code: 'INTERNAL',
|
||||
message: `handler raised: ${(err as Error).message}`,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// Encrypt the response and return it as wire bytes.
|
||||
let responseBytes: Uint8Array;
|
||||
try {
|
||||
const responsePlaintext = encodeRpcEnvelope(result);
|
||||
const responseEnvelope = await shade.send(senderAddress, responsePlaintext);
|
||||
responseBytes = encodeEnvelope(responseEnvelope);
|
||||
} catch (err) {
|
||||
return c.json(
|
||||
{ error: `failed to encrypt response: ${(err as Error).message}` },
|
||||
500,
|
||||
);
|
||||
}
|
||||
|
||||
return new Response(new Blob([responseBytes as unknown as ArrayBuffer]), {
|
||||
status: 200,
|
||||
headers: { 'Content-Type': 'application/octet-stream' },
|
||||
});
|
||||
});
|
||||
|
||||
return app;
|
||||
}
|
||||
@@ -168,13 +168,14 @@ export async function createServerStreamsBridge(
|
||||
|
||||
// No waiter yet — park.
|
||||
parked.set(writeId, arrived);
|
||||
setTimeout(() => {
|
||||
const parkTimer = setTimeout(() => {
|
||||
const stale = parked.get(writeId);
|
||||
if (stale === arrived) {
|
||||
parked.delete(writeId);
|
||||
void handle.abort('rpc-timeout').catch(() => undefined);
|
||||
}
|
||||
}, parkedWriteTtlMs).unref?.();
|
||||
}, parkedWriteTtlMs);
|
||||
(parkTimer as unknown as { unref?: () => void }).unref?.();
|
||||
});
|
||||
|
||||
function cleanupWaiter(w: PendingWaiter): void {
|
||||
|
||||
@@ -0,0 +1,184 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { createShade } from '@shade/sdk';
|
||||
import {
|
||||
createPrekeyServer,
|
||||
MemoryPrekeyStore,
|
||||
PrekeyServerEvents,
|
||||
} from '@shade/server';
|
||||
import { SubtleCryptoProvider } from '@shade/crypto-web';
|
||||
import { Hono } from 'hono';
|
||||
|
||||
const crypto = new SubtleCryptoProvider();
|
||||
|
||||
/**
|
||||
* Concurrent-ratchet hardening tests.
|
||||
*
|
||||
* Reproduces the scenario described in the v4.2.0 ratchet-desync bug
|
||||
* report: with the queue-drainer running on Alice and many concurrent
|
||||
* `shade.send`/RPC operations against the same peer, do
|
||||
* encrypt/decrypt paths share the per-peer mutex on
|
||||
* `ShadeSessionManager` so that no path observes a stale ratchet
|
||||
* state?
|
||||
*
|
||||
* If the lock coverage regresses (a future change re-introduces a
|
||||
* sidekanal bypass), one of these tests will fail with
|
||||
* `DecryptionError: Failed to decrypt message — wrong key or
|
||||
* tampered data`.
|
||||
*/
|
||||
async function setupPullRig() {
|
||||
const prekey = createPrekeyServer({
|
||||
crypto,
|
||||
store: new MemoryPrekeyStore(),
|
||||
disableRateLimit: true,
|
||||
events: new PrekeyServerEvents(),
|
||||
});
|
||||
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
|
||||
const prekeyUrl = `http://localhost:${prekeyServer.port}`;
|
||||
|
||||
const alice = await createShade({ prekeyServer: prekeyUrl, address: 'alice' });
|
||||
const bob = await createShade({ prekeyServer: prekeyUrl, address: 'bob' });
|
||||
|
||||
const queueRoute = await bob.transferQueueRoute({ blockMs: 500 });
|
||||
await bob.files.serve({
|
||||
stat: async () => ({
|
||||
name: '_',
|
||||
kind: 'dir' as const,
|
||||
size: 0,
|
||||
mtime: 0,
|
||||
metadata: {},
|
||||
}),
|
||||
});
|
||||
const rpcRoute = bob.files.rpcRoute({ acceptFirstMessage: true });
|
||||
|
||||
const app = new Hono();
|
||||
app.route('/', queueRoute);
|
||||
app.route('/', rpcRoute);
|
||||
const bobServer = Bun.serve({ port: 0, fetch: app.fetch });
|
||||
const baseUrl = `http://localhost:${bobServer.port}`;
|
||||
|
||||
const fs = alice.files.httpClient('bob', {
|
||||
rpcUrl: `${baseUrl}/rpc`,
|
||||
outboundQueueUrl: `${baseUrl}/queue`,
|
||||
transferBaseUrl: baseUrl,
|
||||
defaultTimeoutMs: 10_000,
|
||||
queueBlockMs: 500,
|
||||
});
|
||||
|
||||
return {
|
||||
alice,
|
||||
bob,
|
||||
fs,
|
||||
baseUrl,
|
||||
teardown: async () => {
|
||||
fs.close();
|
||||
await alice.shutdown();
|
||||
await bob.shutdown();
|
||||
bobServer.stop();
|
||||
prekeyServer.stop();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe('@shade/files — concurrent ratchet under drainer', () => {
|
||||
test('100 parallel httpClient RPCs while drainer runs — no DecryptionError', async () => {
|
||||
const rig = await setupPullRig();
|
||||
try {
|
||||
// Warm-up: establishes the X3DH session (Alice → Bob first message
|
||||
// is a PreKeyMessage; subsequent messages are pure ratchet).
|
||||
const first = await rig.fs.stat('/');
|
||||
expect(first.kind).toBe('dir');
|
||||
|
||||
// Fire 100 concurrent stat RPCs. Each one is a full ratchet
|
||||
// round-trip: encrypt request, POST, decrypt response. They all
|
||||
// contend for `manager.peerOpChains["bob"]` on Alice's side
|
||||
// (encrypt + decrypt) and `manager.peerOpChains["alice"]` on
|
||||
// Bob's side. Drainer is running in the background polling
|
||||
// Bob's queue — its decrypt path also funnels through the same
|
||||
// per-peer lock.
|
||||
// 100 concurrent — minimal repro (after warm-up only).
|
||||
const N = 100;
|
||||
const results = await Promise.allSettled(
|
||||
Array.from({ length: N }, () => rig.fs.stat('/')),
|
||||
);
|
||||
const failures = results.filter((s) => s.status === 'rejected') as Array<
|
||||
PromiseRejectedResult
|
||||
>;
|
||||
if (failures.length > 0) {
|
||||
const sample = failures.slice(0, 1).map((f) => String(f.reason));
|
||||
throw new Error(`${failures.length}/${N} concurrent RPCs failed: ${sample[0]}`);
|
||||
}
|
||||
} finally {
|
||||
await rig.teardown();
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
test('parallel shade.send + drainer + RPCs — ratchet stays in sync', async () => {
|
||||
const rig = await setupPullRig();
|
||||
try {
|
||||
// Establish session via one warm-up RPC.
|
||||
await rig.fs.stat('/');
|
||||
|
||||
// Subscribe Bob to inbound plaintext from Alice — when Alice's
|
||||
// raw `shade.send` plaintext arrives, Bob echoes a reply back
|
||||
// through `shade.send` + `deliverControlEnvelope`, which the
|
||||
// pull-mode envelope transport enqueues for Alice's drainer.
|
||||
// This injects extra inbound traffic into Alice's drainer in
|
||||
// parallel with her ongoing RPCs.
|
||||
const echoes: string[] = [];
|
||||
rig.bob.onMessage(async (from, plaintext) => {
|
||||
if (from !== 'alice') return;
|
||||
if (!plaintext.startsWith('ping:')) return;
|
||||
echoes.push(plaintext);
|
||||
const reply = `pong:${plaintext.slice('ping:'.length)}`;
|
||||
const env = await rig.bob.send('alice', reply);
|
||||
await rig.bob.deliverControlEnvelope('alice', env);
|
||||
});
|
||||
|
||||
const inboundDrained: string[] = [];
|
||||
rig.alice.onMessage((from, plaintext) => {
|
||||
if (from !== 'bob') return;
|
||||
if (plaintext.startsWith('pong:')) inboundDrained.push(plaintext);
|
||||
});
|
||||
|
||||
// Mix three concurrent workloads against the same peer:
|
||||
// - 50 inline file RPCs through httpClient (encrypt + decrypt)
|
||||
// - 50 raw `shade.send` deliveries via control envelope
|
||||
// - drainer pulling Bob's responses + echoes
|
||||
const N = 50;
|
||||
const rpcs = Array.from({ length: N }, () => rig.fs.stat('/'));
|
||||
const sends = Array.from({ length: N }, async (_, i) => {
|
||||
const env = await rig.alice.send('bob', `ping:${i}`);
|
||||
await rig.alice.deliverControlEnvelope('bob', env);
|
||||
});
|
||||
|
||||
const settled = await Promise.allSettled([...rpcs, ...sends]);
|
||||
const failures = settled.filter((s) => s.status === 'rejected') as Array<
|
||||
PromiseRejectedResult
|
||||
>;
|
||||
if (failures.length > 0) {
|
||||
const sample = failures.slice(0, 3).map((f) => String(f.reason));
|
||||
throw new Error(
|
||||
`${failures.length}/${settled.length} concurrent ops failed: ${sample.join(' | ')}`,
|
||||
);
|
||||
}
|
||||
|
||||
// Give Bob's queue + Alice's drainer a beat to drain pongs back.
|
||||
// Echoes round-trip Alice → Bob (control envelope) → Bob's
|
||||
// onMessage → Bob.send + deliver (queue) → Alice's drainer →
|
||||
// Alice's onMessage. We just verify some make it back without
|
||||
// any DecryptionError surfacing.
|
||||
const deadline = Date.now() + 5_000;
|
||||
while (inboundDrained.length < N && Date.now() < deadline) {
|
||||
await new Promise((r) => setTimeout(r, 50));
|
||||
}
|
||||
// Don't gate on every echo arriving — the long-poll cadence and
|
||||
// bun's serve/abort timing can lag a few. We only care that the
|
||||
// ratchet didn't desync; if it had, every subsequent op would
|
||||
// throw DecryptionError above.
|
||||
expect(echoes.length).toBe(N);
|
||||
expect(inboundDrained.length).toBeGreaterThan(0);
|
||||
} finally {
|
||||
await rig.teardown();
|
||||
}
|
||||
}, 30_000);
|
||||
});
|
||||
208
packages/shade-files/tests/integration/http-rpc-streams.test.ts
Normal file
208
packages/shade-files/tests/integration/http-rpc-streams.test.ts
Normal file
@@ -0,0 +1,208 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { createShade } from '@shade/sdk';
|
||||
import {
|
||||
createPrekeyServer,
|
||||
MemoryPrekeyStore,
|
||||
PrekeyServerEvents,
|
||||
} from '@shade/server';
|
||||
import { SubtleCryptoProvider } from '@shade/crypto-web';
|
||||
import { Hono } from 'hono';
|
||||
|
||||
const crypto = new SubtleCryptoProvider();
|
||||
|
||||
/**
|
||||
* Stand up the full pull-mode rig:
|
||||
* - Prekey server (for X3DH)
|
||||
* - Bob: file handler + rpcRoute + transferQueueRoute, all on one server
|
||||
* - Alice: httpClient with outboundQueueUrl + transferBaseUrl wired
|
||||
*
|
||||
* Returns Alice's `FileClient`, which speaks browser-style: ONE base URL,
|
||||
* no inbound listener, streams supported via long-poll.
|
||||
*/
|
||||
async function setupPullRig(opts: {
|
||||
bobHandler: Parameters<NonNullable<Awaited<ReturnType<typeof createShade>>['files']>['serve']>[0];
|
||||
}) {
|
||||
const prekey = createPrekeyServer({
|
||||
crypto,
|
||||
store: new MemoryPrekeyStore(),
|
||||
disableRateLimit: true,
|
||||
events: new PrekeyServerEvents(),
|
||||
});
|
||||
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
|
||||
const prekeyUrl = `http://localhost:${prekeyServer.port}`;
|
||||
|
||||
const alice = await createShade({ prekeyServer: prekeyUrl, address: 'alice' });
|
||||
const bob = await createShade({ prekeyServer: prekeyUrl, address: 'bob' });
|
||||
|
||||
// Bob: queue-route FIRST (configures bob's transports), then files.serve.
|
||||
const queueRoute = await bob.transferQueueRoute({ blockMs: 1_500 });
|
||||
await bob.files.serve(opts.bobHandler);
|
||||
const rpcRoute = bob.files.rpcRoute({ acceptFirstMessage: true });
|
||||
|
||||
const app = new Hono();
|
||||
app.route('/', queueRoute);
|
||||
app.route('/', rpcRoute);
|
||||
|
||||
const bobServer = Bun.serve({ port: 0, fetch: app.fetch });
|
||||
const baseUrl = `http://localhost:${bobServer.port}`;
|
||||
|
||||
const fs = alice.files.httpClient('bob', {
|
||||
rpcUrl: `${baseUrl}/rpc`,
|
||||
outboundQueueUrl: `${baseUrl}/queue`,
|
||||
transferBaseUrl: baseUrl,
|
||||
defaultTimeoutMs: 10_000,
|
||||
queueBlockMs: 1_000,
|
||||
});
|
||||
|
||||
return {
|
||||
alice,
|
||||
bob,
|
||||
fs,
|
||||
baseUrl,
|
||||
teardown: async () => {
|
||||
fs.close();
|
||||
await alice.shutdown();
|
||||
await bob.shutdown();
|
||||
bobServer.stop();
|
||||
prekeyServer.stop();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe('@shade/files HTTP RPC — pull-mode streams', () => {
|
||||
test('streamed read (4 MiB) via long-poll queue', async () => {
|
||||
const payload = new Uint8Array(4 * 1024 * 1024);
|
||||
for (let i = 0; i < payload.length; i++) payload[i] = (i * 97) & 0xff;
|
||||
|
||||
const rig = await setupPullRig({
|
||||
bobHandler: {
|
||||
read: async () => {
|
||||
// Return the payload as a streamed read so the rpc-handler
|
||||
// promotes it via the streams-bridge into a transfer.
|
||||
const stream = new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
const CHUNK = 256 * 1024;
|
||||
for (let off = 0; off < payload.byteLength; off += CHUNK) {
|
||||
controller.enqueue(payload.slice(off, Math.min(off + CHUNK, payload.byteLength)));
|
||||
}
|
||||
controller.close();
|
||||
},
|
||||
});
|
||||
// Need a precomputed sha256 for streamed reads. Use the
|
||||
// crypto provider's sha256 directly.
|
||||
const digest = new Uint8Array(await globalThis.crypto.subtle.digest('SHA-256', payload));
|
||||
const sha256Hex = Array.from(digest, (b) => b.toString(16).padStart(2, '0')).join('');
|
||||
return {
|
||||
kind: 'streams' as const,
|
||||
stream,
|
||||
size: payload.byteLength,
|
||||
sha256: sha256Hex,
|
||||
contentType: 'application/octet-stream',
|
||||
};
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
try {
|
||||
const result = await rig.fs.read('/big.bin');
|
||||
expect(result.kind).toBe('streams');
|
||||
if (result.kind !== 'streams') return;
|
||||
|
||||
// Drain the stream and compare.
|
||||
const reader = result.stream.getReader();
|
||||
const got = new Uint8Array(payload.byteLength);
|
||||
let offset = 0;
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (value !== undefined) {
|
||||
got.set(value, offset);
|
||||
offset += value.byteLength;
|
||||
}
|
||||
}
|
||||
reader.releaseLock();
|
||||
await result.done();
|
||||
|
||||
expect(offset).toBe(payload.byteLength);
|
||||
// Compare in 64KiB strides for speed.
|
||||
let mismatch = -1;
|
||||
for (let i = 0; i < payload.byteLength; i++) {
|
||||
if (got[i] !== payload[i]) {
|
||||
mismatch = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
expect(mismatch).toBe(-1);
|
||||
} finally {
|
||||
await rig.teardown();
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
test('streamed read fails with clear error when outboundQueueUrl is omitted', async () => {
|
||||
const rig = await setupPullRig({
|
||||
bobHandler: {
|
||||
read: async () => {
|
||||
const stream = new ReadableStream<Uint8Array>({
|
||||
start(c) {
|
||||
c.enqueue(new Uint8Array(512 * 1024));
|
||||
c.close();
|
||||
},
|
||||
});
|
||||
const digest = new Uint8Array(await globalThis.crypto.subtle.digest('SHA-256', new Uint8Array(512 * 1024)));
|
||||
const sha256Hex = Array.from(digest, (b) => b.toString(16).padStart(2, '0')).join('');
|
||||
return {
|
||||
kind: 'streams' as const,
|
||||
stream,
|
||||
size: 512 * 1024,
|
||||
sha256: sha256Hex,
|
||||
};
|
||||
},
|
||||
},
|
||||
});
|
||||
// Tear down the rig's drainer so we can construct an inline-only client
|
||||
rig.fs.close();
|
||||
|
||||
const inlineOnly = rig.alice.files.httpClient('bob', {
|
||||
rpcUrl: `${rig.baseUrl}/rpc`,
|
||||
defaultTimeoutMs: 10_000,
|
||||
});
|
||||
try {
|
||||
await expect(inlineOnly.read('/big.bin')).rejects.toThrow(/streamed read/);
|
||||
} finally {
|
||||
inlineOnly.close();
|
||||
await rig.teardown();
|
||||
}
|
||||
}, 15_000);
|
||||
|
||||
test('long-poll returns empty events on idle timeout', async () => {
|
||||
const rig = await setupPullRig({
|
||||
bobHandler: {
|
||||
stat: async () => ({
|
||||
name: '_',
|
||||
kind: 'dir' as const,
|
||||
size: 0,
|
||||
mtime: 0,
|
||||
metadata: {},
|
||||
}),
|
||||
},
|
||||
});
|
||||
try {
|
||||
// Direct poll without any pending events — should return after blockMs.
|
||||
const start = Date.now();
|
||||
const res = await fetch(`${rig.baseUrl}/queue`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'X-Shade-Sender-Address': 'alice',
|
||||
},
|
||||
body: JSON.stringify({ since: 0, blockMs: 500 }),
|
||||
});
|
||||
expect(res.status).toBe(200);
|
||||
const body = (await res.json()) as { events: unknown[]; nextSince: number };
|
||||
expect(body.events).toHaveLength(0);
|
||||
expect(Date.now() - start).toBeGreaterThanOrEqual(400);
|
||||
} finally {
|
||||
await rig.teardown();
|
||||
}
|
||||
}, 10_000);
|
||||
});
|
||||
321
packages/shade-files/tests/integration/http-rpc.test.ts
Normal file
321
packages/shade-files/tests/integration/http-rpc.test.ts
Normal file
@@ -0,0 +1,321 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { createShade } from '@shade/sdk';
|
||||
import {
|
||||
createPrekeyServer,
|
||||
MemoryPrekeyStore,
|
||||
PrekeyServerEvents,
|
||||
} from '@shade/server';
|
||||
import { SubtleCryptoProvider } from '@shade/crypto-web';
|
||||
|
||||
const crypto = new SubtleCryptoProvider();
|
||||
|
||||
/**
|
||||
* Stand up a prekey server + two Shades + Bob's file handler + RPC route
|
||||
* mounted on Bun.serve, then return Alice's HTTP-only `FileClient`.
|
||||
*
|
||||
* Mirrors the request-response setup a browser client would use against a
|
||||
* Bun-style server.
|
||||
*/
|
||||
async function setupHttpRig(opts: {
|
||||
bobHandler: Parameters<NonNullable<Awaited<ReturnType<typeof createShade>>['files']>['serve']>[0];
|
||||
}) {
|
||||
// 1. Prekey server.
|
||||
const prekeyEvents = new PrekeyServerEvents();
|
||||
const prekey = createPrekeyServer({
|
||||
crypto,
|
||||
store: new MemoryPrekeyStore(),
|
||||
disableRateLimit: true,
|
||||
events: prekeyEvents,
|
||||
});
|
||||
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
|
||||
const prekeyUrl = `http://localhost:${prekeyServer.port}`;
|
||||
|
||||
// 2. Two Shades. Alice plays the browser client (no transferRoute);
|
||||
// Bob is the server.
|
||||
const alice = await createShade({ prekeyServer: prekeyUrl, address: 'alice' });
|
||||
const bob = await createShade({ prekeyServer: prekeyUrl, address: 'bob' });
|
||||
|
||||
// 3. Bob: register file handler (HTTP-only — no streams) + mount
|
||||
// the RPC route.
|
||||
await bob.files.serve(opts.bobHandler, { inlineOnly: true });
|
||||
const rpcRoute = bob.files.rpcRoute({ acceptFirstMessage: true });
|
||||
const bobServer = Bun.serve({ port: 0, fetch: rpcRoute.fetch });
|
||||
const rpcUrl = `http://localhost:${bobServer.port}/rpc`;
|
||||
|
||||
// 5. Alice: build the HTTP-only file client.
|
||||
const fs = alice.files.httpClient('bob', { rpcUrl, defaultTimeoutMs: 5000 });
|
||||
|
||||
return {
|
||||
alice,
|
||||
bob,
|
||||
fs,
|
||||
rpcUrl,
|
||||
teardown: async () => {
|
||||
await alice.shutdown();
|
||||
await bob.shutdown();
|
||||
bobServer.stop();
|
||||
prekeyServer.stop();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe('@shade/files HTTP RPC — round-trip', () => {
|
||||
test('list → mkdir → stat → write inline → read inline → delete via httpClient', async () => {
|
||||
interface VfsEntry {
|
||||
kind: 'file' | 'dir';
|
||||
bytes?: Uint8Array;
|
||||
contentType?: string;
|
||||
}
|
||||
const vfs = new Map<string, VfsEntry>([
|
||||
['/', { kind: 'dir' }],
|
||||
['/photos', { kind: 'dir' }],
|
||||
]);
|
||||
|
||||
const rig = await setupHttpRig({
|
||||
bobHandler: {
|
||||
list: async (ctx) => {
|
||||
const prefix = ctx.path.endsWith('/') ? ctx.path : `${ctx.path}/`;
|
||||
const entries = Array.from(vfs.entries())
|
||||
.filter(([p]) => p.startsWith(prefix) && p !== ctx.path && !p.slice(prefix.length).includes('/'))
|
||||
.map(([p, e]) => ({
|
||||
name: p.slice(prefix.length) || p,
|
||||
kind: e.kind,
|
||||
size: e.bytes?.byteLength ?? 0,
|
||||
mtime: 0,
|
||||
metadata: {},
|
||||
}));
|
||||
return { entries, hasMore: false };
|
||||
},
|
||||
stat: async (ctx) => {
|
||||
const e = vfs.get(ctx.path);
|
||||
if (!e) throw new (await import('../../src/index.js')).NotFoundError(`stat ${ctx.path}`);
|
||||
return {
|
||||
name: ctx.path.split('/').pop() ?? ctx.path,
|
||||
kind: e.kind,
|
||||
size: e.bytes?.byteLength ?? 0,
|
||||
mtime: 0,
|
||||
metadata: {},
|
||||
...(e.contentType !== undefined ? { contentType: e.contentType } : {}),
|
||||
};
|
||||
},
|
||||
mkdir: async (ctx) => {
|
||||
vfs.set(ctx.path, { kind: 'dir' });
|
||||
return { entry: { name: ctx.path.split('/').pop() ?? ctx.path, kind: 'dir' as const, size: 0, mtime: 0, metadata: {} } };
|
||||
},
|
||||
delete: async (ctx) => {
|
||||
if (!vfs.has(ctx.path)) {
|
||||
throw new (await import('../../src/index.js')).NotFoundError(`delete ${ctx.path}`);
|
||||
}
|
||||
vfs.delete(ctx.path);
|
||||
return { deletedCount: 1 };
|
||||
},
|
||||
read: async (ctx) => {
|
||||
const e = vfs.get(ctx.path);
|
||||
if (!e || e.kind !== 'file' || !e.bytes) {
|
||||
throw new (await import('../../src/index.js')).NotFoundError(`read ${ctx.path}`);
|
||||
}
|
||||
// Omit sha256 — dispatcher computes it from the bytes.
|
||||
return {
|
||||
kind: 'inline' as const,
|
||||
bytes: e.bytes,
|
||||
...(e.contentType !== undefined ? { contentType: e.contentType } : {}),
|
||||
};
|
||||
},
|
||||
write: async (ctx) => {
|
||||
if (ctx.args.content.kind !== 'inline') {
|
||||
throw new (await import('../../src/index.js')).ConflictError('streams not supported in this test handler');
|
||||
}
|
||||
vfs.set(ctx.args.path, {
|
||||
kind: 'file',
|
||||
bytes: ctx.args.content.bytes,
|
||||
...(ctx.args.contentType !== undefined ? { contentType: ctx.args.contentType } : {}),
|
||||
});
|
||||
return {
|
||||
entry: {
|
||||
name: ctx.args.path.split('/').pop() ?? ctx.args.path,
|
||||
kind: 'file' as const,
|
||||
size: ctx.args.content.bytes.byteLength,
|
||||
mtime: 0,
|
||||
metadata: {},
|
||||
...(ctx.args.contentType !== undefined ? { contentType: ctx.args.contentType } : {}),
|
||||
},
|
||||
};
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
try {
|
||||
// list
|
||||
const listed = await rig.fs.list('/');
|
||||
expect(listed.entries.map((e) => e.name).sort()).toContain('photos');
|
||||
|
||||
// mkdir
|
||||
await rig.fs.mkdir('/docs');
|
||||
const stat = await rig.fs.stat('/docs');
|
||||
expect(stat.kind).toBe('dir');
|
||||
|
||||
// write inline
|
||||
const payload = new TextEncoder().encode('hello browser-friendly world');
|
||||
const writeResult = await rig.fs.write('/docs/greeting.txt', payload, {
|
||||
contentType: 'text/plain',
|
||||
});
|
||||
expect(writeResult.entry.size).toBe(payload.byteLength);
|
||||
|
||||
// read inline
|
||||
const readResult = await rig.fs.read('/docs/greeting.txt');
|
||||
expect(readResult.kind).toBe('inline');
|
||||
if (readResult.kind === 'inline') {
|
||||
expect(new TextDecoder().decode(readResult.bytes)).toBe('hello browser-friendly world');
|
||||
expect(readResult.contentType).toBe('text/plain');
|
||||
}
|
||||
|
||||
// delete
|
||||
const del = await rig.fs.delete('/docs/greeting.txt');
|
||||
expect(del.deletedCount).toBe(1);
|
||||
|
||||
// stat the deleted path → typed NotFoundError
|
||||
const { NotFoundError } = await import('../../src/index.js');
|
||||
await expect(rig.fs.stat('/docs/greeting.txt')).rejects.toBeInstanceOf(NotFoundError);
|
||||
} finally {
|
||||
await rig.teardown();
|
||||
}
|
||||
});
|
||||
|
||||
test('streamed write (> 256 KiB) is rejected with a clear error', async () => {
|
||||
const rig = await setupHttpRig({
|
||||
bobHandler: {
|
||||
write: async () => ({
|
||||
entry: { name: 'unused', kind: 'file' as const, size: 0, mtime: 0, metadata: {} },
|
||||
}),
|
||||
},
|
||||
});
|
||||
try {
|
||||
const big = new Uint8Array(257 * 1024);
|
||||
const { ConflictError } = await import('../../src/index.js');
|
||||
await expect(rig.fs.write('/big.bin', big)).rejects.toBeInstanceOf(ConflictError);
|
||||
} finally {
|
||||
await rig.teardown();
|
||||
}
|
||||
});
|
||||
|
||||
test('rpcRoute() throws when no handler is attached', async () => {
|
||||
// Don't call shade.files.serve(...) — rpcRoute() should refuse.
|
||||
const prekeyEvents = new PrekeyServerEvents();
|
||||
const prekey = createPrekeyServer({
|
||||
crypto,
|
||||
store: new MemoryPrekeyStore(),
|
||||
disableRateLimit: true,
|
||||
events: prekeyEvents,
|
||||
});
|
||||
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
|
||||
const bob = await createShade({
|
||||
prekeyServer: `http://localhost:${prekeyServer.port}`,
|
||||
address: 'bob',
|
||||
});
|
||||
try {
|
||||
expect(() => bob.files.rpcRoute()).toThrow(/no handler attached/);
|
||||
} finally {
|
||||
await bob.shutdown();
|
||||
prekeyServer.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test('missing X-Shade-Sender-Address header → 400', async () => {
|
||||
const rig = await setupHttpRig({
|
||||
bobHandler: {
|
||||
stat: async () => ({
|
||||
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
|
||||
}),
|
||||
},
|
||||
});
|
||||
try {
|
||||
const res = await fetch(rig.rpcUrl, {
|
||||
method: 'POST',
|
||||
body: new Uint8Array([0]),
|
||||
});
|
||||
expect(res.status).toBe(400);
|
||||
const body = (await res.json()) as { error: string };
|
||||
expect(body.error).toMatch(/X-Shade-Sender-Address/);
|
||||
} finally {
|
||||
await rig.teardown();
|
||||
}
|
||||
});
|
||||
|
||||
test('empty body → 400', async () => {
|
||||
const rig = await setupHttpRig({
|
||||
bobHandler: {
|
||||
stat: async () => ({
|
||||
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
|
||||
}),
|
||||
},
|
||||
});
|
||||
try {
|
||||
const res = await fetch(rig.rpcUrl, {
|
||||
method: 'POST',
|
||||
headers: { 'X-Shade-Sender-Address': 'alice' },
|
||||
});
|
||||
expect(res.status).toBe(400);
|
||||
} finally {
|
||||
await rig.teardown();
|
||||
}
|
||||
});
|
||||
|
||||
test('garbage body → 401 decrypt failure', async () => {
|
||||
const rig = await setupHttpRig({
|
||||
bobHandler: {
|
||||
stat: async () => ({
|
||||
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
|
||||
}),
|
||||
},
|
||||
});
|
||||
try {
|
||||
const res = await fetch(rig.rpcUrl, {
|
||||
method: 'POST',
|
||||
headers: { 'X-Shade-Sender-Address': 'alice' },
|
||||
body: new Uint8Array([0x02, 0xff, 0xff, 0xff]),
|
||||
});
|
||||
// 400 from envelope decode failure or 401 from decrypt failure.
|
||||
expect([400, 401]).toContain(res.status);
|
||||
} finally {
|
||||
await rig.teardown();
|
||||
}
|
||||
});
|
||||
|
||||
test('body past maxBodyBytes → 413', async () => {
|
||||
const prekeyEvents = new PrekeyServerEvents();
|
||||
const prekey = createPrekeyServer({
|
||||
crypto,
|
||||
store: new MemoryPrekeyStore(),
|
||||
disableRateLimit: true,
|
||||
events: prekeyEvents,
|
||||
});
|
||||
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
|
||||
const bob = await createShade({
|
||||
prekeyServer: `http://localhost:${prekeyServer.port}`,
|
||||
address: 'bob',
|
||||
});
|
||||
await bob.files.serve(
|
||||
{
|
||||
stat: async () => ({
|
||||
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
|
||||
}),
|
||||
},
|
||||
{ inlineOnly: true },
|
||||
);
|
||||
const route = bob.files.rpcRoute({ maxBodyBytes: 1024 });
|
||||
const server = Bun.serve({ port: 0, fetch: route.fetch });
|
||||
try {
|
||||
const big = new Uint8Array(2048);
|
||||
const res = await fetch(`http://localhost:${server.port}/rpc`, {
|
||||
method: 'POST',
|
||||
headers: { 'X-Shade-Sender-Address': 'alice' },
|
||||
body: big,
|
||||
});
|
||||
expect(res.status).toBe(413);
|
||||
} finally {
|
||||
await bob.shutdown();
|
||||
server.stop();
|
||||
prekeyServer.stop();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shade/inbox-server",
|
||||
"version": "4.0.0",
|
||||
"version": "4.11.0",
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
"types": "src/index.ts",
|
||||
|
||||
268
packages/shade-inbox-server/src/blob-routes.ts
Normal file
268
packages/shade-inbox-server/src/blob-routes.ts
Normal file
@@ -0,0 +1,268 @@
|
||||
import { Hono } from 'hono';
|
||||
import type { CryptoProvider } from '@shade/core';
|
||||
import {
|
||||
errorToHttpStatus,
|
||||
ShadeError,
|
||||
ValidationError,
|
||||
UnauthorizedError,
|
||||
fromBase64,
|
||||
toBase64,
|
||||
constantTimeEqual,
|
||||
} from '@shade/core';
|
||||
import {
|
||||
verifyPayload,
|
||||
RateLimiter,
|
||||
MemoryRateLimitStore,
|
||||
type RateLimitConfig,
|
||||
} from '@shade/server';
|
||||
import {
|
||||
ATTR_ERROR_CODE,
|
||||
ATTR_HTTP_STATUS,
|
||||
ATTR_ROUTE,
|
||||
NOOP_HOOK,
|
||||
type ObservabilityHook,
|
||||
} from '@shade/observability';
|
||||
import type { BlobStore } from './blob-store.js';
|
||||
|
||||
/**
|
||||
* Wire-level wrapper around the V4.9 BlobStore primitive.
|
||||
*
|
||||
* Endpoints:
|
||||
* GET /v1/blob/:slotId → { blob, etag } | 404
|
||||
* PUT /v1/blob/:slotId → { etag, created } | 409 | 412
|
||||
* DELETE /v1/blob/:slotId → { ok }
|
||||
*
|
||||
* SlotId is 64 lowercase hex chars (the HKDF output, 32 bytes). Payloads
|
||||
* are base64-encoded ciphertext; the relay never decrypts. Auth uses
|
||||
* `signPayload` / `verifyPayload` (same canonical-JSON-and-Ed25519
|
||||
* scheme as the inbox routes), keyed off the per-slot pubkey stored
|
||||
* TOFU on the first PUT.
|
||||
*
|
||||
* Quota: a single slot holds one blob. `MAX_BLOB_BYTES` (64 KiB) is
|
||||
* sized for Prism's profile use-case (a few hundred host entries) with
|
||||
* plenty of headroom; future apps can override via `BlobRoutesOptions`.
|
||||
*/
|
||||
const SLOT_ID_REGEX = /^[0-9a-f]{64}$/;
|
||||
const MAX_META_BODY_SIZE = 64 * 1024;
|
||||
/** Default per-slot blob ceiling. Sized for ~500 host entries in JSON form. */
|
||||
export const DEFAULT_MAX_BLOB_BYTES = 64 * 1024;
|
||||
|
||||
const PUT_LIMIT: RateLimitConfig = { capacity: 60, refillPerSecond: 1 };
|
||||
const GET_LIMIT: RateLimitConfig = { capacity: 120, refillPerSecond: 2 };
|
||||
const DELETE_LIMIT: RateLimitConfig = { capacity: 30, refillPerSecond: 1 };
|
||||
|
||||
export interface BlobRoutesOptions {
|
||||
disableRateLimit?: boolean;
|
||||
observability?: ObservabilityHook;
|
||||
/** Per-blob byte ceiling. Defaults to 64 KiB. */
|
||||
maxBlobBytes?: number;
|
||||
}
|
||||
|
||||
export function createBlobRoutes(
|
||||
store: BlobStore,
|
||||
crypto: CryptoProvider,
|
||||
options: BlobRoutesOptions = {},
|
||||
): Hono {
|
||||
const app = new Hono();
|
||||
const observability = options.observability ?? NOOP_HOOK;
|
||||
const maxBlobBytes = options.maxBlobBytes ?? DEFAULT_MAX_BLOB_BYTES;
|
||||
|
||||
app.use('*', async (c, next) => {
|
||||
const route = c.req.routePath ?? c.req.path ?? '<unknown>';
|
||||
const span = observability.startSpan('shade.blob.request', {
|
||||
[ATTR_ROUTE]: route,
|
||||
});
|
||||
try {
|
||||
await next();
|
||||
span.setAttribute(ATTR_HTTP_STATUS, c.res.status);
|
||||
span.setStatus(c.res.status >= 500 ? 'error' : 'ok');
|
||||
} catch (err) {
|
||||
const code =
|
||||
err instanceof ShadeError ? err.code ?? 'SHADE_ERROR' : 'SHADE_INTERNAL';
|
||||
span.setAttribute(ATTR_ERROR_CODE, code);
|
||||
span.recordException(err);
|
||||
span.setStatus('error', code);
|
||||
throw err;
|
||||
} finally {
|
||||
span.end();
|
||||
}
|
||||
});
|
||||
|
||||
const rlStore = new MemoryRateLimitStore();
|
||||
const putRL = new RateLimiter(rlStore, PUT_LIMIT);
|
||||
const getRL = new RateLimiter(rlStore, GET_LIMIT);
|
||||
const deleteRL = new RateLimiter(rlStore, DELETE_LIMIT);
|
||||
const rateLimitEnabled = !options.disableRateLimit;
|
||||
|
||||
const getClientIp = (c: any): string =>
|
||||
c.req.header('x-forwarded-for')?.split(',')[0]?.trim() ??
|
||||
c.req.header('x-real-ip') ??
|
||||
'unknown';
|
||||
|
||||
app.onError((err, c) => {
|
||||
if (err instanceof ShadeError) {
|
||||
const status = errorToHttpStatus(err);
|
||||
const body: any = err.toJSON();
|
||||
if ((err as any).retryAfterSeconds) {
|
||||
c.header('Retry-After', String((err as any).retryAfterSeconds));
|
||||
}
|
||||
return c.json(body, status as any);
|
||||
}
|
||||
console.error('[Shade] Unhandled blob error:', err);
|
||||
return c.json({ error: 'Internal server error' }, 500);
|
||||
});
|
||||
|
||||
function validateSlotId(raw: string | undefined): string {
|
||||
if (typeof raw !== 'string' || !SLOT_ID_REGEX.test(raw)) {
|
||||
throw new ValidationError(
|
||||
'slotId must be 64 lowercase hex chars (32 bytes)',
|
||||
'slotId',
|
||||
);
|
||||
}
|
||||
return raw;
|
||||
}
|
||||
|
||||
// ─── GET ─────────────────────────────────────────────────────
|
||||
// Unauthenticated. SlotId is itself a 256-bit secret derived from the
|
||||
// master key — knowing it implies you derived the master, which is
|
||||
// equivalent to holding the credentials. The blob is AEAD-sealed, so
|
||||
// a relay-side leak of slotId still cannot decrypt the contents.
|
||||
app.get('/v1/blob/:slotId', async (c) => {
|
||||
const slotId = validateSlotId(c.req.param('slotId'));
|
||||
if (rateLimitEnabled) await getRL.consume(`blob-get:${getClientIp(c)}`);
|
||||
|
||||
const row = await store.get(slotId);
|
||||
if (!row) {
|
||||
return c.json({ error: 'Slot not found', code: 'SHADE_NOT_FOUND' }, 404);
|
||||
}
|
||||
return c.json({
|
||||
blob: toBase64(row.blob),
|
||||
etag: String(row.etag),
|
||||
updatedAt: row.updatedAt,
|
||||
});
|
||||
});
|
||||
|
||||
// ─── PUT ─────────────────────────────────────────────────────
|
||||
// Body format:
|
||||
// {
|
||||
// ownerPubkey: b64, // Ed25519 pubkey deterministically
|
||||
// // derived from the master via HKDF.
|
||||
// blob: b64,
|
||||
// ifMatch?: string, // "<etag>" | "*" | undefined
|
||||
// signedAt: number,
|
||||
// signature: b64 // over the canonical body sans signature
|
||||
// }
|
||||
//
|
||||
// First write to a slot is TOFU: we record `ownerPubkey` and require
|
||||
// any future write to verify against it. A different key trying to
|
||||
// overwrite an existing slot is rejected with UnauthorizedError.
|
||||
app.put('/v1/blob/:slotId', async (c) => {
|
||||
const slotId = validateSlotId(c.req.param('slotId'));
|
||||
if (rateLimitEnabled) await putRL.consume(`blob-put:${getClientIp(c)}`);
|
||||
|
||||
const rawBody = await c.req.text();
|
||||
const hardLimit = Math.ceil(maxBlobBytes * 1.4) + MAX_META_BODY_SIZE;
|
||||
if (rawBody.length > hardLimit) {
|
||||
throw new ValidationError(`Request body too large`);
|
||||
}
|
||||
const body = JSON.parse(rawBody);
|
||||
const { ownerPubkey, blob, ifMatch } = body;
|
||||
|
||||
if (typeof ownerPubkey !== 'string') {
|
||||
throw new ValidationError('Missing ownerPubkey', 'ownerPubkey');
|
||||
}
|
||||
if (typeof blob !== 'string') {
|
||||
throw new ValidationError('Missing blob', 'blob');
|
||||
}
|
||||
const claimedKey = fromBase64(ownerPubkey);
|
||||
if (claimedKey.length !== 32) {
|
||||
throw new ValidationError('ownerPubkey must be 32 bytes (Ed25519)', 'ownerPubkey');
|
||||
}
|
||||
const blobBytes = fromBase64(blob);
|
||||
if (blobBytes.length === 0) {
|
||||
throw new ValidationError('blob is empty', 'blob');
|
||||
}
|
||||
if (blobBytes.length > maxBlobBytes) {
|
||||
throw new ValidationError(
|
||||
`blob exceeds maxBlobBytes (${blobBytes.length} > ${maxBlobBytes})`,
|
||||
'blob',
|
||||
);
|
||||
}
|
||||
|
||||
let expectedEtag: number | '*' | undefined;
|
||||
if (ifMatch === undefined) {
|
||||
expectedEtag = undefined;
|
||||
} else if (typeof ifMatch !== 'string') {
|
||||
throw new ValidationError('ifMatch must be a string when present', 'ifMatch');
|
||||
} else if (ifMatch === '*') {
|
||||
expectedEtag = '*';
|
||||
} else {
|
||||
const n = Number(ifMatch);
|
||||
if (!Number.isFinite(n) || !Number.isInteger(n) || n < 0) {
|
||||
throw new ValidationError('ifMatch must be a non-negative integer or "*"', 'ifMatch');
|
||||
}
|
||||
expectedEtag = n;
|
||||
}
|
||||
|
||||
// Existing slot: caller must sign with the original owner key. Use
|
||||
// the stored pubkey for verification. The body's `ownerPubkey` is
|
||||
// bound by the signature too, so an attacker cannot trick us into
|
||||
// verifying with a key they control — the canonicalization includes
|
||||
// every field but `signature`.
|
||||
const existing = await store.get(slotId);
|
||||
const verifyKey = existing ? existing.ownerPubkey : claimedKey;
|
||||
|
||||
// Bind slotId into the signed payload so a signature for slot A
|
||||
// can't be replayed against slot B (the URL is otherwise outside
|
||||
// the signed bytes).
|
||||
await verifyPayload(crypto, verifyKey, { ...body, slotId });
|
||||
|
||||
if (existing && !constantTimeEqual(existing.ownerPubkey, claimedKey)) {
|
||||
throw new UnauthorizedError(
|
||||
`Slot ${slotId} is owned by a different signing key`,
|
||||
);
|
||||
}
|
||||
|
||||
const result = await store.put({
|
||||
slotId,
|
||||
blob: blobBytes,
|
||||
ownerPubkey: claimedKey,
|
||||
expectedEtag,
|
||||
now: Date.now(),
|
||||
});
|
||||
|
||||
return c.json({
|
||||
ok: true,
|
||||
created: result.created,
|
||||
etag: String(result.etag),
|
||||
updatedAt: result.updatedAt,
|
||||
});
|
||||
});
|
||||
|
||||
// ─── DELETE ──────────────────────────────────────────────────
|
||||
// Body format: { signedAt, signature }. Signed by the owner pubkey
|
||||
// recorded on the first PUT. After deletion, the slot is fully gone —
|
||||
// the next PUT TOFU-claims it again (potentially under a different
|
||||
// signing key, e.g. after a rotation).
|
||||
app.delete('/v1/blob/:slotId', async (c) => {
|
||||
const slotId = validateSlotId(c.req.param('slotId'));
|
||||
if (rateLimitEnabled) await deleteRL.consume(`blob-delete:${getClientIp(c)}`);
|
||||
|
||||
const existing = await store.get(slotId);
|
||||
if (!existing) {
|
||||
return c.json({ error: 'Slot not found', code: 'SHADE_NOT_FOUND' }, 404);
|
||||
}
|
||||
|
||||
const rawBody = await c.req.text();
|
||||
if (rawBody.length > MAX_META_BODY_SIZE) {
|
||||
throw new ValidationError(`Request body too large`);
|
||||
}
|
||||
const body = JSON.parse(rawBody);
|
||||
await verifyPayload(crypto, existing.ownerPubkey, { ...body, slotId });
|
||||
|
||||
const removed = await store.delete(slotId);
|
||||
return c.json({ ok: removed });
|
||||
});
|
||||
|
||||
return app;
|
||||
}
|
||||
86
packages/shade-inbox-server/src/blob-store.ts
Normal file
86
packages/shade-inbox-server/src/blob-store.ts
Normal file
@@ -0,0 +1,86 @@
|
||||
/**
|
||||
* BlobStore — server-side storage interface for the V4.9 encrypted-blob
|
||||
* primitive. A "slot" is a single AEAD-sealed blob keyed by a
|
||||
* deterministic 32-byte slotId derived client-side via HKDF from a
|
||||
* master key. The relay never sees plaintext, never holds private keys,
|
||||
* and never decrypts.
|
||||
*
|
||||
* Auth model (TOFU per slot, mirrors the inbox-owner pattern):
|
||||
* - First PUT to an empty slot stores the caller's Ed25519 signing
|
||||
* pubkey alongside the blob. Subsequent writes must produce a valid
|
||||
* signature verifiable by that pubkey.
|
||||
* - GET is unauthenticated — slotId is itself a 256-bit secret derived
|
||||
* from the master key, so knowing it implies you derived the master.
|
||||
* - DELETE clears the blob AND the owner pubkey, allowing future TOFU
|
||||
* re-claim by a fresh signing key derived from the same master (e.g.
|
||||
* after a rotation).
|
||||
*
|
||||
* CAS / etag semantics:
|
||||
* - Every successful PUT bumps a per-slot monotonic etag (returned to
|
||||
* the caller as a string).
|
||||
* - A stale `ifMatch` triggers `PreconditionFailedError` (HTTP 412).
|
||||
* - `ifMatch === undefined` against a populated slot triggers
|
||||
* `ConflictError` (HTTP 409) — clients must read-then-write.
|
||||
* - `ifMatch === '*'` against a populated slot is unconditional
|
||||
* overwrite (escape hatch). Against an empty slot it's still 412
|
||||
* per RFC 7232 (no entity to match).
|
||||
*/
|
||||
export interface BlobSlotRecord {
|
||||
/** Lower-hex 64-char slotId (32 bytes). */
|
||||
slotId: string;
|
||||
/** Raw AEAD ciphertext (bytes). The relay never decrypts. */
|
||||
blob: Uint8Array;
|
||||
/** Owner Ed25519 signing pubkey, established TOFU on the first PUT. */
|
||||
ownerPubkey: Uint8Array;
|
||||
/** Monotonic per-slot version. Used as the ETag on the wire. */
|
||||
etag: number;
|
||||
/** Wall-clock ms of the last successful write. */
|
||||
updatedAt: number;
|
||||
}
|
||||
|
||||
/** Returned to the route layer after a successful PUT. */
|
||||
export interface PutBlobResult {
|
||||
/** Whether the slot was created (true) or updated in place (false). */
|
||||
created: boolean;
|
||||
/** New etag after the write. */
|
||||
etag: number;
|
||||
/** Wall-clock ms of the write. */
|
||||
updatedAt: number;
|
||||
}
|
||||
|
||||
export interface BlobStore {
|
||||
/** Read a slot, or null if it has never been written (or was deleted). */
|
||||
get(slotId: string): Promise<BlobSlotRecord | null>;
|
||||
|
||||
/**
|
||||
* Create or update a slot.
|
||||
*
|
||||
* Implementations MUST treat `(slotId, ownerPubkey)` atomically: the
|
||||
* route layer has already verified the signature, but the store is the
|
||||
* authority on whether the slot exists and what etag it has. Callers
|
||||
* pass the verified `ownerPubkey` (used on first-write to record the
|
||||
* owner; ignored on subsequent writes — the existing pubkey is the
|
||||
* source of truth for who's allowed to write).
|
||||
*
|
||||
* `expectedEtag` semantics (mirror the wire-level If-Match):
|
||||
* - `undefined` : create-only. Slot must be empty.
|
||||
* - `<number>` : compare-and-swap. Must equal the current etag.
|
||||
* - `'*'` : unconditional overwrite. Slot must already exist.
|
||||
*
|
||||
* On precondition mismatch the store throws `PreconditionFailedError`
|
||||
* (stale etag) or `ConflictError` (slot exists, no ifMatch).
|
||||
*/
|
||||
put(args: {
|
||||
slotId: string;
|
||||
blob: Uint8Array;
|
||||
ownerPubkey: Uint8Array;
|
||||
expectedEtag: number | '*' | undefined;
|
||||
now: number;
|
||||
}): Promise<PutBlobResult>;
|
||||
|
||||
/**
|
||||
* Delete a slot. Authentication has already been checked by the route
|
||||
* layer. Returns true if a row was removed (i.e. the slot existed).
|
||||
*/
|
||||
delete(slotId: string): Promise<boolean>;
|
||||
}
|
||||
140
packages/shade-inbox-server/src/bridge-delivery-log.ts
Normal file
140
packages/shade-inbox-server/src/bridge-delivery-log.ts
Normal file
@@ -0,0 +1,140 @@
|
||||
/**
|
||||
* BridgeDeliveryLog — V4.8.4 cross-channel dedup.
|
||||
*
|
||||
* Records per-`(address, msgId)` "delivered via bridge push" timestamps so
|
||||
* the inbox-fetch route can filter out blobs the bridge has already pushed
|
||||
* to the recipient. The relay's contract becomes:
|
||||
*
|
||||
* one `Inbox.send` ⇒ one observable delivery on the recipient
|
||||
*
|
||||
* even when the recipient runs both a bridge subscription (WS / SSE) AND
|
||||
* the regular inbox-poll. Without it, bridge-push and inbox-poll are
|
||||
* independent paths against the same store and the recipient gets the
|
||||
* same envelope twice — bridge-first, then ~30 s later via the next poll
|
||||
* — tripping on already-consumed prekeys (`one-time prekey not found`)
|
||||
* or surfacing as duplicate `shade.receive` work.
|
||||
*
|
||||
* The log is in-memory per process and intentionally bounded: each entry
|
||||
* lives for `graceMs` (default 60 s, well past a typical `pollIntervalMs`
|
||||
* of 30 s). After grace, the entry is forgotten and inbox-poll falls back
|
||||
* to delivering the blob — that's the legitimate "bridge dropped the
|
||||
* frame, poll picked up" recovery path. If the recipient explicitly
|
||||
* acks the blob (HTTP `DELETE /v1/inbox/:addr/:msgId`), the blob is gone
|
||||
* from storage and the log entry is moot.
|
||||
*
|
||||
* Multi-bridge per address (e.g. WS + SSE redundancy on the same client,
|
||||
* or two devices sharing one signing key) is preserved: every bridge
|
||||
* connection still fetches + pushes the blob — each push records its own
|
||||
* timestamp — so each connected bridge gets the frame. Only the *poll*
|
||||
* fetch is filtered, not the bridge fetches themselves.
|
||||
*
|
||||
* @see Prism FR `cross-channel-duplicate-fanout-v4.8.2.md`.
|
||||
*/
|
||||
|
||||
const DEFAULT_GRACE_MS = 60_000;
|
||||
|
||||
export interface BridgeDeliveryLogOptions {
|
||||
/**
|
||||
* How long a `(address, msgId)` mark suppresses inbox-poll delivery.
|
||||
* Defaults to 60_000ms — twice the default `pollIntervalMs` of the
|
||||
* `@shade/inbox` orchestrator, so a poll cycle that races a bridge
|
||||
* push always sees the mark, but a stuck recipient still gets the
|
||||
* blob via poll within ~minutes.
|
||||
*/
|
||||
graceMs?: number;
|
||||
/**
|
||||
* Maximum entries per address. Bounds memory under a busy address.
|
||||
* Oldest entries (by recorded timestamp) are evicted first. Default
|
||||
* 8192 — comfortably above any realistic backlog.
|
||||
*/
|
||||
maxPerAddress?: number;
|
||||
}
|
||||
|
||||
export class BridgeDeliveryLog {
|
||||
private readonly log = new Map<string, Map<string, number>>();
|
||||
private readonly graceMs: number;
|
||||
private readonly maxPerAddress: number;
|
||||
|
||||
constructor(options: BridgeDeliveryLogOptions = {}) {
|
||||
this.graceMs = options.graceMs ?? DEFAULT_GRACE_MS;
|
||||
this.maxPerAddress = options.maxPerAddress ?? 8192;
|
||||
}
|
||||
|
||||
/** Mark `(address, msgId)` as bridge-delivered at `now`. */
|
||||
recordDelivered(address: string, msgId: string, now: number): void {
|
||||
let inner = this.log.get(address);
|
||||
if (!inner) {
|
||||
inner = new Map();
|
||||
this.log.set(address, inner);
|
||||
}
|
||||
inner.set(msgId, now);
|
||||
// Lazy cleanup: drop entries past 2× grace so the map stays bounded
|
||||
// without a separate timer. Bound by `maxPerAddress` as a fallback
|
||||
// for pathological burst scenarios.
|
||||
if (inner.size > this.maxPerAddress) {
|
||||
const cutoff = now - this.graceMs * 2;
|
||||
for (const [id, ts] of inner) {
|
||||
if (ts < cutoff) inner.delete(id);
|
||||
}
|
||||
// Still over cap? Drop the oldest.
|
||||
if (inner.size > this.maxPerAddress) {
|
||||
const sorted = Array.from(inner.entries()).sort((a, b) => a[1] - b[1]);
|
||||
const toDrop = sorted.slice(0, inner.size - this.maxPerAddress);
|
||||
for (const [id] of toDrop) inner.delete(id);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns true if `(address, msgId)` was bridge-delivered within the
|
||||
* grace window.
|
||||
*/
|
||||
isRecentlyDelivered(address: string, msgId: string, now: number): boolean {
|
||||
const inner = this.log.get(address);
|
||||
if (!inner) return false;
|
||||
const ts = inner.get(msgId);
|
||||
if (ts === undefined) return false;
|
||||
if (now - ts > this.graceMs) {
|
||||
inner.delete(msgId); // tombstone the stale entry
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Filter `blobs` down to those not currently in the bridge-delivered
|
||||
* grace window. Used by the inbox-fetch route to suppress duplicates.
|
||||
*/
|
||||
filterRecent<T extends { msgId: string }>(
|
||||
address: string,
|
||||
blobs: T[],
|
||||
now: number,
|
||||
): T[] {
|
||||
const inner = this.log.get(address);
|
||||
if (!inner || inner.size === 0) return blobs;
|
||||
return blobs.filter((b) => {
|
||||
const ts = inner.get(b.msgId);
|
||||
if (ts === undefined) return true;
|
||||
if (now - ts > this.graceMs) {
|
||||
inner.delete(b.msgId);
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
});
|
||||
}
|
||||
|
||||
/** Drop the entry for `(address, msgId)`. Called from blob-delete paths. */
|
||||
forget(address: string, msgId: string): void {
|
||||
this.log.get(address)?.delete(msgId);
|
||||
}
|
||||
|
||||
/** Drop every entry for `address`. Called from address-delete paths. */
|
||||
forgetAddress(address: string): void {
|
||||
this.log.delete(address);
|
||||
}
|
||||
|
||||
/** Test-only inspection. */
|
||||
size(address: string): number {
|
||||
return this.log.get(address)?.size ?? 0;
|
||||
}
|
||||
}
|
||||
@@ -38,8 +38,16 @@ import {
|
||||
import { verifyPayload, validateAddress } from '@shade/server';
|
||||
import type { InboxStore } from './store.js';
|
||||
import type { InboxServerEvents } from './events.js';
|
||||
import { PresenceTracker, type TrackedBridgeKind } from './presence.js';
|
||||
import { BridgeDeliveryLog } from './bridge-delivery-log.js';
|
||||
|
||||
export type BridgeKind = 'stream' | 'poll' | 'ws';
|
||||
/**
|
||||
* Wire-protocol kind tag for `/v1/bridge/presence`. Distinct from
|
||||
* `BridgeKind` because the canonical signed payload is shaped
|
||||
* differently (`watched: string[]` instead of `since: number`).
|
||||
*/
|
||||
export type PresenceKind = 'presence';
|
||||
|
||||
export interface BridgeRoutesOptions {
|
||||
store: InboxStore;
|
||||
@@ -60,6 +68,23 @@ export interface BridgeRoutesOptions {
|
||||
* Default 1_000.
|
||||
*/
|
||||
fallbackPollIntervalMs?: number;
|
||||
/**
|
||||
* Inject an existing presence tracker. Useful when multiple
|
||||
* `createBridgeRoutes` calls need to share state (e.g. mounting the
|
||||
* routes under several hostnames in a single process). When omitted,
|
||||
* the bridge auto-creates an internal tracker bound to `events`.
|
||||
*/
|
||||
presenceTracker?: PresenceTracker;
|
||||
/**
|
||||
* V4.8.4 — shared bridge delivery log. After every successful WS /
|
||||
* SSE push we record `(address, msgId, now)` here so the inbox-fetch
|
||||
* route can suppress the same blob from a subsequent inbox-poll
|
||||
* within the log's grace window. Pass the same instance to
|
||||
* `createInboxRoutes` (or use the auto-created one returned in
|
||||
* `bridgeRoutes.bridgeDeliveryLog`). When omitted, the bridge
|
||||
* auto-creates its own log.
|
||||
*/
|
||||
bridgeDeliveryLog?: BridgeDeliveryLog;
|
||||
}
|
||||
|
||||
interface VerifiedBridgeRequest {
|
||||
@@ -68,6 +93,13 @@ interface VerifiedBridgeRequest {
|
||||
since: number;
|
||||
}
|
||||
|
||||
interface VerifiedPresenceRequest {
|
||||
/** The watcher's address (signer of the request). */
|
||||
address: string;
|
||||
/** Addresses whose presence the watcher is asking to track. */
|
||||
watched: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the bridge Hono router and a paired Bun-WebSocket handler.
|
||||
*
|
||||
@@ -80,6 +112,15 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
|
||||
app: Hono;
|
||||
/** Pass to `Bun.serve({ websocket })`. Undefined if Bun adapter is missing. */
|
||||
websocket: unknown;
|
||||
/** Live presence tracker. Tests + observers can read it; routes update it. */
|
||||
presence: PresenceTracker;
|
||||
/**
|
||||
* V4.8.4 — the shared bridge-delivery log this router writes to on
|
||||
* every successful push. Wire the same instance into
|
||||
* `createInboxRoutes({ bridgeDeliveryLog })` so the inbox-fetch route
|
||||
* can suppress recently-pushed blobs.
|
||||
*/
|
||||
bridgeDeliveryLog: BridgeDeliveryLog;
|
||||
} {
|
||||
const app = new Hono();
|
||||
const pageLimit = opts.pageLimit ?? 50;
|
||||
@@ -87,6 +128,8 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
|
||||
const longPollDefault = opts.longPollTimeoutMs ?? 25_000;
|
||||
const longPollMax = opts.longPollMaxTimeoutMs ?? 55_000;
|
||||
const fallbackPollIntervalMs = opts.fallbackPollIntervalMs ?? 1_000;
|
||||
const presence = opts.presenceTracker ?? new PresenceTracker(opts.events ?? null);
|
||||
const bridgeDeliveryLog = opts.bridgeDeliveryLog ?? new BridgeDeliveryLog();
|
||||
|
||||
app.onError((err, c) => {
|
||||
if (err instanceof ShadeError) {
|
||||
@@ -102,17 +145,37 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
|
||||
const verified = await verifyBridgeAuth(c, opts, 'stream');
|
||||
return streamSSE(c, async (stream) => {
|
||||
const address = verified.address;
|
||||
const connId = presence.newConnectionId();
|
||||
presence.markConnected(address, 'sse', connId);
|
||||
let presenceClosed = false;
|
||||
const closePresence = (reason: 'closed' | 'error'): void => {
|
||||
if (presenceClosed) return;
|
||||
presenceClosed = true;
|
||||
presence.markDisconnected(address, 'sse', connId, reason);
|
||||
};
|
||||
let cursor = verified.since;
|
||||
const writer = makeBlobWriter(opts.store, pageLimit);
|
||||
const delivered = new DeliveredIdLru();
|
||||
|
||||
const recordPush = (msgId: string): void => {
|
||||
bridgeDeliveryLog.recordDelivered(address, msgId, Date.now());
|
||||
};
|
||||
|
||||
// Initial backlog drain.
|
||||
const flushed = await flushTo(writer, address, cursor, async (blob) => {
|
||||
await stream.writeSSE({
|
||||
id: String(blob.receivedAt),
|
||||
event: 'envelope',
|
||||
data: JSON.stringify(serializeBlob(blob)),
|
||||
});
|
||||
});
|
||||
const flushed = await flushTo(
|
||||
writer,
|
||||
address,
|
||||
cursor,
|
||||
async (blob) => {
|
||||
await stream.writeSSE({
|
||||
id: String(blob.receivedAt),
|
||||
event: 'envelope',
|
||||
data: JSON.stringify(serializeBlob(blob)),
|
||||
});
|
||||
recordPush(blob.msgId);
|
||||
},
|
||||
delivered,
|
||||
);
|
||||
cursor = Math.max(cursor, flushed);
|
||||
|
||||
// Hook up event-driven push if available, else fall back to a poll
|
||||
@@ -124,19 +187,32 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
|
||||
const triggerFlush = (): void => {
|
||||
signalled = true;
|
||||
// Serialize fan-in so concurrent triggers don't double-fetch.
|
||||
pendingFlushPromise = pendingFlushPromise.then(async () => {
|
||||
while (signalled) {
|
||||
signalled = false;
|
||||
const drained = await flushTo(writer, address, cursor, async (blob) => {
|
||||
await stream.writeSSE({
|
||||
id: String(blob.receivedAt),
|
||||
event: 'envelope',
|
||||
data: JSON.stringify(serializeBlob(blob)),
|
||||
});
|
||||
});
|
||||
if (drained > cursor) cursor = drained;
|
||||
}
|
||||
});
|
||||
// `.catch(() => {})` keeps the chain alive across transient
|
||||
// emit failures (e.g. a closed SSE write throws) — without it
|
||||
// one rejection silently kills every future flush on this
|
||||
// connection.
|
||||
pendingFlushPromise = pendingFlushPromise
|
||||
.then(async () => {
|
||||
while (signalled) {
|
||||
signalled = false;
|
||||
const drained = await flushTo(
|
||||
writer,
|
||||
address,
|
||||
cursor,
|
||||
async (blob) => {
|
||||
await stream.writeSSE({
|
||||
id: String(blob.receivedAt),
|
||||
event: 'envelope',
|
||||
data: JSON.stringify(serializeBlob(blob)),
|
||||
});
|
||||
recordPush(blob.msgId);
|
||||
},
|
||||
delivered,
|
||||
);
|
||||
if (drained > cursor) cursor = drained;
|
||||
}
|
||||
})
|
||||
.catch(() => {});
|
||||
};
|
||||
|
||||
if (opts.events) {
|
||||
@@ -163,6 +239,68 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
|
||||
clearInterval(fallbackTimer);
|
||||
clearInterval(heartbeat);
|
||||
await pendingFlushPromise.catch(() => {});
|
||||
closePresence('closed');
|
||||
});
|
||||
});
|
||||
|
||||
// ─── Presence (V4.7) ──────────────────────────────────────────
|
||||
// SSE feed of `peer_connected` / `peer_disconnected` events filtered
|
||||
// by a watcher-supplied address list. Subscribing does NOT count as
|
||||
// a peer-bridge connection (it doesn't call `markConnected`) so
|
||||
// monitoring presence doesn't make you appear online to others.
|
||||
app.get('/v1/bridge/presence', async (c) => {
|
||||
const verified = await verifyPresenceAuth(c, opts);
|
||||
return streamSSE(c, async (stream) => {
|
||||
const watched = new Set(verified.watched);
|
||||
|
||||
// Initial snapshot — one frame per watched address with current
|
||||
// status. Lets subscribers render UI immediately rather than
|
||||
// waiting for the next state change.
|
||||
const now = Date.now();
|
||||
for (const addr of verified.watched) {
|
||||
await stream.writeSSE({
|
||||
event: 'presence',
|
||||
data: JSON.stringify({
|
||||
address: addr,
|
||||
status: presence.isOnline(addr) ? 'online' : 'offline',
|
||||
at: now,
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
let unsubscribe: (() => void) | null = null;
|
||||
if (opts.events) {
|
||||
unsubscribe = opts.events.on((e) => {
|
||||
if (e.name !== 'inbox.peer_connected' && e.name !== 'inbox.peer_disconnected') return;
|
||||
const data = e.data as { address: string; bridgeKind: TrackedBridgeKind };
|
||||
if (!watched.has(data.address)) return;
|
||||
const status = e.name === 'inbox.peer_connected' ? 'online' : 'offline';
|
||||
// Fire-and-forget: drop the frame if the stream has gone away.
|
||||
void stream
|
||||
.writeSSE({
|
||||
event: 'presence',
|
||||
data: JSON.stringify({
|
||||
address: data.address,
|
||||
status,
|
||||
at: e.timestamp,
|
||||
via: data.bridgeKind,
|
||||
}),
|
||||
})
|
||||
.catch(() => {});
|
||||
});
|
||||
}
|
||||
const heartbeat = setInterval(() => {
|
||||
stream.write(`: ping ${Date.now()}\n\n`).catch(() => {});
|
||||
}, heartbeatIntervalMs);
|
||||
|
||||
await new Promise<void>((resolve) => {
|
||||
const sig = c.req.raw.signal;
|
||||
if (sig.aborted) return resolve();
|
||||
sig.addEventListener('abort', () => resolve(), { once: true });
|
||||
});
|
||||
|
||||
unsubscribe?.();
|
||||
clearInterval(heartbeat);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -183,6 +321,8 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
|
||||
limit: pageLimit,
|
||||
});
|
||||
if (blobs.length > 0) {
|
||||
const now = Date.now();
|
||||
for (const b of blobs) bridgeDeliveryLog.recordDelivered(verified.address, b.msgId, now);
|
||||
return c.json(buildPollResponse(blobs, verified.since));
|
||||
}
|
||||
|
||||
@@ -197,6 +337,8 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
|
||||
fallbackPollIntervalMs,
|
||||
abortSignal: c.req.raw.signal,
|
||||
});
|
||||
const now = Date.now();
|
||||
for (const b of blobs) bridgeDeliveryLog.recordDelivered(verified.address, b.msgId, now);
|
||||
return c.json(buildPollResponse(blobs, verified.since));
|
||||
});
|
||||
|
||||
@@ -230,30 +372,51 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
|
||||
}
|
||||
|
||||
const address = verified.address;
|
||||
const connId = presence.newConnectionId();
|
||||
let cursor = verified.since;
|
||||
const writer = makeBlobWriter(opts.store, pageLimit);
|
||||
const delivered = new DeliveredIdLru();
|
||||
let unsubscribe: (() => void) | null = null;
|
||||
let fallbackTimer: ReturnType<typeof setInterval> | null = null;
|
||||
let pendingFlushPromise: Promise<void> = Promise.resolve();
|
||||
let signalled = false;
|
||||
let connected = true;
|
||||
let presenceClosed = false;
|
||||
const closePresence = (reason: 'closed' | 'error'): void => {
|
||||
if (presenceClosed) return;
|
||||
presenceClosed = true;
|
||||
presence.markDisconnected(address, 'ws', connId, reason);
|
||||
};
|
||||
|
||||
return {
|
||||
onOpen(_evt: unknown, ws: {
|
||||
send: (data: string) => void;
|
||||
close: (code?: number, reason?: string) => void;
|
||||
}) {
|
||||
presence.markConnected(address, 'ws', connId);
|
||||
const triggerFlush = (): void => {
|
||||
signalled = true;
|
||||
pendingFlushPromise = pendingFlushPromise.then(async () => {
|
||||
while (signalled && connected) {
|
||||
signalled = false;
|
||||
const drained = await flushTo(writer, address, cursor, async (blob) => {
|
||||
ws.send(JSON.stringify(serializeBlob(blob)));
|
||||
});
|
||||
if (drained > cursor) cursor = drained;
|
||||
}
|
||||
});
|
||||
// `.catch(() => {})` mirrors the SSE chain — keeps the
|
||||
// pending-flush queue alive across transient ws.send errors
|
||||
// (e.g. partial close, backpressure overflow).
|
||||
pendingFlushPromise = pendingFlushPromise
|
||||
.then(async () => {
|
||||
while (signalled && connected) {
|
||||
signalled = false;
|
||||
const drained = await flushTo(
|
||||
writer,
|
||||
address,
|
||||
cursor,
|
||||
async (blob) => {
|
||||
ws.send(JSON.stringify(serializeBlob(blob)));
|
||||
bridgeDeliveryLog.recordDelivered(address, blob.msgId, Date.now());
|
||||
},
|
||||
delivered,
|
||||
);
|
||||
if (drained > cursor) cursor = drained;
|
||||
}
|
||||
})
|
||||
.catch(() => {});
|
||||
};
|
||||
if (opts.events) {
|
||||
unsubscribe = opts.events.on((e) => {
|
||||
@@ -269,12 +432,19 @@ export function createBridgeRoutes(opts: BridgeRoutesOptions): {
|
||||
connected = false;
|
||||
unsubscribe?.();
|
||||
if (fallbackTimer) clearInterval(fallbackTimer);
|
||||
closePresence('closed');
|
||||
},
|
||||
onError() {
|
||||
connected = false;
|
||||
unsubscribe?.();
|
||||
if (fallbackTimer) clearInterval(fallbackTimer);
|
||||
closePresence('error');
|
||||
},
|
||||
};
|
||||
}),
|
||||
);
|
||||
|
||||
return { app, websocket };
|
||||
return { app, websocket, presence, bridgeDeliveryLog };
|
||||
}
|
||||
|
||||
// ─── helpers ──────────────────────────────────────────────────
|
||||
@@ -321,11 +491,75 @@ async function verifyBridgeAuth(
|
||||
return { address, kind: kind as BridgeKind, since };
|
||||
}
|
||||
|
||||
const MAX_WATCHED_ADDRESSES = 64;
|
||||
|
||||
/**
|
||||
* Verify a `/v1/bridge/presence` request.
|
||||
*
|
||||
* Signed canonical payload: `{address, kind: 'presence', watched: string[],
|
||||
* signedAt}`. The watcher's address must be a registered inbox; the
|
||||
* signature is verified against the registered owner key for that
|
||||
* address. The `watched` list bounds what the subscription will
|
||||
* receive — server-side filtering is enforced inside the handler.
|
||||
*/
|
||||
async function verifyPresenceAuth(
|
||||
c: Context,
|
||||
opts: BridgeRoutesOptions,
|
||||
): Promise<VerifiedPresenceRequest> {
|
||||
const url = new URL(c.req.url);
|
||||
const qs = url.searchParams;
|
||||
const address = validateAddress(qs.get('address'));
|
||||
const kind = qs.get('kind');
|
||||
if (kind !== 'presence') {
|
||||
throw new ValidationError(`bridge kind mismatch: expected presence`, 'kind');
|
||||
}
|
||||
const watchedRaw = qs.get('watched');
|
||||
if (watchedRaw === null) throw new ValidationError('missing watched', 'watched');
|
||||
// Empty subscription is allowed (subscribe to nothing — useful for a
|
||||
// client that intends to call addPeer right after open). A null/
|
||||
// missing param is still rejected so the canonicalization is
|
||||
// unambiguous.
|
||||
const watched =
|
||||
watchedRaw === ''
|
||||
? []
|
||||
: watchedRaw.split(',').map((a) => validateAddress(a));
|
||||
if (watched.length > MAX_WATCHED_ADDRESSES) {
|
||||
throw new ValidationError(
|
||||
`watched list too large: ${watched.length} > ${MAX_WATCHED_ADDRESSES}`,
|
||||
'watched',
|
||||
);
|
||||
}
|
||||
const signedAtStr = qs.get('signedAt');
|
||||
const signature = qs.get('signature');
|
||||
if (signedAtStr === null) throw new ValidationError('missing signedAt', 'signedAt');
|
||||
if (!signature) throw new ValidationError('missing signature', 'signature');
|
||||
const signedAt = Number(signedAtStr);
|
||||
if (!Number.isFinite(signedAt)) {
|
||||
throw new ValidationError('signedAt must be a number', 'signedAt');
|
||||
}
|
||||
|
||||
const owner = await opts.store.getAddressOwner(address);
|
||||
if (!owner) {
|
||||
throw new UnauthorizedError(`address ${address} is not registered`);
|
||||
}
|
||||
|
||||
await verifyPayload(opts.crypto, owner, {
|
||||
address,
|
||||
kind,
|
||||
watched,
|
||||
signedAt,
|
||||
signature,
|
||||
});
|
||||
return { address, watched };
|
||||
}
|
||||
|
||||
interface BlobRow {
|
||||
msgId: string;
|
||||
ciphertext: Uint8Array;
|
||||
receivedAt: number;
|
||||
expiresAt: number;
|
||||
/** V4.8 — relay-captured sender fingerprint. Optional for legacy rows. */
|
||||
senderFp?: string;
|
||||
}
|
||||
|
||||
interface BlobWriter {
|
||||
@@ -345,11 +579,41 @@ function makeBlobWriter(store: InboxStore, pageLimit: number): BlobWriter {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-connection bounded msgId tracker — defense in depth against duplicate
|
||||
* delivery of the same blob to the same bridge socket. Cursor pagination
|
||||
* already guarantees uniqueness in the happy path, but a dedup gate at the
|
||||
* emit boundary catches any subtle bug (e.g. a flushTo race, a future
|
||||
* refactor, an event-emit retry) without changing wire semantics.
|
||||
*
|
||||
* The cap is intentionally large enough to cover any realistic bridge
|
||||
* pageLimit and small enough to bound memory under long-running streams.
|
||||
*/
|
||||
const DELIVERED_LRU_CAP = 4096;
|
||||
|
||||
class DeliveredIdLru {
|
||||
private readonly seen = new Set<string>();
|
||||
private readonly order: string[] = [];
|
||||
|
||||
/** Returns true if `msgId` has not been seen on this connection yet. */
|
||||
add(msgId: string): boolean {
|
||||
if (this.seen.has(msgId)) return false;
|
||||
this.seen.add(msgId);
|
||||
this.order.push(msgId);
|
||||
if (this.order.length > DELIVERED_LRU_CAP) {
|
||||
const evicted = this.order.shift()!;
|
||||
this.seen.delete(evicted);
|
||||
}
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
async function flushTo(
|
||||
writer: BlobWriter,
|
||||
address: string,
|
||||
startCursor: number,
|
||||
emit: (blob: BlobRow) => Promise<void>,
|
||||
delivered?: DeliveredIdLru,
|
||||
): Promise<number> {
|
||||
let cursor = startCursor;
|
||||
// Drain page-by-page so a backlog larger than `pageLimit` still flushes.
|
||||
@@ -358,7 +622,12 @@ async function flushTo(
|
||||
const page = await writer.fetchPage(address, cursor);
|
||||
if (page.length === 0) break;
|
||||
for (const row of page) {
|
||||
await emit(row);
|
||||
// Per-connection dedup gate — prevents the same msgId from being
|
||||
// emitted twice if flushTo is somehow re-entered before the cursor
|
||||
// catches up. See comment on `DeliveredIdLru`.
|
||||
if (!delivered || delivered.add(row.msgId)) {
|
||||
await emit(row);
|
||||
}
|
||||
if (row.receivedAt > cursor) cursor = row.receivedAt;
|
||||
}
|
||||
if (page.length === 0) break;
|
||||
@@ -371,13 +640,26 @@ function serializeBlob(blob: BlobRow): {
|
||||
ciphertext: string;
|
||||
receivedAt: number;
|
||||
expiresAt: number;
|
||||
from?: string;
|
||||
} {
|
||||
return {
|
||||
const out: {
|
||||
msgId: string;
|
||||
ciphertext: string;
|
||||
receivedAt: number;
|
||||
expiresAt: number;
|
||||
from?: string;
|
||||
} = {
|
||||
msgId: blob.msgId,
|
||||
ciphertext: toBase64(blob.ciphertext),
|
||||
receivedAt: blob.receivedAt,
|
||||
expiresAt: blob.expiresAt,
|
||||
};
|
||||
// V4.8 — relay-captured sender fingerprint. The transport-bridge
|
||||
// wire format already accepts `from`; populating it lets receivers
|
||||
// bootstrap unknown-sender first-contact via `shade.receive('fp:<hex>',
|
||||
// env)` without requiring an out-of-band sender hint.
|
||||
if (blob.senderFp) out.from = blob.senderFp;
|
||||
return out;
|
||||
}
|
||||
|
||||
function buildPollResponse(blobs: BlobRow[], sinceFallback: number): {
|
||||
|
||||
@@ -21,6 +21,19 @@ export interface InboxServerEventMap {
|
||||
'inbox.expired_purged': { count: number };
|
||||
'inbox.rate_limited': { route: string; key: string };
|
||||
'inbox.quota_rejected': { address: string; reason: 'address-quota' | 'sender-quota' | 'body-too-large' };
|
||||
// V4.7 — bridge presence transitions. Emitted on the 0↔1 boundary
|
||||
// across tracked transports for a given address. Long-poll is
|
||||
// intentionally NOT tracked: an LP client toggles in/out of a request
|
||||
// every few seconds, and the resulting flapping would dominate the
|
||||
// event stream. Push transports (WS, SSE) are also the only ones
|
||||
// where the ~50ms revoke window for `BroadcastChannel.removeMember`
|
||||
// matters — long-poll users are already on a slow path.
|
||||
'inbox.peer_connected': { address: string; bridgeKind: 'ws' | 'sse' };
|
||||
'inbox.peer_disconnected': {
|
||||
address: string;
|
||||
bridgeKind: 'ws' | 'sse';
|
||||
reason: 'closed' | 'error';
|
||||
};
|
||||
}
|
||||
|
||||
export type InboxServerEventName = keyof InboxServerEventMap;
|
||||
|
||||
@@ -1,9 +1,12 @@
|
||||
import type { Hono } from 'hono';
|
||||
import { Hono } from 'hono';
|
||||
import type { CryptoProvider } from '@shade/core';
|
||||
import { createInboxRoutes, type InboxRoutesOptions } from './routes.js';
|
||||
import { MemoryInboxStore } from './memory-store.js';
|
||||
import type { InboxStore } from './store.js';
|
||||
import { InboxServerEvents } from './events.js';
|
||||
import { createBlobRoutes, type BlobRoutesOptions } from './blob-routes.js';
|
||||
import { MemoryBlobStore } from './memory-blob-store.js';
|
||||
import type { BlobStore } from './blob-store.js';
|
||||
|
||||
export { createInboxRoutes } from './routes.js';
|
||||
export type { InboxRoutesOptions } from './routes.js';
|
||||
@@ -32,6 +35,14 @@ export {
|
||||
export type { InboxQuotaConfig } from './quota.js';
|
||||
export { createBridgeRoutes } from './bridge.js';
|
||||
export type { BridgeRoutesOptions, BridgeKind } from './bridge.js';
|
||||
export { PresenceTracker } from './presence.js';
|
||||
export type { TrackedBridgeKind } from './presence.js';
|
||||
export { BridgeDeliveryLog } from './bridge-delivery-log.js';
|
||||
export type { BridgeDeliveryLogOptions } from './bridge-delivery-log.js';
|
||||
export { createBlobRoutes, DEFAULT_MAX_BLOB_BYTES } from './blob-routes.js';
|
||||
export type { BlobRoutesOptions } from './blob-routes.js';
|
||||
export { MemoryBlobStore } from './memory-blob-store.js';
|
||||
export type { BlobStore, BlobSlotRecord, PutBlobResult } from './blob-store.js';
|
||||
|
||||
/**
|
||||
* Create a standalone Shade Inbox Server.
|
||||
@@ -44,17 +55,45 @@ export type { BridgeRoutesOptions, BridgeKind } from './bridge.js';
|
||||
* const app = new Hono();
|
||||
* app.route('/', createInboxServer({ crypto }));
|
||||
*/
|
||||
export function createInboxServer(options: {
|
||||
crypto: CryptoProvider;
|
||||
store?: InboxStore;
|
||||
disableRateLimit?: boolean;
|
||||
events?: InboxServerEvents;
|
||||
} & Pick<InboxRoutesOptions, 'observability' | 'quota'>): Hono {
|
||||
export function createInboxServer(
|
||||
options: {
|
||||
crypto: CryptoProvider;
|
||||
store?: InboxStore;
|
||||
disableRateLimit?: boolean;
|
||||
events?: InboxServerEvents;
|
||||
/**
|
||||
* V4.9 — when supplied, mounts the encrypted-blob primitive
|
||||
* (`/v1/blob/<slotId>`) on the same Hono app. Pass `null` to
|
||||
* explicitly opt out; omit to default to a `MemoryBlobStore`.
|
||||
*/
|
||||
blobStore?: BlobStore | null;
|
||||
blobOptions?: Pick<BlobRoutesOptions, 'maxBlobBytes'>;
|
||||
} & Pick<InboxRoutesOptions, 'observability' | 'quota' | 'bridgeDeliveryLog'>,
|
||||
): Hono {
|
||||
const store = options.store ?? new MemoryInboxStore();
|
||||
const routesOptions: InboxRoutesOptions = {};
|
||||
if (options.disableRateLimit !== undefined) routesOptions.disableRateLimit = options.disableRateLimit;
|
||||
if (options.events !== undefined) routesOptions.events = options.events;
|
||||
if (options.observability !== undefined) routesOptions.observability = options.observability;
|
||||
if (options.quota !== undefined) routesOptions.quota = options.quota;
|
||||
return createInboxRoutes(store, options.crypto, routesOptions);
|
||||
if (options.bridgeDeliveryLog !== undefined) routesOptions.bridgeDeliveryLog = options.bridgeDeliveryLog;
|
||||
|
||||
const inboxApp = createInboxRoutes(store, options.crypto, routesOptions);
|
||||
|
||||
// Compose with the blob primitive unless explicitly disabled. The
|
||||
// blob routes share the same Hono app so a single port serves both.
|
||||
if (options.blobStore === null) return inboxApp;
|
||||
const blobStore = options.blobStore ?? new MemoryBlobStore();
|
||||
const blobRoutesOptions: BlobRoutesOptions = {};
|
||||
if (options.disableRateLimit !== undefined) blobRoutesOptions.disableRateLimit = options.disableRateLimit;
|
||||
if (options.observability !== undefined) blobRoutesOptions.observability = options.observability;
|
||||
if (options.blobOptions?.maxBlobBytes !== undefined) {
|
||||
blobRoutesOptions.maxBlobBytes = options.blobOptions.maxBlobBytes;
|
||||
}
|
||||
const blobApp = createBlobRoutes(blobStore, options.crypto, blobRoutesOptions);
|
||||
|
||||
const composed = new Hono();
|
||||
composed.route('/', inboxApp);
|
||||
composed.route('/', blobApp);
|
||||
return composed;
|
||||
}
|
||||
|
||||
85
packages/shade-inbox-server/src/memory-blob-store.ts
Normal file
85
packages/shade-inbox-server/src/memory-blob-store.ts
Normal file
@@ -0,0 +1,85 @@
|
||||
import { ConflictError, PreconditionFailedError } from '@shade/core';
|
||||
import type { BlobStore, BlobSlotRecord, PutBlobResult } from './blob-store.js';
|
||||
|
||||
/**
|
||||
* In-memory BlobStore — used in tests and as the default fallback when
|
||||
* no SQLite/Postgres URL is configured. Rows are kept in a single Map.
|
||||
*
|
||||
* Etag is a strictly-monotonic per-process counter — guarantees a total
|
||||
* order across writes even when many land in the same millisecond. (We
|
||||
* could scope it per-slot, but a global counter keeps the implementation
|
||||
* trivial and the etag values still uniquely identify the write that
|
||||
* produced them, which is all CAS needs.)
|
||||
*/
|
||||
export class MemoryBlobStore implements BlobStore {
|
||||
private slots = new Map<string, BlobSlotRecord>();
|
||||
private nextEtag = 0;
|
||||
|
||||
async get(slotId: string): Promise<BlobSlotRecord | null> {
|
||||
const r = this.slots.get(slotId);
|
||||
if (!r) return null;
|
||||
return {
|
||||
slotId: r.slotId,
|
||||
blob: new Uint8Array(r.blob),
|
||||
ownerPubkey: new Uint8Array(r.ownerPubkey),
|
||||
etag: r.etag,
|
||||
updatedAt: r.updatedAt,
|
||||
};
|
||||
}
|
||||
|
||||
async put(args: {
|
||||
slotId: string;
|
||||
blob: Uint8Array;
|
||||
ownerPubkey: Uint8Array;
|
||||
expectedEtag: number | '*' | undefined;
|
||||
now: number;
|
||||
}): Promise<PutBlobResult> {
|
||||
const existing = this.slots.get(args.slotId);
|
||||
|
||||
if (!existing) {
|
||||
// Empty slot. `ifMatch: '*'` per RFC 7232 still fails — there is
|
||||
// no entity to match. A numeric etag also fails (we have nothing
|
||||
// to compare against).
|
||||
if (args.expectedEtag !== undefined) {
|
||||
throw new PreconditionFailedError(
|
||||
`Slot ${args.slotId} does not exist; cannot match ifMatch=${String(args.expectedEtag)}`,
|
||||
);
|
||||
}
|
||||
this.nextEtag = Math.max(this.nextEtag + 1, args.now);
|
||||
const etag = this.nextEtag;
|
||||
this.slots.set(args.slotId, {
|
||||
slotId: args.slotId,
|
||||
blob: new Uint8Array(args.blob),
|
||||
ownerPubkey: new Uint8Array(args.ownerPubkey),
|
||||
etag,
|
||||
updatedAt: args.now,
|
||||
});
|
||||
return { created: true, etag, updatedAt: args.now };
|
||||
}
|
||||
|
||||
// Slot exists. Pubkey check is the route layer's job — by the time
|
||||
// we're here the signature has already been verified against
|
||||
// `existing.ownerPubkey`.
|
||||
if (args.expectedEtag === undefined) {
|
||||
throw new ConflictError(
|
||||
`Slot ${args.slotId} already exists; supply ifMatch to update`,
|
||||
);
|
||||
}
|
||||
if (args.expectedEtag !== '*' && args.expectedEtag !== existing.etag) {
|
||||
throw new PreconditionFailedError(
|
||||
`Slot ${args.slotId} etag mismatch: expected=${args.expectedEtag} actual=${existing.etag}`,
|
||||
);
|
||||
}
|
||||
|
||||
this.nextEtag = Math.max(this.nextEtag + 1, args.now);
|
||||
const etag = this.nextEtag;
|
||||
existing.blob = new Uint8Array(args.blob);
|
||||
existing.etag = etag;
|
||||
existing.updatedAt = args.now;
|
||||
return { created: false, etag, updatedAt: args.now };
|
||||
}
|
||||
|
||||
async delete(slotId: string): Promise<boolean> {
|
||||
return this.slots.delete(slotId);
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@ interface BlobRow {
|
||||
ciphertext: Uint8Array;
|
||||
receivedAt: number;
|
||||
expiresAt: number;
|
||||
senderFp?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -33,6 +34,7 @@ export class MemoryInboxStore implements InboxStore {
|
||||
msgId: string;
|
||||
ciphertext: Uint8Array;
|
||||
expiresAt: number;
|
||||
senderFp?: string;
|
||||
}): Promise<{ created: boolean; receivedAt: number }> {
|
||||
const list = this.blobs.get(args.address) ?? [];
|
||||
const existing = list.find((r) => r.msgId === args.msgId);
|
||||
@@ -41,12 +43,14 @@ export class MemoryInboxStore implements InboxStore {
|
||||
// multiple blobs land in the same millisecond.
|
||||
const receivedAt = Math.max(this.nextReceivedAt + 1, Date.now());
|
||||
this.nextReceivedAt = receivedAt;
|
||||
list.push({
|
||||
const row: BlobRow = {
|
||||
msgId: args.msgId,
|
||||
ciphertext: new Uint8Array(args.ciphertext),
|
||||
receivedAt,
|
||||
expiresAt: args.expiresAt,
|
||||
});
|
||||
};
|
||||
if (args.senderFp !== undefined) row.senderFp = args.senderFp;
|
||||
list.push(row);
|
||||
this.blobs.set(args.address, list);
|
||||
return { created: true, receivedAt };
|
||||
}
|
||||
@@ -56,18 +60,36 @@ export class MemoryInboxStore implements InboxStore {
|
||||
sinceCursor: number;
|
||||
now: number;
|
||||
limit: number;
|
||||
}): Promise<Array<{ msgId: string; ciphertext: Uint8Array; receivedAt: number; expiresAt: number }>> {
|
||||
}): Promise<
|
||||
Array<{
|
||||
msgId: string;
|
||||
ciphertext: Uint8Array;
|
||||
receivedAt: number;
|
||||
expiresAt: number;
|
||||
senderFp?: string;
|
||||
}>
|
||||
> {
|
||||
const list = this.blobs.get(args.address) ?? [];
|
||||
return list
|
||||
.filter((r) => r.receivedAt > args.sinceCursor && r.expiresAt > args.now)
|
||||
.sort((a, b) => a.receivedAt - b.receivedAt)
|
||||
.slice(0, args.limit)
|
||||
.map((r) => ({
|
||||
msgId: r.msgId,
|
||||
ciphertext: new Uint8Array(r.ciphertext),
|
||||
receivedAt: r.receivedAt,
|
||||
expiresAt: r.expiresAt,
|
||||
}));
|
||||
.map((r) => {
|
||||
const out: {
|
||||
msgId: string;
|
||||
ciphertext: Uint8Array;
|
||||
receivedAt: number;
|
||||
expiresAt: number;
|
||||
senderFp?: string;
|
||||
} = {
|
||||
msgId: r.msgId,
|
||||
ciphertext: new Uint8Array(r.ciphertext),
|
||||
receivedAt: r.receivedAt,
|
||||
expiresAt: r.expiresAt,
|
||||
};
|
||||
if (r.senderFp !== undefined) out.senderFp = r.senderFp;
|
||||
return out;
|
||||
});
|
||||
}
|
||||
|
||||
async deleteBlob(address: string, msgId: string): Promise<boolean> {
|
||||
|
||||
75
packages/shade-inbox-server/src/presence.ts
Normal file
75
packages/shade-inbox-server/src/presence.ts
Normal file
@@ -0,0 +1,75 @@
|
||||
/**
|
||||
* V4.7 — bridge-connection presence tracking.
|
||||
*
|
||||
* The bridge handlers (`/v1/bridge/stream` and `/v1/bridge/ws`) call
|
||||
* `markConnected` on open and `markDisconnected` on close. The tracker
|
||||
* keeps a per-address set of connection ids; the `inbox.peer_connected`
|
||||
* / `inbox.peer_disconnected` events fire only on the 0↔1 boundary so
|
||||
* that two simultaneous bridges (e.g. SSE + WS during a transport-
|
||||
* fallback handover) collapse into a single connected/disconnected
|
||||
* pair from the consumer's point of view.
|
||||
*
|
||||
* Long-poll (`/v1/bridge/poll`) is intentionally NOT tracked — see the
|
||||
* note on `InboxServerEventMap` in `events.ts`.
|
||||
*/
|
||||
|
||||
import type { InboxServerEvents } from './events.js';
|
||||
|
||||
export type TrackedBridgeKind = 'ws' | 'sse';
|
||||
|
||||
export class PresenceTracker {
|
||||
private readonly connections = new Map<string, Set<string>>();
|
||||
private nextConnId = 1;
|
||||
|
||||
constructor(private readonly events: InboxServerEvents | null) {}
|
||||
|
||||
/** Allocate a fresh connection id for `markConnected` / `markDisconnected`. */
|
||||
newConnectionId(): string {
|
||||
return `c${this.nextConnId++}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Snapshot: is `address` currently connected over any tracked transport?
|
||||
* Used by `/v1/bridge/presence` to push the initial state to a new
|
||||
* subscriber.
|
||||
*/
|
||||
isOnline(address: string): boolean {
|
||||
const set = this.connections.get(address);
|
||||
return set !== undefined && set.size > 0;
|
||||
}
|
||||
|
||||
markConnected(address: string, bridgeKind: TrackedBridgeKind, connectionId: string): void {
|
||||
let set = this.connections.get(address);
|
||||
const wasOnline = set !== undefined && set.size > 0;
|
||||
if (!set) {
|
||||
set = new Set();
|
||||
this.connections.set(address, set);
|
||||
}
|
||||
set.add(connectionId);
|
||||
if (!wasOnline) {
|
||||
this.events?.emit('inbox.peer_connected', { address, bridgeKind });
|
||||
}
|
||||
}
|
||||
|
||||
markDisconnected(
|
||||
address: string,
|
||||
bridgeKind: TrackedBridgeKind,
|
||||
connectionId: string,
|
||||
reason: 'closed' | 'error',
|
||||
): void {
|
||||
const set = this.connections.get(address);
|
||||
if (!set) return;
|
||||
if (!set.delete(connectionId)) return;
|
||||
if (set.size === 0) {
|
||||
this.connections.delete(address);
|
||||
this.events?.emit('inbox.peer_disconnected', { address, bridgeKind, reason });
|
||||
}
|
||||
}
|
||||
|
||||
/** Inspect the underlying map. Test/observability use only. */
|
||||
snapshot(): Map<string, ReadonlySet<string>> {
|
||||
return new Map(
|
||||
Array.from(this.connections.entries(), ([k, v]) => [k, v as ReadonlySet<string>]),
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -28,6 +28,7 @@ import type { InboxStore } from './store.js';
|
||||
import { InboxServerEvents, shortHash } from './events.js';
|
||||
import { computeMsgId, isValidMsgId, constantTimeStringEqual } from './msg-id.js';
|
||||
import { clampTtl, DEFAULT_INBOX_QUOTA, type InboxQuotaConfig } from './quota.js';
|
||||
import type { BridgeDeliveryLog } from './bridge-delivery-log.js';
|
||||
|
||||
/** Max metadata-only body size (no ciphertext). 64 KB is enough headroom. */
|
||||
const MAX_META_BODY_SIZE = 64 * 1024;
|
||||
@@ -54,6 +55,16 @@ export interface InboxRoutesOptions {
|
||||
observability?: ObservabilityHook;
|
||||
/** Override quota policy. */
|
||||
quota?: Partial<InboxQuotaConfig>;
|
||||
/**
|
||||
* V4.8.4 — shared bridge delivery log. When provided (and the same
|
||||
* instance is wired into `createBridgeRoutes`), the inbox-fetch route
|
||||
* filters out blobs already pushed via bridge within the log's grace
|
||||
* window. Without this, a recipient that runs both a bridge
|
||||
* subscription and inbox-poll receives the same envelope twice.
|
||||
* Optional — leaving it unset preserves the pre-V4.8.4 behavior of
|
||||
* always returning every blob the cursor matches.
|
||||
*/
|
||||
bridgeDeliveryLog?: BridgeDeliveryLog;
|
||||
}
|
||||
|
||||
export function createInboxRoutes(
|
||||
@@ -171,6 +182,7 @@ export function createInboxRoutes(
|
||||
await verifyPayload(crypto, owner, { ...body, address });
|
||||
|
||||
await store.deleteAddress(address);
|
||||
options.bridgeDeliveryLog?.forgetAddress(address);
|
||||
events?.emit('inbox.address_deleted', { address });
|
||||
return c.json({ ok: true });
|
||||
});
|
||||
@@ -255,11 +267,19 @@ export function createInboxRoutes(
|
||||
);
|
||||
}
|
||||
|
||||
// V4.8: capture sender fingerprint at PUT time. The sender's
|
||||
// signing key was just verified for this request, so the fingerprint
|
||||
// is bound to the same authentication path that authorized the
|
||||
// store. Surfaced on bridge push + inbox-fetch responses to
|
||||
// bootstrap unknown-sender first-contact (X3DH pair handshake).
|
||||
const senderFp = await shortHash(senderKey);
|
||||
|
||||
const result = await store.putBlob({
|
||||
address,
|
||||
msgId,
|
||||
ciphertext: ctBytes,
|
||||
expiresAt,
|
||||
senderFp,
|
||||
});
|
||||
if (result.created) {
|
||||
events?.emit('inbox.blob_stored', {
|
||||
@@ -309,31 +329,63 @@ export function createInboxRoutes(
|
||||
await verifyPayload(crypto, owner, { ...body, address });
|
||||
|
||||
const now = Date.now();
|
||||
const rows = await store.fetchBlobs({
|
||||
const rawRows = await store.fetchBlobs({
|
||||
address,
|
||||
sinceCursor,
|
||||
now,
|
||||
limit: quota.fetchPageLimit,
|
||||
});
|
||||
|
||||
// V4.8.4 — drop blobs the bridge has already pushed to this address
|
||||
// within the grace window. This is the cross-channel dedup gate that
|
||||
// makes "one inbox.send ⇒ one observable delivery" hold even when
|
||||
// the recipient runs both a bridge subscription and inbox-poll. The
|
||||
// cursor still advances over the whole `rawRows` window so the
|
||||
// client doesn't get stuck behind suppressed blobs — pollOnce uses
|
||||
// `nextCursor` (max receivedAt seen by the server, suppressed or
|
||||
// not) for the next fetch.
|
||||
const rows = options.bridgeDeliveryLog
|
||||
? options.bridgeDeliveryLog.filterRecent(address, rawRows, now)
|
||||
: rawRows;
|
||||
|
||||
let bytes = 0;
|
||||
const blobs = rows.map((r) => {
|
||||
bytes += r.ciphertext.length;
|
||||
return {
|
||||
const out: {
|
||||
msgId: string;
|
||||
ciphertext: string;
|
||||
receivedAt: number;
|
||||
expiresAt: number;
|
||||
from?: string;
|
||||
} = {
|
||||
msgId: r.msgId,
|
||||
ciphertext: toBase64(r.ciphertext),
|
||||
receivedAt: r.receivedAt,
|
||||
expiresAt: r.expiresAt,
|
||||
};
|
||||
// V4.8: surface sender fingerprint when present. Empty for blobs
|
||||
// persisted by a pre-4.8 relay that didn't track sender provenance.
|
||||
if (r.senderFp) out.from = r.senderFp;
|
||||
return out;
|
||||
});
|
||||
const nextCursor = rows.length > 0 ? rows[rows.length - 1]!.receivedAt : sinceCursor;
|
||||
// Advance the cursor past the FULL rawRows window — including blobs
|
||||
// we suppressed because the bridge already pushed them. If we
|
||||
// anchored the cursor on `rows` only, suppressed blobs in the
|
||||
// middle of the window would block all subsequent fetches forever
|
||||
// (re-fetched on every poll, re-suppressed, no progress). The
|
||||
// bridge-delivery contract is "the bridge frame is the canonical
|
||||
// delivery"; if the recipient missed processing it, they fall back
|
||||
// to ack-via-DELETE or the blob ages out at TTL — same as a
|
||||
// recipient that crashes mid-handler in the no-bridge case.
|
||||
const cursorAnchor = rawRows.length > 0 ? rawRows[rawRows.length - 1]!.receivedAt : sinceCursor;
|
||||
const nextCursor = cursorAnchor;
|
||||
|
||||
events?.emit('inbox.blob_fetched', { address, count: blobs.length, bytes });
|
||||
|
||||
return c.json({
|
||||
blobs,
|
||||
cursor: nextCursor,
|
||||
hasMore: rows.length === quota.fetchPageLimit,
|
||||
hasMore: rawRows.length === quota.fetchPageLimit,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -359,6 +411,10 @@ export function createInboxRoutes(
|
||||
if (removed) {
|
||||
events?.emit('inbox.blob_acked', { address, msgId });
|
||||
}
|
||||
// Drop any bridge-delivery mark — keeps the log bounded under
|
||||
// sustained traffic (otherwise long-lived addresses accumulate
|
||||
// entries even after the underlying blob is gone).
|
||||
options.bridgeDeliveryLog?.forget(address, msgId);
|
||||
return c.json({ ok: removed });
|
||||
});
|
||||
|
||||
|
||||
@@ -36,12 +36,20 @@ export interface InboxStore {
|
||||
* **Idempotent**: if a row already exists for `(address, msgId)` the
|
||||
* implementation MUST return `{ created: false }` and leave the existing
|
||||
* row untouched. A fresh insert returns `{ created: true, receivedAt }`.
|
||||
*
|
||||
* `senderFp` (V4.8+) is the 8-byte hex of SHA-256(senderSigningKey)
|
||||
* — captured at PUT time when the relay verified the sender's
|
||||
* signature. Optional so legacy 4.7 callers compile, but populated by
|
||||
* `createInboxRoutes` from 4.8 onward and surfaced on bridge push +
|
||||
* inbox-fetch responses to bootstrap unknown-sender first-contact
|
||||
* (X3DH pair handshake).
|
||||
*/
|
||||
putBlob(args: {
|
||||
address: string;
|
||||
msgId: string;
|
||||
ciphertext: Uint8Array;
|
||||
expiresAt: number;
|
||||
senderFp?: string;
|
||||
}): Promise<{ created: boolean; receivedAt: number }>;
|
||||
|
||||
/**
|
||||
@@ -57,7 +65,20 @@ export interface InboxStore {
|
||||
sinceCursor: number;
|
||||
now: number;
|
||||
limit: number;
|
||||
}): Promise<Array<{ msgId: string; ciphertext: Uint8Array; receivedAt: number; expiresAt: number }>>;
|
||||
}): Promise<
|
||||
Array<{
|
||||
msgId: string;
|
||||
ciphertext: Uint8Array;
|
||||
receivedAt: number;
|
||||
expiresAt: number;
|
||||
/**
|
||||
* Sender fingerprint — 8-byte hex of SHA-256(senderSigningKey)
|
||||
* captured at PUT time. Empty/undefined for blobs persisted by a
|
||||
* pre-4.8 relay that didn't track sender provenance.
|
||||
*/
|
||||
senderFp?: string;
|
||||
}>
|
||||
>;
|
||||
|
||||
/**
|
||||
* Delete a single blob by `(address, msgId)`. Returns true if a row was
|
||||
|
||||
295
packages/shade-inbox-server/tests/blob-routes.test.ts
Normal file
295
packages/shade-inbox-server/tests/blob-routes.test.ts
Normal file
@@ -0,0 +1,295 @@
|
||||
import { describe, test, expect, beforeEach } from 'bun:test';
|
||||
import { Hono } from 'hono';
|
||||
import {
|
||||
createBlobRoutes,
|
||||
MemoryBlobStore,
|
||||
type BlobStore,
|
||||
} from '../src/index.js';
|
||||
import { signPayload } from '@shade/server';
|
||||
import { SubtleCryptoProvider, ed25519PublicKeyFromSeed } from '@shade/crypto-web';
|
||||
import { toBase64, fromBase64 } from '@shade/core';
|
||||
|
||||
const crypto = new SubtleCryptoProvider();
|
||||
|
||||
function randBytes(n: number): Uint8Array {
|
||||
const buf = new Uint8Array(n);
|
||||
globalThis.crypto.getRandomValues(buf);
|
||||
return buf;
|
||||
}
|
||||
|
||||
function hex(bytes: Uint8Array): string {
|
||||
let s = '';
|
||||
for (const b of bytes) s += b.toString(16).padStart(2, '0');
|
||||
return s;
|
||||
}
|
||||
|
||||
describe('Shade Blob Routes (V4.9)', () => {
|
||||
let store: BlobStore;
|
||||
let app: Hono;
|
||||
|
||||
beforeEach(() => {
|
||||
store = new MemoryBlobStore();
|
||||
app = createBlobRoutes(store, crypto, { disableRateLimit: true });
|
||||
});
|
||||
|
||||
async function makeOwner() {
|
||||
const seed = randBytes(32);
|
||||
const pubkey = ed25519PublicKeyFromSeed(seed);
|
||||
return { seed, pubkey };
|
||||
}
|
||||
|
||||
function makeSlotId(): string {
|
||||
return hex(randBytes(32));
|
||||
}
|
||||
|
||||
async function signedPut(args: {
|
||||
slotId: string;
|
||||
blob: Uint8Array;
|
||||
seed: Uint8Array;
|
||||
pubkey: Uint8Array;
|
||||
ifMatch?: string;
|
||||
}) {
|
||||
const payload: Record<string, unknown> = {
|
||||
ownerPubkey: toBase64(args.pubkey),
|
||||
blob: toBase64(args.blob),
|
||||
slotId: args.slotId,
|
||||
};
|
||||
if (args.ifMatch !== undefined) payload.ifMatch = args.ifMatch;
|
||||
const signed = await signPayload(crypto, args.seed, payload);
|
||||
const { slotId: _omit, ...wire } = signed as Record<string, unknown>;
|
||||
return app.request(`/v1/blob/${args.slotId}`, {
|
||||
method: 'PUT',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(wire),
|
||||
});
|
||||
}
|
||||
|
||||
async function signedDelete(args: {
|
||||
slotId: string;
|
||||
seed: Uint8Array;
|
||||
}) {
|
||||
const signed = await signPayload(crypto, args.seed, {
|
||||
slotId: args.slotId,
|
||||
});
|
||||
const { slotId: _omit, ...wire } = signed as Record<string, unknown>;
|
||||
return app.request(`/v1/blob/${args.slotId}`, {
|
||||
method: 'DELETE',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(wire),
|
||||
});
|
||||
}
|
||||
|
||||
// ─── GET ─────────────────────────────────────────────────────
|
||||
|
||||
test('GET on missing slot returns 404', async () => {
|
||||
const slotId = makeSlotId();
|
||||
const res = await app.request(`/v1/blob/${slotId}`);
|
||||
expect(res.status).toBe(404);
|
||||
});
|
||||
|
||||
test('GET requires lowercase 64-hex slotId', async () => {
|
||||
const res = await app.request('/v1/blob/notahex');
|
||||
expect(res.status).toBe(400);
|
||||
const res2 = await app.request(`/v1/blob/${'A'.repeat(64)}`);
|
||||
expect(res2.status).toBe(400);
|
||||
});
|
||||
|
||||
// ─── PUT (TOFU) ──────────────────────────────────────────────
|
||||
|
||||
test('first PUT creates slot and returns etag', async () => {
|
||||
const slotId = makeSlotId();
|
||||
const owner = await makeOwner();
|
||||
const blob = randBytes(128);
|
||||
const res = await signedPut({ slotId, blob, ...owner });
|
||||
expect(res.status).toBe(200);
|
||||
const json = (await res.json()) as { created: boolean; etag: string };
|
||||
expect(json.created).toBe(true);
|
||||
expect(typeof json.etag).toBe('string');
|
||||
|
||||
const got = await app.request(`/v1/blob/${slotId}`);
|
||||
expect(got.status).toBe(200);
|
||||
const back = (await got.json()) as { blob: string; etag: string };
|
||||
expect(fromBase64(back.blob)).toEqual(blob);
|
||||
expect(back.etag).toBe(json.etag);
|
||||
});
|
||||
|
||||
test('PUT without ifMatch on populated slot returns 409', async () => {
|
||||
const slotId = makeSlotId();
|
||||
const owner = await makeOwner();
|
||||
await signedPut({ slotId, blob: randBytes(64), ...owner });
|
||||
const res = await signedPut({ slotId, blob: randBytes(64), ...owner });
|
||||
expect(res.status).toBe(409);
|
||||
const json = (await res.json()) as { code: string };
|
||||
expect(json.code).toBe('SHADE_CONFLICT');
|
||||
});
|
||||
|
||||
test('PUT with stale ifMatch returns 412', async () => {
|
||||
const slotId = makeSlotId();
|
||||
const owner = await makeOwner();
|
||||
const r1 = await signedPut({ slotId, blob: randBytes(64), ...owner });
|
||||
const j1 = (await r1.json()) as { etag: string };
|
||||
// Use an etag we know does not match.
|
||||
const stale = String(Number(j1.etag) - 999);
|
||||
const res = await signedPut({
|
||||
slotId,
|
||||
blob: randBytes(64),
|
||||
...owner,
|
||||
ifMatch: stale,
|
||||
});
|
||||
expect(res.status).toBe(412);
|
||||
const json = (await res.json()) as { code: string };
|
||||
expect(json.code).toBe('SHADE_PRECONDITION_FAILED');
|
||||
});
|
||||
|
||||
test('PUT with matching ifMatch updates and bumps etag', async () => {
|
||||
const slotId = makeSlotId();
|
||||
const owner = await makeOwner();
|
||||
const r1 = await signedPut({ slotId, blob: randBytes(64), ...owner });
|
||||
const j1 = (await r1.json()) as { etag: string };
|
||||
const r2 = await signedPut({
|
||||
slotId,
|
||||
blob: randBytes(64),
|
||||
...owner,
|
||||
ifMatch: j1.etag,
|
||||
});
|
||||
expect(r2.status).toBe(200);
|
||||
const j2 = (await r2.json()) as { created: boolean; etag: string };
|
||||
expect(j2.created).toBe(false);
|
||||
expect(Number(j2.etag)).toBeGreaterThan(Number(j1.etag));
|
||||
});
|
||||
|
||||
test('PUT with ifMatch="*" unconditionally overwrites existing slot', async () => {
|
||||
const slotId = makeSlotId();
|
||||
const owner = await makeOwner();
|
||||
await signedPut({ slotId, blob: randBytes(64), ...owner });
|
||||
const res = await signedPut({
|
||||
slotId,
|
||||
blob: randBytes(64),
|
||||
...owner,
|
||||
ifMatch: '*',
|
||||
});
|
||||
expect(res.status).toBe(200);
|
||||
});
|
||||
|
||||
test('PUT with ifMatch="*" on empty slot returns 412', async () => {
|
||||
const slotId = makeSlotId();
|
||||
const owner = await makeOwner();
|
||||
const res = await signedPut({
|
||||
slotId,
|
||||
blob: randBytes(64),
|
||||
...owner,
|
||||
ifMatch: '*',
|
||||
});
|
||||
expect(res.status).toBe(412);
|
||||
});
|
||||
|
||||
test('PUT by a different owner key on existing slot is rejected', async () => {
|
||||
const slotId = makeSlotId();
|
||||
const ownerA = await makeOwner();
|
||||
await signedPut({ slotId, blob: randBytes(64), ...ownerA });
|
||||
|
||||
const ownerB = await makeOwner();
|
||||
const res = await signedPut({
|
||||
slotId,
|
||||
blob: randBytes(64),
|
||||
...ownerB,
|
||||
ifMatch: '*',
|
||||
});
|
||||
expect(res.status).toBe(401);
|
||||
});
|
||||
|
||||
test('PUT with bad signature is rejected', async () => {
|
||||
const slotId = makeSlotId();
|
||||
const owner = await makeOwner();
|
||||
// Sign the payload, then mutate the blob bytes — signature no
|
||||
// longer matches the canonicalized body.
|
||||
const blob = randBytes(64);
|
||||
const payload = {
|
||||
ownerPubkey: toBase64(owner.pubkey),
|
||||
blob: toBase64(blob),
|
||||
slotId,
|
||||
};
|
||||
const signed = await signPayload(crypto, owner.seed, payload);
|
||||
(signed as any).blob = toBase64(randBytes(64));
|
||||
const { slotId: _omit, ...wire } = signed as Record<string, unknown>;
|
||||
const res = await app.request(`/v1/blob/${slotId}`, {
|
||||
method: 'PUT',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(wire),
|
||||
});
|
||||
expect(res.status).toBe(401);
|
||||
});
|
||||
|
||||
test('PUT rejects empty blob and oversized blob', async () => {
|
||||
const slotId = makeSlotId();
|
||||
const owner = await makeOwner();
|
||||
const empty = await signedPut({ slotId, blob: new Uint8Array(0), ...owner });
|
||||
expect(empty.status).toBe(400);
|
||||
const tooBig = await signedPut({
|
||||
slotId,
|
||||
blob: randBytes(70 * 1024),
|
||||
...owner,
|
||||
});
|
||||
expect(tooBig.status).toBe(400);
|
||||
});
|
||||
|
||||
// ─── DELETE ──────────────────────────────────────────────────
|
||||
|
||||
test('DELETE clears slot and lets a fresh key TOFU re-claim', async () => {
|
||||
const slotId = makeSlotId();
|
||||
const ownerA = await makeOwner();
|
||||
await signedPut({ slotId, blob: randBytes(64), ...ownerA });
|
||||
|
||||
const del = await signedDelete({ slotId, seed: ownerA.seed });
|
||||
expect(del.status).toBe(200);
|
||||
|
||||
// Slot is gone.
|
||||
const gone = await app.request(`/v1/blob/${slotId}`);
|
||||
expect(gone.status).toBe(404);
|
||||
|
||||
// A fresh owner can now claim it.
|
||||
const ownerB = await makeOwner();
|
||||
const claim = await signedPut({ slotId, blob: randBytes(64), ...ownerB });
|
||||
expect(claim.status).toBe(200);
|
||||
});
|
||||
|
||||
test('DELETE by a different key is rejected', async () => {
|
||||
const slotId = makeSlotId();
|
||||
const ownerA = await makeOwner();
|
||||
await signedPut({ slotId, blob: randBytes(64), ...ownerA });
|
||||
|
||||
const ownerB = await makeOwner();
|
||||
const res = await signedDelete({ slotId, seed: ownerB.seed });
|
||||
expect(res.status).toBe(401);
|
||||
});
|
||||
|
||||
test('DELETE on missing slot returns 404', async () => {
|
||||
const slotId = makeSlotId();
|
||||
const owner = await makeOwner();
|
||||
const res = await signedDelete({ slotId, seed: owner.seed });
|
||||
expect(res.status).toBe(404);
|
||||
});
|
||||
|
||||
// ─── Cross-slot replay ───────────────────────────────────────
|
||||
|
||||
test('PUT signed for slot A is rejected against slot B', async () => {
|
||||
const slotA = makeSlotId();
|
||||
const slotB = makeSlotId();
|
||||
const owner = await makeOwner();
|
||||
const blob = randBytes(64);
|
||||
// Sign for slotA, send to slotB (URL).
|
||||
const payload = {
|
||||
ownerPubkey: toBase64(owner.pubkey),
|
||||
blob: toBase64(blob),
|
||||
slotId: slotA,
|
||||
};
|
||||
const signed = await signPayload(crypto, owner.seed, payload);
|
||||
const { slotId: _omit, ...wire } = signed as Record<string, unknown>;
|
||||
const res = await app.request(`/v1/blob/${slotB}`, {
|
||||
method: 'PUT',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(wire),
|
||||
});
|
||||
expect(res.status).toBe(401);
|
||||
});
|
||||
});
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shade/inbox",
|
||||
"version": "4.0.0",
|
||||
"version": "4.11.0",
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
"types": "src/index.ts",
|
||||
|
||||
208
packages/shade-inbox/src/blob-client.ts
Normal file
208
packages/shade-inbox/src/blob-client.ts
Normal file
@@ -0,0 +1,208 @@
|
||||
import type { CryptoProvider } from '@shade/core';
|
||||
import {
|
||||
NetworkError,
|
||||
toBase64,
|
||||
fromBase64,
|
||||
ShadeError,
|
||||
ValidationError,
|
||||
} from '@shade/core';
|
||||
import { signPayload } from '@shade/server';
|
||||
|
||||
/**
|
||||
* Low-level HTTP client for the V4.9 encrypted-blob primitive
|
||||
* (`/v1/blob/<slotId>`). Stateless and reusable; higher-level wrappers
|
||||
* (e.g. `Profile` in `@shade/sdk`) compose this client.
|
||||
*
|
||||
* The client doesn't care what the blob bytes mean — it just transports
|
||||
* them. Callers are responsible for AEAD-sealing/opening, deriving the
|
||||
* slotId from the master key, and managing the signing key.
|
||||
*/
|
||||
export interface BlobClientOptions {
|
||||
baseUrl: string;
|
||||
crypto: CryptoProvider;
|
||||
/** Optional fetch override (defaults to globalThis.fetch). */
|
||||
fetch?: typeof fetch;
|
||||
}
|
||||
|
||||
export interface BlobGetResult {
|
||||
blob: Uint8Array;
|
||||
/** ETag string — pass back as `ifMatch` to do a CAS update. */
|
||||
etag: string;
|
||||
updatedAt: number;
|
||||
}
|
||||
|
||||
export interface BlobPutResult {
|
||||
/** True if this PUT created the slot, false if it updated an existing one. */
|
||||
created: boolean;
|
||||
/** New ETag after the write. */
|
||||
etag: string;
|
||||
updatedAt: number;
|
||||
}
|
||||
|
||||
export class BlobClient {
|
||||
private readonly fetchImpl: typeof fetch;
|
||||
|
||||
constructor(private readonly options: BlobClientOptions) {
|
||||
const f = options.fetch ?? globalThis.fetch;
|
||||
this.fetchImpl = f.bind(globalThis);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a slot. Returns null if no blob has ever been written there
|
||||
* (or if it was DELETE'd). GET is unauthenticated — see the
|
||||
* `BlobStore` JSDoc for the threat-model rationale.
|
||||
*/
|
||||
async get(slotIdHex: string): Promise<BlobGetResult | null> {
|
||||
validateSlotIdHex(slotIdHex);
|
||||
const url = joinUrl(this.options.baseUrl, `/v1/blob/${slotIdHex}`);
|
||||
let res: Response;
|
||||
try {
|
||||
res = await this.fetchImpl(url, { method: 'GET' });
|
||||
} catch (err) {
|
||||
throw new NetworkError(`Blob GET failed: ${(err as Error).message}`);
|
||||
}
|
||||
if (res.status === 404) return null;
|
||||
const text = await res.text();
|
||||
let json: any;
|
||||
try {
|
||||
json = text.length > 0 ? JSON.parse(text) : {};
|
||||
} catch {
|
||||
throw new NetworkError(`Blob response not JSON: ${text.slice(0, 200)}`, res.status);
|
||||
}
|
||||
if (!res.ok) {
|
||||
throw new ShadeError(
|
||||
String(json.code ?? 'SHADE_NETWORK'),
|
||||
String(json.message ?? text),
|
||||
);
|
||||
}
|
||||
return {
|
||||
blob: fromBase64(String(json.blob)),
|
||||
etag: String(json.etag),
|
||||
updatedAt: Number(json.updatedAt),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Create or update a slot.
|
||||
*
|
||||
* `ifMatch` semantics:
|
||||
* - `undefined`: create-only. Slot must be empty (else 409).
|
||||
* - `<etag-string>`: compare-and-swap. Must match (else 412).
|
||||
* - `'*'`: unconditional overwrite. Slot must already exist (else 412).
|
||||
*/
|
||||
async put(args: {
|
||||
slotIdHex: string;
|
||||
blob: Uint8Array;
|
||||
/** 32-byte Ed25519 seed (== `signingPrivateKey`). */
|
||||
signingPrivateKey: Uint8Array;
|
||||
/** Pubkey paired to `signingPrivateKey`. */
|
||||
ownerPubkey: Uint8Array;
|
||||
ifMatch?: string;
|
||||
}): Promise<BlobPutResult> {
|
||||
validateSlotIdHex(args.slotIdHex);
|
||||
if (args.blob.length === 0) {
|
||||
throw new ValidationError('Empty blob');
|
||||
}
|
||||
if (args.ownerPubkey.length !== 32) {
|
||||
throw new ValidationError('ownerPubkey must be 32 bytes (Ed25519)');
|
||||
}
|
||||
|
||||
const payload: Record<string, unknown> = {
|
||||
ownerPubkey: toBase64(args.ownerPubkey),
|
||||
blob: toBase64(args.blob),
|
||||
slotId: args.slotIdHex,
|
||||
};
|
||||
if (args.ifMatch !== undefined) payload.ifMatch = args.ifMatch;
|
||||
|
||||
const signed = await signPayload(
|
||||
this.options.crypto,
|
||||
args.signingPrivateKey,
|
||||
payload,
|
||||
);
|
||||
// `slotId` was used for the signature canonicalization to bind it
|
||||
// into the payload; the server rebuilds the same canonical form
|
||||
// by mixing the URL slotId back in. Strip it from the wire body
|
||||
// so we don't send it twice (URL is the path param).
|
||||
const { slotId: _omit, ...wireBody } = signed as Record<string, unknown>;
|
||||
|
||||
const url = joinUrl(this.options.baseUrl, `/v1/blob/${args.slotIdHex}`);
|
||||
const json = await this.requestJson('PUT', url, wireBody);
|
||||
return {
|
||||
created: Boolean(json.created),
|
||||
etag: String(json.etag),
|
||||
updatedAt: Number(json.updatedAt),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete a slot — the next PUT TOFU-claims it again, possibly under
|
||||
* a fresh signing key (e.g. after a rotation). Used by the "forget
|
||||
* everything" path.
|
||||
*/
|
||||
async delete(args: {
|
||||
slotIdHex: string;
|
||||
signingPrivateKey: Uint8Array;
|
||||
}): Promise<boolean> {
|
||||
validateSlotIdHex(args.slotIdHex);
|
||||
const signed = await signPayload(this.options.crypto, args.signingPrivateKey, {
|
||||
slotId: args.slotIdHex,
|
||||
});
|
||||
const { slotId: _omit, ...wireBody } = signed as Record<string, unknown>;
|
||||
const url = joinUrl(this.options.baseUrl, `/v1/blob/${args.slotIdHex}`);
|
||||
const json = await this.requestJson('DELETE', url, wireBody);
|
||||
return Boolean(json.ok);
|
||||
}
|
||||
|
||||
// ─── HTTP plumbing ──────────────────────────────────────────
|
||||
|
||||
private async requestJson(method: string, url: string, body: unknown): Promise<any> {
|
||||
let res: Response;
|
||||
try {
|
||||
res = await this.fetchImpl(url, {
|
||||
method,
|
||||
headers: { 'content-type': 'application/json' },
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
} catch (err) {
|
||||
throw new NetworkError(`Blob request failed: ${(err as Error).message}`);
|
||||
}
|
||||
const text = await res.text();
|
||||
let json: any;
|
||||
try {
|
||||
json = text.length > 0 ? JSON.parse(text) : {};
|
||||
} catch {
|
||||
throw new NetworkError(`Blob response not JSON: ${text.slice(0, 200)}`, res.status);
|
||||
}
|
||||
if (!res.ok) {
|
||||
throw new ShadeError(
|
||||
String(json.code ?? 'SHADE_NETWORK'),
|
||||
String(json.message ?? text),
|
||||
);
|
||||
}
|
||||
return json;
|
||||
}
|
||||
}
|
||||
|
||||
function validateSlotIdHex(s: string): void {
|
||||
if (!/^[0-9a-f]{64}$/.test(s)) {
|
||||
throw new ValidationError('slotIdHex must be 64 lowercase hex chars (32 bytes)');
|
||||
}
|
||||
}
|
||||
|
||||
function joinUrl(base: string, path: string): string {
|
||||
if (base.endsWith('/') && path.startsWith('/')) return base + path.slice(1);
|
||||
if (!base.endsWith('/') && !path.startsWith('/')) return base + '/' + path;
|
||||
return base + path;
|
||||
}
|
||||
|
||||
/** Convert a 32-byte slotId Uint8Array into the lowercase-hex wire form. */
|
||||
export function slotIdToHex(slotId: Uint8Array): string {
|
||||
if (slotId.length !== 32) {
|
||||
throw new ValidationError('slotId must be 32 bytes');
|
||||
}
|
||||
let s = '';
|
||||
for (let i = 0; i < slotId.length; i++) {
|
||||
s += slotId[i]!.toString(16).padStart(2, '0');
|
||||
}
|
||||
return s;
|
||||
}
|
||||
@@ -40,6 +40,16 @@ export interface FetchedBlob {
|
||||
receivedAt: number;
|
||||
/** Absolute expiry time (ms since epoch) reported by the server. */
|
||||
expiresAt: number;
|
||||
/**
|
||||
* Sender fingerprint — 8-byte hex of SHA-256(senderSigningKey),
|
||||
* captured by the relay at PUT time when the sender's signature was
|
||||
* verified. Empty/undefined when the relay is pre-4.8 or the blob
|
||||
* predates sender-fingerprint tracking. Use as an unknown-sender
|
||||
* bootstrap label (`fp:<hex>`) for X3DH first-contact; the
|
||||
* authoritative sender identity is recovered post-decrypt from the
|
||||
* envelope itself, so `from` is a hint, not a trust anchor.
|
||||
*/
|
||||
from?: string;
|
||||
}
|
||||
|
||||
export interface FetchResult {
|
||||
@@ -52,7 +62,14 @@ export class InboxClient {
|
||||
private readonly fetchImpl: typeof fetch;
|
||||
|
||||
constructor(private readonly options: InboxClientOptions) {
|
||||
this.fetchImpl = options.fetch ?? globalThis.fetch;
|
||||
// Bind once. The browser's `globalThis.fetch` is a WebIDL bound
|
||||
// operation that throws "Illegal invocation" when called as a method
|
||||
// on another object (which is what `this.fetchImpl(...)` does).
|
||||
// Node/Bun fetch tolerates a free receiver, but binding is harmless.
|
||||
// A consumer-supplied `options.fetch` is bound to the global too —
|
||||
// a fetch that requires a specific receiver must bind itself.
|
||||
const f = options.fetch ?? globalThis.fetch;
|
||||
this.fetchImpl = f.bind(globalThis);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -144,12 +161,18 @@ export class InboxClient {
|
||||
);
|
||||
const blobs = Array.isArray(json.blobs) ? json.blobs : [];
|
||||
return {
|
||||
blobs: blobs.map((b: any) => ({
|
||||
msgId: String(b.msgId),
|
||||
ciphertext: fromBase64(String(b.ciphertext)),
|
||||
receivedAt: Number(b.receivedAt),
|
||||
expiresAt: Number(b.expiresAt),
|
||||
})),
|
||||
blobs: blobs.map((b: any): FetchedBlob => {
|
||||
const out: FetchedBlob = {
|
||||
msgId: String(b.msgId),
|
||||
ciphertext: fromBase64(String(b.ciphertext)),
|
||||
receivedAt: Number(b.receivedAt),
|
||||
expiresAt: Number(b.expiresAt),
|
||||
};
|
||||
// V4.8 — relay-supplied sender fingerprint hint. Optional; absent
|
||||
// on pre-4.8 relays or for blobs persisted before tracking landed.
|
||||
if (typeof b.from === 'string' && b.from.length > 0) out.from = b.from;
|
||||
return out;
|
||||
}),
|
||||
cursor: Number(json.cursor ?? sinceCursor),
|
||||
hasMore: Boolean(json.hasMore),
|
||||
};
|
||||
|
||||
@@ -16,9 +16,20 @@ import { InboxClientEvents, type InboxClientListener } from './events.js';
|
||||
* decrypt path it owns) and either return a sender-hint for telemetry
|
||||
* (the address the SDK extracted, or `null`) or throw to keep the blob
|
||||
* on the server for a later retry.
|
||||
*
|
||||
* V4.8: `raw.from` is the relay-captured sender fingerprint (8-byte hex
|
||||
* of SHA-256 over the sender's signing key). Empty when the relay is
|
||||
* pre-4.8 or didn't track the sender. Use it as the `fp:<hex>` bootstrap
|
||||
* label for X3DH first-contact handshakes.
|
||||
*/
|
||||
export type DecryptHandler = (
|
||||
raw: { msgId: string; ciphertext: Uint8Array; receivedAt: number; expiresAt: number },
|
||||
raw: {
|
||||
msgId: string;
|
||||
ciphertext: Uint8Array;
|
||||
receivedAt: number;
|
||||
expiresAt: number;
|
||||
from?: string;
|
||||
},
|
||||
) => Promise<string | null | undefined> | string | null | undefined;
|
||||
|
||||
export interface InboxOptions {
|
||||
@@ -52,6 +63,14 @@ export interface InboxOptions {
|
||||
const DEFAULT_TTL_SECONDS = 7 * 24 * 60 * 60;
|
||||
const DEFAULT_POLL_INTERVAL_MS = 30_000;
|
||||
const DEFAULT_MAX_ATTEMPTS = 10;
|
||||
/**
|
||||
* Cap for the cross-channel msgId dedup LRU. Each entry is a 64-char hex
|
||||
* string; 4096 entries ≈ 256 KiB of overhead, plenty of headroom for
|
||||
* bursty traffic (the LRU only needs to span the window between a bridge
|
||||
* push and the next inbox-poll catching up — typically 30 s × the
|
||||
* recipient's throughput).
|
||||
*/
|
||||
const DEFAULT_DEDUP_LRU_CAP = 4096;
|
||||
|
||||
/**
|
||||
* High-level inbox orchestrator.
|
||||
@@ -94,6 +113,23 @@ export class Inbox {
|
||||
private started = false;
|
||||
private registered = false;
|
||||
|
||||
/**
|
||||
* Bounded msgId dedup window. Used by both the inbox-poll path
|
||||
* (`pollOnce` → `handleBlob`) and the bridge-push path
|
||||
* (`acceptBridgeFrame`). The relay stores blobs durably and pushes
|
||||
* them to every active delivery channel; without a shared dedup gate
|
||||
* here the recipient processes the same envelope twice — once from
|
||||
* the bridge, again from the next inbox-poll. The duplicate receive
|
||||
* trips on consumed one-time prekeys ("OPK not found") and pollutes
|
||||
* logs even when the canonical first delivery succeeded. See V4.8.3
|
||||
* Prism FR `cross-channel-duplicate-fanout-v4.8.2.md`.
|
||||
*
|
||||
* Insertion order is FIFO; the oldest msgId is evicted once the LRU
|
||||
* exceeds `DEFAULT_DEDUP_LRU_CAP`.
|
||||
*/
|
||||
private readonly deliveredIds = new Set<string>();
|
||||
private readonly deliveredOrder: string[] = [];
|
||||
|
||||
constructor(private readonly options: InboxOptions) {
|
||||
const clientOptions: ConstructorParameters<typeof InboxClient>[0] = {
|
||||
baseUrl: options.baseUrl,
|
||||
@@ -149,6 +185,14 @@ export class Inbox {
|
||||
signingKey: this.options.signingPublicKey,
|
||||
});
|
||||
this.registered = true;
|
||||
// V4.8: gate the first poll on register success. `start()` calls
|
||||
// `register()` fire-and-forget; without this kick, the very first
|
||||
// `pollOnce()` (scheduled synchronously alongside register) would
|
||||
// race the register HTTP RTT and return SHADE_NOT_FOUND. The
|
||||
// pollOnce() guard skips polls until `registered === true`; this
|
||||
// immediate schedule ensures we don't wait the full pollIntervalMs
|
||||
// for the next attempt once register lands.
|
||||
if (this.started) this.schedulePoll(0);
|
||||
}
|
||||
|
||||
/** Drop the address from the server. Local queue/cursor are preserved. */
|
||||
@@ -203,7 +247,10 @@ export class Inbox {
|
||||
* after a push-trigger arrives). Does not throw on transient errors.
|
||||
*/
|
||||
async tick(): Promise<{ flushed: number; received: number }> {
|
||||
const flushed = await this.flushOnce();
|
||||
const flushResult = await this.flushOnce();
|
||||
// `null` means another flush was concurrent; report 0 newly-flushed
|
||||
// for this caller (the other flush counted them).
|
||||
const flushed = flushResult?.delivered ?? 0;
|
||||
const received = await this.pollOnce();
|
||||
return { flushed, received };
|
||||
}
|
||||
@@ -217,7 +264,11 @@ export class Inbox {
|
||||
this.scheduleRegisterRetry();
|
||||
});
|
||||
this.scheduleFlush();
|
||||
this.schedulePoll(0);
|
||||
// V4.8: do NOT schedule the first poll synchronously. `register()`
|
||||
// success kicks `schedulePoll(0)` so the first poll fires after the
|
||||
// server has acknowledged the address. Pre-fix this raced register
|
||||
// and burned a 404 on every fresh-address `start()`.
|
||||
if (this.registered) this.schedulePoll(0);
|
||||
}
|
||||
|
||||
/** Stop background timers. Pending entries remain in the queue. */
|
||||
@@ -253,11 +304,27 @@ export class Inbox {
|
||||
this.flushTimer = setTimeout(() => {
|
||||
this.flushTimer = null;
|
||||
this.flushOnce()
|
||||
.then(() => {
|
||||
// If anything is still queued, retry with backoff.
|
||||
this.queueStore.size().then((n) => {
|
||||
if (n > 0 && this.started) this.scheduleFlush(15_000);
|
||||
});
|
||||
.then((result) => {
|
||||
// `result === null` means another flush was already in flight
|
||||
// and this call early-returned via the `flushing` guard. The
|
||||
// already-running flush will reschedule itself when it
|
||||
// finishes; do not double-schedule from here.
|
||||
if (result === null) return;
|
||||
if (result.remaining === 0) return;
|
||||
// V4.8.5 — distinguish healthy-drain-but-more-queued from
|
||||
// all-attempts-failed. Pre-fix, both cases used a 15 s
|
||||
// backoff. Under sustained traffic (Prism's typing-into-a-
|
||||
// chatty-shell pattern), bursts of envelopes enqueued
|
||||
// *during* a flush would sit ~10–15 s behind the backoff
|
||||
// timer before the next drain — visible to the receiver as a
|
||||
// "10 s silence then 25-frame burst" wave. Healthy drain
|
||||
// (delivered > 0) means the network is fine and we should
|
||||
// immediately drain whatever piled up; reserve the 15 s
|
||||
// retry for the case where every attempt this round failed.
|
||||
if (this.started) {
|
||||
const delay = result.delivered > 0 ? 0 : 15_000;
|
||||
this.scheduleFlush(delay);
|
||||
}
|
||||
})
|
||||
.catch(() => {
|
||||
if (this.started) this.scheduleFlush(15_000);
|
||||
@@ -276,45 +343,87 @@ export class Inbox {
|
||||
}, delayMs);
|
||||
}
|
||||
|
||||
private async flushOnce(): Promise<number> {
|
||||
if (this.flushing) return 0;
|
||||
/**
|
||||
* Drain the outgoing queue. Returns `null` if another flush is already
|
||||
* in flight (the running flush owns the rescheduling); otherwise
|
||||
* returns the count of newly-delivered envelopes and the queue size
|
||||
* after the drain so the caller can decide whether to immediately
|
||||
* re-flush (more piled up during the drain — healthy network) or
|
||||
* back off (everything failed).
|
||||
*
|
||||
* V4.8.5: drain is parallel-per-recipient. Each `recipientAddress`
|
||||
* gets its own sequential worker (so per-peer order is preserved),
|
||||
* but distinct recipients run concurrently. Pre-fix, a single slow
|
||||
* POST head-of-line-blocked the entire queue — including small
|
||||
* frames bound for unrelated peers. See Prism FR
|
||||
* `per-recipient-flush-concurrency-v4.8.md`.
|
||||
*/
|
||||
private async flushOnce(): Promise<{ delivered: number; remaining: number } | null> {
|
||||
if (this.flushing) return null;
|
||||
this.flushing = true;
|
||||
let delivered = 0;
|
||||
try {
|
||||
const entries = await this.queueStore.list();
|
||||
// Group by recipient. Within a bucket we drain sequentially so
|
||||
// per-peer message order matches enqueue order (the relay
|
||||
// assigns `receivedAt` on PUT arrival; concurrent POSTs to the
|
||||
// same peer would let the second arrive first and the recipient
|
||||
// would observe out-of-order envelopes). Across buckets, no
|
||||
// ordering guarantee exists in Shade's wire model anyway, so
|
||||
// parallel drain is safe.
|
||||
const buckets = new Map<string, OutgoingEntry[]>();
|
||||
for (const entry of entries) {
|
||||
try {
|
||||
const result = await this.client.put({
|
||||
recipientAddress: entry.recipientAddress,
|
||||
senderSigningKey: this.options.signingPublicKey,
|
||||
envelope: entry.ciphertext,
|
||||
ttlSeconds: entry.ttlSeconds,
|
||||
});
|
||||
await this.queueStore.remove(entry.recipientAddress, entry.msgId);
|
||||
delivered++;
|
||||
this.events.emit('inbox.message_delivered', {
|
||||
recipientAddress: entry.recipientAddress,
|
||||
msgId: result.msgId,
|
||||
idempotent: result.idempotent,
|
||||
});
|
||||
} catch (err) {
|
||||
await this.queueStore.bumpAttempts(entry.recipientAddress, entry.msgId);
|
||||
const attempts = entry.attempts + 1;
|
||||
this.events.emit('inbox.message_failed', {
|
||||
recipientAddress: entry.recipientAddress,
|
||||
msgId: entry.msgId,
|
||||
attempts,
|
||||
error: (err as Error).message,
|
||||
});
|
||||
if (attempts >= this.maxAttempts) {
|
||||
let bucket = buckets.get(entry.recipientAddress);
|
||||
if (!bucket) {
|
||||
bucket = [];
|
||||
buckets.set(entry.recipientAddress, bucket);
|
||||
}
|
||||
bucket.push(entry);
|
||||
}
|
||||
|
||||
const drainBucket = async (bucket: OutgoingEntry[]): Promise<number> => {
|
||||
let count = 0;
|
||||
for (const entry of bucket) {
|
||||
try {
|
||||
const result = await this.client.put({
|
||||
recipientAddress: entry.recipientAddress,
|
||||
senderSigningKey: this.options.signingPublicKey,
|
||||
envelope: entry.ciphertext,
|
||||
ttlSeconds: entry.ttlSeconds,
|
||||
});
|
||||
await this.queueStore.remove(entry.recipientAddress, entry.msgId);
|
||||
count++;
|
||||
this.events.emit('inbox.message_delivered', {
|
||||
recipientAddress: entry.recipientAddress,
|
||||
msgId: result.msgId,
|
||||
idempotent: result.idempotent,
|
||||
});
|
||||
} catch (err) {
|
||||
await this.queueStore.bumpAttempts(entry.recipientAddress, entry.msgId);
|
||||
const attempts = entry.attempts + 1;
|
||||
this.events.emit('inbox.message_failed', {
|
||||
recipientAddress: entry.recipientAddress,
|
||||
msgId: entry.msgId,
|
||||
attempts,
|
||||
error: (err as Error).message,
|
||||
});
|
||||
if (attempts >= this.maxAttempts) {
|
||||
await this.queueStore.remove(entry.recipientAddress, entry.msgId);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return count;
|
||||
};
|
||||
|
||||
const counts = await Promise.all(
|
||||
Array.from(buckets.values(), drainBucket),
|
||||
);
|
||||
delivered = counts.reduce((a, b) => a + b, 0);
|
||||
} finally {
|
||||
this.flushing = false;
|
||||
}
|
||||
return delivered;
|
||||
const remaining = await this.queueStore.size();
|
||||
return { delivered, remaining };
|
||||
}
|
||||
|
||||
private async pollOnce(): Promise<number> {
|
||||
@@ -357,9 +466,52 @@ export class Inbox {
|
||||
return total;
|
||||
}
|
||||
|
||||
/**
|
||||
* Feed a blob delivered by a bridge transport (WS / SSE / long-poll
|
||||
* push) into the same dispatch + ack pipeline that `pollOnce` uses.
|
||||
*
|
||||
* Wire-up pattern:
|
||||
* ```ts
|
||||
* const bridge = new FallbackBridgeTransport([...]);
|
||||
* await bridge.connect({
|
||||
* onMessage: async (msg) => {
|
||||
* await inbox.acceptBridgeFrame({
|
||||
* msgId: msg.msgId!, // present on v4.8+ relays
|
||||
* ciphertext: msg.bytes,
|
||||
* receivedAt: msg.receivedAt,
|
||||
* expiresAt: msg.expiresAt ?? Date.now() + 7 * 24 * 3600 * 1000,
|
||||
* ...(msg.from !== undefined ? { from: msg.from } : {}),
|
||||
* });
|
||||
* },
|
||||
* });
|
||||
* ```
|
||||
*
|
||||
* The Inbox's bounded msgId LRU is shared between this path and
|
||||
* `pollOnce`, so whichever channel delivers first wins; the
|
||||
* other channel acks-and-skips when the same msgId comes back
|
||||
* around. Both paths also DELETE the blob from the relay on success
|
||||
* so subsequent polls don't see it either.
|
||||
*
|
||||
* Returns `true` if the blob was newly dispatched, `false` if it
|
||||
* was a duplicate or rejected by the handler (handler still gets a
|
||||
* chance to retry on the next poll if it threw).
|
||||
*/
|
||||
async acceptBridgeFrame(blob: FetchedBlob): Promise<boolean> {
|
||||
return this.handleBlob(blob);
|
||||
}
|
||||
|
||||
private async handleBlob(blob: FetchedBlob): Promise<boolean> {
|
||||
if (!this.incomingHandler) return false;
|
||||
|
||||
// Cross-channel msgId dedup. If the bridge already delivered this
|
||||
// blob, the inbox-poll copy must not re-dispatch (would re-trigger
|
||||
// X3DH / consume an OPK we no longer have). We still ack so the
|
||||
// relay drops the now-redundant copy.
|
||||
if (this.deliveredIds.has(blob.msgId)) {
|
||||
await this.ackQuietly(blob.msgId);
|
||||
return false;
|
||||
}
|
||||
|
||||
// Defense-in-depth: verify msgId ↔ ciphertext at the client too. A
|
||||
// server bug or malicious operator can't sneak a different blob past
|
||||
// the client's hash check.
|
||||
@@ -375,12 +527,20 @@ export class Inbox {
|
||||
|
||||
let senderHint: string | null = null;
|
||||
try {
|
||||
const result = await this.incomingHandler({
|
||||
const raw: {
|
||||
msgId: string;
|
||||
ciphertext: Uint8Array;
|
||||
receivedAt: number;
|
||||
expiresAt: number;
|
||||
from?: string;
|
||||
} = {
|
||||
msgId: blob.msgId,
|
||||
ciphertext: blob.ciphertext,
|
||||
receivedAt: blob.receivedAt,
|
||||
expiresAt: blob.expiresAt,
|
||||
});
|
||||
};
|
||||
if (blob.from !== undefined) raw.from = blob.from;
|
||||
const result = await this.incomingHandler(raw);
|
||||
senderHint = result ?? null;
|
||||
} catch (err) {
|
||||
this.events.emit('inbox.message_decrypt_failed', {
|
||||
@@ -391,17 +551,34 @@ export class Inbox {
|
||||
return false;
|
||||
}
|
||||
|
||||
try {
|
||||
await this.client.ack({ address: this.options.ownAddress, msgId: blob.msgId });
|
||||
} catch (err) {
|
||||
// Decryption succeeded; ack just failed. Will be retried later, and
|
||||
// the duplicate-message ratchet check on `Shade.receive` will dedupe.
|
||||
console.warn('[Shade] Inbox ack failed (will retry):', (err as Error).message);
|
||||
}
|
||||
// Mark before the ack so a slow-network ack doesn't leave a window
|
||||
// where a parallel pollOnce sees the same msgId and re-dispatches.
|
||||
this.recordDelivered(blob.msgId);
|
||||
await this.ackQuietly(blob.msgId);
|
||||
this.events.emit('inbox.message_received', {
|
||||
senderHint,
|
||||
msgId: blob.msgId,
|
||||
});
|
||||
return true;
|
||||
}
|
||||
|
||||
private async ackQuietly(msgId: string): Promise<void> {
|
||||
try {
|
||||
await this.client.ack({ address: this.options.ownAddress, msgId });
|
||||
} catch (err) {
|
||||
// Dispatch (or skip) succeeded; the ack just failed. Next poll
|
||||
// will see the blob again and the dedup gate above will skip it.
|
||||
console.warn('[Shade] Inbox ack failed (will retry):', (err as Error).message);
|
||||
}
|
||||
}
|
||||
|
||||
private recordDelivered(msgId: string): void {
|
||||
if (this.deliveredIds.has(msgId)) return;
|
||||
this.deliveredIds.add(msgId);
|
||||
this.deliveredOrder.push(msgId);
|
||||
if (this.deliveredOrder.length > DEFAULT_DEDUP_LRU_CAP) {
|
||||
const evicted = this.deliveredOrder.shift()!;
|
||||
this.deliveredIds.delete(evicted);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -43,3 +43,11 @@ export type {
|
||||
} from './events.js';
|
||||
|
||||
export { computeMsgId } from './msg-id.js';
|
||||
|
||||
// V4.9 — encrypted-blob primitive (`/v1/blob/<slotId>`).
|
||||
export { BlobClient, slotIdToHex } from './blob-client.js';
|
||||
export type {
|
||||
BlobClientOptions,
|
||||
BlobGetResult,
|
||||
BlobPutResult,
|
||||
} from './blob-client.js';
|
||||
|
||||
@@ -170,6 +170,255 @@ describe('Inbox orchestrator', () => {
|
||||
expect(seen[1]!.to).toBe('carol');
|
||||
});
|
||||
|
||||
test('cross-channel dedup: acceptBridgeFrame + pollOnce never re-dispatch the same msgId (V4.8.3)', async () => {
|
||||
// Reproduces the Prism FR `cross-channel-duplicate-fanout-v4.8.2`:
|
||||
// a single relay PUT was being delivered twice — once via WS bridge
|
||||
// push, again ~30 s later via inbox-poll catching up. Both copies
|
||||
// would dispatch `shade.receive`, the second one tripping on
|
||||
// already-consumed prekeys. The cross-channel msgId LRU inside
|
||||
// Inbox is the dedup gate; this test exercises it directly via
|
||||
// `acceptBridgeFrame` followed by `pollOnce`.
|
||||
const store = new MemoryInboxStore();
|
||||
const app = createInboxServer({ crypto, store, disableRateLimit: true });
|
||||
const bob = await makeIdentity();
|
||||
const alice = await makeIdentity();
|
||||
|
||||
const bobInbox = new Inbox({
|
||||
baseUrl: 'http://localhost',
|
||||
ownAddress: 'bob',
|
||||
crypto,
|
||||
signingPrivateKey: bob.signingPrivateKey,
|
||||
signingPublicKey: bob.signingPublicKey,
|
||||
pollIntervalMs: 0,
|
||||
fetch: honoFetch(app),
|
||||
});
|
||||
await bobInbox.register();
|
||||
|
||||
// Alice PUTs a blob via the relay HTTP API.
|
||||
const ct = randBytes(64);
|
||||
const msgId = await computeMsgId(ct);
|
||||
const aliceClient = new InboxClient({
|
||||
baseUrl: 'http://localhost',
|
||||
crypto,
|
||||
signingPrivateKey: alice.signingPrivateKey,
|
||||
fetch: honoFetch(app),
|
||||
});
|
||||
const putResult = await aliceClient.put({
|
||||
recipientAddress: 'bob',
|
||||
senderSigningKey: alice.signingPublicKey,
|
||||
envelope: ct,
|
||||
});
|
||||
expect(putResult.idempotent).toBe(false);
|
||||
|
||||
const dispatched: string[] = [];
|
||||
bobInbox.onIncoming(async (raw) => {
|
||||
dispatched.push(raw.msgId);
|
||||
return null;
|
||||
});
|
||||
|
||||
// Simulate the bridge push arriving first.
|
||||
await bobInbox.acceptBridgeFrame({
|
||||
msgId,
|
||||
ciphertext: ct,
|
||||
receivedAt: putResult.receivedAt,
|
||||
expiresAt: Date.now() + 60_000,
|
||||
});
|
||||
expect(dispatched).toEqual([msgId]);
|
||||
|
||||
// The inbox-poll path catches up next — without dedup it would
|
||||
// re-dispatch. With the LRU it acks-and-skips.
|
||||
const polled = await bobInbox.tick();
|
||||
expect(polled.received).toBe(0);
|
||||
expect(dispatched).toEqual([msgId]); // still one entry
|
||||
});
|
||||
|
||||
test('cross-channel dedup also covers poll-first then bridge-second order', async () => {
|
||||
const store = new MemoryInboxStore();
|
||||
const app = createInboxServer({ crypto, store, disableRateLimit: true });
|
||||
const bob = await makeIdentity();
|
||||
const alice = await makeIdentity();
|
||||
|
||||
const bobInbox = new Inbox({
|
||||
baseUrl: 'http://localhost',
|
||||
ownAddress: 'bob',
|
||||
crypto,
|
||||
signingPrivateKey: bob.signingPrivateKey,
|
||||
signingPublicKey: bob.signingPublicKey,
|
||||
pollIntervalMs: 0,
|
||||
fetch: honoFetch(app),
|
||||
});
|
||||
await bobInbox.register();
|
||||
const ct = randBytes(48);
|
||||
const msgId = await computeMsgId(ct);
|
||||
const aliceClient = new InboxClient({
|
||||
baseUrl: 'http://localhost',
|
||||
crypto,
|
||||
signingPrivateKey: alice.signingPrivateKey,
|
||||
fetch: honoFetch(app),
|
||||
});
|
||||
const putRes = await aliceClient.put({
|
||||
recipientAddress: 'bob',
|
||||
senderSigningKey: alice.signingPublicKey,
|
||||
envelope: ct,
|
||||
});
|
||||
|
||||
const dispatched: string[] = [];
|
||||
bobInbox.onIncoming(async (raw) => {
|
||||
dispatched.push(raw.msgId);
|
||||
return null;
|
||||
});
|
||||
|
||||
// Poll first.
|
||||
const polled = await bobInbox.tick();
|
||||
expect(polled.received).toBe(1);
|
||||
|
||||
// Bridge frame for the same msgId arrives after the poll already
|
||||
// dispatched + ack'd it — must be a no-op.
|
||||
const handled = await bobInbox.acceptBridgeFrame({
|
||||
msgId,
|
||||
ciphertext: ct,
|
||||
receivedAt: putRes.receivedAt,
|
||||
expiresAt: Date.now() + 60_000,
|
||||
});
|
||||
expect(handled).toBe(false);
|
||||
expect(dispatched).toEqual([msgId]);
|
||||
});
|
||||
|
||||
test('burst enqueued during a flush drains immediately, not after 15 s backoff (V4.8.5)', async () => {
|
||||
// Reproduces Prism FR `per-recipient-flush-concurrency-v4.8`: a
|
||||
// burst of envelopes enqueued *during* a slow POST used to sit
|
||||
// ~15 s behind the next flush because both the success path and
|
||||
// the failure path of `flushOnce` rescheduled with the same 15 s
|
||||
// backoff. The fix uses 0 ms when the round delivered something
|
||||
// (network is healthy — drain remainder) and reserves 15 s for
|
||||
// the all-attempts-failed case.
|
||||
const store = new MemoryInboxStore();
|
||||
const app = createInboxServer({ crypto, store, disableRateLimit: true });
|
||||
const bob = await makeIdentity();
|
||||
const alice = await makeIdentity();
|
||||
const bobClient = new InboxClient({
|
||||
baseUrl: 'http://localhost',
|
||||
crypto,
|
||||
signingPrivateKey: bob.signingPrivateKey,
|
||||
fetch: honoFetch(app),
|
||||
});
|
||||
await bobClient.register({ address: 'bob', signingKey: bob.signingPublicKey });
|
||||
|
||||
// Wrap fetch so the FIRST PUT (only) takes 100 ms — long enough
|
||||
// for many enqueues to land while it's in flight.
|
||||
let firstPutSeen = false;
|
||||
const slowFirstFetch: typeof fetch = (async (input, init) => {
|
||||
const u =
|
||||
typeof input === 'string'
|
||||
? input
|
||||
: input instanceof URL
|
||||
? input.toString()
|
||||
: (input as Request).url;
|
||||
const isPut = u.includes('/v1/inbox/bob') && !u.includes('/fetch');
|
||||
if (isPut && !firstPutSeen) {
|
||||
firstPutSeen = true;
|
||||
await new Promise((r) => setTimeout(r, 100));
|
||||
}
|
||||
return honoFetch(app)(input, init);
|
||||
}) as typeof fetch;
|
||||
|
||||
const aliceInbox = new Inbox({
|
||||
baseUrl: 'http://localhost',
|
||||
ownAddress: 'alice',
|
||||
crypto,
|
||||
signingPrivateKey: alice.signingPrivateKey,
|
||||
signingPublicKey: alice.signingPublicKey,
|
||||
pollIntervalMs: 0,
|
||||
fetch: slowFirstFetch,
|
||||
});
|
||||
|
||||
aliceInbox.start();
|
||||
|
||||
// First send — this kicks the slow-PUT path.
|
||||
await aliceInbox.send({ recipientAddress: 'bob', envelope: randBytes(20) });
|
||||
|
||||
// Pile 24 more on top while the first PUT is still in flight. The
|
||||
// first PUT will finish at ~T+100 ms; the subsequent 24 should
|
||||
// drain immediately after, NOT after a 15 s backoff.
|
||||
for (let i = 0; i < 24; i++) {
|
||||
await aliceInbox.send({ recipientAddress: 'bob', envelope: randBytes(20) });
|
||||
}
|
||||
|
||||
// Wait long enough for the slow first PUT + the immediate
|
||||
// reschedule + the 24-envelope drain. Pre-fix this would still
|
||||
// have ≥1 entry pending after 1 s (waiting for the 15 s timer).
|
||||
await new Promise((r) => setTimeout(r, 1_000));
|
||||
expect(await aliceInbox.pendingCount()).toBe(0);
|
||||
aliceInbox.stop();
|
||||
});
|
||||
|
||||
test('per-recipient parallel drain — slow POST to A does not block POSTs to B (V4.8.5)', async () => {
|
||||
const store = new MemoryInboxStore();
|
||||
const app = createInboxServer({ crypto, store, disableRateLimit: true });
|
||||
const alice = await makeIdentity();
|
||||
const bob = await makeIdentity();
|
||||
const carol = await makeIdentity();
|
||||
// Register bob + carol.
|
||||
const reg = async (name: string, kp: { signingPrivateKey: Uint8Array; signingPublicKey: Uint8Array }) => {
|
||||
const c = new InboxClient({
|
||||
baseUrl: 'http://localhost',
|
||||
crypto,
|
||||
signingPrivateKey: kp.signingPrivateKey,
|
||||
fetch: honoFetch(app),
|
||||
});
|
||||
await c.register({ address: name, signingKey: kp.signingPublicKey });
|
||||
};
|
||||
await reg('bob', bob);
|
||||
await reg('carol', carol);
|
||||
|
||||
// bob's PUT route stalls 200 ms; carol's is instant. Pre-fix this
|
||||
// would head-of-line block carol behind bob.
|
||||
const slowedFetch: typeof fetch = (async (input, init) => {
|
||||
const u =
|
||||
typeof input === 'string'
|
||||
? input
|
||||
: input instanceof URL
|
||||
? input.toString()
|
||||
: (input as Request).url;
|
||||
const m = (init as RequestInit | undefined)?.method ?? 'GET';
|
||||
if (m === 'POST' && u.includes('/v1/inbox/bob') && !u.includes('/fetch')) {
|
||||
await new Promise((r) => setTimeout(r, 200));
|
||||
}
|
||||
return honoFetch(app)(input, init);
|
||||
}) as typeof fetch;
|
||||
|
||||
const aliceInbox = new Inbox({
|
||||
baseUrl: 'http://localhost',
|
||||
ownAddress: 'alice',
|
||||
crypto,
|
||||
signingPrivateKey: alice.signingPrivateKey,
|
||||
signingPublicKey: alice.signingPublicKey,
|
||||
pollIntervalMs: 0,
|
||||
fetch: slowedFetch,
|
||||
});
|
||||
|
||||
const carolDeliveredAt = new Promise<number>((resolve) => {
|
||||
aliceInbox.on((e) => {
|
||||
if (e.name === 'inbox.message_delivered' && e.data.recipientAddress === 'carol') {
|
||||
resolve(Date.now());
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
const t0 = Date.now();
|
||||
// Bob queue first, carol second — pre-fix carol would wait 200 ms
|
||||
// behind bob's slow PUT. With per-recipient parallelism, carol's
|
||||
// PUT runs concurrently and lands first.
|
||||
await aliceInbox.send({ recipientAddress: 'bob', envelope: randBytes(20) });
|
||||
await aliceInbox.send({ recipientAddress: 'carol', envelope: randBytes(20) });
|
||||
|
||||
aliceInbox.start();
|
||||
const carolAt = await carolDeliveredAt;
|
||||
const carolElapsed = carolAt - t0;
|
||||
expect(carolElapsed).toBeLessThan(150);
|
||||
aliceInbox.stop();
|
||||
});
|
||||
|
||||
test('flush retries on transient server failure', async () => {
|
||||
const store = new MemoryInboxStore();
|
||||
const app = createInboxServer({ crypto, store, disableRateLimit: true });
|
||||
@@ -281,3 +530,195 @@ describe('tamper detection', () => {
|
||||
expect(result.received).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('InboxClient — default fetch is bound to globalThis', () => {
|
||||
// Regression: browsers' `fetch` is a WebIDL bound operation that throws
|
||||
// "Illegal invocation" when called as a method on another object. The
|
||||
// class stores `fetchImpl` and calls `this.fetchImpl(...)`, which strips
|
||||
// the Window receiver. Constructor must `bind(globalThis)`.
|
||||
test('default path passes globalThis as `this` (no Illegal invocation)', async () => {
|
||||
const realFetch = globalThis.fetch;
|
||||
let observedReceiver: unknown = 'unset';
|
||||
function strictFetch(this: unknown, _input: unknown, _init?: unknown): Promise<Response> {
|
||||
observedReceiver = this;
|
||||
if (this !== globalThis) {
|
||||
throw new TypeError("Failed to execute 'fetch' on 'Window': Illegal invocation");
|
||||
}
|
||||
return Promise.resolve(
|
||||
new Response('{}', {
|
||||
status: 200,
|
||||
headers: { 'content-type': 'application/json' },
|
||||
}),
|
||||
);
|
||||
}
|
||||
Object.defineProperty(globalThis, 'fetch', {
|
||||
configurable: true,
|
||||
writable: true,
|
||||
value: strictFetch,
|
||||
});
|
||||
try {
|
||||
const id = await makeIdentity();
|
||||
const client = new InboxClient({
|
||||
baseUrl: 'http://example.invalid',
|
||||
crypto,
|
||||
signingPrivateKey: id.signingPrivateKey,
|
||||
// No `fetch` override on purpose — this exercises the default path.
|
||||
});
|
||||
await client.register({ address: 'whoever', signingKey: id.signingPublicKey });
|
||||
expect(observedReceiver).toBe(globalThis);
|
||||
} finally {
|
||||
Object.defineProperty(globalThis, 'fetch', {
|
||||
configurable: true,
|
||||
writable: true,
|
||||
value: realFetch,
|
||||
});
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('Inbox.start() — fresh-address register/poll race (V4.8)', () => {
|
||||
// Regression: pre-4.8 `start()` called `register()` fire-and-forget AND
|
||||
// `schedulePoll(0)` synchronously, so the first poll often beat the
|
||||
// register HTTP RTT and got SHADE_NOT_FOUND on a fresh address. Fix:
|
||||
// start() defers the first poll; register() success kicks it.
|
||||
test('fresh address: no fetch fires before register completes', async () => {
|
||||
const store = new MemoryInboxStore();
|
||||
const app = createInboxServer({ crypto, store, disableRateLimit: true });
|
||||
const alice = await makeIdentity();
|
||||
|
||||
// Order observed by the server: must be register-then-fetch, never
|
||||
// fetch-then-register.
|
||||
const calls: Array<'register' | 'fetch' | 'put'> = [];
|
||||
let registerArrived = false;
|
||||
const recordingFetch: typeof fetch = (async (input, init) => {
|
||||
const u =
|
||||
typeof input === 'string'
|
||||
? input
|
||||
: input instanceof URL
|
||||
? input.toString()
|
||||
: (input as Request).url;
|
||||
if (u.includes('/v1/inbox/register')) {
|
||||
calls.push('register');
|
||||
// Hold register for a tick to widen the race window.
|
||||
await new Promise((r) => setTimeout(r, 25));
|
||||
registerArrived = true;
|
||||
} else if (u.endsWith('/fetch')) {
|
||||
// Any fetch arriving before register is the race we're guarding
|
||||
// against.
|
||||
if (!registerArrived) {
|
||||
throw new Error('fetch fired before register completed (race not fixed)');
|
||||
}
|
||||
calls.push('fetch');
|
||||
} else if (u.includes('/v1/inbox/')) {
|
||||
calls.push('put');
|
||||
}
|
||||
return honoFetch(app)(input, init);
|
||||
}) as typeof fetch;
|
||||
|
||||
const inbox = new Inbox({
|
||||
baseUrl: 'http://localhost',
|
||||
ownAddress: 'alice',
|
||||
crypto,
|
||||
signingPrivateKey: alice.signingPrivateKey,
|
||||
signingPublicKey: alice.signingPublicKey,
|
||||
pollIntervalMs: 30_000, // Long enough that only register's kick triggers.
|
||||
fetch: recordingFetch,
|
||||
});
|
||||
inbox.onIncoming(() => null);
|
||||
inbox.start();
|
||||
|
||||
// Wait until register has completed and the success-kick poll lands.
|
||||
await new Promise((r) => setTimeout(r, 100));
|
||||
inbox.stop();
|
||||
|
||||
expect(calls[0]).toBe('register');
|
||||
// First fetch (if any) must be after register.
|
||||
const firstFetchIdx = calls.indexOf('fetch');
|
||||
if (firstFetchIdx !== -1) {
|
||||
expect(firstFetchIdx).toBeGreaterThan(calls.indexOf('register'));
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('FetchedBlob.from — relay-supplied sender fingerprint (V4.8)', () => {
|
||||
test('inbox-fetch response carries from = 8-byte hex of SHA-256(senderSigningKey)', async () => {
|
||||
const store = new MemoryInboxStore();
|
||||
const app = createInboxServer({ crypto, store, disableRateLimit: true });
|
||||
const bob = await makeIdentity();
|
||||
const alice = await makeIdentity();
|
||||
|
||||
const bobClient = new InboxClient({
|
||||
baseUrl: 'http://localhost',
|
||||
crypto,
|
||||
signingPrivateKey: bob.signingPrivateKey,
|
||||
fetch: honoFetch(app),
|
||||
});
|
||||
const aliceClient = new InboxClient({
|
||||
baseUrl: 'http://localhost',
|
||||
crypto,
|
||||
signingPrivateKey: alice.signingPrivateKey,
|
||||
fetch: honoFetch(app),
|
||||
});
|
||||
|
||||
await bobClient.register({ address: 'bob', signingKey: bob.signingPublicKey });
|
||||
await aliceClient.put({
|
||||
recipientAddress: 'bob',
|
||||
senderSigningKey: alice.signingPublicKey,
|
||||
envelope: randBytes(64),
|
||||
});
|
||||
|
||||
const fetched = await bobClient.fetch({ address: 'bob' });
|
||||
expect(fetched.blobs.length).toBe(1);
|
||||
const fp = fetched.blobs[0]!.from;
|
||||
expect(fp).toBeDefined();
|
||||
expect(fp).toMatch(/^[0-9a-f]{16}$/);
|
||||
|
||||
// Must be reproducible: SHA-256(alice.signingPublicKey) → first 8 bytes hex.
|
||||
const digest = await globalThis.crypto.subtle.digest(
|
||||
'SHA-256',
|
||||
alice.signingPublicKey as unknown as ArrayBuffer,
|
||||
);
|
||||
const expected = Array.from(new Uint8Array(digest).slice(0, 8), (b) =>
|
||||
b.toString(16).padStart(2, '0'),
|
||||
).join('');
|
||||
expect(fp).toBe(expected);
|
||||
});
|
||||
|
||||
test('DecryptHandler raw arg propagates from to the app', async () => {
|
||||
const store = new MemoryInboxStore();
|
||||
const app = createInboxServer({ crypto, store, disableRateLimit: true });
|
||||
const bob = await makeIdentity();
|
||||
const alice = await makeIdentity();
|
||||
|
||||
const aliceClient = new InboxClient({
|
||||
baseUrl: 'http://localhost',
|
||||
crypto,
|
||||
signingPrivateKey: alice.signingPrivateKey,
|
||||
fetch: honoFetch(app),
|
||||
});
|
||||
const bobInbox = new Inbox({
|
||||
baseUrl: 'http://localhost',
|
||||
ownAddress: 'bob',
|
||||
crypto,
|
||||
signingPrivateKey: bob.signingPrivateKey,
|
||||
signingPublicKey: bob.signingPublicKey,
|
||||
pollIntervalMs: 0,
|
||||
fetch: honoFetch(app),
|
||||
});
|
||||
|
||||
await bobInbox.register();
|
||||
await aliceClient.put({
|
||||
recipientAddress: 'bob',
|
||||
senderSigningKey: alice.signingPublicKey,
|
||||
envelope: randBytes(40),
|
||||
});
|
||||
|
||||
let observed: string | undefined = undefined;
|
||||
bobInbox.onIncoming((raw) => {
|
||||
observed = raw.from;
|
||||
return null;
|
||||
});
|
||||
await bobInbox.tick();
|
||||
expect(observed).toMatch(/^[0-9a-f]{16}$/);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shade/key-transparency",
|
||||
"version": "4.0.0",
|
||||
"version": "4.11.0",
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
"types": "src/index.ts",
|
||||
|
||||
@@ -21,7 +21,7 @@
|
||||
* the dataset grows enough that flat re-hash becomes a bottleneck.
|
||||
*/
|
||||
|
||||
import { leafHash, nodeHash, emptyRootHash } from './hashes.js';
|
||||
import { leafHash, emptyRootHash } from './hashes.js';
|
||||
import { sha256Sync } from './sha256.js';
|
||||
import { constantTimeEqual } from './util.js';
|
||||
import { mth, auditPath, recomputeRootFromAuditPath } from './log.js';
|
||||
|
||||
@@ -24,8 +24,6 @@ import { MerkleLog, auditPath } from './log.js';
|
||||
import {
|
||||
AddressIndex,
|
||||
type AddressIndexEntry,
|
||||
type IndexAbsenceProof,
|
||||
type IndexInclusionProof,
|
||||
} from './index-tree.js';
|
||||
import {
|
||||
type SignedTreeHead,
|
||||
|
||||
@@ -111,7 +111,7 @@ interface IndexAbsenceWire {
|
||||
} | null;
|
||||
}
|
||||
|
||||
type IndexProofWire = IndexInclusionWire | IndexAbsenceWire;
|
||||
export type IndexProofWire = IndexInclusionWire | IndexAbsenceWire;
|
||||
|
||||
interface BundleInclusionWire {
|
||||
kind: 'inclusion' | 'tombstone';
|
||||
|
||||
5
packages/shade-key-transparency/tsconfig.json
Normal file
5
packages/shade-key-transparency/tsconfig.json
Normal file
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"extends": "../../tsconfig.json",
|
||||
"compilerOptions": { "outDir": "dist", "rootDir": "src" },
|
||||
"include": ["src"]
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shade/keychain",
|
||||
"version": "4.0.0",
|
||||
"version": "4.11.0",
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
"types": "src/index.ts",
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
{
|
||||
"extends": "../../tsconfig.json",
|
||||
"include": ["src/**/*", "tests/**/*"]
|
||||
"compilerOptions": { "outDir": "dist", "rootDir": "src" },
|
||||
"include": ["src"]
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shade/observability",
|
||||
"version": "4.0.0",
|
||||
"version": "4.11.0",
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
"types": "src/index.ts",
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user