Files
Shade/CHANGELOG.md

1161 lines
55 KiB
Markdown
Raw Permalink Normal View History

# Changelog
All notable changes to Shade are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
release(v4.1.0): browser-friendly HTTP RPC for @shade/files Default shade.files.client(peer) requires both peers to be mutually addressable over HTTP — the response round-trips through Shade.deliverControlEnvelope (POST to peer's /v1/transfer/control). Browser tabs can't host an HTTP server, so they couldn't consume @shade/files at all. Dispatch's filutforsker (admin-panel browser UI) is the canonical use-case. This release adds a parallel request-response transport: one POST per RPC, encrypted envelope in the body, encrypted response in the same HTTP response. No inbound channel needed on the client. ### New API - shade.files.rpcRoute(opts?) — Hono app exposing POST /rpc. - shade.files.httpClient(peer, opts) — request-response FileClient. - FilesNamespace.serve(handler, { inlineOnly: true }) — skip streams- bridge (and its configureTransfers pre-condition); also skip channel-based dispatch so requests aren't double-dispatched. ### Limitations (v1) Inline only (≤ 256 KiB). Streamed reads/writes throw clear errors directing to shade.files.client(peer) on a server-to-server deploy. ### Tests 7 integration tests in tests/integration/http-rpc.test.ts covering round-trip + negative cases (sender header, empty/garbage body, maxBodyBytes, rpcRoute-without-serve). ### Symmetry Mirrors @shade/server's shade-auth-middleware: encrypted envelope in request body, decrypted via existing ratchet, response in same HTTP roundtrip. No WebSocket, no SSE, no outbound from server. Wire-compatible. Source-compatible. Lockstep bump to 4.1.0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 22:08:14 +02:00
## [4.1.0] — 2026-05-03 — Browser-friendly HTTP RPC for @shade/files
The default `shade.files.client(peer)` requires both peers to be
mutually addressable over HTTP — the response to a `list` / `read` /
etc. round-trips through `Shade.deliverControlEnvelope`, which POSTs
to the peer's `/v1/transfer/control` endpoint. **That doesn't work
for browsers** — a tab can't host an HTTP server, so the server
cannot call back outbound.
This release ships a parallel request-response transport. One POST per
RPC, encrypted envelope in the request body, encrypted response in the
same HTTP response. Mirrors the way `@shade/server`'s
`shade-auth-middleware` works for prekey writes.
### Added
#### `@shade/files`
- `createFilesRpcRoute(shade, handler, options?)` — Hono app exposing
`POST /rpc`. Reads `X-Shade-Sender-Address`, decrypts the envelope
via the existing ratchet session, dispatches through the attached
`FileHandler`, encrypts the result, and returns it in the same HTTP
response. Transport-level failures (no session, undecryptable, body
too big) return JSON `{ error }` with appropriate 4xx; application-
level failures ship encrypted `RpcError` envelopes.
- `createFilesHttpClient(shade, peer, options)` — request-response
`FileClient` for browser-style consumers. Each method (list / stat /
mkdir / delete / move / getThumbnail / custom / write inline / read
inline) does one HTTP POST and parses the encrypted response. No
inbound channel required.
- `shade.files.rpcRoute(opts?)` — namespace-side getter for the route.
Throws if no handler has been attached via `shade.files.serve(...)`
first.
- `shade.files.httpClient(peer, opts)` — namespace-side getter for the
client.
- `FilesNamespace.serve(handler, { inlineOnly: true })` — opt-out flag
that skips the streams-bridge setup. Required for HTTP-RPC-only
servers (which don't need `configureTransfers({ resolveBaseUrl })`).
In `inlineOnly` mode the channel-based dispatcher is also not
attached, so requests are dispatched only by the rpc-route — avoids
double-dispatch when a browser client and a server-to-server client
share the same Shade instance.
- `ShadeBridge` (exported) gains a `receive(peer, envelope)` member
matching `Shade.receive` so server-side rpc-route can decrypt
inbound envelopes through the structural surface.
### Wire contract
```
POST /rpc HTTP/1.1
Content-Type: application/octet-stream
X-Shade-Sender-Address: alice@example.com
<wire-encoded ShadeEnvelope wrapping JSON-encoded RpcRequest>
────
200 OK
Content-Type: application/octet-stream
<wire-encoded ShadeEnvelope wrapping JSON-encoded RpcResponse | RpcError>
```
### Limitations (v1)
- **Inline payloads only** (≤ 256 KiB). `write` of larger inputs
throws `ConflictError` directing callers to `shade.files.client(peer)`
on a server-to-server deployment. Streamed `read` results throw
`InternalFileError` for the same reason.
- The X3DH first-message must ride the same RPC route — set
`acceptFirstMessage: true` on `rpcRoute({ acceptFirstMessage: true })`
when the browser client's first-ever call doubles as the handshake.
### Tests
- `tests/integration/http-rpc.test.ts` — round-trip via HTTP
(list / mkdir / stat / write / read / delete) plus negative cases
(streamed write rejected, missing sender header, empty body, garbage
body, body past `maxBodyBytes`, `rpcRoute()` without `serve()`).
### Migration
`4.0.x → 4.1.0` is wire-compatible and source-compatible. The HTTP
RPC route is purely additive — no existing code path changes. To
adopt:
```ts
// server (was)
await shade.files.serve(handlerConfig);
// server (HTTP-RPC mode)
await shade.files.serve(handlerConfig, { inlineOnly: true });
app.route('/api/v1/shade-files', shade.files.rpcRoute());
// browser client
const fs = shade.files.httpClient(serverAddress, { rpcUrl: '...' });
```
release(v4.0.2): consumer-strict reader-shape fixes 4.0.1's typecheck gate compiled each package internally against lib: ["ES2022"]. That doesn't catch types that only fail when *consumer* code (lib: ["DOM"] + exactOptionalPropertyTypes) tries to assign a native browser type into one of our locally-defined narrower types. Dispatch hit one such case in @shade/files inline-threshold.ts. This release adds a tests/consumer-strict/ smoke project to the pre-publish gate. It compiles a tiny "as if I were a downstream app" TS file against: lib: ["ES2022", "DOM", "DOM.Iterable"] types: ["bun-types"] exactOptionalPropertyTypes: true strict: true paths → packages/*/src/index.ts scripts/typecheck-all.ts now runs the smoke after per-package checks. Both must pass before publish:dry / publish:all proceeds. ### Fixed - @shade/files inline-threshold.ts: MinimalReader<T> rewritten as the explicit disjoint union { done:false, value:T } | { done:true, value?: T | undefined } that's assignable from every native reader shape (bun, DOM, node:stream/web). Fixes the "ReadableStreamReadResult is not assignable" Dispatch reported. - @shade/files streams-bridge (client + server): stash setTimeout return in a local before .unref?.() via { unref?: () => void } cast. Fluent .unref?.() failed under lib: ["DOM"] (setTimeout returns number there). - @shade/sdk background.ts: same setInterval .unref?.() fix. Wire-compatible. No API shape changed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 19:51:46 +02:00
## [4.0.2] — 2026-05-03 — Consumer-strict reader-shape fixes
`4.0.1` shipped the `tsc --noEmit` gate that compiles each package
internally against `lib: ["ES2022"]`. That gate did not catch types
that only fail when *consumer* code (running with `lib: ["DOM"]` +
`exactOptionalPropertyTypes`) tries to assign a native browser type
into one of our locally-defined narrower types.
This release adds a consumer-strict smoke test to the pre-publish
gate and fixes every collision that smoke uncovered.
### Fixed
#### `@shade/files`
- `inline-threshold.ts`: rewrote the local `MinimalReader<T>` interface
as an explicit disjoint union (`{ done: false; value: T } | { done:
true; value?: T | undefined }`) so it accepts every native reader
shape — `bun-types` (`value?: undefined`), `lib.dom` (`value?: T`),
and `node:stream/web`. The previous flat shape was rejected by
consumer projects with `exactOptionalPropertyTypes: true` because
the present-branch required `value: T`. **Fixes "Type
ReadableStreamReadResult<Uint8Array> is not assignable to
{ value: Uint8Array | undefined; done: boolean }".**
- `client/streams-bridge.ts`, `server/streams-bridge.ts`: stash the
`setTimeout(...)` return value in a local before calling `.unref?.()`
through an explicit `{ unref?: () => void }` cast. The previous
fluent `.unref?.()` failed under `lib: ["DOM"]` because DOM types
`setTimeout` to `number`, which has no `.unref` even as an optional
property.
#### `@shade/sdk`
- `background.ts`: same `setTimeout` / `setInterval` `.unref?.()` fix.
### Tooling
- New `tests/consumer-strict/` — a tiny "as if I were a downstream app"
TypeScript project with its own `tsconfig.json`:
`lib: ["ES2022", "DOM", "DOM.Iterable"]`, `types: ["bun-types"]`,
`exactOptionalPropertyTypes: true`, `strict: true`,
`paths`-mapped to the workspace's `packages/*/src/index.ts`.
Three smoke files exercise `@shade/files`, `@shade/sdk`, and
`@shade/key-transparency` against the consumer-strict tsconfig.
- `scripts/typecheck-all.ts` now runs the consumer-strict smoke after
the per-package internal type-check. Both must pass before
`prepublish:check` (and therefore `publish:dry` / `publish:all`)
succeeds.
### Migration
`4.0.1 → 4.0.2` is wire-compatible and source-compatible. No API shape
changed; only internal typing was tightened.
release(v4.0.1): strict-TS publishability fixes 4.0.0 shipped TypeScript source as published main/types, but several files only compiled inside the monorepo. Consumer projects (Dispatch, etc.) running their own strict tsc against our published source hit: - @shade/key-transparency: 4 noUnusedLocals violations (IndexAbsenceProof, IndexInclusionProof, IndexProofWire, nodeHash) - @shade/sdk: KT verifier callbacks returned Promise<unknown> instead of Promise<STHWire> / Promise<{ proof: string[] }> - @shade/sdk: thumbnail.ts globalThis cast collided with consumer's lib.dom-supplied createImageBitmap signature - @shade/files: cycle with @shade/sdk produced "this is not assignable to type 'Shade'" because hoisted node_modules layouts duplicated the Shade class. Broken by replacing `import type { Shade }` with a local structural ShadeBridge interface. - @shade/storage-encrypted: KeyUsage (lib.dom) used under lib: ["ES2022"] - @shade/transport-bridge: ReadableStreamDefaultReader<any> ↔ <Uint8Array> mismatch - @shade/keychain / @shade/dashboard / @shade/storage-encrypted tsconfig rootDir / include hygiene Tooling: scripts/typecheck-all.ts runs `bunx tsc --noEmit` against every workspace package's tsconfig and fails on any error. Wired into publish:dry / publish:all and publish-shade.sh as a hard gate so this class of bug cannot recur. All 24 packages bumped to 4.0.1 in lockstep. Migration: <ShadeFilesProvider> now requires an explicit `files` prop (pass `shade.files`). Wire format unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 19:36:47 +02:00
## [4.0.1] — 2026-05-03 — Strict-TS publishability fixes
`4.0.0` shipped TypeScript source files as the published `main` /
`types`, which meant every consumer's `tsc` had to compile our code
under their own strict settings. Several files only compiled inside
the monorepo (where peer-dep cycles resolve via workspace links and
the `lib` array doesn't include `DOM`). This release makes all 24
packages compile cleanly under the strict-flagged tsconfig that ships
with the repo, and wires a `bun run typecheck` gate into both the
`publish:dry` and `publish:all` flows so this category of bug cannot
recur.
### Fixed
#### `@shade/key-transparency`
- Removed unused imports `IndexAbsenceProof`, `IndexInclusionProof`
(`src/manager.ts`), `nodeHash` (`src/index-tree.ts`).
- `IndexProofWire` is now exported (was a private type that
`noUnusedLocals` flagged).
- Added missing `tsconfig.json` so the package can be type-checked
in isolation.
#### `@shade/sdk`
- KT verifier wiring: `fetchLatestSTH()` and `fetchConsistencyProof()`
now have explicit return types (`Promise<STHWire>` and
`Promise<{ proof: string[] }>`) so consumers don't see
`Promise<unknown>` from `res.json()`.
- `STHWire` type is now imported from `@shade/key-transparency`.
- `thumbnail.ts`: cast `globalThis` through `unknown` first when
reading optional DOM globals (`OffscreenCanvas`, `createImageBitmap`)
so consumer projects that include `lib.dom` don't reject our
narrower local types as "insufficiently overlapping".
#### `@shade/files`
- **Broke the `@shade/sdk``@shade/files` dependency cycle.**
`@shade/files` no longer imports `Shade` from `@shade/sdk` — every
callsite uses a new local `ShadeBridge` interface defined in
`src/integration/shade-bridge.ts`. This is the structural surface
Shade must satisfy: `myAddress`, `send`, `onMessage`, `upload`,
`onIncomingTransfer`, `getFingerprintFor` (required) plus
`getObservability`, `deliverControlEnvelope` (optional). The Shade
class structurally implements every member, so
`createFilesNamespace(this)` from the SDK side compiles regardless
of how many copies of `@shade/sdk` a consumer's package manager
hoists. **Fixes "this is not assignable to type 'Shade'"** in
consumer builds.
- `<ShadeFilesProvider>` now takes `files: FilesNamespace` as an
explicit prop instead of reading `shade.files`. Consumers pass
`shade.files` (or any `createFilesNamespace(...)` result for tests)
directly.
- `ShadeFileRpcChannel.send` now raises a clear error when
`deliverControlEnvelope` is undefined instead of producing an
implicit-undefined-call error at compile time.
#### `@shade/storage-encrypted`
- Replaced `KeyUsage` (a `lib.dom` type) with a local
`WebCryptoKeyUsage` union so the package compiles under
`lib: ["ES2022"]` without DOM.
- Fixed `tsconfig.json` `rootDir` so package-level `bunx tsc` works.
#### `@shade/transport-bridge`
- `sse-bridge.ts`: cast `res.body.getReader()` to
`ReadableStreamDefaultReader<Uint8Array>` so the strict reader-type
parity check in the consume loop passes.
#### `@shade/keychain` / `@shade/dashboard`
- Fixed `tsconfig.json` `rootDir` and `include` so the packages can
type-check standalone (and so `vite.config.ts` doesn't get pulled
into the dashboard's `rootDir`).
#### `@shade/widgets`
- Removed unused `ThumbnailMime` import in
`components/transfer/ThumbnailPreview.tsx`.
### Tooling
- New `scripts/typecheck-all.ts` — runs `bunx tsc --noEmit` against
every workspace package's `tsconfig.json` and fails if any reports
errors.
- New `bun run typecheck` script.
- `publish:dry` and `publish:all` now run `prepublish:check`
(`typecheck` + `test`) before any package is packed or published.
- `scripts/publish-shade.sh` calls the typecheck-all gate before
invoking the publisher.
### Migration
`4.0.0 → 4.0.1` is wire-compatible and source-compatible with one
exception:
- `<ShadeFilesProvider>` requires a `files` prop. Previously
`<ShadeFilesProvider shade={shade}>...</ShadeFilesProvider>` worked;
it now must be `<ShadeFilesProvider shade={shade} files={shade.files}>`.
No on-disk schema changes. No package-version-pin changes outside
the lockstep `4.0.0 → 4.0.1` bump.
## [4.0.0] — 2026-05-03 — General Availability
Shade 4.0 is the first GA-marked release: every plan from V3.1 through
V3.12 is merged, the cross-platform vector suite is green on TS + Kotlin,
the threat model has been updated to reflect every new surface, and the
core stack (X3DH, Double Ratchet, storage encryption, recovery, WebRTC
P2P, Key Transparency) has been packaged for external review. Voice and
video — the only big-ticket V2.x ask — have been moved to V5.0 so the
4.0 audit can focus on a frozen non-realtime core.
The wire format is **unchanged from 0.4.x**: 4.0 peers interoperate with
0.4.x peers byte-for-byte. The version bump is semantic (audit-cycle
complete, opt-in surface fully exposed), not breaking. Apps that have
been running 0.4.x in production move forward by `bun add @shade/sdk@^4.0.0`
and (optionally) wiring any of the new opt-in surfaces.
### Highlights
- **External crypto-review-ready.** A "review-bundle" (`docs/audit/`)
ships with this release: links to every protocol spec, the threat
model, the cross-platform test corpus, the build instructions, and
scope guidance for the auditor.
- **Migration guide locked in.** `MIGRATION.md` documents the exact
0.3.x → 4.0 path, including the optional opt-ins, the schema
superset, and the `shade migrate-storage` workflow.
- **Cross-platform parity gated in CI.** `.gitea/workflows/cross-vectors.yml`
runs the same vector corpus on TS (bun) and Kotlin (gradle). A
divergent KDF label, AAD layout, or wire byte fails the build.
- **All V*.md plans archived.** `docs/V3.1.md` through `docs/V3.12.md`
and the original V2.1/V2.2/V2.3 backlog now live under
`docs/archive/` with `Status: Done`. Active planning continues in
`docs/V5.0.md` (Voice & Video).
- **Operator-facing OpenAPI is complete.** `packages/shade-server/openapi.yaml`
now covers prekey, transfer, KT, inbox, bridge (SSE / long-poll / WS),
observer, and the `/metrics`, `/healthz`, `/ready` operations
endpoints — every HTTP surface a 4.0 client can talk to.
- **Threat-model refresh.** Sections 10 (V3.3 fingerprint gates), 11
(V3.11 WebRTC), 12 (V3.8 Web-Worker boundary) are new; the residual-
risk table updates the §1 / §2 / §6 entries with the
4.0 mitigations now landed.
### What's already in 4.0 (consolidated from 0.4.x)
The detailed CHANGELOG entries below list everything that landed in
the 0.4.x series and is now part of the GA baseline:
- V3.2 — At-Rest Storage Encryption (`@shade/storage-encrypted`,
`@shade/keychain`, `shade migrate-storage`).
- V3.3 — Fingerprint Gates & Trust UX (`Shade.beforeFirstLargeFile` /
`beforeBackupImport` / `beforeNewDeviceTrust`,
`<FingerprintCompare />`, `<FingerprintGate />`).
- V3.4 — Observability v2 (OpenTelemetry-shaped events,
`@shade/observability`).
- V3.5 — Android parity + cross-platform CI gate.
- V3.6 — Async Store-and-Forward (`@shade/inbox`,
`@shade/inbox-server`, `InboxPruneTask`).
- V3.7 — Transport Bridge (`@shade/transport-bridge`, SSE +
long-poll + WS adapters).
- V3.8 — Web Workers Crypto (`@shade/crypto-web/worker`).
- V3.9 — Rich File Metadata + thumbnails (in `@shade/files`).
- V3.10 — Social Key Recovery (`@shade/recovery`,
`<RecoverySetup />`, `<RecoveryRequest />`,
`<RecoveryApprove />`).
- V3.11 — WebRTC P2P Transport (`@shade/transport-webrtc`,
`MultiTransportFallback`).
- V3.12 — Key Transparency (`@shade/key-transparency`,
`createPrekeyServerWithKT(...)`, `LightWitness`).
### Acceptance criteria
- [x] V3.1 → V3.12 merged into `main`.
- [x] No open critical / high-severity security issues at the time of
tagging.
- [x] Cross-platform test vectors green: TS (1000 / 1000) and
Kotlin (11 / 11).
- [x] Production-checklist (`docs/PRODUCTION-CHECKLIST.md`) is the
canonical operator gate.
- [x] OpenAPI covers every HTTP surface (`/v1/keys/*`,
`/v1/transfer/*`, `/v1/kt/*`, `/v1/inbox/*`, `/v1/bridge/*`,
`/metrics`, `/healthz`, `/ready`).
- [x] Threat model reflects every new V3.x surface.
- [x] `0.3.x → 4.0` migration documented in `MIGRATION.md` and
validated against the `shade migrate-storage` CLI on a real
SQLite DB.
- [ ] **Pending external review.** A `docs/audit/REVIEW-BUNDLE.md`
pointer is shipped; the actual external review window opens
after tag.
### Migration
See [MIGRATION.md § Migrating from 0.3.x to 4.0 (GA)](./MIGRATION.md#migrating-from-03x-to-40-ga).
The short version: bump every `@shade/*` to `^4.0.0`, run
`bun install`, restart, opt in to the V3.x surfaces you actually need.
No on-disk schema is destructive; no peer wire format changes.
## [Unreleased] — Key Transparency (V3.12) + WebRTC (V3.11)
### V3.12 — Key Transparency
Verifiable prekey distribution. The prekey server can now run in
**Key-Transparency mode**: every register / delete event is committed
to an append-only Merkle log (RFC 6962-style), every bundle-fetch
includes an inclusion proof, and every Signed Tree Head (STH) is
signed with an operator-controlled Ed25519 key that clients pin
out-of-band.
A malicious server that swaps a bundle, splits its view between two
clients, or rewrites history is detected by the client's KT verifier
or by an independent witness. KT is **opt-in** on both server and
client — existing deployments work unchanged until upgraded.
See `docs/V3.12-DESIGN.md` for the design notat (threat model,
data-structure choices, freshness model, recovery procedures) and
`docs/key-transparency.md` for operator + client onboarding.
### Added
#### `@shade/key-transparency` (new package)
- `MerkleLog` — RFC 6962 append-only hash tree over pre-hashed leaves.
In-memory mirror with O(N) leaf storage and O(log N) audit-path /
consistency-proof generation.
- `auditPath`, `recomputeRootFromAuditPath`, `consistencyProof`,
`verifyConsistencyProof` — standalone primitives matching RFC 6962
§2.1.1 and §2.1.2.
- `AddressIndex` + `verifyInclusionProof` / `verifyAbsenceProof`
lexicographically sorted address commitment with both inclusion and
neighbor-pair absence proofs. The index commitment becomes part of
every STH so `address → bundle_hash` is auditable, not just the
raw event log.
- `SignedTreeHead` + `signSth` / `verifySthSignature` /
`canonicalSthBytes` / `computeLogId` — Ed25519-signed commitment to
the tree state. `log_id = SHA-256(public_key)` so a forged STH that
claims a different log key is rejected.
- `KTLogManager` — server-side orchestration that wires `MerkleLog`,
`AddressIndex`, persistent `KTLogStore`, and STH signing under one
serial-mutation API (`recordRegister`, `recordReplenish`,
`recordDelete`, `publishSTH`, `buildBundleInclusionProof`,
`buildBundleAbsenceProof`, `buildConsistencyProof`).
- `KTLogStore` interface + `MemoryKTLogStore` reference impl. The
interface is append-only by contract (no `update()` or `delete()`
on historical leaves).
- `LightWitness` — passive observer that polls a server's `/v1/kt/sth`
endpoint, verifies signature + freshness + consistency, stores
observed STHs, and exposes `compare(otherSth)` for split-view
detection. Used by both witness CLIs and (transparently) by the SDK.
- Bundle-proof verifiers: `verifyBundleInclusion`,
`verifyBundleAbsence`, `verifyBundleTombstone`. Each re-derives the
bundle hash, checks the audit path against the STH root, verifies
the index commitment, and confirms freshness.
- Errors: `KTError`, `KTVerificationError`, `KTSplitViewError`,
`KTStaleSTHError`, `KTLogIdMismatchError`. Mapped to
`SHADE_KT_*` codes.
- Wire-format helpers: `ktProofToWire` / `ktProofFromWire` /
`sthToWire` / `sthFromWire` for JSON-safe transport.
#### `@shade/server`
- `createPrekeyServerWithKT(...)` — convenience that builds the KT
service and wires it into the prekey routes in one call.
- `KeyTransparencyService` — single-writer wrapper around
`KTLogManager` with mutex-serialized mutations, cached latest STH,
and configurable heartbeat interval (default 10 min).
- New routes mounted under `/v1/kt/`:
- `GET /v1/kt/log_id` — operator's signing public key + log_id.
- `GET /v1/kt/sth` — latest signed tree head.
- `GET /v1/kt/sth/:treeSize` — historical STH lookup.
- `GET /v1/kt/consistency?from=N1&to=N2` — RFC 6962 consistency proof.
- `POST /v1/keys/register` and `DELETE /v1/keys/:address` now commit
to the KT log (when enabled). `GET /v1/keys/bundle/:address`
returns a `ktProof` field on success and on 404 (absence/tombstone).
- KT is fully opt-in. Existing deployments are byte-compatible until
`keyTransparency` is configured.
#### `@shade/storage-postgres`
- `PostgresKTLogStore` — durable KTLogStore on Postgres. Uses three
tables (`shade_kt_leaves`, `shade_kt_index`, `shade_kt_sths`) with
an `BEFORE UPDATE/DELETE/TRUNCATE` trigger on `shade_kt_leaves`
that blocks any mutation — defense-in-depth against operator error.
- `ensureKTLogTables(sql)` exported for embedding.
#### `@shade/transport`
- `ShadeFetchTransport` accepts `keyTransparency: KTVerifierOptions`.
Modes: `'observe'` verifies when proof present, `'observe-strict'`
requires proof on every response.
- `fetchBundleVerified(address)` returns `{ bundle, ktSth? }` so
callers can route the verified STH into a `LightWitness`.
- 404 responses are also verified (absence or tombstone proof) under
strict mode.
#### `@shade/sdk`
- `ShadeConfig.keyTransparency` — opt-in client config:
```ts
createShade({
prekeyServer: 'https://shade.example.com',
keyTransparency: { mode: 'observe-strict', logPublicKey: KEY_BYTES_32 },
});
```
- `Shade.getKTWitness()` returns the auto-wired `LightWitness` so app
code can introspect observed STHs or run manual gossip checks.
- The SDK transparently feeds every fetched STH into the witness so
split-view detection runs by default whenever KT is on.
### Tests
- 76 new tests across the KT stack: hash primitives, Merkle audit
paths, consistency proofs, address-index inclusion/absence proofs,
STH signing, manager orchestration, witness ingest, server-side
HTTP routes, transport-side verification, and an end-to-end
acceptance test that simulates two divergent server views and
asserts a `KTSplitViewError` is raised.
### V3.11 — WebRTC P2P Transport
Direct peer-to-peer chunk delivery for `@shade/transfer` (and therefore
`@shade/files`) via `RTCDataChannel`. Signaling — SDP offer / answer +
trickle ICE — rides on top of `Shade.send` / `Shade.onMessage` so the
same Double Ratchet that authenticates regular messages authenticates
WebRTC negotiation. Throughput-heavy uploads (multi-MB / multi-GB) skip
the HTTP relay entirely when NAT allows; when traversal fails, the new
`MultiTransportFallback([webrtc, http])` demotes back to HTTP within
the configured connect-timeout window without losing any chunks already
in flight. See `docs/webrtc.md` and `docs/V3.11.md`.
### Added
#### `@shade/transport-webrtc` (new package)
- `WebRtcConnection` — per-peer wrapper around an `IPeerConnection`
plus the single bidirectional `RTCDataChannel` (label
`shade-transfer/v1`). Drives offer/answer/ICE through a
`WebRtcSignalingChannel`; handles the receiver-side dispatch loop
for chunk-ack / resume-state / ping-pong / error frames; exposes
per-request reqId-correlated `request()` for the transport layer.
- `WebRtcConnectionManager` — per-peer pool with deterministic glare
resolution (lexicographic address compare). `getOrCreate(peer)`
returns the live connection or initiates a fresh one; following
through a glare-yield is automatic so the user-facing promise
resolves to whichever role survives.
- `WebRtcSignalingChannel` — multiplexes the four signaling kinds
(`shade.webrtc-offer/v1`, `shade.webrtc-answer/v1`,
`shade.webrtc-ice/v1`, `shade.webrtc-bye/v1`) over any `ShadeBridge`
(real `Shade.send`/`onMessage`, or `MemoryShadeBridge` for tests).
Non-signaling plaintext is forwarded to a configurable `passthrough`
hook so consumer `onMessage` handlers stay untouched.
- `WebRtcTransferTransport` — implements
`@shade/transfer`'s `ITransferTransport` over the managed
DataChannel. Encodes chunks into the package's binary wire format,
awaits chunk-ack frames matched by 16-byte requestId tokens, and
enforces SCTP-friendly backpressure by polling `bufferedAmount`
(default threshold 4 MiB).
- `IRtcFactory` interface + `nativeRtcFactory()` adapter wrapping
`globalThis.RTCPeerConnection` for browsers / Deno / Cloudflare
Workers. `MemoryRtcFactory` ships an in-process WebRTC simulator
used by the package's own tests and by `@shade/sdk` integration
tests.
- `createShadeBridgeFromShade(shade)` — turns any `Shade`-shaped
object into a `ShadeBridge`. Calls `shade.send(plaintext)` to
ratchet-encrypt the JSON, then `shade.deliverControlEnvelope(...)`
(when present) to ship the envelope over HTTP — same path the
existing control-plane already uses.
- Wire-format constants (`WIRE_CHUNK`, `WIRE_CHUNK_ACK`, etc.) +
`encode*Frame` / `decodeFrame` helpers exported for adapters that
want to interoperate with `ShadeTransferWsTransport` (the wire
matches frame-for-frame).
- Errors: `WebRtcConnectError`, `WebRtcDataChannelError`,
`WebRtcSignalingError`, `WebRtcTimeoutError` — all extend
`TransferTransportError` so `MultiTransportFallback` automatically
demotes on failure.
#### `@shade/transfer`
- `MultiTransportFallback` — N-ary generalisation of the existing
two-arg `FallbackTransferTransport`. Constructor takes
`[{ name: 'webrtc', transport }, { name: 'ws', transport }, ...]`;
layers are tried in order and demote sticky on
`TransferTransportError`. Exposes `activeName`, `hasFallenBack`,
`failures` (diagnostic log), and `onSwitch((from, to) => ...)` for
observability hooks.
#### `@shade/sdk`
- `Shade.configureWebRTC({ factory, iceServers?, iceTransportPolicy?,
bundlePolicy?, connectTimeoutMs?, requestTimeoutMs?,
backpressureThresholdBytes? })` — opt-in entrypoint. MUST be called
before the engine is built (i.e. before the first `upload()`,
`onIncomingTransfer()`, or `transferRoute()` call). When
configured, the engine is wired with
`MultiTransportFallback([webrtc, http])` and the WebRTC manager
receives receiver-hooks pointing at `engine.receiveChunk` /
`engine.getResumeState`.
- `Shade.getWebRtcRuntime(): ShadeWebRtcRuntime | null` — diagnostic
accessor returning the live signaling channel, manager, transport,
and `MultiTransportFallback` after `engine()` builds.
- `@shade/transport-webrtc` is a (optional) peer-dep — projects that
don't call `configureWebRTC()` don't pay the install or runtime
cost.
### Tests
- `packages/shade-transport-webrtc/tests/` — wire-format roundtrips,
signaling routing, full memory-factory caller/callee handshake,
receiver-hook dispatch (chunk + resume-query), glare convergence,
TURN-only configuration plumbing, native-adapter availability
smoke test.
- `packages/shade-transfer/tests/multi-fallback.test.ts` — N-ary
demotion, sticky-after-failure, non-transport-error preservation,
empty-list rejection.
- `packages/shade-sdk/tests/webrtc-integration.test.ts` — two real
Shade instances upload via WebRTC primary; verifies the engine
picks `webrtc` and never demotes during the run.
- `packages/shade-sdk/tests/webrtc-failover.test.ts` — broken-RTC
factory provokes connect timeout; SDK demotes to HTTP within the
V3.11 5-second SLO without losing chunks.
- `packages/shade-sdk/tests/webrtc-throughput.test.ts` — 4 MiB / 4
lanes loopback over WebRTC vs HTTP; integrity match across both
transports + diagnostic speedup ratio.
### Documentation
- `docs/webrtc.md` — full V3.11 guide (NAT-traversal table, TURN
config matrix, connection flow, glare resolution, backpressure,
multi-fallback wiring, diagnostics, wire format, limits, migration).
- `packages/shade-transport-webrtc/README.md` — package quickstart.
- README + CHANGELOG + ROADMAP marked V3.11 as Done.
## [Earlier Unreleased] — Social Key Recovery (V3.10)
The biggest UX hole in any E2EE system — "what happens if I lose my
phone?" — closed without a centralized recovery agent. Pick `n`
guardians from your peers, set a threshold `k`; any `k` of them
together can rebuild your identity onto a new device, but `k-1` or
fewer cannot. Shamir Secret Sharing over GF(2^8) gates the recovery
key; AES-GCM authentication on the backup blob detects forged
shares; an OOB-confirmed fingerprint gate on the guardian side
blocks social-engineering. See `docs/recovery.md` and
`docs/V3.10.md`.
### Added
#### `@shade/recovery` (new package)
- `setupRecovery({ shade, guardians, threshold, deliver })`
primary-device flow. Generates a 32-byte `recoveryKey`,
encrypts an identity backup under the recoveryKey-derived
passphrase via `Shade.exportBackup`, Shamir-splits the key into
`n` shares, and ships one `share-deposit` envelope per guardian
over the existing 1:1 Shade session. Returns a per-guardian
delivery report so partial-distribution is recoverable.
- `attachGuardian({ shade, store, approve, deliver })`
guardian-side receiver. Wires a `Shade.onMessage` handler that
persists incoming deposits in a caller-supplied `RecoveryStore`
and gates `recovery-request` envelopes behind a user-driven
`approve` callback. Auto-declines requests for unknown
`(originalAddress, setupId)` pairs.
- `requestRecovery({ shade, originalAddress, setupId, threshold,
guardians, deliver })` — new-device flow. Sends one
`recovery-request` per guardian, collects `share-grant` /
`share-decline` replies, Shamir-combines the threshold-many
grants, and atomically swaps in the restored identity via
`Shade.importBackup`. Forged shares are detected by the
AES-GCM tag on the backup blob; the loop tries every
threshold-sized subset of grants before giving up.
- Pure-TS Shamir Secret Sharing primitives (`splitSecret`,
`combineShares`, `encodeShare`, `decodeShare`) over GF(2^8)
with constant-time table lookups. Exported for advanced
callers and hardware-token integrations.
- `MemoryRecoveryStore` for tests + a `RecoveryStore` interface
apps implement against IndexedDB / SQLite / AsyncStorage / etc.
- Errors: `RecoveryError`, `RecoveryDeclinedError`,
`RecoveryTimeoutError`, `RecoveryReconstructionError`,
`RecoveryProtocolError`, `RecoveryGuardianRejectedError`.
- Wire protocol: `share-deposit`, `recovery-request`,
`share-grant`, `share-decline` JSON envelopes carried over
Double-Ratchet plaintext.
#### `@shade/widgets`
- `<RecoverySetup />` — primary-device guardian-picker + threshold
slider, drives `setupRecovery` and exposes `formatRecoveryCard`
for the user's offline copy.
- `<RecoveryRequest />` — new-device widget that displays the
temporary fingerprint prominently, drives `requestRecovery`,
and reports per-guardian progress live.
- `<RecoveryApprove />` — guardian-side widget. Renders the
pending request with original-vs-new fingerprint side-by-side
and enforces a two-checkbox gate ("matches" + "OOB-verified")
before the release button is clickable.
- `createApprovalQueue()` — turns the `attachGuardian.approve`
callback into a deferred queue the widget can consume.
#### `@shade/core`
- **Bug fix.** `initReceiverSession` now copies the
`localDHKeyPair` into the session so the eventual zeroize on
DH ratchet step touches a scratch buffer, not the persisted
signed prekey. Pre-V3.10 this corrupted the receiver's signed
prekey after the first incoming X3DH from any sender — a bug
surfaced by V3.10's multi-sender recovery flow but harmful to
any user receiving messages from more than one peer.
Regression test in `packages/shade-core/tests/ratchet.test.ts`.
### Acceptance criteria (V3.10)
- [x] 3-of-5 recovery works end-to-end on two separate Shade
instances. (`packages/shade-recovery/tests/integration.test.ts`)
- [x] No coalition of `(k-1)` guardians can reconstruct the
`recoveryKey` (verified with `fast-check` property tests).
(`packages/shade-recovery/tests/shamir.test.ts`,
`tests/adversarial.test.ts`)
- [x] Guardian-side widget requires fingerprint-confirmation
before sending a share. Two-checkbox enforcement +
symmetric tests of both honest-OOB-confirm and
hostile-fingerprint-mismatch paths.
## [Unreleased] — Web Workers Crypto (V3.8)
Big in-browser uploads stay smooth: AES-GCM, HKDF, HMAC, X25519, Ed25519
and full per-lane stream state now run in a dedicated Web Worker. The
main thread only buffers and forwards plaintext slices over zero-copy
`postMessage`; lane keys never cross the thread boundary. Opt-in via
`shade.configureWorkerCrypto({ workerUrl })`. See `docs/web-workers.md`
and `docs/archive/V3.8.md`.
### Added
#### `@shade/crypto-web`
- `WorkerCryptoProvider` — drop-in `CryptoProvider` proxy that forwards
every async op to a dedicated Web Worker via the `worker-protocol`.
Sync helpers (`randomBytes`, `randomUint32`, `constantTimeEqual`,
`zeroize`) execute on the calling thread — no useless round-trips.
- `createWorkerCryptoProvider({ workerUrl, idleTimeoutMs?, spawn? })`
factory. Spawns lazily, completes a protocol-version handshake, and
self-terminates after 30 s (configurable) of inactivity. Idempotent
re-spawn on next call.
- `WorkerStreamSender` / `WorkerStreamReceiver` — main-thread handles on
`StreamSender` / `StreamReceiver` instances that live entirely inside
the worker. Plaintext is shipped via transferable `ArrayBuffer`s; lane
keys + running sha256 stay worker-side.
- `createEncryptStream` / `createDecryptStream` — TransformStream
factories. `pipeThrough(encryptStream)` consumes plaintext and emits
one wire-encoded `stream-chunk` envelope per write. Both expose a
`laneSha256` promise that resolves once the stream finishes.
- New subpath export: `@shade/crypto-web/worker` is the dedicated
module-worker entrypoint. Bundle with the standard
`new URL('@shade/crypto-web/worker', import.meta.url)` idiom.
- `rotate()` and `destroy()` lifecycle controls — call after identity
rotation to bound the worst-case duration any lane key sits in worker
memory.
#### `@shade/sdk`
- `shade.configureWorkerCrypto({ workerUrl, idleTimeoutMs? })`
opt-in setup. Without it, `encryptStream` / `decryptStream` throw a
clear error pointing to the docs.
- `shade.encryptStream({ streamId, streamSecret, laneId?, chunkSize? })`
`{ stream, laneSha256 }` — TransformStream with an end-of-stream
sha256 promise for end-to-end integrity proofs.
- `shade.decryptStream(...)` — inverse. Strict in-order seq, AAD-bound
AEAD, replay-rejecting.
- `shade.getWorkerCrypto()` — direct access to the worker-backed
`CryptoProvider` for one-off heavy ops.
- `shade.shutdown()` now also `destroy()`s the worker provider.
### Acceptance criteria (V3.8)
- [x] 100 MB upload in Chrome without blocking the main thread
> 16 ms in P99 (verification recipe in
`docs/web-workers.md#verifying-main-thread-budget`).
- [x] Safari works at default chunk-size — every `postMessage` carries
≤ 256 KiB + AEAD overhead, far below Safari's transferable cap.
- [x] Worker terminates within 30 s of last use (default
`idleTimeoutMs`), and re-spawns transparently on the next call.
---
## [Unreleased] — Transport Bridge (V3.7)
A canonical fallback chain for clients that cannot or will not run a
WebSocket: SSE primary, long-poll secondary, plus a thin WS adapter for
the happy path. All three transports surface the same `IncomingMessage`
shape so application code stays portable across browser-extension,
edge-runtime, and proxy-locked environments. See `docs/transport.md`
and `docs/archive/V3.7.md`.
### Added
#### `@shade/transport-bridge` (new)
- `IncomingMessage``{ from, bytes, receivedAt, msgId? }` — single
shape across every transport.
- `BridgeTransport``connect({ onMessage }) → disconnect()` contract.
- `WsBridge`, `SseBridge`, `LongPollBridge` — three concrete transports
consuming the matching `/v1/bridge/{ws,stream,poll}` endpoints.
- `FallbackBridgeTransport` — sticky-after-first-success priority chain.
Exposes `activeKind` and `attempts` for observability.
- `signBridgeQuery` — Ed25519-signed query-string builder (the only
carrier that survives `EventSource`'s no-headers restriction).
- Auto-reconnect with exponential backoff for WS + SSE; `Last-Event-ID`
cursor resume for SSE; bounded one-outstanding-request loop for
long-poll.
#### `@shade/inbox-server`
- `createBridgeRoutes({ store, crypto, events, … })` returns
`{ app, websocket }`.
- `GET /v1/bridge/stream` — SSE feed, one envelope per `event:
envelope`. Heartbeats every 15 s as `: ping` comments.
- `GET /v1/bridge/poll?timeoutMs=…` — long-poll, default 25 s server
hold under typical proxy idle cutoffs, hard cap 55 s.
- `GET /v1/bridge/ws` — Bun-WebSocket upgrade, JSON frame per
envelope.
- Push-style delivery via `InboxServerEvents`
(`inbox.blob_stored`); falls back to a 1 s polling timer when no
events emitter is wired.
- Cross-endpoint replay-protected: `kind` is bound into the canonical
signed payload so a `/poll` signature cannot reach `/stream`.
#### `@shade/server` standalone container
- Bridge routes mount on the same Hono app + Bun.serve as the prekey
and inbox routes — no extra port, no extra env vars.
### Acceptance criteria (V3.7)
- [x] Same "send 100 small messages" suite passes on WS, SSE, and
long-poll.
- [x] Client that starts with WS and is blocked by proxy continues
automatically via SSE — and on through to long-poll if SSE is
also blocked — without message loss.
- [x] Long-poll fallback uses no more than one outstanding request per
client.
---
## [Unreleased] — Async Store-and-Forward (V3.6)
A dedicated relay (`@shade/inbox-server`) holds ciphertext blobs with TTL
+ auth so a sender can deliver to an offline recipient. Server stores
only `address || msgId || ciphertext-bytes || expires_at`; the prekey
server stays public-keys-only, and the relay never holds plaintext or
private keys. See `docs/inbox.md` and `docs/archive/V3.6.md`.
### Added
#### `@shade/inbox` (new)
- `Inbox` — high-level orchestrator. Buffers outgoing PUTs in a durable
queue, polls + acks incoming blobs, and exposes
`onMessageQueued(handler)` (the vendor-neutral push-trigger hook
mandated by V3.6) and `onIncoming(handler)`.
- `InboxClient` — low-level HTTP client (`register`, `put`, `fetch`,
`ack`, `unregister`).
- `OutgoingQueueStore` interface + `MemoryOutgoingQueueStore` default —
swap in a SQLite/IDB backend so queue survives a process restart.
- `CursorStore` interface + `MemoryCursorStore` default for the receive
cursor.
- `computeMsgId(ciphertext)` helper — `lowercase-hex(sha256(ciphertext))`.
#### `@shade/inbox-server` (new)
- `createInboxServer({ crypto, store, ... })` Hono app exposing:
- `POST /v1/inbox/register` — TOFU bind address ↔ signing key.
- `DELETE /v1/inbox/register/:address` — signed unregister.
- `POST /v1/inbox/:address` — signed PUT, idempotent on `(address, msgId)`,
rejects mismatched `msgId !== sha256(ciphertext)` and bodies past
`maxBlobBytes` (default 1 MiB) or per-recipient quota (default 1000).
- `POST /v1/inbox/:address/fetch` — signed challenge, cursor-paginated.
- `DELETE /v1/inbox/:address/:msgId` — signed ack.
- `InboxStore` interface + `MemoryInboxStore` default.
- `InboxPruneTask` — periodic prune of expired blobs (cron, default 5 min).
- `InboxServerEvents` — structural-only event emitter for observability.
#### `@shade/storage-sqlite`
- `SqliteInboxStore``(address, expires_at)` + `(address, received_at)` +
`(expires_at)` indexes. `SHADE_INBOX_DB_PATH` env var for the file path.
#### `@shade/storage-postgres`
- `PostgresInboxStore` — concurrent-safe via `INSERT … ON CONFLICT` and a
per-row `nextval('shade_inbox_seq')`. `ensureInboxServerTables(sql)` is
exported for embedded deployments.
#### `@shade/server` standalone container
- Inbox routes mount alongside prekey routes on the same Hono app.
- New env vars: `SHADE_INBOX_DB_PATH`, `SHADE_INBOX_PG_URL`,
`SHADE_INBOX_PRUNE_INTERVAL_MINUTES`. If `SHADE_INBOX_PG_URL` is unset
the inbox falls back to `SHADE_PREKEY_PG_URL` (single Postgres deploy).
### Acceptance criteria (V3.6)
- [x] Sender → recipient with no online overlap; payload < 1 MiB; first
poll after recipient startup pulls the queued message.
- [x] Server-DB dump exposes no plaintext and no sender-recipient graph
beyond byte-pair sizes (sender pubkey is per-PUT TOFU; only the
recipient address is persisted).
- [x] Replay of PUT with the same `msgId` returns 200 with
`idempotent: true` instead of 409, and no second row is written.
## [0.4.0] — 2026-05-02 — Fingerprint Gates & Trust UX (V3.3)
Blocking verification gates for the handful of operations where MITM risk
is real. Apps stay alert-fatigue-free for ordinary chat, but `upload()`
of a large file, `importBackup()`, and `acceptIdentityChange()` now run
through user-registered handlers before they touch anything sensitive.
See `docs/trust-ux.md` and `docs/archive/V3.3.md`.
### Added
#### `@shade/sdk`
- `Shade.beforeFirstLargeFile(threshold, handler)` — gate runs in
`upload()` when the file size meets the threshold (default 10 MiB) and
the peer is unverified.
- `Shade.beforeBackupImport(handler)` — gate receives the fingerprint of
the identity *embedded in the backup blob*, before any state is written.
- `Shade.beforeNewDeviceTrust(handler)` — gate runs from
`Shade.acceptIdentityChange()`. The peer's identity-version is bumped
first, so any prior verification automatically goes stale.
- `Shade.beforeInboxFanout(handler)` — reserved hook for V3.6 fan-out;
apps can register today.
- `Shade.markPeerVerified(address)` / `isPeerVerified(address)` /
`unmarkPeerVerified(address)` — manual control over persisted
verification state.
- `decryptBackup` / `applyBackupPayload` — split of the backup pipeline
so callers can inspect a backup's identity fingerprint before writing.
- New `FingerprintGateRegistry` exported for advanced integrations.
#### `@shade/core`
- `FingerprintNotVerifiedError` (HTTP 403) — raised when a gate handler
returns `false`, throws, or is missing in environments that policy-
forbid TOFU.
- `PeerVerification` + `PeerVerificationSource` types and storage
methods on `StorageProvider`: `savePeerVerification`,
`getPeerVerification`, `removePeerVerification`,
`getPeerIdentityVersion`, `bumpPeerIdentityVersion`.
#### Storage backends
- `MemoryStorage`, `SQLiteStorage`, `PostgresStorage`,
`EncryptedSQLiteStorage`, `EncryptedPostgresStorage` all carry the new
`peer_verifications` + `peer_identity_versions` tables.
#### `@shade/widgets`
- `<FingerprintGate peerAddress=... />` — render-prop wrapper that blocks
children until the peer's safety number is verified at the current
identity-version. SSR-safe; ships a default fallback with "Copy OOB
text" + "I have verified" actions.
- `<FingerprintCompare onVerified=... />` — existing widget extended with
the same two actions when wired to a callback.
- `formatOobText(peerAddress, fingerprint)` helper exported.
### Changed
- `@shade/sdk` version bumped to 0.4.0 alongside all packages (lockstep
per ROADMAP convention).
### Migration
- No breaking changes. Apps that don't register gate handlers get
warning-mode TOFU automatically (`'tofu-after-warning'` source on the
persisted verification). To upgrade to hard gates, register handlers
for the operations you use. Existing `<FingerprintCompare />` calls
keep working.
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to ship. Apps keep their own UI; this layer ships the typed RPC, the streams bridge for content I/O, and production hooks (rate limit, retention, fingerprint gate, metrics). @shade/files (NEW) - Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with Zod-validated wire schemas + clean user-handler types. - Custom ops: typed via TypeScript declaration merging on CustomOpsMap + per-op Zod schemas; client.custom('app.foo', {...}) is fully typed. - Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams (> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId / shadeFilesReadStreamId correlation. Server-side TransformStream bridges accept inbound transfers immediately (engine rejects chunks that arrive before accept) and park the readable for the matching RPC. - Directory ops: walk(path, opts) async-iterable depth-first walker; uploadDirectory()/downloadDirectory() with bounded concurrency pool (default 4, cap 16), aggregated progress, abort. - Production hooks (callback-based, vendor-neutral): rate-limit (op + byte), idempotency cache (LRU + TTL + in-flight de-dupe), path policy (traversal + percent-decode hardening), fingerprint gate (required/optional/reject), pluggable Ed25519 sig verification with ±5 min replay window, onMetric sink (standard names). - React hooks (subpath @shade/files/react): ShadeFilesProvider, useShadeFiles, useFileList, useFileTransfer/Upload/Download. - Shade.files.serve(handler) + Shade.files.client(peer) high-level entrypoint in @shade/sdk; lazy + memoized; one handler per Shade. Wire format bump - @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from u16 to u32. The previous u16 silently truncated payloads above 64 KiB — a hard correctness ceiling that blocked inline file ops up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions only. Cross-platform Kotlin port (android/shade-android) updated to match; test-vectors/wire-format.json regenerated. Concurrency safety - ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex. Concurrent decryptions of the same peer raced ratchet state (manifested as sporadic "Failed to decrypt — wrong key or tampered data" under load — surfaced once concurrent uploadDirectory pumped many writes in flight). Encrypt was already serialized via Shade.send's encryptChains; decrypt is now serialized at the manager layer too. @shade/streams extension - StreamMetadata.userMetadata?: Record<string, string> for application-level key/value pairs that round-trip verbatim through stream-init plaintext. Used by @shade/files for write/read correlation; available to any consumer. @shade/sdk extension - Shade.files getter (lazy + memoized). - BackgroundHooks.onPruneFiles + periodic timer (default 5 min) + BackgroundTasks.setHook(name, fn) for runtime hook registration. Bundles in-flight 0.2.0 work - packages/shade-streams/, packages/shade-transfer/, related shade-sdk streams-bridge + shade-widgets transfer hooks were uncommitted prior to this session. Including them keeps the workspace consistent at 0.3.0 since @shade/files depends on them. Tests - 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail; 3× stable). Coverage spans unit (inline-threshold + concurrency), integration (read-write inline + streams up to 1 MiB, walk + upload/download directory, custom-op, metrics, SDK namespace end-to-end), and security (tampered-envelope sig verification, replay window, fingerprint gate, rate-limit + quota). Release artifacts - All packages bumped to 0.3.0 via scripts/bump-version.ts. - scripts/publish-all.ts PACKAGES updated with shade-files in topological order (after shade-transfer, before shade-sdk). - bun run publish:dry clean (14 packed, 0 failed). - examples/08-files-browser/ — three-process CLI demo (prekey + Bob server + Alice CLI) covering list/stat/mkdir/delete/upload/download. - docs/files.md — full API + design doc. - CHANGELOG.md 0.3.0 entry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
## [0.3.0] — 2026-05-02 — Shade Files
E2EE filesystem RPC primitive — drop-in entrypoints for any consumer that
wants to expose a filesystem (or filesystem-like surface) over Shade. Apps
keep their own UI; this layer ships the typed RPC, the streams bridge for
content I/O over 256 KiB, and production hooks (rate limit, retention,
fingerprint gate, metrics).
### Added
#### `@shade/files` (NEW)
- Standard ops: `list`, `stat`, `mkdir`, `delete`, `move`, `read`, `write`,
`getThumbnail` — Zod-validated wire schemas + clean user-handler types.
- Custom ops: `client.custom('app.foo', {...})` with full type-safety via
TypeScript declaration merging on `CustomOpsMap` + per-op Zod schemas
registered server-side.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams (> 256 KiB)
ride `@shade/transfer` with automatic correlation via
`userMetadata.shadeFilesWriteId` / `shadeFilesReadStreamId`.
- Directory ops: `walk(path, opts)` async-iterable depth-first walker;
`uploadDirectory()` / `downloadDirectory()` with bounded concurrency
pool (default 4, cap 16), aggregated progress events, abort support.
- Production hooks (all callback-based, vendor-neutral):
- **Rate limit**: token-bucket per sender, op-cost + byte-quota,
`FsRateLimitError` / `QuotaExceededError` with `retryAfterMs`.
- **Idempotency cache**: per-sender LRU + TTL, in-flight de-dupe,
periodic prune via `BackgroundHooks.onPruneFiles`.
- **Path policy**: built-in traversal hardening, percent-decode,
forbidden-bytes check, root-scope, symlink toggle, `extra` predicate.
- **Fingerprint gate**: `requireFingerprintVerifiedFor(ctx)`
`'required' | 'optional' | 'reject'` + `isFingerprintVerified(sender)`.
- **Signature verification**: pluggable `verifySender(sender, canonical, sig)`
with replay-window enforcement (±5 min `signedAt` skew rejected).
- **Metrics**: `onMetric(name, value, tags)` with standard names
(`shade_files_op_duration_ms`, `_op_total`, `_bytes_in/out`,
`_idempotency_hit/conflict_total`, `_rate_limit_reject_total`,
`_fingerprint_reject_total`, `_signature_reject_total`).
- React hooks (subpath import `@shade/files/react`):
`<ShadeFilesProvider>`, `useShadeFiles`, `useFileList`,
`useFileTransfer` / `useFileUpload` / `useFileDownload`. SSR-safe; no UI
components — apps bring their own.
- High-level entry: `Shade.files.serve(handler)` and `Shade.files.client(peer)`
in `@shade/sdk`. Lazy + memoized; one handler per Shade instance.
- Drop-in adapter: `createMemoryDirectory()` for tests; structurally
compatible with browser `FileSystemDirectoryHandle`.
#### Wire format bump
- `@shade/proto` wire VERSION bumped from `0x01` to `0x02`. Length prefixes
changed from u16 to u32 — previous limit was 64 KiB ratchet payloads,
which blocked inline file ops up to 256 KiB.
**Wire-incompatible with 0.2.x peers.** New sessions only.
- Cross-platform Kotlin port (`android/shade-android`) updated to match.
#### Concurrency safety
- `ShadeSessionManager.encrypt` / `.decrypt` now run under per-peer mutex.
Previously, concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic `Failed to decrypt — wrong key or tampered data`
under load). Encrypt was already serialized via `Shade.send`'s
`encryptChains`; decrypt is now serialized at the manager layer too.
#### `@shade/streams` extension
- `StreamMetadata` gets optional `userMetadata?: Record<string, string>`
application-level key/value pairs that round-trip verbatim through
`stream-init` plaintext. Used by `@shade/files` for write/read correlation
but available to any consumer.
#### `@shade/sdk` extension
- `Shade.files` getter (lazy + memoized).
- `BackgroundHooks.onPruneFiles?: () => void` + periodic timer (default 5 min)
for `@shade/files` retention.
- `BackgroundTasks.setHook(name, fn)` for runtime hook registration.
### Examples
- `examples/08-files-browser/` — three-process demo (prekey + Bob server +
Alice CLI) covering list/stat/mkdir/delete/upload/download with both
inline and streamed paths.
### Tests
- 100+ new tests across `tests/{unit,integration,security}/` in
`@shade/files`. End-to-end coverage for streams I/O up to 1 MiB, custom-op
registration + Zod validation, fingerprint-gate rejection, replay-window
enforcement, idempotent retries, rate-limit + quota enforcement, walk
+ bulk transfer aggregated progress.
## [0.2.0] — 2026-05-01 — Shade Streams
E2EE chunked upload/download with parallel lanes, resumable transfers, and a
"magic drop-in" UX for any Shade-using app. Adds two new packages
(`@shade/streams`, `@shade/transfer`) and extends `@shade/sdk` and
`@shade/widgets` with high-level transfer APIs.
### Added
#### Streams crypto layer (`@shade/streams`)
- HKDF stream/lane key derivation (`deriveStreamKey`, `deriveLaneKey`)
- Deterministic AES-GCM nonce construction `nonce = laneId(4) || seq(8)`
- Streaming SHA-256 via `@noble/hashes/sha2.js` for memory-bounded integrity
- `StreamSender` / `StreamReceiver` per-lane state machines with strict
in-order seq + replay detection (`StreamReplayError`,
`StreamOutOfOrderError`, `StreamDecryptionError`, `StreamProtocolError`)
- `MultiLaneSender` / `MultiLaneReceiver` coordinators for parallel transfers
- Range and round-robin partitioning helpers (`planRangePartition`,
`planRoundRobinPartition`, `chunkRange`)
- Wire format: new envelope type `0x11` (stream-chunk) in `@shade/proto`,
control envelopes (`stream-init` / `-finish` / `-abort` / `-resume-*`)
ride existing `0x02` ratchet messages with JSON `kind` discriminator
#### Transfer orchestration (`@shade/transfer`)
- `TransferEngine` — single class wrapping outgoing + incoming lifecycle
- Default `ShadeTransferHttpTransport` for chunk POSTs, opt-in
`ShadeTransferWsTransport` with `FallbackTransferTransport` for auto-fallback
- `createTransferRoutes()` Hono factory mounts `/v1/transfer/*` routes
(`chunk`, `state`, `health`)
- `IControlChannel` + `MemoryControlChannel` for in-process testing;
the SDK provides `ShadeControlChannel` over `Shade.send`/`receive`
- Resume protocol: `MemoryResumeStore`, `StorageBackedResumeStore`,
`deriveDeviceKey()` for at-rest streamSecret encryption,
`engine.resumeUpload(streamId, freshInput)` for kill-restart-verify flows
- `ProgressTracker` with EMA-smoothed throughput + ETA
- Retry/backoff (`withRetry`) with exponential delay + jitter
- Error hierarchy: `TransferError`, `TransferAbortError`,
`TransferIntegrityError`, `TransferProtocolError`, `TransferOfflineError`,
`TransferResumeError`, `TransferTransportError`
#### SDK (`@shade/sdk`)
- `Shade.upload(opts)` — high-level entry; encrypts + chunks + ships
- `Shade.onIncomingTransfer(handler)` — receiver-side subscription
- `Shade.transferRoute()` — Hono router to mount on the consumer's HTTP server
- `Shade.acceptTransferEnvelope(from, env)` — low-level entry for custom transports
- `Shade.resumeUpload(streamId, freshInput)` — pick up an interrupted transfer
- `Shade.listTransfers(filter?)` — list resumable / active transfers from storage
- `ShadeTransferAuthenticator` — Ed25519-signing authenticator for HTTP/WS transports
- `Shade.onMessage(handler)` now accepts `Promise<void>`-returning handlers
(awaited in sequence) — supports flow-control over the control plane
#### Storage (all backends)
- New optional `StorageProvider` methods: `saveStreamState`,
`getStreamState`, `removeStreamState`, `listActiveStreamStates`,
`pruneStreamStates`. Existing v0.1.x providers compile cleanly (optional methods)
- SQLite (`stream_state` table) and Postgres (`shade_stream_state` table)
schemas with at-rest encrypted streamSecret
- `MemoryStorage` extended with in-memory stream-state map
#### Widgets (`@shade/widgets`)
- `<ShadeRuntimeProvider runtime={shade}>` — separate React context for
upload/download widgets (distinct from the observer-dashboard `<ShadeProvider>`)
- `useShadeUpload()` / `useShadeDownload()` headless hooks
- `<ShadeUploader />` / `<ShadeDownloader />` composite components with
render-prop pattern for full UI replacement
- Sub-components: `<DropZone />`, `<TransferRow />`, `<ProgressBar />`,
`<SpeedReadout />`, `<ETAReadout />`, `<LaneIndicator />`
- Theme-token additions for progress, drop zone, and lane indicator colors
### Security properties
- Per-chunk AES-256-GCM with deterministic nonce; AAD binds
`streamId || laneId || seq || isLast` so any header tamper invalidates AEAD
- streamSecret never on the wire in plaintext — shipped via Double Ratchet
control envelope; lane keys derived locally and never transmitted
- Resume state encrypted at rest with `deviceKey` derived from identity's
signing private key (rotation invalidates in-flight resume — by design)
- Receiver enforces strict in-order seq per lane (`StreamOutOfOrderError`,
`StreamReplayError`); finish-time integrity check verifies per-lane sha256
+ overall sha256 over original byte order
### Tests added (118 new across 47 files; 444 total)
- Unit: KDF, nonce, AEAD, streaming SHA, sender/receiver, partition
- Integration: 1/4/16-lane parity, range vs round-robin parity,
Bun.serve loopback at 100 KiB / 1 MiB / 8 MiB, two real Shade instances
end-to-end at 64 KiB / 512 KiB / 4 MiB
- Resume: kill-restart-verify on 256 KiB with 4 lanes
- WS fallback: WS connect failure → transparent HTTP completion
- Tamper: bit-flip ciphertext / tag / header field; replay; out-of-order
- Wire: 0x11 envelope encode/decode roundtrip + edge cases
### Backward compatibility
- `Shade.send`/`receive`/`onMessage`/`fingerprint`/`rotate` unchanged
(`onMessage` widened to support async handlers — sync handlers still work)
- Existing wire types `0x01` (PreKeyMessage) / `0x02` (RatchetMessage) unchanged
- `StorageProvider` interface extension uses optional methods
- `@shade/streams` and `@shade/transfer` are new packages; no migration
## [1.0.0] — 2026-04-10
### First production release
Shade implements the Signal Protocol (X3DH + Double Ratchet) as a standalone, audit-friendly E2EE library for TypeScript/Bun.
### Added
#### Core protocol
- **X3DH** key agreement (X25519 + Ed25519, supports asynchronous bundles)
- **Double Ratchet** with forward secrecy and post-compromise recovery
- Skipped message key cache for out-of-order delivery (max 1000 per chain)
- Header-bound AAD on AES-256-GCM encrypts (tampered headers fail decryption)
- Memory zeroization of message keys, chain keys, root keys, and DH private keys after use
#### Storage
- `MemoryStorage` (in-memory, for tests/embedded)
- `SQLiteStorage` (`@shade/storage-sqlite`) — bun:sqlite, WAL mode, crash-safe
- `PostgresStorage` (`@shade/storage-postgres`) — Drizzle, FOR UPDATE SKIP LOCKED
- All backends survive container restarts and SIGKILL
- Identity history with 7-day grace period for rotation
#### Prekey server (`@shade/server`)
- Hono-based REST API with self-authenticated registration (Ed25519 signatures)
- Anonymous bundle fetches (read-only)
- Per-IP and per-identity rate limiting (token bucket)
- Address validation (NFKC normalization, alphanumeric + `:_-.`)
- ±5 minute replay window on signed requests
- Health endpoints (`/health`, `/healthz`, `/ready`)
- Prometheus metrics (`/metrics`)
- Structured JSON logging
- Graceful shutdown on SIGTERM/SIGINT
- Production Dockerfile with non-root user, healthcheck, multi-stage build
- docker-compose.yml example for Dokploy
#### Session manager (`@shade/core`)
- `ShadeSessionManager` high-level API (`encrypt`, `decrypt`, `initSessionFromBundle`)
- `getIdentityFingerprint()` — Signal-style 60-digit safety numbers
- `ensurePreKeyStock()` — auto-replenish when below threshold
- `resetSession()` and `acceptIdentityChange()` for recovery scenarios
- `rotateIdentity()` with archived previous identities
#### Transport (`@shade/transport`)
- `ShadeFetchTransport` — HTTP client for the prekey server with auto-signing
- `ShadeWebSocket` — WebSocket wrapper with transparent encrypt/decrypt
#### Wire format (`@shade/proto`)
- Compact binary encoding (significantly smaller than JSON)
- Length-prefixed byte arrays, big-endian integers
- Version-tagged envelopes for forward compatibility
#### Cryptographic hardening
- `constantTimeEqual` (XOR-accumulator, no early exit)
- `randomUint32` via crypto.getRandomValues (no Math.random)
- Timing-attack regression test
- Constant-time trust verification in all storage backends
#### Errors
- Stable `SHADE_*` error codes
- `errorToHttpStatus` for consistent HTTP mapping
- `toJSON()` for network serialization
- 14 specific error types (Validation, Network, Storage, RateLimit, etc.)
#### Documentation
- README, SECURITY.md, THREAT-MODEL.md
- 5 runnable examples (basic conversation, prekey server, WebSocket tunnel, identity verification, Dokploy deployment)
- Per-package READMEs
- Inline TSDoc throughout
#### Testing
- 195+ tests across all packages
- Crash recovery integration test
- Cross-platform PostgreSQL tests (skip without `SHADE_TEST_PG_URL`)
- CI workflow with PostgreSQL service
- Benchmark suite
### Security properties
- Forward secrecy
- Post-compromise security
- Authenticated identity verification
- Replay protection
- Constant-time secret comparisons
- Memory zeroization (best-effort)