Compare commits
7 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 7520b11b25 | |||
| da93b97cce | |||
| 0bdf9e859c | |||
| 70e319fef8 | |||
| f301b391a5 | |||
| 40766c60f4 | |||
| de25b19033 |
391
CHANGELOG.md
391
CHANGELOG.md
@@ -5,6 +5,397 @@ All notable changes to Shade are documented in this file.
|
|||||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||||
|
|
||||||
|
## [4.2.0] — 2026-05-03 — Pull-mode streams for browser @shade/files
|
||||||
|
|
||||||
|
`4.1.0` shipped HTTP RPC for browser clients but capped them at inline
|
||||||
|
payloads (≤ 256 KiB). Larger reads/writes — mod-jars (1–50 MB),
|
||||||
|
world-backups (100+ MB), the things that actually need streaming —
|
||||||
|
threw `ConflictError` directing callers to the server-to-server
|
||||||
|
pathway. That made browser-side `@shade/files` insufficient for
|
||||||
|
admin-panel-style apps where the client is a browser tab and the
|
||||||
|
server is a Bun process.
|
||||||
|
|
||||||
|
`4.2.0` flips the direction: when the browser supplies
|
||||||
|
`outboundQueueUrl` + `transferBaseUrl`, server-to-browser chunks +
|
||||||
|
control envelopes ride a per-peer queue that the browser long-polls,
|
||||||
|
and browser-to-server chunks POST directly to the server's existing
|
||||||
|
chunk-receive routes. No WebSockets, no SSE, no inbound listener on
|
||||||
|
the browser. Long-polling + a request-response inbound queue is
|
||||||
|
the entire wire surface.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
#### `@shade/transfer`
|
||||||
|
- `OutboundQueue` — per-peer monotonic event log with long-poll
|
||||||
|
semantics. `enqueue(peer, event)` appends, `drain(peer, since,
|
||||||
|
blockMs, signal)` returns events with `id > since` (blocking up
|
||||||
|
to `blockMs` if none are ready). Idle-eviction GC drops peers
|
||||||
|
that haven't polled in `idleEvictionMs` (default 10 min). Ring-
|
||||||
|
buffered to `maxEventsPerPeer` (default 1000) — overflow drops
|
||||||
|
oldest, receivers pick up the gap via re-resume from `since=0`.
|
||||||
|
- `QueuedEvent` discriminated union: `{ kind: 'envelope', bytes }`
|
||||||
|
or `{ kind: 'chunk', bytes, meta: { streamId, laneId, seq } }`.
|
||||||
|
- `QueueTransferTransport` (implements `ITransferTransport`) —
|
||||||
|
enqueues outbound chunks instead of POSTing. Returns optimistic
|
||||||
|
`ChunkAck` because the queue *is* the delivery; chunk-resume picks
|
||||||
|
up dropped events on receiver-side reconnect.
|
||||||
|
|
||||||
|
#### `@shade/sdk`
|
||||||
|
- `Shade.transferQueueRoute(opts?)` — Hono app with all five routes a
|
||||||
|
pull-mode receiver needs:
|
||||||
|
- `POST /queue` — long-poll the per-peer outbound queue.
|
||||||
|
- `POST /v1/transfer/:streamId/chunk` — receive incoming chunks
|
||||||
|
(browser → server writes).
|
||||||
|
- `GET /v1/transfer/:streamId/state` — resume-state lookup.
|
||||||
|
- `POST /v1/transfer/control` — receive incoming control envelopes
|
||||||
|
(browser → server stream-init / abort).
|
||||||
|
- `GET /v1/transfer/health` — peer reachability probe.
|
||||||
|
Auto-configures `shade.configureTransfers(...)` with the queue
|
||||||
|
transport + `QueueEnvelopeTransport` if not already configured.
|
||||||
|
- `Shade.configureTransfers(opts)` extended: `resolveBaseUrl` is now
|
||||||
|
optional when `transport` and `envelopeTransport` are both supplied
|
||||||
|
(lets pure-queue servers omit the baseUrl entirely). New
|
||||||
|
`transport?: ITransferTransport` override slot.
|
||||||
|
- `QueueEnvelopeTransport` — `ControlEnvelopeTransport` impl that
|
||||||
|
enqueues outbound envelopes for browser receivers.
|
||||||
|
|
||||||
|
#### `@shade/files`
|
||||||
|
- `createFilesHttpClient` (and `shade.files.httpClient`) accept new
|
||||||
|
options:
|
||||||
|
- `outboundQueueUrl` — `/queue` endpoint to long-poll.
|
||||||
|
- `transferBaseUrl` — base URL for outbound chunk POSTs and control
|
||||||
|
envelope POSTs (browser → server writes).
|
||||||
|
- `queueBlockMs` — long-poll timeout (default 30 s; server clamps
|
||||||
|
at `maxBlockMs`).
|
||||||
|
When set, the client:
|
||||||
|
1. Configures `shade.configureTransfers({ resolveBaseUrl })` so
|
||||||
|
outbound chunks POST to `<transferBaseUrl>/v1/transfer/...`.
|
||||||
|
2. Builds a `ClientStreamsBridge` eagerly so the engine's
|
||||||
|
incoming-transfer subscription is in place before the drainer
|
||||||
|
dispatches the first envelope.
|
||||||
|
3. Starts a long-poll `startQueueDrainer(...)` that pulls queued
|
||||||
|
events and dispatches them via `shade.acceptTransferEnvelope`.
|
||||||
|
- Streamed reads (`fs.read` of files > 256 KiB) and streamed writes
|
||||||
|
(`fs.write` of large inputs) now work end-to-end on the browser
|
||||||
|
client when the queue options are set.
|
||||||
|
- `startQueueDrainer(shade, opts)` exported for advanced consumers
|
||||||
|
that want to drive their own drainer (e.g. service-worker setups
|
||||||
|
that want a single shared drainer across multiple `httpClient`s).
|
||||||
|
- `client.close()` now stops the drainer and tears down the streams-
|
||||||
|
bridge — important on tab unload to free the long-poll socket.
|
||||||
|
|
||||||
|
#### `@shade/files` (internal)
|
||||||
|
- `ClientStreamsBridge` uses a TransformStream with `highWaterMark:
|
||||||
|
64` instead of the default `0` so the receive-side write loop
|
||||||
|
doesn't stall on backpressure before the consumer attaches its
|
||||||
|
reader (default HWM stalled at chunk 4 in pull-mode where the
|
||||||
|
drainer races the consumer's `getReader()` call).
|
||||||
|
|
||||||
|
### Wire contract
|
||||||
|
|
||||||
|
```
|
||||||
|
POST <base>/queue HTTP/1.1
|
||||||
|
X-Shade-Sender-Address: alice@example.com
|
||||||
|
{ "since": 42, "blockMs": 30000 }
|
||||||
|
|
||||||
|
────
|
||||||
|
|
||||||
|
200 OK
|
||||||
|
{
|
||||||
|
"events": [
|
||||||
|
{ "id": 43, "kind": "envelope", "bytesB64": "...", "timestampMs": 1730... },
|
||||||
|
{ "id": 44, "kind": "chunk", "bytesB64": "...", "meta": { "streamId": "...", "laneId": 0, "seq": 0 } },
|
||||||
|
...
|
||||||
|
],
|
||||||
|
"nextSince": 47
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
|
||||||
|
`tests/integration/http-rpc-streams.test.ts` — three integration tests:
|
||||||
|
- 4 MiB streamed read end-to-end via long-poll queue (verifies bytes
|
||||||
|
match the source).
|
||||||
|
- Inline-only client throws clear error on streamed read.
|
||||||
|
- Long-poll returns empty events on idle timeout (verifies the
|
||||||
|
`blockMs` pathway).
|
||||||
|
|
||||||
|
### Migration
|
||||||
|
|
||||||
|
`4.1.0 → 4.2.0` is wire-compatible and source-compatible — the
|
||||||
|
queue route is purely additive. To enable streamed transfers in a
|
||||||
|
browser app:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
// Server
|
||||||
|
const queue = await shade.transferQueueRoute({ blockMs: 30_000 });
|
||||||
|
await shade.files.serve(handler);
|
||||||
|
const rpc = shade.files.rpcRoute({ acceptFirstMessage: true });
|
||||||
|
|
||||||
|
const app = new Hono();
|
||||||
|
app.route('/api/v1/shade-files', queue);
|
||||||
|
app.route('/api/v1/shade-files', rpc);
|
||||||
|
|
||||||
|
// Browser
|
||||||
|
const fs = shade.files.httpClient(serverAddress, {
|
||||||
|
rpcUrl: 'https://server/api/v1/shade-files/rpc',
|
||||||
|
outboundQueueUrl: 'https://server/api/v1/shade-files/queue',
|
||||||
|
transferBaseUrl: 'https://server/api/v1/shade-files',
|
||||||
|
});
|
||||||
|
await fs.write('/mods/some-mod.jar', new Uint8Array(/* 50 MB */));
|
||||||
|
const result = await fs.read('/backups/world.tar.gz'); // streamed
|
||||||
|
```
|
||||||
|
|
||||||
|
`shade.files.serve(handler, { inlineOnly: true })` is still supported
|
||||||
|
for HTTP-RPC-without-streams deployments — it skips the streams-bridge
|
||||||
|
setup entirely.
|
||||||
|
|
||||||
|
## [4.1.0] — 2026-05-03 — Browser-friendly HTTP RPC for @shade/files
|
||||||
|
|
||||||
|
The default `shade.files.client(peer)` requires both peers to be
|
||||||
|
mutually addressable over HTTP — the response to a `list` / `read` /
|
||||||
|
etc. round-trips through `Shade.deliverControlEnvelope`, which POSTs
|
||||||
|
to the peer's `/v1/transfer/control` endpoint. **That doesn't work
|
||||||
|
for browsers** — a tab can't host an HTTP server, so the server
|
||||||
|
cannot call back outbound.
|
||||||
|
|
||||||
|
This release ships a parallel request-response transport. One POST per
|
||||||
|
RPC, encrypted envelope in the request body, encrypted response in the
|
||||||
|
same HTTP response. Mirrors the way `@shade/server`'s
|
||||||
|
`shade-auth-middleware` works for prekey writes.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
#### `@shade/files`
|
||||||
|
- `createFilesRpcRoute(shade, handler, options?)` — Hono app exposing
|
||||||
|
`POST /rpc`. Reads `X-Shade-Sender-Address`, decrypts the envelope
|
||||||
|
via the existing ratchet session, dispatches through the attached
|
||||||
|
`FileHandler`, encrypts the result, and returns it in the same HTTP
|
||||||
|
response. Transport-level failures (no session, undecryptable, body
|
||||||
|
too big) return JSON `{ error }` with appropriate 4xx; application-
|
||||||
|
level failures ship encrypted `RpcError` envelopes.
|
||||||
|
- `createFilesHttpClient(shade, peer, options)` — request-response
|
||||||
|
`FileClient` for browser-style consumers. Each method (list / stat /
|
||||||
|
mkdir / delete / move / getThumbnail / custom / write inline / read
|
||||||
|
inline) does one HTTP POST and parses the encrypted response. No
|
||||||
|
inbound channel required.
|
||||||
|
- `shade.files.rpcRoute(opts?)` — namespace-side getter for the route.
|
||||||
|
Throws if no handler has been attached via `shade.files.serve(...)`
|
||||||
|
first.
|
||||||
|
- `shade.files.httpClient(peer, opts)` — namespace-side getter for the
|
||||||
|
client.
|
||||||
|
- `FilesNamespace.serve(handler, { inlineOnly: true })` — opt-out flag
|
||||||
|
that skips the streams-bridge setup. Required for HTTP-RPC-only
|
||||||
|
servers (which don't need `configureTransfers({ resolveBaseUrl })`).
|
||||||
|
In `inlineOnly` mode the channel-based dispatcher is also not
|
||||||
|
attached, so requests are dispatched only by the rpc-route — avoids
|
||||||
|
double-dispatch when a browser client and a server-to-server client
|
||||||
|
share the same Shade instance.
|
||||||
|
- `ShadeBridge` (exported) gains a `receive(peer, envelope)` member
|
||||||
|
matching `Shade.receive` so server-side rpc-route can decrypt
|
||||||
|
inbound envelopes through the structural surface.
|
||||||
|
|
||||||
|
### Wire contract
|
||||||
|
|
||||||
|
```
|
||||||
|
POST /rpc HTTP/1.1
|
||||||
|
Content-Type: application/octet-stream
|
||||||
|
X-Shade-Sender-Address: alice@example.com
|
||||||
|
|
||||||
|
<wire-encoded ShadeEnvelope wrapping JSON-encoded RpcRequest>
|
||||||
|
|
||||||
|
────
|
||||||
|
|
||||||
|
200 OK
|
||||||
|
Content-Type: application/octet-stream
|
||||||
|
|
||||||
|
<wire-encoded ShadeEnvelope wrapping JSON-encoded RpcResponse | RpcError>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Limitations (v1)
|
||||||
|
|
||||||
|
- **Inline payloads only** (≤ 256 KiB). `write` of larger inputs
|
||||||
|
throws `ConflictError` directing callers to `shade.files.client(peer)`
|
||||||
|
on a server-to-server deployment. Streamed `read` results throw
|
||||||
|
`InternalFileError` for the same reason.
|
||||||
|
- The X3DH first-message must ride the same RPC route — set
|
||||||
|
`acceptFirstMessage: true` on `rpcRoute({ acceptFirstMessage: true })`
|
||||||
|
when the browser client's first-ever call doubles as the handshake.
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
|
||||||
|
- `tests/integration/http-rpc.test.ts` — round-trip via HTTP
|
||||||
|
(list / mkdir / stat / write / read / delete) plus negative cases
|
||||||
|
(streamed write rejected, missing sender header, empty body, garbage
|
||||||
|
body, body past `maxBodyBytes`, `rpcRoute()` without `serve()`).
|
||||||
|
|
||||||
|
### Migration
|
||||||
|
|
||||||
|
`4.0.x → 4.1.0` is wire-compatible and source-compatible. The HTTP
|
||||||
|
RPC route is purely additive — no existing code path changes. To
|
||||||
|
adopt:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
// server (was)
|
||||||
|
await shade.files.serve(handlerConfig);
|
||||||
|
|
||||||
|
// server (HTTP-RPC mode)
|
||||||
|
await shade.files.serve(handlerConfig, { inlineOnly: true });
|
||||||
|
app.route('/api/v1/shade-files', shade.files.rpcRoute());
|
||||||
|
|
||||||
|
// browser client
|
||||||
|
const fs = shade.files.httpClient(serverAddress, { rpcUrl: '...' });
|
||||||
|
```
|
||||||
|
|
||||||
|
## [4.0.2] — 2026-05-03 — Consumer-strict reader-shape fixes
|
||||||
|
|
||||||
|
`4.0.1` shipped the `tsc --noEmit` gate that compiles each package
|
||||||
|
internally against `lib: ["ES2022"]`. That gate did not catch types
|
||||||
|
that only fail when *consumer* code (running with `lib: ["DOM"]` +
|
||||||
|
`exactOptionalPropertyTypes`) tries to assign a native browser type
|
||||||
|
into one of our locally-defined narrower types.
|
||||||
|
|
||||||
|
This release adds a consumer-strict smoke test to the pre-publish
|
||||||
|
gate and fixes every collision that smoke uncovered.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
#### `@shade/files`
|
||||||
|
- `inline-threshold.ts`: rewrote the local `MinimalReader<T>` interface
|
||||||
|
as an explicit disjoint union (`{ done: false; value: T } | { done:
|
||||||
|
true; value?: T | undefined }`) so it accepts every native reader
|
||||||
|
shape — `bun-types` (`value?: undefined`), `lib.dom` (`value?: T`),
|
||||||
|
and `node:stream/web`. The previous flat shape was rejected by
|
||||||
|
consumer projects with `exactOptionalPropertyTypes: true` because
|
||||||
|
the present-branch required `value: T`. **Fixes "Type
|
||||||
|
ReadableStreamReadResult<Uint8Array> is not assignable to
|
||||||
|
{ value: Uint8Array | undefined; done: boolean }".**
|
||||||
|
- `client/streams-bridge.ts`, `server/streams-bridge.ts`: stash the
|
||||||
|
`setTimeout(...)` return value in a local before calling `.unref?.()`
|
||||||
|
through an explicit `{ unref?: () => void }` cast. The previous
|
||||||
|
fluent `.unref?.()` failed under `lib: ["DOM"]` because DOM types
|
||||||
|
`setTimeout` to `number`, which has no `.unref` even as an optional
|
||||||
|
property.
|
||||||
|
|
||||||
|
#### `@shade/sdk`
|
||||||
|
- `background.ts`: same `setTimeout` / `setInterval` `.unref?.()` fix.
|
||||||
|
|
||||||
|
### Tooling
|
||||||
|
|
||||||
|
- New `tests/consumer-strict/` — a tiny "as if I were a downstream app"
|
||||||
|
TypeScript project with its own `tsconfig.json`:
|
||||||
|
`lib: ["ES2022", "DOM", "DOM.Iterable"]`, `types: ["bun-types"]`,
|
||||||
|
`exactOptionalPropertyTypes: true`, `strict: true`,
|
||||||
|
`paths`-mapped to the workspace's `packages/*/src/index.ts`.
|
||||||
|
Three smoke files exercise `@shade/files`, `@shade/sdk`, and
|
||||||
|
`@shade/key-transparency` against the consumer-strict tsconfig.
|
||||||
|
- `scripts/typecheck-all.ts` now runs the consumer-strict smoke after
|
||||||
|
the per-package internal type-check. Both must pass before
|
||||||
|
`prepublish:check` (and therefore `publish:dry` / `publish:all`)
|
||||||
|
succeeds.
|
||||||
|
|
||||||
|
### Migration
|
||||||
|
|
||||||
|
`4.0.1 → 4.0.2` is wire-compatible and source-compatible. No API shape
|
||||||
|
changed; only internal typing was tightened.
|
||||||
|
|
||||||
|
## [4.0.1] — 2026-05-03 — Strict-TS publishability fixes
|
||||||
|
|
||||||
|
`4.0.0` shipped TypeScript source files as the published `main` /
|
||||||
|
`types`, which meant every consumer's `tsc` had to compile our code
|
||||||
|
under their own strict settings. Several files only compiled inside
|
||||||
|
the monorepo (where peer-dep cycles resolve via workspace links and
|
||||||
|
the `lib` array doesn't include `DOM`). This release makes all 24
|
||||||
|
packages compile cleanly under the strict-flagged tsconfig that ships
|
||||||
|
with the repo, and wires a `bun run typecheck` gate into both the
|
||||||
|
`publish:dry` and `publish:all` flows so this category of bug cannot
|
||||||
|
recur.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
#### `@shade/key-transparency`
|
||||||
|
- Removed unused imports `IndexAbsenceProof`, `IndexInclusionProof`
|
||||||
|
(`src/manager.ts`), `nodeHash` (`src/index-tree.ts`).
|
||||||
|
- `IndexProofWire` is now exported (was a private type that
|
||||||
|
`noUnusedLocals` flagged).
|
||||||
|
- Added missing `tsconfig.json` so the package can be type-checked
|
||||||
|
in isolation.
|
||||||
|
|
||||||
|
#### `@shade/sdk`
|
||||||
|
- KT verifier wiring: `fetchLatestSTH()` and `fetchConsistencyProof()`
|
||||||
|
now have explicit return types (`Promise<STHWire>` and
|
||||||
|
`Promise<{ proof: string[] }>`) so consumers don't see
|
||||||
|
`Promise<unknown>` from `res.json()`.
|
||||||
|
- `STHWire` type is now imported from `@shade/key-transparency`.
|
||||||
|
- `thumbnail.ts`: cast `globalThis` through `unknown` first when
|
||||||
|
reading optional DOM globals (`OffscreenCanvas`, `createImageBitmap`)
|
||||||
|
so consumer projects that include `lib.dom` don't reject our
|
||||||
|
narrower local types as "insufficiently overlapping".
|
||||||
|
|
||||||
|
#### `@shade/files`
|
||||||
|
- **Broke the `@shade/sdk` ↔ `@shade/files` dependency cycle.**
|
||||||
|
`@shade/files` no longer imports `Shade` from `@shade/sdk` — every
|
||||||
|
callsite uses a new local `ShadeBridge` interface defined in
|
||||||
|
`src/integration/shade-bridge.ts`. This is the structural surface
|
||||||
|
Shade must satisfy: `myAddress`, `send`, `onMessage`, `upload`,
|
||||||
|
`onIncomingTransfer`, `getFingerprintFor` (required) plus
|
||||||
|
`getObservability`, `deliverControlEnvelope` (optional). The Shade
|
||||||
|
class structurally implements every member, so
|
||||||
|
`createFilesNamespace(this)` from the SDK side compiles regardless
|
||||||
|
of how many copies of `@shade/sdk` a consumer's package manager
|
||||||
|
hoists. **Fixes "this is not assignable to type 'Shade'"** in
|
||||||
|
consumer builds.
|
||||||
|
- `<ShadeFilesProvider>` now takes `files: FilesNamespace` as an
|
||||||
|
explicit prop instead of reading `shade.files`. Consumers pass
|
||||||
|
`shade.files` (or any `createFilesNamespace(...)` result for tests)
|
||||||
|
directly.
|
||||||
|
- `ShadeFileRpcChannel.send` now raises a clear error when
|
||||||
|
`deliverControlEnvelope` is undefined instead of producing an
|
||||||
|
implicit-undefined-call error at compile time.
|
||||||
|
|
||||||
|
#### `@shade/storage-encrypted`
|
||||||
|
- Replaced `KeyUsage` (a `lib.dom` type) with a local
|
||||||
|
`WebCryptoKeyUsage` union so the package compiles under
|
||||||
|
`lib: ["ES2022"]` without DOM.
|
||||||
|
- Fixed `tsconfig.json` `rootDir` so package-level `bunx tsc` works.
|
||||||
|
|
||||||
|
#### `@shade/transport-bridge`
|
||||||
|
- `sse-bridge.ts`: cast `res.body.getReader()` to
|
||||||
|
`ReadableStreamDefaultReader<Uint8Array>` so the strict reader-type
|
||||||
|
parity check in the consume loop passes.
|
||||||
|
|
||||||
|
#### `@shade/keychain` / `@shade/dashboard`
|
||||||
|
- Fixed `tsconfig.json` `rootDir` and `include` so the packages can
|
||||||
|
type-check standalone (and so `vite.config.ts` doesn't get pulled
|
||||||
|
into the dashboard's `rootDir`).
|
||||||
|
|
||||||
|
#### `@shade/widgets`
|
||||||
|
- Removed unused `ThumbnailMime` import in
|
||||||
|
`components/transfer/ThumbnailPreview.tsx`.
|
||||||
|
|
||||||
|
### Tooling
|
||||||
|
|
||||||
|
- New `scripts/typecheck-all.ts` — runs `bunx tsc --noEmit` against
|
||||||
|
every workspace package's `tsconfig.json` and fails if any reports
|
||||||
|
errors.
|
||||||
|
- New `bun run typecheck` script.
|
||||||
|
- `publish:dry` and `publish:all` now run `prepublish:check`
|
||||||
|
(`typecheck` + `test`) before any package is packed or published.
|
||||||
|
- `scripts/publish-shade.sh` calls the typecheck-all gate before
|
||||||
|
invoking the publisher.
|
||||||
|
|
||||||
|
### Migration
|
||||||
|
|
||||||
|
`4.0.0 → 4.0.1` is wire-compatible and source-compatible with one
|
||||||
|
exception:
|
||||||
|
|
||||||
|
- `<ShadeFilesProvider>` requires a `files` prop. Previously
|
||||||
|
`<ShadeFilesProvider shade={shade}>...</ShadeFilesProvider>` worked;
|
||||||
|
it now must be `<ShadeFilesProvider shade={shade} files={shade.files}>`.
|
||||||
|
|
||||||
|
No on-disk schema changes. No package-version-pin changes outside
|
||||||
|
the lockstep `4.0.0 → 4.0.1` bump.
|
||||||
|
|
||||||
## [4.0.0] — 2026-05-03 — General Availability
|
## [4.0.0] — 2026-05-03 — General Availability
|
||||||
|
|
||||||
Shade 4.0 is the first GA-marked release: every plan from V3.1 through
|
Shade 4.0 is the first GA-marked release: every plan from V3.1 through
|
||||||
|
|||||||
322
README.md
322
README.md
@@ -3,37 +3,82 @@
|
|||||||
End-to-end encryption library implementing the Signal Protocol (X3DH + Double Ratchet) for TypeScript/Bun. Drop into any project — frontend, backend, mobile — to get forward secrecy, post-compromise recovery, and self-healing security.
|
End-to-end encryption library implementing the Signal Protocol (X3DH + Double Ratchet) for TypeScript/Bun. Drop into any project — frontend, backend, mobile — to get forward secrecy, post-compromise recovery, and self-healing security.
|
||||||
|
|
||||||
> **4.0.0 — General Availability.** All V3.1 → V3.12 work is merged,
|
> **4.0.0 — General Availability.** All V3.1 → V3.12 work is merged,
|
||||||
> the cross-platform vector suite is green on TS + Kotlin, the threat
|
> the cross-platform vector suite is green on TS + Kotlin (1000 / 1000
|
||||||
> model has been refreshed for every new surface, and the core stack
|
> + 11 / 11), the threat model has been refreshed for every new
|
||||||
> (X3DH, ratchet, storage encryption, recovery, WebRTC P2P, Key
|
> surface, and the core stack (X3DH, ratchet, storage encryption,
|
||||||
> Transparency) has been packaged for external review. The wire
|
> recovery, WebRTC P2P, Key Transparency) has been packaged for
|
||||||
> format is **unchanged from 0.4.x** — 4.0 peers interoperate with
|
> external review. The wire format is **unchanged from 0.4.x** — 4.0
|
||||||
> 0.4.x peers byte-for-byte. See
|
> peers interoperate with 0.4.x peers byte-for-byte. See
|
||||||
> [MIGRATION.md § 0.3.x → 4.0](./MIGRATION.md#migrating-from-03x-to-40-ga)
|
> [MIGRATION.md § 0.3.x → 4.0](./MIGRATION.md#migrating-from-03x-to-40-ga)
|
||||||
> for the upgrade path and [CHANGELOG.md § 4.0.0](./CHANGELOG.md) for
|
> for the upgrade path and [CHANGELOG.md § 4.0.0](./CHANGELOG.md) for
|
||||||
> the consolidated release notes. Voice / Video have been moved to
|
> the consolidated release notes. Voice / Video have been moved to
|
||||||
> [V5.0](./docs/V5.0.md), to be built on top of the frozen 4.0
|
> [V5.0](./docs/V5.0.md), to be built on top of the frozen 4.0
|
||||||
> baseline.
|
> baseline.
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
| Area | 4.0 status | Pointers |
|
||||||
|
|------|-----------|----------|
|
||||||
|
| Protocol core (X3DH + ratchet + sender keys) | ✅ Done — frozen | [`packages/shade-core`](./packages/shade-core) |
|
||||||
|
| Storage encryption (V3.2) | ✅ Done — opt-in `EncryptedSQLiteStorage` / `EncryptedPostgresStorage`, key sources: passphrase / OS keychain / app-injected | [`docs/storage-encryption.md`](./docs/storage-encryption.md) |
|
||||||
|
| Fingerprint gates & trust UX (V3.3) | ✅ Done — `Shade.beforeFirstLargeFile / beforeBackupImport / beforeNewDeviceTrust` | [`docs/trust-ux.md`](./docs/trust-ux.md) |
|
||||||
|
| Observability v2 (V3.4) | ✅ Done — OpenTelemetry-shaped events, `/metrics`, observer dashboard | [`docs/observability.md`](./docs/observability.md) |
|
||||||
|
| Android parity & cross-platform CI (V3.5) | ✅ Done — TS + Kotlin vector-gate live; Android `KeystoreStorage` is post-GA | [`android/shade-android/README.md`](./android/shade-android/README.md), [`docs/cross-platform.md`](./docs/cross-platform.md) |
|
||||||
|
| Async store-and-forward (V3.6) | ✅ Done — `@shade/inbox` + `@shade/inbox-server` | [`docs/inbox.md`](./docs/inbox.md) |
|
||||||
|
| Transport bridge (V3.7) | ✅ Done — SSE / long-poll / WS adapters | [`docs/transport.md`](./docs/transport.md) |
|
||||||
|
| Web Workers crypto (V3.8) | ✅ Done — lane keys never cross the thread boundary | [`docs/web-workers.md`](./docs/web-workers.md) |
|
||||||
|
| Rich file metadata + thumbnails (V3.9) | ✅ Done — in `@shade/files` | [`docs/files.md`](./docs/files.md) |
|
||||||
|
| Social key recovery (V3.10) | ✅ Done — Shamir + AEAD-gated reconstruction + guardian widgets | [`docs/recovery.md`](./docs/recovery.md) |
|
||||||
|
| WebRTC P2P transport (V3.11) | ✅ Done — `RTCDataChannel` with `MultiTransportFallback([webrtc, http])` | [`docs/webrtc.md`](./docs/webrtc.md) |
|
||||||
|
| Key Transparency (V3.12) | ✅ Done — opt-in Merkle log, signed STH, witness gossip | [`docs/key-transparency.md`](./docs/key-transparency.md) |
|
||||||
|
| External crypto review | 🟡 Bundle ready — review window open after tag | [`docs/audit/REVIEW-BUNDLE.md`](./docs/audit/REVIEW-BUNDLE.md) |
|
||||||
|
| Soak (≥ 2 weeks under load) | 🟡 Harness shipped — operator runs it | [`scripts/soak.ts`](./scripts/soak.ts) (`bun run soak --hours 336`) |
|
||||||
|
| Voice / Video / Broadcast | 🔜 V5.0 — built on top of frozen 4.0 stack | [`docs/V5.0.md`](./docs/V5.0.md) |
|
||||||
|
|
||||||
## What you get
|
## What you get
|
||||||
|
|
||||||
|
**Protocol core**
|
||||||
- **X3DH** initial key agreement (works asynchronously via prekey bundles)
|
- **X3DH** initial key agreement (works asynchronously via prekey bundles)
|
||||||
- **Double Ratchet** for per-message forward secrecy and post-compromise security
|
- **Double Ratchet** for per-message forward secrecy and post-compromise security
|
||||||
- **Self-authenticated prekey server** (Hono, Docker-ready) with rate limiting, metrics, health checks
|
- **Sender keys** for group ratchet (1:N broadcast key derivation)
|
||||||
- **Persistent storage backends**: SQLite (zero-config) and PostgreSQL (Drizzle)
|
|
||||||
- **Identity rotation** with grace period for old sessions
|
- **Identity rotation** with grace period for old sessions
|
||||||
- **Safety numbers** (Signal-style fingerprints) for out-of-band verification
|
- **Safety numbers** (Signal-style fingerprints) for out-of-band verification
|
||||||
- **Constant-time comparisons** and **memory zeroization** for hardened operation
|
- **Constant-time comparisons** and **memory zeroization** for hardened operation
|
||||||
- **Binary wire format** that's significantly smaller than JSON
|
- **Binary wire format** (`@shade/proto`) — significantly smaller than JSON
|
||||||
|
|
||||||
|
**Storage**
|
||||||
|
- **Persistent backends**: SQLite (zero-config, `bun:sqlite`) and PostgreSQL (Drizzle, FOR UPDATE SKIP LOCKED)
|
||||||
- **Crash-safe** — sessions survive container restarts, power outages, SIGKILL
|
- **Crash-safe** — sessions survive container restarts, power outages, SIGKILL
|
||||||
- **Live observability** — bundled dashboard SPA + embeddable React widgets to see what's happening between every step
|
- **At-rest encryption (V3.2, opt-in)** — AES-256-GCM under per-(table,column) DEKs; key sources: passphrase (scrypt), OS keychain (`@shade/keychain`), or app-injected. Online re-key, no downtime.
|
||||||
- **E2EE file transfers** — multi-lane chunked uploads/downloads with resume, integrity checks, and HTTP/WS fallback (`@shade/streams` + `@shade/transfer`)
|
|
||||||
- **WebRTC P2P transport (V3.11)** — opt-in `RTCDataChannel` upload path with public-STUN defaults, TURN-relay support, glare-safe peer pool, and automatic `MultiTransportFallback` back to HTTP when NAT traversal fails (`@shade/transport-webrtc`, [docs/webrtc.md](./docs/webrtc.md))
|
**Servers**
|
||||||
- **Web Workers crypto** — AEAD, HKDF, HMAC, X25519, Ed25519 and per-lane stream state run in a dedicated worker. 100 MB+ uploads stay smooth without frame drops, lane keys never cross the thread boundary (`@shade/crypto-web/worker`, [docs/web-workers.md](./docs/web-workers.md))
|
- **Self-authenticated prekey server** (`@shade/server`, Hono, Docker-ready) with rate limiting, metrics, health checks
|
||||||
- **E2EE filesystem RPC** — typed `list/stat/mkdir/delete/move/read/write/getThumbnail` + custom ops between peers, with rate-limit, retention, and fingerprint-gate hooks (`@shade/files`)
|
- **Async store-and-forward relay** (`@shade/inbox-server`) — TTL-bound ciphertext blobs, signed PUT/FETCH/ACK, idempotent on `(address, msgId)`, per-recipient quota
|
||||||
- **Async store-and-forward** — deliver to offline recipients via a relay that holds ciphertext-only blobs with TTL, idempotent PUT, signed fetch/ack, and an `onMessageQueued` push-trigger hook (`@shade/inbox` + `@shade/inbox-server`)
|
- **Bridge transports** (`@shade/transport-bridge`) — WS → SSE → long-poll fallback chain for clients that can't keep a WebSocket open. Same `IncomingMessage` shape across all three.
|
||||||
- **Social key recovery** — Shamir-split your identity to `n` guardians; any threshold-many `k` together restore it on a new device. No centralized recovery agent; OOB-fingerprint gate on every guardian release; AES-GCM authenticates the reconstruction (`@shade/recovery` + `<RecoverySetup />` / `<RecoveryRequest />` / `<RecoveryApprove />` widgets, [docs/recovery.md](./docs/recovery.md))
|
- **Standalone container** — one image bundles prekey + inbox + bridge + transfer + KT + observer
|
||||||
- **Key Transparency (V3.12)** — opt-in append-only Merkle log over the prekey server. Every `register` / `delete` becomes a signed leaf; every bundle-fetch carries an inclusion proof; an Ed25519-signed Tree Head ties roots to a fixed `log_id`. A `LightWitness` cross-checks STHs across clients so a malicious server that splits its view or rewrites history is caught (`@shade/key-transparency`, [docs/key-transparency.md](./docs/key-transparency.md))
|
|
||||||
|
**Trust UX**
|
||||||
|
- **Fingerprint gates (V3.3)** — `Shade.beforeFirstLargeFile(threshold, handler)`, `beforeBackupImport`, `beforeNewDeviceTrust`. Gates raise `FingerprintNotVerifiedError` on the operations that matter, default-warn TOFU otherwise.
|
||||||
|
- **`<FingerprintCompare />` / `<FingerprintGate />`** widgets for the matching UI side.
|
||||||
|
|
||||||
|
**File transfer & filesystem**
|
||||||
|
- **E2EE file transfers** (`@shade/streams` + `@shade/transfer`) — multi-lane chunked uploads/downloads with resume, integrity checks (per-lane sha256 + overall sha256), HTTP/WS fallback, `MultiTransportFallback` for N-ary demotion
|
||||||
|
- **WebRTC P2P transport (V3.11, opt-in)** — `RTCDataChannel` chunk path with public-STUN defaults, TURN-relay support, glare-safe peer pool, automatic fallback to HTTP when NAT traversal fails (`@shade/transport-webrtc`)
|
||||||
|
- **Web Workers crypto (V3.8, opt-in)** — AEAD, HKDF, HMAC, X25519, Ed25519 and per-lane stream state run in a dedicated worker. 100 MB+ uploads stay smooth without frame drops; lane keys never cross the thread boundary (`@shade/crypto-web/worker`)
|
||||||
|
- **E2EE filesystem RPC** (`@shade/files`) — typed `list/stat/mkdir/delete/move/read/write/getThumbnail` + custom ops, with rate-limit, retention, fingerprint-gate, and metrics hooks. React hooks under `@shade/files/react`.
|
||||||
|
|
||||||
|
**Recovery**
|
||||||
|
- **Social key recovery (V3.10)** — Shamir-split your identity to `n` guardians; any threshold-many `k` together restore it on a new device. No centralized recovery agent; OOB-fingerprint gate per guardian release; AES-GCM-authenticated reconstruction (`@shade/recovery` + `<RecoverySetup />` / `<RecoveryRequest />` / `<RecoveryApprove />`)
|
||||||
|
|
||||||
|
**Verifiable distribution**
|
||||||
|
- **Key Transparency (V3.12, opt-in)** — append-only Merkle log over the prekey server. Every `register` / `delete` becomes a signed leaf; every bundle-fetch carries an inclusion proof; an Ed25519-signed Tree Head ties roots to a fixed `log_id`. A `LightWitness` cross-checks STHs across clients so a malicious server that splits its view or rewrites history is caught (`@shade/key-transparency`).
|
||||||
|
|
||||||
|
**Observability**
|
||||||
|
- **Live observability** — OpenTelemetry-shaped events, bundled dashboard SPA + embeddable React widgets to see what's happening between every step (`@shade/observability` + `@shade/observer` + `@shade/dashboard`)
|
||||||
|
|
||||||
|
**Tooling**
|
||||||
|
- **CLI** — `shade init` scaffolder, `shade migrate-storage` (V3.2), `shade rotate-storage-key`, `shade fingerprint`, `shade rotate`, `shade peer`, `shade dashboard`, `shade doctor`, `shade backup` (`@shade/cli`)
|
||||||
|
- **Soak harness** — `bun run soak --hours 336` for the 2-week GA-stable window (`scripts/soak.ts`)
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
@@ -84,7 +129,38 @@ const plaintext = await shade.receive('alice@example.com', incomingEnvelope);
|
|||||||
console.log(await shade.fingerprint);
|
console.log(await shade.fingerprint);
|
||||||
```
|
```
|
||||||
|
|
||||||
Need to ship a file or expose a filesystem to a peer? `Shade.files` is the high-level entrypoint:
|
### Opt-in surfaces (V3.x → 4.0 GA)
|
||||||
|
|
||||||
|
All of these are off by default. Wire them only where you need them.
|
||||||
|
|
||||||
|
```ts
|
||||||
|
// V3.3 — Fingerprint gates: enforce verification on the operations that matter
|
||||||
|
shade.beforeFirstLargeFile(10 * 1024 * 1024, async ({ peer, fingerprint }) => {
|
||||||
|
return await ui.confirmSafetyNumberMatches(peer, fingerprint);
|
||||||
|
});
|
||||||
|
shade.beforeBackupImport(async ({ embeddedFingerprint }) => { /* ... */ });
|
||||||
|
shade.beforeNewDeviceTrust(async ({ peer, oldFp, newFp }) => { /* ... */ });
|
||||||
|
|
||||||
|
// V3.8 — Web Workers crypto: opt-in, lane keys stay off the main thread
|
||||||
|
shade.configureWorkerCrypto({
|
||||||
|
workerUrl: new URL('@shade/crypto-web/worker', import.meta.url),
|
||||||
|
});
|
||||||
|
|
||||||
|
// V3.11 — WebRTC P2P transport: file transfers ride RTCDataChannel where NAT allows
|
||||||
|
import { nativeRtcFactory } from '@shade/transport-webrtc';
|
||||||
|
shade.configureWebRTC({
|
||||||
|
factory: nativeRtcFactory(),
|
||||||
|
iceServers: [{ urls: 'stun:stun.l.google.com:19302' }],
|
||||||
|
});
|
||||||
|
|
||||||
|
// V3.12 — Key Transparency: detect server-side bundle swaps
|
||||||
|
const shade = await createShade({
|
||||||
|
prekeyServer: 'https://shade.example.com',
|
||||||
|
keyTransparency: { mode: 'observe-strict', logPublicKey: PINNED_KEY_BYTES_32 },
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### Files RPC (`@shade/files`)
|
||||||
|
|
||||||
```ts
|
```ts
|
||||||
// Server side — Bob exposes a virtual filesystem
|
// Server side — Bob exposes a virtual filesystem
|
||||||
@@ -97,13 +173,13 @@ const stop = await shade.files.serve({
|
|||||||
|
|
||||||
// Client side — Alice consumes Bob's filesystem
|
// Client side — Alice consumes Bob's filesystem
|
||||||
const fs = await shade.files.client('bob');
|
const fs = await shade.files.client('bob');
|
||||||
await fs.write('/photos/cover.png', new Uint8Array(...)); // auto inline/streams
|
await fs.write('/photos/cover.png', new Uint8Array([/* ... */])); // auto inline/streams
|
||||||
const result = await fs.read('/photos/cover.png');
|
const result = await fs.read('/photos/cover.png');
|
||||||
```
|
```
|
||||||
|
|
||||||
Files ≤ 256 KiB ride inline in the RPC envelope; larger files automatically promote to multi-lane `@shade/transfer` streams with sha256 integrity. See [`docs/files.md`](./docs/files.md) for the full API.
|
Files ≤ 256 KiB ride inline in the RPC envelope; larger files automatically promote to multi-lane `@shade/transfer` streams with sha256 integrity. See [`docs/files.md`](./docs/files.md) for the full API.
|
||||||
|
|
||||||
Or use the lower-level packages directly if you need full control:
|
### Lower-level access
|
||||||
|
|
||||||
```ts
|
```ts
|
||||||
import { ShadeSessionManager } from '@shade/core';
|
import { ShadeSessionManager } from '@shade/core';
|
||||||
@@ -117,20 +193,43 @@ const manager = new ShadeSessionManager(
|
|||||||
await manager.initialize();
|
await manager.initialize();
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### At-rest encryption (V3.2)
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { KeyManager, EncryptedSQLiteStorage } from '@shade/storage-encrypted';
|
||||||
|
|
||||||
|
const km = await KeyManager.open({
|
||||||
|
kind: 'passphrase',
|
||||||
|
passphrase: process.env.SHADE_STORAGE_PASSPHRASE!,
|
||||||
|
salt: loadSaltFromDisk(),
|
||||||
|
});
|
||||||
|
const storage = await EncryptedSQLiteStorage.open({
|
||||||
|
dbPath: '/data/shade-client.db',
|
||||||
|
keyManager: km,
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
To migrate an existing 0.3.x SQLite DB in place:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
shade migrate-storage \
|
||||||
|
--key-source passphrase \
|
||||||
|
--passphrase "$SHADE_STORAGE_PASSPHRASE" \
|
||||||
|
--salt-file /data/shade-client.db.salt
|
||||||
|
```
|
||||||
|
|
||||||
## Architecture — keys vs. payloads
|
## Architecture — keys vs. payloads
|
||||||
|
|
||||||
Shade splits the network into two planes. The **prekey server** only
|
Shade splits the network into a **public-key plane** (the prekey
|
||||||
sees public keys; **everything else** rides the encrypted Double
|
server) and an **encrypted plane** (everything else). The prekey
|
||||||
Ratchet between peers. If you remember nothing else from this README,
|
server only sees public key material. If you remember nothing else
|
||||||
remember this picture:
|
from this README, remember this picture:
|
||||||
|
|
||||||
```
|
```
|
||||||
Shade Prekey Server (Hono, public keys only)
|
Shade Prekey Container (Hono — public keys only)
|
||||||
│
|
│
|
||||||
POST /v1/keys/register (signed)
|
/v1/keys/* /v1/inbox/* /v1/bridge/* /v1/transfer/*
|
||||||
GET /v1/keys/bundle/:address
|
/v1/kt/* /metrics /healthz /ready
|
||||||
POST /v1/keys/replenish (signed)
|
|
||||||
DELETE /v1/keys/:address (signed)
|
|
||||||
│
|
│
|
||||||
┌──────────────────┴──────────────────┐
|
┌──────────────────┴──────────────────┐
|
||||||
│ │
|
│ │
|
||||||
@@ -142,11 +241,20 @@ remember this picture:
|
|||||||
│◄── Double Ratchet messages ────────►│ ← end-to-end,
|
│◄── Double Ratchet messages ────────►│ ← end-to-end,
|
||||||
│ (ratchet 0x02 / chunks 0x11) │ never on the
|
│ (ratchet 0x02 / chunks 0x11) │ never on the
|
||||||
│ │ prekey server
|
│ │ prekey server
|
||||||
│◄── @shade/transfer chunks ─────────►│
|
│
|
||||||
│ POST /v1/transfer/:id/chunk │ ← peer-to-peer
|
│◄── @shade/transfer chunks ─────────►│ ← peer-to-peer
|
||||||
│ GET /v1/transfer/:id/state │ HTTP, opaque
|
│ POST /v1/transfer/:id/chunk │ HTTP, opaque
|
||||||
│ │ ciphertext
|
│ GET /v1/transfer/:id/state │ ciphertext
|
||||||
|
│
|
||||||
|
│◄── @shade/inbox blobs (offline) ───►│ ← TTL-bound
|
||||||
|
│ POST /v1/inbox/:address │ ciphertext-only
|
||||||
|
│ POST /v1/inbox/:address/fetch │ relay
|
||||||
|
│
|
||||||
|
│◄── @shade/transport-webrtc ────────►│ ← optional P2P
|
||||||
|
│ RTCDataChannel `shade-transfer/v1` │ `MultiTransportFallback`
|
||||||
|
│ │ auto-demotes to HTTP
|
||||||
SQLiteStorage / PostgresStorage SQLiteStorage / PostgresStorage
|
SQLiteStorage / PostgresStorage SQLiteStorage / PostgresStorage
|
||||||
|
+ EncryptedSQLiteStorage (V3.2) + EncryptedSQLiteStorage (V3.2)
|
||||||
(private keys + sessions) (private keys + sessions)
|
(private keys + sessions) (private keys + sessions)
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -155,17 +263,23 @@ remember this picture:
|
|||||||
- Identity public keys (Ed25519 + X25519)
|
- Identity public keys (Ed25519 + X25519)
|
||||||
- Signed prekeys + one-time prekey bundles
|
- Signed prekeys + one-time prekey bundles
|
||||||
- Registration / replenish / delete writes, all Ed25519-signed
|
- Registration / replenish / delete writes, all Ed25519-signed
|
||||||
- Operator-only metrics, `/health`, and the optional observer
|
- (V3.6) Inbox ciphertext blobs with TTL — same container, separate
|
||||||
dashboard
|
routes; the relay only sees `address || msgId || ciphertext-bytes`
|
||||||
|
- (V3.7) Bridge transports (SSE / long-poll / WS) — also delivered by
|
||||||
|
the same Hono app for clients that can't hold a WebSocket
|
||||||
|
- (V3.12, opt-in) KT inclusion proofs + signed tree heads on
|
||||||
|
`/v1/kt/*` — verifiable distribution
|
||||||
|
- Operator-only metrics and the optional observer dashboard
|
||||||
|
|
||||||
### What does **not** go via the prekey server
|
### What does **not** go via the prekey server
|
||||||
|
|
||||||
- **Message plaintext, ever.** Encrypted ratchet envelopes flow peer-
|
- **Message plaintext, ever.** Encrypted ratchet envelopes flow peer-
|
||||||
to-peer over whatever transport you choose (HTTP, WebSocket, your
|
to-peer over whatever transport you choose (HTTP, WebSocket, your
|
||||||
own broker).
|
own broker, or the inbox relay above — which carries ciphertext only).
|
||||||
- **File chunks.** `@shade/transfer` POSTs ciphertext directly to the
|
- **File chunks.** `@shade/transfer` POSTs ciphertext directly to the
|
||||||
receiver's `/v1/transfer/:streamId/chunk` route — the prekey server
|
receiver's `/v1/transfer/:streamId/chunk` route — the prekey server
|
||||||
is not involved.
|
is not involved. With V3.11 + `configureWebRTC()`, chunks ride
|
||||||
|
`RTCDataChannel` peer-to-peer; the relay is bypassed entirely.
|
||||||
- **Identity private keys.** They never leave the device's storage.
|
- **Identity private keys.** They never leave the device's storage.
|
||||||
- **Filesystem RPC.** `@shade/files` rides the Double Ratchet for
|
- **Filesystem RPC.** `@shade/files` rides the Double Ratchet for
|
||||||
control + small payloads, then promotes to direct `@shade/transfer`
|
control + small payloads, then promotes to direct `@shade/transfer`
|
||||||
@@ -176,6 +290,8 @@ remember this picture:
|
|||||||
The prekey server is metadata-bearing (see `THREAT-MODEL.md § 2`):
|
The prekey server is metadata-bearing (see `THREAT-MODEL.md § 2`):
|
||||||
it sees who registers, who fetches whose bundle, and when. It does
|
it sees who registers, who fetches whose bundle, and when. It does
|
||||||
**not** see message contents, transfer contents, or session state.
|
**not** see message contents, transfer contents, or session state.
|
||||||
|
**V3.12 Key Transparency** (opt-in) makes its bundle distribution
|
||||||
|
*verifiable* so a malicious server that swaps a bundle is caught.
|
||||||
|
|
||||||
For the full threat model and mitigations, read
|
For the full threat model and mitigations, read
|
||||||
[THREAT-MODEL.md](./THREAT-MODEL.md). For deployment-time guarantees,
|
[THREAT-MODEL.md](./THREAT-MODEL.md). For deployment-time guarantees,
|
||||||
@@ -183,53 +299,84 @@ read [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md).
|
|||||||
|
|
||||||
## Packages
|
## Packages
|
||||||
|
|
||||||
|
All packages publish in lockstep at `4.0.0`.
|
||||||
|
|
||||||
| Package | Purpose |
|
| Package | Purpose |
|
||||||
|---------|---------|
|
|---------|---------|
|
||||||
| `@shade/core` | Protocol logic (X3DH, Double Ratchet, session manager, errors, events) |
|
| `@shade/core` | Protocol logic (X3DH, Double Ratchet, sender keys, session manager, errors, events) |
|
||||||
|
| `@shade/proto` | Compact binary wire format (`0x01` PreKeyMessage, `0x02` RatchetMessage, `0x11` StreamChunk) |
|
||||||
| `@shade/crypto-web` | SubtleCrypto + @noble/curves provider, in-memory storage. Includes the V3.8 Web Workers entrypoint (`@shade/crypto-web/worker`) — drop-in `WorkerCryptoProvider` plus `createEncryptStream` / `createDecryptStream` TransformStream factories |
|
| `@shade/crypto-web` | SubtleCrypto + @noble/curves provider, in-memory storage. Includes the V3.8 Web Workers entrypoint (`@shade/crypto-web/worker`) — drop-in `WorkerCryptoProvider` plus `createEncryptStream` / `createDecryptStream` TransformStream factories |
|
||||||
| `@shade/storage-sqlite` | Persistent SQLite storage (zero-config, bun:sqlite) |
|
| `@shade/observability` | OpenTelemetry-shaped event bus consumed by `@shade/observer`, server hooks, and the dashboard |
|
||||||
| `@shade/storage-postgres` | PostgreSQL storage with Drizzle for shared databases |
|
| `@shade/keychain` | OS keychain bindings (libsecret / Keychain / Credential Manager) used by `@shade/storage-encrypted` and the CLI |
|
||||||
| `@shade/server` | Prekey server (Hono routes, auth, rate limit, health, metrics) |
|
| `@shade/key-transparency` | Key Transparency (V3.12) — RFC 6962-style append-only Merkle log, address-index commitment, signed tree heads, and a `LightWitness` for split-view detection. Opt-in on both server and client. |
|
||||||
| `@shade/transport` | HTTP + WebSocket transport wrappers with auto-encryption |
|
| `@shade/storage-sqlite` | Persistent SQLite storage (zero-config, `bun:sqlite`); also ships `SqliteInboxStore` |
|
||||||
|
| `@shade/storage-postgres` | PostgreSQL storage with Drizzle for shared databases; also ships `PostgresInboxStore` + `PostgresKTLogStore` |
|
||||||
|
| `@shade/storage-encrypted` | At-rest encryption (V3.2) — `EncryptedSQLiteStorage` / `EncryptedPostgresStorage`, `KeyManager`, online re-key |
|
||||||
|
| `@shade/streams` | Multi-lane chunk encryption — HKDF-derived per-lane keys, deterministic AES-GCM nonces, streaming SHA-256, file metadata + thumbnails (V3.9) |
|
||||||
|
| `@shade/transport` | HTTP + WebSocket transport wrappers with auto-encryption; KT-verifying `fetchBundleVerified` |
|
||||||
| `@shade/transport-bridge` | WS → SSE → long-poll fallback chain (V3.7) — single `IncomingMessage` shape across transports for clients that can't keep a WebSocket open |
|
| `@shade/transport-bridge` | WS → SSE → long-poll fallback chain (V3.7) — single `IncomingMessage` shape across transports for clients that can't keep a WebSocket open |
|
||||||
| `@shade/transport-webrtc` | V3.11 P2P chunk transport via `RTCDataChannel`. Plugs into `@shade/transfer` as an `ITransferTransport`; signaling rides Shade's own ratchet. Memory factory + native (`globalThis.RTCPeerConnection`) factory included; `MultiTransportFallback([webrtc, http])` wired automatically when `shade.configureWebRTC()` is called. |
|
| `@shade/transport-webrtc` | V3.11 P2P chunk transport via `RTCDataChannel`. Plugs into `@shade/transfer` as an `ITransferTransport`; signaling rides Shade's own ratchet. Memory factory + native (`globalThis.RTCPeerConnection`) factory included; `MultiTransportFallback([webrtc, http])` wired automatically when `shade.configureWebRTC()` is called. |
|
||||||
| `@shade/proto` | Compact binary wire format (smaller than JSON) |
|
| `@shade/server` | Prekey server (Hono routes, auth, rate limit, health, metrics). `createPrekeyServerWithKT(...)` opts into V3.12 KT mode |
|
||||||
| `@shade/streams` | Multi-lane chunk encryption — HKDF-derived per-lane keys, deterministic AES-GCM nonces, streaming SHA-256 |
|
|
||||||
| `@shade/transfer` | Transfer engine on top of streams: parallel lanes, resume, HTTP + WS transport with auto-fallback, integrity verification |
|
|
||||||
| `@shade/files` | Typed E2EE filesystem RPC — list/stat/mkdir/delete/move/read/write/getThumbnail + custom ops, auto inline/streams routing, production hooks (rate limit, retention, fingerprint gate, metrics), React hooks |
|
|
||||||
| `@shade/recovery` | Social key recovery (V3.10) — Shamir-split identity to `n` guardians; threshold-many `k` reconstruct on a new device. AES-GCM-authenticated reconstruction; OOB-fingerprint gate per guardian release |
|
|
||||||
| `@shade/key-transparency` | Key Transparency (V3.12) — RFC 6962-style append-only Merkle log, address-index commitment, signed tree heads, and a `LightWitness` for split-view detection. Opt-in on both server and client. See [docs/key-transparency.md](./docs/key-transparency.md) |
|
|
||||||
| `@shade/inbox-server` | Async store-and-forward relay (V3.6) — Hono routes, signed PUT/FETCH/DELETE, per-recipient TTL + quota, idempotent on `(address, msgId)`. Bundles into the same standalone container as the prekey server |
|
| `@shade/inbox-server` | Async store-and-forward relay (V3.6) — Hono routes, signed PUT/FETCH/DELETE, per-recipient TTL + quota, idempotent on `(address, msgId)`. Bundles into the same standalone container as the prekey server |
|
||||||
| `@shade/inbox` | Inbox client + durable outgoing queue + receive cursor + push-trigger hook (`onMessageQueued`); composes on top of `Shade.send`/`Shade.receive` for offline-recipient delivery |
|
| `@shade/inbox` | Inbox client + durable outgoing queue + receive cursor + push-trigger hook (`onMessageQueued`); composes on top of `Shade.send`/`Shade.receive` for offline-recipient delivery |
|
||||||
|
| `@shade/transfer` | Transfer engine on top of streams: parallel lanes, resume, HTTP + WS transport with auto-fallback, `MultiTransportFallback` (N-ary demotion), integrity verification |
|
||||||
|
| `@shade/files` | Typed E2EE filesystem RPC — list/stat/mkdir/delete/move/read/write/getThumbnail + custom ops, auto inline/streams routing, production hooks (rate limit, retention, fingerprint gate, metrics), React hooks under `@shade/files/react` |
|
||||||
|
| `@shade/recovery` | Social key recovery (V3.10) — Shamir-split identity to `n` guardians; threshold-many `k` reconstruct on a new device. AES-GCM-authenticated reconstruction; OOB-fingerprint gate per guardian release |
|
||||||
| `@shade/observer` | Live debugger backend (snapshot, SSE, dashboard) — see [README](./packages/shade-observer/README.md) |
|
| `@shade/observer` | Live debugger backend (snapshot, SSE, dashboard) — see [README](./packages/shade-observer/README.md) |
|
||||||
| `@shade/widgets` | Embeddable React widgets including transfer uploader/downloader — see [README](./packages/shade-widgets/README.md) |
|
|
||||||
| `@shade/dashboard` | Standalone dashboard SPA bundled into the observer |
|
| `@shade/dashboard` | Standalone dashboard SPA bundled into the observer |
|
||||||
| `@shade/sdk` | High-level wrapper with `createShade()` one-liner, auto-publish, auto-establish, auto-replenish, `Shade.files` namespace |
|
| `@shade/sdk` | High-level wrapper with `createShade()` one-liner, auto-publish, auto-establish, auto-replenish, `Shade.files` namespace, fingerprint gates, KT integration, WebRTC opt-in |
|
||||||
| `@shade/cli` | `shade init` scaffolder + utilities (fingerprint, rotate, peer, dashboard, doctor) |
|
| `@shade/widgets` | Embeddable React widgets — fingerprint compare/gate, recovery setup/request/approve, transfer uploader/downloader, observer panels |
|
||||||
|
| `@shade/cli` | `shade init` scaffolder + utilities (fingerprint, rotate, peer, dashboard, doctor, backup, migrate-storage, rotate-storage-key) |
|
||||||
|
|
||||||
## Shade as a modular toolkit
|
## Shade as a modular toolkit
|
||||||
|
|
||||||
Shade is split into packages so each project can depend on **only what it needs**—encrypted messaging, file transfer, prekey hosting, or lower-level building blocks. You do not need one giant stack for every use case.
|
Shade is split into packages so each project can depend on **only what it needs**—encrypted messaging, file transfer, prekey hosting, social recovery, KT verification, or lower-level building blocks. You do not need one giant stack for every use case.
|
||||||
|
|
||||||
For a **plain-language map** (which packages to add, what the prekey server does vs your own wiring, and where to start in code), see **[docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md)**.
|
For a **plain-language map** (which packages to add, what the prekey server does vs your own wiring, and where to start in code), see **[docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md)**.
|
||||||
|
|
||||||
## Publishing
|
## Publishing
|
||||||
|
|
||||||
All packages publish to a self-hosted Gitea npm registry on `gt.zyon.no`.
|
All packages publish to a self-hosted Gitea npm registry on `gt.zyon.no`. The Docker image of the standalone container ships at `gt.zyon.no/stian/shade-prekey:<tag>`.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Bump all packages in lockstep
|
# Bump all packages in lockstep
|
||||||
bun run version 1.1.0
|
bun run version 4.0.1
|
||||||
|
|
||||||
# Dry-run (pack all tarballs without publishing)
|
# Dry-run (pack all tarballs without publishing) — no token required
|
||||||
bun run publish:dry
|
bun run publish:dry
|
||||||
|
|
||||||
# Real publish (requires GITEA_TOKEN env var)
|
# Real publish — interactive (prompts for GITEA_TOKEN, checks
|
||||||
|
# registry for conflicts, publishes via scripts/publish-all.ts)
|
||||||
bun run publish:all
|
bun run publish:all
|
||||||
|
|
||||||
# Or via CI: push a git tag v1.1.0 and .gitea/workflows/publish.yml runs
|
# Build + push the standalone Docker image
|
||||||
|
bun run scripts/build-docker.ts -- --tag 4.0.1 --push
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The interactive `scripts/publish-shade.sh` is the human entrypoint;
|
||||||
|
`scripts/publish-all.ts` is the headless variant used by CI and
|
||||||
|
`publish:dry`. They share a single `PACKAGES` list (24 entries at
|
||||||
|
4.0.0) so the two flows can never drift.
|
||||||
|
|
||||||
|
## Soak / GA-stable
|
||||||
|
|
||||||
|
Before tagging `4.0.0` as `latest` and recommending production
|
||||||
|
upgrades, run the combined soak harness for ≥ 2 weeks:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Full GA window (V4.0 §Soak): 14 days × 24 hours
|
||||||
|
bun run soak --hours 336
|
||||||
|
|
||||||
|
# Smoke (~3 minutes — ratchet ping-pong, integrity check)
|
||||||
|
bun run soak:smoke
|
||||||
|
```
|
||||||
|
|
||||||
|
The harness fans out N concurrent ratchet pairs, ping-pongs at
|
||||||
|
~400 ops/sec/pair, and reports cumulative counters every minute.
|
||||||
|
Any exception in any pair is captured and re-raised at shutdown so
|
||||||
|
silent failures cannot hide. Wrap it in `systemd-run --user`,
|
||||||
|
`nohup`, or a Gitea scheduled job for the actual 2-week window.
|
||||||
|
|
||||||
## Security properties
|
## Security properties
|
||||||
|
|
||||||
| Property | Description |
|
| Property | Description |
|
||||||
@@ -242,27 +389,57 @@ bun run publish:all
|
|||||||
| **Memory zeroization** | Key material is zeroed after use (best-effort in JS) |
|
| **Memory zeroization** | Key material is zeroed after use (best-effort in JS) |
|
||||||
| **Identity verification** | Safety numbers (60 digits) for out-of-band comparison |
|
| **Identity verification** | Safety numbers (60 digits) for out-of-band comparison |
|
||||||
| **Identity rotation** | 7-day grace period for old sessions during rotation |
|
| **Identity rotation** | 7-day grace period for old sessions during rotation |
|
||||||
|
| **At-rest encryption** *(V3.2, opt-in)* | AES-256-GCM under per-(table, column) DEKs; AAD binds `(table, column, pk)`; passphrase / OS keychain / app-injected master key; online re-key |
|
||||||
|
| **Fingerprint gates** *(V3.3)* | `Shade.beforeFirstLargeFile / beforeBackupImport / beforeNewDeviceTrust` raise `FingerprintNotVerifiedError` on the operations that matter; defaults to TOFU + warning when no gate is registered |
|
||||||
|
| **Async store-and-forward** *(V3.6)* | Relay only sees `address || msgId || ciphertext`; idempotent PUT; signed FETCH/ACK; TTL-bounded |
|
||||||
|
| **Web-Worker key isolation** *(V3.8)* | Lane keys, identity keys, and ratchet chain keys live inside a dedicated worker; main thread only ferries plaintext via transferable buffers; idle terminate releases worker memory |
|
||||||
|
| **Social key recovery** *(V3.10)* | Shamir over GF(2^8); AEAD-authenticated reconstruction (forged shares fail); guardian-side fingerprint gate before share release |
|
||||||
|
| **WebRTC P2P transport** *(V3.11)* | Same Double Ratchet authenticates SDP/ICE signaling; chunk frames AEAD-bound to `streamId/laneId/seq`; deterministic glare resolution; `MultiTransportFallback` auto-demotes to HTTP |
|
||||||
| **Key Transparency** *(V3.12, opt-in)* | Append-only Merkle log + signed tree heads + witness gossip — split-view and history-rewrite are detected by clients |
|
| **Key Transparency** *(V3.12, opt-in)* | Append-only Merkle log + signed tree heads + witness gossip — split-view and history-rewrite are detected by clients |
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
|
**Operator + integrator**
|
||||||
|
|
||||||
- [docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md) — **Modular toolkit**: pick packages by scenario (messages, files, browser, ops)
|
- [docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md) — **Modular toolkit**: pick packages by scenario (messages, files, browser, ops)
|
||||||
- [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md) — Pre-flight gates for going to production
|
- [docs/PRODUCTION-CHECKLIST.md](./docs/PRODUCTION-CHECKLIST.md) — Pre-flight gates for going to production
|
||||||
|
- [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md) — Full deployment guide (Docker, env vars, PostgreSQL, backup, Dokploy)
|
||||||
|
- [docs/ROADMAP.md](./docs/ROADMAP.md) — V3.x → 4.0 GA → V5.0 trajectory
|
||||||
|
|
||||||
|
**Per-surface deep-dives**
|
||||||
|
|
||||||
- [docs/files.md](./docs/files.md) — `@shade/files` API + design (filesystem RPC, custom ops, hooks, React)
|
- [docs/files.md](./docs/files.md) — `@shade/files` API + design (filesystem RPC, custom ops, hooks, React)
|
||||||
- [docs/recovery.md](./docs/recovery.md) — `@shade/recovery` social key recovery (V3.10): Shamir setup, guardian-side gates, threshold tuning
|
|
||||||
- [docs/streams.md](./docs/streams.md) — `@shade/streams` + `@shade/transfer` deep dive (incl. hardening + retention)
|
- [docs/streams.md](./docs/streams.md) — `@shade/streams` + `@shade/transfer` deep dive (incl. hardening + retention)
|
||||||
- [docs/inbox.md](./docs/inbox.md) — `@shade/inbox` + `@shade/inbox-server` async store-and-forward relay (V3.6)
|
- [docs/inbox.md](./docs/inbox.md) — `@shade/inbox` + `@shade/inbox-server` async store-and-forward relay (V3.6)
|
||||||
- [docs/transport.md](./docs/transport.md) — `@shade/transport-bridge` SSE / long-poll / WS bridge layer (V3.7)
|
- [docs/transport.md](./docs/transport.md) — `@shade/transport-bridge` SSE / long-poll / WS bridge layer (V3.7)
|
||||||
|
- [docs/web-workers.md](./docs/web-workers.md) — V3.8 Web Workers crypto: setup, bundler recipes (Vite/Webpack/Rollup), Safari notes, lifecycle, threat-model
|
||||||
|
- [docs/recovery.md](./docs/recovery.md) — `@shade/recovery` social key recovery (V3.10): Shamir setup, guardian-side gates, threshold tuning
|
||||||
- [docs/webrtc.md](./docs/webrtc.md) — `@shade/transport-webrtc` P2P transport (V3.11): NAT-traversal, TURN config, glare resolution, wire format, multi-fallback wiring
|
- [docs/webrtc.md](./docs/webrtc.md) — `@shade/transport-webrtc` P2P transport (V3.11): NAT-traversal, TURN config, glare resolution, wire format, multi-fallback wiring
|
||||||
- [docs/key-transparency.md](./docs/key-transparency.md) — `@shade/key-transparency` (V3.12): operator + client onboarding, witness role, recovery procedures
|
- [docs/key-transparency.md](./docs/key-transparency.md) — `@shade/key-transparency` (V3.12): operator + client onboarding, witness role, recovery procedures
|
||||||
- [docs/V3.12-DESIGN.md](./docs/V3.12-DESIGN.md) — V3.12 design notat (threat model, RFC 6962 vs CONIKS choice, freshness model, open-questions resolution)
|
- [docs/storage-encryption.md](./docs/storage-encryption.md) — V3.2 at-rest encryption: design, key sources, rotation
|
||||||
- [docs/web-workers.md](./docs/web-workers.md) — V3.8 Web Workers crypto: setup, bundler recipes (Vite/Webpack/Rollup), Safari notes, lifecycle, threat-model
|
- [docs/trust-ux.md](./docs/trust-ux.md) — V3.3 fingerprint gates: when each fires, handler patterns, widget integration
|
||||||
|
- [docs/observability.md](./docs/observability.md) — V3.4 event bus + dashboard
|
||||||
|
- [docs/cross-platform.md](./docs/cross-platform.md) — V3.5 Android parity + cross-platform vector regime
|
||||||
|
|
||||||
|
**Threat model + audit**
|
||||||
|
|
||||||
- [SECURITY.md](./SECURITY.md) — Reporting vulnerabilities, security policy, threat-/test-matrix
|
- [SECURITY.md](./SECURITY.md) — Reporting vulnerabilities, security policy, threat-/test-matrix
|
||||||
- [THREAT-MODEL.md](./THREAT-MODEL.md) — Honest threat model and assumptions
|
- [THREAT-MODEL.md](./THREAT-MODEL.md) — Honest threat model and assumptions (12 numbered sections + residual-risks table)
|
||||||
|
- [docs/audit/REVIEW-BUNDLE.md](./docs/audit/REVIEW-BUNDLE.md) — External crypto-review entrypoint (scope, build instructions, reporting)
|
||||||
|
- [docs/audit/SCOPE.md](./docs/audit/SCOPE.md) — One-page audit-scope summary
|
||||||
|
|
||||||
|
**Migration + history**
|
||||||
|
|
||||||
|
- [MIGRATION.md](./MIGRATION.md) — How to replace existing crypto with Shade + the [0.3.x → 4.0 upgrade path](./MIGRATION.md#migrating-from-03x-to-40-ga)
|
||||||
|
- [CHANGELOG.md](./CHANGELOG.md) — `4.0.0` GA section + every prior release
|
||||||
|
- [docs/archive/](./docs/archive/) — V2.1 / V2.2 / V2.3 backlog and V3.1 → V3.12 implementation plans (all `Status: Done`)
|
||||||
|
- [docs/V5.0.md](./docs/V5.0.md) — Voice / Video / Broadcast (post-GA, built on the frozen 4.0 stack)
|
||||||
|
|
||||||
|
**Examples**
|
||||||
|
|
||||||
- [examples/](./examples/) — Runnable example applications, including
|
- [examples/](./examples/) — Runnable example applications, including
|
||||||
[`07-streams-upload`](./examples/07-streams-upload) (multi-lane file transfer)
|
[`07-streams-upload`](./examples/07-streams-upload) (multi-lane file transfer)
|
||||||
and [`08-files-browser`](./examples/08-files-browser) (filesystem RPC)
|
and [`08-files-browser`](./examples/08-files-browser) (filesystem RPC)
|
||||||
- [MIGRATION.md](./MIGRATION.md) — How to replace existing crypto with Shade
|
|
||||||
|
|
||||||
## Deployment — one container per project
|
## Deployment — one container per project
|
||||||
|
|
||||||
@@ -274,15 +451,30 @@ docker run -d \
|
|||||||
-v my-project-shade:/data \
|
-v my-project-shade:/data \
|
||||||
-p 3900:3900 \
|
-p 3900:3900 \
|
||||||
-e SHADE_OBSERVER_TOKEN=change-me-to-at-least-16-chars \
|
-e SHADE_OBSERVER_TOKEN=change-me-to-at-least-16-chars \
|
||||||
gt.zyon.no/stian/shade-prekey:latest
|
gt.zyon.no/stian/shade-prekey:4.0.0
|
||||||
```
|
```
|
||||||
|
|
||||||
The container includes:
|
The container includes:
|
||||||
|
|
||||||
- **Prekey server** — `/v1/keys/*` REST API
|
- **Prekey server** — `/v1/keys/*` REST API
|
||||||
- **Observer dashboard** — `/shade-observer/dashboard/` (off unless token is set)
|
- **Inbox relay (V3.6)** — `/v1/inbox/*` async store-and-forward; enable
|
||||||
|
with `SHADE_INBOX_DB_PATH=/data/inbox.db` (or `SHADE_INBOX_PG_URL`).
|
||||||
|
`SHADE_INBOX_PRUNE_INTERVAL_MINUTES` controls TTL prune cadence.
|
||||||
|
- **Bridge transports (V3.7)** — `/v1/bridge/{stream,poll,ws}` SSE +
|
||||||
|
long-poll + WS adapters for clients that can't keep a WebSocket open.
|
||||||
|
- **Transfer routes** — `/v1/transfer/*` chunk + state + control routes
|
||||||
|
for `@shade/transfer`.
|
||||||
|
- **Key Transparency (V3.12)** — `/v1/kt/*` exposes `log_id`, latest +
|
||||||
|
historical STH, and consistency proofs. Enable with
|
||||||
|
`SHADE_KT_*` env vars; off by default.
|
||||||
|
- **Observer dashboard** — `/shade-observer/dashboard/` (off unless
|
||||||
|
`SHADE_OBSERVER_TOKEN` is set)
|
||||||
- **OpenAPI spec** — `/openapi.yaml` and interactive `/docs` viewer
|
- **OpenAPI spec** — `/openapi.yaml` and interactive `/docs` viewer
|
||||||
|
(covers all 27 routes — prekey, inbox, bridge, transfer, KT,
|
||||||
|
observer, `/metrics`, `/healthz`, `/ready`)
|
||||||
- **Prometheus metrics** — `/metrics`
|
- **Prometheus metrics** — `/metrics`
|
||||||
- **Health check** — `/health`
|
- **Health probes** — `/health` (full), `/healthz` (liveness),
|
||||||
|
`/ready` (readiness)
|
||||||
- **Stale cleanup** — purges inactive identities automatically
|
- **Stale cleanup** — purges inactive identities automatically
|
||||||
|
|
||||||
See [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md) for the full deployment guide, environment variables, PostgreSQL config, backup strategy, and Dokploy instructions.
|
See [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md) for the full deployment guide, environment variables, PostgreSQL config, backup strategy, and Dokploy instructions.
|
||||||
|
|||||||
@@ -1,8 +1,11 @@
|
|||||||
# Shade V2.1 — Improvements (infrastructure, storage, operations, security)
|
# Shade V2.1 — Improvements (infrastructure, storage, operations, security)
|
||||||
|
|
||||||
This document describes **improvements** agreed for next-generation work on Shade: clearer product story, stronger storage, mobile parity, operational hardening, transfer abuse, and a formal security narrative.
|
**Status:** Done — superseded by the V3.1 → V3.12 plans, all of which
|
||||||
|
landed in the 4.0 GA release. This document is preserved as historical
|
||||||
|
context for the original V2.1 backlog; the concrete deliverables live
|
||||||
|
under [`docs/archive/V3.*.md`](./).
|
||||||
|
|
||||||
**Audience:** **Maintainers and contributors** implementing the changes. Add status fields as items land in code/docs.
|
This document describes **improvements** agreed for next-generation work on Shade: clearer product story, stronger storage, mobile parity, operational hardening, transfer abuse, and a formal security narrative.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -1,8 +1,10 @@
|
|||||||
# Shade V2.2 — Feature plan: product, platform, and developer experience
|
# Shade V2.2 — Feature plan: product, platform, and developer experience
|
||||||
|
|
||||||
This document gathers **planned features** that extend Shade beyond today’s core (X3DH + Double Ratchet + Streams/transfer): groups, asynchronous delivery, richer file UX, web workers, CLI, API docs, and scaffolding.
|
**Status:** Done — superseded by V3.6 (inbox), V3.7 (bridge), V3.8
|
||||||
|
(workers), V3.9 (file metadata), V3.10 (recovery), V3.12 (KT). All
|
||||||
|
landed in the 4.0 GA release; see [`docs/ROADMAP.md`](../ROADMAP.md).
|
||||||
|
|
||||||
Add optional per-feature status (Idea / Design / IMP / Done).
|
This document gathers **planned features** that extend Shade beyond today’s core (X3DH + Double Ratchet + Streams/transfer): groups, asynchronous delivery, richer file UX, web workers, CLI, API docs, and scaffolding.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,9 @@
|
|||||||
# Shade V2.3 — Tillit, retention, integrasjon og observability
|
# Shade V2.3 — Tillit, retention, integrasjon og observability
|
||||||
|
|
||||||
|
**Status:** Done — superseded by V3.3 (trust UX), V3.4 (observability),
|
||||||
|
V3.10 (recovery), V3.12 (KT). Alt levert i 4.0 GA — se
|
||||||
|
[`docs/ROADMAP.md`](../ROADMAP.md).
|
||||||
|
|
||||||
Dette dokumentet beskriver **høyere ambisjonsnivå** og **plattformkryssende** arbeid: brukertilfeller der tillit må være **eksplisitt**, data må **utkrympes**/ryddes automatisk, apper må **enkelt koble seg på** kjente transportmønstre, og drift må **observere** uten å lekke innhold.
|
Dette dokumentet beskriver **høyere ambisjonsnivå** og **plattformkryssende** arbeid: brukertilfeller der tillit må være **eksplisitt**, data må **utkrympes**/ryddes automatisk, apper må **enkelt koble seg på** kjente transportmønstre, og drift må **observere** uten å lekke innhold.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# V3.12 — Key Transparency: Designnotat
|
# V3.12 — Key Transparency: Designnotat
|
||||||
|
|
||||||
**Status:** Approved (in-tree review — markeres `Design` i ROADMAP)
|
**Status:** Done — implementert i `@shade/key-transparency` 0.4.0, frosset i 4.0 GA.
|
||||||
**Forfatter:** Shade-teamet
|
**Forfatter:** Shade-teamet
|
||||||
**Reviewer-mål:** ekstern crypto-orientert reviewer før produksjons-deploy.
|
**Reviewer-mål:** ekstern crypto-orientert reviewer før produksjons-deploy.
|
||||||
**Implementasjons-target:** `@shade/key-transparency` + utvidelser i
|
**Implementasjons-target:** `@shade/key-transparency` + utvidelser i
|
||||||
|
|||||||
105
docs/files.md
105
docs/files.md
@@ -53,6 +53,111 @@ if (result.kind === 'inline') console.log(result.bytes.byteLength);
|
|||||||
else for await (const _chunk of /* result.stream */) { /* ... */ }
|
else for await (const _chunk of /* result.stream */) { /* ... */ }
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## HTTP RPC — browser-friendly request-response (4.1+)
|
||||||
|
|
||||||
|
The default `shade.files.client(peer)` requires both peers to be mutually
|
||||||
|
addressable over HTTP — the response to a `list`/`read` etc. round-trips
|
||||||
|
through `Shade.deliverControlEnvelope`, which POSTs to the peer's
|
||||||
|
`/v1/transfer/control` endpoint. **That doesn't work for browsers** —
|
||||||
|
a tab can't host an HTTP server, so the server cannot call back outbound.
|
||||||
|
|
||||||
|
`@shade/files` 4.1 ships a parallel **request-response** transport that
|
||||||
|
lets browser-style clients fully consume the file-RPC surface without
|
||||||
|
any inbound channel. It mirrors the way `@shade/server`'s
|
||||||
|
`shade-auth-middleware` works: one POST per RPC, encrypted envelope in
|
||||||
|
the request body, encrypted response in the same HTTP response.
|
||||||
|
|
||||||
|
### Server side — mount the RPC route
|
||||||
|
|
||||||
|
```ts
|
||||||
|
// 1. Register the file handler. `inlineOnly: true` skips the
|
||||||
|
// streams-bridge (which would require @shade/transfer).
|
||||||
|
await shade.files.serve(handlerConfig, { inlineOnly: true });
|
||||||
|
|
||||||
|
// 2. Mount the route on your Hono app under any base path.
|
||||||
|
app.route('/api/v1/shade-files', shade.files.rpcRoute());
|
||||||
|
// ^^^^^^^^^^^^^^
|
||||||
|
// POST <base>/rpc
|
||||||
|
```
|
||||||
|
|
||||||
|
`rpcRoute()` accepts:
|
||||||
|
|
||||||
|
| Option | Default | Purpose |
|
||||||
|
|---------------------|---------|----------------------------------------------------------------------------------------------------|
|
||||||
|
| `maxBodyBytes` | 1 MiB | Max request body. The protocol caps inline payloads at 256 KiB; the headroom is for base64 inflation + custom-op envelopes. |
|
||||||
|
| `acceptFirstMessage`| `false` | Accept `0x01` PreKeyMessage envelopes — required when the RPC route also doubles as the X3DH handshake (browser's first-ever request). |
|
||||||
|
|
||||||
|
### Browser client
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { createShade } from '@shade/sdk';
|
||||||
|
|
||||||
|
const shade = await createShade({
|
||||||
|
prekeyServer: 'https://shade.example.com',
|
||||||
|
storage: 'memory',
|
||||||
|
address: 'alice@example.com',
|
||||||
|
});
|
||||||
|
|
||||||
|
const fs = shade.files.httpClient('bob@example.com', {
|
||||||
|
rpcUrl: 'https://dispatch.example.com/api/v1/shade-files/rpc',
|
||||||
|
// Optional: thread CSRF / auth tokens, override fetch, etc.
|
||||||
|
headers: { 'X-CSRF-Token': csrfToken },
|
||||||
|
});
|
||||||
|
|
||||||
|
await fs.mkdir('/photos');
|
||||||
|
await fs.write('/photos/cover.png', new Uint8Array([/* ... */]), {
|
||||||
|
contentType: 'image/png',
|
||||||
|
});
|
||||||
|
const result = await fs.read('/photos/cover.png');
|
||||||
|
```
|
||||||
|
|
||||||
|
### What works in HTTP-RPC mode
|
||||||
|
|
||||||
|
- `list`, `stat`, `mkdir`, `delete`, `move`, `getThumbnail`, `custom<K>` — full parity.
|
||||||
|
- `write` — **inline only** (≤ 256 KiB plaintext). Larger inputs throw `ConflictError`.
|
||||||
|
- `read` — **inline only**. If the server returns a streamed `read` result, the client throws `InternalFileError` directing callers to the stateful pathway.
|
||||||
|
|
||||||
|
### Wire contract
|
||||||
|
|
||||||
|
```
|
||||||
|
POST /rpc HTTP/1.1
|
||||||
|
Content-Type: application/octet-stream
|
||||||
|
X-Shade-Sender-Address: alice@example.com
|
||||||
|
|
||||||
|
<wire-encoded ShadeEnvelope (0x01 first-time, 0x02 after) wrapping
|
||||||
|
JSON-encoded RpcRequest>
|
||||||
|
|
||||||
|
────
|
||||||
|
|
||||||
|
200 OK
|
||||||
|
Content-Type: application/octet-stream
|
||||||
|
|
||||||
|
<wire-encoded ShadeEnvelope (0x02) wrapping JSON-encoded RpcResponse | RpcError>
|
||||||
|
```
|
||||||
|
|
||||||
|
Transport-level failures (no session, undecryptable envelope, body too
|
||||||
|
big) return JSON `{ "error": "..." }` with appropriate 4xx status.
|
||||||
|
Application-level failures (file not found, permission denied) ship
|
||||||
|
encrypted `RpcError` envelopes — the client maps them back to typed
|
||||||
|
`FileError` subclasses (`NotFoundError`, `ConflictError`, etc.).
|
||||||
|
|
||||||
|
### Symmetry with `@shade/server`
|
||||||
|
|
||||||
|
The shape mirrors `@shade/server`'s shade-auth-middleware: encrypted
|
||||||
|
envelope rides the request body, server decrypts via the existing
|
||||||
|
ratchet session, performs the protected operation, returns an encrypted
|
||||||
|
envelope in the response. No bidirectional channel required, no
|
||||||
|
WebSocket, no SSE.
|
||||||
|
|
||||||
|
### When to use which
|
||||||
|
|
||||||
|
| Setup | Use |
|
||||||
|
|-----------------------------------------------|-----------------------------------------------|
|
||||||
|
| Browser client ↔ Bun/Hono server | `httpClient()` + `rpcRoute()` |
|
||||||
|
| Server ↔ server (both can host HTTP) | `client()` (default) — supports streams |
|
||||||
|
| Service-worker / extension ↔ server | `httpClient()` (no inbound listener) |
|
||||||
|
| CLI / daemon ↔ daemon | Either; `client()` if you need streams |
|
||||||
|
|
||||||
## Op surface
|
## Op surface
|
||||||
|
|
||||||
| Op | Args | Result |
|
| Op | Args | Result |
|
||||||
|
|||||||
@@ -16,8 +16,10 @@
|
|||||||
"version": "bun run scripts/bump-version.ts",
|
"version": "bun run scripts/bump-version.ts",
|
||||||
"soak": "bun run scripts/soak.ts",
|
"soak": "bun run scripts/soak.ts",
|
||||||
"soak:smoke": "bun run scripts/soak.ts --hours 0.05 --pairs 4",
|
"soak:smoke": "bun run scripts/soak.ts --hours 0.05 --pairs 4",
|
||||||
"publish:dry": "DRY_RUN=1 bun run scripts/publish-all.ts",
|
"typecheck": "bun run scripts/typecheck-all.ts",
|
||||||
"publish:all": "bash scripts/publish-shade.sh",
|
"prepublish:check": "bun run typecheck",
|
||||||
|
"publish:dry": "bun run prepublish:check && DRY_RUN=1 bun run scripts/publish-all.ts",
|
||||||
|
"publish:all": "bun run prepublish:check && bash scripts/publish-shade.sh",
|
||||||
"build:docker": "bun run scripts/build-docker.ts",
|
"build:docker": "bun run scripts/build-docker.ts",
|
||||||
"publish:docker": "bun run scripts/build-docker.ts -- --push"
|
"publish:docker": "bun run scripts/build-docker.ts -- --push"
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/cli",
|
"name": "@shade/cli",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/cli.ts",
|
"main": "src/cli.ts",
|
||||||
"bin": {
|
"bin": {
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/core",
|
"name": "@shade/core",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/crypto-web",
|
"name": "@shade/crypto-web",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/dashboard",
|
"name": "@shade/dashboard",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"dev": "vite",
|
"dev": "vite",
|
||||||
|
|||||||
@@ -8,5 +8,5 @@
|
|||||||
"module": "ESNext",
|
"module": "ESNext",
|
||||||
"moduleResolution": "bundler"
|
"moduleResolution": "bundler"
|
||||||
},
|
},
|
||||||
"include": ["src", "vite.config.ts"]
|
"include": ["src"]
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/files",
|
"name": "@shade/files",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
import type { Shade } from '@shade/sdk';
|
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||||
import {
|
import {
|
||||||
KIND_CUSTOM_V1,
|
KIND_CUSTOM_V1,
|
||||||
KIND_DELETE_V1,
|
KIND_DELETE_V1,
|
||||||
@@ -184,7 +184,7 @@ export interface CreateFileClientOptions {
|
|||||||
* transfers that carry the actual bytes.
|
* transfers that carry the actual bytes.
|
||||||
*/
|
*/
|
||||||
export function createFileClient(
|
export function createFileClient(
|
||||||
shade: Shade,
|
shade: ShadeBridge,
|
||||||
channel: ShadeFileRpcChannel,
|
channel: ShadeFileRpcChannel,
|
||||||
pending: PendingRpcRegistry,
|
pending: PendingRpcRegistry,
|
||||||
peerAddress: string,
|
peerAddress: string,
|
||||||
|
|||||||
590
packages/shade-files/src/client/http-client.ts
Normal file
590
packages/shade-files/src/client/http-client.ts
Normal file
@@ -0,0 +1,590 @@
|
|||||||
|
/**
|
||||||
|
* Browser-friendly request-response `FileClient` for `@shade/files`.
|
||||||
|
*
|
||||||
|
* The default `shade.files.client(peer)` ships RPC envelopes via
|
||||||
|
* `Shade.send` + `Shade.deliverControlEnvelope`, which means the
|
||||||
|
* server has to be able to call back outbound to the client. That
|
||||||
|
* doesn't work for browser tabs (no inbound HTTP listener). This
|
||||||
|
* client posts each RPC envelope to a single server endpoint and
|
||||||
|
* reads the encrypted response from the same HTTP response — pure
|
||||||
|
* request-response, no inbound channel required.
|
||||||
|
*
|
||||||
|
* Inline payloads only (≤ 256 KiB). For larger reads/writes, use the
|
||||||
|
* stateful path: `shade.files.client(peer)` server-to-server, with
|
||||||
|
* `@shade/transfer` chunk routes for content I/O.
|
||||||
|
*
|
||||||
|
* @see {@link createFilesRpcRoute} for the matching server-side route.
|
||||||
|
*/
|
||||||
|
import type { ZodTypeAny } from 'zod';
|
||||||
|
import { decodeEnvelope, encodeEnvelope as encodeWireEnvelope } from '@shade/proto';
|
||||||
|
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||||
|
import {
|
||||||
|
encodeEnvelope as encodeRpcEnvelope,
|
||||||
|
tryParseEnvelope,
|
||||||
|
} from '../protocol/envelope-codec.js';
|
||||||
|
import {
|
||||||
|
KIND_CUSTOM_V1,
|
||||||
|
KIND_DELETE_V1,
|
||||||
|
KIND_GET_THUMBNAIL_V1,
|
||||||
|
KIND_LIST_V1,
|
||||||
|
KIND_MKDIR_V1,
|
||||||
|
KIND_MOVE_V1,
|
||||||
|
KIND_READ_V1,
|
||||||
|
KIND_STAT_V1,
|
||||||
|
KIND_WRITE_V1,
|
||||||
|
} from '../protocol/kinds.js';
|
||||||
|
import {
|
||||||
|
CustomArgsSchema,
|
||||||
|
CustomResultSchema,
|
||||||
|
DeleteArgsSchema,
|
||||||
|
DeleteResultSchema,
|
||||||
|
GetThumbnailArgsSchema,
|
||||||
|
GetThumbnailResultSchema,
|
||||||
|
ListArgsSchema,
|
||||||
|
ListResultSchema,
|
||||||
|
MkdirArgsSchema,
|
||||||
|
MkdirResultSchema,
|
||||||
|
MoveArgsSchema,
|
||||||
|
MoveResultSchema,
|
||||||
|
ReadArgsSchema,
|
||||||
|
ReadResultSchema,
|
||||||
|
StatArgsSchema,
|
||||||
|
StatResultSchema,
|
||||||
|
WriteArgsSchema,
|
||||||
|
WriteResultSchema,
|
||||||
|
type ListResult,
|
||||||
|
type MkdirResult,
|
||||||
|
type DeleteResult,
|
||||||
|
type MoveResult,
|
||||||
|
type StatResult,
|
||||||
|
type ThumbnailSize,
|
||||||
|
type WriteResult,
|
||||||
|
} from '../schemas/ops.js';
|
||||||
|
import {
|
||||||
|
fileErrorFromPayload,
|
||||||
|
CancelledError,
|
||||||
|
InternalFileError,
|
||||||
|
ConflictError,
|
||||||
|
} from '../schemas/errors.js';
|
||||||
|
import { buildRpcRequest } from '../protocol/rpc-builder.js';
|
||||||
|
import { decideInline, INLINE_THRESHOLD, type WriteSource } from './inline-threshold.js';
|
||||||
|
import { base64ToBytes, bytesToBase64 } from '../protocol/canonical.js';
|
||||||
|
import { startQueueDrainer, type QueueDrainerHandle } from './queue-drainer.js';
|
||||||
|
import {
|
||||||
|
createClientStreamsBridge,
|
||||||
|
type ClientStreamsBridge,
|
||||||
|
} from './streams-bridge.js';
|
||||||
|
import type {
|
||||||
|
FileClient,
|
||||||
|
ReadOpts,
|
||||||
|
ReadOutput,
|
||||||
|
ThumbnailResult,
|
||||||
|
WriteOpts,
|
||||||
|
CreateFileClientOptions,
|
||||||
|
BaseOpts,
|
||||||
|
} from './client.js';
|
||||||
|
|
||||||
|
export interface FilesHttpClientOptions
|
||||||
|
extends Omit<CreateFileClientOptions, 'streamsBridge'> {
|
||||||
|
/**
|
||||||
|
* Server endpoint that hosts `createFilesRpcRoute(...)`. Typically:
|
||||||
|
* `https://server.example.com/api/v1/shade-files/rpc`.
|
||||||
|
*/
|
||||||
|
rpcUrl: string;
|
||||||
|
/**
|
||||||
|
* Optional `fetch` override. Defaults to `globalThis.fetch`. Wire a
|
||||||
|
* custom `fetch` to thread auth-cookies, CSRF tokens, or
|
||||||
|
* service-worker interception.
|
||||||
|
*/
|
||||||
|
fetch?: typeof globalThis.fetch;
|
||||||
|
/**
|
||||||
|
* Extra HTTP headers applied to every RPC POST. Useful for app-level
|
||||||
|
* auth (CSRF, session cookies via custom header, etc.) — these are
|
||||||
|
* orthogonal to the ratchet authentication on the envelope itself.
|
||||||
|
*/
|
||||||
|
headers?: Record<string, string>;
|
||||||
|
/**
|
||||||
|
* Server endpoint that hosts `transferQueueRoute()`'s long-poll
|
||||||
|
* endpoint. Typically:
|
||||||
|
* `https://server.example.com/api/v1/shade-files/queue`.
|
||||||
|
*
|
||||||
|
* When supplied, the client starts a background long-poll that
|
||||||
|
* drains queued envelopes + chunks from the server and dispatches
|
||||||
|
* them via `shade.acceptTransferEnvelope`. This unlocks
|
||||||
|
* **streamed reads** (>256 KiB) for browser-style consumers.
|
||||||
|
*/
|
||||||
|
outboundQueueUrl?: string;
|
||||||
|
/**
|
||||||
|
* Base URL for outbound transfer routes (browser → server). Required
|
||||||
|
* alongside `outboundQueueUrl` to enable streamed writes. Typically:
|
||||||
|
* `https://server.example.com/api/v1/shade-files`.
|
||||||
|
*
|
||||||
|
* The client POSTs:
|
||||||
|
* - chunks to `<base>/v1/transfer/<streamId>/chunk`
|
||||||
|
* - control envelopes to `<base>/v1/transfer/control`
|
||||||
|
*/
|
||||||
|
transferBaseUrl?: string;
|
||||||
|
/**
|
||||||
|
* Long-poll block timeout, milliseconds. Default 30_000. Server
|
||||||
|
* clamps to its own `maxBlockMs` (default 55_000).
|
||||||
|
*/
|
||||||
|
queueBlockMs?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface RoundTripOpts {
|
||||||
|
signal?: AbortSignal;
|
||||||
|
timeoutMs?: number;
|
||||||
|
idempotencyKey?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Create a request-response `FileClient` bound to `peerAddress` and a
|
||||||
|
* server-side RPC URL. The session must already be established
|
||||||
|
* (via `shade.initSessionFromBundle(peerAddress, bundle)` or an
|
||||||
|
* incoming first-message). Otherwise the first RPC will fail with
|
||||||
|
* "decrypt failed: no session for peer".
|
||||||
|
*
|
||||||
|
* When `outboundQueueUrl` + `transferBaseUrl` are supplied, the
|
||||||
|
* client also unlocks **streamed reads/writes** for files larger than
|
||||||
|
* the inline threshold (256 KiB). The browser polls the server's
|
||||||
|
* outbound queue for chunks/envelopes and POSTs its own outbound
|
||||||
|
* chunks to the server's transfer-receive routes.
|
||||||
|
*/
|
||||||
|
export function createFilesHttpClient(
|
||||||
|
shade: ShadeBridge,
|
||||||
|
peerAddress: string,
|
||||||
|
options: FilesHttpClientOptions,
|
||||||
|
): FileClient {
|
||||||
|
const rpcUrl = options.rpcUrl;
|
||||||
|
const fetchFn = options.fetch ?? globalThis.fetch.bind(globalThis);
|
||||||
|
const extraHeaders = options.headers ?? {};
|
||||||
|
const defaultTimeoutMs = options.defaultTimeoutMs ?? 30_000;
|
||||||
|
const ioTimeoutMs = options.ioTimeoutMs ?? 60_000;
|
||||||
|
const signRequest = options.signRequest;
|
||||||
|
const senderAddress = shade.myAddress;
|
||||||
|
|
||||||
|
// ─── Streamed-mode bootstrap ─────────────────────────────────
|
||||||
|
//
|
||||||
|
// When `outboundQueueUrl` is supplied, the client:
|
||||||
|
// 1. Configures `shade.configureTransfers(...)` so outbound
|
||||||
|
// chunks POST to `<transferBaseUrl>/v1/transfer/<streamId>/chunk`
|
||||||
|
// and outbound control envelopes POST to
|
||||||
|
// `<transferBaseUrl>/v1/transfer/control`.
|
||||||
|
// 2. Spawns a streams-bridge so streamed reads can be awaited.
|
||||||
|
// 3. Starts a long-poll drainer that pulls queued envelopes +
|
||||||
|
// chunks from the server and dispatches via
|
||||||
|
// `shade.acceptTransferEnvelope`.
|
||||||
|
|
||||||
|
let drainer: QueueDrainerHandle | null = null;
|
||||||
|
let streamsBridgePromise: Promise<ClientStreamsBridge> | null = null;
|
||||||
|
let streamsBridge: ClientStreamsBridge | null = null;
|
||||||
|
|
||||||
|
if (options.outboundQueueUrl !== undefined) {
|
||||||
|
const outboundQueueUrl = options.outboundQueueUrl;
|
||||||
|
if (options.transferBaseUrl === undefined) {
|
||||||
|
throw new Error(
|
||||||
|
'createFilesHttpClient: outboundQueueUrl was supplied without transferBaseUrl. Pass `transferBaseUrl` (the server prefix that hosts /v1/transfer/...) so outbound chunks have a destination.',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (shade.configureTransfers === undefined) {
|
||||||
|
throw new Error(
|
||||||
|
'createFilesHttpClient: shade.configureTransfers is required for streamed mode (the underlying ShadeBridge must surface it).',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const transferBaseUrl = options.transferBaseUrl.replace(/\/$/, '');
|
||||||
|
shade.configureTransfers({
|
||||||
|
resolveBaseUrl: async (peer) => {
|
||||||
|
if (peer !== peerAddress) {
|
||||||
|
throw new Error(
|
||||||
|
`httpClient is bound to peer "${peerAddress}" — refusing to resolve outgoing chunks for "${peer}" without a multi-peer registry. Use shade.files.client(peer) for server-to-server multi-peer.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return transferBaseUrl;
|
||||||
|
},
|
||||||
|
});
|
||||||
|
// Build the streams-bridge eagerly. The engine's incoming-transfer
|
||||||
|
// subscription has to be in place BEFORE the drainer dispatches the
|
||||||
|
// first stream-init envelope, otherwise the engine emits the
|
||||||
|
// IncomingTransfer to zero handlers and the read silently never
|
||||||
|
// accepts. We kick off the drainer once the bridge has subscribed.
|
||||||
|
streamsBridgePromise = createClientStreamsBridge(shade).then((bridge) => {
|
||||||
|
streamsBridge = bridge;
|
||||||
|
drainer = startQueueDrainer(shade, {
|
||||||
|
outboundQueueUrl,
|
||||||
|
peerAddress,
|
||||||
|
senderAddress,
|
||||||
|
...(options.fetch !== undefined ? { fetch: options.fetch } : {}),
|
||||||
|
...(options.headers !== undefined ? { headers: options.headers } : {}),
|
||||||
|
...(options.queueBlockMs !== undefined ? { blockMs: options.queueBlockMs } : {}),
|
||||||
|
});
|
||||||
|
return bridge;
|
||||||
|
});
|
||||||
|
// Surface bridge-construction failures eagerly via a rejected
|
||||||
|
// promise the next read/write picks up.
|
||||||
|
streamsBridgePromise.catch(() => {
|
||||||
|
/* observed via getStreamsBridge() */
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async function getStreamsBridge(): Promise<ClientStreamsBridge> {
|
||||||
|
if (streamsBridge !== null) return streamsBridge;
|
||||||
|
if (streamsBridgePromise === null) {
|
||||||
|
throw new ConflictError(
|
||||||
|
`http RPC client supports inline writes/reads only (≤ ${INLINE_THRESHOLD} bytes) — pass { outboundQueueUrl, transferBaseUrl } to enable streamed transfers.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
streamsBridge = await streamsBridgePromise;
|
||||||
|
return streamsBridge;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Encrypt + POST + decrypt + parse one RPC round-trip.
|
||||||
|
*
|
||||||
|
* Throws a typed `FileError` subclass when the server returns an
|
||||||
|
* encrypted `RpcError`, or `InternalFileError` for transport-level
|
||||||
|
* failures (network, 4xx/5xx, malformed body).
|
||||||
|
*/
|
||||||
|
async function roundTrip<TResult>(
|
||||||
|
kind: string,
|
||||||
|
op: 'list' | 'stat' | 'mkdir' | 'delete' | 'move' | 'read' | 'write' | 'getThumbnail' | 'custom',
|
||||||
|
args: unknown,
|
||||||
|
resultSchema: ZodTypeAny,
|
||||||
|
opts: RoundTripOpts | undefined,
|
||||||
|
): Promise<TResult> {
|
||||||
|
const requestEnv = await buildRpcRequest({
|
||||||
|
senderAddress,
|
||||||
|
kind,
|
||||||
|
op,
|
||||||
|
args,
|
||||||
|
...(opts?.idempotencyKey !== undefined ? { idempotencyKey: opts.idempotencyKey } : {}),
|
||||||
|
...(signRequest !== undefined ? { signRequest } : {}),
|
||||||
|
});
|
||||||
|
|
||||||
|
const plaintext = encodeRpcEnvelope(requestEnv);
|
||||||
|
const ratchetEnvelope = await shade.send(peerAddress, plaintext);
|
||||||
|
const wireBytes = encodeWireEnvelope(ratchetEnvelope);
|
||||||
|
|
||||||
|
const ac = new AbortController();
|
||||||
|
const timeoutMs = opts?.timeoutMs ?? defaultTimeoutMs;
|
||||||
|
const timer = setTimeout(
|
||||||
|
() => ac.abort(new Error(`RPC timeout after ${timeoutMs}ms`)),
|
||||||
|
timeoutMs,
|
||||||
|
);
|
||||||
|
(timer as unknown as { unref?: () => void }).unref?.();
|
||||||
|
if (opts?.signal !== undefined) {
|
||||||
|
const userSignal = opts.signal;
|
||||||
|
if (userSignal.aborted) ac.abort(userSignal.reason);
|
||||||
|
else userSignal.addEventListener('abort', () => ac.abort(userSignal.reason), { once: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
let response: Response;
|
||||||
|
try {
|
||||||
|
// Wrap the wire bytes in a Blob so the body type satisfies the
|
||||||
|
// common-denominator `BodyInit` across DOM, Bun, and node-fetch
|
||||||
|
// (some runtimes accept `Uint8Array` directly, others don't).
|
||||||
|
// Cast through `unknown` because TS's `bun-types` and `lib.dom`
|
||||||
|
// disagree about whether `Uint8Array<ArrayBufferLike>` is itself
|
||||||
|
// a `BlobPart`; the runtime accepts it on every platform.
|
||||||
|
response = await fetchFn(rpcUrl, {
|
||||||
|
method: 'POST',
|
||||||
|
body: new Blob([wireBytes as unknown as ArrayBuffer]),
|
||||||
|
signal: ac.signal,
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/octet-stream',
|
||||||
|
'X-Shade-Sender-Address': senderAddress,
|
||||||
|
...extraHeaders,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
} catch (err) {
|
||||||
|
clearTimeout(timer);
|
||||||
|
if ((err as Error).name === 'AbortError') {
|
||||||
|
throw new CancelledError(`RPC ${kind} aborted: ${(err as Error).message}`);
|
||||||
|
}
|
||||||
|
throw new InternalFileError(`RPC ${kind} fetch failed: ${(err as Error).message}`);
|
||||||
|
}
|
||||||
|
clearTimeout(timer);
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
let body: { error?: string } | null = null;
|
||||||
|
try {
|
||||||
|
body = (await response.json()) as { error?: string };
|
||||||
|
} catch {
|
||||||
|
/* server emitted non-JSON body */
|
||||||
|
}
|
||||||
|
throw new InternalFileError(
|
||||||
|
`RPC ${kind} → ${response.status} ${response.statusText}: ${
|
||||||
|
body?.error ?? '(no error body)'
|
||||||
|
}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const ab = await response.arrayBuffer();
|
||||||
|
if (ab.byteLength === 0) {
|
||||||
|
throw new InternalFileError(`RPC ${kind}: empty response body`);
|
||||||
|
}
|
||||||
|
|
||||||
|
let responseRatchet;
|
||||||
|
try {
|
||||||
|
responseRatchet = decodeEnvelope(new Uint8Array(ab));
|
||||||
|
} catch (err) {
|
||||||
|
throw new InternalFileError(
|
||||||
|
`RPC ${kind}: response body is not a valid wire envelope: ${(err as Error).message}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let responsePlaintext: string;
|
||||||
|
try {
|
||||||
|
responsePlaintext = await shade.receive(peerAddress, responseRatchet);
|
||||||
|
} catch (err) {
|
||||||
|
throw new InternalFileError(
|
||||||
|
`RPC ${kind}: response decrypt failed: ${(err as Error).message}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const classified = tryParseEnvelope(responsePlaintext);
|
||||||
|
if (classified === null) {
|
||||||
|
throw new InternalFileError(
|
||||||
|
`RPC ${kind}: response plaintext is not a valid @shade/files envelope`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (classified.kind === 'error') {
|
||||||
|
throw fileErrorFromPayload(classified.envelope.error);
|
||||||
|
}
|
||||||
|
if (classified.kind !== 'response') {
|
||||||
|
throw new InternalFileError(
|
||||||
|
`RPC ${kind}: unexpected response envelope kind: ${classified.kind}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (classified.envelope.id !== requestEnv.id) {
|
||||||
|
throw new InternalFileError(
|
||||||
|
`RPC ${kind}: response correlation id mismatch (got ${classified.envelope.id}, expected ${requestEnv.id})`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return resultSchema.parse(classified.envelope.result) as TResult;
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
async list(path, opts): Promise<ListResult> {
|
||||||
|
const args = ListArgsSchema.parse({
|
||||||
|
path,
|
||||||
|
...(opts?.cursor !== undefined ? { cursor: opts.cursor } : {}),
|
||||||
|
...(opts?.pageSize !== undefined ? { pageSize: opts.pageSize } : {}),
|
||||||
|
...(opts?.filter !== undefined ? { filter: opts.filter } : {}),
|
||||||
|
});
|
||||||
|
return await roundTrip<ListResult>(
|
||||||
|
KIND_LIST_V1,
|
||||||
|
'list',
|
||||||
|
args,
|
||||||
|
ListResultSchema,
|
||||||
|
opts,
|
||||||
|
);
|
||||||
|
},
|
||||||
|
|
||||||
|
async stat(path, opts): Promise<StatResult> {
|
||||||
|
const args = StatArgsSchema.parse({ path });
|
||||||
|
return await roundTrip<StatResult>(KIND_STAT_V1, 'stat', args, StatResultSchema, opts);
|
||||||
|
},
|
||||||
|
|
||||||
|
async mkdir(path, opts): Promise<MkdirResult> {
|
||||||
|
const args = MkdirArgsSchema.parse({
|
||||||
|
path,
|
||||||
|
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
|
||||||
|
});
|
||||||
|
return await roundTrip<MkdirResult>(
|
||||||
|
KIND_MKDIR_V1,
|
||||||
|
'mkdir',
|
||||||
|
args,
|
||||||
|
MkdirResultSchema,
|
||||||
|
opts,
|
||||||
|
);
|
||||||
|
},
|
||||||
|
|
||||||
|
async delete(path, opts): Promise<DeleteResult> {
|
||||||
|
const args = DeleteArgsSchema.parse({
|
||||||
|
path,
|
||||||
|
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
|
||||||
|
});
|
||||||
|
return await roundTrip<DeleteResult>(
|
||||||
|
KIND_DELETE_V1,
|
||||||
|
'delete',
|
||||||
|
args,
|
||||||
|
DeleteResultSchema,
|
||||||
|
opts,
|
||||||
|
);
|
||||||
|
},
|
||||||
|
|
||||||
|
async move(src, dst, opts): Promise<MoveResult> {
|
||||||
|
const args = MoveArgsSchema.parse({
|
||||||
|
src,
|
||||||
|
dst,
|
||||||
|
...(opts?.overwrite !== undefined ? { overwrite: opts.overwrite } : {}),
|
||||||
|
});
|
||||||
|
return await roundTrip<MoveResult>(KIND_MOVE_V1, 'move', args, MoveResultSchema, opts);
|
||||||
|
},
|
||||||
|
|
||||||
|
async read(path, opts: ReadOpts = {}): Promise<ReadOutput> {
|
||||||
|
const args = ReadArgsSchema.parse({
|
||||||
|
path,
|
||||||
|
...(opts.range !== undefined ? { range: opts.range } : {}),
|
||||||
|
...(opts.preferInline !== undefined ? { preferInline: opts.preferInline } : {}),
|
||||||
|
});
|
||||||
|
const wire = await roundTrip<import('../schemas/ops.js').ReadResult>(
|
||||||
|
KIND_READ_V1,
|
||||||
|
'read',
|
||||||
|
args,
|
||||||
|
ReadResultSchema,
|
||||||
|
opts,
|
||||||
|
);
|
||||||
|
if (wire.kind === 'inline') {
|
||||||
|
const bytes = base64ToBytes(wire.bytesB64);
|
||||||
|
const out: ReadOutput = {
|
||||||
|
kind: 'inline',
|
||||||
|
bytes,
|
||||||
|
size: wire.size,
|
||||||
|
sha256: wire.sha256,
|
||||||
|
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
|
||||||
|
};
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
// Streamed read — only supported when the queue drainer is wired.
|
||||||
|
if (drainer === null) {
|
||||||
|
throw new InternalFileError(
|
||||||
|
`http RPC client received a streamed read (size ${wire.size}) but is in inline-only mode. Pass { outboundQueueUrl, transferBaseUrl } when constructing the client to enable streamed reads.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const bridge = await getStreamsBridge();
|
||||||
|
const bridgeSignal = opts.signal ?? new AbortController().signal;
|
||||||
|
const parked = await bridge.awaitRead(wire.streamId, {
|
||||||
|
expectedFrom: peerAddress,
|
||||||
|
signal: bridgeSignal,
|
||||||
|
timeoutMs: ioTimeoutMs,
|
||||||
|
});
|
||||||
|
const out: ReadOutput = {
|
||||||
|
kind: 'streams',
|
||||||
|
stream: parked.readable,
|
||||||
|
size: wire.size,
|
||||||
|
sha256: wire.sha256,
|
||||||
|
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
|
||||||
|
done: async () => {
|
||||||
|
await parked.done;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
return out;
|
||||||
|
},
|
||||||
|
|
||||||
|
async write(path, input: WriteSource, opts: WriteOpts = {}): Promise<WriteResult> {
|
||||||
|
const decision = await decideInline(input);
|
||||||
|
const overwrite = opts.overwrite ?? false;
|
||||||
|
const contentType = opts.contentType ?? decision.contentType;
|
||||||
|
|
||||||
|
if (decision.kind === 'inline' || opts.forceInline === true) {
|
||||||
|
const bytes = decision.kind === 'inline' ? decision.bytes : null;
|
||||||
|
if (bytes === null) {
|
||||||
|
// forceInline === true with a streams-typed decision —
|
||||||
|
// decideInline always produced a `streams` shape because the
|
||||||
|
// input was a bare ReadableStream. We can't drain a stream
|
||||||
|
// synchronously here without a streams-bridge.
|
||||||
|
throw new ConflictError(
|
||||||
|
'http RPC client cannot forceInline a streamed input — pass a Uint8Array / Blob, or pre-buffer the stream.',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (bytes.byteLength > INLINE_THRESHOLD) {
|
||||||
|
throw new ConflictError(
|
||||||
|
`inline write exceeds ${INLINE_THRESHOLD}-byte threshold (got ${bytes.byteLength}); pass forceInline=true to override`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const args = WriteArgsSchema.parse({
|
||||||
|
kind: 'inline',
|
||||||
|
path,
|
||||||
|
bytesB64: bytesToBase64(bytes),
|
||||||
|
...(contentType !== undefined ? { contentType } : {}),
|
||||||
|
overwrite,
|
||||||
|
});
|
||||||
|
return await roundTrip<WriteResult>(
|
||||||
|
KIND_WRITE_V1,
|
||||||
|
'write',
|
||||||
|
args,
|
||||||
|
WriteResultSchema,
|
||||||
|
opts,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Streamed write — requires the queue drainer + streams-bridge.
|
||||||
|
if (drainer === null) {
|
||||||
|
throw new ConflictError(
|
||||||
|
`http RPC client supports inline writes only (≤ ${INLINE_THRESHOLD} bytes). The supplied input was promoted to streams (size ${decision.size ?? 'unknown'}). Pass { outboundQueueUrl, transferBaseUrl } to enable streamed writes.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const bridge = await getStreamsBridge();
|
||||||
|
const size = decision.size;
|
||||||
|
if (size === undefined) {
|
||||||
|
throw new ConflictError(
|
||||||
|
'streams write requires a known plaintext size; pass `{ stream, size }` instead of a bare ReadableStream',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const { writeId, handle } = await bridge.initiateWrite({
|
||||||
|
peer: peerAddress,
|
||||||
|
stream: decision.stream,
|
||||||
|
size,
|
||||||
|
...(contentType !== undefined ? { contentType } : {}),
|
||||||
|
name: path,
|
||||||
|
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
|
||||||
|
});
|
||||||
|
const args = WriteArgsSchema.parse({
|
||||||
|
kind: 'streams',
|
||||||
|
path,
|
||||||
|
size,
|
||||||
|
...(contentType !== undefined ? { contentType } : {}),
|
||||||
|
overwrite,
|
||||||
|
writeId,
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
const [result] = await Promise.all([
|
||||||
|
roundTrip<WriteResult>(KIND_WRITE_V1, 'write', args, WriteResultSchema, opts),
|
||||||
|
handle.done(),
|
||||||
|
]);
|
||||||
|
return result;
|
||||||
|
} catch (err) {
|
||||||
|
await handle.abort('rpc-failed').catch(() => undefined);
|
||||||
|
throw err;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
|
async getThumbnail(path, size: ThumbnailSize, opts): Promise<ThumbnailResult> {
|
||||||
|
const args = GetThumbnailArgsSchema.parse({
|
||||||
|
path,
|
||||||
|
size,
|
||||||
|
...(opts?.format !== undefined ? { format: opts.format } : {}),
|
||||||
|
});
|
||||||
|
const raw = await roundTrip<import('../schemas/ops.js').GetThumbnailResult>(
|
||||||
|
KIND_GET_THUMBNAIL_V1,
|
||||||
|
'getThumbnail',
|
||||||
|
args,
|
||||||
|
GetThumbnailResultSchema,
|
||||||
|
opts,
|
||||||
|
);
|
||||||
|
return {
|
||||||
|
bytes: base64ToBytes(raw.bytesB64),
|
||||||
|
format: raw.format,
|
||||||
|
width: raw.width,
|
||||||
|
height: raw.height,
|
||||||
|
sha256: raw.sha256,
|
||||||
|
};
|
||||||
|
},
|
||||||
|
|
||||||
|
async custom(name, args, opts?: BaseOpts): Promise<unknown> {
|
||||||
|
const wireArgs = CustomArgsSchema.parse({ name, args });
|
||||||
|
return await roundTrip(KIND_CUSTOM_V1, 'custom', wireArgs, CustomResultSchema, opts);
|
||||||
|
},
|
||||||
|
|
||||||
|
close(): void {
|
||||||
|
// Stop the long-poll drainer + tear down the streams-bridge if
|
||||||
|
// we built one. Idempotent — safe to call multiple times.
|
||||||
|
drainer?.stop();
|
||||||
|
drainer = null;
|
||||||
|
if (streamsBridge !== null) {
|
||||||
|
void streamsBridge.destroy().catch(() => undefined);
|
||||||
|
streamsBridge = null;
|
||||||
|
}
|
||||||
|
streamsBridgePromise = null;
|
||||||
|
},
|
||||||
|
} as FileClient;
|
||||||
|
}
|
||||||
@@ -161,15 +161,33 @@ async function peekStream(stream: ReadableStream<Uint8Array>): Promise<InlineDec
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
interface MinimalReader {
|
/**
|
||||||
read(): Promise<{ value: Uint8Array | undefined; done: boolean }>;
|
* Structural mirror of WHATWG `ReadableStreamDefaultReader<Uint8Array>`.
|
||||||
|
*
|
||||||
|
* The disjoint union shape with `value?: T | undefined` is the lowest
|
||||||
|
* common denominator across every lib environment we care about:
|
||||||
|
* - `bun-types` emits `{ done: true; value?: undefined }`
|
||||||
|
* - `lib.dom` emits `{ done: true; value?: T }`
|
||||||
|
* - `node:stream/web` emits the union form
|
||||||
|
*
|
||||||
|
* `value?: T | undefined` is assignable from all three. A flat
|
||||||
|
* `{ value?: T; done: boolean }` is rejected by
|
||||||
|
* `exactOptionalPropertyTypes` because the present branches require
|
||||||
|
* `value: T`. Defining it as an explicit union avoids the trap.
|
||||||
|
*/
|
||||||
|
type MinimalReadResult<T> =
|
||||||
|
| { done: false; value: T }
|
||||||
|
| { done: true; value?: T | undefined };
|
||||||
|
|
||||||
|
interface MinimalReader<T> {
|
||||||
|
read(): Promise<MinimalReadResult<T>>;
|
||||||
cancel(reason?: unknown): Promise<void>;
|
cancel(reason?: unknown): Promise<void>;
|
||||||
releaseLock(): void;
|
releaseLock(): void;
|
||||||
}
|
}
|
||||||
|
|
||||||
function reconstructStream(
|
function reconstructStream(
|
||||||
prefix: Uint8Array[],
|
prefix: Uint8Array[],
|
||||||
reader: MinimalReader,
|
reader: MinimalReader<Uint8Array>,
|
||||||
): ReadableStream<Uint8Array> {
|
): ReadableStream<Uint8Array> {
|
||||||
let prefixIdx = 0;
|
let prefixIdx = 0;
|
||||||
return new ReadableStream<Uint8Array>({
|
return new ReadableStream<Uint8Array>({
|
||||||
|
|||||||
172
packages/shade-files/src/client/queue-drainer.ts
Normal file
172
packages/shade-files/src/client/queue-drainer.ts
Normal file
@@ -0,0 +1,172 @@
|
|||||||
|
/**
|
||||||
|
* Browser-side drainer for the pull-mode outbound queue.
|
||||||
|
*
|
||||||
|
* Background task that long-polls the server's `/queue` endpoint,
|
||||||
|
* decodes each event, and dispatches it into the consumer's Shade
|
||||||
|
* instance via `shade.acceptTransferEnvelope`. Same wire-shape as the
|
||||||
|
* server-to-server case where the engine receives chunks via direct
|
||||||
|
* HTTP POSTs — we just flip the direction so the browser pulls
|
||||||
|
* instead of accepts.
|
||||||
|
*/
|
||||||
|
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||||
|
|
||||||
|
export interface QueueDrainerOptions {
|
||||||
|
/**
|
||||||
|
* Server endpoint that hosts `transferQueueRoute()`. Typically:
|
||||||
|
* `https://server.example.com/api/v1/shade-files/queue`.
|
||||||
|
*/
|
||||||
|
outboundQueueUrl: string;
|
||||||
|
/** Peer the queue is pulled FROM (the server's address). */
|
||||||
|
peerAddress: string;
|
||||||
|
/** Address we identify ourselves with on the long-poll. */
|
||||||
|
senderAddress: string;
|
||||||
|
/** Optional `fetch` override. Default `globalThis.fetch`. */
|
||||||
|
fetch?: typeof globalThis.fetch;
|
||||||
|
/** Extra headers applied to every poll request. */
|
||||||
|
headers?: Record<string, string>;
|
||||||
|
/**
|
||||||
|
* Long-poll request timeout (server-side block). Default 30_000.
|
||||||
|
* Server clamps to its own `maxBlockMs` (default 55_000).
|
||||||
|
*/
|
||||||
|
blockMs?: number;
|
||||||
|
/**
|
||||||
|
* Backoff after a network error before re-polling. Default 2_000.
|
||||||
|
* Doubles up to `maxBackoffMs` on consecutive failures.
|
||||||
|
*/
|
||||||
|
initialBackoffMs?: number;
|
||||||
|
maxBackoffMs?: number;
|
||||||
|
/**
|
||||||
|
* Called when a poll cycle fails. Defaults to logging via `console.error`.
|
||||||
|
* Throwing from this hook does NOT stop the drainer — it backs off
|
||||||
|
* and retries.
|
||||||
|
*/
|
||||||
|
onError?: (err: unknown) => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface QueueDrainerHandle {
|
||||||
|
/** Stop the drainer. Pending fetch is aborted; the loop exits. */
|
||||||
|
stop(): void;
|
||||||
|
/** Promise that resolves once the drainer has fully stopped. */
|
||||||
|
stopped: Promise<void>;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface PolledEvent {
|
||||||
|
id: number;
|
||||||
|
timestampMs: number;
|
||||||
|
kind: 'envelope' | 'chunk';
|
||||||
|
bytesB64: string;
|
||||||
|
meta?: { streamId: string; laneId: number; seq: number };
|
||||||
|
}
|
||||||
|
|
||||||
|
interface PollResponseBody {
|
||||||
|
events: PolledEvent[];
|
||||||
|
nextSince: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
const DEFAULT_BLOCK_MS = 30_000;
|
||||||
|
const DEFAULT_INITIAL_BACKOFF_MS = 2_000;
|
||||||
|
const DEFAULT_MAX_BACKOFF_MS = 30_000;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Start a long-poll loop that drains queued envelopes + chunks from
|
||||||
|
* the server and dispatches them into the local Shade instance.
|
||||||
|
*
|
||||||
|
* Returns a handle the caller can use to stop the drainer when the
|
||||||
|
* `httpClient` is closed (e.g. on tab unload).
|
||||||
|
*/
|
||||||
|
export function startQueueDrainer(
|
||||||
|
shade: ShadeBridge,
|
||||||
|
options: QueueDrainerOptions,
|
||||||
|
): QueueDrainerHandle {
|
||||||
|
if (shade.acceptTransferEnvelope === undefined) {
|
||||||
|
throw new Error(
|
||||||
|
'startQueueDrainer: shade.acceptTransferEnvelope is required for pull-mode streams. The supplied ShadeBridge implementation must surface it.',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const accept = shade.acceptTransferEnvelope.bind(shade);
|
||||||
|
const fetchFn = options.fetch ?? globalThis.fetch.bind(globalThis);
|
||||||
|
const blockMs = options.blockMs ?? DEFAULT_BLOCK_MS;
|
||||||
|
const onError = options.onError ?? ((err: unknown) => console.error('[shade.files queue-drainer]', err));
|
||||||
|
const initialBackoffMs = options.initialBackoffMs ?? DEFAULT_INITIAL_BACKOFF_MS;
|
||||||
|
const maxBackoffMs = options.maxBackoffMs ?? DEFAULT_MAX_BACKOFF_MS;
|
||||||
|
const ac = new AbortController();
|
||||||
|
let stopped = false;
|
||||||
|
let resolveStopped!: () => void;
|
||||||
|
const stoppedPromise = new Promise<void>((r) => {
|
||||||
|
resolveStopped = r;
|
||||||
|
});
|
||||||
|
|
||||||
|
void (async () => {
|
||||||
|
let since = 0;
|
||||||
|
let backoff = initialBackoffMs;
|
||||||
|
while (!stopped) {
|
||||||
|
try {
|
||||||
|
const response = await fetchFn(options.outboundQueueUrl, {
|
||||||
|
method: 'POST',
|
||||||
|
signal: ac.signal,
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
'X-Shade-Sender-Address': options.senderAddress,
|
||||||
|
...(options.headers ?? {}),
|
||||||
|
},
|
||||||
|
body: JSON.stringify({ since, blockMs }),
|
||||||
|
});
|
||||||
|
if (!response.ok) {
|
||||||
|
throw new Error(`queue poll → ${response.status} ${response.statusText}`);
|
||||||
|
}
|
||||||
|
const body = (await response.json()) as PollResponseBody;
|
||||||
|
if (Array.isArray(body.events) && body.events.length > 0) {
|
||||||
|
for (const event of body.events) {
|
||||||
|
if (stopped) break;
|
||||||
|
try {
|
||||||
|
const bytes = base64ToBytes(event.bytesB64);
|
||||||
|
await accept(options.peerAddress, bytes);
|
||||||
|
} catch (err) {
|
||||||
|
// Per-event dispatch failure should not kill the loop —
|
||||||
|
// resume picks up missing chunks via @shade/transfer's
|
||||||
|
// built-in lane-resume protocol.
|
||||||
|
onError(err);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
since = typeof body.nextSince === 'number' ? body.nextSince : since;
|
||||||
|
backoff = initialBackoffMs;
|
||||||
|
} catch (err) {
|
||||||
|
if (stopped || ac.signal.aborted) break;
|
||||||
|
onError(err);
|
||||||
|
// Exponential backoff with jitter — caps at maxBackoffMs.
|
||||||
|
const jitter = Math.floor(Math.random() * Math.min(backoff, 1_000));
|
||||||
|
await new Promise<void>((r) => {
|
||||||
|
const t = setTimeout(r, backoff + jitter);
|
||||||
|
(t as unknown as { unref?: () => void }).unref?.();
|
||||||
|
ac.signal.addEventListener(
|
||||||
|
'abort',
|
||||||
|
() => {
|
||||||
|
clearTimeout(t);
|
||||||
|
r();
|
||||||
|
},
|
||||||
|
{ once: true },
|
||||||
|
);
|
||||||
|
});
|
||||||
|
backoff = Math.min(maxBackoffMs, backoff * 2);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
resolveStopped();
|
||||||
|
})();
|
||||||
|
|
||||||
|
return {
|
||||||
|
stop(): void {
|
||||||
|
if (stopped) return;
|
||||||
|
stopped = true;
|
||||||
|
ac.abort(new Error('queue drainer stopped'));
|
||||||
|
},
|
||||||
|
stopped: stoppedPromise,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function base64ToBytes(b64: string): Uint8Array {
|
||||||
|
const bin = atob(b64);
|
||||||
|
const out = new Uint8Array(bin.length);
|
||||||
|
for (let i = 0; i < bin.length; i++) out[i] = bin.charCodeAt(i);
|
||||||
|
return out;
|
||||||
|
}
|
||||||
@@ -102,7 +102,16 @@ export async function createClientStreamsBridge(
|
|||||||
const readStreamId = incoming.metadata.userMetadata?.[META_KEY_READ_STREAM_ID];
|
const readStreamId = incoming.metadata.userMetadata?.[META_KEY_READ_STREAM_ID];
|
||||||
if (readStreamId === undefined) return;
|
if (readStreamId === undefined) return;
|
||||||
|
|
||||||
const ts = new TransformStream<Uint8Array, Uint8Array>();
|
// Generous HWM so the receiver-side write loop (drainer →
|
||||||
|
// engine.receiveChunk → sink.write) doesn't stall on backpressure
|
||||||
|
// before the consumer's reader is wired up. The reader still
|
||||||
|
// applies its own backpressure once it's consuming, but we no
|
||||||
|
// longer race fs.read's await on stream-init against the consumer
|
||||||
|
// attaching its reader.
|
||||||
|
const ts = new TransformStream<Uint8Array, Uint8Array>(undefined, undefined, {
|
||||||
|
highWaterMark: 64,
|
||||||
|
size: (chunk?: Uint8Array) => (chunk === undefined ? 0 : 1),
|
||||||
|
});
|
||||||
let handle: TransferHandle;
|
let handle: TransferHandle;
|
||||||
try {
|
try {
|
||||||
handle = await incoming.accept({
|
handle = await incoming.accept({
|
||||||
@@ -142,13 +151,14 @@ export async function createClientStreamsBridge(
|
|||||||
}
|
}
|
||||||
|
|
||||||
parked.set(readStreamId, arrival);
|
parked.set(readStreamId, arrival);
|
||||||
setTimeout(() => {
|
const t = setTimeout(() => {
|
||||||
const stale = parked.get(readStreamId);
|
const stale = parked.get(readStreamId);
|
||||||
if (stale === arrival) {
|
if (stale === arrival) {
|
||||||
parked.delete(readStreamId);
|
parked.delete(readStreamId);
|
||||||
void handle.abort('rpc-timeout').catch(() => undefined);
|
void handle.abort('rpc-timeout').catch(() => undefined);
|
||||||
}
|
}
|
||||||
}, parkedReadTtlMs).unref?.();
|
}, parkedReadTtlMs);
|
||||||
|
(t as unknown as { unref?: () => void }).unref?.();
|
||||||
});
|
});
|
||||||
|
|
||||||
function cleanupWaiter(w: PendingReadWaiter): void {
|
function cleanupWaiter(w: PendingReadWaiter): void {
|
||||||
|
|||||||
@@ -183,6 +183,24 @@ export { MAX_SIGNATURE_AGE_MS } from './server/handler.js';
|
|||||||
export { createFilesNamespace } from './integration/files-namespace.js';
|
export { createFilesNamespace } from './integration/files-namespace.js';
|
||||||
export type { FilesNamespace } from './integration/files-namespace.js';
|
export type { FilesNamespace } from './integration/files-namespace.js';
|
||||||
|
|
||||||
|
// Request-response HTTP transport — for browser-style consumers
|
||||||
|
// (one HTTP POST per RPC, no inbound channel needed). See
|
||||||
|
// `docs/files.md § HTTP RPC`.
|
||||||
|
export { createFilesRpcRoute } from './server/rpc-route.js';
|
||||||
|
export type { FilesRpcRouteOptions } from './server/rpc-route.js';
|
||||||
|
export { createFilesHttpClient } from './client/http-client.js';
|
||||||
|
export type { FilesHttpClientOptions } from './client/http-client.js';
|
||||||
|
export { startQueueDrainer } from './client/queue-drainer.js';
|
||||||
|
export type {
|
||||||
|
QueueDrainerHandle,
|
||||||
|
QueueDrainerOptions,
|
||||||
|
} from './client/queue-drainer.js';
|
||||||
|
|
||||||
|
// Shared structural surface @shade/files needs from a Shade instance —
|
||||||
|
// exposed so consumers building custom Shade-shaped bridges can verify
|
||||||
|
// they implement every required member.
|
||||||
|
export type { ShadeBridge } from './integration/shade-bridge.js';
|
||||||
|
|
||||||
// Integration helpers — wire handler + pending registry onto a channel
|
// Integration helpers — wire handler + pending registry onto a channel
|
||||||
export { attachFileHandler } from './integration/wire-server.js';
|
export { attachFileHandler } from './integration/wire-server.js';
|
||||||
export { attachClientRouting } from './integration/wire-client.js';
|
export { attachClientRouting } from './integration/wire-client.js';
|
||||||
|
|||||||
@@ -4,13 +4,16 @@
|
|||||||
* so a single Shade can simultaneously serve files AND consume them from
|
* so a single Shade can simultaneously serve files AND consume them from
|
||||||
* peers without paying the setup cost twice.
|
* peers without paying the setup cost twice.
|
||||||
*/
|
*/
|
||||||
import type { Shade } from '@shade/sdk';
|
import type { Hono } from 'hono';
|
||||||
|
import type { ShadeBridge } from './shade-bridge.js';
|
||||||
import {
|
import {
|
||||||
attachClientRouting,
|
attachClientRouting,
|
||||||
attachFileHandler,
|
attachFileHandler,
|
||||||
createClientStreamsBridge,
|
createClientStreamsBridge,
|
||||||
createFileClient,
|
createFileClient,
|
||||||
createFileHandler,
|
createFileHandler,
|
||||||
|
createFilesHttpClient,
|
||||||
|
createFilesRpcRoute,
|
||||||
createServerStreamsBridge,
|
createServerStreamsBridge,
|
||||||
PendingRpcRegistry,
|
PendingRpcRegistry,
|
||||||
ShadeFileRpcChannel,
|
ShadeFileRpcChannel,
|
||||||
@@ -19,22 +22,79 @@ import {
|
|||||||
type FileClient,
|
type FileClient,
|
||||||
type FileHandler,
|
type FileHandler,
|
||||||
type FileHandlerConfig,
|
type FileHandlerConfig,
|
||||||
|
type FilesHttpClientOptions,
|
||||||
|
type FilesRpcRouteOptions,
|
||||||
type ServerStreamsBridge,
|
type ServerStreamsBridge,
|
||||||
} from '../index.js';
|
} from '../index.js';
|
||||||
import { IdempotencyCache } from '../server/idempotency-cache.js';
|
import { IdempotencyCache } from '../server/idempotency-cache.js';
|
||||||
|
|
||||||
|
export interface ServeOptions {
|
||||||
|
/**
|
||||||
|
* Skip the streams bridge setup. Required for deployments that only
|
||||||
|
* use the HTTP RPC route ({@link FilesNamespace.rpcRoute}) — those
|
||||||
|
* deployments don't need to configure `@shade/transfer` because the
|
||||||
|
* RPC route only services inline payloads (≤ 256 KiB). Without this
|
||||||
|
* flag, `serve()` calls `createServerStreamsBridge(shade)` which
|
||||||
|
* eagerly instantiates the transfer engine and fails when
|
||||||
|
* `configureTransfers({ resolveBaseUrl })` has not been called.
|
||||||
|
*
|
||||||
|
* Default: `false` (build the streams bridge — full server-to-server
|
||||||
|
* stack with streamed reads/writes).
|
||||||
|
*/
|
||||||
|
inlineOnly?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
export interface FilesNamespace {
|
export interface FilesNamespace {
|
||||||
/**
|
/**
|
||||||
* Register a file handler. Throws if a handler is already attached on
|
* Register a file handler. Throws if a handler is already attached on
|
||||||
* this Shade — only one server per Shade. The returned function detaches
|
* this Shade — only one server per Shade. The returned function detaches
|
||||||
* the handler and tears down its idempotency / retention timers.
|
* the handler and tears down its idempotency / retention timers.
|
||||||
|
*
|
||||||
|
* Pass `{ inlineOnly: true }` for HTTP-RPC-only deployments to skip
|
||||||
|
* the streams-bridge setup (and the implied `configureTransfers`
|
||||||
|
* pre-condition).
|
||||||
*/
|
*/
|
||||||
serve(handler: FileHandlerConfig): Promise<() => Promise<void>>;
|
serve(
|
||||||
|
handler: FileHandlerConfig,
|
||||||
|
options?: ServeOptions,
|
||||||
|
): Promise<() => Promise<void>>;
|
||||||
/**
|
/**
|
||||||
* Build a typed file client for `peer`. Multiple concurrent clients to
|
* Build a typed file client for `peer`. Multiple concurrent clients to
|
||||||
* different peers share the same channel + streams bridge.
|
* different peers share the same channel + streams bridge.
|
||||||
|
*
|
||||||
|
* Use this for **server-to-server** deployments where both peers can
|
||||||
|
* receive inbound HTTP. For browser clients (no inbound listener),
|
||||||
|
* use {@link httpClient} instead.
|
||||||
*/
|
*/
|
||||||
client(peer: string, opts?: Omit<CreateFileClientOptions, 'streamsBridge'>): Promise<FileClient>;
|
client(peer: string, opts?: Omit<CreateFileClientOptions, 'streamsBridge'>): Promise<FileClient>;
|
||||||
|
/**
|
||||||
|
* Build a request-response `FileClient` for browser-style consumers.
|
||||||
|
* Each RPC is one HTTP POST to the supplied `rpcUrl`; the encrypted
|
||||||
|
* response rides back in the same response body. No inbound channel
|
||||||
|
* required on the client side.
|
||||||
|
*
|
||||||
|
* Inline payloads only (≤ 256 KiB). Streamed reads/writes throw a
|
||||||
|
* clear error directing callers to {@link client} instead.
|
||||||
|
*
|
||||||
|
* Pre-condition: the session for `peer` must already be established
|
||||||
|
* (typically via `shade.initSessionFromBundle(peer, bundle)`).
|
||||||
|
*/
|
||||||
|
httpClient(peer: string, opts: FilesHttpClientOptions): FileClient;
|
||||||
|
/**
|
||||||
|
* Mount the server-side request-response RPC route. Returns a Hono
|
||||||
|
* app exposing `POST /rpc` that accepts encrypted file-RPC envelopes
|
||||||
|
* and returns encrypted responses in the same HTTP roundtrip.
|
||||||
|
*
|
||||||
|
* Mount under any base path:
|
||||||
|
* ```ts
|
||||||
|
* app.route('/api/v1/shade-files', shade.files.rpcRoute());
|
||||||
|
* ```
|
||||||
|
*
|
||||||
|
* Requires `shade.files.serve(...)` to have been called first —
|
||||||
|
* the route dispatches incoming requests through the attached
|
||||||
|
* handler.
|
||||||
|
*/
|
||||||
|
rpcRoute(opts?: FilesRpcRouteOptions): Hono;
|
||||||
/** Tear down channel + bridges. After destroy(), serve()/client() throw. */
|
/** Tear down channel + bridges. After destroy(), serve()/client() throw. */
|
||||||
destroy(): Promise<void>;
|
destroy(): Promise<void>;
|
||||||
}
|
}
|
||||||
@@ -54,7 +114,7 @@ interface NamespaceState {
|
|||||||
* Construct a `FilesNamespace` bound to a Shade instance. The SDK's
|
* Construct a `FilesNamespace` bound to a Shade instance. The SDK's
|
||||||
* `Shade.files` getter calls this lazily and memoizes the result.
|
* `Shade.files` getter calls this lazily and memoizes the result.
|
||||||
*/
|
*/
|
||||||
export function createFilesNamespace(shade: Shade): FilesNamespace {
|
export function createFilesNamespace(shade: ShadeBridge): FilesNamespace {
|
||||||
const state: NamespaceState = {
|
const state: NamespaceState = {
|
||||||
channel: new ShadeFileRpcChannel(shade),
|
channel: new ShadeFileRpcChannel(shade),
|
||||||
pending: new PendingRpcRegistry(),
|
pending: new PendingRpcRegistry(),
|
||||||
@@ -71,24 +131,39 @@ export function createFilesNamespace(shade: Shade): FilesNamespace {
|
|||||||
}
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
async serve(handlerConfig) {
|
async serve(handlerConfig, options = {}) {
|
||||||
ensureAlive();
|
ensureAlive();
|
||||||
if (state.serverHandler !== null) {
|
if (state.serverHandler !== null) {
|
||||||
throw new Error('FilesNamespace: a handler is already registered (one per Shade)');
|
throw new Error('FilesNamespace: a handler is already registered (one per Shade)');
|
||||||
}
|
}
|
||||||
// Lazy server-side streams bridge.
|
// Lazy server-side streams bridge — skip when the deployment is
|
||||||
if (state.serverBridge === null) {
|
// HTTP-RPC-only and does not need `@shade/transfer` wired up.
|
||||||
|
if (!options.inlineOnly && state.serverBridge === null) {
|
||||||
state.serverBridge = await createServerStreamsBridge(shade);
|
state.serverBridge = await createServerStreamsBridge(shade);
|
||||||
}
|
}
|
||||||
const inheritedObservability = shade.getObservability?.();
|
const inheritedObservability = shade.getObservability?.();
|
||||||
const handler = createFileHandler(shade, {
|
const handler = createFileHandler(shade, {
|
||||||
...handlerConfig,
|
...handlerConfig,
|
||||||
streamsBridge: state.serverBridge,
|
...(state.serverBridge !== null
|
||||||
|
? { streamsBridge: state.serverBridge }
|
||||||
|
: {}),
|
||||||
...(handlerConfig.observability === undefined && inheritedObservability !== undefined
|
...(handlerConfig.observability === undefined && inheritedObservability !== undefined
|
||||||
? { observability: inheritedObservability }
|
? { observability: inheritedObservability }
|
||||||
: {}),
|
: {}),
|
||||||
});
|
});
|
||||||
const detach = attachFileHandler(state.channel, handler);
|
// In inlineOnly mode, the rpc-route is the sole inbound path —
|
||||||
|
// do NOT also subscribe the channel's onMessage handler to this
|
||||||
|
// file handler, because that would cause every incoming request
|
||||||
|
// to be dispatched twice (once by the rpc-route's direct call,
|
||||||
|
// once by the channel's onMessage handler) and the channel-side
|
||||||
|
// response would attempt an outbound POST via
|
||||||
|
// `deliverControlEnvelope`, which is exactly the path that fails
|
||||||
|
// for browser clients.
|
||||||
|
const detach: () => void = options.inlineOnly
|
||||||
|
? () => {
|
||||||
|
/* no channel subscription to detach */
|
||||||
|
}
|
||||||
|
: attachFileHandler(state.channel, handler);
|
||||||
state.serverHandler = handler;
|
state.serverHandler = handler;
|
||||||
state.serverDetach = detach;
|
state.serverDetach = detach;
|
||||||
|
|
||||||
@@ -131,6 +206,21 @@ export function createFilesNamespace(shade: Shade): FilesNamespace {
|
|||||||
});
|
});
|
||||||
},
|
},
|
||||||
|
|
||||||
|
httpClient(peer, opts) {
|
||||||
|
ensureAlive();
|
||||||
|
return createFilesHttpClient(shade, peer, opts);
|
||||||
|
},
|
||||||
|
|
||||||
|
rpcRoute(opts = {}) {
|
||||||
|
ensureAlive();
|
||||||
|
if (state.serverHandler === null) {
|
||||||
|
throw new Error(
|
||||||
|
'FilesNamespace.rpcRoute(): no handler attached. Call shade.files.serve(...) before mounting the RPC route.',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return createFilesRpcRoute(shade, state.serverHandler, opts);
|
||||||
|
},
|
||||||
|
|
||||||
async destroy() {
|
async destroy() {
|
||||||
if (state.destroyed) return;
|
if (state.destroyed) return;
|
||||||
state.destroyed = true;
|
state.destroyed = true;
|
||||||
|
|||||||
94
packages/shade-files/src/integration/shade-bridge.ts
Normal file
94
packages/shade-files/src/integration/shade-bridge.ts
Normal file
@@ -0,0 +1,94 @@
|
|||||||
|
/**
|
||||||
|
* Structural surface @shade/files needs from a Shade instance.
|
||||||
|
*
|
||||||
|
* Defining this locally — instead of `import type { Shade } from '@shade/sdk'`
|
||||||
|
* — breaks the @shade/sdk ↔ @shade/files dependency cycle. Without this
|
||||||
|
* break, a consumer that installs @shade/sdk from a registry ends up with
|
||||||
|
* two distinct `Shade` classes in `node_modules` (one from
|
||||||
|
* `@shade/sdk/node_modules/@shade/files/.../Shade`, one from
|
||||||
|
* `@shade/sdk/Shade`). TypeScript treats them as nominally different types,
|
||||||
|
* raising `this is not assignable to Shade` from inside SDK methods that
|
||||||
|
* pass `this` into `createFilesNamespace`.
|
||||||
|
*
|
||||||
|
* The Shade class structurally implements every member listed below, so
|
||||||
|
* `createFilesNamespace(this)` from the SDK side compiles regardless of
|
||||||
|
* how many copies of @shade/sdk a consumer's package manager installs.
|
||||||
|
*
|
||||||
|
* Member signatures match Shade's exactly so this is a structural
|
||||||
|
* subtype, not a parallel API.
|
||||||
|
*/
|
||||||
|
import type { ShadeEnvelope } from '@shade/core';
|
||||||
|
import type {
|
||||||
|
IncomingTransfer,
|
||||||
|
TransferHandle,
|
||||||
|
TransferOptions,
|
||||||
|
} from '@shade/transfer';
|
||||||
|
import type { ObservabilityHook } from '@shade/observability';
|
||||||
|
|
||||||
|
export interface ShadeBridge {
|
||||||
|
/** Address that names this Shade instance to peers. */
|
||||||
|
readonly myAddress: string;
|
||||||
|
|
||||||
|
/** Encrypt + send `plaintext` to `peer`; returns the wire envelope. */
|
||||||
|
send(peer: string, plaintext: string): Promise<ShadeEnvelope>;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Decrypt an inbound envelope from `peer` and return the plaintext.
|
||||||
|
* Used by the request-response RPC route on the server side.
|
||||||
|
*/
|
||||||
|
receive(peer: string, envelope: ShadeEnvelope): Promise<string>;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Subscribe to incoming ratchet plaintext. Returns an unsubscribe.
|
||||||
|
* Handlers may be sync or async; async handlers are awaited in
|
||||||
|
* registration order.
|
||||||
|
*/
|
||||||
|
onMessage(
|
||||||
|
handler: (from: string, plaintext: string) => void | Promise<void>,
|
||||||
|
): () => void;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Upload bytes via the SDK's transfer engine. Required when the bridge
|
||||||
|
* is used with `streams` content I/O (read/write > 256 KiB).
|
||||||
|
*/
|
||||||
|
upload(opts: TransferOptions): Promise<TransferHandle>;
|
||||||
|
|
||||||
|
/** Subscribe to incoming transfers initiated by a peer. */
|
||||||
|
onIncomingTransfer(
|
||||||
|
handler: (incoming: IncomingTransfer) => void | Promise<void>,
|
||||||
|
): Promise<() => void>;
|
||||||
|
|
||||||
|
/** Fingerprint accessor for the trust-gate hooks. */
|
||||||
|
getFingerprintFor(peer: string): Promise<string>;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Optional inheritable observability bus. Files inherits the bus when
|
||||||
|
* the SDK passes one in via the namespace; otherwise files runs without
|
||||||
|
* observability hooks.
|
||||||
|
*/
|
||||||
|
getObservability?(): ObservabilityHook | undefined;
|
||||||
|
|
||||||
|
/** Optional control-envelope passthrough used by the WebRTC bridge. */
|
||||||
|
deliverControlEnvelope?(peer: string, envelope: ShadeEnvelope): Promise<void>;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Hand a freshly-decoded wire envelope (control or chunk) to the
|
||||||
|
* transfer engine. Required by the pull-mode HTTP client when it
|
||||||
|
* drains queued events from the server: each polled chunk / control
|
||||||
|
* envelope is dispatched here so the engine sees it just as if it
|
||||||
|
* had arrived via an HTTP POST on `/v1/transfer/...`.
|
||||||
|
*/
|
||||||
|
acceptTransferEnvelope?(from: string, env: ShadeEnvelope | Uint8Array): Promise<void>;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Configure the transfer stack. Called by the pull-mode HTTP client
|
||||||
|
* to point the browser's outgoing chunks + control envelopes at the
|
||||||
|
* server's transferQueueRoute mount. Optional because the
|
||||||
|
* server-to-server path uses a separate, app-driven configuration.
|
||||||
|
*/
|
||||||
|
configureTransfers?(opts: {
|
||||||
|
resolveBaseUrl?: (peerAddress: string) => Promise<string>;
|
||||||
|
transport?: unknown;
|
||||||
|
envelopeTransport?: unknown;
|
||||||
|
}): void;
|
||||||
|
}
|
||||||
126
packages/shade-files/src/protocol/rpc-builder.ts
Normal file
126
packages/shade-files/src/protocol/rpc-builder.ts
Normal file
@@ -0,0 +1,126 @@
|
|||||||
|
/**
|
||||||
|
* Shared RPC-request construction.
|
||||||
|
*
|
||||||
|
* Both the channel-based `FileClient` (`createFileClient`) and the
|
||||||
|
* HTTP-based `FilesHttpClient` build identical `RpcRequest` envelopes
|
||||||
|
* — they differ only in *transport* (channel.send → ratchet via
|
||||||
|
* Shade.send/onMessage vs HTTP POST → ratchet via single
|
||||||
|
* request-response). This module is the single source of truth for
|
||||||
|
* the wire shape so the two clients can never drift.
|
||||||
|
*/
|
||||||
|
import type { ZodTypeAny } from 'zod';
|
||||||
|
import {
|
||||||
|
KIND_DELETE_V1,
|
||||||
|
KIND_GET_THUMBNAIL_V1,
|
||||||
|
KIND_LIST_V1,
|
||||||
|
KIND_MKDIR_V1,
|
||||||
|
KIND_MOVE_V1,
|
||||||
|
KIND_READ_V1,
|
||||||
|
KIND_STAT_V1,
|
||||||
|
KIND_WRITE_V1,
|
||||||
|
MUTATION_OPS,
|
||||||
|
type StandardOp,
|
||||||
|
} from './kinds.js';
|
||||||
|
import { generateIdempotencyKey, generateRequestId } from './correlate.js';
|
||||||
|
import { canonicalRpcBytes, hashArgs } from './canonical.js';
|
||||||
|
import type { RpcRequest } from '../schemas/envelope.js';
|
||||||
|
|
||||||
|
export const KIND_BY_OP: Record<StandardOp, string> = {
|
||||||
|
list: KIND_LIST_V1,
|
||||||
|
stat: KIND_STAT_V1,
|
||||||
|
mkdir: KIND_MKDIR_V1,
|
||||||
|
delete: KIND_DELETE_V1,
|
||||||
|
move: KIND_MOVE_V1,
|
||||||
|
read: KIND_READ_V1,
|
||||||
|
write: KIND_WRITE_V1,
|
||||||
|
getThumbnail: KIND_GET_THUMBNAIL_V1,
|
||||||
|
};
|
||||||
|
|
||||||
|
export type SignRequest = (canonicalBytes: Uint8Array) => Promise<string> | string;
|
||||||
|
|
||||||
|
export interface BuildRpcRequestOptions {
|
||||||
|
/** Address that this RPC call originates from. */
|
||||||
|
senderAddress: string;
|
||||||
|
/** RPC kind — `KIND_*_V1` constants for standard ops, `KIND_CUSTOM_V1` otherwise. */
|
||||||
|
kind: string;
|
||||||
|
/** Op classifier so mutations get an auto-generated idempotency key. */
|
||||||
|
op: StandardOp | 'custom';
|
||||||
|
/** Validated args object. The caller is responsible for `Zod.parse(args)`. */
|
||||||
|
args: unknown;
|
||||||
|
/** Caller-supplied idempotency key. Mutations get one auto-generated. */
|
||||||
|
idempotencyKey?: string;
|
||||||
|
/** Optional Ed25519-style signer over the canonical bytes. */
|
||||||
|
signRequest?: SignRequest;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Build a single `RpcRequest` envelope. Generates `id`, `signedAt`,
|
||||||
|
* idempotency key (mutations only), and the canonical signature.
|
||||||
|
*/
|
||||||
|
export async function buildRpcRequest(
|
||||||
|
options: BuildRpcRequestOptions,
|
||||||
|
): Promise<RpcRequest> {
|
||||||
|
const { senderAddress, kind, op, args, signRequest } = options;
|
||||||
|
const requestId = generateRequestId();
|
||||||
|
const isMutation = MUTATION_OPS.has(op);
|
||||||
|
const idempotencyKey =
|
||||||
|
options.idempotencyKey ?? (isMutation ? generateIdempotencyKey() : undefined);
|
||||||
|
const signedAt = Date.now();
|
||||||
|
|
||||||
|
let sig = 'unsigned';
|
||||||
|
if (signRequest !== undefined) {
|
||||||
|
const canonical = canonicalRpcBytes({
|
||||||
|
address: senderAddress,
|
||||||
|
signedAt,
|
||||||
|
kind,
|
||||||
|
id: requestId,
|
||||||
|
argsHash: hashArgs(args),
|
||||||
|
});
|
||||||
|
sig = await signRequest(canonical);
|
||||||
|
}
|
||||||
|
|
||||||
|
const env: RpcRequest = {
|
||||||
|
kind,
|
||||||
|
id: requestId,
|
||||||
|
args,
|
||||||
|
...(idempotencyKey !== undefined ? { idempotencyKey } : {}),
|
||||||
|
sig,
|
||||||
|
signedAt,
|
||||||
|
};
|
||||||
|
return env;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Helper for the typed standard ops: validates args via the supplied Zod
|
||||||
|
* schema, calls {@link buildRpcRequest}, and forwards the result.
|
||||||
|
*
|
||||||
|
* The HTTP and channel clients both call this from each op-method to
|
||||||
|
* guarantee identical wire-shape. Custom ops use {@link buildRpcRequest}
|
||||||
|
* directly because they ship `KIND_CUSTOM_V1` plus runtime-validated args.
|
||||||
|
*/
|
||||||
|
export async function buildStandardRpcRequest<TArgs>(
|
||||||
|
schema: { parse(input: unknown): TArgs },
|
||||||
|
rawArgs: unknown,
|
||||||
|
options: Omit<BuildRpcRequestOptions, 'kind' | 'op' | 'args'> & {
|
||||||
|
op: StandardOp;
|
||||||
|
},
|
||||||
|
): Promise<{ request: RpcRequest; args: TArgs }> {
|
||||||
|
const args = schema.parse(rawArgs) as TArgs;
|
||||||
|
const kind = KIND_BY_OP[options.op];
|
||||||
|
const request = await buildRpcRequest({
|
||||||
|
...options,
|
||||||
|
kind,
|
||||||
|
op: options.op,
|
||||||
|
args,
|
||||||
|
});
|
||||||
|
return { request, args };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Validate a returned RpcResponse `result` against the supplied schema.
|
||||||
|
* Centralised so the HTTP and channel clients fail-loudly with the same
|
||||||
|
* error type when a server returns an off-spec response.
|
||||||
|
*/
|
||||||
|
export function parseResult<T>(schema: ZodTypeAny, raw: unknown): T {
|
||||||
|
return schema.parse(raw) as T;
|
||||||
|
}
|
||||||
@@ -1,17 +1,23 @@
|
|||||||
import React, { createContext, useContext, useMemo } from 'react';
|
import React, { createContext, useContext, useMemo } from 'react';
|
||||||
import type { Shade } from '@shade/sdk';
|
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||||
import type { FilesNamespace } from '../integration/files-namespace.js';
|
import type { FilesNamespace } from '../integration/files-namespace.js';
|
||||||
|
|
||||||
export interface ShadeFilesContextValue {
|
export interface ShadeFilesContextValue {
|
||||||
shade: Shade;
|
shade: ShadeBridge;
|
||||||
files: FilesNamespace;
|
files: FilesNamespace;
|
||||||
}
|
}
|
||||||
|
|
||||||
const ShadeFilesContext = createContext<ShadeFilesContextValue | null>(null);
|
const ShadeFilesContext = createContext<ShadeFilesContextValue | null>(null);
|
||||||
|
|
||||||
export interface ShadeFilesProviderProps {
|
export interface ShadeFilesProviderProps {
|
||||||
/** Initialized `Shade` instance. `files` namespace is read off it lazily. */
|
/** Initialized `Shade` instance (or any `ShadeBridge`-shaped object). */
|
||||||
shade: Shade;
|
shade: ShadeBridge;
|
||||||
|
/**
|
||||||
|
* The `FilesNamespace` to expose to children. Pass `shade.files` from
|
||||||
|
* `@shade/sdk`, or a `createFilesNamespace(...)` result for tests /
|
||||||
|
* custom bridges.
|
||||||
|
*/
|
||||||
|
files: FilesNamespace;
|
||||||
children: React.ReactNode;
|
children: React.ReactNode;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -20,8 +26,8 @@ export interface ShadeFilesProviderProps {
|
|||||||
* `<ShadeRuntimeProvider>` in `@shade/widgets` so file-RPC consumers
|
* `<ShadeRuntimeProvider>` in `@shade/widgets` so file-RPC consumers
|
||||||
* don't pull in the widget tree.
|
* don't pull in the widget tree.
|
||||||
*/
|
*/
|
||||||
export function ShadeFilesProvider({ shade, children }: ShadeFilesProviderProps): React.ReactElement {
|
export function ShadeFilesProvider({ shade, files, children }: ShadeFilesProviderProps): React.ReactElement {
|
||||||
const value = useMemo<ShadeFilesContextValue>(() => ({ shade, files: shade.files }), [shade]);
|
const value = useMemo<ShadeFilesContextValue>(() => ({ shade, files }), [shade, files]);
|
||||||
return React.createElement(ShadeFilesContext.Provider, { value }, children);
|
return React.createElement(ShadeFilesContext.Provider, { value }, children);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
import type { Shade } from '@shade/sdk';
|
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||||
import {
|
import {
|
||||||
encodeEnvelope,
|
encodeEnvelope,
|
||||||
looksLikeFileEnvelope,
|
looksLikeFileEnvelope,
|
||||||
@@ -35,7 +35,7 @@ export class ShadeFileRpcChannel {
|
|||||||
private readonly unsubscribe: () => void;
|
private readonly unsubscribe: () => void;
|
||||||
private destroyed = false;
|
private destroyed = false;
|
||||||
|
|
||||||
constructor(private readonly shade: Shade) {
|
constructor(private readonly shade: ShadeBridge) {
|
||||||
this.unsubscribe = shade.onMessage(async (from, plaintext) => {
|
this.unsubscribe = shade.onMessage(async (from, plaintext) => {
|
||||||
if (!looksLikeFileEnvelope(plaintext)) return;
|
if (!looksLikeFileEnvelope(plaintext)) return;
|
||||||
const classified = tryParseEnvelope(plaintext);
|
const classified = tryParseEnvelope(plaintext);
|
||||||
@@ -72,6 +72,11 @@ export class ShadeFileRpcChannel {
|
|||||||
if (this.destroyed) throw new Error('ShadeFileRpcChannel: destroyed');
|
if (this.destroyed) throw new Error('ShadeFileRpcChannel: destroyed');
|
||||||
const plaintext = encodeEnvelope(envelope);
|
const plaintext = encodeEnvelope(envelope);
|
||||||
const ratchetEnvelope = await this.shade.send(peerAddress, plaintext);
|
const ratchetEnvelope = await this.shade.send(peerAddress, plaintext);
|
||||||
|
if (this.shade.deliverControlEnvelope === undefined) {
|
||||||
|
throw new Error(
|
||||||
|
'ShadeFileRpcChannel: shade.deliverControlEnvelope is required — call shade.configureTransfers({ resolveBaseUrl }) before using the files namespace.',
|
||||||
|
);
|
||||||
|
}
|
||||||
await this.shade.deliverControlEnvelope(peerAddress, ratchetEnvelope);
|
await this.shade.deliverControlEnvelope(peerAddress, ratchetEnvelope);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
import type { Shade } from '@shade/sdk';
|
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||||
import type { StandardOp } from '../protocol/kinds.js';
|
import type { StandardOp } from '../protocol/kinds.js';
|
||||||
|
|
||||||
export type OpKind = StandardOp | `custom:${string}`;
|
export type OpKind = StandardOp | `custom:${string}`;
|
||||||
@@ -42,7 +42,7 @@ export function buildOpContext<TArgs>(args: {
|
|||||||
signal: AbortSignal;
|
signal: AbortSignal;
|
||||||
idempotencyKey: string | undefined;
|
idempotencyKey: string | undefined;
|
||||||
attemptNumber: number;
|
attemptNumber: number;
|
||||||
shade: Shade;
|
shade: ShadeBridge;
|
||||||
}): OpContext<TArgs> {
|
}): OpContext<TArgs> {
|
||||||
return {
|
return {
|
||||||
op: args.op,
|
op: args.op,
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
import type { Shade } from '@shade/sdk';
|
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||||
import type { ZodTypeAny } from 'zod';
|
import type { ZodTypeAny } from 'zod';
|
||||||
import {
|
import {
|
||||||
MUTATION_OPS,
|
MUTATION_OPS,
|
||||||
@@ -215,7 +215,7 @@ const OP_SCHEMAS: Record<StandardOp, OpSchemaPair> = {
|
|||||||
* via `Shade.files.serve(...)` in the SDK).
|
* via `Shade.files.serve(...)` in the SDK).
|
||||||
*/
|
*/
|
||||||
export function createFileHandler(
|
export function createFileHandler(
|
||||||
shade: Shade,
|
shade: ShadeBridge,
|
||||||
config: FileHandlerConfig,
|
config: FileHandlerConfig,
|
||||||
): FileHandler {
|
): FileHandler {
|
||||||
const idempotency = new IdempotencyCache(config.idempotency);
|
const idempotency = new IdempotencyCache(config.idempotency);
|
||||||
|
|||||||
218
packages/shade-files/src/server/rpc-route.ts
Normal file
218
packages/shade-files/src/server/rpc-route.ts
Normal file
@@ -0,0 +1,218 @@
|
|||||||
|
/**
|
||||||
|
* Request-response RPC route for `@shade/files`.
|
||||||
|
*
|
||||||
|
* Mounts a single `POST /rpc` Hono endpoint that accepts an encrypted
|
||||||
|
* `RpcRequest` envelope, dispatches it through the file handler, and
|
||||||
|
* returns the encrypted `RpcResponse` (or `RpcError`) envelope in the
|
||||||
|
* SAME HTTP response.
|
||||||
|
*
|
||||||
|
* This is the browser-friendly transport: the server never needs to
|
||||||
|
* make outbound calls back to the client, so a browser tab — which
|
||||||
|
* cannot host an HTTP server — can fully consume `@shade/files`.
|
||||||
|
*
|
||||||
|
* ### Wire contract
|
||||||
|
*
|
||||||
|
* Request:
|
||||||
|
* ```
|
||||||
|
* POST <mount>/rpc HTTP/1.1
|
||||||
|
* Content-Type: application/octet-stream
|
||||||
|
* X-Shade-Sender-Address: <peer address>
|
||||||
|
*
|
||||||
|
* <wire-encoded ShadeEnvelope (0x01 PreKeyMessage or 0x02 RatchetMessage)
|
||||||
|
* containing JSON-encoded RpcRequest>
|
||||||
|
* ```
|
||||||
|
*
|
||||||
|
* Response (success):
|
||||||
|
* ```
|
||||||
|
* 200 OK
|
||||||
|
* Content-Type: application/octet-stream
|
||||||
|
*
|
||||||
|
* <wire-encoded ShadeEnvelope (0x02 RatchetMessage)
|
||||||
|
* containing JSON-encoded RpcResponse | RpcError>
|
||||||
|
* ```
|
||||||
|
*
|
||||||
|
* Response (transport-level failure — no session, undecryptable, etc.):
|
||||||
|
* ```
|
||||||
|
* 4xx
|
||||||
|
* Content-Type: application/json
|
||||||
|
*
|
||||||
|
* { "error": "..." }
|
||||||
|
* ```
|
||||||
|
*
|
||||||
|
* ### Symmetry with shade-auth-middleware
|
||||||
|
*
|
||||||
|
* The shape mirrors `@shade/server`'s shade-auth-middleware: an
|
||||||
|
* encrypted envelope rides the request body, the server decrypts via
|
||||||
|
* the existing ratchet session, performs the protected operation,
|
||||||
|
* and returns an encrypted envelope in the response. No bidirectional
|
||||||
|
* channel required.
|
||||||
|
*
|
||||||
|
* @see {@link createFilesHttpClient} for the matching browser client.
|
||||||
|
*/
|
||||||
|
import { Hono } from 'hono';
|
||||||
|
import { decodeEnvelope, encodeEnvelope } from '@shade/proto';
|
||||||
|
import type { ShadeBridge } from '../integration/shade-bridge.js';
|
||||||
|
import {
|
||||||
|
encodeEnvelope as encodeRpcEnvelope,
|
||||||
|
tryParseEnvelope,
|
||||||
|
} from '../protocol/envelope-codec.js';
|
||||||
|
import type { FileHandler } from './handler.js';
|
||||||
|
import type { RpcError, RpcRequest, RpcResponse } from '../schemas/envelope.js';
|
||||||
|
import { KIND_ERROR_V1 } from '../protocol/kinds.js';
|
||||||
|
|
||||||
|
export interface FilesRpcRouteOptions {
|
||||||
|
/**
|
||||||
|
* Maximum request body size in bytes. Default 1 MiB. Inline payloads
|
||||||
|
* are capped at 256 KiB by the protocol; the headroom is for
|
||||||
|
* custom-op payloads and base64 inflation.
|
||||||
|
*/
|
||||||
|
maxBodyBytes?: number;
|
||||||
|
/**
|
||||||
|
* Allow this server to accept the very first message (PreKeyMessage,
|
||||||
|
* `0x01`) over the RPC route. Disabled by default — most browser
|
||||||
|
* clients establish a session via `shade.initSessionFromBundle`
|
||||||
|
* before the first RPC. Enable when you want the RPC route to also
|
||||||
|
* be the X3DH carrier (uncommon but supported).
|
||||||
|
*/
|
||||||
|
acceptFirstMessage?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
const DEFAULT_MAX_BODY_BYTES = 1 * 1024 * 1024;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Build a Hono app with a single `POST /rpc` route. Mount under any
|
||||||
|
* base path: `app.route('/api/v1/shade-files', shade.files.rpcRoute())`.
|
||||||
|
*
|
||||||
|
* The `handler` must already be attached (typically via
|
||||||
|
* `shade.files.serve(handlerConfig)`); this route only ships the
|
||||||
|
* transport — it does not register a new file handler.
|
||||||
|
*/
|
||||||
|
export function createFilesRpcRoute(
|
||||||
|
shade: ShadeBridge,
|
||||||
|
handler: FileHandler,
|
||||||
|
options: FilesRpcRouteOptions = {},
|
||||||
|
): Hono {
|
||||||
|
const maxBodyBytes = options.maxBodyBytes ?? DEFAULT_MAX_BODY_BYTES;
|
||||||
|
const app = new Hono();
|
||||||
|
|
||||||
|
app.post('/rpc', async (c) => {
|
||||||
|
const senderAddress = c.req.header('X-Shade-Sender-Address');
|
||||||
|
if (senderAddress === undefined || senderAddress === '') {
|
||||||
|
return c.json({ error: 'missing X-Shade-Sender-Address header' }, 400);
|
||||||
|
}
|
||||||
|
|
||||||
|
const contentLengthHeader = c.req.header('Content-Length');
|
||||||
|
if (contentLengthHeader !== undefined) {
|
||||||
|
const contentLength = Number.parseInt(contentLengthHeader, 10);
|
||||||
|
if (Number.isFinite(contentLength) && contentLength > maxBodyBytes) {
|
||||||
|
return c.json(
|
||||||
|
{ error: `body exceeds maxBodyBytes (${contentLength} > ${maxBodyBytes})` },
|
||||||
|
413,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let bodyBytes: Uint8Array;
|
||||||
|
try {
|
||||||
|
const ab = await c.req.arrayBuffer();
|
||||||
|
if (ab.byteLength > maxBodyBytes) {
|
||||||
|
return c.json(
|
||||||
|
{ error: `body exceeds maxBodyBytes (${ab.byteLength} > ${maxBodyBytes})` },
|
||||||
|
413,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
bodyBytes = new Uint8Array(ab);
|
||||||
|
} catch (err) {
|
||||||
|
return c.json({ error: `failed to read request body: ${(err as Error).message}` }, 400);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (bodyBytes.byteLength === 0) {
|
||||||
|
return c.json({ error: 'empty request body' }, 400);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Decode the wire envelope. `decodeEnvelope` handles both `0x01`
|
||||||
|
// PreKeyMessage and `0x02` RatchetMessage shapes.
|
||||||
|
let plaintext: string;
|
||||||
|
try {
|
||||||
|
const envelope = decodeEnvelope(bodyBytes);
|
||||||
|
// First-message gate: only allow `prekey` envelopes when the
|
||||||
|
// operator has explicitly opted in.
|
||||||
|
if (options.acceptFirstMessage !== true && envelope.type === 'prekey') {
|
||||||
|
return c.json(
|
||||||
|
{
|
||||||
|
error:
|
||||||
|
'PreKeyMessage envelopes are not accepted on this RPC route — establish the session first via shade.initSessionFromBundle, or set acceptFirstMessage: true',
|
||||||
|
},
|
||||||
|
400,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
plaintext = await shade.receive(senderAddress, envelope);
|
||||||
|
} catch (err) {
|
||||||
|
// Decryption failure — could be no session, corrupted envelope,
|
||||||
|
// or sender address mismatch. Treat as 401 since the envelope is
|
||||||
|
// self-authenticating: a valid sender would decrypt cleanly.
|
||||||
|
return c.json({ error: `decrypt failed: ${(err as Error).message}` }, 401);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse the plaintext as an RpcRequest.
|
||||||
|
const classified = tryParseEnvelope(plaintext);
|
||||||
|
if (classified === null) {
|
||||||
|
return c.json({ error: 'plaintext is not a valid @shade/files envelope' }, 400);
|
||||||
|
}
|
||||||
|
if (classified.kind !== 'request') {
|
||||||
|
// Cancel envelopes are silently dropped — RPC route is request/
|
||||||
|
// response only. Cancellation across HTTP is achieved via
|
||||||
|
// AbortController on the client side, not protocol-level.
|
||||||
|
if (classified.kind === 'cancel') {
|
||||||
|
handler.handleCancel(senderAddress, classified.envelope);
|
||||||
|
// No response body — the cancel was best-effort.
|
||||||
|
return new Response(null, { status: 204 });
|
||||||
|
}
|
||||||
|
return c.json(
|
||||||
|
{ error: `unexpected envelope kind on RPC route: ${classified.kind}` },
|
||||||
|
400,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const request: RpcRequest = classified.envelope;
|
||||||
|
|
||||||
|
// Dispatch through the file handler.
|
||||||
|
let result: RpcResponse | RpcError;
|
||||||
|
try {
|
||||||
|
result = await handler.handleRequest(senderAddress, request);
|
||||||
|
} catch (err) {
|
||||||
|
// Should never happen — handler.handleRequest catches its own
|
||||||
|
// errors and returns RpcError. If it didn't, that's a bug; emit
|
||||||
|
// a generic transport-level RpcError so the client can surface
|
||||||
|
// it deterministically.
|
||||||
|
result = {
|
||||||
|
kind: KIND_ERROR_V1,
|
||||||
|
id: request.id,
|
||||||
|
error: {
|
||||||
|
code: 'INTERNAL',
|
||||||
|
message: `handler raised: ${(err as Error).message}`,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Encrypt the response and return it as wire bytes.
|
||||||
|
let responseBytes: Uint8Array;
|
||||||
|
try {
|
||||||
|
const responsePlaintext = encodeRpcEnvelope(result);
|
||||||
|
const responseEnvelope = await shade.send(senderAddress, responsePlaintext);
|
||||||
|
responseBytes = encodeEnvelope(responseEnvelope);
|
||||||
|
} catch (err) {
|
||||||
|
return c.json(
|
||||||
|
{ error: `failed to encrypt response: ${(err as Error).message}` },
|
||||||
|
500,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return new Response(new Blob([responseBytes as unknown as ArrayBuffer]), {
|
||||||
|
status: 200,
|
||||||
|
headers: { 'Content-Type': 'application/octet-stream' },
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
return app;
|
||||||
|
}
|
||||||
@@ -168,13 +168,14 @@ export async function createServerStreamsBridge(
|
|||||||
|
|
||||||
// No waiter yet — park.
|
// No waiter yet — park.
|
||||||
parked.set(writeId, arrived);
|
parked.set(writeId, arrived);
|
||||||
setTimeout(() => {
|
const parkTimer = setTimeout(() => {
|
||||||
const stale = parked.get(writeId);
|
const stale = parked.get(writeId);
|
||||||
if (stale === arrived) {
|
if (stale === arrived) {
|
||||||
parked.delete(writeId);
|
parked.delete(writeId);
|
||||||
void handle.abort('rpc-timeout').catch(() => undefined);
|
void handle.abort('rpc-timeout').catch(() => undefined);
|
||||||
}
|
}
|
||||||
}, parkedWriteTtlMs).unref?.();
|
}, parkedWriteTtlMs);
|
||||||
|
(parkTimer as unknown as { unref?: () => void }).unref?.();
|
||||||
});
|
});
|
||||||
|
|
||||||
function cleanupWaiter(w: PendingWaiter): void {
|
function cleanupWaiter(w: PendingWaiter): void {
|
||||||
|
|||||||
208
packages/shade-files/tests/integration/http-rpc-streams.test.ts
Normal file
208
packages/shade-files/tests/integration/http-rpc-streams.test.ts
Normal file
@@ -0,0 +1,208 @@
|
|||||||
|
import { describe, expect, test } from 'bun:test';
|
||||||
|
import { createShade } from '@shade/sdk';
|
||||||
|
import {
|
||||||
|
createPrekeyServer,
|
||||||
|
MemoryPrekeyStore,
|
||||||
|
PrekeyServerEvents,
|
||||||
|
} from '@shade/server';
|
||||||
|
import { SubtleCryptoProvider } from '@shade/crypto-web';
|
||||||
|
import { Hono } from 'hono';
|
||||||
|
|
||||||
|
const crypto = new SubtleCryptoProvider();
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Stand up the full pull-mode rig:
|
||||||
|
* - Prekey server (for X3DH)
|
||||||
|
* - Bob: file handler + rpcRoute + transferQueueRoute, all on one server
|
||||||
|
* - Alice: httpClient with outboundQueueUrl + transferBaseUrl wired
|
||||||
|
*
|
||||||
|
* Returns Alice's `FileClient`, which speaks browser-style: ONE base URL,
|
||||||
|
* no inbound listener, streams supported via long-poll.
|
||||||
|
*/
|
||||||
|
async function setupPullRig(opts: {
|
||||||
|
bobHandler: Parameters<NonNullable<Awaited<ReturnType<typeof createShade>>['files']>['serve']>[0];
|
||||||
|
}) {
|
||||||
|
const prekey = createPrekeyServer({
|
||||||
|
crypto,
|
||||||
|
store: new MemoryPrekeyStore(),
|
||||||
|
disableRateLimit: true,
|
||||||
|
events: new PrekeyServerEvents(),
|
||||||
|
});
|
||||||
|
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
|
||||||
|
const prekeyUrl = `http://localhost:${prekeyServer.port}`;
|
||||||
|
|
||||||
|
const alice = await createShade({ prekeyServer: prekeyUrl, address: 'alice' });
|
||||||
|
const bob = await createShade({ prekeyServer: prekeyUrl, address: 'bob' });
|
||||||
|
|
||||||
|
// Bob: queue-route FIRST (configures bob's transports), then files.serve.
|
||||||
|
const queueRoute = await bob.transferQueueRoute({ blockMs: 1_500 });
|
||||||
|
await bob.files.serve(opts.bobHandler);
|
||||||
|
const rpcRoute = bob.files.rpcRoute({ acceptFirstMessage: true });
|
||||||
|
|
||||||
|
const app = new Hono();
|
||||||
|
app.route('/', queueRoute);
|
||||||
|
app.route('/', rpcRoute);
|
||||||
|
|
||||||
|
const bobServer = Bun.serve({ port: 0, fetch: app.fetch });
|
||||||
|
const baseUrl = `http://localhost:${bobServer.port}`;
|
||||||
|
|
||||||
|
const fs = alice.files.httpClient('bob', {
|
||||||
|
rpcUrl: `${baseUrl}/rpc`,
|
||||||
|
outboundQueueUrl: `${baseUrl}/queue`,
|
||||||
|
transferBaseUrl: baseUrl,
|
||||||
|
defaultTimeoutMs: 10_000,
|
||||||
|
queueBlockMs: 1_000,
|
||||||
|
});
|
||||||
|
|
||||||
|
return {
|
||||||
|
alice,
|
||||||
|
bob,
|
||||||
|
fs,
|
||||||
|
baseUrl,
|
||||||
|
teardown: async () => {
|
||||||
|
fs.close();
|
||||||
|
await alice.shutdown();
|
||||||
|
await bob.shutdown();
|
||||||
|
bobServer.stop();
|
||||||
|
prekeyServer.stop();
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('@shade/files HTTP RPC — pull-mode streams', () => {
|
||||||
|
test('streamed read (4 MiB) via long-poll queue', async () => {
|
||||||
|
const payload = new Uint8Array(4 * 1024 * 1024);
|
||||||
|
for (let i = 0; i < payload.length; i++) payload[i] = (i * 97) & 0xff;
|
||||||
|
|
||||||
|
const rig = await setupPullRig({
|
||||||
|
bobHandler: {
|
||||||
|
read: async () => {
|
||||||
|
// Return the payload as a streamed read so the rpc-handler
|
||||||
|
// promotes it via the streams-bridge into a transfer.
|
||||||
|
const stream = new ReadableStream<Uint8Array>({
|
||||||
|
start(controller) {
|
||||||
|
const CHUNK = 256 * 1024;
|
||||||
|
for (let off = 0; off < payload.byteLength; off += CHUNK) {
|
||||||
|
controller.enqueue(payload.slice(off, Math.min(off + CHUNK, payload.byteLength)));
|
||||||
|
}
|
||||||
|
controller.close();
|
||||||
|
},
|
||||||
|
});
|
||||||
|
// Need a precomputed sha256 for streamed reads. Use the
|
||||||
|
// crypto provider's sha256 directly.
|
||||||
|
const digest = new Uint8Array(await globalThis.crypto.subtle.digest('SHA-256', payload));
|
||||||
|
const sha256Hex = Array.from(digest, (b) => b.toString(16).padStart(2, '0')).join('');
|
||||||
|
return {
|
||||||
|
kind: 'streams' as const,
|
||||||
|
stream,
|
||||||
|
size: payload.byteLength,
|
||||||
|
sha256: sha256Hex,
|
||||||
|
contentType: 'application/octet-stream',
|
||||||
|
};
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
try {
|
||||||
|
const result = await rig.fs.read('/big.bin');
|
||||||
|
expect(result.kind).toBe('streams');
|
||||||
|
if (result.kind !== 'streams') return;
|
||||||
|
|
||||||
|
// Drain the stream and compare.
|
||||||
|
const reader = result.stream.getReader();
|
||||||
|
const got = new Uint8Array(payload.byteLength);
|
||||||
|
let offset = 0;
|
||||||
|
while (true) {
|
||||||
|
const { value, done } = await reader.read();
|
||||||
|
if (done) break;
|
||||||
|
if (value !== undefined) {
|
||||||
|
got.set(value, offset);
|
||||||
|
offset += value.byteLength;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
reader.releaseLock();
|
||||||
|
await result.done();
|
||||||
|
|
||||||
|
expect(offset).toBe(payload.byteLength);
|
||||||
|
// Compare in 64KiB strides for speed.
|
||||||
|
let mismatch = -1;
|
||||||
|
for (let i = 0; i < payload.byteLength; i++) {
|
||||||
|
if (got[i] !== payload[i]) {
|
||||||
|
mismatch = i;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
expect(mismatch).toBe(-1);
|
||||||
|
} finally {
|
||||||
|
await rig.teardown();
|
||||||
|
}
|
||||||
|
}, 30_000);
|
||||||
|
|
||||||
|
test('streamed read fails with clear error when outboundQueueUrl is omitted', async () => {
|
||||||
|
const rig = await setupPullRig({
|
||||||
|
bobHandler: {
|
||||||
|
read: async () => {
|
||||||
|
const stream = new ReadableStream<Uint8Array>({
|
||||||
|
start(c) {
|
||||||
|
c.enqueue(new Uint8Array(512 * 1024));
|
||||||
|
c.close();
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const digest = new Uint8Array(await globalThis.crypto.subtle.digest('SHA-256', new Uint8Array(512 * 1024)));
|
||||||
|
const sha256Hex = Array.from(digest, (b) => b.toString(16).padStart(2, '0')).join('');
|
||||||
|
return {
|
||||||
|
kind: 'streams' as const,
|
||||||
|
stream,
|
||||||
|
size: 512 * 1024,
|
||||||
|
sha256: sha256Hex,
|
||||||
|
};
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
// Tear down the rig's drainer so we can construct an inline-only client
|
||||||
|
rig.fs.close();
|
||||||
|
|
||||||
|
const inlineOnly = rig.alice.files.httpClient('bob', {
|
||||||
|
rpcUrl: `${rig.baseUrl}/rpc`,
|
||||||
|
defaultTimeoutMs: 10_000,
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
await expect(inlineOnly.read('/big.bin')).rejects.toThrow(/streamed read/);
|
||||||
|
} finally {
|
||||||
|
inlineOnly.close();
|
||||||
|
await rig.teardown();
|
||||||
|
}
|
||||||
|
}, 15_000);
|
||||||
|
|
||||||
|
test('long-poll returns empty events on idle timeout', async () => {
|
||||||
|
const rig = await setupPullRig({
|
||||||
|
bobHandler: {
|
||||||
|
stat: async () => ({
|
||||||
|
name: '_',
|
||||||
|
kind: 'dir' as const,
|
||||||
|
size: 0,
|
||||||
|
mtime: 0,
|
||||||
|
metadata: {},
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
// Direct poll without any pending events — should return after blockMs.
|
||||||
|
const start = Date.now();
|
||||||
|
const res = await fetch(`${rig.baseUrl}/queue`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
'X-Shade-Sender-Address': 'alice',
|
||||||
|
},
|
||||||
|
body: JSON.stringify({ since: 0, blockMs: 500 }),
|
||||||
|
});
|
||||||
|
expect(res.status).toBe(200);
|
||||||
|
const body = (await res.json()) as { events: unknown[]; nextSince: number };
|
||||||
|
expect(body.events).toHaveLength(0);
|
||||||
|
expect(Date.now() - start).toBeGreaterThanOrEqual(400);
|
||||||
|
} finally {
|
||||||
|
await rig.teardown();
|
||||||
|
}
|
||||||
|
}, 10_000);
|
||||||
|
});
|
||||||
321
packages/shade-files/tests/integration/http-rpc.test.ts
Normal file
321
packages/shade-files/tests/integration/http-rpc.test.ts
Normal file
@@ -0,0 +1,321 @@
|
|||||||
|
import { describe, expect, test } from 'bun:test';
|
||||||
|
import { createShade } from '@shade/sdk';
|
||||||
|
import {
|
||||||
|
createPrekeyServer,
|
||||||
|
MemoryPrekeyStore,
|
||||||
|
PrekeyServerEvents,
|
||||||
|
} from '@shade/server';
|
||||||
|
import { SubtleCryptoProvider } from '@shade/crypto-web';
|
||||||
|
|
||||||
|
const crypto = new SubtleCryptoProvider();
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Stand up a prekey server + two Shades + Bob's file handler + RPC route
|
||||||
|
* mounted on Bun.serve, then return Alice's HTTP-only `FileClient`.
|
||||||
|
*
|
||||||
|
* Mirrors the request-response setup a browser client would use against a
|
||||||
|
* Bun-style server.
|
||||||
|
*/
|
||||||
|
async function setupHttpRig(opts: {
|
||||||
|
bobHandler: Parameters<NonNullable<Awaited<ReturnType<typeof createShade>>['files']>['serve']>[0];
|
||||||
|
}) {
|
||||||
|
// 1. Prekey server.
|
||||||
|
const prekeyEvents = new PrekeyServerEvents();
|
||||||
|
const prekey = createPrekeyServer({
|
||||||
|
crypto,
|
||||||
|
store: new MemoryPrekeyStore(),
|
||||||
|
disableRateLimit: true,
|
||||||
|
events: prekeyEvents,
|
||||||
|
});
|
||||||
|
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
|
||||||
|
const prekeyUrl = `http://localhost:${prekeyServer.port}`;
|
||||||
|
|
||||||
|
// 2. Two Shades. Alice plays the browser client (no transferRoute);
|
||||||
|
// Bob is the server.
|
||||||
|
const alice = await createShade({ prekeyServer: prekeyUrl, address: 'alice' });
|
||||||
|
const bob = await createShade({ prekeyServer: prekeyUrl, address: 'bob' });
|
||||||
|
|
||||||
|
// 3. Bob: register file handler (HTTP-only — no streams) + mount
|
||||||
|
// the RPC route.
|
||||||
|
await bob.files.serve(opts.bobHandler, { inlineOnly: true });
|
||||||
|
const rpcRoute = bob.files.rpcRoute({ acceptFirstMessage: true });
|
||||||
|
const bobServer = Bun.serve({ port: 0, fetch: rpcRoute.fetch });
|
||||||
|
const rpcUrl = `http://localhost:${bobServer.port}/rpc`;
|
||||||
|
|
||||||
|
// 5. Alice: build the HTTP-only file client.
|
||||||
|
const fs = alice.files.httpClient('bob', { rpcUrl, defaultTimeoutMs: 5000 });
|
||||||
|
|
||||||
|
return {
|
||||||
|
alice,
|
||||||
|
bob,
|
||||||
|
fs,
|
||||||
|
rpcUrl,
|
||||||
|
teardown: async () => {
|
||||||
|
await alice.shutdown();
|
||||||
|
await bob.shutdown();
|
||||||
|
bobServer.stop();
|
||||||
|
prekeyServer.stop();
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('@shade/files HTTP RPC — round-trip', () => {
|
||||||
|
test('list → mkdir → stat → write inline → read inline → delete via httpClient', async () => {
|
||||||
|
interface VfsEntry {
|
||||||
|
kind: 'file' | 'dir';
|
||||||
|
bytes?: Uint8Array;
|
||||||
|
contentType?: string;
|
||||||
|
}
|
||||||
|
const vfs = new Map<string, VfsEntry>([
|
||||||
|
['/', { kind: 'dir' }],
|
||||||
|
['/photos', { kind: 'dir' }],
|
||||||
|
]);
|
||||||
|
|
||||||
|
const rig = await setupHttpRig({
|
||||||
|
bobHandler: {
|
||||||
|
list: async (ctx) => {
|
||||||
|
const prefix = ctx.path.endsWith('/') ? ctx.path : `${ctx.path}/`;
|
||||||
|
const entries = Array.from(vfs.entries())
|
||||||
|
.filter(([p]) => p.startsWith(prefix) && p !== ctx.path && !p.slice(prefix.length).includes('/'))
|
||||||
|
.map(([p, e]) => ({
|
||||||
|
name: p.slice(prefix.length) || p,
|
||||||
|
kind: e.kind,
|
||||||
|
size: e.bytes?.byteLength ?? 0,
|
||||||
|
mtime: 0,
|
||||||
|
metadata: {},
|
||||||
|
}));
|
||||||
|
return { entries, hasMore: false };
|
||||||
|
},
|
||||||
|
stat: async (ctx) => {
|
||||||
|
const e = vfs.get(ctx.path);
|
||||||
|
if (!e) throw new (await import('../../src/index.js')).NotFoundError(`stat ${ctx.path}`);
|
||||||
|
return {
|
||||||
|
name: ctx.path.split('/').pop() ?? ctx.path,
|
||||||
|
kind: e.kind,
|
||||||
|
size: e.bytes?.byteLength ?? 0,
|
||||||
|
mtime: 0,
|
||||||
|
metadata: {},
|
||||||
|
...(e.contentType !== undefined ? { contentType: e.contentType } : {}),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
mkdir: async (ctx) => {
|
||||||
|
vfs.set(ctx.path, { kind: 'dir' });
|
||||||
|
return { entry: { name: ctx.path.split('/').pop() ?? ctx.path, kind: 'dir' as const, size: 0, mtime: 0, metadata: {} } };
|
||||||
|
},
|
||||||
|
delete: async (ctx) => {
|
||||||
|
if (!vfs.has(ctx.path)) {
|
||||||
|
throw new (await import('../../src/index.js')).NotFoundError(`delete ${ctx.path}`);
|
||||||
|
}
|
||||||
|
vfs.delete(ctx.path);
|
||||||
|
return { deletedCount: 1 };
|
||||||
|
},
|
||||||
|
read: async (ctx) => {
|
||||||
|
const e = vfs.get(ctx.path);
|
||||||
|
if (!e || e.kind !== 'file' || !e.bytes) {
|
||||||
|
throw new (await import('../../src/index.js')).NotFoundError(`read ${ctx.path}`);
|
||||||
|
}
|
||||||
|
// Omit sha256 — dispatcher computes it from the bytes.
|
||||||
|
return {
|
||||||
|
kind: 'inline' as const,
|
||||||
|
bytes: e.bytes,
|
||||||
|
...(e.contentType !== undefined ? { contentType: e.contentType } : {}),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
write: async (ctx) => {
|
||||||
|
if (ctx.args.content.kind !== 'inline') {
|
||||||
|
throw new (await import('../../src/index.js')).ConflictError('streams not supported in this test handler');
|
||||||
|
}
|
||||||
|
vfs.set(ctx.args.path, {
|
||||||
|
kind: 'file',
|
||||||
|
bytes: ctx.args.content.bytes,
|
||||||
|
...(ctx.args.contentType !== undefined ? { contentType: ctx.args.contentType } : {}),
|
||||||
|
});
|
||||||
|
return {
|
||||||
|
entry: {
|
||||||
|
name: ctx.args.path.split('/').pop() ?? ctx.args.path,
|
||||||
|
kind: 'file' as const,
|
||||||
|
size: ctx.args.content.bytes.byteLength,
|
||||||
|
mtime: 0,
|
||||||
|
metadata: {},
|
||||||
|
...(ctx.args.contentType !== undefined ? { contentType: ctx.args.contentType } : {}),
|
||||||
|
},
|
||||||
|
};
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
try {
|
||||||
|
// list
|
||||||
|
const listed = await rig.fs.list('/');
|
||||||
|
expect(listed.entries.map((e) => e.name).sort()).toContain('photos');
|
||||||
|
|
||||||
|
// mkdir
|
||||||
|
await rig.fs.mkdir('/docs');
|
||||||
|
const stat = await rig.fs.stat('/docs');
|
||||||
|
expect(stat.kind).toBe('dir');
|
||||||
|
|
||||||
|
// write inline
|
||||||
|
const payload = new TextEncoder().encode('hello browser-friendly world');
|
||||||
|
const writeResult = await rig.fs.write('/docs/greeting.txt', payload, {
|
||||||
|
contentType: 'text/plain',
|
||||||
|
});
|
||||||
|
expect(writeResult.entry.size).toBe(payload.byteLength);
|
||||||
|
|
||||||
|
// read inline
|
||||||
|
const readResult = await rig.fs.read('/docs/greeting.txt');
|
||||||
|
expect(readResult.kind).toBe('inline');
|
||||||
|
if (readResult.kind === 'inline') {
|
||||||
|
expect(new TextDecoder().decode(readResult.bytes)).toBe('hello browser-friendly world');
|
||||||
|
expect(readResult.contentType).toBe('text/plain');
|
||||||
|
}
|
||||||
|
|
||||||
|
// delete
|
||||||
|
const del = await rig.fs.delete('/docs/greeting.txt');
|
||||||
|
expect(del.deletedCount).toBe(1);
|
||||||
|
|
||||||
|
// stat the deleted path → typed NotFoundError
|
||||||
|
const { NotFoundError } = await import('../../src/index.js');
|
||||||
|
await expect(rig.fs.stat('/docs/greeting.txt')).rejects.toBeInstanceOf(NotFoundError);
|
||||||
|
} finally {
|
||||||
|
await rig.teardown();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('streamed write (> 256 KiB) is rejected with a clear error', async () => {
|
||||||
|
const rig = await setupHttpRig({
|
||||||
|
bobHandler: {
|
||||||
|
write: async () => ({
|
||||||
|
entry: { name: 'unused', kind: 'file' as const, size: 0, mtime: 0, metadata: {} },
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
const big = new Uint8Array(257 * 1024);
|
||||||
|
const { ConflictError } = await import('../../src/index.js');
|
||||||
|
await expect(rig.fs.write('/big.bin', big)).rejects.toBeInstanceOf(ConflictError);
|
||||||
|
} finally {
|
||||||
|
await rig.teardown();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('rpcRoute() throws when no handler is attached', async () => {
|
||||||
|
// Don't call shade.files.serve(...) — rpcRoute() should refuse.
|
||||||
|
const prekeyEvents = new PrekeyServerEvents();
|
||||||
|
const prekey = createPrekeyServer({
|
||||||
|
crypto,
|
||||||
|
store: new MemoryPrekeyStore(),
|
||||||
|
disableRateLimit: true,
|
||||||
|
events: prekeyEvents,
|
||||||
|
});
|
||||||
|
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
|
||||||
|
const bob = await createShade({
|
||||||
|
prekeyServer: `http://localhost:${prekeyServer.port}`,
|
||||||
|
address: 'bob',
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
expect(() => bob.files.rpcRoute()).toThrow(/no handler attached/);
|
||||||
|
} finally {
|
||||||
|
await bob.shutdown();
|
||||||
|
prekeyServer.stop();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('missing X-Shade-Sender-Address header → 400', async () => {
|
||||||
|
const rig = await setupHttpRig({
|
||||||
|
bobHandler: {
|
||||||
|
stat: async () => ({
|
||||||
|
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
const res = await fetch(rig.rpcUrl, {
|
||||||
|
method: 'POST',
|
||||||
|
body: new Uint8Array([0]),
|
||||||
|
});
|
||||||
|
expect(res.status).toBe(400);
|
||||||
|
const body = (await res.json()) as { error: string };
|
||||||
|
expect(body.error).toMatch(/X-Shade-Sender-Address/);
|
||||||
|
} finally {
|
||||||
|
await rig.teardown();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('empty body → 400', async () => {
|
||||||
|
const rig = await setupHttpRig({
|
||||||
|
bobHandler: {
|
||||||
|
stat: async () => ({
|
||||||
|
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
const res = await fetch(rig.rpcUrl, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'X-Shade-Sender-Address': 'alice' },
|
||||||
|
});
|
||||||
|
expect(res.status).toBe(400);
|
||||||
|
} finally {
|
||||||
|
await rig.teardown();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('garbage body → 401 decrypt failure', async () => {
|
||||||
|
const rig = await setupHttpRig({
|
||||||
|
bobHandler: {
|
||||||
|
stat: async () => ({
|
||||||
|
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
const res = await fetch(rig.rpcUrl, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'X-Shade-Sender-Address': 'alice' },
|
||||||
|
body: new Uint8Array([0x02, 0xff, 0xff, 0xff]),
|
||||||
|
});
|
||||||
|
// 400 from envelope decode failure or 401 from decrypt failure.
|
||||||
|
expect([400, 401]).toContain(res.status);
|
||||||
|
} finally {
|
||||||
|
await rig.teardown();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('body past maxBodyBytes → 413', async () => {
|
||||||
|
const prekeyEvents = new PrekeyServerEvents();
|
||||||
|
const prekey = createPrekeyServer({
|
||||||
|
crypto,
|
||||||
|
store: new MemoryPrekeyStore(),
|
||||||
|
disableRateLimit: true,
|
||||||
|
events: prekeyEvents,
|
||||||
|
});
|
||||||
|
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
|
||||||
|
const bob = await createShade({
|
||||||
|
prekeyServer: `http://localhost:${prekeyServer.port}`,
|
||||||
|
address: 'bob',
|
||||||
|
});
|
||||||
|
await bob.files.serve(
|
||||||
|
{
|
||||||
|
stat: async () => ({
|
||||||
|
name: '_', kind: 'dir' as const, size: 0, mtime: 0, metadata: {},
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
{ inlineOnly: true },
|
||||||
|
);
|
||||||
|
const route = bob.files.rpcRoute({ maxBodyBytes: 1024 });
|
||||||
|
const server = Bun.serve({ port: 0, fetch: route.fetch });
|
||||||
|
try {
|
||||||
|
const big = new Uint8Array(2048);
|
||||||
|
const res = await fetch(`http://localhost:${server.port}/rpc`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'X-Shade-Sender-Address': 'alice' },
|
||||||
|
body: big,
|
||||||
|
});
|
||||||
|
expect(res.status).toBe(413);
|
||||||
|
} finally {
|
||||||
|
await bob.shutdown();
|
||||||
|
server.stop();
|
||||||
|
prekeyServer.stop();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/inbox-server",
|
"name": "@shade/inbox-server",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/inbox",
|
"name": "@shade/inbox",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/key-transparency",
|
"name": "@shade/key-transparency",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -21,7 +21,7 @@
|
|||||||
* the dataset grows enough that flat re-hash becomes a bottleneck.
|
* the dataset grows enough that flat re-hash becomes a bottleneck.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { leafHash, nodeHash, emptyRootHash } from './hashes.js';
|
import { leafHash, emptyRootHash } from './hashes.js';
|
||||||
import { sha256Sync } from './sha256.js';
|
import { sha256Sync } from './sha256.js';
|
||||||
import { constantTimeEqual } from './util.js';
|
import { constantTimeEqual } from './util.js';
|
||||||
import { mth, auditPath, recomputeRootFromAuditPath } from './log.js';
|
import { mth, auditPath, recomputeRootFromAuditPath } from './log.js';
|
||||||
|
|||||||
@@ -24,8 +24,6 @@ import { MerkleLog, auditPath } from './log.js';
|
|||||||
import {
|
import {
|
||||||
AddressIndex,
|
AddressIndex,
|
||||||
type AddressIndexEntry,
|
type AddressIndexEntry,
|
||||||
type IndexAbsenceProof,
|
|
||||||
type IndexInclusionProof,
|
|
||||||
} from './index-tree.js';
|
} from './index-tree.js';
|
||||||
import {
|
import {
|
||||||
type SignedTreeHead,
|
type SignedTreeHead,
|
||||||
|
|||||||
@@ -111,7 +111,7 @@ interface IndexAbsenceWire {
|
|||||||
} | null;
|
} | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
type IndexProofWire = IndexInclusionWire | IndexAbsenceWire;
|
export type IndexProofWire = IndexInclusionWire | IndexAbsenceWire;
|
||||||
|
|
||||||
interface BundleInclusionWire {
|
interface BundleInclusionWire {
|
||||||
kind: 'inclusion' | 'tombstone';
|
kind: 'inclusion' | 'tombstone';
|
||||||
|
|||||||
5
packages/shade-key-transparency/tsconfig.json
Normal file
5
packages/shade-key-transparency/tsconfig.json
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
{
|
||||||
|
"extends": "../../tsconfig.json",
|
||||||
|
"compilerOptions": { "outDir": "dist", "rootDir": "src" },
|
||||||
|
"include": ["src"]
|
||||||
|
}
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/keychain",
|
"name": "@shade/keychain",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
{
|
{
|
||||||
"extends": "../../tsconfig.json",
|
"extends": "../../tsconfig.json",
|
||||||
"include": ["src/**/*", "tests/**/*"]
|
"compilerOptions": { "outDir": "dist", "rootDir": "src" },
|
||||||
|
"include": ["src"]
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/observability",
|
"name": "@shade/observability",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/observer",
|
"name": "@shade/observer",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/proto",
|
"name": "@shade/proto",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/recovery",
|
"name": "@shade/recovery",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/sdk",
|
"name": "@shade/sdk",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -96,7 +96,7 @@ export class BackgroundTasks {
|
|||||||
this.hooks.onError?.(err as Error, 'prune-files');
|
this.hooks.onError?.(err as Error, 'prune-files');
|
||||||
}
|
}
|
||||||
}, this.pruneFilesIntervalMs);
|
}, this.pruneFilesIntervalMs);
|
||||||
this.pruneFilesTimer.unref?.();
|
(this.pruneFilesTimer as unknown as { unref?: () => void }).unref?.();
|
||||||
}
|
}
|
||||||
|
|
||||||
stop(): void {
|
stop(): void {
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ import {
|
|||||||
import { encodeEnvelope, decodeEnvelope, inspectEnvelopeType } from '@shade/proto';
|
import { encodeEnvelope, decodeEnvelope, inspectEnvelopeType } from '@shade/proto';
|
||||||
import { ShadeFetchTransport, type KTVerifierOptions } from '@shade/transport';
|
import { ShadeFetchTransport, type KTVerifierOptions } from '@shade/transport';
|
||||||
import { LightWitness } from '@shade/key-transparency';
|
import { LightWitness } from '@shade/key-transparency';
|
||||||
import type { SignedTreeHead } from '@shade/key-transparency';
|
import type { SignedTreeHead, STHWire } from '@shade/key-transparency';
|
||||||
import {
|
import {
|
||||||
TransferEngine,
|
TransferEngine,
|
||||||
ShadeTransferHttpTransport,
|
ShadeTransferHttpTransport,
|
||||||
@@ -31,6 +31,8 @@ import {
|
|||||||
type TransferHandle,
|
type TransferHandle,
|
||||||
type TransferOptions,
|
type TransferOptions,
|
||||||
type TransferSummary,
|
type TransferSummary,
|
||||||
|
type OutboundQueue as OutboundQueueLike,
|
||||||
|
type QueuedEventInput,
|
||||||
} from '@shade/transfer';
|
} from '@shade/transfer';
|
||||||
import type { Hono } from 'hono';
|
import type { Hono } from 'hono';
|
||||||
import { BackgroundTasks } from './background.js';
|
import { BackgroundTasks } from './background.js';
|
||||||
@@ -151,6 +153,8 @@ export class Shade {
|
|||||||
private controlChannel: ShadeControlChannel | null = null;
|
private controlChannel: ShadeControlChannel | null = null;
|
||||||
private peerBaseUrlResolver: ((peerAddress: string) => Promise<string>) | null = null;
|
private peerBaseUrlResolver: ((peerAddress: string) => Promise<string>) | null = null;
|
||||||
private envelopeOutboxes: ControlEnvelopeTransport | null = null;
|
private envelopeOutboxes: ControlEnvelopeTransport | null = null;
|
||||||
|
private transferTransportOverride: ITransferTransport | null = null;
|
||||||
|
private transferQueue: OutboundQueueLike | null = null;
|
||||||
|
|
||||||
// `@shade/files` namespace, lazy + memoized.
|
// `@shade/files` namespace, lazy + memoized.
|
||||||
private filesNamespace: FilesNamespace | null = null;
|
private filesNamespace: FilesNamespace | null = null;
|
||||||
@@ -217,15 +221,15 @@ export class Shade {
|
|||||||
maxStaleMs: this.config.keyTransparency.maxStaleMs,
|
maxStaleMs: this.config.keyTransparency.maxStaleMs,
|
||||||
maxStored: this.config.keyTransparency.witnessMaxStored,
|
maxStored: this.config.keyTransparency.witnessMaxStored,
|
||||||
fetcher: {
|
fetcher: {
|
||||||
async fetchLatestSTH() {
|
async fetchLatestSTH(): Promise<STHWire> {
|
||||||
const res = await fetch(`${baseUrl}/v1/kt/sth`);
|
const res = await fetch(`${baseUrl}/v1/kt/sth`);
|
||||||
if (!res.ok) throw new Error(`KT /sth: ${res.status}`);
|
if (!res.ok) throw new Error(`KT /sth: ${res.status}`);
|
||||||
return res.json();
|
return (await res.json()) as STHWire;
|
||||||
},
|
},
|
||||||
async fetchConsistencyProof(from, to) {
|
async fetchConsistencyProof(from, to): Promise<{ proof: string[] }> {
|
||||||
const res = await fetch(`${baseUrl}/v1/kt/consistency?from=${from}&to=${to}`);
|
const res = await fetch(`${baseUrl}/v1/kt/consistency?from=${from}&to=${to}`);
|
||||||
if (!res.ok) throw new Error(`KT /consistency: ${res.status}`);
|
if (!res.ok) throw new Error(`KT /consistency: ${res.status}`);
|
||||||
return res.json();
|
return (await res.json()) as { proof: string[] };
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
@@ -746,12 +750,52 @@ export class Shade {
|
|||||||
* HTTP POSTs to `<base>/v1/transfer/control`).
|
* HTTP POSTs to `<base>/v1/transfer/control`).
|
||||||
*/
|
*/
|
||||||
configureTransfers(opts: {
|
configureTransfers(opts: {
|
||||||
resolveBaseUrl: (peerAddress: string) => Promise<string>;
|
/**
|
||||||
|
* Resolver for the peer's HTTP base URL (used by the default
|
||||||
|
* `ShadeTransferHttpTransport` to POST chunks). Optional when a
|
||||||
|
* custom `transport` and `envelopeTransport` are supplied — e.g.
|
||||||
|
* for pull-mode browser servers (`@shade/files transferQueueRoute`)
|
||||||
|
* which never POST chunks anywhere.
|
||||||
|
*/
|
||||||
|
resolveBaseUrl?: (peerAddress: string) => Promise<string>;
|
||||||
|
/**
|
||||||
|
* Override the chunk-level transport. Defaults to
|
||||||
|
* `ShadeTransferHttpTransport` (HTTP POSTs per chunk) when
|
||||||
|
* `resolveBaseUrl` is supplied. Required when `resolveBaseUrl`
|
||||||
|
* is omitted.
|
||||||
|
*/
|
||||||
|
transport?: ITransferTransport;
|
||||||
|
/**
|
||||||
|
* Override the control-envelope transport. Defaults to HTTP POSTs
|
||||||
|
* to `<base>/v1/transfer/control` when `resolveBaseUrl` is
|
||||||
|
* supplied. Required when `resolveBaseUrl` is omitted.
|
||||||
|
*/
|
||||||
envelopeTransport?: ControlEnvelopeTransport;
|
envelopeTransport?: ControlEnvelopeTransport;
|
||||||
}): void {
|
}): void {
|
||||||
this.peerBaseUrlResolver = opts.resolveBaseUrl;
|
if (opts.resolveBaseUrl === undefined) {
|
||||||
this.envelopeOutboxes =
|
if (opts.transport === undefined || opts.envelopeTransport === undefined) {
|
||||||
opts.envelopeTransport ?? new HttpEnvelopeTransport(opts.resolveBaseUrl, this.address);
|
throw new Error(
|
||||||
|
'configureTransfers: resolveBaseUrl is required unless both `transport` and `envelopeTransport` are supplied (e.g. for pull-mode queue servers).',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
this.peerBaseUrlResolver = async () => {
|
||||||
|
throw new Error(
|
||||||
|
'resolveBaseUrl was not configured — this Shade is in queue/pull mode and does not POST chunks. Configure a custom transport instead.',
|
||||||
|
);
|
||||||
|
};
|
||||||
|
} else {
|
||||||
|
this.peerBaseUrlResolver = opts.resolveBaseUrl;
|
||||||
|
}
|
||||||
|
this.transferTransportOverride = opts.transport ?? null;
|
||||||
|
if (opts.envelopeTransport !== undefined) {
|
||||||
|
this.envelopeOutboxes = opts.envelopeTransport;
|
||||||
|
} else if (opts.resolveBaseUrl !== undefined) {
|
||||||
|
this.envelopeOutboxes = new HttpEnvelopeTransport(opts.resolveBaseUrl, this.address);
|
||||||
|
} else {
|
||||||
|
throw new Error(
|
||||||
|
'configureTransfers: envelopeTransport is required when resolveBaseUrl is omitted.',
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -898,6 +942,109 @@ export class Shade {
|
|||||||
return (await this.engine()).onIncomingTransfer(handler);
|
return (await this.engine()).onIncomingTransfer(handler);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Mount the **pull-mode** transfer routes on a Hono app. Mount under
|
||||||
|
* any base path: `app.route('/api/v1/shade-files', shade.transferQueueRoute())`.
|
||||||
|
*
|
||||||
|
* Configures this Shade instance to queue all outbound chunks +
|
||||||
|
* control envelopes per peer instead of POSTing them. Browser-style
|
||||||
|
* receivers drain the queue via long-polling — no inbound HTTP
|
||||||
|
* listener required on the receiver.
|
||||||
|
*
|
||||||
|
* Routes mounted (relative to the base path):
|
||||||
|
* POST /queue — long-poll the per-peer outbound queue
|
||||||
|
* POST /v1/transfer/:streamId/chunk — receive incoming chunks (browser → server)
|
||||||
|
* GET /v1/transfer/:streamId/state — resume-state lookup
|
||||||
|
* POST /v1/transfer/control — receive incoming control envelopes
|
||||||
|
* GET /v1/transfer/health — peer reachability probe
|
||||||
|
*
|
||||||
|
* **Idempotent**: calling twice returns a fresh `Hono` app each
|
||||||
|
* time but reuses the underlying queue + transport (so the engine
|
||||||
|
* stays single).
|
||||||
|
*
|
||||||
|
* **Ordering**: must be called **before** `shade.files.serve(...)`
|
||||||
|
* (or any other path that builds the engine), because configuring
|
||||||
|
* the queue transport mutates the transfer stack. Calling after the
|
||||||
|
* engine is built throws.
|
||||||
|
*/
|
||||||
|
async transferQueueRoute(opts: TransferQueueRouteOptions = {}): Promise<Hono> {
|
||||||
|
if (this.transferEngine !== null && this.transferTransportOverride === null) {
|
||||||
|
throw new Error(
|
||||||
|
'transferQueueRoute(): the transfer engine has already been built with the default HTTP transport. Call transferQueueRoute() before any upload()/onIncomingTransfer()/configureTransfers().',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const { OutboundQueue, QueueTransferTransport } = await import('@shade/transfer');
|
||||||
|
if (this.transferQueue === null) {
|
||||||
|
this.transferQueue = new OutboundQueue({
|
||||||
|
...(opts.maxEventsPerPeer !== undefined ? { maxEventsPerPeer: opts.maxEventsPerPeer } : {}),
|
||||||
|
...(opts.idleEvictionMs !== undefined ? { idleEvictionMs: opts.idleEvictionMs } : {}),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if (this.transferTransportOverride === null) {
|
||||||
|
const queueTransport = new QueueTransferTransport(this.transferQueue);
|
||||||
|
const queueEnvelopeTransport = new QueueEnvelopeTransport(this.transferQueue);
|
||||||
|
this.configureTransfers({
|
||||||
|
transport: queueTransport,
|
||||||
|
envelopeTransport: queueEnvelopeTransport,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
const queue = this.transferQueue;
|
||||||
|
const blockMs = opts.blockMs ?? 30_000;
|
||||||
|
const maxBlockMs = opts.maxBlockMs ?? 55_000;
|
||||||
|
const engine = await this.engine();
|
||||||
|
const { createTransferRoutes, PermissiveAuthenticator } = await import('@shade/transfer');
|
||||||
|
const app = await createTransferRoutes(engine, {
|
||||||
|
authenticator: PermissiveAuthenticator,
|
||||||
|
});
|
||||||
|
app.post('/v1/transfer/control', async (c) => {
|
||||||
|
const senderAddress = c.req.header('X-Shade-Sender-Address');
|
||||||
|
if (senderAddress === undefined || senderAddress === '') {
|
||||||
|
return c.json({ error: 'missing X-Shade-Sender-Address' }, 400);
|
||||||
|
}
|
||||||
|
const ab = await c.req.arrayBuffer();
|
||||||
|
const bytes = new Uint8Array(ab);
|
||||||
|
try {
|
||||||
|
await this.acceptTransferEnvelope(senderAddress, bytes);
|
||||||
|
} catch (err) {
|
||||||
|
return c.json({ error: (err as Error).message }, 400);
|
||||||
|
}
|
||||||
|
return c.json({ ok: true });
|
||||||
|
});
|
||||||
|
// Long-poll endpoint.
|
||||||
|
app.post('/queue', async (c) => {
|
||||||
|
const senderAddress = c.req.header('X-Shade-Sender-Address');
|
||||||
|
if (senderAddress === undefined || senderAddress === '') {
|
||||||
|
return c.json({ error: 'missing X-Shade-Sender-Address' }, 400);
|
||||||
|
}
|
||||||
|
let body: { since?: unknown; blockMs?: unknown };
|
||||||
|
try {
|
||||||
|
body = (await c.req.json()) as { since?: unknown; blockMs?: unknown };
|
||||||
|
} catch {
|
||||||
|
return c.json({ error: 'invalid JSON body' }, 400);
|
||||||
|
}
|
||||||
|
const since = typeof body.since === 'number' && Number.isFinite(body.since) ? body.since : 0;
|
||||||
|
const requestedBlockMs =
|
||||||
|
typeof body.blockMs === 'number' && Number.isFinite(body.blockMs)
|
||||||
|
? Math.max(0, Math.min(maxBlockMs, body.blockMs))
|
||||||
|
: blockMs;
|
||||||
|
// Bun-side short-circuit if the request was aborted while we
|
||||||
|
// were holding the long-poll. AbortSignal from the request body
|
||||||
|
// is already surfaced via `c.req.raw.signal` in Hono.
|
||||||
|
const events = await queue.drain(senderAddress, since, requestedBlockMs, c.req.raw.signal);
|
||||||
|
return c.json({
|
||||||
|
events: events.map((e) => ({
|
||||||
|
id: e.id,
|
||||||
|
timestampMs: e.timestampMs,
|
||||||
|
kind: e.kind,
|
||||||
|
bytesB64: bytesToBase64Std(e.bytes),
|
||||||
|
...(e.kind === 'chunk' ? { meta: e.meta } : {}),
|
||||||
|
})),
|
||||||
|
nextSince: events.length > 0 ? events[events.length - 1]!.id : since,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
return app;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Mount the receiver-side HTTP routes on a Hono app. Mount under any
|
* Mount the receiver-side HTTP routes on a Hono app. Mount under any
|
||||||
* base path: `app.route('/shade', await shade.transferRoute())`.
|
* base path: `app.route('/shade', await shade.transferRoute())`.
|
||||||
@@ -1019,16 +1166,23 @@ export class Shade {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
this.controlChannel = new ShadeControlChannel(this, this.envelopeOutboxes);
|
this.controlChannel = new ShadeControlChannel(this, this.envelopeOutboxes);
|
||||||
const httpTransport: ITransferTransport = new ShadeTransferHttpTransport({
|
let transport: ITransferTransport;
|
||||||
resolveBaseUrl: this.peerBaseUrlResolver,
|
|
||||||
authenticator: await this.makeAuthenticator(),
|
|
||||||
});
|
|
||||||
|
|
||||||
let transport: ITransferTransport = httpTransport;
|
|
||||||
let webrtcRuntime: ShadeWebRtcRuntime | null = null;
|
let webrtcRuntime: ShadeWebRtcRuntime | null = null;
|
||||||
if (this.webrtcConfig !== null) {
|
if (this.transferTransportOverride !== null) {
|
||||||
webrtcRuntime = await this.buildWebRtcRuntime(this.webrtcConfig, httpTransport);
|
// Custom transport (queue, in-memory, custom adapter) — used as-is.
|
||||||
transport = webrtcRuntime.fallback;
|
// WebRTC fallback only attaches when the default HTTP transport is
|
||||||
|
// active because WebRTC's `MultiTransportFallback` is HTTP-shaped.
|
||||||
|
transport = this.transferTransportOverride;
|
||||||
|
} else {
|
||||||
|
const httpTransport: ITransferTransport = new ShadeTransferHttpTransport({
|
||||||
|
resolveBaseUrl: this.peerBaseUrlResolver,
|
||||||
|
authenticator: await this.makeAuthenticator(),
|
||||||
|
});
|
||||||
|
transport = httpTransport;
|
||||||
|
if (this.webrtcConfig !== null) {
|
||||||
|
webrtcRuntime = await this.buildWebRtcRuntime(this.webrtcConfig, httpTransport);
|
||||||
|
transport = webrtcRuntime.fallback;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
this.transferEngine = new TransferEngine({
|
this.transferEngine = new TransferEngine({
|
||||||
@@ -1256,6 +1410,53 @@ function parseChunkHeader(bytes: Uint8Array): {
|
|||||||
return { streamId, laneId, seq };
|
return { streamId, laneId, seq };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ─── Queue-mode (pull) envelope transport ─────────────────────
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Configuration for {@link Shade.transferQueueRoute}. All fields are
|
||||||
|
* optional with sensible production defaults.
|
||||||
|
*/
|
||||||
|
export interface TransferQueueRouteOptions {
|
||||||
|
/**
|
||||||
|
* Long-poll timeout in milliseconds. Server holds the request open
|
||||||
|
* up to this long before returning an empty `events` array. Default
|
||||||
|
* 30_000.
|
||||||
|
*/
|
||||||
|
blockMs?: number;
|
||||||
|
/**
|
||||||
|
* Hard cap on long-poll timeout (clamps client-supplied `blockMs`).
|
||||||
|
* Default 55_000 — under typical reverse-proxy idle thresholds (60s
|
||||||
|
* on most CDNs).
|
||||||
|
*/
|
||||||
|
maxBlockMs?: number;
|
||||||
|
/**
|
||||||
|
* Per-peer ring-buffer size. When the queue is full, oldest events
|
||||||
|
* are dropped on enqueue. Receivers detect the gap via missing
|
||||||
|
* sequence numbers and re-resume from `since=0`. Default 1000.
|
||||||
|
*/
|
||||||
|
maxEventsPerPeer?: number;
|
||||||
|
/**
|
||||||
|
* Drop a peer's queue + reject pending pollers after this much
|
||||||
|
* silence. Default 10 minutes. Setting to `0` disables idle-eviction.
|
||||||
|
*/
|
||||||
|
idleEvictionMs?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* `ControlEnvelopeTransport` that enqueues outbound envelopes into an
|
||||||
|
* `OutboundQueue` for browser-style receivers to long-poll. Mirrors
|
||||||
|
* `HttpEnvelopeTransport` shape (one `send(peer, envelope)` method);
|
||||||
|
* the difference is the destination — local queue, not remote HTTP.
|
||||||
|
*/
|
||||||
|
class QueueEnvelopeTransport implements ControlEnvelopeTransport {
|
||||||
|
constructor(private readonly queue: OutboundQueueLike) {}
|
||||||
|
async send(peerAddress: string, envelope: ShadeEnvelope): Promise<void> {
|
||||||
|
const bytes = encodeEnvelope(envelope);
|
||||||
|
const event: QueuedEventInput = { kind: 'envelope', bytes };
|
||||||
|
this.queue.enqueue(peerAddress, event);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ─── Default HTTP envelope transport ──────────────────────────
|
// ─── Default HTTP envelope transport ──────────────────────────
|
||||||
|
|
||||||
class HttpEnvelopeTransport implements ControlEnvelopeTransport {
|
class HttpEnvelopeTransport implements ControlEnvelopeTransport {
|
||||||
|
|||||||
@@ -59,12 +59,16 @@ interface CreateImageBitmapFn {
|
|||||||
}
|
}
|
||||||
|
|
||||||
function getOffscreenCanvasCtor(): OffscreenCanvasCtor | null {
|
function getOffscreenCanvasCtor(): OffscreenCanvasCtor | null {
|
||||||
const g = globalThis as { OffscreenCanvas?: OffscreenCanvasCtor };
|
const g = globalThis as unknown as { OffscreenCanvas?: OffscreenCanvasCtor };
|
||||||
return g.OffscreenCanvas ?? null;
|
return g.OffscreenCanvas ?? null;
|
||||||
}
|
}
|
||||||
|
|
||||||
function getCreateImageBitmap(): CreateImageBitmapFn | null {
|
function getCreateImageBitmap(): CreateImageBitmapFn | null {
|
||||||
const g = globalThis as { createImageBitmap?: CreateImageBitmapFn };
|
// `globalThis.createImageBitmap` (when DOM lib is loaded) has a wider
|
||||||
|
// signature than our minimal `CreateImageBitmapFn`. Cast through
|
||||||
|
// `unknown` so consumer tsconfigs that include "DOM" don't reject the
|
||||||
|
// narrower local type as "insufficiently overlapping".
|
||||||
|
const g = globalThis as unknown as { createImageBitmap?: CreateImageBitmapFn };
|
||||||
return g.createImageBitmap ?? null;
|
return g.createImageBitmap ?? null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/server",
|
"name": "@shade/server",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/storage-encrypted",
|
"name": "@shade/storage-encrypted",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -11,11 +11,23 @@
|
|||||||
|
|
||||||
const NONCE_LEN = 12;
|
const NONCE_LEN = 12;
|
||||||
|
|
||||||
|
// Local mirror of the WebCrypto KeyUsage union — avoids depending on
|
||||||
|
// `lib.dom` (we run on Bun + plain ES2022) while keeping API parity.
|
||||||
|
type WebCryptoKeyUsage =
|
||||||
|
| 'encrypt'
|
||||||
|
| 'decrypt'
|
||||||
|
| 'sign'
|
||||||
|
| 'verify'
|
||||||
|
| 'deriveKey'
|
||||||
|
| 'deriveBits'
|
||||||
|
| 'wrapKey'
|
||||||
|
| 'unwrapKey';
|
||||||
|
|
||||||
function bs(u: Uint8Array): ArrayBuffer {
|
function bs(u: Uint8Array): ArrayBuffer {
|
||||||
return u as unknown as ArrayBuffer;
|
return u as unknown as ArrayBuffer;
|
||||||
}
|
}
|
||||||
|
|
||||||
async function importKey(key: Uint8Array, usages: KeyUsage[]): Promise<CryptoKey> {
|
async function importKey(key: Uint8Array, usages: WebCryptoKeyUsage[]): Promise<CryptoKey> {
|
||||||
if (key.length !== 32) throw new Error(`AES-256-GCM key must be 32 bytes, got ${key.length}`);
|
if (key.length !== 32) throw new Error(`AES-256-GCM key must be 32 bytes, got ${key.length}`);
|
||||||
return globalThis.crypto.subtle.importKey('raw', bs(key), 'AES-GCM', false, usages);
|
return globalThis.crypto.subtle.importKey('raw', bs(key), 'AES-GCM', false, usages);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
{
|
{
|
||||||
"extends": "../../tsconfig.json",
|
"extends": "../../tsconfig.json",
|
||||||
"include": ["src/**/*", "tests/**/*"]
|
"compilerOptions": { "outDir": "dist", "rootDir": "src" },
|
||||||
|
"include": ["src"]
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/storage-postgres",
|
"name": "@shade/storage-postgres",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/storage-sqlite",
|
"name": "@shade/storage-sqlite",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/streams",
|
"name": "@shade/streams",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/transfer",
|
"name": "@shade/transfer",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ export * from './transport/memory.js';
|
|||||||
export * from './transport/http-transport.js';
|
export * from './transport/http-transport.js';
|
||||||
export * from './transport/ws-transport.js';
|
export * from './transport/ws-transport.js';
|
||||||
export * from './transport/multi-fallback.js';
|
export * from './transport/multi-fallback.js';
|
||||||
|
export * from './transport/queue-transport.js';
|
||||||
export * from './engine.js';
|
export * from './engine.js';
|
||||||
export {
|
export {
|
||||||
createTransferRoutes,
|
createTransferRoutes,
|
||||||
|
|||||||
319
packages/shade-transfer/src/transport/queue-transport.ts
Normal file
319
packages/shade-transfer/src/transport/queue-transport.ts
Normal file
@@ -0,0 +1,319 @@
|
|||||||
|
/**
|
||||||
|
* Per-peer outbound queue + queue-transport for browser-friendly
|
||||||
|
* pull-mode streams.
|
||||||
|
*
|
||||||
|
* The default `ShadeTransferHttpTransport` POSTs each chunk directly
|
||||||
|
* to the receiver's `/v1/transfer/<streamId>/chunk` route. That
|
||||||
|
* requires the receiver to host an HTTP server, which a browser tab
|
||||||
|
* cannot. `QueueTransferTransport` flips the direction: it queues
|
||||||
|
* chunks per peer and lets the receiver pull them via a long-poll
|
||||||
|
* endpoint.
|
||||||
|
*
|
||||||
|
* The companion `OutboundQueue` data structure is plain server-side
|
||||||
|
* state — wired up by `@shade/files`'s `transferQueueRoute()` (and any
|
||||||
|
* future consumer) to expose the long-poll surface and feed envelopes
|
||||||
|
* + chunks into the browser receiver.
|
||||||
|
*/
|
||||||
|
import type {
|
||||||
|
ChunkAck,
|
||||||
|
ChunkSendOptions,
|
||||||
|
ITransferTransport,
|
||||||
|
TransferResumeState,
|
||||||
|
} from './transport.js';
|
||||||
|
import { TransferTransportError } from '../errors.js';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Discriminated union of items the queue ships to the browser
|
||||||
|
* receiver. Both kinds carry **wire-encoded bytes** of an envelope —
|
||||||
|
* the receiver decodes via `decodeEnvelope` (control envelopes) or
|
||||||
|
* forwards directly to `engine.receiveChunk` (chunk envelopes).
|
||||||
|
*/
|
||||||
|
export type QueuedEvent =
|
||||||
|
| {
|
||||||
|
id: number;
|
||||||
|
timestampMs: number;
|
||||||
|
kind: 'envelope';
|
||||||
|
/** Wire-encoded `0x02` ratchet envelope (or `0x01` first-message). */
|
||||||
|
bytes: Uint8Array;
|
||||||
|
}
|
||||||
|
| {
|
||||||
|
id: number;
|
||||||
|
timestampMs: number;
|
||||||
|
kind: 'chunk';
|
||||||
|
/** Wire-encoded `0x11` stream-chunk envelope. */
|
||||||
|
bytes: Uint8Array;
|
||||||
|
meta: {
|
||||||
|
streamId: string;
|
||||||
|
laneId: number;
|
||||||
|
seq: number;
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
/** Caller-supplied shape for {@link OutboundQueue.enqueue} — the queue assigns `id` + `timestampMs`. */
|
||||||
|
export type QueuedEventInput =
|
||||||
|
| { kind: 'envelope'; bytes: Uint8Array }
|
||||||
|
| {
|
||||||
|
kind: 'chunk';
|
||||||
|
bytes: Uint8Array;
|
||||||
|
meta: { streamId: string; laneId: number; seq: number };
|
||||||
|
};
|
||||||
|
|
||||||
|
export interface OutboundQueueOptions {
|
||||||
|
/**
|
||||||
|
* Maximum events held per peer. When the queue is full, the oldest
|
||||||
|
* unacked event is dropped on next enqueue. Default 1000 — at the
|
||||||
|
* default chunk size (256 KiB plaintext) this caps a single peer's
|
||||||
|
* outbound buffer at ~256 MiB. Tune up for fewer/bigger streams,
|
||||||
|
* down for many concurrent small flows.
|
||||||
|
*/
|
||||||
|
maxEventsPerPeer?: number;
|
||||||
|
/**
|
||||||
|
* After a peer has not polled for this long, the queue's events are
|
||||||
|
* dropped and any pending waiters are released. Default 10 minutes.
|
||||||
|
* Setting to `0` disables idle-eviction.
|
||||||
|
*/
|
||||||
|
idleEvictionMs?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
const DEFAULT_MAX_EVENTS = 1000;
|
||||||
|
const DEFAULT_IDLE_EVICTION_MS = 10 * 60 * 1000;
|
||||||
|
|
||||||
|
interface PendingWaiter {
|
||||||
|
resolve(events: QueuedEvent[]): void;
|
||||||
|
reject(err: Error): void;
|
||||||
|
timer: ReturnType<typeof setTimeout>;
|
||||||
|
abortHandler?: () => void;
|
||||||
|
signal?: AbortSignal;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface PerPeerState {
|
||||||
|
nextId: number;
|
||||||
|
events: QueuedEvent[];
|
||||||
|
waiters: PendingWaiter[];
|
||||||
|
lastTouchedMs: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Per-peer monotonic event log with long-poll semantics.
|
||||||
|
*
|
||||||
|
* `enqueue` appends; `drain` returns all events with `id > since`,
|
||||||
|
* blocking up to `blockMs` if there are none. `since`-based pagination
|
||||||
|
* is the resume mechanism: a client crashing mid-stream restarts with
|
||||||
|
* its last-processed id and the queue replays everything after it
|
||||||
|
* (subject to `maxEventsPerPeer` retention).
|
||||||
|
*/
|
||||||
|
export class OutboundQueue {
|
||||||
|
private peers = new Map<string, PerPeerState>();
|
||||||
|
private readonly maxEvents: number;
|
||||||
|
private readonly idleEvictionMs: number;
|
||||||
|
private evictTimer: ReturnType<typeof setTimeout> | null = null;
|
||||||
|
private destroyed = false;
|
||||||
|
|
||||||
|
constructor(opts: OutboundQueueOptions = {}) {
|
||||||
|
this.maxEvents = opts.maxEventsPerPeer ?? DEFAULT_MAX_EVENTS;
|
||||||
|
this.idleEvictionMs = opts.idleEvictionMs ?? DEFAULT_IDLE_EVICTION_MS;
|
||||||
|
if (this.idleEvictionMs > 0) this.scheduleEviction();
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Append an event and wake any waiters for that peer. */
|
||||||
|
enqueue(peer: string, ev: QueuedEventInput): QueuedEvent {
|
||||||
|
if (this.destroyed) throw new Error('OutboundQueue: destroyed');
|
||||||
|
const state = this.getOrCreate(peer);
|
||||||
|
const event: QueuedEvent =
|
||||||
|
ev.kind === 'chunk'
|
||||||
|
? {
|
||||||
|
id: state.nextId++,
|
||||||
|
timestampMs: Date.now(),
|
||||||
|
kind: 'chunk',
|
||||||
|
bytes: ev.bytes,
|
||||||
|
meta: ev.meta,
|
||||||
|
}
|
||||||
|
: {
|
||||||
|
id: state.nextId++,
|
||||||
|
timestampMs: Date.now(),
|
||||||
|
kind: 'envelope',
|
||||||
|
bytes: ev.bytes,
|
||||||
|
};
|
||||||
|
state.events.push(event);
|
||||||
|
state.lastTouchedMs = Date.now();
|
||||||
|
// Cap: drop oldest. Lost events trigger receiver-side resume from
|
||||||
|
// last polled id; the @shade/transfer engine handles missing seqs
|
||||||
|
// by re-sending on resume.
|
||||||
|
while (state.events.length > this.maxEvents) state.events.shift();
|
||||||
|
// Wake all waiters with whatever has accumulated.
|
||||||
|
const drained = this.collect(state, 0);
|
||||||
|
if (drained.length > 0) {
|
||||||
|
const waiters = state.waiters.splice(0);
|
||||||
|
for (const w of waiters) {
|
||||||
|
clearTimeout(w.timer);
|
||||||
|
if (w.abortHandler !== undefined && w.signal !== undefined) {
|
||||||
|
w.signal.removeEventListener('abort', w.abortHandler);
|
||||||
|
}
|
||||||
|
w.resolve(drained);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return event;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Drain events with `id > since`. If none are available, block up
|
||||||
|
* to `blockMs` until any arrive. `signal` cancels the wait early.
|
||||||
|
*/
|
||||||
|
async drain(
|
||||||
|
peer: string,
|
||||||
|
since: number,
|
||||||
|
blockMs: number,
|
||||||
|
signal?: AbortSignal,
|
||||||
|
): Promise<QueuedEvent[]> {
|
||||||
|
if (this.destroyed) throw new Error('OutboundQueue: destroyed');
|
||||||
|
const state = this.getOrCreate(peer);
|
||||||
|
state.lastTouchedMs = Date.now();
|
||||||
|
const ready = this.collect(state, since);
|
||||||
|
if (ready.length > 0 || blockMs <= 0) return ready;
|
||||||
|
if (signal?.aborted) return [];
|
||||||
|
return await new Promise<QueuedEvent[]>((resolve, reject) => {
|
||||||
|
const timer = setTimeout(() => {
|
||||||
|
const idx = state.waiters.indexOf(waiter);
|
||||||
|
if (idx >= 0) state.waiters.splice(idx, 1);
|
||||||
|
if (waiter.abortHandler !== undefined && waiter.signal !== undefined) {
|
||||||
|
waiter.signal.removeEventListener('abort', waiter.abortHandler);
|
||||||
|
}
|
||||||
|
// Empty drain on timeout — that's the "no new events" signal.
|
||||||
|
resolve([]);
|
||||||
|
}, blockMs);
|
||||||
|
const waiter: PendingWaiter = { resolve, reject, timer };
|
||||||
|
if (signal !== undefined) {
|
||||||
|
const handler = () => {
|
||||||
|
const idx = state.waiters.indexOf(waiter);
|
||||||
|
if (idx >= 0) state.waiters.splice(idx, 1);
|
||||||
|
clearTimeout(timer);
|
||||||
|
resolve([]);
|
||||||
|
};
|
||||||
|
signal.addEventListener('abort', handler, { once: true });
|
||||||
|
waiter.abortHandler = handler;
|
||||||
|
waiter.signal = signal;
|
||||||
|
}
|
||||||
|
state.waiters.push(waiter);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Drop a peer's queue + reject waiters. */
|
||||||
|
evict(peer: string): void {
|
||||||
|
const state = this.peers.get(peer);
|
||||||
|
if (state === undefined) return;
|
||||||
|
this.peers.delete(peer);
|
||||||
|
for (const w of state.waiters) {
|
||||||
|
clearTimeout(w.timer);
|
||||||
|
if (w.abortHandler !== undefined && w.signal !== undefined) {
|
||||||
|
w.signal.removeEventListener('abort', w.abortHandler);
|
||||||
|
}
|
||||||
|
w.reject(new Error('OutboundQueue: peer evicted'));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Peer-specific snapshot for diagnostics. */
|
||||||
|
stats(peer: string): { eventCount: number; nextId: number; waiters: number } | null {
|
||||||
|
const state = this.peers.get(peer);
|
||||||
|
if (state === undefined) return null;
|
||||||
|
return {
|
||||||
|
eventCount: state.events.length,
|
||||||
|
nextId: state.nextId,
|
||||||
|
waiters: state.waiters.length,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Tear everything down. Pending waiters are rejected. */
|
||||||
|
destroy(): void {
|
||||||
|
if (this.destroyed) return;
|
||||||
|
this.destroyed = true;
|
||||||
|
if (this.evictTimer !== null) clearTimeout(this.evictTimer);
|
||||||
|
for (const peer of [...this.peers.keys()]) this.evict(peer);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── internals ──────────────────────────────────────────────
|
||||||
|
|
||||||
|
private getOrCreate(peer: string): PerPeerState {
|
||||||
|
let state = this.peers.get(peer);
|
||||||
|
if (state === undefined) {
|
||||||
|
state = {
|
||||||
|
nextId: 1,
|
||||||
|
events: [],
|
||||||
|
waiters: [],
|
||||||
|
lastTouchedMs: Date.now(),
|
||||||
|
};
|
||||||
|
this.peers.set(peer, state);
|
||||||
|
}
|
||||||
|
return state;
|
||||||
|
}
|
||||||
|
|
||||||
|
private collect(state: PerPeerState, since: number): QueuedEvent[] {
|
||||||
|
if (state.events.length === 0) return [];
|
||||||
|
return state.events.filter((e) => e.id > since);
|
||||||
|
}
|
||||||
|
|
||||||
|
private scheduleEviction(): void {
|
||||||
|
const interval = Math.max(60_000, Math.floor(this.idleEvictionMs / 4));
|
||||||
|
this.evictTimer = setTimeout(() => {
|
||||||
|
if (this.destroyed) return;
|
||||||
|
const cutoff = Date.now() - this.idleEvictionMs;
|
||||||
|
for (const [peer, state] of this.peers.entries()) {
|
||||||
|
if (state.lastTouchedMs < cutoff) this.evict(peer);
|
||||||
|
}
|
||||||
|
this.scheduleEviction();
|
||||||
|
}, interval);
|
||||||
|
(this.evictTimer as unknown as { unref?: () => void }).unref?.();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Chunk transport that enqueues into an {@link OutboundQueue} instead
|
||||||
|
* of POSTing.
|
||||||
|
*
|
||||||
|
* Returns an optimistic `ChunkAck` immediately because the queue *is*
|
||||||
|
* the delivery — the receiver polls and dispatches. Browser receivers
|
||||||
|
* cannot synchronously confirm receipt before the next chunk; the
|
||||||
|
* engine's stream-protocol catches dropped chunks at finish-time
|
||||||
|
* integrity check, and chunk-resume restarts the lane from the last
|
||||||
|
* polled `since`.
|
||||||
|
*/
|
||||||
|
export class QueueTransferTransport implements ITransferTransport {
|
||||||
|
constructor(private readonly queue: OutboundQueue) {}
|
||||||
|
|
||||||
|
async probe(_peer: string): Promise<void> {
|
||||||
|
// The queue is local. Reachability is "is there a poller?" which
|
||||||
|
// is decided by `idleEvictionMs`. We don't synchronously check
|
||||||
|
// here; the engine retries via `withRetry` on `sendChunk` errors.
|
||||||
|
}
|
||||||
|
|
||||||
|
async sendChunk(
|
||||||
|
peer: string,
|
||||||
|
streamId: string,
|
||||||
|
laneId: number,
|
||||||
|
seq: number | bigint,
|
||||||
|
bytes: Uint8Array,
|
||||||
|
options?: ChunkSendOptions,
|
||||||
|
): Promise<ChunkAck> {
|
||||||
|
if (options?.signal?.aborted) {
|
||||||
|
throw new TransferTransportError('sendChunk aborted by caller');
|
||||||
|
}
|
||||||
|
const seqNum = typeof seq === 'bigint' ? Number(seq) : seq;
|
||||||
|
this.queue.enqueue(peer, {
|
||||||
|
kind: 'chunk',
|
||||||
|
bytes,
|
||||||
|
meta: { streamId, laneId, seq: seqNum },
|
||||||
|
});
|
||||||
|
return { lastSeq: seqNum };
|
||||||
|
}
|
||||||
|
|
||||||
|
async fetchResumeState(
|
||||||
|
_peer: string,
|
||||||
|
_streamId: string,
|
||||||
|
): Promise<TransferResumeState | null> {
|
||||||
|
// Pull-mode receivers report resume state by re-polling with the
|
||||||
|
// `since` cursor they last successfully processed; the queue does
|
||||||
|
// not need to query the receiver. Return null so the engine
|
||||||
|
// restarts from seq 0 (deterministic), and the queue replays from
|
||||||
|
// `since=0` if the client reconnects fresh.
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/transport-bridge",
|
"name": "@shade/transport-bridge",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -127,7 +127,7 @@ export class SseBridge implements BridgeTransport {
|
|||||||
if (!res.body) {
|
if (!res.body) {
|
||||||
throw new BridgeError('SSE response has no body');
|
throw new BridgeError('SSE response has no body');
|
||||||
}
|
}
|
||||||
this.currentReader = res.body.getReader();
|
this.currentReader = res.body.getReader() as ReadableStreamDefaultReader<Uint8Array>;
|
||||||
this.connected = true;
|
this.connected = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/transport-webrtc",
|
"name": "@shade/transport-webrtc",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/transport",
|
"name": "@shade/transport",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@shade/widgets",
|
"name": "@shade/widgets",
|
||||||
"version": "4.0.0",
|
"version": "4.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
"types": "src/index.ts",
|
"types": "src/index.ts",
|
||||||
|
|||||||
@@ -2,7 +2,6 @@ import React, { useEffect, useRef, useState } from 'react';
|
|||||||
import {
|
import {
|
||||||
isAllowedThumbnailMime,
|
isAllowedThumbnailMime,
|
||||||
THUMBNAIL_MAX_BYTES,
|
THUMBNAIL_MAX_BYTES,
|
||||||
type ThumbnailMime,
|
|
||||||
} from '@shade/sdk';
|
} from '@shade/sdk';
|
||||||
import { useShadeRuntimeTheme } from '../../ShadeRuntimeProvider.js';
|
import { useShadeRuntimeTheme } from '../../ShadeRuntimeProvider.js';
|
||||||
|
|
||||||
|
|||||||
@@ -127,6 +127,10 @@ export GITEA_TOKEN="$TOKEN"
|
|||||||
|
|
||||||
cd "$SHADE_DIR"
|
cd "$SHADE_DIR"
|
||||||
echo
|
echo
|
||||||
echo "Kjører bun run publish:all i $SHADE_DIR"
|
echo "Type-check (strict TS) før publish ..."
|
||||||
echo "----------------------------------------"
|
echo "----------------------------------------"
|
||||||
bun run publish:all
|
bun run scripts/typecheck-all.ts
|
||||||
|
echo
|
||||||
|
echo "Kjører scripts/publish-all.ts i $SHADE_DIR"
|
||||||
|
echo "----------------------------------------"
|
||||||
|
bun run scripts/publish-all.ts
|
||||||
|
|||||||
108
scripts/typecheck-all.ts
Normal file
108
scripts/typecheck-all.ts
Normal file
@@ -0,0 +1,108 @@
|
|||||||
|
#!/usr/bin/env bun
|
||||||
|
/**
|
||||||
|
* Pre-publish gate — type-check every workspace package against the
|
||||||
|
* monorepo's strict tsconfig.
|
||||||
|
*
|
||||||
|
* Required before any publish. The Bun test runner is intentionally
|
||||||
|
* permissive (it transpiles, doesn't type-check), so without this gate
|
||||||
|
* a package can pass `bun test` and still ship code that fails to
|
||||||
|
* compile in a downstream consumer's strict TS project.
|
||||||
|
*
|
||||||
|
* Usage:
|
||||||
|
* bun run scripts/typecheck-all.ts # check every package
|
||||||
|
* bun run scripts/typecheck-all.ts core sdk # check only listed
|
||||||
|
*
|
||||||
|
* Exit code 0 if every package compiles, 1 otherwise.
|
||||||
|
*/
|
||||||
|
import { readdirSync, statSync, existsSync } from 'fs';
|
||||||
|
import { join } from 'path';
|
||||||
|
import { $ } from 'bun';
|
||||||
|
|
||||||
|
const ROOT = join(import.meta.dir, '..');
|
||||||
|
const PACKAGES_DIR = join(ROOT, 'packages');
|
||||||
|
|
||||||
|
const filter = new Set(process.argv.slice(2));
|
||||||
|
|
||||||
|
const packages = readdirSync(PACKAGES_DIR).filter((name) => {
|
||||||
|
const p = join(PACKAGES_DIR, name);
|
||||||
|
if (!statSync(p).isDirectory()) return false;
|
||||||
|
if (!existsSync(join(p, 'tsconfig.json'))) return false;
|
||||||
|
if (filter.size > 0 && !filter.has(name) && !filter.has(name.replace(/^shade-/, ''))) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
|
||||||
|
let failures = 0;
|
||||||
|
const failed: { pkg: string; out: string }[] = [];
|
||||||
|
|
||||||
|
for (const pkg of packages) {
|
||||||
|
const dir = join(PACKAGES_DIR, pkg);
|
||||||
|
const proc = Bun.spawnSync(['bunx', 'tsc', '--noEmit', '-p', 'tsconfig.json'], {
|
||||||
|
cwd: dir,
|
||||||
|
stdout: 'pipe',
|
||||||
|
stderr: 'pipe',
|
||||||
|
});
|
||||||
|
const stdout = proc.stdout.toString();
|
||||||
|
const stderr = proc.stderr.toString();
|
||||||
|
const out = (stdout + stderr)
|
||||||
|
.split('\n')
|
||||||
|
.filter((l) => !/^Resolving|^Resolved|^Saved/.test(l))
|
||||||
|
.join('\n')
|
||||||
|
.trim();
|
||||||
|
|
||||||
|
if (proc.exitCode === 0 && out.length === 0) {
|
||||||
|
console.log(` ✓ ${pkg}`);
|
||||||
|
} else {
|
||||||
|
failures++;
|
||||||
|
failed.push({ pkg, out });
|
||||||
|
console.log(` ✗ ${pkg}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log();
|
||||||
|
|
||||||
|
// Step 2 — consumer-strict smoke. Compiles a tiny "as if I were a
|
||||||
|
// downstream app" project against our public API surface under the
|
||||||
|
// consumer-likely tsconfig (`lib: ["DOM"]` + `exactOptionalPropertyTypes`).
|
||||||
|
// Catches type-bugs that ONLY surface when our internal narrower types
|
||||||
|
// meet a consumer's standard-library types — the class of bug `tsc`
|
||||||
|
// inside our own packages does not see (because our packages compile
|
||||||
|
// against `lib: ["ES2022"]` only).
|
||||||
|
if (filter.size === 0) {
|
||||||
|
console.log('Consumer-strict smoke (lib: DOM, exactOptional, paths→workspace) ...');
|
||||||
|
const consumerDir = join(ROOT, 'tests', 'consumer-strict');
|
||||||
|
if (existsSync(join(consumerDir, 'tsconfig.json'))) {
|
||||||
|
const proc = Bun.spawnSync(['bunx', 'tsc', '--noEmit', '-p', 'tsconfig.json'], {
|
||||||
|
cwd: consumerDir,
|
||||||
|
stdout: 'pipe',
|
||||||
|
stderr: 'pipe',
|
||||||
|
});
|
||||||
|
const out = (proc.stdout.toString() + proc.stderr.toString())
|
||||||
|
.split('\n')
|
||||||
|
.filter((l) => !/^Resolving|^Resolved|^Saved/.test(l))
|
||||||
|
.join('\n')
|
||||||
|
.trim();
|
||||||
|
if (proc.exitCode === 0 && out.length === 0) {
|
||||||
|
console.log(' ✓ consumer-strict');
|
||||||
|
} else {
|
||||||
|
failures++;
|
||||||
|
failed.push({ pkg: 'consumer-strict', out });
|
||||||
|
console.log(' ✗ consumer-strict');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
console.log();
|
||||||
|
}
|
||||||
|
|
||||||
|
if (failures === 0) {
|
||||||
|
console.log(`All ${packages.length} packages type-check cleanly.`);
|
||||||
|
process.exit(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
console.error(`${failures} of ${packages.length} packages failed:\n`);
|
||||||
|
for (const f of failed) {
|
||||||
|
console.error(`── ${f.pkg} ──`);
|
||||||
|
console.error(f.out);
|
||||||
|
console.error();
|
||||||
|
}
|
||||||
|
process.exit(1);
|
||||||
45
tests/consumer-strict/tsconfig.json
Normal file
45
tests/consumer-strict/tsconfig.json
Normal file
@@ -0,0 +1,45 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ESNext",
|
||||||
|
"module": "ESNext",
|
||||||
|
"moduleResolution": "bundler",
|
||||||
|
"strict": true,
|
||||||
|
"noUnusedLocals": false,
|
||||||
|
"noUnusedParameters": true,
|
||||||
|
"noImplicitOverride": true,
|
||||||
|
"exactOptionalPropertyTypes": true,
|
||||||
|
"lib": ["ES2022", "DOM", "DOM.Iterable"],
|
||||||
|
"esModuleInterop": true,
|
||||||
|
"skipLibCheck": true,
|
||||||
|
"noEmit": true,
|
||||||
|
"types": ["bun-types"],
|
||||||
|
"ignoreDeprecations": "6.0",
|
||||||
|
"baseUrl": ".",
|
||||||
|
"paths": {
|
||||||
|
"@shade/core": ["../../packages/shade-core/src/index.ts"],
|
||||||
|
"@shade/proto": ["../../packages/shade-proto/src/index.ts"],
|
||||||
|
"@shade/crypto-web": ["../../packages/shade-crypto-web/src/index.ts"],
|
||||||
|
"@shade/observability": ["../../packages/shade-observability/src/index.ts"],
|
||||||
|
"@shade/keychain": ["../../packages/shade-keychain/src/index.ts"],
|
||||||
|
"@shade/key-transparency": ["../../packages/shade-key-transparency/src/index.ts"],
|
||||||
|
"@shade/storage-sqlite": ["../../packages/shade-storage-sqlite/src/index.ts"],
|
||||||
|
"@shade/storage-postgres": ["../../packages/shade-storage-postgres/src/index.ts"],
|
||||||
|
"@shade/storage-encrypted": ["../../packages/shade-storage-encrypted/src/index.ts"],
|
||||||
|
"@shade/streams": ["../../packages/shade-streams/src/index.ts"],
|
||||||
|
"@shade/transport": ["../../packages/shade-transport/src/index.ts"],
|
||||||
|
"@shade/transport-bridge": ["../../packages/shade-transport-bridge/src/index.ts"],
|
||||||
|
"@shade/transport-webrtc": ["../../packages/shade-transport-webrtc/src/index.ts"],
|
||||||
|
"@shade/server": ["../../packages/shade-server/src/index.ts"],
|
||||||
|
"@shade/inbox-server": ["../../packages/shade-inbox-server/src/index.ts"],
|
||||||
|
"@shade/inbox": ["../../packages/shade-inbox/src/index.ts"],
|
||||||
|
"@shade/transfer": ["../../packages/shade-transfer/src/index.ts"],
|
||||||
|
"@shade/files": ["../../packages/shade-files/src/index.ts"],
|
||||||
|
"@shade/recovery": ["../../packages/shade-recovery/src/index.ts"],
|
||||||
|
"@shade/observer": ["../../packages/shade-observer/src/index.ts"],
|
||||||
|
"@shade/dashboard": ["../../packages/shade-dashboard/src/index.ts"],
|
||||||
|
"@shade/sdk": ["../../packages/shade-sdk/src/index.ts"],
|
||||||
|
"@shade/widgets": ["../../packages/shade-widgets/src/index.ts"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"include": ["./*.ts"]
|
||||||
|
}
|
||||||
43
tests/consumer-strict/use-files.ts
Normal file
43
tests/consumer-strict/use-files.ts
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
/**
|
||||||
|
* Consumer-strict smoke for `@shade/files`.
|
||||||
|
*
|
||||||
|
* Compiled with `lib: ["ES2022", "DOM"]` + `exactOptionalPropertyTypes` +
|
||||||
|
* `skipLibCheck: false` to mimic a downstream consumer like Dispatch.
|
||||||
|
* Catches the class of bug where our internal narrower types (e.g. a
|
||||||
|
* locally-defined `MinimalReader`) reject native browser types
|
||||||
|
* (e.g. `ReadableStreamDefaultReader`) the consumer would naturally
|
||||||
|
* pass in.
|
||||||
|
*
|
||||||
|
* If this file fails to compile, the published packages will fail in
|
||||||
|
* any consumer's strict tsc — pre-publish gate must catch it.
|
||||||
|
*/
|
||||||
|
import { decideInline, type WriteSource } from '@shade/files';
|
||||||
|
|
||||||
|
declare const blob: Blob;
|
||||||
|
declare const stream: ReadableStream<Uint8Array>;
|
||||||
|
declare const bytes: Uint8Array;
|
||||||
|
|
||||||
|
async function smoke(): Promise<void> {
|
||||||
|
// Each branch of WriteSource must round-trip through decideInline()
|
||||||
|
// when given the natively-typed inputs a browser app would supply.
|
||||||
|
const sources: WriteSource[] = [
|
||||||
|
bytes,
|
||||||
|
blob,
|
||||||
|
stream,
|
||||||
|
{ stream, size: 1024 },
|
||||||
|
{ stream, size: 1024, contentType: 'image/png' },
|
||||||
|
];
|
||||||
|
for (const src of sources) {
|
||||||
|
const decision = await decideInline(src);
|
||||||
|
if (decision.kind === 'streams') {
|
||||||
|
const reader = decision.stream.getReader();
|
||||||
|
const { value, done } = await reader.read();
|
||||||
|
if (!done && value !== undefined) {
|
||||||
|
void value.byteLength;
|
||||||
|
}
|
||||||
|
reader.releaseLock();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void smoke;
|
||||||
42
tests/consumer-strict/use-key-transparency.ts
Normal file
42
tests/consumer-strict/use-key-transparency.ts
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
/**
|
||||||
|
* Consumer-strict smoke for `@shade/key-transparency`.
|
||||||
|
*
|
||||||
|
* The package was the source of the 4 noUnusedLocals + the IndexProofWire
|
||||||
|
* privacy bug in 4.0.0. This smoke imports every public type and
|
||||||
|
* exercises the witness-fetcher contract that the SDK plugs into.
|
||||||
|
*/
|
||||||
|
import {
|
||||||
|
LightWitness,
|
||||||
|
type SignedTreeHead,
|
||||||
|
type STHWire,
|
||||||
|
type WitnessFetcher,
|
||||||
|
} from '@shade/key-transparency';
|
||||||
|
|
||||||
|
declare const crypto: import('@shade/core').CryptoProvider;
|
||||||
|
declare const logPublicKey: Uint8Array;
|
||||||
|
|
||||||
|
async function smoke(): Promise<void> {
|
||||||
|
const fetcher: WitnessFetcher = {
|
||||||
|
async fetchLatestSTH(): Promise<STHWire> {
|
||||||
|
const res = await fetch('https://shade.example.com/v1/kt/sth');
|
||||||
|
return (await res.json()) as STHWire;
|
||||||
|
},
|
||||||
|
async fetchConsistencyProof(
|
||||||
|
from: number,
|
||||||
|
to: number,
|
||||||
|
): Promise<{ proof: string[] }> {
|
||||||
|
const res = await fetch(
|
||||||
|
`https://shade.example.com/v1/kt/consistency?from=${from}&to=${to}`,
|
||||||
|
);
|
||||||
|
return (await res.json()) as { proof: string[] };
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const witness = new LightWitness({ crypto, logPublicKey, fetcher });
|
||||||
|
void witness;
|
||||||
|
}
|
||||||
|
|
||||||
|
void smoke;
|
||||||
|
|
||||||
|
// Verify the type is reachable with no `any` leak.
|
||||||
|
declare const sth: SignedTreeHead;
|
||||||
|
void sth.treeSize;
|
||||||
50
tests/consumer-strict/use-sdk.ts
Normal file
50
tests/consumer-strict/use-sdk.ts
Normal file
@@ -0,0 +1,50 @@
|
|||||||
|
/**
|
||||||
|
* Consumer-strict smoke for `@shade/sdk`.
|
||||||
|
*
|
||||||
|
* Exercises the high-level `createShade()` flow + the V3.x opt-in
|
||||||
|
* surfaces (KT, WebRTC, fingerprint gates). Compiles under DOM-lib +
|
||||||
|
* exactOptionalPropertyTypes to flag any private-type leaks like the
|
||||||
|
* `Promise<unknown>` on `fetchLatestSTH` that 4.0.0 shipped.
|
||||||
|
*/
|
||||||
|
import {
|
||||||
|
createShade,
|
||||||
|
type Shade,
|
||||||
|
type ShadeConfig,
|
||||||
|
type ShadeWebRtcConfig,
|
||||||
|
} from '@shade/sdk';
|
||||||
|
|
||||||
|
declare const factory: ShadeWebRtcConfig['factory'];
|
||||||
|
|
||||||
|
async function smoke(): Promise<void> {
|
||||||
|
const config: ShadeConfig = {
|
||||||
|
prekeyServer: 'https://shade.example.com',
|
||||||
|
storage: 'memory',
|
||||||
|
address: 'alice@example.com',
|
||||||
|
keyTransparency: {
|
||||||
|
mode: 'observe-strict',
|
||||||
|
logPublicKey: new Uint8Array(32),
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const shade: Shade = await createShade(config);
|
||||||
|
|
||||||
|
const env = await shade.send('bob', 'hi');
|
||||||
|
void env;
|
||||||
|
|
||||||
|
shade.onMessage(async (from: string, plaintext: string) => {
|
||||||
|
void from;
|
||||||
|
void plaintext;
|
||||||
|
});
|
||||||
|
|
||||||
|
await shade.beforeFirstLargeFile(10 * 1024 * 1024, async (ctx) => {
|
||||||
|
void ctx.peerAddress;
|
||||||
|
void ctx.fingerprint;
|
||||||
|
void ctx.gate;
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
|
||||||
|
shade.configureWebRTC({ factory });
|
||||||
|
|
||||||
|
void (await shade.fingerprint);
|
||||||
|
}
|
||||||
|
|
||||||
|
void smoke;
|
||||||
Reference in New Issue
Block a user