release(v4.2.0): pull-mode streams for browser @shade/files
Some checks failed
Test / test (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled

4.1.0's HTTP RPC for browsers capped at inline payloads (≤ 256 KiB).
4.2.0 unlocks streams: server queues outbound chunks + control
envelopes per peer, browser long-polls the queue. Browser-to-server
writes ride the existing /v1/transfer/<id>/chunk POST routes
unchanged.

For Dispatch this unlocks mod-jar uploads (50 MB) and world-backup
downloads (100+ MB) — the actual reason browser-side @shade/files
matters.

### New API

@shade/sdk:
- shade.transferQueueRoute(opts?) — Hono app with /queue +
  /v1/transfer/* routes. Auto-configures the queue transport.
- shade.configureTransfers extended: transport + envelopeTransport
  override slots; resolveBaseUrl optional when both supplied.

@shade/transfer:
- OutboundQueue — per-peer monotonic event log with long-poll
  semantics, idle-eviction GC, ring-buffered to maxEventsPerPeer.
- QueueTransferTransport — enqueues instead of POSTing.

@shade/files:
- httpClient({ outboundQueueUrl, transferBaseUrl }) — when set,
  starts a long-poll drainer + builds a streams-bridge. fs.read /
  fs.write of >256 KiB work end-to-end.
- startQueueDrainer(shade, opts) — exported helper for advanced
  consumers driving their own drainer.

### Implementation notes

- ClientStreamsBridge's TransformStream had HWM=0 by default which
  stalled the drainer's await chain at chunk 4 (writer.write pended
  before the consumer's reader was attached). Bumped to HWM=64 so
  the receive loop can buffer ahead of the consumer.

### Tests

3 new integration tests in tests/integration/http-rpc-streams.test.ts:
4 MiB streamed read round-trip, inline-only error path, idle-timeout
long-poll behaviour.

Wire-compatible. Source-compatible. Lockstep bump to 4.2.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-03 23:27:06 +02:00
parent da93b97cce
commit 7520b11b25
34 changed files with 1331 additions and 59 deletions

View File

@@ -5,6 +5,151 @@ All notable changes to Shade are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [4.2.0] — 2026-05-03 — Pull-mode streams for browser @shade/files
`4.1.0` shipped HTTP RPC for browser clients but capped them at inline
payloads (≤ 256 KiB). Larger reads/writes — mod-jars (150 MB),
world-backups (100+ MB), the things that actually need streaming —
threw `ConflictError` directing callers to the server-to-server
pathway. That made browser-side `@shade/files` insufficient for
admin-panel-style apps where the client is a browser tab and the
server is a Bun process.
`4.2.0` flips the direction: when the browser supplies
`outboundQueueUrl` + `transferBaseUrl`, server-to-browser chunks +
control envelopes ride a per-peer queue that the browser long-polls,
and browser-to-server chunks POST directly to the server's existing
chunk-receive routes. No WebSockets, no SSE, no inbound listener on
the browser. Long-polling + a request-response inbound queue is
the entire wire surface.
### Added
#### `@shade/transfer`
- `OutboundQueue` — per-peer monotonic event log with long-poll
semantics. `enqueue(peer, event)` appends, `drain(peer, since,
blockMs, signal)` returns events with `id > since` (blocking up
to `blockMs` if none are ready). Idle-eviction GC drops peers
that haven't polled in `idleEvictionMs` (default 10 min). Ring-
buffered to `maxEventsPerPeer` (default 1000) — overflow drops
oldest, receivers pick up the gap via re-resume from `since=0`.
- `QueuedEvent` discriminated union: `{ kind: 'envelope', bytes }`
or `{ kind: 'chunk', bytes, meta: { streamId, laneId, seq } }`.
- `QueueTransferTransport` (implements `ITransferTransport`) —
enqueues outbound chunks instead of POSTing. Returns optimistic
`ChunkAck` because the queue *is* the delivery; chunk-resume picks
up dropped events on receiver-side reconnect.
#### `@shade/sdk`
- `Shade.transferQueueRoute(opts?)` — Hono app with all five routes a
pull-mode receiver needs:
- `POST /queue` — long-poll the per-peer outbound queue.
- `POST /v1/transfer/:streamId/chunk` — receive incoming chunks
(browser → server writes).
- `GET /v1/transfer/:streamId/state` — resume-state lookup.
- `POST /v1/transfer/control` — receive incoming control envelopes
(browser → server stream-init / abort).
- `GET /v1/transfer/health` — peer reachability probe.
Auto-configures `shade.configureTransfers(...)` with the queue
transport + `QueueEnvelopeTransport` if not already configured.
- `Shade.configureTransfers(opts)` extended: `resolveBaseUrl` is now
optional when `transport` and `envelopeTransport` are both supplied
(lets pure-queue servers omit the baseUrl entirely). New
`transport?: ITransferTransport` override slot.
- `QueueEnvelopeTransport``ControlEnvelopeTransport` impl that
enqueues outbound envelopes for browser receivers.
#### `@shade/files`
- `createFilesHttpClient` (and `shade.files.httpClient`) accept new
options:
- `outboundQueueUrl``/queue` endpoint to long-poll.
- `transferBaseUrl` — base URL for outbound chunk POSTs and control
envelope POSTs (browser → server writes).
- `queueBlockMs` — long-poll timeout (default 30 s; server clamps
at `maxBlockMs`).
When set, the client:
1. Configures `shade.configureTransfers({ resolveBaseUrl })` so
outbound chunks POST to `<transferBaseUrl>/v1/transfer/...`.
2. Builds a `ClientStreamsBridge` eagerly so the engine's
incoming-transfer subscription is in place before the drainer
dispatches the first envelope.
3. Starts a long-poll `startQueueDrainer(...)` that pulls queued
events and dispatches them via `shade.acceptTransferEnvelope`.
- Streamed reads (`fs.read` of files > 256 KiB) and streamed writes
(`fs.write` of large inputs) now work end-to-end on the browser
client when the queue options are set.
- `startQueueDrainer(shade, opts)` exported for advanced consumers
that want to drive their own drainer (e.g. service-worker setups
that want a single shared drainer across multiple `httpClient`s).
- `client.close()` now stops the drainer and tears down the streams-
bridge — important on tab unload to free the long-poll socket.
#### `@shade/files` (internal)
- `ClientStreamsBridge` uses a TransformStream with `highWaterMark:
64` instead of the default `0` so the receive-side write loop
doesn't stall on backpressure before the consumer attaches its
reader (default HWM stalled at chunk 4 in pull-mode where the
drainer races the consumer's `getReader()` call).
### Wire contract
```
POST <base>/queue HTTP/1.1
X-Shade-Sender-Address: alice@example.com
{ "since": 42, "blockMs": 30000 }
────
200 OK
{
"events": [
{ "id": 43, "kind": "envelope", "bytesB64": "...", "timestampMs": 1730... },
{ "id": 44, "kind": "chunk", "bytesB64": "...", "meta": { "streamId": "...", "laneId": 0, "seq": 0 } },
...
],
"nextSince": 47
}
```
### Tests
`tests/integration/http-rpc-streams.test.ts` — three integration tests:
- 4 MiB streamed read end-to-end via long-poll queue (verifies bytes
match the source).
- Inline-only client throws clear error on streamed read.
- Long-poll returns empty events on idle timeout (verifies the
`blockMs` pathway).
### Migration
`4.1.0 → 4.2.0` is wire-compatible and source-compatible — the
queue route is purely additive. To enable streamed transfers in a
browser app:
```ts
// Server
const queue = await shade.transferQueueRoute({ blockMs: 30_000 });
await shade.files.serve(handler);
const rpc = shade.files.rpcRoute({ acceptFirstMessage: true });
const app = new Hono();
app.route('/api/v1/shade-files', queue);
app.route('/api/v1/shade-files', rpc);
// Browser
const fs = shade.files.httpClient(serverAddress, {
rpcUrl: 'https://server/api/v1/shade-files/rpc',
outboundQueueUrl: 'https://server/api/v1/shade-files/queue',
transferBaseUrl: 'https://server/api/v1/shade-files',
});
await fs.write('/mods/some-mod.jar', new Uint8Array(/* 50 MB */));
const result = await fs.read('/backups/world.tar.gz'); // streamed
```
`shade.files.serve(handler, { inlineOnly: true })` is still supported
for HTTP-RPC-without-streams deployments — it skips the streams-bridge
setup entirely.
## [4.1.0] — 2026-05-03 — Browser-friendly HTTP RPC for @shade/files
The default `shade.files.client(peer)` requires both peers to be