release(v4.2.0): pull-mode streams for browser @shade/files
4.1.0's HTTP RPC for browsers capped at inline payloads (≤ 256 KiB).
4.2.0 unlocks streams: server queues outbound chunks + control
envelopes per peer, browser long-polls the queue. Browser-to-server
writes ride the existing /v1/transfer/<id>/chunk POST routes
unchanged.
For Dispatch this unlocks mod-jar uploads (50 MB) and world-backup
downloads (100+ MB) — the actual reason browser-side @shade/files
matters.
### New API
@shade/sdk:
- shade.transferQueueRoute(opts?) — Hono app with /queue +
/v1/transfer/* routes. Auto-configures the queue transport.
- shade.configureTransfers extended: transport + envelopeTransport
override slots; resolveBaseUrl optional when both supplied.
@shade/transfer:
- OutboundQueue — per-peer monotonic event log with long-poll
semantics, idle-eviction GC, ring-buffered to maxEventsPerPeer.
- QueueTransferTransport — enqueues instead of POSTing.
@shade/files:
- httpClient({ outboundQueueUrl, transferBaseUrl }) — when set,
starts a long-poll drainer + builds a streams-bridge. fs.read /
fs.write of >256 KiB work end-to-end.
- startQueueDrainer(shade, opts) — exported helper for advanced
consumers driving their own drainer.
### Implementation notes
- ClientStreamsBridge's TransformStream had HWM=0 by default which
stalled the drainer's await chain at chunk 4 (writer.write pended
before the consumer's reader was attached). Bumped to HWM=64 so
the receive loop can buffer ahead of the consumer.
### Tests
3 new integration tests in tests/integration/http-rpc-streams.test.ts:
4 MiB streamed read round-trip, inline-only error path, idle-timeout
long-poll behaviour.
Wire-compatible. Source-compatible. Lockstep bump to 4.2.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
145
CHANGELOG.md
145
CHANGELOG.md
@@ -5,6 +5,151 @@ All notable changes to Shade are documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [4.2.0] — 2026-05-03 — Pull-mode streams for browser @shade/files
|
||||
|
||||
`4.1.0` shipped HTTP RPC for browser clients but capped them at inline
|
||||
payloads (≤ 256 KiB). Larger reads/writes — mod-jars (1–50 MB),
|
||||
world-backups (100+ MB), the things that actually need streaming —
|
||||
threw `ConflictError` directing callers to the server-to-server
|
||||
pathway. That made browser-side `@shade/files` insufficient for
|
||||
admin-panel-style apps where the client is a browser tab and the
|
||||
server is a Bun process.
|
||||
|
||||
`4.2.0` flips the direction: when the browser supplies
|
||||
`outboundQueueUrl` + `transferBaseUrl`, server-to-browser chunks +
|
||||
control envelopes ride a per-peer queue that the browser long-polls,
|
||||
and browser-to-server chunks POST directly to the server's existing
|
||||
chunk-receive routes. No WebSockets, no SSE, no inbound listener on
|
||||
the browser. Long-polling + a request-response inbound queue is
|
||||
the entire wire surface.
|
||||
|
||||
### Added
|
||||
|
||||
#### `@shade/transfer`
|
||||
- `OutboundQueue` — per-peer monotonic event log with long-poll
|
||||
semantics. `enqueue(peer, event)` appends, `drain(peer, since,
|
||||
blockMs, signal)` returns events with `id > since` (blocking up
|
||||
to `blockMs` if none are ready). Idle-eviction GC drops peers
|
||||
that haven't polled in `idleEvictionMs` (default 10 min). Ring-
|
||||
buffered to `maxEventsPerPeer` (default 1000) — overflow drops
|
||||
oldest, receivers pick up the gap via re-resume from `since=0`.
|
||||
- `QueuedEvent` discriminated union: `{ kind: 'envelope', bytes }`
|
||||
or `{ kind: 'chunk', bytes, meta: { streamId, laneId, seq } }`.
|
||||
- `QueueTransferTransport` (implements `ITransferTransport`) —
|
||||
enqueues outbound chunks instead of POSTing. Returns optimistic
|
||||
`ChunkAck` because the queue *is* the delivery; chunk-resume picks
|
||||
up dropped events on receiver-side reconnect.
|
||||
|
||||
#### `@shade/sdk`
|
||||
- `Shade.transferQueueRoute(opts?)` — Hono app with all five routes a
|
||||
pull-mode receiver needs:
|
||||
- `POST /queue` — long-poll the per-peer outbound queue.
|
||||
- `POST /v1/transfer/:streamId/chunk` — receive incoming chunks
|
||||
(browser → server writes).
|
||||
- `GET /v1/transfer/:streamId/state` — resume-state lookup.
|
||||
- `POST /v1/transfer/control` — receive incoming control envelopes
|
||||
(browser → server stream-init / abort).
|
||||
- `GET /v1/transfer/health` — peer reachability probe.
|
||||
Auto-configures `shade.configureTransfers(...)` with the queue
|
||||
transport + `QueueEnvelopeTransport` if not already configured.
|
||||
- `Shade.configureTransfers(opts)` extended: `resolveBaseUrl` is now
|
||||
optional when `transport` and `envelopeTransport` are both supplied
|
||||
(lets pure-queue servers omit the baseUrl entirely). New
|
||||
`transport?: ITransferTransport` override slot.
|
||||
- `QueueEnvelopeTransport` — `ControlEnvelopeTransport` impl that
|
||||
enqueues outbound envelopes for browser receivers.
|
||||
|
||||
#### `@shade/files`
|
||||
- `createFilesHttpClient` (and `shade.files.httpClient`) accept new
|
||||
options:
|
||||
- `outboundQueueUrl` — `/queue` endpoint to long-poll.
|
||||
- `transferBaseUrl` — base URL for outbound chunk POSTs and control
|
||||
envelope POSTs (browser → server writes).
|
||||
- `queueBlockMs` — long-poll timeout (default 30 s; server clamps
|
||||
at `maxBlockMs`).
|
||||
When set, the client:
|
||||
1. Configures `shade.configureTransfers({ resolveBaseUrl })` so
|
||||
outbound chunks POST to `<transferBaseUrl>/v1/transfer/...`.
|
||||
2. Builds a `ClientStreamsBridge` eagerly so the engine's
|
||||
incoming-transfer subscription is in place before the drainer
|
||||
dispatches the first envelope.
|
||||
3. Starts a long-poll `startQueueDrainer(...)` that pulls queued
|
||||
events and dispatches them via `shade.acceptTransferEnvelope`.
|
||||
- Streamed reads (`fs.read` of files > 256 KiB) and streamed writes
|
||||
(`fs.write` of large inputs) now work end-to-end on the browser
|
||||
client when the queue options are set.
|
||||
- `startQueueDrainer(shade, opts)` exported for advanced consumers
|
||||
that want to drive their own drainer (e.g. service-worker setups
|
||||
that want a single shared drainer across multiple `httpClient`s).
|
||||
- `client.close()` now stops the drainer and tears down the streams-
|
||||
bridge — important on tab unload to free the long-poll socket.
|
||||
|
||||
#### `@shade/files` (internal)
|
||||
- `ClientStreamsBridge` uses a TransformStream with `highWaterMark:
|
||||
64` instead of the default `0` so the receive-side write loop
|
||||
doesn't stall on backpressure before the consumer attaches its
|
||||
reader (default HWM stalled at chunk 4 in pull-mode where the
|
||||
drainer races the consumer's `getReader()` call).
|
||||
|
||||
### Wire contract
|
||||
|
||||
```
|
||||
POST <base>/queue HTTP/1.1
|
||||
X-Shade-Sender-Address: alice@example.com
|
||||
{ "since": 42, "blockMs": 30000 }
|
||||
|
||||
────
|
||||
|
||||
200 OK
|
||||
{
|
||||
"events": [
|
||||
{ "id": 43, "kind": "envelope", "bytesB64": "...", "timestampMs": 1730... },
|
||||
{ "id": 44, "kind": "chunk", "bytesB64": "...", "meta": { "streamId": "...", "laneId": 0, "seq": 0 } },
|
||||
...
|
||||
],
|
||||
"nextSince": 47
|
||||
}
|
||||
```
|
||||
|
||||
### Tests
|
||||
|
||||
`tests/integration/http-rpc-streams.test.ts` — three integration tests:
|
||||
- 4 MiB streamed read end-to-end via long-poll queue (verifies bytes
|
||||
match the source).
|
||||
- Inline-only client throws clear error on streamed read.
|
||||
- Long-poll returns empty events on idle timeout (verifies the
|
||||
`blockMs` pathway).
|
||||
|
||||
### Migration
|
||||
|
||||
`4.1.0 → 4.2.0` is wire-compatible and source-compatible — the
|
||||
queue route is purely additive. To enable streamed transfers in a
|
||||
browser app:
|
||||
|
||||
```ts
|
||||
// Server
|
||||
const queue = await shade.transferQueueRoute({ blockMs: 30_000 });
|
||||
await shade.files.serve(handler);
|
||||
const rpc = shade.files.rpcRoute({ acceptFirstMessage: true });
|
||||
|
||||
const app = new Hono();
|
||||
app.route('/api/v1/shade-files', queue);
|
||||
app.route('/api/v1/shade-files', rpc);
|
||||
|
||||
// Browser
|
||||
const fs = shade.files.httpClient(serverAddress, {
|
||||
rpcUrl: 'https://server/api/v1/shade-files/rpc',
|
||||
outboundQueueUrl: 'https://server/api/v1/shade-files/queue',
|
||||
transferBaseUrl: 'https://server/api/v1/shade-files',
|
||||
});
|
||||
await fs.write('/mods/some-mod.jar', new Uint8Array(/* 50 MB */));
|
||||
const result = await fs.read('/backups/world.tar.gz'); // streamed
|
||||
```
|
||||
|
||||
`shade.files.serve(handler, { inlineOnly: true })` is still supported
|
||||
for HTTP-RPC-without-streams deployments — it skips the streams-bridge
|
||||
setup entirely.
|
||||
|
||||
## [4.1.0] — 2026-05-03 — Browser-friendly HTTP RPC for @shade/files
|
||||
|
||||
The default `shade.files.client(peer)` requires both peers to be
|
||||
|
||||
Reference in New Issue
Block a user