feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
Some checks failed
Test / test (push) Has been cancelled

M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).

@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
  Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
  + per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
  (> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
  / shadeFilesReadStreamId correlation. Server-side TransformStream
  bridges accept inbound transfers immediately (engine rejects chunks
  that arrive before accept) and park the readable for the matching
  RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
  uploadDirectory()/downloadDirectory() with bounded concurrency pool
  (default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
  byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
  policy (traversal + percent-decode hardening), fingerprint gate
  (required/optional/reject), pluggable Ed25519 sig verification with
  ±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
  useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
  entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.

Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
  u16 to u32. The previous u16 silently truncated payloads above
  64 KiB — a hard correctness ceiling that blocked inline file ops
  up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
  only. Cross-platform Kotlin port (android/shade-android) updated to
  match; test-vectors/wire-format.json regenerated.

Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
  Concurrent decryptions of the same peer raced ratchet state
  (manifested as sporadic "Failed to decrypt — wrong key or tampered
  data" under load — surfaced once concurrent uploadDirectory pumped
  many writes in flight). Encrypt was already serialized via
  Shade.send's encryptChains; decrypt is now serialized at the
  manager layer too.

@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
  application-level key/value pairs that round-trip verbatim through
  stream-init plaintext. Used by @shade/files for write/read
  correlation; available to any consumer.

@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
  BackgroundTasks.setHook(name, fn) for runtime hook registration.

Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
  shade-sdk streams-bridge + shade-widgets transfer hooks were
  uncommitted prior to this session. Including them keeps the
  workspace consistent at 0.3.0 since @shade/files depends on them.

Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
  3× stable). Coverage spans unit (inline-threshold + concurrency),
  integration (read-write inline + streams up to 1 MiB, walk +
  upload/download directory, custom-op, metrics, SDK namespace
  end-to-end), and security (tampered-envelope sig verification,
  replay window, fingerprint gate, rate-limit + quota).

Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
  topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
  server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-02 14:00:01 +02:00
parent 7e0f7320a9
commit fa770d3063
198 changed files with 20412 additions and 256 deletions

View File

@@ -0,0 +1,483 @@
import type { Shade } from '@shade/sdk';
import {
KIND_CUSTOM_V1,
KIND_DELETE_V1,
KIND_GET_THUMBNAIL_V1,
KIND_LIST_V1,
KIND_MKDIR_V1,
KIND_MOVE_V1,
KIND_READ_V1,
KIND_STAT_V1,
KIND_WRITE_V1,
MUTATION_OPS,
type StandardOp,
} from '../protocol/kinds.js';
import type { CustomOpsMap } from '../server/custom-ops.js';
import { generateIdempotencyKey, generateRequestId } from '../protocol/correlate.js';
import {
base64ToBytes,
bytesToBase64,
canonicalRpcBytes,
hashArgs,
} from '../protocol/canonical.js';
import { ShadeFileRpcChannel } from '../rpc/channel.js';
import { PendingRpcRegistry, type RegisterOptions } from '../rpc/pending.js';
import type { RpcRequest } from '../schemas/envelope.js';
import {
CustomArgsSchema,
CustomResultSchema,
DeleteArgsSchema,
DeleteResultSchema,
GetThumbnailArgsSchema,
GetThumbnailResultSchema,
ListArgsSchema,
ListResultSchema,
MkdirArgsSchema,
MkdirResultSchema,
MoveArgsSchema,
MoveResultSchema,
ReadArgsSchema,
ReadResultSchema,
StatArgsSchema,
StatResultSchema,
WriteArgsSchema,
WriteResultSchema,
type DeleteArgs,
type DeleteResult,
type GetThumbnailArgs,
type ListArgs,
type ListResult,
type MkdirArgs,
type MkdirResult,
type MoveArgs,
type MoveResult,
type ReadArgs,
type StatResult,
type ThumbnailSize,
type WriteResult,
} from '../schemas/ops.js';
import { ConflictError, InternalFileError } from '../schemas/errors.js';
import { decideInline, INLINE_THRESHOLD, type WriteSource } from './inline-threshold.js';
import type { ClientStreamsBridge } from './streams-bridge.js';
export interface BaseOpts {
signal?: AbortSignal;
/** Auto-generated for mutations if not provided. */
idempotencyKey?: string;
/** Per-call timeout. Default 30_000 ms. */
timeoutMs?: number;
}
// ─── read/write public types ─────────────────────────────────
export interface ReadInlineOutput {
kind: 'inline';
bytes: Uint8Array;
size: number;
sha256: string;
contentType?: string;
}
export interface ReadStreamsOutput {
kind: 'streams';
stream: ReadableStream<Uint8Array>;
size: number;
sha256: string;
contentType?: string;
/** Resolves once the entire transfer has been received and verified. */
done(): Promise<void>;
}
export type ReadOutput = ReadInlineOutput | ReadStreamsOutput;
export interface WriteOpts extends BaseOpts {
contentType?: string;
overwrite?: boolean;
/** Force inline even if size > 256 KiB. Throws if input is too big. */
forceInline?: boolean;
}
export interface ReadOpts extends BaseOpts {
range?: { start: number; end: number };
preferInline?: boolean;
}
export interface ThumbnailResult {
bytes: Uint8Array;
format: 'png' | 'webp' | 'jpeg';
width: number;
height: number;
sha256: string;
}
// ─── FileClient interface ────────────────────────────────────
/**
* Untyped fallback for `FileClient.custom()` — used when the consumer
* hasn't extended `CustomOpsMap` for a given op name.
*/
type CustomOpArgs<K> = K extends keyof CustomOpsMap
? CustomOpsMap[K] extends { args: infer A }
? A
: unknown
: unknown;
type CustomOpResponse<K> = K extends keyof CustomOpsMap
? CustomOpsMap[K] extends { response: infer R }
? R
: unknown
: unknown;
export interface FileClient {
list(path: string, opts?: BaseOpts & Partial<Omit<ListArgs, 'path'>>): Promise<ListResult>;
stat(path: string, opts?: BaseOpts): Promise<StatResult>;
mkdir(path: string, opts?: BaseOpts & Partial<Omit<MkdirArgs, 'path'>>): Promise<MkdirResult>;
delete(path: string, opts?: BaseOpts & Partial<Omit<DeleteArgs, 'path'>>): Promise<DeleteResult>;
move(src: string, dst: string, opts?: BaseOpts & Partial<Omit<MoveArgs, 'src' | 'dst'>>): Promise<MoveResult>;
read(path: string, opts?: ReadOpts): Promise<ReadOutput>;
write(path: string, input: WriteSource, opts?: WriteOpts): Promise<WriteResult>;
getThumbnail(
path: string,
size: ThumbnailSize,
opts?: BaseOpts & { format?: 'png' | 'webp' | 'jpeg' },
): Promise<ThumbnailResult>;
/**
* Invoke a custom op registered on the server. Args/response types are
* pulled from `CustomOpsMap` via TypeScript declaration merging — see
* `server/custom-ops.ts` for the registration pattern.
*/
custom<K extends keyof CustomOpsMap | string>(
name: K & string,
args: CustomOpArgs<K>,
opts?: BaseOpts,
): Promise<CustomOpResponse<K>>;
close(): void;
}
export interface CreateFileClientOptions {
/** Default per-call timeout. Default 30_000. */
defaultTimeoutMs?: number;
/** Hard deadline for incoming-read awaits. Default 60_000. */
ioTimeoutMs?: number;
/**
* Required for read/write `streams` ops. Coordinates inbound/outbound
* `@shade/transfer` transfers via `userMetadata.shadeFiles*Id` keys.
*/
streamsBridge?: ClientStreamsBridge;
/**
* Optional: sign the canonical bytes of every outgoing RPC envelope.
* Pluggable so apps can plug their own signing-key store (e.g.,
* Ed25519-as-a-service, browser SubtleCrypto). When omitted, ships
* `'unsigned'` — the server's `verifySender` should also be unset, or
* be configured to accept the placeholder.
*/
signRequest?: (canonicalBytes: Uint8Array) => Promise<string> | string;
}
/**
* Client-side proxy for `@shade/files` ops. Each method ships an
* `RpcRequest` over `Shade.send`/`Shade.receive` and awaits the matching
* response (or error/timeout) from `PendingRpcRegistry`.
*
* Mutations auto-generate an idempotency key per logical call (not per
* attempt) so transparent retries under the SDK don't produce duplicates.
*
* Read/write content I/O over 256 KiB requires a `streamsBridge` to be
* passed via options — it coordinates the inbound/outbound `@shade/transfer`
* transfers that carry the actual bytes.
*/
export function createFileClient(
shade: Shade,
channel: ShadeFileRpcChannel,
pending: PendingRpcRegistry,
peerAddress: string,
options: CreateFileClientOptions = {},
): FileClient {
const defaultTimeout = options.defaultTimeoutMs ?? 30_000;
const ioTimeoutMs = options.ioTimeoutMs ?? 60_000;
const streamsBridge = options.streamsBridge;
const signRequest = options.signRequest;
const senderAddress = shade.myAddress;
async function request<TResult>(
kind: string,
op: StandardOp | 'custom',
args: unknown,
opts: BaseOpts | undefined,
): Promise<TResult> {
const requestId = generateRequestId();
const isMutation = MUTATION_OPS.has(op);
const idempotencyKey =
opts?.idempotencyKey ?? (isMutation ? generateIdempotencyKey() : undefined);
const signedAt = Date.now();
let sig = 'unsigned';
if (signRequest !== undefined) {
// Server reconstructs canonical bytes using `address = from`, which
// is OUR own address as authenticated by the ratchet. So we sign
// over the same identifier here.
const canonical = canonicalRpcBytes({
address: senderAddress,
signedAt,
kind,
id: requestId,
argsHash: hashArgs(args),
});
sig = await signRequest(canonical);
}
const env: RpcRequest = {
kind,
id: requestId,
args,
...(idempotencyKey !== undefined ? { idempotencyKey } : {}),
sig,
signedAt,
};
const registerOpts: RegisterOptions = {
timeoutMs: opts?.timeoutMs ?? defaultTimeout,
onCancel: (reason) => {
// Fire-and-forget cancel envelope so server can release resources.
void channel
.send(peerAddress, {
kind: 'shade.fs.cancel/v1',
id: requestId,
reason,
})
.catch(() => {
/* swallow — cancellation is best-effort */
});
},
};
if (opts?.signal !== undefined) registerOpts.signal = opts.signal;
const pendingPromise = pending.register<TResult>(requestId, registerOpts);
try {
await channel.send(peerAddress, env);
} catch (err) {
// If the send itself fails, the pending entry will never resolve;
// reject it directly.
pending.rejectAll(err);
throw err;
}
return pendingPromise;
}
return {
async list(path, opts) {
const args: ListArgs = ListArgsSchema.parse({
path,
...(opts?.cursor !== undefined ? { cursor: opts.cursor } : {}),
...(opts?.pageSize !== undefined ? { pageSize: opts.pageSize } : {}),
...(opts?.filter !== undefined ? { filter: opts.filter } : {}),
});
const raw = await request<unknown>(KIND_LIST_V1, 'list', args, opts);
return ListResultSchema.parse(raw);
},
async stat(path, opts) {
const args = StatArgsSchema.parse({ path });
const raw = await request<unknown>(KIND_STAT_V1, 'stat', args, opts);
return StatResultSchema.parse(raw);
},
async mkdir(path, opts) {
const args = MkdirArgsSchema.parse({
path,
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
});
const raw = await request<unknown>(KIND_MKDIR_V1, 'mkdir', args, opts);
return MkdirResultSchema.parse(raw);
},
async delete(path, opts) {
const args = DeleteArgsSchema.parse({
path,
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
});
const raw = await request<unknown>(KIND_DELETE_V1, 'delete', args, opts);
return DeleteResultSchema.parse(raw);
},
async move(src, dst, opts) {
const args = MoveArgsSchema.parse({
src,
dst,
...(opts?.overwrite !== undefined ? { overwrite: opts.overwrite } : {}),
});
const raw = await request<unknown>(KIND_MOVE_V1, 'move', args, opts);
return MoveResultSchema.parse(raw);
},
async read(path, opts) {
const args: ReadArgs = ReadArgsSchema.parse({
path,
...(opts?.range !== undefined ? { range: opts.range } : {}),
...(opts?.preferInline !== undefined ? { preferInline: opts.preferInline } : {}),
});
const raw = await request<unknown>(KIND_READ_V1, 'read', args, opts);
const wire = ReadResultSchema.parse(raw);
if (wire.kind === 'inline') {
const bytes = base64ToBytes(wire.bytesB64);
const out: ReadInlineOutput = {
kind: 'inline',
bytes,
size: wire.size,
sha256: wire.sha256,
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
};
return out;
}
// streams — wait for the matching incoming transfer via the bridge.
if (streamsBridge === undefined) {
throw new InternalFileError(
'streams-bridge not configured: cannot consume streamed read',
);
}
const bridgeSignal = opts?.signal ?? new AbortController().signal;
const parked = await streamsBridge.awaitRead(wire.streamId, {
expectedFrom: peerAddress,
signal: bridgeSignal,
timeoutMs: ioTimeoutMs,
});
const out: ReadStreamsOutput = {
kind: 'streams',
stream: parked.readable,
size: wire.size,
sha256: wire.sha256,
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
done: async () => {
await parked.done;
},
};
return out;
},
async write(path, input, opts) {
const decision = await decideInline(input);
const overwrite = opts?.overwrite ?? false;
const contentType = opts?.contentType ?? decision.contentType;
if (decision.kind === 'inline' || opts?.forceInline === true) {
// Inline path — base64 in the RPC envelope.
const bytes =
decision.kind === 'inline'
? decision.bytes
: await drainToUint8Array(decision.stream, decision.size ?? Number.POSITIVE_INFINITY);
if (bytes.byteLength > INLINE_THRESHOLD && opts?.forceInline !== true) {
throw new ConflictError(
`inline write exceeds ${INLINE_THRESHOLD}-byte threshold (got ${bytes.byteLength}); pass forceInline=true to override`,
);
}
const args = WriteArgsSchema.parse({
kind: 'inline',
path,
bytesB64: bytesToBase64(bytes),
...(contentType !== undefined ? { contentType } : {}),
overwrite,
});
const raw = await request<unknown>(KIND_WRITE_V1, 'write', args, opts);
return WriteResultSchema.parse(raw);
}
// Streams path — kick the upload, then ship the RPC.
if (streamsBridge === undefined) {
throw new InternalFileError(
'streams-bridge not configured: cannot ship streamed write',
);
}
const size = decision.size;
if (size === undefined) {
throw new ConflictError(
'streams write requires a known plaintext size; pass `{ stream, size }` instead of a bare ReadableStream',
);
}
const { writeId, handle } = await streamsBridge.initiateWrite({
peer: peerAddress,
stream: decision.stream,
size,
...(contentType !== undefined ? { contentType } : {}),
name: path,
...(opts?.signal !== undefined ? { signal: opts.signal } : {}),
});
const args = WriteArgsSchema.parse({
kind: 'streams',
path,
size,
...(contentType !== undefined ? { contentType } : {}),
overwrite,
writeId,
});
try {
const [raw] = await Promise.all([
request<unknown>(KIND_WRITE_V1, 'write', args, opts),
handle.done(),
]);
return WriteResultSchema.parse(raw);
} catch (err) {
// Best-effort cancel of the transfer on RPC failure.
await handle.abort('rpc-failed').catch(() => undefined);
throw err;
}
},
async getThumbnail(path, size, opts) {
const args: GetThumbnailArgs = GetThumbnailArgsSchema.parse({
path,
size,
...(opts?.format !== undefined ? { format: opts.format } : {}),
});
const raw = await request<unknown>(KIND_GET_THUMBNAIL_V1, 'getThumbnail', args, opts);
const wire = GetThumbnailResultSchema.parse(raw);
return {
bytes: base64ToBytes(wire.bytesB64),
format: wire.format,
width: wire.width,
height: wire.height,
sha256: wire.sha256,
};
},
async custom(name, args, opts) {
const wireArgs = CustomArgsSchema.parse({ name, payload: args });
// `custom` is a mutation in the rate-limit sense; auto-key for retries.
const raw = await request<unknown>(KIND_CUSTOM_V1, 'custom' as StandardOp, wireArgs, opts);
const wire = CustomResultSchema.parse(raw);
// The result schema is `{ result: unknown }` — the inner `result` is
// already validated against the consumer's response schema on the
// server side, so we trust it here.
return wire.result as never;
},
close() {
pending.rejectAll(new Error('FileClient closed'));
},
};
}
/** Drain a stream into a single buffer; used for the inline-write fallback. */
async function drainToUint8Array(
stream: ReadableStream<Uint8Array>,
cap: number,
): Promise<Uint8Array> {
const reader = stream.getReader();
const chunks: Uint8Array[] = [];
let total = 0;
try {
while (true) {
const { value, done } = await reader.read();
if (done) break;
if (value === undefined) continue;
chunks.push(value);
total += value.byteLength;
if (total > cap) {
throw new Error(`stream produced more than declared size cap (${cap})`);
}
}
} finally {
reader.releaseLock();
}
const out = new Uint8Array(total);
let offset = 0;
for (const c of chunks) {
out.set(c, offset);
offset += c.byteLength;
}
return out;
}

View File

@@ -0,0 +1,91 @@
/**
* Bounded-concurrency async map.
*
* Pulls items from an `AsyncIterable` source (lazily — never materializing
* the whole sequence) and runs `fn` on at most `concurrency` of them at
* once. Failures bubble unless `continueOnError` is set, in which case they
* are reported via `onError` and the pool keeps draining.
*/
import { CancelledError } from '../schemas/errors.js';
export interface ConcurrentMapOptions<T> {
concurrency: number;
signal?: AbortSignal;
continueOnError?: boolean;
onError?: (item: T, err: unknown) => void;
}
export async function runWithConcurrency<T>(
source: AsyncIterable<T>,
fn: (item: T) => Promise<void>,
opts: ConcurrentMapOptions<T>,
): Promise<void> {
if (opts.concurrency < 1) throw new Error('concurrency must be ≥ 1');
const iter = source[Symbol.asyncIterator]();
const inFlight = new Set<Promise<void>>();
let firstError: unknown = null;
let aborted = false;
function checkAbort(): void {
if (opts.signal?.aborted) {
aborted = true;
const reason = opts.signal.reason;
throw reason instanceof Error ? reason : new CancelledError(String(reason ?? 'aborted'));
}
}
async function pumpOne(): Promise<boolean> {
checkAbort();
const next = await iter.next();
if (next.done === true) return false;
const item = next.value;
const task = (async () => {
try {
await fn(item);
} catch (err) {
if (opts.continueOnError === true) {
opts.onError?.(item, err);
} else if (firstError === null) {
firstError = err;
}
}
})().finally(() => {
inFlight.delete(task);
});
inFlight.add(task);
return true;
}
try {
// Prime the pool
while (inFlight.size < opts.concurrency) {
if (firstError !== null || aborted) break;
const more = await pumpOne();
if (!more) break;
}
// Maintain saturation
while (inFlight.size > 0) {
if (firstError !== null && opts.continueOnError !== true) break;
await Promise.race(inFlight);
while (
inFlight.size < opts.concurrency &&
firstError === null &&
!aborted
) {
const more = await pumpOne();
if (!more) break;
}
}
// Drain whatever survived
if (inFlight.size > 0) await Promise.allSettled(inFlight);
} finally {
if (typeof iter.return === 'function') {
await iter.return().catch(() => undefined);
}
}
if (firstError !== null && opts.continueOnError !== true) {
throw firstError;
}
}

View File

@@ -0,0 +1,97 @@
/**
* Public types for directory operations (`walk`, `uploadDirectory`,
* `downloadDirectory`).
*
* `DirectoryHandleLike` and `FileHandleLike` are structurally compatible
* with the browser File System Access API
* (`FileSystemDirectoryHandle` / `FileSystemFileHandle`) so that browser
* consumers can pass them directly. For Bun/Node, use the adapter from
* `node-directory-handle.ts`.
*/
export interface FileHandleLike {
readonly kind: 'file';
readonly name: string;
/** Read-side accessor — used by `uploadDirectory`. */
getFile(): Promise<FileLike>;
/** Write-side accessor — used by `downloadDirectory`. */
createWritable(): Promise<WritableStreamLike>;
}
export interface FileLike {
readonly name: string;
readonly size: number;
readonly type: string;
stream(): ReadableStream<Uint8Array>;
arrayBuffer(): Promise<ArrayBuffer>;
}
export interface WritableStreamLike {
write(chunk: Uint8Array): Promise<void>;
close(): Promise<void>;
abort(reason?: unknown): Promise<void>;
}
export interface DirectoryHandleLike {
readonly kind: 'directory';
readonly name: string;
/** Yield child entries (file or directory). Used by `uploadDirectory`. */
entries(): AsyncIterable<[string, DirectoryHandleLike | FileHandleLike]>;
/** Used by `downloadDirectory` to create remote → local mapping. */
getDirectoryHandle(name: string, opts?: { create?: boolean }): Promise<DirectoryHandleLike>;
getFileHandle(name: string, opts?: { create?: boolean }): Promise<FileHandleLike>;
}
// ─── Bulk transfer types ─────────────────────────────────────
export type BulkTransferEvent =
| { type: 'plan'; totalFiles: number; totalBytes: number | undefined }
| { type: 'file-start'; path: string; size: number }
| { type: 'file-progress'; path: string; bytesDone: number; bytesTotal: number }
| { type: 'file-done'; path: string; bytesDone: number }
| { type: 'file-error'; path: string; error: unknown }
| {
type: 'progress';
filesDone: number;
filesTotal: number;
bytesDone: number;
bytesTotal: number | undefined;
}
| { type: 'complete'; filesDone: number; bytesDone: number; durationMs: number }
| { type: 'abort'; reason: string };
export interface BulkTransferResult {
filesDone: number;
bytesDone: number;
durationMs: number;
}
export interface BulkTransferHandle {
/** Async-iterable event stream — plan, per-file events, aggregate progress. */
readonly events: AsyncIterable<BulkTransferEvent>;
abort(reason?: string): Promise<void>;
done(): Promise<BulkTransferResult>;
}
export interface BulkOpts {
/**
* Max files in flight at once. Default 4. Capped at 16 to bound memory
* (each in-flight file may hold a buffered chunk; with 1 MiB chunks +
* 16 lanes that's still bounded).
*/
concurrency?: number;
/**
* Continue past per-file failures. Default false (fail-fast).
* When true, errors are emitted via `file-error` events and the whole
* transfer still resolves (with `done()` returning the count of
* successful files).
*/
continueOnError?: boolean;
/** Cancellation. */
signal?: AbortSignal;
}
/** Hard cap to bound memory regardless of caller-supplied concurrency. */
export const MAX_BULK_CONCURRENCY = 16;
/** Default concurrency. */
export const DEFAULT_BULK_CONCURRENCY = 4;

View File

@@ -0,0 +1,316 @@
/**
* Download an entire remote directory tree to a local `DirectoryHandleLike`.
*
* Mirror image of `uploadDirectory`: walks the remote tree via the shared
* `walk()` helper, creates local directories on the fly, and downloads each
* file with `client.read` (which routes inline or streams based on the
* server's response). Bounded concurrency keeps RPC inflight count low.
*/
import { walk, type WalkItem } from './walk.js';
import { runWithConcurrency } from './concurrency.js';
import type { FileClient } from './client.js';
import {
DEFAULT_BULK_CONCURRENCY,
MAX_BULK_CONCURRENCY,
type BulkOpts,
type BulkTransferEvent,
type BulkTransferHandle,
type BulkTransferResult,
type DirectoryHandleLike,
type FileHandleLike,
} from './directory-types.js';
import { CancelledError } from '../schemas/errors.js';
export interface DownloadDirectoryOptions extends BulkOpts {
/** Page size hint forwarded to the underlying `walk`. Default 200. */
pageSize?: number;
/** Skip files already present locally. Default false (overwrite). */
skipExisting?: boolean;
}
export function downloadDirectory(
client: Pick<FileClient, 'read' | 'list'>,
remoteRoot: string,
local: DirectoryHandleLike,
opts: DownloadDirectoryOptions = {},
): BulkTransferHandle {
const concurrency = Math.max(
1,
Math.min(MAX_BULK_CONCURRENCY, opts.concurrency ?? DEFAULT_BULK_CONCURRENCY),
);
const continueOnError = opts.continueOnError ?? false;
const externalSignal = opts.signal;
const pageSize = opts.pageSize ?? 200;
const internalAbort = new AbortController();
const combinedSignal = mergeSignals(externalSignal, internalAbort.signal);
const events: BulkTransferEvent[] = [];
const eventResolvers: ((v: IteratorResult<BulkTransferEvent>) => void)[] = [];
let eventsClosed = false;
function emit(event: BulkTransferEvent): void {
if (eventsClosed) return;
if (eventResolvers.length > 0) {
eventResolvers.shift()!({ value: event, done: false });
} else {
events.push(event);
}
}
function closeEvents(): void {
if (eventsClosed) return;
eventsClosed = true;
while (eventResolvers.length > 0) {
eventResolvers.shift()!({ value: undefined as never, done: true });
}
}
let resolveDone!: (r: BulkTransferResult) => void;
let rejectDone!: (err: unknown) => void;
const donePromise = new Promise<BulkTransferResult>((resolve, reject) => {
resolveDone = resolve;
rejectDone = reject;
});
const startedAt = Date.now();
let filesDone = 0;
let bytesDone = 0;
void (async () => {
try {
// 1. Plan: walk the remote tree into [dirs, files]
const dirItems: WalkItem[] = [];
const fileItems: WalkItem[] = [];
try {
for await (const item of walk(client, remoteRoot, {
pageSize,
...(combinedSignal !== undefined ? { signal: combinedSignal } : {}),
})) {
if (item.entry.kind === 'dir') dirItems.push(item);
else fileItems.push(item);
}
} catch (err) {
emit({ type: 'abort', reason: errMsg(err) });
closeEvents();
rejectDone(err);
return;
}
const totalFiles = fileItems.length;
const bytesTotal = fileItems.reduce((acc, f) => acc + f.entry.size, 0);
emit({ type: 'plan', totalFiles, totalBytes: bytesTotal });
if (combinedSignal.aborted) {
emit({ type: 'abort', reason: errMsg(combinedSignal.reason) });
closeEvents();
rejectDone(combinedSignal.reason ?? new CancelledError());
return;
}
// 2. Pre-create local directories sequentially.
for (const d of dirItems) {
if (combinedSignal.aborted) break;
try {
await ensureLocalDir(local, d.relativePath);
} catch (err) {
if (!continueOnError) {
emit({ type: 'abort', reason: errMsg(err) });
closeEvents();
rejectDone(err);
return;
}
}
}
// 3. Download files with bounded concurrency.
try {
await runWithConcurrency(
asyncIterableOf(fileItems),
async (item) => {
if (combinedSignal.aborted) {
throw new CancelledError('download aborted');
}
emit({ type: 'file-start', path: item.relativePath, size: item.entry.size });
try {
const fileHandle = await ensureLocalFile(local, item.relativePath);
if (opts.skipExisting === true) {
// Skip if file already exists with non-zero size
try {
const existing = await fileHandle.getFile();
if (existing.size === item.entry.size) {
filesDone++;
bytesDone += existing.size;
emit({ type: 'file-done', path: item.relativePath, bytesDone: existing.size });
emit({
type: 'progress',
filesDone,
filesTotal: totalFiles,
bytesDone,
bytesTotal,
});
return;
}
} catch {
/* not present yet — fall through to download */
}
}
const writable = await fileHandle.createWritable();
try {
const result = await client.read(item.absolutePath, {
signal: combinedSignal,
});
if (result.kind === 'inline') {
if (result.bytes.byteLength > 0) await writable.write(result.bytes);
await writable.close();
filesDone++;
bytesDone += result.bytes.byteLength;
emit({ type: 'file-done', path: item.relativePath, bytesDone: result.bytes.byteLength });
} else {
await pipeReadableToWritable(result.stream, writable);
await writable.close();
await result.done();
filesDone++;
bytesDone += result.size;
emit({ type: 'file-done', path: item.relativePath, bytesDone: result.size });
}
emit({
type: 'progress',
filesDone,
filesTotal: totalFiles,
bytesDone,
bytesTotal,
});
} catch (err) {
await writable.abort(err).catch(() => undefined);
throw err;
}
} catch (err) {
emit({ type: 'file-error', path: item.relativePath, error: err });
throw err;
}
},
{
concurrency,
signal: combinedSignal,
continueOnError,
onError: () => {
/* already emitted as 'file-error' */
},
},
);
} catch (err) {
emit({ type: 'abort', reason: errMsg(err) });
closeEvents();
rejectDone(err);
return;
}
const durationMs = Date.now() - startedAt;
emit({ type: 'complete', filesDone, bytesDone, durationMs });
closeEvents();
resolveDone({ filesDone, bytesDone, durationMs });
} catch (err) {
closeEvents();
rejectDone(err);
}
})();
donePromise.catch(() => {
/* deliberate */
});
return {
events: {
[Symbol.asyncIterator]() {
return {
next(): Promise<IteratorResult<BulkTransferEvent>> {
if (events.length > 0) {
return Promise.resolve({ value: events.shift()!, done: false });
}
if (eventsClosed) {
return Promise.resolve({ value: undefined as never, done: true });
}
return new Promise((resolve) => eventResolvers.push(resolve));
},
return(): Promise<IteratorResult<BulkTransferEvent>> {
return Promise.resolve({ value: undefined as never, done: true });
},
};
},
},
async abort(reason) {
internalAbort.abort(new CancelledError(reason ?? 'manual abort'));
},
done: () => donePromise,
};
}
// ─── Helpers ─────────────────────────────────────────────────
async function ensureLocalDir(
root: DirectoryHandleLike,
relativePath: string,
): Promise<DirectoryHandleLike> {
const segments = relativePath.split('/').filter((s) => s !== '');
let current = root;
for (const seg of segments) {
current = await current.getDirectoryHandle(seg, { create: true });
}
return current;
}
async function ensureLocalFile(
root: DirectoryHandleLike,
relativePath: string,
): Promise<FileHandleLike> {
const segments = relativePath.split('/').filter((s) => s !== '');
if (segments.length === 0) throw new Error('empty file path');
let current = root;
for (let i = 0; i < segments.length - 1; i++) {
current = await current.getDirectoryHandle(segments[i]!, { create: true });
}
return await current.getFileHandle(segments[segments.length - 1]!, { create: true });
}
async function pipeReadableToWritable(
readable: ReadableStream<Uint8Array>,
writable: { write(chunk: Uint8Array): Promise<void> },
): Promise<void> {
const reader = readable.getReader();
try {
while (true) {
const { value, done } = await reader.read();
if (done) break;
if (value !== undefined && value.byteLength > 0) {
await writable.write(value);
}
}
} finally {
reader.releaseLock();
}
}
async function* asyncIterableOf<T>(arr: T[]): AsyncIterable<T> {
for (const item of arr) yield item;
}
function errMsg(err: unknown): string {
if (err instanceof Error) return err.message;
return String(err ?? 'unknown error');
}
function mergeSignals(
external: AbortSignal | undefined,
internal: AbortSignal,
): AbortSignal {
if (external === undefined) return internal;
if (external.aborted) return external;
const controller = new AbortController();
const onAbort = (sig: AbortSignal): void => {
controller.abort(sig.reason);
};
external.addEventListener('abort', () => onAbort(external), { once: true });
internal.addEventListener('abort', () => onAbort(internal), { once: true });
return controller.signal;
}

View File

@@ -0,0 +1,218 @@
/**
* Decide whether a write should travel inline (base64 inside the RPC
* envelope) or via a dedicated `@shade/transfer` stream.
*
* Threshold: 256 KiB plaintext. Anything strictly above goes through the
* stream path; anything ≤ goes inline.
*
* For known-size inputs (`Uint8Array`, `Blob`, `File`) the decision is
* cheap: compare `byteLength`/`size` against the threshold.
*
* For `ReadableStream` we cannot know the size up front. We pull chunks
* into a temporary buffer until we either:
* - hit EOF before the threshold → inline (we have all the bytes)
* - cross the threshold → streams (the buffered prefix + the rest of the
* stream are returned via a fresh `ReadableStream` so the caller can
* feed it to `shade.upload`).
*/
/** Plaintext size at which inline transitions to streams. */
export const INLINE_THRESHOLD = 256 * 1024;
export type InlineDecision =
| {
kind: 'inline';
bytes: Uint8Array;
contentType?: string;
}
| {
kind: 'streams';
stream: ReadableStream<Uint8Array>;
/** Plaintext size when known (Blob/File). undefined for raw streams. */
size?: number;
contentType?: string;
};
/**
* Public input shape for `FileClient.write`. Runtime discriminator helper.
*/
export type WriteSource =
| Uint8Array
| Blob
| File
| ReadableStream<Uint8Array>
| { stream: ReadableStream<Uint8Array>; size: number; contentType?: string };
export async function decideInline(input: WriteSource): Promise<InlineDecision> {
// 1. Uint8Array — direct size check
if (input instanceof Uint8Array) {
if (input.byteLength <= INLINE_THRESHOLD) {
return { kind: 'inline', bytes: input };
}
return {
kind: 'streams',
stream: uint8ArrayToStream(input),
size: input.byteLength,
};
}
// 2. Blob / File — known size (Blob.type is a string; File extends Blob)
if (typeof Blob !== 'undefined' && input instanceof Blob) {
const blob: Blob = input;
const contentType = blob.type === '' ? undefined : blob.type;
if (blob.size <= INLINE_THRESHOLD) {
const bytes = new Uint8Array(await blob.arrayBuffer());
return contentType !== undefined
? { kind: 'inline', bytes, contentType }
: { kind: 'inline', bytes };
}
return contentType !== undefined
? { kind: 'streams', stream: blob.stream(), size: blob.size, contentType }
: { kind: 'streams', stream: blob.stream(), size: blob.size };
}
// 3. Pre-wrapped { stream, size } — trust caller's declared size
if (
typeof input === 'object' &&
input !== null &&
'stream' in input &&
'size' in input &&
(input as { stream: unknown }).stream instanceof ReadableStream
) {
const wrapped = input as { stream: ReadableStream<Uint8Array>; size: number; contentType?: string };
if (wrapped.size <= INLINE_THRESHOLD) {
const bytes = await drainStream(wrapped.stream, wrapped.size);
const contentType = wrapped.contentType;
return contentType !== undefined
? { kind: 'inline', bytes, contentType }
: { kind: 'inline', bytes };
}
const contentType = wrapped.contentType;
return contentType !== undefined
? { kind: 'streams', stream: wrapped.stream, size: wrapped.size, contentType }
: { kind: 'streams', stream: wrapped.stream, size: wrapped.size };
}
// 4. Bare ReadableStream — peek until threshold or EOF
if (input instanceof ReadableStream) {
return await peekStream(input);
}
throw new TypeError(
`decideInline: unsupported input type ${Object.prototype.toString.call(input)}`,
);
}
/**
* Drain a stream into a Uint8Array, with a soft cap at `expected` + slack.
* Used for the inline path when the caller declared a size.
*/
async function drainStream(
stream: ReadableStream<Uint8Array>,
expected: number,
): Promise<Uint8Array> {
const reader = stream.getReader();
const chunks: Uint8Array[] = [];
let total = 0;
try {
while (true) {
const { value, done } = await reader.read();
if (done) break;
if (value === undefined) continue;
chunks.push(value);
total += value.byteLength;
if (total > expected + INLINE_THRESHOLD) {
throw new Error(
`decideInline: stream produced more bytes (${total}) than declared size (${expected})`,
);
}
}
} finally {
reader.releaseLock();
}
return concat(chunks, total);
}
/**
* Peek a `ReadableStream` of unknown length: buffer up to `INLINE_THRESHOLD + 1`
* bytes. If EOF first, return inline. Otherwise reconstruct a stream that
* yields the buffered prefix followed by the remainder.
*/
async function peekStream(stream: ReadableStream<Uint8Array>): Promise<InlineDecision> {
const reader = stream.getReader();
const buffered: Uint8Array[] = [];
let total = 0;
try {
while (total <= INLINE_THRESHOLD) {
const { value, done } = await reader.read();
if (done) {
reader.releaseLock();
return { kind: 'inline', bytes: concat(buffered, total) };
}
if (value === undefined) continue;
buffered.push(value);
total += value.byteLength;
}
// We have at least INLINE_THRESHOLD + 1 bytes buffered. Promote to streams.
const reconstructed = reconstructStream(buffered, reader);
return { kind: 'streams', stream: reconstructed };
} catch (err) {
reader.releaseLock();
throw err;
}
}
interface MinimalReader {
read(): Promise<{ value: Uint8Array | undefined; done: boolean }>;
cancel(reason?: unknown): Promise<void>;
releaseLock(): void;
}
function reconstructStream(
prefix: Uint8Array[],
reader: MinimalReader,
): ReadableStream<Uint8Array> {
let prefixIdx = 0;
return new ReadableStream<Uint8Array>({
async pull(controller) {
if (prefixIdx < prefix.length) {
controller.enqueue(prefix[prefixIdx]!);
prefixIdx++;
return;
}
const { value, done } = await reader.read();
if (done) {
controller.close();
reader.releaseLock();
return;
}
if (value !== undefined) controller.enqueue(value);
},
async cancel(reason) {
try {
await reader.cancel(reason);
} finally {
reader.releaseLock();
}
},
});
}
function uint8ArrayToStream(bytes: Uint8Array): ReadableStream<Uint8Array> {
return new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(bytes);
controller.close();
},
});
}
function concat(chunks: Uint8Array[], total: number): Uint8Array {
const out = new Uint8Array(total);
let offset = 0;
for (const c of chunks) {
out.set(c, offset);
offset += c.byteLength;
}
return out;
}

View File

@@ -0,0 +1,145 @@
/**
* In-memory `DirectoryHandleLike` implementation.
*
* Useful for tests and Node/Bun environments without the browser File
* System Access API. The shape is intentionally compatible with browser
* `FileSystemDirectoryHandle` so consumers can swap implementations.
*/
import type {
DirectoryHandleLike,
FileHandleLike,
FileLike,
WritableStreamLike,
} from './directory-types.js';
interface MemoryNode {
bytes: Uint8Array;
type: string;
}
class MemoryFileLike implements FileLike {
constructor(
public readonly name: string,
private readonly node: MemoryNode,
) {}
get size(): number { return this.node.bytes.byteLength; }
get type(): string { return this.node.type; }
stream(): ReadableStream<Uint8Array> {
const bytes = this.node.bytes;
return new ReadableStream<Uint8Array>({
start(controller) {
if (bytes.byteLength > 0) controller.enqueue(bytes);
controller.close();
},
});
}
async arrayBuffer(): Promise<ArrayBuffer> {
const out = new ArrayBuffer(this.node.bytes.byteLength);
new Uint8Array(out).set(this.node.bytes);
return out;
}
}
class MemoryWritable implements WritableStreamLike {
private chunks: Uint8Array[] = [];
constructor(private readonly file: MemoryFile) {}
async write(chunk: Uint8Array): Promise<void> {
this.chunks.push(chunk);
}
async close(): Promise<void> {
let total = 0;
for (const c of this.chunks) total += c.byteLength;
const out = new Uint8Array(total);
let offset = 0;
for (const c of this.chunks) {
out.set(c, offset);
offset += c.byteLength;
}
this.file.commit(out);
}
async abort(): Promise<void> {
this.chunks = [];
}
}
class MemoryFile implements FileHandleLike {
readonly kind: 'file' = 'file';
constructor(
public readonly name: string,
public readonly node: MemoryNode,
) {}
async getFile(): Promise<FileLike> {
return new MemoryFileLike(this.name, this.node);
}
async createWritable(): Promise<WritableStreamLike> {
return new MemoryWritable(this);
}
/** Internal — used by `MemoryWritable.close()`. */
commit(bytes: Uint8Array): void {
this.node.bytes = bytes;
}
}
class MemoryDirectory implements DirectoryHandleLike {
readonly kind: 'directory' = 'directory';
private readonly children = new Map<string, MemoryDirectory | MemoryFile>();
constructor(public readonly name: string) {}
async *entries(): AsyncIterable<[string, DirectoryHandleLike | FileHandleLike]> {
for (const [name, child] of this.children) {
yield [name, child];
}
}
async getDirectoryHandle(name: string, opts?: { create?: boolean }): Promise<DirectoryHandleLike> {
const existing = this.children.get(name);
if (existing !== undefined) {
if (existing.kind !== 'directory') {
throw new Error(`'${name}' exists but is a file`);
}
return existing;
}
if (opts?.create !== true) throw new Error(`directory not found: ${name}`);
const dir = new MemoryDirectory(name);
this.children.set(name, dir);
return dir;
}
async getFileHandle(name: string, opts?: { create?: boolean }): Promise<FileHandleLike> {
const existing = this.children.get(name);
if (existing !== undefined) {
if (existing.kind !== 'file') {
throw new Error(`'${name}' exists but is a directory`);
}
return existing;
}
if (opts?.create !== true) throw new Error(`file not found: ${name}`);
const node: MemoryNode = { bytes: new Uint8Array(0), type: '' };
const file = new MemoryFile(name, node);
this.children.set(name, file);
return file;
}
/** Test helper: synchronously add a file with given bytes. */
addFile(name: string, bytes: Uint8Array, type: string = ''): MemoryFile {
const node: MemoryNode = { bytes, type };
const file = new MemoryFile(name, node);
this.children.set(name, file);
return file;
}
/** Test helper: synchronously add a subdirectory. */
addDir(name: string): MemoryDirectory {
const dir = new MemoryDirectory(name);
this.children.set(name, dir);
return dir;
}
}
/** Construct an empty in-memory directory tree. */
export function createMemoryDirectory(name: string = ''): DirectoryHandleLike & {
addFile(name: string, bytes: Uint8Array, type?: string): FileHandleLike;
addDir(name: string): DirectoryHandleLike;
} {
return new MemoryDirectory(name);
}

View File

@@ -0,0 +1,251 @@
/**
* Client-side bridge between `@shade/files` content RPC ops and the
* `@shade/transfer` engine.
*
* Two responsibilities, mirror-image of `server/streams-bridge.ts`:
* 1. **Outbound writes (client → server, > 256 KiB).** When `FileClient.write`
* promotes to the streams path, this bridge calls `shade.upload(...)`
* with `userMetadata.shadeFilesWriteId = <id>` so the server can
* correlate the inbound transfer with the parallel `write` RPC.
*
* 2. **Inbound reads (server → client, > 256 KiB).** When `FileClient.read`
* gets a `{ kind: 'streams', streamId, ... }` RPC response, it asks
* this bridge for the matching incoming transfer's plaintext stream.
* The bridge subscribes to `shade.onIncomingTransfer` once and, on
* each transfer tagged with `userMetadata.shadeFilesReadStreamId`,
* **immediately** calls `accept(...)` (the engine rejects chunks that
* arrive before accept), pipes plaintext into a TransformStream, and
* parks the readable side until the matching read RPC awaits it.
*/
import type { TransferHandle, TransferProgress } from '@shade/transfer';
import { generateRequestId } from '../protocol/correlate.js';
import { OperationTimeoutError } from '../schemas/errors.js';
import {
META_KEY_READ_STREAM_ID,
META_KEY_WRITE_ID,
type StreamsBridgeShade,
} from '../server/streams-bridge.js';
export interface AwaitReadOptions {
/** Sender address — must match `incoming.from` for delivery. */
expectedFrom: string;
signal: AbortSignal;
/** Hard deadline (ms from now). Default 60_000. */
timeoutMs?: number;
}
export interface ParkedRead {
from: string;
/** Plaintext stream. The bridge already accepted the transfer. */
readable: ReadableStream<Uint8Array>;
/** Resolves when the transfer fully completes (verified sha256 available). */
done: Promise<{ sha256: string; bytesSent: number }>;
/** Underlying transfer handle — for abort propagation. */
handle: TransferHandle;
arrivedAt: number;
}
export interface ClientStreamsBridge {
/**
* Generate a fresh writeId, kick `shade.upload(...)` to `peer` with that
* id stamped in `userMetadata`, and return both the id (for the parallel
* RPC envelope) and the transfer handle (for `done()`/`abort()`).
*/
initiateWrite(opts: {
peer: string;
stream: ReadableStream<Uint8Array>;
size: number;
contentType?: string;
name?: string;
signal?: AbortSignal;
onProgress?: (p: TransferProgress) => void;
}): Promise<{ writeId: string; handle: TransferHandle }>;
/**
* Wait for an inbound transfer carrying `userMetadata.shadeFilesReadStreamId
* === streamId` from `expectedFrom`. Resolves with the parked entry whose
* `readable` can be consumed by the caller.
*/
awaitRead(streamId: string, opts: AwaitReadOptions): Promise<ParkedRead>;
destroy(): Promise<void>;
}
interface PendingReadWaiter {
resolve: (parked: ParkedRead) => void;
reject: (err: unknown) => void;
expectedFrom: string;
timer: ReturnType<typeof setTimeout> | null;
abortListener: (() => void) | null;
signal: AbortSignal;
}
export interface CreateClientStreamsBridgeOptions {
/** Default deadline for `awaitRead` if the caller doesn't supply one. */
defaultAwaitReadTimeoutMs?: number;
/** How long to retain a parked transfer waiting for its RPC. Default 60_000. */
parkedReadTtlMs?: number;
}
export async function createClientStreamsBridge(
shade: StreamsBridgeShade,
options: CreateClientStreamsBridgeOptions = {},
): Promise<ClientStreamsBridge> {
const parkedReadTtlMs = options.parkedReadTtlMs ?? 60_000;
const defaultAwaitTimeoutMs = options.defaultAwaitReadTimeoutMs ?? 60_000;
const parked = new Map<string, ParkedRead>();
const waiters = new Map<string, PendingReadWaiter>();
let destroyed = false;
const unsubscribe = await shade.onIncomingTransfer(async (incoming) => {
const readStreamId = incoming.metadata.userMetadata?.[META_KEY_READ_STREAM_ID];
if (readStreamId === undefined) return;
const ts = new TransformStream<Uint8Array, Uint8Array>();
let handle: TransferHandle;
try {
handle = await incoming.accept({
output: { kind: 'pipe', pipeTo: ts.writable },
});
} catch (err) {
console.error('[shade-files client streams-bridge] accept failed:', err);
return;
}
const arrival: ParkedRead = {
from: incoming.from,
readable: ts.readable,
done: handle.done().then((r) => ({ sha256: r.sha256, bytesSent: r.bytesSent })),
handle,
arrivedAt: Date.now(),
};
arrival.done.catch(() => {
/* swallow until consumer awaits */
});
const waiter = waiters.get(readStreamId);
if (waiter !== undefined) {
waiters.delete(readStreamId);
cleanupWaiter(waiter);
if (incoming.from !== waiter.expectedFrom) {
void handle.abort('sender-mismatch').catch(() => undefined);
waiter.reject(
new Error(
`streams-bridge: readStreamId=${readStreamId} delivered by ${incoming.from}, expected ${waiter.expectedFrom}`,
),
);
return;
}
waiter.resolve(arrival);
return;
}
parked.set(readStreamId, arrival);
setTimeout(() => {
const stale = parked.get(readStreamId);
if (stale === arrival) {
parked.delete(readStreamId);
void handle.abort('rpc-timeout').catch(() => undefined);
}
}, parkedReadTtlMs).unref?.();
});
function cleanupWaiter(w: PendingReadWaiter): void {
if (w.timer !== null) clearTimeout(w.timer);
if (w.abortListener !== null) {
w.signal.removeEventListener('abort', w.abortListener);
}
}
return {
async initiateWrite(opts) {
if (destroyed) throw new Error('streams-bridge: destroyed');
const writeId = generateRequestId();
const transferOpts: import('@shade/transfer').TransferOptions = {
to: opts.peer,
input: opts.stream,
metadata: {
...(opts.name !== undefined ? { name: opts.name } : {}),
...(opts.contentType !== undefined ? { contentType: opts.contentType } : {}),
sizeBytes: opts.size,
userMetadata: { [META_KEY_WRITE_ID]: writeId },
},
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
...(opts.onProgress !== undefined ? { onProgress: opts.onProgress } : {}),
};
const handle = await shade.upload(transferOpts);
return { writeId, handle };
},
async awaitRead(streamId, opts) {
if (destroyed) throw new Error('streams-bridge: destroyed');
const ready = parked.get(streamId);
if (ready !== undefined) {
parked.delete(streamId);
if (ready.from !== opts.expectedFrom) {
void ready.handle.abort('sender-mismatch').catch(() => undefined);
throw new Error(
`streams-bridge: readStreamId=${streamId} delivered by ${ready.from}, expected ${opts.expectedFrom}`,
);
}
return ready;
}
if (waiters.has(streamId)) {
throw new Error(`streams-bridge: readStreamId=${streamId} already awaited`);
}
const timeoutMs = opts.timeoutMs ?? defaultAwaitTimeoutMs;
return await new Promise<ParkedRead>((resolve, reject) => {
const w: PendingReadWaiter = {
resolve,
reject,
expectedFrom: opts.expectedFrom,
timer: null,
abortListener: null,
signal: opts.signal,
};
w.timer = setTimeout(() => {
if (waiters.get(streamId) === w) {
waiters.delete(streamId);
cleanupWaiter(w);
reject(new OperationTimeoutError(`streams-bridge: readStreamId=${streamId} timed out after ${timeoutMs}ms`));
}
}, timeoutMs);
if (opts.signal.aborted) {
cleanupWaiter(w);
reject(opts.signal.reason ?? new Error('aborted before await'));
return;
}
const onAbort = (): void => {
if (waiters.get(streamId) === w) {
waiters.delete(streamId);
cleanupWaiter(w);
reject(opts.signal.reason ?? new Error('aborted'));
}
};
w.abortListener = onAbort;
opts.signal.addEventListener('abort', onAbort, { once: true });
waiters.set(streamId, w);
});
},
async destroy() {
if (destroyed) return;
destroyed = true;
unsubscribe();
for (const w of waiters.values()) {
cleanupWaiter(w);
w.reject(new Error('streams-bridge: destroyed'));
}
waiters.clear();
for (const p of parked.values()) {
try {
await p.handle.abort('bridge-destroyed');
} catch {
/* swallow */
}
}
parked.clear();
},
};
}

View File

@@ -0,0 +1,310 @@
/**
* Upload an entire local directory tree to a remote peer via a `FileClient`.
*
* Walks the local `DirectoryHandleLike` lazily, creates remote directories
* via `client.mkdir({ recursive: true })`, and uploads each file with
* `client.write` (which routes inline or streams based on size). A bounded
* concurrency pool keeps memory and inflight RPCs in check.
*
* Returns a `BulkTransferHandle` whose `events` stream emits `'plan'`,
* `'file-start'`, `'file-progress'` (currently emitted at file start +
* end), `'file-done'` / `'file-error'`, aggregate `'progress'`, and a
* final `'complete'` (or `'abort'`).
*/
import { posixJoin } from '../utils/path.js';
import { runWithConcurrency } from './concurrency.js';
import type { FileClient } from './client.js';
import {
DEFAULT_BULK_CONCURRENCY,
MAX_BULK_CONCURRENCY,
type BulkOpts,
type BulkTransferEvent,
type BulkTransferHandle,
type BulkTransferResult,
type DirectoryHandleLike,
type FileHandleLike,
} from './directory-types.js';
import { CancelledError } from '../schemas/errors.js';
interface PlannedFile {
/** Local file handle. */
handle: FileHandleLike;
/** Path relative to the upload root. */
relativePath: string;
/** Absolute remote path. */
remoteAbsPath: string;
}
interface PlannedDir {
remoteAbsPath: string;
}
export interface UploadDirectoryOptions extends BulkOpts {
/**
* Pre-create remote directories before uploading files. Default true.
* Disable if the server-side `write` already mkdir-p's parents.
*/
precreateDirs?: boolean;
}
export function uploadDirectory(
client: Pick<FileClient, 'write' | 'mkdir'>,
local: DirectoryHandleLike,
remoteRoot: string,
opts: UploadDirectoryOptions = {},
): BulkTransferHandle {
const concurrency = Math.max(
1,
Math.min(MAX_BULK_CONCURRENCY, opts.concurrency ?? DEFAULT_BULK_CONCURRENCY),
);
const precreateDirs = opts.precreateDirs ?? true;
const continueOnError = opts.continueOnError ?? false;
const externalSignal = opts.signal;
const internalAbort = new AbortController();
const combinedSignal = mergeSignals(externalSignal, internalAbort.signal);
const events: BulkTransferEvent[] = [];
const eventResolvers: ((v: IteratorResult<BulkTransferEvent>) => void)[] = [];
let eventsClosed = false;
function emit(event: BulkTransferEvent): void {
if (eventsClosed) return;
if (eventResolvers.length > 0) {
eventResolvers.shift()!({ value: event, done: false });
} else {
events.push(event);
}
}
function closeEvents(): void {
if (eventsClosed) return;
eventsClosed = true;
while (eventResolvers.length > 0) {
eventResolvers.shift()!({ value: undefined as never, done: true });
}
}
let resolveDone!: (r: BulkTransferResult) => void;
let rejectDone!: (err: unknown) => void;
const donePromise = new Promise<BulkTransferResult>((resolve, reject) => {
resolveDone = resolve;
rejectDone = reject;
});
const startedAt = Date.now();
let filesDone = 0;
let bytesDone = 0;
let bytesTotal = 0;
// Run the upload pipeline asynchronously.
void (async () => {
try {
// 1. Plan: walk local tree, collect dirs + files
const plannedDirs: PlannedDir[] = [];
const plannedFiles: PlannedFile[] = [];
try {
await collect(local, '', remoteRoot, plannedDirs, plannedFiles, combinedSignal);
} catch (err) {
emit({ type: 'abort', reason: errMsg(err) });
closeEvents();
rejectDone(err);
return;
}
const totalFiles = plannedFiles.length;
bytesTotal = plannedFiles.reduce(
(acc, f) => acc + (f.handle as FileHandleLike & { _size?: number })._size!,
0,
);
// Note: bytesTotal is computed lazily in collect() via cached sizes.
emit({ type: 'plan', totalFiles, totalBytes: bytesTotal });
if (combinedSignal.aborted) {
emit({ type: 'abort', reason: errMsg(combinedSignal.reason) });
closeEvents();
rejectDone(combinedSignal.reason ?? new CancelledError());
return;
}
// 2. Pre-create remote directories sequentially (cheap, avoids races
// when many uploads target the same parent).
if (precreateDirs) {
for (const d of plannedDirs) {
if (combinedSignal.aborted) break;
try {
await client.mkdir(d.remoteAbsPath, { recursive: true });
} catch (err) {
if (!isAlreadyExistsError(err)) {
if (!continueOnError) {
emit({ type: 'abort', reason: errMsg(err) });
closeEvents();
rejectDone(err);
return;
}
}
}
}
}
// 3. Upload files with bounded concurrency
try {
await runWithConcurrency(
asyncIterableOf(plannedFiles),
async (planned) => {
if (combinedSignal.aborted) {
throw new CancelledError('upload aborted');
}
const file = await planned.handle.getFile();
const size = file.size;
emit({ type: 'file-start', path: planned.relativePath, size });
try {
const writeOpts: { contentType?: string; signal?: AbortSignal } = {};
if (file.type !== '') writeOpts.contentType = file.type;
writeOpts.signal = combinedSignal;
if (size === 0) {
// Edge case: empty file — write empty buffer.
await client.write(planned.remoteAbsPath, new Uint8Array(0), writeOpts);
} else {
await client.write(
planned.remoteAbsPath,
{ stream: file.stream(), size, ...(file.type !== '' ? { contentType: file.type } : {}) },
writeOpts,
);
}
filesDone++;
bytesDone += size;
emit({ type: 'file-done', path: planned.relativePath, bytesDone: size });
emit({
type: 'progress',
filesDone,
filesTotal: totalFiles,
bytesDone,
bytesTotal,
});
} catch (err) {
emit({ type: 'file-error', path: planned.relativePath, error: err });
throw err;
}
},
{
concurrency,
signal: combinedSignal,
continueOnError,
onError: () => {
/* already emitted as 'file-error' */
},
},
);
} catch (err) {
emit({ type: 'abort', reason: errMsg(err) });
closeEvents();
rejectDone(err);
return;
}
const durationMs = Date.now() - startedAt;
emit({ type: 'complete', filesDone, bytesDone, durationMs });
closeEvents();
resolveDone({ filesDone, bytesDone, durationMs });
} catch (err) {
// Belt-and-suspenders: any unhandled error reaches here.
closeEvents();
rejectDone(err);
}
})();
// Suppress unhandled-rejection until consumer awaits done().
donePromise.catch(() => {
/* deliberate */
});
return {
events: {
[Symbol.asyncIterator]() {
return {
next(): Promise<IteratorResult<BulkTransferEvent>> {
if (events.length > 0) {
return Promise.resolve({ value: events.shift()!, done: false });
}
if (eventsClosed) {
return Promise.resolve({ value: undefined as never, done: true });
}
return new Promise((resolve) => eventResolvers.push(resolve));
},
return(): Promise<IteratorResult<BulkTransferEvent>> {
// Caller broke out — stop iterating but keep the bulk going.
return Promise.resolve({ value: undefined as never, done: true });
},
};
},
},
async abort(reason) {
internalAbort.abort(new CancelledError(reason ?? 'manual abort'));
},
done: () => donePromise,
};
}
// ─── Helpers ─────────────────────────────────────────────────
async function collect(
dir: DirectoryHandleLike,
relPrefix: string,
remoteRoot: string,
plannedDirs: PlannedDir[],
plannedFiles: PlannedFile[],
signal: AbortSignal,
): Promise<void> {
for await (const [name, handle] of dir.entries()) {
if (signal.aborted) return;
const relPath = relPrefix === '' ? name : `${relPrefix}/${name}`;
if (handle.kind === 'directory') {
const remoteAbs = posixJoin(remoteRoot, relPath);
plannedDirs.push({ remoteAbsPath: remoteAbs });
await collect(handle as DirectoryHandleLike, relPath, remoteRoot, plannedDirs, plannedFiles, signal);
} else {
// Cache size on the handle so the planning total is accurate.
const file = await (handle as FileHandleLike).getFile();
(handle as FileHandleLike & { _size?: number })._size = file.size;
plannedFiles.push({
handle: handle as FileHandleLike,
relativePath: relPath,
remoteAbsPath: posixJoin(remoteRoot, relPath),
});
}
}
}
async function* asyncIterableOf<T>(arr: T[]): AsyncIterable<T> {
for (const item of arr) yield item;
}
function isAlreadyExistsError(err: unknown): boolean {
if (err === null || typeof err !== 'object') return false;
const code = (err as { code?: string; payload?: { code?: string } }).code;
if (code === 'SHADE_FS_CONFLICT') return true;
const payloadCode = (err as { payload?: { code?: string } }).payload?.code;
return payloadCode === 'CONFLICT';
}
function errMsg(err: unknown): string {
if (err instanceof Error) return err.message;
return String(err ?? 'unknown error');
}
function mergeSignals(
external: AbortSignal | undefined,
internal: AbortSignal,
): AbortSignal {
if (external === undefined) return internal;
if (external.aborted) return external;
const controller = new AbortController();
const onAbort = (sig: AbortSignal): void => {
controller.abort(sig.reason);
};
external.addEventListener('abort', () => onAbort(external), { once: true });
internal.addEventListener('abort', () => onAbort(internal), { once: true });
return controller.signal;
}

View File

@@ -0,0 +1,89 @@
/**
* Async-iterable directory walker.
*
* Depth-first by default — yields a directory's entries before descending
* into the next sibling. Memory-bounded: never materializes the whole tree;
* uses `client.list` page-by-page.
*
* Designed for arbitrarily-large remote trees: the consumer can break out
* of the iterator at any point and the walk halts cleanly.
*/
import { posixJoin } from '../utils/path.js';
import { CancelledError } from '../schemas/errors.js';
import type { FileEntry } from '../schemas/file-entry.js';
import type { FileClient } from './client.js';
export interface WalkOpts {
/** Hard cap on recursion depth. Default 32. */
maxDepth?: number;
/** Cancellation. */
signal?: AbortSignal;
/**
* Apply a per-entry filter. Return `false` to skip an entry (and, for
* directories, to skip descending into it). Default: include all.
*/
filter?: (entry: FileEntry, relativePath: string) => boolean;
/** Page size hint for `client.list`. Default 200. */
pageSize?: number;
}
export interface WalkItem {
entry: FileEntry;
/** Path relative to the walk root. Includes the entry's own name. */
relativePath: string;
/** Absolute path on the remote (root + relativePath). */
absolutePath: string;
/** Depth from root — 1 for direct children. */
depth: number;
}
const DEFAULT_MAX_DEPTH = 32;
const DEFAULT_PAGE_SIZE = 200;
export async function* walk(
client: Pick<FileClient, 'list'>,
rootPath: string,
opts: WalkOpts = {},
): AsyncIterable<WalkItem> {
const maxDepth = opts.maxDepth ?? DEFAULT_MAX_DEPTH;
const pageSize = opts.pageSize ?? DEFAULT_PAGE_SIZE;
function checkAbort(): void {
if (opts.signal?.aborted) {
throw new CancelledError(
opts.signal.reason instanceof Error
? opts.signal.reason.message
: 'walk aborted',
);
}
}
async function* descend(absDir: string, depth: number, relPrefix: string): AsyncIterable<WalkItem> {
if (depth > maxDepth) return;
let cursor: string | undefined;
do {
checkAbort();
const page = await client.list(absDir, {
pageSize,
...(cursor !== undefined ? { cursor } : {}),
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
});
for (const entry of page.entries) {
checkAbort();
const relPath = relPrefix === '' ? entry.name : `${relPrefix}/${entry.name}`;
const absPath = posixJoin(absDir, entry.name);
if (opts.filter !== undefined && !opts.filter(entry, relPath)) {
continue;
}
const item: WalkItem = { entry, relativePath: relPath, absolutePath: absPath, depth };
yield item;
if (entry.kind === 'dir') {
yield* descend(absPath, depth + 1, relPath);
}
}
cursor = page.hasMore ? page.nextCursor : undefined;
} while (cursor !== undefined);
}
yield* descend(rootPath, 1, '');
}