feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
Some checks failed
Test / test (push) Has been cancelled

M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).

@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
  Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
  + per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
  (> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
  / shadeFilesReadStreamId correlation. Server-side TransformStream
  bridges accept inbound transfers immediately (engine rejects chunks
  that arrive before accept) and park the readable for the matching
  RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
  uploadDirectory()/downloadDirectory() with bounded concurrency pool
  (default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
  byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
  policy (traversal + percent-decode hardening), fingerprint gate
  (required/optional/reject), pluggable Ed25519 sig verification with
  ±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
  useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
  entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.

Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
  u16 to u32. The previous u16 silently truncated payloads above
  64 KiB — a hard correctness ceiling that blocked inline file ops
  up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
  only. Cross-platform Kotlin port (android/shade-android) updated to
  match; test-vectors/wire-format.json regenerated.

Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
  Concurrent decryptions of the same peer raced ratchet state
  (manifested as sporadic "Failed to decrypt — wrong key or tampered
  data" under load — surfaced once concurrent uploadDirectory pumped
  many writes in flight). Encrypt was already serialized via
  Shade.send's encryptChains; decrypt is now serialized at the
  manager layer too.

@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
  application-level key/value pairs that round-trip verbatim through
  stream-init plaintext. Used by @shade/files for write/read
  correlation; available to any consumer.

@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
  BackgroundTasks.setHook(name, fn) for runtime hook registration.

Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
  shade-sdk streams-bridge + shade-widgets transfer hooks were
  uncommitted prior to this session. Including them keeps the
  workspace consistent at 0.3.0 since @shade/files depends on them.

Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
  3× stable). Coverage spans unit (inline-threshold + concurrency),
  integration (read-write inline + streams up to 1 MiB, walk +
  upload/download directory, custom-op, metrics, SDK namespace
  end-to-end), and security (tampered-envelope sig verification,
  replay window, fingerprint gate, rate-limit + quota).

Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
  topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
  server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-02 14:00:01 +02:00
parent 7e0f7320a9
commit fa770d3063
198 changed files with 20412 additions and 256 deletions

View File

@@ -0,0 +1,483 @@
import type { Shade } from '@shade/sdk';
import {
KIND_CUSTOM_V1,
KIND_DELETE_V1,
KIND_GET_THUMBNAIL_V1,
KIND_LIST_V1,
KIND_MKDIR_V1,
KIND_MOVE_V1,
KIND_READ_V1,
KIND_STAT_V1,
KIND_WRITE_V1,
MUTATION_OPS,
type StandardOp,
} from '../protocol/kinds.js';
import type { CustomOpsMap } from '../server/custom-ops.js';
import { generateIdempotencyKey, generateRequestId } from '../protocol/correlate.js';
import {
base64ToBytes,
bytesToBase64,
canonicalRpcBytes,
hashArgs,
} from '../protocol/canonical.js';
import { ShadeFileRpcChannel } from '../rpc/channel.js';
import { PendingRpcRegistry, type RegisterOptions } from '../rpc/pending.js';
import type { RpcRequest } from '../schemas/envelope.js';
import {
CustomArgsSchema,
CustomResultSchema,
DeleteArgsSchema,
DeleteResultSchema,
GetThumbnailArgsSchema,
GetThumbnailResultSchema,
ListArgsSchema,
ListResultSchema,
MkdirArgsSchema,
MkdirResultSchema,
MoveArgsSchema,
MoveResultSchema,
ReadArgsSchema,
ReadResultSchema,
StatArgsSchema,
StatResultSchema,
WriteArgsSchema,
WriteResultSchema,
type DeleteArgs,
type DeleteResult,
type GetThumbnailArgs,
type ListArgs,
type ListResult,
type MkdirArgs,
type MkdirResult,
type MoveArgs,
type MoveResult,
type ReadArgs,
type StatResult,
type ThumbnailSize,
type WriteResult,
} from '../schemas/ops.js';
import { ConflictError, InternalFileError } from '../schemas/errors.js';
import { decideInline, INLINE_THRESHOLD, type WriteSource } from './inline-threshold.js';
import type { ClientStreamsBridge } from './streams-bridge.js';
export interface BaseOpts {
signal?: AbortSignal;
/** Auto-generated for mutations if not provided. */
idempotencyKey?: string;
/** Per-call timeout. Default 30_000 ms. */
timeoutMs?: number;
}
// ─── read/write public types ─────────────────────────────────
export interface ReadInlineOutput {
kind: 'inline';
bytes: Uint8Array;
size: number;
sha256: string;
contentType?: string;
}
export interface ReadStreamsOutput {
kind: 'streams';
stream: ReadableStream<Uint8Array>;
size: number;
sha256: string;
contentType?: string;
/** Resolves once the entire transfer has been received and verified. */
done(): Promise<void>;
}
export type ReadOutput = ReadInlineOutput | ReadStreamsOutput;
export interface WriteOpts extends BaseOpts {
contentType?: string;
overwrite?: boolean;
/** Force inline even if size > 256 KiB. Throws if input is too big. */
forceInline?: boolean;
}
export interface ReadOpts extends BaseOpts {
range?: { start: number; end: number };
preferInline?: boolean;
}
export interface ThumbnailResult {
bytes: Uint8Array;
format: 'png' | 'webp' | 'jpeg';
width: number;
height: number;
sha256: string;
}
// ─── FileClient interface ────────────────────────────────────
/**
* Untyped fallback for `FileClient.custom()` — used when the consumer
* hasn't extended `CustomOpsMap` for a given op name.
*/
type CustomOpArgs<K> = K extends keyof CustomOpsMap
? CustomOpsMap[K] extends { args: infer A }
? A
: unknown
: unknown;
type CustomOpResponse<K> = K extends keyof CustomOpsMap
? CustomOpsMap[K] extends { response: infer R }
? R
: unknown
: unknown;
export interface FileClient {
list(path: string, opts?: BaseOpts & Partial<Omit<ListArgs, 'path'>>): Promise<ListResult>;
stat(path: string, opts?: BaseOpts): Promise<StatResult>;
mkdir(path: string, opts?: BaseOpts & Partial<Omit<MkdirArgs, 'path'>>): Promise<MkdirResult>;
delete(path: string, opts?: BaseOpts & Partial<Omit<DeleteArgs, 'path'>>): Promise<DeleteResult>;
move(src: string, dst: string, opts?: BaseOpts & Partial<Omit<MoveArgs, 'src' | 'dst'>>): Promise<MoveResult>;
read(path: string, opts?: ReadOpts): Promise<ReadOutput>;
write(path: string, input: WriteSource, opts?: WriteOpts): Promise<WriteResult>;
getThumbnail(
path: string,
size: ThumbnailSize,
opts?: BaseOpts & { format?: 'png' | 'webp' | 'jpeg' },
): Promise<ThumbnailResult>;
/**
* Invoke a custom op registered on the server. Args/response types are
* pulled from `CustomOpsMap` via TypeScript declaration merging — see
* `server/custom-ops.ts` for the registration pattern.
*/
custom<K extends keyof CustomOpsMap | string>(
name: K & string,
args: CustomOpArgs<K>,
opts?: BaseOpts,
): Promise<CustomOpResponse<K>>;
close(): void;
}
export interface CreateFileClientOptions {
/** Default per-call timeout. Default 30_000. */
defaultTimeoutMs?: number;
/** Hard deadline for incoming-read awaits. Default 60_000. */
ioTimeoutMs?: number;
/**
* Required for read/write `streams` ops. Coordinates inbound/outbound
* `@shade/transfer` transfers via `userMetadata.shadeFiles*Id` keys.
*/
streamsBridge?: ClientStreamsBridge;
/**
* Optional: sign the canonical bytes of every outgoing RPC envelope.
* Pluggable so apps can plug their own signing-key store (e.g.,
* Ed25519-as-a-service, browser SubtleCrypto). When omitted, ships
* `'unsigned'` — the server's `verifySender` should also be unset, or
* be configured to accept the placeholder.
*/
signRequest?: (canonicalBytes: Uint8Array) => Promise<string> | string;
}
/**
* Client-side proxy for `@shade/files` ops. Each method ships an
* `RpcRequest` over `Shade.send`/`Shade.receive` and awaits the matching
* response (or error/timeout) from `PendingRpcRegistry`.
*
* Mutations auto-generate an idempotency key per logical call (not per
* attempt) so transparent retries under the SDK don't produce duplicates.
*
* Read/write content I/O over 256 KiB requires a `streamsBridge` to be
* passed via options — it coordinates the inbound/outbound `@shade/transfer`
* transfers that carry the actual bytes.
*/
export function createFileClient(
shade: Shade,
channel: ShadeFileRpcChannel,
pending: PendingRpcRegistry,
peerAddress: string,
options: CreateFileClientOptions = {},
): FileClient {
const defaultTimeout = options.defaultTimeoutMs ?? 30_000;
const ioTimeoutMs = options.ioTimeoutMs ?? 60_000;
const streamsBridge = options.streamsBridge;
const signRequest = options.signRequest;
const senderAddress = shade.myAddress;
async function request<TResult>(
kind: string,
op: StandardOp | 'custom',
args: unknown,
opts: BaseOpts | undefined,
): Promise<TResult> {
const requestId = generateRequestId();
const isMutation = MUTATION_OPS.has(op);
const idempotencyKey =
opts?.idempotencyKey ?? (isMutation ? generateIdempotencyKey() : undefined);
const signedAt = Date.now();
let sig = 'unsigned';
if (signRequest !== undefined) {
// Server reconstructs canonical bytes using `address = from`, which
// is OUR own address as authenticated by the ratchet. So we sign
// over the same identifier here.
const canonical = canonicalRpcBytes({
address: senderAddress,
signedAt,
kind,
id: requestId,
argsHash: hashArgs(args),
});
sig = await signRequest(canonical);
}
const env: RpcRequest = {
kind,
id: requestId,
args,
...(idempotencyKey !== undefined ? { idempotencyKey } : {}),
sig,
signedAt,
};
const registerOpts: RegisterOptions = {
timeoutMs: opts?.timeoutMs ?? defaultTimeout,
onCancel: (reason) => {
// Fire-and-forget cancel envelope so server can release resources.
void channel
.send(peerAddress, {
kind: 'shade.fs.cancel/v1',
id: requestId,
reason,
})
.catch(() => {
/* swallow — cancellation is best-effort */
});
},
};
if (opts?.signal !== undefined) registerOpts.signal = opts.signal;
const pendingPromise = pending.register<TResult>(requestId, registerOpts);
try {
await channel.send(peerAddress, env);
} catch (err) {
// If the send itself fails, the pending entry will never resolve;
// reject it directly.
pending.rejectAll(err);
throw err;
}
return pendingPromise;
}
return {
async list(path, opts) {
const args: ListArgs = ListArgsSchema.parse({
path,
...(opts?.cursor !== undefined ? { cursor: opts.cursor } : {}),
...(opts?.pageSize !== undefined ? { pageSize: opts.pageSize } : {}),
...(opts?.filter !== undefined ? { filter: opts.filter } : {}),
});
const raw = await request<unknown>(KIND_LIST_V1, 'list', args, opts);
return ListResultSchema.parse(raw);
},
async stat(path, opts) {
const args = StatArgsSchema.parse({ path });
const raw = await request<unknown>(KIND_STAT_V1, 'stat', args, opts);
return StatResultSchema.parse(raw);
},
async mkdir(path, opts) {
const args = MkdirArgsSchema.parse({
path,
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
});
const raw = await request<unknown>(KIND_MKDIR_V1, 'mkdir', args, opts);
return MkdirResultSchema.parse(raw);
},
async delete(path, opts) {
const args = DeleteArgsSchema.parse({
path,
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
});
const raw = await request<unknown>(KIND_DELETE_V1, 'delete', args, opts);
return DeleteResultSchema.parse(raw);
},
async move(src, dst, opts) {
const args = MoveArgsSchema.parse({
src,
dst,
...(opts?.overwrite !== undefined ? { overwrite: opts.overwrite } : {}),
});
const raw = await request<unknown>(KIND_MOVE_V1, 'move', args, opts);
return MoveResultSchema.parse(raw);
},
async read(path, opts) {
const args: ReadArgs = ReadArgsSchema.parse({
path,
...(opts?.range !== undefined ? { range: opts.range } : {}),
...(opts?.preferInline !== undefined ? { preferInline: opts.preferInline } : {}),
});
const raw = await request<unknown>(KIND_READ_V1, 'read', args, opts);
const wire = ReadResultSchema.parse(raw);
if (wire.kind === 'inline') {
const bytes = base64ToBytes(wire.bytesB64);
const out: ReadInlineOutput = {
kind: 'inline',
bytes,
size: wire.size,
sha256: wire.sha256,
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
};
return out;
}
// streams — wait for the matching incoming transfer via the bridge.
if (streamsBridge === undefined) {
throw new InternalFileError(
'streams-bridge not configured: cannot consume streamed read',
);
}
const bridgeSignal = opts?.signal ?? new AbortController().signal;
const parked = await streamsBridge.awaitRead(wire.streamId, {
expectedFrom: peerAddress,
signal: bridgeSignal,
timeoutMs: ioTimeoutMs,
});
const out: ReadStreamsOutput = {
kind: 'streams',
stream: parked.readable,
size: wire.size,
sha256: wire.sha256,
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
done: async () => {
await parked.done;
},
};
return out;
},
async write(path, input, opts) {
const decision = await decideInline(input);
const overwrite = opts?.overwrite ?? false;
const contentType = opts?.contentType ?? decision.contentType;
if (decision.kind === 'inline' || opts?.forceInline === true) {
// Inline path — base64 in the RPC envelope.
const bytes =
decision.kind === 'inline'
? decision.bytes
: await drainToUint8Array(decision.stream, decision.size ?? Number.POSITIVE_INFINITY);
if (bytes.byteLength > INLINE_THRESHOLD && opts?.forceInline !== true) {
throw new ConflictError(
`inline write exceeds ${INLINE_THRESHOLD}-byte threshold (got ${bytes.byteLength}); pass forceInline=true to override`,
);
}
const args = WriteArgsSchema.parse({
kind: 'inline',
path,
bytesB64: bytesToBase64(bytes),
...(contentType !== undefined ? { contentType } : {}),
overwrite,
});
const raw = await request<unknown>(KIND_WRITE_V1, 'write', args, opts);
return WriteResultSchema.parse(raw);
}
// Streams path — kick the upload, then ship the RPC.
if (streamsBridge === undefined) {
throw new InternalFileError(
'streams-bridge not configured: cannot ship streamed write',
);
}
const size = decision.size;
if (size === undefined) {
throw new ConflictError(
'streams write requires a known plaintext size; pass `{ stream, size }` instead of a bare ReadableStream',
);
}
const { writeId, handle } = await streamsBridge.initiateWrite({
peer: peerAddress,
stream: decision.stream,
size,
...(contentType !== undefined ? { contentType } : {}),
name: path,
...(opts?.signal !== undefined ? { signal: opts.signal } : {}),
});
const args = WriteArgsSchema.parse({
kind: 'streams',
path,
size,
...(contentType !== undefined ? { contentType } : {}),
overwrite,
writeId,
});
try {
const [raw] = await Promise.all([
request<unknown>(KIND_WRITE_V1, 'write', args, opts),
handle.done(),
]);
return WriteResultSchema.parse(raw);
} catch (err) {
// Best-effort cancel of the transfer on RPC failure.
await handle.abort('rpc-failed').catch(() => undefined);
throw err;
}
},
async getThumbnail(path, size, opts) {
const args: GetThumbnailArgs = GetThumbnailArgsSchema.parse({
path,
size,
...(opts?.format !== undefined ? { format: opts.format } : {}),
});
const raw = await request<unknown>(KIND_GET_THUMBNAIL_V1, 'getThumbnail', args, opts);
const wire = GetThumbnailResultSchema.parse(raw);
return {
bytes: base64ToBytes(wire.bytesB64),
format: wire.format,
width: wire.width,
height: wire.height,
sha256: wire.sha256,
};
},
async custom(name, args, opts) {
const wireArgs = CustomArgsSchema.parse({ name, payload: args });
// `custom` is a mutation in the rate-limit sense; auto-key for retries.
const raw = await request<unknown>(KIND_CUSTOM_V1, 'custom' as StandardOp, wireArgs, opts);
const wire = CustomResultSchema.parse(raw);
// The result schema is `{ result: unknown }` — the inner `result` is
// already validated against the consumer's response schema on the
// server side, so we trust it here.
return wire.result as never;
},
close() {
pending.rejectAll(new Error('FileClient closed'));
},
};
}
/** Drain a stream into a single buffer; used for the inline-write fallback. */
async function drainToUint8Array(
stream: ReadableStream<Uint8Array>,
cap: number,
): Promise<Uint8Array> {
const reader = stream.getReader();
const chunks: Uint8Array[] = [];
let total = 0;
try {
while (true) {
const { value, done } = await reader.read();
if (done) break;
if (value === undefined) continue;
chunks.push(value);
total += value.byteLength;
if (total > cap) {
throw new Error(`stream produced more than declared size cap (${cap})`);
}
}
} finally {
reader.releaseLock();
}
const out = new Uint8Array(total);
let offset = 0;
for (const c of chunks) {
out.set(c, offset);
offset += c.byteLength;
}
return out;
}

View File

@@ -0,0 +1,91 @@
/**
* Bounded-concurrency async map.
*
* Pulls items from an `AsyncIterable` source (lazily — never materializing
* the whole sequence) and runs `fn` on at most `concurrency` of them at
* once. Failures bubble unless `continueOnError` is set, in which case they
* are reported via `onError` and the pool keeps draining.
*/
import { CancelledError } from '../schemas/errors.js';
export interface ConcurrentMapOptions<T> {
concurrency: number;
signal?: AbortSignal;
continueOnError?: boolean;
onError?: (item: T, err: unknown) => void;
}
export async function runWithConcurrency<T>(
source: AsyncIterable<T>,
fn: (item: T) => Promise<void>,
opts: ConcurrentMapOptions<T>,
): Promise<void> {
if (opts.concurrency < 1) throw new Error('concurrency must be ≥ 1');
const iter = source[Symbol.asyncIterator]();
const inFlight = new Set<Promise<void>>();
let firstError: unknown = null;
let aborted = false;
function checkAbort(): void {
if (opts.signal?.aborted) {
aborted = true;
const reason = opts.signal.reason;
throw reason instanceof Error ? reason : new CancelledError(String(reason ?? 'aborted'));
}
}
async function pumpOne(): Promise<boolean> {
checkAbort();
const next = await iter.next();
if (next.done === true) return false;
const item = next.value;
const task = (async () => {
try {
await fn(item);
} catch (err) {
if (opts.continueOnError === true) {
opts.onError?.(item, err);
} else if (firstError === null) {
firstError = err;
}
}
})().finally(() => {
inFlight.delete(task);
});
inFlight.add(task);
return true;
}
try {
// Prime the pool
while (inFlight.size < opts.concurrency) {
if (firstError !== null || aborted) break;
const more = await pumpOne();
if (!more) break;
}
// Maintain saturation
while (inFlight.size > 0) {
if (firstError !== null && opts.continueOnError !== true) break;
await Promise.race(inFlight);
while (
inFlight.size < opts.concurrency &&
firstError === null &&
!aborted
) {
const more = await pumpOne();
if (!more) break;
}
}
// Drain whatever survived
if (inFlight.size > 0) await Promise.allSettled(inFlight);
} finally {
if (typeof iter.return === 'function') {
await iter.return().catch(() => undefined);
}
}
if (firstError !== null && opts.continueOnError !== true) {
throw firstError;
}
}

View File

@@ -0,0 +1,97 @@
/**
* Public types for directory operations (`walk`, `uploadDirectory`,
* `downloadDirectory`).
*
* `DirectoryHandleLike` and `FileHandleLike` are structurally compatible
* with the browser File System Access API
* (`FileSystemDirectoryHandle` / `FileSystemFileHandle`) so that browser
* consumers can pass them directly. For Bun/Node, use the adapter from
* `node-directory-handle.ts`.
*/
export interface FileHandleLike {
readonly kind: 'file';
readonly name: string;
/** Read-side accessor — used by `uploadDirectory`. */
getFile(): Promise<FileLike>;
/** Write-side accessor — used by `downloadDirectory`. */
createWritable(): Promise<WritableStreamLike>;
}
export interface FileLike {
readonly name: string;
readonly size: number;
readonly type: string;
stream(): ReadableStream<Uint8Array>;
arrayBuffer(): Promise<ArrayBuffer>;
}
export interface WritableStreamLike {
write(chunk: Uint8Array): Promise<void>;
close(): Promise<void>;
abort(reason?: unknown): Promise<void>;
}
export interface DirectoryHandleLike {
readonly kind: 'directory';
readonly name: string;
/** Yield child entries (file or directory). Used by `uploadDirectory`. */
entries(): AsyncIterable<[string, DirectoryHandleLike | FileHandleLike]>;
/** Used by `downloadDirectory` to create remote → local mapping. */
getDirectoryHandle(name: string, opts?: { create?: boolean }): Promise<DirectoryHandleLike>;
getFileHandle(name: string, opts?: { create?: boolean }): Promise<FileHandleLike>;
}
// ─── Bulk transfer types ─────────────────────────────────────
export type BulkTransferEvent =
| { type: 'plan'; totalFiles: number; totalBytes: number | undefined }
| { type: 'file-start'; path: string; size: number }
| { type: 'file-progress'; path: string; bytesDone: number; bytesTotal: number }
| { type: 'file-done'; path: string; bytesDone: number }
| { type: 'file-error'; path: string; error: unknown }
| {
type: 'progress';
filesDone: number;
filesTotal: number;
bytesDone: number;
bytesTotal: number | undefined;
}
| { type: 'complete'; filesDone: number; bytesDone: number; durationMs: number }
| { type: 'abort'; reason: string };
export interface BulkTransferResult {
filesDone: number;
bytesDone: number;
durationMs: number;
}
export interface BulkTransferHandle {
/** Async-iterable event stream — plan, per-file events, aggregate progress. */
readonly events: AsyncIterable<BulkTransferEvent>;
abort(reason?: string): Promise<void>;
done(): Promise<BulkTransferResult>;
}
export interface BulkOpts {
/**
* Max files in flight at once. Default 4. Capped at 16 to bound memory
* (each in-flight file may hold a buffered chunk; with 1 MiB chunks +
* 16 lanes that's still bounded).
*/
concurrency?: number;
/**
* Continue past per-file failures. Default false (fail-fast).
* When true, errors are emitted via `file-error` events and the whole
* transfer still resolves (with `done()` returning the count of
* successful files).
*/
continueOnError?: boolean;
/** Cancellation. */
signal?: AbortSignal;
}
/** Hard cap to bound memory regardless of caller-supplied concurrency. */
export const MAX_BULK_CONCURRENCY = 16;
/** Default concurrency. */
export const DEFAULT_BULK_CONCURRENCY = 4;

View File

@@ -0,0 +1,316 @@
/**
* Download an entire remote directory tree to a local `DirectoryHandleLike`.
*
* Mirror image of `uploadDirectory`: walks the remote tree via the shared
* `walk()` helper, creates local directories on the fly, and downloads each
* file with `client.read` (which routes inline or streams based on the
* server's response). Bounded concurrency keeps RPC inflight count low.
*/
import { walk, type WalkItem } from './walk.js';
import { runWithConcurrency } from './concurrency.js';
import type { FileClient } from './client.js';
import {
DEFAULT_BULK_CONCURRENCY,
MAX_BULK_CONCURRENCY,
type BulkOpts,
type BulkTransferEvent,
type BulkTransferHandle,
type BulkTransferResult,
type DirectoryHandleLike,
type FileHandleLike,
} from './directory-types.js';
import { CancelledError } from '../schemas/errors.js';
export interface DownloadDirectoryOptions extends BulkOpts {
/** Page size hint forwarded to the underlying `walk`. Default 200. */
pageSize?: number;
/** Skip files already present locally. Default false (overwrite). */
skipExisting?: boolean;
}
export function downloadDirectory(
client: Pick<FileClient, 'read' | 'list'>,
remoteRoot: string,
local: DirectoryHandleLike,
opts: DownloadDirectoryOptions = {},
): BulkTransferHandle {
const concurrency = Math.max(
1,
Math.min(MAX_BULK_CONCURRENCY, opts.concurrency ?? DEFAULT_BULK_CONCURRENCY),
);
const continueOnError = opts.continueOnError ?? false;
const externalSignal = opts.signal;
const pageSize = opts.pageSize ?? 200;
const internalAbort = new AbortController();
const combinedSignal = mergeSignals(externalSignal, internalAbort.signal);
const events: BulkTransferEvent[] = [];
const eventResolvers: ((v: IteratorResult<BulkTransferEvent>) => void)[] = [];
let eventsClosed = false;
function emit(event: BulkTransferEvent): void {
if (eventsClosed) return;
if (eventResolvers.length > 0) {
eventResolvers.shift()!({ value: event, done: false });
} else {
events.push(event);
}
}
function closeEvents(): void {
if (eventsClosed) return;
eventsClosed = true;
while (eventResolvers.length > 0) {
eventResolvers.shift()!({ value: undefined as never, done: true });
}
}
let resolveDone!: (r: BulkTransferResult) => void;
let rejectDone!: (err: unknown) => void;
const donePromise = new Promise<BulkTransferResult>((resolve, reject) => {
resolveDone = resolve;
rejectDone = reject;
});
const startedAt = Date.now();
let filesDone = 0;
let bytesDone = 0;
void (async () => {
try {
// 1. Plan: walk the remote tree into [dirs, files]
const dirItems: WalkItem[] = [];
const fileItems: WalkItem[] = [];
try {
for await (const item of walk(client, remoteRoot, {
pageSize,
...(combinedSignal !== undefined ? { signal: combinedSignal } : {}),
})) {
if (item.entry.kind === 'dir') dirItems.push(item);
else fileItems.push(item);
}
} catch (err) {
emit({ type: 'abort', reason: errMsg(err) });
closeEvents();
rejectDone(err);
return;
}
const totalFiles = fileItems.length;
const bytesTotal = fileItems.reduce((acc, f) => acc + f.entry.size, 0);
emit({ type: 'plan', totalFiles, totalBytes: bytesTotal });
if (combinedSignal.aborted) {
emit({ type: 'abort', reason: errMsg(combinedSignal.reason) });
closeEvents();
rejectDone(combinedSignal.reason ?? new CancelledError());
return;
}
// 2. Pre-create local directories sequentially.
for (const d of dirItems) {
if (combinedSignal.aborted) break;
try {
await ensureLocalDir(local, d.relativePath);
} catch (err) {
if (!continueOnError) {
emit({ type: 'abort', reason: errMsg(err) });
closeEvents();
rejectDone(err);
return;
}
}
}
// 3. Download files with bounded concurrency.
try {
await runWithConcurrency(
asyncIterableOf(fileItems),
async (item) => {
if (combinedSignal.aborted) {
throw new CancelledError('download aborted');
}
emit({ type: 'file-start', path: item.relativePath, size: item.entry.size });
try {
const fileHandle = await ensureLocalFile(local, item.relativePath);
if (opts.skipExisting === true) {
// Skip if file already exists with non-zero size
try {
const existing = await fileHandle.getFile();
if (existing.size === item.entry.size) {
filesDone++;
bytesDone += existing.size;
emit({ type: 'file-done', path: item.relativePath, bytesDone: existing.size });
emit({
type: 'progress',
filesDone,
filesTotal: totalFiles,
bytesDone,
bytesTotal,
});
return;
}
} catch {
/* not present yet — fall through to download */
}
}
const writable = await fileHandle.createWritable();
try {
const result = await client.read(item.absolutePath, {
signal: combinedSignal,
});
if (result.kind === 'inline') {
if (result.bytes.byteLength > 0) await writable.write(result.bytes);
await writable.close();
filesDone++;
bytesDone += result.bytes.byteLength;
emit({ type: 'file-done', path: item.relativePath, bytesDone: result.bytes.byteLength });
} else {
await pipeReadableToWritable(result.stream, writable);
await writable.close();
await result.done();
filesDone++;
bytesDone += result.size;
emit({ type: 'file-done', path: item.relativePath, bytesDone: result.size });
}
emit({
type: 'progress',
filesDone,
filesTotal: totalFiles,
bytesDone,
bytesTotal,
});
} catch (err) {
await writable.abort(err).catch(() => undefined);
throw err;
}
} catch (err) {
emit({ type: 'file-error', path: item.relativePath, error: err });
throw err;
}
},
{
concurrency,
signal: combinedSignal,
continueOnError,
onError: () => {
/* already emitted as 'file-error' */
},
},
);
} catch (err) {
emit({ type: 'abort', reason: errMsg(err) });
closeEvents();
rejectDone(err);
return;
}
const durationMs = Date.now() - startedAt;
emit({ type: 'complete', filesDone, bytesDone, durationMs });
closeEvents();
resolveDone({ filesDone, bytesDone, durationMs });
} catch (err) {
closeEvents();
rejectDone(err);
}
})();
donePromise.catch(() => {
/* deliberate */
});
return {
events: {
[Symbol.asyncIterator]() {
return {
next(): Promise<IteratorResult<BulkTransferEvent>> {
if (events.length > 0) {
return Promise.resolve({ value: events.shift()!, done: false });
}
if (eventsClosed) {
return Promise.resolve({ value: undefined as never, done: true });
}
return new Promise((resolve) => eventResolvers.push(resolve));
},
return(): Promise<IteratorResult<BulkTransferEvent>> {
return Promise.resolve({ value: undefined as never, done: true });
},
};
},
},
async abort(reason) {
internalAbort.abort(new CancelledError(reason ?? 'manual abort'));
},
done: () => donePromise,
};
}
// ─── Helpers ─────────────────────────────────────────────────
async function ensureLocalDir(
root: DirectoryHandleLike,
relativePath: string,
): Promise<DirectoryHandleLike> {
const segments = relativePath.split('/').filter((s) => s !== '');
let current = root;
for (const seg of segments) {
current = await current.getDirectoryHandle(seg, { create: true });
}
return current;
}
async function ensureLocalFile(
root: DirectoryHandleLike,
relativePath: string,
): Promise<FileHandleLike> {
const segments = relativePath.split('/').filter((s) => s !== '');
if (segments.length === 0) throw new Error('empty file path');
let current = root;
for (let i = 0; i < segments.length - 1; i++) {
current = await current.getDirectoryHandle(segments[i]!, { create: true });
}
return await current.getFileHandle(segments[segments.length - 1]!, { create: true });
}
async function pipeReadableToWritable(
readable: ReadableStream<Uint8Array>,
writable: { write(chunk: Uint8Array): Promise<void> },
): Promise<void> {
const reader = readable.getReader();
try {
while (true) {
const { value, done } = await reader.read();
if (done) break;
if (value !== undefined && value.byteLength > 0) {
await writable.write(value);
}
}
} finally {
reader.releaseLock();
}
}
async function* asyncIterableOf<T>(arr: T[]): AsyncIterable<T> {
for (const item of arr) yield item;
}
function errMsg(err: unknown): string {
if (err instanceof Error) return err.message;
return String(err ?? 'unknown error');
}
function mergeSignals(
external: AbortSignal | undefined,
internal: AbortSignal,
): AbortSignal {
if (external === undefined) return internal;
if (external.aborted) return external;
const controller = new AbortController();
const onAbort = (sig: AbortSignal): void => {
controller.abort(sig.reason);
};
external.addEventListener('abort', () => onAbort(external), { once: true });
internal.addEventListener('abort', () => onAbort(internal), { once: true });
return controller.signal;
}

View File

@@ -0,0 +1,218 @@
/**
* Decide whether a write should travel inline (base64 inside the RPC
* envelope) or via a dedicated `@shade/transfer` stream.
*
* Threshold: 256 KiB plaintext. Anything strictly above goes through the
* stream path; anything ≤ goes inline.
*
* For known-size inputs (`Uint8Array`, `Blob`, `File`) the decision is
* cheap: compare `byteLength`/`size` against the threshold.
*
* For `ReadableStream` we cannot know the size up front. We pull chunks
* into a temporary buffer until we either:
* - hit EOF before the threshold → inline (we have all the bytes)
* - cross the threshold → streams (the buffered prefix + the rest of the
* stream are returned via a fresh `ReadableStream` so the caller can
* feed it to `shade.upload`).
*/
/** Plaintext size at which inline transitions to streams. */
export const INLINE_THRESHOLD = 256 * 1024;
export type InlineDecision =
| {
kind: 'inline';
bytes: Uint8Array;
contentType?: string;
}
| {
kind: 'streams';
stream: ReadableStream<Uint8Array>;
/** Plaintext size when known (Blob/File). undefined for raw streams. */
size?: number;
contentType?: string;
};
/**
* Public input shape for `FileClient.write`. Runtime discriminator helper.
*/
export type WriteSource =
| Uint8Array
| Blob
| File
| ReadableStream<Uint8Array>
| { stream: ReadableStream<Uint8Array>; size: number; contentType?: string };
export async function decideInline(input: WriteSource): Promise<InlineDecision> {
// 1. Uint8Array — direct size check
if (input instanceof Uint8Array) {
if (input.byteLength <= INLINE_THRESHOLD) {
return { kind: 'inline', bytes: input };
}
return {
kind: 'streams',
stream: uint8ArrayToStream(input),
size: input.byteLength,
};
}
// 2. Blob / File — known size (Blob.type is a string; File extends Blob)
if (typeof Blob !== 'undefined' && input instanceof Blob) {
const blob: Blob = input;
const contentType = blob.type === '' ? undefined : blob.type;
if (blob.size <= INLINE_THRESHOLD) {
const bytes = new Uint8Array(await blob.arrayBuffer());
return contentType !== undefined
? { kind: 'inline', bytes, contentType }
: { kind: 'inline', bytes };
}
return contentType !== undefined
? { kind: 'streams', stream: blob.stream(), size: blob.size, contentType }
: { kind: 'streams', stream: blob.stream(), size: blob.size };
}
// 3. Pre-wrapped { stream, size } — trust caller's declared size
if (
typeof input === 'object' &&
input !== null &&
'stream' in input &&
'size' in input &&
(input as { stream: unknown }).stream instanceof ReadableStream
) {
const wrapped = input as { stream: ReadableStream<Uint8Array>; size: number; contentType?: string };
if (wrapped.size <= INLINE_THRESHOLD) {
const bytes = await drainStream(wrapped.stream, wrapped.size);
const contentType = wrapped.contentType;
return contentType !== undefined
? { kind: 'inline', bytes, contentType }
: { kind: 'inline', bytes };
}
const contentType = wrapped.contentType;
return contentType !== undefined
? { kind: 'streams', stream: wrapped.stream, size: wrapped.size, contentType }
: { kind: 'streams', stream: wrapped.stream, size: wrapped.size };
}
// 4. Bare ReadableStream — peek until threshold or EOF
if (input instanceof ReadableStream) {
return await peekStream(input);
}
throw new TypeError(
`decideInline: unsupported input type ${Object.prototype.toString.call(input)}`,
);
}
/**
* Drain a stream into a Uint8Array, with a soft cap at `expected` + slack.
* Used for the inline path when the caller declared a size.
*/
async function drainStream(
stream: ReadableStream<Uint8Array>,
expected: number,
): Promise<Uint8Array> {
const reader = stream.getReader();
const chunks: Uint8Array[] = [];
let total = 0;
try {
while (true) {
const { value, done } = await reader.read();
if (done) break;
if (value === undefined) continue;
chunks.push(value);
total += value.byteLength;
if (total > expected + INLINE_THRESHOLD) {
throw new Error(
`decideInline: stream produced more bytes (${total}) than declared size (${expected})`,
);
}
}
} finally {
reader.releaseLock();
}
return concat(chunks, total);
}
/**
* Peek a `ReadableStream` of unknown length: buffer up to `INLINE_THRESHOLD + 1`
* bytes. If EOF first, return inline. Otherwise reconstruct a stream that
* yields the buffered prefix followed by the remainder.
*/
async function peekStream(stream: ReadableStream<Uint8Array>): Promise<InlineDecision> {
const reader = stream.getReader();
const buffered: Uint8Array[] = [];
let total = 0;
try {
while (total <= INLINE_THRESHOLD) {
const { value, done } = await reader.read();
if (done) {
reader.releaseLock();
return { kind: 'inline', bytes: concat(buffered, total) };
}
if (value === undefined) continue;
buffered.push(value);
total += value.byteLength;
}
// We have at least INLINE_THRESHOLD + 1 bytes buffered. Promote to streams.
const reconstructed = reconstructStream(buffered, reader);
return { kind: 'streams', stream: reconstructed };
} catch (err) {
reader.releaseLock();
throw err;
}
}
interface MinimalReader {
read(): Promise<{ value: Uint8Array | undefined; done: boolean }>;
cancel(reason?: unknown): Promise<void>;
releaseLock(): void;
}
function reconstructStream(
prefix: Uint8Array[],
reader: MinimalReader,
): ReadableStream<Uint8Array> {
let prefixIdx = 0;
return new ReadableStream<Uint8Array>({
async pull(controller) {
if (prefixIdx < prefix.length) {
controller.enqueue(prefix[prefixIdx]!);
prefixIdx++;
return;
}
const { value, done } = await reader.read();
if (done) {
controller.close();
reader.releaseLock();
return;
}
if (value !== undefined) controller.enqueue(value);
},
async cancel(reason) {
try {
await reader.cancel(reason);
} finally {
reader.releaseLock();
}
},
});
}
function uint8ArrayToStream(bytes: Uint8Array): ReadableStream<Uint8Array> {
return new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(bytes);
controller.close();
},
});
}
function concat(chunks: Uint8Array[], total: number): Uint8Array {
const out = new Uint8Array(total);
let offset = 0;
for (const c of chunks) {
out.set(c, offset);
offset += c.byteLength;
}
return out;
}

View File

@@ -0,0 +1,145 @@
/**
* In-memory `DirectoryHandleLike` implementation.
*
* Useful for tests and Node/Bun environments without the browser File
* System Access API. The shape is intentionally compatible with browser
* `FileSystemDirectoryHandle` so consumers can swap implementations.
*/
import type {
DirectoryHandleLike,
FileHandleLike,
FileLike,
WritableStreamLike,
} from './directory-types.js';
interface MemoryNode {
bytes: Uint8Array;
type: string;
}
class MemoryFileLike implements FileLike {
constructor(
public readonly name: string,
private readonly node: MemoryNode,
) {}
get size(): number { return this.node.bytes.byteLength; }
get type(): string { return this.node.type; }
stream(): ReadableStream<Uint8Array> {
const bytes = this.node.bytes;
return new ReadableStream<Uint8Array>({
start(controller) {
if (bytes.byteLength > 0) controller.enqueue(bytes);
controller.close();
},
});
}
async arrayBuffer(): Promise<ArrayBuffer> {
const out = new ArrayBuffer(this.node.bytes.byteLength);
new Uint8Array(out).set(this.node.bytes);
return out;
}
}
class MemoryWritable implements WritableStreamLike {
private chunks: Uint8Array[] = [];
constructor(private readonly file: MemoryFile) {}
async write(chunk: Uint8Array): Promise<void> {
this.chunks.push(chunk);
}
async close(): Promise<void> {
let total = 0;
for (const c of this.chunks) total += c.byteLength;
const out = new Uint8Array(total);
let offset = 0;
for (const c of this.chunks) {
out.set(c, offset);
offset += c.byteLength;
}
this.file.commit(out);
}
async abort(): Promise<void> {
this.chunks = [];
}
}
class MemoryFile implements FileHandleLike {
readonly kind: 'file' = 'file';
constructor(
public readonly name: string,
public readonly node: MemoryNode,
) {}
async getFile(): Promise<FileLike> {
return new MemoryFileLike(this.name, this.node);
}
async createWritable(): Promise<WritableStreamLike> {
return new MemoryWritable(this);
}
/** Internal — used by `MemoryWritable.close()`. */
commit(bytes: Uint8Array): void {
this.node.bytes = bytes;
}
}
class MemoryDirectory implements DirectoryHandleLike {
readonly kind: 'directory' = 'directory';
private readonly children = new Map<string, MemoryDirectory | MemoryFile>();
constructor(public readonly name: string) {}
async *entries(): AsyncIterable<[string, DirectoryHandleLike | FileHandleLike]> {
for (const [name, child] of this.children) {
yield [name, child];
}
}
async getDirectoryHandle(name: string, opts?: { create?: boolean }): Promise<DirectoryHandleLike> {
const existing = this.children.get(name);
if (existing !== undefined) {
if (existing.kind !== 'directory') {
throw new Error(`'${name}' exists but is a file`);
}
return existing;
}
if (opts?.create !== true) throw new Error(`directory not found: ${name}`);
const dir = new MemoryDirectory(name);
this.children.set(name, dir);
return dir;
}
async getFileHandle(name: string, opts?: { create?: boolean }): Promise<FileHandleLike> {
const existing = this.children.get(name);
if (existing !== undefined) {
if (existing.kind !== 'file') {
throw new Error(`'${name}' exists but is a directory`);
}
return existing;
}
if (opts?.create !== true) throw new Error(`file not found: ${name}`);
const node: MemoryNode = { bytes: new Uint8Array(0), type: '' };
const file = new MemoryFile(name, node);
this.children.set(name, file);
return file;
}
/** Test helper: synchronously add a file with given bytes. */
addFile(name: string, bytes: Uint8Array, type: string = ''): MemoryFile {
const node: MemoryNode = { bytes, type };
const file = new MemoryFile(name, node);
this.children.set(name, file);
return file;
}
/** Test helper: synchronously add a subdirectory. */
addDir(name: string): MemoryDirectory {
const dir = new MemoryDirectory(name);
this.children.set(name, dir);
return dir;
}
}
/** Construct an empty in-memory directory tree. */
export function createMemoryDirectory(name: string = ''): DirectoryHandleLike & {
addFile(name: string, bytes: Uint8Array, type?: string): FileHandleLike;
addDir(name: string): DirectoryHandleLike;
} {
return new MemoryDirectory(name);
}

View File

@@ -0,0 +1,251 @@
/**
* Client-side bridge between `@shade/files` content RPC ops and the
* `@shade/transfer` engine.
*
* Two responsibilities, mirror-image of `server/streams-bridge.ts`:
* 1. **Outbound writes (client → server, > 256 KiB).** When `FileClient.write`
* promotes to the streams path, this bridge calls `shade.upload(...)`
* with `userMetadata.shadeFilesWriteId = <id>` so the server can
* correlate the inbound transfer with the parallel `write` RPC.
*
* 2. **Inbound reads (server → client, > 256 KiB).** When `FileClient.read`
* gets a `{ kind: 'streams', streamId, ... }` RPC response, it asks
* this bridge for the matching incoming transfer's plaintext stream.
* The bridge subscribes to `shade.onIncomingTransfer` once and, on
* each transfer tagged with `userMetadata.shadeFilesReadStreamId`,
* **immediately** calls `accept(...)` (the engine rejects chunks that
* arrive before accept), pipes plaintext into a TransformStream, and
* parks the readable side until the matching read RPC awaits it.
*/
import type { TransferHandle, TransferProgress } from '@shade/transfer';
import { generateRequestId } from '../protocol/correlate.js';
import { OperationTimeoutError } from '../schemas/errors.js';
import {
META_KEY_READ_STREAM_ID,
META_KEY_WRITE_ID,
type StreamsBridgeShade,
} from '../server/streams-bridge.js';
export interface AwaitReadOptions {
/** Sender address — must match `incoming.from` for delivery. */
expectedFrom: string;
signal: AbortSignal;
/** Hard deadline (ms from now). Default 60_000. */
timeoutMs?: number;
}
export interface ParkedRead {
from: string;
/** Plaintext stream. The bridge already accepted the transfer. */
readable: ReadableStream<Uint8Array>;
/** Resolves when the transfer fully completes (verified sha256 available). */
done: Promise<{ sha256: string; bytesSent: number }>;
/** Underlying transfer handle — for abort propagation. */
handle: TransferHandle;
arrivedAt: number;
}
export interface ClientStreamsBridge {
/**
* Generate a fresh writeId, kick `shade.upload(...)` to `peer` with that
* id stamped in `userMetadata`, and return both the id (for the parallel
* RPC envelope) and the transfer handle (for `done()`/`abort()`).
*/
initiateWrite(opts: {
peer: string;
stream: ReadableStream<Uint8Array>;
size: number;
contentType?: string;
name?: string;
signal?: AbortSignal;
onProgress?: (p: TransferProgress) => void;
}): Promise<{ writeId: string; handle: TransferHandle }>;
/**
* Wait for an inbound transfer carrying `userMetadata.shadeFilesReadStreamId
* === streamId` from `expectedFrom`. Resolves with the parked entry whose
* `readable` can be consumed by the caller.
*/
awaitRead(streamId: string, opts: AwaitReadOptions): Promise<ParkedRead>;
destroy(): Promise<void>;
}
interface PendingReadWaiter {
resolve: (parked: ParkedRead) => void;
reject: (err: unknown) => void;
expectedFrom: string;
timer: ReturnType<typeof setTimeout> | null;
abortListener: (() => void) | null;
signal: AbortSignal;
}
export interface CreateClientStreamsBridgeOptions {
/** Default deadline for `awaitRead` if the caller doesn't supply one. */
defaultAwaitReadTimeoutMs?: number;
/** How long to retain a parked transfer waiting for its RPC. Default 60_000. */
parkedReadTtlMs?: number;
}
export async function createClientStreamsBridge(
shade: StreamsBridgeShade,
options: CreateClientStreamsBridgeOptions = {},
): Promise<ClientStreamsBridge> {
const parkedReadTtlMs = options.parkedReadTtlMs ?? 60_000;
const defaultAwaitTimeoutMs = options.defaultAwaitReadTimeoutMs ?? 60_000;
const parked = new Map<string, ParkedRead>();
const waiters = new Map<string, PendingReadWaiter>();
let destroyed = false;
const unsubscribe = await shade.onIncomingTransfer(async (incoming) => {
const readStreamId = incoming.metadata.userMetadata?.[META_KEY_READ_STREAM_ID];
if (readStreamId === undefined) return;
const ts = new TransformStream<Uint8Array, Uint8Array>();
let handle: TransferHandle;
try {
handle = await incoming.accept({
output: { kind: 'pipe', pipeTo: ts.writable },
});
} catch (err) {
console.error('[shade-files client streams-bridge] accept failed:', err);
return;
}
const arrival: ParkedRead = {
from: incoming.from,
readable: ts.readable,
done: handle.done().then((r) => ({ sha256: r.sha256, bytesSent: r.bytesSent })),
handle,
arrivedAt: Date.now(),
};
arrival.done.catch(() => {
/* swallow until consumer awaits */
});
const waiter = waiters.get(readStreamId);
if (waiter !== undefined) {
waiters.delete(readStreamId);
cleanupWaiter(waiter);
if (incoming.from !== waiter.expectedFrom) {
void handle.abort('sender-mismatch').catch(() => undefined);
waiter.reject(
new Error(
`streams-bridge: readStreamId=${readStreamId} delivered by ${incoming.from}, expected ${waiter.expectedFrom}`,
),
);
return;
}
waiter.resolve(arrival);
return;
}
parked.set(readStreamId, arrival);
setTimeout(() => {
const stale = parked.get(readStreamId);
if (stale === arrival) {
parked.delete(readStreamId);
void handle.abort('rpc-timeout').catch(() => undefined);
}
}, parkedReadTtlMs).unref?.();
});
function cleanupWaiter(w: PendingReadWaiter): void {
if (w.timer !== null) clearTimeout(w.timer);
if (w.abortListener !== null) {
w.signal.removeEventListener('abort', w.abortListener);
}
}
return {
async initiateWrite(opts) {
if (destroyed) throw new Error('streams-bridge: destroyed');
const writeId = generateRequestId();
const transferOpts: import('@shade/transfer').TransferOptions = {
to: opts.peer,
input: opts.stream,
metadata: {
...(opts.name !== undefined ? { name: opts.name } : {}),
...(opts.contentType !== undefined ? { contentType: opts.contentType } : {}),
sizeBytes: opts.size,
userMetadata: { [META_KEY_WRITE_ID]: writeId },
},
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
...(opts.onProgress !== undefined ? { onProgress: opts.onProgress } : {}),
};
const handle = await shade.upload(transferOpts);
return { writeId, handle };
},
async awaitRead(streamId, opts) {
if (destroyed) throw new Error('streams-bridge: destroyed');
const ready = parked.get(streamId);
if (ready !== undefined) {
parked.delete(streamId);
if (ready.from !== opts.expectedFrom) {
void ready.handle.abort('sender-mismatch').catch(() => undefined);
throw new Error(
`streams-bridge: readStreamId=${streamId} delivered by ${ready.from}, expected ${opts.expectedFrom}`,
);
}
return ready;
}
if (waiters.has(streamId)) {
throw new Error(`streams-bridge: readStreamId=${streamId} already awaited`);
}
const timeoutMs = opts.timeoutMs ?? defaultAwaitTimeoutMs;
return await new Promise<ParkedRead>((resolve, reject) => {
const w: PendingReadWaiter = {
resolve,
reject,
expectedFrom: opts.expectedFrom,
timer: null,
abortListener: null,
signal: opts.signal,
};
w.timer = setTimeout(() => {
if (waiters.get(streamId) === w) {
waiters.delete(streamId);
cleanupWaiter(w);
reject(new OperationTimeoutError(`streams-bridge: readStreamId=${streamId} timed out after ${timeoutMs}ms`));
}
}, timeoutMs);
if (opts.signal.aborted) {
cleanupWaiter(w);
reject(opts.signal.reason ?? new Error('aborted before await'));
return;
}
const onAbort = (): void => {
if (waiters.get(streamId) === w) {
waiters.delete(streamId);
cleanupWaiter(w);
reject(opts.signal.reason ?? new Error('aborted'));
}
};
w.abortListener = onAbort;
opts.signal.addEventListener('abort', onAbort, { once: true });
waiters.set(streamId, w);
});
},
async destroy() {
if (destroyed) return;
destroyed = true;
unsubscribe();
for (const w of waiters.values()) {
cleanupWaiter(w);
w.reject(new Error('streams-bridge: destroyed'));
}
waiters.clear();
for (const p of parked.values()) {
try {
await p.handle.abort('bridge-destroyed');
} catch {
/* swallow */
}
}
parked.clear();
},
};
}

View File

@@ -0,0 +1,310 @@
/**
* Upload an entire local directory tree to a remote peer via a `FileClient`.
*
* Walks the local `DirectoryHandleLike` lazily, creates remote directories
* via `client.mkdir({ recursive: true })`, and uploads each file with
* `client.write` (which routes inline or streams based on size). A bounded
* concurrency pool keeps memory and inflight RPCs in check.
*
* Returns a `BulkTransferHandle` whose `events` stream emits `'plan'`,
* `'file-start'`, `'file-progress'` (currently emitted at file start +
* end), `'file-done'` / `'file-error'`, aggregate `'progress'`, and a
* final `'complete'` (or `'abort'`).
*/
import { posixJoin } from '../utils/path.js';
import { runWithConcurrency } from './concurrency.js';
import type { FileClient } from './client.js';
import {
DEFAULT_BULK_CONCURRENCY,
MAX_BULK_CONCURRENCY,
type BulkOpts,
type BulkTransferEvent,
type BulkTransferHandle,
type BulkTransferResult,
type DirectoryHandleLike,
type FileHandleLike,
} from './directory-types.js';
import { CancelledError } from '../schemas/errors.js';
interface PlannedFile {
/** Local file handle. */
handle: FileHandleLike;
/** Path relative to the upload root. */
relativePath: string;
/** Absolute remote path. */
remoteAbsPath: string;
}
interface PlannedDir {
remoteAbsPath: string;
}
export interface UploadDirectoryOptions extends BulkOpts {
/**
* Pre-create remote directories before uploading files. Default true.
* Disable if the server-side `write` already mkdir-p's parents.
*/
precreateDirs?: boolean;
}
export function uploadDirectory(
client: Pick<FileClient, 'write' | 'mkdir'>,
local: DirectoryHandleLike,
remoteRoot: string,
opts: UploadDirectoryOptions = {},
): BulkTransferHandle {
const concurrency = Math.max(
1,
Math.min(MAX_BULK_CONCURRENCY, opts.concurrency ?? DEFAULT_BULK_CONCURRENCY),
);
const precreateDirs = opts.precreateDirs ?? true;
const continueOnError = opts.continueOnError ?? false;
const externalSignal = opts.signal;
const internalAbort = new AbortController();
const combinedSignal = mergeSignals(externalSignal, internalAbort.signal);
const events: BulkTransferEvent[] = [];
const eventResolvers: ((v: IteratorResult<BulkTransferEvent>) => void)[] = [];
let eventsClosed = false;
function emit(event: BulkTransferEvent): void {
if (eventsClosed) return;
if (eventResolvers.length > 0) {
eventResolvers.shift()!({ value: event, done: false });
} else {
events.push(event);
}
}
function closeEvents(): void {
if (eventsClosed) return;
eventsClosed = true;
while (eventResolvers.length > 0) {
eventResolvers.shift()!({ value: undefined as never, done: true });
}
}
let resolveDone!: (r: BulkTransferResult) => void;
let rejectDone!: (err: unknown) => void;
const donePromise = new Promise<BulkTransferResult>((resolve, reject) => {
resolveDone = resolve;
rejectDone = reject;
});
const startedAt = Date.now();
let filesDone = 0;
let bytesDone = 0;
let bytesTotal = 0;
// Run the upload pipeline asynchronously.
void (async () => {
try {
// 1. Plan: walk local tree, collect dirs + files
const plannedDirs: PlannedDir[] = [];
const plannedFiles: PlannedFile[] = [];
try {
await collect(local, '', remoteRoot, plannedDirs, plannedFiles, combinedSignal);
} catch (err) {
emit({ type: 'abort', reason: errMsg(err) });
closeEvents();
rejectDone(err);
return;
}
const totalFiles = plannedFiles.length;
bytesTotal = plannedFiles.reduce(
(acc, f) => acc + (f.handle as FileHandleLike & { _size?: number })._size!,
0,
);
// Note: bytesTotal is computed lazily in collect() via cached sizes.
emit({ type: 'plan', totalFiles, totalBytes: bytesTotal });
if (combinedSignal.aborted) {
emit({ type: 'abort', reason: errMsg(combinedSignal.reason) });
closeEvents();
rejectDone(combinedSignal.reason ?? new CancelledError());
return;
}
// 2. Pre-create remote directories sequentially (cheap, avoids races
// when many uploads target the same parent).
if (precreateDirs) {
for (const d of plannedDirs) {
if (combinedSignal.aborted) break;
try {
await client.mkdir(d.remoteAbsPath, { recursive: true });
} catch (err) {
if (!isAlreadyExistsError(err)) {
if (!continueOnError) {
emit({ type: 'abort', reason: errMsg(err) });
closeEvents();
rejectDone(err);
return;
}
}
}
}
}
// 3. Upload files with bounded concurrency
try {
await runWithConcurrency(
asyncIterableOf(plannedFiles),
async (planned) => {
if (combinedSignal.aborted) {
throw new CancelledError('upload aborted');
}
const file = await planned.handle.getFile();
const size = file.size;
emit({ type: 'file-start', path: planned.relativePath, size });
try {
const writeOpts: { contentType?: string; signal?: AbortSignal } = {};
if (file.type !== '') writeOpts.contentType = file.type;
writeOpts.signal = combinedSignal;
if (size === 0) {
// Edge case: empty file — write empty buffer.
await client.write(planned.remoteAbsPath, new Uint8Array(0), writeOpts);
} else {
await client.write(
planned.remoteAbsPath,
{ stream: file.stream(), size, ...(file.type !== '' ? { contentType: file.type } : {}) },
writeOpts,
);
}
filesDone++;
bytesDone += size;
emit({ type: 'file-done', path: planned.relativePath, bytesDone: size });
emit({
type: 'progress',
filesDone,
filesTotal: totalFiles,
bytesDone,
bytesTotal,
});
} catch (err) {
emit({ type: 'file-error', path: planned.relativePath, error: err });
throw err;
}
},
{
concurrency,
signal: combinedSignal,
continueOnError,
onError: () => {
/* already emitted as 'file-error' */
},
},
);
} catch (err) {
emit({ type: 'abort', reason: errMsg(err) });
closeEvents();
rejectDone(err);
return;
}
const durationMs = Date.now() - startedAt;
emit({ type: 'complete', filesDone, bytesDone, durationMs });
closeEvents();
resolveDone({ filesDone, bytesDone, durationMs });
} catch (err) {
// Belt-and-suspenders: any unhandled error reaches here.
closeEvents();
rejectDone(err);
}
})();
// Suppress unhandled-rejection until consumer awaits done().
donePromise.catch(() => {
/* deliberate */
});
return {
events: {
[Symbol.asyncIterator]() {
return {
next(): Promise<IteratorResult<BulkTransferEvent>> {
if (events.length > 0) {
return Promise.resolve({ value: events.shift()!, done: false });
}
if (eventsClosed) {
return Promise.resolve({ value: undefined as never, done: true });
}
return new Promise((resolve) => eventResolvers.push(resolve));
},
return(): Promise<IteratorResult<BulkTransferEvent>> {
// Caller broke out — stop iterating but keep the bulk going.
return Promise.resolve({ value: undefined as never, done: true });
},
};
},
},
async abort(reason) {
internalAbort.abort(new CancelledError(reason ?? 'manual abort'));
},
done: () => donePromise,
};
}
// ─── Helpers ─────────────────────────────────────────────────
async function collect(
dir: DirectoryHandleLike,
relPrefix: string,
remoteRoot: string,
plannedDirs: PlannedDir[],
plannedFiles: PlannedFile[],
signal: AbortSignal,
): Promise<void> {
for await (const [name, handle] of dir.entries()) {
if (signal.aborted) return;
const relPath = relPrefix === '' ? name : `${relPrefix}/${name}`;
if (handle.kind === 'directory') {
const remoteAbs = posixJoin(remoteRoot, relPath);
plannedDirs.push({ remoteAbsPath: remoteAbs });
await collect(handle as DirectoryHandleLike, relPath, remoteRoot, plannedDirs, plannedFiles, signal);
} else {
// Cache size on the handle so the planning total is accurate.
const file = await (handle as FileHandleLike).getFile();
(handle as FileHandleLike & { _size?: number })._size = file.size;
plannedFiles.push({
handle: handle as FileHandleLike,
relativePath: relPath,
remoteAbsPath: posixJoin(remoteRoot, relPath),
});
}
}
}
async function* asyncIterableOf<T>(arr: T[]): AsyncIterable<T> {
for (const item of arr) yield item;
}
function isAlreadyExistsError(err: unknown): boolean {
if (err === null || typeof err !== 'object') return false;
const code = (err as { code?: string; payload?: { code?: string } }).code;
if (code === 'SHADE_FS_CONFLICT') return true;
const payloadCode = (err as { payload?: { code?: string } }).payload?.code;
return payloadCode === 'CONFLICT';
}
function errMsg(err: unknown): string {
if (err instanceof Error) return err.message;
return String(err ?? 'unknown error');
}
function mergeSignals(
external: AbortSignal | undefined,
internal: AbortSignal,
): AbortSignal {
if (external === undefined) return internal;
if (external.aborted) return external;
const controller = new AbortController();
const onAbort = (sig: AbortSignal): void => {
controller.abort(sig.reason);
};
external.addEventListener('abort', () => onAbort(external), { once: true });
internal.addEventListener('abort', () => onAbort(internal), { once: true });
return controller.signal;
}

View File

@@ -0,0 +1,89 @@
/**
* Async-iterable directory walker.
*
* Depth-first by default — yields a directory's entries before descending
* into the next sibling. Memory-bounded: never materializes the whole tree;
* uses `client.list` page-by-page.
*
* Designed for arbitrarily-large remote trees: the consumer can break out
* of the iterator at any point and the walk halts cleanly.
*/
import { posixJoin } from '../utils/path.js';
import { CancelledError } from '../schemas/errors.js';
import type { FileEntry } from '../schemas/file-entry.js';
import type { FileClient } from './client.js';
export interface WalkOpts {
/** Hard cap on recursion depth. Default 32. */
maxDepth?: number;
/** Cancellation. */
signal?: AbortSignal;
/**
* Apply a per-entry filter. Return `false` to skip an entry (and, for
* directories, to skip descending into it). Default: include all.
*/
filter?: (entry: FileEntry, relativePath: string) => boolean;
/** Page size hint for `client.list`. Default 200. */
pageSize?: number;
}
export interface WalkItem {
entry: FileEntry;
/** Path relative to the walk root. Includes the entry's own name. */
relativePath: string;
/** Absolute path on the remote (root + relativePath). */
absolutePath: string;
/** Depth from root — 1 for direct children. */
depth: number;
}
const DEFAULT_MAX_DEPTH = 32;
const DEFAULT_PAGE_SIZE = 200;
export async function* walk(
client: Pick<FileClient, 'list'>,
rootPath: string,
opts: WalkOpts = {},
): AsyncIterable<WalkItem> {
const maxDepth = opts.maxDepth ?? DEFAULT_MAX_DEPTH;
const pageSize = opts.pageSize ?? DEFAULT_PAGE_SIZE;
function checkAbort(): void {
if (opts.signal?.aborted) {
throw new CancelledError(
opts.signal.reason instanceof Error
? opts.signal.reason.message
: 'walk aborted',
);
}
}
async function* descend(absDir: string, depth: number, relPrefix: string): AsyncIterable<WalkItem> {
if (depth > maxDepth) return;
let cursor: string | undefined;
do {
checkAbort();
const page = await client.list(absDir, {
pageSize,
...(cursor !== undefined ? { cursor } : {}),
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
});
for (const entry of page.entries) {
checkAbort();
const relPath = relPrefix === '' ? entry.name : `${relPrefix}/${entry.name}`;
const absPath = posixJoin(absDir, entry.name);
if (opts.filter !== undefined && !opts.filter(entry, relPath)) {
continue;
}
const item: WalkItem = { entry, relativePath: relPath, absolutePath: absPath, depth };
yield item;
if (entry.kind === 'dir') {
yield* descend(absPath, depth + 1, relPath);
}
}
cursor = page.hasMore ? page.nextCursor : undefined;
} while (cursor !== undefined);
}
yield* descend(rootPath, 1, '');
}

View File

@@ -0,0 +1,198 @@
// Schemas — Zod runtime + compile-time types
export * from './schemas/index.js';
// Protocol primitives — kinds, correlation, canonical bytes, envelope codec
export {
SHADE_FILES_VERSION,
KIND_PREFIX,
SUPPORTED_KINDS,
isSupportedKind,
} from './protocol/version.js';
export type { SupportedKind } from './protocol/version.js';
export {
KIND_LIST_V1,
KIND_STAT_V1,
KIND_MKDIR_V1,
KIND_DELETE_V1,
KIND_MOVE_V1,
KIND_READ_V1,
KIND_WRITE_V1,
KIND_GET_THUMBNAIL_V1,
KIND_CUSTOM_V1,
KIND_ERROR_V1,
KIND_CANCEL_V1,
responseKindOf,
opOfKind,
MUTATION_OPS,
} from './protocol/kinds.js';
export type { StandardOp } from './protocol/kinds.js';
export {
generateRequestId,
generateIdempotencyKey,
base64UrlEncode,
base64UrlDecode,
} from './protocol/correlate.js';
export {
canonicalRpcBytes,
canonicalJsonStringify,
hashArgs,
bytesToHex,
bytesToBase64,
base64ToBytes,
} from './protocol/canonical.js';
export {
encodeEnvelope,
looksLikeFileEnvelope,
tryParseEnvelope,
classify,
} from './protocol/envelope-codec.js';
export type { ClassifiedEnvelope } from './protocol/envelope-codec.js';
// RPC channel — wires Shade.send/onMessage to the file-RPC routing layer
export { ShadeFileRpcChannel } from './rpc/channel.js';
export type { RpcChannelHooks } from './rpc/channel.js';
export { PendingRpcRegistry } from './rpc/pending.js';
export type { RegisterOptions } from './rpc/pending.js';
// Server side
export { createFileHandler, INTERNAL_SYMBOL } from './server/handler.js';
export type {
FileHandler,
FileHandlerConfig,
FileHandlerOps,
} from './server/handler.js';
export type { OpContext, OpKind } from './server/handler-context.js';
export { validatePath } from './server/path-policy.js';
export type { PathPolicy, PathValidationResult } from './server/path-policy.js';
export { IdempotencyCache } from './server/idempotency-cache.js';
export type { IdempotencyCacheOptions } from './server/idempotency-cache.js';
export { RateLimiter } from './server/rate-limiter.js';
export type { RateLimitConfig } from './server/rate-limiter.js';
export { createCursorBuilder } from './server/cursor.js';
export type { CursorBuilder } from './server/cursor.js';
export {
createServerStreamsBridge,
META_KEY_READ_STREAM_ID,
META_KEY_WRITE_ID,
} from './server/streams-bridge.js';
export type {
ServerStreamsBridge,
StreamsBridgeShade,
CreateServerStreamsBridgeOptions,
ParkedWrite,
AwaitWriteOptions,
} from './server/streams-bridge.js';
export type {
UserReadResult,
UserReadResultInline,
UserReadResultStreams,
UserWriteArgs,
UserWriteContent,
UserWriteContentInline,
UserWriteContentStreams,
UserThumbnailResult,
} from './server/io-types.js';
export {
assertThumbnailFormat,
isThumbnailFormat,
} from './server/thumbnail.js';
export type { ThumbnailFormat } from './server/thumbnail.js';
// Client side
export { createFileClient } from './client/client.js';
export type {
FileClient,
BaseOpts,
CreateFileClientOptions,
ReadOpts,
WriteOpts,
ReadOutput,
ReadInlineOutput,
ReadStreamsOutput,
ThumbnailResult,
} from './client/client.js';
export {
createClientStreamsBridge,
} from './client/streams-bridge.js';
export type {
ClientStreamsBridge,
CreateClientStreamsBridgeOptions,
AwaitReadOptions,
ParkedRead,
} from './client/streams-bridge.js';
export {
decideInline,
INLINE_THRESHOLD,
} from './client/inline-threshold.js';
export type {
InlineDecision,
WriteSource,
} from './client/inline-threshold.js';
// Directory ops — walk + bulk transfers
export { walk } from './client/walk.js';
export type { WalkOpts, WalkItem } from './client/walk.js';
export { uploadDirectory } from './client/upload-directory.js';
export type { UploadDirectoryOptions } from './client/upload-directory.js';
export { downloadDirectory } from './client/download-directory.js';
export type { DownloadDirectoryOptions } from './client/download-directory.js';
export { runWithConcurrency } from './client/concurrency.js';
export type { ConcurrentMapOptions } from './client/concurrency.js';
export {
DEFAULT_BULK_CONCURRENCY,
MAX_BULK_CONCURRENCY,
} from './client/directory-types.js';
export type {
BulkOpts,
BulkTransferEvent,
BulkTransferHandle,
BulkTransferResult,
DirectoryHandleLike,
FileHandleLike,
FileLike,
WritableStreamLike,
} from './client/directory-types.js';
export { createMemoryDirectory } from './client/memory-directory.js';
// Custom-op registry typing (declaration-merged by consumers)
export type {
CustomOpDef,
CustomOpsMap,
CustomOpRegistration,
CustomOpRegistrations,
} from './server/custom-ops.js';
// Production hooks: metrics
export {
METRIC_BYTES_IN,
METRIC_BYTES_OUT,
METRIC_FINGERPRINT_REJECT_TOTAL,
METRIC_IDEMPOTENCY_CONFLICT_TOTAL,
METRIC_IDEMPOTENCY_HIT_TOTAL,
METRIC_OP_DURATION_MS,
METRIC_OP_TOTAL,
METRIC_RATE_LIMIT_REJECT_TOTAL,
METRIC_SIGNATURE_REJECT_TOTAL,
NOOP_METRIC_SINK,
} from './server/metrics.js';
export type { MetricSink, MetricTags } from './server/metrics.js';
export { MAX_SIGNATURE_AGE_MS } from './server/handler.js';
// High-level SDK integration entrypoint
export { createFilesNamespace } from './integration/files-namespace.js';
export type { FilesNamespace } from './integration/files-namespace.js';
// Integration helpers — wire handler + pending registry onto a channel
export { attachFileHandler } from './integration/wire-server.js';
export { attachClientRouting } from './integration/wire-client.js';
// Path utilities
export {
posixNormalize,
posixJoin,
posixDirname,
posixBasename,
decodePercentEscapes,
isPathInside,
} from './utils/path.js';

View File

@@ -0,0 +1,142 @@
/**
* High-level `FilesNamespace` — the entry point that the SDK exposes via
* `Shade.files`. Memoizes the underlying `ShadeFileRpcChannel` and bridges
* so a single Shade can simultaneously serve files AND consume them from
* peers without paying the setup cost twice.
*/
import type { Shade } from '@shade/sdk';
import {
attachClientRouting,
attachFileHandler,
createClientStreamsBridge,
createFileClient,
createFileHandler,
createServerStreamsBridge,
PendingRpcRegistry,
ShadeFileRpcChannel,
type ClientStreamsBridge,
type CreateFileClientOptions,
type FileClient,
type FileHandler,
type FileHandlerConfig,
type ServerStreamsBridge,
} from '../index.js';
import { IdempotencyCache } from '../server/idempotency-cache.js';
export interface FilesNamespace {
/**
* Register a file handler. Throws if a handler is already attached on
* this Shade — only one server per Shade. The returned function detaches
* the handler and tears down its idempotency / retention timers.
*/
serve(handler: FileHandlerConfig): Promise<() => Promise<void>>;
/**
* Build a typed file client for `peer`. Multiple concurrent clients to
* different peers share the same channel + streams bridge.
*/
client(peer: string, opts?: Omit<CreateFileClientOptions, 'streamsBridge'>): Promise<FileClient>;
/** Tear down channel + bridges. After destroy(), serve()/client() throw. */
destroy(): Promise<void>;
}
interface NamespaceState {
channel: ShadeFileRpcChannel;
pending: PendingRpcRegistry;
serverBridge: ServerStreamsBridge | null;
clientBridge: ClientStreamsBridge | null;
serverHandler: FileHandler | null;
serverDetach: (() => void) | null;
clientDetach: (() => void) | null;
destroyed: boolean;
}
/**
* Construct a `FilesNamespace` bound to a Shade instance. The SDK's
* `Shade.files` getter calls this lazily and memoizes the result.
*/
export function createFilesNamespace(shade: Shade): FilesNamespace {
const state: NamespaceState = {
channel: new ShadeFileRpcChannel(shade),
pending: new PendingRpcRegistry(),
serverBridge: null,
clientBridge: null,
serverHandler: null,
serverDetach: null,
clientDetach: null,
destroyed: false,
};
function ensureAlive(): void {
if (state.destroyed) throw new Error('FilesNamespace: destroyed');
}
return {
async serve(handlerConfig) {
ensureAlive();
if (state.serverHandler !== null) {
throw new Error('FilesNamespace: a handler is already registered (one per Shade)');
}
// Lazy server-side streams bridge.
if (state.serverBridge === null) {
state.serverBridge = await createServerStreamsBridge(shade);
}
const handler = createFileHandler(shade, {
...handlerConfig,
streamsBridge: state.serverBridge,
});
const detach = attachFileHandler(state.channel, handler);
state.serverHandler = handler;
state.serverDetach = detach;
// Wire BackgroundHooks.onPruneFiles to the new handler's idempotency
// cache. Use the symbol-exposed internals (works because FileHandler
// attaches them via Object.assign).
const internals = (handler as unknown as { [k: symbol]: { idempotency: IdempotencyCache } })[
Symbol.for('@shade/files/internal')
];
const background = (shade as unknown as { background?: { setHook?: (n: string, f: () => void) => void } }).background;
if (background?.setHook !== undefined && internals !== undefined) {
background.setHook('onPruneFiles', () => {
internals.idempotency.prune();
});
}
return async () => {
if (state.serverDetach !== null) state.serverDetach();
state.serverHandler = null;
state.serverDetach = null;
if (background?.setHook !== undefined) {
background.setHook('onPruneFiles', undefined as unknown as () => void);
}
};
},
async client(peer, opts = {}) {
ensureAlive();
// Lazy client-side streams bridge.
if (state.clientBridge === null) {
state.clientBridge = await createClientStreamsBridge(shade);
}
// Attach response routing once.
if (state.clientDetach === null) {
state.clientDetach = attachClientRouting(state.channel, state.pending);
}
return createFileClient(shade, state.channel, state.pending, peer, {
...opts,
streamsBridge: state.clientBridge,
});
},
async destroy() {
if (state.destroyed) return;
state.destroyed = true;
if (state.serverDetach !== null) state.serverDetach();
if (state.clientDetach !== null) state.clientDetach();
if (state.serverHandler !== null) state.serverHandler.destroy();
if (state.serverBridge !== null) await state.serverBridge.destroy();
if (state.clientBridge !== null) await state.clientBridge.destroy();
state.channel.destroy();
state.pending.rejectAll(new Error('FilesNamespace destroyed'));
},
};
}

View File

@@ -0,0 +1,23 @@
import { ShadeFileRpcChannel } from '../rpc/channel.js';
import { PendingRpcRegistry } from '../rpc/pending.js';
/**
* Wire a `PendingRpcRegistry` onto a channel so incoming `response` and
* `error` envelopes resolve the matching client-side promises.
*
* The same channel can serve multiple clients (one per peer); the
* registry is shared because correlation IDs are globally unique.
*/
export function attachClientRouting(
channel: ShadeFileRpcChannel,
pending: PendingRpcRegistry,
): () => void {
const previous = channel.setHooks({
onResponse: (_from, env) => pending.resolveResponse(env.envelope),
onError: (_from, env) => pending.resolveError(env.envelope),
});
return () => {
pending.rejectAll(new Error('client routing detached'));
channel.setHooks(previous);
};
}

View File

@@ -0,0 +1,35 @@
import { ShadeFileRpcChannel } from '../rpc/channel.js';
import type { FileHandler } from '../server/handler.js';
/**
* Connect a `FileHandler` to a `ShadeFileRpcChannel`. Returns an unsubscribe
* function that detaches the handler. Throws if a handler is already wired
* on this channel.
*
* The wiring:
* - On `request` envelopes: invoke `handler.handleRequest`, then send the
* returned response/error envelope back to the requester.
* - On `cancel` envelopes: forward to `handler.handleCancel`.
*/
export function attachFileHandler(
channel: ShadeFileRpcChannel,
handler: FileHandler,
): () => void {
const previous = channel.setHooks({
onRequest: async (from, env) => {
const response = await handler.handleRequest(from, env.envelope);
try {
await channel.send(from, response);
} catch (err) {
console.error('[shade-files] failed to send response:', err);
}
},
onCancel: (from, env) => {
handler.handleCancel(from, env.envelope);
},
});
return () => {
handler.destroy();
channel.setHooks(previous);
};
}

View File

@@ -0,0 +1,72 @@
import { sha256 } from '@noble/hashes/sha2.js';
/**
* Canonical bytes for an `@shade/files` RPC request signature.
*
* "rpc\0addr=<sender>\0at=<signedAt>\0kind=<kind>\0id=<id>\0argsHash=<hex>\0"
*
* Mirror of `canonicalControlBytes` in `@shade/sdk/src/streams-bridge.ts:188`.
* The signature binds the (sender, op kind, request id, args, signedAt)
* tuple — defense-in-depth on top of Double Ratchet authentication.
*/
export function canonicalRpcBytes(args: {
address: string;
signedAt: number;
kind: string;
id: string;
argsHash: Uint8Array;
}): Uint8Array {
const enc = new TextEncoder();
const fields = [
`rpc\0`,
`addr=${args.address}\0`,
`at=${args.signedAt}\0`,
`kind=${args.kind}\0`,
`id=${args.id}\0`,
`argsHash=${bytesToHex(args.argsHash)}\0`,
];
return enc.encode(fields.join(''));
}
/**
* Stable canonical-JSON serialization for hashing args. Sorts object keys
* recursively so two equivalent inputs hash to identical bytes regardless
* of property order.
*/
export function canonicalJsonStringify(value: unknown): string {
if (value === null || typeof value !== 'object') return JSON.stringify(value);
if (Array.isArray(value)) {
return `[${value.map((v) => canonicalJsonStringify(v)).join(',')}]`;
}
const obj = value as Record<string, unknown>;
const keys = Object.keys(obj).sort();
const parts: string[] = [];
for (const key of keys) {
if (obj[key] === undefined) continue;
parts.push(`${JSON.stringify(key)}:${canonicalJsonStringify(obj[key])}`);
}
return `{${parts.join(',')}}`;
}
/** Hash args via canonical-JSON → SHA-256 (32 bytes). */
export function hashArgs(args: unknown): Uint8Array {
const enc = new TextEncoder();
return sha256(enc.encode(canonicalJsonStringify(args)));
}
export function bytesToHex(b: Uint8Array): string {
return Array.from(b, (x) => x.toString(16).padStart(2, '0')).join('');
}
export function bytesToBase64(b: Uint8Array): string {
let bin = '';
for (let i = 0; i < b.length; i++) bin += String.fromCharCode(b[i]!);
return btoa(bin);
}
export function base64ToBytes(s: string): Uint8Array {
const bin = atob(s);
const out = new Uint8Array(bin.length);
for (let i = 0; i < bin.length; i++) out[i] = bin.charCodeAt(i);
return out;
}

View File

@@ -0,0 +1,32 @@
/**
* Generate a correlation ID — 16 random bytes, base64url-encoded (22 chars
* after stripping padding). Same shape as `RequestIdSchema`.
*
* Uses `globalThis.crypto.getRandomValues` so it works in Bun, Node ≥19,
* and browsers without any platform-specific imports.
*/
export function generateRequestId(): string {
const buf = new Uint8Array(16);
globalThis.crypto.getRandomValues(buf);
return base64UrlEncode(buf);
}
/** Same as `generateRequestId` — distinct name for callsite clarity. */
export function generateIdempotencyKey(): string {
return generateRequestId();
}
export function base64UrlEncode(bytes: Uint8Array): string {
let bin = '';
for (let i = 0; i < bytes.length; i++) bin += String.fromCharCode(bytes[i]!);
return btoa(bin).replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, '');
}
export function base64UrlDecode(s: string): Uint8Array {
const padded = s.replace(/-/g, '+').replace(/_/g, '/');
const pad = padded.length % 4 === 0 ? '' : '='.repeat(4 - (padded.length % 4));
const bin = atob(padded + pad);
const out = new Uint8Array(bin.length);
for (let i = 0; i < bin.length; i++) out[i] = bin.charCodeAt(i);
return out;
}

View File

@@ -0,0 +1,70 @@
import { z } from 'zod';
import {
RpcCancelSchema,
RpcErrorSchema,
RpcRequestSchema,
RpcResponseSchema,
type RpcCancel,
type RpcEnvelope,
type RpcError,
type RpcRequest,
type RpcResponse,
} from '../schemas/envelope.js';
import { KIND_PREFIX } from './version.js';
/** Tagged classification of any incoming envelope. */
export type ClassifiedEnvelope =
| { kind: 'request'; envelope: RpcRequest }
| { kind: 'response'; envelope: RpcResponse }
| { kind: 'error'; envelope: RpcError }
| { kind: 'cancel'; envelope: RpcCancel };
const RpcAnySchema = z.union([
RpcRequestSchema,
RpcResponseSchema,
RpcErrorSchema,
RpcCancelSchema,
]);
/** Encode an envelope to JSON plaintext for `Shade.send`. */
export function encodeEnvelope(env: RpcEnvelope): string {
return JSON.stringify(env);
}
/**
* Quick-detection: does this plaintext look like an `@shade/files` envelope?
* Used by `ShadeFileRpcChannel` to skip non-files messages cheaply.
*/
export function looksLikeFileEnvelope(plaintext: string): boolean {
return plaintext.includes(KIND_PREFIX);
}
/**
* Parse and classify an incoming plaintext. Returns null on any malformed
* input — the channel ignores those silently (could be a different
* protocol on the same Shade.send pipe).
*/
export function tryParseEnvelope(plaintext: string): ClassifiedEnvelope | null {
let raw: unknown;
try {
raw = JSON.parse(plaintext);
} catch {
return null;
}
const result = RpcAnySchema.safeParse(raw);
if (!result.success) return null;
return classify(result.data);
}
export function classify(env: RpcEnvelope): ClassifiedEnvelope {
if (env.kind === 'shade.fs.cancel/v1') {
return { kind: 'cancel', envelope: env as RpcCancel };
}
if (env.kind === 'shade.fs.error/v1') {
return { kind: 'error', envelope: env as RpcError };
}
if (env.kind.endsWith('.response')) {
return { kind: 'response', envelope: env as RpcResponse };
}
return { kind: 'request', envelope: env as RpcRequest };
}

View File

@@ -0,0 +1,53 @@
/** Canonical kind names for v1 ops. */
export const KIND_LIST_V1 = 'shade.fs.list/v1' as const;
export const KIND_STAT_V1 = 'shade.fs.stat/v1' as const;
export const KIND_MKDIR_V1 = 'shade.fs.mkdir/v1' as const;
export const KIND_DELETE_V1 = 'shade.fs.delete/v1' as const;
export const KIND_MOVE_V1 = 'shade.fs.move/v1' as const;
export const KIND_READ_V1 = 'shade.fs.read/v1' as const;
export const KIND_WRITE_V1 = 'shade.fs.write/v1' as const;
export const KIND_GET_THUMBNAIL_V1 = 'shade.fs.getThumbnail/v1' as const;
export const KIND_CUSTOM_V1 = 'shade.fs.custom/v1' as const;
export const KIND_ERROR_V1 = 'shade.fs.error/v1' as const;
export const KIND_CANCEL_V1 = 'shade.fs.cancel/v1' as const;
/** Compute the response kind from a request kind: `'shade.fs.foo/v1'` → `'shade.fs.foo/v1.response'`. */
export function responseKindOf<K extends string>(kind: K): `${K}.response` {
return `${kind}.response`;
}
/** Standard op identifier — used in `OpContext.op`. */
export type StandardOp =
| 'list'
| 'stat'
| 'mkdir'
| 'delete'
| 'move'
| 'read'
| 'write'
| 'getThumbnail';
const KIND_TO_OP: Record<string, StandardOp> = {
[KIND_LIST_V1]: 'list',
[KIND_STAT_V1]: 'stat',
[KIND_MKDIR_V1]: 'mkdir',
[KIND_DELETE_V1]: 'delete',
[KIND_MOVE_V1]: 'move',
[KIND_READ_V1]: 'read',
[KIND_WRITE_V1]: 'write',
[KIND_GET_THUMBNAIL_V1]: 'getThumbnail',
};
export function opOfKind(kind: string): StandardOp | null {
return KIND_TO_OP[kind] ?? null;
}
/** Mutation ops require an idempotency key on retries. */
export const MUTATION_OPS = new Set<StandardOp | 'custom'>([
'mkdir',
'delete',
'move',
'write',
'custom',
]);

View File

@@ -0,0 +1,24 @@
/** Wire-protocol version. Bump when introducing breaking changes. */
export const SHADE_FILES_VERSION = '0.2.0';
/** Substring every `@shade/files` plaintext starts with — used by the channel filter. */
export const KIND_PREFIX = 'shade.fs';
/** Currently-supported op kinds. Extend when introducing new versions. */
export const SUPPORTED_KINDS = [
'shade.fs.list/v1',
'shade.fs.stat/v1',
'shade.fs.mkdir/v1',
'shade.fs.delete/v1',
'shade.fs.move/v1',
'shade.fs.read/v1',
'shade.fs.write/v1',
'shade.fs.getThumbnail/v1',
'shade.fs.custom/v1',
] as const;
export type SupportedKind = (typeof SUPPORTED_KINDS)[number];
const SUPPORTED_KIND_SET = new Set<string>(SUPPORTED_KINDS);
export function isSupportedKind(kind: string): kind is SupportedKind {
return SUPPORTED_KIND_SET.has(kind);
}

View File

@@ -0,0 +1,35 @@
import React, { createContext, useContext, useMemo } from 'react';
import type { Shade } from '@shade/sdk';
import type { FilesNamespace } from '../integration/files-namespace.js';
export interface ShadeFilesContextValue {
shade: Shade;
files: FilesNamespace;
}
const ShadeFilesContext = createContext<ShadeFilesContextValue | null>(null);
export interface ShadeFilesProviderProps {
/** Initialized `Shade` instance. `files` namespace is read off it lazily. */
shade: Shade;
children: React.ReactNode;
}
/**
* Provider for `@shade/files` React hooks. Distinct from
* `<ShadeRuntimeProvider>` in `@shade/widgets` so file-RPC consumers
* don't pull in the widget tree.
*/
export function ShadeFilesProvider({ shade, children }: ShadeFilesProviderProps): React.ReactElement {
const value = useMemo<ShadeFilesContextValue>(() => ({ shade, files: shade.files }), [shade]);
return React.createElement(ShadeFilesContext.Provider, { value }, children);
}
/** Throws if no `<ShadeFilesProvider>` is mounted above. */
export function useShadeFilesContext(): ShadeFilesContextValue {
const ctx = useContext(ShadeFilesContext);
if (ctx === null) {
throw new Error('useShadeFilesContext: missing <ShadeFilesProvider>');
}
return ctx;
}

View File

@@ -0,0 +1,26 @@
/**
* React entry point for `@shade/files`.
*
* Note: hooks/components are tree-shakable. Importing from
* `@shade/files/react` (this entry) avoids pulling React into Bun-only
* server consumers that import from the root `@shade/files`.
*/
export { ShadeFilesProvider, useShadeFilesContext } from './ShadeFilesProvider.js';
export type { ShadeFilesContextValue, ShadeFilesProviderProps } from './ShadeFilesProvider.js';
export { useShadeFiles } from './useShadeFiles.js';
export type { UseShadeFilesOptions, UseShadeFilesResult } from './useShadeFiles.js';
export { useFileList } from './useFileList.js';
export type { UseFileListOptions, UseFileListResult } from './useFileList.js';
export {
useFileTransfer,
useFileUpload,
useFileDownload,
} from './useFileTransfer.js';
export type {
BulkTransferStatus,
UseFileTransferResult,
} from './useFileTransfer.js';

View File

@@ -0,0 +1,90 @@
import { useCallback, useEffect, useRef, useState } from 'react';
import type { FileEntry } from '../schemas/file-entry.js';
import { useShadeFiles, type UseShadeFilesOptions } from './useShadeFiles.js';
export interface UseFileListResult {
entries: FileEntry[];
isLoading: boolean;
error: unknown;
hasMore: boolean;
/** Reset and reload the first page. */
refresh(): void;
/** Append the next page (no-op when `hasMore === false`). */
loadMore(): Promise<void>;
}
export interface UseFileListOptions extends UseShadeFilesOptions {
pageSize?: number;
}
/**
* Paginated list of `path` on `peer`. Fetches the first page on mount,
* exposes `loadMore()` for subsequent pages.
*/
export function useFileList(
peer: string,
path: string,
opts: UseFileListOptions = {},
): UseFileListResult {
const filesOpts: UseShadeFilesOptions = {};
if (opts.probeIntervalMs !== undefined) filesOpts.probeIntervalMs = opts.probeIntervalMs;
const { fs } = useShadeFiles(peer, filesOpts);
const pageSize = opts.pageSize ?? 100;
const [entries, setEntries] = useState<FileEntry[]>([]);
const [cursor, setCursor] = useState<string | undefined>(undefined);
const [hasMore, setHasMore] = useState(false);
const [isLoading, setIsLoading] = useState(false);
const [error, setError] = useState<unknown>(null);
const [tick, setTick] = useState(0);
const inflightRef = useRef<AbortController | null>(null);
// Initial load + reload on path/peer/refresh.
useEffect(() => {
if (fs === null) return;
inflightRef.current?.abort();
const ctrl = new AbortController();
inflightRef.current = ctrl;
setIsLoading(true);
setError(null);
fs.list(path, { pageSize, signal: ctrl.signal })
.then((page) => {
if (ctrl.signal.aborted) return;
setEntries(page.entries);
setHasMore(page.hasMore);
setCursor(page.nextCursor);
})
.catch((err) => {
if (!ctrl.signal.aborted) setError(err);
})
.finally(() => {
if (!ctrl.signal.aborted) setIsLoading(false);
});
return () => {
ctrl.abort();
};
}, [fs, path, pageSize, tick]);
const loadMore = useCallback(async (): Promise<void> => {
if (fs === null || !hasMore || cursor === undefined) return;
setIsLoading(true);
try {
const page = await fs.list(path, { pageSize, cursor });
setEntries((prev) => [...prev, ...page.entries]);
setHasMore(page.hasMore);
setCursor(page.nextCursor);
} catch (err) {
setError(err);
} finally {
setIsLoading(false);
}
}, [fs, path, pageSize, cursor, hasMore]);
return {
entries,
isLoading,
error,
hasMore,
refresh: () => setTick((t) => t + 1),
loadMore,
};
}

View File

@@ -0,0 +1,103 @@
import { useCallback, useEffect, useRef, useState } from 'react';
import type { BulkTransferEvent, BulkTransferHandle, BulkTransferResult } from '../client/directory-types.js';
export interface BulkTransferStatus {
state: 'idle' | 'running' | 'done' | 'error' | 'aborted';
filesDone: number;
filesTotal: number;
bytesDone: number;
bytesTotal: number;
error: unknown;
result: BulkTransferResult | null;
/** Last event emitted by the underlying handle. */
lastEvent: BulkTransferEvent | null;
}
export interface UseFileTransferResult extends BulkTransferStatus {
start(handle: BulkTransferHandle): void;
abort(reason?: string): Promise<void>;
}
const INITIAL: BulkTransferStatus = {
state: 'idle',
filesDone: 0,
filesTotal: 0,
bytesDone: 0,
bytesTotal: 0,
error: null,
result: null,
lastEvent: null,
};
/**
* Generic React-state wrapper around a `BulkTransferHandle`. Apps wire the
* handle returned from `uploadDirectory()` / `downloadDirectory()` here to
* surface progress in their UI.
*
* `useFileUpload` and `useFileDownload` are thin presets — they call
* `start(...)` with the appropriate handle automatically.
*/
export function useFileTransfer(): UseFileTransferResult {
const [status, setStatus] = useState<BulkTransferStatus>(INITIAL);
const handleRef = useRef<BulkTransferHandle | null>(null);
const start = useCallback((handle: BulkTransferHandle): void => {
handleRef.current = handle;
setStatus({ ...INITIAL, state: 'running' });
void (async () => {
try {
for await (const ev of handle.events) {
setStatus((prev) => {
const next: BulkTransferStatus = { ...prev, lastEvent: ev };
if (ev.type === 'plan') {
next.filesTotal = ev.totalFiles;
next.bytesTotal = ev.totalBytes ?? 0;
} else if (ev.type === 'progress') {
next.filesDone = ev.filesDone;
next.bytesDone = ev.bytesDone;
next.bytesTotal = ev.bytesTotal ?? prev.bytesTotal;
next.filesTotal = ev.filesTotal;
} else if (ev.type === 'complete') {
next.state = 'done';
next.filesDone = ev.filesDone;
next.bytesDone = ev.bytesDone;
} else if (ev.type === 'abort') {
next.state = 'aborted';
next.error = new Error(ev.reason);
} else if (ev.type === 'file-error') {
next.error = ev.error;
}
return next;
});
}
const result = await handle.done();
setStatus((prev) => ({ ...prev, state: 'done', result }));
} catch (err) {
setStatus((prev) => ({ ...prev, state: 'error', error: err }));
}
})();
}, []);
const abort = useCallback(async (reason?: string): Promise<void> => {
if (handleRef.current === null) return;
await handleRef.current.abort(reason);
}, []);
// Cleanup on unmount: abort any in-flight transfer.
useEffect(() => {
return () => {
if (handleRef.current !== null) {
void handleRef.current.abort('unmount').catch(() => undefined);
}
};
}, []);
return { ...status, start, abort };
}
/**
* Preset: `useFileUpload` — semantically identical to `useFileTransfer`,
* named distinctly to mirror the `useFileDownload` pair.
*/
export const useFileUpload = useFileTransfer;
export const useFileDownload = useFileTransfer;

View File

@@ -0,0 +1,88 @@
import { useEffect, useRef, useState } from 'react';
import type { FileClient } from '../client/client.js';
import { useShadeFilesContext } from './ShadeFilesProvider.js';
export interface UseShadeFilesResult {
fs: FileClient | null;
isLoading: boolean;
error: unknown;
/** True after a successful `stat('/')` probe; false until proven online. */
isOnline: boolean;
/** Force a re-probe. */
refresh(): void;
}
export interface UseShadeFilesOptions {
/** Probe interval (ms) for `isOnline`. Default 30_000. Disable with 0. */
probeIntervalMs?: number;
}
/**
* Get a typed `FileClient` for `peer`. Memoizes per (shade, peer) so
* multiple components sharing the same peer reuse the same client.
*/
export function useShadeFiles(
peer: string,
opts: UseShadeFilesOptions = {},
): UseShadeFilesResult {
const { files } = useShadeFilesContext();
const [fs, setFs] = useState<FileClient | null>(null);
const [error, setError] = useState<unknown>(null);
const [isOnline, setIsOnline] = useState(false);
const [probeTick, setProbeTick] = useState(0);
const cancelledRef = useRef(false);
useEffect(() => {
cancelledRef.current = false;
let mounted = true;
setError(null);
setFs(null);
files
.client(peer)
.then((client) => {
if (!mounted) {
client.close();
return;
}
setFs(client);
})
.catch((err) => {
if (mounted) setError(err);
});
return () => {
mounted = false;
cancelledRef.current = true;
};
}, [files, peer]);
// Periodic probe: stat('/') as a cheap reachability check.
useEffect(() => {
if (fs === null) return undefined;
const intervalMs = opts.probeIntervalMs ?? 30_000;
let cancelled = false;
const probe = (): void => {
fs.stat('/')
.then(() => {
if (!cancelled) setIsOnline(true);
})
.catch(() => {
if (!cancelled) setIsOnline(false);
});
};
probe();
if (intervalMs <= 0) return () => undefined;
const timer = setInterval(probe, intervalMs);
return () => {
cancelled = true;
clearInterval(timer);
};
}, [fs, opts.probeIntervalMs, probeTick]);
return {
fs,
isLoading: fs === null && error === null,
error,
isOnline,
refresh: () => setProbeTick((t) => t + 1),
};
}

View File

@@ -0,0 +1,107 @@
import type { Shade } from '@shade/sdk';
import {
encodeEnvelope,
looksLikeFileEnvelope,
tryParseEnvelope,
type ClassifiedEnvelope,
} from '../protocol/envelope-codec.js';
import type { RpcEnvelope } from '../schemas/envelope.js';
/**
* Routing hooks called by the channel when it receives different envelope
* classes. M-Files-1 ships the skeleton; M-Files-2 wires up real handlers
* (server-side dispatcher, client-side response routing).
*/
export interface RpcChannelHooks {
onRequest?: (from: string, env: ClassifiedEnvelope & { kind: 'request' }) => Promise<void>;
onResponse?: (from: string, env: ClassifiedEnvelope & { kind: 'response' }) => void;
onError?: (from: string, env: ClassifiedEnvelope & { kind: 'error' }) => void;
onCancel?: (from: string, env: ClassifiedEnvelope & { kind: 'cancel' }) => void;
}
/**
* `ShadeFileRpcChannel` rides on top of `Shade.send`/`Shade.onMessage`
* and routes `shade.fs.*` JSON envelopes to registered hooks.
*
* Counterpart to `ShadeControlChannel` from `@shade/sdk` — but with
* request-response semantics instead of fire-and-forget.
*
* One channel per `Shade` instance; the SDK's `files` getter memoizes it.
* Both server and client roles share the same channel: the same Shade can
* both serve files and consume them from peers.
*/
export class ShadeFileRpcChannel {
private hooks: RpcChannelHooks = {};
private readonly unsubscribe: () => void;
private destroyed = false;
constructor(private readonly shade: Shade) {
this.unsubscribe = shade.onMessage(async (from, plaintext) => {
if (!looksLikeFileEnvelope(plaintext)) return;
const classified = tryParseEnvelope(plaintext);
if (classified === null) return;
await this.dispatch(from, classified);
});
}
/**
* Merge hooks into the current set. Returns the previous values for the
* keys being set so callers can restore them on cleanup. Other hook
* slots are left unchanged — the same channel can simultaneously serve
* (`onRequest`+`onCancel`) and consume (`onResponse`+`onError`).
*/
setHooks(hooks: RpcChannelHooks): RpcChannelHooks {
if (this.destroyed) throw new Error('ShadeFileRpcChannel: destroyed');
const prev: RpcChannelHooks = {};
const keys = ['onRequest', 'onResponse', 'onError', 'onCancel'] as const;
for (const key of keys) {
if (key in hooks) {
prev[key] = this.hooks[key] as never;
this.hooks[key] = hooks[key] as never;
}
}
return prev;
}
/**
* Send an envelope to a peer. Encrypts via `Shade.send` (Double Ratchet)
* then delivers the ratchet envelope to the peer's `transferRoute()`
* control endpoint via the SDK's configured envelope transport.
*/
async send(peerAddress: string, envelope: RpcEnvelope): Promise<void> {
if (this.destroyed) throw new Error('ShadeFileRpcChannel: destroyed');
const plaintext = encodeEnvelope(envelope);
const ratchetEnvelope = await this.shade.send(peerAddress, plaintext);
await this.shade.deliverControlEnvelope(peerAddress, ratchetEnvelope);
}
destroy(): void {
if (this.destroyed) return;
this.destroyed = true;
this.unsubscribe();
this.hooks = {};
}
private async dispatch(from: string, classified: ClassifiedEnvelope): Promise<void> {
try {
switch (classified.kind) {
case 'request':
if (this.hooks.onRequest !== undefined) {
await this.hooks.onRequest(from, classified);
}
return;
case 'response':
this.hooks.onResponse?.(from, classified);
return;
case 'error':
this.hooks.onError?.(from, classified);
return;
case 'cancel':
this.hooks.onCancel?.(from, classified);
return;
}
} catch (err) {
console.error('[ShadeFileRpcChannel] dispatch error:', err);
}
}
}

View File

@@ -0,0 +1,113 @@
import { CancelledError, OperationTimeoutError, fileErrorFromPayload } from '../schemas/errors.js';
import type { RpcError, RpcResponse } from '../schemas/envelope.js';
interface PendingEntry {
resolve: (result: unknown) => void;
reject: (err: unknown) => void;
timer: ReturnType<typeof setTimeout> | null;
abortListener: (() => void) | null;
signal?: AbortSignal | undefined;
}
export interface RegisterOptions {
timeoutMs?: number;
signal?: AbortSignal;
/** Called when client cancels (signal aborted or timeout); transport can ship an `RpcCancel`. */
onCancel?: (reason: 'timeout' | 'signal') => void;
}
/**
* Client-side tracker for in-flight RPC calls. Routes incoming
* `RpcResponse`/`RpcError` to the right pending promise via correlation id.
*/
export class PendingRpcRegistry {
private readonly entries = new Map<string, PendingEntry>();
/** Register a pending request. Returns a Promise resolved/rejected by routing. */
register<T = unknown>(requestId: string, opts: RegisterOptions = {}): Promise<T> {
const p = new Promise<T>((resolve, reject) => {
const cleanup = (): void => {
const e = this.entries.get(requestId);
if (e === undefined) return;
if (e.timer !== null) clearTimeout(e.timer);
if (e.abortListener !== null && e.signal !== undefined) {
e.signal.removeEventListener('abort', e.abortListener);
}
this.entries.delete(requestId);
};
const entry: PendingEntry = {
resolve: (v) => {
cleanup();
resolve(v as T);
},
reject: (err) => {
cleanup();
reject(err);
},
timer: null,
abortListener: null,
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
};
if (opts.timeoutMs !== undefined && opts.timeoutMs > 0) {
entry.timer = setTimeout(() => {
opts.onCancel?.('timeout');
entry.reject(new OperationTimeoutError());
}, opts.timeoutMs);
}
if (opts.signal !== undefined) {
if (opts.signal.aborted) {
// Defer microtask so caller's await sees the rejection.
queueMicrotask(() => {
opts.onCancel?.('signal');
entry.reject(new CancelledError());
});
} else {
const listener = (): void => {
opts.onCancel?.('signal');
entry.reject(new CancelledError());
};
entry.abortListener = listener;
opts.signal.addEventListener('abort', listener, { once: true });
}
}
this.entries.set(requestId, entry);
});
// Attach a no-op rejection handler so Bun's strict unhandled-rejection
// detection doesn't flag the registered promise BEFORE the caller's
// `await` attaches its handler. The caller's await still observes the
// rejection through its own handler chain.
p.catch(() => {
/* suppress unhandled-rejection warning */
});
return p;
}
/** Route an incoming response. */
resolveResponse(env: RpcResponse): void {
const entry = this.entries.get(env.id);
if (entry === undefined) return; // unknown id — possibly a stale duplicate
entry.resolve(env.result);
}
/** Route an incoming error envelope. */
resolveError(env: RpcError): void {
const entry = this.entries.get(env.id);
if (entry === undefined) return;
entry.reject(fileErrorFromPayload(env.error));
}
/** Reject all pending entries — for shutdown. */
rejectAll(reason: unknown): void {
for (const entry of this.entries.values()) entry.reject(reason);
this.entries.clear();
}
/** For tests / metrics. */
size(): number {
return this.entries.size;
}
}

View File

@@ -0,0 +1,46 @@
import { z } from 'zod';
import { FileErrorPayloadSchema } from './errors.js';
import { IdempotencyKeySchema, RequestIdSchema } from './primitives.js';
const KIND_REQUEST_PATTERN = /^shade\.fs\.[A-Za-z0-9.-]+\/v\d+$/;
const KIND_RESPONSE_PATTERN = /^shade\.fs\.[A-Za-z0-9.-]+\/v\d+\.response$/;
/**
* Request envelope. Carries op kind, correlation id, args (validated per
* op), idempotency key (mutations only), and an Ed25519 signature binding
* `(sender, kind, id, argsHash, signedAt)`.
*/
export const RpcRequestSchema = z.object({
kind: z.string().regex(KIND_REQUEST_PATTERN, 'invalid request kind'),
id: RequestIdSchema,
args: z.unknown(),
idempotencyKey: IdempotencyKeySchema.optional(),
attempt: z.number().int().positive().optional(),
deadlineMs: z.number().int().positive().optional(),
sig: z.string().min(1).max(128),
signedAt: z.number().int().positive(),
});
export type RpcRequest = z.infer<typeof RpcRequestSchema>;
export const RpcResponseSchema = z.object({
kind: z.string().regex(KIND_RESPONSE_PATTERN, 'invalid response kind'),
id: RequestIdSchema,
result: z.unknown(),
});
export type RpcResponse = z.infer<typeof RpcResponseSchema>;
export const RpcErrorSchema = z.object({
kind: z.literal('shade.fs.error/v1'),
id: RequestIdSchema,
error: FileErrorPayloadSchema,
});
export type RpcError = z.infer<typeof RpcErrorSchema>;
export const RpcCancelSchema = z.object({
kind: z.literal('shade.fs.cancel/v1'),
id: RequestIdSchema,
reason: z.string().max(512).optional(),
});
export type RpcCancel = z.infer<typeof RpcCancelSchema>;
export type RpcEnvelope = RpcRequest | RpcResponse | RpcError | RpcCancel;

View File

@@ -0,0 +1,228 @@
import { z } from 'zod';
import { ShadeError } from '@shade/core';
export const FileErrorCodeSchema = z.enum([
'NOT_FOUND',
'PERMISSION_DENIED',
'CONFLICT',
'QUOTA_EXCEEDED',
'RATE_LIMIT',
'PATH_VALIDATION',
'FINGERPRINT_REQUIRED',
'OPERATION_TIMEOUT',
'IDEMPOTENCY_CONFLICT',
'CANCELLED',
'INTERNAL',
'NOT_IMPLEMENTED',
'CUSTOM_OP_REJECTED',
'INVALID_SIGNATURE',
'INVALID_ARGS',
]);
export type FileErrorCode = z.infer<typeof FileErrorCodeSchema>;
export const FileErrorPayloadSchema = z.object({
code: FileErrorCodeSchema,
message: z.string().max(2048),
/** Suggested retry delay in ms; only set for retriable errors (rate limit, transient). */
retryAfterMs: z.number().int().nonnegative().optional(),
/** Path or arg field the error refers to. */
field: z.string().max(128).optional(),
/** Optional cause string (sanitized — no stack traces leak from server). */
cause: z.string().max(2048).optional(),
});
export type FileErrorPayload = z.infer<typeof FileErrorPayloadSchema>;
// ─── Class hierarchy ─────────────────────────────────────────
/**
* Base class for all `@shade/files` errors. Extends `ShadeError` so the
* existing `errorToHttpStatus()` mapping in `@shade/core` continues to apply.
*/
export class FileError extends ShadeError {
readonly payload: FileErrorPayload;
constructor(payload: FileErrorPayload) {
super(`SHADE_FS_${payload.code}`, payload.message);
this.name = 'FileError';
this.payload = payload;
}
override toJSON(): { name: string; code: string; message: string; payload: FileErrorPayload } {
return {
name: this.name,
code: this.code,
message: this.message,
payload: this.payload,
};
}
}
export class NotFoundError extends FileError {
constructor(message = 'Not found', field?: string) {
super({ code: 'NOT_FOUND', message, ...(field !== undefined ? { field } : {}) });
this.name = 'NotFoundError';
}
}
export class PermissionDeniedError extends FileError {
constructor(message = 'Permission denied') {
super({ code: 'PERMISSION_DENIED', message });
this.name = 'PermissionDeniedError';
}
}
export class ConflictError extends FileError {
constructor(message: string) {
super({ code: 'CONFLICT', message });
this.name = 'ConflictError';
}
}
export class QuotaExceededError extends FileError {
constructor(message = 'Quota exceeded', retryAfterMs?: number) {
super({
code: 'QUOTA_EXCEEDED',
message,
...(retryAfterMs !== undefined ? { retryAfterMs } : {}),
});
this.name = 'QuotaExceededError';
}
}
export class FsRateLimitError extends FileError {
constructor(message = 'Rate limit exceeded', retryAfterMs?: number) {
super({
code: 'RATE_LIMIT',
message,
...(retryAfterMs !== undefined ? { retryAfterMs } : {}),
});
this.name = 'FsRateLimitError';
}
}
export class PathValidationError extends FileError {
constructor(message: string, field = 'path') {
super({ code: 'PATH_VALIDATION', message, field });
this.name = 'PathValidationError';
}
}
export class FingerprintRequiredError extends FileError {
constructor(message = 'Peer fingerprint must be verified before this operation') {
super({ code: 'FINGERPRINT_REQUIRED', message });
this.name = 'FingerprintRequiredError';
}
}
export class OperationTimeoutError extends FileError {
constructor(message = 'Operation timed out') {
super({ code: 'OPERATION_TIMEOUT', message });
this.name = 'OperationTimeoutError';
}
}
export class IdempotencyConflictError extends FileError {
constructor(
message = 'Idempotency key reused with different arguments',
) {
super({ code: 'IDEMPOTENCY_CONFLICT', message });
this.name = 'IdempotencyConflictError';
}
}
export class CancelledError extends FileError {
constructor(message = 'Cancelled') {
super({ code: 'CANCELLED', message });
this.name = 'CancelledError';
}
}
export class InternalFileError extends FileError {
constructor(message = 'Internal server error', cause?: string) {
super({
code: 'INTERNAL',
message,
...(cause !== undefined ? { cause } : {}),
});
this.name = 'InternalFileError';
}
}
export class NotImplementedError extends FileError {
constructor(op: string) {
super({ code: 'NOT_IMPLEMENTED', message: `Operation not implemented: ${op}` });
this.name = 'NotImplementedError';
}
}
export class CustomOpRejectedError extends FileError {
constructor(message: string) {
super({ code: 'CUSTOM_OP_REJECTED', message });
this.name = 'CustomOpRejectedError';
}
}
export class InvalidSignatureError extends FileError {
constructor(message = 'RPC envelope signature verification failed') {
super({ code: 'INVALID_SIGNATURE', message });
this.name = 'InvalidSignatureError';
}
}
export class InvalidArgsError extends FileError {
constructor(message: string, field?: string) {
super({ code: 'INVALID_ARGS', message, ...(field !== undefined ? { field } : {}) });
this.name = 'InvalidArgsError';
}
}
/**
* Reconstruct the right `FileError` subclass from a wire payload. Used by
* the client to surface typed errors from server responses.
*/
export function fileErrorFromPayload(payload: FileErrorPayload): FileError {
switch (payload.code) {
case 'NOT_FOUND':
return new NotFoundError(payload.message, payload.field);
case 'PERMISSION_DENIED':
return new PermissionDeniedError(payload.message);
case 'CONFLICT':
return new ConflictError(payload.message);
case 'QUOTA_EXCEEDED':
return new QuotaExceededError(payload.message, payload.retryAfterMs);
case 'RATE_LIMIT':
return new FsRateLimitError(payload.message, payload.retryAfterMs);
case 'PATH_VALIDATION':
return new PathValidationError(payload.message, payload.field);
case 'FINGERPRINT_REQUIRED':
return new FingerprintRequiredError(payload.message);
case 'OPERATION_TIMEOUT':
return new OperationTimeoutError(payload.message);
case 'IDEMPOTENCY_CONFLICT':
return new IdempotencyConflictError(payload.message);
case 'CANCELLED':
return new CancelledError(payload.message);
case 'INTERNAL':
return new InternalFileError(payload.message, payload.cause);
case 'NOT_IMPLEMENTED':
return new NotImplementedError(payload.message);
case 'CUSTOM_OP_REJECTED':
return new CustomOpRejectedError(payload.message);
case 'INVALID_SIGNATURE':
return new InvalidSignatureError(payload.message);
case 'INVALID_ARGS':
return new InvalidArgsError(payload.message, payload.field);
}
}
/** Convert any thrown value into a `FileErrorPayload` for wire serialization. */
export function payloadFromError(err: unknown): FileErrorPayload {
if (err instanceof FileError) return err.payload;
if (err instanceof Error) {
return {
code: 'INTERNAL',
message: err.message.slice(0, 2048),
};
}
return { code: 'INTERNAL', message: 'unknown error' };
}

View File

@@ -0,0 +1,33 @@
import { z } from 'zod';
import { CursorSchema } from './primitives.js';
export const FileKindSchema = z.enum(['file', 'dir']);
export type FileKind = z.infer<typeof FileKindSchema>;
/**
* Directory entry shape. App-specific extension via `metadata` — keep it
* lean; large blobs (thumbnails) belong in `getThumbnail`, not here.
*/
export const FileEntrySchema = z.object({
/** Base name only — no path separators. */
name: z.string().min(1).max(512).refine((s) => !s.includes('/') && !s.includes('\\'), {
message: 'name must not contain path separators',
}),
kind: FileKindSchema,
/** Plaintext byte size; 0 for directories. */
size: z.number().int().nonnegative(),
/** Last-modified time, unix milliseconds. */
mtime: z.number().int(),
/** RFC 6838 media type if known. */
contentType: z.string().max(255).optional(),
/** App-specific extension fields. Default empty. */
metadata: z.record(z.string(), z.unknown()).default({}),
});
export type FileEntry = z.infer<typeof FileEntrySchema>;
export const ListPageSchema = z.object({
entries: z.array(FileEntrySchema),
nextCursor: CursorSchema.optional(),
hasMore: z.boolean(),
});
export type ListPage = z.infer<typeof ListPageSchema>;

View File

@@ -0,0 +1,5 @@
export * from './primitives.js';
export * from './file-entry.js';
export * from './ops.js';
export * from './envelope.js';
export * from './errors.js';

View File

@@ -0,0 +1,157 @@
import { z } from 'zod';
import { Base64Schema, CursorSchema, PathSchema, Sha256HexSchema } from './primitives.js';
import { FileEntrySchema, FileKindSchema, ListPageSchema } from './file-entry.js';
// ─── list ────────────────────────────────────────────────────
export const ListArgsSchema = z.object({
path: PathSchema,
cursor: CursorSchema.optional(),
pageSize: z.number().int().min(1).max(1000).default(100),
filter: z
.object({
prefix: z.string().max(256).optional(),
kind: FileKindSchema.optional(),
})
.optional(),
});
export type ListArgs = z.infer<typeof ListArgsSchema>;
export const ListResultSchema = ListPageSchema;
export type ListResult = z.infer<typeof ListResultSchema>;
// ─── stat ────────────────────────────────────────────────────
export const StatArgsSchema = z.object({ path: PathSchema });
export type StatArgs = z.infer<typeof StatArgsSchema>;
export const StatResultSchema = FileEntrySchema;
export type StatResult = z.infer<typeof StatResultSchema>;
// ─── mkdir ───────────────────────────────────────────────────
export const MkdirArgsSchema = z.object({
path: PathSchema,
recursive: z.boolean().default(false),
});
export type MkdirArgs = z.infer<typeof MkdirArgsSchema>;
export const MkdirResultSchema = z.object({ entry: FileEntrySchema });
export type MkdirResult = z.infer<typeof MkdirResultSchema>;
// ─── delete ──────────────────────────────────────────────────
export const DeleteArgsSchema = z.object({
path: PathSchema,
recursive: z.boolean().default(false),
});
export type DeleteArgs = z.infer<typeof DeleteArgsSchema>;
export const DeleteResultSchema = z.object({
deletedCount: z.number().int().nonnegative(),
});
export type DeleteResult = z.infer<typeof DeleteResultSchema>;
// ─── move ────────────────────────────────────────────────────
export const MoveArgsSchema = z.object({
src: PathSchema,
dst: PathSchema,
overwrite: z.boolean().default(false),
});
export type MoveArgs = z.infer<typeof MoveArgsSchema>;
export const MoveResultSchema = z.object({ entry: FileEntrySchema });
export type MoveResult = z.infer<typeof MoveResultSchema>;
// ─── read ────────────────────────────────────────────────────
export const ReadArgsSchema = z.object({
path: PathSchema,
range: z
.object({
start: z.number().int().nonnegative(),
end: z.number().int().positive(),
})
.optional(),
/** Force inline path even if size unknown / large. Use with care. */
preferInline: z.boolean().optional(),
});
export type ReadArgs = z.infer<typeof ReadArgsSchema>;
export const ReadResultSchema = z.discriminatedUnion('kind', [
z.object({
kind: z.literal('inline'),
bytesB64: Base64Schema,
size: z.number().int().nonnegative(),
sha256: Sha256HexSchema,
contentType: z.string().optional(),
}),
z.object({
kind: z.literal('streams'),
streamId: z.string(),
size: z.number().int().nonnegative(),
sha256: Sha256HexSchema,
contentType: z.string().optional(),
}),
]);
export type ReadResult = z.infer<typeof ReadResultSchema>;
// ─── write ───────────────────────────────────────────────────
export const WriteArgsSchema = z.discriminatedUnion('kind', [
z.object({
kind: z.literal('inline'),
path: PathSchema,
bytesB64: Base64Schema,
contentType: z.string().optional(),
overwrite: z.boolean().default(false),
}),
z.object({
kind: z.literal('streams'),
path: PathSchema,
/** Declared plaintext size; server clamps quota against this. */
size: z.number().int().nonnegative(),
contentType: z.string().optional(),
overwrite: z.boolean().default(false),
/** base64url(16) — matches `metadata.shadeFilesWriteId` on the streams transfer. */
writeId: z.string(),
}),
]);
export type WriteArgs = z.infer<typeof WriteArgsSchema>;
export const WriteResultSchema = z.object({ entry: FileEntrySchema });
export type WriteResult = z.infer<typeof WriteResultSchema>;
// ─── getThumbnail ────────────────────────────────────────────
export const ThumbnailSizeSchema = z.union([
z.literal(64),
z.literal(128),
z.literal(256),
z.literal(512),
]);
export type ThumbnailSize = z.infer<typeof ThumbnailSizeSchema>;
export const GetThumbnailArgsSchema = z.object({
path: PathSchema,
size: ThumbnailSizeSchema,
format: z.enum(['png', 'webp', 'jpeg']).default('png'),
});
export type GetThumbnailArgs = z.infer<typeof GetThumbnailArgsSchema>;
export const GetThumbnailResultSchema = z.object({
bytesB64: Base64Schema,
format: z.enum(['png', 'webp', 'jpeg']),
width: z.number().int().positive().max(4096),
height: z.number().int().positive().max(4096),
sha256: Sha256HexSchema,
});
export type GetThumbnailResult = z.infer<typeof GetThumbnailResultSchema>;
// ─── custom ──────────────────────────────────────────────────
export const CustomArgsSchema = z.object({
name: z.string().min(1).max(128).regex(/^[a-z0-9.-]+$/i, 'invalid custom op name'),
payload: z.unknown(),
});
export type CustomArgs = z.infer<typeof CustomArgsSchema>;
export const CustomResultSchema = z.object({
result: z.unknown(),
});
export type CustomResult = z.infer<typeof CustomResultSchema>;

View File

@@ -0,0 +1,42 @@
import { z } from 'zod';
/**
* POSIX-style absolute path. Must start with `/`, max 4096 chars, no NULs,
* no CR/LF/DEL, no backslashes (Windows-style rejected). Further policy
* (root-scope, traversal hardening) applied by `validatePath` server-side.
*/
export const PathSchema = z
.string()
.min(1)
.max(4096)
.regex(/^\//, 'must be absolute (start with /)')
.refine((s) => !/[\x00-\x08\x0a-\x1f\x7f\\]/.test(s), {
message: 'contains forbidden control or backslash characters',
});
export type Path = z.infer<typeof PathSchema>;
/** Base64url(16 bytes) — used for both requestId and idempotencyKey. */
export const RequestIdSchema = z
.string()
.min(22)
.max(22)
.regex(/^[A-Za-z0-9_-]+$/, 'invalid base64url');
export type RequestId = z.infer<typeof RequestIdSchema>;
export const IdempotencyKeySchema = RequestIdSchema;
export type IdempotencyKey = z.infer<typeof IdempotencyKeySchema>;
/**
* Opaque cursor token. Server-defined shape; clients pass through verbatim.
* Capped at 2 KiB to prevent over-sized cursors leaking server state.
*/
export const CursorSchema = z.string().min(1).max(2048);
export type Cursor = z.infer<typeof CursorSchema>;
/** Hex-encoded SHA-256 (64 chars). */
export const Sha256HexSchema = z.string().length(64).regex(/^[0-9a-f]+$/);
/** Standard base64 (with padding) used for binary blobs in JSON envelopes. */
export const Base64Schema = z.string().regex(/^[A-Za-z0-9+/]*=*$/);

View File

@@ -0,0 +1,73 @@
import { sha256 } from '@noble/hashes/sha2.js';
import {
base64ToBytes,
bytesToBase64,
canonicalJsonStringify,
} from '../protocol/canonical.js';
/**
* Opaque pagination cursor. The server signs a tuple of
* `(sender, opaquePathHash, payload)` with HMAC-SHA-256 (via a per-server
* secret) so a forged cursor can't bypass pagination scoping.
*
* The payload itself is server-defined; in tests we use simple `{ offset }`.
*/
export interface CursorBuilder {
encode(sender: string, pathHash: string, payload: unknown): string;
decode(sender: string, pathHash: string, cursor: string): unknown | null;
}
export function createCursorBuilder(serverSecret: Uint8Array): CursorBuilder {
if (serverSecret.length < 16) {
throw new Error('serverSecret must be at least 16 bytes');
}
function mac(input: Uint8Array): Uint8Array {
// Simple HMAC-SHA-256 implementation using @noble/hashes — keyed
// construction over secret || input. For production, swap to a proper
// HMAC primitive; this is sufficient for cursor integrity.
const padded = new Uint8Array(serverSecret.length + input.length);
padded.set(serverSecret, 0);
padded.set(input, serverSecret.length);
return sha256(padded);
}
return {
encode(sender, pathHash, payload) {
const json = canonicalJsonStringify({ s: sender, p: pathHash, d: payload });
const enc = new TextEncoder().encode(json);
const tag = mac(enc).slice(0, 16); // 128-bit truncation is plenty
const out = new Uint8Array(enc.length + tag.length);
out.set(enc, 0);
out.set(tag, enc.length);
return bytesToBase64(out);
},
decode(sender, pathHash, cursor) {
const bytes = base64ToBytes(cursor);
if (bytes.length < 17) return null;
const enc = bytes.slice(0, bytes.length - 16);
const tag = bytes.slice(bytes.length - 16);
const expected = mac(enc).slice(0, 16);
if (!constantTimeEqualBytes(tag, expected)) return null;
try {
const parsed = JSON.parse(new TextDecoder().decode(enc)) as {
s: string;
p: string;
d: unknown;
};
if (parsed.s !== sender || parsed.p !== pathHash) return null;
return parsed.d;
} catch {
return null;
}
},
};
}
function constantTimeEqualBytes(a: Uint8Array, b: Uint8Array): boolean {
if (a.length !== b.length) return false;
let diff = 0;
for (let i = 0; i < a.length; i++) diff |= a[i]! ^ b[i]!;
return diff === 0;
}

View File

@@ -0,0 +1,86 @@
/**
* Custom op registry — typed via TypeScript declaration merging.
*
* Consumers register a custom op by:
* 1. Adding to `CustomOpsMap` via declaration merging (compile-time typing).
* 2. Passing a Zod-backed handler in `FileHandlerConfig.custom`.
*
* Server validates wire args against the registered Zod schema before
* invoking the user handler, then validates the user's response against
* the schema before shipping it.
*
* @example
* ```ts
* // In your app, once globally:
* declare module '@shade/files' {
* interface CustomOpsMap {
* 'app.deploy-mod': CustomOpDef<{ jarPath: string }, { deploymentId: string }>;
* }
* }
*
* // Server registers a handler:
* shade.files.serve({
* custom: {
* 'app.deploy-mod': {
* args: z.object({ jarPath: z.string() }),
* response: z.object({ deploymentId: z.string() }),
* handler: async (args, ctx) => ({ deploymentId: '...' }),
* },
* },
* });
*
* // Client gets typed I/O:
* const result = await fs.custom('app.deploy-mod', { jarPath: '...' });
* // ^? { deploymentId: string }
* ```
*/
import type { ZodType } from 'zod';
import type { OpContext } from './handler-context.js';
// ─── Public typing scaffold (declaration-merged by consumers) ─
/** Marker shape for one custom op entry — consumers extend `CustomOpsMap`. */
export interface CustomOpDef<A = unknown, R = unknown> {
args: A;
response: R;
}
/**
* Registry of all custom ops known at compile time. Empty by default —
* consumers extend via TypeScript declaration merging:
*
* ```ts
* declare module '@shade/files' {
* interface CustomOpsMap {
* 'app.foo': CustomOpDef<{ x: number }, { y: string }>;
* }
* }
* ```
*/
// eslint-disable-next-line @typescript-eslint/no-empty-object-type
export interface CustomOpsMap {}
// ─── Server-side handler config ──────────────────────────────
/**
* Server-side definition for one custom op: the args schema (validated
* inbound), the response schema (validated outbound), and the handler.
*/
export interface CustomOpRegistration<TArgs = unknown, TResponse = unknown> {
args: ZodType<TArgs>;
response: ZodType<TResponse>;
handler: (args: TArgs, ctx: OpContext<{ name: string; payload: TArgs }>) => Promise<TResponse>;
/**
* Optional cost override for the rate limiter. Default: same as
* `opCost.custom` (1 unless overridden in `RateLimitConfig`).
*/
cost?: number;
}
/**
* Map from custom-op name to registration. The keys are the same names
* declared in `CustomOpsMap`. Required typing via declaration merging is
* not strict here — runtime validation is enforced by the registered Zod
* schemas, so consumers using untyped names still get safety.
*/
export type CustomOpRegistrations = Record<string, CustomOpRegistration<any, any>>;

View File

@@ -0,0 +1,59 @@
import type { Shade } from '@shade/sdk';
import type { StandardOp } from '../protocol/kinds.js';
export type OpKind = StandardOp | `custom:${string}`;
/**
* Per-call context handed to every server-side handler/middleware.
*
* Path-related ops set `path` to the (validated, normalized) primary path.
* For `move`, `path` is `args.src`. For ops without a path (`custom`), the
* field is the empty string.
*/
export interface OpContext<TArgs = unknown> {
/** The op being executed. */
op: OpKind;
/** Validated + normalized primary path; empty string for path-less ops. */
path: string;
/** Zod-validated args for this op. */
args: TArgs;
/** Peer address (from Shade.receive). */
sender: string;
/** Aborted on client cancel or timeout. */
signal: AbortSignal;
/** Idempotency key (mutations only). */
idempotencyKey: string | undefined;
/** Attempt counter (1-based). */
attemptNumber: number;
/** Wall-clock when dispatch started (for metrics). */
startedAt: number;
/** Lazy fingerprint accessor — call only if needed. */
fingerprint: () => Promise<string>;
/** Mutable scratchpad for middleware (e.g. cache fingerprint hit). */
metadata: Map<string, unknown>;
}
/** Build an `OpContext`. Internal — used by the dispatcher. */
export function buildOpContext<TArgs>(args: {
op: OpKind;
path: string;
parsedArgs: TArgs;
sender: string;
signal: AbortSignal;
idempotencyKey: string | undefined;
attemptNumber: number;
shade: Shade;
}): OpContext<TArgs> {
return {
op: args.op,
path: args.path,
args: args.parsedArgs,
sender: args.sender,
signal: args.signal,
idempotencyKey: args.idempotencyKey,
attemptNumber: args.attemptNumber,
startedAt: Date.now(),
fingerprint: () => args.shade.getFingerprintFor(args.sender),
metadata: new Map(),
};
}

View File

@@ -0,0 +1,701 @@
import type { Shade } from '@shade/sdk';
import type { ZodTypeAny } from 'zod';
import {
MUTATION_OPS,
opOfKind,
responseKindOf,
type StandardOp,
} from '../protocol/kinds.js';
import {
DeleteArgsSchema,
DeleteResultSchema,
GetThumbnailArgsSchema,
GetThumbnailResultSchema,
ListArgsSchema,
ListResultSchema,
MkdirArgsSchema,
MkdirResultSchema,
MoveArgsSchema,
MoveResultSchema,
ReadArgsSchema,
ReadResultSchema,
StatArgsSchema,
StatResultSchema,
WriteArgsSchema,
WriteResultSchema,
type DeleteArgs,
type DeleteResult,
type GetThumbnailArgs,
type ListArgs,
type ListResult,
type MkdirArgs,
type MkdirResult,
type MoveArgs,
type MoveResult,
type ReadArgs,
type StatArgs,
type StatResult,
type WriteArgs,
type WriteResult,
} from '../schemas/ops.js';
import {
CancelledError,
CustomOpRejectedError,
FileError,
FingerprintRequiredError,
InvalidArgsError,
InvalidSignatureError,
NotImplementedError,
PathValidationError,
payloadFromError,
} from '../schemas/errors.js';
import type { RpcCancel, RpcError, RpcRequest, RpcResponse } from '../schemas/envelope.js';
import { buildOpContext, type OpContext } from './handler-context.js';
import { IdempotencyCache, type IdempotencyCacheOptions } from './idempotency-cache.js';
import { RateLimiter, type RateLimitConfig } from './rate-limiter.js';
import { validatePath, type PathPolicy } from './path-policy.js';
import {
adaptReadResult,
adaptThumbnailResult,
adaptWriteArgs,
} from './io-adapters.js';
import type {
UserReadResult,
UserThumbnailResult,
UserWriteArgs,
} from './io-types.js';
import type { ServerStreamsBridge } from './streams-bridge.js';
import type { CustomOpRegistrations } from './custom-ops.js';
import {
METRIC_BYTES_IN,
METRIC_BYTES_OUT,
METRIC_FINGERPRINT_REJECT_TOTAL,
METRIC_IDEMPOTENCY_CONFLICT_TOTAL,
METRIC_IDEMPOTENCY_HIT_TOTAL,
METRIC_OP_DURATION_MS,
METRIC_OP_TOTAL,
METRIC_RATE_LIMIT_REJECT_TOTAL,
METRIC_SIGNATURE_REJECT_TOTAL,
NOOP_METRIC_SINK,
type MetricSink,
} from './metrics.js';
import {
CustomArgsSchema,
CustomResultSchema,
type CustomArgs,
} from '../schemas/ops.js';
import { canonicalRpcBytes, hashArgs } from '../protocol/canonical.js';
/** Replay window for the `signedAt` field on inbound RPC envelopes. */
export const MAX_SIGNATURE_AGE_MS = 5 * 60 * 1000;
// ─── Public types ────────────────────────────────────────────
export interface FileHandlerOps {
list?: (ctx: OpContext<ListArgs>) => Promise<ListResult>;
stat?: (ctx: OpContext<StatArgs>) => Promise<StatResult>;
mkdir?: (ctx: OpContext<MkdirArgs>) => Promise<MkdirResult>;
delete?: (ctx: OpContext<DeleteArgs>) => Promise<DeleteResult>;
move?: (ctx: OpContext<MoveArgs>) => Promise<MoveResult>;
/**
* User-supplied read handler. Returns either an `inline` payload (≤ 256
* KiB) or a `streams` payload with a precomputed sha256. The dispatcher
* adapts to the wire shape.
*/
read?: (ctx: OpContext<ReadArgs>) => Promise<UserReadResult>;
/**
* User-supplied write handler. Receives `UserWriteArgs` with a clean
* `Uint8Array` (inline) or `ReadableStream` + sha256-promise (streams)
* shape — the dispatcher hides the base64 / writeId wire details.
*/
write?: (ctx: OpContext<UserWriteArgs>) => Promise<WriteResult>;
/**
* User-supplied thumbnail handler. Bytes are validated for format magic
* before they're shipped to prevent format misclassification attacks.
*/
getThumbnail?: (ctx: OpContext<GetThumbnailArgs>) => Promise<UserThumbnailResult>;
}
export interface FileHandlerConfig extends FileHandlerOps {
pathPolicy?: PathPolicy;
rateLimits?: RateLimitConfig;
idempotency?: IdempotencyCacheOptions;
/**
* Required for read/write `streams` ops. Coordinates the inbound/outbound
* `@shade/transfer` transfers via `userMetadata.shadeFiles*Id` keys.
*/
streamsBridge?: ServerStreamsBridge;
/** Custom ops registry — see `CustomOpsMap` declaration-merging. */
custom?: CustomOpRegistrations;
/**
* Verify the Ed25519 signature on inbound RPC envelopes. Pluggable so
* apps can plug their own peer-identity store. Returning `false` rejects
* with `InvalidSignatureError`. Default: skip (Double Ratchet AEAD on
* the underlying envelope already authenticates the sender).
*
* The `signedAt` replay-window check (±5 min) is enforced regardless.
*/
verifySender?: (
sender: string,
canonicalBytes: Uint8Array,
sig: string,
) => boolean | Promise<boolean>;
/**
* Per-op fingerprint-verification gate. Return `'required'` to demand
* the peer's fingerprint has been verified out-of-band (via
* `isFingerprintVerified`); `'reject'` to deny outright;
* `'optional'` (default) to allow.
*/
requireFingerprintVerifiedFor?: (
ctx: OpContext<unknown>,
) => 'required' | 'optional' | 'reject' | Promise<'required' | 'optional' | 'reject'>;
/** Lookup whether the consumer has out-of-band verified the peer. */
isFingerprintVerified?: (sender: string) => boolean | Promise<boolean>;
/** Vendor-neutral metrics sink. */
onMetric?: MetricSink;
/** Called BEFORE the handler runs. Throw to deny. */
beforeOp?: (ctx: OpContext<unknown>) => void | Promise<void>;
/** Called AFTER the handler returns. Result is the validated response. */
afterOp?: (ctx: OpContext<unknown>, result: unknown) => void | Promise<void>;
/** Called when an op fails. Receives the error and the context. */
onError?: (err: unknown, ctx: OpContext<unknown>) => void;
/** Default per-op timeout in ms. Default 60_000. */
defaultTimeoutMs?: number;
/** Hard deadline for streams-bridge awaits / outbound transfers. Default 60_000. */
ioTimeoutMs?: number;
}
export interface FileHandler {
/** Handle an incoming request envelope. Returns the envelope to send back. */
handleRequest(from: string, request: RpcRequest): Promise<RpcResponse | RpcError>;
/** Handle an incoming cancel envelope. */
handleCancel(from: string, cancel: RpcCancel): void;
/** Free up internal state (timers, abort listeners). */
destroy(): void;
}
interface OpSchemaPair {
args: ZodTypeAny;
result: ZodTypeAny;
}
const OP_SCHEMAS: Record<StandardOp, OpSchemaPair> = {
list: { args: ListArgsSchema, result: ListResultSchema },
stat: { args: StatArgsSchema, result: StatResultSchema },
mkdir: { args: MkdirArgsSchema, result: MkdirResultSchema },
delete: { args: DeleteArgsSchema, result: DeleteResultSchema },
move: { args: MoveArgsSchema, result: MoveResultSchema },
read: { args: ReadArgsSchema, result: ReadResultSchema },
write: { args: WriteArgsSchema, result: WriteResultSchema },
getThumbnail: { args: GetThumbnailArgsSchema, result: GetThumbnailResultSchema },
};
// ─── createFileHandler ───────────────────────────────────────
/**
* Build a `FileHandler` for the server side. The returned object is
* registered with `ShadeFileRpcChannel.setHooks({ onRequest })` (typically
* via `Shade.files.serve(...)` in the SDK).
*/
export function createFileHandler(
shade: Shade,
config: FileHandlerConfig,
): FileHandler {
const idempotency = new IdempotencyCache(config.idempotency);
const rateLimiter = new RateLimiter(config.rateLimits);
const inflightCancellers = new Map<string, AbortController>();
const defaultTimeoutMs = config.defaultTimeoutMs ?? 60_000;
const ioTimeoutMs = config.ioTimeoutMs ?? 60_000;
const metrics: MetricSink = config.onMetric ?? NOOP_METRIC_SINK;
const customRegistrations = config.custom ?? {};
const isCustomKind = (kind: string): boolean => kind === 'shade.fs.custom/v1';
async function handleRequest(
from: string,
request: RpcRequest,
): Promise<RpcResponse | RpcError> {
// 0. Replay-window check (independent of sig — defends against
// intercept-and-resend even when sig verification is disabled).
const skewMs = Math.abs(Date.now() - request.signedAt);
if (skewMs > MAX_SIGNATURE_AGE_MS) {
metrics(METRIC_SIGNATURE_REJECT_TOTAL, 1, { reason: 'skew' });
return makeErrorEnvelope(
request,
new InvalidSignatureError(
`signedAt is ${skewMs}ms outside the ±${MAX_SIGNATURE_AGE_MS}ms replay window`,
),
);
}
// 0b. Optional Ed25519 sig verification — pluggable. The canonical
// bytes bind sender + kind + id + signedAt + sha256(args).
if (config.verifySender !== undefined) {
const argsHashBytes = hashArgs(request.args);
const canonical = canonicalRpcBytes({
address: from,
signedAt: request.signedAt,
kind: request.kind,
id: request.id,
argsHash: argsHashBytes,
});
let ok = false;
try {
ok = await config.verifySender(from, canonical, request.sig);
} catch (err) {
metrics(METRIC_SIGNATURE_REJECT_TOTAL, 1, { reason: 'throw' });
return makeErrorEnvelope(request, err);
}
if (!ok) {
metrics(METRIC_SIGNATURE_REJECT_TOTAL, 1, { reason: 'mismatch' });
return makeErrorEnvelope(request, new InvalidSignatureError());
}
}
// 1. Resolve op + handler. Custom ops route through `config.custom`.
let op: StandardOp | 'custom';
let argSchema: ZodTypeAny;
let resultSchema: ZodTypeAny;
let customHandler: CustomOpRegistrations[string] | undefined;
if (isCustomKind(request.kind)) {
op = 'custom';
argSchema = CustomArgsSchema;
resultSchema = CustomResultSchema;
} else {
const std = opOfKind(request.kind);
if (std === null) {
return makeErrorEnvelope(request, new NotImplementedError(request.kind));
}
op = std;
argSchema = OP_SCHEMAS[std].args;
resultSchema = OP_SCHEMAS[std].result;
const handler = config[std];
if (handler === undefined) {
return makeErrorEnvelope(request, new NotImplementedError(std));
}
}
// 2. Args validation (wire shape)
const argParse = argSchema.safeParse(request.args);
if (!argParse.success) {
const issue = argParse.error.issues[0];
return makeErrorEnvelope(
request,
new InvalidArgsError(
issue?.message ?? 'invalid arguments',
issue?.path.join('.') || undefined,
),
);
}
const parsedArgs = argParse.data as unknown;
// 2b. Custom op resolution
let resolvedOpKind: string = op;
if (op === 'custom') {
const customArgs = parsedArgs as CustomArgs;
customHandler = customRegistrations[customArgs.name];
if (customHandler === undefined) {
return makeErrorEnvelope(
request,
new NotImplementedError(`custom:${customArgs.name}`),
);
}
resolvedOpKind = `custom:${customArgs.name}`;
// Validate inner payload against the custom op's args schema
const payloadParse = customHandler.args.safeParse(customArgs.payload);
if (!payloadParse.success) {
const issue = payloadParse.error.issues[0];
return makeErrorEnvelope(
request,
new InvalidArgsError(
issue?.message ?? 'invalid custom-op payload',
issue?.path.join('.') || undefined,
),
);
}
// Replace payload with validated value (Zod may apply defaults).
(parsedArgs as CustomArgs).payload = payloadParse.data;
}
// 3. Path validation (skip ops without a path)
let primaryPath = '';
if (op === 'move') {
primaryPath = (parsedArgs as MoveArgs).src;
} else if (op !== 'custom' && 'path' in (parsedArgs as object)) {
primaryPath = (parsedArgs as { path: string }).path;
}
let normalizedPath = primaryPath;
if (primaryPath !== '') {
const validated = validatePath(primaryPath, config.pathPolicy);
if (!validated.ok) {
return makeErrorEnvelope(request, new PathValidationError(validated.reason));
}
normalizedPath = validated.normalized;
if (op === 'move') {
const dstValid = validatePath((parsedArgs as MoveArgs).dst, config.pathPolicy);
if (!dstValid.ok) {
return makeErrorEnvelope(
request,
new PathValidationError(dstValid.reason, 'dst'),
);
}
(parsedArgs as MoveArgs).src = normalizedPath;
(parsedArgs as MoveArgs).dst = dstValid.normalized;
} else {
(parsedArgs as { path: string }).path = normalizedPath;
}
}
// 4. Rate-limit acquire
const opCostKey = op === 'custom' ? 'custom' : op;
const customCost = customHandler?.cost;
const estimatedBytes = estimateBytes(op === 'custom' ? 'custom' : op, parsedArgs);
try {
if (customCost !== undefined) {
// Custom op-specific cost: acquire that many op-tokens manually.
// Fall back to standard `custom` bucket cost for non-overridden.
rateLimiter.acquire(from, opCostKey, estimatedBytes);
// For overridden cost > 1, deduct extra tokens from the same bucket.
// The simplest route: re-acquire (cost - 1) more.
for (let i = 1; i < customCost; i++) {
rateLimiter.acquire(from, opCostKey, 0);
}
} else {
rateLimiter.acquire(from, opCostKey, estimatedBytes);
}
} catch (err) {
metrics(METRIC_RATE_LIMIT_REJECT_TOTAL, 1, { op: resolvedOpKind });
return makeErrorEnvelope(request, err);
}
// 5. Idempotency (mutations only)
const isMutation = MUTATION_OPS.has(opCostKey);
let commitIdem: ((response: unknown) => void) | null = null;
let abandonIdem: (() => void) | null = null;
if (isMutation && request.idempotencyKey !== undefined) {
try {
const begin = idempotency.begin(from, request.idempotencyKey, parsedArgs);
if (begin.status === 'replay') {
rateLimiter.release(from, opCostKey, estimatedBytes);
metrics(METRIC_IDEMPOTENCY_HIT_TOTAL, 1, { op: resolvedOpKind });
return makeResponseEnvelope(request, begin.response);
}
if (begin.status === 'wait') {
const cached = await begin.promise;
rateLimiter.release(from, opCostKey, estimatedBytes);
metrics(METRIC_IDEMPOTENCY_HIT_TOTAL, 1, { op: resolvedOpKind });
return makeResponseEnvelope(request, cached);
}
commitIdem = begin.commit;
abandonIdem = begin.abandon;
} catch (err) {
rateLimiter.release(from, opCostKey, estimatedBytes);
if (
err !== null &&
typeof err === 'object' &&
(err as { code?: string }).code === 'SHADE_FS_IDEMPOTENCY_CONFLICT'
) {
metrics(METRIC_IDEMPOTENCY_CONFLICT_TOTAL, 1, { op: resolvedOpKind });
}
return makeErrorEnvelope(request, err);
}
}
// 6. Build context + abort controller
const controller = new AbortController();
inflightCancellers.set(request.id, controller);
const ctx = buildOpContext({
op: op === 'custom' ? (resolvedOpKind as `custom:${string}`) : op,
path: normalizedPath,
parsedArgs,
sender: from,
signal: controller.signal,
idempotencyKey: request.idempotencyKey,
attemptNumber: request.attempt ?? 1,
shade,
});
// 7. Fingerprint gate
if (config.requireFingerprintVerifiedFor !== undefined) {
let gate: 'required' | 'optional' | 'reject';
try {
gate = await config.requireFingerprintVerifiedFor(ctx as OpContext<unknown>);
} catch (err) {
cleanup({ release: true });
return makeErrorEnvelope(request, err);
}
if (gate === 'reject') {
cleanup({ release: true });
metrics(METRIC_FINGERPRINT_REJECT_TOTAL, 1, { op: resolvedOpKind, gate: 'reject' });
return makeErrorEnvelope(
request,
new FingerprintRequiredError('operation rejected by fingerprint policy'),
);
}
if (gate === 'required') {
let verified = false;
try {
verified = config.isFingerprintVerified !== undefined
? Boolean(await config.isFingerprintVerified(from))
: false;
} catch (err) {
cleanup({ release: true });
return makeErrorEnvelope(request, err);
}
if (!verified) {
cleanup({ release: true });
metrics(METRIC_FINGERPRINT_REJECT_TOTAL, 1, { op: resolvedOpKind, gate: 'required' });
return makeErrorEnvelope(request, new FingerprintRequiredError());
}
}
}
// 8. beforeOp
try {
if (config.beforeOp !== undefined) {
await config.beforeOp(ctx as OpContext<unknown>);
}
} catch (err) {
cleanup({ release: true });
return makeErrorEnvelope(request, err);
}
// 9. Run handler with timeout race — adapting I/O ops as needed.
const timeoutMs = Math.min(
request.deadlineMs ?? defaultTimeoutMs,
defaultTimeoutMs,
);
let wireResult: unknown;
const startedAt = Date.now();
try {
wireResult = await runWithTimeout(
() => invokeOpHandler({
op,
stdHandler: op === 'custom' ? undefined : (config[op] as unknown),
customHandler,
ctx: ctx as OpContext<unknown>,
parsedArgs,
sender: from,
signal: controller.signal,
streamsBridge: config.streamsBridge,
ioTimeoutMs,
}),
timeoutMs,
controller,
);
} catch (err) {
const durationMs = Date.now() - startedAt;
metrics(METRIC_OP_DURATION_MS, durationMs, { op: resolvedOpKind, result: 'error' });
metrics(METRIC_OP_TOTAL, 1, { op: resolvedOpKind, result: 'error' });
cleanup({ release: true });
if (config.onError !== undefined) {
try {
config.onError(err, ctx as OpContext<unknown>);
} catch (hookErr) {
console.error('[FileHandler] onError hook threw:', hookErr);
}
}
return makeErrorEnvelope(request, err);
}
// 10. Custom-op response validation against the registered schema.
if (op === 'custom' && customHandler !== undefined) {
const innerParse = customHandler.response.safeParse(wireResult);
if (!innerParse.success) {
cleanup({ release: true });
return makeErrorEnvelope(
request,
new CustomOpRejectedError(
`custom-op response shape rejected by registered schema: ${innerParse.error.issues[0]?.message ?? 'unknown'}`,
),
);
}
wireResult = { result: innerParse.data };
}
// 11. Defensive response validation against the wire schema.
const resultParse = resultSchema.safeParse(wireResult);
if (!resultParse.success) {
cleanup({ release: true });
return makeErrorEnvelope(
request,
new InvalidArgsError(
`handler for ${resolvedOpKind} returned invalid response shape`,
),
);
}
// 12. afterOp
try {
if (config.afterOp !== undefined) {
await config.afterOp(ctx as OpContext<unknown>, resultParse.data);
}
} catch (err) {
cleanup({ release: true });
return makeErrorEnvelope(request, err);
}
// 13. Commit idempotency + emit metrics
commitIdem?.(resultParse.data);
cleanup({ release: false });
const durationMs = Date.now() - startedAt;
metrics(METRIC_OP_DURATION_MS, durationMs, { op: resolvedOpKind, result: 'ok' });
metrics(METRIC_OP_TOTAL, 1, { op: resolvedOpKind, result: 'ok' });
if (estimatedBytes > 0) {
// Inbound bytes (write) vs outbound (read) — both reuse the same
// pre-call `estimatedBytes`, since post-execution reconciliation
// would require deeper plumbing.
const direction = op === 'write' ? METRIC_BYTES_IN : op === 'read' ? METRIC_BYTES_OUT : null;
if (direction !== null) {
metrics(direction, estimatedBytes, { op: resolvedOpKind });
}
}
return makeResponseEnvelope(request, resultParse.data);
function cleanup(opts: { release: boolean }): void {
inflightCancellers.delete(request.id);
if (opts.release) {
abandonIdem?.();
rateLimiter.release(from, opCostKey, estimatedBytes);
}
}
}
function handleCancel(_from: string, cancel: RpcCancel): void {
const controller = inflightCancellers.get(cancel.id);
if (controller !== undefined) {
controller.abort(new CancelledError(cancel.reason ?? 'cancelled by sender'));
inflightCancellers.delete(cancel.id);
}
}
function destroy(): void {
for (const c of inflightCancellers.values()) {
c.abort(new CancelledError('handler destroyed'));
}
inflightCancellers.clear();
}
return Object.assign({ handleRequest, handleCancel, destroy }, {
[INTERNAL_SYMBOL]: { idempotency, rateLimiter },
});
}
export const INTERNAL_SYMBOL = Symbol.for('@shade/files/internal');
// ─── Op invoker (handles I/O adapters) ───────────────────────
interface InvokeArgs {
op: StandardOp | 'custom';
stdHandler: unknown;
customHandler: CustomOpRegistrations[string] | undefined;
ctx: OpContext<unknown>;
parsedArgs: unknown;
sender: string;
signal: AbortSignal;
streamsBridge: ServerStreamsBridge | undefined;
ioTimeoutMs: number;
}
async function invokeOpHandler(args: InvokeArgs): Promise<unknown> {
const { op, stdHandler, customHandler, ctx, parsedArgs, sender, signal, streamsBridge, ioTimeoutMs } = args;
const adapterDeps = { streamsBridge, sender, signal, ioTimeoutMs };
switch (op) {
case 'write': {
const wireArgs = parsedArgs as WriteArgs;
const { userArgs, awaitTransferDone } = await adaptWriteArgs(wireArgs, adapterDeps);
const userCtx = { ...ctx, args: userArgs } as OpContext<UserWriteArgs>;
const userResult = await (stdHandler as (c: OpContext<UserWriteArgs>) => Promise<WriteResult>)(userCtx);
await awaitTransferDone();
return userResult;
}
case 'read': {
const readArgs = parsedArgs as ReadArgs;
const userResult = await (stdHandler as (c: OpContext<ReadArgs>) => Promise<UserReadResult>)(ctx as OpContext<ReadArgs>);
return await adaptReadResult(userResult, readArgs, adapterDeps);
}
case 'getThumbnail': {
const userResult = await (stdHandler as (c: OpContext<GetThumbnailArgs>) => Promise<UserThumbnailResult>)(ctx as OpContext<GetThumbnailArgs>);
return adaptThumbnailResult(userResult);
}
case 'custom': {
if (customHandler === undefined) {
throw new NotImplementedError('custom op without registration');
}
const customArgs = parsedArgs as CustomArgs;
const innerCtx = { ...ctx, args: customArgs } as OpContext<{ name: string; payload: unknown }>;
// Pass the validated inner payload as the first arg, the OpContext as the second.
return await customHandler.handler(
customArgs.payload,
innerCtx as OpContext<{ name: string; payload: unknown }>,
);
}
default:
// Pass-through for list/stat/mkdir/delete/move.
return await (stdHandler as (c: OpContext<unknown>) => Promise<unknown>)(ctx);
}
}
// ─── Helpers ─────────────────────────────────────────────────
function makeResponseEnvelope(req: RpcRequest, result: unknown): RpcResponse {
return {
kind: responseKindOf(req.kind),
id: req.id,
result,
};
}
function makeErrorEnvelope(req: RpcRequest, err: unknown): RpcError {
return {
kind: 'shade.fs.error/v1',
id: req.id,
error: payloadFromError(err),
};
}
function estimateBytes(op: StandardOp | 'custom', args: unknown): number {
if (op === 'write') {
const w = args as { kind: 'inline' | 'streams'; bytesB64?: string; size?: number };
if (w.kind === 'inline' && typeof w.bytesB64 === 'string') {
return Math.floor((w.bytesB64.length * 3) / 4);
}
return w.size ?? 0;
}
if (op === 'read') {
const r = args as { range?: { start: number; end: number } };
if (r.range !== undefined) return r.range.end - r.range.start;
return 0;
}
return 0;
}
async function runWithTimeout<T>(
fn: () => Promise<T>,
timeoutMs: number,
controller: AbortController,
): Promise<T> {
let timer: ReturnType<typeof setTimeout> | null = null;
const timeout = new Promise<never>((_, reject) => {
timer = setTimeout(() => {
controller.abort(new Error('timeout'));
reject(new (FileError as unknown as { new (p: { code: string; message: string }): FileError })({
code: 'OPERATION_TIMEOUT',
message: `operation timed out after ${timeoutMs}ms`,
}));
}, timeoutMs);
});
try {
return await Promise.race([fn(), timeout]);
} finally {
if (timer !== null) clearTimeout(timer);
}
}

View File

@@ -0,0 +1,160 @@
import { hashArgs, bytesToHex } from '../protocol/canonical.js';
import { IdempotencyConflictError } from '../schemas/errors.js';
interface Entry {
argsHash: string;
/** `undefined` while inflight; resolved value once handler completes. */
status: 'inflight' | 'done';
promise?: Promise<unknown>;
response?: unknown;
insertedAt: number;
lastAccessAt: number;
}
export interface IdempotencyCacheOptions {
/** Time in ms after which an entry is eligible for eviction. Default 1h. */
ttlMs?: number;
/** Per-sender hard cap on cached entries; oldest-by-insert evicted first. */
maxEntriesPerSender?: number;
}
/**
* Per-sender LRU + TTL cache for idempotent RPC calls.
*
* - On lookup, returns cached response if `argsHash` matches the original
* call. Throws `IdempotencyConflictError` on argsHash mismatch.
* - On concurrent retry of the same key (still inflight), the second
* caller awaits the first's promise — no double-execution.
* - LRU eviction caps memory at `maxEntriesPerSender` per sender.
*
* Keyed on `(senderAddress, idempotencyKey)`. Internally a 2-level map.
*/
export class IdempotencyCache {
private readonly ttlMs: number;
private readonly maxPerSender: number;
private readonly bySender = new Map<string, Map<string, Entry>>();
constructor(opts: IdempotencyCacheOptions = {}) {
this.ttlMs = opts.ttlMs ?? 60 * 60 * 1000;
this.maxPerSender = opts.maxEntriesPerSender ?? 10_000;
}
/**
* Begin (or rejoin) an idempotent operation. Returns:
* - `{ status: 'fresh', commit }` — handler should run, then call `commit(response)`.
* - `{ status: 'wait', promise }` — concurrent retry; await the in-flight promise.
* - `{ status: 'replay', response }` — completed earlier; return cached.
*
* Mismatched argsHash throws `IdempotencyConflictError` immediately.
*/
begin(
sender: string,
key: string,
args: unknown,
):
| { status: 'fresh'; commit: (response: unknown) => void; abandon: () => void }
| { status: 'wait'; promise: Promise<unknown> }
| { status: 'replay'; response: unknown } {
const argsHash = bytesToHex(hashArgs(args));
const senderMap = this.getOrCreate(sender);
this.evictExpired(senderMap);
const existing = senderMap.get(key);
if (existing !== undefined) {
if (existing.argsHash !== argsHash) {
throw new IdempotencyConflictError();
}
existing.lastAccessAt = Date.now();
if (existing.status === 'done') {
return { status: 'replay', response: existing.response };
}
// Inflight — return the same promise so concurrent retries de-dupe.
return { status: 'wait', promise: existing.promise! };
}
let resolveInflight: (value: unknown) => void = () => {};
let rejectInflight: (err: unknown) => void = () => {};
const inflightPromise = new Promise<unknown>((resolve, reject) => {
resolveInflight = resolve;
rejectInflight = reject;
});
// Suppress unhandled-rejection warnings when no concurrent retry is
// awaiting this promise. Real waiters attach their own catch via
// `await begin().promise` and still see the rejection.
inflightPromise.catch(() => {
/* swallow — see comment above */
});
const entry: Entry = {
argsHash,
status: 'inflight',
promise: inflightPromise,
insertedAt: Date.now(),
lastAccessAt: Date.now(),
};
senderMap.set(key, entry);
this.evictOverflow(senderMap);
const commit = (response: unknown): void => {
entry.status = 'done';
entry.response = response;
entry.lastAccessAt = Date.now();
resolveInflight(response);
};
const abandon = (): void => {
// Failed/cancelled call — remove from cache so retries can proceed.
senderMap.delete(key);
rejectInflight(new Error('idempotency abandoned'));
};
return { status: 'fresh', commit, abandon };
}
/** Manual prune; called by the periodic retention job. */
prune(now: number = Date.now()): number {
let removed = 0;
for (const [sender, senderMap] of this.bySender) {
for (const [key, entry] of senderMap) {
if (now - entry.insertedAt > this.ttlMs) {
senderMap.delete(key);
removed++;
}
}
if (senderMap.size === 0) this.bySender.delete(sender);
}
return removed;
}
/** Total cached entries across all senders (mainly for tests/metrics). */
size(): number {
let total = 0;
for (const m of this.bySender.values()) total += m.size;
return total;
}
private getOrCreate(sender: string): Map<string, Entry> {
let m = this.bySender.get(sender);
if (m === undefined) {
m = new Map();
this.bySender.set(sender, m);
}
return m;
}
private evictExpired(senderMap: Map<string, Entry>): void {
const now = Date.now();
for (const [key, entry] of senderMap) {
if (now - entry.insertedAt > this.ttlMs) senderMap.delete(key);
}
}
private evictOverflow(senderMap: Map<string, Entry>): void {
if (senderMap.size <= this.maxPerSender) return;
// Map iteration order = insertion order; first key is oldest.
const overflow = senderMap.size - this.maxPerSender;
let i = 0;
for (const key of senderMap.keys()) {
if (i >= overflow) break;
senderMap.delete(key);
i++;
}
}
}

View File

@@ -0,0 +1,229 @@
/**
* Adapt between wire-shaped I/O ops and user-handler-shaped I/O ops.
*
* client → server (write): WriteArgs (wire) → UserWriteArgs (handler)
* server → client (read): UserReadResult (handler) → ReadResult (wire)
* server → client (thumb): UserThumbnailResult → GetThumbnailResult
*
* The streams-bridge is consulted whenever we're crossing the > 256 KiB
* boundary: outbound reads kick a new transfer; inbound writes await one.
*/
import { sha256 } from '@noble/hashes/sha2.js';
import type {
GetThumbnailResult,
ReadArgs,
ReadResult,
WriteArgs,
} from '../schemas/ops.js';
import { base64ToBytes, bytesToBase64, bytesToHex } from '../protocol/canonical.js';
import { INLINE_THRESHOLD } from '../client/inline-threshold.js';
import {
ConflictError,
InternalFileError,
InvalidArgsError,
} from '../schemas/errors.js';
import type {
UserReadResult,
UserThumbnailResult,
UserWriteArgs,
} from './io-types.js';
import type { ServerStreamsBridge } from './streams-bridge.js';
export interface IoAdapterDeps {
streamsBridge: ServerStreamsBridge | undefined;
sender: string;
signal: AbortSignal;
/** Hard deadline for streams-bridge awaits / outbound transfers. */
ioTimeoutMs: number;
}
// ─── write: wire args → user args ────────────────────────────
/**
* Convert validated `WriteArgs` (wire shape) into `UserWriteArgs`. For
* streams writes, this awaits the matching inbound transfer via the
* streams-bridge and synthesizes a `ReadableStream` for the user handler.
*/
export async function adaptWriteArgs(
wireArgs: WriteArgs,
deps: IoAdapterDeps,
): Promise<{
userArgs: UserWriteArgs;
/** Resolves once the inbound transfer (if any) has completed. */
awaitTransferDone: () => Promise<void>;
}> {
if (wireArgs.kind === 'inline') {
const bytes = base64ToBytes(wireArgs.bytesB64);
if (bytes.byteLength > INLINE_THRESHOLD) {
throw new InvalidArgsError(
`inline write exceeds ${INLINE_THRESHOLD}-byte threshold (got ${bytes.byteLength})`,
'bytesB64',
);
}
const hashHex = bytesToHex(sha256(bytes));
const userArgs: UserWriteArgs = {
path: wireArgs.path,
overwrite: wireArgs.overwrite,
...(wireArgs.contentType !== undefined ? { contentType: wireArgs.contentType } : {}),
content: {
kind: 'inline',
bytes,
size: bytes.byteLength,
sha256: hashHex,
},
};
return { userArgs, awaitTransferDone: async () => undefined };
}
// Streams write — must have a streams-bridge configured.
if (deps.streamsBridge === undefined) {
throw new InternalFileError(
'streams-bridge not configured: cannot accept streamed write',
);
}
const parked = await deps.streamsBridge.awaitWrite(wireArgs.writeId, {
expectedFrom: deps.sender,
signal: deps.signal,
timeoutMs: deps.ioTimeoutMs,
});
// sha256 from the inbound transfer's done() — user handler can `await`
// it immediately after draining the readable.
const sha256Promise = parked.done.then((r) => r.sha256);
sha256Promise.catch(() => {
/* swallow until consumer awaits */
});
// Cancellation: if the user handler aborts (or the dispatcher times out),
// also tear down the inbound transfer.
const onAbort = (): void => {
void parked.handle.abort('user-cancel').catch(() => undefined);
};
if (deps.signal.aborted) onAbort();
else deps.signal.addEventListener('abort', onAbort, { once: true });
// The dispatcher calls this AFTER the user handler resolves to ensure
// the transfer is fully drained (so any wire-level integrity errors
// surface here before we send the RPC ack to the client).
const awaitTransferDone = async (): Promise<void> => {
try {
await parked.done;
} finally {
deps.signal.removeEventListener('abort', onAbort);
}
};
const userArgs: UserWriteArgs = {
path: wireArgs.path,
overwrite: wireArgs.overwrite,
...(wireArgs.contentType !== undefined ? { contentType: wireArgs.contentType } : {}),
content: {
kind: 'streams',
stream: parked.readable,
size: wireArgs.size,
sha256: sha256Promise,
},
};
return { userArgs, awaitTransferDone };
}
// ─── read: user result → wire result ─────────────────────────
/**
* Convert a user-supplied `UserReadResult` into a wire `ReadResult`. For
* streams results, this kicks an outbound `@shade/transfer` and embeds the
* resulting `streamId` in the response so the client-side bridge can
* subscribe.
*/
export async function adaptReadResult(
userResult: UserReadResult,
args: ReadArgs,
deps: IoAdapterDeps,
): Promise<ReadResult> {
if (userResult.kind === 'inline') {
if (userResult.bytes.byteLength > INLINE_THRESHOLD) {
throw new InternalFileError(
`inline read result exceeds ${INLINE_THRESHOLD} bytes (got ${userResult.bytes.byteLength})`,
);
}
const sha256Hex = userResult.sha256 ?? bytesToHex(sha256(userResult.bytes));
return {
kind: 'inline',
bytesB64: bytesToBase64(userResult.bytes),
size: userResult.bytes.byteLength,
sha256: sha256Hex,
...(userResult.contentType !== undefined ? { contentType: userResult.contentType } : {}),
};
}
if (deps.streamsBridge === undefined) {
throw new InternalFileError(
'streams-bridge not configured: cannot ship streamed read result',
);
}
const { readStreamId } = await deps.streamsBridge.initiateRead({
peer: deps.sender,
stream: userResult.stream,
size: userResult.size,
...(userResult.contentType !== undefined ? { contentType: userResult.contentType } : {}),
name: args.path,
signal: deps.signal,
});
return {
kind: 'streams',
streamId: readStreamId,
size: userResult.size,
sha256: userResult.sha256,
...(userResult.contentType !== undefined ? { contentType: userResult.contentType } : {}),
};
}
// ─── getThumbnail: user result → wire result ─────────────────
const FORMAT_MAGIC: Record<string, Uint8Array> = {
png: new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
jpeg: new Uint8Array([0xff, 0xd8, 0xff]),
webp: new Uint8Array([0x52, 0x49, 0x46, 0x46]), // 'RIFF', then size, then 'WEBP'
};
const WEBP_HEAD = new Uint8Array([0x57, 0x45, 0x42, 0x50]); // 'WEBP' at offset 8
/**
* Validate a user thumbnail result for format magic-bytes match. The user
* MAY produce any image format internally, but this guards against
* misclassification (e.g. claiming PNG with JPEG bytes), which can confuse
* downstream rendering and security scanners.
*/
function checkFormatMagic(bytes: Uint8Array, format: 'png' | 'webp' | 'jpeg'): boolean {
const magic = FORMAT_MAGIC[format];
if (magic === undefined) return false;
if (bytes.byteLength < magic.byteLength) return false;
for (let i = 0; i < magic.byteLength; i++) {
if (bytes[i] !== magic[i]) return false;
}
if (format === 'webp') {
if (bytes.byteLength < 12) return false;
for (let i = 0; i < 4; i++) {
if (bytes[8 + i] !== WEBP_HEAD[i]) return false;
}
}
return true;
}
export function adaptThumbnailResult(
userResult: UserThumbnailResult,
): GetThumbnailResult {
if (!checkFormatMagic(userResult.bytes, userResult.format)) {
throw new ConflictError(
`thumbnail bytes do not match declared format=${userResult.format}`,
);
}
const hash = userResult.sha256 ?? bytesToHex(sha256(userResult.bytes));
return {
bytesB64: bytesToBase64(userResult.bytes),
format: userResult.format,
width: userResult.width,
height: userResult.height,
sha256: hash,
};
}

View File

@@ -0,0 +1,102 @@
/**
* Server-facing types for the I/O ops (`read`, `write`, `getThumbnail`).
*
* These differ from the wire schemas in `schemas/ops.ts` so user-supplied
* handlers see clean, typed values (`Uint8Array`, `ReadableStream`) instead
* of base64 strings and opaque transfer ids. The dispatcher in
* `server/handler.ts` adapts between the two.
*/
import type { ThumbnailSize, WriteResult } from '../schemas/ops.js';
// ─── read handler types ──────────────────────────────────────
/**
* Value returned by a user-supplied `read` handler.
*
* - `inline`: the bytes are returned in-band; the dispatcher base64-encodes
* them and computes the sha256 if the user did not provide one. Inline
* reads MUST be ≤ 256 KiB plaintext.
* - `streams`: the user provides a `ReadableStream`, the declared `size`,
* and a precomputed `sha256` (e.g. cached on the file's metadata row).
* The dispatcher initiates an outbound `@shade/transfer` transfer and
* echoes the transfer's `streamId` back to the client over RPC so the
* client-side bridge can subscribe.
*/
export type UserReadResult =
| UserReadResultInline
| UserReadResultStreams;
export interface UserReadResultInline {
kind: 'inline';
bytes: Uint8Array;
/** Optional — dispatcher computes if absent. */
sha256?: string;
contentType?: string;
}
export interface UserReadResultStreams {
kind: 'streams';
stream: ReadableStream<Uint8Array>;
size: number;
/** REQUIRED — the dispatcher cannot synchronously hash a stream. */
sha256: string;
contentType?: string;
}
// ─── write handler types ─────────────────────────────────────
/**
* Value passed to a user-supplied `write` handler.
*
* The dispatcher decodes the wire `WriteArgs` and (for `streams`) parks
* the inbound transfer via the streams-bridge, then synthesizes this
* cleaner shape for the user.
*/
export interface UserWriteArgs {
path: string;
contentType?: string;
overwrite: boolean;
content: UserWriteContent;
}
export type UserWriteContent =
| UserWriteContentInline
| UserWriteContentStreams;
export interface UserWriteContentInline {
kind: 'inline';
bytes: Uint8Array;
size: number;
/** sha256 of `bytes`, computed by the dispatcher. */
sha256: string;
}
export interface UserWriteContentStreams {
kind: 'streams';
/** Plaintext stream. Closes when the inbound `@shade/transfer` completes. */
stream: ReadableStream<Uint8Array>;
/** Declared plaintext size from the client's RPC args. */
size: number;
/**
* Promise that resolves with the transfer-verified sha256 once the entire
* stream has been received. Use this to attach a content hash to the
* user's storage record.
*/
sha256: Promise<string>;
}
// ─── getThumbnail handler types ──────────────────────────────
/** Value returned by a user-supplied `getThumbnail` handler. */
export interface UserThumbnailResult {
bytes: Uint8Array;
format: 'png' | 'webp' | 'jpeg';
width: number;
height: number;
/** Optional — dispatcher computes if absent. */
sha256?: string;
}
// ─── Re-exports ──────────────────────────────────────────────
export type { ThumbnailSize, WriteResult };

View File

@@ -0,0 +1,25 @@
/**
* Standard metric names emitted by the dispatcher.
*
* The user-supplied `onMetric(name, value, tags)` callback is vendor-neutral —
* pipe to Prometheus, OpenTelemetry, statsd, or just append to a log file.
*/
export const METRIC_OP_DURATION_MS = 'shade_files_op_duration_ms';
export const METRIC_OP_TOTAL = 'shade_files_op_total';
export const METRIC_BYTES_IN = 'shade_files_bytes_in';
export const METRIC_BYTES_OUT = 'shade_files_bytes_out';
export const METRIC_IDEMPOTENCY_HIT_TOTAL = 'shade_files_idempotency_hit_total';
export const METRIC_IDEMPOTENCY_CONFLICT_TOTAL = 'shade_files_idempotency_conflict_total';
export const METRIC_RATE_LIMIT_REJECT_TOTAL = 'shade_files_rate_limit_reject_total';
export const METRIC_FINGERPRINT_REJECT_TOTAL = 'shade_files_fingerprint_reject_total';
export const METRIC_SIGNATURE_REJECT_TOTAL = 'shade_files_signature_reject_total';
/** Tags attached to every metric event. */
export type MetricTags = Record<string, string | number | boolean>;
/** User-supplied metric sink. Synchronous to keep hot-path fast. */
export type MetricSink = (name: string, value: number, tags: MetricTags) => void;
/** No-op sink used when the user doesn't supply one. */
export const NOOP_METRIC_SINK: MetricSink = () => undefined;

View File

@@ -0,0 +1,95 @@
import {
decodePercentEscapes,
isPathInside,
posixNormalize,
} from '../utils/path.js';
export interface PathPolicy {
/** Chroot-style absolute root. All paths must lie inside this directory. */
rootScope?: string;
/** Reject `..` segments after normalization. Default true. */
forbidTraversal?: boolean;
/** Hard cap on raw-input length (post-decode). Default 4096. */
maxLength?: number;
/** Allow symlink components. Default false (consumer enforces realpath if true). */
allowSymlinks?: boolean;
/** App-specific extra check. Returns 'allow' | 'reject'. */
extra?: (normalizedPath: string) => 'allow' | 'reject';
}
export type PathValidationResult =
| { ok: true; normalized: string }
| { ok: false; reason: string };
const DEFAULT_MAX_LENGTH = 4096;
/** Bytes that must never appear in any path. */
const FORBIDDEN_RE = /[\x00-\x08\x0a-\x1f\x7f\\]/;
/**
* Validate and normalize a path against the configured policy.
*
* Performs (in order):
* 1. Length check on raw input.
* 2. Forbidden-bytes check on raw input (NUL/CR/LF/DEL/backslash).
* 3. Percent-decode (defense against `%2e%2e` smuggling).
* 4. Forbidden-bytes check on decoded form.
* 5. POSIX normalization.
* 6. `..` traversal rejection.
* 7. Root-scope boundary check.
* 8. Consumer-supplied `extra` predicate.
*
* Returns the normalized path on success. The handler MUST use the
* normalized form, not the raw input — the raw could trip TOCTOU on
* the caller's filesystem.
*/
export function validatePath(
rawPath: string,
policy: PathPolicy = {},
): PathValidationResult {
const maxLength = policy.maxLength ?? DEFAULT_MAX_LENGTH;
const forbidTraversal = policy.forbidTraversal ?? true;
if (typeof rawPath !== 'string') {
return { ok: false, reason: 'path must be a string' };
}
if (rawPath.length === 0) {
return { ok: false, reason: 'path is empty' };
}
if (rawPath.length > maxLength) {
return { ok: false, reason: `path exceeds ${maxLength} characters` };
}
if (FORBIDDEN_RE.test(rawPath)) {
return { ok: false, reason: 'path contains forbidden control or backslash characters' };
}
const decoded = decodePercentEscapes(rawPath);
if (FORBIDDEN_RE.test(decoded)) {
return { ok: false, reason: 'percent-decoded path contains forbidden characters' };
}
if (!decoded.startsWith('/')) {
return { ok: false, reason: 'path must be absolute' };
}
if (forbidTraversal) {
// Detect `..` segments BEFORE normalization (otherwise they'd be silently
// collapsed). Handles both raw and decoded forms.
if (/(^|\/)\.\.(\/|$)/.test(rawPath) || /(^|\/)\.\.(\/|$)/.test(decoded)) {
return { ok: false, reason: 'path contains `..` traversal segment' };
}
}
const normalized = posixNormalize(decoded);
if (policy.rootScope !== undefined) {
const root = posixNormalize(policy.rootScope);
if (!isPathInside(normalized, root)) {
return { ok: false, reason: 'path is outside root scope' };
}
}
if (policy.extra !== undefined && policy.extra(normalized) === 'reject') {
return { ok: false, reason: 'path rejected by policy.extra' };
}
return { ok: true, normalized };
}

View File

@@ -0,0 +1,157 @@
import { FsRateLimitError, QuotaExceededError } from '../schemas/errors.js';
export interface RateLimitConfig {
/** Cap on op-tokens per minute per sender. Default 600. */
maxOpsPerMinutePerSender?: number;
/** Cap on byte-tokens per hour per sender. Default 10 GiB. */
maxBytesPerHourPerSender?: number;
/** Per-op token cost. Default 1 per op except write=5, delete=3. */
opCost?: Partial<Record<string, number>> & { default?: number };
}
interface Bucket {
tokens: number;
capacity: number;
/** Tokens replenished per millisecond. */
refillRate: number;
lastRefill: number;
}
interface SenderBuckets {
ops: Bucket;
bytes: Bucket;
}
/**
* Two-bucket per-sender rate limiter. Each operation consumes:
* - `opCost(op)` from the ops bucket.
* - `estimatedBytes` from the bytes bucket (post-call reconciliation possible).
*
* Rejection throws either `FsRateLimitError` (op-bucket) or `QuotaExceededError`
* (bytes-bucket), each with a `retryAfterMs` hint.
*
* Cancellation MUST call `release()` to return reserved tokens.
*/
export class RateLimiter {
private readonly opsCapacity: number;
private readonly opsRefillPerMs: number;
private readonly bytesCapacity: number;
private readonly bytesRefillPerMs: number;
private readonly opCost: Required<NonNullable<RateLimitConfig['opCost']>>;
private readonly bySender = new Map<string, SenderBuckets>();
constructor(config: RateLimitConfig = {}) {
this.opsCapacity = config.maxOpsPerMinutePerSender ?? 600;
this.opsRefillPerMs = this.opsCapacity / 60_000;
this.bytesCapacity = config.maxBytesPerHourPerSender ?? 10 * 1024 * 1024 * 1024;
this.bytesRefillPerMs = this.bytesCapacity / 3_600_000;
this.opCost = {
list: config.opCost?.list ?? 1,
stat: config.opCost?.stat ?? 1,
mkdir: config.opCost?.mkdir ?? 2,
delete: config.opCost?.delete ?? 3,
move: config.opCost?.move ?? 2,
read: config.opCost?.read ?? 1,
write: config.opCost?.write ?? 5,
getThumbnail: config.opCost?.getThumbnail ?? 2,
custom: config.opCost?.custom ?? 1,
default: config.opCost?.default ?? 1,
} as Required<NonNullable<RateLimitConfig['opCost']>>;
}
/**
* Acquire tokens for an operation. Throws if either bucket is empty;
* otherwise atomically deducts from both. `estimatedBytes` may be 0 for
* non-I/O ops.
*/
acquire(sender: string, op: string, estimatedBytes: number = 0): void {
const buckets = this.getBuckets(sender);
this.refill(buckets, Date.now());
const cost = this.costFor(op);
if (buckets.ops.tokens < cost) {
const need = cost - buckets.ops.tokens;
const retryMs = Math.ceil(need / this.opsRefillPerMs);
throw new FsRateLimitError(`op rate limit exceeded`, retryMs);
}
if (estimatedBytes > 0 && buckets.bytes.tokens < estimatedBytes) {
const need = estimatedBytes - buckets.bytes.tokens;
const retryMs = Math.ceil(need / this.bytesRefillPerMs);
throw new QuotaExceededError(`byte quota exceeded`, retryMs);
}
buckets.ops.tokens -= cost;
if (estimatedBytes > 0) buckets.bytes.tokens -= estimatedBytes;
}
/**
* Reconcile after the actual byte cost is known. `actualBytes` may be
* higher or lower than the estimate; caps to capacity on release.
*/
reconcile(sender: string, estimatedBytes: number, actualBytes: number): void {
if (estimatedBytes === actualBytes) return;
const buckets = this.getBuckets(sender);
const delta = estimatedBytes - actualBytes;
if (delta > 0) {
buckets.bytes.tokens = Math.min(buckets.bytes.capacity, buckets.bytes.tokens + delta);
} else {
buckets.bytes.tokens = Math.max(0, buckets.bytes.tokens + delta);
}
}
/** Return tokens to both buckets (cancellation cleanup). */
release(sender: string, op: string, bytesReserved: number = 0): void {
const buckets = this.bySender.get(sender);
if (buckets === undefined) return;
const cost = this.costFor(op);
buckets.ops.tokens = Math.min(buckets.ops.capacity, buckets.ops.tokens + cost);
if (bytesReserved > 0) {
buckets.bytes.tokens = Math.min(buckets.bytes.capacity, buckets.bytes.tokens + bytesReserved);
}
}
/** Available token snapshot — for tests/metrics. */
snapshot(sender: string): { ops: number; bytes: number } | null {
const buckets = this.bySender.get(sender);
if (buckets === undefined) return null;
this.refill(buckets, Date.now());
return { ops: buckets.ops.tokens, bytes: buckets.bytes.tokens };
}
private costFor(op: string): number {
const explicit = (this.opCost as Record<string, number>)[op];
return explicit ?? this.opCost.default;
}
private getBuckets(sender: string): SenderBuckets {
let b = this.bySender.get(sender);
if (b !== undefined) return b;
const now = Date.now();
b = {
ops: {
tokens: this.opsCapacity,
capacity: this.opsCapacity,
refillRate: this.opsRefillPerMs,
lastRefill: now,
},
bytes: {
tokens: this.bytesCapacity,
capacity: this.bytesCapacity,
refillRate: this.bytesRefillPerMs,
lastRefill: now,
},
};
this.bySender.set(sender, b);
return b;
}
private refill(buckets: SenderBuckets, now: number): void {
const refillBucket = (b: Bucket): void => {
const elapsed = now - b.lastRefill;
if (elapsed <= 0) return;
b.tokens = Math.min(b.capacity, b.tokens + elapsed * b.refillRate);
b.lastRefill = now;
};
refillBucket(buckets.ops);
refillBucket(buckets.bytes);
}
}

View File

@@ -0,0 +1,289 @@
/**
* Server-side bridge between `@shade/files` content RPC ops and the
* `@shade/transfer` engine.
*
* Two responsibilities:
* 1. **Inbound writes (client → server, > 256 KiB).** When a client kicks
* `shade.upload(...)` with `userMetadata.shadeFilesWriteId = <id>`, this
* bridge intercepts the corresponding `IncomingTransfer` from
* `shade.onIncomingTransfer`, **immediately** calls `accept(...)` (the
* transfer engine requires synchronous accept — it rejects chunks that
* arrive before accept), pipes the plaintext into a TransformStream,
* and parks the readable side. The `write` RPC handler then awaits
* `awaitWrite(writeId, ...)` to pick up the readable, hand it to the
* user handler, and observe the transfer's done() result.
*
* 2. **Outbound reads (server → client, > 256 KiB).** When the user-supplied
* `read` handler returns `{ kind: 'streams', stream, size, sha256 }`,
* this bridge calls `shade.upload(...)` with
* `userMetadata.shadeFilesReadStreamId = <id>`. The id is then echoed
* back to the client in the `read` RPC response so the client-side
* bridge can subscribe to the matching incoming transfer.
*/
import type { IncomingTransfer, TransferHandle, TransferProgress } from '@shade/transfer';
import { generateRequestId } from '../protocol/correlate.js';
import { OperationTimeoutError } from '../schemas/errors.js';
/** Metadata key carried on inbound (client → server) write streams. */
export const META_KEY_WRITE_ID = 'shadeFilesWriteId';
/** Metadata key carried on outbound (server → client) read streams. */
export const META_KEY_READ_STREAM_ID = 'shadeFilesReadStreamId';
/**
* Subset of `Shade` we depend on. Lets us unit-test without a real Shade
* runtime and keeps the bridge framework-agnostic.
*/
export interface StreamsBridgeShade {
upload(opts: import('@shade/transfer').TransferOptions): Promise<TransferHandle>;
onIncomingTransfer(
handler: (incoming: IncomingTransfer) => void | Promise<void>,
): Promise<() => void>;
}
export interface ParkedWrite {
/** Sender address from the transfer engine. */
from: string;
/** Plaintext stream. The bridge already accepted the transfer; just consume. */
readable: ReadableStream<Uint8Array>;
/** Resolves when the transfer fully completes (sha256 available). */
done: Promise<{ sha256: string; bytesSent: number }>;
/** Underlying transfer handle — for abort propagation. */
handle: TransferHandle;
/** Unix epoch ms when the transfer arrived. Used for stale-eviction. */
arrivedAt: number;
}
export interface AwaitWriteOptions {
/** Sender address — must match `parked.from` for delivery. */
expectedFrom: string;
/** AbortSignal from the OpContext (cancellation/timeout). */
signal: AbortSignal;
/** Hard deadline (ms from now). Default 60_000. */
timeoutMs?: number;
}
export interface ServerStreamsBridge {
/**
* Wait for an inbound transfer carrying `userMetadata.shadeFilesWriteId
* === writeId` from `expectedFrom`. Resolves with the parked entry whose
* `readable` can be consumed by the user handler.
*
* If the transfer has already arrived (race: stream-init beat the RPC),
* resolves immediately. Otherwise awaits up to `timeoutMs`.
*/
awaitWrite(writeId: string, opts: AwaitWriteOptions): Promise<ParkedWrite>;
/**
* Initiate an outbound transfer to `peer` carrying the given stream.
* Stamps `userMetadata.shadeFilesReadStreamId = <streamId>`. Returns the
* new streamId so the caller can echo it back to the client over RPC.
*/
initiateRead(opts: {
peer: string;
stream: ReadableStream<Uint8Array>;
size: number;
contentType?: string;
name?: string;
signal?: AbortSignal;
onProgress?: (p: TransferProgress) => void;
}): Promise<{ readStreamId: string; handle: TransferHandle }>;
/** Tear down the incoming-transfer subscription and reject all pending awaits. */
destroy(): Promise<void>;
}
interface PendingWaiter {
resolve: (parked: ParkedWrite) => void;
reject: (err: unknown) => void;
expectedFrom: string;
timer: ReturnType<typeof setTimeout> | null;
abortListener: (() => void) | null;
signal: AbortSignal;
}
export interface CreateServerStreamsBridgeOptions {
/** Default deadline for `awaitWrite` if the caller doesn't supply one. */
defaultAwaitWriteTimeoutMs?: number;
/** How long to retain a parked transfer waiting for its RPC. Default 60_000. */
parkedWriteTtlMs?: number;
}
export async function createServerStreamsBridge(
shade: StreamsBridgeShade,
options: CreateServerStreamsBridgeOptions = {},
): Promise<ServerStreamsBridge> {
const parkedWriteTtlMs = options.parkedWriteTtlMs ?? 60_000;
const defaultAwaitTimeoutMs = options.defaultAwaitWriteTimeoutMs ?? 60_000;
const parked = new Map<string, ParkedWrite>();
const waiters = new Map<string, PendingWaiter>();
let destroyed = false;
const unsubscribeIncoming = await shade.onIncomingTransfer(async (incoming) => {
const writeId = incoming.metadata.userMetadata?.[META_KEY_WRITE_ID];
if (writeId === undefined) return; // not a files-bridge transfer; ignore
// Accept synchronously into a TransformStream so the engine can deliver
// chunks immediately. The readable side is parked for the matching RPC.
const ts = new TransformStream<Uint8Array, Uint8Array>();
let handle: TransferHandle;
try {
handle = await incoming.accept({
output: { kind: 'pipe', pipeTo: ts.writable },
});
} catch (err) {
// Accept failed — likely double-init or shutdown. Drop the parked
// entry; the awaiting RPC will time out.
console.error('[shade-files streams-bridge] accept failed:', err);
return;
}
const arrived: ParkedWrite = {
from: incoming.from,
readable: ts.readable,
done: handle.done().then((r) => ({ sha256: r.sha256, bytesSent: r.bytesSent })),
handle,
arrivedAt: Date.now(),
};
arrived.done.catch(() => {
/* swallow until consumer awaits */
});
const waiter = waiters.get(writeId);
if (waiter !== undefined) {
waiters.delete(writeId);
cleanupWaiter(waiter);
if (incoming.from !== waiter.expectedFrom) {
void handle.abort('sender-mismatch').catch(() => undefined);
waiter.reject(
new Error(
`streams-bridge: writeId=${writeId} delivered by ${incoming.from}, expected ${waiter.expectedFrom}`,
),
);
return;
}
waiter.resolve(arrived);
return;
}
// No waiter yet — park.
parked.set(writeId, arrived);
setTimeout(() => {
const stale = parked.get(writeId);
if (stale === arrived) {
parked.delete(writeId);
void handle.abort('rpc-timeout').catch(() => undefined);
}
}, parkedWriteTtlMs).unref?.();
});
function cleanupWaiter(w: PendingWaiter): void {
if (w.timer !== null) clearTimeout(w.timer);
if (w.abortListener !== null) {
w.signal.removeEventListener('abort', w.abortListener);
}
}
return {
async awaitWrite(writeId, opts) {
if (destroyed) throw new Error('streams-bridge: destroyed');
// 1. Already parked?
const ready = parked.get(writeId);
if (ready !== undefined) {
parked.delete(writeId);
if (ready.from !== opts.expectedFrom) {
void ready.handle.abort('sender-mismatch').catch(() => undefined);
throw new Error(
`streams-bridge: writeId=${writeId} delivered by ${ready.from}, expected ${opts.expectedFrom}`,
);
}
return ready;
}
if (waiters.has(writeId)) {
throw new Error(`streams-bridge: writeId=${writeId} already awaited`);
}
// 2. Park a waiter
const timeoutMs = opts.timeoutMs ?? defaultAwaitTimeoutMs;
return await new Promise<ParkedWrite>((resolve, reject) => {
const w: PendingWaiter = {
resolve,
reject,
expectedFrom: opts.expectedFrom,
timer: null,
abortListener: null,
signal: opts.signal,
};
w.timer = setTimeout(() => {
if (waiters.get(writeId) === w) {
waiters.delete(writeId);
cleanupWaiter(w);
reject(new OperationTimeoutError(`streams-bridge: writeId=${writeId} timed out after ${timeoutMs}ms`));
}
}, timeoutMs);
if (opts.signal.aborted) {
cleanupWaiter(w);
reject(opts.signal.reason ?? new Error('aborted before await'));
return;
}
const onAbort = (): void => {
if (waiters.get(writeId) === w) {
waiters.delete(writeId);
cleanupWaiter(w);
reject(opts.signal.reason ?? new Error('aborted'));
}
};
w.abortListener = onAbort;
opts.signal.addEventListener('abort', onAbort, { once: true });
waiters.set(writeId, w);
});
},
async initiateRead(opts) {
if (destroyed) throw new Error('streams-bridge: destroyed');
const readStreamId = generateRequestId();
const transferOpts: import('@shade/transfer').TransferOptions = {
to: opts.peer,
input: opts.stream,
metadata: {
...(opts.name !== undefined ? { name: opts.name } : {}),
...(opts.contentType !== undefined ? { contentType: opts.contentType } : {}),
sizeBytes: opts.size,
userMetadata: { [META_KEY_READ_STREAM_ID]: readStreamId },
},
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
...(opts.onProgress !== undefined ? { onProgress: opts.onProgress } : {}),
};
const handle = await shade.upload(transferOpts);
// The dispatcher returns the response immediately and doesn't itself
// await this handle — the receiver awaits its own handle.done() on
// the inbound side. Attach a no-op catch so engine teardown during
// an in-flight outbound transfer doesn't surface as an unhandled
// rejection in test environments.
handle.done().catch(() => undefined);
return { readStreamId, handle };
},
async destroy() {
if (destroyed) return;
destroyed = true;
unsubscribeIncoming();
// Reject all waiting RPCs
for (const w of waiters.values()) {
cleanupWaiter(w);
w.reject(new Error('streams-bridge: destroyed'));
}
waiters.clear();
// Abort all parked transfers
for (const p of parked.values()) {
try {
await p.handle.abort('bridge-destroyed');
} catch {
/* swallow */
}
}
parked.clear();
},
};
}

View File

@@ -0,0 +1,61 @@
/**
* Thumbnail format-hardening helper.
*
* Magic-byte verification for the three formats `@shade/files` advertises
* over the wire (`png`, `webp`, `jpeg`). Apps producing thumbnails outside
* the dispatcher (e.g. precomputing on disk) can call `assertThumbnailFormat`
* to fail-fast on mislabelled bytes before they reach the network.
*
* The same magic-byte table is used inside `io-adapters.ts:adaptThumbnailResult`
* to enforce the invariant on every outbound thumbnail.
*/
import { ConflictError } from '../schemas/errors.js';
const PNG_MAGIC = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
const JPEG_MAGIC = new Uint8Array([0xff, 0xd8, 0xff]);
const WEBP_RIFF = new Uint8Array([0x52, 0x49, 0x46, 0x46]);
const WEBP_HEAD = new Uint8Array([0x57, 0x45, 0x42, 0x50]);
export type ThumbnailFormat = 'png' | 'webp' | 'jpeg';
/**
* Verify the bytes look like the declared format. Returns boolean — does
* not throw. For a throwing variant, use `assertThumbnailFormat`.
*/
export function isThumbnailFormat(bytes: Uint8Array, format: ThumbnailFormat): boolean {
switch (format) {
case 'png':
return startsWith(bytes, PNG_MAGIC);
case 'jpeg':
return startsWith(bytes, JPEG_MAGIC);
case 'webp': {
if (bytes.byteLength < 12) return false;
if (!startsWith(bytes, WEBP_RIFF)) return false;
for (let i = 0; i < 4; i++) {
if (bytes[8 + i] !== WEBP_HEAD[i]) return false;
}
return true;
}
}
}
/** Throws `ConflictError` if `bytes` doesn't match the declared `format`. */
export function assertThumbnailFormat(
bytes: Uint8Array,
format: ThumbnailFormat,
): void {
if (!isThumbnailFormat(bytes, format)) {
throw new ConflictError(
`thumbnail bytes do not match declared format=${format}`,
);
}
}
function startsWith(bytes: Uint8Array, prefix: Uint8Array): boolean {
if (bytes.byteLength < prefix.byteLength) return false;
for (let i = 0; i < prefix.byteLength; i++) {
if (bytes[i] !== prefix[i]) return false;
}
return true;
}

View File

@@ -0,0 +1,86 @@
/**
* Minimal POSIX path utilities — works in browser and Bun without depending
* on Node's built-in `node:path`. Only handles forward-slash absolute paths.
*/
/**
* Normalize a POSIX absolute path: collapse `//`, resolve `.` and `..`,
* preserve trailing slash semantics. Returns `'/'` for `''`/`'/'` input.
*
* Throws nothing on invalid input — sanitization is the caller's job
* (`validatePath` in `server/path-policy.ts` does that).
*/
export function posixNormalize(path: string): string {
if (path === '' || path === '/') return '/';
const isAbs = path.startsWith('/');
const segments = path.split('/');
const stack: string[] = [];
for (const seg of segments) {
if (seg === '' || seg === '.') continue;
if (seg === '..') {
if (stack.length > 0 && stack[stack.length - 1] !== '..') {
stack.pop();
} else if (!isAbs) {
stack.push('..');
}
// For absolute paths, `..` past root is silently dropped.
continue;
}
stack.push(seg);
}
const joined = stack.join('/');
if (isAbs) return '/' + joined;
return joined === '' ? '.' : joined;
}
/** Join path segments with POSIX semantics. */
export function posixJoin(...parts: string[]): string {
if (parts.length === 0) return '.';
const joined = parts.filter((p) => p !== '').join('/');
return posixNormalize(joined);
}
/** Return the directory portion of a path, or `'/'` for root. */
export function posixDirname(path: string): string {
const normalized = posixNormalize(path);
if (normalized === '/') return '/';
const lastSlash = normalized.lastIndexOf('/');
if (lastSlash === 0) return '/';
if (lastSlash === -1) return '.';
return normalized.slice(0, lastSlash);
}
/** Return the file/dir name (last component). */
export function posixBasename(path: string): string {
const normalized = posixNormalize(path);
if (normalized === '/') return '';
const lastSlash = normalized.lastIndexOf('/');
return lastSlash === -1 ? normalized : normalized.slice(lastSlash + 1);
}
/**
* Decode percent-escapes BEFORE security checks so attackers can't
* smuggle `..` via `%2e%2e`. Decodes UTF-8 sequences where possible;
* leaves invalid sequences untouched (the caller still rejects them).
*/
export function decodePercentEscapes(s: string): string {
if (!s.includes('%')) return s;
try {
return decodeURIComponent(s);
} catch {
// Invalid percent-encoding — leave as-is so policy rejects on the raw form.
return s;
}
}
/**
* Predicate: does `child` lie under `root` after both are normalized?
* Both must be absolute. `root` is treated as a directory boundary.
*/
export function isPathInside(child: string, root: string): boolean {
const c = posixNormalize(child);
const r = posixNormalize(root);
if (r === '/') return c.startsWith('/');
// Ensure boundary alignment so `/foo` is NOT inside `/foobar`.
return c === r || c.startsWith(r + '/');
}