feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
Some checks failed
Test / test (push) Has been cancelled
Some checks failed
Test / test (push) Has been cancelled
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
483
packages/shade-files/src/client/client.ts
Normal file
483
packages/shade-files/src/client/client.ts
Normal file
@@ -0,0 +1,483 @@
|
||||
import type { Shade } from '@shade/sdk';
|
||||
import {
|
||||
KIND_CUSTOM_V1,
|
||||
KIND_DELETE_V1,
|
||||
KIND_GET_THUMBNAIL_V1,
|
||||
KIND_LIST_V1,
|
||||
KIND_MKDIR_V1,
|
||||
KIND_MOVE_V1,
|
||||
KIND_READ_V1,
|
||||
KIND_STAT_V1,
|
||||
KIND_WRITE_V1,
|
||||
MUTATION_OPS,
|
||||
type StandardOp,
|
||||
} from '../protocol/kinds.js';
|
||||
import type { CustomOpsMap } from '../server/custom-ops.js';
|
||||
import { generateIdempotencyKey, generateRequestId } from '../protocol/correlate.js';
|
||||
import {
|
||||
base64ToBytes,
|
||||
bytesToBase64,
|
||||
canonicalRpcBytes,
|
||||
hashArgs,
|
||||
} from '../protocol/canonical.js';
|
||||
import { ShadeFileRpcChannel } from '../rpc/channel.js';
|
||||
import { PendingRpcRegistry, type RegisterOptions } from '../rpc/pending.js';
|
||||
import type { RpcRequest } from '../schemas/envelope.js';
|
||||
import {
|
||||
CustomArgsSchema,
|
||||
CustomResultSchema,
|
||||
DeleteArgsSchema,
|
||||
DeleteResultSchema,
|
||||
GetThumbnailArgsSchema,
|
||||
GetThumbnailResultSchema,
|
||||
ListArgsSchema,
|
||||
ListResultSchema,
|
||||
MkdirArgsSchema,
|
||||
MkdirResultSchema,
|
||||
MoveArgsSchema,
|
||||
MoveResultSchema,
|
||||
ReadArgsSchema,
|
||||
ReadResultSchema,
|
||||
StatArgsSchema,
|
||||
StatResultSchema,
|
||||
WriteArgsSchema,
|
||||
WriteResultSchema,
|
||||
type DeleteArgs,
|
||||
type DeleteResult,
|
||||
type GetThumbnailArgs,
|
||||
type ListArgs,
|
||||
type ListResult,
|
||||
type MkdirArgs,
|
||||
type MkdirResult,
|
||||
type MoveArgs,
|
||||
type MoveResult,
|
||||
type ReadArgs,
|
||||
type StatResult,
|
||||
type ThumbnailSize,
|
||||
type WriteResult,
|
||||
} from '../schemas/ops.js';
|
||||
import { ConflictError, InternalFileError } from '../schemas/errors.js';
|
||||
import { decideInline, INLINE_THRESHOLD, type WriteSource } from './inline-threshold.js';
|
||||
import type { ClientStreamsBridge } from './streams-bridge.js';
|
||||
|
||||
export interface BaseOpts {
|
||||
signal?: AbortSignal;
|
||||
/** Auto-generated for mutations if not provided. */
|
||||
idempotencyKey?: string;
|
||||
/** Per-call timeout. Default 30_000 ms. */
|
||||
timeoutMs?: number;
|
||||
}
|
||||
|
||||
// ─── read/write public types ─────────────────────────────────
|
||||
|
||||
export interface ReadInlineOutput {
|
||||
kind: 'inline';
|
||||
bytes: Uint8Array;
|
||||
size: number;
|
||||
sha256: string;
|
||||
contentType?: string;
|
||||
}
|
||||
export interface ReadStreamsOutput {
|
||||
kind: 'streams';
|
||||
stream: ReadableStream<Uint8Array>;
|
||||
size: number;
|
||||
sha256: string;
|
||||
contentType?: string;
|
||||
/** Resolves once the entire transfer has been received and verified. */
|
||||
done(): Promise<void>;
|
||||
}
|
||||
export type ReadOutput = ReadInlineOutput | ReadStreamsOutput;
|
||||
|
||||
export interface WriteOpts extends BaseOpts {
|
||||
contentType?: string;
|
||||
overwrite?: boolean;
|
||||
/** Force inline even if size > 256 KiB. Throws if input is too big. */
|
||||
forceInline?: boolean;
|
||||
}
|
||||
|
||||
export interface ReadOpts extends BaseOpts {
|
||||
range?: { start: number; end: number };
|
||||
preferInline?: boolean;
|
||||
}
|
||||
|
||||
export interface ThumbnailResult {
|
||||
bytes: Uint8Array;
|
||||
format: 'png' | 'webp' | 'jpeg';
|
||||
width: number;
|
||||
height: number;
|
||||
sha256: string;
|
||||
}
|
||||
|
||||
// ─── FileClient interface ────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Untyped fallback for `FileClient.custom()` — used when the consumer
|
||||
* hasn't extended `CustomOpsMap` for a given op name.
|
||||
*/
|
||||
type CustomOpArgs<K> = K extends keyof CustomOpsMap
|
||||
? CustomOpsMap[K] extends { args: infer A }
|
||||
? A
|
||||
: unknown
|
||||
: unknown;
|
||||
type CustomOpResponse<K> = K extends keyof CustomOpsMap
|
||||
? CustomOpsMap[K] extends { response: infer R }
|
||||
? R
|
||||
: unknown
|
||||
: unknown;
|
||||
|
||||
export interface FileClient {
|
||||
list(path: string, opts?: BaseOpts & Partial<Omit<ListArgs, 'path'>>): Promise<ListResult>;
|
||||
stat(path: string, opts?: BaseOpts): Promise<StatResult>;
|
||||
mkdir(path: string, opts?: BaseOpts & Partial<Omit<MkdirArgs, 'path'>>): Promise<MkdirResult>;
|
||||
delete(path: string, opts?: BaseOpts & Partial<Omit<DeleteArgs, 'path'>>): Promise<DeleteResult>;
|
||||
move(src: string, dst: string, opts?: BaseOpts & Partial<Omit<MoveArgs, 'src' | 'dst'>>): Promise<MoveResult>;
|
||||
read(path: string, opts?: ReadOpts): Promise<ReadOutput>;
|
||||
write(path: string, input: WriteSource, opts?: WriteOpts): Promise<WriteResult>;
|
||||
getThumbnail(
|
||||
path: string,
|
||||
size: ThumbnailSize,
|
||||
opts?: BaseOpts & { format?: 'png' | 'webp' | 'jpeg' },
|
||||
): Promise<ThumbnailResult>;
|
||||
/**
|
||||
* Invoke a custom op registered on the server. Args/response types are
|
||||
* pulled from `CustomOpsMap` via TypeScript declaration merging — see
|
||||
* `server/custom-ops.ts` for the registration pattern.
|
||||
*/
|
||||
custom<K extends keyof CustomOpsMap | string>(
|
||||
name: K & string,
|
||||
args: CustomOpArgs<K>,
|
||||
opts?: BaseOpts,
|
||||
): Promise<CustomOpResponse<K>>;
|
||||
close(): void;
|
||||
}
|
||||
|
||||
export interface CreateFileClientOptions {
|
||||
/** Default per-call timeout. Default 30_000. */
|
||||
defaultTimeoutMs?: number;
|
||||
/** Hard deadline for incoming-read awaits. Default 60_000. */
|
||||
ioTimeoutMs?: number;
|
||||
/**
|
||||
* Required for read/write `streams` ops. Coordinates inbound/outbound
|
||||
* `@shade/transfer` transfers via `userMetadata.shadeFiles*Id` keys.
|
||||
*/
|
||||
streamsBridge?: ClientStreamsBridge;
|
||||
/**
|
||||
* Optional: sign the canonical bytes of every outgoing RPC envelope.
|
||||
* Pluggable so apps can plug their own signing-key store (e.g.,
|
||||
* Ed25519-as-a-service, browser SubtleCrypto). When omitted, ships
|
||||
* `'unsigned'` — the server's `verifySender` should also be unset, or
|
||||
* be configured to accept the placeholder.
|
||||
*/
|
||||
signRequest?: (canonicalBytes: Uint8Array) => Promise<string> | string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Client-side proxy for `@shade/files` ops. Each method ships an
|
||||
* `RpcRequest` over `Shade.send`/`Shade.receive` and awaits the matching
|
||||
* response (or error/timeout) from `PendingRpcRegistry`.
|
||||
*
|
||||
* Mutations auto-generate an idempotency key per logical call (not per
|
||||
* attempt) so transparent retries under the SDK don't produce duplicates.
|
||||
*
|
||||
* Read/write content I/O over 256 KiB requires a `streamsBridge` to be
|
||||
* passed via options — it coordinates the inbound/outbound `@shade/transfer`
|
||||
* transfers that carry the actual bytes.
|
||||
*/
|
||||
export function createFileClient(
|
||||
shade: Shade,
|
||||
channel: ShadeFileRpcChannel,
|
||||
pending: PendingRpcRegistry,
|
||||
peerAddress: string,
|
||||
options: CreateFileClientOptions = {},
|
||||
): FileClient {
|
||||
const defaultTimeout = options.defaultTimeoutMs ?? 30_000;
|
||||
const ioTimeoutMs = options.ioTimeoutMs ?? 60_000;
|
||||
const streamsBridge = options.streamsBridge;
|
||||
const signRequest = options.signRequest;
|
||||
const senderAddress = shade.myAddress;
|
||||
|
||||
async function request<TResult>(
|
||||
kind: string,
|
||||
op: StandardOp | 'custom',
|
||||
args: unknown,
|
||||
opts: BaseOpts | undefined,
|
||||
): Promise<TResult> {
|
||||
const requestId = generateRequestId();
|
||||
const isMutation = MUTATION_OPS.has(op);
|
||||
const idempotencyKey =
|
||||
opts?.idempotencyKey ?? (isMutation ? generateIdempotencyKey() : undefined);
|
||||
const signedAt = Date.now();
|
||||
let sig = 'unsigned';
|
||||
if (signRequest !== undefined) {
|
||||
// Server reconstructs canonical bytes using `address = from`, which
|
||||
// is OUR own address as authenticated by the ratchet. So we sign
|
||||
// over the same identifier here.
|
||||
const canonical = canonicalRpcBytes({
|
||||
address: senderAddress,
|
||||
signedAt,
|
||||
kind,
|
||||
id: requestId,
|
||||
argsHash: hashArgs(args),
|
||||
});
|
||||
sig = await signRequest(canonical);
|
||||
}
|
||||
const env: RpcRequest = {
|
||||
kind,
|
||||
id: requestId,
|
||||
args,
|
||||
...(idempotencyKey !== undefined ? { idempotencyKey } : {}),
|
||||
sig,
|
||||
signedAt,
|
||||
};
|
||||
|
||||
const registerOpts: RegisterOptions = {
|
||||
timeoutMs: opts?.timeoutMs ?? defaultTimeout,
|
||||
onCancel: (reason) => {
|
||||
// Fire-and-forget cancel envelope so server can release resources.
|
||||
void channel
|
||||
.send(peerAddress, {
|
||||
kind: 'shade.fs.cancel/v1',
|
||||
id: requestId,
|
||||
reason,
|
||||
})
|
||||
.catch(() => {
|
||||
/* swallow — cancellation is best-effort */
|
||||
});
|
||||
},
|
||||
};
|
||||
if (opts?.signal !== undefined) registerOpts.signal = opts.signal;
|
||||
|
||||
const pendingPromise = pending.register<TResult>(requestId, registerOpts);
|
||||
try {
|
||||
await channel.send(peerAddress, env);
|
||||
} catch (err) {
|
||||
// If the send itself fails, the pending entry will never resolve;
|
||||
// reject it directly.
|
||||
pending.rejectAll(err);
|
||||
throw err;
|
||||
}
|
||||
return pendingPromise;
|
||||
}
|
||||
|
||||
return {
|
||||
async list(path, opts) {
|
||||
const args: ListArgs = ListArgsSchema.parse({
|
||||
path,
|
||||
...(opts?.cursor !== undefined ? { cursor: opts.cursor } : {}),
|
||||
...(opts?.pageSize !== undefined ? { pageSize: opts.pageSize } : {}),
|
||||
...(opts?.filter !== undefined ? { filter: opts.filter } : {}),
|
||||
});
|
||||
const raw = await request<unknown>(KIND_LIST_V1, 'list', args, opts);
|
||||
return ListResultSchema.parse(raw);
|
||||
},
|
||||
|
||||
async stat(path, opts) {
|
||||
const args = StatArgsSchema.parse({ path });
|
||||
const raw = await request<unknown>(KIND_STAT_V1, 'stat', args, opts);
|
||||
return StatResultSchema.parse(raw);
|
||||
},
|
||||
|
||||
async mkdir(path, opts) {
|
||||
const args = MkdirArgsSchema.parse({
|
||||
path,
|
||||
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
|
||||
});
|
||||
const raw = await request<unknown>(KIND_MKDIR_V1, 'mkdir', args, opts);
|
||||
return MkdirResultSchema.parse(raw);
|
||||
},
|
||||
|
||||
async delete(path, opts) {
|
||||
const args = DeleteArgsSchema.parse({
|
||||
path,
|
||||
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
|
||||
});
|
||||
const raw = await request<unknown>(KIND_DELETE_V1, 'delete', args, opts);
|
||||
return DeleteResultSchema.parse(raw);
|
||||
},
|
||||
|
||||
async move(src, dst, opts) {
|
||||
const args = MoveArgsSchema.parse({
|
||||
src,
|
||||
dst,
|
||||
...(opts?.overwrite !== undefined ? { overwrite: opts.overwrite } : {}),
|
||||
});
|
||||
const raw = await request<unknown>(KIND_MOVE_V1, 'move', args, opts);
|
||||
return MoveResultSchema.parse(raw);
|
||||
},
|
||||
|
||||
async read(path, opts) {
|
||||
const args: ReadArgs = ReadArgsSchema.parse({
|
||||
path,
|
||||
...(opts?.range !== undefined ? { range: opts.range } : {}),
|
||||
...(opts?.preferInline !== undefined ? { preferInline: opts.preferInline } : {}),
|
||||
});
|
||||
const raw = await request<unknown>(KIND_READ_V1, 'read', args, opts);
|
||||
const wire = ReadResultSchema.parse(raw);
|
||||
if (wire.kind === 'inline') {
|
||||
const bytes = base64ToBytes(wire.bytesB64);
|
||||
const out: ReadInlineOutput = {
|
||||
kind: 'inline',
|
||||
bytes,
|
||||
size: wire.size,
|
||||
sha256: wire.sha256,
|
||||
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
|
||||
};
|
||||
return out;
|
||||
}
|
||||
// streams — wait for the matching incoming transfer via the bridge.
|
||||
if (streamsBridge === undefined) {
|
||||
throw new InternalFileError(
|
||||
'streams-bridge not configured: cannot consume streamed read',
|
||||
);
|
||||
}
|
||||
const bridgeSignal = opts?.signal ?? new AbortController().signal;
|
||||
const parked = await streamsBridge.awaitRead(wire.streamId, {
|
||||
expectedFrom: peerAddress,
|
||||
signal: bridgeSignal,
|
||||
timeoutMs: ioTimeoutMs,
|
||||
});
|
||||
const out: ReadStreamsOutput = {
|
||||
kind: 'streams',
|
||||
stream: parked.readable,
|
||||
size: wire.size,
|
||||
sha256: wire.sha256,
|
||||
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
|
||||
done: async () => {
|
||||
await parked.done;
|
||||
},
|
||||
};
|
||||
return out;
|
||||
},
|
||||
|
||||
async write(path, input, opts) {
|
||||
const decision = await decideInline(input);
|
||||
const overwrite = opts?.overwrite ?? false;
|
||||
const contentType = opts?.contentType ?? decision.contentType;
|
||||
|
||||
if (decision.kind === 'inline' || opts?.forceInline === true) {
|
||||
// Inline path — base64 in the RPC envelope.
|
||||
const bytes =
|
||||
decision.kind === 'inline'
|
||||
? decision.bytes
|
||||
: await drainToUint8Array(decision.stream, decision.size ?? Number.POSITIVE_INFINITY);
|
||||
if (bytes.byteLength > INLINE_THRESHOLD && opts?.forceInline !== true) {
|
||||
throw new ConflictError(
|
||||
`inline write exceeds ${INLINE_THRESHOLD}-byte threshold (got ${bytes.byteLength}); pass forceInline=true to override`,
|
||||
);
|
||||
}
|
||||
const args = WriteArgsSchema.parse({
|
||||
kind: 'inline',
|
||||
path,
|
||||
bytesB64: bytesToBase64(bytes),
|
||||
...(contentType !== undefined ? { contentType } : {}),
|
||||
overwrite,
|
||||
});
|
||||
const raw = await request<unknown>(KIND_WRITE_V1, 'write', args, opts);
|
||||
return WriteResultSchema.parse(raw);
|
||||
}
|
||||
|
||||
// Streams path — kick the upload, then ship the RPC.
|
||||
if (streamsBridge === undefined) {
|
||||
throw new InternalFileError(
|
||||
'streams-bridge not configured: cannot ship streamed write',
|
||||
);
|
||||
}
|
||||
const size = decision.size;
|
||||
if (size === undefined) {
|
||||
throw new ConflictError(
|
||||
'streams write requires a known plaintext size; pass `{ stream, size }` instead of a bare ReadableStream',
|
||||
);
|
||||
}
|
||||
const { writeId, handle } = await streamsBridge.initiateWrite({
|
||||
peer: peerAddress,
|
||||
stream: decision.stream,
|
||||
size,
|
||||
...(contentType !== undefined ? { contentType } : {}),
|
||||
name: path,
|
||||
...(opts?.signal !== undefined ? { signal: opts.signal } : {}),
|
||||
});
|
||||
const args = WriteArgsSchema.parse({
|
||||
kind: 'streams',
|
||||
path,
|
||||
size,
|
||||
...(contentType !== undefined ? { contentType } : {}),
|
||||
overwrite,
|
||||
writeId,
|
||||
});
|
||||
try {
|
||||
const [raw] = await Promise.all([
|
||||
request<unknown>(KIND_WRITE_V1, 'write', args, opts),
|
||||
handle.done(),
|
||||
]);
|
||||
return WriteResultSchema.parse(raw);
|
||||
} catch (err) {
|
||||
// Best-effort cancel of the transfer on RPC failure.
|
||||
await handle.abort('rpc-failed').catch(() => undefined);
|
||||
throw err;
|
||||
}
|
||||
},
|
||||
|
||||
async getThumbnail(path, size, opts) {
|
||||
const args: GetThumbnailArgs = GetThumbnailArgsSchema.parse({
|
||||
path,
|
||||
size,
|
||||
...(opts?.format !== undefined ? { format: opts.format } : {}),
|
||||
});
|
||||
const raw = await request<unknown>(KIND_GET_THUMBNAIL_V1, 'getThumbnail', args, opts);
|
||||
const wire = GetThumbnailResultSchema.parse(raw);
|
||||
return {
|
||||
bytes: base64ToBytes(wire.bytesB64),
|
||||
format: wire.format,
|
||||
width: wire.width,
|
||||
height: wire.height,
|
||||
sha256: wire.sha256,
|
||||
};
|
||||
},
|
||||
|
||||
async custom(name, args, opts) {
|
||||
const wireArgs = CustomArgsSchema.parse({ name, payload: args });
|
||||
// `custom` is a mutation in the rate-limit sense; auto-key for retries.
|
||||
const raw = await request<unknown>(KIND_CUSTOM_V1, 'custom' as StandardOp, wireArgs, opts);
|
||||
const wire = CustomResultSchema.parse(raw);
|
||||
// The result schema is `{ result: unknown }` — the inner `result` is
|
||||
// already validated against the consumer's response schema on the
|
||||
// server side, so we trust it here.
|
||||
return wire.result as never;
|
||||
},
|
||||
|
||||
close() {
|
||||
pending.rejectAll(new Error('FileClient closed'));
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** Drain a stream into a single buffer; used for the inline-write fallback. */
|
||||
async function drainToUint8Array(
|
||||
stream: ReadableStream<Uint8Array>,
|
||||
cap: number,
|
||||
): Promise<Uint8Array> {
|
||||
const reader = stream.getReader();
|
||||
const chunks: Uint8Array[] = [];
|
||||
let total = 0;
|
||||
try {
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (value === undefined) continue;
|
||||
chunks.push(value);
|
||||
total += value.byteLength;
|
||||
if (total > cap) {
|
||||
throw new Error(`stream produced more than declared size cap (${cap})`);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
reader.releaseLock();
|
||||
}
|
||||
const out = new Uint8Array(total);
|
||||
let offset = 0;
|
||||
for (const c of chunks) {
|
||||
out.set(c, offset);
|
||||
offset += c.byteLength;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
91
packages/shade-files/src/client/concurrency.ts
Normal file
91
packages/shade-files/src/client/concurrency.ts
Normal file
@@ -0,0 +1,91 @@
|
||||
/**
|
||||
* Bounded-concurrency async map.
|
||||
*
|
||||
* Pulls items from an `AsyncIterable` source (lazily — never materializing
|
||||
* the whole sequence) and runs `fn` on at most `concurrency` of them at
|
||||
* once. Failures bubble unless `continueOnError` is set, in which case they
|
||||
* are reported via `onError` and the pool keeps draining.
|
||||
*/
|
||||
import { CancelledError } from '../schemas/errors.js';
|
||||
|
||||
export interface ConcurrentMapOptions<T> {
|
||||
concurrency: number;
|
||||
signal?: AbortSignal;
|
||||
continueOnError?: boolean;
|
||||
onError?: (item: T, err: unknown) => void;
|
||||
}
|
||||
|
||||
export async function runWithConcurrency<T>(
|
||||
source: AsyncIterable<T>,
|
||||
fn: (item: T) => Promise<void>,
|
||||
opts: ConcurrentMapOptions<T>,
|
||||
): Promise<void> {
|
||||
if (opts.concurrency < 1) throw new Error('concurrency must be ≥ 1');
|
||||
|
||||
const iter = source[Symbol.asyncIterator]();
|
||||
const inFlight = new Set<Promise<void>>();
|
||||
let firstError: unknown = null;
|
||||
let aborted = false;
|
||||
|
||||
function checkAbort(): void {
|
||||
if (opts.signal?.aborted) {
|
||||
aborted = true;
|
||||
const reason = opts.signal.reason;
|
||||
throw reason instanceof Error ? reason : new CancelledError(String(reason ?? 'aborted'));
|
||||
}
|
||||
}
|
||||
|
||||
async function pumpOne(): Promise<boolean> {
|
||||
checkAbort();
|
||||
const next = await iter.next();
|
||||
if (next.done === true) return false;
|
||||
const item = next.value;
|
||||
const task = (async () => {
|
||||
try {
|
||||
await fn(item);
|
||||
} catch (err) {
|
||||
if (opts.continueOnError === true) {
|
||||
opts.onError?.(item, err);
|
||||
} else if (firstError === null) {
|
||||
firstError = err;
|
||||
}
|
||||
}
|
||||
})().finally(() => {
|
||||
inFlight.delete(task);
|
||||
});
|
||||
inFlight.add(task);
|
||||
return true;
|
||||
}
|
||||
|
||||
try {
|
||||
// Prime the pool
|
||||
while (inFlight.size < opts.concurrency) {
|
||||
if (firstError !== null || aborted) break;
|
||||
const more = await pumpOne();
|
||||
if (!more) break;
|
||||
}
|
||||
// Maintain saturation
|
||||
while (inFlight.size > 0) {
|
||||
if (firstError !== null && opts.continueOnError !== true) break;
|
||||
await Promise.race(inFlight);
|
||||
while (
|
||||
inFlight.size < opts.concurrency &&
|
||||
firstError === null &&
|
||||
!aborted
|
||||
) {
|
||||
const more = await pumpOne();
|
||||
if (!more) break;
|
||||
}
|
||||
}
|
||||
// Drain whatever survived
|
||||
if (inFlight.size > 0) await Promise.allSettled(inFlight);
|
||||
} finally {
|
||||
if (typeof iter.return === 'function') {
|
||||
await iter.return().catch(() => undefined);
|
||||
}
|
||||
}
|
||||
|
||||
if (firstError !== null && opts.continueOnError !== true) {
|
||||
throw firstError;
|
||||
}
|
||||
}
|
||||
97
packages/shade-files/src/client/directory-types.ts
Normal file
97
packages/shade-files/src/client/directory-types.ts
Normal file
@@ -0,0 +1,97 @@
|
||||
/**
|
||||
* Public types for directory operations (`walk`, `uploadDirectory`,
|
||||
* `downloadDirectory`).
|
||||
*
|
||||
* `DirectoryHandleLike` and `FileHandleLike` are structurally compatible
|
||||
* with the browser File System Access API
|
||||
* (`FileSystemDirectoryHandle` / `FileSystemFileHandle`) so that browser
|
||||
* consumers can pass them directly. For Bun/Node, use the adapter from
|
||||
* `node-directory-handle.ts`.
|
||||
*/
|
||||
|
||||
export interface FileHandleLike {
|
||||
readonly kind: 'file';
|
||||
readonly name: string;
|
||||
/** Read-side accessor — used by `uploadDirectory`. */
|
||||
getFile(): Promise<FileLike>;
|
||||
/** Write-side accessor — used by `downloadDirectory`. */
|
||||
createWritable(): Promise<WritableStreamLike>;
|
||||
}
|
||||
|
||||
export interface FileLike {
|
||||
readonly name: string;
|
||||
readonly size: number;
|
||||
readonly type: string;
|
||||
stream(): ReadableStream<Uint8Array>;
|
||||
arrayBuffer(): Promise<ArrayBuffer>;
|
||||
}
|
||||
|
||||
export interface WritableStreamLike {
|
||||
write(chunk: Uint8Array): Promise<void>;
|
||||
close(): Promise<void>;
|
||||
abort(reason?: unknown): Promise<void>;
|
||||
}
|
||||
|
||||
export interface DirectoryHandleLike {
|
||||
readonly kind: 'directory';
|
||||
readonly name: string;
|
||||
/** Yield child entries (file or directory). Used by `uploadDirectory`. */
|
||||
entries(): AsyncIterable<[string, DirectoryHandleLike | FileHandleLike]>;
|
||||
/** Used by `downloadDirectory` to create remote → local mapping. */
|
||||
getDirectoryHandle(name: string, opts?: { create?: boolean }): Promise<DirectoryHandleLike>;
|
||||
getFileHandle(name: string, opts?: { create?: boolean }): Promise<FileHandleLike>;
|
||||
}
|
||||
|
||||
// ─── Bulk transfer types ─────────────────────────────────────
|
||||
|
||||
export type BulkTransferEvent =
|
||||
| { type: 'plan'; totalFiles: number; totalBytes: number | undefined }
|
||||
| { type: 'file-start'; path: string; size: number }
|
||||
| { type: 'file-progress'; path: string; bytesDone: number; bytesTotal: number }
|
||||
| { type: 'file-done'; path: string; bytesDone: number }
|
||||
| { type: 'file-error'; path: string; error: unknown }
|
||||
| {
|
||||
type: 'progress';
|
||||
filesDone: number;
|
||||
filesTotal: number;
|
||||
bytesDone: number;
|
||||
bytesTotal: number | undefined;
|
||||
}
|
||||
| { type: 'complete'; filesDone: number; bytesDone: number; durationMs: number }
|
||||
| { type: 'abort'; reason: string };
|
||||
|
||||
export interface BulkTransferResult {
|
||||
filesDone: number;
|
||||
bytesDone: number;
|
||||
durationMs: number;
|
||||
}
|
||||
|
||||
export interface BulkTransferHandle {
|
||||
/** Async-iterable event stream — plan, per-file events, aggregate progress. */
|
||||
readonly events: AsyncIterable<BulkTransferEvent>;
|
||||
abort(reason?: string): Promise<void>;
|
||||
done(): Promise<BulkTransferResult>;
|
||||
}
|
||||
|
||||
export interface BulkOpts {
|
||||
/**
|
||||
* Max files in flight at once. Default 4. Capped at 16 to bound memory
|
||||
* (each in-flight file may hold a buffered chunk; with 1 MiB chunks +
|
||||
* 16 lanes that's still bounded).
|
||||
*/
|
||||
concurrency?: number;
|
||||
/**
|
||||
* Continue past per-file failures. Default false (fail-fast).
|
||||
* When true, errors are emitted via `file-error` events and the whole
|
||||
* transfer still resolves (with `done()` returning the count of
|
||||
* successful files).
|
||||
*/
|
||||
continueOnError?: boolean;
|
||||
/** Cancellation. */
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
|
||||
/** Hard cap to bound memory regardless of caller-supplied concurrency. */
|
||||
export const MAX_BULK_CONCURRENCY = 16;
|
||||
/** Default concurrency. */
|
||||
export const DEFAULT_BULK_CONCURRENCY = 4;
|
||||
316
packages/shade-files/src/client/download-directory.ts
Normal file
316
packages/shade-files/src/client/download-directory.ts
Normal file
@@ -0,0 +1,316 @@
|
||||
/**
|
||||
* Download an entire remote directory tree to a local `DirectoryHandleLike`.
|
||||
*
|
||||
* Mirror image of `uploadDirectory`: walks the remote tree via the shared
|
||||
* `walk()` helper, creates local directories on the fly, and downloads each
|
||||
* file with `client.read` (which routes inline or streams based on the
|
||||
* server's response). Bounded concurrency keeps RPC inflight count low.
|
||||
*/
|
||||
import { walk, type WalkItem } from './walk.js';
|
||||
import { runWithConcurrency } from './concurrency.js';
|
||||
import type { FileClient } from './client.js';
|
||||
import {
|
||||
DEFAULT_BULK_CONCURRENCY,
|
||||
MAX_BULK_CONCURRENCY,
|
||||
type BulkOpts,
|
||||
type BulkTransferEvent,
|
||||
type BulkTransferHandle,
|
||||
type BulkTransferResult,
|
||||
type DirectoryHandleLike,
|
||||
type FileHandleLike,
|
||||
} from './directory-types.js';
|
||||
import { CancelledError } from '../schemas/errors.js';
|
||||
|
||||
export interface DownloadDirectoryOptions extends BulkOpts {
|
||||
/** Page size hint forwarded to the underlying `walk`. Default 200. */
|
||||
pageSize?: number;
|
||||
/** Skip files already present locally. Default false (overwrite). */
|
||||
skipExisting?: boolean;
|
||||
}
|
||||
|
||||
export function downloadDirectory(
|
||||
client: Pick<FileClient, 'read' | 'list'>,
|
||||
remoteRoot: string,
|
||||
local: DirectoryHandleLike,
|
||||
opts: DownloadDirectoryOptions = {},
|
||||
): BulkTransferHandle {
|
||||
const concurrency = Math.max(
|
||||
1,
|
||||
Math.min(MAX_BULK_CONCURRENCY, opts.concurrency ?? DEFAULT_BULK_CONCURRENCY),
|
||||
);
|
||||
const continueOnError = opts.continueOnError ?? false;
|
||||
const externalSignal = opts.signal;
|
||||
const pageSize = opts.pageSize ?? 200;
|
||||
|
||||
const internalAbort = new AbortController();
|
||||
const combinedSignal = mergeSignals(externalSignal, internalAbort.signal);
|
||||
|
||||
const events: BulkTransferEvent[] = [];
|
||||
const eventResolvers: ((v: IteratorResult<BulkTransferEvent>) => void)[] = [];
|
||||
let eventsClosed = false;
|
||||
|
||||
function emit(event: BulkTransferEvent): void {
|
||||
if (eventsClosed) return;
|
||||
if (eventResolvers.length > 0) {
|
||||
eventResolvers.shift()!({ value: event, done: false });
|
||||
} else {
|
||||
events.push(event);
|
||||
}
|
||||
}
|
||||
|
||||
function closeEvents(): void {
|
||||
if (eventsClosed) return;
|
||||
eventsClosed = true;
|
||||
while (eventResolvers.length > 0) {
|
||||
eventResolvers.shift()!({ value: undefined as never, done: true });
|
||||
}
|
||||
}
|
||||
|
||||
let resolveDone!: (r: BulkTransferResult) => void;
|
||||
let rejectDone!: (err: unknown) => void;
|
||||
const donePromise = new Promise<BulkTransferResult>((resolve, reject) => {
|
||||
resolveDone = resolve;
|
||||
rejectDone = reject;
|
||||
});
|
||||
|
||||
const startedAt = Date.now();
|
||||
let filesDone = 0;
|
||||
let bytesDone = 0;
|
||||
|
||||
void (async () => {
|
||||
try {
|
||||
// 1. Plan: walk the remote tree into [dirs, files]
|
||||
const dirItems: WalkItem[] = [];
|
||||
const fileItems: WalkItem[] = [];
|
||||
try {
|
||||
for await (const item of walk(client, remoteRoot, {
|
||||
pageSize,
|
||||
...(combinedSignal !== undefined ? { signal: combinedSignal } : {}),
|
||||
})) {
|
||||
if (item.entry.kind === 'dir') dirItems.push(item);
|
||||
else fileItems.push(item);
|
||||
}
|
||||
} catch (err) {
|
||||
emit({ type: 'abort', reason: errMsg(err) });
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
return;
|
||||
}
|
||||
|
||||
const totalFiles = fileItems.length;
|
||||
const bytesTotal = fileItems.reduce((acc, f) => acc + f.entry.size, 0);
|
||||
emit({ type: 'plan', totalFiles, totalBytes: bytesTotal });
|
||||
|
||||
if (combinedSignal.aborted) {
|
||||
emit({ type: 'abort', reason: errMsg(combinedSignal.reason) });
|
||||
closeEvents();
|
||||
rejectDone(combinedSignal.reason ?? new CancelledError());
|
||||
return;
|
||||
}
|
||||
|
||||
// 2. Pre-create local directories sequentially.
|
||||
for (const d of dirItems) {
|
||||
if (combinedSignal.aborted) break;
|
||||
try {
|
||||
await ensureLocalDir(local, d.relativePath);
|
||||
} catch (err) {
|
||||
if (!continueOnError) {
|
||||
emit({ type: 'abort', reason: errMsg(err) });
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Download files with bounded concurrency.
|
||||
try {
|
||||
await runWithConcurrency(
|
||||
asyncIterableOf(fileItems),
|
||||
async (item) => {
|
||||
if (combinedSignal.aborted) {
|
||||
throw new CancelledError('download aborted');
|
||||
}
|
||||
emit({ type: 'file-start', path: item.relativePath, size: item.entry.size });
|
||||
try {
|
||||
const fileHandle = await ensureLocalFile(local, item.relativePath);
|
||||
if (opts.skipExisting === true) {
|
||||
// Skip if file already exists with non-zero size
|
||||
try {
|
||||
const existing = await fileHandle.getFile();
|
||||
if (existing.size === item.entry.size) {
|
||||
filesDone++;
|
||||
bytesDone += existing.size;
|
||||
emit({ type: 'file-done', path: item.relativePath, bytesDone: existing.size });
|
||||
emit({
|
||||
type: 'progress',
|
||||
filesDone,
|
||||
filesTotal: totalFiles,
|
||||
bytesDone,
|
||||
bytesTotal,
|
||||
});
|
||||
return;
|
||||
}
|
||||
} catch {
|
||||
/* not present yet — fall through to download */
|
||||
}
|
||||
}
|
||||
const writable = await fileHandle.createWritable();
|
||||
try {
|
||||
const result = await client.read(item.absolutePath, {
|
||||
signal: combinedSignal,
|
||||
});
|
||||
if (result.kind === 'inline') {
|
||||
if (result.bytes.byteLength > 0) await writable.write(result.bytes);
|
||||
await writable.close();
|
||||
filesDone++;
|
||||
bytesDone += result.bytes.byteLength;
|
||||
emit({ type: 'file-done', path: item.relativePath, bytesDone: result.bytes.byteLength });
|
||||
} else {
|
||||
await pipeReadableToWritable(result.stream, writable);
|
||||
await writable.close();
|
||||
await result.done();
|
||||
filesDone++;
|
||||
bytesDone += result.size;
|
||||
emit({ type: 'file-done', path: item.relativePath, bytesDone: result.size });
|
||||
}
|
||||
emit({
|
||||
type: 'progress',
|
||||
filesDone,
|
||||
filesTotal: totalFiles,
|
||||
bytesDone,
|
||||
bytesTotal,
|
||||
});
|
||||
} catch (err) {
|
||||
await writable.abort(err).catch(() => undefined);
|
||||
throw err;
|
||||
}
|
||||
} catch (err) {
|
||||
emit({ type: 'file-error', path: item.relativePath, error: err });
|
||||
throw err;
|
||||
}
|
||||
},
|
||||
{
|
||||
concurrency,
|
||||
signal: combinedSignal,
|
||||
continueOnError,
|
||||
onError: () => {
|
||||
/* already emitted as 'file-error' */
|
||||
},
|
||||
},
|
||||
);
|
||||
} catch (err) {
|
||||
emit({ type: 'abort', reason: errMsg(err) });
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
return;
|
||||
}
|
||||
|
||||
const durationMs = Date.now() - startedAt;
|
||||
emit({ type: 'complete', filesDone, bytesDone, durationMs });
|
||||
closeEvents();
|
||||
resolveDone({ filesDone, bytesDone, durationMs });
|
||||
} catch (err) {
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
}
|
||||
})();
|
||||
|
||||
donePromise.catch(() => {
|
||||
/* deliberate */
|
||||
});
|
||||
|
||||
return {
|
||||
events: {
|
||||
[Symbol.asyncIterator]() {
|
||||
return {
|
||||
next(): Promise<IteratorResult<BulkTransferEvent>> {
|
||||
if (events.length > 0) {
|
||||
return Promise.resolve({ value: events.shift()!, done: false });
|
||||
}
|
||||
if (eventsClosed) {
|
||||
return Promise.resolve({ value: undefined as never, done: true });
|
||||
}
|
||||
return new Promise((resolve) => eventResolvers.push(resolve));
|
||||
},
|
||||
return(): Promise<IteratorResult<BulkTransferEvent>> {
|
||||
return Promise.resolve({ value: undefined as never, done: true });
|
||||
},
|
||||
};
|
||||
},
|
||||
},
|
||||
async abort(reason) {
|
||||
internalAbort.abort(new CancelledError(reason ?? 'manual abort'));
|
||||
},
|
||||
done: () => donePromise,
|
||||
};
|
||||
}
|
||||
|
||||
// ─── Helpers ─────────────────────────────────────────────────
|
||||
|
||||
async function ensureLocalDir(
|
||||
root: DirectoryHandleLike,
|
||||
relativePath: string,
|
||||
): Promise<DirectoryHandleLike> {
|
||||
const segments = relativePath.split('/').filter((s) => s !== '');
|
||||
let current = root;
|
||||
for (const seg of segments) {
|
||||
current = await current.getDirectoryHandle(seg, { create: true });
|
||||
}
|
||||
return current;
|
||||
}
|
||||
|
||||
async function ensureLocalFile(
|
||||
root: DirectoryHandleLike,
|
||||
relativePath: string,
|
||||
): Promise<FileHandleLike> {
|
||||
const segments = relativePath.split('/').filter((s) => s !== '');
|
||||
if (segments.length === 0) throw new Error('empty file path');
|
||||
let current = root;
|
||||
for (let i = 0; i < segments.length - 1; i++) {
|
||||
current = await current.getDirectoryHandle(segments[i]!, { create: true });
|
||||
}
|
||||
return await current.getFileHandle(segments[segments.length - 1]!, { create: true });
|
||||
}
|
||||
|
||||
async function pipeReadableToWritable(
|
||||
readable: ReadableStream<Uint8Array>,
|
||||
writable: { write(chunk: Uint8Array): Promise<void> },
|
||||
): Promise<void> {
|
||||
const reader = readable.getReader();
|
||||
try {
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (value !== undefined && value.byteLength > 0) {
|
||||
await writable.write(value);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
reader.releaseLock();
|
||||
}
|
||||
}
|
||||
|
||||
async function* asyncIterableOf<T>(arr: T[]): AsyncIterable<T> {
|
||||
for (const item of arr) yield item;
|
||||
}
|
||||
|
||||
function errMsg(err: unknown): string {
|
||||
if (err instanceof Error) return err.message;
|
||||
return String(err ?? 'unknown error');
|
||||
}
|
||||
|
||||
function mergeSignals(
|
||||
external: AbortSignal | undefined,
|
||||
internal: AbortSignal,
|
||||
): AbortSignal {
|
||||
if (external === undefined) return internal;
|
||||
if (external.aborted) return external;
|
||||
const controller = new AbortController();
|
||||
const onAbort = (sig: AbortSignal): void => {
|
||||
controller.abort(sig.reason);
|
||||
};
|
||||
external.addEventListener('abort', () => onAbort(external), { once: true });
|
||||
internal.addEventListener('abort', () => onAbort(internal), { once: true });
|
||||
return controller.signal;
|
||||
}
|
||||
218
packages/shade-files/src/client/inline-threshold.ts
Normal file
218
packages/shade-files/src/client/inline-threshold.ts
Normal file
@@ -0,0 +1,218 @@
|
||||
/**
|
||||
* Decide whether a write should travel inline (base64 inside the RPC
|
||||
* envelope) or via a dedicated `@shade/transfer` stream.
|
||||
*
|
||||
* Threshold: 256 KiB plaintext. Anything strictly above goes through the
|
||||
* stream path; anything ≤ goes inline.
|
||||
*
|
||||
* For known-size inputs (`Uint8Array`, `Blob`, `File`) the decision is
|
||||
* cheap: compare `byteLength`/`size` against the threshold.
|
||||
*
|
||||
* For `ReadableStream` we cannot know the size up front. We pull chunks
|
||||
* into a temporary buffer until we either:
|
||||
* - hit EOF before the threshold → inline (we have all the bytes)
|
||||
* - cross the threshold → streams (the buffered prefix + the rest of the
|
||||
* stream are returned via a fresh `ReadableStream` so the caller can
|
||||
* feed it to `shade.upload`).
|
||||
*/
|
||||
/** Plaintext size at which inline transitions to streams. */
|
||||
export const INLINE_THRESHOLD = 256 * 1024;
|
||||
|
||||
export type InlineDecision =
|
||||
| {
|
||||
kind: 'inline';
|
||||
bytes: Uint8Array;
|
||||
contentType?: string;
|
||||
}
|
||||
| {
|
||||
kind: 'streams';
|
||||
stream: ReadableStream<Uint8Array>;
|
||||
/** Plaintext size when known (Blob/File). undefined for raw streams. */
|
||||
size?: number;
|
||||
contentType?: string;
|
||||
};
|
||||
|
||||
/**
|
||||
* Public input shape for `FileClient.write`. Runtime discriminator helper.
|
||||
*/
|
||||
export type WriteSource =
|
||||
| Uint8Array
|
||||
| Blob
|
||||
| File
|
||||
| ReadableStream<Uint8Array>
|
||||
| { stream: ReadableStream<Uint8Array>; size: number; contentType?: string };
|
||||
|
||||
export async function decideInline(input: WriteSource): Promise<InlineDecision> {
|
||||
// 1. Uint8Array — direct size check
|
||||
if (input instanceof Uint8Array) {
|
||||
if (input.byteLength <= INLINE_THRESHOLD) {
|
||||
return { kind: 'inline', bytes: input };
|
||||
}
|
||||
return {
|
||||
kind: 'streams',
|
||||
stream: uint8ArrayToStream(input),
|
||||
size: input.byteLength,
|
||||
};
|
||||
}
|
||||
|
||||
// 2. Blob / File — known size (Blob.type is a string; File extends Blob)
|
||||
if (typeof Blob !== 'undefined' && input instanceof Blob) {
|
||||
const blob: Blob = input;
|
||||
const contentType = blob.type === '' ? undefined : blob.type;
|
||||
if (blob.size <= INLINE_THRESHOLD) {
|
||||
const bytes = new Uint8Array(await blob.arrayBuffer());
|
||||
return contentType !== undefined
|
||||
? { kind: 'inline', bytes, contentType }
|
||||
: { kind: 'inline', bytes };
|
||||
}
|
||||
return contentType !== undefined
|
||||
? { kind: 'streams', stream: blob.stream(), size: blob.size, contentType }
|
||||
: { kind: 'streams', stream: blob.stream(), size: blob.size };
|
||||
}
|
||||
|
||||
// 3. Pre-wrapped { stream, size } — trust caller's declared size
|
||||
if (
|
||||
typeof input === 'object' &&
|
||||
input !== null &&
|
||||
'stream' in input &&
|
||||
'size' in input &&
|
||||
(input as { stream: unknown }).stream instanceof ReadableStream
|
||||
) {
|
||||
const wrapped = input as { stream: ReadableStream<Uint8Array>; size: number; contentType?: string };
|
||||
if (wrapped.size <= INLINE_THRESHOLD) {
|
||||
const bytes = await drainStream(wrapped.stream, wrapped.size);
|
||||
const contentType = wrapped.contentType;
|
||||
return contentType !== undefined
|
||||
? { kind: 'inline', bytes, contentType }
|
||||
: { kind: 'inline', bytes };
|
||||
}
|
||||
const contentType = wrapped.contentType;
|
||||
return contentType !== undefined
|
||||
? { kind: 'streams', stream: wrapped.stream, size: wrapped.size, contentType }
|
||||
: { kind: 'streams', stream: wrapped.stream, size: wrapped.size };
|
||||
}
|
||||
|
||||
// 4. Bare ReadableStream — peek until threshold or EOF
|
||||
if (input instanceof ReadableStream) {
|
||||
return await peekStream(input);
|
||||
}
|
||||
|
||||
throw new TypeError(
|
||||
`decideInline: unsupported input type ${Object.prototype.toString.call(input)}`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Drain a stream into a Uint8Array, with a soft cap at `expected` + slack.
|
||||
* Used for the inline path when the caller declared a size.
|
||||
*/
|
||||
async function drainStream(
|
||||
stream: ReadableStream<Uint8Array>,
|
||||
expected: number,
|
||||
): Promise<Uint8Array> {
|
||||
const reader = stream.getReader();
|
||||
const chunks: Uint8Array[] = [];
|
||||
let total = 0;
|
||||
try {
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (value === undefined) continue;
|
||||
chunks.push(value);
|
||||
total += value.byteLength;
|
||||
if (total > expected + INLINE_THRESHOLD) {
|
||||
throw new Error(
|
||||
`decideInline: stream produced more bytes (${total}) than declared size (${expected})`,
|
||||
);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
reader.releaseLock();
|
||||
}
|
||||
return concat(chunks, total);
|
||||
}
|
||||
|
||||
/**
|
||||
* Peek a `ReadableStream` of unknown length: buffer up to `INLINE_THRESHOLD + 1`
|
||||
* bytes. If EOF first, return inline. Otherwise reconstruct a stream that
|
||||
* yields the buffered prefix followed by the remainder.
|
||||
*/
|
||||
async function peekStream(stream: ReadableStream<Uint8Array>): Promise<InlineDecision> {
|
||||
const reader = stream.getReader();
|
||||
const buffered: Uint8Array[] = [];
|
||||
let total = 0;
|
||||
try {
|
||||
while (total <= INLINE_THRESHOLD) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) {
|
||||
reader.releaseLock();
|
||||
return { kind: 'inline', bytes: concat(buffered, total) };
|
||||
}
|
||||
if (value === undefined) continue;
|
||||
buffered.push(value);
|
||||
total += value.byteLength;
|
||||
}
|
||||
// We have at least INLINE_THRESHOLD + 1 bytes buffered. Promote to streams.
|
||||
const reconstructed = reconstructStream(buffered, reader);
|
||||
return { kind: 'streams', stream: reconstructed };
|
||||
} catch (err) {
|
||||
reader.releaseLock();
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
interface MinimalReader {
|
||||
read(): Promise<{ value: Uint8Array | undefined; done: boolean }>;
|
||||
cancel(reason?: unknown): Promise<void>;
|
||||
releaseLock(): void;
|
||||
}
|
||||
|
||||
function reconstructStream(
|
||||
prefix: Uint8Array[],
|
||||
reader: MinimalReader,
|
||||
): ReadableStream<Uint8Array> {
|
||||
let prefixIdx = 0;
|
||||
return new ReadableStream<Uint8Array>({
|
||||
async pull(controller) {
|
||||
if (prefixIdx < prefix.length) {
|
||||
controller.enqueue(prefix[prefixIdx]!);
|
||||
prefixIdx++;
|
||||
return;
|
||||
}
|
||||
const { value, done } = await reader.read();
|
||||
if (done) {
|
||||
controller.close();
|
||||
reader.releaseLock();
|
||||
return;
|
||||
}
|
||||
if (value !== undefined) controller.enqueue(value);
|
||||
},
|
||||
async cancel(reason) {
|
||||
try {
|
||||
await reader.cancel(reason);
|
||||
} finally {
|
||||
reader.releaseLock();
|
||||
}
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function uint8ArrayToStream(bytes: Uint8Array): ReadableStream<Uint8Array> {
|
||||
return new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
controller.enqueue(bytes);
|
||||
controller.close();
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function concat(chunks: Uint8Array[], total: number): Uint8Array {
|
||||
const out = new Uint8Array(total);
|
||||
let offset = 0;
|
||||
for (const c of chunks) {
|
||||
out.set(c, offset);
|
||||
offset += c.byteLength;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
145
packages/shade-files/src/client/memory-directory.ts
Normal file
145
packages/shade-files/src/client/memory-directory.ts
Normal file
@@ -0,0 +1,145 @@
|
||||
/**
|
||||
* In-memory `DirectoryHandleLike` implementation.
|
||||
*
|
||||
* Useful for tests and Node/Bun environments without the browser File
|
||||
* System Access API. The shape is intentionally compatible with browser
|
||||
* `FileSystemDirectoryHandle` so consumers can swap implementations.
|
||||
*/
|
||||
import type {
|
||||
DirectoryHandleLike,
|
||||
FileHandleLike,
|
||||
FileLike,
|
||||
WritableStreamLike,
|
||||
} from './directory-types.js';
|
||||
|
||||
interface MemoryNode {
|
||||
bytes: Uint8Array;
|
||||
type: string;
|
||||
}
|
||||
|
||||
class MemoryFileLike implements FileLike {
|
||||
constructor(
|
||||
public readonly name: string,
|
||||
private readonly node: MemoryNode,
|
||||
) {}
|
||||
get size(): number { return this.node.bytes.byteLength; }
|
||||
get type(): string { return this.node.type; }
|
||||
stream(): ReadableStream<Uint8Array> {
|
||||
const bytes = this.node.bytes;
|
||||
return new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
if (bytes.byteLength > 0) controller.enqueue(bytes);
|
||||
controller.close();
|
||||
},
|
||||
});
|
||||
}
|
||||
async arrayBuffer(): Promise<ArrayBuffer> {
|
||||
const out = new ArrayBuffer(this.node.bytes.byteLength);
|
||||
new Uint8Array(out).set(this.node.bytes);
|
||||
return out;
|
||||
}
|
||||
}
|
||||
|
||||
class MemoryWritable implements WritableStreamLike {
|
||||
private chunks: Uint8Array[] = [];
|
||||
constructor(private readonly file: MemoryFile) {}
|
||||
async write(chunk: Uint8Array): Promise<void> {
|
||||
this.chunks.push(chunk);
|
||||
}
|
||||
async close(): Promise<void> {
|
||||
let total = 0;
|
||||
for (const c of this.chunks) total += c.byteLength;
|
||||
const out = new Uint8Array(total);
|
||||
let offset = 0;
|
||||
for (const c of this.chunks) {
|
||||
out.set(c, offset);
|
||||
offset += c.byteLength;
|
||||
}
|
||||
this.file.commit(out);
|
||||
}
|
||||
async abort(): Promise<void> {
|
||||
this.chunks = [];
|
||||
}
|
||||
}
|
||||
|
||||
class MemoryFile implements FileHandleLike {
|
||||
readonly kind: 'file' = 'file';
|
||||
constructor(
|
||||
public readonly name: string,
|
||||
public readonly node: MemoryNode,
|
||||
) {}
|
||||
async getFile(): Promise<FileLike> {
|
||||
return new MemoryFileLike(this.name, this.node);
|
||||
}
|
||||
async createWritable(): Promise<WritableStreamLike> {
|
||||
return new MemoryWritable(this);
|
||||
}
|
||||
/** Internal — used by `MemoryWritable.close()`. */
|
||||
commit(bytes: Uint8Array): void {
|
||||
this.node.bytes = bytes;
|
||||
}
|
||||
}
|
||||
|
||||
class MemoryDirectory implements DirectoryHandleLike {
|
||||
readonly kind: 'directory' = 'directory';
|
||||
private readonly children = new Map<string, MemoryDirectory | MemoryFile>();
|
||||
constructor(public readonly name: string) {}
|
||||
|
||||
async *entries(): AsyncIterable<[string, DirectoryHandleLike | FileHandleLike]> {
|
||||
for (const [name, child] of this.children) {
|
||||
yield [name, child];
|
||||
}
|
||||
}
|
||||
|
||||
async getDirectoryHandle(name: string, opts?: { create?: boolean }): Promise<DirectoryHandleLike> {
|
||||
const existing = this.children.get(name);
|
||||
if (existing !== undefined) {
|
||||
if (existing.kind !== 'directory') {
|
||||
throw new Error(`'${name}' exists but is a file`);
|
||||
}
|
||||
return existing;
|
||||
}
|
||||
if (opts?.create !== true) throw new Error(`directory not found: ${name}`);
|
||||
const dir = new MemoryDirectory(name);
|
||||
this.children.set(name, dir);
|
||||
return dir;
|
||||
}
|
||||
|
||||
async getFileHandle(name: string, opts?: { create?: boolean }): Promise<FileHandleLike> {
|
||||
const existing = this.children.get(name);
|
||||
if (existing !== undefined) {
|
||||
if (existing.kind !== 'file') {
|
||||
throw new Error(`'${name}' exists but is a directory`);
|
||||
}
|
||||
return existing;
|
||||
}
|
||||
if (opts?.create !== true) throw new Error(`file not found: ${name}`);
|
||||
const node: MemoryNode = { bytes: new Uint8Array(0), type: '' };
|
||||
const file = new MemoryFile(name, node);
|
||||
this.children.set(name, file);
|
||||
return file;
|
||||
}
|
||||
|
||||
/** Test helper: synchronously add a file with given bytes. */
|
||||
addFile(name: string, bytes: Uint8Array, type: string = ''): MemoryFile {
|
||||
const node: MemoryNode = { bytes, type };
|
||||
const file = new MemoryFile(name, node);
|
||||
this.children.set(name, file);
|
||||
return file;
|
||||
}
|
||||
|
||||
/** Test helper: synchronously add a subdirectory. */
|
||||
addDir(name: string): MemoryDirectory {
|
||||
const dir = new MemoryDirectory(name);
|
||||
this.children.set(name, dir);
|
||||
return dir;
|
||||
}
|
||||
}
|
||||
|
||||
/** Construct an empty in-memory directory tree. */
|
||||
export function createMemoryDirectory(name: string = ''): DirectoryHandleLike & {
|
||||
addFile(name: string, bytes: Uint8Array, type?: string): FileHandleLike;
|
||||
addDir(name: string): DirectoryHandleLike;
|
||||
} {
|
||||
return new MemoryDirectory(name);
|
||||
}
|
||||
251
packages/shade-files/src/client/streams-bridge.ts
Normal file
251
packages/shade-files/src/client/streams-bridge.ts
Normal file
@@ -0,0 +1,251 @@
|
||||
/**
|
||||
* Client-side bridge between `@shade/files` content RPC ops and the
|
||||
* `@shade/transfer` engine.
|
||||
*
|
||||
* Two responsibilities, mirror-image of `server/streams-bridge.ts`:
|
||||
* 1. **Outbound writes (client → server, > 256 KiB).** When `FileClient.write`
|
||||
* promotes to the streams path, this bridge calls `shade.upload(...)`
|
||||
* with `userMetadata.shadeFilesWriteId = <id>` so the server can
|
||||
* correlate the inbound transfer with the parallel `write` RPC.
|
||||
*
|
||||
* 2. **Inbound reads (server → client, > 256 KiB).** When `FileClient.read`
|
||||
* gets a `{ kind: 'streams', streamId, ... }` RPC response, it asks
|
||||
* this bridge for the matching incoming transfer's plaintext stream.
|
||||
* The bridge subscribes to `shade.onIncomingTransfer` once and, on
|
||||
* each transfer tagged with `userMetadata.shadeFilesReadStreamId`,
|
||||
* **immediately** calls `accept(...)` (the engine rejects chunks that
|
||||
* arrive before accept), pipes plaintext into a TransformStream, and
|
||||
* parks the readable side until the matching read RPC awaits it.
|
||||
*/
|
||||
import type { TransferHandle, TransferProgress } from '@shade/transfer';
|
||||
import { generateRequestId } from '../protocol/correlate.js';
|
||||
import { OperationTimeoutError } from '../schemas/errors.js';
|
||||
import {
|
||||
META_KEY_READ_STREAM_ID,
|
||||
META_KEY_WRITE_ID,
|
||||
type StreamsBridgeShade,
|
||||
} from '../server/streams-bridge.js';
|
||||
|
||||
export interface AwaitReadOptions {
|
||||
/** Sender address — must match `incoming.from` for delivery. */
|
||||
expectedFrom: string;
|
||||
signal: AbortSignal;
|
||||
/** Hard deadline (ms from now). Default 60_000. */
|
||||
timeoutMs?: number;
|
||||
}
|
||||
|
||||
export interface ParkedRead {
|
||||
from: string;
|
||||
/** Plaintext stream. The bridge already accepted the transfer. */
|
||||
readable: ReadableStream<Uint8Array>;
|
||||
/** Resolves when the transfer fully completes (verified sha256 available). */
|
||||
done: Promise<{ sha256: string; bytesSent: number }>;
|
||||
/** Underlying transfer handle — for abort propagation. */
|
||||
handle: TransferHandle;
|
||||
arrivedAt: number;
|
||||
}
|
||||
|
||||
export interface ClientStreamsBridge {
|
||||
/**
|
||||
* Generate a fresh writeId, kick `shade.upload(...)` to `peer` with that
|
||||
* id stamped in `userMetadata`, and return both the id (for the parallel
|
||||
* RPC envelope) and the transfer handle (for `done()`/`abort()`).
|
||||
*/
|
||||
initiateWrite(opts: {
|
||||
peer: string;
|
||||
stream: ReadableStream<Uint8Array>;
|
||||
size: number;
|
||||
contentType?: string;
|
||||
name?: string;
|
||||
signal?: AbortSignal;
|
||||
onProgress?: (p: TransferProgress) => void;
|
||||
}): Promise<{ writeId: string; handle: TransferHandle }>;
|
||||
|
||||
/**
|
||||
* Wait for an inbound transfer carrying `userMetadata.shadeFilesReadStreamId
|
||||
* === streamId` from `expectedFrom`. Resolves with the parked entry whose
|
||||
* `readable` can be consumed by the caller.
|
||||
*/
|
||||
awaitRead(streamId: string, opts: AwaitReadOptions): Promise<ParkedRead>;
|
||||
|
||||
destroy(): Promise<void>;
|
||||
}
|
||||
|
||||
interface PendingReadWaiter {
|
||||
resolve: (parked: ParkedRead) => void;
|
||||
reject: (err: unknown) => void;
|
||||
expectedFrom: string;
|
||||
timer: ReturnType<typeof setTimeout> | null;
|
||||
abortListener: (() => void) | null;
|
||||
signal: AbortSignal;
|
||||
}
|
||||
|
||||
export interface CreateClientStreamsBridgeOptions {
|
||||
/** Default deadline for `awaitRead` if the caller doesn't supply one. */
|
||||
defaultAwaitReadTimeoutMs?: number;
|
||||
/** How long to retain a parked transfer waiting for its RPC. Default 60_000. */
|
||||
parkedReadTtlMs?: number;
|
||||
}
|
||||
|
||||
export async function createClientStreamsBridge(
|
||||
shade: StreamsBridgeShade,
|
||||
options: CreateClientStreamsBridgeOptions = {},
|
||||
): Promise<ClientStreamsBridge> {
|
||||
const parkedReadTtlMs = options.parkedReadTtlMs ?? 60_000;
|
||||
const defaultAwaitTimeoutMs = options.defaultAwaitReadTimeoutMs ?? 60_000;
|
||||
|
||||
const parked = new Map<string, ParkedRead>();
|
||||
const waiters = new Map<string, PendingReadWaiter>();
|
||||
let destroyed = false;
|
||||
|
||||
const unsubscribe = await shade.onIncomingTransfer(async (incoming) => {
|
||||
const readStreamId = incoming.metadata.userMetadata?.[META_KEY_READ_STREAM_ID];
|
||||
if (readStreamId === undefined) return;
|
||||
|
||||
const ts = new TransformStream<Uint8Array, Uint8Array>();
|
||||
let handle: TransferHandle;
|
||||
try {
|
||||
handle = await incoming.accept({
|
||||
output: { kind: 'pipe', pipeTo: ts.writable },
|
||||
});
|
||||
} catch (err) {
|
||||
console.error('[shade-files client streams-bridge] accept failed:', err);
|
||||
return;
|
||||
}
|
||||
|
||||
const arrival: ParkedRead = {
|
||||
from: incoming.from,
|
||||
readable: ts.readable,
|
||||
done: handle.done().then((r) => ({ sha256: r.sha256, bytesSent: r.bytesSent })),
|
||||
handle,
|
||||
arrivedAt: Date.now(),
|
||||
};
|
||||
arrival.done.catch(() => {
|
||||
/* swallow until consumer awaits */
|
||||
});
|
||||
|
||||
const waiter = waiters.get(readStreamId);
|
||||
if (waiter !== undefined) {
|
||||
waiters.delete(readStreamId);
|
||||
cleanupWaiter(waiter);
|
||||
if (incoming.from !== waiter.expectedFrom) {
|
||||
void handle.abort('sender-mismatch').catch(() => undefined);
|
||||
waiter.reject(
|
||||
new Error(
|
||||
`streams-bridge: readStreamId=${readStreamId} delivered by ${incoming.from}, expected ${waiter.expectedFrom}`,
|
||||
),
|
||||
);
|
||||
return;
|
||||
}
|
||||
waiter.resolve(arrival);
|
||||
return;
|
||||
}
|
||||
|
||||
parked.set(readStreamId, arrival);
|
||||
setTimeout(() => {
|
||||
const stale = parked.get(readStreamId);
|
||||
if (stale === arrival) {
|
||||
parked.delete(readStreamId);
|
||||
void handle.abort('rpc-timeout').catch(() => undefined);
|
||||
}
|
||||
}, parkedReadTtlMs).unref?.();
|
||||
});
|
||||
|
||||
function cleanupWaiter(w: PendingReadWaiter): void {
|
||||
if (w.timer !== null) clearTimeout(w.timer);
|
||||
if (w.abortListener !== null) {
|
||||
w.signal.removeEventListener('abort', w.abortListener);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
async initiateWrite(opts) {
|
||||
if (destroyed) throw new Error('streams-bridge: destroyed');
|
||||
const writeId = generateRequestId();
|
||||
const transferOpts: import('@shade/transfer').TransferOptions = {
|
||||
to: opts.peer,
|
||||
input: opts.stream,
|
||||
metadata: {
|
||||
...(opts.name !== undefined ? { name: opts.name } : {}),
|
||||
...(opts.contentType !== undefined ? { contentType: opts.contentType } : {}),
|
||||
sizeBytes: opts.size,
|
||||
userMetadata: { [META_KEY_WRITE_ID]: writeId },
|
||||
},
|
||||
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
|
||||
...(opts.onProgress !== undefined ? { onProgress: opts.onProgress } : {}),
|
||||
};
|
||||
const handle = await shade.upload(transferOpts);
|
||||
return { writeId, handle };
|
||||
},
|
||||
|
||||
async awaitRead(streamId, opts) {
|
||||
if (destroyed) throw new Error('streams-bridge: destroyed');
|
||||
const ready = parked.get(streamId);
|
||||
if (ready !== undefined) {
|
||||
parked.delete(streamId);
|
||||
if (ready.from !== opts.expectedFrom) {
|
||||
void ready.handle.abort('sender-mismatch').catch(() => undefined);
|
||||
throw new Error(
|
||||
`streams-bridge: readStreamId=${streamId} delivered by ${ready.from}, expected ${opts.expectedFrom}`,
|
||||
);
|
||||
}
|
||||
return ready;
|
||||
}
|
||||
if (waiters.has(streamId)) {
|
||||
throw new Error(`streams-bridge: readStreamId=${streamId} already awaited`);
|
||||
}
|
||||
const timeoutMs = opts.timeoutMs ?? defaultAwaitTimeoutMs;
|
||||
return await new Promise<ParkedRead>((resolve, reject) => {
|
||||
const w: PendingReadWaiter = {
|
||||
resolve,
|
||||
reject,
|
||||
expectedFrom: opts.expectedFrom,
|
||||
timer: null,
|
||||
abortListener: null,
|
||||
signal: opts.signal,
|
||||
};
|
||||
w.timer = setTimeout(() => {
|
||||
if (waiters.get(streamId) === w) {
|
||||
waiters.delete(streamId);
|
||||
cleanupWaiter(w);
|
||||
reject(new OperationTimeoutError(`streams-bridge: readStreamId=${streamId} timed out after ${timeoutMs}ms`));
|
||||
}
|
||||
}, timeoutMs);
|
||||
if (opts.signal.aborted) {
|
||||
cleanupWaiter(w);
|
||||
reject(opts.signal.reason ?? new Error('aborted before await'));
|
||||
return;
|
||||
}
|
||||
const onAbort = (): void => {
|
||||
if (waiters.get(streamId) === w) {
|
||||
waiters.delete(streamId);
|
||||
cleanupWaiter(w);
|
||||
reject(opts.signal.reason ?? new Error('aborted'));
|
||||
}
|
||||
};
|
||||
w.abortListener = onAbort;
|
||||
opts.signal.addEventListener('abort', onAbort, { once: true });
|
||||
waiters.set(streamId, w);
|
||||
});
|
||||
},
|
||||
|
||||
async destroy() {
|
||||
if (destroyed) return;
|
||||
destroyed = true;
|
||||
unsubscribe();
|
||||
for (const w of waiters.values()) {
|
||||
cleanupWaiter(w);
|
||||
w.reject(new Error('streams-bridge: destroyed'));
|
||||
}
|
||||
waiters.clear();
|
||||
for (const p of parked.values()) {
|
||||
try {
|
||||
await p.handle.abort('bridge-destroyed');
|
||||
} catch {
|
||||
/* swallow */
|
||||
}
|
||||
}
|
||||
parked.clear();
|
||||
},
|
||||
};
|
||||
}
|
||||
310
packages/shade-files/src/client/upload-directory.ts
Normal file
310
packages/shade-files/src/client/upload-directory.ts
Normal file
@@ -0,0 +1,310 @@
|
||||
/**
|
||||
* Upload an entire local directory tree to a remote peer via a `FileClient`.
|
||||
*
|
||||
* Walks the local `DirectoryHandleLike` lazily, creates remote directories
|
||||
* via `client.mkdir({ recursive: true })`, and uploads each file with
|
||||
* `client.write` (which routes inline or streams based on size). A bounded
|
||||
* concurrency pool keeps memory and inflight RPCs in check.
|
||||
*
|
||||
* Returns a `BulkTransferHandle` whose `events` stream emits `'plan'`,
|
||||
* `'file-start'`, `'file-progress'` (currently emitted at file start +
|
||||
* end), `'file-done'` / `'file-error'`, aggregate `'progress'`, and a
|
||||
* final `'complete'` (or `'abort'`).
|
||||
*/
|
||||
import { posixJoin } from '../utils/path.js';
|
||||
import { runWithConcurrency } from './concurrency.js';
|
||||
import type { FileClient } from './client.js';
|
||||
import {
|
||||
DEFAULT_BULK_CONCURRENCY,
|
||||
MAX_BULK_CONCURRENCY,
|
||||
type BulkOpts,
|
||||
type BulkTransferEvent,
|
||||
type BulkTransferHandle,
|
||||
type BulkTransferResult,
|
||||
type DirectoryHandleLike,
|
||||
type FileHandleLike,
|
||||
} from './directory-types.js';
|
||||
import { CancelledError } from '../schemas/errors.js';
|
||||
|
||||
interface PlannedFile {
|
||||
/** Local file handle. */
|
||||
handle: FileHandleLike;
|
||||
/** Path relative to the upload root. */
|
||||
relativePath: string;
|
||||
/** Absolute remote path. */
|
||||
remoteAbsPath: string;
|
||||
}
|
||||
|
||||
interface PlannedDir {
|
||||
remoteAbsPath: string;
|
||||
}
|
||||
|
||||
export interface UploadDirectoryOptions extends BulkOpts {
|
||||
/**
|
||||
* Pre-create remote directories before uploading files. Default true.
|
||||
* Disable if the server-side `write` already mkdir-p's parents.
|
||||
*/
|
||||
precreateDirs?: boolean;
|
||||
}
|
||||
|
||||
export function uploadDirectory(
|
||||
client: Pick<FileClient, 'write' | 'mkdir'>,
|
||||
local: DirectoryHandleLike,
|
||||
remoteRoot: string,
|
||||
opts: UploadDirectoryOptions = {},
|
||||
): BulkTransferHandle {
|
||||
const concurrency = Math.max(
|
||||
1,
|
||||
Math.min(MAX_BULK_CONCURRENCY, opts.concurrency ?? DEFAULT_BULK_CONCURRENCY),
|
||||
);
|
||||
const precreateDirs = opts.precreateDirs ?? true;
|
||||
const continueOnError = opts.continueOnError ?? false;
|
||||
const externalSignal = opts.signal;
|
||||
|
||||
const internalAbort = new AbortController();
|
||||
const combinedSignal = mergeSignals(externalSignal, internalAbort.signal);
|
||||
|
||||
const events: BulkTransferEvent[] = [];
|
||||
const eventResolvers: ((v: IteratorResult<BulkTransferEvent>) => void)[] = [];
|
||||
let eventsClosed = false;
|
||||
|
||||
function emit(event: BulkTransferEvent): void {
|
||||
if (eventsClosed) return;
|
||||
if (eventResolvers.length > 0) {
|
||||
eventResolvers.shift()!({ value: event, done: false });
|
||||
} else {
|
||||
events.push(event);
|
||||
}
|
||||
}
|
||||
|
||||
function closeEvents(): void {
|
||||
if (eventsClosed) return;
|
||||
eventsClosed = true;
|
||||
while (eventResolvers.length > 0) {
|
||||
eventResolvers.shift()!({ value: undefined as never, done: true });
|
||||
}
|
||||
}
|
||||
|
||||
let resolveDone!: (r: BulkTransferResult) => void;
|
||||
let rejectDone!: (err: unknown) => void;
|
||||
const donePromise = new Promise<BulkTransferResult>((resolve, reject) => {
|
||||
resolveDone = resolve;
|
||||
rejectDone = reject;
|
||||
});
|
||||
|
||||
const startedAt = Date.now();
|
||||
let filesDone = 0;
|
||||
let bytesDone = 0;
|
||||
let bytesTotal = 0;
|
||||
|
||||
// Run the upload pipeline asynchronously.
|
||||
void (async () => {
|
||||
try {
|
||||
// 1. Plan: walk local tree, collect dirs + files
|
||||
const plannedDirs: PlannedDir[] = [];
|
||||
const plannedFiles: PlannedFile[] = [];
|
||||
try {
|
||||
await collect(local, '', remoteRoot, plannedDirs, plannedFiles, combinedSignal);
|
||||
} catch (err) {
|
||||
emit({ type: 'abort', reason: errMsg(err) });
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
return;
|
||||
}
|
||||
|
||||
const totalFiles = plannedFiles.length;
|
||||
bytesTotal = plannedFiles.reduce(
|
||||
(acc, f) => acc + (f.handle as FileHandleLike & { _size?: number })._size!,
|
||||
0,
|
||||
);
|
||||
// Note: bytesTotal is computed lazily in collect() via cached sizes.
|
||||
|
||||
emit({ type: 'plan', totalFiles, totalBytes: bytesTotal });
|
||||
|
||||
if (combinedSignal.aborted) {
|
||||
emit({ type: 'abort', reason: errMsg(combinedSignal.reason) });
|
||||
closeEvents();
|
||||
rejectDone(combinedSignal.reason ?? new CancelledError());
|
||||
return;
|
||||
}
|
||||
|
||||
// 2. Pre-create remote directories sequentially (cheap, avoids races
|
||||
// when many uploads target the same parent).
|
||||
if (precreateDirs) {
|
||||
for (const d of plannedDirs) {
|
||||
if (combinedSignal.aborted) break;
|
||||
try {
|
||||
await client.mkdir(d.remoteAbsPath, { recursive: true });
|
||||
} catch (err) {
|
||||
if (!isAlreadyExistsError(err)) {
|
||||
if (!continueOnError) {
|
||||
emit({ type: 'abort', reason: errMsg(err) });
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Upload files with bounded concurrency
|
||||
try {
|
||||
await runWithConcurrency(
|
||||
asyncIterableOf(plannedFiles),
|
||||
async (planned) => {
|
||||
if (combinedSignal.aborted) {
|
||||
throw new CancelledError('upload aborted');
|
||||
}
|
||||
const file = await planned.handle.getFile();
|
||||
const size = file.size;
|
||||
emit({ type: 'file-start', path: planned.relativePath, size });
|
||||
try {
|
||||
const writeOpts: { contentType?: string; signal?: AbortSignal } = {};
|
||||
if (file.type !== '') writeOpts.contentType = file.type;
|
||||
writeOpts.signal = combinedSignal;
|
||||
if (size === 0) {
|
||||
// Edge case: empty file — write empty buffer.
|
||||
await client.write(planned.remoteAbsPath, new Uint8Array(0), writeOpts);
|
||||
} else {
|
||||
await client.write(
|
||||
planned.remoteAbsPath,
|
||||
{ stream: file.stream(), size, ...(file.type !== '' ? { contentType: file.type } : {}) },
|
||||
writeOpts,
|
||||
);
|
||||
}
|
||||
filesDone++;
|
||||
bytesDone += size;
|
||||
emit({ type: 'file-done', path: planned.relativePath, bytesDone: size });
|
||||
emit({
|
||||
type: 'progress',
|
||||
filesDone,
|
||||
filesTotal: totalFiles,
|
||||
bytesDone,
|
||||
bytesTotal,
|
||||
});
|
||||
} catch (err) {
|
||||
emit({ type: 'file-error', path: planned.relativePath, error: err });
|
||||
throw err;
|
||||
}
|
||||
},
|
||||
{
|
||||
concurrency,
|
||||
signal: combinedSignal,
|
||||
continueOnError,
|
||||
onError: () => {
|
||||
/* already emitted as 'file-error' */
|
||||
},
|
||||
},
|
||||
);
|
||||
} catch (err) {
|
||||
emit({ type: 'abort', reason: errMsg(err) });
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
return;
|
||||
}
|
||||
|
||||
const durationMs = Date.now() - startedAt;
|
||||
emit({ type: 'complete', filesDone, bytesDone, durationMs });
|
||||
closeEvents();
|
||||
resolveDone({ filesDone, bytesDone, durationMs });
|
||||
} catch (err) {
|
||||
// Belt-and-suspenders: any unhandled error reaches here.
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
}
|
||||
})();
|
||||
|
||||
// Suppress unhandled-rejection until consumer awaits done().
|
||||
donePromise.catch(() => {
|
||||
/* deliberate */
|
||||
});
|
||||
|
||||
return {
|
||||
events: {
|
||||
[Symbol.asyncIterator]() {
|
||||
return {
|
||||
next(): Promise<IteratorResult<BulkTransferEvent>> {
|
||||
if (events.length > 0) {
|
||||
return Promise.resolve({ value: events.shift()!, done: false });
|
||||
}
|
||||
if (eventsClosed) {
|
||||
return Promise.resolve({ value: undefined as never, done: true });
|
||||
}
|
||||
return new Promise((resolve) => eventResolvers.push(resolve));
|
||||
},
|
||||
return(): Promise<IteratorResult<BulkTransferEvent>> {
|
||||
// Caller broke out — stop iterating but keep the bulk going.
|
||||
return Promise.resolve({ value: undefined as never, done: true });
|
||||
},
|
||||
};
|
||||
},
|
||||
},
|
||||
async abort(reason) {
|
||||
internalAbort.abort(new CancelledError(reason ?? 'manual abort'));
|
||||
},
|
||||
done: () => donePromise,
|
||||
};
|
||||
}
|
||||
|
||||
// ─── Helpers ─────────────────────────────────────────────────
|
||||
|
||||
async function collect(
|
||||
dir: DirectoryHandleLike,
|
||||
relPrefix: string,
|
||||
remoteRoot: string,
|
||||
plannedDirs: PlannedDir[],
|
||||
plannedFiles: PlannedFile[],
|
||||
signal: AbortSignal,
|
||||
): Promise<void> {
|
||||
for await (const [name, handle] of dir.entries()) {
|
||||
if (signal.aborted) return;
|
||||
const relPath = relPrefix === '' ? name : `${relPrefix}/${name}`;
|
||||
if (handle.kind === 'directory') {
|
||||
const remoteAbs = posixJoin(remoteRoot, relPath);
|
||||
plannedDirs.push({ remoteAbsPath: remoteAbs });
|
||||
await collect(handle as DirectoryHandleLike, relPath, remoteRoot, plannedDirs, plannedFiles, signal);
|
||||
} else {
|
||||
// Cache size on the handle so the planning total is accurate.
|
||||
const file = await (handle as FileHandleLike).getFile();
|
||||
(handle as FileHandleLike & { _size?: number })._size = file.size;
|
||||
plannedFiles.push({
|
||||
handle: handle as FileHandleLike,
|
||||
relativePath: relPath,
|
||||
remoteAbsPath: posixJoin(remoteRoot, relPath),
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function* asyncIterableOf<T>(arr: T[]): AsyncIterable<T> {
|
||||
for (const item of arr) yield item;
|
||||
}
|
||||
|
||||
function isAlreadyExistsError(err: unknown): boolean {
|
||||
if (err === null || typeof err !== 'object') return false;
|
||||
const code = (err as { code?: string; payload?: { code?: string } }).code;
|
||||
if (code === 'SHADE_FS_CONFLICT') return true;
|
||||
const payloadCode = (err as { payload?: { code?: string } }).payload?.code;
|
||||
return payloadCode === 'CONFLICT';
|
||||
}
|
||||
|
||||
function errMsg(err: unknown): string {
|
||||
if (err instanceof Error) return err.message;
|
||||
return String(err ?? 'unknown error');
|
||||
}
|
||||
|
||||
function mergeSignals(
|
||||
external: AbortSignal | undefined,
|
||||
internal: AbortSignal,
|
||||
): AbortSignal {
|
||||
if (external === undefined) return internal;
|
||||
if (external.aborted) return external;
|
||||
const controller = new AbortController();
|
||||
const onAbort = (sig: AbortSignal): void => {
|
||||
controller.abort(sig.reason);
|
||||
};
|
||||
external.addEventListener('abort', () => onAbort(external), { once: true });
|
||||
internal.addEventListener('abort', () => onAbort(internal), { once: true });
|
||||
return controller.signal;
|
||||
}
|
||||
89
packages/shade-files/src/client/walk.ts
Normal file
89
packages/shade-files/src/client/walk.ts
Normal file
@@ -0,0 +1,89 @@
|
||||
/**
|
||||
* Async-iterable directory walker.
|
||||
*
|
||||
* Depth-first by default — yields a directory's entries before descending
|
||||
* into the next sibling. Memory-bounded: never materializes the whole tree;
|
||||
* uses `client.list` page-by-page.
|
||||
*
|
||||
* Designed for arbitrarily-large remote trees: the consumer can break out
|
||||
* of the iterator at any point and the walk halts cleanly.
|
||||
*/
|
||||
import { posixJoin } from '../utils/path.js';
|
||||
import { CancelledError } from '../schemas/errors.js';
|
||||
import type { FileEntry } from '../schemas/file-entry.js';
|
||||
import type { FileClient } from './client.js';
|
||||
|
||||
export interface WalkOpts {
|
||||
/** Hard cap on recursion depth. Default 32. */
|
||||
maxDepth?: number;
|
||||
/** Cancellation. */
|
||||
signal?: AbortSignal;
|
||||
/**
|
||||
* Apply a per-entry filter. Return `false` to skip an entry (and, for
|
||||
* directories, to skip descending into it). Default: include all.
|
||||
*/
|
||||
filter?: (entry: FileEntry, relativePath: string) => boolean;
|
||||
/** Page size hint for `client.list`. Default 200. */
|
||||
pageSize?: number;
|
||||
}
|
||||
|
||||
export interface WalkItem {
|
||||
entry: FileEntry;
|
||||
/** Path relative to the walk root. Includes the entry's own name. */
|
||||
relativePath: string;
|
||||
/** Absolute path on the remote (root + relativePath). */
|
||||
absolutePath: string;
|
||||
/** Depth from root — 1 for direct children. */
|
||||
depth: number;
|
||||
}
|
||||
|
||||
const DEFAULT_MAX_DEPTH = 32;
|
||||
const DEFAULT_PAGE_SIZE = 200;
|
||||
|
||||
export async function* walk(
|
||||
client: Pick<FileClient, 'list'>,
|
||||
rootPath: string,
|
||||
opts: WalkOpts = {},
|
||||
): AsyncIterable<WalkItem> {
|
||||
const maxDepth = opts.maxDepth ?? DEFAULT_MAX_DEPTH;
|
||||
const pageSize = opts.pageSize ?? DEFAULT_PAGE_SIZE;
|
||||
|
||||
function checkAbort(): void {
|
||||
if (opts.signal?.aborted) {
|
||||
throw new CancelledError(
|
||||
opts.signal.reason instanceof Error
|
||||
? opts.signal.reason.message
|
||||
: 'walk aborted',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function* descend(absDir: string, depth: number, relPrefix: string): AsyncIterable<WalkItem> {
|
||||
if (depth > maxDepth) return;
|
||||
let cursor: string | undefined;
|
||||
do {
|
||||
checkAbort();
|
||||
const page = await client.list(absDir, {
|
||||
pageSize,
|
||||
...(cursor !== undefined ? { cursor } : {}),
|
||||
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
|
||||
});
|
||||
for (const entry of page.entries) {
|
||||
checkAbort();
|
||||
const relPath = relPrefix === '' ? entry.name : `${relPrefix}/${entry.name}`;
|
||||
const absPath = posixJoin(absDir, entry.name);
|
||||
if (opts.filter !== undefined && !opts.filter(entry, relPath)) {
|
||||
continue;
|
||||
}
|
||||
const item: WalkItem = { entry, relativePath: relPath, absolutePath: absPath, depth };
|
||||
yield item;
|
||||
if (entry.kind === 'dir') {
|
||||
yield* descend(absPath, depth + 1, relPath);
|
||||
}
|
||||
}
|
||||
cursor = page.hasMore ? page.nextCursor : undefined;
|
||||
} while (cursor !== undefined);
|
||||
}
|
||||
|
||||
yield* descend(rootPath, 1, '');
|
||||
}
|
||||
Reference in New Issue
Block a user