feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
Some checks failed
Test / test (push) Has been cancelled

M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).

@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
  Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
  + per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
  (> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
  / shadeFilesReadStreamId correlation. Server-side TransformStream
  bridges accept inbound transfers immediately (engine rejects chunks
  that arrive before accept) and park the readable for the matching
  RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
  uploadDirectory()/downloadDirectory() with bounded concurrency pool
  (default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
  byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
  policy (traversal + percent-decode hardening), fingerprint gate
  (required/optional/reject), pluggable Ed25519 sig verification with
  ±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
  useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
  entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.

Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
  u16 to u32. The previous u16 silently truncated payloads above
  64 KiB — a hard correctness ceiling that blocked inline file ops
  up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
  only. Cross-platform Kotlin port (android/shade-android) updated to
  match; test-vectors/wire-format.json regenerated.

Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
  Concurrent decryptions of the same peer raced ratchet state
  (manifested as sporadic "Failed to decrypt — wrong key or tampered
  data" under load — surfaced once concurrent uploadDirectory pumped
  many writes in flight). Encrypt was already serialized via
  Shade.send's encryptChains; decrypt is now serialized at the
  manager layer too.

@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
  application-level key/value pairs that round-trip verbatim through
  stream-init plaintext. Used by @shade/files for write/read
  correlation; available to any consumer.

@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
  BackgroundTasks.setHook(name, fn) for runtime hook registration.

Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
  shade-sdk streams-bridge + shade-widgets transfer hooks were
  uncommitted prior to this session. Including them keeps the
  workspace consistent at 0.3.0 since @shade/files depends on them.

Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
  3× stable). Coverage spans unit (inline-threshold + concurrency),
  integration (read-write inline + streams up to 1 MiB, walk +
  upload/download directory, custom-op, metrics, SDK namespace
  end-to-end), and security (tampered-envelope sig verification,
  replay window, fingerprint gate, rate-limit + quota).

Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
  topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
  server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-02 14:00:01 +02:00
parent 7e0f7320a9
commit fa770d3063
198 changed files with 20412 additions and 256 deletions

View File

@@ -0,0 +1,76 @@
import { StreamDecryptionError } from './errors.js';
import { STREAM_NONCE_BYTES } from './nonce.js';
/** Authentication tag length for AES-256-GCM (always 16 bytes). */
export const AEAD_TAG_BYTES = 16;
/**
* SubtleCrypto-style typed buffer source bridge. The DOM `BufferSource` alias
* isn't available without `lib: ["DOM"]`, but SubtleCrypto runtime accepts
* `ArrayBuffer` or `ArrayBufferView` interchangeably. Cast through `unknown`
* to satisfy TS without dragging in DOM lib (matches the pattern in
* `@shade/crypto-web/src/provider.ts:14`).
*/
function bs(u: Uint8Array): ArrayBuffer {
return u as unknown as ArrayBuffer;
}
function resolveSubtle(subtle?: SubtleCrypto): SubtleCrypto {
return subtle ?? globalThis.crypto.subtle;
}
/**
* AES-256-GCM encrypt with a CALLER-SUPPLIED 12-byte nonce.
*
* Unlike `CryptoProvider.aesGcmEncrypt` (which generates a random nonce
* internally), streams require deterministic nonces derived from
* `(laneId, seq)`. Returns the ciphertext concatenated with the 16-byte
* authentication tag (SubtleCrypto's standard layout).
*/
export async function aesGcmEncryptWithNonce(
key: Uint8Array,
nonce: Uint8Array,
plaintext: Uint8Array,
aad: Uint8Array,
subtle?: SubtleCrypto,
): Promise<Uint8Array> {
if (nonce.length !== STREAM_NONCE_BYTES) {
throw new Error(`AES-GCM nonce must be ${STREAM_NONCE_BYTES} bytes`);
}
const s = resolveSubtle(subtle);
const aesKey = await s.importKey('raw', bs(key), 'AES-GCM', false, ['encrypt']);
const out = await s.encrypt(
{ name: 'AES-GCM', iv: bs(nonce), additionalData: bs(aad) },
aesKey,
bs(plaintext),
);
return new Uint8Array(out);
}
/**
* AES-256-GCM decrypt with a CALLER-SUPPLIED nonce. Throws
* `StreamDecryptionError` on authentication failure.
*/
export async function aesGcmDecryptWithNonce(
key: Uint8Array,
nonce: Uint8Array,
ciphertext: Uint8Array,
aad: Uint8Array,
subtle?: SubtleCrypto,
): Promise<Uint8Array> {
if (nonce.length !== STREAM_NONCE_BYTES) {
throw new Error(`AES-GCM nonce must be ${STREAM_NONCE_BYTES} bytes`);
}
const s = resolveSubtle(subtle);
const aesKey = await s.importKey('raw', bs(key), 'AES-GCM', false, ['decrypt']);
try {
const out = await s.decrypt(
{ name: 'AES-GCM', iv: bs(nonce), additionalData: bs(aad) },
aesKey,
bs(ciphertext),
);
return new Uint8Array(out);
} catch {
throw new StreamDecryptionError();
}
}

View File

@@ -0,0 +1,265 @@
import type { CryptoProvider } from '@shade/core';
import { ValidationError } from '@shade/core';
import { decodeStreamChunk } from '@shade/proto';
import { StreamFinishedError, StreamProtocolError } from './errors.js';
import { StreamingSha256 } from './hash.js';
import { StreamReceiver } from './receiver.js';
import { StreamSender } from './sender.js';
import type { DecryptedChunk, EncryptedChunk, LaneInitSpec } from './types.js';
export interface MultiLaneSenderInit {
crypto: CryptoProvider;
subtle?: SubtleCrypto;
streamId: Uint8Array;
streamSecret: Uint8Array;
lanes: LaneInitSpec[];
/** Per-lane resume offsets; defaults to 0 for each lane. */
startSeqByLane?: Map<number, number | bigint>;
}
export interface MultiLaneReceiverInit extends MultiLaneSenderInit {}
/** Per-lane sha256 fingerprint for the stream-finish envelope. */
export interface LaneFingerprint {
laneId: number;
/** sha256 over plaintext bytes carried by this lane in seq order. */
sha256: Uint8Array;
chunkCount: number;
byteCount: number;
}
/**
* Multi-lane stream sender.
*
* Wraps N independent `StreamSender`s sharing the same (streamSecret, streamId)
* and tracks an `overallSha256` over the original plaintext as the consumer
* dispatches bytes to lanes.
*
* The coordinator is partition-AGNOSTIC — the consumer (transfer layer)
* decides which bytes go to which lane. The coordinator only enforces:
* - lane lookup by laneId
* - update of overallSha256 from `appendOverall(...)` (called BEFORE bytes
* are dispatched to lanes, in original byte order)
*/
export class MultiLaneSender {
private readonly senders = new Map<number, StreamSender>();
private readonly overallHasher = new StreamingSha256();
private destroyed = false;
private constructor(senders: StreamSender[]) {
for (const s of senders) {
this.senders.set(s.laneId, s);
}
}
static async create(opts: MultiLaneSenderInit): Promise<MultiLaneSender> {
if (opts.lanes.length === 0) {
throw new ValidationError('at least one lane is required', 'lanes');
}
const seen = new Set<number>();
for (const l of opts.lanes) {
if (seen.has(l.laneId)) {
throw new ValidationError(`duplicate laneId ${l.laneId}`, 'lanes');
}
seen.add(l.laneId);
}
const senders: StreamSender[] = [];
for (const lane of opts.lanes) {
const startSeq = opts.startSeqByLane?.get(lane.laneId);
senders.push(
await StreamSender.create({
crypto: opts.crypto,
...(opts.subtle !== undefined ? { subtle: opts.subtle } : {}),
streamId: opts.streamId,
streamSecret: opts.streamSecret,
laneId: lane.laneId,
...(startSeq !== undefined ? { startSeq } : {}),
}),
);
}
return new MultiLaneSender(senders);
}
/**
* Update the overall (cross-lane, original-order) sha256 hasher.
*
* The consumer must call this once with each plaintext slice IN ORIGINAL
* BYTE ORDER (regardless of which lane the slice ends up on). For range
* partitioning this is straightforward (lanes are contiguous); for
* round-robin the consumer hashes each chunk in the order it was read
* from the input.
*/
appendOverall(plaintext: Uint8Array): void {
if (this.destroyed) throw new StreamFinishedError('MultiLaneSender: destroyed');
this.overallHasher.update(plaintext);
}
/** Encrypt `plaintext` for the given lane. */
async encryptForLane(
laneId: number,
plaintext: Uint8Array,
isLast: boolean,
): Promise<EncryptedChunk> {
const lane = this.senders.get(laneId);
if (lane === undefined) {
throw new StreamProtocolError(`Unknown laneId ${laneId}`);
}
return lane.encryptChunk(plaintext, isLast);
}
/** Resume helper: feed plaintext into a lane's hasher without advancing seq. */
preHashForLane(laneId: number, plaintext: Uint8Array): void {
const lane = this.senders.get(laneId);
if (lane === undefined) {
throw new StreamProtocolError(`Unknown laneId ${laneId}`);
}
lane.preHash(plaintext);
}
/** Snapshot per-lane fingerprints (call once when all lanes are finished). */
getLaneFingerprints(): LaneFingerprint[] {
const out: LaneFingerprint[] = [];
for (const [laneId, sender] of this.senders) {
out.push({
laneId,
sha256: sender.getLaneSha256Digest(),
chunkCount: Number(sender.nextSequence),
byteCount: Number(sender.bytesSent),
});
}
out.sort((a, b) => a.laneId - b.laneId);
return out;
}
/** Snapshot overall sha256 over original-order plaintext. */
getOverallSha256(): Uint8Array {
return this.overallHasher.digest();
}
/** All lanes have emitted their `isLast` chunks. */
get allLanesFinished(): boolean {
for (const lane of this.senders.values()) {
if (!lane.isFinished) return false;
}
return true;
}
destroy(): void {
if (this.destroyed) return;
for (const sender of this.senders.values()) sender.destroy();
this.destroyed = true;
}
}
/**
* Multi-lane stream receiver.
*
* Routes incoming wire envelopes to per-lane `StreamReceiver`s by laneId,
* and exposes a hook for the consumer to update the overall sha256 in
* original byte order (the consumer's reorder logic feeds plaintext here
* after collecting it from individual lanes).
*/
export class MultiLaneReceiver {
private readonly receivers = new Map<number, StreamReceiver>();
private readonly overallHasher = new StreamingSha256();
private destroyed = false;
private constructor(receivers: StreamReceiver[]) {
for (const r of receivers) {
this.receivers.set(r.laneId, r);
}
}
static async create(opts: MultiLaneReceiverInit): Promise<MultiLaneReceiver> {
if (opts.lanes.length === 0) {
throw new ValidationError('at least one lane is required', 'lanes');
}
const seen = new Set<number>();
for (const l of opts.lanes) {
if (seen.has(l.laneId)) {
throw new ValidationError(`duplicate laneId ${l.laneId}`, 'lanes');
}
seen.add(l.laneId);
}
const receivers: StreamReceiver[] = [];
for (const lane of opts.lanes) {
const startSeq = opts.startSeqByLane?.get(lane.laneId);
receivers.push(
await StreamReceiver.create({
crypto: opts.crypto,
...(opts.subtle !== undefined ? { subtle: opts.subtle } : {}),
streamId: opts.streamId,
streamSecret: opts.streamSecret,
laneId: lane.laneId,
...(startSeq !== undefined ? { startSeq } : {}),
}),
);
}
return new MultiLaneReceiver(receivers);
}
/**
* Decrypt one wire-level chunk. Routes to the lane indicated by the
* envelope's laneId. Throws if no receiver exists for that laneId.
*/
async decryptChunk(wireBytes: Uint8Array): Promise<DecryptedChunk & { laneId: number }> {
if (this.destroyed) throw new StreamFinishedError('MultiLaneReceiver: destroyed');
const env = decodeStreamChunk(wireBytes);
const lane = this.receivers.get(env.laneId);
if (lane === undefined) {
throw new StreamProtocolError(`Unknown laneId ${env.laneId}`);
}
const dec = await lane.decryptChunk(wireBytes);
return { ...dec, laneId: env.laneId };
}
/**
* Update the overall sha256 hasher with plaintext IN ORIGINAL BYTE ORDER.
* The consumer's reorder logic decides when to call this (as bytes leave
* the reorder buffer in original order, OR for range-partitioning,
* concatenated lane outputs in laneId order).
*/
appendOverall(plaintext: Uint8Array): void {
if (this.destroyed) throw new StreamFinishedError('MultiLaneReceiver: destroyed');
this.overallHasher.update(plaintext);
}
/** Snapshot per-lane fingerprints (call once when all lanes are finished). */
getLaneFingerprints(): LaneFingerprint[] {
const out: LaneFingerprint[] = [];
for (const [laneId, receiver] of this.receivers) {
out.push({
laneId,
sha256: receiver.getLaneSha256Digest(),
chunkCount: Number(receiver.nextExpectedSequence),
byteCount: Number(receiver.bytesReceived),
});
}
out.sort((a, b) => a.laneId - b.laneId);
return out;
}
/** Snapshot overall sha256 over original-order plaintext. */
getOverallSha256(): Uint8Array {
return this.overallHasher.digest();
}
/** All lanes have received their `isLast` chunks. */
get allLanesFinished(): boolean {
for (const lane of this.receivers.values()) {
if (!lane.isFinished) return false;
}
return true;
}
/** Get a specific lane receiver (for inspection/resume). */
getLane(laneId: number): StreamReceiver | undefined {
return this.receivers.get(laneId);
}
destroy(): void {
if (this.destroyed) return;
for (const receiver of this.receivers.values()) receiver.destroy();
this.destroyed = true;
}
}

View File

@@ -0,0 +1,103 @@
/**
* Plaintext schemas for stream control messages.
*
* Stream-init / -finish / -abort / -resume ride the existing 0x02 ratchet
* envelope — Double Ratchet AEAD already authenticates them and rejects
* replays. Each carries a JSON plaintext with a `kind` discriminator.
* Stream-chunk uses dedicated wire type 0x11 (see `@shade/proto/wire.ts`).
*/
import { ValidationError } from '@shade/core';
import type { LaneInitSpec, StreamMetadata } from './types.js';
export type StreamControlKind =
| 'shade.stream-init/v1'
| 'shade.stream-finish/v1'
| 'shade.stream-abort/v1'
| 'shade.stream-resume-request/v1'
| 'shade.stream-resume-state/v1';
export interface StreamInitMessage {
kind: 'shade.stream-init/v1';
/** base64url-encoded 16-byte streamId. */
streamId: string;
/** base64url-encoded 32-byte streamSecret. */
streamSecret: string;
metadata: StreamMetadata;
lanes: LaneInitSpec[];
}
export interface StreamFinishMessage {
kind: 'shade.stream-finish/v1';
streamId: string;
/** Per-lane integrity fingerprints, base64-encoded sha256 of plaintext bytes the lane carried. */
laneSha256: Array<{
laneId: number;
sha256: string;
chunkCount: number;
byteCount: number;
}>;
/** base64-encoded sha256 over the original plaintext file in original byte order. */
overallSha256: string;
finishedAt: number;
}
export type StreamAbortReason = 'sender-cancel' | 'receiver-cancel' | 'fatal-error';
export interface StreamAbortMessage {
kind: 'shade.stream-abort/v1';
streamId: string;
reason: StreamAbortReason;
message?: string;
abortedAt: number;
}
export interface StreamResumeRequestMessage {
kind: 'shade.stream-resume-request/v1';
streamId: string;
requestedAt: number;
}
export interface StreamResumeStateMessage {
kind: 'shade.stream-resume-state/v1';
streamId: string;
/** lastSeqAcked = -1 means no chunk for this lane has been received yet. */
lanes: Array<{ laneId: number; lastSeqAcked: number }>;
resumedAt: number;
}
export type StreamControlMessage =
| StreamInitMessage
| StreamFinishMessage
| StreamAbortMessage
| StreamResumeRequestMessage
| StreamResumeStateMessage;
/** Type guard: does the value look like a stream control message? */
export function isStreamControlMessage(value: unknown): value is StreamControlMessage {
if (typeof value !== 'object' || value === null) return false;
const kind = (value as { kind?: unknown }).kind;
if (typeof kind !== 'string') return false;
return kind.startsWith('shade.stream-');
}
/** Parse JSON plaintext; throw ValidationError on shape mismatch. */
export function parseStreamControl(plaintext: string): StreamControlMessage {
let parsed: unknown;
try {
parsed = JSON.parse(plaintext);
} catch (err) {
throw new ValidationError(
`stream control plaintext is not valid JSON: ${(err as Error).message}`,
'plaintext',
);
}
if (!isStreamControlMessage(parsed)) {
throw new ValidationError('plaintext is not a stream control message', 'plaintext');
}
return parsed;
}
/** Encode a stream control message as JSON; throws on circular refs. */
export function encodeStreamControl(msg: StreamControlMessage): string {
return JSON.stringify(msg);
}

View File

@@ -0,0 +1,53 @@
import { ShadeError } from '@shade/core';
export class StreamError extends ShadeError {
constructor(code: string, message: string) {
super(code, message);
this.name = 'StreamError';
}
}
export class StreamProtocolError extends StreamError {
constructor(message: string) {
super('SHADE_STREAM_PROTOCOL', message);
this.name = 'StreamProtocolError';
}
}
export class StreamIntegrityError extends StreamError {
constructor(message: string) {
super('SHADE_STREAM_INTEGRITY', message);
this.name = 'StreamIntegrityError';
}
}
export class StreamReplayError extends StreamError {
constructor(message = 'Stream chunk replay detected') {
super('SHADE_STREAM_REPLAY', message);
this.name = 'StreamReplayError';
}
}
export class StreamOutOfOrderError extends StreamError {
constructor(expected: number, received: number) {
super(
'SHADE_STREAM_OUT_OF_ORDER',
`Out-of-order chunk: expected seq=${expected}, got ${received}`,
);
this.name = 'StreamOutOfOrderError';
}
}
export class StreamDecryptionError extends StreamError {
constructor(message = 'Stream chunk authenticated decryption failed') {
super('SHADE_STREAM_DECRYPTION', message);
this.name = 'StreamDecryptionError';
}
}
export class StreamFinishedError extends StreamError {
constructor(message = 'Stream is already finished or aborted') {
super('SHADE_STREAM_FINISHED', message);
this.name = 'StreamFinishedError';
}
}

View File

@@ -0,0 +1,41 @@
import { sha256 } from '@noble/hashes/sha2.js';
/**
* Streaming SHA-256 wrapper.
*
* SubtleCrypto exposes only one-shot `digest`. Streams need incremental
* hashing so the receiver doesn't materialize the full plaintext before
* verifying integrity (NF2: O(chunkSize) memory). `@noble/hashes/sha2`
* provides a streaming API that works in Bun, Node, and browsers without
* any native bindings.
*/
export class StreamingSha256 {
private h = sha256.create();
private finalized = false;
/** Feed bytes into the hasher. No-op on empty input. */
update(data: Uint8Array): this {
if (this.finalized) throw new Error('StreamingSha256: already finalized');
if (data.length > 0) this.h.update(data);
return this;
}
/**
* Produce the 32-byte digest. After `digest()` the hasher is finalized;
* subsequent `update()` calls will throw.
*/
digest(): Uint8Array {
this.finalized = true;
return this.h.digest();
}
/** Whether `digest()` has been called. */
get isFinalized(): boolean {
return this.finalized;
}
}
/** Convenience: hash a single buffer in one shot. */
export function sha256Once(data: Uint8Array): Uint8Array {
return sha256(data);
}

View File

@@ -0,0 +1,47 @@
import type { CryptoProvider } from '@shade/core';
import { ValidationError } from '@shade/core';
export const STREAM_ID_BYTES = 16;
export const STREAM_SECRET_BYTES = 32;
/** Generate a fresh 16-byte random streamId. */
export function generateStreamId(crypto: CryptoProvider): Uint8Array {
return crypto.randomBytes(STREAM_ID_BYTES);
}
/** Generate a fresh 32-byte random streamSecret. */
export function generateStreamSecret(crypto: CryptoProvider): Uint8Array {
return crypto.randomBytes(STREAM_SECRET_BYTES);
}
/** Encode a streamId as URL-safe base64 (no padding). */
export function streamIdToString(streamId: Uint8Array): string {
if (streamId.length !== STREAM_ID_BYTES) {
throw new ValidationError(`streamId must be ${STREAM_ID_BYTES} bytes`, 'streamId');
}
return base64UrlEncode(streamId);
}
/** Decode a URL-safe base64 streamId back to bytes. */
export function streamIdFromString(s: string): Uint8Array {
const bytes = base64UrlDecode(s);
if (bytes.length !== STREAM_ID_BYTES) {
throw new ValidationError(`streamId must decode to ${STREAM_ID_BYTES} bytes`, 'streamId');
}
return bytes;
}
function base64UrlEncode(bytes: Uint8Array): string {
let bin = '';
for (let i = 0; i < bytes.length; i++) bin += String.fromCharCode(bytes[i]!);
return btoa(bin).replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, '');
}
function base64UrlDecode(s: string): Uint8Array {
const padded = s.replace(/-/g, '+').replace(/_/g, '/');
const pad = padded.length % 4 === 0 ? '' : '='.repeat(4 - (padded.length % 4));
const bin = atob(padded + pad);
const out = new Uint8Array(bin.length);
for (let i = 0; i < bin.length; i++) out[i] = bin.charCodeAt(i);
return out;
}

View File

@@ -0,0 +1,12 @@
export * from './errors.js';
export * from './types.js';
export * from './ids.js';
export * from './kdf.js';
export * from './nonce.js';
export * from './aead.js';
export * from './hash.js';
export * from './envelope.js';
export * from './sender.js';
export * from './receiver.js';
export * from './partition.js';
export * from './coordinator.js';

View File

@@ -0,0 +1,66 @@
import type { CryptoProvider } from '@shade/core';
import { ValidationError } from '@shade/core';
import { STREAM_ID_BYTES, STREAM_SECRET_BYTES } from './ids.js';
const TEXT = new TextEncoder();
const STREAM_KEY_INFO = TEXT.encode('shade-stream/v1\0master');
const LANE_KEY_INFO_PREFIX = TEXT.encode('shade-stream/v1\0lane\0');
const STREAM_KEY_BYTES = 32;
export const LANE_KEY_BYTES = 32;
/**
* Derive the master streamKey from a streamSecret + streamId.
*
* streamKey = HKDF(ikm=streamSecret, salt=streamId,
* info="shade-stream/v1\0master", length=32)
*
* The streamKey is NEVER used to encrypt chunks directly — it is a root for
* per-lane key derivation (see `deriveLaneKey`).
*/
export async function deriveStreamKey(
crypto: CryptoProvider,
streamSecret: Uint8Array,
streamId: Uint8Array,
): Promise<Uint8Array> {
if (streamSecret.length !== STREAM_SECRET_BYTES) {
throw new ValidationError(
`streamSecret must be ${STREAM_SECRET_BYTES} bytes`,
'streamSecret',
);
}
if (streamId.length !== STREAM_ID_BYTES) {
throw new ValidationError(`streamId must be ${STREAM_ID_BYTES} bytes`, 'streamId');
}
return crypto.hkdf(streamSecret, streamId, STREAM_KEY_INFO, STREAM_KEY_BYTES);
}
/**
* Derive a lane-specific AEAD key.
*
* laneKey[i] = HKDF(ikm=streamKey, salt=streamId,
* info="shade-stream/v1\0lane\0" || u32_be(laneId), length=32)
*
* Distinct laneIds produce independent keys; receiver derives the same key
* given the same (streamSecret, streamId, laneId).
*/
export async function deriveLaneKey(
crypto: CryptoProvider,
streamKey: Uint8Array,
streamId: Uint8Array,
laneId: number,
): Promise<Uint8Array> {
if (streamKey.length !== STREAM_KEY_BYTES) {
throw new ValidationError(`streamKey must be ${STREAM_KEY_BYTES} bytes`, 'streamKey');
}
if (streamId.length !== STREAM_ID_BYTES) {
throw new ValidationError(`streamId must be ${STREAM_ID_BYTES} bytes`, 'streamId');
}
if (!Number.isInteger(laneId) || laneId < 0 || laneId > 0xffff_ffff) {
throw new ValidationError(`laneId must fit in u32: ${laneId}`, 'laneId');
}
const info = new Uint8Array(LANE_KEY_INFO_PREFIX.length + 4);
info.set(LANE_KEY_INFO_PREFIX, 0);
new DataView(info.buffer).setUint32(LANE_KEY_INFO_PREFIX.length, laneId, false);
return crypto.hkdf(streamKey, streamId, info, LANE_KEY_BYTES);
}

View File

@@ -0,0 +1,62 @@
import { ValidationError } from '@shade/core';
export const STREAM_NONCE_BYTES = 12;
/** Maximum chunk seq value (u64 max). Hard-spec'd hard limit. */
export const MAX_SEQ = 0xffff_ffff_ffff_ffffn;
/**
* Construct the deterministic AES-GCM nonce for a stream chunk.
*
* nonce[0..4] = u32_be(laneId)
* nonce[4..12] = u64_be(seq)
*
* Per (laneId, seq) is unique — combined with the lane-specific key, this
* guarantees AES-GCM nonce-uniqueness even across multiple parallel lanes.
*/
export function buildChunkNonce(laneId: number, seq: number | bigint): Uint8Array {
if (!Number.isInteger(laneId) || laneId < 0 || laneId > 0xffff_ffff) {
throw new ValidationError(`laneId must fit in u32: ${laneId}`, 'laneId');
}
const seqBig = typeof seq === 'bigint' ? seq : BigInt(seq);
if (seqBig < 0n || seqBig > MAX_SEQ) {
throw new ValidationError(`seq must fit in u64 (>= 0): ${seq}`, 'seq');
}
const out = new Uint8Array(STREAM_NONCE_BYTES);
const view = new DataView(out.buffer);
view.setUint32(0, laneId, false);
view.setBigUint64(4, seqBig, false);
return out;
}
/**
* Build the AAD bound to a stream chunk. Computed implicitly on both sides
* from the chunk header (never transmitted as-is). Tampering with any header
* field invalidates the AEAD tag.
*
* aad = streamId(16) || u32_be(laneId) || u64_be(seq) || u8(isLast)
*/
export function buildChunkAad(
streamId: Uint8Array,
laneId: number,
seq: number | bigint,
isLast: boolean,
): Uint8Array {
if (streamId.length !== 16) {
throw new ValidationError('streamId must be 16 bytes', 'streamId');
}
if (!Number.isInteger(laneId) || laneId < 0 || laneId > 0xffff_ffff) {
throw new ValidationError(`laneId must fit in u32: ${laneId}`, 'laneId');
}
const seqBig = typeof seq === 'bigint' ? seq : BigInt(seq);
if (seqBig < 0n || seqBig > MAX_SEQ) {
throw new ValidationError(`seq must fit in u64 (>= 0): ${seq}`, 'seq');
}
const out = new Uint8Array(16 + 4 + 8 + 1);
out.set(streamId, 0);
const view = new DataView(out.buffer);
view.setUint32(16, laneId, false);
view.setBigUint64(20, seqBig, false);
out[28] = isLast ? 0x01 : 0x00;
return out;
}

View File

@@ -0,0 +1,87 @@
import { ValidationError } from '@shade/core';
import type { LaneInitSpec, LanePartition } from './types.js';
/** Build a range-partition plan: contiguous byte ranges, one per lane. */
export function planRangePartition(
totalBytes: number,
laneCount: number,
): LaneInitSpec[] {
if (!Number.isInteger(totalBytes) || totalBytes < 0) {
throw new ValidationError(`totalBytes must be a non-negative integer`, 'totalBytes');
}
if (!Number.isInteger(laneCount) || laneCount < 1) {
throw new ValidationError(`laneCount must be >= 1`, 'laneCount');
}
const lanes: LaneInitSpec[] = [];
// Use ceil so the first lanes get the extra byte when not evenly divisible.
// Each lane's start = previous end. Last lane's end = totalBytes.
const baseSize = Math.floor(totalBytes / laneCount);
const remainder = totalBytes - baseSize * laneCount;
let cursor = 0;
for (let i = 0; i < laneCount; i++) {
const extra = i < remainder ? 1 : 0;
const size = baseSize + extra;
const startByte = cursor;
const endByte = cursor + size;
lanes.push({
laneId: i,
partition: { kind: 'range', startByte, endByte, startChunk: 0 },
});
cursor = endByte;
}
return lanes;
}
/** Build a round-robin partition plan: chunk i goes to lane (i mod count). */
export function planRoundRobinPartition(laneCount: number): LaneInitSpec[] {
if (!Number.isInteger(laneCount) || laneCount < 1) {
throw new ValidationError(`laneCount must be >= 1`, 'laneCount');
}
const lanes: LaneInitSpec[] = [];
for (let i = 0; i < laneCount; i++) {
lanes.push({
laneId: i,
partition: { kind: 'round-robin', lane: i, count: laneCount },
});
}
return lanes;
}
/**
* Split an arbitrary byte range into chunkSize-sized slices.
* Returns an array of [startByte, endByte) tuples for each chunk.
*/
export function chunkRange(
startByte: number,
endByte: number,
chunkSize: number,
): Array<{ start: number; end: number }> {
if (chunkSize <= 0) {
throw new ValidationError(`chunkSize must be positive`, 'chunkSize');
}
const out: Array<{ start: number; end: number }> = [];
if (endByte === startByte) {
// Empty range: emit one empty chunk so isLast can be carried.
out.push({ start: startByte, end: startByte });
return out;
}
for (let off = startByte; off < endByte; off += chunkSize) {
out.push({ start: off, end: Math.min(off + chunkSize, endByte) });
}
return out;
}
/**
* Validate that a `LanePartition` matches the global stream parameters.
* Used by the receiver to detect partition-mismatch on resume.
*/
export function partitionsEqual(a: LanePartition, b: LanePartition): boolean {
if (a.kind !== b.kind) return false;
if (a.kind === 'range' && b.kind === 'range') {
return a.startByte === b.startByte && a.endByte === b.endByte && a.startChunk === b.startChunk;
}
if (a.kind === 'round-robin' && b.kind === 'round-robin') {
return a.lane === b.lane && a.count === b.count;
}
return false;
}

View File

@@ -0,0 +1,169 @@
import type { CryptoProvider } from '@shade/core';
import { ValidationError, constantTimeEqual } from '@shade/core';
import { decodeStreamChunk } from '@shade/proto';
import { aesGcmDecryptWithNonce } from './aead.js';
import {
StreamFinishedError,
StreamOutOfOrderError,
StreamProtocolError,
StreamReplayError,
} from './errors.js';
import { StreamingSha256 } from './hash.js';
import { deriveLaneKey, deriveStreamKey } from './kdf.js';
import { buildChunkAad, buildChunkNonce, MAX_SEQ } from './nonce.js';
import type { DecryptedChunk } from './types.js';
export interface StreamReceiverInit {
crypto: CryptoProvider;
subtle?: SubtleCrypto;
streamId: Uint8Array;
streamSecret: Uint8Array;
laneId: number;
/** First seq this receiver will accept; defaults to 0. Used for resume. */
startSeq?: number | bigint;
}
/**
* Per-lane stream receiver state machine.
*
* Verifies AEAD, enforces strict in-order seq (rejects replay + out-of-order),
* and updates a running lane sha256 over the decrypted plaintext.
*/
export class StreamReceiver {
private constructor(
private readonly subtle: SubtleCrypto | undefined,
private readonly crypto: CryptoProvider,
private readonly streamIdBytes: Uint8Array,
public readonly laneId: number,
private laneKey: Uint8Array | null,
private expectedSeq: bigint,
private readonly hasher: StreamingSha256,
private finished: boolean,
private bytesReceivedInternal: bigint,
) {}
/** 16-byte streamId this receiver decrypts under. Defensive copy. */
get streamId(): Uint8Array {
return this.streamIdBytes.slice();
}
static async create(opts: StreamReceiverInit): Promise<StreamReceiver> {
const streamKey = await deriveStreamKey(opts.crypto, opts.streamSecret, opts.streamId);
try {
const laneKey = await deriveLaneKey(opts.crypto, streamKey, opts.streamId, opts.laneId);
const startSeq =
opts.startSeq === undefined
? 0n
: typeof opts.startSeq === 'bigint'
? opts.startSeq
: BigInt(opts.startSeq);
if (startSeq < 0n || startSeq > MAX_SEQ) {
throw new ValidationError(`startSeq out of range: ${opts.startSeq}`, 'startSeq');
}
return new StreamReceiver(
opts.subtle,
opts.crypto,
opts.streamId.slice(),
opts.laneId,
laneKey,
startSeq,
new StreamingSha256(),
false,
0n,
);
} finally {
opts.crypto.zeroize(streamKey);
}
}
/**
* Decrypt and authenticate a wire-level chunk envelope. Throws on:
* - mismatched streamId / laneId
* - replayed seq (already accepted)
* - out-of-order seq (gap or backwards)
* - tampered nonce / ciphertext / aad
* - any chunk after `isLast`
*/
async decryptChunk(wireBytes: Uint8Array): Promise<DecryptedChunk> {
if (this.finished) {
throw new StreamFinishedError('StreamReceiver: lane already finished');
}
if (this.laneKey === null) {
throw new StreamFinishedError('StreamReceiver: destroyed');
}
const env = decodeStreamChunk(wireBytes);
if (!constantTimeEqual(env.streamId, this.streamIdBytes)) {
throw new StreamProtocolError('stream-chunk streamId does not match this receiver');
}
if (env.laneId !== this.laneId) {
throw new StreamProtocolError(
`stream-chunk laneId=${env.laneId} routed to laneId=${this.laneId}`,
);
}
const seqBig = typeof env.seq === 'bigint' ? env.seq : BigInt(env.seq);
if (seqBig < this.expectedSeq) {
throw new StreamReplayError(`Replay: seq=${seqBig}, already accepted < ${this.expectedSeq}`);
}
if (seqBig > this.expectedSeq) {
throw new StreamOutOfOrderError(Number(this.expectedSeq), Number(seqBig));
}
// Defense-in-depth: the wire nonce MUST equal the deterministically derived one.
const expectedNonce = buildChunkNonce(this.laneId, seqBig);
if (!constantTimeEqual(env.nonce, expectedNonce)) {
throw new StreamProtocolError('stream-chunk nonce mismatch');
}
if (env.aad.length !== 0) {
throw new StreamProtocolError('stream-chunk aad must be empty in v0.2.0');
}
const aad = buildChunkAad(this.streamIdBytes, this.laneId, seqBig, env.isLast);
const plaintext = await aesGcmDecryptWithNonce(
this.laneKey,
env.nonce,
env.ciphertext,
aad,
this.subtle,
);
this.hasher.update(plaintext);
this.bytesReceivedInternal += BigInt(plaintext.length);
this.expectedSeq = seqBig + 1n;
if (env.isLast) this.finished = true;
return { plaintext, seq: Number(seqBig), isLast: env.isLast };
}
/** Snapshot the lane sha256 digest. Hasher is frozen after this call. */
getLaneSha256Digest(): Uint8Array {
return this.hasher.digest();
}
/** Total plaintext bytes accepted so far in this lane. */
get bytesReceived(): bigint {
return this.bytesReceivedInternal;
}
/** Next sequence number this receiver expects. */
get nextExpectedSequence(): bigint {
return this.expectedSeq;
}
/** Has this lane received its `isLast` chunk? */
get isFinished(): boolean {
return this.finished;
}
/** Zero the lane key in memory. After destroy, decrypt calls throw. */
destroy(): void {
if (this.laneKey !== null) {
this.crypto.zeroize(this.laneKey);
this.laneKey = null;
}
this.finished = true;
}
}

View File

@@ -0,0 +1,165 @@
import type { CryptoProvider } from '@shade/core';
import { ValidationError } from '@shade/core';
import { encodeStreamChunk } from '@shade/proto';
import { aesGcmEncryptWithNonce } from './aead.js';
import { StreamFinishedError } from './errors.js';
import { StreamingSha256 } from './hash.js';
import { deriveLaneKey, deriveStreamKey } from './kdf.js';
import { buildChunkAad, buildChunkNonce, MAX_SEQ } from './nonce.js';
import type { EncryptedChunk, StreamChunkEnvelope } from './types.js';
export interface StreamSenderInit {
crypto: CryptoProvider;
/** Optional SubtleCrypto for AEAD; defaults to globalThis.crypto.subtle. */
subtle?: SubtleCrypto;
streamId: Uint8Array;
/** 32-byte streamSecret. The lane key is derived internally and stays in this instance. */
streamSecret: Uint8Array;
laneId: number;
/** First seq this sender will emit; defaults to 0. Used for resume. */
startSeq?: number | bigint;
}
/**
* Per-lane stream sender state machine.
*
* Encapsulates the lane-specific AEAD key + monotonic seq counter + running
* lane-sha256. Multiple senders (one per lane) sharing the same
* `(streamSecret, streamId)` make up a parallel transfer.
*/
export class StreamSender {
private constructor(
private readonly subtle: SubtleCrypto | undefined,
private readonly crypto: CryptoProvider,
private readonly streamIdBytes: Uint8Array,
public readonly laneId: number,
private laneKey: Uint8Array | null,
private nextSeq: bigint,
private readonly hasher: StreamingSha256,
private finished: boolean,
private bytesSentInternal: bigint,
) {}
/** 16-byte streamId this sender encrypts under. Defensive copy. */
get streamId(): Uint8Array {
return this.streamIdBytes.slice();
}
static async create(opts: StreamSenderInit): Promise<StreamSender> {
const streamKey = await deriveStreamKey(opts.crypto, opts.streamSecret, opts.streamId);
try {
const laneKey = await deriveLaneKey(opts.crypto, streamKey, opts.streamId, opts.laneId);
const startSeq =
opts.startSeq === undefined
? 0n
: typeof opts.startSeq === 'bigint'
? opts.startSeq
: BigInt(opts.startSeq);
if (startSeq < 0n || startSeq > MAX_SEQ) {
throw new ValidationError(`startSeq out of range: ${opts.startSeq}`, 'startSeq');
}
return new StreamSender(
opts.subtle,
opts.crypto,
opts.streamId.slice(),
opts.laneId,
laneKey,
startSeq,
new StreamingSha256(),
false,
0n,
);
} finally {
opts.crypto.zeroize(streamKey);
}
}
/**
* Encrypt one plaintext chunk. Updates lane sha256 with the plaintext bytes
* BEFORE encryption (so receiver can independently verify by hashing
* decrypted plaintext in the same order).
*/
async encryptChunk(plaintext: Uint8Array, isLast: boolean): Promise<EncryptedChunk> {
if (this.finished) {
throw new StreamFinishedError('StreamSender: lane already finished');
}
if (this.laneKey === null) {
throw new StreamFinishedError('StreamSender: destroyed');
}
if (this.nextSeq > MAX_SEQ) {
throw new ValidationError('seq overflow', 'seq');
}
const seq = this.nextSeq;
const nonce = buildChunkNonce(this.laneId, seq);
const aad = buildChunkAad(this.streamIdBytes, this.laneId, seq, isLast);
const ciphertext = await aesGcmEncryptWithNonce(
this.laneKey,
nonce,
plaintext,
aad,
this.subtle,
);
this.hasher.update(plaintext);
this.bytesSentInternal += BigInt(plaintext.length);
this.nextSeq = seq + 1n;
const envelope: StreamChunkEnvelope = {
streamId: this.streamIdBytes,
laneId: this.laneId,
seq,
isLast,
nonce,
aad: new Uint8Array(0),
ciphertext,
};
const bytes = encodeStreamChunk(envelope);
if (isLast) this.finished = true;
return { envelope, bytes, seq: Number(seq) };
}
/** Snapshot the lane sha256 digest. Hasher is frozen after this call. */
getLaneSha256Digest(): Uint8Array {
return this.hasher.digest();
}
/**
* Feed plaintext into the lane sha256 WITHOUT advancing seq or
* producing wire bytes. Used by resume flows to re-build the running
* lane hash from already-shipped bytes (since `@noble/hashes` v2 doesn't
* expose serializable state in v0.2.0).
*/
preHash(plaintext: Uint8Array): void {
if (this.finished) {
throw new StreamFinishedError('StreamSender: lane already finished');
}
this.hasher.update(plaintext);
}
/** Number of plaintext bytes encrypted so far in this lane. */
get bytesSent(): bigint {
return this.bytesSentInternal;
}
/** Next sequence number this sender will emit. */
get nextSequence(): bigint {
return this.nextSeq;
}
/** Has this lane emitted its `isLast` chunk? */
get isFinished(): boolean {
return this.finished;
}
/** Zero the lane key in memory. After destroy, encrypt calls throw. */
destroy(): void {
if (this.laneKey !== null) {
this.crypto.zeroize(this.laneKey);
this.laneKey = null;
}
this.finished = true;
}
}

View File

@@ -0,0 +1,72 @@
/**
* Public types for @shade/streams.
*
* Higher-level wrappers for transfer-orchestration (parallel lanes, resume,
* progress, etc.) live in @shade/transfer.
*/
/** Plaintext metadata sent in a stream-init control envelope. */
export interface StreamMetadata {
name?: string;
/** Total plaintext size in bytes. Omit for streams of unknown length. */
sizeBytes?: number;
contentType?: string;
/** Plaintext bytes per chunk for this stream. */
chunkSize: number;
/** Total chunk count across all lanes. Omit for unknown-length streams. */
totalChunks?: number;
/** Sender's local clock at init time (advisory; never used for security decisions). */
sentAt: number;
/**
* Optional application-level metadata, JSON-stringified-safe key/value
* pairs. Round-tripped verbatim through stream-init plaintext. Used by
* higher layers (e.g. `@shade/files` writes a `shadeFilesWriteId` here so
* a server-side bridge can correlate an inbound transfer with a pending
* RPC). The transport itself does not interpret these values.
*/
userMetadata?: Record<string, string>;
}
/** Per-lane partition assignment carried in stream-init. */
export type LanePartition =
| {
kind: 'range';
/** Inclusive start byte of this lane's slice. */
startByte: number;
/** Exclusive end byte of this lane's slice. */
endByte: number;
/** First chunk seq this lane's region begins at (always 0 for per-lane numbering). */
startChunk: number;
}
| {
kind: 'round-robin';
/** Chunk i goes to lane (i mod count). */
lane: number;
count: number;
};
/** Per-lane state included in stream-init plaintext. laneKey is NOT shipped — it is derived. */
export interface LaneInitSpec {
laneId: number;
partition: LanePartition;
}
import type { StreamChunkWire } from '@shade/proto';
/** Wire-decoded stream-chunk envelope (alias for @shade/proto's `StreamChunkWire`). */
export type StreamChunkEnvelope = StreamChunkWire;
/** Result returned from `StreamSender.encryptChunk`. */
export interface EncryptedChunk {
envelope: StreamChunkEnvelope;
/** Raw bytes ready to ship — output of `encodeStreamChunk`. */
bytes: Uint8Array;
seq: number;
}
/** Result returned from `StreamReceiver.decryptChunk`. */
export interface DecryptedChunk {
plaintext: Uint8Array;
seq: number;
isLast: boolean;
}