feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
Some checks failed
Test / test (push) Has been cancelled
Some checks failed
Test / test (push) Has been cancelled
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
76
packages/shade-streams/src/aead.ts
Normal file
76
packages/shade-streams/src/aead.ts
Normal file
@@ -0,0 +1,76 @@
|
||||
import { StreamDecryptionError } from './errors.js';
|
||||
import { STREAM_NONCE_BYTES } from './nonce.js';
|
||||
|
||||
/** Authentication tag length for AES-256-GCM (always 16 bytes). */
|
||||
export const AEAD_TAG_BYTES = 16;
|
||||
|
||||
/**
|
||||
* SubtleCrypto-style typed buffer source bridge. The DOM `BufferSource` alias
|
||||
* isn't available without `lib: ["DOM"]`, but SubtleCrypto runtime accepts
|
||||
* `ArrayBuffer` or `ArrayBufferView` interchangeably. Cast through `unknown`
|
||||
* to satisfy TS without dragging in DOM lib (matches the pattern in
|
||||
* `@shade/crypto-web/src/provider.ts:14`).
|
||||
*/
|
||||
function bs(u: Uint8Array): ArrayBuffer {
|
||||
return u as unknown as ArrayBuffer;
|
||||
}
|
||||
|
||||
function resolveSubtle(subtle?: SubtleCrypto): SubtleCrypto {
|
||||
return subtle ?? globalThis.crypto.subtle;
|
||||
}
|
||||
|
||||
/**
|
||||
* AES-256-GCM encrypt with a CALLER-SUPPLIED 12-byte nonce.
|
||||
*
|
||||
* Unlike `CryptoProvider.aesGcmEncrypt` (which generates a random nonce
|
||||
* internally), streams require deterministic nonces derived from
|
||||
* `(laneId, seq)`. Returns the ciphertext concatenated with the 16-byte
|
||||
* authentication tag (SubtleCrypto's standard layout).
|
||||
*/
|
||||
export async function aesGcmEncryptWithNonce(
|
||||
key: Uint8Array,
|
||||
nonce: Uint8Array,
|
||||
plaintext: Uint8Array,
|
||||
aad: Uint8Array,
|
||||
subtle?: SubtleCrypto,
|
||||
): Promise<Uint8Array> {
|
||||
if (nonce.length !== STREAM_NONCE_BYTES) {
|
||||
throw new Error(`AES-GCM nonce must be ${STREAM_NONCE_BYTES} bytes`);
|
||||
}
|
||||
const s = resolveSubtle(subtle);
|
||||
const aesKey = await s.importKey('raw', bs(key), 'AES-GCM', false, ['encrypt']);
|
||||
const out = await s.encrypt(
|
||||
{ name: 'AES-GCM', iv: bs(nonce), additionalData: bs(aad) },
|
||||
aesKey,
|
||||
bs(plaintext),
|
||||
);
|
||||
return new Uint8Array(out);
|
||||
}
|
||||
|
||||
/**
|
||||
* AES-256-GCM decrypt with a CALLER-SUPPLIED nonce. Throws
|
||||
* `StreamDecryptionError` on authentication failure.
|
||||
*/
|
||||
export async function aesGcmDecryptWithNonce(
|
||||
key: Uint8Array,
|
||||
nonce: Uint8Array,
|
||||
ciphertext: Uint8Array,
|
||||
aad: Uint8Array,
|
||||
subtle?: SubtleCrypto,
|
||||
): Promise<Uint8Array> {
|
||||
if (nonce.length !== STREAM_NONCE_BYTES) {
|
||||
throw new Error(`AES-GCM nonce must be ${STREAM_NONCE_BYTES} bytes`);
|
||||
}
|
||||
const s = resolveSubtle(subtle);
|
||||
const aesKey = await s.importKey('raw', bs(key), 'AES-GCM', false, ['decrypt']);
|
||||
try {
|
||||
const out = await s.decrypt(
|
||||
{ name: 'AES-GCM', iv: bs(nonce), additionalData: bs(aad) },
|
||||
aesKey,
|
||||
bs(ciphertext),
|
||||
);
|
||||
return new Uint8Array(out);
|
||||
} catch {
|
||||
throw new StreamDecryptionError();
|
||||
}
|
||||
}
|
||||
265
packages/shade-streams/src/coordinator.ts
Normal file
265
packages/shade-streams/src/coordinator.ts
Normal file
@@ -0,0 +1,265 @@
|
||||
import type { CryptoProvider } from '@shade/core';
|
||||
import { ValidationError } from '@shade/core';
|
||||
import { decodeStreamChunk } from '@shade/proto';
|
||||
import { StreamFinishedError, StreamProtocolError } from './errors.js';
|
||||
import { StreamingSha256 } from './hash.js';
|
||||
import { StreamReceiver } from './receiver.js';
|
||||
import { StreamSender } from './sender.js';
|
||||
import type { DecryptedChunk, EncryptedChunk, LaneInitSpec } from './types.js';
|
||||
|
||||
export interface MultiLaneSenderInit {
|
||||
crypto: CryptoProvider;
|
||||
subtle?: SubtleCrypto;
|
||||
streamId: Uint8Array;
|
||||
streamSecret: Uint8Array;
|
||||
lanes: LaneInitSpec[];
|
||||
/** Per-lane resume offsets; defaults to 0 for each lane. */
|
||||
startSeqByLane?: Map<number, number | bigint>;
|
||||
}
|
||||
|
||||
export interface MultiLaneReceiverInit extends MultiLaneSenderInit {}
|
||||
|
||||
/** Per-lane sha256 fingerprint for the stream-finish envelope. */
|
||||
export interface LaneFingerprint {
|
||||
laneId: number;
|
||||
/** sha256 over plaintext bytes carried by this lane in seq order. */
|
||||
sha256: Uint8Array;
|
||||
chunkCount: number;
|
||||
byteCount: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Multi-lane stream sender.
|
||||
*
|
||||
* Wraps N independent `StreamSender`s sharing the same (streamSecret, streamId)
|
||||
* and tracks an `overallSha256` over the original plaintext as the consumer
|
||||
* dispatches bytes to lanes.
|
||||
*
|
||||
* The coordinator is partition-AGNOSTIC — the consumer (transfer layer)
|
||||
* decides which bytes go to which lane. The coordinator only enforces:
|
||||
* - lane lookup by laneId
|
||||
* - update of overallSha256 from `appendOverall(...)` (called BEFORE bytes
|
||||
* are dispatched to lanes, in original byte order)
|
||||
*/
|
||||
export class MultiLaneSender {
|
||||
private readonly senders = new Map<number, StreamSender>();
|
||||
private readonly overallHasher = new StreamingSha256();
|
||||
private destroyed = false;
|
||||
|
||||
private constructor(senders: StreamSender[]) {
|
||||
for (const s of senders) {
|
||||
this.senders.set(s.laneId, s);
|
||||
}
|
||||
}
|
||||
|
||||
static async create(opts: MultiLaneSenderInit): Promise<MultiLaneSender> {
|
||||
if (opts.lanes.length === 0) {
|
||||
throw new ValidationError('at least one lane is required', 'lanes');
|
||||
}
|
||||
const seen = new Set<number>();
|
||||
for (const l of opts.lanes) {
|
||||
if (seen.has(l.laneId)) {
|
||||
throw new ValidationError(`duplicate laneId ${l.laneId}`, 'lanes');
|
||||
}
|
||||
seen.add(l.laneId);
|
||||
}
|
||||
const senders: StreamSender[] = [];
|
||||
for (const lane of opts.lanes) {
|
||||
const startSeq = opts.startSeqByLane?.get(lane.laneId);
|
||||
senders.push(
|
||||
await StreamSender.create({
|
||||
crypto: opts.crypto,
|
||||
...(opts.subtle !== undefined ? { subtle: opts.subtle } : {}),
|
||||
streamId: opts.streamId,
|
||||
streamSecret: opts.streamSecret,
|
||||
laneId: lane.laneId,
|
||||
...(startSeq !== undefined ? { startSeq } : {}),
|
||||
}),
|
||||
);
|
||||
}
|
||||
return new MultiLaneSender(senders);
|
||||
}
|
||||
|
||||
/**
|
||||
* Update the overall (cross-lane, original-order) sha256 hasher.
|
||||
*
|
||||
* The consumer must call this once with each plaintext slice IN ORIGINAL
|
||||
* BYTE ORDER (regardless of which lane the slice ends up on). For range
|
||||
* partitioning this is straightforward (lanes are contiguous); for
|
||||
* round-robin the consumer hashes each chunk in the order it was read
|
||||
* from the input.
|
||||
*/
|
||||
appendOverall(plaintext: Uint8Array): void {
|
||||
if (this.destroyed) throw new StreamFinishedError('MultiLaneSender: destroyed');
|
||||
this.overallHasher.update(plaintext);
|
||||
}
|
||||
|
||||
/** Encrypt `plaintext` for the given lane. */
|
||||
async encryptForLane(
|
||||
laneId: number,
|
||||
plaintext: Uint8Array,
|
||||
isLast: boolean,
|
||||
): Promise<EncryptedChunk> {
|
||||
const lane = this.senders.get(laneId);
|
||||
if (lane === undefined) {
|
||||
throw new StreamProtocolError(`Unknown laneId ${laneId}`);
|
||||
}
|
||||
return lane.encryptChunk(plaintext, isLast);
|
||||
}
|
||||
|
||||
/** Resume helper: feed plaintext into a lane's hasher without advancing seq. */
|
||||
preHashForLane(laneId: number, plaintext: Uint8Array): void {
|
||||
const lane = this.senders.get(laneId);
|
||||
if (lane === undefined) {
|
||||
throw new StreamProtocolError(`Unknown laneId ${laneId}`);
|
||||
}
|
||||
lane.preHash(plaintext);
|
||||
}
|
||||
|
||||
/** Snapshot per-lane fingerprints (call once when all lanes are finished). */
|
||||
getLaneFingerprints(): LaneFingerprint[] {
|
||||
const out: LaneFingerprint[] = [];
|
||||
for (const [laneId, sender] of this.senders) {
|
||||
out.push({
|
||||
laneId,
|
||||
sha256: sender.getLaneSha256Digest(),
|
||||
chunkCount: Number(sender.nextSequence),
|
||||
byteCount: Number(sender.bytesSent),
|
||||
});
|
||||
}
|
||||
out.sort((a, b) => a.laneId - b.laneId);
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Snapshot overall sha256 over original-order plaintext. */
|
||||
getOverallSha256(): Uint8Array {
|
||||
return this.overallHasher.digest();
|
||||
}
|
||||
|
||||
/** All lanes have emitted their `isLast` chunks. */
|
||||
get allLanesFinished(): boolean {
|
||||
for (const lane of this.senders.values()) {
|
||||
if (!lane.isFinished) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
destroy(): void {
|
||||
if (this.destroyed) return;
|
||||
for (const sender of this.senders.values()) sender.destroy();
|
||||
this.destroyed = true;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Multi-lane stream receiver.
|
||||
*
|
||||
* Routes incoming wire envelopes to per-lane `StreamReceiver`s by laneId,
|
||||
* and exposes a hook for the consumer to update the overall sha256 in
|
||||
* original byte order (the consumer's reorder logic feeds plaintext here
|
||||
* after collecting it from individual lanes).
|
||||
*/
|
||||
export class MultiLaneReceiver {
|
||||
private readonly receivers = new Map<number, StreamReceiver>();
|
||||
private readonly overallHasher = new StreamingSha256();
|
||||
private destroyed = false;
|
||||
|
||||
private constructor(receivers: StreamReceiver[]) {
|
||||
for (const r of receivers) {
|
||||
this.receivers.set(r.laneId, r);
|
||||
}
|
||||
}
|
||||
|
||||
static async create(opts: MultiLaneReceiverInit): Promise<MultiLaneReceiver> {
|
||||
if (opts.lanes.length === 0) {
|
||||
throw new ValidationError('at least one lane is required', 'lanes');
|
||||
}
|
||||
const seen = new Set<number>();
|
||||
for (const l of opts.lanes) {
|
||||
if (seen.has(l.laneId)) {
|
||||
throw new ValidationError(`duplicate laneId ${l.laneId}`, 'lanes');
|
||||
}
|
||||
seen.add(l.laneId);
|
||||
}
|
||||
const receivers: StreamReceiver[] = [];
|
||||
for (const lane of opts.lanes) {
|
||||
const startSeq = opts.startSeqByLane?.get(lane.laneId);
|
||||
receivers.push(
|
||||
await StreamReceiver.create({
|
||||
crypto: opts.crypto,
|
||||
...(opts.subtle !== undefined ? { subtle: opts.subtle } : {}),
|
||||
streamId: opts.streamId,
|
||||
streamSecret: opts.streamSecret,
|
||||
laneId: lane.laneId,
|
||||
...(startSeq !== undefined ? { startSeq } : {}),
|
||||
}),
|
||||
);
|
||||
}
|
||||
return new MultiLaneReceiver(receivers);
|
||||
}
|
||||
|
||||
/**
|
||||
* Decrypt one wire-level chunk. Routes to the lane indicated by the
|
||||
* envelope's laneId. Throws if no receiver exists for that laneId.
|
||||
*/
|
||||
async decryptChunk(wireBytes: Uint8Array): Promise<DecryptedChunk & { laneId: number }> {
|
||||
if (this.destroyed) throw new StreamFinishedError('MultiLaneReceiver: destroyed');
|
||||
const env = decodeStreamChunk(wireBytes);
|
||||
const lane = this.receivers.get(env.laneId);
|
||||
if (lane === undefined) {
|
||||
throw new StreamProtocolError(`Unknown laneId ${env.laneId}`);
|
||||
}
|
||||
const dec = await lane.decryptChunk(wireBytes);
|
||||
return { ...dec, laneId: env.laneId };
|
||||
}
|
||||
|
||||
/**
|
||||
* Update the overall sha256 hasher with plaintext IN ORIGINAL BYTE ORDER.
|
||||
* The consumer's reorder logic decides when to call this (as bytes leave
|
||||
* the reorder buffer in original order, OR for range-partitioning,
|
||||
* concatenated lane outputs in laneId order).
|
||||
*/
|
||||
appendOverall(plaintext: Uint8Array): void {
|
||||
if (this.destroyed) throw new StreamFinishedError('MultiLaneReceiver: destroyed');
|
||||
this.overallHasher.update(plaintext);
|
||||
}
|
||||
|
||||
/** Snapshot per-lane fingerprints (call once when all lanes are finished). */
|
||||
getLaneFingerprints(): LaneFingerprint[] {
|
||||
const out: LaneFingerprint[] = [];
|
||||
for (const [laneId, receiver] of this.receivers) {
|
||||
out.push({
|
||||
laneId,
|
||||
sha256: receiver.getLaneSha256Digest(),
|
||||
chunkCount: Number(receiver.nextExpectedSequence),
|
||||
byteCount: Number(receiver.bytesReceived),
|
||||
});
|
||||
}
|
||||
out.sort((a, b) => a.laneId - b.laneId);
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Snapshot overall sha256 over original-order plaintext. */
|
||||
getOverallSha256(): Uint8Array {
|
||||
return this.overallHasher.digest();
|
||||
}
|
||||
|
||||
/** All lanes have received their `isLast` chunks. */
|
||||
get allLanesFinished(): boolean {
|
||||
for (const lane of this.receivers.values()) {
|
||||
if (!lane.isFinished) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Get a specific lane receiver (for inspection/resume). */
|
||||
getLane(laneId: number): StreamReceiver | undefined {
|
||||
return this.receivers.get(laneId);
|
||||
}
|
||||
|
||||
destroy(): void {
|
||||
if (this.destroyed) return;
|
||||
for (const receiver of this.receivers.values()) receiver.destroy();
|
||||
this.destroyed = true;
|
||||
}
|
||||
}
|
||||
103
packages/shade-streams/src/envelope.ts
Normal file
103
packages/shade-streams/src/envelope.ts
Normal file
@@ -0,0 +1,103 @@
|
||||
/**
|
||||
* Plaintext schemas for stream control messages.
|
||||
*
|
||||
* Stream-init / -finish / -abort / -resume ride the existing 0x02 ratchet
|
||||
* envelope — Double Ratchet AEAD already authenticates them and rejects
|
||||
* replays. Each carries a JSON plaintext with a `kind` discriminator.
|
||||
* Stream-chunk uses dedicated wire type 0x11 (see `@shade/proto/wire.ts`).
|
||||
*/
|
||||
import { ValidationError } from '@shade/core';
|
||||
import type { LaneInitSpec, StreamMetadata } from './types.js';
|
||||
|
||||
export type StreamControlKind =
|
||||
| 'shade.stream-init/v1'
|
||||
| 'shade.stream-finish/v1'
|
||||
| 'shade.stream-abort/v1'
|
||||
| 'shade.stream-resume-request/v1'
|
||||
| 'shade.stream-resume-state/v1';
|
||||
|
||||
export interface StreamInitMessage {
|
||||
kind: 'shade.stream-init/v1';
|
||||
/** base64url-encoded 16-byte streamId. */
|
||||
streamId: string;
|
||||
/** base64url-encoded 32-byte streamSecret. */
|
||||
streamSecret: string;
|
||||
metadata: StreamMetadata;
|
||||
lanes: LaneInitSpec[];
|
||||
}
|
||||
|
||||
export interface StreamFinishMessage {
|
||||
kind: 'shade.stream-finish/v1';
|
||||
streamId: string;
|
||||
/** Per-lane integrity fingerprints, base64-encoded sha256 of plaintext bytes the lane carried. */
|
||||
laneSha256: Array<{
|
||||
laneId: number;
|
||||
sha256: string;
|
||||
chunkCount: number;
|
||||
byteCount: number;
|
||||
}>;
|
||||
/** base64-encoded sha256 over the original plaintext file in original byte order. */
|
||||
overallSha256: string;
|
||||
finishedAt: number;
|
||||
}
|
||||
|
||||
export type StreamAbortReason = 'sender-cancel' | 'receiver-cancel' | 'fatal-error';
|
||||
|
||||
export interface StreamAbortMessage {
|
||||
kind: 'shade.stream-abort/v1';
|
||||
streamId: string;
|
||||
reason: StreamAbortReason;
|
||||
message?: string;
|
||||
abortedAt: number;
|
||||
}
|
||||
|
||||
export interface StreamResumeRequestMessage {
|
||||
kind: 'shade.stream-resume-request/v1';
|
||||
streamId: string;
|
||||
requestedAt: number;
|
||||
}
|
||||
|
||||
export interface StreamResumeStateMessage {
|
||||
kind: 'shade.stream-resume-state/v1';
|
||||
streamId: string;
|
||||
/** lastSeqAcked = -1 means no chunk for this lane has been received yet. */
|
||||
lanes: Array<{ laneId: number; lastSeqAcked: number }>;
|
||||
resumedAt: number;
|
||||
}
|
||||
|
||||
export type StreamControlMessage =
|
||||
| StreamInitMessage
|
||||
| StreamFinishMessage
|
||||
| StreamAbortMessage
|
||||
| StreamResumeRequestMessage
|
||||
| StreamResumeStateMessage;
|
||||
|
||||
/** Type guard: does the value look like a stream control message? */
|
||||
export function isStreamControlMessage(value: unknown): value is StreamControlMessage {
|
||||
if (typeof value !== 'object' || value === null) return false;
|
||||
const kind = (value as { kind?: unknown }).kind;
|
||||
if (typeof kind !== 'string') return false;
|
||||
return kind.startsWith('shade.stream-');
|
||||
}
|
||||
|
||||
/** Parse JSON plaintext; throw ValidationError on shape mismatch. */
|
||||
export function parseStreamControl(plaintext: string): StreamControlMessage {
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(plaintext);
|
||||
} catch (err) {
|
||||
throw new ValidationError(
|
||||
`stream control plaintext is not valid JSON: ${(err as Error).message}`,
|
||||
'plaintext',
|
||||
);
|
||||
}
|
||||
if (!isStreamControlMessage(parsed)) {
|
||||
throw new ValidationError('plaintext is not a stream control message', 'plaintext');
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
/** Encode a stream control message as JSON; throws on circular refs. */
|
||||
export function encodeStreamControl(msg: StreamControlMessage): string {
|
||||
return JSON.stringify(msg);
|
||||
}
|
||||
53
packages/shade-streams/src/errors.ts
Normal file
53
packages/shade-streams/src/errors.ts
Normal file
@@ -0,0 +1,53 @@
|
||||
import { ShadeError } from '@shade/core';
|
||||
|
||||
export class StreamError extends ShadeError {
|
||||
constructor(code: string, message: string) {
|
||||
super(code, message);
|
||||
this.name = 'StreamError';
|
||||
}
|
||||
}
|
||||
|
||||
export class StreamProtocolError extends StreamError {
|
||||
constructor(message: string) {
|
||||
super('SHADE_STREAM_PROTOCOL', message);
|
||||
this.name = 'StreamProtocolError';
|
||||
}
|
||||
}
|
||||
|
||||
export class StreamIntegrityError extends StreamError {
|
||||
constructor(message: string) {
|
||||
super('SHADE_STREAM_INTEGRITY', message);
|
||||
this.name = 'StreamIntegrityError';
|
||||
}
|
||||
}
|
||||
|
||||
export class StreamReplayError extends StreamError {
|
||||
constructor(message = 'Stream chunk replay detected') {
|
||||
super('SHADE_STREAM_REPLAY', message);
|
||||
this.name = 'StreamReplayError';
|
||||
}
|
||||
}
|
||||
|
||||
export class StreamOutOfOrderError extends StreamError {
|
||||
constructor(expected: number, received: number) {
|
||||
super(
|
||||
'SHADE_STREAM_OUT_OF_ORDER',
|
||||
`Out-of-order chunk: expected seq=${expected}, got ${received}`,
|
||||
);
|
||||
this.name = 'StreamOutOfOrderError';
|
||||
}
|
||||
}
|
||||
|
||||
export class StreamDecryptionError extends StreamError {
|
||||
constructor(message = 'Stream chunk authenticated decryption failed') {
|
||||
super('SHADE_STREAM_DECRYPTION', message);
|
||||
this.name = 'StreamDecryptionError';
|
||||
}
|
||||
}
|
||||
|
||||
export class StreamFinishedError extends StreamError {
|
||||
constructor(message = 'Stream is already finished or aborted') {
|
||||
super('SHADE_STREAM_FINISHED', message);
|
||||
this.name = 'StreamFinishedError';
|
||||
}
|
||||
}
|
||||
41
packages/shade-streams/src/hash.ts
Normal file
41
packages/shade-streams/src/hash.ts
Normal file
@@ -0,0 +1,41 @@
|
||||
import { sha256 } from '@noble/hashes/sha2.js';
|
||||
|
||||
/**
|
||||
* Streaming SHA-256 wrapper.
|
||||
*
|
||||
* SubtleCrypto exposes only one-shot `digest`. Streams need incremental
|
||||
* hashing so the receiver doesn't materialize the full plaintext before
|
||||
* verifying integrity (NF2: O(chunkSize) memory). `@noble/hashes/sha2`
|
||||
* provides a streaming API that works in Bun, Node, and browsers without
|
||||
* any native bindings.
|
||||
*/
|
||||
export class StreamingSha256 {
|
||||
private h = sha256.create();
|
||||
private finalized = false;
|
||||
|
||||
/** Feed bytes into the hasher. No-op on empty input. */
|
||||
update(data: Uint8Array): this {
|
||||
if (this.finalized) throw new Error('StreamingSha256: already finalized');
|
||||
if (data.length > 0) this.h.update(data);
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Produce the 32-byte digest. After `digest()` the hasher is finalized;
|
||||
* subsequent `update()` calls will throw.
|
||||
*/
|
||||
digest(): Uint8Array {
|
||||
this.finalized = true;
|
||||
return this.h.digest();
|
||||
}
|
||||
|
||||
/** Whether `digest()` has been called. */
|
||||
get isFinalized(): boolean {
|
||||
return this.finalized;
|
||||
}
|
||||
}
|
||||
|
||||
/** Convenience: hash a single buffer in one shot. */
|
||||
export function sha256Once(data: Uint8Array): Uint8Array {
|
||||
return sha256(data);
|
||||
}
|
||||
47
packages/shade-streams/src/ids.ts
Normal file
47
packages/shade-streams/src/ids.ts
Normal file
@@ -0,0 +1,47 @@
|
||||
import type { CryptoProvider } from '@shade/core';
|
||||
import { ValidationError } from '@shade/core';
|
||||
|
||||
export const STREAM_ID_BYTES = 16;
|
||||
export const STREAM_SECRET_BYTES = 32;
|
||||
|
||||
/** Generate a fresh 16-byte random streamId. */
|
||||
export function generateStreamId(crypto: CryptoProvider): Uint8Array {
|
||||
return crypto.randomBytes(STREAM_ID_BYTES);
|
||||
}
|
||||
|
||||
/** Generate a fresh 32-byte random streamSecret. */
|
||||
export function generateStreamSecret(crypto: CryptoProvider): Uint8Array {
|
||||
return crypto.randomBytes(STREAM_SECRET_BYTES);
|
||||
}
|
||||
|
||||
/** Encode a streamId as URL-safe base64 (no padding). */
|
||||
export function streamIdToString(streamId: Uint8Array): string {
|
||||
if (streamId.length !== STREAM_ID_BYTES) {
|
||||
throw new ValidationError(`streamId must be ${STREAM_ID_BYTES} bytes`, 'streamId');
|
||||
}
|
||||
return base64UrlEncode(streamId);
|
||||
}
|
||||
|
||||
/** Decode a URL-safe base64 streamId back to bytes. */
|
||||
export function streamIdFromString(s: string): Uint8Array {
|
||||
const bytes = base64UrlDecode(s);
|
||||
if (bytes.length !== STREAM_ID_BYTES) {
|
||||
throw new ValidationError(`streamId must decode to ${STREAM_ID_BYTES} bytes`, 'streamId');
|
||||
}
|
||||
return bytes;
|
||||
}
|
||||
|
||||
function base64UrlEncode(bytes: Uint8Array): string {
|
||||
let bin = '';
|
||||
for (let i = 0; i < bytes.length; i++) bin += String.fromCharCode(bytes[i]!);
|
||||
return btoa(bin).replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, '');
|
||||
}
|
||||
|
||||
function base64UrlDecode(s: string): Uint8Array {
|
||||
const padded = s.replace(/-/g, '+').replace(/_/g, '/');
|
||||
const pad = padded.length % 4 === 0 ? '' : '='.repeat(4 - (padded.length % 4));
|
||||
const bin = atob(padded + pad);
|
||||
const out = new Uint8Array(bin.length);
|
||||
for (let i = 0; i < bin.length; i++) out[i] = bin.charCodeAt(i);
|
||||
return out;
|
||||
}
|
||||
12
packages/shade-streams/src/index.ts
Normal file
12
packages/shade-streams/src/index.ts
Normal file
@@ -0,0 +1,12 @@
|
||||
export * from './errors.js';
|
||||
export * from './types.js';
|
||||
export * from './ids.js';
|
||||
export * from './kdf.js';
|
||||
export * from './nonce.js';
|
||||
export * from './aead.js';
|
||||
export * from './hash.js';
|
||||
export * from './envelope.js';
|
||||
export * from './sender.js';
|
||||
export * from './receiver.js';
|
||||
export * from './partition.js';
|
||||
export * from './coordinator.js';
|
||||
66
packages/shade-streams/src/kdf.ts
Normal file
66
packages/shade-streams/src/kdf.ts
Normal file
@@ -0,0 +1,66 @@
|
||||
import type { CryptoProvider } from '@shade/core';
|
||||
import { ValidationError } from '@shade/core';
|
||||
import { STREAM_ID_BYTES, STREAM_SECRET_BYTES } from './ids.js';
|
||||
|
||||
const TEXT = new TextEncoder();
|
||||
const STREAM_KEY_INFO = TEXT.encode('shade-stream/v1\0master');
|
||||
const LANE_KEY_INFO_PREFIX = TEXT.encode('shade-stream/v1\0lane\0');
|
||||
|
||||
const STREAM_KEY_BYTES = 32;
|
||||
export const LANE_KEY_BYTES = 32;
|
||||
|
||||
/**
|
||||
* Derive the master streamKey from a streamSecret + streamId.
|
||||
*
|
||||
* streamKey = HKDF(ikm=streamSecret, salt=streamId,
|
||||
* info="shade-stream/v1\0master", length=32)
|
||||
*
|
||||
* The streamKey is NEVER used to encrypt chunks directly — it is a root for
|
||||
* per-lane key derivation (see `deriveLaneKey`).
|
||||
*/
|
||||
export async function deriveStreamKey(
|
||||
crypto: CryptoProvider,
|
||||
streamSecret: Uint8Array,
|
||||
streamId: Uint8Array,
|
||||
): Promise<Uint8Array> {
|
||||
if (streamSecret.length !== STREAM_SECRET_BYTES) {
|
||||
throw new ValidationError(
|
||||
`streamSecret must be ${STREAM_SECRET_BYTES} bytes`,
|
||||
'streamSecret',
|
||||
);
|
||||
}
|
||||
if (streamId.length !== STREAM_ID_BYTES) {
|
||||
throw new ValidationError(`streamId must be ${STREAM_ID_BYTES} bytes`, 'streamId');
|
||||
}
|
||||
return crypto.hkdf(streamSecret, streamId, STREAM_KEY_INFO, STREAM_KEY_BYTES);
|
||||
}
|
||||
|
||||
/**
|
||||
* Derive a lane-specific AEAD key.
|
||||
*
|
||||
* laneKey[i] = HKDF(ikm=streamKey, salt=streamId,
|
||||
* info="shade-stream/v1\0lane\0" || u32_be(laneId), length=32)
|
||||
*
|
||||
* Distinct laneIds produce independent keys; receiver derives the same key
|
||||
* given the same (streamSecret, streamId, laneId).
|
||||
*/
|
||||
export async function deriveLaneKey(
|
||||
crypto: CryptoProvider,
|
||||
streamKey: Uint8Array,
|
||||
streamId: Uint8Array,
|
||||
laneId: number,
|
||||
): Promise<Uint8Array> {
|
||||
if (streamKey.length !== STREAM_KEY_BYTES) {
|
||||
throw new ValidationError(`streamKey must be ${STREAM_KEY_BYTES} bytes`, 'streamKey');
|
||||
}
|
||||
if (streamId.length !== STREAM_ID_BYTES) {
|
||||
throw new ValidationError(`streamId must be ${STREAM_ID_BYTES} bytes`, 'streamId');
|
||||
}
|
||||
if (!Number.isInteger(laneId) || laneId < 0 || laneId > 0xffff_ffff) {
|
||||
throw new ValidationError(`laneId must fit in u32: ${laneId}`, 'laneId');
|
||||
}
|
||||
const info = new Uint8Array(LANE_KEY_INFO_PREFIX.length + 4);
|
||||
info.set(LANE_KEY_INFO_PREFIX, 0);
|
||||
new DataView(info.buffer).setUint32(LANE_KEY_INFO_PREFIX.length, laneId, false);
|
||||
return crypto.hkdf(streamKey, streamId, info, LANE_KEY_BYTES);
|
||||
}
|
||||
62
packages/shade-streams/src/nonce.ts
Normal file
62
packages/shade-streams/src/nonce.ts
Normal file
@@ -0,0 +1,62 @@
|
||||
import { ValidationError } from '@shade/core';
|
||||
|
||||
export const STREAM_NONCE_BYTES = 12;
|
||||
|
||||
/** Maximum chunk seq value (u64 max). Hard-spec'd hard limit. */
|
||||
export const MAX_SEQ = 0xffff_ffff_ffff_ffffn;
|
||||
|
||||
/**
|
||||
* Construct the deterministic AES-GCM nonce for a stream chunk.
|
||||
*
|
||||
* nonce[0..4] = u32_be(laneId)
|
||||
* nonce[4..12] = u64_be(seq)
|
||||
*
|
||||
* Per (laneId, seq) is unique — combined with the lane-specific key, this
|
||||
* guarantees AES-GCM nonce-uniqueness even across multiple parallel lanes.
|
||||
*/
|
||||
export function buildChunkNonce(laneId: number, seq: number | bigint): Uint8Array {
|
||||
if (!Number.isInteger(laneId) || laneId < 0 || laneId > 0xffff_ffff) {
|
||||
throw new ValidationError(`laneId must fit in u32: ${laneId}`, 'laneId');
|
||||
}
|
||||
const seqBig = typeof seq === 'bigint' ? seq : BigInt(seq);
|
||||
if (seqBig < 0n || seqBig > MAX_SEQ) {
|
||||
throw new ValidationError(`seq must fit in u64 (>= 0): ${seq}`, 'seq');
|
||||
}
|
||||
const out = new Uint8Array(STREAM_NONCE_BYTES);
|
||||
const view = new DataView(out.buffer);
|
||||
view.setUint32(0, laneId, false);
|
||||
view.setBigUint64(4, seqBig, false);
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the AAD bound to a stream chunk. Computed implicitly on both sides
|
||||
* from the chunk header (never transmitted as-is). Tampering with any header
|
||||
* field invalidates the AEAD tag.
|
||||
*
|
||||
* aad = streamId(16) || u32_be(laneId) || u64_be(seq) || u8(isLast)
|
||||
*/
|
||||
export function buildChunkAad(
|
||||
streamId: Uint8Array,
|
||||
laneId: number,
|
||||
seq: number | bigint,
|
||||
isLast: boolean,
|
||||
): Uint8Array {
|
||||
if (streamId.length !== 16) {
|
||||
throw new ValidationError('streamId must be 16 bytes', 'streamId');
|
||||
}
|
||||
if (!Number.isInteger(laneId) || laneId < 0 || laneId > 0xffff_ffff) {
|
||||
throw new ValidationError(`laneId must fit in u32: ${laneId}`, 'laneId');
|
||||
}
|
||||
const seqBig = typeof seq === 'bigint' ? seq : BigInt(seq);
|
||||
if (seqBig < 0n || seqBig > MAX_SEQ) {
|
||||
throw new ValidationError(`seq must fit in u64 (>= 0): ${seq}`, 'seq');
|
||||
}
|
||||
const out = new Uint8Array(16 + 4 + 8 + 1);
|
||||
out.set(streamId, 0);
|
||||
const view = new DataView(out.buffer);
|
||||
view.setUint32(16, laneId, false);
|
||||
view.setBigUint64(20, seqBig, false);
|
||||
out[28] = isLast ? 0x01 : 0x00;
|
||||
return out;
|
||||
}
|
||||
87
packages/shade-streams/src/partition.ts
Normal file
87
packages/shade-streams/src/partition.ts
Normal file
@@ -0,0 +1,87 @@
|
||||
import { ValidationError } from '@shade/core';
|
||||
import type { LaneInitSpec, LanePartition } from './types.js';
|
||||
|
||||
/** Build a range-partition plan: contiguous byte ranges, one per lane. */
|
||||
export function planRangePartition(
|
||||
totalBytes: number,
|
||||
laneCount: number,
|
||||
): LaneInitSpec[] {
|
||||
if (!Number.isInteger(totalBytes) || totalBytes < 0) {
|
||||
throw new ValidationError(`totalBytes must be a non-negative integer`, 'totalBytes');
|
||||
}
|
||||
if (!Number.isInteger(laneCount) || laneCount < 1) {
|
||||
throw new ValidationError(`laneCount must be >= 1`, 'laneCount');
|
||||
}
|
||||
const lanes: LaneInitSpec[] = [];
|
||||
// Use ceil so the first lanes get the extra byte when not evenly divisible.
|
||||
// Each lane's start = previous end. Last lane's end = totalBytes.
|
||||
const baseSize = Math.floor(totalBytes / laneCount);
|
||||
const remainder = totalBytes - baseSize * laneCount;
|
||||
let cursor = 0;
|
||||
for (let i = 0; i < laneCount; i++) {
|
||||
const extra = i < remainder ? 1 : 0;
|
||||
const size = baseSize + extra;
|
||||
const startByte = cursor;
|
||||
const endByte = cursor + size;
|
||||
lanes.push({
|
||||
laneId: i,
|
||||
partition: { kind: 'range', startByte, endByte, startChunk: 0 },
|
||||
});
|
||||
cursor = endByte;
|
||||
}
|
||||
return lanes;
|
||||
}
|
||||
|
||||
/** Build a round-robin partition plan: chunk i goes to lane (i mod count). */
|
||||
export function planRoundRobinPartition(laneCount: number): LaneInitSpec[] {
|
||||
if (!Number.isInteger(laneCount) || laneCount < 1) {
|
||||
throw new ValidationError(`laneCount must be >= 1`, 'laneCount');
|
||||
}
|
||||
const lanes: LaneInitSpec[] = [];
|
||||
for (let i = 0; i < laneCount; i++) {
|
||||
lanes.push({
|
||||
laneId: i,
|
||||
partition: { kind: 'round-robin', lane: i, count: laneCount },
|
||||
});
|
||||
}
|
||||
return lanes;
|
||||
}
|
||||
|
||||
/**
|
||||
* Split an arbitrary byte range into chunkSize-sized slices.
|
||||
* Returns an array of [startByte, endByte) tuples for each chunk.
|
||||
*/
|
||||
export function chunkRange(
|
||||
startByte: number,
|
||||
endByte: number,
|
||||
chunkSize: number,
|
||||
): Array<{ start: number; end: number }> {
|
||||
if (chunkSize <= 0) {
|
||||
throw new ValidationError(`chunkSize must be positive`, 'chunkSize');
|
||||
}
|
||||
const out: Array<{ start: number; end: number }> = [];
|
||||
if (endByte === startByte) {
|
||||
// Empty range: emit one empty chunk so isLast can be carried.
|
||||
out.push({ start: startByte, end: startByte });
|
||||
return out;
|
||||
}
|
||||
for (let off = startByte; off < endByte; off += chunkSize) {
|
||||
out.push({ start: off, end: Math.min(off + chunkSize, endByte) });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate that a `LanePartition` matches the global stream parameters.
|
||||
* Used by the receiver to detect partition-mismatch on resume.
|
||||
*/
|
||||
export function partitionsEqual(a: LanePartition, b: LanePartition): boolean {
|
||||
if (a.kind !== b.kind) return false;
|
||||
if (a.kind === 'range' && b.kind === 'range') {
|
||||
return a.startByte === b.startByte && a.endByte === b.endByte && a.startChunk === b.startChunk;
|
||||
}
|
||||
if (a.kind === 'round-robin' && b.kind === 'round-robin') {
|
||||
return a.lane === b.lane && a.count === b.count;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
169
packages/shade-streams/src/receiver.ts
Normal file
169
packages/shade-streams/src/receiver.ts
Normal file
@@ -0,0 +1,169 @@
|
||||
import type { CryptoProvider } from '@shade/core';
|
||||
import { ValidationError, constantTimeEqual } from '@shade/core';
|
||||
import { decodeStreamChunk } from '@shade/proto';
|
||||
import { aesGcmDecryptWithNonce } from './aead.js';
|
||||
import {
|
||||
StreamFinishedError,
|
||||
StreamOutOfOrderError,
|
||||
StreamProtocolError,
|
||||
StreamReplayError,
|
||||
} from './errors.js';
|
||||
import { StreamingSha256 } from './hash.js';
|
||||
import { deriveLaneKey, deriveStreamKey } from './kdf.js';
|
||||
import { buildChunkAad, buildChunkNonce, MAX_SEQ } from './nonce.js';
|
||||
import type { DecryptedChunk } from './types.js';
|
||||
|
||||
export interface StreamReceiverInit {
|
||||
crypto: CryptoProvider;
|
||||
subtle?: SubtleCrypto;
|
||||
streamId: Uint8Array;
|
||||
streamSecret: Uint8Array;
|
||||
laneId: number;
|
||||
/** First seq this receiver will accept; defaults to 0. Used for resume. */
|
||||
startSeq?: number | bigint;
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-lane stream receiver state machine.
|
||||
*
|
||||
* Verifies AEAD, enforces strict in-order seq (rejects replay + out-of-order),
|
||||
* and updates a running lane sha256 over the decrypted plaintext.
|
||||
*/
|
||||
export class StreamReceiver {
|
||||
private constructor(
|
||||
private readonly subtle: SubtleCrypto | undefined,
|
||||
private readonly crypto: CryptoProvider,
|
||||
private readonly streamIdBytes: Uint8Array,
|
||||
public readonly laneId: number,
|
||||
private laneKey: Uint8Array | null,
|
||||
private expectedSeq: bigint,
|
||||
private readonly hasher: StreamingSha256,
|
||||
private finished: boolean,
|
||||
private bytesReceivedInternal: bigint,
|
||||
) {}
|
||||
|
||||
/** 16-byte streamId this receiver decrypts under. Defensive copy. */
|
||||
get streamId(): Uint8Array {
|
||||
return this.streamIdBytes.slice();
|
||||
}
|
||||
|
||||
static async create(opts: StreamReceiverInit): Promise<StreamReceiver> {
|
||||
const streamKey = await deriveStreamKey(opts.crypto, opts.streamSecret, opts.streamId);
|
||||
try {
|
||||
const laneKey = await deriveLaneKey(opts.crypto, streamKey, opts.streamId, opts.laneId);
|
||||
const startSeq =
|
||||
opts.startSeq === undefined
|
||||
? 0n
|
||||
: typeof opts.startSeq === 'bigint'
|
||||
? opts.startSeq
|
||||
: BigInt(opts.startSeq);
|
||||
if (startSeq < 0n || startSeq > MAX_SEQ) {
|
||||
throw new ValidationError(`startSeq out of range: ${opts.startSeq}`, 'startSeq');
|
||||
}
|
||||
return new StreamReceiver(
|
||||
opts.subtle,
|
||||
opts.crypto,
|
||||
opts.streamId.slice(),
|
||||
opts.laneId,
|
||||
laneKey,
|
||||
startSeq,
|
||||
new StreamingSha256(),
|
||||
false,
|
||||
0n,
|
||||
);
|
||||
} finally {
|
||||
opts.crypto.zeroize(streamKey);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Decrypt and authenticate a wire-level chunk envelope. Throws on:
|
||||
* - mismatched streamId / laneId
|
||||
* - replayed seq (already accepted)
|
||||
* - out-of-order seq (gap or backwards)
|
||||
* - tampered nonce / ciphertext / aad
|
||||
* - any chunk after `isLast`
|
||||
*/
|
||||
async decryptChunk(wireBytes: Uint8Array): Promise<DecryptedChunk> {
|
||||
if (this.finished) {
|
||||
throw new StreamFinishedError('StreamReceiver: lane already finished');
|
||||
}
|
||||
if (this.laneKey === null) {
|
||||
throw new StreamFinishedError('StreamReceiver: destroyed');
|
||||
}
|
||||
|
||||
const env = decodeStreamChunk(wireBytes);
|
||||
|
||||
if (!constantTimeEqual(env.streamId, this.streamIdBytes)) {
|
||||
throw new StreamProtocolError('stream-chunk streamId does not match this receiver');
|
||||
}
|
||||
if (env.laneId !== this.laneId) {
|
||||
throw new StreamProtocolError(
|
||||
`stream-chunk laneId=${env.laneId} routed to laneId=${this.laneId}`,
|
||||
);
|
||||
}
|
||||
|
||||
const seqBig = typeof env.seq === 'bigint' ? env.seq : BigInt(env.seq);
|
||||
if (seqBig < this.expectedSeq) {
|
||||
throw new StreamReplayError(`Replay: seq=${seqBig}, already accepted < ${this.expectedSeq}`);
|
||||
}
|
||||
if (seqBig > this.expectedSeq) {
|
||||
throw new StreamOutOfOrderError(Number(this.expectedSeq), Number(seqBig));
|
||||
}
|
||||
|
||||
// Defense-in-depth: the wire nonce MUST equal the deterministically derived one.
|
||||
const expectedNonce = buildChunkNonce(this.laneId, seqBig);
|
||||
if (!constantTimeEqual(env.nonce, expectedNonce)) {
|
||||
throw new StreamProtocolError('stream-chunk nonce mismatch');
|
||||
}
|
||||
if (env.aad.length !== 0) {
|
||||
throw new StreamProtocolError('stream-chunk aad must be empty in v0.2.0');
|
||||
}
|
||||
|
||||
const aad = buildChunkAad(this.streamIdBytes, this.laneId, seqBig, env.isLast);
|
||||
const plaintext = await aesGcmDecryptWithNonce(
|
||||
this.laneKey,
|
||||
env.nonce,
|
||||
env.ciphertext,
|
||||
aad,
|
||||
this.subtle,
|
||||
);
|
||||
|
||||
this.hasher.update(plaintext);
|
||||
this.bytesReceivedInternal += BigInt(plaintext.length);
|
||||
this.expectedSeq = seqBig + 1n;
|
||||
|
||||
if (env.isLast) this.finished = true;
|
||||
|
||||
return { plaintext, seq: Number(seqBig), isLast: env.isLast };
|
||||
}
|
||||
|
||||
/** Snapshot the lane sha256 digest. Hasher is frozen after this call. */
|
||||
getLaneSha256Digest(): Uint8Array {
|
||||
return this.hasher.digest();
|
||||
}
|
||||
|
||||
/** Total plaintext bytes accepted so far in this lane. */
|
||||
get bytesReceived(): bigint {
|
||||
return this.bytesReceivedInternal;
|
||||
}
|
||||
|
||||
/** Next sequence number this receiver expects. */
|
||||
get nextExpectedSequence(): bigint {
|
||||
return this.expectedSeq;
|
||||
}
|
||||
|
||||
/** Has this lane received its `isLast` chunk? */
|
||||
get isFinished(): boolean {
|
||||
return this.finished;
|
||||
}
|
||||
|
||||
/** Zero the lane key in memory. After destroy, decrypt calls throw. */
|
||||
destroy(): void {
|
||||
if (this.laneKey !== null) {
|
||||
this.crypto.zeroize(this.laneKey);
|
||||
this.laneKey = null;
|
||||
}
|
||||
this.finished = true;
|
||||
}
|
||||
}
|
||||
165
packages/shade-streams/src/sender.ts
Normal file
165
packages/shade-streams/src/sender.ts
Normal file
@@ -0,0 +1,165 @@
|
||||
import type { CryptoProvider } from '@shade/core';
|
||||
import { ValidationError } from '@shade/core';
|
||||
import { encodeStreamChunk } from '@shade/proto';
|
||||
import { aesGcmEncryptWithNonce } from './aead.js';
|
||||
import { StreamFinishedError } from './errors.js';
|
||||
import { StreamingSha256 } from './hash.js';
|
||||
import { deriveLaneKey, deriveStreamKey } from './kdf.js';
|
||||
import { buildChunkAad, buildChunkNonce, MAX_SEQ } from './nonce.js';
|
||||
import type { EncryptedChunk, StreamChunkEnvelope } from './types.js';
|
||||
|
||||
export interface StreamSenderInit {
|
||||
crypto: CryptoProvider;
|
||||
/** Optional SubtleCrypto for AEAD; defaults to globalThis.crypto.subtle. */
|
||||
subtle?: SubtleCrypto;
|
||||
streamId: Uint8Array;
|
||||
/** 32-byte streamSecret. The lane key is derived internally and stays in this instance. */
|
||||
streamSecret: Uint8Array;
|
||||
laneId: number;
|
||||
/** First seq this sender will emit; defaults to 0. Used for resume. */
|
||||
startSeq?: number | bigint;
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-lane stream sender state machine.
|
||||
*
|
||||
* Encapsulates the lane-specific AEAD key + monotonic seq counter + running
|
||||
* lane-sha256. Multiple senders (one per lane) sharing the same
|
||||
* `(streamSecret, streamId)` make up a parallel transfer.
|
||||
*/
|
||||
export class StreamSender {
|
||||
private constructor(
|
||||
private readonly subtle: SubtleCrypto | undefined,
|
||||
private readonly crypto: CryptoProvider,
|
||||
private readonly streamIdBytes: Uint8Array,
|
||||
public readonly laneId: number,
|
||||
private laneKey: Uint8Array | null,
|
||||
private nextSeq: bigint,
|
||||
private readonly hasher: StreamingSha256,
|
||||
private finished: boolean,
|
||||
private bytesSentInternal: bigint,
|
||||
) {}
|
||||
|
||||
/** 16-byte streamId this sender encrypts under. Defensive copy. */
|
||||
get streamId(): Uint8Array {
|
||||
return this.streamIdBytes.slice();
|
||||
}
|
||||
|
||||
static async create(opts: StreamSenderInit): Promise<StreamSender> {
|
||||
const streamKey = await deriveStreamKey(opts.crypto, opts.streamSecret, opts.streamId);
|
||||
try {
|
||||
const laneKey = await deriveLaneKey(opts.crypto, streamKey, opts.streamId, opts.laneId);
|
||||
const startSeq =
|
||||
opts.startSeq === undefined
|
||||
? 0n
|
||||
: typeof opts.startSeq === 'bigint'
|
||||
? opts.startSeq
|
||||
: BigInt(opts.startSeq);
|
||||
if (startSeq < 0n || startSeq > MAX_SEQ) {
|
||||
throw new ValidationError(`startSeq out of range: ${opts.startSeq}`, 'startSeq');
|
||||
}
|
||||
return new StreamSender(
|
||||
opts.subtle,
|
||||
opts.crypto,
|
||||
opts.streamId.slice(),
|
||||
opts.laneId,
|
||||
laneKey,
|
||||
startSeq,
|
||||
new StreamingSha256(),
|
||||
false,
|
||||
0n,
|
||||
);
|
||||
} finally {
|
||||
opts.crypto.zeroize(streamKey);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Encrypt one plaintext chunk. Updates lane sha256 with the plaintext bytes
|
||||
* BEFORE encryption (so receiver can independently verify by hashing
|
||||
* decrypted plaintext in the same order).
|
||||
*/
|
||||
async encryptChunk(plaintext: Uint8Array, isLast: boolean): Promise<EncryptedChunk> {
|
||||
if (this.finished) {
|
||||
throw new StreamFinishedError('StreamSender: lane already finished');
|
||||
}
|
||||
if (this.laneKey === null) {
|
||||
throw new StreamFinishedError('StreamSender: destroyed');
|
||||
}
|
||||
if (this.nextSeq > MAX_SEQ) {
|
||||
throw new ValidationError('seq overflow', 'seq');
|
||||
}
|
||||
|
||||
const seq = this.nextSeq;
|
||||
const nonce = buildChunkNonce(this.laneId, seq);
|
||||
const aad = buildChunkAad(this.streamIdBytes, this.laneId, seq, isLast);
|
||||
const ciphertext = await aesGcmEncryptWithNonce(
|
||||
this.laneKey,
|
||||
nonce,
|
||||
plaintext,
|
||||
aad,
|
||||
this.subtle,
|
||||
);
|
||||
|
||||
this.hasher.update(plaintext);
|
||||
this.bytesSentInternal += BigInt(plaintext.length);
|
||||
this.nextSeq = seq + 1n;
|
||||
|
||||
const envelope: StreamChunkEnvelope = {
|
||||
streamId: this.streamIdBytes,
|
||||
laneId: this.laneId,
|
||||
seq,
|
||||
isLast,
|
||||
nonce,
|
||||
aad: new Uint8Array(0),
|
||||
ciphertext,
|
||||
};
|
||||
const bytes = encodeStreamChunk(envelope);
|
||||
|
||||
if (isLast) this.finished = true;
|
||||
|
||||
return { envelope, bytes, seq: Number(seq) };
|
||||
}
|
||||
|
||||
/** Snapshot the lane sha256 digest. Hasher is frozen after this call. */
|
||||
getLaneSha256Digest(): Uint8Array {
|
||||
return this.hasher.digest();
|
||||
}
|
||||
|
||||
/**
|
||||
* Feed plaintext into the lane sha256 WITHOUT advancing seq or
|
||||
* producing wire bytes. Used by resume flows to re-build the running
|
||||
* lane hash from already-shipped bytes (since `@noble/hashes` v2 doesn't
|
||||
* expose serializable state in v0.2.0).
|
||||
*/
|
||||
preHash(plaintext: Uint8Array): void {
|
||||
if (this.finished) {
|
||||
throw new StreamFinishedError('StreamSender: lane already finished');
|
||||
}
|
||||
this.hasher.update(plaintext);
|
||||
}
|
||||
|
||||
/** Number of plaintext bytes encrypted so far in this lane. */
|
||||
get bytesSent(): bigint {
|
||||
return this.bytesSentInternal;
|
||||
}
|
||||
|
||||
/** Next sequence number this sender will emit. */
|
||||
get nextSequence(): bigint {
|
||||
return this.nextSeq;
|
||||
}
|
||||
|
||||
/** Has this lane emitted its `isLast` chunk? */
|
||||
get isFinished(): boolean {
|
||||
return this.finished;
|
||||
}
|
||||
|
||||
/** Zero the lane key in memory. After destroy, encrypt calls throw. */
|
||||
destroy(): void {
|
||||
if (this.laneKey !== null) {
|
||||
this.crypto.zeroize(this.laneKey);
|
||||
this.laneKey = null;
|
||||
}
|
||||
this.finished = true;
|
||||
}
|
||||
}
|
||||
72
packages/shade-streams/src/types.ts
Normal file
72
packages/shade-streams/src/types.ts
Normal file
@@ -0,0 +1,72 @@
|
||||
/**
|
||||
* Public types for @shade/streams.
|
||||
*
|
||||
* Higher-level wrappers for transfer-orchestration (parallel lanes, resume,
|
||||
* progress, etc.) live in @shade/transfer.
|
||||
*/
|
||||
|
||||
/** Plaintext metadata sent in a stream-init control envelope. */
|
||||
export interface StreamMetadata {
|
||||
name?: string;
|
||||
/** Total plaintext size in bytes. Omit for streams of unknown length. */
|
||||
sizeBytes?: number;
|
||||
contentType?: string;
|
||||
/** Plaintext bytes per chunk for this stream. */
|
||||
chunkSize: number;
|
||||
/** Total chunk count across all lanes. Omit for unknown-length streams. */
|
||||
totalChunks?: number;
|
||||
/** Sender's local clock at init time (advisory; never used for security decisions). */
|
||||
sentAt: number;
|
||||
/**
|
||||
* Optional application-level metadata, JSON-stringified-safe key/value
|
||||
* pairs. Round-tripped verbatim through stream-init plaintext. Used by
|
||||
* higher layers (e.g. `@shade/files` writes a `shadeFilesWriteId` here so
|
||||
* a server-side bridge can correlate an inbound transfer with a pending
|
||||
* RPC). The transport itself does not interpret these values.
|
||||
*/
|
||||
userMetadata?: Record<string, string>;
|
||||
}
|
||||
|
||||
/** Per-lane partition assignment carried in stream-init. */
|
||||
export type LanePartition =
|
||||
| {
|
||||
kind: 'range';
|
||||
/** Inclusive start byte of this lane's slice. */
|
||||
startByte: number;
|
||||
/** Exclusive end byte of this lane's slice. */
|
||||
endByte: number;
|
||||
/** First chunk seq this lane's region begins at (always 0 for per-lane numbering). */
|
||||
startChunk: number;
|
||||
}
|
||||
| {
|
||||
kind: 'round-robin';
|
||||
/** Chunk i goes to lane (i mod count). */
|
||||
lane: number;
|
||||
count: number;
|
||||
};
|
||||
|
||||
/** Per-lane state included in stream-init plaintext. laneKey is NOT shipped — it is derived. */
|
||||
export interface LaneInitSpec {
|
||||
laneId: number;
|
||||
partition: LanePartition;
|
||||
}
|
||||
|
||||
import type { StreamChunkWire } from '@shade/proto';
|
||||
|
||||
/** Wire-decoded stream-chunk envelope (alias for @shade/proto's `StreamChunkWire`). */
|
||||
export type StreamChunkEnvelope = StreamChunkWire;
|
||||
|
||||
/** Result returned from `StreamSender.encryptChunk`. */
|
||||
export interface EncryptedChunk {
|
||||
envelope: StreamChunkEnvelope;
|
||||
/** Raw bytes ready to ship — output of `encodeStreamChunk`. */
|
||||
bytes: Uint8Array;
|
||||
seq: number;
|
||||
}
|
||||
|
||||
/** Result returned from `StreamReceiver.decryptChunk`. */
|
||||
export interface DecryptedChunk {
|
||||
plaintext: Uint8Array;
|
||||
seq: number;
|
||||
isLast: boolean;
|
||||
}
|
||||
Reference in New Issue
Block a user