feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive

M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).

@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
  Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
  + per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
  (> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
  / shadeFilesReadStreamId correlation. Server-side TransformStream
  bridges accept inbound transfers immediately (engine rejects chunks
  that arrive before accept) and park the readable for the matching
  RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
  uploadDirectory()/downloadDirectory() with bounded concurrency pool
  (default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
  byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
  policy (traversal + percent-decode hardening), fingerprint gate
  (required/optional/reject), pluggable Ed25519 sig verification with
  ±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
  useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
  entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.

Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
  u16 to u32. The previous u16 silently truncated payloads above
  64 KiB — a hard correctness ceiling that blocked inline file ops
  up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
  only. Cross-platform Kotlin port (android/shade-android) updated to
  match; test-vectors/wire-format.json regenerated.

Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
  Concurrent decryptions of the same peer raced ratchet state
  (manifested as sporadic "Failed to decrypt — wrong key or tampered
  data" under load — surfaced once concurrent uploadDirectory pumped
  many writes in flight). Encrypt was already serialized via
  Shade.send's encryptChains; decrypt is now serialized at the
  manager layer too.

@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
  application-level key/value pairs that round-trip verbatim through
  stream-init plaintext. Used by @shade/files for write/read
  correlation; available to any consumer.

@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
  BackgroundTasks.setHook(name, fn) for runtime hook registration.

Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
  shade-sdk streams-bridge + shade-widgets transfer hooks were
  uncommitted prior to this session. Including them keeps the
  workspace consistent at 0.3.0 since @shade/files depends on them.

Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
  3× stable). Coverage spans unit (inline-threshold + concurrency),
  integration (read-write inline + streams up to 1 MiB, walk +
  upload/download directory, custom-op, metrics, SDK namespace
  end-to-end), and security (tampered-envelope sig verification,
  replay window, fingerprint gate, rate-limit + quota).

Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
  topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
  server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

packages/shade-proto/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@shade/proto",
-  "version": "0.1.0",
+  "version": "0.3.0",
   "type": "module",
   "main": "src/index.ts",
   "types": "src/index.ts",

packages/shade-proto/src/index.ts

@@ -1 +1,11 @@
-export { encodeEnvelope, decodeEnvelope, encodePreKeyMessage, encodeRatchetMessage } from './wire.js';
+export {
+  encodeEnvelope,
+  decodeEnvelope,
+  encodePreKeyMessage,
+  encodeRatchetMessage,
+  encodeStreamChunk,
+  decodeStreamChunk,
+  inspectEnvelopeType,
+  TYPE_STREAM_CHUNK,
+} from './wire.js';
+export type { StreamChunkWire } from './wire.js';

packages/shade-proto/src/wire.ts

@@ -6,17 +6,45 @@
  * Types:
  *   0x01 = PreKeyMessage
  *   0x02 = RatchetMessage
+ *   0x11 = StreamChunk
  *
  * All multi-byte integers are big-endian.
- * All byte arrays are length-prefixed (2-byte length + data).
+ *
+ * Length prefixes are 4-byte (u32) since wire VERSION 0x02. (VERSION 0x01
+ * used 2-byte/u16 length prefixes which silently truncated payloads larger
+ * than 64 KiB — a hard correctness ceiling that blocked inline file ops
+ * up to 256 KiB. The bump is incompatible with 0.2.x peers.)
  */
 
 import type { PreKeyMessage, RatchetMessage, ShadeEnvelope } from '@shade/core';
 
-const VERSION = 0x01;
+const VERSION = 0x02;
 
 const TYPE_PREKEY = 0x01;
 const TYPE_RATCHET = 0x02;
+export const TYPE_STREAM_CHUNK = 0x11;
+
+// ─── Stream chunk types ──────────────────────────────────────
+
+/**
+ * Wire-decoded stream-chunk envelope (type 0x11).
+ *
+ * See spec §2.2. The nonce is deterministic — derived from
+ * (laneId, seq) on both sides — but is also serialized over the wire for
+ * self-description and validated by the receiver.
+ */
+export interface StreamChunkWire {
+  streamId: Uint8Array;     // 16 bytes
+  laneId: number;           // u32
+  seq: number | bigint;     // u64
+  isLast: boolean;
+  nonce: Uint8Array;        // 12 bytes
+  aad: Uint8Array;          // additional bound data (length=0 in v0.2.0, reserved)
+  ciphertext: Uint8Array;   // AES-GCM ciphertext including 16-byte tag
+}
+
+const STREAM_ID_BYTES = 16;
+const STREAM_NONCE_BYTES = 12;
 
 // ─── Encode ──────────────────────────────────────────────────
 
@@ -72,6 +100,59 @@ function encodeRatchetMessageInner(msg: RatchetMessage): Uint8Array {
   return concat(parts);
 }
 
+/**
+ * Encode a stream-chunk envelope to wire bytes (type 0x11).
+ *
+ * Layout: see `shade-streams-spec.md` §2.2.
+ */
+export function encodeStreamChunk(c: StreamChunkWire): Uint8Array {
+  if (c.streamId.length !== STREAM_ID_BYTES) {
+    throw new Error(`streamId must be ${STREAM_ID_BYTES} bytes`);
+  }
+  if (c.nonce.length !== STREAM_NONCE_BYTES) {
+    throw new Error(`nonce must be ${STREAM_NONCE_BYTES} bytes`);
+  }
+  if (!Number.isInteger(c.laneId) || c.laneId < 0 || c.laneId > 0xffff_ffff) {
+    throw new Error(`laneId out of u32 range: ${c.laneId}`);
+  }
+  const seqBig = typeof c.seq === 'bigint' ? c.seq : BigInt(c.seq);
+  if (seqBig < 0n || seqBig > 0xffff_ffff_ffff_ffffn) {
+    throw new Error(`seq out of u64 range: ${c.seq}`);
+  }
+
+  const headerSize = 1 + 1 + STREAM_ID_BYTES + 4 + 8 + 1 + STREAM_NONCE_BYTES + 4 + c.aad.length + 4;
+  const out = new Uint8Array(headerSize + c.ciphertext.length);
+  const view = new DataView(out.buffer);
+  let offset = 0;
+
+  out[offset++] = VERSION;
+  out[offset++] = TYPE_STREAM_CHUNK;
+  out.set(c.streamId, offset);
+  offset += STREAM_ID_BYTES;
+
+  view.setUint32(offset, c.laneId, false);
+  offset += 4;
+
+  view.setBigUint64(offset, seqBig, false);
+  offset += 8;
+
+  out[offset++] = c.isLast ? 0x01 : 0x00;
+
+  out.set(c.nonce, offset);
+  offset += STREAM_NONCE_BYTES;
+
+  view.setUint32(offset, c.aad.length, false);
+  offset += 4;
+  out.set(c.aad, offset);
+  offset += c.aad.length;
+
+  view.setUint32(offset, c.ciphertext.length, false);
+  offset += 4;
+  out.set(c.ciphertext, offset);
+
+  return out;
+}
+
 // ─── Decode ──────────────────────────────────────────────────
 
 export function decodeEnvelope(data: Uint8Array): ShadeEnvelope {
@@ -93,6 +174,80 @@ export function decodeEnvelope(data: Uint8Array): ShadeEnvelope {
   throw new Error(`Unknown type: ${type}`);
 }
 
+/**
+ * Decode a stream-chunk envelope from wire bytes (type 0x11).
+ * Throws if the data is malformed or the type tag is wrong.
+ */
+export function decodeStreamChunk(data: Uint8Array): StreamChunkWire {
+  const minHeaderSize = 2 + STREAM_ID_BYTES + 4 + 8 + 1 + STREAM_NONCE_BYTES + 4 + 4;
+  if (data.length < minHeaderSize) {
+    throw new Error(`stream-chunk too short: ${data.length} < ${minHeaderSize}`);
+  }
+  if (data[0] !== VERSION) {
+    throw new Error(`Unknown version: ${data[0]}`);
+  }
+  if (data[1] !== TYPE_STREAM_CHUNK) {
+    throw new Error(`Not a stream-chunk: type=${data[1]}`);
+  }
+
+  const view = new DataView(data.buffer, data.byteOffset);
+  let offset = 2;
+
+  const streamId = data.slice(offset, offset + STREAM_ID_BYTES);
+  offset += STREAM_ID_BYTES;
+
+  const laneId = view.getUint32(offset, false);
+  offset += 4;
+
+  const seq = view.getBigUint64(offset, false);
+  offset += 8;
+
+  const isLast = data[offset] === 0x01;
+  offset += 1;
+
+  const nonce = data.slice(offset, offset + STREAM_NONCE_BYTES);
+  offset += STREAM_NONCE_BYTES;
+
+  const aadLen = view.getUint32(offset, false);
+  offset += 4;
+  if (offset + aadLen + 4 > data.length) {
+    throw new Error('stream-chunk truncated in aad/ctLen');
+  }
+  const aad = data.slice(offset, offset + aadLen);
+  offset += aadLen;
+
+  const ctLen = view.getUint32(offset, false);
+  offset += 4;
+  if (offset + ctLen !== data.length) {
+    throw new Error(
+      `stream-chunk length mismatch: declared ${offset + ctLen}, actual ${data.length}`,
+    );
+  }
+  const ciphertext = data.slice(offset, offset + ctLen);
+
+  return { streamId, laneId, seq, isLast, nonce, aad, ciphertext };
+}
+
+/**
+ * Inspect the type tag of an arbitrary envelope without full parsing.
+ * Returns one of `'prekey' | 'ratchet' | 'stream-chunk' | 'unknown'`.
+ */
+export function inspectEnvelopeType(
+  data: Uint8Array,
+): 'prekey' | 'ratchet' | 'stream-chunk' | 'unknown' {
+  if (data.length < 2 || data[0] !== VERSION) return 'unknown';
+  switch (data[1]) {
+    case TYPE_PREKEY:
+      return 'prekey';
+    case TYPE_RATCHET:
+      return 'ratchet';
+    case TYPE_STREAM_CHUNK:
+      return 'stream-chunk';
+    default:
+      return 'unknown';
+  }
+}
+
 function decodePreKeyMessageInner(data: Uint8Array): PreKeyMessage {
   let offset = 0;
 
@@ -107,14 +262,15 @@ function decodePreKeyMessageInner(data: Uint8Array): PreKeyMessage {
 
   const ratchet = decodeRatchetMessageInner(ratchetData.value, 0);
 
-  return {
+  const msg: PreKeyMessage = {
     registrationId,
-    preKeyId,
     signedPreKeyId,
     ephemeralKey: ephemeral.value,
     identityDHKey: identityDH.value,
     message: ratchet.value,
   };
+  if (preKeyId !== undefined) msg.preKeyId = preKeyId;
+  return msg;
 }
 
 function decodeRatchetMessageInner(data: Uint8Array, offset: number): { value: RatchetMessage; end: number } {
@@ -145,8 +301,8 @@ function uint32(n: number): Uint8Array {
 }
 
 function lpBytes(data: Uint8Array): Uint8Array {
-  const len = new Uint8Array(2);
-  new DataView(len.buffer).setUint16(0, data.length, false);
+  const len = new Uint8Array(4);
+  new DataView(len.buffer).setUint32(0, data.length, false);
   return concat([len, data]);
 }
 
@@ -155,9 +311,9 @@ function readUint32(data: Uint8Array, offset: number): number {
 }
 
 function readLP(data: Uint8Array, offset: number): { value: Uint8Array; end: number } {
-  const len = new DataView(data.buffer, data.byteOffset + offset).getUint16(0, false);
-  const value = data.slice(offset + 2, offset + 2 + len);
-  return { value, end: offset + 2 + len };
+  const len = new DataView(data.buffer, data.byteOffset + offset).getUint32(0, false);
+  const value = data.slice(offset + 4, offset + 4 + len);
+  return { value, end: offset + 4 + len };
 }
 
 function concat(parts: Uint8Array[]): Uint8Array {

packages/shade-proto/tests/wire.test.ts

@@ -1,5 +1,14 @@
 import { describe, test, expect } from 'bun:test';
-import { encodeEnvelope, decodeEnvelope, encodePreKeyMessage, encodeRatchetMessage } from '../src/index.js';
+import {
+  encodeEnvelope,
+  decodeEnvelope,
+  encodeRatchetMessage,
+  encodeStreamChunk,
+  decodeStreamChunk,
+  inspectEnvelopeType,
+  TYPE_STREAM_CHUNK,
+} from '../src/index.js';
+import type { StreamChunkWire } from '../src/index.js';
 import type { PreKeyMessage, RatchetMessage, ShadeEnvelope } from '@shade/core';
 
 function randBytes(n: number): Uint8Array {
@@ -173,13 +182,124 @@ describe('Wire Format', () => {
     });
 
     test('rejects unknown type', () => {
-      const data = new Uint8Array([0x01, 0xFF]);
+      const data = new Uint8Array([0x02, 0xFF]);
       expect(() => decodeEnvelope(data)).toThrow('Unknown type');
     });
 
     test('rejects too-short data', () => {
-      expect(() => decodeEnvelope(new Uint8Array([0x01]))).toThrow('Too short');
+      expect(() => decodeEnvelope(new Uint8Array([0x02]))).toThrow('Too short');
       expect(() => decodeEnvelope(new Uint8Array([]))).toThrow('Too short');
     });
   });
+
+  // ─── StreamChunk (type 0x11) ───────────────────────────────
+
+  describe('StreamChunk', () => {
+    function makeChunk(overrides: Partial<StreamChunkWire> = {}): StreamChunkWire {
+      return {
+        streamId: randBytes(16),
+        laneId: 0,
+        seq: 0n,
+        isLast: false,
+        nonce: randBytes(12),
+        aad: new Uint8Array(0),
+        ciphertext: randBytes(64),
+        ...overrides,
+      };
+    }
+
+    test('encode/decode roundtrip', () => {
+      const c = makeChunk({ laneId: 7, seq: 42n, isLast: true });
+      const encoded = encodeStreamChunk(c);
+      const decoded = decodeStreamChunk(encoded);
+      expect(decoded.streamId).toEqual(c.streamId);
+      expect(decoded.laneId).toBe(7);
+      expect(decoded.seq).toBe(42n);
+      expect(decoded.isLast).toBe(true);
+      expect(decoded.nonce).toEqual(c.nonce);
+      expect(decoded.aad.length).toBe(0);
+      expect(decoded.ciphertext).toEqual(c.ciphertext);
+    });
+
+    test('emits the correct type tag', () => {
+      const encoded = encodeStreamChunk(makeChunk());
+      expect(encoded[0]).toBe(0x02); // version
+      expect(encoded[1]).toBe(TYPE_STREAM_CHUNK);
+    });
+
+    test('handles empty ciphertext', () => {
+      const c = makeChunk({ ciphertext: new Uint8Array(0) });
+      const decoded = decodeStreamChunk(encodeStreamChunk(c));
+      expect(decoded.ciphertext.length).toBe(0);
+    });
+
+    test('handles 16 MiB ciphertext (max chunk + tag)', () => {
+      const c = makeChunk({ ciphertext: randBytes(16 * 1024 * 1024 + 16) });
+      const encoded = encodeStreamChunk(c);
+      const decoded = decodeStreamChunk(encoded);
+      expect(decoded.ciphertext.length).toBe(c.ciphertext.length);
+    });
+
+    test('handles MAX_SEQ', () => {
+      const max = 0xffff_ffff_ffff_ffffn;
+      const c = makeChunk({ seq: max });
+      const decoded = decodeStreamChunk(encodeStreamChunk(c));
+      expect(decoded.seq).toBe(max);
+    });
+
+    test('rejects wrong-length streamId', () => {
+      expect(() => encodeStreamChunk(makeChunk({ streamId: randBytes(15) }))).toThrow();
+    });
+
+    test('rejects wrong-length nonce', () => {
+      expect(() => encodeStreamChunk(makeChunk({ nonce: randBytes(11) }))).toThrow();
+    });
+
+    test('rejects out-of-range laneId', () => {
+      expect(() => encodeStreamChunk(makeChunk({ laneId: 0x1_0000_0000 }))).toThrow();
+      expect(() => encodeStreamChunk(makeChunk({ laneId: -1 }))).toThrow();
+    });
+
+    test('decode rejects wrong type tag', () => {
+      const valid = encodeStreamChunk(makeChunk());
+      const tampered = valid.slice();
+      tampered[1] = 0x02; // ratchet
+      expect(() => decodeStreamChunk(tampered)).toThrow();
+    });
+
+    test('decode rejects wrong version', () => {
+      const valid = encodeStreamChunk(makeChunk());
+      const tampered = valid.slice();
+      tampered[0] = 0x01; // bumped to 0x02 in v0.3.0; 0x01 is the legacy version
+      expect(() => decodeStreamChunk(tampered)).toThrow();
+    });
+
+    test('decode rejects truncated body', () => {
+      const valid = encodeStreamChunk(makeChunk());
+      expect(() => decodeStreamChunk(valid.slice(0, valid.length - 5))).toThrow();
+    });
+
+    test('decode rejects body extended beyond ctLen', () => {
+      const valid = encodeStreamChunk(makeChunk());
+      const tampered = new Uint8Array(valid.length + 1);
+      tampered.set(valid);
+      expect(() => decodeStreamChunk(tampered)).toThrow();
+    });
+
+    test('inspectEnvelopeType identifies stream-chunk', () => {
+      expect(inspectEnvelopeType(encodeStreamChunk(makeChunk()))).toBe('stream-chunk');
+    });
+
+    test('inspectEnvelopeType identifies ratchet/prekey/unknown', () => {
+      const ratchet: ShadeEnvelope = {
+        type: 'ratchet',
+        content: makeRatchetMessage(),
+        timestamp: 0,
+        senderAddress: '',
+      };
+      expect(inspectEnvelopeType(encodeEnvelope(ratchet))).toBe('ratchet');
+      expect(inspectEnvelopeType(new Uint8Array([0x02, 0x99]))).toBe('unknown');
+      expect(inspectEnvelopeType(new Uint8Array([]))).toBe('unknown');
+    });
+  });
 });