feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive

M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).

@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
  Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
  + per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
  (> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
  / shadeFilesReadStreamId correlation. Server-side TransformStream
  bridges accept inbound transfers immediately (engine rejects chunks
  that arrive before accept) and park the readable for the matching
  RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
  uploadDirectory()/downloadDirectory() with bounded concurrency pool
  (default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
  byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
  policy (traversal + percent-decode hardening), fingerprint gate
  (required/optional/reject), pluggable Ed25519 sig verification with
  ±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
  useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
  entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.

Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
  u16 to u32. The previous u16 silently truncated payloads above
  64 KiB — a hard correctness ceiling that blocked inline file ops
  up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
  only. Cross-platform Kotlin port (android/shade-android) updated to
  match; test-vectors/wire-format.json regenerated.

Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
  Concurrent decryptions of the same peer raced ratchet state
  (manifested as sporadic "Failed to decrypt — wrong key or tampered
  data" under load — surfaced once concurrent uploadDirectory pumped
  many writes in flight). Encrypt was already serialized via
  Shade.send's encryptChains; decrypt is now serialized at the
  manager layer too.

@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
  application-level key/value pairs that round-trip verbatim through
  stream-init plaintext. Used by @shade/files for write/read
  correlation; available to any consumer.

@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
  BackgroundTasks.setHook(name, fn) for runtime hook registration.

Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
  shade-sdk streams-bridge + shade-widgets transfer hooks were
  uncommitted prior to this session. Including them keeps the
  workspace consistent at 0.3.0 since @shade/files depends on them.

Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
  3× stable). Coverage spans unit (inline-threshold + concurrency),
  integration (read-write inline + streams up to 1 MiB, walk +
  upload/download directory, custom-op, metrics, SDK namespace
  end-to-end), and security (tampered-envelope sig verification,
  replay window, fingerprint gate, rate-limit + quota).

Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
  topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
  server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

packages/shade-sdk/src/shade.ts

@@ -5,10 +5,27 @@ import {
   NoSessionError,
 } from '@shade/core';
 import { SubtleCryptoProvider, MemoryStorage } from '@shade/crypto-web';
+import { encodeEnvelope, decodeEnvelope, inspectEnvelopeType } from '@shade/proto';
 import { ShadeFetchTransport } from '@shade/transport';
-import { BackgroundTasks, type BackgroundHooks } from './background.js';
-import { exportBackup, importBackup, backupToString, backupFromString, type BackupBlob } from './backup.js';
+import {
+  TransferEngine,
+  ShadeTransferHttpTransport,
+  type ITransferTransport,
+  type IncomingTransfer,
+  type TransferHandle,
+  type TransferOptions,
+  type TransferSummary,
+} from '@shade/transfer';
+import type { Hono } from 'hono';
+import { BackgroundTasks } from './background.js';
+import { exportBackup, importBackup, backupToString, backupFromString } from './backup.js';
 import type { ResolvedConfig } from './config.js';
+import {
+  ShadeControlChannel,
+  ShadeTransferAuthenticator,
+  type ControlEnvelopeTransport,
+} from './streams-bridge.js';
+import { createFilesNamespace, type FilesNamespace } from '@shade/files';
 
 /**
  * The high-level Shade API.
@@ -23,7 +40,7 @@ export class Shade {
   private storage!: StorageProvider;
   private manager!: ShadeSessionManager;
   private transport!: ShadeFetchTransport;
-  private background: BackgroundTasks | null = null;
+  private _background: BackgroundTasks | null = null;
   private address!: string;
   private initialized = false;
 
@@ -32,8 +49,19 @@ export class Shade {
   // Per-address encrypt queue to serialize ratchet mutations
   private encryptChains = new Map<string, Promise<unknown>>();
 
-  // Message handlers
-  private messageHandlers: Array<(from: string, plaintext: string) => void> = [];
+  // Message handlers — may be sync or async; receive() awaits each.
+  private messageHandlers: Array<
+    (from: string, plaintext: string) => void | Promise<void>
+  > = [];
+
+  // Stream-transfer engine, lazily constructed on first use.
+  private transferEngine: TransferEngine | null = null;
+  private controlChannel: ShadeControlChannel | null = null;
+  private peerBaseUrlResolver: ((peerAddress: string) => Promise<string>) | null = null;
+  private envelopeOutboxes: ControlEnvelopeTransport | null = null;
+
+  // `@shade/files` namespace, lazy + memoized.
+  private filesNamespace: FilesNamespace | null = null;
 
   constructor(private readonly config: ResolvedConfig) {}
 
@@ -88,13 +116,13 @@ export class Shade {
     }
 
     // Step 6: Background tasks
-    this.background = new BackgroundTasks(
+    this._background = new BackgroundTasks(
       this.manager,
       this.transport,
       this.address,
       this.config,
     );
-    this.background.start();
+    this._background.start();
 
     this.initialized = true;
   }
@@ -111,6 +139,35 @@ export class Shade {
     return this.address;
   }
 
+  /**
+   * `@shade/files` namespace — high-level entry point for E2EE filesystem
+   * RPC. Lazily creates the underlying channel + streams bridges on first
+   * access; subsequent accesses return the same instance.
+   *
+   * ```ts
+   * const files = shade.files;
+   * const stop = await files.serve({ list: ..., write: ..., ... });
+   * const fs = await files.client('bob');
+   * await fs.list('/');
+   * ```
+   *
+   * Requires `configureTransfers({ resolveBaseUrl })` to be called first
+   * (same as `upload`/`onIncomingTransfer`).
+   */
+  get files(): FilesNamespace {
+    if (!this.initialized) throw new Error('Not initialized');
+    if (this.filesNamespace !== null) return this.filesNamespace;
+    // `@shade/files` only imports `Shade` as a type, so the cyclic ESM
+    // import is type-only at the value layer — safe to bind synchronously.
+    this.filesNamespace = createFilesNamespace(this);
+    return this.filesNamespace;
+  }
+
+  /** Internal — exposes the BackgroundTasks for `@shade/files` to wire prune. */
+  get background(): BackgroundTasks | null {
+    return this._background;
+  }
+
   /** Access the underlying event emitter (for observer integration) */
   getEvents(): ShadeEventEmitter {
     return this.events;
@@ -164,7 +221,7 @@ export class Shade {
     const plaintext = await this.manager.decrypt(from, envelope);
     for (const handler of this.messageHandlers) {
       try {
-        handler(from, plaintext);
+        await handler(from, plaintext);
       } catch (err) {
         console.error('[Shade] Message handler threw:', err);
       }
@@ -172,8 +229,10 @@ export class Shade {
     return plaintext;
   }
 
-  /** Register a handler for incoming messages */
-  onMessage(handler: (from: string, plaintext: string) => void): () => void {
+  /** Register a handler for incoming messages. Async handlers are awaited. */
+  onMessage(
+    handler: (from: string, plaintext: string) => void | Promise<void>,
+  ): () => void {
     this.messageHandlers.push(handler);
     return () => {
       this.messageHandlers = this.messageHandlers.filter((h) => h !== handler);
@@ -218,23 +277,23 @@ export class Shade {
     );
 
     // Rebuild background tasks so they use the new transport
-    if (this.background) {
-      this.background.stop();
-      this.background = new BackgroundTasks(
+    if (this._background) {
+      this._background.stop();
+      this._background = new BackgroundTasks(
         this.manager,
         this.transport,
         this.address,
         this.config,
       );
-      this.background.start();
+      this._background.start();
     }
   }
 
   /** Manually trigger replenishment (normally background task handles this) */
   async replenish(): Promise<number> {
     if (!this.initialized) throw new Error('Not initialized');
-    if (!this.background) return 0;
-    return this.background.runReplenish();
+    if (!this._background) return 0;
+    return this._background.runReplenish();
   }
 
   /**
@@ -273,7 +332,9 @@ export class Shade {
 
   /** Clean shutdown: stop timers, close storage if it supports it */
   async shutdown(): Promise<void> {
-    this.background?.stop();
+    this._background?.stop();
+    if (this.transferEngine !== null) this.transferEngine.destroy();
+    if (this.controlChannel !== null) this.controlChannel.destroy();
     // Close storage if it has a close method (SQLite)
     const closable = this.storage as unknown as { close?: () => void | Promise<void> };
     if (typeof closable.close === 'function') {
@@ -282,8 +343,166 @@ export class Shade {
     this.initialized = false;
   }
 
+  // ─── Stream transfers (v0.2.0) ─────────────────────────────
+
+  /**
+   * Configure how stream-transfer chunks reach peers. Provide a resolver
+   * that returns the peer's HTTP base URL (e.g. by looking up a
+   * `transfer.baseUrl` field in your prekey-bundle metadata or a static
+   * directory map). If unset, `upload()` rejects with a clear error.
+   *
+   * Optionally also override the control-envelope transport (defaults to
+   * HTTP POSTs to `<base>/v1/transfer/control`).
+   */
+  configureTransfers(opts: {
+    resolveBaseUrl: (peerAddress: string) => Promise<string>;
+    envelopeTransport?: ControlEnvelopeTransport;
+  }): void {
+    this.peerBaseUrlResolver = opts.resolveBaseUrl;
+    this.envelopeOutboxes =
+      opts.envelopeTransport ?? new HttpEnvelopeTransport(opts.resolveBaseUrl, this.address);
+  }
+
+  /**
+   * Deliver a freshly-encrypted ratchet envelope to a peer using the
+   * configured envelope transport (HTTP POST to `/v1/transfer/control` by
+   * default). Used by `@shade/files` for RPC plaintext delivery.
+   */
+  async deliverControlEnvelope(peerAddress: string, envelope: ShadeEnvelope): Promise<void> {
+    if (this.envelopeOutboxes === null) {
+      throw new Error(
+        'Call shade.configureTransfers({ resolveBaseUrl }) before deliverControlEnvelope()',
+      );
+    }
+    await this.envelopeOutboxes.send(peerAddress, envelope);
+  }
+
+  /**
+   * Upload bytes to a peer. Returns a `TransferHandle` that can be paused/
+   * aborted and awaited. Requires `configureTransfers` to be called first.
+   */
+  async upload(opts: TransferOptions): Promise<TransferHandle> {
+    return (await this.engine()).upload(opts);
+  }
+
+  /**
+   * Subscribe to incoming transfers from peers. Handler is invoked when a
+   * `stream-init` arrives; the handler MUST call `incoming.accept(...)` to
+   * begin receiving (or `incoming.decline(...)` to reject).
+   */
+  async onIncomingTransfer(
+    handler: (incoming: IncomingTransfer) => void | Promise<void>,
+  ): Promise<() => void> {
+    return (await this.engine()).onIncomingTransfer(handler);
+  }
+
+  /**
+   * Mount the receiver-side HTTP routes on a Hono app. Mount under any
+   * base path: `app.route('/shade', await shade.transferRoute())`.
+   *
+   * Routes:
+   *   POST /v1/transfer/:streamId/chunk  — wire-encoded 0x11 chunks
+   *   GET  /v1/transfer/:streamId/state  — resume-state lookup
+   *   POST /v1/transfer/control          — wire-encoded 0x02 control envelopes
+   *   GET  /v1/transfer/health           — peer reachability probe
+   */
+  async transferRoute(): Promise<Hono> {
+    const engine = await this.engine();
+    const { createTransferRoutes, PermissiveAuthenticator } = await import('@shade/transfer');
+    const app = await createTransferRoutes(engine, {
+      authenticator: PermissiveAuthenticator,
+    });
+    // Add the control-envelope POST route on top.
+    app.post('/v1/transfer/control', async (c) => {
+      const senderAddress = c.req.header('X-Shade-Sender-Address');
+      if (senderAddress === undefined || senderAddress === '') {
+        return c.json({ error: 'missing X-Shade-Sender-Address' }, 400);
+      }
+      const ab = await c.req.arrayBuffer();
+      const bytes = new Uint8Array(ab);
+      try {
+        await this.acceptTransferEnvelope(senderAddress, bytes);
+      } catch (err) {
+        return c.json({ error: (err as Error).message }, 400);
+      }
+      return c.json({ ok: true });
+    });
+    return app;
+  }
+
+  /**
+   * Low-level entry for custom transports: hand a `0x02` ratchet envelope
+   * (control-plane) or a `0x11` stream-chunk envelope to the engine.
+   * Used internally by `transferRoute()`.
+   */
+  async acceptTransferEnvelope(from: string, env: ShadeEnvelope | Uint8Array): Promise<void> {
+    const engine = await this.engine();
+    if (env instanceof Uint8Array) {
+      const kind = inspectEnvelopeType(env);
+      if (kind === 'stream-chunk') {
+        // Engine extracts laneId/seq from the wire bytes via decodeStreamChunk.
+        const headers = parseChunkHeader(env);
+        await engine.receiveChunk(from, headers.streamId, headers.laneId, headers.seq, env);
+        return;
+      }
+      if (kind === 'ratchet' || kind === 'prekey') {
+        const decoded = decodeEnvelope(env);
+        await this.controlChannel!.acceptEnvelope(from, decoded);
+        return;
+      }
+      throw new Error(`Unknown envelope type ${kind}`);
+    }
+    // Already-decoded envelope (ratchet or prekey)
+    await this.controlChannel!.acceptEnvelope(from, env);
+  }
+
   // ─── Internals ─────────────────────────────────────────────
 
+  private async engine(): Promise<TransferEngine> {
+    if (this.transferEngine !== null) return this.transferEngine;
+    if (this.peerBaseUrlResolver === null || this.envelopeOutboxes === null) {
+      throw new Error(
+        'Call shade.configureTransfers({ resolveBaseUrl }) before using upload()/onIncomingTransfer()',
+      );
+    }
+    this.controlChannel = new ShadeControlChannel(this, this.envelopeOutboxes);
+    const transport: ITransferTransport = new ShadeTransferHttpTransport({
+      resolveBaseUrl: this.peerBaseUrlResolver,
+      authenticator: await this.makeAuthenticator(),
+    });
+    this.transferEngine = new TransferEngine({
+      crypto: this.crypto,
+      controlChannel: this.controlChannel,
+      transport,
+      myAddress: this.address,
+    });
+    return this.transferEngine;
+  }
+
+  private async makeAuthenticator(): Promise<ShadeTransferAuthenticator> {
+    const identity = await this.storage.getIdentityKeyPair();
+    if (identity === null) throw new Error('Identity not initialized');
+    return new ShadeTransferAuthenticator(this.crypto, this.address, identity.signingPrivateKey);
+  }
+
+  /** Returns a list of in-flight stream transfers from storage (resume support). */
+  async listTransfers(filter?: {
+    direction?: 'send' | 'receive';
+  }): Promise<TransferSummary[]> {
+    if (this.storage.listActiveStreamStates === undefined) return [];
+    const rows = await this.storage.listActiveStreamStates(filter?.direction);
+    return rows.map((s) => ({
+      streamId: s.streamId,
+      direction: s.direction,
+      peerAddress: s.peerAddress,
+      status: s.status,
+      bytesProcessed: 0, // computed from laneState
+      createdAt: s.createdAt,
+      updatedAt: s.updatedAt,
+      metadata: tryParseMetadata(s.metadataJson),
+    }));
+  }
+
   private async ensureSession(address: string): Promise<void> {
     // Deduplicate concurrent establishment requests
     const existing = this.establishing.get(address);
@@ -306,6 +525,57 @@ export class Shade {
   }
 }
 
+function tryParseMetadata(json: string): import('@shade/streams').StreamMetadata | null {
+  try {
+    return JSON.parse(json) as import('@shade/streams').StreamMetadata;
+  } catch {
+    return null;
+  }
+}
+
+function parseChunkHeader(bytes: Uint8Array): {
+  streamId: string;
+  laneId: number;
+  seq: bigint;
+} {
+  // [0]=ver [1]=type [2..18]=streamId(16) [18..22]=laneId u32 [22..30]=seq u64
+  if (bytes.length < 30) throw new Error('truncated stream-chunk header');
+  const view = new DataView(bytes.buffer, bytes.byteOffset);
+  const sidBytes = bytes.slice(2, 18);
+  const laneId = view.getUint32(18, false);
+  const seq = view.getBigUint64(22, false);
+  // Encode streamId as base64url
+  let bin = '';
+  for (let i = 0; i < sidBytes.length; i++) bin += String.fromCharCode(sidBytes[i]!);
+  const streamId = btoa(bin).replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, '');
+  return { streamId, laneId, seq };
+}
+
+// ─── Default HTTP envelope transport ──────────────────────────
+
+class HttpEnvelopeTransport implements ControlEnvelopeTransport {
+  constructor(
+    private readonly resolveBaseUrl: (peerAddress: string) => Promise<string>,
+    private readonly myAddress: string,
+  ) {}
+  async send(peerAddress: string, envelope: ShadeEnvelope): Promise<void> {
+    const base = (await this.resolveBaseUrl(peerAddress)).replace(/\/$/, '');
+    const url = `${base}/v1/transfer/control`;
+    const bytes = encodeEnvelope(envelope);
+    const res = await fetch(url, {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/octet-stream',
+        'X-Shade-Sender-Address': this.myAddress,
+      },
+      body: bytes as unknown as never,
+    });
+    if (!res.ok) {
+      throw new Error(`control envelope POST failed: ${res.status} ${await res.text()}`);
+    }
+  }
+}
+
 // ─── Helpers ─────────────────────────────────────────────────
 
 async function resolveStorage(
@@ -326,8 +596,14 @@ async function resolveStorage(
   }
 
   if (typeof spec === 'object' && spec.type === 'postgres') {
-    const { PostgresStorage } = await import('@shade/storage-postgres');
-    return PostgresStorage.create(spec.url);
+    // Dynamic import keeps @shade/storage-postgres optional — consumers that
+    // never use postgres don't need to install it. The string-form import
+    // path makes the resolver lazy at type-check time too.
+    const moduleId = '@shade/storage-postgres';
+    const mod = (await import(moduleId)) as {
+      PostgresStorage: { create(url: string): Promise<StorageProvider> };
+    };
+    return mod.PostgresStorage.create(spec.url);
   }
 
   throw new Error(`Unsupported storage spec: ${JSON.stringify(spec)}`);