# Shade Streams 0.2.0 E2EE chunked upload/download for Shade. Drop into any Shade-using app: ```ts const handle = await shade.upload({ to: 'bob', input: file }); const result = await handle.done(); // { sha256, bytesSent, durationMs } ``` …and on the receiver: ```ts shade.onIncomingTransfer(async (incoming) => { const handle = await incoming.accept({ output: { kind: 'file', path: '/uploads/x' } }); await handle.done(); }); ``` Or in React: ```tsx console.log(r.sha256)} /> ``` ## How it works A transfer has two planes: - **Control plane** — `stream-init`, `stream-finish`, `stream-abort`, and `stream-resume-*` messages, encoded as JSON plaintext and shipped through the existing Double Ratchet (envelope type `0x02`). One ratchet step establishes a stream; the rest is per-chunk AEAD. - **Data plane** — `stream-chunk` envelopes (envelope type `0x11`), AES-256-GCM-encrypted under a per-lane key, shipped over HTTP POST (or WebSocket if opted-in). Lanes run in parallel for throughput. ``` Sender Receiver ────── ──────── streamSecret = randomBytes(32) streamId = randomBytes(16) streamKey = HKDF(streamSecret, streamId, "shade-stream/v1\0master") laneKey[i] = HKDF(streamKey, streamId, "...\0lane\0" || u32(i)) [stream-init JSON over Double Ratchet] ─▶ parses streamSecret, derives same keys spawns L per-lane receivers [chunk 0x11 over HTTP] ─▶ AES-GCM(laneKey[i], plaintext, nonce=laneId||seq, aad=streamId||laneId||seq||isLast) decrypts, verifies, writes to sink (× 4 lanes in parallel) [stream-finish JSON over Double Ratchet] ─▶ verifies per-lane sha256 + overall sha256 throws TransferIntegrityError on mismatch ``` ## Partition strategies - **Range** (default for known-size inputs) — lane `i` owns bytes `[i·N/L, (i+1)·N/L)`. Receiver reconstructs by concatenating lane outputs in laneId order. - **Round-robin** (default for unknown-size streams) — chunk `i` goes to lane `i mod L`. Receiver reorders via a per-stream chunk-index buffer. ## Resume Persistence is opt-in via a `ResumeStore` (memory, SQLite, Postgres, IndexedDB-ready). State persisted on init; sender's resume queries the receiver's `lastSeqAcked` per lane via `GET /v1/transfer/:streamId/state`, then continues from there. The streamSecret is encrypted at rest under a device-key derived from the local identity's signing private key — a stolen DB without the identity key cannot resume. ```ts const handle = await shade.resumeUpload(streamId, sameInputAsBefore); await handle.done(); ``` Resume across **identity rotation** is not supported (rotation invalidates the device key — by design, to prevent a stolen pre-rotation DB from deriving keys for any post-rotation transfer). Restart the transfer manually after rotation. ## Throughput - Default 4 lanes × 1 MiB chunks × 4 in-flight chunks per lane = 16 MiB peak in-flight per direction. - Memory-bounded: receivers stream chunks to the configured sink without buffering the full payload. 1 GB transfer = O(chunkSize) RSS, not O(file). - AES-GCM is hardware-accelerated via `SubtleCrypto`; SHA-256 streaming via `@noble/hashes`. ## Security properties | ID | Property | |---|---| | S1 | streamSecret never on the wire in plaintext (Double Ratchet only) | | S2 | Unique per-(streamKey, laneId, seq) AEAD nonce — no nonce reuse | | S3 | Tampered chunk header / ciphertext / tag → AEAD reject | | S4 | Per-lane sha256 + overall sha256 verified at finish | | S5 | streamKey/laneKey zeroized on abort/finish (`destroy()`) | | S6 | Concurrent streams have independent lane keys | | S7 | seq overflow practical-impossible (u64 max) | | S8 | At-rest streamSecret encrypted under device-key | ## Hardening `@shade/streams` ships unbounded by default — a peer can declare a 1 PiB transfer and the receiver will dutifully allocate lane state for it. Production receivers must enforce limits at the boundary. The `@shade/files` package wires the same patterns up for its filesystem RPC; copy the shapes that fit your app. ### Per-stream caps The receiver sees the declared plaintext size in the `stream-init` control message before it accepts. Reject above your tolerance: ```ts shade.onIncomingTransfer(async (incoming) => { if (incoming.metadata.totalBytes > 256 * 1024 * 1024) { await incoming.decline({ reason: 'stream too large' }); return; } await incoming.accept({ output: ... }); }); ``` Recommended ceilings (tune to your product, not these): | Tier | totalBytes ceiling | Rationale | |------|--------------------|-----------| | Chat attachment | 25 MiB | matches mobile MMS / Slack expectations | | Photo / doc share | 256 MiB | covers raw RAW + most desktop docs | | Backup / dataset | 4 GiB | larger needs explicit operator opt-in | ### Per-chunk cap `createTransferRoutes` accepts `maxChunkBytes` (default ≈ 16 MiB + header). Lower it if your sink can't absorb that — the receiver will 413 anything over the limit before the chunk is decrypted, which keeps DoS cost bounded. ### Per-sender quotas `@shade/files` ships a `RateLimiter` (`packages/shade-files/src/server/rate-limiter.ts`) that enforces both ops-per-window and bytes-per-hour caps per sender address. The same shape is the recommended template for guarding raw streams: wrap `incoming.accept` in a check that consumes from a token bucket keyed by `incoming.fromAddress`, and reject with `decline()` when the bucket is empty. See `packages/shade-files/tests/security/quota.test.ts` for the test shape. ### TTL on idle streams A `paused` stream-state record consumes a row in your storage and an encrypted streamSecret slot until it expires. Use the **Retention** defaults below to expire abandoned streams; pair with a metric (`shade_stream_states_active`) and an alert when the count grows unbounded. A peer that opens streams and never finishes them is the dominant abuse pattern for resumable transfer. ### Trust gates For high-stakes transfers (backups, key material, internal docs), gate `accept()` on a verified fingerprint. The pattern mirrors `@shade/files`'s fingerprint gate — see `packages/shade-files/tests/security/fingerprint-gate.test.ts`. ## Retention Resumable streams persist a `PersistedStreamState` per in-flight transfer, encrypted under a device key. Without retention, every crashed or abandoned upload leaves a row behind forever. ### Defaults The shipped `bun-server` SDK template (`shade init --template bun-server`) schedules `pruneStreamStates` on a daily cron with a **14-day** horizon. That is: any stream-state record whose `updatedAt` is older than 14 days is removed at the next sweep. If a sender resumes a 14-day-old stream, it will get a "no state" 404 and start over — which is the right answer for a transfer that has been idle for two weeks. ### Tuning the horizon Set `SHADE_STREAM_RETENTION_DAYS` in the template's environment to override the 14-day default. Recommended ranges: | Use case | Horizon | Why | |----------|---------|-----| | Synchronous chat | 1–3 days | resume-after-crash, not resume-after-vacation | | File-share product | 7–14 days | covers a typical user vacation | | Cold backup target | 30+ days | deliberate, but plan for storage growth | ### Hooking the prune call manually If you bring your own server (no `bun-server` template), call the storage method on your own schedule: ```ts import { setInterval } from 'node:timers'; const ONE_DAY_MS = 24 * 60 * 60 * 1000; const HORIZON_MS = 14 * ONE_DAY_MS; setInterval(async () => { if (storage.pruneStreamStates !== undefined) { await storage.pruneStreamStates(Date.now() - HORIZON_MS); } }, ONE_DAY_MS); ``` `pruneStreamStates(olderThan)` removes records whose `updatedAt` is strictly less than `olderThan`. It is idempotent and safe to call concurrently. ## Rich file metadata + previews (V3.9) `stream-init` plaintext can carry an optional `fileMetadata` field that ships filename, MIME-type, and a thumbnail-stream pointer **end-to-end encrypted**. Older receivers ignore the field — backwards-compatible with 0.2.x / 0.3.x peers. ```jsonc { "kind": "shade.stream-init/v1", "streamId": "...", "streamSecret": "...", "metadata": { "chunkSize": 1048576, "sentAt": 1730000000000, "fileMetadata": { "filename": "report.pdf", "mimeType": "application/pdf", "thumbnailStreamId": "Ej1z...", "thumbnailHash": "9a7c...", "thumbnailMime": "image/webp", "thumbnailBytes": 18342 } }, "lanes": [ /* ... */ ] } ``` ### What rides where | Field | Plane | Visible to server? | |-------|-------|--------------------| | `filename` | inside Double Ratchet plaintext | no | | `mimeType` | inside Double Ratchet plaintext | no | | `thumbnailStreamId` | streamId of companion stream | yes (random ID, no info leak) | | `thumbnailHash` | sha256 of preview plaintext | base64 hash only, no pixels | | `thumbnailMime` | one of `image/jpeg / image/webp / image/png` | yes (allowlist enforced) | | `thumbnailBytes` | declared length, capped at 64 KiB | yes | | thumbnail bytes themselves | separate AEAD stream, own lane | no | The thumbnail rides as its **own stream-transfer**, keyed independently from the main stream. A server compromise leaks neither preview pixels nor original bytes. ### Sender — attach a preview ```ts // Pre-computed preview (server-side pipeline path): await shade.upload({ to: 'bob', input: pdfBytes, thumbnail: { bytes: previewWebp, mime: 'image/webp' }, metadata: { fileMetadata: { filename: 'report.pdf', mimeType: 'application/pdf' } }, }); // Browser auto-generation (image File / Blob → 256×256 preview): await shade.upload({ to: 'bob', input: imageFile, // a `File` from generateThumbnail: true, // OffscreenCanvas + createImageBitmap }); ``` `generateThumbnail` is a no-op on runtimes lacking `OffscreenCanvas + createImageBitmap` (Bun, Node) — those callers should pre-generate and pass `thumbnail` directly, or skip the preview entirely. ### Receiver — render in widgets The bundled `@shade/widgets` `useShadeDownload` hook auto-accepts thumbnail streams (marked by `userMetadata.shadeThumbnail = '1'`) into an in-memory `ShadeThumbnailCache`. `` reads from the same cache and renders inside an `` element so the browser's image-decoding sandbox is the trust boundary for format parsing. ```tsx ``` ### Format-hardening (sender + receiver) Both sides enforce the same rules — single source of truth in `@shade/streams/file-metadata.ts`: | Rule | Limit | |------|-------| | `thumbnailMime` allowlist | `image/jpeg`, `image/webp`, `image/png` | | `thumbnailBytes` cap | 64 KiB (`THUMBNAIL_MAX_BYTES`) | | `filename` length | ≤ 1024 chars, no control characters | | `mimeType` shape | RFC 7231 `type/subtype` token | | Hash binding | declared `thumbnailHash` = sha256(preview bytes); mismatched bytes are dropped at the cache before any render | A hostile peer cannot: - smuggle exotic image formats past the allowlist (envelope parser rejects at decode-time), - substitute different bytes for a declared preview (cache verifies sha256 before exposing bytes to a renderer), - inflate the cache to OOM the receiver (LRU + 1 MiB total cap). ### Risks consciously accepted - **Preview-arrival ≠ send completion.** A receiver may see the thumbnail before the main upload finishes. For high-stakes flows where "did Alice send X?" is itself sensitive, send the preview *only* after main completion (set `thumbnail` to `null` and instead ship a follow-up `stream-init` with the preview). The default ordering optimizes UX, not metadata-secrecy. - **Renderer trust.** We render through a Blob-URL ``. A 0-day in the browser's image decoder would still reach the receiver. Keep browsers patched; rely on the CSP of your embedding app. ## API surface See package READMEs: - `packages/shade-streams/README.md` — crypto + state machines - `packages/shade-transfer/README.md` — orchestration, transports, persistence - `packages/shade-transport-webrtc/README.md` — V3.11 P2P transport plug-in - `packages/shade-sdk/README.md` — magic drop-in - `packages/shade-widgets/README.md` — React UI ## Transports `@shade/transfer` ships HTTP + WebSocket chunk transports. V3.11 adds an opt-in P2P chunk transport via `RTCDataChannel`: - HTTP — `ShadeTransferHttpTransport`. POST per chunk; the receiver- side route is `app.route('/v1/transfer', await shade.transferRoute())`. - WebSocket — `ShadeTransferWsTransport`. One connection per peer, binary-framed chunks, JSON acks; same wire format inside the frame as the WebRTC transport. - WebRTC — `WebRtcTransferTransport` from `@shade/transport-webrtc`. Wired automatically by `shade.configureWebRTC()` as the primary layer of a `MultiTransportFallback([webrtc, http])`. See [docs/webrtc.md](./webrtc.md). `MultiTransportFallback` is the N-ary generalisation of `FallbackTransferTransport`: pass an ordered list of named transports and the engine demotes sticky on `TransferTransportError`.