Shade/docs/streams.md

# Shade Streams 0.2.0

E2EE chunked upload/download for Shade. Drop into any Shade-using app:

```ts
const handle = await shade.upload({ to: 'bob', input: file });
const result = await handle.done(); // { sha256, bytesSent, durationMs }
```

…and on the receiver:

```ts
shade.onIncomingTransfer(async (incoming) => {
  const handle = await incoming.accept({ output: { kind: 'file', path: '/uploads/x' } });
  await handle.done();
});
```

Or in React:

```tsx
<ShadeRuntimeProvider runtime={shade}>
  <ShadeUploader to="bob" onComplete={(r) => console.log(r.sha256)} />
</ShadeRuntimeProvider>
```

## How it works

A transfer has two planes:

- **Control plane** — `stream-init`, `stream-finish`, `stream-abort`, and
  `stream-resume-*` messages, encoded as JSON plaintext and shipped through
  the existing Double Ratchet (envelope type `0x02`). One ratchet step
  establishes a stream; the rest is per-chunk AEAD.
- **Data plane** — `stream-chunk` envelopes (envelope type `0x11`),
  AES-256-GCM-encrypted under a per-lane key, shipped over HTTP POST (or
  WebSocket if opted-in). Lanes run in parallel for throughput.

```
Sender                                  Receiver
──────                                  ────────
streamSecret = randomBytes(32)
streamId     = randomBytes(16)
streamKey    = HKDF(streamSecret, streamId, "shade-stream/v1\0master")
laneKey[i]   = HKDF(streamKey, streamId, "...\0lane\0" || u32(i))

[stream-init JSON over Double Ratchet] ─▶
                                          parses streamSecret, derives same keys
                                          spawns L per-lane receivers

[chunk 0x11 over HTTP]                 ─▶
   AES-GCM(laneKey[i], plaintext, nonce=laneId||seq, aad=streamId||laneId||seq||isLast)
                                          decrypts, verifies, writes to sink
   (× 4 lanes in parallel)

[stream-finish JSON over Double Ratchet] ─▶
                                          verifies per-lane sha256 + overall sha256
                                          throws TransferIntegrityError on mismatch
```

## Partition strategies

- **Range** (default for known-size inputs) — lane `i` owns bytes
  `[i·N/L, (i+1)·N/L)`. Receiver reconstructs by concatenating lane outputs
  in laneId order.
- **Round-robin** (default for unknown-size streams) — chunk `i` goes to
  lane `i mod L`. Receiver reorders via a per-stream chunk-index buffer.

## Resume

Persistence is opt-in via a `ResumeStore` (memory, SQLite, Postgres,
IndexedDB-ready). State persisted on init; sender's resume queries the
receiver's `lastSeqAcked` per lane via `GET /v1/transfer/:streamId/state`,
then continues from there. The streamSecret is encrypted at rest under a
device-key derived from the local identity's signing private key — a
stolen DB without the identity key cannot resume.

```ts
const handle = await shade.resumeUpload(streamId, sameInputAsBefore);
await handle.done();
```

Resume across **identity rotation** is not supported (rotation invalidates
the device key — by design, to prevent a stolen pre-rotation DB from
deriving keys for any post-rotation transfer). Restart the transfer
manually after rotation.

## Throughput

- Default 4 lanes × 1 MiB chunks × 4 in-flight chunks per lane =
  16 MiB peak in-flight per direction.
- Memory-bounded: receivers stream chunks to the configured sink without
  buffering the full payload. 1 GB transfer = O(chunkSize) RSS, not O(file).
- AES-GCM is hardware-accelerated via `SubtleCrypto`; SHA-256 streaming via
  `@noble/hashes`.

## Security properties

| ID | Property |
|---|---|
| S1 | streamSecret never on the wire in plaintext (Double Ratchet only) |
| S2 | Unique per-(streamKey, laneId, seq) AEAD nonce — no nonce reuse |
| S3 | Tampered chunk header / ciphertext / tag → AEAD reject |
| S4 | Per-lane sha256 + overall sha256 verified at finish |
| S5 | streamKey/laneKey zeroized on abort/finish (`destroy()`) |
| S6 | Concurrent streams have independent lane keys |
| S7 | seq overflow practical-impossible (u64 max) |
| S8 | At-rest streamSecret encrypted under device-key |

## Hardening

`@shade/streams` ships unbounded by default — a peer can declare a
1 PiB transfer and the receiver will dutifully allocate lane state for
it. Production receivers must enforce limits at the boundary. The
`@shade/files` package wires the same patterns up for its filesystem
RPC; copy the shapes that fit your app.

### Per-stream caps

The receiver sees the declared plaintext size in the `stream-init`
control message before it accepts. Reject above your tolerance:

```ts
shade.onIncomingTransfer(async (incoming) => {
  if (incoming.metadata.totalBytes > 256 * 1024 * 1024) {
    await incoming.decline({ reason: 'stream too large' });
    return;
  }
  await incoming.accept({ output: ... });
});
```

Recommended ceilings (tune to your product, not these):

| Tier | totalBytes ceiling | Rationale |
|------|--------------------|-----------|
| Chat attachment | 25 MiB | matches mobile MMS / Slack expectations |
| Photo / doc share | 256 MiB | covers raw RAW + most desktop docs |
| Backup / dataset | 4 GiB | larger needs explicit operator opt-in |

### Per-chunk cap

`createTransferRoutes` accepts `maxChunkBytes` (default ≈ 16 MiB +
header). Lower it if your sink can't absorb that — the receiver will
413 anything over the limit before the chunk is decrypted, which
keeps DoS cost bounded.

### Per-sender quotas

`@shade/files` ships a `RateLimiter` (`packages/shade-files/src/server/rate-limiter.ts`)
that enforces both ops-per-window and bytes-per-hour caps per sender
address. The same shape is the recommended template for guarding raw
streams: wrap `incoming.accept` in a check that consumes from a token
bucket keyed by `incoming.fromAddress`, and reject with `decline()`
when the bucket is empty. See
`packages/shade-files/tests/security/quota.test.ts` for the test
shape.

### TTL on idle streams

A `paused` stream-state record consumes a row in your storage and an
encrypted streamSecret slot until it expires. Use the **Retention**
defaults below to expire abandoned streams; pair with a metric
(`shade_stream_states_active`) and an alert when the count grows
unbounded. A peer that opens streams and never finishes them is the
dominant abuse pattern for resumable transfer.

### Trust gates

For high-stakes transfers (backups, key material, internal docs),
gate `accept()` on a verified fingerprint. The pattern mirrors
`@shade/files`'s fingerprint gate — see
`packages/shade-files/tests/security/fingerprint-gate.test.ts`.

## Retention

Resumable streams persist a `PersistedStreamState` per in-flight
transfer, encrypted under a device key. Without retention, every
crashed or abandoned upload leaves a row behind forever.

### Defaults

The shipped `bun-server` SDK template (`shade init --template bun-server`)
schedules `pruneStreamStates` on a daily cron with a **14-day**
horizon. That is: any stream-state record whose `updatedAt` is older
than 14 days is removed at the next sweep. If a sender resumes a
14-day-old stream, it will get a "no state" 404 and start over —
which is the right answer for a transfer that has been idle for two
weeks.

### Tuning the horizon

Set `SHADE_STREAM_RETENTION_DAYS` in the template's environment to
override the 14-day default. Recommended ranges:

| Use case | Horizon | Why |
|----------|---------|-----|
| Synchronous chat | 1–3 days | resume-after-crash, not resume-after-vacation |
| File-share product | 7–14 days | covers a typical user vacation |
| Cold backup target | 30+ days | deliberate, but plan for storage growth |

### Hooking the prune call manually

If you bring your own server (no `bun-server` template), call the
storage method on your own schedule:

```ts
import { setInterval } from 'node:timers';

const ONE_DAY_MS = 24 * 60 * 60 * 1000;
const HORIZON_MS = 14 * ONE_DAY_MS;

setInterval(async () => {
  if (storage.pruneStreamStates !== undefined) {
    await storage.pruneStreamStates(Date.now() - HORIZON_MS);
  }
}, ONE_DAY_MS);
```

`pruneStreamStates(olderThan)` removes records whose `updatedAt` is
strictly less than `olderThan`. It is idempotent and safe to call
concurrently.

## Rich file metadata + previews (V3.9)

`stream-init` plaintext can carry an optional `fileMetadata` field that
ships filename, MIME-type, and a thumbnail-stream pointer **end-to-end
encrypted**. Older receivers ignore the field — backwards-compatible
with 0.2.x / 0.3.x peers.

```jsonc
{
  "kind": "shade.stream-init/v1",
  "streamId": "...",
  "streamSecret": "...",
  "metadata": {
    "chunkSize": 1048576,
    "sentAt": 1730000000000,
    "fileMetadata": {
      "filename": "report.pdf",
      "mimeType": "application/pdf",
      "thumbnailStreamId": "Ej1z...",
      "thumbnailHash": "9a7c...",
      "thumbnailMime": "image/webp",
      "thumbnailBytes": 18342
    }
  },
  "lanes": [ /* ... */ ]
}
```

### What rides where

| Field | Plane | Visible to server? |
|-------|-------|--------------------|
| `filename` | inside Double Ratchet plaintext | no |
| `mimeType` | inside Double Ratchet plaintext | no |
| `thumbnailStreamId` | streamId of companion stream | yes (random ID, no info leak) |
| `thumbnailHash` | sha256 of preview plaintext | base64 hash only, no pixels |
| `thumbnailMime` | one of `image/jpeg / image/webp / image/png` | yes (allowlist enforced) |
| `thumbnailBytes` | declared length, capped at 64 KiB | yes |
| thumbnail bytes themselves | separate AEAD stream, own lane | no |

The thumbnail rides as its **own stream-transfer**, keyed independently
from the main stream. A server compromise leaks neither preview pixels
nor original bytes.

### Sender — attach a preview

```ts
// Pre-computed preview (server-side pipeline path):
await shade.upload({
  to: 'bob',
  input: pdfBytes,
  thumbnail: { bytes: previewWebp, mime: 'image/webp' },
  metadata: { fileMetadata: { filename: 'report.pdf', mimeType: 'application/pdf' } },
});

// Browser auto-generation (image File / Blob → 256×256 preview):
await shade.upload({
  to: 'bob',
  input: imageFile,                 // a `File` from <input type="file">
  generateThumbnail: true,          // OffscreenCanvas + createImageBitmap
});
```

`generateThumbnail` is a no-op on runtimes lacking
`OffscreenCanvas + createImageBitmap` (Bun, Node) — those callers should
pre-generate and pass `thumbnail` directly, or skip the preview entirely.

### Receiver — render in widgets

The bundled `@shade/widgets` `useShadeDownload` hook auto-accepts
thumbnail streams (marked by `userMetadata.shadeThumbnail = '1'`) into
an in-memory `ShadeThumbnailCache`. `<TransferRow showThumbnail
fileMetadata={...} />` reads from the same cache and renders inside an
`<img>` element so the browser's image-decoding sandbox is the trust
boundary for format parsing.

```tsx
<ShadeThumbnailProvider>
  <TransferRow
    handle={handle}
    progress={progress}
    showThumbnail
    fileMetadata={incoming.metadata.fileMetadata}
  />
</ShadeThumbnailProvider>
```

### Format-hardening (sender + receiver)

Both sides enforce the same rules — single source of truth in
`@shade/streams/file-metadata.ts`:

| Rule | Limit |
|------|-------|
| `thumbnailMime` allowlist | `image/jpeg`, `image/webp`, `image/png` |
| `thumbnailBytes` cap | 64 KiB (`THUMBNAIL_MAX_BYTES`) |
| `filename` length | ≤ 1024 chars, no control characters |
| `mimeType` shape | RFC 7231 `type/subtype` token |
| Hash binding | declared `thumbnailHash` = sha256(preview bytes); mismatched bytes are dropped at the cache before any render |

A hostile peer cannot:
- smuggle exotic image formats past the allowlist (envelope parser
  rejects at decode-time),
- substitute different bytes for a declared preview (cache verifies
  sha256 before exposing bytes to a renderer),
- inflate the cache to OOM the receiver (LRU + 1 MiB total cap).

### Risks consciously accepted

- **Preview-arrival ≠ send completion.** A receiver may see the
  thumbnail before the main upload finishes. For high-stakes flows
  where "did Alice send X?" is itself sensitive, send the preview
  *only* after main completion (set `thumbnail` to `null` and instead
  ship a follow-up `stream-init` with the preview). The default
  ordering optimizes UX, not metadata-secrecy.
- **Renderer trust.** We render through a Blob-URL `<img>`. A 0-day
  in the browser's image decoder would still reach the receiver. Keep
  browsers patched; rely on the CSP of your embedding app.

## API surface

See package READMEs:

- `packages/shade-streams/README.md` — crypto + state machines
- `packages/shade-transfer/README.md` — orchestration, transports, persistence
- `packages/shade-transport-webrtc/README.md` — V3.11 P2P transport plug-in
- `packages/shade-sdk/README.md` — magic drop-in
- `packages/shade-widgets/README.md` — React UI

## Transports

`@shade/transfer` ships HTTP + WebSocket chunk transports. V3.11 adds an
opt-in P2P chunk transport via `RTCDataChannel`:

- HTTP — `ShadeTransferHttpTransport`. POST per chunk; the receiver-
  side route is `app.route('/v1/transfer', await shade.transferRoute())`.
- WebSocket — `ShadeTransferWsTransport`. One connection per peer,
  binary-framed chunks, JSON acks; same wire format inside the frame as
  the WebRTC transport.
- WebRTC — `WebRtcTransferTransport` from `@shade/transport-webrtc`.
  Wired automatically by `shade.configureWebRTC()` as the primary
  layer of a `MultiTransportFallback([webrtc, http])`. See
  [docs/webrtc.md](./webrtc.md).

`MultiTransportFallback` is the N-ary generalisation of
`FallbackTransferTransport`: pass an ordered list of named transports
and the engine demotes sticky on `TransferTransportError`.