Shade/docs/streams.md

Shade Streams 0.2.0

E2EE chunked upload/download for Shade. Drop into any Shade-using app:

const handle = await shade.upload({ to: 'bob', input: file });
const result = await handle.done(); // { sha256, bytesSent, durationMs }

…and on the receiver:

shade.onIncomingTransfer(async (incoming) => {
  const handle = await incoming.accept({ output: { kind: 'file', path: '/uploads/x' } });
  await handle.done();
});

Or in React:

<ShadeRuntimeProvider runtime={shade}>
  <ShadeUploader to="bob" onComplete={(r) => console.log(r.sha256)} />
</ShadeRuntimeProvider>

How it works

A transfer has two planes:

• Control plane — stream-init, stream-finish, stream-abort, and stream-resume-* messages, encoded as JSON plaintext and shipped through the existing Double Ratchet (envelope type 0x02). One ratchet step establishes a stream; the rest is per-chunk AEAD.
• Data plane — stream-chunk envelopes (envelope type 0x11), AES-256-GCM-encrypted under a per-lane key, shipped over HTTP POST (or WebSocket if opted-in). Lanes run in parallel for throughput.

Sender                                  Receiver
──────                                  ────────
streamSecret = randomBytes(32)
streamId     = randomBytes(16)
streamKey    = HKDF(streamSecret, streamId, "shade-stream/v1\0master")
laneKey[i]   = HKDF(streamKey, streamId, "...\0lane\0" || u32(i))

[stream-init JSON over Double Ratchet] ─▶
                                          parses streamSecret, derives same keys
                                          spawns L per-lane receivers

[chunk 0x11 over HTTP]                 ─▶
   AES-GCM(laneKey[i], plaintext, nonce=laneId||seq, aad=streamId||laneId||seq||isLast)
                                          decrypts, verifies, writes to sink
   (× 4 lanes in parallel)

[stream-finish JSON over Double Ratchet] ─▶
                                          verifies per-lane sha256 + overall sha256
                                          throws TransferIntegrityError on mismatch

Partition strategies

• Range (default for known-size inputs) — lane i owns bytes [i·N/L, (i+1)·N/L). Receiver reconstructs by concatenating lane outputs in laneId order.
• Round-robin (default for unknown-size streams) — chunk i goes to lane i mod L. Receiver reorders via a per-stream chunk-index buffer.

Resume

Persistence is opt-in via a ResumeStore (memory, SQLite, Postgres, IndexedDB-ready). State persisted on init; sender's resume queries the receiver's lastSeqAcked per lane via GET /v1/transfer/:streamId/state, then continues from there. The streamSecret is encrypted at rest under a device-key derived from the local identity's signing private key — a stolen DB without the identity key cannot resume.

const handle = await shade.resumeUpload(streamId, sameInputAsBefore);
await handle.done();

Resume across identity rotation is not supported (rotation invalidates the device key — by design, to prevent a stolen pre-rotation DB from deriving keys for any post-rotation transfer). Restart the transfer manually after rotation.

Throughput

• Default 4 lanes × 1 MiB chunks × 4 in-flight chunks per lane = 16 MiB peak in-flight per direction.
• Memory-bounded: receivers stream chunks to the configured sink without buffering the full payload. 1 GB transfer = O(chunkSize) RSS, not O(file).
• AES-GCM is hardware-accelerated via SubtleCrypto; SHA-256 streaming via @noble/hashes.

Security properties

ID	Property
S1	streamSecret never on the wire in plaintext (Double Ratchet only)
S2	Unique per-(streamKey, laneId, seq) AEAD nonce — no nonce reuse
S3	Tampered chunk header / ciphertext / tag → AEAD reject
S4	Per-lane sha256 + overall sha256 verified at finish
S5	streamKey/laneKey zeroized on abort/finish (destroy())
S6	Concurrent streams have independent lane keys
S7	seq overflow practical-impossible (u64 max)
S8	At-rest streamSecret encrypted under device-key

Hardening

@shade/streams ships unbounded by default — a peer can declare a 1 PiB transfer and the receiver will dutifully allocate lane state for it. Production receivers must enforce limits at the boundary. The @shade/files package wires the same patterns up for its filesystem RPC; copy the shapes that fit your app.

Per-stream caps

The receiver sees the declared plaintext size in the stream-init control message before it accepts. Reject above your tolerance:

shade.onIncomingTransfer(async (incoming) => {
  if (incoming.metadata.totalBytes > 256 * 1024 * 1024) {
    await incoming.decline({ reason: 'stream too large' });
    return;
  }
  await incoming.accept({ output: ... });
});

Recommended ceilings (tune to your product, not these):

Tier	totalBytes ceiling	Rationale
Chat attachment	25 MiB	matches mobile MMS / Slack expectations
Photo / doc share	256 MiB	covers raw RAW + most desktop docs
Backup / dataset	4 GiB	larger needs explicit operator opt-in

Per-chunk cap

createTransferRoutes accepts maxChunkBytes (default ≈ 16 MiB + header). Lower it if your sink can't absorb that — the receiver will 413 anything over the limit before the chunk is decrypted, which keeps DoS cost bounded.

Per-sender quotas

@shade/files ships a RateLimiter (packages/shade-files/src/server/rate-limiter.ts) that enforces both ops-per-window and bytes-per-hour caps per sender address. The same shape is the recommended template for guarding raw streams: wrap incoming.accept in a check that consumes from a token bucket keyed by incoming.fromAddress, and reject with decline() when the bucket is empty. See packages/shade-files/tests/security/quota.test.ts for the test shape.

TTL on idle streams

A paused stream-state record consumes a row in your storage and an encrypted streamSecret slot until it expires. Use the Retention defaults below to expire abandoned streams; pair with a metric (shade_stream_states_active) and an alert when the count grows unbounded. A peer that opens streams and never finishes them is the dominant abuse pattern for resumable transfer.

Trust gates

For high-stakes transfers (backups, key material, internal docs), gate accept() on a verified fingerprint. The pattern mirrors @shade/files's fingerprint gate — see packages/shade-files/tests/security/fingerprint-gate.test.ts.

Retention

Resumable streams persist a PersistedStreamState per in-flight transfer, encrypted under a device key. Without retention, every crashed or abandoned upload leaves a row behind forever.

Defaults

The shipped bun-server SDK template (shade init --template bun-server) schedules pruneStreamStates on a daily cron with a 14-day horizon. That is: any stream-state record whose updatedAt is older than 14 days is removed at the next sweep. If a sender resumes a 14-day-old stream, it will get a "no state" 404 and start over — which is the right answer for a transfer that has been idle for two weeks.

Tuning the horizon

Set SHADE_STREAM_RETENTION_DAYS in the template's environment to override the 14-day default. Recommended ranges:

Use case	Horizon	Why
Synchronous chat	1–3 days	resume-after-crash, not resume-after-vacation
File-share product	7–14 days	covers a typical user vacation
Cold backup target	30+ days	deliberate, but plan for storage growth

Hooking the prune call manually

If you bring your own server (no bun-server template), call the storage method on your own schedule:

import { setInterval } from 'node:timers';

const ONE_DAY_MS = 24 * 60 * 60 * 1000;
const HORIZON_MS = 14 * ONE_DAY_MS;

setInterval(async () => {
  if (storage.pruneStreamStates !== undefined) {
    await storage.pruneStreamStates(Date.now() - HORIZON_MS);
  }
}, ONE_DAY_MS);

pruneStreamStates(olderThan) removes records whose updatedAt is strictly less than olderThan. It is idempotent and safe to call concurrently.

Rich file metadata + previews (V3.9)

stream-init plaintext can carry an optional fileMetadata field that ships filename, MIME-type, and a thumbnail-stream pointer end-to-end encrypted. Older receivers ignore the field — backwards-compatible with 0.2.x / 0.3.x peers.

{
  "kind": "shade.stream-init/v1",
  "streamId": "...",
  "streamSecret": "...",
  "metadata": {
    "chunkSize": 1048576,
    "sentAt": 1730000000000,
    "fileMetadata": {
      "filename": "report.pdf",
      "mimeType": "application/pdf",
      "thumbnailStreamId": "Ej1z...",
      "thumbnailHash": "9a7c...",
      "thumbnailMime": "image/webp",
      "thumbnailBytes": 18342
    }
  },
  "lanes": [ /* ... */ ]
}

What rides where

Field	Plane	Visible to server?
filename	inside Double Ratchet plaintext	no
mimeType	inside Double Ratchet plaintext	no
thumbnailStreamId	streamId of companion stream	yes (random ID, no info leak)
thumbnailHash	sha256 of preview plaintext	base64 hash only, no pixels
thumbnailMime	one of image/jpeg / image/webp / image/png	yes (allowlist enforced)
thumbnailBytes	declared length, capped at 64 KiB	yes
thumbnail bytes themselves	separate AEAD stream, own lane	no

The thumbnail rides as its own stream-transfer, keyed independently from the main stream. A server compromise leaks neither preview pixels nor original bytes.

Sender — attach a preview

// Pre-computed preview (server-side pipeline path):
await shade.upload({
  to: 'bob',
  input: pdfBytes,
  thumbnail: { bytes: previewWebp, mime: 'image/webp' },
  metadata: { fileMetadata: { filename: 'report.pdf', mimeType: 'application/pdf' } },
});

// Browser auto-generation (image File / Blob → 256×256 preview):
await shade.upload({
  to: 'bob',
  input: imageFile,                 // a `File` from <input type="file">
  generateThumbnail: true,          // OffscreenCanvas + createImageBitmap
});

generateThumbnail is a no-op on runtimes lacking OffscreenCanvas + createImageBitmap (Bun, Node) — those callers should pre-generate and pass thumbnail directly, or skip the preview entirely.

Receiver — render in widgets

The bundled @shade/widgets useShadeDownload hook auto-accepts thumbnail streams (marked by userMetadata.shadeThumbnail = '1') into an in-memory ShadeThumbnailCache. <TransferRow showThumbnail fileMetadata={...} /> reads from the same cache and renders inside an <img> element so the browser's image-decoding sandbox is the trust boundary for format parsing.

<ShadeThumbnailProvider>
  <TransferRow
    handle={handle}
    progress={progress}
    showThumbnail
    fileMetadata={incoming.metadata.fileMetadata}
  />
</ShadeThumbnailProvider>

Format-hardening (sender + receiver)

Both sides enforce the same rules — single source of truth in @shade/streams/file-metadata.ts:

Rule	Limit
thumbnailMime allowlist	image/jpeg, image/webp, image/png
thumbnailBytes cap	64 KiB (THUMBNAIL_MAX_BYTES)
filename length	≤ 1024 chars, no control characters
mimeType shape	RFC 7231 type/subtype token
Hash binding	declared thumbnailHash = sha256(preview bytes); mismatched bytes are dropped at the cache before any render

A hostile peer cannot:

• smuggle exotic image formats past the allowlist (envelope parser rejects at decode-time),
• substitute different bytes for a declared preview (cache verifies sha256 before exposing bytes to a renderer),
• inflate the cache to OOM the receiver (LRU + 1 MiB total cap).

Risks consciously accepted

• Preview-arrival ≠ send completion. A receiver may see the thumbnail before the main upload finishes. For high-stakes flows where "did Alice send X?" is itself sensitive, send the preview only after main completion (set thumbnail to null and instead ship a follow-up stream-init with the preview). The default ordering optimizes UX, not metadata-secrecy.
• Renderer trust. We render through a Blob-URL <img>. A 0-day in the browser's image decoder would still reach the receiver. Keep browsers patched; rely on the CSP of your embedding app.

API surface

See package READMEs:

• packages/shade-streams/README.md — crypto + state machines
• packages/shade-transfer/README.md — orchestration, transports, persistence
• packages/shade-transport-webrtc/README.md — V3.11 P2P transport plug-in
• packages/shade-sdk/README.md — magic drop-in
• packages/shade-widgets/README.md — React UI

Transports

@shade/transfer ships HTTP + WebSocket chunk transports. V3.11 adds an opt-in P2P chunk transport via RTCDataChannel:

• HTTP — ShadeTransferHttpTransport. POST per chunk; the receiver- side route is app.route('/v1/transfer', await shade.transferRoute()).
• WebSocket — ShadeTransferWsTransport. One connection per peer, binary-framed chunks, JSON acks; same wire format inside the frame as the WebRTC transport.
• WebRTC — WebRtcTransferTransport from @shade/transport-webrtc. Wired automatically by shade.configureWebRTC() as the primary layer of a MultiTransportFallback([webrtc, http]). See docs/webrtc.md.

MultiTransportFallback is the N-ary generalisation of FallbackTransferTransport: pass an ordered list of named transports and the engine demotes sticky on TransferTransportError.