release(v4.0.0): Shade GA — V3.x consolidation + audit prep
Some checks failed
Test / test (push) Has been cancelled
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
Some checks failed
Test / test (push) Has been cancelled
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers byte-for-byte. The version bump is semantic: audit-cycle complete, opt-in surface fully exposed, threat model refreshed for every new surface. Highlights: - All 24 @shade/* packages bumped to 4.0.0 in lockstep. - CHANGELOG 4.0.0 section is the canonical manifest of what landed. - THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12 Web-Worker boundary) + residual-risks table refreshed. - OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox, bridge, observer, /metrics, /healthz, /ready. - MIGRATION 0.3.x → 4.0 documented + smoke-tested against shade migrate-storage on a real SQLite DB. - docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer. - scripts/soak.ts harness for the GA-stable 2-week soak window. - All V*.md plans archived under docs/archive/ with Status: Done. - Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen non-realtime stack. Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green. Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports version 4.0.0 on /health. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
253
docs/streams.md
253
docs/streams.md
@@ -107,11 +107,264 @@ manually after rotation.
|
||||
| S7 | seq overflow practical-impossible (u64 max) |
|
||||
| S8 | At-rest streamSecret encrypted under device-key |
|
||||
|
||||
## Hardening
|
||||
|
||||
`@shade/streams` ships unbounded by default — a peer can declare a
|
||||
1 PiB transfer and the receiver will dutifully allocate lane state for
|
||||
it. Production receivers must enforce limits at the boundary. The
|
||||
`@shade/files` package wires the same patterns up for its filesystem
|
||||
RPC; copy the shapes that fit your app.
|
||||
|
||||
### Per-stream caps
|
||||
|
||||
The receiver sees the declared plaintext size in the `stream-init`
|
||||
control message before it accepts. Reject above your tolerance:
|
||||
|
||||
```ts
|
||||
shade.onIncomingTransfer(async (incoming) => {
|
||||
if (incoming.metadata.totalBytes > 256 * 1024 * 1024) {
|
||||
await incoming.decline({ reason: 'stream too large' });
|
||||
return;
|
||||
}
|
||||
await incoming.accept({ output: ... });
|
||||
});
|
||||
```
|
||||
|
||||
Recommended ceilings (tune to your product, not these):
|
||||
|
||||
| Tier | totalBytes ceiling | Rationale |
|
||||
|------|--------------------|-----------|
|
||||
| Chat attachment | 25 MiB | matches mobile MMS / Slack expectations |
|
||||
| Photo / doc share | 256 MiB | covers raw RAW + most desktop docs |
|
||||
| Backup / dataset | 4 GiB | larger needs explicit operator opt-in |
|
||||
|
||||
### Per-chunk cap
|
||||
|
||||
`createTransferRoutes` accepts `maxChunkBytes` (default ≈ 16 MiB +
|
||||
header). Lower it if your sink can't absorb that — the receiver will
|
||||
413 anything over the limit before the chunk is decrypted, which
|
||||
keeps DoS cost bounded.
|
||||
|
||||
### Per-sender quotas
|
||||
|
||||
`@shade/files` ships a `RateLimiter` (`packages/shade-files/src/server/rate-limiter.ts`)
|
||||
that enforces both ops-per-window and bytes-per-hour caps per sender
|
||||
address. The same shape is the recommended template for guarding raw
|
||||
streams: wrap `incoming.accept` in a check that consumes from a token
|
||||
bucket keyed by `incoming.fromAddress`, and reject with `decline()`
|
||||
when the bucket is empty. See
|
||||
`packages/shade-files/tests/security/quota.test.ts` for the test
|
||||
shape.
|
||||
|
||||
### TTL on idle streams
|
||||
|
||||
A `paused` stream-state record consumes a row in your storage and an
|
||||
encrypted streamSecret slot until it expires. Use the **Retention**
|
||||
defaults below to expire abandoned streams; pair with a metric
|
||||
(`shade_stream_states_active`) and an alert when the count grows
|
||||
unbounded. A peer that opens streams and never finishes them is the
|
||||
dominant abuse pattern for resumable transfer.
|
||||
|
||||
### Trust gates
|
||||
|
||||
For high-stakes transfers (backups, key material, internal docs),
|
||||
gate `accept()` on a verified fingerprint. The pattern mirrors
|
||||
`@shade/files`'s fingerprint gate — see
|
||||
`packages/shade-files/tests/security/fingerprint-gate.test.ts`.
|
||||
|
||||
## Retention
|
||||
|
||||
Resumable streams persist a `PersistedStreamState` per in-flight
|
||||
transfer, encrypted under a device key. Without retention, every
|
||||
crashed or abandoned upload leaves a row behind forever.
|
||||
|
||||
### Defaults
|
||||
|
||||
The shipped `bun-server` SDK template (`shade init --template bun-server`)
|
||||
schedules `pruneStreamStates` on a daily cron with a **14-day**
|
||||
horizon. That is: any stream-state record whose `updatedAt` is older
|
||||
than 14 days is removed at the next sweep. If a sender resumes a
|
||||
14-day-old stream, it will get a "no state" 404 and start over —
|
||||
which is the right answer for a transfer that has been idle for two
|
||||
weeks.
|
||||
|
||||
### Tuning the horizon
|
||||
|
||||
Set `SHADE_STREAM_RETENTION_DAYS` in the template's environment to
|
||||
override the 14-day default. Recommended ranges:
|
||||
|
||||
| Use case | Horizon | Why |
|
||||
|----------|---------|-----|
|
||||
| Synchronous chat | 1–3 days | resume-after-crash, not resume-after-vacation |
|
||||
| File-share product | 7–14 days | covers a typical user vacation |
|
||||
| Cold backup target | 30+ days | deliberate, but plan for storage growth |
|
||||
|
||||
### Hooking the prune call manually
|
||||
|
||||
If you bring your own server (no `bun-server` template), call the
|
||||
storage method on your own schedule:
|
||||
|
||||
```ts
|
||||
import { setInterval } from 'node:timers';
|
||||
|
||||
const ONE_DAY_MS = 24 * 60 * 60 * 1000;
|
||||
const HORIZON_MS = 14 * ONE_DAY_MS;
|
||||
|
||||
setInterval(async () => {
|
||||
if (storage.pruneStreamStates !== undefined) {
|
||||
await storage.pruneStreamStates(Date.now() - HORIZON_MS);
|
||||
}
|
||||
}, ONE_DAY_MS);
|
||||
```
|
||||
|
||||
`pruneStreamStates(olderThan)` removes records whose `updatedAt` is
|
||||
strictly less than `olderThan`. It is idempotent and safe to call
|
||||
concurrently.
|
||||
|
||||
## Rich file metadata + previews (V3.9)
|
||||
|
||||
`stream-init` plaintext can carry an optional `fileMetadata` field that
|
||||
ships filename, MIME-type, and a thumbnail-stream pointer **end-to-end
|
||||
encrypted**. Older receivers ignore the field — backwards-compatible
|
||||
with 0.2.x / 0.3.x peers.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"kind": "shade.stream-init/v1",
|
||||
"streamId": "...",
|
||||
"streamSecret": "...",
|
||||
"metadata": {
|
||||
"chunkSize": 1048576,
|
||||
"sentAt": 1730000000000,
|
||||
"fileMetadata": {
|
||||
"filename": "report.pdf",
|
||||
"mimeType": "application/pdf",
|
||||
"thumbnailStreamId": "Ej1z...",
|
||||
"thumbnailHash": "9a7c...",
|
||||
"thumbnailMime": "image/webp",
|
||||
"thumbnailBytes": 18342
|
||||
}
|
||||
},
|
||||
"lanes": [ /* ... */ ]
|
||||
}
|
||||
```
|
||||
|
||||
### What rides where
|
||||
|
||||
| Field | Plane | Visible to server? |
|
||||
|-------|-------|--------------------|
|
||||
| `filename` | inside Double Ratchet plaintext | no |
|
||||
| `mimeType` | inside Double Ratchet plaintext | no |
|
||||
| `thumbnailStreamId` | streamId of companion stream | yes (random ID, no info leak) |
|
||||
| `thumbnailHash` | sha256 of preview plaintext | base64 hash only, no pixels |
|
||||
| `thumbnailMime` | one of `image/jpeg / image/webp / image/png` | yes (allowlist enforced) |
|
||||
| `thumbnailBytes` | declared length, capped at 64 KiB | yes |
|
||||
| thumbnail bytes themselves | separate AEAD stream, own lane | no |
|
||||
|
||||
The thumbnail rides as its **own stream-transfer**, keyed independently
|
||||
from the main stream. A server compromise leaks neither preview pixels
|
||||
nor original bytes.
|
||||
|
||||
### Sender — attach a preview
|
||||
|
||||
```ts
|
||||
// Pre-computed preview (server-side pipeline path):
|
||||
await shade.upload({
|
||||
to: 'bob',
|
||||
input: pdfBytes,
|
||||
thumbnail: { bytes: previewWebp, mime: 'image/webp' },
|
||||
metadata: { fileMetadata: { filename: 'report.pdf', mimeType: 'application/pdf' } },
|
||||
});
|
||||
|
||||
// Browser auto-generation (image File / Blob → 256×256 preview):
|
||||
await shade.upload({
|
||||
to: 'bob',
|
||||
input: imageFile, // a `File` from <input type="file">
|
||||
generateThumbnail: true, // OffscreenCanvas + createImageBitmap
|
||||
});
|
||||
```
|
||||
|
||||
`generateThumbnail` is a no-op on runtimes lacking
|
||||
`OffscreenCanvas + createImageBitmap` (Bun, Node) — those callers should
|
||||
pre-generate and pass `thumbnail` directly, or skip the preview entirely.
|
||||
|
||||
### Receiver — render in widgets
|
||||
|
||||
The bundled `@shade/widgets` `useShadeDownload` hook auto-accepts
|
||||
thumbnail streams (marked by `userMetadata.shadeThumbnail = '1'`) into
|
||||
an in-memory `ShadeThumbnailCache`. `<TransferRow showThumbnail
|
||||
fileMetadata={...} />` reads from the same cache and renders inside an
|
||||
`<img>` element so the browser's image-decoding sandbox is the trust
|
||||
boundary for format parsing.
|
||||
|
||||
```tsx
|
||||
<ShadeThumbnailProvider>
|
||||
<TransferRow
|
||||
handle={handle}
|
||||
progress={progress}
|
||||
showThumbnail
|
||||
fileMetadata={incoming.metadata.fileMetadata}
|
||||
/>
|
||||
</ShadeThumbnailProvider>
|
||||
```
|
||||
|
||||
### Format-hardening (sender + receiver)
|
||||
|
||||
Both sides enforce the same rules — single source of truth in
|
||||
`@shade/streams/file-metadata.ts`:
|
||||
|
||||
| Rule | Limit |
|
||||
|------|-------|
|
||||
| `thumbnailMime` allowlist | `image/jpeg`, `image/webp`, `image/png` |
|
||||
| `thumbnailBytes` cap | 64 KiB (`THUMBNAIL_MAX_BYTES`) |
|
||||
| `filename` length | ≤ 1024 chars, no control characters |
|
||||
| `mimeType` shape | RFC 7231 `type/subtype` token |
|
||||
| Hash binding | declared `thumbnailHash` = sha256(preview bytes); mismatched bytes are dropped at the cache before any render |
|
||||
|
||||
A hostile peer cannot:
|
||||
- smuggle exotic image formats past the allowlist (envelope parser
|
||||
rejects at decode-time),
|
||||
- substitute different bytes for a declared preview (cache verifies
|
||||
sha256 before exposing bytes to a renderer),
|
||||
- inflate the cache to OOM the receiver (LRU + 1 MiB total cap).
|
||||
|
||||
### Risks consciously accepted
|
||||
|
||||
- **Preview-arrival ≠ send completion.** A receiver may see the
|
||||
thumbnail before the main upload finishes. For high-stakes flows
|
||||
where "did Alice send X?" is itself sensitive, send the preview
|
||||
*only* after main completion (set `thumbnail` to `null` and instead
|
||||
ship a follow-up `stream-init` with the preview). The default
|
||||
ordering optimizes UX, not metadata-secrecy.
|
||||
- **Renderer trust.** We render through a Blob-URL `<img>`. A 0-day
|
||||
in the browser's image decoder would still reach the receiver. Keep
|
||||
browsers patched; rely on the CSP of your embedding app.
|
||||
|
||||
## API surface
|
||||
|
||||
See package READMEs:
|
||||
|
||||
- `packages/shade-streams/README.md` — crypto + state machines
|
||||
- `packages/shade-transfer/README.md` — orchestration, transports, persistence
|
||||
- `packages/shade-transport-webrtc/README.md` — V3.11 P2P transport plug-in
|
||||
- `packages/shade-sdk/README.md` — magic drop-in
|
||||
- `packages/shade-widgets/README.md` — React UI
|
||||
|
||||
## Transports
|
||||
|
||||
`@shade/transfer` ships HTTP + WebSocket chunk transports. V3.11 adds an
|
||||
opt-in P2P chunk transport via `RTCDataChannel`:
|
||||
|
||||
- HTTP — `ShadeTransferHttpTransport`. POST per chunk; the receiver-
|
||||
side route is `app.route('/v1/transfer', await shade.transferRoute())`.
|
||||
- WebSocket — `ShadeTransferWsTransport`. One connection per peer,
|
||||
binary-framed chunks, JSON acks; same wire format inside the frame as
|
||||
the WebRTC transport.
|
||||
- WebRTC — `WebRtcTransferTransport` from `@shade/transport-webrtc`.
|
||||
Wired automatically by `shade.configureWebRTC()` as the primary
|
||||
layer of a `MultiTransportFallback([webrtc, http])`. See
|
||||
[docs/webrtc.md](./webrtc.md).
|
||||
|
||||
`MultiTransportFallback` is the N-ary generalisation of
|
||||
`FallbackTransferTransport`: pass an ordered list of named transports
|
||||
and the engine demotes sticky on `TransferTransportError`.
|
||||
|
||||
Reference in New Issue
Block a user