release(v4.0.0): Shade GA — V3.x consolidation + audit prep
Some checks failed
Test / test (push) Has been cancelled
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled

V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.

Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
  Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
  bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
  shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
  non-realtime stack.

Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
  version 4.0.0 on /health.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-03 18:35:35 +02:00
parent 8b055912b7
commit e6fdf31b49
298 changed files with 37909 additions and 256 deletions

View File

@@ -107,11 +107,264 @@ manually after rotation.
| S7 | seq overflow practical-impossible (u64 max) |
| S8 | At-rest streamSecret encrypted under device-key |
## Hardening
`@shade/streams` ships unbounded by default — a peer can declare a
1 PiB transfer and the receiver will dutifully allocate lane state for
it. Production receivers must enforce limits at the boundary. The
`@shade/files` package wires the same patterns up for its filesystem
RPC; copy the shapes that fit your app.
### Per-stream caps
The receiver sees the declared plaintext size in the `stream-init`
control message before it accepts. Reject above your tolerance:
```ts
shade.onIncomingTransfer(async (incoming) => {
if (incoming.metadata.totalBytes > 256 * 1024 * 1024) {
await incoming.decline({ reason: 'stream too large' });
return;
}
await incoming.accept({ output: ... });
});
```
Recommended ceilings (tune to your product, not these):
| Tier | totalBytes ceiling | Rationale |
|------|--------------------|-----------|
| Chat attachment | 25 MiB | matches mobile MMS / Slack expectations |
| Photo / doc share | 256 MiB | covers raw RAW + most desktop docs |
| Backup / dataset | 4 GiB | larger needs explicit operator opt-in |
### Per-chunk cap
`createTransferRoutes` accepts `maxChunkBytes` (default ≈ 16 MiB +
header). Lower it if your sink can't absorb that — the receiver will
413 anything over the limit before the chunk is decrypted, which
keeps DoS cost bounded.
### Per-sender quotas
`@shade/files` ships a `RateLimiter` (`packages/shade-files/src/server/rate-limiter.ts`)
that enforces both ops-per-window and bytes-per-hour caps per sender
address. The same shape is the recommended template for guarding raw
streams: wrap `incoming.accept` in a check that consumes from a token
bucket keyed by `incoming.fromAddress`, and reject with `decline()`
when the bucket is empty. See
`packages/shade-files/tests/security/quota.test.ts` for the test
shape.
### TTL on idle streams
A `paused` stream-state record consumes a row in your storage and an
encrypted streamSecret slot until it expires. Use the **Retention**
defaults below to expire abandoned streams; pair with a metric
(`shade_stream_states_active`) and an alert when the count grows
unbounded. A peer that opens streams and never finishes them is the
dominant abuse pattern for resumable transfer.
### Trust gates
For high-stakes transfers (backups, key material, internal docs),
gate `accept()` on a verified fingerprint. The pattern mirrors
`@shade/files`'s fingerprint gate — see
`packages/shade-files/tests/security/fingerprint-gate.test.ts`.
## Retention
Resumable streams persist a `PersistedStreamState` per in-flight
transfer, encrypted under a device key. Without retention, every
crashed or abandoned upload leaves a row behind forever.
### Defaults
The shipped `bun-server` SDK template (`shade init --template bun-server`)
schedules `pruneStreamStates` on a daily cron with a **14-day**
horizon. That is: any stream-state record whose `updatedAt` is older
than 14 days is removed at the next sweep. If a sender resumes a
14-day-old stream, it will get a "no state" 404 and start over —
which is the right answer for a transfer that has been idle for two
weeks.
### Tuning the horizon
Set `SHADE_STREAM_RETENTION_DAYS` in the template's environment to
override the 14-day default. Recommended ranges:
| Use case | Horizon | Why |
|----------|---------|-----|
| Synchronous chat | 13 days | resume-after-crash, not resume-after-vacation |
| File-share product | 714 days | covers a typical user vacation |
| Cold backup target | 30+ days | deliberate, but plan for storage growth |
### Hooking the prune call manually
If you bring your own server (no `bun-server` template), call the
storage method on your own schedule:
```ts
import { setInterval } from 'node:timers';
const ONE_DAY_MS = 24 * 60 * 60 * 1000;
const HORIZON_MS = 14 * ONE_DAY_MS;
setInterval(async () => {
if (storage.pruneStreamStates !== undefined) {
await storage.pruneStreamStates(Date.now() - HORIZON_MS);
}
}, ONE_DAY_MS);
```
`pruneStreamStates(olderThan)` removes records whose `updatedAt` is
strictly less than `olderThan`. It is idempotent and safe to call
concurrently.
## Rich file metadata + previews (V3.9)
`stream-init` plaintext can carry an optional `fileMetadata` field that
ships filename, MIME-type, and a thumbnail-stream pointer **end-to-end
encrypted**. Older receivers ignore the field — backwards-compatible
with 0.2.x / 0.3.x peers.
```jsonc
{
"kind": "shade.stream-init/v1",
"streamId": "...",
"streamSecret": "...",
"metadata": {
"chunkSize": 1048576,
"sentAt": 1730000000000,
"fileMetadata": {
"filename": "report.pdf",
"mimeType": "application/pdf",
"thumbnailStreamId": "Ej1z...",
"thumbnailHash": "9a7c...",
"thumbnailMime": "image/webp",
"thumbnailBytes": 18342
}
},
"lanes": [ /* ... */ ]
}
```
### What rides where
| Field | Plane | Visible to server? |
|-------|-------|--------------------|
| `filename` | inside Double Ratchet plaintext | no |
| `mimeType` | inside Double Ratchet plaintext | no |
| `thumbnailStreamId` | streamId of companion stream | yes (random ID, no info leak) |
| `thumbnailHash` | sha256 of preview plaintext | base64 hash only, no pixels |
| `thumbnailMime` | one of `image/jpeg / image/webp / image/png` | yes (allowlist enforced) |
| `thumbnailBytes` | declared length, capped at 64 KiB | yes |
| thumbnail bytes themselves | separate AEAD stream, own lane | no |
The thumbnail rides as its **own stream-transfer**, keyed independently
from the main stream. A server compromise leaks neither preview pixels
nor original bytes.
### Sender — attach a preview
```ts
// Pre-computed preview (server-side pipeline path):
await shade.upload({
to: 'bob',
input: pdfBytes,
thumbnail: { bytes: previewWebp, mime: 'image/webp' },
metadata: { fileMetadata: { filename: 'report.pdf', mimeType: 'application/pdf' } },
});
// Browser auto-generation (image File / Blob → 256×256 preview):
await shade.upload({
to: 'bob',
input: imageFile, // a `File` from <input type="file">
generateThumbnail: true, // OffscreenCanvas + createImageBitmap
});
```
`generateThumbnail` is a no-op on runtimes lacking
`OffscreenCanvas + createImageBitmap` (Bun, Node) — those callers should
pre-generate and pass `thumbnail` directly, or skip the preview entirely.
### Receiver — render in widgets
The bundled `@shade/widgets` `useShadeDownload` hook auto-accepts
thumbnail streams (marked by `userMetadata.shadeThumbnail = '1'`) into
an in-memory `ShadeThumbnailCache`. `<TransferRow showThumbnail
fileMetadata={...} />` reads from the same cache and renders inside an
`<img>` element so the browser's image-decoding sandbox is the trust
boundary for format parsing.
```tsx
<ShadeThumbnailProvider>
<TransferRow
handle={handle}
progress={progress}
showThumbnail
fileMetadata={incoming.metadata.fileMetadata}
/>
</ShadeThumbnailProvider>
```
### Format-hardening (sender + receiver)
Both sides enforce the same rules — single source of truth in
`@shade/streams/file-metadata.ts`:
| Rule | Limit |
|------|-------|
| `thumbnailMime` allowlist | `image/jpeg`, `image/webp`, `image/png` |
| `thumbnailBytes` cap | 64 KiB (`THUMBNAIL_MAX_BYTES`) |
| `filename` length | ≤ 1024 chars, no control characters |
| `mimeType` shape | RFC 7231 `type/subtype` token |
| Hash binding | declared `thumbnailHash` = sha256(preview bytes); mismatched bytes are dropped at the cache before any render |
A hostile peer cannot:
- smuggle exotic image formats past the allowlist (envelope parser
rejects at decode-time),
- substitute different bytes for a declared preview (cache verifies
sha256 before exposing bytes to a renderer),
- inflate the cache to OOM the receiver (LRU + 1 MiB total cap).
### Risks consciously accepted
- **Preview-arrival ≠ send completion.** A receiver may see the
thumbnail before the main upload finishes. For high-stakes flows
where "did Alice send X?" is itself sensitive, send the preview
*only* after main completion (set `thumbnail` to `null` and instead
ship a follow-up `stream-init` with the preview). The default
ordering optimizes UX, not metadata-secrecy.
- **Renderer trust.** We render through a Blob-URL `<img>`. A 0-day
in the browser's image decoder would still reach the receiver. Keep
browsers patched; rely on the CSP of your embedding app.
## API surface
See package READMEs:
- `packages/shade-streams/README.md` — crypto + state machines
- `packages/shade-transfer/README.md` — orchestration, transports, persistence
- `packages/shade-transport-webrtc/README.md` — V3.11 P2P transport plug-in
- `packages/shade-sdk/README.md` — magic drop-in
- `packages/shade-widgets/README.md` — React UI
## Transports
`@shade/transfer` ships HTTP + WebSocket chunk transports. V3.11 adds an
opt-in P2P chunk transport via `RTCDataChannel`:
- HTTP — `ShadeTransferHttpTransport`. POST per chunk; the receiver-
side route is `app.route('/v1/transfer', await shade.transferRoute())`.
- WebSocket — `ShadeTransferWsTransport`. One connection per peer,
binary-framed chunks, JSON acks; same wire format inside the frame as
the WebRTC transport.
- WebRTC — `WebRtcTransferTransport` from `@shade/transport-webrtc`.
Wired automatically by `shade.configureWebRTC()` as the primary
layer of a `MultiTransportFallback([webrtc, http])`. See
[docs/webrtc.md](./webrtc.md).
`MultiTransportFallback` is the N-ary generalisation of
`FallbackTransferTransport`: pass an ordered list of named transports
and the engine demotes sticky on `TransferTransportError`.