Shade/CHANGELOG.md

# Changelog

All notable changes to Shade are documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.3.0] — 2026-05-02 — Shade Files

E2EE filesystem RPC primitive — drop-in entrypoints for any consumer that
wants to expose a filesystem (or filesystem-like surface) over Shade. Apps
keep their own UI; this layer ships the typed RPC, the streams bridge for
content I/O over 256 KiB, and production hooks (rate limit, retention,
fingerprint gate, metrics).

### Added

#### `@shade/files` (NEW)
- Standard ops: `list`, `stat`, `mkdir`, `delete`, `move`, `read`, `write`,
  `getThumbnail` — Zod-validated wire schemas + clean user-handler types.
- Custom ops: `client.custom('app.foo', {...})` with full type-safety via
  TypeScript declaration merging on `CustomOpsMap` + per-op Zod schemas
  registered server-side.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams (> 256 KiB)
  ride `@shade/transfer` with automatic correlation via
  `userMetadata.shadeFilesWriteId` / `shadeFilesReadStreamId`.
- Directory ops: `walk(path, opts)` async-iterable depth-first walker;
  `uploadDirectory()` / `downloadDirectory()` with bounded concurrency
  pool (default 4, cap 16), aggregated progress events, abort support.
- Production hooks (all callback-based, vendor-neutral):
  - **Rate limit**: token-bucket per sender, op-cost + byte-quota,
    `FsRateLimitError` / `QuotaExceededError` with `retryAfterMs`.
  - **Idempotency cache**: per-sender LRU + TTL, in-flight de-dupe,
    periodic prune via `BackgroundHooks.onPruneFiles`.
  - **Path policy**: built-in traversal hardening, percent-decode,
    forbidden-bytes check, root-scope, symlink toggle, `extra` predicate.
  - **Fingerprint gate**: `requireFingerprintVerifiedFor(ctx)` →
    `'required' | 'optional' | 'reject'` + `isFingerprintVerified(sender)`.
  - **Signature verification**: pluggable `verifySender(sender, canonical, sig)`
    with replay-window enforcement (±5 min `signedAt` skew rejected).
  - **Metrics**: `onMetric(name, value, tags)` with standard names
    (`shade_files_op_duration_ms`, `_op_total`, `_bytes_in/out`,
    `_idempotency_hit/conflict_total`, `_rate_limit_reject_total`,
    `_fingerprint_reject_total`, `_signature_reject_total`).
- React hooks (subpath import `@shade/files/react`):
  `<ShadeFilesProvider>`, `useShadeFiles`, `useFileList`,
  `useFileTransfer` / `useFileUpload` / `useFileDownload`. SSR-safe; no UI
  components — apps bring their own.
- High-level entry: `Shade.files.serve(handler)` and `Shade.files.client(peer)`
  in `@shade/sdk`. Lazy + memoized; one handler per Shade instance.
- Drop-in adapter: `createMemoryDirectory()` for tests; structurally
  compatible with browser `FileSystemDirectoryHandle`.

#### Wire format bump
- `@shade/proto` wire VERSION bumped from `0x01` to `0x02`. Length prefixes
  changed from u16 to u32 — previous limit was 64 KiB ratchet payloads,
  which blocked inline file ops up to 256 KiB.
  **Wire-incompatible with 0.2.x peers.** New sessions only.
- Cross-platform Kotlin port (`android/shade-android`) updated to match.

#### Concurrency safety
- `ShadeSessionManager.encrypt` / `.decrypt` now run under per-peer mutex.
  Previously, concurrent decryptions of the same peer raced ratchet state
  (manifested as sporadic `Failed to decrypt — wrong key or tampered data`
  under load). Encrypt was already serialized via `Shade.send`'s
  `encryptChains`; decrypt is now serialized at the manager layer too.

#### `@shade/streams` extension
- `StreamMetadata` gets optional `userMetadata?: Record<string, string>` —
  application-level key/value pairs that round-trip verbatim through
  `stream-init` plaintext. Used by `@shade/files` for write/read correlation
  but available to any consumer.

#### `@shade/sdk` extension
- `Shade.files` getter (lazy + memoized).
- `BackgroundHooks.onPruneFiles?: () => void` + periodic timer (default 5 min)
  for `@shade/files` retention.
- `BackgroundTasks.setHook(name, fn)` for runtime hook registration.

### Examples
- `examples/08-files-browser/` — three-process demo (prekey + Bob server +
  Alice CLI) covering list/stat/mkdir/delete/upload/download with both
  inline and streamed paths.

### Tests
- 100+ new tests across `tests/{unit,integration,security}/` in
  `@shade/files`. End-to-end coverage for streams I/O up to 1 MiB, custom-op
  registration + Zod validation, fingerprint-gate rejection, replay-window
  enforcement, idempotent retries, rate-limit + quota enforcement, walk
  + bulk transfer aggregated progress.

## [0.2.0] — 2026-05-01 — Shade Streams

E2EE chunked upload/download with parallel lanes, resumable transfers, and a
"magic drop-in" UX for any Shade-using app. Adds two new packages
(`@shade/streams`, `@shade/transfer`) and extends `@shade/sdk` and
`@shade/widgets` with high-level transfer APIs.

### Added

#### Streams crypto layer (`@shade/streams`)
- HKDF stream/lane key derivation (`deriveStreamKey`, `deriveLaneKey`)
- Deterministic AES-GCM nonce construction `nonce = laneId(4) || seq(8)`
- Streaming SHA-256 via `@noble/hashes/sha2.js` for memory-bounded integrity
- `StreamSender` / `StreamReceiver` per-lane state machines with strict
  in-order seq + replay detection (`StreamReplayError`,
  `StreamOutOfOrderError`, `StreamDecryptionError`, `StreamProtocolError`)
- `MultiLaneSender` / `MultiLaneReceiver` coordinators for parallel transfers
- Range and round-robin partitioning helpers (`planRangePartition`,
  `planRoundRobinPartition`, `chunkRange`)
- Wire format: new envelope type `0x11` (stream-chunk) in `@shade/proto`,
  control envelopes (`stream-init` / `-finish` / `-abort` / `-resume-*`)
  ride existing `0x02` ratchet messages with JSON `kind` discriminator

#### Transfer orchestration (`@shade/transfer`)
- `TransferEngine` — single class wrapping outgoing + incoming lifecycle
- Default `ShadeTransferHttpTransport` for chunk POSTs, opt-in
  `ShadeTransferWsTransport` with `FallbackTransferTransport` for auto-fallback
- `createTransferRoutes()` Hono factory mounts `/v1/transfer/*` routes
  (`chunk`, `state`, `health`)
- `IControlChannel` + `MemoryControlChannel` for in-process testing;
  the SDK provides `ShadeControlChannel` over `Shade.send`/`receive`
- Resume protocol: `MemoryResumeStore`, `StorageBackedResumeStore`,
  `deriveDeviceKey()` for at-rest streamSecret encryption,
  `engine.resumeUpload(streamId, freshInput)` for kill-restart-verify flows
- `ProgressTracker` with EMA-smoothed throughput + ETA
- Retry/backoff (`withRetry`) with exponential delay + jitter
- Error hierarchy: `TransferError`, `TransferAbortError`,
  `TransferIntegrityError`, `TransferProtocolError`, `TransferOfflineError`,
  `TransferResumeError`, `TransferTransportError`

#### SDK (`@shade/sdk`)
- `Shade.upload(opts)` — high-level entry; encrypts + chunks + ships
- `Shade.onIncomingTransfer(handler)` — receiver-side subscription
- `Shade.transferRoute()` — Hono router to mount on the consumer's HTTP server
- `Shade.acceptTransferEnvelope(from, env)` — low-level entry for custom transports
- `Shade.resumeUpload(streamId, freshInput)` — pick up an interrupted transfer
- `Shade.listTransfers(filter?)` — list resumable / active transfers from storage
- `ShadeTransferAuthenticator` — Ed25519-signing authenticator for HTTP/WS transports
- `Shade.onMessage(handler)` now accepts `Promise<void>`-returning handlers
  (awaited in sequence) — supports flow-control over the control plane

#### Storage (all backends)
- New optional `StorageProvider` methods: `saveStreamState`,
  `getStreamState`, `removeStreamState`, `listActiveStreamStates`,
  `pruneStreamStates`. Existing v0.1.x providers compile cleanly (optional methods)
- SQLite (`stream_state` table) and Postgres (`shade_stream_state` table)
  schemas with at-rest encrypted streamSecret
- `MemoryStorage` extended with in-memory stream-state map

#### Widgets (`@shade/widgets`)
- `<ShadeRuntimeProvider runtime={shade}>` — separate React context for
  upload/download widgets (distinct from the observer-dashboard `<ShadeProvider>`)
- `useShadeUpload()` / `useShadeDownload()` headless hooks
- `<ShadeUploader />` / `<ShadeDownloader />` composite components with
  render-prop pattern for full UI replacement
- Sub-components: `<DropZone />`, `<TransferRow />`, `<ProgressBar />`,
  `<SpeedReadout />`, `<ETAReadout />`, `<LaneIndicator />`
- Theme-token additions for progress, drop zone, and lane indicator colors

### Security properties

- Per-chunk AES-256-GCM with deterministic nonce; AAD binds
  `streamId || laneId || seq || isLast` so any header tamper invalidates AEAD
- streamSecret never on the wire in plaintext — shipped via Double Ratchet
  control envelope; lane keys derived locally and never transmitted
- Resume state encrypted at rest with `deviceKey` derived from identity's
  signing private key (rotation invalidates in-flight resume — by design)
- Receiver enforces strict in-order seq per lane (`StreamOutOfOrderError`,
  `StreamReplayError`); finish-time integrity check verifies per-lane sha256
  + overall sha256 over original byte order

### Tests added (118 new across 47 files; 444 total)

- Unit: KDF, nonce, AEAD, streaming SHA, sender/receiver, partition
- Integration: 1/4/16-lane parity, range vs round-robin parity,
  Bun.serve loopback at 100 KiB / 1 MiB / 8 MiB, two real Shade instances
  end-to-end at 64 KiB / 512 KiB / 4 MiB
- Resume: kill-restart-verify on 256 KiB with 4 lanes
- WS fallback: WS connect failure → transparent HTTP completion
- Tamper: bit-flip ciphertext / tag / header field; replay; out-of-order
- Wire: 0x11 envelope encode/decode roundtrip + edge cases

### Backward compatibility

- `Shade.send`/`receive`/`onMessage`/`fingerprint`/`rotate` unchanged
  (`onMessage` widened to support async handlers — sync handlers still work)
- Existing wire types `0x01` (PreKeyMessage) / `0x02` (RatchetMessage) unchanged
- `StorageProvider` interface extension uses optional methods
- `@shade/streams` and `@shade/transfer` are new packages; no migration

## [1.0.0] — 2026-04-10

### First production release

Shade implements the Signal Protocol (X3DH + Double Ratchet) as a standalone, audit-friendly E2EE library for TypeScript/Bun.

### Added

#### Core protocol
- **X3DH** key agreement (X25519 + Ed25519, supports asynchronous bundles)
- **Double Ratchet** with forward secrecy and post-compromise recovery
- Skipped message key cache for out-of-order delivery (max 1000 per chain)
- Header-bound AAD on AES-256-GCM encrypts (tampered headers fail decryption)
- Memory zeroization of message keys, chain keys, root keys, and DH private keys after use

#### Storage
- `MemoryStorage` (in-memory, for tests/embedded)
- `SQLiteStorage` (`@shade/storage-sqlite`) — bun:sqlite, WAL mode, crash-safe
- `PostgresStorage` (`@shade/storage-postgres`) — Drizzle, FOR UPDATE SKIP LOCKED
- All backends survive container restarts and SIGKILL
- Identity history with 7-day grace period for rotation

#### Prekey server (`@shade/server`)
- Hono-based REST API with self-authenticated registration (Ed25519 signatures)
- Anonymous bundle fetches (read-only)
- Per-IP and per-identity rate limiting (token bucket)
- Address validation (NFKC normalization, alphanumeric + `:_-.`)
- ±5 minute replay window on signed requests
- Health endpoints (`/health`, `/healthz`, `/ready`)
- Prometheus metrics (`/metrics`)
- Structured JSON logging
- Graceful shutdown on SIGTERM/SIGINT
- Production Dockerfile with non-root user, healthcheck, multi-stage build
- docker-compose.yml example for Dokploy

#### Session manager (`@shade/core`)
- `ShadeSessionManager` high-level API (`encrypt`, `decrypt`, `initSessionFromBundle`)
- `getIdentityFingerprint()` — Signal-style 60-digit safety numbers
- `ensurePreKeyStock()` — auto-replenish when below threshold
- `resetSession()` and `acceptIdentityChange()` for recovery scenarios
- `rotateIdentity()` with archived previous identities

#### Transport (`@shade/transport`)
- `ShadeFetchTransport` — HTTP client for the prekey server with auto-signing
- `ShadeWebSocket` — WebSocket wrapper with transparent encrypt/decrypt

#### Wire format (`@shade/proto`)
- Compact binary encoding (significantly smaller than JSON)
- Length-prefixed byte arrays, big-endian integers
- Version-tagged envelopes for forward compatibility

#### Cryptographic hardening
- `constantTimeEqual` (XOR-accumulator, no early exit)
- `randomUint32` via crypto.getRandomValues (no Math.random)
- Timing-attack regression test
- Constant-time trust verification in all storage backends

#### Errors
- Stable `SHADE_*` error codes
- `errorToHttpStatus` for consistent HTTP mapping
- `toJSON()` for network serialization
- 14 specific error types (Validation, Network, Storage, RateLimit, etc.)

#### Documentation
- README, SECURITY.md, THREAT-MODEL.md
- 5 runnable examples (basic conversation, prekey server, WebSocket tunnel, identity verification, Dokploy deployment)
- Per-package READMEs
- Inline TSDoc throughout

#### Testing
- 195+ tests across all packages
- Crash recovery integration test
- Cross-platform PostgreSQL tests (skip without `SHADE_TEST_PG_URL`)
- CI workflow with PostgreSQL service
- Benchmark suite

### Security properties
- Forward secrecy
- Post-compromise security
- Authenticated identity verification
- Replay protection
- Constant-time secret comparisons
- Memory zeroization (best-effort)