release(v4.11.0): streaming Double-Ratchet sub-sessions (ShadeStream)
Answers Vyvern FR shade-ws-streaming-ratchet.md with a first-class
streaming-session API rather than the documented-contract fallback.
The Double-Ratchet crypto was already safe for high-frequency
one-directional use; the send/receive wrapper was not (per-frame
saveSession keystore write; shared per-peer mutex + single stored
session row coupling reuse to the HTTP path).
- @shade/core: stream.ts — identity-bound 3-DH seeding (X3DH-minus-
prekeys, no prekey-server round trip, mutually authenticated against
the parent session's pinned identities), bootstrapStreamSession
reusing init{Sender,Receiver}Session verbatim, in-memory-only
StreamRatchet (own op-mutex, never persisted, zeroized on close).
beginStream/acceptStream on ShadeSessionManager; Stream{Closed,
Handshake}Error; stream.opened/closed events.
- @shade/proto: STREAM_OPEN/OPEN_ACK/FRAME wire (0x31/0x32/0x33),
additive; inspectEnvelopeType extended.
- @shade/sdk: Shade.openStream/acceptStream → ShadeStream
(handshakeFrame/handleHandshake/seal/open/close), transport-
agnostic, independent of encrypt/decrypt queues + parent session,
identical server (sqlite:) and browser (IndexedDB) — touches no
storage.
- Tests: 5000-frame one-directional burst (bounded skipped keys + FS
zeroize), parent-session independence, replay/rewind rejection,
mutual-auth, proto wire round-trips. Full suite green (1159 pass).
- docs/streaming-sessions.md (R1–R7 contract); SECURITY.md matrix rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 11:29:09 +02:00
import type { ShadeEnvelope , StorageProvider , RatchetMessage } from '@shade/core' ;
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
import {
ShadeSessionManager ,
ShadeEventEmitter ,
NoSessionError ,
release(v4.11.0): streaming Double-Ratchet sub-sessions (ShadeStream)
Answers Vyvern FR shade-ws-streaming-ratchet.md with a first-class
streaming-session API rather than the documented-contract fallback.
The Double-Ratchet crypto was already safe for high-frequency
one-directional use; the send/receive wrapper was not (per-frame
saveSession keystore write; shared per-peer mutex + single stored
session row coupling reuse to the HTTP path).
- @shade/core: stream.ts — identity-bound 3-DH seeding (X3DH-minus-
prekeys, no prekey-server round trip, mutually authenticated against
the parent session's pinned identities), bootstrapStreamSession
reusing init{Sender,Receiver}Session verbatim, in-memory-only
StreamRatchet (own op-mutex, never persisted, zeroized on close).
beginStream/acceptStream on ShadeSessionManager; Stream{Closed,
Handshake}Error; stream.opened/closed events.
- @shade/proto: STREAM_OPEN/OPEN_ACK/FRAME wire (0x31/0x32/0x33),
additive; inspectEnvelopeType extended.
- @shade/sdk: Shade.openStream/acceptStream → ShadeStream
(handshakeFrame/handleHandshake/seal/open/close), transport-
agnostic, independent of encrypt/decrypt queues + parent session,
identical server (sqlite:) and browser (IndexedDB) — touches no
storage.
- Tests: 5000-frame one-directional burst (bounded skipped keys + FS
zeroize), parent-session independence, replay/rewind rejection,
mutual-auth, proto wire round-trips. Full suite green (1159 pass).
- docs/streaming-sessions.md (R1–R7 contract); SECURITY.md matrix rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 11:29:09 +02:00
StreamRatchet ,
StreamHandshakeError ,
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
} from '@shade/core' ;
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
import {
FingerprintGateRegistry ,
type FingerprintGateHandler ,
} from './gates.js' ;
import {
SubtleCryptoProvider ,
MemoryStorage ,
createWorkerCryptoProvider ,
type WorkerCryptoProvider ,
createEncryptStream ,
createDecryptStream ,
type CreateEncryptStreamOptions ,
type CreateDecryptStreamOptions ,
} from '@shade/crypto-web' ;
release(v4.11.0): streaming Double-Ratchet sub-sessions (ShadeStream)
Answers Vyvern FR shade-ws-streaming-ratchet.md with a first-class
streaming-session API rather than the documented-contract fallback.
The Double-Ratchet crypto was already safe for high-frequency
one-directional use; the send/receive wrapper was not (per-frame
saveSession keystore write; shared per-peer mutex + single stored
session row coupling reuse to the HTTP path).
- @shade/core: stream.ts — identity-bound 3-DH seeding (X3DH-minus-
prekeys, no prekey-server round trip, mutually authenticated against
the parent session's pinned identities), bootstrapStreamSession
reusing init{Sender,Receiver}Session verbatim, in-memory-only
StreamRatchet (own op-mutex, never persisted, zeroized on close).
beginStream/acceptStream on ShadeSessionManager; Stream{Closed,
Handshake}Error; stream.opened/closed events.
- @shade/proto: STREAM_OPEN/OPEN_ACK/FRAME wire (0x31/0x32/0x33),
additive; inspectEnvelopeType extended.
- @shade/sdk: Shade.openStream/acceptStream → ShadeStream
(handshakeFrame/handleHandshake/seal/open/close), transport-
agnostic, independent of encrypt/decrypt queues + parent session,
identical server (sqlite:) and browser (IndexedDB) — touches no
storage.
- Tests: 5000-frame one-directional burst (bounded skipped keys + FS
zeroize), parent-session independence, replay/rewind rejection,
mutual-auth, proto wire round-trips. Full suite green (1159 pass).
- docs/streaming-sessions.md (R1–R7 contract); SECURITY.md matrix rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 11:29:09 +02:00
import {
encodeEnvelope ,
decodeEnvelope ,
inspectEnvelopeType ,
encodeStreamOpen ,
encodeStreamOpenAck ,
decodeStreamHandshake ,
encodeStreamFrame ,
decodeStreamFrame ,
} from '@shade/proto' ;
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
import { ShadeFetchTransport , type KTVerifierOptions } from '@shade/transport' ;
import { LightWitness } from '@shade/key-transparency' ;
2026-05-03 19:36:47 +02:00
import type { SignedTreeHead , STHWire } from '@shade/key-transparency' ;
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
import {
TransferEngine ,
ShadeTransferHttpTransport ,
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
MultiTransportFallback ,
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
type ITransferTransport ,
type IncomingTransfer ,
type TransferHandle ,
type TransferOptions ,
type TransferSummary ,
2026-05-03 23:27:06 +02:00
type OutboundQueue as OutboundQueueLike ,
type QueuedEventInput ,
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
} from '@shade/transfer' ;
import type { Hono } from 'hono' ;
import { BackgroundTasks } from './background.js' ;
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
import {
exportBackup ,
applyBackupPayload ,
decryptBackup ,
backupToString ,
backupFromString ,
} from './backup.js' ;
import { computeFingerprint , deserializeIdentityKeyPair } from '@shade/core' ;
2026-05-07 15:55:34 +02:00
import {
acceptBroadcastEnvelope ,
createBroadcastChannelImpl ,
getBroadcastChannelImpl ,
listBroadcastChannelsImpl ,
maybeHandleControlPlaintext ,
type BroadcastChannel ,
type BroadcastChannelSummary ,
type BroadcastSdkHooks ,
type MessageMeta ,
} from './broadcast.js' ;
release(v4.3.0): browser persistence via @shade/storage-indexeddb
Ship an official IndexedDB-backed StorageProvider so browser-based Shade
consumers persist identity, prekeys, sessions, retired identities,
peer-verification state and stream-resume rows across tab refresh and
browser restart. Closes the gap that forced browser apps onto
storage:"memory" (regenerated identity each load, orphaned device
records server-side).
- New package @shade/storage-indexeddb (4.3.0): full StorageProvider
conformance, schema v1, idb-backed; bumpPeerIdentityVersion is wrapped
in a single readwrite IDB transaction (atomic, vs SQLite's
read-then-upsert race).
- @shade/sdk resolveStorage() accepts { type: 'indexeddb', dbName? } via
dynamic import (lazy, optional dep — same pattern as
@shade/storage-postgres). Named StorageSpec type now reused by
ResolvedConfig.
- Tests: 16 new tests in shade-storage-indexeddb (StorageProvider
surface + peer-verifications + full E2EE conversation surviving a
simulated tab reload). Run on fake-indexeddb.
- Lockstep version bump 4.2.1 → 4.3.0 across all 25 packages.
- Publish scripts updated to include the new package.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 17:35:02 +02:00
import type { ResolvedConfig , StorageSpec } from './config.js' ;
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
import {
ShadeControlChannel ,
ShadeTransferAuthenticator ,
type ControlEnvelopeTransport ,
} from './streams-bridge.js' ;
import { createFilesNamespace , type FilesNamespace } from '@shade/files' ;
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
import type { ObservabilityHook } from '@shade/observability' ;
import {
isAllowedThumbnailMime ,
sha256Once ,
THUMBNAIL_MAX_BYTES ,
type StreamFileMetadata ,
type ThumbnailMime ,
} from '@shade/streams' ;
import {
generateThumbnail as generateThumbnailFromBlob ,
type ThumbnailGenerationOptions ,
} from './thumbnail.js' ;
/ * *
* V3 . 9 — extended upload options . The base ` TransferOptions ` is forwarded
* verbatim to ` @shade/transfer ` ; the extra fields control the thumbnail
* companion - stream and never leak past the SDK boundary .
* /
/ * *
* V3 . 11 — opt - in WebRTC P2P transport . Pass to ` shade.configureWebRTC() `
* before the engine is built . The shape mirrors ` WebRtcConnectionManager ` ' s
* options without forcing the SDK to import ` @shade/transport-webrtc ` at
* the value layer ( the import happens lazily inside ` engine() ` ) .
* /
export interface ShadeWebRtcConfig {
/ * *
* WebRTC adapter . Use ` nativeRtcFactory() ` from ` @shade/transport-webrtc `
* in browsers / Deno / Cloudflare Workers ; supply your own
* ` IRtcFactory ` for Node - class environments ( ` node-datachannel ` , ` wrtc ` ) .
* /
factory : import ( '@shade/transport-webrtc' ) . IRtcFactory ;
iceServers? : import ( '@shade/transport-webrtc' ) . ShadeRtcConfig [ 'iceServers' ] ;
iceTransportPolicy? : import ( '@shade/transport-webrtc' ) . ShadeRtcConfig [ 'iceTransportPolicy' ] ;
bundlePolicy? : import ( '@shade/transport-webrtc' ) . ShadeRtcConfig [ 'bundlePolicy' ] ;
/** Default 30s. */
connectTimeoutMs? : number ;
/** Default 30s. */
requestTimeoutMs? : number ;
/** Default 4 MiB. */
backpressureThresholdBytes? : number ;
}
/** Live WebRTC runtime returned by `shade.getWebRtcRuntime()`. */
export interface ShadeWebRtcRuntime {
signaling : import ( '@shade/transport-webrtc' ) . WebRtcSignalingChannel ;
manager : import ( '@shade/transport-webrtc' ) . WebRtcConnectionManager ;
transport : import ( '@shade/transport-webrtc' ) . WebRtcTransferTransport ;
fallback : MultiTransportFallback ;
/** Internal — wires `engine` into the receiver hooks once it's built. */
attachEngine ( engine : TransferEngine ) : void ;
/** Tear down the manager, the signaling channel, and any open peer connections. */
destroy ( ) : void ;
}
export interface ShadeUploadOptions extends TransferOptions {
/ * *
* Pre - generated thumbnail bytes + MIME . Use this on server runtimes
* where in - process image processing is already part of your pipeline ,
* or when you have a richer thumbnail than the auto - generator would
* produce ( e . g . center - cropped , branded watermark , etc . ) .
* /
thumbnail ? : { bytes : Uint8Array ; mime : ThumbnailMime } ;
/ * *
* Browser auto - generation . Set to ` true ` for defaults , or pass a
* config object . Returns ` null ` ( silently skips the thumbnail ) on
* runtimes lacking ` OffscreenCanvas ` + ` createImageBitmap ` .
* /
generateThumbnail? : boolean | ThumbnailGenerationOptions ;
}
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
/ * *
* The high - level Shade API .
*
* Wraps crypto , storage , session management , transport , and optional
* observer into a single object . Provides magic auto - establish + auto -
* publish + auto - replenish behavior .
* /
export class Shade {
private readonly crypto = new SubtleCryptoProvider ( ) ;
private readonly events = new ShadeEventEmitter ( ) ;
private storage ! : StorageProvider ;
private manager ! : ShadeSessionManager ;
private transport ! : ShadeFetchTransport ;
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
private _background : BackgroundTasks | null = null ;
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
private address ! : string ;
private initialized = false ;
// Per-address mutex to serialize session establishment under concurrent sends
private establishing = new Map < string , Promise < void > > ( ) ;
// Per-address encrypt queue to serialize ratchet mutations
private encryptChains = new Map < string , Promise < unknown > > ( ) ;
2026-05-08 12:13:46 +02:00
// Per-`from` decrypt queue: serializes incoming receives so two concurrent
// shade.receive(from, env) calls can't race the ratchet/storage. Without
// this, parallel deliveries (relay duplicate fan-out, fast pipelined
// sends) hit `database is locked` (sqlite) or transaction conflicts (IDB)
// because the underlying StorageProvider isn't required to be a
// concurrent-safe writer. See V4.8.2 changelog.
private decryptChains = new Map < string , Promise < unknown > > ( ) ;
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
2026-05-07 15:55:34 +02:00
// Message handlers — may be sync or async; receive() awaits each. The
// optional third arg distinguishes direct vs broadcast plaintexts;
// handlers registered without it work unchanged (V4.6 back-compat).
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
private messageHandlers : Array <
2026-05-07 15:55:34 +02:00
( from : string , plaintext : string , meta? : MessageMeta ) = > void | Promise < void >
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
> = [ ] ;
// Stream-transfer engine, lazily constructed on first use.
private transferEngine : TransferEngine | null = null ;
private controlChannel : ShadeControlChannel | null = null ;
private peerBaseUrlResolver : ( ( peerAddress : string ) = > Promise < string > ) | null = null ;
private envelopeOutboxes : ControlEnvelopeTransport | null = null ;
2026-05-03 23:27:06 +02:00
private transferTransportOverride : ITransferTransport | null = null ;
private transferQueue : OutboundQueueLike | null = null ;
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
// `@shade/files` namespace, lazy + memoized.
private filesNamespace : FilesNamespace | null = null ;
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
// V3.12 — light-witness for split-view detection.
private ktWitness : LightWitness | null = null ;
// V3.11 WebRTC P2P transport. Lazy-built on first engine() if configured.
private webrtcConfig : ShadeWebRtcConfig | null = null ;
private webrtcRuntime : ShadeWebRtcRuntime | null = null ;
// V3.8 Worker-Crypto. Lazy: configured via `configureWorkerCrypto()`,
// spawned the first time `encryptStream`/`decryptStream` is used.
private workerCryptoConfig : {
workerUrl : URL | string ;
idleTimeoutMs? : number ;
} | null = null ;
private workerCrypto : WorkerCryptoProvider | null = null ;
private workerCryptoBoot : Promise < WorkerCryptoProvider > | null = null ;
// V3.3 fingerprint gates. Created in `initialize()` once storage is up.
private gates ! : FingerprintGateRegistry ;
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
constructor ( private readonly config : ResolvedConfig ) { }
/ * *
* Initialize the SDK :
* 1 . Resolve storage backend
* 2 . Create session manager + generate identity if needed
* 3 . Create transport
* 4 . Generate initial one - time prekeys
* 5 . Register with prekey server
* 6 . Start background tasks
* /
async initialize ( ) : Promise < void > {
if ( this . initialized ) return ;
// Step 1: Storage
this . storage = await resolveStorage ( this . config . storage ) ;
// Step 2: Session manager with event bus attached
this . manager = new ShadeSessionManager ( this . crypto , this . storage , {
events : this.events ,
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
. . . ( this . config . observability !== undefined
? { observability : this.config.observability }
: { } ) ,
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
} ) ;
await this . manager . initialize ( ) ;
// Step 3: Address (user-provided or persisted UUID)
this . address = this . config . address ? ? ( await resolveAddress ( this . storage ) ) ;
// Step 4: Transport with our signing key
const identity = await this . storage . getIdentityKeyPair ( ) ;
if ( ! identity ) throw new Error ( 'Identity not available after initialize' ) ;
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
// V3.12 — wire up KT verifier + light-witness if configured.
let ktOpts : KTVerifierOptions | undefined ;
if ( this . config . keyTransparency ) {
const baseUrl = this . config . prekeyServer ;
this . ktWitness = new LightWitness ( {
crypto : this.crypto ,
logPublicKey : this.config.keyTransparency.logPublicKey ,
maxStaleMs : this.config.keyTransparency.maxStaleMs ,
maxStored : this.config.keyTransparency.witnessMaxStored ,
fetcher : {
2026-05-03 19:36:47 +02:00
async fetchLatestSTH ( ) : Promise < STHWire > {
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
const res = await fetch ( ` ${ baseUrl } /v1/kt/sth ` ) ;
if ( ! res . ok ) throw new Error ( ` KT /sth: ${ res . status } ` ) ;
2026-05-03 19:36:47 +02:00
return ( await res . json ( ) ) as STHWire ;
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
} ,
2026-05-03 19:36:47 +02:00
async fetchConsistencyProof ( from , to ) : Promise < { proof : string [ ] } > {
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
const res = await fetch ( ` ${ baseUrl } /v1/kt/consistency?from= ${ from } &to= ${ to } ` ) ;
if ( ! res . ok ) throw new Error ( ` KT /consistency: ${ res . status } ` ) ;
2026-05-03 19:36:47 +02:00
return ( await res . json ( ) ) as { proof : string [ ] } ;
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
} ,
} ,
} ) ;
ktOpts = {
mode : this.config.keyTransparency.mode ,
logPublicKey : this.config.keyTransparency.logPublicKey ,
maxStaleMs : this.config.keyTransparency.maxStaleMs ,
onObserveSth : async ( sth : SignedTreeHead ) = > {
if ( this . ktWitness ) {
// The fetched STH was already verified by the transport; feed
// it to the witness for split-view tracking. `observe` may also
// throw on split-view — we let it propagate to the caller.
await this . ktWitness . observe ( sth ) ;
}
} ,
} ;
}
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
this . transport = new ShadeFetchTransport ( {
baseUrl : this.config.prekeyServer ,
crypto : this.crypto ,
signingPrivateKey : identity.signingPrivateKey ,
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
. . . ( ktOpts ? { keyTransparency : ktOpts } : { } ) ,
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
} ) ;
// Step 5: Initial prekeys + register
const otpks = await this . manager . generateOneTimePreKeys ( 20 ) ;
const bundle = await this . manager . createPreKeyBundle ( ) ;
try {
await this . transport . register (
this . address ,
this . manager . getPublicIdentity ( ) ,
bundle . signedPreKey ,
otpks ,
) ;
} catch ( err ) {
console . warn (
` [Shade] Failed to register with prekey server at ${ this . config . prekeyServer } : ${ ( err as Error ) . message } . Will retry on next replenish. ` ,
) ;
}
// Step 6: Background tasks
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
this . _background = new BackgroundTasks (
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
this . manager ,
this . transport ,
this . address ,
this . config ,
) ;
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
this . _background . start ( ) ;
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
// Step 7: V3.3 fingerprint gates
this . gates = new FingerprintGateRegistry ( this . storage ) ;
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
this . initialized = true ;
}
/** Your identity's safety number (12 groups × 5 digits) */
get fingerprint ( ) : Promise < string > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
return this . manager . getIdentityFingerprint ( ) ;
}
/** Your address on the prekey server */
get myAddress ( ) : string {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
return this . address ;
}
2026-05-05 17:58:45 +02:00
/ * *
* The local device ' s Ed25519 identity public key ( 32 bytes ) .
*
* Stable for the lifetime of the identity . After { @link rotate } this
* reflects the new key ; the previous key is preserved in retired -
* identities storage for the configured grace period .
*
* Hand this to your application ' s backend at enrollment time so it
* can verify signatures from this device , compute its own safety -
* number representation , or pin the key for later attestation . Use
* { @link fingerprint } instead for human side - channel comparison .
* /
get identityPublicKey ( ) : Promise < Uint8Array > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
return this . storage . getIdentityKeyPair ( ) . then ( ( kp ) = > {
if ( ! kp ) throw new Error ( 'Identity not yet generated' ) ;
return kp . signingPublicKey ;
} ) ;
}
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
/ * *
* ` @shade/files ` namespace — high - level entry point for E2EE filesystem
* RPC . Lazily creates the underlying channel + streams bridges on first
* access ; subsequent accesses return the same instance .
*
* ` ` ` ts
* const files = shade . files ;
* const stop = await files . serve ( { list : ... , write : ... , . . . } ) ;
* const fs = await files . client ( 'bob' ) ;
* await fs . list ( '/' ) ;
* ` ` `
*
* Requires ` configureTransfers({ resolveBaseUrl }) ` to be called first
* ( same as ` upload ` / ` onIncomingTransfer ` ) .
* /
get files ( ) : FilesNamespace {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
if ( this . filesNamespace !== null ) return this . filesNamespace ;
// `@shade/files` only imports `Shade` as a type, so the cyclic ESM
// import is type-only at the value layer — safe to bind synchronously.
this . filesNamespace = createFilesNamespace ( this ) ;
return this . filesNamespace ;
}
/** Internal — exposes the BackgroundTasks for `@shade/files` to wire prune. */
get background ( ) : BackgroundTasks | null {
return this . _background ;
}
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
/** Access the underlying event emitter (for observer integration) */
getEvents ( ) : ShadeEventEmitter {
return this . events ;
}
/** Access the underlying session manager (for advanced usage) */
getManager ( ) : ShadeSessionManager {
return this . manager ;
}
/** Access the underlying transport (for advanced usage) */
getTransport ( ) : ShadeFetchTransport {
return this . transport ;
}
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
/ * *
* V3 . 12 — access the configured Key - Transparency light - witness , or
* ` null ` when KT was not configured . Useful for surfacing observed
* STHs to UI / dashboards , or for manual gossip checks against
* trusted peers .
* /
getKTWitness ( ) : LightWitness | null {
return this . ktWitness ;
}
/ * *
* Returns the OTel observability hook the SDK was configured with , or
* ` undefined ` if observability is off . Used by ` @shade/files ` and other
* sub - modules to inherit the same tracer the rest of the SDK uses .
* /
getObservability ( ) : ObservabilityHook | undefined {
return this . config . observability ;
}
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
/ * *
* Encrypt a message to a peer . Auto - establishes a session if none exists .
* Returns the ShadeEnvelope ready to send over any transport .
* /
async send ( address : string , plaintext : string ) : Promise < ShadeEnvelope > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
// Serialize all sends to the same peer: the SessionManager mutates
// ratchet state in place, and interleaved mutations corrupt it.
const previous = this . encryptChains . get ( address ) ? ? Promise . resolve ( ) ;
const next = previous
. catch ( ( ) = > { } ) // don't propagate upstream failures to later sends
. then ( async ( ) = > {
try {
return await this . manager . encrypt ( address , plaintext ) ;
} catch ( err ) {
if ( ! ( err instanceof NoSessionError ) ) throw err ;
await this . ensureSession ( address ) ;
return this . manager . encrypt ( address , plaintext ) ;
}
} ) ;
this . encryptChains . set ( address , next ) ;
return next as Promise < ShadeEnvelope > ;
}
/ * *
* Decrypt an incoming envelope and notify registered message handlers .
* Returns the plaintext .
*
* The caller provides the ` from ` address because the envelope itself
* doesn 't authenticate the sender — that' s determined by your transport
* layer ( auth header , WebSocket peer , push notification metadata , etc . ) .
2026-05-07 15:55:34 +02:00
*
* V4.6 : when the decrypted plaintext is a broadcast control message
* ( sender - key distribution / revocation ) , the SDK consumes it
* internally and returns an empty string ; user handlers do NOT fire .
* Apps therefore see only direct plaintexts here . Broadcast payloads
* arrive via { @link Shade . acceptBroadcast } .
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
* /
async receive ( from : string , envelope : ShadeEnvelope ) : Promise < string > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
2026-05-08 12:13:46 +02:00
// Serialize ONLY the ratchet/storage write portion of receive (the
// call into `manager.decrypt`). Concurrent decrypts race the
// SessionManager ratchet (mutated in place) and the StorageProvider
// (not required to be a concurrent-safe writer — `bun:sqlite`
// throws `database is locked`, IDB throws transaction conflicts).
// The Prism FR called this out: a relay-duplicated WS fan-out
// dispatched 8 parallel `shade.receive(from, env)` calls, one won
// the X3DH prekey race and the other 7 failed with
// `database is locked` / `one-time prekey not found`. The fix is
// to queue per-`from` decrypts so the ratchet step is sequential.
//
// Crucially the user-visible MESSAGE HANDLERS run *outside* the
// queue. Streams + file-RPC issue nested `shade.receive` calls for
// the same peer from inside their handlers (e.g. `stream-end`
// arrives while a write-RPC is still waiting on chunks); holding
// the queue across the handler would self-deadlock. The atomic
// unit we have to protect is just the ratchet+storage step, not
// the consumer's reaction to it.
const previous = this . decryptChains . get ( from ) ? ? Promise . resolve ( ) ;
const decryptPromise = previous
. catch ( ( ) = > undefined ) // don't propagate upstream failures
. then ( ( ) = > this . manager . decrypt ( from , envelope ) ) ;
// Store a never-rejecting copy so the next chained receive doesn't
// see a rejection from this one (we still surface our own rejection
// to *this* caller via the original `decryptPromise`).
this . decryptChains . set ( from , decryptPromise . catch ( ( ) = > undefined ) ) ;
const plaintext = await decryptPromise ;
2026-05-07 15:55:34 +02:00
const consumed = await maybeHandleControlPlaintext (
this . broadcastHooks ( ) ,
from ,
plaintext ,
) ;
if ( consumed ) return '' ;
const meta : MessageMeta = { kind : 'direct' } ;
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
for ( const handler of this . messageHandlers ) {
try {
2026-05-07 15:55:34 +02:00
await handler ( from , plaintext , meta ) ;
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
} catch ( err ) {
console . error ( '[Shade] Message handler threw:' , err ) ;
}
}
return plaintext ;
}
2026-05-07 15:55:34 +02:00
/ * *
* Register a handler for incoming messages . Async handlers are awaited .
*
* V4.6 : handlers may declare an optional ` meta ` parameter to discriminate
* direct ( ` meta.kind === 'direct' ` ) from broadcast ( ` meta.kind === 'broadcast' ` )
* deliveries . Handlers that ignore the third arg keep working unchanged
* for direct messages .
* /
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
onMessage (
2026-05-07 15:55:34 +02:00
handler : ( from : string , plaintext : string , meta? : MessageMeta ) = > void | Promise < void > ,
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
) : ( ) = > void {
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
this . messageHandlers . push ( handler ) ;
return ( ) = > {
this . messageHandlers = this . messageHandlers . filter ( ( h ) = > h !== handler ) ;
} ;
}
2026-05-07 15:55:34 +02:00
// ─── V4.6 Broadcast channels ───────────────────────────────
/ * *
* Create a new broadcast channel owned by this device . Returns a
* handle for adding / removing members , encrypting a single payload ,
* and rotating on revocation . The channel id is opaque , stable
* across ` shutdown() ` / re - open , and persisted via the configured
* ` StorageProvider ` .
* /
async createBroadcastChannel ( opts : { label? : string } = { } ) : Promise < BroadcastChannel > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
return createBroadcastChannelImpl ( this . broadcastHooks ( ) , opts ) ;
}
/ * *
* Look up an existing sender - side broadcast channel by id . Returns
* ` null ` when the id is unknown OR when this device only holds a
* receiver - side copy ( the receiver path uses ` onMessage ` for delivery
* — there is no app - facing handle on the receive side ) .
* /
async getBroadcastChannel ( channelId : string ) : Promise < BroadcastChannel | null > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
return getBroadcastChannelImpl ( this . broadcastHooks ( ) , channelId ) ;
}
/ * *
* Snapshot of every broadcast channel persisted on this device ,
* including receiver - side channels that we joined . Useful for
* rebuilding UI state on startup .
* /
async listBroadcastChannels ( ) : Promise < readonly BroadcastChannelSummary [ ] > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
return listBroadcastChannelsImpl ( this . broadcastHooks ( ) ) ;
}
/ * *
* Hand a wire - encoded broadcast envelope ( type 0x21 ) to the SDK .
* Decrypts via the matching channel , advances the chain , and dispatches
* the plaintext to ` onMessage ` handlers with ` meta.kind === 'broadcast' ` .
*
* Stale generations ( sender ' s old chain after a rotation we already
* received ) are silently dropped . Future generations ( we haven ' t seen
* the rotation distribution yet ) throw — the app should ensure the
* distribution envelope is delivered before the broadcast .
* /
async acceptBroadcast ( envelope : Uint8Array ) : Promise < void > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
const result = await acceptBroadcastEnvelope ( this . broadcastHooks ( ) , envelope ) ;
if ( result === null ) return ;
for ( const handler of this . messageHandlers ) {
try {
await handler ( result . meta . kind === 'broadcast' ? result . meta . sender : '' , result . plaintext , result . meta ) ;
} catch ( err ) {
console . error ( '[Shade] Broadcast handler threw:' , err ) ;
}
}
}
private broadcastHooks ( ) : BroadcastSdkHooks {
return {
bilateralSend : ( peer , pt ) = > this . send ( peer , pt ) ,
myAddress : ( ) = > this . address ,
crypto : this.crypto ,
storage : this.storage ,
} ;
}
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
/** Get a peer's fingerprint (requires an existing session) */
async getFingerprintFor ( address : string ) : Promise < string > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
return this . manager . getRemoteFingerprint ( address ) ;
}
/** Verify a fingerprint matches the pinned identity for an address */
async verify ( address : string , fingerprint : string ) : Promise < boolean > {
const remote = await this . getFingerprintFor ( address ) ;
return normalize ( remote ) === normalize ( fingerprint ) ;
}
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
// ─── V3.3 fingerprint gates ───────────────────────────────
/ * *
* Register a handler that runs before ` upload() ` proceeds when the file
* is at or above ` threshold ` bytes and the peer is not yet verified .
* Return ` true ` to allow + persist the verification , ` false ` to abort .
*
* Default threshold ( when this method is never called ) : 10 MiB .
* /
beforeFirstLargeFile ( threshold : number , handler : FingerprintGateHandler ) : void {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
this . gates . registerFirstLargeFile ( threshold , handler ) ;
}
/ * *
* Register a handler that runs before ` importBackup() ` writes to storage .
* The handler receives the fingerprint of the identity * embedded in the
* backup blob * , so the user can OOB - confirm the backup is theirs .
* /
beforeBackupImport ( handler : FingerprintGateHandler ) : void {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
this . gates . registerBackupImport ( handler ) ;
}
/ * *
* Register a handler that runs the first time a peer ' s rotated identity
* is observed ( via ` acceptIdentityChange ` or X3DH against a new bundle ) .
* /
beforeNewDeviceTrust ( handler : FingerprintGateHandler ) : void {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
this . gates . registerNewDeviceTrust ( handler ) ;
}
/ * *
* Register a handler that runs per - recipient before an inbox fan - out
* delivery ( V3 . 6 ) . Reserved hook — wired here so apps can register it
* today and have it active automatically when V3 . 6 ships .
* /
beforeInboxFanout ( handler : FingerprintGateHandler ) : void {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
this . gates . registerInboxFanout ( handler ) ;
}
/ * *
* Mark a peer as verified at their current fingerprint . Call this from
* your own UI ( e . g . after the user scans a QR code or reads the safety
* number aloud ) to satisfy any gate without going through the handler .
* /
async markPeerVerified ( address : string ) : Promise < void > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
const fingerprint = await this . manager . getRemoteFingerprint ( address ) ;
await this . gates . markVerified ( address , fingerprint , 'user' ) ;
}
/ * *
* Returns whether ` address ` has a current verification ( fingerprint and
* identity - version both still match ) .
* /
async isPeerVerified ( address : string ) : Promise < boolean > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
const fingerprint = await this . manager . getRemoteFingerprint ( address ) ;
return this . gates . isVerified ( address , fingerprint ) ;
}
/** Drop any persisted verification for `address`. */
async unmarkPeerVerified ( address : string ) : Promise < void > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
await this . gates . revoke ( address ) ;
}
release(v4.8.3): cross-channel msgId dedup + Shade.aliasSession
Two follow-ups to the V4.8.2 duplicate-fan-out fixes Prism filed.
1. `Inbox.acceptBridgeFrame(blob)` + shared 4096-entry msgId LRU.
The relay durably stores blobs and pushes them to every active
delivery channel; without a cross-channel ack the bridge frame
ran first and the next inbox-poll re-dispatched the same blob
~30 s later, tripping on consumed prekeys. Bridge consumers now
plumb pushed frames through `acceptBridgeFrame`, which shares
the dedup gate + ack path with `pollOnce`. Whichever channel
delivers first wins; the other acks-and-skips. Inbox records
the msgId before the ack so a parallel poll can't observe an
in-flight ack window.
2. `Shade.aliasSession(oldLabel, newLabel)`. First-contact forces
the receiver to label the new session by the relay's sender
fingerprint hint (`fp:<senderfp>`); the post-decrypt plaintext
typically announces the peer's real address. Aliasing moves
session, trusted identity, peer-verification, and identity-
version under the canonical label. Holds the per-peer mutex on
both labels (lexicographic order) so concurrent crypto ops can't
observe a half-moved state. Refuses to overwrite an existing
session at the new label.
Wire change: `IncomingMessage.expiresAt?` now surfaces the relay's
expiry so receivers can pass bridge frames straight to
`acceptBridgeFrame` without inventing a TTL.
Tests cover bridge-then-poll, poll-then-bridge, aliasSession happy
path, refuse-to-overwrite, and same-label no-op.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 15:49:36 +02:00
/ * *
* Move every per - peer storage row for ` oldLabel ` ( session , trusted
* identity , peer - verification , identity - version counter ) to
* ` newLabel ` . Use this when first - contact forced you to label a
* session by the relay ' s sender - fingerprint hint
* ( ` fp:<hex> ` — see ` IncomingMessage.from ` / ` FetchedBlob.from ` ) and
* the just - decrypted plaintext announces the peer ' s canonical
* address : alias once and every subsequent
* ` send ` / ` receive ` / broadcast cross - check operates under the
* announced label , no app - side fp ↔ address mapping needed for the
* receive path .
*
* The rename is atomic from a per - peer - mutex perspective — both
* labels are locked for the duration so concurrent encrypt / decrypt
* can ' t observe a half - moved state . Throws if ` oldLabel ` has no
* session , or if ` newLabel ` already does ( refuses to overwrite —
* call ` resetSession ` first if that ' s intentional ) .
*
* After alias , the SDK ' s internal serialization queues
* ( ` encryptChains ` , ` decryptChains ` ) for ` oldLabel ` are dropped so
* future operations don ' t queue behind a stale chain .
*
* V4 . 8.3 — Prism FR ` session-label-asymmetry-v4.8.2.md ` .
* /
async aliasSession ( oldLabel : string , newLabel : string ) : Promise < void > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
await this . manager . aliasSession ( oldLabel , newLabel ) ;
// The SDK's per-`from` chains are keyed by label; drop the old
// entries so future `send`/`receive` to either label start with a
// fresh queue rather than chaining off whatever was last in flight
// for `oldLabel`.
this . encryptChains . delete ( oldLabel ) ;
this . decryptChains . delete ( oldLabel ) ;
}
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
/ * *
* Accept a peer ' s rotated identity . Bumps the per - peer identity - version
* counter so any earlier verification automatically goes stale , then
* runs the ` beforeNewDeviceTrust ` gate before the new key is pinned .
* /
async acceptIdentityChange ( address : string , newIdentityKey : Uint8Array ) : Promise < void > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
await this . storage . bumpPeerIdentityVersion ( address ) ;
const newFingerprint = await computeFingerprint (
this . crypto ,
// X3DH stores DH-only "trusted identity"; in this SDK the trusted
// entry IS the DH public key. We feed it as both args so the
// fingerprint binds to the rotated key material the user is asked
// to confirm.
newIdentityKey ,
newIdentityKey ,
) ;
await this . gates . checkNewDeviceTrust ( address , newFingerprint ) ;
await this . manager . acceptIdentityChange ( address , newIdentityKey ) ;
}
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
/** Manually rotate the identity (destructive — see docs) */
async rotate ( ) : Promise < void > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
// Rotate locally first
const newBundle = await this . manager . rotateIdentity ( ) ;
// Rebuild the transport with the new signing key so subsequent
// signed operations (replenish, delete, register) work
const identity = await this . storage . getIdentityKeyPair ( ) ;
if ( ! identity ) throw new Error ( 'Identity missing after rotate' ) ;
this . transport = new ShadeFetchTransport ( {
baseUrl : this.config.prekeyServer ,
crypto : this.crypto ,
signingPrivateKey : identity.signingPrivateKey ,
} ) ;
// Re-upload the new bundle
await this . transport . register (
this . address ,
this . manager . getPublicIdentity ( ) ,
newBundle . signedPreKey ,
[ ] ,
) ;
// Rebuild background tasks so they use the new transport
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
if ( this . _background ) {
this . _background . stop ( ) ;
this . _background = new BackgroundTasks (
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
this . manager ,
this . transport ,
this . address ,
this . config ,
) ;
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
this . _background . start ( ) ;
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
}
}
/** Manually trigger replenishment (normally background task handles this) */
async replenish ( ) : Promise < number > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
if ( ! this . _background ) return 0 ;
return this . _background . runReplenish ( ) ;
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
}
2026-04-11 00:51:34 +02:00
/ * *
* Export an encrypted backup blob that can be restored to a new device .
*
* @param passphrase User passphrase ( minimum 12 characters )
* @param knownAddresses Peer addresses whose sessions should be included
* /
async exportBackup ( passphrase : string , knownAddresses : string [ ] = [ ] ) : Promise < string > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
const blob = await exportBackup ( this . crypto , this . storage , passphrase , knownAddresses ) ;
return backupToString ( blob ) ;
}
/ * *
* Restore state from a backup string . Overwrites existing state .
* Call this BEFORE initialize ( ) on a fresh device , or after shutdown ( ) + re - init .
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
*
* V3.3 : invokes the ` beforeBackupImport ` gate . The handler receives the
* fingerprint of the identity * embedded in the backup * — this lets the
* user OOB - confirm that the backup is theirs before sessions and
* pinned - trust entries are written to disk .
2026-04-11 00:51:34 +02:00
* /
async importBackup ( backupString : string , passphrase : string ) : Promise < void > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
const blob = backupFromString ( backupString ) ;
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
const payload = await decryptBackup ( this . crypto , blob , passphrase ) ;
const backupFingerprint = await fingerprintFromBackupPayload ( this . crypto , payload ) ;
await this . gates . checkBackupImport ( this . address , backupFingerprint ) ;
await applyBackupPayload ( this . storage , payload ) ;
2026-04-11 00:51:34 +02:00
// Reload identity after restore
const restored = await this . storage . getIdentityKeyPair ( ) ;
if ( restored ) {
// Rebuild the manager and transport with the restored identity
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
this . manager = new ShadeSessionManager ( this . crypto , this . storage , {
events : this.events ,
. . . ( this . config . observability !== undefined
? { observability : this.config.observability }
: { } ) ,
} ) ;
2026-04-11 00:51:34 +02:00
await this . manager . initialize ( ) ;
this . transport = new ShadeFetchTransport ( {
baseUrl : this.config.prekeyServer ,
crypto : this.crypto ,
signingPrivateKey : restored.signingPrivateKey ,
} ) ;
}
}
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
/** Clean shutdown: stop timers, close storage if it supports it */
async shutdown ( ) : Promise < void > {
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
this . _background ? . stop ( ) ;
if ( this . transferEngine !== null ) this . transferEngine . destroy ( ) ;
if ( this . controlChannel !== null ) this . controlChannel . destroy ( ) ;
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
if ( this . webrtcRuntime !== null ) {
this . webrtcRuntime . destroy ( ) ;
this . webrtcRuntime = null ;
}
if ( this . workerCrypto !== null ) {
await this . workerCrypto . destroy ( ) ;
this . workerCrypto = null ;
}
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
// Close storage if it has a close method (SQLite)
const closable = this . storage as unknown as { close ? : ( ) = > void | Promise < void > } ;
if ( typeof closable . close === 'function' ) {
await closable . close ( ) ;
}
this . initialized = false ;
}
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
// ─── Worker-Crypto streams (V3.8) ──────────────────────────
/ * *
* Opt in to Web Workers crypto : subsequent ` encryptStream ` /
* ` decryptStream ` calls offload all AEAD work to a dedicated worker so
* the main thread stays under the 16 ms - per - frame budget for big
* uploads . The worker is spawned on first use and self - terminates
* after ` idleTimeoutMs ` of inactivity ( default 30 s ) .
*
* Bundlers resolve worker URLs differently — the recommended idiom is :
*
* ` ` ` ts
* shade . configureWorkerCrypto ( {
* workerUrl : new URL ( '@shade/crypto-web/worker' , import . meta . url ) ,
* } ) ;
* ` ` `
*
* See ` docs/web-workers.md ` for Vite / Webpack / Rollup recipes and
* Safari notes .
* /
configureWorkerCrypto ( opts : {
workerUrl : URL | string ;
idleTimeoutMs? : number ;
} ) : void {
this . workerCryptoConfig = opts ;
}
/ * *
* Encrypt a ` ReadableStream<Uint8Array> ` of plaintext into stream - chunk
* wire envelopes via a Web Worker .
*
* The caller pre - negotiates ` streamId ` + ` streamSecret ` with the peer
* ( typically through ` shade.upload() ` for HTTP - based delivery , or any
* other channel ) . The returned ` stream ` is a TransformStream :
* pipe plaintext in , get encrypted chunks out .
*
* ` laneSha256 ` resolves once the stream finishes ( final chunk emitted
* with ` isLast=true ` ) . Compare it against the receiver ' s lane sha256
* for end - to - end integrity proof .
*
* Requires ` configureWorkerCrypto() ` to be called first .
* /
encryptStream (
opts : Omit < CreateEncryptStreamOptions , 'provider' > ,
) : Promise < {
stream : TransformStream < Uint8Array , Uint8Array > ;
laneSha256 : Promise < Uint8Array > ;
} > {
return this . ensureWorkerCrypto ( ) . then ( ( provider ) = >
createEncryptStream ( { provider , . . . opts } ) ,
) ;
}
/ * *
* Inverse of { @link Shade . encryptStream } — decrypt incoming wire
* envelopes back into plaintext . Each input chunk MUST be a complete
* stream - chunk envelope ( the wire framing is the caller ' s job ) .
*
* Requires ` configureWorkerCrypto() ` to be called first .
* /
decryptStream (
opts : Omit < CreateDecryptStreamOptions , 'provider' > ,
) : Promise < {
stream : TransformStream < Uint8Array , Uint8Array > ;
laneSha256 : Promise < Uint8Array > ;
} > {
return this . ensureWorkerCrypto ( ) . then ( ( provider ) = >
createDecryptStream ( { provider , . . . opts } ) ,
) ;
}
/ * *
* Direct access to the worker - backed ` CryptoProvider ` . Use when you
* want to run a one - off heavy crypto op ( X25519 batch DH , big HKDF
* derivation , etc . ) off the main thread without setting up a stream .
* /
async getWorkerCrypto ( ) : Promise < WorkerCryptoProvider > {
return this . ensureWorkerCrypto ( ) ;
}
private async ensureWorkerCrypto ( ) : Promise < WorkerCryptoProvider > {
if ( this . workerCrypto !== null ) return this . workerCrypto ;
if ( this . workerCryptoBoot !== null ) return this . workerCryptoBoot ;
if ( this . workerCryptoConfig === null ) {
throw new Error (
'Call shade.configureWorkerCrypto({ workerUrl }) before encryptStream()/decryptStream(). See docs/web-workers.md.' ,
) ;
}
const cfg = this . workerCryptoConfig ;
this . workerCryptoBoot = ( async ( ) = > {
const provider = await createWorkerCryptoProvider ( {
workerUrl : cfg.workerUrl ,
. . . ( cfg . idleTimeoutMs !== undefined ? { idleTimeoutMs : cfg.idleTimeoutMs } : { } ) ,
} ) ;
this . workerCrypto = provider ;
return provider ;
} ) ( ) ;
try {
return await this . workerCryptoBoot ;
} finally {
this . workerCryptoBoot = null ;
}
}
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
// ─── Stream transfers (v0.2.0) ─────────────────────────────
/ * *
* Configure how stream - transfer chunks reach peers . Provide a resolver
* that returns the peer ' s HTTP base URL ( e . g . by looking up a
* ` transfer.baseUrl ` field in your prekey - bundle metadata or a static
* directory map ) . If unset , ` upload() ` rejects with a clear error .
*
* Optionally also override the control - envelope transport ( defaults to
* HTTP POSTs to ` <base>/v1/transfer/control ` ) .
* /
configureTransfers ( opts : {
2026-05-03 23:27:06 +02:00
/ * *
* Resolver for the peer ' s HTTP base URL ( used by the default
* ` ShadeTransferHttpTransport ` to POST chunks ) . Optional when a
* custom ` transport ` and ` envelopeTransport ` are supplied — e . g .
* for pull - mode browser servers ( ` @shade/files transferQueueRoute ` )
* which never POST chunks anywhere .
* /
resolveBaseUrl ? : ( peerAddress : string ) = > Promise < string > ;
/ * *
* Override the chunk - level transport . Defaults to
* ` ShadeTransferHttpTransport ` ( HTTP POSTs per chunk ) when
* ` resolveBaseUrl ` is supplied . Required when ` resolveBaseUrl `
* is omitted .
* /
transport? : ITransferTransport ;
/ * *
* Override the control - envelope transport . Defaults to HTTP POSTs
* to ` <base>/v1/transfer/control ` when ` resolveBaseUrl ` is
* supplied . Required when ` resolveBaseUrl ` is omitted .
* /
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
envelopeTransport? : ControlEnvelopeTransport ;
} ) : void {
2026-05-03 23:27:06 +02:00
if ( opts . resolveBaseUrl === undefined ) {
if ( opts . transport === undefined || opts . envelopeTransport === undefined ) {
throw new Error (
'configureTransfers: resolveBaseUrl is required unless both `transport` and `envelopeTransport` are supplied (e.g. for pull-mode queue servers).' ,
) ;
}
this . peerBaseUrlResolver = async ( ) = > {
throw new Error (
'resolveBaseUrl was not configured — this Shade is in queue/pull mode and does not POST chunks. Configure a custom transport instead.' ,
) ;
} ;
} else {
this . peerBaseUrlResolver = opts . resolveBaseUrl ;
}
this . transferTransportOverride = opts . transport ? ? null ;
if ( opts . envelopeTransport !== undefined ) {
this . envelopeOutboxes = opts . envelopeTransport ;
} else if ( opts . resolveBaseUrl !== undefined ) {
this . envelopeOutboxes = new HttpEnvelopeTransport ( opts . resolveBaseUrl , this . address ) ;
} else {
throw new Error (
'configureTransfers: envelopeTransport is required when resolveBaseUrl is omitted.' ,
) ;
}
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
}
/ * *
* Deliver a freshly - encrypted ratchet envelope to a peer using the
* configured envelope transport ( HTTP POST to ` /v1/transfer/control ` by
* default ) . Used by ` @shade/files ` for RPC plaintext delivery .
* /
async deliverControlEnvelope ( peerAddress : string , envelope : ShadeEnvelope ) : Promise < void > {
if ( this . envelopeOutboxes === null ) {
throw new Error (
'Call shade.configureTransfers({ resolveBaseUrl }) before deliverControlEnvelope()' ,
) ;
}
await this . envelopeOutboxes . send ( peerAddress , envelope ) ;
}
/ * *
* Upload bytes to a peer . Returns a ` TransferHandle ` that can be paused /
* aborted and awaited . Requires ` configureTransfers ` to be called first .
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
*
* V3.3 : when the file size is at or above the configured threshold
* ( default 10 MiB ) and the peer is not yet verified , the registered
* ` beforeFirstLargeFile ` handler is invoked . Rejection throws
* ` FingerprintNotVerifiedError ` before any bytes hit the wire .
*
* V3.9 : pass ` thumbnail: { bytes, mime } ` to attach a pre - generated
* preview , or ` generateThumbnail: true ` to auto - derive a 256 x256 preview
* from an image input in browser - class runtimes ( no - op elsewhere ) . The
* thumbnail is shipped as a * separate * E2EE stream and referenced from
* the main stream ' s ` fileMetadata ` .
* /
async upload ( opts : ShadeUploadOptions ) : Promise < TransferHandle > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
const size = inferTransferSize ( opts ) ;
if ( size !== null && size >= this . gates . getFirstLargeFileThreshold ( ) ) {
// Establish the session up-front so we have a fingerprint to gate on.
// For peers we've never contacted, this is the TOFU moment where the
// gate matters most.
if ( ( await this . storage . getSession ( opts . to ) ) === null ) {
await this . ensureSession ( opts . to ) ;
}
const fingerprint = await this . manager . getRemoteFingerprint ( opts . to ) ;
await this . gates . checkFirstLargeFile ( opts . to , fingerprint , size ) ;
}
const engine = await this . engine ( ) ;
const thumbnail = await this . resolveThumbnail ( opts ) ;
if ( thumbnail !== null ) {
const fileMeta : StreamFileMetadata = {
. . . ( opts . metadata ? . fileMetadata ? ? { } ) ,
thumbnailStreamId : thumbnail.streamId ,
thumbnailHash : thumbnail.hashB64 ,
thumbnailMime : thumbnail.mime ,
thumbnailBytes : thumbnail.bytes ,
} ;
const merged : TransferOptions = {
. . . opts ,
metadata : {
. . . ( opts . metadata ? ? { } ) ,
fileMetadata : fileMeta ,
} ,
} ;
return engine . upload ( merged ) ;
}
return engine . upload ( opts ) ;
}
/ * *
* Coordinate the thumbnail - side of a V3 . 9 upload . Resolves to either
* - ` null ` — no thumbnail will be attached ( caller passed neither
* ` thumbnail ` nor a generator that produced bytes ) , or
* - the streamId + sha256 + mime + bytes of the thumbnail - stream that
* has now been kicked off ( it runs to completion in the background ;
* the main upload ' s ` done() ` is independent ) .
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
* /
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
private async resolveThumbnail ( opts : ShadeUploadOptions ) : Promise < {
streamId : string ;
hashB64 : string ;
mime : ThumbnailMime ;
bytes : number ;
} | null > {
let bytes : Uint8Array | null = null ;
let mime : ThumbnailMime | null = null ;
if ( opts . thumbnail !== undefined ) {
bytes = opts . thumbnail . bytes ;
mime = opts . thumbnail . mime ;
} else if ( opts . generateThumbnail !== undefined && opts . generateThumbnail !== false ) {
const genOpts : ThumbnailGenerationOptions =
opts . generateThumbnail === true ? { } : opts . generateThumbnail ;
const gen = await generateThumbnailFromBlob ( opts . input , genOpts ) ;
if ( gen !== null ) {
bytes = gen . bytes ;
mime = gen . mime ;
}
}
if ( bytes === null || mime === null ) return null ;
if ( bytes . byteLength > THUMBNAIL_MAX_BYTES ) {
throw new Error (
` thumbnail size ${ bytes . byteLength } exceeds THUMBNAIL_MAX_BYTES ( ${ THUMBNAIL_MAX_BYTES } ) ` ,
) ;
}
if ( ! isAllowedThumbnailMime ( mime ) ) {
throw new Error ( ` thumbnail mime ${ mime } not in allowlist ` ) ;
}
const hash = sha256Once ( bytes ) ;
const hashB64 = bytesToBase64Std ( hash ) ;
const engine = await this . engine ( ) ;
// Ship the thumbnail FIRST so the receiver can present a preview the
// moment the main `stream-init` references it. Single lane, single
// chunk — at ≤ 64 KiB the parallelism overhead would dominate.
const handle = await engine . upload ( {
to : opts.to ,
input : bytes ,
lanes : 1 ,
chunkSize : Math.max ( 1 , bytes . byteLength ) ,
metadata : {
contentType : mime ,
userMetadata : {
shadeThumbnail : '1' ,
} ,
} ,
} ) ;
// Don't await `done()` — the main upload should not block on the
// thumbnail finishing. Errors on the preview are surfaced via the
// returned handle's events (consumer can listen if they care).
handle . done ( ) . catch ( ( err ) = > {
console . warn ( '[Shade] thumbnail transfer failed:' , err ) ;
} ) ;
return {
streamId : handle.streamId ,
hashB64 ,
mime ,
bytes : bytes.byteLength ,
} ;
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
}
/ * *
* Subscribe to incoming transfers from peers . Handler is invoked when a
* ` stream-init ` arrives ; the handler MUST call ` incoming.accept(...) ` to
* begin receiving ( or ` incoming.decline(...) ` to reject ) .
* /
async onIncomingTransfer (
handler : ( incoming : IncomingTransfer ) = > void | Promise < void > ,
) : Promise < ( ) = > void > {
return ( await this . engine ( ) ) . onIncomingTransfer ( handler ) ;
}
2026-05-03 23:27:06 +02:00
/ * *
* Mount the * * pull - mode * * transfer routes on a Hono app . Mount under
* any base path : ` app.route('/api/v1/shade-files', shade.transferQueueRoute()) ` .
*
* Configures this Shade instance to queue all outbound chunks +
* control envelopes per peer instead of POSTing them . Browser - style
* receivers drain the queue via long - polling — no inbound HTTP
* listener required on the receiver .
*
* Routes mounted ( relative to the base path ) :
* POST / queue — long - poll the per - peer outbound queue
* POST / v1 / transfer / : streamId / chunk — receive incoming chunks ( browser → server )
* GET / v1 / transfer / : streamId / state — resume - state lookup
* POST / v1 / transfer / control — receive incoming control envelopes
* GET / v1 / transfer / health — peer reachability probe
*
* * * Idempotent * * : calling twice returns a fresh ` Hono ` app each
* time but reuses the underlying queue + transport ( so the engine
* stays single ) .
*
* * * Ordering * * : must be called * * before * * ` shade.files.serve(...) `
* ( or any other path that builds the engine ) , because configuring
* the queue transport mutates the transfer stack . Calling after the
* engine is built throws .
* /
async transferQueueRoute ( opts : TransferQueueRouteOptions = { } ) : Promise < Hono > {
if ( this . transferEngine !== null && this . transferTransportOverride === null ) {
throw new Error (
'transferQueueRoute(): the transfer engine has already been built with the default HTTP transport. Call transferQueueRoute() before any upload()/onIncomingTransfer()/configureTransfers().' ,
) ;
}
const { OutboundQueue , QueueTransferTransport } = await import ( '@shade/transfer' ) ;
if ( this . transferQueue === null ) {
this . transferQueue = new OutboundQueue ( {
. . . ( opts . maxEventsPerPeer !== undefined ? { maxEventsPerPeer : opts.maxEventsPerPeer } : { } ) ,
. . . ( opts . idleEvictionMs !== undefined ? { idleEvictionMs : opts.idleEvictionMs } : { } ) ,
} ) ;
}
if ( this . transferTransportOverride === null ) {
const queueTransport = new QueueTransferTransport ( this . transferQueue ) ;
const queueEnvelopeTransport = new QueueEnvelopeTransport ( this . transferQueue ) ;
this . configureTransfers ( {
transport : queueTransport ,
envelopeTransport : queueEnvelopeTransport ,
} ) ;
}
const queue = this . transferQueue ;
const blockMs = opts . blockMs ? ? 30 _000 ;
const maxBlockMs = opts . maxBlockMs ? ? 55 _000 ;
const engine = await this . engine ( ) ;
const { createTransferRoutes , PermissiveAuthenticator } = await import ( '@shade/transfer' ) ;
const app = await createTransferRoutes ( engine , {
authenticator : PermissiveAuthenticator ,
} ) ;
app . post ( '/v1/transfer/control' , async ( c ) = > {
const senderAddress = c . req . header ( 'X-Shade-Sender-Address' ) ;
if ( senderAddress === undefined || senderAddress === '' ) {
return c . json ( { error : 'missing X-Shade-Sender-Address' } , 400 ) ;
}
const ab = await c . req . arrayBuffer ( ) ;
const bytes = new Uint8Array ( ab ) ;
try {
await this . acceptTransferEnvelope ( senderAddress , bytes ) ;
} catch ( err ) {
return c . json ( { error : ( err as Error ) . message } , 400 ) ;
}
return c . json ( { ok : true } ) ;
} ) ;
// Long-poll endpoint.
app . post ( '/queue' , async ( c ) = > {
const senderAddress = c . req . header ( 'X-Shade-Sender-Address' ) ;
if ( senderAddress === undefined || senderAddress === '' ) {
return c . json ( { error : 'missing X-Shade-Sender-Address' } , 400 ) ;
}
let body : { since? : unknown ; blockMs? : unknown } ;
try {
body = ( await c . req . json ( ) ) as { since? : unknown ; blockMs? : unknown } ;
} catch {
return c . json ( { error : 'invalid JSON body' } , 400 ) ;
}
const since = typeof body . since === 'number' && Number . isFinite ( body . since ) ? body.since : 0 ;
const requestedBlockMs =
typeof body . blockMs === 'number' && Number . isFinite ( body . blockMs )
? Math . max ( 0 , Math . min ( maxBlockMs , body . blockMs ) )
: blockMs ;
// Bun-side short-circuit if the request was aborted while we
// were holding the long-poll. AbortSignal from the request body
// is already surfaced via `c.req.raw.signal` in Hono.
const events = await queue . drain ( senderAddress , since , requestedBlockMs , c . req . raw . signal ) ;
return c . json ( {
events : events.map ( ( e ) = > ( {
id : e.id ,
timestampMs : e.timestampMs ,
kind : e.kind ,
bytesB64 : bytesToBase64Std ( e . bytes ) ,
. . . ( e . kind === 'chunk' ? { meta : e.meta } : { } ) ,
} ) ) ,
nextSince : events.length > 0 ? events [ events . length - 1 ] ! . id : since ,
} ) ;
} ) ;
return app ;
}
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
/ * *
* Mount the receiver - side HTTP routes on a Hono app . Mount under any
* base path : ` app.route('/shade', await shade.transferRoute()) ` .
*
* Routes :
* POST / v1 / transfer / : streamId / chunk — wire - encoded 0x11 chunks
* GET / v1 / transfer / : streamId / state — resume - state lookup
* POST / v1 / transfer / control — wire - encoded 0x02 control envelopes
* GET / v1 / transfer / health — peer reachability probe
* /
async transferRoute ( ) : Promise < Hono > {
const engine = await this . engine ( ) ;
const { createTransferRoutes , PermissiveAuthenticator } = await import ( '@shade/transfer' ) ;
const app = await createTransferRoutes ( engine , {
authenticator : PermissiveAuthenticator ,
} ) ;
// Add the control-envelope POST route on top.
app . post ( '/v1/transfer/control' , async ( c ) = > {
const senderAddress = c . req . header ( 'X-Shade-Sender-Address' ) ;
if ( senderAddress === undefined || senderAddress === '' ) {
return c . json ( { error : 'missing X-Shade-Sender-Address' } , 400 ) ;
}
const ab = await c . req . arrayBuffer ( ) ;
const bytes = new Uint8Array ( ab ) ;
try {
await this . acceptTransferEnvelope ( senderAddress , bytes ) ;
} catch ( err ) {
return c . json ( { error : ( err as Error ) . message } , 400 ) ;
}
return c . json ( { ok : true } ) ;
} ) ;
return app ;
}
/ * *
* Low - level entry for custom transports : hand a ` 0x02 ` ratchet envelope
* ( control - plane ) or a ` 0x11 ` stream - chunk envelope to the engine .
* Used internally by ` transferRoute() ` .
* /
async acceptTransferEnvelope ( from : string , env : ShadeEnvelope | Uint8Array ) : Promise < void > {
const engine = await this . engine ( ) ;
if ( env instanceof Uint8Array ) {
const kind = inspectEnvelopeType ( env ) ;
if ( kind === 'stream-chunk' ) {
// Engine extracts laneId/seq from the wire bytes via decodeStreamChunk.
const headers = parseChunkHeader ( env ) ;
await engine . receiveChunk ( from , headers . streamId , headers . laneId , headers . seq , env ) ;
return ;
}
if ( kind === 'ratchet' || kind === 'prekey' ) {
const decoded = decodeEnvelope ( env ) ;
await this . controlChannel ! . acceptEnvelope ( from , decoded ) ;
return ;
}
throw new Error ( ` Unknown envelope type ${ kind } ` ) ;
}
// Already-decoded envelope (ratchet or prekey)
await this . controlChannel ! . acceptEnvelope ( from , env ) ;
}
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
// ─── V3.11 WebRTC P2P transport ────────────────────────────
/ * *
* Opt in to direct peer - to - peer chunk delivery via WebRTC .
*
* When configured , ` upload() ` builds a ` [WebRTC, HTTP] `
* { @link MultiTransportFallback } : P2P first , HTTP as automatic
* fallback . Signaling ( SDP offer / answer + trickle - ICE ) rides on top
* of ` Shade.send ` / ` Shade.onMessage ` — no out - of - band server needed .
*
* Must be called BEFORE the first ` upload() ` / ` onIncomingTransfer() `
* ( those instantiate the transfer engine , which captures the
* transport stack at construction time ) . Calling later throws .
*
* The ` factory ` argument is the WebRTC adapter — ` nativeRtcFactory() `
* for browsers , a custom one for Node - class environments
* ( ` node-datachannel ` , ` wrtc ` , etc . ) . Set ` iceServers ` to override the
* default public STUN list , or supply TURN credentials for paranoid
* NATs :
*
* ` ` ` ts
* import { nativeRtcFactory } from '@shade/transport-webrtc' ;
* shade . configureWebRTC ( {
* factory : nativeRtcFactory ( ) ,
* iceServers : [
* { urls : 'stun:stun.l.google.com:19302' } ,
* { urls : 'turn:turn.example.com:3478' , username : 'u' , credential : 'p' } ,
* ] ,
* } ) ;
* ` ` `
* /
configureWebRTC ( opts : ShadeWebRtcConfig ) : void {
if ( this . transferEngine !== null ) {
throw new Error (
'shade.configureWebRTC() must be called before upload()/onIncomingTransfer() builds the engine' ,
) ;
}
this . webrtcConfig = opts ;
}
/ * *
* Returns the live WebRTC runtime ( signaling channel + connection
* manager + transport ) if ` configureWebRTC ` was called and ` engine() `
* has been instantiated . Useful for diagnostics : peek
* ` runtime.manager.isConnected('alice') ` to see whether a P2P link is
* live , or wire ` runtime.fallback.onSwitch(...) ` to log demotions .
* /
getWebRtcRuntime ( ) : ShadeWebRtcRuntime | null {
return this . webrtcRuntime ;
}
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
// ─── Internals ─────────────────────────────────────────────
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
private async engine ( ) : Promise < TransferEngine > {
if ( this . transferEngine !== null ) return this . transferEngine ;
if ( this . peerBaseUrlResolver === null || this . envelopeOutboxes === null ) {
throw new Error (
'Call shade.configureTransfers({ resolveBaseUrl }) before using upload()/onIncomingTransfer()' ,
) ;
}
this . controlChannel = new ShadeControlChannel ( this , this . envelopeOutboxes ) ;
2026-05-03 23:27:06 +02:00
let transport : ITransferTransport ;
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
let webrtcRuntime : ShadeWebRtcRuntime | null = null ;
2026-05-03 23:27:06 +02:00
if ( this . transferTransportOverride !== null ) {
// Custom transport (queue, in-memory, custom adapter) — used as-is.
// WebRTC fallback only attaches when the default HTTP transport is
// active because WebRTC's `MultiTransportFallback` is HTTP-shaped.
transport = this . transferTransportOverride ;
} else {
const httpTransport : ITransferTransport = new ShadeTransferHttpTransport ( {
resolveBaseUrl : this.peerBaseUrlResolver ,
authenticator : await this . makeAuthenticator ( ) ,
} ) ;
transport = httpTransport ;
if ( this . webrtcConfig !== null ) {
webrtcRuntime = await this . buildWebRtcRuntime ( this . webrtcConfig , httpTransport ) ;
transport = webrtcRuntime . fallback ;
}
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
}
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
this . transferEngine = new TransferEngine ( {
crypto : this.crypto ,
controlChannel : this.controlChannel ,
transport ,
myAddress : this.address ,
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
. . . ( this . config . observability !== undefined
? { observability : this.config.observability }
: { } ) ,
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
} ) ;
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
if ( webrtcRuntime !== null ) {
// Receiver-hooks need to dispatch into the freshly-built engine.
webrtcRuntime . attachEngine ( this . transferEngine ) ;
this . webrtcRuntime = webrtcRuntime ;
}
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
return this . transferEngine ;
}
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
/ * *
* Dynamically import ` @shade/transport-webrtc ` , wire its signaling
* channel onto our ` Shade.send ` / ` Shade.onMessage ` , and build a
* MultiTransportFallback that prefers WebRTC then falls back to HTTP .
* /
private async buildWebRtcRuntime (
cfg : ShadeWebRtcConfig ,
httpTransport : ITransferTransport ,
) : Promise < ShadeWebRtcRuntime > {
// `@shade/transport-webrtc` is an optional peer dep — keep the
// import lazy so consumers that don't use WebRTC don't pay for it.
const moduleId = '@shade/transport-webrtc' ;
const mod = ( await import ( moduleId ) ) as typeof import ( '@shade/transport-webrtc' ) ;
const {
WebRtcSignalingChannel ,
WebRtcConnectionManager ,
WebRtcTransferTransport ,
createShadeBridgeFromShade ,
} = mod ;
const signaling = new WebRtcSignalingChannel ( createShadeBridgeFromShade ( this ) ) ;
let engineRef : TransferEngine | null = null ;
const manager = new WebRtcConnectionManager ( {
factory : cfg.factory ,
signaling ,
. . . ( cfg . iceServers !== undefined ||
cfg . iceTransportPolicy !== undefined ||
cfg . bundlePolicy !== undefined
? {
config : {
. . . ( cfg . iceServers !== undefined ? { iceServers : cfg.iceServers } : { } ) ,
. . . ( cfg . iceTransportPolicy !== undefined
? { iceTransportPolicy : cfg.iceTransportPolicy }
: { } ) ,
. . . ( cfg . bundlePolicy !== undefined ? { bundlePolicy : cfg.bundlePolicy } : { } ) ,
} ,
}
: { } ) ,
. . . ( cfg . connectTimeoutMs !== undefined ? { connectTimeoutMs : cfg.connectTimeoutMs } : { } ) ,
receiver : {
onChunk : async ( from , streamId , laneId , seq , envelope ) = > {
if ( engineRef === null ) {
throw new Error ( 'webrtc receiver hook fired before engine attached' ) ;
}
return engineRef . receiveChunk ( from , streamId , laneId , seq , envelope ) ;
} ,
onResumeQuery : async ( from , streamId ) = > {
if ( engineRef === null ) return null ;
return engineRef . getResumeState ( from , streamId ) ;
} ,
} ,
} ) ;
const webrtcTransport = new WebRtcTransferTransport ( {
manager ,
. . . ( cfg . requestTimeoutMs !== undefined ? { requestTimeoutMs : cfg.requestTimeoutMs } : { } ) ,
. . . ( cfg . backpressureThresholdBytes !== undefined
? { backpressureThresholdBytes : cfg.backpressureThresholdBytes }
: { } ) ,
} ) ;
const fallback = new MultiTransportFallback ( [
{ name : 'webrtc' , transport : webrtcTransport } ,
{ name : 'http' , transport : httpTransport } ,
] ) ;
return {
signaling ,
manager ,
transport : webrtcTransport ,
fallback ,
attachEngine ( engine ) {
engineRef = engine ;
} ,
destroy() {
manager . destroy ( ) ;
signaling . destroy ( ) ;
} ,
} ;
}
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
private async makeAuthenticator ( ) : Promise < ShadeTransferAuthenticator > {
const identity = await this . storage . getIdentityKeyPair ( ) ;
if ( identity === null ) throw new Error ( 'Identity not initialized' ) ;
return new ShadeTransferAuthenticator ( this . crypto , this . address , identity . signingPrivateKey ) ;
}
/** Returns a list of in-flight stream transfers from storage (resume support). */
async listTransfers ( filter ? : {
direction ? : 'send' | 'receive' ;
} ) : Promise < TransferSummary [ ] > {
if ( this . storage . listActiveStreamStates === undefined ) return [ ] ;
const rows = await this . storage . listActiveStreamStates ( filter ? . direction ) ;
return rows . map ( ( s ) = > ( {
streamId : s.streamId ,
direction : s.direction ,
peerAddress : s.peerAddress ,
status : s.status ,
bytesProcessed : 0 , // computed from laneState
createdAt : s.createdAt ,
updatedAt : s.updatedAt ,
metadata : tryParseMetadata ( s . metadataJson ) ,
} ) ) ;
}
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
/ * *
* Drop persisted stream - state records whose ` updatedAt ` is strictly
* less than ` olderThan ` ( Unix ms ) . Idempotent . Returns silently when
* the configured storage backend does not implement stream - state
* persistence ( e . g . memory storage in tests ) .
*
* Recommended usage : schedule on a daily cron with a 14 - day horizon
* — see ` docs/streams.md ` § Retention . The ` bun-server ` SDK template
* wires this up by default .
* /
async pruneStreamStates ( olderThan : number ) : Promise < void > {
if ( this . storage . pruneStreamStates === undefined ) return ;
await this . storage . pruneStreamStates ( olderThan ) ;
}
release(v4.11.0): streaming Double-Ratchet sub-sessions (ShadeStream)
Answers Vyvern FR shade-ws-streaming-ratchet.md with a first-class
streaming-session API rather than the documented-contract fallback.
The Double-Ratchet crypto was already safe for high-frequency
one-directional use; the send/receive wrapper was not (per-frame
saveSession keystore write; shared per-peer mutex + single stored
session row coupling reuse to the HTTP path).
- @shade/core: stream.ts — identity-bound 3-DH seeding (X3DH-minus-
prekeys, no prekey-server round trip, mutually authenticated against
the parent session's pinned identities), bootstrapStreamSession
reusing init{Sender,Receiver}Session verbatim, in-memory-only
StreamRatchet (own op-mutex, never persisted, zeroized on close).
beginStream/acceptStream on ShadeSessionManager; Stream{Closed,
Handshake}Error; stream.opened/closed events.
- @shade/proto: STREAM_OPEN/OPEN_ACK/FRAME wire (0x31/0x32/0x33),
additive; inspectEnvelopeType extended.
- @shade/sdk: Shade.openStream/acceptStream → ShadeStream
(handshakeFrame/handleHandshake/seal/open/close), transport-
agnostic, independent of encrypt/decrypt queues + parent session,
identical server (sqlite:) and browser (IndexedDB) — touches no
storage.
- Tests: 5000-frame one-directional burst (bounded skipped keys + FS
zeroize), parent-session independence, replay/rewind rejection,
mutual-auth, proto wire round-trips. Full suite green (1159 pass).
- docs/streaming-sessions.md (R1–R7 contract); SECURITY.md matrix rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 11:29:09 +02:00
// ─── Streaming sub-sessions (V4.11) ────────────────────────
/ * *
* Open a long - lived streaming Double - Ratchet sub - session to an
* already - known peer , for wrapping individual frames on a
* bidirectional , often server - heavy channel ( e . g . a console - log
* WebSocket ) with the same confidentiality / forward - secrecy /
* replay guarantees as the HTTP ` send ` / ` receive ` path .
*
* This is the * * initiator * * half . Like the rest of the SDK it is
* transport - agnostic : it produces handshake / frame bytes you put on
* your WebSocket , and consumes the bytes you receive from it .
*
* ` ` ` ts
* const stream = await shade . openStream ( peerAddr ) ;
* ws . send ( stream . handshakeFrame ( ) ) ; // → STREAM_OPEN
* // … first inbound WS frame is the peer's STREAM_OPEN_ACK …
* await stream . handleHandshake ( ackBytes ) ; // stream now usable
* ws . send ( await stream . seal ( utf8 ( line ) ) ) ; // outbound frame
* onLog ( await stream . open ( inboundBytes ) ) ; // inbound frame
* await stream . close ( ) ; // on ws close
* ` ` `
*
* Independence ( R5 ) : this never touches the stored parent session ,
* its prekeys , or the per - peer ` send ` / ` receive ` queues — it runs
* concurrently against the same peer . The ratchet lives only in
* memory and is zeroized by { @link ShadeStream . close } ; a dropped
* connection is re - opened with a fresh ` openStream ` , never resumed
* ( persisting per - frame ratchet secrets would defeat forward
* secrecy ) .
*
* Note ( Double - Ratchet semantics ) : a responder cannot ` seal ` until
* it has ` open ` ed at least one frame from the initiator ( standard
* Signal behaviour ) . For a server - heavy stream either make the bursty
* sender the initiator , or have the initiator send one priming frame
* right after the handshake .
*
* Requires an established parent session ; one is auto - established
* ( same path as { @link send } ) if missing .
* /
async openStream ( peerAddress : string ) : Promise < ShadeStream > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
let begun ;
try {
begun = await this . manager . beginStream ( peerAddress ) ;
} catch ( err ) {
if ( ! ( err instanceof NoSessionError ) ) throw err ;
await this . ensureSession ( peerAddress ) ;
begun = await this . manager . beginStream ( peerAddress ) ;
}
return new ShadeStream ( {
peer : peerAddress ,
role : 'initiator' ,
streamId : begun.streamId ,
events : this.events ,
handshakeOut : encodeStreamOpen ( begun . streamId , begun . ephemeralPublicKey ) ,
complete : begun.complete ,
} ) ;
}
/ * *
* Accept an inbound stream — the * * responder * * half . Feed it the
* peer ' s ` STREAM_OPEN ` bytes ( route by { @link inspectEnvelopeType }
* ` === 'stream-open' ` ) . The returned stream is immediately usable for
* ` open() ` ; send ` handshakeFrame() ` ( the ` STREAM_OPEN_ACK ` ) back over
* the transport so the initiator can complete its side .
* /
async acceptStream ( peerAddress : string , openBytes : Uint8Array ) : Promise < ShadeStream > {
if ( ! this . initialized ) throw new Error ( 'Not initialized' ) ;
const hs = decodeStreamHandshake ( openBytes ) ;
if ( hs . kind !== 'open' ) {
throw new StreamHandshakeError ( ` expected STREAM_OPEN, got ${ hs . kind } ` ) ;
}
let accepted ;
try {
accepted = await this . manager . acceptStream ( peerAddress , hs . streamId , hs . ephemeralPub ) ;
} catch ( err ) {
if ( ! ( err instanceof NoSessionError ) ) throw err ;
await this . ensureSession ( peerAddress ) ;
accepted = await this . manager . acceptStream ( peerAddress , hs . streamId , hs . ephemeralPub ) ;
}
return new ShadeStream ( {
peer : peerAddress ,
role : 'responder' ,
streamId : hs.streamId ,
events : this.events ,
handshakeOut : encodeStreamOpenAck ( hs . streamId , accepted . ephemeralPublicKey ) ,
ratchet : accepted.stream ,
} ) ;
}
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
private async ensureSession ( address : string ) : Promise < void > {
// Deduplicate concurrent establishment requests
const existing = this . establishing . get ( address ) ;
if ( existing ) {
await existing ;
return ;
}
const promise = ( async ( ) = > {
const bundle = await this . transport . fetchBundle ( address ) ;
await this . manager . initSessionFromBundle ( address , bundle ) ;
} ) ( ) ;
this . establishing . set ( address , promise ) ;
try {
await promise ;
} finally {
this . establishing . delete ( address ) ;
}
}
}
release(v4.11.0): streaming Double-Ratchet sub-sessions (ShadeStream)
Answers Vyvern FR shade-ws-streaming-ratchet.md with a first-class
streaming-session API rather than the documented-contract fallback.
The Double-Ratchet crypto was already safe for high-frequency
one-directional use; the send/receive wrapper was not (per-frame
saveSession keystore write; shared per-peer mutex + single stored
session row coupling reuse to the HTTP path).
- @shade/core: stream.ts — identity-bound 3-DH seeding (X3DH-minus-
prekeys, no prekey-server round trip, mutually authenticated against
the parent session's pinned identities), bootstrapStreamSession
reusing init{Sender,Receiver}Session verbatim, in-memory-only
StreamRatchet (own op-mutex, never persisted, zeroized on close).
beginStream/acceptStream on ShadeSessionManager; Stream{Closed,
Handshake}Error; stream.opened/closed events.
- @shade/proto: STREAM_OPEN/OPEN_ACK/FRAME wire (0x31/0x32/0x33),
additive; inspectEnvelopeType extended.
- @shade/sdk: Shade.openStream/acceptStream → ShadeStream
(handshakeFrame/handleHandshake/seal/open/close), transport-
agnostic, independent of encrypt/decrypt queues + parent session,
identical server (sqlite:) and browser (IndexedDB) — touches no
storage.
- Tests: 5000-frame one-directional burst (bounded skipped keys + FS
zeroize), parent-session independence, replay/rewind rejection,
mutual-auth, proto wire round-trips. Full suite green (1159 pass).
- docs/streaming-sessions.md (R1–R7 contract); SECURITY.md matrix rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 11:29:09 +02:00
// ─── ShadeStream (V4.11) ─────────────────────────────────────
interface ShadeStreamInit {
peer : string ;
role : 'initiator' | 'responder' ;
streamId : Uint8Array ;
events : ShadeEventEmitter ;
/** Bytes to put on the wire for our half of the handshake. */
handshakeOut : Uint8Array ;
/** Initiator only: continuation that derives the ratchet from the ACK. */
complete ? : ( peerEphemeralPub : Uint8Array ) = > Promise < StreamRatchet > ;
/** Responder only: ratchet is ready at accept time. */
ratchet? : StreamRatchet ;
}
function streamIdsEqual ( a : Uint8Array , b : Uint8Array ) : boolean {
if ( a . length !== b . length ) return false ;
for ( let i = 0 ; i < a . length ; i ++ ) if ( a [ i ] !== b [ i ] ) return false ;
return true ;
}
/ * *
* A live streaming Double - Ratchet sub - session . Transport - agnostic : it
* emits / consumes wire bytes , the caller owns the WebSocket ( or any
* other ordered frame transport ) .
*
* Lifecycle :
* - * * initiator * * : ` handshakeFrame() ` → ` STREAM_OPEN ` ; after the peer ' s
* ` STREAM_OPEN_ACK ` arrives call ` handleHandshake(ack) ` ; then
* ` seal ` / ` open ` .
* - * * responder * * : usable immediately ; ` handshakeFrame() ` →
* ` STREAM_OPEN_ACK ` to send back ; ` open ` the initiator ' s first frame
* before ` seal ` ( standard Double - Ratchet ordering ) .
* /
export class ShadeStream {
private readonly _streamId : Uint8Array ;
private readonly _peer : string ;
private readonly _role : 'initiator' | 'responder' ;
private readonly events : ShadeEventEmitter ;
private readonly handshakeOut : Uint8Array ;
private readonly complete ? : ( peerEphemeralPub : Uint8Array ) = > Promise < StreamRatchet > ;
private ratchet : StreamRatchet | null ;
private state : 'await-ack' | 'open' | 'closed' ;
constructor ( init : ShadeStreamInit ) {
this . _streamId = init . streamId ;
this . _peer = init . peer ;
this . _role = init . role ;
this . events = init . events ;
this . handshakeOut = init . handshakeOut ;
if ( init . role === 'initiator' ) {
if ( init . complete ) this . complete = init . complete ;
this . ratchet = null ;
this . state = 'await-ack' ;
} else {
this . ratchet = init . ratchet ? ? null ;
this . state = 'open' ;
}
}
/** Peer address this stream is bound to. */
get peer ( ) : string {
return this . _peer ;
}
/** Which half of the handshake this end performed. */
get role ( ) : 'initiator' | 'responder' {
return this . _role ;
}
/** Lowercase-hex stream id (stable for the stream's lifetime). */
get streamId ( ) : string {
return Array . from ( this . _streamId , ( b ) = > b . toString ( 16 ) . padStart ( 2 , '0' ) ) . join ( '' ) ;
}
/** True once the ratchet is established and not yet closed. */
get isOpen ( ) : boolean {
return this . state === 'open' && this . ratchet !== null ;
}
/ * *
* The bytes for our half of the handshake to put on the transport
* ( ` STREAM_OPEN ` for an initiator , ` STREAM_OPEN_ACK ` for a responder ) .
* Stable ; safe to read once and send .
* /
handshakeFrame ( ) : Uint8Array {
return this . handshakeOut ;
}
/ * *
* Initiator only : consume the peer ' s ` STREAM_OPEN_ACK ` and derive the
* ratchet . Idempotent - safe to call exactly once ; throws if called on
* a responder , out of order , or with a mismatched streamId .
* /
async handleHandshake ( ackBytes : Uint8Array ) : Promise < void > {
if ( this . _role !== 'initiator' ) {
throw new StreamHandshakeError ( 'handleHandshake is initiator-only' ) ;
}
if ( this . state !== 'await-ack' || ! this . complete ) {
throw new StreamHandshakeError ( 'handshake already completed or stream closed' ) ;
}
const hs = decodeStreamHandshake ( ackBytes ) ;
if ( hs . kind !== 'open-ack' ) {
throw new StreamHandshakeError ( ` expected STREAM_OPEN_ACK, got ${ hs . kind } ` ) ;
}
if ( ! streamIdsEqual ( hs . streamId , this . _streamId ) ) {
throw new StreamHandshakeError ( 'STREAM_OPEN_ACK streamId mismatch' ) ;
}
this . ratchet = await this . complete ( hs . ephemeralPub ) ;
this . state = 'open' ;
}
/ * *
* Seal one logical frame . Returns ` STREAM_FRAME ` wire bytes — put
* exactly one in one WS frame . Advances the sending chain one step .
* /
async seal ( plaintext : Uint8Array ) : Promise < Uint8Array > {
if ( ! this . ratchet || this . state !== 'open' ) {
throw new StreamHandshakeError ( 'stream not open (complete the handshake first)' ) ;
}
const msg = await this . ratchet . seal ( plaintext ) ;
return encodeStreamFrame ( this . _streamId , msg ) ;
}
/ * *
* Open one inbound ` STREAM_FRAME ` . Correct and memory - bounded across
* long one - directional bursts ; replays / counter - rewinds are rejected
* by the underlying ratchet .
* /
async open ( wire : Uint8Array ) : Promise < Uint8Array > {
if ( ! this . ratchet || this . state !== 'open' ) {
throw new StreamHandshakeError ( 'stream not open (complete the handshake first)' ) ;
}
const frame : { streamId : Uint8Array ; message : RatchetMessage } = decodeStreamFrame ( wire ) ;
if ( ! streamIdsEqual ( frame . streamId , this . _streamId ) ) {
throw new StreamHandshakeError ( 'STREAM_FRAME streamId mismatch' ) ;
}
return this . ratchet . open ( frame . message ) ;
}
/** Zeroize and drop the ratchet. Idempotent. */
async close ( ) : Promise < void > {
if ( this . state === 'closed' ) return ;
this . state = 'closed' ;
if ( this . ratchet ) {
await this . ratchet . close ( ) ;
this . ratchet = null ;
}
this . events . emit ( 'stream.closed' , { address : this._peer } ) ;
}
}
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
function bytesToBase64Std ( bytes : Uint8Array ) : string {
let bin = '' ;
for ( let i = 0 ; i < bytes . length ; i ++ ) bin += String . fromCharCode ( bytes [ i ] ! ) ;
return btoa ( bin ) ;
}
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
function tryParseMetadata ( json : string ) : import ( '@shade/streams' ) . StreamMetadata | null {
try {
return JSON . parse ( json ) as import ( '@shade/streams' ) . StreamMetadata ;
} catch {
return null ;
}
}
release(v4.0.0): Shade GA — V3.x consolidation + audit prep
V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.
Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
non-realtime stack.
Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
version 4.0.0 on /health.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:35:35 +02:00
/ * *
* Best - effort plaintext - size inference for a ` TransferOptions.input ` .
* Returns null when the size is genuinely unknowable ( raw ` ReadableStream `
* without a metadata hint ) , so the caller can decide whether to gate .
* /
function inferTransferSize ( opts : TransferOptions ) : number | null {
if ( typeof opts . metadata ? . sizeBytes === 'number' ) return opts . metadata . sizeBytes ;
const input = opts . input ;
if ( input instanceof Uint8Array ) return input . byteLength ;
// Blob and File both expose `.size`. Use a structural check so we don't
// depend on lib.dom typings inside the SDK build.
if ( typeof ( input as unknown as { size? : unknown } ) . size === 'number' ) {
return ( input as unknown as { size : number } ) . size ;
}
return null ;
}
/ * *
* Compute the safety - number fingerprint for the identity embedded in a
* decrypted backup payload . Used by ` Shade.importBackup ` to drive the
* ` beforeBackupImport ` gate before any state is overwritten .
* /
async function fingerprintFromBackupPayload (
crypto : SubtleCryptoProvider ,
payload : import ( './backup.js' ) . BackupPayload ,
) : Promise < string > {
if ( payload . identity === null ) {
// No identity in the backup means there's nothing to fingerprint.
// Return a stable sentinel so the gate handler can still display
// something meaningful instead of throwing here.
return 'no-identity-in-backup' ;
}
const id = deserializeIdentityKeyPair ( payload . identity ) ;
return computeFingerprint ( crypto , id . signingPublicKey , id . dhPublicKey ) ;
}
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
function parseChunkHeader ( bytes : Uint8Array ) : {
streamId : string ;
laneId : number ;
seq : bigint ;
} {
// [0]=ver [1]=type [2..18]=streamId(16) [18..22]=laneId u32 [22..30]=seq u64
if ( bytes . length < 30 ) throw new Error ( 'truncated stream-chunk header' ) ;
const view = new DataView ( bytes . buffer , bytes . byteOffset ) ;
const sidBytes = bytes . slice ( 2 , 18 ) ;
const laneId = view . getUint32 ( 18 , false ) ;
const seq = view . getBigUint64 ( 22 , false ) ;
// Encode streamId as base64url
let bin = '' ;
for ( let i = 0 ; i < sidBytes . length ; i ++ ) bin += String . fromCharCode ( sidBytes [ i ] ! ) ;
const streamId = btoa ( bin ) . replace ( /\+/g , '-' ) . replace ( /\//g , '_' ) . replace ( /=+$/ , '' ) ;
return { streamId , laneId , seq } ;
}
2026-05-03 23:27:06 +02:00
// ─── Queue-mode (pull) envelope transport ─────────────────────
/ * *
* Configuration for { @link Shade . transferQueueRoute } . All fields are
* optional with sensible production defaults .
* /
export interface TransferQueueRouteOptions {
/ * *
* Long - poll timeout in milliseconds . Server holds the request open
* up to this long before returning an empty ` events ` array . Default
* 30 _000 .
* /
blockMs? : number ;
/ * *
* Hard cap on long - poll timeout ( clamps client - supplied ` blockMs ` ) .
* Default 55 _000 — under typical reverse - proxy idle thresholds ( 60 s
* on most CDNs ) .
* /
maxBlockMs? : number ;
/ * *
* Per - peer ring - buffer size . When the queue is full , oldest events
* are dropped on enqueue . Receivers detect the gap via missing
* sequence numbers and re - resume from ` since=0 ` . Default 1000 .
* /
maxEventsPerPeer? : number ;
/ * *
* Drop a peer ' s queue + reject pending pollers after this much
* silence . Default 10 minutes . Setting to ` 0 ` disables idle - eviction .
* /
idleEvictionMs? : number ;
}
/ * *
* ` ControlEnvelopeTransport ` that enqueues outbound envelopes into an
* ` OutboundQueue ` for browser - style receivers to long - poll . Mirrors
* ` HttpEnvelopeTransport ` shape ( one ` send(peer, envelope) ` method ) ;
* the difference is the destination — local queue , not remote HTTP .
* /
class QueueEnvelopeTransport implements ControlEnvelopeTransport {
constructor ( private readonly queue : OutboundQueueLike ) { }
async send ( peerAddress : string , envelope : ShadeEnvelope ) : Promise < void > {
const bytes = encodeEnvelope ( envelope ) ;
const event : QueuedEventInput = { kind : 'envelope' , bytes } ;
this . queue . enqueue ( peerAddress , event ) ;
}
}
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
// ─── Default HTTP envelope transport ──────────────────────────
class HttpEnvelopeTransport implements ControlEnvelopeTransport {
constructor (
private readonly resolveBaseUrl : ( peerAddress : string ) = > Promise < string > ,
private readonly myAddress : string ,
) { }
async send ( peerAddress : string , envelope : ShadeEnvelope ) : Promise < void > {
const base = ( await this . resolveBaseUrl ( peerAddress ) ) . replace ( /\/$/ , '' ) ;
const url = ` ${ base } /v1/transfer/control ` ;
const bytes = encodeEnvelope ( envelope ) ;
const res = await fetch ( url , {
method : 'POST' ,
headers : {
'Content-Type' : 'application/octet-stream' ,
'X-Shade-Sender-Address' : this . myAddress ,
} ,
body : bytes as unknown as never ,
} ) ;
if ( ! res . ok ) {
throw new Error ( ` control envelope POST failed: ${ res . status } ${ await res . text ( ) } ` ) ;
}
}
}
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
// ─── Helpers ─────────────────────────────────────────────────
release(v4.3.0): browser persistence via @shade/storage-indexeddb
Ship an official IndexedDB-backed StorageProvider so browser-based Shade
consumers persist identity, prekeys, sessions, retired identities,
peer-verification state and stream-resume rows across tab refresh and
browser restart. Closes the gap that forced browser apps onto
storage:"memory" (regenerated identity each load, orphaned device
records server-side).
- New package @shade/storage-indexeddb (4.3.0): full StorageProvider
conformance, schema v1, idb-backed; bumpPeerIdentityVersion is wrapped
in a single readwrite IDB transaction (atomic, vs SQLite's
read-then-upsert race).
- @shade/sdk resolveStorage() accepts { type: 'indexeddb', dbName? } via
dynamic import (lazy, optional dep — same pattern as
@shade/storage-postgres). Named StorageSpec type now reused by
ResolvedConfig.
- Tests: 16 new tests in shade-storage-indexeddb (StorageProvider
surface + peer-verifications + full E2EE conversation surviving a
simulated tab reload). Run on fake-indexeddb.
- Lockstep version bump 4.2.1 → 4.3.0 across all 25 packages.
- Publish scripts updated to include the new package.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 17:35:02 +02:00
async function resolveStorage ( spec : StorageSpec ) : Promise < StorageProvider > {
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
if ( typeof spec === 'object' && 'getIdentityKeyPair' in spec ) {
return spec ;
}
if ( spec === 'memory' ) {
return new MemoryStorage ( ) ;
}
if ( typeof spec === 'string' && spec . startsWith ( 'sqlite:' ) ) {
const path = spec . slice ( 'sqlite:' . length ) ;
const { SQLiteStorage } = await import ( '@shade/storage-sqlite' ) ;
return new SQLiteStorage ( path ) ;
}
if ( typeof spec === 'object' && spec . type === 'postgres' ) {
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
// Dynamic import keeps @shade/storage-postgres optional — consumers that
// never use postgres don't need to install it. The string-form import
// path makes the resolver lazy at type-check time too.
const moduleId = '@shade/storage-postgres' ;
const mod = ( await import ( moduleId ) ) as {
PostgresStorage : { create ( url : string ) : Promise < StorageProvider > } ;
} ;
return mod . PostgresStorage . create ( spec . url ) ;
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
}
release(v4.3.0): browser persistence via @shade/storage-indexeddb
Ship an official IndexedDB-backed StorageProvider so browser-based Shade
consumers persist identity, prekeys, sessions, retired identities,
peer-verification state and stream-resume rows across tab refresh and
browser restart. Closes the gap that forced browser apps onto
storage:"memory" (regenerated identity each load, orphaned device
records server-side).
- New package @shade/storage-indexeddb (4.3.0): full StorageProvider
conformance, schema v1, idb-backed; bumpPeerIdentityVersion is wrapped
in a single readwrite IDB transaction (atomic, vs SQLite's
read-then-upsert race).
- @shade/sdk resolveStorage() accepts { type: 'indexeddb', dbName? } via
dynamic import (lazy, optional dep — same pattern as
@shade/storage-postgres). Named StorageSpec type now reused by
ResolvedConfig.
- Tests: 16 new tests in shade-storage-indexeddb (StorageProvider
surface + peer-verifications + full E2EE conversation surviving a
simulated tab reload). Run on fake-indexeddb.
- Lockstep version bump 4.2.1 → 4.3.0 across all 25 packages.
- Publish scripts updated to include the new package.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 17:35:02 +02:00
if ( typeof spec === 'object' && spec . type === 'indexeddb' ) {
// Dynamic import keeps @shade/storage-indexeddb optional — Node-only
// consumers don't need to install a browser-only adapter.
const moduleId = '@shade/storage-indexeddb' ;
const mod = ( await import ( moduleId ) ) as {
IndexedDBStorage : { create ( opts : { dbName? : string } ) : Promise < StorageProvider > } ;
} ;
const opts : { dbName? : string } = { } ;
if ( spec . dbName !== undefined ) opts . dbName = spec . dbName ;
return mod . IndexedDBStorage . create ( opts ) ;
}
feat(sdk): M-Magic 1-4 — high-level SDK with magic drop-in
Phase A complete: createShade() one-liner with auto-establish, auto-publish,
and auto-replenish.
M-Magic 1-4 rolled into @shade/sdk:
- createShade() factory with config validation and storage resolution
(memory | sqlite:... | { type: 'postgres', url: ... } | explicit instance)
- Shade class wraps crypto + storage + session manager + transport
- Auto-publish: initialize() automatically registers with the prekey server
- Auto-establish: send() transparently fetches bundles and creates sessions
on first message to a new peer
- Per-address mutex serializes concurrent sends to prevent ratchet corruption
- BackgroundTasks class for periodic replenishment + opt-in identity rotation
- rotate() rebuilds the transport with the new signing key so subsequent
signed operations work after rotation
- onMessage() handler API for incoming plaintext
API:
const shade = await createShade({ prekeyServer, storage });
await shade.send('bob', 'hello');
await shade.receive('alice', envelope);
shade.onMessage((from, msg) => ...);
await shade.rotate();
await shade.shutdown();
13 new SDK tests covering: happy path, auto-publish, two-process
conversation, onMessage handlers, concurrent sends, unknown peer,
fingerprint verification, shutdown, manual replenish, auto-replenish
off, rotate, and config validation.
233 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:27:59 +02:00
throw new Error ( ` Unsupported storage spec: ${ JSON . stringify ( spec ) } ` ) ;
}
async function resolveAddress ( storage : StorageProvider ) : Promise < string > {
// Try to load a persisted address, else generate a random one and save it.
// We reuse the config table by storing a special key.
// Since StorageProvider doesn't expose a generic key-value, we just use
// the local registration ID as a deterministic fallback.
const id = await storage . getLocalRegistrationId ( ) ;
return ` device: ${ id } ` ;
}
function normalize ( fp : string ) : string {
return fp . replace ( /\s+/g , ' ' ) . trim ( ) ;
}