feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
Some checks failed
Test / test (push) Has been cancelled
Some checks failed
Test / test (push) Has been cancelled
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
4
.gitignore
vendored
4
.gitignore
vendored
@@ -2,3 +2,7 @@ node_modules/
|
||||
dist/
|
||||
*.tsbuildinfo
|
||||
.DS_Store
|
||||
|
||||
# Maintainer-only publish-scripts (ikke for offentlig)
|
||||
scripts/publish-all.ts
|
||||
scripts/publish-shade.sh
|
||||
|
||||
184
CHANGELOG.md
184
CHANGELOG.md
@@ -5,6 +5,190 @@ All notable changes to Shade are documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [0.3.0] — 2026-05-02 — Shade Files
|
||||
|
||||
E2EE filesystem RPC primitive — drop-in entrypoints for any consumer that
|
||||
wants to expose a filesystem (or filesystem-like surface) over Shade. Apps
|
||||
keep their own UI; this layer ships the typed RPC, the streams bridge for
|
||||
content I/O over 256 KiB, and production hooks (rate limit, retention,
|
||||
fingerprint gate, metrics).
|
||||
|
||||
### Added
|
||||
|
||||
#### `@shade/files` (NEW)
|
||||
- Standard ops: `list`, `stat`, `mkdir`, `delete`, `move`, `read`, `write`,
|
||||
`getThumbnail` — Zod-validated wire schemas + clean user-handler types.
|
||||
- Custom ops: `client.custom('app.foo', {...})` with full type-safety via
|
||||
TypeScript declaration merging on `CustomOpsMap` + per-op Zod schemas
|
||||
registered server-side.
|
||||
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams (> 256 KiB)
|
||||
ride `@shade/transfer` with automatic correlation via
|
||||
`userMetadata.shadeFilesWriteId` / `shadeFilesReadStreamId`.
|
||||
- Directory ops: `walk(path, opts)` async-iterable depth-first walker;
|
||||
`uploadDirectory()` / `downloadDirectory()` with bounded concurrency
|
||||
pool (default 4, cap 16), aggregated progress events, abort support.
|
||||
- Production hooks (all callback-based, vendor-neutral):
|
||||
- **Rate limit**: token-bucket per sender, op-cost + byte-quota,
|
||||
`FsRateLimitError` / `QuotaExceededError` with `retryAfterMs`.
|
||||
- **Idempotency cache**: per-sender LRU + TTL, in-flight de-dupe,
|
||||
periodic prune via `BackgroundHooks.onPruneFiles`.
|
||||
- **Path policy**: built-in traversal hardening, percent-decode,
|
||||
forbidden-bytes check, root-scope, symlink toggle, `extra` predicate.
|
||||
- **Fingerprint gate**: `requireFingerprintVerifiedFor(ctx)` →
|
||||
`'required' | 'optional' | 'reject'` + `isFingerprintVerified(sender)`.
|
||||
- **Signature verification**: pluggable `verifySender(sender, canonical, sig)`
|
||||
with replay-window enforcement (±5 min `signedAt` skew rejected).
|
||||
- **Metrics**: `onMetric(name, value, tags)` with standard names
|
||||
(`shade_files_op_duration_ms`, `_op_total`, `_bytes_in/out`,
|
||||
`_idempotency_hit/conflict_total`, `_rate_limit_reject_total`,
|
||||
`_fingerprint_reject_total`, `_signature_reject_total`).
|
||||
- React hooks (subpath import `@shade/files/react`):
|
||||
`<ShadeFilesProvider>`, `useShadeFiles`, `useFileList`,
|
||||
`useFileTransfer` / `useFileUpload` / `useFileDownload`. SSR-safe; no UI
|
||||
components — apps bring their own.
|
||||
- High-level entry: `Shade.files.serve(handler)` and `Shade.files.client(peer)`
|
||||
in `@shade/sdk`. Lazy + memoized; one handler per Shade instance.
|
||||
- Drop-in adapter: `createMemoryDirectory()` for tests; structurally
|
||||
compatible with browser `FileSystemDirectoryHandle`.
|
||||
|
||||
#### Wire format bump
|
||||
- `@shade/proto` wire VERSION bumped from `0x01` to `0x02`. Length prefixes
|
||||
changed from u16 to u32 — previous limit was 64 KiB ratchet payloads,
|
||||
which blocked inline file ops up to 256 KiB.
|
||||
**Wire-incompatible with 0.2.x peers.** New sessions only.
|
||||
- Cross-platform Kotlin port (`android/shade-android`) updated to match.
|
||||
|
||||
#### Concurrency safety
|
||||
- `ShadeSessionManager.encrypt` / `.decrypt` now run under per-peer mutex.
|
||||
Previously, concurrent decryptions of the same peer raced ratchet state
|
||||
(manifested as sporadic `Failed to decrypt — wrong key or tampered data`
|
||||
under load). Encrypt was already serialized via `Shade.send`'s
|
||||
`encryptChains`; decrypt is now serialized at the manager layer too.
|
||||
|
||||
#### `@shade/streams` extension
|
||||
- `StreamMetadata` gets optional `userMetadata?: Record<string, string>` —
|
||||
application-level key/value pairs that round-trip verbatim through
|
||||
`stream-init` plaintext. Used by `@shade/files` for write/read correlation
|
||||
but available to any consumer.
|
||||
|
||||
#### `@shade/sdk` extension
|
||||
- `Shade.files` getter (lazy + memoized).
|
||||
- `BackgroundHooks.onPruneFiles?: () => void` + periodic timer (default 5 min)
|
||||
for `@shade/files` retention.
|
||||
- `BackgroundTasks.setHook(name, fn)` for runtime hook registration.
|
||||
|
||||
### Examples
|
||||
- `examples/08-files-browser/` — three-process demo (prekey + Bob server +
|
||||
Alice CLI) covering list/stat/mkdir/delete/upload/download with both
|
||||
inline and streamed paths.
|
||||
|
||||
### Tests
|
||||
- 100+ new tests across `tests/{unit,integration,security}/` in
|
||||
`@shade/files`. End-to-end coverage for streams I/O up to 1 MiB, custom-op
|
||||
registration + Zod validation, fingerprint-gate rejection, replay-window
|
||||
enforcement, idempotent retries, rate-limit + quota enforcement, walk
|
||||
+ bulk transfer aggregated progress.
|
||||
|
||||
## [0.2.0] — 2026-05-01 — Shade Streams
|
||||
|
||||
E2EE chunked upload/download with parallel lanes, resumable transfers, and a
|
||||
"magic drop-in" UX for any Shade-using app. Adds two new packages
|
||||
(`@shade/streams`, `@shade/transfer`) and extends `@shade/sdk` and
|
||||
`@shade/widgets` with high-level transfer APIs.
|
||||
|
||||
### Added
|
||||
|
||||
#### Streams crypto layer (`@shade/streams`)
|
||||
- HKDF stream/lane key derivation (`deriveStreamKey`, `deriveLaneKey`)
|
||||
- Deterministic AES-GCM nonce construction `nonce = laneId(4) || seq(8)`
|
||||
- Streaming SHA-256 via `@noble/hashes/sha2.js` for memory-bounded integrity
|
||||
- `StreamSender` / `StreamReceiver` per-lane state machines with strict
|
||||
in-order seq + replay detection (`StreamReplayError`,
|
||||
`StreamOutOfOrderError`, `StreamDecryptionError`, `StreamProtocolError`)
|
||||
- `MultiLaneSender` / `MultiLaneReceiver` coordinators for parallel transfers
|
||||
- Range and round-robin partitioning helpers (`planRangePartition`,
|
||||
`planRoundRobinPartition`, `chunkRange`)
|
||||
- Wire format: new envelope type `0x11` (stream-chunk) in `@shade/proto`,
|
||||
control envelopes (`stream-init` / `-finish` / `-abort` / `-resume-*`)
|
||||
ride existing `0x02` ratchet messages with JSON `kind` discriminator
|
||||
|
||||
#### Transfer orchestration (`@shade/transfer`)
|
||||
- `TransferEngine` — single class wrapping outgoing + incoming lifecycle
|
||||
- Default `ShadeTransferHttpTransport` for chunk POSTs, opt-in
|
||||
`ShadeTransferWsTransport` with `FallbackTransferTransport` for auto-fallback
|
||||
- `createTransferRoutes()` Hono factory mounts `/v1/transfer/*` routes
|
||||
(`chunk`, `state`, `health`)
|
||||
- `IControlChannel` + `MemoryControlChannel` for in-process testing;
|
||||
the SDK provides `ShadeControlChannel` over `Shade.send`/`receive`
|
||||
- Resume protocol: `MemoryResumeStore`, `StorageBackedResumeStore`,
|
||||
`deriveDeviceKey()` for at-rest streamSecret encryption,
|
||||
`engine.resumeUpload(streamId, freshInput)` for kill-restart-verify flows
|
||||
- `ProgressTracker` with EMA-smoothed throughput + ETA
|
||||
- Retry/backoff (`withRetry`) with exponential delay + jitter
|
||||
- Error hierarchy: `TransferError`, `TransferAbortError`,
|
||||
`TransferIntegrityError`, `TransferProtocolError`, `TransferOfflineError`,
|
||||
`TransferResumeError`, `TransferTransportError`
|
||||
|
||||
#### SDK (`@shade/sdk`)
|
||||
- `Shade.upload(opts)` — high-level entry; encrypts + chunks + ships
|
||||
- `Shade.onIncomingTransfer(handler)` — receiver-side subscription
|
||||
- `Shade.transferRoute()` — Hono router to mount on the consumer's HTTP server
|
||||
- `Shade.acceptTransferEnvelope(from, env)` — low-level entry for custom transports
|
||||
- `Shade.resumeUpload(streamId, freshInput)` — pick up an interrupted transfer
|
||||
- `Shade.listTransfers(filter?)` — list resumable / active transfers from storage
|
||||
- `ShadeTransferAuthenticator` — Ed25519-signing authenticator for HTTP/WS transports
|
||||
- `Shade.onMessage(handler)` now accepts `Promise<void>`-returning handlers
|
||||
(awaited in sequence) — supports flow-control over the control plane
|
||||
|
||||
#### Storage (all backends)
|
||||
- New optional `StorageProvider` methods: `saveStreamState`,
|
||||
`getStreamState`, `removeStreamState`, `listActiveStreamStates`,
|
||||
`pruneStreamStates`. Existing v0.1.x providers compile cleanly (optional methods)
|
||||
- SQLite (`stream_state` table) and Postgres (`shade_stream_state` table)
|
||||
schemas with at-rest encrypted streamSecret
|
||||
- `MemoryStorage` extended with in-memory stream-state map
|
||||
|
||||
#### Widgets (`@shade/widgets`)
|
||||
- `<ShadeRuntimeProvider runtime={shade}>` — separate React context for
|
||||
upload/download widgets (distinct from the observer-dashboard `<ShadeProvider>`)
|
||||
- `useShadeUpload()` / `useShadeDownload()` headless hooks
|
||||
- `<ShadeUploader />` / `<ShadeDownloader />` composite components with
|
||||
render-prop pattern for full UI replacement
|
||||
- Sub-components: `<DropZone />`, `<TransferRow />`, `<ProgressBar />`,
|
||||
`<SpeedReadout />`, `<ETAReadout />`, `<LaneIndicator />`
|
||||
- Theme-token additions for progress, drop zone, and lane indicator colors
|
||||
|
||||
### Security properties
|
||||
|
||||
- Per-chunk AES-256-GCM with deterministic nonce; AAD binds
|
||||
`streamId || laneId || seq || isLast` so any header tamper invalidates AEAD
|
||||
- streamSecret never on the wire in plaintext — shipped via Double Ratchet
|
||||
control envelope; lane keys derived locally and never transmitted
|
||||
- Resume state encrypted at rest with `deviceKey` derived from identity's
|
||||
signing private key (rotation invalidates in-flight resume — by design)
|
||||
- Receiver enforces strict in-order seq per lane (`StreamOutOfOrderError`,
|
||||
`StreamReplayError`); finish-time integrity check verifies per-lane sha256
|
||||
+ overall sha256 over original byte order
|
||||
|
||||
### Tests added (118 new across 47 files; 444 total)
|
||||
|
||||
- Unit: KDF, nonce, AEAD, streaming SHA, sender/receiver, partition
|
||||
- Integration: 1/4/16-lane parity, range vs round-robin parity,
|
||||
Bun.serve loopback at 100 KiB / 1 MiB / 8 MiB, two real Shade instances
|
||||
end-to-end at 64 KiB / 512 KiB / 4 MiB
|
||||
- Resume: kill-restart-verify on 256 KiB with 4 lanes
|
||||
- WS fallback: WS connect failure → transparent HTTP completion
|
||||
- Tamper: bit-flip ciphertext / tag / header field; replay; out-of-order
|
||||
- Wire: 0x11 envelope encode/decode roundtrip + edge cases
|
||||
|
||||
### Backward compatibility
|
||||
|
||||
- `Shade.send`/`receive`/`onMessage`/`fingerprint`/`rotate` unchanged
|
||||
(`onMessage` widened to support async handlers — sync handlers still work)
|
||||
- Existing wire types `0x01` (PreKeyMessage) / `0x02` (RatchetMessage) unchanged
|
||||
- `StorageProvider` interface extension uses optional methods
|
||||
- `@shade/streams` and `@shade/transfer` are new packages; no migration
|
||||
|
||||
## [1.0.0] — 2026-04-10
|
||||
|
||||
### First production release
|
||||
|
||||
@@ -117,6 +117,12 @@ await manager.initialize();
|
||||
| `@shade/sdk` | High-level wrapper with `createShade()` one-liner, auto-publish, auto-establish, auto-replenish |
|
||||
| `@shade/cli` | `shade init` scaffolder + utilities (fingerprint, rotate, peer, dashboard, doctor) |
|
||||
|
||||
## Shade as a modular toolkit
|
||||
|
||||
Shade is split into packages so each project can depend on **only what it needs**—encrypted messaging, file transfer, prekey hosting, or lower-level building blocks. You do not need one giant stack for every use case.
|
||||
|
||||
For a **plain-language map** (which packages to add, what the prekey server does vs your own wiring, and where to start in code), see **[docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md)**.
|
||||
|
||||
## Publishing
|
||||
|
||||
All packages publish to a self-hosted Gitea npm registry on `gt.zyon.no`.
|
||||
@@ -149,6 +155,7 @@ bun run publish:all
|
||||
|
||||
## Documentation
|
||||
|
||||
- [docs/SHADE-BY-SCENARIO.md](./docs/SHADE-BY-SCENARIO.md) — **Modular toolkit**: pick packages by scenario (messages, files, browser, ops)
|
||||
- [SECURITY.md](./SECURITY.md) — Reporting vulnerabilities, security policy
|
||||
- [THREAT-MODEL.md](./THREAT-MODEL.md) — Honest threat model and assumptions
|
||||
- [examples/](./examples/) — Runnable example applications
|
||||
|
||||
@@ -11,10 +11,13 @@ import java.nio.ByteBuffer
|
||||
* Format: [version:1][type:1][payload...]
|
||||
* Types: 0x01 = PreKeyMessage, 0x02 = RatchetMessage
|
||||
* Integers: big-endian
|
||||
* Byte arrays: 2-byte length prefix + data
|
||||
* Byte arrays: 4-byte (u32) length prefix + data (since wire VERSION 0x02).
|
||||
*
|
||||
* VERSION 0x01 used a 2-byte length prefix and was capped at 64 KiB
|
||||
* payloads — incompatible with inline file ops up to 256 KiB.
|
||||
*/
|
||||
object WireFormat {
|
||||
private const val VERSION: Byte = 0x01
|
||||
private const val VERSION: Byte = 0x02
|
||||
private const val TYPE_PREKEY: Byte = 0x01
|
||||
private const val TYPE_RATCHET: Byte = 0x02
|
||||
private const val PREKEY_NONE: Long = 0xFFFFFFFFL
|
||||
@@ -138,8 +141,8 @@ object WireFormat {
|
||||
}
|
||||
|
||||
private fun lpBytes(data: ByteArray): ByteArray {
|
||||
val len = ByteBuffer.allocate(2)
|
||||
len.putShort(data.size.toShort())
|
||||
val len = ByteBuffer.allocate(4)
|
||||
len.putInt(data.size)
|
||||
return concat(listOf(len.array(), data))
|
||||
}
|
||||
|
||||
@@ -151,9 +154,9 @@ object WireFormat {
|
||||
}
|
||||
|
||||
private fun readLP(data: ByteArray, offset: Int): Pair<ByteArray, Int> {
|
||||
val len = ((data[offset].toInt() and 0xff) shl 8) or (data[offset + 1].toInt() and 0xff)
|
||||
val value = data.copyOfRange(offset + 2, offset + 2 + len)
|
||||
return value to (offset + 2 + len)
|
||||
val len = readUint32(data, offset).toInt()
|
||||
val value = data.copyOfRange(offset + 4, offset + 4 + len)
|
||||
return value to (offset + 4 + len)
|
||||
}
|
||||
|
||||
private fun concat(parts: List<ByteArray>): ByteArray {
|
||||
|
||||
107
bun.lock
107
bun.lock
@@ -8,14 +8,16 @@
|
||||
"@noble/curves": "^2.0.1",
|
||||
"@noble/hashes": "^2.0.1",
|
||||
"hono": "^4.12.12",
|
||||
"zod": "^3.23.8",
|
||||
},
|
||||
"devDependencies": {
|
||||
"bun-types": "^1.3.11",
|
||||
"fast-check": "^3.22.0",
|
||||
},
|
||||
},
|
||||
"packages/shade-cli": {
|
||||
"name": "@shade/cli",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"bin": {
|
||||
"shade": "src/cli.ts",
|
||||
},
|
||||
@@ -32,7 +34,7 @@
|
||||
},
|
||||
"packages/shade-core": {
|
||||
"name": "@shade/core",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"devDependencies": {
|
||||
"@shade/proto": "workspace:*",
|
||||
},
|
||||
@@ -42,7 +44,7 @@
|
||||
},
|
||||
"packages/shade-crypto-web": {
|
||||
"name": "@shade/crypto-web",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"dependencies": {
|
||||
"@noble/curves": "^2.0.1",
|
||||
"@noble/hashes": "^2.0.1",
|
||||
@@ -51,7 +53,7 @@
|
||||
},
|
||||
"packages/shade-dashboard": {
|
||||
"name": "@shade/dashboard",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"dependencies": {
|
||||
"@shade/widgets": "workspace:*",
|
||||
"react": "^19.0.0",
|
||||
@@ -64,9 +66,35 @@
|
||||
"vite": "^6.0.0",
|
||||
},
|
||||
},
|
||||
"packages/shade-files": {
|
||||
"name": "@shade/files",
|
||||
"version": "0.3.0",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
"@shade/proto": "workspace:*",
|
||||
"@shade/sdk": "workspace:*",
|
||||
"@shade/streams": "workspace:*",
|
||||
"@shade/transfer": "workspace:*",
|
||||
"zod": "^3.23.8",
|
||||
},
|
||||
"devDependencies": {
|
||||
"@shade/server": "workspace:*",
|
||||
"@types/react": "^19.2.14",
|
||||
"fast-check": "^3.22.0",
|
||||
"happy-dom": "^15.11.7",
|
||||
"react": "^19.2.5",
|
||||
},
|
||||
"peerDependencies": {
|
||||
"react": "^18.0.0 || ^19.0.0",
|
||||
},
|
||||
"optionalPeers": [
|
||||
"react",
|
||||
],
|
||||
},
|
||||
"packages/shade-observer": {
|
||||
"name": "@shade/observer",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/server": "workspace:*",
|
||||
@@ -78,27 +106,30 @@
|
||||
},
|
||||
"packages/shade-proto": {
|
||||
"name": "@shade/proto",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
},
|
||||
},
|
||||
"packages/shade-sdk": {
|
||||
"name": "@shade/sdk",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
"@shade/files": "workspace:*",
|
||||
"@shade/observer": "workspace:*",
|
||||
"@shade/proto": "workspace:*",
|
||||
"@shade/server": "workspace:*",
|
||||
"@shade/storage-sqlite": "workspace:*",
|
||||
"@shade/streams": "workspace:*",
|
||||
"@shade/transfer": "workspace:*",
|
||||
"@shade/transport": "workspace:*",
|
||||
},
|
||||
},
|
||||
"packages/shade-server": {
|
||||
"name": "@shade/server",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"hono": "^4.12.12",
|
||||
@@ -115,7 +146,7 @@
|
||||
},
|
||||
"packages/shade-storage-postgres": {
|
||||
"name": "@shade/storage-postgres",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/server": "workspace:*",
|
||||
@@ -128,16 +159,42 @@
|
||||
},
|
||||
"packages/shade-storage-sqlite": {
|
||||
"name": "@shade/storage-sqlite",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
"@shade/server": "workspace:*",
|
||||
},
|
||||
},
|
||||
"packages/shade-streams": {
|
||||
"name": "@shade/streams",
|
||||
"version": "0.3.0",
|
||||
"dependencies": {
|
||||
"@noble/hashes": "^2.0.1",
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
"@shade/proto": "workspace:*",
|
||||
},
|
||||
},
|
||||
"packages/shade-transfer": {
|
||||
"name": "@shade/transfer",
|
||||
"version": "0.3.0",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
"@shade/proto": "workspace:*",
|
||||
"@shade/streams": "workspace:*",
|
||||
},
|
||||
"peerDependencies": {
|
||||
"hono": "^4",
|
||||
},
|
||||
"optionalPeers": [
|
||||
"hono",
|
||||
],
|
||||
},
|
||||
"packages/shade-transport": {
|
||||
"name": "@shade/transport",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
@@ -147,10 +204,16 @@
|
||||
},
|
||||
"packages/shade-widgets": {
|
||||
"name": "@shade/widgets",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"dependencies": {
|
||||
"@shade/sdk": "workspace:*",
|
||||
"@shade/streams": "workspace:*",
|
||||
"@shade/transfer": "workspace:*",
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/react": "^19.2.14",
|
||||
"@types/react-dom": "^19.2.3",
|
||||
"happy-dom": "^15.11.7",
|
||||
"react": "^19.2.5",
|
||||
"react-dom": "^19.2.5",
|
||||
},
|
||||
@@ -325,6 +388,8 @@
|
||||
|
||||
"@shade/dashboard": ["@shade/dashboard@workspace:packages/shade-dashboard"],
|
||||
|
||||
"@shade/files": ["@shade/files@workspace:packages/shade-files"],
|
||||
|
||||
"@shade/observer": ["@shade/observer@workspace:packages/shade-observer"],
|
||||
|
||||
"@shade/proto": ["@shade/proto@workspace:packages/shade-proto"],
|
||||
@@ -337,6 +402,10 @@
|
||||
|
||||
"@shade/storage-sqlite": ["@shade/storage-sqlite@workspace:packages/shade-storage-sqlite"],
|
||||
|
||||
"@shade/streams": ["@shade/streams@workspace:packages/shade-streams"],
|
||||
|
||||
"@shade/transfer": ["@shade/transfer@workspace:packages/shade-transfer"],
|
||||
|
||||
"@shade/transport": ["@shade/transport@workspace:packages/shade-transport"],
|
||||
|
||||
"@shade/widgets": ["@shade/widgets@workspace:packages/shade-widgets"],
|
||||
@@ -377,16 +446,22 @@
|
||||
|
||||
"electron-to-chromium": ["electron-to-chromium@1.5.334", "", {}, "sha512-mgjZAz7Jyx1SRCwEpy9wefDS7GvNPazLthHg8eQMJ76wBdGQQDW33TCrUTvQ4wzpmOrv2zrFoD3oNufMdyMpog=="],
|
||||
|
||||
"entities": ["entities@4.5.0", "", {}, "sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw=="],
|
||||
|
||||
"esbuild": ["esbuild@0.25.12", "", { "optionalDependencies": { "@esbuild/aix-ppc64": "0.25.12", "@esbuild/android-arm": "0.25.12", "@esbuild/android-arm64": "0.25.12", "@esbuild/android-x64": "0.25.12", "@esbuild/darwin-arm64": "0.25.12", "@esbuild/darwin-x64": "0.25.12", "@esbuild/freebsd-arm64": "0.25.12", "@esbuild/freebsd-x64": "0.25.12", "@esbuild/linux-arm": "0.25.12", "@esbuild/linux-arm64": "0.25.12", "@esbuild/linux-ia32": "0.25.12", "@esbuild/linux-loong64": "0.25.12", "@esbuild/linux-mips64el": "0.25.12", "@esbuild/linux-ppc64": "0.25.12", "@esbuild/linux-riscv64": "0.25.12", "@esbuild/linux-s390x": "0.25.12", "@esbuild/linux-x64": "0.25.12", "@esbuild/netbsd-arm64": "0.25.12", "@esbuild/netbsd-x64": "0.25.12", "@esbuild/openbsd-arm64": "0.25.12", "@esbuild/openbsd-x64": "0.25.12", "@esbuild/openharmony-arm64": "0.25.12", "@esbuild/sunos-x64": "0.25.12", "@esbuild/win32-arm64": "0.25.12", "@esbuild/win32-ia32": "0.25.12", "@esbuild/win32-x64": "0.25.12" }, "bin": { "esbuild": "bin/esbuild" } }, "sha512-bbPBYYrtZbkt6Os6FiTLCTFxvq4tt3JKall1vRwshA3fdVztsLAatFaZobhkBC8/BrPetoa0oksYoKXoG4ryJg=="],
|
||||
|
||||
"escalade": ["escalade@3.2.0", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="],
|
||||
|
||||
"fast-check": ["fast-check@3.23.2", "", { "dependencies": { "pure-rand": "^6.1.0" } }, "sha512-h5+1OzzfCC3Ef7VbtKdcv7zsstUQwUDlYpUTvjeUsJAssPgLn7QzbboPtL5ro04Mq0rPOsMzl7q5hIbRs2wD1A=="],
|
||||
|
||||
"fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="],
|
||||
|
||||
"fsevents": ["fsevents@2.3.3", "", { "os": "darwin" }, "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw=="],
|
||||
|
||||
"gensync": ["gensync@1.0.0-beta.2", "", {}, "sha512-3hN7NaskYvMDLQY55gnW3NQ+mesEAepTqlg+VEbj7zzqEMBVNhzcGYYeqFo/TlYz6eQiFcp1HcsCZO+nGgS8zg=="],
|
||||
|
||||
"happy-dom": ["happy-dom@15.11.7", "", { "dependencies": { "entities": "^4.5.0", "webidl-conversions": "^7.0.0", "whatwg-mimetype": "^3.0.0" } }, "sha512-KyrFvnl+J9US63TEzwoiJOQzZBJY7KgBushJA8X61DMbNsH+2ONkDuLDnCnwUiPTF42tLoEmrPyoqbenVA5zrg=="],
|
||||
|
||||
"hono": ["hono@4.12.12", "", {}, "sha512-p1JfQMKaceuCbpJKAPKVqyqviZdS0eUxH9v82oWo1kb9xjQ5wA6iP3FNVAPDFlz5/p7d45lO+BpSk1tuSZMF4Q=="],
|
||||
|
||||
"js-tokens": ["js-tokens@4.0.0", "", {}, "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ=="],
|
||||
@@ -411,6 +486,8 @@
|
||||
|
||||
"postgres": ["postgres@3.4.9", "", {}, "sha512-GD3qdB0x1z9xgFI6cdRD6xu2Sp2WCOEoe3mtnyB5Ee0XrrL5Pe+e4CCnJrRMnL1zYtRDZmQQVbvOttLnKDLnaw=="],
|
||||
|
||||
"pure-rand": ["pure-rand@6.1.0", "", {}, "sha512-bVWawvoZoBYpp6yIoQtQXHZjmz35RSVHnUOTefl8Vcjr8snTPY1wnpSPMWekcFwbxI6gtmT7rSYPFvz71ldiOA=="],
|
||||
|
||||
"react": ["react@19.2.5", "", {}, "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA=="],
|
||||
|
||||
"react-dom": ["react-dom@19.2.5", "", { "dependencies": { "scheduler": "^0.27.0" }, "peerDependencies": { "react": "^19.2.5" } }, "sha512-J5bAZz+DXMMwW/wV3xzKke59Af6CHY7G4uYLN1OvBcKEsWOs4pQExj86BBKamxl/Ik5bx9whOrvBlSDfWzgSag=="],
|
||||
@@ -433,6 +510,12 @@
|
||||
|
||||
"vite": ["vite@6.4.2", "", { "dependencies": { "esbuild": "^0.25.0", "fdir": "^6.4.4", "picomatch": "^4.0.2", "postcss": "^8.5.3", "rollup": "^4.34.9", "tinyglobby": "^0.2.13" }, "optionalDependencies": { "fsevents": "~2.3.3" }, "peerDependencies": { "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", "jiti": ">=1.21.0", "less": "*", "lightningcss": "^1.21.0", "sass": "*", "sass-embedded": "*", "stylus": "*", "sugarss": "*", "terser": "^5.16.0", "tsx": "^4.8.1", "yaml": "^2.4.2" }, "optionalPeers": ["@types/node", "jiti", "less", "lightningcss", "sass", "sass-embedded", "stylus", "sugarss", "terser", "tsx", "yaml"], "bin": { "vite": "bin/vite.js" } }, "sha512-2N/55r4JDJ4gdrCvGgINMy+HH3iRpNIz8K6SFwVsA+JbQScLiC+clmAxBgwiSPgcG9U15QmvqCGWzMbqda5zGQ=="],
|
||||
|
||||
"webidl-conversions": ["webidl-conversions@7.0.0", "", {}, "sha512-VwddBukDzu71offAQR975unBIGqfKZpM+8ZX6ySk8nYhVoo5CYaZyzt3YBvYtRtO+aoGlqxPg/B87NGVZ/fu6g=="],
|
||||
|
||||
"whatwg-mimetype": ["whatwg-mimetype@3.0.0", "", {}, "sha512-nt+N2dzIutVRxARx1nghPKGv1xHikU7HKdfafKkLNLindmPU/ch3U31NOCGGA/dmPcmb1VlofO0vnKAcsm0o/Q=="],
|
||||
|
||||
"yallist": ["yallist@3.1.1", "", {}, "sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g=="],
|
||||
|
||||
"zod": ["zod@3.25.76", "", {}, "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ=="],
|
||||
}
|
||||
}
|
||||
|
||||
63
docs/SHADE-BY-SCENARIO.md
Normal file
63
docs/SHADE-BY-SCENARIO.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# Shade by scenario — modular E2EE toolkit
|
||||
|
||||
This page is for **builders**, not cryptography specialists. Shade is packaged so you can **drop in small pieces** per project instead of importing a heavy “everything” stack.
|
||||
|
||||
---
|
||||
|
||||
## Plain-language mental model
|
||||
|
||||
1. **Identity & session (the hard crypto):** Shade establishes a **secret channel** between two parties using the same kind of cryptographic core as Signal (initial setup + ongoing “ratchet” updates). **You mostly call high-level APIs** (`send`, `receive`, fingerprints) rather than assembling primitives by hand.
|
||||
|
||||
2. **Who is “the server”?**
|
||||
The **prekey server** only helps with **public key material** (so Alice can fetch Bob’s public bundle before the first message). It is **not** your general-purpose message relay unless **you** build that separately. Normal **message payloads and file chunks** typically flow over **your** transport (your HTTP routes, websocket, bridge, queue, etc.).
|
||||
|
||||
3. **Small vs large payloads:**
|
||||
Short messages ride the usual ratchet envelopes. Very large payloads use **Streams + Transfer**: secrets are negotiated over the ratchet; ** ciphertext chunks** ship over optimized HTTP/WebSocket transports with parallelism and resume.
|
||||
|
||||
4. **Trust:** Strong encryption does **not** replace **verifying who you are talking to**. For high-stakes use, compare **safety numbers** out-of-band (see [THREAT-MODEL.md](../THREAT-MODEL.md)).
|
||||
|
||||
---
|
||||
|
||||
## Scenario → minimum packages
|
||||
|
||||
Pull in **one row** that matches your project; add optional columns only when needed.
|
||||
|
||||
| Scenario | What you need | Minimum packages / surface | Where to start |
|
||||
|----------|----------------|----------------------------|----------------|
|
||||
| **Backend or Bun service** — encrypted messages between users | Session storage + crypto provider + prekey URL | `@shade/sdk` + `sqlite:` or `@shade/storage-postgres` | `createShade()` → `send` / `receive` |
|
||||
| **Browser / frontend** — same, in the client | Web crypto + durable or memory storage | `@shade/sdk` or `@shade/core` + `@shade/crypto-web` (+ storage you provide) | Same APIs; ensure `prekeyServer` is reachable from the browser (CORS, etc.) |
|
||||
| **Large files** — resumable E2EE upload/download | Above + stream protocol + HTTP (or WS) transport | `@shade/sdk` (re-exports transfer) + mount transfer routes on **your** HTTP server | `shade.upload` / `onIncomingTransfer` — see [streams.md](./streams.md) |
|
||||
| **React UI** — upload/download widgets | Runtime from SDK + widgets | `@shade/sdk` + `@shade/widgets` | `ShadeRuntimeProvider`, `useShadeUpload` / `useShadeDownload` |
|
||||
| **Prekey hosting only** — one container per product | No app crypto in the container | Docker image / `@shade/server` | Deploy prekey image; point `prekeyServer` at it from apps |
|
||||
| **Maximum control** — custom wire, custom transport | Wire + session manager | `@shade/core` + `@shade/proto` (+ your storage + crypto provider) | `ShadeSessionManager`, encode/decode envelopes yourself |
|
||||
| **HTTP or WebSocket convenience** | Auto-wrap application bytes | `@shade/transport` on top of your stack | Use when you want transport helpers, not a new protocol |
|
||||
| **Android** | Byte-compatible with TS (roadmap) | `shade-android` module | See [android/shade-android/README.md](../android/shade-android/README.md) — parity work in progress |
|
||||
|
||||
You can **mix rows**: e.g. backend with `@shade/sdk` + SQLite for sessions, separate service mounting `transfer` routes, browser clients using `@shade/widgets`.
|
||||
|
||||
---
|
||||
|
||||
## New project checklist (lightweight)
|
||||
|
||||
1. Run a **prekey server** (Docker or embedded `@shade/server`) for your environment.
|
||||
2. Pick **storage** (`sqlite:…`, Postgres, or project-specific adapter implementing the core storage interfaces).
|
||||
3. Choose **surface**: usually `@shade/sdk` unless you truly need `@shade/core` only.
|
||||
4. For files: enable **transfer routes** and authenticate chunk uploads using the patterns in the SDK (see streams doc).
|
||||
5. Run **`shade doctor`** when something fails in production-ish setups (install the CLI as in repository [Quick start](../README.md#quick-start)); coverage is evolving — roadmap in [V2.2](./V2.2.md).
|
||||
|
||||
---
|
||||
|
||||
## Related docs & roadmap
|
||||
|
||||
| Topic | Doc |
|
||||
|--------|-----|
|
||||
| File transfer architecture | [streams.md](./streams.md) |
|
||||
| Deployment & operations | [DEPLOYMENT.md](./DEPLOYMENT.md) |
|
||||
| Threat model | [THREAT-MODEL.md](../THREAT-MODEL.md) |
|
||||
| Planned improvements | [V2.1](./V2.1.md), feature backlog [V2.2](./V2.2.md), trust/ops [V2.3](./V2.3.md) |
|
||||
|
||||
---
|
||||
|
||||
## Version note
|
||||
|
||||
This file describes how Shade is **intended** to be composed. Package names and re-exports may gain small aliases over time; the **scenario table** should remain the source of truth for “what to install for what job.” Update this page when adding major surfaces (new transport bridges, richer `shade init` templates, etc.).
|
||||
151
docs/V2.1.md
Normal file
151
docs/V2.1.md
Normal file
@@ -0,0 +1,151 @@
|
||||
# Shade V2.1 — Improvements (infrastructure, storage, operations, security)
|
||||
|
||||
This document describes **improvements** agreed for next-generation work on Shade: clearer product story, stronger storage, mobile parity, operational hardening, transfer abuse, and a formal security narrative.
|
||||
|
||||
**Audience:** **Maintainers and contributors** implementing the changes. Add status fields as items land in code/docs.
|
||||
|
||||
---
|
||||
|
||||
## 1. Clear “who is the server?” and data flow
|
||||
|
||||
**Problem:** New users may think the prekey server is a message hub or that all E2EE traffic goes through the Shade container.
|
||||
|
||||
**Goal:** One consistent explanation across the root README, package READMEs, and optional onboarding: **the prekey server distributes public keys and bundles**; **actual messages and (typically) file chunks go through your app’s own channel** (your transport, your backend, your URLs).
|
||||
|
||||
**Deliverables (proposal):**
|
||||
|
||||
- Diagram + short “keys vs payloads” text in the root README and in `@shade/server` README.
|
||||
- Link to `THREAT-MODEL.md` from the same section (MITM on first contact ↔ safety numbers).
|
||||
- Optionally one “concept page” (or extend `MIGRATION.md`) with typical architecture: *A ↔ B via app; both talk to the prekey host for X3DH material*.
|
||||
|
||||
**Acceptance criteria:** A new developer without domain background understands in one reading *what* goes to the Shade server and *what* does not.
|
||||
|
||||
---
|
||||
|
||||
## 2. Optional encryption of storage (at-rest)
|
||||
|
||||
**Problem:** `THREAT-MODEL.md` states that a stolen DB + filesystem can expose private keys because Shade does not encrypt the storage layer by default.
|
||||
|
||||
**Goal:** **Opt-in** protection for sensitive state (identity, session, optional stream resume secrets) with keys that **do not** live in plaintext in the DB — e.g. OS keychain/Keystore, passphrase + KDF, or an explicit device key injected by the app.
|
||||
|
||||
**Design principles:**
|
||||
|
||||
- Default developer experience (dev, simple demos) stays unchanged or includes a clear “insecure mode” warning in docs.
|
||||
- APIs implementable per platform (Bun/SQLite, Postgres, web/IndexedDB, Android).
|
||||
- Document limitations: what remains uncovered (e.g. active memory compromise).
|
||||
|
||||
**Acceptance criteria:** Threat model updated for “when encrypted storage is enabled”; at least one reference implementation + migration note.
|
||||
|
||||
---
|
||||
|
||||
## 3. Android parity and a published roadmap
|
||||
|
||||
**Problem:** `shade-android` is under development; drift from the TS SDK undermines the “byte-compatible” promise.
|
||||
|
||||
**Goal:** A **published roadmap** (milestones + what counts as parity vs TS-only) and **CI running shared test vectors** as a merge gate before release.
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
- Roadmap section in `android/shade-android/README.md` or dedicated `ROADMAP-ANDROID.md` with explicit cross-checkpoints: wire format, fingerprints, rotations, streams (`0x11`) where applicable, resume semantics.
|
||||
- CI job that fails on Kotlin vs TS vector mismatch.
|
||||
|
||||
**Acceptance criteria:** Parity coverage is visible and enforceable; the first critical cross-surface (e.g. core ratchet + proto) is green before a “production” label.
|
||||
|
||||
---
|
||||
|
||||
## 4. Operational hardening — prekey container and production
|
||||
|
||||
**Problem:** Many teams deploy the Docker image quickly; mistakes around TLS, backups, and secrets add avoidable risk.
|
||||
|
||||
**Goal:** A **production checklist**: TLS termination, volume backup (`/data`), rotation of `SHADE_OBSERVER_TOKEN`, use of `SHADE_PREKEY_PG_URL` vs SQLite, observability hooks, logging levels, meaning of stale cleanup parameters.
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
- Extend `docs/DEPLOYMENT.md` or add short `docs/PRODUCTION-CHECKLIST.md` with bullet defaults.
|
||||
- Link from the main README under “Deployment”.
|
||||
|
||||
**Acceptance criteria:** A checklist operators can follow without reading the whole codebase first.
|
||||
|
||||
---
|
||||
|
||||
## 5. Abuse and resource limits on the transfer plane
|
||||
|
||||
**Problem:** Parallel lanes and large uploads can be abused for resource or storage if consumer mounts of `createTransferRoutes()` share no coherent policy.
|
||||
|
||||
**Goal:** Documented **limits and patterns**: authentication (already an active SDK topic), max stream size, TTL for temporary chunk storage, quotas per identity or IP where sensible.
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
- Guidelines in `docs/streams.md` or a dedicated “Transfer hardening” section.
|
||||
- Optional helpers or middleware examples in `@shade/transfer` / server routes for common limits (without forcing every deployment into one DB model).
|
||||
|
||||
**Acceptance criteria:** A clear “recommended minimum” for production that teams can copy.
|
||||
|
||||
---
|
||||
|
||||
## 6. Security review and formal test / narrative
|
||||
|
||||
**Problem:** Enterprises and security-conscious users often ask for independent review and a traceable test matrix.
|
||||
|
||||
**Goal:** Plan for **independent crypto review** (timing, scope, deliverables) and a **published test / threat matrix** linking `THREAT-MODEL.md` to concrete automated tests (replay, tamper, out-of-order, resume, etc.).
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
- Internal checklist “preparing for external review” (which files, assumptions, known limits).
|
||||
- Short section in `SECURITY.md` on review status and how to report findings.
|
||||
|
||||
**Acceptance criteria:** One authoritative source for “what is tested automatically” vs “what needs manual/MITM/out-of-band process”.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies and order (proposal)
|
||||
|
||||
| Priority | Topic | Note |
|
||||
|---------|-------|------|
|
||||
| Low friction | §1, §4 | Documentation and checklists only |
|
||||
| Medium | §5 | Design + docs + possibly small API helpers |
|
||||
| High effort | §2, §3 | Platform work and long-term maintenance |
|
||||
| Coordinated | §6 | Parallel with maturity in §2–§5 |
|
||||
|
||||
---
|
||||
|
||||
## Document versioning
|
||||
|
||||
- **V2.1** — first consolidated plan for the six improvement areas above. Update with dates and short per-section status lines as work starts and completes.
|
||||
|
||||
Gitt der du er nå (post-streams 0.2.0), ville jeg prioritert:
|
||||
|
||||
1. Sosial nøkkel-recovery — løser "hva skjer hvis jeg mister telefonen?" som er det store hullet i alle
|
||||
E2EE-systemer
|
||||
2. P2P WebRTC transport — naturlig kompletterende til streams; gir massiv throughput-gevinst gratis
|
||||
3. E2EE Pub/Sub — lavt scope, høy nytte, og passer perfekt med din "magic drop-in"-filosofi
|
||||
|
||||
Disse tre har alle samme egenskap: de er infrastruktur-features som hver app du bygger fremover vil dra nytte av,
|
||||
ikke bare Dispatch. Helt i tråd med "Shade som komponent som dras inn i hvilket som helst prosjekt".
|
||||
|
||||
|
||||
/btw Hva med Shade videostreaming, VOIP streaming, og lignende?
|
||||
|
||||
Magic drop-in-tilnærming (din filosofi)
|
||||
|
||||
Tenk deg at i en Dispatch-app kan du bare gjøre:
|
||||
|
||||
<ShadeVideoCall to="device:server-admin" />
|
||||
<ShadeVoiceButton to={peerAddress} />
|
||||
<ShadeBroadcaster streamKey="game-stream-1" />
|
||||
<ShadeViewer streamKey="game-stream-1" />
|
||||
|
||||
…og det bare fungerer, E2EE, P2P når mulig, fallback til relay automatisk. Det er det samme mønsteret som streams
|
||||
0.2.0, bare for sanntid.
|
||||
|
||||
Realisme-sjekk
|
||||
|
||||
Video/VOIP er det vanskeligste i hele E2EE-verdenen. Signal brukte år på å få det riktig. Du bør:
|
||||
1. Ferdigstille streams 0.2.0 først (verifiserer crypto-fundamentet)
|
||||
2. Bygge P2P WebRTC-transport som separat milestone
|
||||
3. Da har du alle byggeklossene og Voice 0.4.0 blir 70% gjenbruk
|
||||
|
||||
Men ja — dette hører absolutt hjemme i Shade. Shade som "alt-i-ett E2EE-platform" er en mye sterkere posisjon enn
|
||||
"bare messaging + filer". Du kan bli til E2EE hva Twilio er til vanlig kommunikasjon.
|
||||
|
||||
|
||||
126
docs/V2.2.md
Normal file
126
docs/V2.2.md
Normal file
@@ -0,0 +1,126 @@
|
||||
# Shade V2.2 — Feature plan: product, platform, and developer experience
|
||||
|
||||
This document gathers **planned features** that extend Shade beyond today’s core (X3DH + Double Ratchet + Streams/transfer): groups, asynchronous delivery, richer file UX, web workers, CLI, API docs, and scaffolding.
|
||||
|
||||
Add optional per-feature status (Idea / Design / IMP / Done).
|
||||
|
||||
---
|
||||
|
||||
## 1. Groups / multiple participants
|
||||
|
||||
**Vision:** Beyond strict 1:1 — multiple identities in the same “conversation” or shared context (messages and possibly shared file/stream policy).
|
||||
|
||||
**Challenges:**
|
||||
|
||||
- Today’s Signal-like core is naturally 1:1; groups need either **pairwise sessions per member**, **sender keys / fan-out**, or a **dedicated group protocol** (larger architectural step).
|
||||
- Lifecycle: invites, member removal, compromised device, history, and group PCS.
|
||||
|
||||
**Goals for early milestones (proposal):**
|
||||
|
||||
1. **Document** a recommended pattern for “group-lite” (e.g. coordinator relays ciphertext without decrypting + clients encrypt per-recipient).
|
||||
2. **Optional** minimal API making fan-out easier in the SDK (without promising full MLS).
|
||||
|
||||
**Acceptance criteria (MVP definition):** Explicit scope in docs + one reference architecture; no ambiguous “group is done” without an updated threat model.
|
||||
|
||||
---
|
||||
|
||||
## 2. Async store-and-forward messaging
|
||||
|
||||
**Vision:** Recipient need not be online; **ciphertext** is stored temporarily and fetched when the recipient returns — the server never sees plaintext.
|
||||
|
||||
**Distinction from the prekey server:**
|
||||
|
||||
- Prekey stays **public keys only** (or extended only under strict policy).
|
||||
- A **dedicated relay/inbox service** (or app-owned backend) holds **encrypted blobs only** with TTL, idempotency, and authorization (who may list/fetch).
|
||||
|
||||
**Deliverables (proposal):**
|
||||
|
||||
- Protocol sketch: address registration, `PUT` blob, `GET`/`DELETE` or lease, replay protection at the application layer.
|
||||
- SDK helpers: outgoing queue, poll/pull, or push-notification hook (without dictating mobile platform).
|
||||
|
||||
**Acceptance criteria:** Threat-model section “what the relay sees” + reference implementation or example app.
|
||||
|
||||
---
|
||||
|
||||
## 3. File metadata and preview
|
||||
|
||||
**Vision:** Richer UX **without** leaking sensitive content to the server: filename, MIME type, length where known; **optional** client-generated thumbnails or previews encrypted as separate blocks or small payloads on the control init path.
|
||||
|
||||
**Technical considerations:**
|
||||
|
||||
- Anything sent must be **E2EE** or omitted; plaintext metadata on the server must be deliberate and minimal.
|
||||
- Thumbnails should use **format hardening** on the client (size limits, sandboxing in UI).
|
||||
|
||||
**Acceptance criteria:** Extended `stream-init` (or sidecar envelope) with optional fields + widget support for “preview when available”.
|
||||
|
||||
---
|
||||
|
||||
## 4. Web: worker-based crypto and streaming
|
||||
|
||||
**Vision:** Large files in the browser without blocking the main thread or blowing RAM — **Web Crypto / noble** inside a **Worker**, **ReadableStream/WritableStream** end-to-end chunk pipeline aligned with `@shade/streams` / transfer.
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
- `@shade/crypto-web` (or companion) patterns: transferable buffers, lifecycle, errors surfaced to the UI.
|
||||
- Documented constraints (Safari, chunk sizing, Service Worker vs dedicated worker).
|
||||
|
||||
**Acceptance criteria:** E2E demo or test that sends multi‑MiB through a worker without a blocking UI.
|
||||
|
||||
---
|
||||
|
||||
## 5. CLI: `shade doctor`
|
||||
|
||||
**Vision:** One command that **diagnoses the environment** before production debugging.
|
||||
|
||||
**Typical checks (proposal):**
|
||||
|
||||
- Reachability of `prekeyServer` (`/health`, optional OpenAPI).
|
||||
- Local config: storage path, rotation headers, clock skew (relevant for signed requests).
|
||||
- **Streams:** transfer routes mounted, auth matches expected key, `GET .../state` behaves as expected in test mode.
|
||||
- CLI / Node/Bun runtime versions and `@shade/*` packages where readable from `package.json`.
|
||||
|
||||
**Acceptance criteria:** `shade doctor` with exit codes suitable for CI (warn vs fail).
|
||||
|
||||
---
|
||||
|
||||
## 6. OpenAPI / docs
|
||||
|
||||
**Vision:** All HTTP contracts teams are expected to implement (prekey **and** transfer) appear in **one** OpenAPI story or clearly linked specs — not only README examples.
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
- Consolidate or cross-reference `openapi.yaml` with transfer endpoints (`/v1/transfer/*`) and security schemes for chunk upload.
|
||||
- `/docs` (Redoc or similar) or published static artifacts for versioned specs.
|
||||
|
||||
**Acceptance criteria:** Generated client (e.g. Python/Go) from spec without manual fixes for the happy path.
|
||||
|
||||
---
|
||||
|
||||
## 7. `shade init`
|
||||
|
||||
**Vision:** Scaffolding from **empty repo** to a **minimal runnable** app with Shade and optional streams.
|
||||
|
||||
**Extensions:**
|
||||
|
||||
- New or extended template: **minimal Hono/Fastify** app with `Shade.transferRoute()` mounted, auth example matching the SDK authenticator, `.env` template.
|
||||
- Optional: demo `shade doctor` after init.
|
||||
|
||||
**Acceptance criteria:** `shade init …` produces a project that `bun install && bun run start` runs with documented env vars.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies between items
|
||||
|
||||
```text
|
||||
shade init ─────► doctor (same conventions for URLs and secrets)
|
||||
openapi/docs ◄── transfer + prekey (single source)
|
||||
web workers ───► streams UX (large file in browser)
|
||||
groups ◄──────── store-and-forward (often related socially/technically)
|
||||
metadata/preview► widgets + proto/control plane
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Document versioning
|
||||
|
||||
- **V2.2** — feature backlog as described. Split into issues/ADR per feature when implementation starts.
|
||||
102
docs/V2.3.md
Normal file
102
docs/V2.3.md
Normal file
@@ -0,0 +1,102 @@
|
||||
# Shade V2.3 — Tillit, retention, integrasjon og observability
|
||||
|
||||
Dette dokumentet beskriver **høyere ambisjonsnivå** og **plattformkryssende** arbeid: brukertilfeller der tillit må være **eksplisitt**, data må **utkrympes**/ryddes automatisk, apper må **enkelt koble seg på** kjente transportmønstre, og drift må **observere** uten å lekke innhold.
|
||||
|
||||
---
|
||||
|
||||
## 1. Key transparency / bundle-attestasjon **eller** kritiske fingerprint-øyeblikk i UI
|
||||
|
||||
Dette kan sees som **to spor** samme mål: redusere tillit til én korrumperbar prekey-server uten brukerens merknad.
|
||||
|
||||
### Spor A — Key transparency / bundle-attestasjon (ambisiøst)
|
||||
|
||||
**Idé:** Logging av **hvilket bundle som ble utlevert når**, kryptografisk forankret og **verifiserbar av klienter** eller tredjeparts-audit (inspirert av KT-litteratur, tilpasset Shade sin trusselmodell).
|
||||
|
||||
**Leveranser (målpris høyt):**
|
||||
|
||||
- Trusselmodell-oppdatering: *hva* CT/attest løser vs *fortsatt MITM før første verifisering*.
|
||||
- Designnotat: datastruktur, friskhetsbevis, klient-verifikasjonssteg, operatørkost.
|
||||
|
||||
**Risiko:** Kompleks drift og kryptodesign — bør være **valgfritt** lag for motiverte deploys.
|
||||
|
||||
### Spor B — Fingerprints i app-UI på kritiske hendelser (pragmatisk)
|
||||
|
||||
**Idé:** Gjør **safety numbers** synlige og **handlingspålagte** i presiserte flyt:
|
||||
|
||||
- **Før første stor fil** (eller før første stream over terskel i bytes).
|
||||
- **Før «trust this device»** / backup-importer / ny enhet som gjenbruker identitet.
|
||||
|
||||
**Leveranser:**
|
||||
|
||||
- `@shade/widgets` + SDK-hooks: modal/sheet med fingerprint, kopier-OOB-tekst, «jeg har verifisert».
|
||||
- Dokumenterte UX-retningslinjer (unngå «alert fatigue» kun på lave risiko-events).
|
||||
|
||||
### Felles akseptansekriterier
|
||||
|
||||
Enten spor A eller B må ha **eksplisitt testcoverage** på «blokkerer/handhever verifisering» der det er lovet, og trusselmodellen må nevne kombinasjonen av OOB + tekniske grep.
|
||||
|
||||
---
|
||||
|
||||
## 2. Retention policies
|
||||
|
||||
**Vision:** Standardiserte **TTL- og oppryddingsregler** for data Shade-økosystemet etterlater på server eller i klient-lagring — spesielt:
|
||||
|
||||
- **Stream-state** og midlertidige chunk-referanser etter fullførte/avbrutte transfers.
|
||||
- **Eventuell** inbox/relay-ciphertext (`V2.2`).
|
||||
- Prekey-server: kobling til eksisterende `SHADE_STALE_DAYS` / cleanup, plus **policy-dokumentasjon** for operatører.
|
||||
|
||||
**Leveranser:**
|
||||
|
||||
- Default-anbefalinger (f.eks. «ferdige streams: prune etter N dager») i `@shade/storage-*` helpers eller server-factory.
|
||||
- Konfigurerbare hooks: `maxAge`, `maxBytesPerIdentity`, cron vs on-access prune.
|
||||
|
||||
**Akseptansekriterier:** Ingen «uendelig vekst» som default i nye templates; dokumentert adferd i `streams.md` / deployment.
|
||||
|
||||
---
|
||||
|
||||
## 3. Ferdig «bridge» (transport)
|
||||
|
||||
**Vision:** Apper som ikke kan eller vil bruke WebSocket, får **ferdig mønster** for mottak av små meldinger eller kontroll-signaler.
|
||||
|
||||
**Eksempler:**
|
||||
|
||||
- **Server-Sent Events (SSE)** som **ren ciphertext-pipe** eller som signal «hent fra inbox» uten plaintext på server (kombinasjon med `V2.2`).
|
||||
- **Lang-poll fallback** dokumentert ved siden av WS i `@shade/transport`.
|
||||
|
||||
**Leveranser:**
|
||||
|
||||
- Modulær `bridge`-pakke eller tydelig undermodul med få eksponerte typer og én felles `IncomingMessage`-modell på klient.
|
||||
|
||||
**Akseptansekriterier:** Ett fungende eksempel + test som viser dekryptering likt eksisterende transport.
|
||||
|
||||
---
|
||||
|
||||
## 4. Observability
|
||||
|
||||
**Vision:** Produksjonsteam får **målepunkter og spor** uten **innholdslekkasjer** eller kryptomatrise i logger.
|
||||
|
||||
**Forslag til innhold:**
|
||||
|
||||
- **Metrics (Prometheus-stil eller vendor-nøytralt):** opplastings-/nedlastings-varighet, lane-telling, retry counts, abort vs complete rates, HTTP/WS-feilkoder (aggregert).
|
||||
- **OpenTelemetry:** spans rundt `TransferEngine` og prekey-endepunkter (uten payload-lengde i klartekst som PII — bruk binære størrelser eller binning).
|
||||
- **Sampling og PII-policy** dokumentert (ikke logg adresser i full hvis compliance krever maskering).
|
||||
|
||||
**Akseptansekriterier:** Opt-in flags (default av i lib, på i server-container der det gir mening), og `docs/` avsnitt om hva som **aldri** skal logges.
|
||||
|
||||
---
|
||||
|
||||
## Prioritering mot V2.1 / V2.2
|
||||
|
||||
| Dette dokument (V2.3) | Naturlig forutsetning |
|
||||
|----------------------|------------------------|
|
||||
| Retention | Streams/transfer i bruk (`V2.1` §5, `V2.2` metadata) |
|
||||
| Bridge | `V2.2` store-and-forward eller egen meldingskanal |
|
||||
| UI fingerprints | Widgets/SDK allerede i bruk |
|
||||
| KT / attest | Moden trusselmodell + juridisk/operativ vilje |
|
||||
| Observability | Stabil nok intern API for hooks |
|
||||
|
||||
---
|
||||
|
||||
## Versjonering
|
||||
|
||||
- **V2.3** — første samlet plan for tillit, retention, bridge og observability. Splitt i ADR-er når konkret design er valgt.
|
||||
201
docs/files.md
Normal file
201
docs/files.md
Normal file
@@ -0,0 +1,201 @@
|
||||
# `@shade/files` — E2EE filesystem RPC
|
||||
|
||||
`@shade/files` exposes a typed, end-to-end-encrypted RPC surface for
|
||||
filesystem-style operations between two Shade peers. Both sides ride a
|
||||
single Double Ratchet session for control envelopes; large-file content
|
||||
(`> 256 KiB`) flows through `@shade/transfer` lanes, correlated back to
|
||||
the RPC by an opaque `userMetadata.shadeFilesWriteId` /
|
||||
`shadeFilesReadStreamId`.
|
||||
|
||||
The package is a **primitive**, not a UI: it ships hooks and clients, not
|
||||
file managers. Consumers (e.g. Dispatch, Mail, Drive-style apps) keep
|
||||
their own UI and plug `@shade/files` underneath.
|
||||
|
||||
## Quick start
|
||||
|
||||
### Server (Bob exposes a filesystem)
|
||||
|
||||
```ts
|
||||
import { createShade } from '@shade/sdk';
|
||||
|
||||
const bob = await createShade({ prekeyServer, address: 'bob' });
|
||||
bob.configureTransfers({ resolveBaseUrl: ... });
|
||||
Bun.serve({ port: 8080, fetch: (await bob.transferRoute()).fetch });
|
||||
|
||||
const stop = await bob.files.serve({
|
||||
list: async (ctx) => ({ entries: await readdirAt(ctx.path), hasMore: false }),
|
||||
stat: async (ctx) => statAt(ctx.path),
|
||||
mkdir: async (ctx) => mkdirAt(ctx.path, ctx.args.recursive),
|
||||
delete: async (ctx) => deleteAt(ctx.path, ctx.args.recursive),
|
||||
move: async (ctx) => moveAt(ctx.args.src, ctx.args.dst, ctx.args.overwrite),
|
||||
read: async (ctx) => readAt(ctx.path), // returns inline | streams
|
||||
write: async (ctx) => writeAt(ctx.args), // receives inline | streams
|
||||
getThumbnail: async (ctx) => thumbnailAt(ctx.path, ctx.args.size, ctx.args.format),
|
||||
});
|
||||
|
||||
// Later: stop the handler.
|
||||
await stop();
|
||||
```
|
||||
|
||||
### Client (Alice consumes Bob's filesystem)
|
||||
|
||||
```ts
|
||||
const alice = await createShade({ prekeyServer, address: 'alice' });
|
||||
alice.configureTransfers({ resolveBaseUrl: ... });
|
||||
Bun.serve({ port: 8081, fetch: (await alice.transferRoute()).fetch });
|
||||
|
||||
const fs = await alice.files.client('bob');
|
||||
|
||||
const page = await fs.list('/photos');
|
||||
await fs.write('/photos/cover.png', new Uint8Array(...)); // auto inline/streams
|
||||
const result = await fs.read('/photos/cover.png');
|
||||
if (result.kind === 'inline') console.log(result.bytes.byteLength);
|
||||
else for await (const _chunk of /* result.stream */) { /* ... */ }
|
||||
```
|
||||
|
||||
## Op surface
|
||||
|
||||
| Op | Args | Result |
|
||||
|----------------|------------------------------------------|-----------------------------------------|
|
||||
| `list` | `path`, `cursor?`, `pageSize?`, `filter?`| `{ entries, hasMore, nextCursor? }` |
|
||||
| `stat` | `path` | `FileEntry` |
|
||||
| `mkdir` | `path`, `recursive?` | `{ entry: FileEntry }` |
|
||||
| `delete` | `path`, `recursive?` | `{ deletedCount }` |
|
||||
| `move` | `src`, `dst`, `overwrite?` | `{ entry: FileEntry }` |
|
||||
| `read` | `path`, `range?`, `preferInline?` | inline `{ bytes, sha256, size }` or streams `{ stream, size, sha256 }` |
|
||||
| `write` | `path`, `Uint8Array \| Blob \| Stream` | `{ entry: FileEntry }` |
|
||||
| `getThumbnail` | `path`, `size: 64\|128\|256\|512`, `format?` | `{ bytes, format, width, height, sha256 }` |
|
||||
| `custom<K>` | typed via `CustomOpsMap[K]` | typed via `CustomOpsMap[K]` |
|
||||
|
||||
`MUTATION_OPS = { mkdir, delete, move, write, custom }` — these auto-generate
|
||||
an idempotency key per logical call so transparent retries under the SDK
|
||||
don't produce duplicates.
|
||||
|
||||
## Inline vs streams
|
||||
|
||||
The threshold is `INLINE_THRESHOLD = 256 * 1024` bytes (plaintext). The
|
||||
client's `decideInline()` runs at `write` time:
|
||||
|
||||
* `Uint8Array` / `Blob` with known size → direct comparison.
|
||||
* Bare `ReadableStream` → peek the first chunks; promote to streams as
|
||||
soon as the buffered prefix exceeds the threshold.
|
||||
|
||||
Streams paths kick `shade.upload(...)` with `userMetadata.shadeFilesWriteId`
|
||||
in parallel with the RPC envelope. The server's streams-bridge accepts the
|
||||
inbound transfer immediately into a `TransformStream` and parks the
|
||||
readable side until the matching `write` RPC arrives.
|
||||
|
||||
## Custom ops
|
||||
|
||||
Augment `CustomOpsMap` once globally for type-safe consumer-defined ops:
|
||||
|
||||
```ts
|
||||
declare module '@shade/files' {
|
||||
interface CustomOpsMap {
|
||||
'app.deploy': CustomOpDef<{ jarPath: string }, { deploymentId: string }>;
|
||||
}
|
||||
}
|
||||
|
||||
// Server registers a Zod-backed handler:
|
||||
await shade.files.serve({
|
||||
custom: {
|
||||
'app.deploy': {
|
||||
args: z.object({ jarPath: z.string() }),
|
||||
response: z.object({ deploymentId: z.string() }),
|
||||
handler: async (args, ctx) => ({ deploymentId: deploy(args.jarPath) }),
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
// Client gets typed I/O:
|
||||
const result = await fs.custom('app.deploy', { jarPath: '/mods/foo.jar' });
|
||||
// ^? { deploymentId: string }
|
||||
```
|
||||
|
||||
## Production hooks
|
||||
|
||||
All hooks are callbacks the consumer plugs in — `@shade/files` enforces
|
||||
the *mechanism*; the app owns the *policy*.
|
||||
|
||||
```ts
|
||||
await shade.files.serve({
|
||||
pathPolicy: { rootScope: '/srv/shade-root', forbidTraversal: true },
|
||||
rateLimits: {
|
||||
maxOpsPerMinutePerSender: 600,
|
||||
maxBytesPerHourPerSender: 10 * 1024 ** 3,
|
||||
opCost: { read: 1, write: 5, delete: 3, default: 1 },
|
||||
},
|
||||
idempotency: { ttlMs: 60 * 60 * 1000, maxEntriesPerSender: 10_000 },
|
||||
requireFingerprintVerifiedFor: (ctx) =>
|
||||
['delete', 'write', 'mkdir'].includes(String(ctx.op)) ? 'required' : 'optional',
|
||||
isFingerprintVerified: (sender) => verifiedSet.has(sender),
|
||||
verifySender: async (sender, canonical, sig) => {
|
||||
return ed25519.verify(base64ToBytes(sig), canonical, getPubKey(sender));
|
||||
},
|
||||
onMetric: (name, value, tags) => prometheus.observe(name, value, tags),
|
||||
onError: (err, ctx) => logger.error({ op: ctx.op, sender: ctx.sender }, err),
|
||||
});
|
||||
```
|
||||
|
||||
## React
|
||||
|
||||
```tsx
|
||||
import { ShadeFilesProvider, useFileList } from '@shade/files/react';
|
||||
|
||||
function FileBrowser({ shade, peer }: { shade: Shade; peer: string }) {
|
||||
return (
|
||||
<ShadeFilesProvider shade={shade}>
|
||||
<Listing peer={peer} path="/" />
|
||||
</ShadeFilesProvider>
|
||||
);
|
||||
}
|
||||
|
||||
function Listing({ peer, path }) {
|
||||
const { entries, isLoading, hasMore, loadMore, refresh } = useFileList(peer, path);
|
||||
if (isLoading) return <Spinner />;
|
||||
return (
|
||||
<ul>
|
||||
{entries.map((e) => <li key={e.name}>{e.name} ({e.kind})</li>)}
|
||||
{hasMore && <button onClick={loadMore}>More</button>}
|
||||
</ul>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
`useFileTransfer` exposes a generic state machine for `BulkTransferHandle`s
|
||||
returned by `uploadDirectory()` / `downloadDirectory()`:
|
||||
|
||||
```tsx
|
||||
const { start, abort, state, filesDone, filesTotal, bytesDone } = useFileUpload();
|
||||
const handle = uploadDirectory(fs, localDir, '/uploaded');
|
||||
useEffect(() => { start(handle); return () => void abort(); }, []);
|
||||
```
|
||||
|
||||
## Path safety
|
||||
|
||||
The dispatcher applies `validatePath()` before invoking the user handler:
|
||||
|
||||
1. Length check on raw input.
|
||||
2. Forbidden-bytes check (NUL/CR/LF/DEL/backslash).
|
||||
3. Percent-decode (defends against `%2e%2e` smuggling).
|
||||
4. POSIX normalization.
|
||||
5. `..` traversal rejection.
|
||||
6. Root-scope boundary check.
|
||||
7. Consumer-supplied `extra` predicate.
|
||||
|
||||
The user handler receives the *normalized* path; using `args.path` raw is a
|
||||
TOCTOU bug.
|
||||
|
||||
## Wire compatibility
|
||||
|
||||
`@shade/files` 0.3.0 requires `@shade/proto` 0.3.0+. The proto layer's wire
|
||||
VERSION was bumped from `0x01` to `0x02` to lift the 64 KiB length-prefix
|
||||
ceiling that previously capped ratchet payloads. **Sessions are
|
||||
incompatible across the bump**; both peers must run 0.3.0+.
|
||||
|
||||
## Related modules
|
||||
|
||||
* `@shade/streams` — chunk encryption, lane key derivation. Indirect dep.
|
||||
* `@shade/transfer` — multi-lane transport with HTTP / WS fallback.
|
||||
* `@shade/sdk` — `Shade.files` getter; `BackgroundHooks.onPruneFiles` for
|
||||
retention.
|
||||
@@ -1,9 +1,9 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="no">
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>Shade — ende-til-ende kryptering som modul</title>
|
||||
<title>Shade — end-to-end encryption as a module</title>
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com" />
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
|
||||
<link href="https://fonts.googleapis.com/css2?family=Fraunces:ital,opsz,wght@0,9..144,400;0,9..144,600;0,9..144,700;1,9..144,400&family=Source+Serif+4:ital,opsz,wght@0,8..60,400;0,8..60,600;1,8..60,400&display=swap" rel="stylesheet" />
|
||||
@@ -364,32 +364,32 @@
|
||||
<header>
|
||||
<h1>Shade</h1>
|
||||
<p class="lede">
|
||||
En gjenbrukbar modul for <strong style="color: var(--text); font-weight: 600;">ende-til-ende-kryptert</strong> kommunikasjon i egne apper — med samme type protokoll som brukes i Signal.
|
||||
A reusable module for <strong style="color: var(--text); font-weight: 600;">end-to-end encrypted</strong> communication in your own apps — using the same kind of protocol as Signal.
|
||||
</p>
|
||||
<div class="badge-row">
|
||||
<span class="badge">X3DH</span>
|
||||
<span class="badge">Double Ratchet</span>
|
||||
<span class="badge">TypeScript</span>
|
||||
<span class="badge">Plattformagnostisk crypto</span>
|
||||
<span class="badge">Platform-agnostic crypto</span>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<section id="hva">
|
||||
<h2>Hva gjør prosjektet?</h2>
|
||||
<section id="what">
|
||||
<h2>What does the project do?</h2>
|
||||
<p>
|
||||
<strong>Shade</strong> er et monorepo som implementerer sikker meldingskryptering mellom to parter (for eksempel nettleser og server, eller to klienter). Meldingene er kryptert slik at transportlaget (HTTP, WebSocket, e.l.) bare ser uleselige bytes — ikke innholdet.
|
||||
<strong>Shade</strong> is a monorepo that implements secure messaging encryption between two parties (for example browser and server, or two clients). Messages are encrypted so the transport layer (HTTP, WebSocket, etc.) only sees opaque bytes — not the content.
|
||||
</p>
|
||||
<div class="callout">
|
||||
<strong>Kjerneideen:</strong> Du bygger inn <code>ShadeSessionManager</code> (fra <code>@shade/core</code>) sammen med en <code>CryptoProvider</code> (f.eks. Web Crypto i nettleser/Bun) og lagring. Deretter kan du kalle <code>encrypt</code> / <code>decrypt</code> per motpart, akkurat som i demo-koden <code>demo.ts</code>.
|
||||
<strong>Core idea:</strong> Embed <code>ShadeSessionManager</code> (from <code>@shade/core</code>) together with a <code>CryptoProvider</code> (e.g. Web Crypto in the browser/Bun) and storage. Then call <code>encrypt</code> / <code>decrypt</code> per peer, just like in the demo code <code>demo.ts</code>.
|
||||
</div>
|
||||
<p>
|
||||
Første melding til noen ny inneholder nøkkelavtale (X3DH). Etterpå bruker hver melding <em>Double Ratchet</em>: nye meldingsnøkler og periodiske DH-steg gir <strong>forward secrecy</strong> (gamle meldinger overlever ikke nøkkellekkasje) og <strong>post-compromise security</strong> (systemet «helbreder» seg over tid etter kompromittering).
|
||||
The first message to someone new performs key agreement (X3DH). After that each message uses the <em>Double Ratchet</em>: fresh message keys and periodic DH steps provide <strong>forward secrecy</strong> (past messages do not survive key compromise) and <strong>post-compromise security</strong> (the system “recovers” over time after a compromise).
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section id="pakker">
|
||||
<h2>Pakkene (hvordan det henger sammen)</h2>
|
||||
<div class="tabs" role="tablist" aria-label="Pakkeoversikt">
|
||||
<section id="packages">
|
||||
<h2>Packages (how they fit)</h2>
|
||||
<div class="tabs" role="tablist" aria-label="Package overview">
|
||||
<button type="button" class="tab-btn" role="tab" id="tab-core" aria-selected="true" aria-controls="panel-core">shade-core</button>
|
||||
<button type="button" class="tab-btn" role="tab" id="tab-crypto" aria-selected="false" aria-controls="panel-crypto">shade-crypto-web</button>
|
||||
<button type="button" class="tab-btn" role="tab" id="tab-proto" aria-selected="false" aria-controls="panel-proto">shade-proto</button>
|
||||
@@ -397,117 +397,117 @@
|
||||
<button type="button" class="tab-btn" role="tab" id="tab-server" aria-selected="false" aria-controls="panel-server">shade-server</button>
|
||||
</div>
|
||||
<div id="panel-core" class="tab-panel active" role="tabpanel" aria-labelledby="tab-core">
|
||||
<p style="margin-top:0"><strong>Protokollen.</strong> X3DH, Double Ratchet, sesjonstyper, feiltyper. Ingen plattformkrypto her — bare grensesnittet <code>CryptoProvider</code>.</p>
|
||||
<p style="margin-top:0"><strong>The protocol.</strong> X3DH, Double Ratchet, session shapes, errors. No platform crypto here — only the <code>CryptoProvider</code> interface.</p>
|
||||
<ul>
|
||||
<li><code>ShadeSessionManager</code> — høynivå-API: <code>initialize</code>, <code>createPreKeyBundle</code>, <code>initSessionFromBundle</code>, <code>encrypt</code>, <code>decrypt</code></li>
|
||||
<li>Symmetrisk kryptering: <strong>AES-256-GCM</strong> med AAD fra ratchet-header</li>
|
||||
<li><code>ShadeSessionManager</code> — high-level API: <code>initialize</code>, <code>createPreKeyBundle</code>, <code>initSessionFromBundle</code>, <code>encrypt</code>, <code>decrypt</code></li>
|
||||
<li>Symmetric encryption: <strong>AES-256-GCM</strong> with AAD from the ratchet header</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div id="panel-crypto" class="tab-panel" role="tabpanel" aria-labelledby="tab-crypto" hidden>
|
||||
<p style="margin-top:0"><strong>Implementasjon av crypto for web/Bun/Node</strong> via SubtleCrypto — X25519, Ed25519, HKDF, HMAC, tilfeldige bytes.</p>
|
||||
<p style="margin-top:0"><strong>Crypto implementation for web/Bun/Node</strong> via SubtleCrypto — X25519, Ed25519, HKDF, HMAC, random bytes.</p>
|
||||
<ul>
|
||||
<li>Gjør det mulig å bruke <code>shade-core</code> i nettleser og i servere som støtter Web Crypto</li>
|
||||
<li>Kommentarer i koden peker på fremtidig Android (f.eks. Tink) som egen provider</li>
|
||||
<li>Lets you use <code>shade-core</code> in the browser and on servers that support Web Crypto</li>
|
||||
<li>Comments in source point toward future Android (e.g. Tink) as a separate provider</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div id="panel-proto" class="tab-panel" role="tabpanel" aria-labelledby="tab-proto" hidden>
|
||||
<p style="margin-top:0"><strong>Binært wire-format</strong> for meldinger: versjon + type + lengdeprefiksede felt (big-endian).</p>
|
||||
<p style="margin-top:0"><strong>Binary wire format</strong> for messages: version + type + length-prefixed fields (big-endian).</p>
|
||||
<ul>
|
||||
<li>Type <code>0x01</code> = PreKeyMessage, <code>0x02</code> = RatchetMessage</li>
|
||||
<li>Brukes når du vil serialisere <code>ShadeEnvelope</code> effektivt over nettet</li>
|
||||
<li>Use when you want to serialize <code>ShadeEnvelope</code> efficiently on the wire</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div id="panel-transport" class="tab-panel" role="tabpanel" aria-labelledby="tab-transport" hidden>
|
||||
<p style="margin-top:0"><strong>Transportadaptere</strong> — ikke selve krypteringen, men hvordan du sender bytes (f.eks. fetch eller WebSocket).</p>
|
||||
<p style="margin-top:0"><strong>Transport adapters</strong> — not encryption itself, but how you move bytes (e.g. fetch or WebSocket).</p>
|
||||
<ul>
|
||||
<li>Kobler applikasjonen din til den kanalen du allerede bruker</li>
|
||||
<li>Hooks your application to the channel you already use</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div id="panel-server" class="tab-panel" role="tabpanel" aria-labelledby="tab-server" hidden>
|
||||
<p style="margin-top:0"><strong>Prekey-server (Hono)</strong> — lagrer offentlige nøkler slik at Alice kan starte samtale mens Bob er «offline».</p>
|
||||
<p style="margin-top:0"><strong>Prekey server (Hono)</strong> — stores public keys so Alice can start a conversation while Bob is “offline”.</p>
|
||||
<ul>
|
||||
<li><code>POST /v1/keys/register</code> — registrer identitet + bundle</li>
|
||||
<li><code>GET /v1/keys/bundle/:address</code> — hent bundle (forbruker én engangsnøkkel om tilgjengelig)</li>
|
||||
<li><code>POST /v1/keys/replenish</code> — etterfyll engangsnøkler</li>
|
||||
<li><code>POST /v1/keys/register</code> — register identity + bundle</li>
|
||||
<li><code>GET /v1/keys/bundle/:address</code> — fetch bundle (consumes one-time prekey when available)</li>
|
||||
<li><code>POST /v1/keys/replenish</code> — replenish one-time prekeys</li>
|
||||
</ul>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="nokler">
|
||||
<h2>Nøkler i korthet</h2>
|
||||
<section id="keys-in-brief">
|
||||
<h2>Keys at a glance</h2>
|
||||
<div class="accordion" id="key-acc">
|
||||
<div class="acc-item">
|
||||
<button type="button" class="acc-trigger" aria-expanded="true" aria-controls="acc-identity" id="btn-identity">
|
||||
Identitetsnøkkel (langvarig)
|
||||
Identity key (long-term)
|
||||
</button>
|
||||
<div class="acc-panel" id="acc-identity" role="region" aria-labelledby="btn-identity">
|
||||
<strong>Ed25519</strong> brukes til å signere den «signerte prekeyen». <strong>X25519</strong> brukes i Diffie-Hellman i X3DH og i ratchet. Én identitet per enhet/bruker i typisk oppsett.
|
||||
<strong>Ed25519</strong> signs the “signed prekey”. <strong>X25519</strong> is used in Diffie–Hellman in X3DH and in the ratchet. One identity per device/user is typical.
|
||||
</div>
|
||||
</div>
|
||||
<div class="acc-item">
|
||||
<button type="button" class="acc-trigger" aria-expanded="false" aria-controls="acc-spk" id="btn-spk">
|
||||
Signert prekey (mediumvarig, roteres)
|
||||
Signed prekey (medium-lived, rotated)
|
||||
</button>
|
||||
<div class="acc-panel" id="acc-spk" role="region" aria-labelledby="btn-spk" hidden>
|
||||
X25519-nøkkel som publiseres og signeres med identiteten. Mottaker verifiserer signaturen før DH. I koden anbefales rotasjon omtrent hver 1–7 dag.
|
||||
An X25519 key that is published and signed by the identity. The recipient verifies the signature before DH. The codebase recommends rotation on the order of 1–7 days.
|
||||
</div>
|
||||
</div>
|
||||
<div class="acc-item">
|
||||
<button type="button" class="acc-trigger" aria-expanded="false" aria-controls="acc-otpk" id="btn-otpk">
|
||||
Engangsnøkler (one-time prekeys)
|
||||
One-time prekeys
|
||||
</button>
|
||||
<div class="acc-panel" id="acc-otpk" role="region" aria-labelledby="btn-otpk" hidden>
|
||||
Valgfrie, men viktige for ekstra sikkerhet: hver hentet bundle kan inkludere én engangsnøkkel som slettes etter bruk (<code>processPreKeyMessage</code> fjerner den fra lager). Gir bedre beskyttelse mot visse angrep når mange klienter kobler til samme mottaker.
|
||||
Optional but important for stronger security: each fetched bundle can include one one-time key that is removed after use (<code>processPreKeyMessage</code> clears it from storage). Improves protection against certain attacks when many clients connect to the same recipient.
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="flyt">
|
||||
<h2>Interaktiv flyt: fra null til kryptert melding</h2>
|
||||
<section id="flow-demo">
|
||||
<h2>Interactive flow: zero to encrypted message</h2>
|
||||
<p>
|
||||
Klikk «Neste» for å gå gjennom rekkefølgen slik Shade er bygget. Dette speiler <code>ShadeSessionManager</code> og demoen i repoet.
|
||||
Click “Next step” to walk through the sequence as Shade builds it. This mirrors <code>ShadeSessionManager</code> and the demo in the repo.
|
||||
</p>
|
||||
<div class="flow">
|
||||
<h3>Sesjon og meldinger</h3>
|
||||
<h3>Sessions and messages</h3>
|
||||
<div class="flow-steps" id="flow-steps"></div>
|
||||
<div class="flow-actions">
|
||||
<button type="button" class="btn" id="flow-next">Neste steg</button>
|
||||
<button type="button" class="btn btn-secondary" id="flow-reset">Start på nytt</button>
|
||||
<button type="button" class="btn" id="flow-next">Next step</button>
|
||||
<button type="button" class="btn btn-secondary" id="flow-reset">Start over</button>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="x3dh-ratchet">
|
||||
<h2>X3DH og Double Ratchet (kort forklart)</h2>
|
||||
<h2>X3DH and Double Ratchet (brief)</h2>
|
||||
<p>
|
||||
<strong>X3DH</strong> løser problemet «jeg vil snakke med Bob nå, men Bob svarer ikke før senere». Bob legger ut en <em>prekey bundle</em> på serveren. Alice henter den, gjør 3 eller 4 DH-operasjoner (avhengig av om engangsnøkkel brukes), og deriverer en felles rot-nøkkel som begge kan rekonstruere uten at serveren kjenner hemmeligheten.
|
||||
<strong>X3DH</strong> solves “I want to talk to Bob now, but Bob may not reply until later”. Bob publishes a <em>prekey bundle</em> on the server. Alice fetches it, runs 3 or 4 DH operations (depending on whether a one-time key is used), and derives a shared root key both parties can reconstruct — without the server learning the secret.
|
||||
</p>
|
||||
<p>
|
||||
<strong>Double Ratchet</strong> bruker den roten som startpunkt. For hver melding (eller ved nye DH-nøkler) avledes nye nøkler; meldinger på ledningen er AES-GCM med autentisering (AAD binder kryptoteksten til ratchet-header). Protokollen håndterer også meldinger i feil rekkefølge innenfor grenser (<code>MAX_SKIP</code>).
|
||||
<strong>Double Ratchet</strong> uses that root as a starting point. For each message (or when new DH keys spin), keys are derived; on the wire payloads are AES-GCM with authentication (AAD binds ciphertext to the ratchet header). The protocol also tolerates messages arriving out of order within limits (<code>MAX_SKIP</code>).
|
||||
</p>
|
||||
<p>
|
||||
Spesifikasjoner fra Signal (engelsk): <a href="https://signal.org/docs/specifications/x3dh/" target="_blank" rel="noopener">X3DH</a> · <a href="https://signal.org/docs/specifications/doubleratchet/" target="_blank" rel="noopener">Double Ratchet</a>.
|
||||
Signal specifications (English): <a href="https://signal.org/docs/specifications/x3dh/" target="_blank" rel="noopener">X3DH</a> · <a href="https://signal.org/docs/specifications/doubleratchet/" target="_blank" rel="noopener">Double Ratchet</a>.
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section id="gjenbruk">
|
||||
<h2>Bruke Shade i flere prosjekter</h2>
|
||||
<section id="reuse">
|
||||
<h2>Using Shade across projects</h2>
|
||||
<p>
|
||||
Tenk på Shade som tre lag du kan kombinere etter behov:
|
||||
Treat Shade as three layers you combine as needed:
|
||||
</p>
|
||||
<ol>
|
||||
<li><strong>Core + crypto-provider + storage</strong> — selve E2EE-motoren (kan kjøre i klient eller serverprosess som skal dekryptere).</li>
|
||||
<li><strong>Proto</strong> — når du vil ha kompakt binær serialisering.</li>
|
||||
<li><strong>Transport + prekey-server</strong> — når du vil standardisere nøkkelutveksling og kanaler.</li>
|
||||
<li><strong>Core + crypto provider + storage</strong> — the E2EE engine itself (runs in a client or a server process that must decrypt).</li>
|
||||
<li><strong>Proto</strong> — when you want compact binary serialization.</li>
|
||||
<li><strong>Transport + prekey server</strong> — when you want standardized key discovery and channels.</li>
|
||||
</ol>
|
||||
<p>
|
||||
Referansekjøring: <code class="mono">bun demo.ts</code> i rotmappen viser frontend/backend-flyt med minnelager og ekte kryptoprimitiver.
|
||||
Reference path: <code class="mono">bun demo.ts</code> from the repo root shows a frontend/backend flow with memory storage and real crypto primitives.
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<footer>
|
||||
<p>Shade — oversikt generert som statisk HTML i <code class="mono">docs/shade-overview.html</code>. Åpne filen direkte i nettleseren eller server den statisk.</p>
|
||||
<p>Shade — overview written as static HTML in <code class="mono">docs/shade-overview.html</code>. Open the file directly in the browser or serve it as static assets.</p>
|
||||
</footer>
|
||||
</div>
|
||||
|
||||
@@ -561,28 +561,28 @@
|
||||
// Flow steps
|
||||
var steps = [
|
||||
{
|
||||
title: "Initialiser klient",
|
||||
body: "Kall initialize(): last eller generer identitetsnøkkel (Ed25519 + X25519), registrationId og signert prekey.",
|
||||
title: "Initialize client",
|
||||
body: "Call initialize(): load or generate the identity keys (Ed25519 + X25519), registrationId, and signed prekey.",
|
||||
},
|
||||
{
|
||||
title: "Publiser prekey bundle",
|
||||
body: "Bygg bundle med createPreKeyBundle() / generateOneTimePreKeys() og registrer på prekey-server (eller del ut av band for demo).",
|
||||
title: "Publish prekey bundle",
|
||||
body: "Build a bundle with createPreKeyBundle() / generateOneTimePreKeys() and register it on the prekey server (or share out-of-band for a demo).",
|
||||
},
|
||||
{
|
||||
title: "Start sesjon mot peer",
|
||||
body: "Hent motpartens bundle, kjør initSessionFromBundle(address, bundle). X3DH kjører og ratchet-sesjon lagres i StorageProvider.",
|
||||
title: "Start session with peer",
|
||||
body: "Fetch the peer bundle, run initSessionFromBundle(address, bundle). X3DH runs and the ratchet session is stored in StorageProvider.",
|
||||
},
|
||||
{
|
||||
title: "Første encrypt",
|
||||
body: "encrypt() returnerer ShadeEnvelope type 'prekey': inneholder ephemeral nøkkel, prekey-ID-er og første RatchetMessage (AES-GCM).",
|
||||
title: "First encrypt",
|
||||
body: "encrypt() returns a ShadeEnvelope of type 'prekey': includes ephemeral keys, prekey IDs, and the first RatchetMessage (AES-GCM).",
|
||||
},
|
||||
{
|
||||
title: "Motpart decrypt",
|
||||
body: "decrypt() på PreKeyMessage: gjenskaper samme root key, initReceiverSession, ratchetDecrypt — plaintext ut.",
|
||||
title: "Peer decrypt",
|
||||
body: "decrypt() on PreKeyMessage: restores the same root key, initReceiverSession, ratchetDecrypt — plaintext out.",
|
||||
},
|
||||
{
|
||||
title: "Videre meldinger",
|
||||
body: "Neste kall til encrypt() gir type 'ratchet'. DH-ratchet steg gir nye kjeder og forbedret sikkerhet over tid.",
|
||||
title: "Further messages",
|
||||
body: "The next encrypt() calls yield type 'ratchet'. DH ratchet steps rotate chains and improve security over time.",
|
||||
},
|
||||
];
|
||||
|
||||
|
||||
117
docs/streams.md
Normal file
117
docs/streams.md
Normal file
@@ -0,0 +1,117 @@
|
||||
# Shade Streams 0.2.0
|
||||
|
||||
E2EE chunked upload/download for Shade. Drop into any Shade-using app:
|
||||
|
||||
```ts
|
||||
const handle = await shade.upload({ to: 'bob', input: file });
|
||||
const result = await handle.done(); // { sha256, bytesSent, durationMs }
|
||||
```
|
||||
|
||||
…and on the receiver:
|
||||
|
||||
```ts
|
||||
shade.onIncomingTransfer(async (incoming) => {
|
||||
const handle = await incoming.accept({ output: { kind: 'file', path: '/uploads/x' } });
|
||||
await handle.done();
|
||||
});
|
||||
```
|
||||
|
||||
Or in React:
|
||||
|
||||
```tsx
|
||||
<ShadeRuntimeProvider runtime={shade}>
|
||||
<ShadeUploader to="bob" onComplete={(r) => console.log(r.sha256)} />
|
||||
</ShadeRuntimeProvider>
|
||||
```
|
||||
|
||||
## How it works
|
||||
|
||||
A transfer has two planes:
|
||||
|
||||
- **Control plane** — `stream-init`, `stream-finish`, `stream-abort`, and
|
||||
`stream-resume-*` messages, encoded as JSON plaintext and shipped through
|
||||
the existing Double Ratchet (envelope type `0x02`). One ratchet step
|
||||
establishes a stream; the rest is per-chunk AEAD.
|
||||
- **Data plane** — `stream-chunk` envelopes (envelope type `0x11`),
|
||||
AES-256-GCM-encrypted under a per-lane key, shipped over HTTP POST (or
|
||||
WebSocket if opted-in). Lanes run in parallel for throughput.
|
||||
|
||||
```
|
||||
Sender Receiver
|
||||
────── ────────
|
||||
streamSecret = randomBytes(32)
|
||||
streamId = randomBytes(16)
|
||||
streamKey = HKDF(streamSecret, streamId, "shade-stream/v1\0master")
|
||||
laneKey[i] = HKDF(streamKey, streamId, "...\0lane\0" || u32(i))
|
||||
|
||||
[stream-init JSON over Double Ratchet] ─▶
|
||||
parses streamSecret, derives same keys
|
||||
spawns L per-lane receivers
|
||||
|
||||
[chunk 0x11 over HTTP] ─▶
|
||||
AES-GCM(laneKey[i], plaintext, nonce=laneId||seq, aad=streamId||laneId||seq||isLast)
|
||||
decrypts, verifies, writes to sink
|
||||
(× 4 lanes in parallel)
|
||||
|
||||
[stream-finish JSON over Double Ratchet] ─▶
|
||||
verifies per-lane sha256 + overall sha256
|
||||
throws TransferIntegrityError on mismatch
|
||||
```
|
||||
|
||||
## Partition strategies
|
||||
|
||||
- **Range** (default for known-size inputs) — lane `i` owns bytes
|
||||
`[i·N/L, (i+1)·N/L)`. Receiver reconstructs by concatenating lane outputs
|
||||
in laneId order.
|
||||
- **Round-robin** (default for unknown-size streams) — chunk `i` goes to
|
||||
lane `i mod L`. Receiver reorders via a per-stream chunk-index buffer.
|
||||
|
||||
## Resume
|
||||
|
||||
Persistence is opt-in via a `ResumeStore` (memory, SQLite, Postgres,
|
||||
IndexedDB-ready). State persisted on init; sender's resume queries the
|
||||
receiver's `lastSeqAcked` per lane via `GET /v1/transfer/:streamId/state`,
|
||||
then continues from there. The streamSecret is encrypted at rest under a
|
||||
device-key derived from the local identity's signing private key — a
|
||||
stolen DB without the identity key cannot resume.
|
||||
|
||||
```ts
|
||||
const handle = await shade.resumeUpload(streamId, sameInputAsBefore);
|
||||
await handle.done();
|
||||
```
|
||||
|
||||
Resume across **identity rotation** is not supported (rotation invalidates
|
||||
the device key — by design, to prevent a stolen pre-rotation DB from
|
||||
deriving keys for any post-rotation transfer). Restart the transfer
|
||||
manually after rotation.
|
||||
|
||||
## Throughput
|
||||
|
||||
- Default 4 lanes × 1 MiB chunks × 4 in-flight chunks per lane =
|
||||
16 MiB peak in-flight per direction.
|
||||
- Memory-bounded: receivers stream chunks to the configured sink without
|
||||
buffering the full payload. 1 GB transfer = O(chunkSize) RSS, not O(file).
|
||||
- AES-GCM is hardware-accelerated via `SubtleCrypto`; SHA-256 streaming via
|
||||
`@noble/hashes`.
|
||||
|
||||
## Security properties
|
||||
|
||||
| ID | Property |
|
||||
|---|---|
|
||||
| S1 | streamSecret never on the wire in plaintext (Double Ratchet only) |
|
||||
| S2 | Unique per-(streamKey, laneId, seq) AEAD nonce — no nonce reuse |
|
||||
| S3 | Tampered chunk header / ciphertext / tag → AEAD reject |
|
||||
| S4 | Per-lane sha256 + overall sha256 verified at finish |
|
||||
| S5 | streamKey/laneKey zeroized on abort/finish (`destroy()`) |
|
||||
| S6 | Concurrent streams have independent lane keys |
|
||||
| S7 | seq overflow practical-impossible (u64 max) |
|
||||
| S8 | At-rest streamSecret encrypted under device-key |
|
||||
|
||||
## API surface
|
||||
|
||||
See package READMEs:
|
||||
|
||||
- `packages/shade-streams/README.md` — crypto + state machines
|
||||
- `packages/shade-transfer/README.md` — orchestration, transports, persistence
|
||||
- `packages/shade-sdk/README.md` — magic drop-in
|
||||
- `packages/shade-widgets/README.md` — React UI
|
||||
28
examples/07-streams-upload/README.md
Normal file
28
examples/07-streams-upload/README.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# Example 07 — E2EE file upload with Shade Streams
|
||||
|
||||
Two Bun processes, both running real `Shade` instances. Alice uploads a file
|
||||
to Bob; the file is chunked, encrypted per-chunk under a per-lane key, sent
|
||||
over HTTP across 4 parallel lanes, and verified end-to-end.
|
||||
|
||||
## Files
|
||||
|
||||
- `prekey-server.ts` — boots a `@shade/server` prekey server on port 9991.
|
||||
- `bob-receiver.ts` — Bob's Shade runtime + a Bun HTTP server that mounts
|
||||
`Shade.transferRoute()`. Saves incoming files to `./received/`.
|
||||
- `alice-sender.ts` — Alice's Shade runtime; uploads a file to Bob.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
# Terminal 1 — prekey server
|
||||
bun run examples/07-streams-upload/prekey-server.ts
|
||||
|
||||
# Terminal 2 — Bob (receiver)
|
||||
bun run examples/07-streams-upload/bob-receiver.ts
|
||||
|
||||
# Terminal 3 — Alice (sender)
|
||||
bun run examples/07-streams-upload/alice-sender.ts ./path/to/file
|
||||
```
|
||||
|
||||
The receiver writes the decrypted file to `./received/<filename>` and prints
|
||||
the matching sha256.
|
||||
54
examples/07-streams-upload/alice-sender.ts
Normal file
54
examples/07-streams-upload/alice-sender.ts
Normal file
@@ -0,0 +1,54 @@
|
||||
/**
|
||||
* Alice — sender. Uploads a local file to Bob using 4 parallel lanes,
|
||||
* 1 MiB chunks, and the default HTTP transport. Prints sender-side
|
||||
* progress + the verified sha256 on completion.
|
||||
*
|
||||
* Usage:
|
||||
* bun run examples/07-streams-upload/alice-sender.ts <path/to/file>
|
||||
*/
|
||||
import { readFile } from 'node:fs/promises';
|
||||
import { basename } from 'node:path';
|
||||
import { createShade } from '@shade/sdk';
|
||||
|
||||
const PREKEY_URL = 'http://localhost:9991';
|
||||
const BOB_BASE_URL = 'http://localhost:9992';
|
||||
|
||||
const filePath = process.argv[2];
|
||||
if (filePath === undefined || filePath === '') {
|
||||
console.error('usage: alice-sender.ts <path/to/file>');
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const bytes = await readFile(filePath);
|
||||
const name = basename(filePath);
|
||||
|
||||
const alice = await createShade({ prekeyServer: PREKEY_URL, address: 'alice' });
|
||||
|
||||
alice.configureTransfers({
|
||||
resolveBaseUrl: async (peer) => {
|
||||
if (peer === 'bob') return BOB_BASE_URL;
|
||||
throw new Error(`unknown peer ${peer}`);
|
||||
},
|
||||
});
|
||||
|
||||
console.log(`[alice] uploading ${name} (${bytes.length} bytes) → bob`);
|
||||
|
||||
const handle = await alice.upload({
|
||||
to: 'bob',
|
||||
input: new Uint8Array(bytes),
|
||||
lanes: 4,
|
||||
chunkSize: 1024 * 1024,
|
||||
metadata: { name },
|
||||
onProgress: (p) => {
|
||||
const pct = p.percent !== undefined ? `${(p.percent * 100).toFixed(1)}%` : '—';
|
||||
const mbps = (p.bytesPerSecond / (1024 * 1024)).toFixed(2);
|
||||
process.stdout.write(`\r[alice] ${pct} ${mbps} MB/s ${p.bytesSent}/${p.bytesTotal ?? '?'} B`);
|
||||
},
|
||||
});
|
||||
|
||||
const result = await handle.done();
|
||||
process.stdout.write('\n');
|
||||
console.log(`[alice] done — ${result.bytesSent} B in ${result.durationMs.toFixed(0)} ms`);
|
||||
console.log(`[alice] sha256 = ${result.sha256}`);
|
||||
|
||||
await alice.shutdown();
|
||||
36
examples/07-streams-upload/bob-receiver.ts
Normal file
36
examples/07-streams-upload/bob-receiver.ts
Normal file
@@ -0,0 +1,36 @@
|
||||
/**
|
||||
* Bob — receiver. Boots a Shade runtime and mounts the transfer route on
|
||||
* port 9992. Incoming uploads are decrypted, integrity-checked, and saved
|
||||
* to ./received/<filename>.
|
||||
*/
|
||||
import { mkdirSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { createShade } from '@shade/sdk';
|
||||
|
||||
const PREKEY_URL = 'http://localhost:9991';
|
||||
const PORT = 9992;
|
||||
const OUT_DIR = join(import.meta.dir, 'received');
|
||||
mkdirSync(OUT_DIR, { recursive: true });
|
||||
|
||||
const bob = await createShade({ prekeyServer: PREKEY_URL, address: 'bob' });
|
||||
|
||||
bob.configureTransfers({
|
||||
resolveBaseUrl: async () => {
|
||||
throw new Error('bob is receive-only');
|
||||
},
|
||||
});
|
||||
|
||||
bob.onIncomingTransfer(async (incoming) => {
|
||||
const name = incoming.metadata.name ?? incoming.streamId;
|
||||
const path = join(OUT_DIR, name.replace(/[^a-zA-Z0-9._-]/g, '_'));
|
||||
console.log(`[bob] incoming "${name}" from ${incoming.from} → ${path}`);
|
||||
const handle = await incoming.accept({ output: { kind: 'file', path } });
|
||||
void handle.done().then((result) => {
|
||||
console.log(`[bob] done "${name}" — ${result.bytesSent} B, sha256=${result.sha256}`);
|
||||
});
|
||||
});
|
||||
|
||||
const app = await bob.transferRoute();
|
||||
Bun.serve({ port: PORT, fetch: app.fetch });
|
||||
console.log(`Bob's transfer endpoint listening on http://localhost:${PORT}`);
|
||||
console.log(`Saving incoming files to ${OUT_DIR}`);
|
||||
17
examples/07-streams-upload/prekey-server.ts
Normal file
17
examples/07-streams-upload/prekey-server.ts
Normal file
@@ -0,0 +1,17 @@
|
||||
/**
|
||||
* Minimal Shade prekey server for the streams example.
|
||||
* Run on port 9991 — both Alice and Bob register here.
|
||||
*/
|
||||
import { createPrekeyServer, MemoryPrekeyStore } from '@shade/server';
|
||||
import { SubtleCryptoProvider } from '@shade/crypto-web';
|
||||
|
||||
const crypto = new SubtleCryptoProvider();
|
||||
const server = createPrekeyServer({
|
||||
crypto,
|
||||
store: new MemoryPrekeyStore(),
|
||||
disableRateLimit: true,
|
||||
});
|
||||
|
||||
const port = 9991;
|
||||
Bun.serve({ port, fetch: server.fetch });
|
||||
console.log(`Prekey server listening on http://localhost:${port}`);
|
||||
31
examples/08-files-browser/README.md
Normal file
31
examples/08-files-browser/README.md
Normal file
@@ -0,0 +1,31 @@
|
||||
# 08 — Files Browser (E2EE filesystem RPC)
|
||||
|
||||
Two-process demo of `@shade/files`: Alice (CLI) talks to Bob (file server) over
|
||||
Double Ratchet–encrypted RPC. Bob's filesystem is rooted at `./files-root/`.
|
||||
|
||||
## Run
|
||||
|
||||
```sh
|
||||
# Terminal 1 — prekey server
|
||||
bun run examples/08-files-browser/prekey-server.ts
|
||||
|
||||
# Terminal 2 — Bob's file server
|
||||
bun run examples/08-files-browser/bob-server.ts
|
||||
|
||||
# Terminal 3 — Alice issues commands
|
||||
bun run examples/08-files-browser/alice-cli.ts mkdir /docs
|
||||
bun run examples/08-files-browser/alice-cli.ts upload ./README.md /docs/readme.md
|
||||
bun run examples/08-files-browser/alice-cli.ts list /docs
|
||||
bun run examples/08-files-browser/alice-cli.ts stat /docs/readme.md
|
||||
bun run examples/08-files-browser/alice-cli.ts download /docs/readme.md /tmp/out.md
|
||||
bun run examples/08-files-browser/alice-cli.ts delete /docs/readme.md
|
||||
```
|
||||
|
||||
## What it shows
|
||||
|
||||
* Standard ops: `list`, `stat`, `mkdir`, `delete`, `move`.
|
||||
* Content I/O: `read`/`write` with automatic inline (≤ 256 KiB) /
|
||||
streams (> 256 KiB) routing — handled transparently by the SDK.
|
||||
* sha256 paritet: streaming downloads return the verified hash.
|
||||
* Two ratchets, end-to-end encrypted. No content ever leaves either
|
||||
process unencrypted.
|
||||
108
examples/08-files-browser/alice-cli.ts
Normal file
108
examples/08-files-browser/alice-cli.ts
Normal file
@@ -0,0 +1,108 @@
|
||||
/**
|
||||
* Alice — the file CLIENT. Demonstrates the typed `@shade/files` API:
|
||||
*
|
||||
* bun run examples/08-files-browser/alice-cli.ts list /
|
||||
* bun run examples/08-files-browser/alice-cli.ts mkdir /docs
|
||||
* bun run examples/08-files-browser/alice-cli.ts upload ./README.md /docs/readme.md
|
||||
* bun run examples/08-files-browser/alice-cli.ts download /docs/readme.md ./out.md
|
||||
* bun run examples/08-files-browser/alice-cli.ts stat /docs/readme.md
|
||||
* bun run examples/08-files-browser/alice-cli.ts delete /docs/readme.md
|
||||
*/
|
||||
import { readFile, writeFile, stat as fsstat } from 'node:fs/promises';
|
||||
import { createShade } from '@shade/sdk';
|
||||
|
||||
const PREKEY = 'http://localhost:9992';
|
||||
const ALICE_BASE_URL = 'http://localhost:9993';
|
||||
const BOB_BASE_URL = 'http://localhost:9994';
|
||||
|
||||
const alice = await createShade({ prekeyServer: PREKEY, address: 'alice' });
|
||||
alice.configureTransfers({
|
||||
resolveBaseUrl: async (peer) => {
|
||||
if (peer === 'bob') return BOB_BASE_URL;
|
||||
throw new Error(`alice: unknown peer ${peer}`);
|
||||
},
|
||||
});
|
||||
const app = await alice.transferRoute();
|
||||
Bun.serve({ port: 9993, fetch: app.fetch });
|
||||
|
||||
const fs = await alice.files.client('bob', { defaultTimeoutMs: 30_000 });
|
||||
|
||||
const [cmd, ...args] = process.argv.slice(2);
|
||||
try {
|
||||
switch (cmd) {
|
||||
case 'list': {
|
||||
const path = args[0] ?? '/';
|
||||
const result = await fs.list(path);
|
||||
for (const e of result.entries) {
|
||||
console.log(`${e.kind === 'dir' ? 'd' : '-'} ${String(e.size).padStart(10)} ${e.name}`);
|
||||
}
|
||||
break;
|
||||
}
|
||||
case 'stat': {
|
||||
const path = args[0]!;
|
||||
const e = await fs.stat(path);
|
||||
console.log(JSON.stringify(e, null, 2));
|
||||
break;
|
||||
}
|
||||
case 'mkdir': {
|
||||
const path = args[0]!;
|
||||
const r = await fs.mkdir(path, { recursive: true });
|
||||
console.log(`created ${r.entry.name}`);
|
||||
break;
|
||||
}
|
||||
case 'delete': {
|
||||
const path = args[0]!;
|
||||
const r = await fs.delete(path, { recursive: true });
|
||||
console.log(`deleted ${r.deletedCount} entrie(s)`);
|
||||
break;
|
||||
}
|
||||
case 'upload': {
|
||||
const localPath = args[0]!;
|
||||
const remotePath = args[1]!;
|
||||
const bytes = new Uint8Array(await readFile(localPath));
|
||||
const s = await fsstat(localPath);
|
||||
console.log(`uploading ${s.size} bytes → ${remotePath}`);
|
||||
const r = await fs.write(remotePath, bytes, { overwrite: true });
|
||||
console.log(`done — remote size: ${r.entry.size}`);
|
||||
break;
|
||||
}
|
||||
case 'download': {
|
||||
const remotePath = args[0]!;
|
||||
const localPath = args[1]!;
|
||||
const result = await fs.read(remotePath);
|
||||
if (result.kind === 'inline') {
|
||||
await writeFile(localPath, result.bytes);
|
||||
console.log(`downloaded ${result.bytes.byteLength} bytes (inline) sha256=${result.sha256}`);
|
||||
} else {
|
||||
const reader = result.stream.getReader();
|
||||
const chunks: Uint8Array[] = [];
|
||||
let total = 0;
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (value !== undefined) {
|
||||
chunks.push(value);
|
||||
total += value.byteLength;
|
||||
}
|
||||
}
|
||||
reader.releaseLock();
|
||||
const out = new Uint8Array(total);
|
||||
let offset = 0;
|
||||
for (const c of chunks) {
|
||||
out.set(c, offset);
|
||||
offset += c.byteLength;
|
||||
}
|
||||
await writeFile(localPath, out);
|
||||
await result.done();
|
||||
console.log(`downloaded ${total} bytes (streamed) sha256=${result.sha256}`);
|
||||
}
|
||||
break;
|
||||
}
|
||||
default:
|
||||
console.error('Usage: alice-cli.ts <list|stat|mkdir|delete|upload|download> [args...]');
|
||||
process.exit(1);
|
||||
}
|
||||
} finally {
|
||||
await alice.shutdown();
|
||||
process.exit(0);
|
||||
}
|
||||
170
examples/08-files-browser/bob-server.ts
Normal file
170
examples/08-files-browser/bob-server.ts
Normal file
@@ -0,0 +1,170 @@
|
||||
/**
|
||||
* Bob — the file SERVER. Mounts a virtual filesystem rooted in this
|
||||
* directory's `./files-root/` dir (created on first run) and serves it
|
||||
* over `@shade/files` E2EE RPC.
|
||||
*
|
||||
* Demonstrates: list/stat/mkdir/delete/move + read/write inline+streams.
|
||||
*/
|
||||
import { mkdir, readdir, readFile, rename, rm, stat, writeFile } from 'node:fs/promises';
|
||||
import { createReadStream } from 'node:fs';
|
||||
import { Readable } from 'node:stream';
|
||||
import { join, basename, dirname } from 'node:path';
|
||||
import { createShade } from '@shade/sdk';
|
||||
import {
|
||||
ConflictError,
|
||||
NotFoundError,
|
||||
type FileEntry,
|
||||
type UserReadResult,
|
||||
} from '@shade/files';
|
||||
|
||||
const ROOT = join(import.meta.dir, 'files-root');
|
||||
await mkdir(ROOT, { recursive: true });
|
||||
|
||||
const PREKEY = 'http://localhost:9992';
|
||||
const BOB_BASE_URL = 'http://localhost:9994';
|
||||
const ALICE_BASE_URL = 'http://localhost:9993';
|
||||
|
||||
const bob = await createShade({ prekeyServer: PREKEY, address: 'bob' });
|
||||
bob.configureTransfers({
|
||||
resolveBaseUrl: async (peer) => {
|
||||
if (peer === 'alice') return ALICE_BASE_URL;
|
||||
throw new Error(`bob: unknown peer ${peer}`);
|
||||
},
|
||||
});
|
||||
const app = await bob.transferRoute();
|
||||
Bun.serve({ port: 9994, fetch: app.fetch });
|
||||
console.log(`[bob] listening on ${BOB_BASE_URL}, files at ${ROOT}`);
|
||||
|
||||
function safePath(remote: string): string {
|
||||
// Strip leading slash + reject any '..' (the dispatcher already does
|
||||
// this, but we defend in depth).
|
||||
const segments = remote.split('/').filter((s) => s !== '' && s !== '..');
|
||||
return join(ROOT, ...segments);
|
||||
}
|
||||
|
||||
await bob.files.serve({
|
||||
list: async (ctx) => {
|
||||
const local = safePath(ctx.path);
|
||||
let names: string[];
|
||||
try {
|
||||
names = await readdir(local);
|
||||
} catch {
|
||||
throw new NotFoundError(ctx.path);
|
||||
}
|
||||
const entries: FileEntry[] = [];
|
||||
for (const name of names) {
|
||||
const s = await stat(join(local, name));
|
||||
entries.push({
|
||||
name,
|
||||
kind: s.isDirectory() ? 'dir' : 'file',
|
||||
size: s.isDirectory() ? 0 : s.size,
|
||||
mtime: s.mtimeMs,
|
||||
metadata: {},
|
||||
});
|
||||
}
|
||||
return { entries, hasMore: false };
|
||||
},
|
||||
stat: async (ctx) => {
|
||||
let s: import('node:fs').Stats;
|
||||
try {
|
||||
s = await stat(safePath(ctx.path));
|
||||
} catch {
|
||||
throw new NotFoundError(ctx.path);
|
||||
}
|
||||
return {
|
||||
name: basename(ctx.path),
|
||||
kind: s.isDirectory() ? 'dir' : 'file',
|
||||
size: s.isDirectory() ? 0 : s.size,
|
||||
mtime: s.mtimeMs,
|
||||
metadata: {},
|
||||
};
|
||||
},
|
||||
mkdir: async (ctx) => {
|
||||
const local = safePath(ctx.path);
|
||||
await mkdir(local, { recursive: ctx.args.recursive });
|
||||
return {
|
||||
entry: {
|
||||
name: basename(ctx.path),
|
||||
kind: 'dir',
|
||||
size: 0,
|
||||
mtime: Date.now(),
|
||||
metadata: {},
|
||||
},
|
||||
};
|
||||
},
|
||||
delete: async (ctx) => {
|
||||
await rm(safePath(ctx.path), { recursive: ctx.args.recursive, force: false });
|
||||
return { deletedCount: 1 };
|
||||
},
|
||||
move: async (ctx) => {
|
||||
await rename(safePath(ctx.args.src), safePath(ctx.args.dst));
|
||||
return {
|
||||
entry: {
|
||||
name: basename(ctx.args.dst),
|
||||
kind: 'file',
|
||||
size: 0,
|
||||
mtime: Date.now(),
|
||||
metadata: {},
|
||||
},
|
||||
};
|
||||
},
|
||||
read: async (ctx): Promise<UserReadResult> => {
|
||||
const local = safePath(ctx.path);
|
||||
const s = await stat(local);
|
||||
if (s.isDirectory()) throw new ConflictError(`${ctx.path} is a directory`);
|
||||
if (s.size <= 256 * 1024) {
|
||||
const bytes = new Uint8Array(await readFile(local));
|
||||
return { kind: 'inline', bytes };
|
||||
}
|
||||
// Stream the file.
|
||||
const sha256 = await computeSha256OfFile(local);
|
||||
const stream = Readable.toWeb(createReadStream(local)) as ReadableStream<Uint8Array>;
|
||||
return { kind: 'streams', stream, size: s.size, sha256 };
|
||||
},
|
||||
write: async (ctx) => {
|
||||
const local = safePath(ctx.args.path);
|
||||
await mkdir(dirname(local), { recursive: true });
|
||||
if (ctx.args.content.kind === 'inline') {
|
||||
await writeFile(local, ctx.args.content.bytes);
|
||||
} else {
|
||||
// Drain stream → temp buffer → file. For huge files use createWriteStream.
|
||||
const reader = ctx.args.content.stream.getReader();
|
||||
const chunks: Uint8Array[] = [];
|
||||
let total = 0;
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (value !== undefined) {
|
||||
chunks.push(value);
|
||||
total += value.byteLength;
|
||||
}
|
||||
}
|
||||
reader.releaseLock();
|
||||
const out = new Uint8Array(total);
|
||||
let offset = 0;
|
||||
for (const c of chunks) {
|
||||
out.set(c, offset);
|
||||
offset += c.byteLength;
|
||||
}
|
||||
await writeFile(local, out);
|
||||
// Verify integrity (optional)
|
||||
await ctx.args.content.sha256;
|
||||
}
|
||||
const s = await stat(local);
|
||||
return {
|
||||
entry: {
|
||||
name: basename(ctx.args.path),
|
||||
kind: 'file',
|
||||
size: s.size,
|
||||
mtime: s.mtimeMs,
|
||||
metadata: {},
|
||||
},
|
||||
};
|
||||
},
|
||||
});
|
||||
|
||||
async function computeSha256OfFile(path: string): Promise<string> {
|
||||
const { sha256 } = await import('@noble/hashes/sha2.js');
|
||||
const data = new Uint8Array(await readFile(path));
|
||||
return Array.from(sha256(data), (b) => b.toString(16).padStart(2, '0')).join('');
|
||||
}
|
||||
17
examples/08-files-browser/prekey-server.ts
Normal file
17
examples/08-files-browser/prekey-server.ts
Normal file
@@ -0,0 +1,17 @@
|
||||
/**
|
||||
* Minimal Shade prekey server for the files-browser example.
|
||||
* Run on port 9992 — both Alice and Bob register here.
|
||||
*/
|
||||
import { createPrekeyServer, MemoryPrekeyStore } from '@shade/server';
|
||||
import { SubtleCryptoProvider } from '@shade/crypto-web';
|
||||
|
||||
const crypto = new SubtleCryptoProvider();
|
||||
const server = createPrekeyServer({
|
||||
crypto,
|
||||
store: new MemoryPrekeyStore(),
|
||||
disableRateLimit: true,
|
||||
});
|
||||
|
||||
const port = 9992;
|
||||
Bun.serve({ port, fetch: server.fetch });
|
||||
console.log(`[prekey] listening on http://localhost:${port}`);
|
||||
@@ -18,11 +18,13 @@
|
||||
"publish:docker": "bun run scripts/build-docker.ts -- --push"
|
||||
},
|
||||
"devDependencies": {
|
||||
"bun-types": "^1.3.11"
|
||||
"bun-types": "^1.3.11",
|
||||
"fast-check": "^3.22.0"
|
||||
},
|
||||
"dependencies": {
|
||||
"@noble/curves": "^2.0.1",
|
||||
"@noble/hashes": "^2.0.1",
|
||||
"hono": "^4.12.12"
|
||||
"hono": "^4.12.12",
|
||||
"zod": "^3.23.8"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,17 +1,17 @@
|
||||
{
|
||||
"name": "@shade/cli",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"type": "module",
|
||||
"main": "src/cli.ts",
|
||||
"bin": {
|
||||
"shade": "src/cli.ts"
|
||||
},
|
||||
"dependencies": {
|
||||
"@shade/sdk": "workspace:*",
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
"@shade/sdk": "workspace:*",
|
||||
"@shade/storage-sqlite": "workspace:*",
|
||||
"@shade/transport": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*"
|
||||
"@shade/transport": "workspace:*"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@shade/server": "workspace:*"
|
||||
|
||||
@@ -27,7 +27,9 @@ export async function peerAddCommand(address: string): Promise<void> {
|
||||
}
|
||||
|
||||
export async function peerListCommand(): Promise<void> {
|
||||
const config = loadConfig();
|
||||
// Loading the config validates the environment; we don't need its values
|
||||
// until session enumeration is implemented (TODO).
|
||||
loadConfig();
|
||||
|
||||
// For list, we need to enumerate sessions from storage. The StorageProvider
|
||||
// doesn't currently expose a "list all sessions" method. For v1, we show
|
||||
|
||||
@@ -4,9 +4,9 @@ import { join } from 'path';
|
||||
export interface CliConfig {
|
||||
prekeyServer: string;
|
||||
storage: string;
|
||||
observerToken?: string;
|
||||
observerUrl?: string;
|
||||
address?: string;
|
||||
observerToken?: string | undefined;
|
||||
observerUrl?: string | undefined;
|
||||
address?: string | undefined;
|
||||
}
|
||||
|
||||
const DEFAULT_STORAGE = 'sqlite:./.shade/client.db';
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shade/core",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
"types": "src/index.ts",
|
||||
|
||||
@@ -98,7 +98,7 @@ export class NetworkError extends ShadeError {
|
||||
}
|
||||
|
||||
export class StorageError extends ShadeError {
|
||||
constructor(message: string, public readonly cause?: unknown) {
|
||||
constructor(message: string, public override readonly cause?: unknown) {
|
||||
super('SHADE_STORAGE', message);
|
||||
this.name = 'StorageError';
|
||||
}
|
||||
|
||||
@@ -2,7 +2,7 @@ import type { CryptoProvider } from './crypto.js';
|
||||
import type { KeyPair, SessionState, ChainState, RatchetMessage } from './types.js';
|
||||
import { MAX_SKIP, MAX_CACHED_SKIPPED_KEYS } from './types.js';
|
||||
import { kdfRootKey, kdfChainKey } from './keys.js';
|
||||
import { DecryptionError, MaxSkipExceededError, DuplicateMessageError } from './errors.js';
|
||||
import { DecryptionError, MaxSkipExceededError } from './errors.js';
|
||||
|
||||
/**
|
||||
* Double Ratchet — per-message forward secrecy and post-compromise recovery.
|
||||
|
||||
@@ -23,9 +23,8 @@ import {
|
||||
ratchetEncrypt,
|
||||
ratchetDecrypt,
|
||||
} from './ratchet.js';
|
||||
import { NoSessionError, UntrustedIdentityError } from './errors.js';
|
||||
import { NoSessionError } from './errors.js';
|
||||
import { computeFingerprint, shortFingerprint } from './fingerprint.js';
|
||||
import { constantTimeEqual } from './crypto.js';
|
||||
import { ShadeEventEmitter, shortHash } from './events.js';
|
||||
|
||||
const enc = new TextEncoder();
|
||||
@@ -61,14 +60,43 @@ export class ShadeSessionManager {
|
||||
private registrationId: number = 0;
|
||||
private currentSignedPreKeyId: number = 0;
|
||||
private readonly events?: ShadeEventEmitter;
|
||||
/**
|
||||
* Per-address operation chain. Both encrypt and decrypt mutate ratchet
|
||||
* state in place (counter, DH key, skipped-keys cache); concurrent
|
||||
* operations on the same peer can corrupt the session. We serialize
|
||||
* per-peer by chaining promises — operations to different peers stay
|
||||
* fully concurrent.
|
||||
*/
|
||||
private readonly peerOpChains = new Map<string, Promise<unknown>>();
|
||||
|
||||
constructor(
|
||||
private readonly crypto: CryptoProvider,
|
||||
private readonly storage: StorageProvider,
|
||||
options: { events?: ShadeEventEmitter } = {},
|
||||
) {
|
||||
if (options.events !== undefined) {
|
||||
this.events = options.events;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run `fn` under the per-address mutex so encrypt/decrypt for the same
|
||||
* peer never interleave their session-state mutations.
|
||||
*/
|
||||
private async runUnderPeerLock<T>(address: string, fn: () => Promise<T>): Promise<T> {
|
||||
const previous = this.peerOpChains.get(address) ?? Promise.resolve();
|
||||
const next = previous.catch(() => undefined).then(fn);
|
||||
this.peerOpChains.set(address, next);
|
||||
try {
|
||||
return await next;
|
||||
} finally {
|
||||
// Best-effort cleanup so finished chains can be garbage collected
|
||||
// when a peer goes idle. If a newer op has chained on, we leave it.
|
||||
if (this.peerOpChains.get(address) === next) {
|
||||
this.peerOpChains.delete(address);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Get the event emitter (if observability is enabled) */
|
||||
getEvents(): ShadeEventEmitter | undefined {
|
||||
@@ -368,6 +396,7 @@ export class ShadeSessionManager {
|
||||
* Subsequent messages are standard RatchetMessages.
|
||||
*/
|
||||
async encrypt(address: string, plaintext: string): Promise<ShadeEnvelope> {
|
||||
return this.runUnderPeerLock(address, async () => {
|
||||
const session = await this.storage.getSession(address);
|
||||
if (!session) throw new NoSessionError(address);
|
||||
|
||||
@@ -408,16 +437,19 @@ export class ShadeSessionManager {
|
||||
timestamp: Date.now(),
|
||||
senderAddress: address,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Decrypt a message from a peer. Handles both PreKeyMessage and RatchetMessage.
|
||||
*/
|
||||
async decrypt(address: string, envelope: ShadeEnvelope): Promise<string> {
|
||||
return this.runUnderPeerLock(address, async () => {
|
||||
if (envelope.type === 'prekey') {
|
||||
return this.decryptPreKeyMessage(address, envelope.content as PreKeyMessage);
|
||||
}
|
||||
return this.decryptRatchetMessage(address, envelope.content as RatchetMessage);
|
||||
});
|
||||
}
|
||||
|
||||
private async decryptPreKeyMessage(address: string, message: PreKeyMessage): Promise<string> {
|
||||
|
||||
@@ -6,6 +6,38 @@ export interface RetiredIdentity {
|
||||
retiredAt: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Persisted stream-transfer resume record. Holds enough state for either
|
||||
* side of a transfer to resume after restart. The `streamSecret` MUST be
|
||||
* encrypted before storage (see `@shade/transfer` for the deviceKey-based
|
||||
* AES-GCM at-rest scheme).
|
||||
*/
|
||||
export interface PersistedStreamState {
|
||||
streamId: string;
|
||||
direction: 'send' | 'receive';
|
||||
peerAddress: string;
|
||||
status: 'active' | 'paused' | 'finished' | 'aborted';
|
||||
/** JSON-serialized `StreamMetadata`. */
|
||||
metadataJson: string;
|
||||
/** JSON-serialized `LaneInitSpec[]`. */
|
||||
partitionJson: string;
|
||||
/** JSON-serialized per-lane progress array (laneId/nextSeq/bytesProcessed). */
|
||||
laneStateJson: string;
|
||||
/** JSON-serialized I/O descriptor (file path / file handle reference / buffer). */
|
||||
ioDescriptorJson: string;
|
||||
/** AES-GCM-encrypted streamSecret (under deviceKey). */
|
||||
secretEnc: Uint8Array;
|
||||
/** AES-GCM nonce used for `secretEnc`. */
|
||||
secretNonce: Uint8Array;
|
||||
/**
|
||||
* Reserved for future hasher serialization. Empty in v0.2.0; resume
|
||||
* re-hashes received bytes from disk.
|
||||
*/
|
||||
overallHashState?: string;
|
||||
createdAt: number;
|
||||
updatedAt: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* StorageProvider — abstract interface for persisting cryptographic state.
|
||||
*
|
||||
@@ -82,4 +114,27 @@ export interface StorageProvider {
|
||||
|
||||
/** Remove retired identities older than the given timestamp */
|
||||
pruneRetiredIdentities(olderThan: number): Promise<void>;
|
||||
|
||||
// ─── Stream-transfer resume state (optional, added in v0.2.0) ──
|
||||
|
||||
/**
|
||||
* Persist or update the stream-state row for a given streamId. Idempotent:
|
||||
* upserts on `streamId`. Optional — providers that don't support resume
|
||||
* can omit this and consumers will fall back to in-memory state.
|
||||
*/
|
||||
saveStreamState?(state: PersistedStreamState): Promise<void>;
|
||||
|
||||
/** Look up the stream-state row by streamId. Returns null if absent. */
|
||||
getStreamState?(streamId: string): Promise<PersistedStreamState | null>;
|
||||
|
||||
/** Remove a stream-state row (e.g. on completion or abort). */
|
||||
removeStreamState?(streamId: string): Promise<void>;
|
||||
|
||||
/** List active or paused stream-state rows (for resume on startup). */
|
||||
listActiveStreamStates?(
|
||||
direction?: 'send' | 'receive',
|
||||
): Promise<PersistedStreamState[]>;
|
||||
|
||||
/** Prune stream-state rows in `'finished' | 'aborted'` status older than `olderThan`. */
|
||||
pruneStreamStates?(olderThan: number): Promise<void>;
|
||||
}
|
||||
|
||||
@@ -2,16 +2,14 @@ import type { CryptoProvider } from './crypto.js';
|
||||
import type { StorageProvider } from './storage.js';
|
||||
import type {
|
||||
IdentityKeyPair,
|
||||
KeyPair,
|
||||
SignedPreKey,
|
||||
OneTimePreKey,
|
||||
PreKeyBundle,
|
||||
PreKeyMessage,
|
||||
RatchetMessage,
|
||||
SessionState,
|
||||
} from './types.js';
|
||||
import { deriveInitialRootKey } from './keys.js';
|
||||
import { InvalidSignatureError, PreKeyNotFoundError, UntrustedIdentityError } from './errors.js';
|
||||
import { InvalidSignatureError, PreKeyNotFoundError } from './errors.js';
|
||||
|
||||
/**
|
||||
* X3DH — Extended Triple Diffie-Hellman key agreement.
|
||||
@@ -75,7 +73,7 @@ export function createPreKeyBundle(
|
||||
signedPreKey: SignedPreKey,
|
||||
oneTimePreKey?: OneTimePreKey,
|
||||
): PreKeyBundle {
|
||||
return {
|
||||
const bundle: PreKeyBundle = {
|
||||
registrationId,
|
||||
identitySigningKey: identityKey.signingPublicKey,
|
||||
identityDHKey: identityKey.dhPublicKey,
|
||||
@@ -84,10 +82,14 @@ export function createPreKeyBundle(
|
||||
publicKey: signedPreKey.keyPair.publicKey,
|
||||
signature: signedPreKey.signature,
|
||||
},
|
||||
oneTimePreKey: oneTimePreKey
|
||||
? { keyId: oneTimePreKey.keyId, publicKey: oneTimePreKey.keyPair.publicKey }
|
||||
: undefined,
|
||||
};
|
||||
if (oneTimePreKey) {
|
||||
bundle.oneTimePreKey = {
|
||||
keyId: oneTimePreKey.keyId,
|
||||
publicKey: oneTimePreKey.keyPair.publicKey,
|
||||
};
|
||||
}
|
||||
return bundle;
|
||||
}
|
||||
|
||||
// ─── Alice: Initiate Session ─────────────────────────────────
|
||||
@@ -164,14 +166,15 @@ export async function processPreKeyBundle(
|
||||
// 6. Save trust for remote identity
|
||||
await storage.saveTrustedIdentity('pending', bundle.identityDHKey);
|
||||
|
||||
return {
|
||||
const result: X3DHInitResult = {
|
||||
rootKey,
|
||||
ephemeralPublicKey: ephemeral.publicKey,
|
||||
signedPreKeyId: bundle.signedPreKey.keyId,
|
||||
preKeyId,
|
||||
remoteIdentityKey: bundle.identityDHKey,
|
||||
remoteSignedPreKey: bundle.signedPreKey.publicKey,
|
||||
};
|
||||
if (preKeyId !== undefined) result.preKeyId = preKeyId;
|
||||
return result;
|
||||
}
|
||||
|
||||
// ─── Bob: Respond to PreKeyMessage ───────────────────────────
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shade/crypto-web",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
"types": "src/index.ts",
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import type { StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey, SessionState, RetiredIdentity } from '@shade/core';
|
||||
import type { StorageProvider, IdentityKeyPair, SignedPreKey, OneTimePreKey, SessionState, RetiredIdentity, PersistedStreamState } from '@shade/core';
|
||||
import { constantTimeEqual } from '@shade/core';
|
||||
|
||||
/**
|
||||
@@ -103,4 +103,40 @@ export class MemoryStorage implements StorageProvider {
|
||||
async pruneRetiredIdentities(olderThan: number): Promise<void> {
|
||||
this.retiredIdentities = this.retiredIdentities.filter((r) => r.retiredAt >= olderThan);
|
||||
}
|
||||
|
||||
// ─── Stream-transfer resume state (v0.2.0) ────────────────
|
||||
|
||||
private streamStates = new Map<string, PersistedStreamState>();
|
||||
|
||||
async saveStreamState(state: PersistedStreamState): Promise<void> {
|
||||
this.streamStates.set(state.streamId, { ...state });
|
||||
}
|
||||
|
||||
async getStreamState(streamId: string): Promise<PersistedStreamState | null> {
|
||||
const v = this.streamStates.get(streamId);
|
||||
return v ? { ...v } : null;
|
||||
}
|
||||
|
||||
async removeStreamState(streamId: string): Promise<void> {
|
||||
this.streamStates.delete(streamId);
|
||||
}
|
||||
|
||||
async listActiveStreamStates(direction?: 'send' | 'receive'): Promise<PersistedStreamState[]> {
|
||||
const out: PersistedStreamState[] = [];
|
||||
for (const s of this.streamStates.values()) {
|
||||
if (s.status !== 'active' && s.status !== 'paused') continue;
|
||||
if (direction !== undefined && s.direction !== direction) continue;
|
||||
out.push({ ...s });
|
||||
}
|
||||
out.sort((a, b) => b.updatedAt - a.updatedAt);
|
||||
return out;
|
||||
}
|
||||
|
||||
async pruneStreamStates(olderThan: number): Promise<void> {
|
||||
for (const [id, s] of this.streamStates) {
|
||||
if ((s.status === 'finished' || s.status === 'aborted') && s.updatedAt < olderThan) {
|
||||
this.streamStates.delete(id);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,6 +2,19 @@ import type { CryptoProvider } from '@shade/core';
|
||||
import { x25519 } from '@noble/curves/ed25519.js';
|
||||
import { ed25519 } from '@noble/curves/ed25519.js';
|
||||
|
||||
/**
|
||||
* Cast a Uint8Array to a SubtleCrypto-compatible buffer source. TS 5.7+
|
||||
* tightened Uint8Array's buffer generic to ArrayBuffer, but our public API
|
||||
* accepts Uint8Array<ArrayBufferLike>. SubtleCrypto methods are runtime-
|
||||
* compatible with both — only the type system needs the bridge.
|
||||
*
|
||||
* Returns `ArrayBuffer` (rather than the DOM `BufferSource` alias) so this
|
||||
* file type-checks without DOM lib in the consumer's tsconfig.
|
||||
*/
|
||||
function bs(u: Uint8Array | undefined): ArrayBuffer {
|
||||
return u as unknown as ArrayBuffer;
|
||||
}
|
||||
|
||||
/**
|
||||
* SubtleCrypto + noble/curves implementation of CryptoProvider.
|
||||
*
|
||||
@@ -55,11 +68,11 @@ export class SubtleCryptoProvider implements CryptoProvider {
|
||||
aad?: Uint8Array,
|
||||
): Promise<{ ciphertext: Uint8Array; nonce: Uint8Array }> {
|
||||
const nonce = this.randomBytes(12);
|
||||
const aesKey = await this.subtle.importKey('raw', key, 'AES-GCM', false, ['encrypt']);
|
||||
const aesKey = await this.subtle.importKey('raw', bs(key), 'AES-GCM', false, ['encrypt']);
|
||||
const encrypted = await this.subtle.encrypt(
|
||||
{ name: 'AES-GCM', iv: nonce, additionalData: aad },
|
||||
{ name: 'AES-GCM', iv: bs(nonce), additionalData: aad ? bs(aad) : undefined },
|
||||
aesKey,
|
||||
plaintext,
|
||||
bs(plaintext),
|
||||
);
|
||||
return { ciphertext: new Uint8Array(encrypted), nonce };
|
||||
}
|
||||
@@ -70,11 +83,11 @@ export class SubtleCryptoProvider implements CryptoProvider {
|
||||
nonce: Uint8Array,
|
||||
aad?: Uint8Array,
|
||||
): Promise<Uint8Array> {
|
||||
const aesKey = await this.subtle.importKey('raw', key, 'AES-GCM', false, ['decrypt']);
|
||||
const aesKey = await this.subtle.importKey('raw', bs(key), 'AES-GCM', false, ['decrypt']);
|
||||
const decrypted = await this.subtle.decrypt(
|
||||
{ name: 'AES-GCM', iv: nonce, additionalData: aad },
|
||||
{ name: 'AES-GCM', iv: bs(nonce), additionalData: aad ? bs(aad) : undefined },
|
||||
aesKey,
|
||||
ciphertext,
|
||||
bs(ciphertext),
|
||||
);
|
||||
return new Uint8Array(decrypted);
|
||||
}
|
||||
@@ -87,9 +100,9 @@ export class SubtleCryptoProvider implements CryptoProvider {
|
||||
info: Uint8Array,
|
||||
length: number,
|
||||
): Promise<Uint8Array> {
|
||||
const baseKey = await this.subtle.importKey('raw', ikm, 'HKDF', false, ['deriveBits']);
|
||||
const baseKey = await this.subtle.importKey('raw', bs(ikm), 'HKDF', false, ['deriveBits']);
|
||||
const bits = await this.subtle.deriveBits(
|
||||
{ name: 'HKDF', hash: 'SHA-256', salt, info },
|
||||
{ name: 'HKDF', hash: 'SHA-256', salt: bs(salt), info: bs(info) },
|
||||
baseKey,
|
||||
length * 8,
|
||||
);
|
||||
@@ -99,12 +112,12 @@ export class SubtleCryptoProvider implements CryptoProvider {
|
||||
async hmacSha256(key: Uint8Array, data: Uint8Array): Promise<Uint8Array> {
|
||||
const hmacKey = await this.subtle.importKey(
|
||||
'raw',
|
||||
key,
|
||||
bs(key),
|
||||
{ name: 'HMAC', hash: 'SHA-256' },
|
||||
false,
|
||||
['sign'],
|
||||
);
|
||||
const sig = await this.subtle.sign('HMAC', hmacKey, data);
|
||||
const sig = await this.subtle.sign('HMAC', hmacKey, bs(data));
|
||||
return new Uint8Array(sig);
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shade/dashboard",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
|
||||
41
packages/shade-files/package.json
Normal file
41
packages/shade-files/package.json
Normal file
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"name": "@shade/files",
|
||||
"version": "0.3.0",
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
"types": "src/index.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./src/index.ts",
|
||||
"import": "./src/index.ts"
|
||||
},
|
||||
"./react": {
|
||||
"types": "./src/react/index.ts",
|
||||
"import": "./src/react/index.ts"
|
||||
}
|
||||
},
|
||||
"dependencies": {
|
||||
"@shade/core": "workspace:*",
|
||||
"@shade/crypto-web": "workspace:*",
|
||||
"@shade/proto": "workspace:*",
|
||||
"@shade/sdk": "workspace:*",
|
||||
"@shade/streams": "workspace:*",
|
||||
"@shade/transfer": "workspace:*",
|
||||
"zod": "^3.23.8"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"react": "^18.0.0 || ^19.0.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"react": {
|
||||
"optional": true
|
||||
}
|
||||
},
|
||||
"devDependencies": {
|
||||
"@shade/server": "workspace:*",
|
||||
"@types/react": "^19.2.14",
|
||||
"fast-check": "^3.22.0",
|
||||
"happy-dom": "^15.11.7",
|
||||
"react": "^19.2.5"
|
||||
}
|
||||
}
|
||||
483
packages/shade-files/src/client/client.ts
Normal file
483
packages/shade-files/src/client/client.ts
Normal file
@@ -0,0 +1,483 @@
|
||||
import type { Shade } from '@shade/sdk';
|
||||
import {
|
||||
KIND_CUSTOM_V1,
|
||||
KIND_DELETE_V1,
|
||||
KIND_GET_THUMBNAIL_V1,
|
||||
KIND_LIST_V1,
|
||||
KIND_MKDIR_V1,
|
||||
KIND_MOVE_V1,
|
||||
KIND_READ_V1,
|
||||
KIND_STAT_V1,
|
||||
KIND_WRITE_V1,
|
||||
MUTATION_OPS,
|
||||
type StandardOp,
|
||||
} from '../protocol/kinds.js';
|
||||
import type { CustomOpsMap } from '../server/custom-ops.js';
|
||||
import { generateIdempotencyKey, generateRequestId } from '../protocol/correlate.js';
|
||||
import {
|
||||
base64ToBytes,
|
||||
bytesToBase64,
|
||||
canonicalRpcBytes,
|
||||
hashArgs,
|
||||
} from '../protocol/canonical.js';
|
||||
import { ShadeFileRpcChannel } from '../rpc/channel.js';
|
||||
import { PendingRpcRegistry, type RegisterOptions } from '../rpc/pending.js';
|
||||
import type { RpcRequest } from '../schemas/envelope.js';
|
||||
import {
|
||||
CustomArgsSchema,
|
||||
CustomResultSchema,
|
||||
DeleteArgsSchema,
|
||||
DeleteResultSchema,
|
||||
GetThumbnailArgsSchema,
|
||||
GetThumbnailResultSchema,
|
||||
ListArgsSchema,
|
||||
ListResultSchema,
|
||||
MkdirArgsSchema,
|
||||
MkdirResultSchema,
|
||||
MoveArgsSchema,
|
||||
MoveResultSchema,
|
||||
ReadArgsSchema,
|
||||
ReadResultSchema,
|
||||
StatArgsSchema,
|
||||
StatResultSchema,
|
||||
WriteArgsSchema,
|
||||
WriteResultSchema,
|
||||
type DeleteArgs,
|
||||
type DeleteResult,
|
||||
type GetThumbnailArgs,
|
||||
type ListArgs,
|
||||
type ListResult,
|
||||
type MkdirArgs,
|
||||
type MkdirResult,
|
||||
type MoveArgs,
|
||||
type MoveResult,
|
||||
type ReadArgs,
|
||||
type StatResult,
|
||||
type ThumbnailSize,
|
||||
type WriteResult,
|
||||
} from '../schemas/ops.js';
|
||||
import { ConflictError, InternalFileError } from '../schemas/errors.js';
|
||||
import { decideInline, INLINE_THRESHOLD, type WriteSource } from './inline-threshold.js';
|
||||
import type { ClientStreamsBridge } from './streams-bridge.js';
|
||||
|
||||
export interface BaseOpts {
|
||||
signal?: AbortSignal;
|
||||
/** Auto-generated for mutations if not provided. */
|
||||
idempotencyKey?: string;
|
||||
/** Per-call timeout. Default 30_000 ms. */
|
||||
timeoutMs?: number;
|
||||
}
|
||||
|
||||
// ─── read/write public types ─────────────────────────────────
|
||||
|
||||
export interface ReadInlineOutput {
|
||||
kind: 'inline';
|
||||
bytes: Uint8Array;
|
||||
size: number;
|
||||
sha256: string;
|
||||
contentType?: string;
|
||||
}
|
||||
export interface ReadStreamsOutput {
|
||||
kind: 'streams';
|
||||
stream: ReadableStream<Uint8Array>;
|
||||
size: number;
|
||||
sha256: string;
|
||||
contentType?: string;
|
||||
/** Resolves once the entire transfer has been received and verified. */
|
||||
done(): Promise<void>;
|
||||
}
|
||||
export type ReadOutput = ReadInlineOutput | ReadStreamsOutput;
|
||||
|
||||
export interface WriteOpts extends BaseOpts {
|
||||
contentType?: string;
|
||||
overwrite?: boolean;
|
||||
/** Force inline even if size > 256 KiB. Throws if input is too big. */
|
||||
forceInline?: boolean;
|
||||
}
|
||||
|
||||
export interface ReadOpts extends BaseOpts {
|
||||
range?: { start: number; end: number };
|
||||
preferInline?: boolean;
|
||||
}
|
||||
|
||||
export interface ThumbnailResult {
|
||||
bytes: Uint8Array;
|
||||
format: 'png' | 'webp' | 'jpeg';
|
||||
width: number;
|
||||
height: number;
|
||||
sha256: string;
|
||||
}
|
||||
|
||||
// ─── FileClient interface ────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Untyped fallback for `FileClient.custom()` — used when the consumer
|
||||
* hasn't extended `CustomOpsMap` for a given op name.
|
||||
*/
|
||||
type CustomOpArgs<K> = K extends keyof CustomOpsMap
|
||||
? CustomOpsMap[K] extends { args: infer A }
|
||||
? A
|
||||
: unknown
|
||||
: unknown;
|
||||
type CustomOpResponse<K> = K extends keyof CustomOpsMap
|
||||
? CustomOpsMap[K] extends { response: infer R }
|
||||
? R
|
||||
: unknown
|
||||
: unknown;
|
||||
|
||||
export interface FileClient {
|
||||
list(path: string, opts?: BaseOpts & Partial<Omit<ListArgs, 'path'>>): Promise<ListResult>;
|
||||
stat(path: string, opts?: BaseOpts): Promise<StatResult>;
|
||||
mkdir(path: string, opts?: BaseOpts & Partial<Omit<MkdirArgs, 'path'>>): Promise<MkdirResult>;
|
||||
delete(path: string, opts?: BaseOpts & Partial<Omit<DeleteArgs, 'path'>>): Promise<DeleteResult>;
|
||||
move(src: string, dst: string, opts?: BaseOpts & Partial<Omit<MoveArgs, 'src' | 'dst'>>): Promise<MoveResult>;
|
||||
read(path: string, opts?: ReadOpts): Promise<ReadOutput>;
|
||||
write(path: string, input: WriteSource, opts?: WriteOpts): Promise<WriteResult>;
|
||||
getThumbnail(
|
||||
path: string,
|
||||
size: ThumbnailSize,
|
||||
opts?: BaseOpts & { format?: 'png' | 'webp' | 'jpeg' },
|
||||
): Promise<ThumbnailResult>;
|
||||
/**
|
||||
* Invoke a custom op registered on the server. Args/response types are
|
||||
* pulled from `CustomOpsMap` via TypeScript declaration merging — see
|
||||
* `server/custom-ops.ts` for the registration pattern.
|
||||
*/
|
||||
custom<K extends keyof CustomOpsMap | string>(
|
||||
name: K & string,
|
||||
args: CustomOpArgs<K>,
|
||||
opts?: BaseOpts,
|
||||
): Promise<CustomOpResponse<K>>;
|
||||
close(): void;
|
||||
}
|
||||
|
||||
export interface CreateFileClientOptions {
|
||||
/** Default per-call timeout. Default 30_000. */
|
||||
defaultTimeoutMs?: number;
|
||||
/** Hard deadline for incoming-read awaits. Default 60_000. */
|
||||
ioTimeoutMs?: number;
|
||||
/**
|
||||
* Required for read/write `streams` ops. Coordinates inbound/outbound
|
||||
* `@shade/transfer` transfers via `userMetadata.shadeFiles*Id` keys.
|
||||
*/
|
||||
streamsBridge?: ClientStreamsBridge;
|
||||
/**
|
||||
* Optional: sign the canonical bytes of every outgoing RPC envelope.
|
||||
* Pluggable so apps can plug their own signing-key store (e.g.,
|
||||
* Ed25519-as-a-service, browser SubtleCrypto). When omitted, ships
|
||||
* `'unsigned'` — the server's `verifySender` should also be unset, or
|
||||
* be configured to accept the placeholder.
|
||||
*/
|
||||
signRequest?: (canonicalBytes: Uint8Array) => Promise<string> | string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Client-side proxy for `@shade/files` ops. Each method ships an
|
||||
* `RpcRequest` over `Shade.send`/`Shade.receive` and awaits the matching
|
||||
* response (or error/timeout) from `PendingRpcRegistry`.
|
||||
*
|
||||
* Mutations auto-generate an idempotency key per logical call (not per
|
||||
* attempt) so transparent retries under the SDK don't produce duplicates.
|
||||
*
|
||||
* Read/write content I/O over 256 KiB requires a `streamsBridge` to be
|
||||
* passed via options — it coordinates the inbound/outbound `@shade/transfer`
|
||||
* transfers that carry the actual bytes.
|
||||
*/
|
||||
export function createFileClient(
|
||||
shade: Shade,
|
||||
channel: ShadeFileRpcChannel,
|
||||
pending: PendingRpcRegistry,
|
||||
peerAddress: string,
|
||||
options: CreateFileClientOptions = {},
|
||||
): FileClient {
|
||||
const defaultTimeout = options.defaultTimeoutMs ?? 30_000;
|
||||
const ioTimeoutMs = options.ioTimeoutMs ?? 60_000;
|
||||
const streamsBridge = options.streamsBridge;
|
||||
const signRequest = options.signRequest;
|
||||
const senderAddress = shade.myAddress;
|
||||
|
||||
async function request<TResult>(
|
||||
kind: string,
|
||||
op: StandardOp | 'custom',
|
||||
args: unknown,
|
||||
opts: BaseOpts | undefined,
|
||||
): Promise<TResult> {
|
||||
const requestId = generateRequestId();
|
||||
const isMutation = MUTATION_OPS.has(op);
|
||||
const idempotencyKey =
|
||||
opts?.idempotencyKey ?? (isMutation ? generateIdempotencyKey() : undefined);
|
||||
const signedAt = Date.now();
|
||||
let sig = 'unsigned';
|
||||
if (signRequest !== undefined) {
|
||||
// Server reconstructs canonical bytes using `address = from`, which
|
||||
// is OUR own address as authenticated by the ratchet. So we sign
|
||||
// over the same identifier here.
|
||||
const canonical = canonicalRpcBytes({
|
||||
address: senderAddress,
|
||||
signedAt,
|
||||
kind,
|
||||
id: requestId,
|
||||
argsHash: hashArgs(args),
|
||||
});
|
||||
sig = await signRequest(canonical);
|
||||
}
|
||||
const env: RpcRequest = {
|
||||
kind,
|
||||
id: requestId,
|
||||
args,
|
||||
...(idempotencyKey !== undefined ? { idempotencyKey } : {}),
|
||||
sig,
|
||||
signedAt,
|
||||
};
|
||||
|
||||
const registerOpts: RegisterOptions = {
|
||||
timeoutMs: opts?.timeoutMs ?? defaultTimeout,
|
||||
onCancel: (reason) => {
|
||||
// Fire-and-forget cancel envelope so server can release resources.
|
||||
void channel
|
||||
.send(peerAddress, {
|
||||
kind: 'shade.fs.cancel/v1',
|
||||
id: requestId,
|
||||
reason,
|
||||
})
|
||||
.catch(() => {
|
||||
/* swallow — cancellation is best-effort */
|
||||
});
|
||||
},
|
||||
};
|
||||
if (opts?.signal !== undefined) registerOpts.signal = opts.signal;
|
||||
|
||||
const pendingPromise = pending.register<TResult>(requestId, registerOpts);
|
||||
try {
|
||||
await channel.send(peerAddress, env);
|
||||
} catch (err) {
|
||||
// If the send itself fails, the pending entry will never resolve;
|
||||
// reject it directly.
|
||||
pending.rejectAll(err);
|
||||
throw err;
|
||||
}
|
||||
return pendingPromise;
|
||||
}
|
||||
|
||||
return {
|
||||
async list(path, opts) {
|
||||
const args: ListArgs = ListArgsSchema.parse({
|
||||
path,
|
||||
...(opts?.cursor !== undefined ? { cursor: opts.cursor } : {}),
|
||||
...(opts?.pageSize !== undefined ? { pageSize: opts.pageSize } : {}),
|
||||
...(opts?.filter !== undefined ? { filter: opts.filter } : {}),
|
||||
});
|
||||
const raw = await request<unknown>(KIND_LIST_V1, 'list', args, opts);
|
||||
return ListResultSchema.parse(raw);
|
||||
},
|
||||
|
||||
async stat(path, opts) {
|
||||
const args = StatArgsSchema.parse({ path });
|
||||
const raw = await request<unknown>(KIND_STAT_V1, 'stat', args, opts);
|
||||
return StatResultSchema.parse(raw);
|
||||
},
|
||||
|
||||
async mkdir(path, opts) {
|
||||
const args = MkdirArgsSchema.parse({
|
||||
path,
|
||||
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
|
||||
});
|
||||
const raw = await request<unknown>(KIND_MKDIR_V1, 'mkdir', args, opts);
|
||||
return MkdirResultSchema.parse(raw);
|
||||
},
|
||||
|
||||
async delete(path, opts) {
|
||||
const args = DeleteArgsSchema.parse({
|
||||
path,
|
||||
...(opts?.recursive !== undefined ? { recursive: opts.recursive } : {}),
|
||||
});
|
||||
const raw = await request<unknown>(KIND_DELETE_V1, 'delete', args, opts);
|
||||
return DeleteResultSchema.parse(raw);
|
||||
},
|
||||
|
||||
async move(src, dst, opts) {
|
||||
const args = MoveArgsSchema.parse({
|
||||
src,
|
||||
dst,
|
||||
...(opts?.overwrite !== undefined ? { overwrite: opts.overwrite } : {}),
|
||||
});
|
||||
const raw = await request<unknown>(KIND_MOVE_V1, 'move', args, opts);
|
||||
return MoveResultSchema.parse(raw);
|
||||
},
|
||||
|
||||
async read(path, opts) {
|
||||
const args: ReadArgs = ReadArgsSchema.parse({
|
||||
path,
|
||||
...(opts?.range !== undefined ? { range: opts.range } : {}),
|
||||
...(opts?.preferInline !== undefined ? { preferInline: opts.preferInline } : {}),
|
||||
});
|
||||
const raw = await request<unknown>(KIND_READ_V1, 'read', args, opts);
|
||||
const wire = ReadResultSchema.parse(raw);
|
||||
if (wire.kind === 'inline') {
|
||||
const bytes = base64ToBytes(wire.bytesB64);
|
||||
const out: ReadInlineOutput = {
|
||||
kind: 'inline',
|
||||
bytes,
|
||||
size: wire.size,
|
||||
sha256: wire.sha256,
|
||||
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
|
||||
};
|
||||
return out;
|
||||
}
|
||||
// streams — wait for the matching incoming transfer via the bridge.
|
||||
if (streamsBridge === undefined) {
|
||||
throw new InternalFileError(
|
||||
'streams-bridge not configured: cannot consume streamed read',
|
||||
);
|
||||
}
|
||||
const bridgeSignal = opts?.signal ?? new AbortController().signal;
|
||||
const parked = await streamsBridge.awaitRead(wire.streamId, {
|
||||
expectedFrom: peerAddress,
|
||||
signal: bridgeSignal,
|
||||
timeoutMs: ioTimeoutMs,
|
||||
});
|
||||
const out: ReadStreamsOutput = {
|
||||
kind: 'streams',
|
||||
stream: parked.readable,
|
||||
size: wire.size,
|
||||
sha256: wire.sha256,
|
||||
...(wire.contentType !== undefined ? { contentType: wire.contentType } : {}),
|
||||
done: async () => {
|
||||
await parked.done;
|
||||
},
|
||||
};
|
||||
return out;
|
||||
},
|
||||
|
||||
async write(path, input, opts) {
|
||||
const decision = await decideInline(input);
|
||||
const overwrite = opts?.overwrite ?? false;
|
||||
const contentType = opts?.contentType ?? decision.contentType;
|
||||
|
||||
if (decision.kind === 'inline' || opts?.forceInline === true) {
|
||||
// Inline path — base64 in the RPC envelope.
|
||||
const bytes =
|
||||
decision.kind === 'inline'
|
||||
? decision.bytes
|
||||
: await drainToUint8Array(decision.stream, decision.size ?? Number.POSITIVE_INFINITY);
|
||||
if (bytes.byteLength > INLINE_THRESHOLD && opts?.forceInline !== true) {
|
||||
throw new ConflictError(
|
||||
`inline write exceeds ${INLINE_THRESHOLD}-byte threshold (got ${bytes.byteLength}); pass forceInline=true to override`,
|
||||
);
|
||||
}
|
||||
const args = WriteArgsSchema.parse({
|
||||
kind: 'inline',
|
||||
path,
|
||||
bytesB64: bytesToBase64(bytes),
|
||||
...(contentType !== undefined ? { contentType } : {}),
|
||||
overwrite,
|
||||
});
|
||||
const raw = await request<unknown>(KIND_WRITE_V1, 'write', args, opts);
|
||||
return WriteResultSchema.parse(raw);
|
||||
}
|
||||
|
||||
// Streams path — kick the upload, then ship the RPC.
|
||||
if (streamsBridge === undefined) {
|
||||
throw new InternalFileError(
|
||||
'streams-bridge not configured: cannot ship streamed write',
|
||||
);
|
||||
}
|
||||
const size = decision.size;
|
||||
if (size === undefined) {
|
||||
throw new ConflictError(
|
||||
'streams write requires a known plaintext size; pass `{ stream, size }` instead of a bare ReadableStream',
|
||||
);
|
||||
}
|
||||
const { writeId, handle } = await streamsBridge.initiateWrite({
|
||||
peer: peerAddress,
|
||||
stream: decision.stream,
|
||||
size,
|
||||
...(contentType !== undefined ? { contentType } : {}),
|
||||
name: path,
|
||||
...(opts?.signal !== undefined ? { signal: opts.signal } : {}),
|
||||
});
|
||||
const args = WriteArgsSchema.parse({
|
||||
kind: 'streams',
|
||||
path,
|
||||
size,
|
||||
...(contentType !== undefined ? { contentType } : {}),
|
||||
overwrite,
|
||||
writeId,
|
||||
});
|
||||
try {
|
||||
const [raw] = await Promise.all([
|
||||
request<unknown>(KIND_WRITE_V1, 'write', args, opts),
|
||||
handle.done(),
|
||||
]);
|
||||
return WriteResultSchema.parse(raw);
|
||||
} catch (err) {
|
||||
// Best-effort cancel of the transfer on RPC failure.
|
||||
await handle.abort('rpc-failed').catch(() => undefined);
|
||||
throw err;
|
||||
}
|
||||
},
|
||||
|
||||
async getThumbnail(path, size, opts) {
|
||||
const args: GetThumbnailArgs = GetThumbnailArgsSchema.parse({
|
||||
path,
|
||||
size,
|
||||
...(opts?.format !== undefined ? { format: opts.format } : {}),
|
||||
});
|
||||
const raw = await request<unknown>(KIND_GET_THUMBNAIL_V1, 'getThumbnail', args, opts);
|
||||
const wire = GetThumbnailResultSchema.parse(raw);
|
||||
return {
|
||||
bytes: base64ToBytes(wire.bytesB64),
|
||||
format: wire.format,
|
||||
width: wire.width,
|
||||
height: wire.height,
|
||||
sha256: wire.sha256,
|
||||
};
|
||||
},
|
||||
|
||||
async custom(name, args, opts) {
|
||||
const wireArgs = CustomArgsSchema.parse({ name, payload: args });
|
||||
// `custom` is a mutation in the rate-limit sense; auto-key for retries.
|
||||
const raw = await request<unknown>(KIND_CUSTOM_V1, 'custom' as StandardOp, wireArgs, opts);
|
||||
const wire = CustomResultSchema.parse(raw);
|
||||
// The result schema is `{ result: unknown }` — the inner `result` is
|
||||
// already validated against the consumer's response schema on the
|
||||
// server side, so we trust it here.
|
||||
return wire.result as never;
|
||||
},
|
||||
|
||||
close() {
|
||||
pending.rejectAll(new Error('FileClient closed'));
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** Drain a stream into a single buffer; used for the inline-write fallback. */
|
||||
async function drainToUint8Array(
|
||||
stream: ReadableStream<Uint8Array>,
|
||||
cap: number,
|
||||
): Promise<Uint8Array> {
|
||||
const reader = stream.getReader();
|
||||
const chunks: Uint8Array[] = [];
|
||||
let total = 0;
|
||||
try {
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (value === undefined) continue;
|
||||
chunks.push(value);
|
||||
total += value.byteLength;
|
||||
if (total > cap) {
|
||||
throw new Error(`stream produced more than declared size cap (${cap})`);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
reader.releaseLock();
|
||||
}
|
||||
const out = new Uint8Array(total);
|
||||
let offset = 0;
|
||||
for (const c of chunks) {
|
||||
out.set(c, offset);
|
||||
offset += c.byteLength;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
91
packages/shade-files/src/client/concurrency.ts
Normal file
91
packages/shade-files/src/client/concurrency.ts
Normal file
@@ -0,0 +1,91 @@
|
||||
/**
|
||||
* Bounded-concurrency async map.
|
||||
*
|
||||
* Pulls items from an `AsyncIterable` source (lazily — never materializing
|
||||
* the whole sequence) and runs `fn` on at most `concurrency` of them at
|
||||
* once. Failures bubble unless `continueOnError` is set, in which case they
|
||||
* are reported via `onError` and the pool keeps draining.
|
||||
*/
|
||||
import { CancelledError } from '../schemas/errors.js';
|
||||
|
||||
export interface ConcurrentMapOptions<T> {
|
||||
concurrency: number;
|
||||
signal?: AbortSignal;
|
||||
continueOnError?: boolean;
|
||||
onError?: (item: T, err: unknown) => void;
|
||||
}
|
||||
|
||||
export async function runWithConcurrency<T>(
|
||||
source: AsyncIterable<T>,
|
||||
fn: (item: T) => Promise<void>,
|
||||
opts: ConcurrentMapOptions<T>,
|
||||
): Promise<void> {
|
||||
if (opts.concurrency < 1) throw new Error('concurrency must be ≥ 1');
|
||||
|
||||
const iter = source[Symbol.asyncIterator]();
|
||||
const inFlight = new Set<Promise<void>>();
|
||||
let firstError: unknown = null;
|
||||
let aborted = false;
|
||||
|
||||
function checkAbort(): void {
|
||||
if (opts.signal?.aborted) {
|
||||
aborted = true;
|
||||
const reason = opts.signal.reason;
|
||||
throw reason instanceof Error ? reason : new CancelledError(String(reason ?? 'aborted'));
|
||||
}
|
||||
}
|
||||
|
||||
async function pumpOne(): Promise<boolean> {
|
||||
checkAbort();
|
||||
const next = await iter.next();
|
||||
if (next.done === true) return false;
|
||||
const item = next.value;
|
||||
const task = (async () => {
|
||||
try {
|
||||
await fn(item);
|
||||
} catch (err) {
|
||||
if (opts.continueOnError === true) {
|
||||
opts.onError?.(item, err);
|
||||
} else if (firstError === null) {
|
||||
firstError = err;
|
||||
}
|
||||
}
|
||||
})().finally(() => {
|
||||
inFlight.delete(task);
|
||||
});
|
||||
inFlight.add(task);
|
||||
return true;
|
||||
}
|
||||
|
||||
try {
|
||||
// Prime the pool
|
||||
while (inFlight.size < opts.concurrency) {
|
||||
if (firstError !== null || aborted) break;
|
||||
const more = await pumpOne();
|
||||
if (!more) break;
|
||||
}
|
||||
// Maintain saturation
|
||||
while (inFlight.size > 0) {
|
||||
if (firstError !== null && opts.continueOnError !== true) break;
|
||||
await Promise.race(inFlight);
|
||||
while (
|
||||
inFlight.size < opts.concurrency &&
|
||||
firstError === null &&
|
||||
!aborted
|
||||
) {
|
||||
const more = await pumpOne();
|
||||
if (!more) break;
|
||||
}
|
||||
}
|
||||
// Drain whatever survived
|
||||
if (inFlight.size > 0) await Promise.allSettled(inFlight);
|
||||
} finally {
|
||||
if (typeof iter.return === 'function') {
|
||||
await iter.return().catch(() => undefined);
|
||||
}
|
||||
}
|
||||
|
||||
if (firstError !== null && opts.continueOnError !== true) {
|
||||
throw firstError;
|
||||
}
|
||||
}
|
||||
97
packages/shade-files/src/client/directory-types.ts
Normal file
97
packages/shade-files/src/client/directory-types.ts
Normal file
@@ -0,0 +1,97 @@
|
||||
/**
|
||||
* Public types for directory operations (`walk`, `uploadDirectory`,
|
||||
* `downloadDirectory`).
|
||||
*
|
||||
* `DirectoryHandleLike` and `FileHandleLike` are structurally compatible
|
||||
* with the browser File System Access API
|
||||
* (`FileSystemDirectoryHandle` / `FileSystemFileHandle`) so that browser
|
||||
* consumers can pass them directly. For Bun/Node, use the adapter from
|
||||
* `node-directory-handle.ts`.
|
||||
*/
|
||||
|
||||
export interface FileHandleLike {
|
||||
readonly kind: 'file';
|
||||
readonly name: string;
|
||||
/** Read-side accessor — used by `uploadDirectory`. */
|
||||
getFile(): Promise<FileLike>;
|
||||
/** Write-side accessor — used by `downloadDirectory`. */
|
||||
createWritable(): Promise<WritableStreamLike>;
|
||||
}
|
||||
|
||||
export interface FileLike {
|
||||
readonly name: string;
|
||||
readonly size: number;
|
||||
readonly type: string;
|
||||
stream(): ReadableStream<Uint8Array>;
|
||||
arrayBuffer(): Promise<ArrayBuffer>;
|
||||
}
|
||||
|
||||
export interface WritableStreamLike {
|
||||
write(chunk: Uint8Array): Promise<void>;
|
||||
close(): Promise<void>;
|
||||
abort(reason?: unknown): Promise<void>;
|
||||
}
|
||||
|
||||
export interface DirectoryHandleLike {
|
||||
readonly kind: 'directory';
|
||||
readonly name: string;
|
||||
/** Yield child entries (file or directory). Used by `uploadDirectory`. */
|
||||
entries(): AsyncIterable<[string, DirectoryHandleLike | FileHandleLike]>;
|
||||
/** Used by `downloadDirectory` to create remote → local mapping. */
|
||||
getDirectoryHandle(name: string, opts?: { create?: boolean }): Promise<DirectoryHandleLike>;
|
||||
getFileHandle(name: string, opts?: { create?: boolean }): Promise<FileHandleLike>;
|
||||
}
|
||||
|
||||
// ─── Bulk transfer types ─────────────────────────────────────
|
||||
|
||||
export type BulkTransferEvent =
|
||||
| { type: 'plan'; totalFiles: number; totalBytes: number | undefined }
|
||||
| { type: 'file-start'; path: string; size: number }
|
||||
| { type: 'file-progress'; path: string; bytesDone: number; bytesTotal: number }
|
||||
| { type: 'file-done'; path: string; bytesDone: number }
|
||||
| { type: 'file-error'; path: string; error: unknown }
|
||||
| {
|
||||
type: 'progress';
|
||||
filesDone: number;
|
||||
filesTotal: number;
|
||||
bytesDone: number;
|
||||
bytesTotal: number | undefined;
|
||||
}
|
||||
| { type: 'complete'; filesDone: number; bytesDone: number; durationMs: number }
|
||||
| { type: 'abort'; reason: string };
|
||||
|
||||
export interface BulkTransferResult {
|
||||
filesDone: number;
|
||||
bytesDone: number;
|
||||
durationMs: number;
|
||||
}
|
||||
|
||||
export interface BulkTransferHandle {
|
||||
/** Async-iterable event stream — plan, per-file events, aggregate progress. */
|
||||
readonly events: AsyncIterable<BulkTransferEvent>;
|
||||
abort(reason?: string): Promise<void>;
|
||||
done(): Promise<BulkTransferResult>;
|
||||
}
|
||||
|
||||
export interface BulkOpts {
|
||||
/**
|
||||
* Max files in flight at once. Default 4. Capped at 16 to bound memory
|
||||
* (each in-flight file may hold a buffered chunk; with 1 MiB chunks +
|
||||
* 16 lanes that's still bounded).
|
||||
*/
|
||||
concurrency?: number;
|
||||
/**
|
||||
* Continue past per-file failures. Default false (fail-fast).
|
||||
* When true, errors are emitted via `file-error` events and the whole
|
||||
* transfer still resolves (with `done()` returning the count of
|
||||
* successful files).
|
||||
*/
|
||||
continueOnError?: boolean;
|
||||
/** Cancellation. */
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
|
||||
/** Hard cap to bound memory regardless of caller-supplied concurrency. */
|
||||
export const MAX_BULK_CONCURRENCY = 16;
|
||||
/** Default concurrency. */
|
||||
export const DEFAULT_BULK_CONCURRENCY = 4;
|
||||
316
packages/shade-files/src/client/download-directory.ts
Normal file
316
packages/shade-files/src/client/download-directory.ts
Normal file
@@ -0,0 +1,316 @@
|
||||
/**
|
||||
* Download an entire remote directory tree to a local `DirectoryHandleLike`.
|
||||
*
|
||||
* Mirror image of `uploadDirectory`: walks the remote tree via the shared
|
||||
* `walk()` helper, creates local directories on the fly, and downloads each
|
||||
* file with `client.read` (which routes inline or streams based on the
|
||||
* server's response). Bounded concurrency keeps RPC inflight count low.
|
||||
*/
|
||||
import { walk, type WalkItem } from './walk.js';
|
||||
import { runWithConcurrency } from './concurrency.js';
|
||||
import type { FileClient } from './client.js';
|
||||
import {
|
||||
DEFAULT_BULK_CONCURRENCY,
|
||||
MAX_BULK_CONCURRENCY,
|
||||
type BulkOpts,
|
||||
type BulkTransferEvent,
|
||||
type BulkTransferHandle,
|
||||
type BulkTransferResult,
|
||||
type DirectoryHandleLike,
|
||||
type FileHandleLike,
|
||||
} from './directory-types.js';
|
||||
import { CancelledError } from '../schemas/errors.js';
|
||||
|
||||
export interface DownloadDirectoryOptions extends BulkOpts {
|
||||
/** Page size hint forwarded to the underlying `walk`. Default 200. */
|
||||
pageSize?: number;
|
||||
/** Skip files already present locally. Default false (overwrite). */
|
||||
skipExisting?: boolean;
|
||||
}
|
||||
|
||||
export function downloadDirectory(
|
||||
client: Pick<FileClient, 'read' | 'list'>,
|
||||
remoteRoot: string,
|
||||
local: DirectoryHandleLike,
|
||||
opts: DownloadDirectoryOptions = {},
|
||||
): BulkTransferHandle {
|
||||
const concurrency = Math.max(
|
||||
1,
|
||||
Math.min(MAX_BULK_CONCURRENCY, opts.concurrency ?? DEFAULT_BULK_CONCURRENCY),
|
||||
);
|
||||
const continueOnError = opts.continueOnError ?? false;
|
||||
const externalSignal = opts.signal;
|
||||
const pageSize = opts.pageSize ?? 200;
|
||||
|
||||
const internalAbort = new AbortController();
|
||||
const combinedSignal = mergeSignals(externalSignal, internalAbort.signal);
|
||||
|
||||
const events: BulkTransferEvent[] = [];
|
||||
const eventResolvers: ((v: IteratorResult<BulkTransferEvent>) => void)[] = [];
|
||||
let eventsClosed = false;
|
||||
|
||||
function emit(event: BulkTransferEvent): void {
|
||||
if (eventsClosed) return;
|
||||
if (eventResolvers.length > 0) {
|
||||
eventResolvers.shift()!({ value: event, done: false });
|
||||
} else {
|
||||
events.push(event);
|
||||
}
|
||||
}
|
||||
|
||||
function closeEvents(): void {
|
||||
if (eventsClosed) return;
|
||||
eventsClosed = true;
|
||||
while (eventResolvers.length > 0) {
|
||||
eventResolvers.shift()!({ value: undefined as never, done: true });
|
||||
}
|
||||
}
|
||||
|
||||
let resolveDone!: (r: BulkTransferResult) => void;
|
||||
let rejectDone!: (err: unknown) => void;
|
||||
const donePromise = new Promise<BulkTransferResult>((resolve, reject) => {
|
||||
resolveDone = resolve;
|
||||
rejectDone = reject;
|
||||
});
|
||||
|
||||
const startedAt = Date.now();
|
||||
let filesDone = 0;
|
||||
let bytesDone = 0;
|
||||
|
||||
void (async () => {
|
||||
try {
|
||||
// 1. Plan: walk the remote tree into [dirs, files]
|
||||
const dirItems: WalkItem[] = [];
|
||||
const fileItems: WalkItem[] = [];
|
||||
try {
|
||||
for await (const item of walk(client, remoteRoot, {
|
||||
pageSize,
|
||||
...(combinedSignal !== undefined ? { signal: combinedSignal } : {}),
|
||||
})) {
|
||||
if (item.entry.kind === 'dir') dirItems.push(item);
|
||||
else fileItems.push(item);
|
||||
}
|
||||
} catch (err) {
|
||||
emit({ type: 'abort', reason: errMsg(err) });
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
return;
|
||||
}
|
||||
|
||||
const totalFiles = fileItems.length;
|
||||
const bytesTotal = fileItems.reduce((acc, f) => acc + f.entry.size, 0);
|
||||
emit({ type: 'plan', totalFiles, totalBytes: bytesTotal });
|
||||
|
||||
if (combinedSignal.aborted) {
|
||||
emit({ type: 'abort', reason: errMsg(combinedSignal.reason) });
|
||||
closeEvents();
|
||||
rejectDone(combinedSignal.reason ?? new CancelledError());
|
||||
return;
|
||||
}
|
||||
|
||||
// 2. Pre-create local directories sequentially.
|
||||
for (const d of dirItems) {
|
||||
if (combinedSignal.aborted) break;
|
||||
try {
|
||||
await ensureLocalDir(local, d.relativePath);
|
||||
} catch (err) {
|
||||
if (!continueOnError) {
|
||||
emit({ type: 'abort', reason: errMsg(err) });
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Download files with bounded concurrency.
|
||||
try {
|
||||
await runWithConcurrency(
|
||||
asyncIterableOf(fileItems),
|
||||
async (item) => {
|
||||
if (combinedSignal.aborted) {
|
||||
throw new CancelledError('download aborted');
|
||||
}
|
||||
emit({ type: 'file-start', path: item.relativePath, size: item.entry.size });
|
||||
try {
|
||||
const fileHandle = await ensureLocalFile(local, item.relativePath);
|
||||
if (opts.skipExisting === true) {
|
||||
// Skip if file already exists with non-zero size
|
||||
try {
|
||||
const existing = await fileHandle.getFile();
|
||||
if (existing.size === item.entry.size) {
|
||||
filesDone++;
|
||||
bytesDone += existing.size;
|
||||
emit({ type: 'file-done', path: item.relativePath, bytesDone: existing.size });
|
||||
emit({
|
||||
type: 'progress',
|
||||
filesDone,
|
||||
filesTotal: totalFiles,
|
||||
bytesDone,
|
||||
bytesTotal,
|
||||
});
|
||||
return;
|
||||
}
|
||||
} catch {
|
||||
/* not present yet — fall through to download */
|
||||
}
|
||||
}
|
||||
const writable = await fileHandle.createWritable();
|
||||
try {
|
||||
const result = await client.read(item.absolutePath, {
|
||||
signal: combinedSignal,
|
||||
});
|
||||
if (result.kind === 'inline') {
|
||||
if (result.bytes.byteLength > 0) await writable.write(result.bytes);
|
||||
await writable.close();
|
||||
filesDone++;
|
||||
bytesDone += result.bytes.byteLength;
|
||||
emit({ type: 'file-done', path: item.relativePath, bytesDone: result.bytes.byteLength });
|
||||
} else {
|
||||
await pipeReadableToWritable(result.stream, writable);
|
||||
await writable.close();
|
||||
await result.done();
|
||||
filesDone++;
|
||||
bytesDone += result.size;
|
||||
emit({ type: 'file-done', path: item.relativePath, bytesDone: result.size });
|
||||
}
|
||||
emit({
|
||||
type: 'progress',
|
||||
filesDone,
|
||||
filesTotal: totalFiles,
|
||||
bytesDone,
|
||||
bytesTotal,
|
||||
});
|
||||
} catch (err) {
|
||||
await writable.abort(err).catch(() => undefined);
|
||||
throw err;
|
||||
}
|
||||
} catch (err) {
|
||||
emit({ type: 'file-error', path: item.relativePath, error: err });
|
||||
throw err;
|
||||
}
|
||||
},
|
||||
{
|
||||
concurrency,
|
||||
signal: combinedSignal,
|
||||
continueOnError,
|
||||
onError: () => {
|
||||
/* already emitted as 'file-error' */
|
||||
},
|
||||
},
|
||||
);
|
||||
} catch (err) {
|
||||
emit({ type: 'abort', reason: errMsg(err) });
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
return;
|
||||
}
|
||||
|
||||
const durationMs = Date.now() - startedAt;
|
||||
emit({ type: 'complete', filesDone, bytesDone, durationMs });
|
||||
closeEvents();
|
||||
resolveDone({ filesDone, bytesDone, durationMs });
|
||||
} catch (err) {
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
}
|
||||
})();
|
||||
|
||||
donePromise.catch(() => {
|
||||
/* deliberate */
|
||||
});
|
||||
|
||||
return {
|
||||
events: {
|
||||
[Symbol.asyncIterator]() {
|
||||
return {
|
||||
next(): Promise<IteratorResult<BulkTransferEvent>> {
|
||||
if (events.length > 0) {
|
||||
return Promise.resolve({ value: events.shift()!, done: false });
|
||||
}
|
||||
if (eventsClosed) {
|
||||
return Promise.resolve({ value: undefined as never, done: true });
|
||||
}
|
||||
return new Promise((resolve) => eventResolvers.push(resolve));
|
||||
},
|
||||
return(): Promise<IteratorResult<BulkTransferEvent>> {
|
||||
return Promise.resolve({ value: undefined as never, done: true });
|
||||
},
|
||||
};
|
||||
},
|
||||
},
|
||||
async abort(reason) {
|
||||
internalAbort.abort(new CancelledError(reason ?? 'manual abort'));
|
||||
},
|
||||
done: () => donePromise,
|
||||
};
|
||||
}
|
||||
|
||||
// ─── Helpers ─────────────────────────────────────────────────
|
||||
|
||||
async function ensureLocalDir(
|
||||
root: DirectoryHandleLike,
|
||||
relativePath: string,
|
||||
): Promise<DirectoryHandleLike> {
|
||||
const segments = relativePath.split('/').filter((s) => s !== '');
|
||||
let current = root;
|
||||
for (const seg of segments) {
|
||||
current = await current.getDirectoryHandle(seg, { create: true });
|
||||
}
|
||||
return current;
|
||||
}
|
||||
|
||||
async function ensureLocalFile(
|
||||
root: DirectoryHandleLike,
|
||||
relativePath: string,
|
||||
): Promise<FileHandleLike> {
|
||||
const segments = relativePath.split('/').filter((s) => s !== '');
|
||||
if (segments.length === 0) throw new Error('empty file path');
|
||||
let current = root;
|
||||
for (let i = 0; i < segments.length - 1; i++) {
|
||||
current = await current.getDirectoryHandle(segments[i]!, { create: true });
|
||||
}
|
||||
return await current.getFileHandle(segments[segments.length - 1]!, { create: true });
|
||||
}
|
||||
|
||||
async function pipeReadableToWritable(
|
||||
readable: ReadableStream<Uint8Array>,
|
||||
writable: { write(chunk: Uint8Array): Promise<void> },
|
||||
): Promise<void> {
|
||||
const reader = readable.getReader();
|
||||
try {
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (value !== undefined && value.byteLength > 0) {
|
||||
await writable.write(value);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
reader.releaseLock();
|
||||
}
|
||||
}
|
||||
|
||||
async function* asyncIterableOf<T>(arr: T[]): AsyncIterable<T> {
|
||||
for (const item of arr) yield item;
|
||||
}
|
||||
|
||||
function errMsg(err: unknown): string {
|
||||
if (err instanceof Error) return err.message;
|
||||
return String(err ?? 'unknown error');
|
||||
}
|
||||
|
||||
function mergeSignals(
|
||||
external: AbortSignal | undefined,
|
||||
internal: AbortSignal,
|
||||
): AbortSignal {
|
||||
if (external === undefined) return internal;
|
||||
if (external.aborted) return external;
|
||||
const controller = new AbortController();
|
||||
const onAbort = (sig: AbortSignal): void => {
|
||||
controller.abort(sig.reason);
|
||||
};
|
||||
external.addEventListener('abort', () => onAbort(external), { once: true });
|
||||
internal.addEventListener('abort', () => onAbort(internal), { once: true });
|
||||
return controller.signal;
|
||||
}
|
||||
218
packages/shade-files/src/client/inline-threshold.ts
Normal file
218
packages/shade-files/src/client/inline-threshold.ts
Normal file
@@ -0,0 +1,218 @@
|
||||
/**
|
||||
* Decide whether a write should travel inline (base64 inside the RPC
|
||||
* envelope) or via a dedicated `@shade/transfer` stream.
|
||||
*
|
||||
* Threshold: 256 KiB plaintext. Anything strictly above goes through the
|
||||
* stream path; anything ≤ goes inline.
|
||||
*
|
||||
* For known-size inputs (`Uint8Array`, `Blob`, `File`) the decision is
|
||||
* cheap: compare `byteLength`/`size` against the threshold.
|
||||
*
|
||||
* For `ReadableStream` we cannot know the size up front. We pull chunks
|
||||
* into a temporary buffer until we either:
|
||||
* - hit EOF before the threshold → inline (we have all the bytes)
|
||||
* - cross the threshold → streams (the buffered prefix + the rest of the
|
||||
* stream are returned via a fresh `ReadableStream` so the caller can
|
||||
* feed it to `shade.upload`).
|
||||
*/
|
||||
/** Plaintext size at which inline transitions to streams. */
|
||||
export const INLINE_THRESHOLD = 256 * 1024;
|
||||
|
||||
export type InlineDecision =
|
||||
| {
|
||||
kind: 'inline';
|
||||
bytes: Uint8Array;
|
||||
contentType?: string;
|
||||
}
|
||||
| {
|
||||
kind: 'streams';
|
||||
stream: ReadableStream<Uint8Array>;
|
||||
/** Plaintext size when known (Blob/File). undefined for raw streams. */
|
||||
size?: number;
|
||||
contentType?: string;
|
||||
};
|
||||
|
||||
/**
|
||||
* Public input shape for `FileClient.write`. Runtime discriminator helper.
|
||||
*/
|
||||
export type WriteSource =
|
||||
| Uint8Array
|
||||
| Blob
|
||||
| File
|
||||
| ReadableStream<Uint8Array>
|
||||
| { stream: ReadableStream<Uint8Array>; size: number; contentType?: string };
|
||||
|
||||
export async function decideInline(input: WriteSource): Promise<InlineDecision> {
|
||||
// 1. Uint8Array — direct size check
|
||||
if (input instanceof Uint8Array) {
|
||||
if (input.byteLength <= INLINE_THRESHOLD) {
|
||||
return { kind: 'inline', bytes: input };
|
||||
}
|
||||
return {
|
||||
kind: 'streams',
|
||||
stream: uint8ArrayToStream(input),
|
||||
size: input.byteLength,
|
||||
};
|
||||
}
|
||||
|
||||
// 2. Blob / File — known size (Blob.type is a string; File extends Blob)
|
||||
if (typeof Blob !== 'undefined' && input instanceof Blob) {
|
||||
const blob: Blob = input;
|
||||
const contentType = blob.type === '' ? undefined : blob.type;
|
||||
if (blob.size <= INLINE_THRESHOLD) {
|
||||
const bytes = new Uint8Array(await blob.arrayBuffer());
|
||||
return contentType !== undefined
|
||||
? { kind: 'inline', bytes, contentType }
|
||||
: { kind: 'inline', bytes };
|
||||
}
|
||||
return contentType !== undefined
|
||||
? { kind: 'streams', stream: blob.stream(), size: blob.size, contentType }
|
||||
: { kind: 'streams', stream: blob.stream(), size: blob.size };
|
||||
}
|
||||
|
||||
// 3. Pre-wrapped { stream, size } — trust caller's declared size
|
||||
if (
|
||||
typeof input === 'object' &&
|
||||
input !== null &&
|
||||
'stream' in input &&
|
||||
'size' in input &&
|
||||
(input as { stream: unknown }).stream instanceof ReadableStream
|
||||
) {
|
||||
const wrapped = input as { stream: ReadableStream<Uint8Array>; size: number; contentType?: string };
|
||||
if (wrapped.size <= INLINE_THRESHOLD) {
|
||||
const bytes = await drainStream(wrapped.stream, wrapped.size);
|
||||
const contentType = wrapped.contentType;
|
||||
return contentType !== undefined
|
||||
? { kind: 'inline', bytes, contentType }
|
||||
: { kind: 'inline', bytes };
|
||||
}
|
||||
const contentType = wrapped.contentType;
|
||||
return contentType !== undefined
|
||||
? { kind: 'streams', stream: wrapped.stream, size: wrapped.size, contentType }
|
||||
: { kind: 'streams', stream: wrapped.stream, size: wrapped.size };
|
||||
}
|
||||
|
||||
// 4. Bare ReadableStream — peek until threshold or EOF
|
||||
if (input instanceof ReadableStream) {
|
||||
return await peekStream(input);
|
||||
}
|
||||
|
||||
throw new TypeError(
|
||||
`decideInline: unsupported input type ${Object.prototype.toString.call(input)}`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Drain a stream into a Uint8Array, with a soft cap at `expected` + slack.
|
||||
* Used for the inline path when the caller declared a size.
|
||||
*/
|
||||
async function drainStream(
|
||||
stream: ReadableStream<Uint8Array>,
|
||||
expected: number,
|
||||
): Promise<Uint8Array> {
|
||||
const reader = stream.getReader();
|
||||
const chunks: Uint8Array[] = [];
|
||||
let total = 0;
|
||||
try {
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (value === undefined) continue;
|
||||
chunks.push(value);
|
||||
total += value.byteLength;
|
||||
if (total > expected + INLINE_THRESHOLD) {
|
||||
throw new Error(
|
||||
`decideInline: stream produced more bytes (${total}) than declared size (${expected})`,
|
||||
);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
reader.releaseLock();
|
||||
}
|
||||
return concat(chunks, total);
|
||||
}
|
||||
|
||||
/**
|
||||
* Peek a `ReadableStream` of unknown length: buffer up to `INLINE_THRESHOLD + 1`
|
||||
* bytes. If EOF first, return inline. Otherwise reconstruct a stream that
|
||||
* yields the buffered prefix followed by the remainder.
|
||||
*/
|
||||
async function peekStream(stream: ReadableStream<Uint8Array>): Promise<InlineDecision> {
|
||||
const reader = stream.getReader();
|
||||
const buffered: Uint8Array[] = [];
|
||||
let total = 0;
|
||||
try {
|
||||
while (total <= INLINE_THRESHOLD) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) {
|
||||
reader.releaseLock();
|
||||
return { kind: 'inline', bytes: concat(buffered, total) };
|
||||
}
|
||||
if (value === undefined) continue;
|
||||
buffered.push(value);
|
||||
total += value.byteLength;
|
||||
}
|
||||
// We have at least INLINE_THRESHOLD + 1 bytes buffered. Promote to streams.
|
||||
const reconstructed = reconstructStream(buffered, reader);
|
||||
return { kind: 'streams', stream: reconstructed };
|
||||
} catch (err) {
|
||||
reader.releaseLock();
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
interface MinimalReader {
|
||||
read(): Promise<{ value: Uint8Array | undefined; done: boolean }>;
|
||||
cancel(reason?: unknown): Promise<void>;
|
||||
releaseLock(): void;
|
||||
}
|
||||
|
||||
function reconstructStream(
|
||||
prefix: Uint8Array[],
|
||||
reader: MinimalReader,
|
||||
): ReadableStream<Uint8Array> {
|
||||
let prefixIdx = 0;
|
||||
return new ReadableStream<Uint8Array>({
|
||||
async pull(controller) {
|
||||
if (prefixIdx < prefix.length) {
|
||||
controller.enqueue(prefix[prefixIdx]!);
|
||||
prefixIdx++;
|
||||
return;
|
||||
}
|
||||
const { value, done } = await reader.read();
|
||||
if (done) {
|
||||
controller.close();
|
||||
reader.releaseLock();
|
||||
return;
|
||||
}
|
||||
if (value !== undefined) controller.enqueue(value);
|
||||
},
|
||||
async cancel(reason) {
|
||||
try {
|
||||
await reader.cancel(reason);
|
||||
} finally {
|
||||
reader.releaseLock();
|
||||
}
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function uint8ArrayToStream(bytes: Uint8Array): ReadableStream<Uint8Array> {
|
||||
return new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
controller.enqueue(bytes);
|
||||
controller.close();
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function concat(chunks: Uint8Array[], total: number): Uint8Array {
|
||||
const out = new Uint8Array(total);
|
||||
let offset = 0;
|
||||
for (const c of chunks) {
|
||||
out.set(c, offset);
|
||||
offset += c.byteLength;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
145
packages/shade-files/src/client/memory-directory.ts
Normal file
145
packages/shade-files/src/client/memory-directory.ts
Normal file
@@ -0,0 +1,145 @@
|
||||
/**
|
||||
* In-memory `DirectoryHandleLike` implementation.
|
||||
*
|
||||
* Useful for tests and Node/Bun environments without the browser File
|
||||
* System Access API. The shape is intentionally compatible with browser
|
||||
* `FileSystemDirectoryHandle` so consumers can swap implementations.
|
||||
*/
|
||||
import type {
|
||||
DirectoryHandleLike,
|
||||
FileHandleLike,
|
||||
FileLike,
|
||||
WritableStreamLike,
|
||||
} from './directory-types.js';
|
||||
|
||||
interface MemoryNode {
|
||||
bytes: Uint8Array;
|
||||
type: string;
|
||||
}
|
||||
|
||||
class MemoryFileLike implements FileLike {
|
||||
constructor(
|
||||
public readonly name: string,
|
||||
private readonly node: MemoryNode,
|
||||
) {}
|
||||
get size(): number { return this.node.bytes.byteLength; }
|
||||
get type(): string { return this.node.type; }
|
||||
stream(): ReadableStream<Uint8Array> {
|
||||
const bytes = this.node.bytes;
|
||||
return new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
if (bytes.byteLength > 0) controller.enqueue(bytes);
|
||||
controller.close();
|
||||
},
|
||||
});
|
||||
}
|
||||
async arrayBuffer(): Promise<ArrayBuffer> {
|
||||
const out = new ArrayBuffer(this.node.bytes.byteLength);
|
||||
new Uint8Array(out).set(this.node.bytes);
|
||||
return out;
|
||||
}
|
||||
}
|
||||
|
||||
class MemoryWritable implements WritableStreamLike {
|
||||
private chunks: Uint8Array[] = [];
|
||||
constructor(private readonly file: MemoryFile) {}
|
||||
async write(chunk: Uint8Array): Promise<void> {
|
||||
this.chunks.push(chunk);
|
||||
}
|
||||
async close(): Promise<void> {
|
||||
let total = 0;
|
||||
for (const c of this.chunks) total += c.byteLength;
|
||||
const out = new Uint8Array(total);
|
||||
let offset = 0;
|
||||
for (const c of this.chunks) {
|
||||
out.set(c, offset);
|
||||
offset += c.byteLength;
|
||||
}
|
||||
this.file.commit(out);
|
||||
}
|
||||
async abort(): Promise<void> {
|
||||
this.chunks = [];
|
||||
}
|
||||
}
|
||||
|
||||
class MemoryFile implements FileHandleLike {
|
||||
readonly kind: 'file' = 'file';
|
||||
constructor(
|
||||
public readonly name: string,
|
||||
public readonly node: MemoryNode,
|
||||
) {}
|
||||
async getFile(): Promise<FileLike> {
|
||||
return new MemoryFileLike(this.name, this.node);
|
||||
}
|
||||
async createWritable(): Promise<WritableStreamLike> {
|
||||
return new MemoryWritable(this);
|
||||
}
|
||||
/** Internal — used by `MemoryWritable.close()`. */
|
||||
commit(bytes: Uint8Array): void {
|
||||
this.node.bytes = bytes;
|
||||
}
|
||||
}
|
||||
|
||||
class MemoryDirectory implements DirectoryHandleLike {
|
||||
readonly kind: 'directory' = 'directory';
|
||||
private readonly children = new Map<string, MemoryDirectory | MemoryFile>();
|
||||
constructor(public readonly name: string) {}
|
||||
|
||||
async *entries(): AsyncIterable<[string, DirectoryHandleLike | FileHandleLike]> {
|
||||
for (const [name, child] of this.children) {
|
||||
yield [name, child];
|
||||
}
|
||||
}
|
||||
|
||||
async getDirectoryHandle(name: string, opts?: { create?: boolean }): Promise<DirectoryHandleLike> {
|
||||
const existing = this.children.get(name);
|
||||
if (existing !== undefined) {
|
||||
if (existing.kind !== 'directory') {
|
||||
throw new Error(`'${name}' exists but is a file`);
|
||||
}
|
||||
return existing;
|
||||
}
|
||||
if (opts?.create !== true) throw new Error(`directory not found: ${name}`);
|
||||
const dir = new MemoryDirectory(name);
|
||||
this.children.set(name, dir);
|
||||
return dir;
|
||||
}
|
||||
|
||||
async getFileHandle(name: string, opts?: { create?: boolean }): Promise<FileHandleLike> {
|
||||
const existing = this.children.get(name);
|
||||
if (existing !== undefined) {
|
||||
if (existing.kind !== 'file') {
|
||||
throw new Error(`'${name}' exists but is a directory`);
|
||||
}
|
||||
return existing;
|
||||
}
|
||||
if (opts?.create !== true) throw new Error(`file not found: ${name}`);
|
||||
const node: MemoryNode = { bytes: new Uint8Array(0), type: '' };
|
||||
const file = new MemoryFile(name, node);
|
||||
this.children.set(name, file);
|
||||
return file;
|
||||
}
|
||||
|
||||
/** Test helper: synchronously add a file with given bytes. */
|
||||
addFile(name: string, bytes: Uint8Array, type: string = ''): MemoryFile {
|
||||
const node: MemoryNode = { bytes, type };
|
||||
const file = new MemoryFile(name, node);
|
||||
this.children.set(name, file);
|
||||
return file;
|
||||
}
|
||||
|
||||
/** Test helper: synchronously add a subdirectory. */
|
||||
addDir(name: string): MemoryDirectory {
|
||||
const dir = new MemoryDirectory(name);
|
||||
this.children.set(name, dir);
|
||||
return dir;
|
||||
}
|
||||
}
|
||||
|
||||
/** Construct an empty in-memory directory tree. */
|
||||
export function createMemoryDirectory(name: string = ''): DirectoryHandleLike & {
|
||||
addFile(name: string, bytes: Uint8Array, type?: string): FileHandleLike;
|
||||
addDir(name: string): DirectoryHandleLike;
|
||||
} {
|
||||
return new MemoryDirectory(name);
|
||||
}
|
||||
251
packages/shade-files/src/client/streams-bridge.ts
Normal file
251
packages/shade-files/src/client/streams-bridge.ts
Normal file
@@ -0,0 +1,251 @@
|
||||
/**
|
||||
* Client-side bridge between `@shade/files` content RPC ops and the
|
||||
* `@shade/transfer` engine.
|
||||
*
|
||||
* Two responsibilities, mirror-image of `server/streams-bridge.ts`:
|
||||
* 1. **Outbound writes (client → server, > 256 KiB).** When `FileClient.write`
|
||||
* promotes to the streams path, this bridge calls `shade.upload(...)`
|
||||
* with `userMetadata.shadeFilesWriteId = <id>` so the server can
|
||||
* correlate the inbound transfer with the parallel `write` RPC.
|
||||
*
|
||||
* 2. **Inbound reads (server → client, > 256 KiB).** When `FileClient.read`
|
||||
* gets a `{ kind: 'streams', streamId, ... }` RPC response, it asks
|
||||
* this bridge for the matching incoming transfer's plaintext stream.
|
||||
* The bridge subscribes to `shade.onIncomingTransfer` once and, on
|
||||
* each transfer tagged with `userMetadata.shadeFilesReadStreamId`,
|
||||
* **immediately** calls `accept(...)` (the engine rejects chunks that
|
||||
* arrive before accept), pipes plaintext into a TransformStream, and
|
||||
* parks the readable side until the matching read RPC awaits it.
|
||||
*/
|
||||
import type { TransferHandle, TransferProgress } from '@shade/transfer';
|
||||
import { generateRequestId } from '../protocol/correlate.js';
|
||||
import { OperationTimeoutError } from '../schemas/errors.js';
|
||||
import {
|
||||
META_KEY_READ_STREAM_ID,
|
||||
META_KEY_WRITE_ID,
|
||||
type StreamsBridgeShade,
|
||||
} from '../server/streams-bridge.js';
|
||||
|
||||
export interface AwaitReadOptions {
|
||||
/** Sender address — must match `incoming.from` for delivery. */
|
||||
expectedFrom: string;
|
||||
signal: AbortSignal;
|
||||
/** Hard deadline (ms from now). Default 60_000. */
|
||||
timeoutMs?: number;
|
||||
}
|
||||
|
||||
export interface ParkedRead {
|
||||
from: string;
|
||||
/** Plaintext stream. The bridge already accepted the transfer. */
|
||||
readable: ReadableStream<Uint8Array>;
|
||||
/** Resolves when the transfer fully completes (verified sha256 available). */
|
||||
done: Promise<{ sha256: string; bytesSent: number }>;
|
||||
/** Underlying transfer handle — for abort propagation. */
|
||||
handle: TransferHandle;
|
||||
arrivedAt: number;
|
||||
}
|
||||
|
||||
export interface ClientStreamsBridge {
|
||||
/**
|
||||
* Generate a fresh writeId, kick `shade.upload(...)` to `peer` with that
|
||||
* id stamped in `userMetadata`, and return both the id (for the parallel
|
||||
* RPC envelope) and the transfer handle (for `done()`/`abort()`).
|
||||
*/
|
||||
initiateWrite(opts: {
|
||||
peer: string;
|
||||
stream: ReadableStream<Uint8Array>;
|
||||
size: number;
|
||||
contentType?: string;
|
||||
name?: string;
|
||||
signal?: AbortSignal;
|
||||
onProgress?: (p: TransferProgress) => void;
|
||||
}): Promise<{ writeId: string; handle: TransferHandle }>;
|
||||
|
||||
/**
|
||||
* Wait for an inbound transfer carrying `userMetadata.shadeFilesReadStreamId
|
||||
* === streamId` from `expectedFrom`. Resolves with the parked entry whose
|
||||
* `readable` can be consumed by the caller.
|
||||
*/
|
||||
awaitRead(streamId: string, opts: AwaitReadOptions): Promise<ParkedRead>;
|
||||
|
||||
destroy(): Promise<void>;
|
||||
}
|
||||
|
||||
interface PendingReadWaiter {
|
||||
resolve: (parked: ParkedRead) => void;
|
||||
reject: (err: unknown) => void;
|
||||
expectedFrom: string;
|
||||
timer: ReturnType<typeof setTimeout> | null;
|
||||
abortListener: (() => void) | null;
|
||||
signal: AbortSignal;
|
||||
}
|
||||
|
||||
export interface CreateClientStreamsBridgeOptions {
|
||||
/** Default deadline for `awaitRead` if the caller doesn't supply one. */
|
||||
defaultAwaitReadTimeoutMs?: number;
|
||||
/** How long to retain a parked transfer waiting for its RPC. Default 60_000. */
|
||||
parkedReadTtlMs?: number;
|
||||
}
|
||||
|
||||
export async function createClientStreamsBridge(
|
||||
shade: StreamsBridgeShade,
|
||||
options: CreateClientStreamsBridgeOptions = {},
|
||||
): Promise<ClientStreamsBridge> {
|
||||
const parkedReadTtlMs = options.parkedReadTtlMs ?? 60_000;
|
||||
const defaultAwaitTimeoutMs = options.defaultAwaitReadTimeoutMs ?? 60_000;
|
||||
|
||||
const parked = new Map<string, ParkedRead>();
|
||||
const waiters = new Map<string, PendingReadWaiter>();
|
||||
let destroyed = false;
|
||||
|
||||
const unsubscribe = await shade.onIncomingTransfer(async (incoming) => {
|
||||
const readStreamId = incoming.metadata.userMetadata?.[META_KEY_READ_STREAM_ID];
|
||||
if (readStreamId === undefined) return;
|
||||
|
||||
const ts = new TransformStream<Uint8Array, Uint8Array>();
|
||||
let handle: TransferHandle;
|
||||
try {
|
||||
handle = await incoming.accept({
|
||||
output: { kind: 'pipe', pipeTo: ts.writable },
|
||||
});
|
||||
} catch (err) {
|
||||
console.error('[shade-files client streams-bridge] accept failed:', err);
|
||||
return;
|
||||
}
|
||||
|
||||
const arrival: ParkedRead = {
|
||||
from: incoming.from,
|
||||
readable: ts.readable,
|
||||
done: handle.done().then((r) => ({ sha256: r.sha256, bytesSent: r.bytesSent })),
|
||||
handle,
|
||||
arrivedAt: Date.now(),
|
||||
};
|
||||
arrival.done.catch(() => {
|
||||
/* swallow until consumer awaits */
|
||||
});
|
||||
|
||||
const waiter = waiters.get(readStreamId);
|
||||
if (waiter !== undefined) {
|
||||
waiters.delete(readStreamId);
|
||||
cleanupWaiter(waiter);
|
||||
if (incoming.from !== waiter.expectedFrom) {
|
||||
void handle.abort('sender-mismatch').catch(() => undefined);
|
||||
waiter.reject(
|
||||
new Error(
|
||||
`streams-bridge: readStreamId=${readStreamId} delivered by ${incoming.from}, expected ${waiter.expectedFrom}`,
|
||||
),
|
||||
);
|
||||
return;
|
||||
}
|
||||
waiter.resolve(arrival);
|
||||
return;
|
||||
}
|
||||
|
||||
parked.set(readStreamId, arrival);
|
||||
setTimeout(() => {
|
||||
const stale = parked.get(readStreamId);
|
||||
if (stale === arrival) {
|
||||
parked.delete(readStreamId);
|
||||
void handle.abort('rpc-timeout').catch(() => undefined);
|
||||
}
|
||||
}, parkedReadTtlMs).unref?.();
|
||||
});
|
||||
|
||||
function cleanupWaiter(w: PendingReadWaiter): void {
|
||||
if (w.timer !== null) clearTimeout(w.timer);
|
||||
if (w.abortListener !== null) {
|
||||
w.signal.removeEventListener('abort', w.abortListener);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
async initiateWrite(opts) {
|
||||
if (destroyed) throw new Error('streams-bridge: destroyed');
|
||||
const writeId = generateRequestId();
|
||||
const transferOpts: import('@shade/transfer').TransferOptions = {
|
||||
to: opts.peer,
|
||||
input: opts.stream,
|
||||
metadata: {
|
||||
...(opts.name !== undefined ? { name: opts.name } : {}),
|
||||
...(opts.contentType !== undefined ? { contentType: opts.contentType } : {}),
|
||||
sizeBytes: opts.size,
|
||||
userMetadata: { [META_KEY_WRITE_ID]: writeId },
|
||||
},
|
||||
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
|
||||
...(opts.onProgress !== undefined ? { onProgress: opts.onProgress } : {}),
|
||||
};
|
||||
const handle = await shade.upload(transferOpts);
|
||||
return { writeId, handle };
|
||||
},
|
||||
|
||||
async awaitRead(streamId, opts) {
|
||||
if (destroyed) throw new Error('streams-bridge: destroyed');
|
||||
const ready = parked.get(streamId);
|
||||
if (ready !== undefined) {
|
||||
parked.delete(streamId);
|
||||
if (ready.from !== opts.expectedFrom) {
|
||||
void ready.handle.abort('sender-mismatch').catch(() => undefined);
|
||||
throw new Error(
|
||||
`streams-bridge: readStreamId=${streamId} delivered by ${ready.from}, expected ${opts.expectedFrom}`,
|
||||
);
|
||||
}
|
||||
return ready;
|
||||
}
|
||||
if (waiters.has(streamId)) {
|
||||
throw new Error(`streams-bridge: readStreamId=${streamId} already awaited`);
|
||||
}
|
||||
const timeoutMs = opts.timeoutMs ?? defaultAwaitTimeoutMs;
|
||||
return await new Promise<ParkedRead>((resolve, reject) => {
|
||||
const w: PendingReadWaiter = {
|
||||
resolve,
|
||||
reject,
|
||||
expectedFrom: opts.expectedFrom,
|
||||
timer: null,
|
||||
abortListener: null,
|
||||
signal: opts.signal,
|
||||
};
|
||||
w.timer = setTimeout(() => {
|
||||
if (waiters.get(streamId) === w) {
|
||||
waiters.delete(streamId);
|
||||
cleanupWaiter(w);
|
||||
reject(new OperationTimeoutError(`streams-bridge: readStreamId=${streamId} timed out after ${timeoutMs}ms`));
|
||||
}
|
||||
}, timeoutMs);
|
||||
if (opts.signal.aborted) {
|
||||
cleanupWaiter(w);
|
||||
reject(opts.signal.reason ?? new Error('aborted before await'));
|
||||
return;
|
||||
}
|
||||
const onAbort = (): void => {
|
||||
if (waiters.get(streamId) === w) {
|
||||
waiters.delete(streamId);
|
||||
cleanupWaiter(w);
|
||||
reject(opts.signal.reason ?? new Error('aborted'));
|
||||
}
|
||||
};
|
||||
w.abortListener = onAbort;
|
||||
opts.signal.addEventListener('abort', onAbort, { once: true });
|
||||
waiters.set(streamId, w);
|
||||
});
|
||||
},
|
||||
|
||||
async destroy() {
|
||||
if (destroyed) return;
|
||||
destroyed = true;
|
||||
unsubscribe();
|
||||
for (const w of waiters.values()) {
|
||||
cleanupWaiter(w);
|
||||
w.reject(new Error('streams-bridge: destroyed'));
|
||||
}
|
||||
waiters.clear();
|
||||
for (const p of parked.values()) {
|
||||
try {
|
||||
await p.handle.abort('bridge-destroyed');
|
||||
} catch {
|
||||
/* swallow */
|
||||
}
|
||||
}
|
||||
parked.clear();
|
||||
},
|
||||
};
|
||||
}
|
||||
310
packages/shade-files/src/client/upload-directory.ts
Normal file
310
packages/shade-files/src/client/upload-directory.ts
Normal file
@@ -0,0 +1,310 @@
|
||||
/**
|
||||
* Upload an entire local directory tree to a remote peer via a `FileClient`.
|
||||
*
|
||||
* Walks the local `DirectoryHandleLike` lazily, creates remote directories
|
||||
* via `client.mkdir({ recursive: true })`, and uploads each file with
|
||||
* `client.write` (which routes inline or streams based on size). A bounded
|
||||
* concurrency pool keeps memory and inflight RPCs in check.
|
||||
*
|
||||
* Returns a `BulkTransferHandle` whose `events` stream emits `'plan'`,
|
||||
* `'file-start'`, `'file-progress'` (currently emitted at file start +
|
||||
* end), `'file-done'` / `'file-error'`, aggregate `'progress'`, and a
|
||||
* final `'complete'` (or `'abort'`).
|
||||
*/
|
||||
import { posixJoin } from '../utils/path.js';
|
||||
import { runWithConcurrency } from './concurrency.js';
|
||||
import type { FileClient } from './client.js';
|
||||
import {
|
||||
DEFAULT_BULK_CONCURRENCY,
|
||||
MAX_BULK_CONCURRENCY,
|
||||
type BulkOpts,
|
||||
type BulkTransferEvent,
|
||||
type BulkTransferHandle,
|
||||
type BulkTransferResult,
|
||||
type DirectoryHandleLike,
|
||||
type FileHandleLike,
|
||||
} from './directory-types.js';
|
||||
import { CancelledError } from '../schemas/errors.js';
|
||||
|
||||
interface PlannedFile {
|
||||
/** Local file handle. */
|
||||
handle: FileHandleLike;
|
||||
/** Path relative to the upload root. */
|
||||
relativePath: string;
|
||||
/** Absolute remote path. */
|
||||
remoteAbsPath: string;
|
||||
}
|
||||
|
||||
interface PlannedDir {
|
||||
remoteAbsPath: string;
|
||||
}
|
||||
|
||||
export interface UploadDirectoryOptions extends BulkOpts {
|
||||
/**
|
||||
* Pre-create remote directories before uploading files. Default true.
|
||||
* Disable if the server-side `write` already mkdir-p's parents.
|
||||
*/
|
||||
precreateDirs?: boolean;
|
||||
}
|
||||
|
||||
export function uploadDirectory(
|
||||
client: Pick<FileClient, 'write' | 'mkdir'>,
|
||||
local: DirectoryHandleLike,
|
||||
remoteRoot: string,
|
||||
opts: UploadDirectoryOptions = {},
|
||||
): BulkTransferHandle {
|
||||
const concurrency = Math.max(
|
||||
1,
|
||||
Math.min(MAX_BULK_CONCURRENCY, opts.concurrency ?? DEFAULT_BULK_CONCURRENCY),
|
||||
);
|
||||
const precreateDirs = opts.precreateDirs ?? true;
|
||||
const continueOnError = opts.continueOnError ?? false;
|
||||
const externalSignal = opts.signal;
|
||||
|
||||
const internalAbort = new AbortController();
|
||||
const combinedSignal = mergeSignals(externalSignal, internalAbort.signal);
|
||||
|
||||
const events: BulkTransferEvent[] = [];
|
||||
const eventResolvers: ((v: IteratorResult<BulkTransferEvent>) => void)[] = [];
|
||||
let eventsClosed = false;
|
||||
|
||||
function emit(event: BulkTransferEvent): void {
|
||||
if (eventsClosed) return;
|
||||
if (eventResolvers.length > 0) {
|
||||
eventResolvers.shift()!({ value: event, done: false });
|
||||
} else {
|
||||
events.push(event);
|
||||
}
|
||||
}
|
||||
|
||||
function closeEvents(): void {
|
||||
if (eventsClosed) return;
|
||||
eventsClosed = true;
|
||||
while (eventResolvers.length > 0) {
|
||||
eventResolvers.shift()!({ value: undefined as never, done: true });
|
||||
}
|
||||
}
|
||||
|
||||
let resolveDone!: (r: BulkTransferResult) => void;
|
||||
let rejectDone!: (err: unknown) => void;
|
||||
const donePromise = new Promise<BulkTransferResult>((resolve, reject) => {
|
||||
resolveDone = resolve;
|
||||
rejectDone = reject;
|
||||
});
|
||||
|
||||
const startedAt = Date.now();
|
||||
let filesDone = 0;
|
||||
let bytesDone = 0;
|
||||
let bytesTotal = 0;
|
||||
|
||||
// Run the upload pipeline asynchronously.
|
||||
void (async () => {
|
||||
try {
|
||||
// 1. Plan: walk local tree, collect dirs + files
|
||||
const plannedDirs: PlannedDir[] = [];
|
||||
const plannedFiles: PlannedFile[] = [];
|
||||
try {
|
||||
await collect(local, '', remoteRoot, plannedDirs, plannedFiles, combinedSignal);
|
||||
} catch (err) {
|
||||
emit({ type: 'abort', reason: errMsg(err) });
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
return;
|
||||
}
|
||||
|
||||
const totalFiles = plannedFiles.length;
|
||||
bytesTotal = plannedFiles.reduce(
|
||||
(acc, f) => acc + (f.handle as FileHandleLike & { _size?: number })._size!,
|
||||
0,
|
||||
);
|
||||
// Note: bytesTotal is computed lazily in collect() via cached sizes.
|
||||
|
||||
emit({ type: 'plan', totalFiles, totalBytes: bytesTotal });
|
||||
|
||||
if (combinedSignal.aborted) {
|
||||
emit({ type: 'abort', reason: errMsg(combinedSignal.reason) });
|
||||
closeEvents();
|
||||
rejectDone(combinedSignal.reason ?? new CancelledError());
|
||||
return;
|
||||
}
|
||||
|
||||
// 2. Pre-create remote directories sequentially (cheap, avoids races
|
||||
// when many uploads target the same parent).
|
||||
if (precreateDirs) {
|
||||
for (const d of plannedDirs) {
|
||||
if (combinedSignal.aborted) break;
|
||||
try {
|
||||
await client.mkdir(d.remoteAbsPath, { recursive: true });
|
||||
} catch (err) {
|
||||
if (!isAlreadyExistsError(err)) {
|
||||
if (!continueOnError) {
|
||||
emit({ type: 'abort', reason: errMsg(err) });
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Upload files with bounded concurrency
|
||||
try {
|
||||
await runWithConcurrency(
|
||||
asyncIterableOf(plannedFiles),
|
||||
async (planned) => {
|
||||
if (combinedSignal.aborted) {
|
||||
throw new CancelledError('upload aborted');
|
||||
}
|
||||
const file = await planned.handle.getFile();
|
||||
const size = file.size;
|
||||
emit({ type: 'file-start', path: planned.relativePath, size });
|
||||
try {
|
||||
const writeOpts: { contentType?: string; signal?: AbortSignal } = {};
|
||||
if (file.type !== '') writeOpts.contentType = file.type;
|
||||
writeOpts.signal = combinedSignal;
|
||||
if (size === 0) {
|
||||
// Edge case: empty file — write empty buffer.
|
||||
await client.write(planned.remoteAbsPath, new Uint8Array(0), writeOpts);
|
||||
} else {
|
||||
await client.write(
|
||||
planned.remoteAbsPath,
|
||||
{ stream: file.stream(), size, ...(file.type !== '' ? { contentType: file.type } : {}) },
|
||||
writeOpts,
|
||||
);
|
||||
}
|
||||
filesDone++;
|
||||
bytesDone += size;
|
||||
emit({ type: 'file-done', path: planned.relativePath, bytesDone: size });
|
||||
emit({
|
||||
type: 'progress',
|
||||
filesDone,
|
||||
filesTotal: totalFiles,
|
||||
bytesDone,
|
||||
bytesTotal,
|
||||
});
|
||||
} catch (err) {
|
||||
emit({ type: 'file-error', path: planned.relativePath, error: err });
|
||||
throw err;
|
||||
}
|
||||
},
|
||||
{
|
||||
concurrency,
|
||||
signal: combinedSignal,
|
||||
continueOnError,
|
||||
onError: () => {
|
||||
/* already emitted as 'file-error' */
|
||||
},
|
||||
},
|
||||
);
|
||||
} catch (err) {
|
||||
emit({ type: 'abort', reason: errMsg(err) });
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
return;
|
||||
}
|
||||
|
||||
const durationMs = Date.now() - startedAt;
|
||||
emit({ type: 'complete', filesDone, bytesDone, durationMs });
|
||||
closeEvents();
|
||||
resolveDone({ filesDone, bytesDone, durationMs });
|
||||
} catch (err) {
|
||||
// Belt-and-suspenders: any unhandled error reaches here.
|
||||
closeEvents();
|
||||
rejectDone(err);
|
||||
}
|
||||
})();
|
||||
|
||||
// Suppress unhandled-rejection until consumer awaits done().
|
||||
donePromise.catch(() => {
|
||||
/* deliberate */
|
||||
});
|
||||
|
||||
return {
|
||||
events: {
|
||||
[Symbol.asyncIterator]() {
|
||||
return {
|
||||
next(): Promise<IteratorResult<BulkTransferEvent>> {
|
||||
if (events.length > 0) {
|
||||
return Promise.resolve({ value: events.shift()!, done: false });
|
||||
}
|
||||
if (eventsClosed) {
|
||||
return Promise.resolve({ value: undefined as never, done: true });
|
||||
}
|
||||
return new Promise((resolve) => eventResolvers.push(resolve));
|
||||
},
|
||||
return(): Promise<IteratorResult<BulkTransferEvent>> {
|
||||
// Caller broke out — stop iterating but keep the bulk going.
|
||||
return Promise.resolve({ value: undefined as never, done: true });
|
||||
},
|
||||
};
|
||||
},
|
||||
},
|
||||
async abort(reason) {
|
||||
internalAbort.abort(new CancelledError(reason ?? 'manual abort'));
|
||||
},
|
||||
done: () => donePromise,
|
||||
};
|
||||
}
|
||||
|
||||
// ─── Helpers ─────────────────────────────────────────────────
|
||||
|
||||
async function collect(
|
||||
dir: DirectoryHandleLike,
|
||||
relPrefix: string,
|
||||
remoteRoot: string,
|
||||
plannedDirs: PlannedDir[],
|
||||
plannedFiles: PlannedFile[],
|
||||
signal: AbortSignal,
|
||||
): Promise<void> {
|
||||
for await (const [name, handle] of dir.entries()) {
|
||||
if (signal.aborted) return;
|
||||
const relPath = relPrefix === '' ? name : `${relPrefix}/${name}`;
|
||||
if (handle.kind === 'directory') {
|
||||
const remoteAbs = posixJoin(remoteRoot, relPath);
|
||||
plannedDirs.push({ remoteAbsPath: remoteAbs });
|
||||
await collect(handle as DirectoryHandleLike, relPath, remoteRoot, plannedDirs, plannedFiles, signal);
|
||||
} else {
|
||||
// Cache size on the handle so the planning total is accurate.
|
||||
const file = await (handle as FileHandleLike).getFile();
|
||||
(handle as FileHandleLike & { _size?: number })._size = file.size;
|
||||
plannedFiles.push({
|
||||
handle: handle as FileHandleLike,
|
||||
relativePath: relPath,
|
||||
remoteAbsPath: posixJoin(remoteRoot, relPath),
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function* asyncIterableOf<T>(arr: T[]): AsyncIterable<T> {
|
||||
for (const item of arr) yield item;
|
||||
}
|
||||
|
||||
function isAlreadyExistsError(err: unknown): boolean {
|
||||
if (err === null || typeof err !== 'object') return false;
|
||||
const code = (err as { code?: string; payload?: { code?: string } }).code;
|
||||
if (code === 'SHADE_FS_CONFLICT') return true;
|
||||
const payloadCode = (err as { payload?: { code?: string } }).payload?.code;
|
||||
return payloadCode === 'CONFLICT';
|
||||
}
|
||||
|
||||
function errMsg(err: unknown): string {
|
||||
if (err instanceof Error) return err.message;
|
||||
return String(err ?? 'unknown error');
|
||||
}
|
||||
|
||||
function mergeSignals(
|
||||
external: AbortSignal | undefined,
|
||||
internal: AbortSignal,
|
||||
): AbortSignal {
|
||||
if (external === undefined) return internal;
|
||||
if (external.aborted) return external;
|
||||
const controller = new AbortController();
|
||||
const onAbort = (sig: AbortSignal): void => {
|
||||
controller.abort(sig.reason);
|
||||
};
|
||||
external.addEventListener('abort', () => onAbort(external), { once: true });
|
||||
internal.addEventListener('abort', () => onAbort(internal), { once: true });
|
||||
return controller.signal;
|
||||
}
|
||||
89
packages/shade-files/src/client/walk.ts
Normal file
89
packages/shade-files/src/client/walk.ts
Normal file
@@ -0,0 +1,89 @@
|
||||
/**
|
||||
* Async-iterable directory walker.
|
||||
*
|
||||
* Depth-first by default — yields a directory's entries before descending
|
||||
* into the next sibling. Memory-bounded: never materializes the whole tree;
|
||||
* uses `client.list` page-by-page.
|
||||
*
|
||||
* Designed for arbitrarily-large remote trees: the consumer can break out
|
||||
* of the iterator at any point and the walk halts cleanly.
|
||||
*/
|
||||
import { posixJoin } from '../utils/path.js';
|
||||
import { CancelledError } from '../schemas/errors.js';
|
||||
import type { FileEntry } from '../schemas/file-entry.js';
|
||||
import type { FileClient } from './client.js';
|
||||
|
||||
export interface WalkOpts {
|
||||
/** Hard cap on recursion depth. Default 32. */
|
||||
maxDepth?: number;
|
||||
/** Cancellation. */
|
||||
signal?: AbortSignal;
|
||||
/**
|
||||
* Apply a per-entry filter. Return `false` to skip an entry (and, for
|
||||
* directories, to skip descending into it). Default: include all.
|
||||
*/
|
||||
filter?: (entry: FileEntry, relativePath: string) => boolean;
|
||||
/** Page size hint for `client.list`. Default 200. */
|
||||
pageSize?: number;
|
||||
}
|
||||
|
||||
export interface WalkItem {
|
||||
entry: FileEntry;
|
||||
/** Path relative to the walk root. Includes the entry's own name. */
|
||||
relativePath: string;
|
||||
/** Absolute path on the remote (root + relativePath). */
|
||||
absolutePath: string;
|
||||
/** Depth from root — 1 for direct children. */
|
||||
depth: number;
|
||||
}
|
||||
|
||||
const DEFAULT_MAX_DEPTH = 32;
|
||||
const DEFAULT_PAGE_SIZE = 200;
|
||||
|
||||
export async function* walk(
|
||||
client: Pick<FileClient, 'list'>,
|
||||
rootPath: string,
|
||||
opts: WalkOpts = {},
|
||||
): AsyncIterable<WalkItem> {
|
||||
const maxDepth = opts.maxDepth ?? DEFAULT_MAX_DEPTH;
|
||||
const pageSize = opts.pageSize ?? DEFAULT_PAGE_SIZE;
|
||||
|
||||
function checkAbort(): void {
|
||||
if (opts.signal?.aborted) {
|
||||
throw new CancelledError(
|
||||
opts.signal.reason instanceof Error
|
||||
? opts.signal.reason.message
|
||||
: 'walk aborted',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function* descend(absDir: string, depth: number, relPrefix: string): AsyncIterable<WalkItem> {
|
||||
if (depth > maxDepth) return;
|
||||
let cursor: string | undefined;
|
||||
do {
|
||||
checkAbort();
|
||||
const page = await client.list(absDir, {
|
||||
pageSize,
|
||||
...(cursor !== undefined ? { cursor } : {}),
|
||||
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
|
||||
});
|
||||
for (const entry of page.entries) {
|
||||
checkAbort();
|
||||
const relPath = relPrefix === '' ? entry.name : `${relPrefix}/${entry.name}`;
|
||||
const absPath = posixJoin(absDir, entry.name);
|
||||
if (opts.filter !== undefined && !opts.filter(entry, relPath)) {
|
||||
continue;
|
||||
}
|
||||
const item: WalkItem = { entry, relativePath: relPath, absolutePath: absPath, depth };
|
||||
yield item;
|
||||
if (entry.kind === 'dir') {
|
||||
yield* descend(absPath, depth + 1, relPath);
|
||||
}
|
||||
}
|
||||
cursor = page.hasMore ? page.nextCursor : undefined;
|
||||
} while (cursor !== undefined);
|
||||
}
|
||||
|
||||
yield* descend(rootPath, 1, '');
|
||||
}
|
||||
198
packages/shade-files/src/index.ts
Normal file
198
packages/shade-files/src/index.ts
Normal file
@@ -0,0 +1,198 @@
|
||||
// Schemas — Zod runtime + compile-time types
|
||||
export * from './schemas/index.js';
|
||||
|
||||
// Protocol primitives — kinds, correlation, canonical bytes, envelope codec
|
||||
export {
|
||||
SHADE_FILES_VERSION,
|
||||
KIND_PREFIX,
|
||||
SUPPORTED_KINDS,
|
||||
isSupportedKind,
|
||||
} from './protocol/version.js';
|
||||
export type { SupportedKind } from './protocol/version.js';
|
||||
export {
|
||||
KIND_LIST_V1,
|
||||
KIND_STAT_V1,
|
||||
KIND_MKDIR_V1,
|
||||
KIND_DELETE_V1,
|
||||
KIND_MOVE_V1,
|
||||
KIND_READ_V1,
|
||||
KIND_WRITE_V1,
|
||||
KIND_GET_THUMBNAIL_V1,
|
||||
KIND_CUSTOM_V1,
|
||||
KIND_ERROR_V1,
|
||||
KIND_CANCEL_V1,
|
||||
responseKindOf,
|
||||
opOfKind,
|
||||
MUTATION_OPS,
|
||||
} from './protocol/kinds.js';
|
||||
export type { StandardOp } from './protocol/kinds.js';
|
||||
export {
|
||||
generateRequestId,
|
||||
generateIdempotencyKey,
|
||||
base64UrlEncode,
|
||||
base64UrlDecode,
|
||||
} from './protocol/correlate.js';
|
||||
export {
|
||||
canonicalRpcBytes,
|
||||
canonicalJsonStringify,
|
||||
hashArgs,
|
||||
bytesToHex,
|
||||
bytesToBase64,
|
||||
base64ToBytes,
|
||||
} from './protocol/canonical.js';
|
||||
export {
|
||||
encodeEnvelope,
|
||||
looksLikeFileEnvelope,
|
||||
tryParseEnvelope,
|
||||
classify,
|
||||
} from './protocol/envelope-codec.js';
|
||||
export type { ClassifiedEnvelope } from './protocol/envelope-codec.js';
|
||||
|
||||
// RPC channel — wires Shade.send/onMessage to the file-RPC routing layer
|
||||
export { ShadeFileRpcChannel } from './rpc/channel.js';
|
||||
export type { RpcChannelHooks } from './rpc/channel.js';
|
||||
export { PendingRpcRegistry } from './rpc/pending.js';
|
||||
export type { RegisterOptions } from './rpc/pending.js';
|
||||
|
||||
// Server side
|
||||
export { createFileHandler, INTERNAL_SYMBOL } from './server/handler.js';
|
||||
export type {
|
||||
FileHandler,
|
||||
FileHandlerConfig,
|
||||
FileHandlerOps,
|
||||
} from './server/handler.js';
|
||||
export type { OpContext, OpKind } from './server/handler-context.js';
|
||||
export { validatePath } from './server/path-policy.js';
|
||||
export type { PathPolicy, PathValidationResult } from './server/path-policy.js';
|
||||
export { IdempotencyCache } from './server/idempotency-cache.js';
|
||||
export type { IdempotencyCacheOptions } from './server/idempotency-cache.js';
|
||||
export { RateLimiter } from './server/rate-limiter.js';
|
||||
export type { RateLimitConfig } from './server/rate-limiter.js';
|
||||
export { createCursorBuilder } from './server/cursor.js';
|
||||
export type { CursorBuilder } from './server/cursor.js';
|
||||
export {
|
||||
createServerStreamsBridge,
|
||||
META_KEY_READ_STREAM_ID,
|
||||
META_KEY_WRITE_ID,
|
||||
} from './server/streams-bridge.js';
|
||||
export type {
|
||||
ServerStreamsBridge,
|
||||
StreamsBridgeShade,
|
||||
CreateServerStreamsBridgeOptions,
|
||||
ParkedWrite,
|
||||
AwaitWriteOptions,
|
||||
} from './server/streams-bridge.js';
|
||||
export type {
|
||||
UserReadResult,
|
||||
UserReadResultInline,
|
||||
UserReadResultStreams,
|
||||
UserWriteArgs,
|
||||
UserWriteContent,
|
||||
UserWriteContentInline,
|
||||
UserWriteContentStreams,
|
||||
UserThumbnailResult,
|
||||
} from './server/io-types.js';
|
||||
export {
|
||||
assertThumbnailFormat,
|
||||
isThumbnailFormat,
|
||||
} from './server/thumbnail.js';
|
||||
export type { ThumbnailFormat } from './server/thumbnail.js';
|
||||
|
||||
// Client side
|
||||
export { createFileClient } from './client/client.js';
|
||||
export type {
|
||||
FileClient,
|
||||
BaseOpts,
|
||||
CreateFileClientOptions,
|
||||
ReadOpts,
|
||||
WriteOpts,
|
||||
ReadOutput,
|
||||
ReadInlineOutput,
|
||||
ReadStreamsOutput,
|
||||
ThumbnailResult,
|
||||
} from './client/client.js';
|
||||
export {
|
||||
createClientStreamsBridge,
|
||||
} from './client/streams-bridge.js';
|
||||
export type {
|
||||
ClientStreamsBridge,
|
||||
CreateClientStreamsBridgeOptions,
|
||||
AwaitReadOptions,
|
||||
ParkedRead,
|
||||
} from './client/streams-bridge.js';
|
||||
export {
|
||||
decideInline,
|
||||
INLINE_THRESHOLD,
|
||||
} from './client/inline-threshold.js';
|
||||
export type {
|
||||
InlineDecision,
|
||||
WriteSource,
|
||||
} from './client/inline-threshold.js';
|
||||
|
||||
// Directory ops — walk + bulk transfers
|
||||
export { walk } from './client/walk.js';
|
||||
export type { WalkOpts, WalkItem } from './client/walk.js';
|
||||
export { uploadDirectory } from './client/upload-directory.js';
|
||||
export type { UploadDirectoryOptions } from './client/upload-directory.js';
|
||||
export { downloadDirectory } from './client/download-directory.js';
|
||||
export type { DownloadDirectoryOptions } from './client/download-directory.js';
|
||||
export { runWithConcurrency } from './client/concurrency.js';
|
||||
export type { ConcurrentMapOptions } from './client/concurrency.js';
|
||||
export {
|
||||
DEFAULT_BULK_CONCURRENCY,
|
||||
MAX_BULK_CONCURRENCY,
|
||||
} from './client/directory-types.js';
|
||||
export type {
|
||||
BulkOpts,
|
||||
BulkTransferEvent,
|
||||
BulkTransferHandle,
|
||||
BulkTransferResult,
|
||||
DirectoryHandleLike,
|
||||
FileHandleLike,
|
||||
FileLike,
|
||||
WritableStreamLike,
|
||||
} from './client/directory-types.js';
|
||||
export { createMemoryDirectory } from './client/memory-directory.js';
|
||||
|
||||
// Custom-op registry typing (declaration-merged by consumers)
|
||||
export type {
|
||||
CustomOpDef,
|
||||
CustomOpsMap,
|
||||
CustomOpRegistration,
|
||||
CustomOpRegistrations,
|
||||
} from './server/custom-ops.js';
|
||||
|
||||
// Production hooks: metrics
|
||||
export {
|
||||
METRIC_BYTES_IN,
|
||||
METRIC_BYTES_OUT,
|
||||
METRIC_FINGERPRINT_REJECT_TOTAL,
|
||||
METRIC_IDEMPOTENCY_CONFLICT_TOTAL,
|
||||
METRIC_IDEMPOTENCY_HIT_TOTAL,
|
||||
METRIC_OP_DURATION_MS,
|
||||
METRIC_OP_TOTAL,
|
||||
METRIC_RATE_LIMIT_REJECT_TOTAL,
|
||||
METRIC_SIGNATURE_REJECT_TOTAL,
|
||||
NOOP_METRIC_SINK,
|
||||
} from './server/metrics.js';
|
||||
export type { MetricSink, MetricTags } from './server/metrics.js';
|
||||
|
||||
export { MAX_SIGNATURE_AGE_MS } from './server/handler.js';
|
||||
|
||||
// High-level SDK integration entrypoint
|
||||
export { createFilesNamespace } from './integration/files-namespace.js';
|
||||
export type { FilesNamespace } from './integration/files-namespace.js';
|
||||
|
||||
// Integration helpers — wire handler + pending registry onto a channel
|
||||
export { attachFileHandler } from './integration/wire-server.js';
|
||||
export { attachClientRouting } from './integration/wire-client.js';
|
||||
|
||||
// Path utilities
|
||||
export {
|
||||
posixNormalize,
|
||||
posixJoin,
|
||||
posixDirname,
|
||||
posixBasename,
|
||||
decodePercentEscapes,
|
||||
isPathInside,
|
||||
} from './utils/path.js';
|
||||
142
packages/shade-files/src/integration/files-namespace.ts
Normal file
142
packages/shade-files/src/integration/files-namespace.ts
Normal file
@@ -0,0 +1,142 @@
|
||||
/**
|
||||
* High-level `FilesNamespace` — the entry point that the SDK exposes via
|
||||
* `Shade.files`. Memoizes the underlying `ShadeFileRpcChannel` and bridges
|
||||
* so a single Shade can simultaneously serve files AND consume them from
|
||||
* peers without paying the setup cost twice.
|
||||
*/
|
||||
import type { Shade } from '@shade/sdk';
|
||||
import {
|
||||
attachClientRouting,
|
||||
attachFileHandler,
|
||||
createClientStreamsBridge,
|
||||
createFileClient,
|
||||
createFileHandler,
|
||||
createServerStreamsBridge,
|
||||
PendingRpcRegistry,
|
||||
ShadeFileRpcChannel,
|
||||
type ClientStreamsBridge,
|
||||
type CreateFileClientOptions,
|
||||
type FileClient,
|
||||
type FileHandler,
|
||||
type FileHandlerConfig,
|
||||
type ServerStreamsBridge,
|
||||
} from '../index.js';
|
||||
import { IdempotencyCache } from '../server/idempotency-cache.js';
|
||||
|
||||
export interface FilesNamespace {
|
||||
/**
|
||||
* Register a file handler. Throws if a handler is already attached on
|
||||
* this Shade — only one server per Shade. The returned function detaches
|
||||
* the handler and tears down its idempotency / retention timers.
|
||||
*/
|
||||
serve(handler: FileHandlerConfig): Promise<() => Promise<void>>;
|
||||
/**
|
||||
* Build a typed file client for `peer`. Multiple concurrent clients to
|
||||
* different peers share the same channel + streams bridge.
|
||||
*/
|
||||
client(peer: string, opts?: Omit<CreateFileClientOptions, 'streamsBridge'>): Promise<FileClient>;
|
||||
/** Tear down channel + bridges. After destroy(), serve()/client() throw. */
|
||||
destroy(): Promise<void>;
|
||||
}
|
||||
|
||||
interface NamespaceState {
|
||||
channel: ShadeFileRpcChannel;
|
||||
pending: PendingRpcRegistry;
|
||||
serverBridge: ServerStreamsBridge | null;
|
||||
clientBridge: ClientStreamsBridge | null;
|
||||
serverHandler: FileHandler | null;
|
||||
serverDetach: (() => void) | null;
|
||||
clientDetach: (() => void) | null;
|
||||
destroyed: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Construct a `FilesNamespace` bound to a Shade instance. The SDK's
|
||||
* `Shade.files` getter calls this lazily and memoizes the result.
|
||||
*/
|
||||
export function createFilesNamespace(shade: Shade): FilesNamespace {
|
||||
const state: NamespaceState = {
|
||||
channel: new ShadeFileRpcChannel(shade),
|
||||
pending: new PendingRpcRegistry(),
|
||||
serverBridge: null,
|
||||
clientBridge: null,
|
||||
serverHandler: null,
|
||||
serverDetach: null,
|
||||
clientDetach: null,
|
||||
destroyed: false,
|
||||
};
|
||||
|
||||
function ensureAlive(): void {
|
||||
if (state.destroyed) throw new Error('FilesNamespace: destroyed');
|
||||
}
|
||||
|
||||
return {
|
||||
async serve(handlerConfig) {
|
||||
ensureAlive();
|
||||
if (state.serverHandler !== null) {
|
||||
throw new Error('FilesNamespace: a handler is already registered (one per Shade)');
|
||||
}
|
||||
// Lazy server-side streams bridge.
|
||||
if (state.serverBridge === null) {
|
||||
state.serverBridge = await createServerStreamsBridge(shade);
|
||||
}
|
||||
const handler = createFileHandler(shade, {
|
||||
...handlerConfig,
|
||||
streamsBridge: state.serverBridge,
|
||||
});
|
||||
const detach = attachFileHandler(state.channel, handler);
|
||||
state.serverHandler = handler;
|
||||
state.serverDetach = detach;
|
||||
|
||||
// Wire BackgroundHooks.onPruneFiles to the new handler's idempotency
|
||||
// cache. Use the symbol-exposed internals (works because FileHandler
|
||||
// attaches them via Object.assign).
|
||||
const internals = (handler as unknown as { [k: symbol]: { idempotency: IdempotencyCache } })[
|
||||
Symbol.for('@shade/files/internal')
|
||||
];
|
||||
const background = (shade as unknown as { background?: { setHook?: (n: string, f: () => void) => void } }).background;
|
||||
if (background?.setHook !== undefined && internals !== undefined) {
|
||||
background.setHook('onPruneFiles', () => {
|
||||
internals.idempotency.prune();
|
||||
});
|
||||
}
|
||||
|
||||
return async () => {
|
||||
if (state.serverDetach !== null) state.serverDetach();
|
||||
state.serverHandler = null;
|
||||
state.serverDetach = null;
|
||||
if (background?.setHook !== undefined) {
|
||||
background.setHook('onPruneFiles', undefined as unknown as () => void);
|
||||
}
|
||||
};
|
||||
},
|
||||
|
||||
async client(peer, opts = {}) {
|
||||
ensureAlive();
|
||||
// Lazy client-side streams bridge.
|
||||
if (state.clientBridge === null) {
|
||||
state.clientBridge = await createClientStreamsBridge(shade);
|
||||
}
|
||||
// Attach response routing once.
|
||||
if (state.clientDetach === null) {
|
||||
state.clientDetach = attachClientRouting(state.channel, state.pending);
|
||||
}
|
||||
return createFileClient(shade, state.channel, state.pending, peer, {
|
||||
...opts,
|
||||
streamsBridge: state.clientBridge,
|
||||
});
|
||||
},
|
||||
|
||||
async destroy() {
|
||||
if (state.destroyed) return;
|
||||
state.destroyed = true;
|
||||
if (state.serverDetach !== null) state.serverDetach();
|
||||
if (state.clientDetach !== null) state.clientDetach();
|
||||
if (state.serverHandler !== null) state.serverHandler.destroy();
|
||||
if (state.serverBridge !== null) await state.serverBridge.destroy();
|
||||
if (state.clientBridge !== null) await state.clientBridge.destroy();
|
||||
state.channel.destroy();
|
||||
state.pending.rejectAll(new Error('FilesNamespace destroyed'));
|
||||
},
|
||||
};
|
||||
}
|
||||
23
packages/shade-files/src/integration/wire-client.ts
Normal file
23
packages/shade-files/src/integration/wire-client.ts
Normal file
@@ -0,0 +1,23 @@
|
||||
import { ShadeFileRpcChannel } from '../rpc/channel.js';
|
||||
import { PendingRpcRegistry } from '../rpc/pending.js';
|
||||
|
||||
/**
|
||||
* Wire a `PendingRpcRegistry` onto a channel so incoming `response` and
|
||||
* `error` envelopes resolve the matching client-side promises.
|
||||
*
|
||||
* The same channel can serve multiple clients (one per peer); the
|
||||
* registry is shared because correlation IDs are globally unique.
|
||||
*/
|
||||
export function attachClientRouting(
|
||||
channel: ShadeFileRpcChannel,
|
||||
pending: PendingRpcRegistry,
|
||||
): () => void {
|
||||
const previous = channel.setHooks({
|
||||
onResponse: (_from, env) => pending.resolveResponse(env.envelope),
|
||||
onError: (_from, env) => pending.resolveError(env.envelope),
|
||||
});
|
||||
return () => {
|
||||
pending.rejectAll(new Error('client routing detached'));
|
||||
channel.setHooks(previous);
|
||||
};
|
||||
}
|
||||
35
packages/shade-files/src/integration/wire-server.ts
Normal file
35
packages/shade-files/src/integration/wire-server.ts
Normal file
@@ -0,0 +1,35 @@
|
||||
import { ShadeFileRpcChannel } from '../rpc/channel.js';
|
||||
import type { FileHandler } from '../server/handler.js';
|
||||
|
||||
/**
|
||||
* Connect a `FileHandler` to a `ShadeFileRpcChannel`. Returns an unsubscribe
|
||||
* function that detaches the handler. Throws if a handler is already wired
|
||||
* on this channel.
|
||||
*
|
||||
* The wiring:
|
||||
* - On `request` envelopes: invoke `handler.handleRequest`, then send the
|
||||
* returned response/error envelope back to the requester.
|
||||
* - On `cancel` envelopes: forward to `handler.handleCancel`.
|
||||
*/
|
||||
export function attachFileHandler(
|
||||
channel: ShadeFileRpcChannel,
|
||||
handler: FileHandler,
|
||||
): () => void {
|
||||
const previous = channel.setHooks({
|
||||
onRequest: async (from, env) => {
|
||||
const response = await handler.handleRequest(from, env.envelope);
|
||||
try {
|
||||
await channel.send(from, response);
|
||||
} catch (err) {
|
||||
console.error('[shade-files] failed to send response:', err);
|
||||
}
|
||||
},
|
||||
onCancel: (from, env) => {
|
||||
handler.handleCancel(from, env.envelope);
|
||||
},
|
||||
});
|
||||
return () => {
|
||||
handler.destroy();
|
||||
channel.setHooks(previous);
|
||||
};
|
||||
}
|
||||
72
packages/shade-files/src/protocol/canonical.ts
Normal file
72
packages/shade-files/src/protocol/canonical.ts
Normal file
@@ -0,0 +1,72 @@
|
||||
import { sha256 } from '@noble/hashes/sha2.js';
|
||||
|
||||
/**
|
||||
* Canonical bytes for an `@shade/files` RPC request signature.
|
||||
*
|
||||
* "rpc\0addr=<sender>\0at=<signedAt>\0kind=<kind>\0id=<id>\0argsHash=<hex>\0"
|
||||
*
|
||||
* Mirror of `canonicalControlBytes` in `@shade/sdk/src/streams-bridge.ts:188`.
|
||||
* The signature binds the (sender, op kind, request id, args, signedAt)
|
||||
* tuple — defense-in-depth on top of Double Ratchet authentication.
|
||||
*/
|
||||
export function canonicalRpcBytes(args: {
|
||||
address: string;
|
||||
signedAt: number;
|
||||
kind: string;
|
||||
id: string;
|
||||
argsHash: Uint8Array;
|
||||
}): Uint8Array {
|
||||
const enc = new TextEncoder();
|
||||
const fields = [
|
||||
`rpc\0`,
|
||||
`addr=${args.address}\0`,
|
||||
`at=${args.signedAt}\0`,
|
||||
`kind=${args.kind}\0`,
|
||||
`id=${args.id}\0`,
|
||||
`argsHash=${bytesToHex(args.argsHash)}\0`,
|
||||
];
|
||||
return enc.encode(fields.join(''));
|
||||
}
|
||||
|
||||
/**
|
||||
* Stable canonical-JSON serialization for hashing args. Sorts object keys
|
||||
* recursively so two equivalent inputs hash to identical bytes regardless
|
||||
* of property order.
|
||||
*/
|
||||
export function canonicalJsonStringify(value: unknown): string {
|
||||
if (value === null || typeof value !== 'object') return JSON.stringify(value);
|
||||
if (Array.isArray(value)) {
|
||||
return `[${value.map((v) => canonicalJsonStringify(v)).join(',')}]`;
|
||||
}
|
||||
const obj = value as Record<string, unknown>;
|
||||
const keys = Object.keys(obj).sort();
|
||||
const parts: string[] = [];
|
||||
for (const key of keys) {
|
||||
if (obj[key] === undefined) continue;
|
||||
parts.push(`${JSON.stringify(key)}:${canonicalJsonStringify(obj[key])}`);
|
||||
}
|
||||
return `{${parts.join(',')}}`;
|
||||
}
|
||||
|
||||
/** Hash args via canonical-JSON → SHA-256 (32 bytes). */
|
||||
export function hashArgs(args: unknown): Uint8Array {
|
||||
const enc = new TextEncoder();
|
||||
return sha256(enc.encode(canonicalJsonStringify(args)));
|
||||
}
|
||||
|
||||
export function bytesToHex(b: Uint8Array): string {
|
||||
return Array.from(b, (x) => x.toString(16).padStart(2, '0')).join('');
|
||||
}
|
||||
|
||||
export function bytesToBase64(b: Uint8Array): string {
|
||||
let bin = '';
|
||||
for (let i = 0; i < b.length; i++) bin += String.fromCharCode(b[i]!);
|
||||
return btoa(bin);
|
||||
}
|
||||
|
||||
export function base64ToBytes(s: string): Uint8Array {
|
||||
const bin = atob(s);
|
||||
const out = new Uint8Array(bin.length);
|
||||
for (let i = 0; i < bin.length; i++) out[i] = bin.charCodeAt(i);
|
||||
return out;
|
||||
}
|
||||
32
packages/shade-files/src/protocol/correlate.ts
Normal file
32
packages/shade-files/src/protocol/correlate.ts
Normal file
@@ -0,0 +1,32 @@
|
||||
/**
|
||||
* Generate a correlation ID — 16 random bytes, base64url-encoded (22 chars
|
||||
* after stripping padding). Same shape as `RequestIdSchema`.
|
||||
*
|
||||
* Uses `globalThis.crypto.getRandomValues` so it works in Bun, Node ≥19,
|
||||
* and browsers without any platform-specific imports.
|
||||
*/
|
||||
export function generateRequestId(): string {
|
||||
const buf = new Uint8Array(16);
|
||||
globalThis.crypto.getRandomValues(buf);
|
||||
return base64UrlEncode(buf);
|
||||
}
|
||||
|
||||
/** Same as `generateRequestId` — distinct name for callsite clarity. */
|
||||
export function generateIdempotencyKey(): string {
|
||||
return generateRequestId();
|
||||
}
|
||||
|
||||
export function base64UrlEncode(bytes: Uint8Array): string {
|
||||
let bin = '';
|
||||
for (let i = 0; i < bytes.length; i++) bin += String.fromCharCode(bytes[i]!);
|
||||
return btoa(bin).replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, '');
|
||||
}
|
||||
|
||||
export function base64UrlDecode(s: string): Uint8Array {
|
||||
const padded = s.replace(/-/g, '+').replace(/_/g, '/');
|
||||
const pad = padded.length % 4 === 0 ? '' : '='.repeat(4 - (padded.length % 4));
|
||||
const bin = atob(padded + pad);
|
||||
const out = new Uint8Array(bin.length);
|
||||
for (let i = 0; i < bin.length; i++) out[i] = bin.charCodeAt(i);
|
||||
return out;
|
||||
}
|
||||
70
packages/shade-files/src/protocol/envelope-codec.ts
Normal file
70
packages/shade-files/src/protocol/envelope-codec.ts
Normal file
@@ -0,0 +1,70 @@
|
||||
import { z } from 'zod';
|
||||
import {
|
||||
RpcCancelSchema,
|
||||
RpcErrorSchema,
|
||||
RpcRequestSchema,
|
||||
RpcResponseSchema,
|
||||
type RpcCancel,
|
||||
type RpcEnvelope,
|
||||
type RpcError,
|
||||
type RpcRequest,
|
||||
type RpcResponse,
|
||||
} from '../schemas/envelope.js';
|
||||
import { KIND_PREFIX } from './version.js';
|
||||
|
||||
/** Tagged classification of any incoming envelope. */
|
||||
export type ClassifiedEnvelope =
|
||||
| { kind: 'request'; envelope: RpcRequest }
|
||||
| { kind: 'response'; envelope: RpcResponse }
|
||||
| { kind: 'error'; envelope: RpcError }
|
||||
| { kind: 'cancel'; envelope: RpcCancel };
|
||||
|
||||
const RpcAnySchema = z.union([
|
||||
RpcRequestSchema,
|
||||
RpcResponseSchema,
|
||||
RpcErrorSchema,
|
||||
RpcCancelSchema,
|
||||
]);
|
||||
|
||||
/** Encode an envelope to JSON plaintext for `Shade.send`. */
|
||||
export function encodeEnvelope(env: RpcEnvelope): string {
|
||||
return JSON.stringify(env);
|
||||
}
|
||||
|
||||
/**
|
||||
* Quick-detection: does this plaintext look like an `@shade/files` envelope?
|
||||
* Used by `ShadeFileRpcChannel` to skip non-files messages cheaply.
|
||||
*/
|
||||
export function looksLikeFileEnvelope(plaintext: string): boolean {
|
||||
return plaintext.includes(KIND_PREFIX);
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse and classify an incoming plaintext. Returns null on any malformed
|
||||
* input — the channel ignores those silently (could be a different
|
||||
* protocol on the same Shade.send pipe).
|
||||
*/
|
||||
export function tryParseEnvelope(plaintext: string): ClassifiedEnvelope | null {
|
||||
let raw: unknown;
|
||||
try {
|
||||
raw = JSON.parse(plaintext);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
const result = RpcAnySchema.safeParse(raw);
|
||||
if (!result.success) return null;
|
||||
return classify(result.data);
|
||||
}
|
||||
|
||||
export function classify(env: RpcEnvelope): ClassifiedEnvelope {
|
||||
if (env.kind === 'shade.fs.cancel/v1') {
|
||||
return { kind: 'cancel', envelope: env as RpcCancel };
|
||||
}
|
||||
if (env.kind === 'shade.fs.error/v1') {
|
||||
return { kind: 'error', envelope: env as RpcError };
|
||||
}
|
||||
if (env.kind.endsWith('.response')) {
|
||||
return { kind: 'response', envelope: env as RpcResponse };
|
||||
}
|
||||
return { kind: 'request', envelope: env as RpcRequest };
|
||||
}
|
||||
53
packages/shade-files/src/protocol/kinds.ts
Normal file
53
packages/shade-files/src/protocol/kinds.ts
Normal file
@@ -0,0 +1,53 @@
|
||||
/** Canonical kind names for v1 ops. */
|
||||
export const KIND_LIST_V1 = 'shade.fs.list/v1' as const;
|
||||
export const KIND_STAT_V1 = 'shade.fs.stat/v1' as const;
|
||||
export const KIND_MKDIR_V1 = 'shade.fs.mkdir/v1' as const;
|
||||
export const KIND_DELETE_V1 = 'shade.fs.delete/v1' as const;
|
||||
export const KIND_MOVE_V1 = 'shade.fs.move/v1' as const;
|
||||
export const KIND_READ_V1 = 'shade.fs.read/v1' as const;
|
||||
export const KIND_WRITE_V1 = 'shade.fs.write/v1' as const;
|
||||
export const KIND_GET_THUMBNAIL_V1 = 'shade.fs.getThumbnail/v1' as const;
|
||||
export const KIND_CUSTOM_V1 = 'shade.fs.custom/v1' as const;
|
||||
|
||||
export const KIND_ERROR_V1 = 'shade.fs.error/v1' as const;
|
||||
export const KIND_CANCEL_V1 = 'shade.fs.cancel/v1' as const;
|
||||
|
||||
/** Compute the response kind from a request kind: `'shade.fs.foo/v1'` → `'shade.fs.foo/v1.response'`. */
|
||||
export function responseKindOf<K extends string>(kind: K): `${K}.response` {
|
||||
return `${kind}.response`;
|
||||
}
|
||||
|
||||
/** Standard op identifier — used in `OpContext.op`. */
|
||||
export type StandardOp =
|
||||
| 'list'
|
||||
| 'stat'
|
||||
| 'mkdir'
|
||||
| 'delete'
|
||||
| 'move'
|
||||
| 'read'
|
||||
| 'write'
|
||||
| 'getThumbnail';
|
||||
|
||||
const KIND_TO_OP: Record<string, StandardOp> = {
|
||||
[KIND_LIST_V1]: 'list',
|
||||
[KIND_STAT_V1]: 'stat',
|
||||
[KIND_MKDIR_V1]: 'mkdir',
|
||||
[KIND_DELETE_V1]: 'delete',
|
||||
[KIND_MOVE_V1]: 'move',
|
||||
[KIND_READ_V1]: 'read',
|
||||
[KIND_WRITE_V1]: 'write',
|
||||
[KIND_GET_THUMBNAIL_V1]: 'getThumbnail',
|
||||
};
|
||||
|
||||
export function opOfKind(kind: string): StandardOp | null {
|
||||
return KIND_TO_OP[kind] ?? null;
|
||||
}
|
||||
|
||||
/** Mutation ops require an idempotency key on retries. */
|
||||
export const MUTATION_OPS = new Set<StandardOp | 'custom'>([
|
||||
'mkdir',
|
||||
'delete',
|
||||
'move',
|
||||
'write',
|
||||
'custom',
|
||||
]);
|
||||
24
packages/shade-files/src/protocol/version.ts
Normal file
24
packages/shade-files/src/protocol/version.ts
Normal file
@@ -0,0 +1,24 @@
|
||||
/** Wire-protocol version. Bump when introducing breaking changes. */
|
||||
export const SHADE_FILES_VERSION = '0.2.0';
|
||||
|
||||
/** Substring every `@shade/files` plaintext starts with — used by the channel filter. */
|
||||
export const KIND_PREFIX = 'shade.fs';
|
||||
|
||||
/** Currently-supported op kinds. Extend when introducing new versions. */
|
||||
export const SUPPORTED_KINDS = [
|
||||
'shade.fs.list/v1',
|
||||
'shade.fs.stat/v1',
|
||||
'shade.fs.mkdir/v1',
|
||||
'shade.fs.delete/v1',
|
||||
'shade.fs.move/v1',
|
||||
'shade.fs.read/v1',
|
||||
'shade.fs.write/v1',
|
||||
'shade.fs.getThumbnail/v1',
|
||||
'shade.fs.custom/v1',
|
||||
] as const;
|
||||
export type SupportedKind = (typeof SUPPORTED_KINDS)[number];
|
||||
|
||||
const SUPPORTED_KIND_SET = new Set<string>(SUPPORTED_KINDS);
|
||||
export function isSupportedKind(kind: string): kind is SupportedKind {
|
||||
return SUPPORTED_KIND_SET.has(kind);
|
||||
}
|
||||
35
packages/shade-files/src/react/ShadeFilesProvider.tsx
Normal file
35
packages/shade-files/src/react/ShadeFilesProvider.tsx
Normal file
@@ -0,0 +1,35 @@
|
||||
import React, { createContext, useContext, useMemo } from 'react';
|
||||
import type { Shade } from '@shade/sdk';
|
||||
import type { FilesNamespace } from '../integration/files-namespace.js';
|
||||
|
||||
export interface ShadeFilesContextValue {
|
||||
shade: Shade;
|
||||
files: FilesNamespace;
|
||||
}
|
||||
|
||||
const ShadeFilesContext = createContext<ShadeFilesContextValue | null>(null);
|
||||
|
||||
export interface ShadeFilesProviderProps {
|
||||
/** Initialized `Shade` instance. `files` namespace is read off it lazily. */
|
||||
shade: Shade;
|
||||
children: React.ReactNode;
|
||||
}
|
||||
|
||||
/**
|
||||
* Provider for `@shade/files` React hooks. Distinct from
|
||||
* `<ShadeRuntimeProvider>` in `@shade/widgets` so file-RPC consumers
|
||||
* don't pull in the widget tree.
|
||||
*/
|
||||
export function ShadeFilesProvider({ shade, children }: ShadeFilesProviderProps): React.ReactElement {
|
||||
const value = useMemo<ShadeFilesContextValue>(() => ({ shade, files: shade.files }), [shade]);
|
||||
return React.createElement(ShadeFilesContext.Provider, { value }, children);
|
||||
}
|
||||
|
||||
/** Throws if no `<ShadeFilesProvider>` is mounted above. */
|
||||
export function useShadeFilesContext(): ShadeFilesContextValue {
|
||||
const ctx = useContext(ShadeFilesContext);
|
||||
if (ctx === null) {
|
||||
throw new Error('useShadeFilesContext: missing <ShadeFilesProvider>');
|
||||
}
|
||||
return ctx;
|
||||
}
|
||||
26
packages/shade-files/src/react/index.ts
Normal file
26
packages/shade-files/src/react/index.ts
Normal file
@@ -0,0 +1,26 @@
|
||||
/**
|
||||
* React entry point for `@shade/files`.
|
||||
*
|
||||
* Note: hooks/components are tree-shakable. Importing from
|
||||
* `@shade/files/react` (this entry) avoids pulling React into Bun-only
|
||||
* server consumers that import from the root `@shade/files`.
|
||||
*/
|
||||
|
||||
export { ShadeFilesProvider, useShadeFilesContext } from './ShadeFilesProvider.js';
|
||||
export type { ShadeFilesContextValue, ShadeFilesProviderProps } from './ShadeFilesProvider.js';
|
||||
|
||||
export { useShadeFiles } from './useShadeFiles.js';
|
||||
export type { UseShadeFilesOptions, UseShadeFilesResult } from './useShadeFiles.js';
|
||||
|
||||
export { useFileList } from './useFileList.js';
|
||||
export type { UseFileListOptions, UseFileListResult } from './useFileList.js';
|
||||
|
||||
export {
|
||||
useFileTransfer,
|
||||
useFileUpload,
|
||||
useFileDownload,
|
||||
} from './useFileTransfer.js';
|
||||
export type {
|
||||
BulkTransferStatus,
|
||||
UseFileTransferResult,
|
||||
} from './useFileTransfer.js';
|
||||
90
packages/shade-files/src/react/useFileList.ts
Normal file
90
packages/shade-files/src/react/useFileList.ts
Normal file
@@ -0,0 +1,90 @@
|
||||
import { useCallback, useEffect, useRef, useState } from 'react';
|
||||
import type { FileEntry } from '../schemas/file-entry.js';
|
||||
import { useShadeFiles, type UseShadeFilesOptions } from './useShadeFiles.js';
|
||||
|
||||
export interface UseFileListResult {
|
||||
entries: FileEntry[];
|
||||
isLoading: boolean;
|
||||
error: unknown;
|
||||
hasMore: boolean;
|
||||
/** Reset and reload the first page. */
|
||||
refresh(): void;
|
||||
/** Append the next page (no-op when `hasMore === false`). */
|
||||
loadMore(): Promise<void>;
|
||||
}
|
||||
|
||||
export interface UseFileListOptions extends UseShadeFilesOptions {
|
||||
pageSize?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Paginated list of `path` on `peer`. Fetches the first page on mount,
|
||||
* exposes `loadMore()` for subsequent pages.
|
||||
*/
|
||||
export function useFileList(
|
||||
peer: string,
|
||||
path: string,
|
||||
opts: UseFileListOptions = {},
|
||||
): UseFileListResult {
|
||||
const filesOpts: UseShadeFilesOptions = {};
|
||||
if (opts.probeIntervalMs !== undefined) filesOpts.probeIntervalMs = opts.probeIntervalMs;
|
||||
const { fs } = useShadeFiles(peer, filesOpts);
|
||||
const pageSize = opts.pageSize ?? 100;
|
||||
const [entries, setEntries] = useState<FileEntry[]>([]);
|
||||
const [cursor, setCursor] = useState<string | undefined>(undefined);
|
||||
const [hasMore, setHasMore] = useState(false);
|
||||
const [isLoading, setIsLoading] = useState(false);
|
||||
const [error, setError] = useState<unknown>(null);
|
||||
const [tick, setTick] = useState(0);
|
||||
const inflightRef = useRef<AbortController | null>(null);
|
||||
|
||||
// Initial load + reload on path/peer/refresh.
|
||||
useEffect(() => {
|
||||
if (fs === null) return;
|
||||
inflightRef.current?.abort();
|
||||
const ctrl = new AbortController();
|
||||
inflightRef.current = ctrl;
|
||||
setIsLoading(true);
|
||||
setError(null);
|
||||
fs.list(path, { pageSize, signal: ctrl.signal })
|
||||
.then((page) => {
|
||||
if (ctrl.signal.aborted) return;
|
||||
setEntries(page.entries);
|
||||
setHasMore(page.hasMore);
|
||||
setCursor(page.nextCursor);
|
||||
})
|
||||
.catch((err) => {
|
||||
if (!ctrl.signal.aborted) setError(err);
|
||||
})
|
||||
.finally(() => {
|
||||
if (!ctrl.signal.aborted) setIsLoading(false);
|
||||
});
|
||||
return () => {
|
||||
ctrl.abort();
|
||||
};
|
||||
}, [fs, path, pageSize, tick]);
|
||||
|
||||
const loadMore = useCallback(async (): Promise<void> => {
|
||||
if (fs === null || !hasMore || cursor === undefined) return;
|
||||
setIsLoading(true);
|
||||
try {
|
||||
const page = await fs.list(path, { pageSize, cursor });
|
||||
setEntries((prev) => [...prev, ...page.entries]);
|
||||
setHasMore(page.hasMore);
|
||||
setCursor(page.nextCursor);
|
||||
} catch (err) {
|
||||
setError(err);
|
||||
} finally {
|
||||
setIsLoading(false);
|
||||
}
|
||||
}, [fs, path, pageSize, cursor, hasMore]);
|
||||
|
||||
return {
|
||||
entries,
|
||||
isLoading,
|
||||
error,
|
||||
hasMore,
|
||||
refresh: () => setTick((t) => t + 1),
|
||||
loadMore,
|
||||
};
|
||||
}
|
||||
103
packages/shade-files/src/react/useFileTransfer.ts
Normal file
103
packages/shade-files/src/react/useFileTransfer.ts
Normal file
@@ -0,0 +1,103 @@
|
||||
import { useCallback, useEffect, useRef, useState } from 'react';
|
||||
import type { BulkTransferEvent, BulkTransferHandle, BulkTransferResult } from '../client/directory-types.js';
|
||||
|
||||
export interface BulkTransferStatus {
|
||||
state: 'idle' | 'running' | 'done' | 'error' | 'aborted';
|
||||
filesDone: number;
|
||||
filesTotal: number;
|
||||
bytesDone: number;
|
||||
bytesTotal: number;
|
||||
error: unknown;
|
||||
result: BulkTransferResult | null;
|
||||
/** Last event emitted by the underlying handle. */
|
||||
lastEvent: BulkTransferEvent | null;
|
||||
}
|
||||
|
||||
export interface UseFileTransferResult extends BulkTransferStatus {
|
||||
start(handle: BulkTransferHandle): void;
|
||||
abort(reason?: string): Promise<void>;
|
||||
}
|
||||
|
||||
const INITIAL: BulkTransferStatus = {
|
||||
state: 'idle',
|
||||
filesDone: 0,
|
||||
filesTotal: 0,
|
||||
bytesDone: 0,
|
||||
bytesTotal: 0,
|
||||
error: null,
|
||||
result: null,
|
||||
lastEvent: null,
|
||||
};
|
||||
|
||||
/**
|
||||
* Generic React-state wrapper around a `BulkTransferHandle`. Apps wire the
|
||||
* handle returned from `uploadDirectory()` / `downloadDirectory()` here to
|
||||
* surface progress in their UI.
|
||||
*
|
||||
* `useFileUpload` and `useFileDownload` are thin presets — they call
|
||||
* `start(...)` with the appropriate handle automatically.
|
||||
*/
|
||||
export function useFileTransfer(): UseFileTransferResult {
|
||||
const [status, setStatus] = useState<BulkTransferStatus>(INITIAL);
|
||||
const handleRef = useRef<BulkTransferHandle | null>(null);
|
||||
|
||||
const start = useCallback((handle: BulkTransferHandle): void => {
|
||||
handleRef.current = handle;
|
||||
setStatus({ ...INITIAL, state: 'running' });
|
||||
void (async () => {
|
||||
try {
|
||||
for await (const ev of handle.events) {
|
||||
setStatus((prev) => {
|
||||
const next: BulkTransferStatus = { ...prev, lastEvent: ev };
|
||||
if (ev.type === 'plan') {
|
||||
next.filesTotal = ev.totalFiles;
|
||||
next.bytesTotal = ev.totalBytes ?? 0;
|
||||
} else if (ev.type === 'progress') {
|
||||
next.filesDone = ev.filesDone;
|
||||
next.bytesDone = ev.bytesDone;
|
||||
next.bytesTotal = ev.bytesTotal ?? prev.bytesTotal;
|
||||
next.filesTotal = ev.filesTotal;
|
||||
} else if (ev.type === 'complete') {
|
||||
next.state = 'done';
|
||||
next.filesDone = ev.filesDone;
|
||||
next.bytesDone = ev.bytesDone;
|
||||
} else if (ev.type === 'abort') {
|
||||
next.state = 'aborted';
|
||||
next.error = new Error(ev.reason);
|
||||
} else if (ev.type === 'file-error') {
|
||||
next.error = ev.error;
|
||||
}
|
||||
return next;
|
||||
});
|
||||
}
|
||||
const result = await handle.done();
|
||||
setStatus((prev) => ({ ...prev, state: 'done', result }));
|
||||
} catch (err) {
|
||||
setStatus((prev) => ({ ...prev, state: 'error', error: err }));
|
||||
}
|
||||
})();
|
||||
}, []);
|
||||
|
||||
const abort = useCallback(async (reason?: string): Promise<void> => {
|
||||
if (handleRef.current === null) return;
|
||||
await handleRef.current.abort(reason);
|
||||
}, []);
|
||||
|
||||
// Cleanup on unmount: abort any in-flight transfer.
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
if (handleRef.current !== null) {
|
||||
void handleRef.current.abort('unmount').catch(() => undefined);
|
||||
}
|
||||
};
|
||||
}, []);
|
||||
|
||||
return { ...status, start, abort };
|
||||
}
|
||||
|
||||
/**
|
||||
* Preset: `useFileUpload` — semantically identical to `useFileTransfer`,
|
||||
* named distinctly to mirror the `useFileDownload` pair.
|
||||
*/
|
||||
export const useFileUpload = useFileTransfer;
|
||||
export const useFileDownload = useFileTransfer;
|
||||
88
packages/shade-files/src/react/useShadeFiles.ts
Normal file
88
packages/shade-files/src/react/useShadeFiles.ts
Normal file
@@ -0,0 +1,88 @@
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
import type { FileClient } from '../client/client.js';
|
||||
import { useShadeFilesContext } from './ShadeFilesProvider.js';
|
||||
|
||||
export interface UseShadeFilesResult {
|
||||
fs: FileClient | null;
|
||||
isLoading: boolean;
|
||||
error: unknown;
|
||||
/** True after a successful `stat('/')` probe; false until proven online. */
|
||||
isOnline: boolean;
|
||||
/** Force a re-probe. */
|
||||
refresh(): void;
|
||||
}
|
||||
|
||||
export interface UseShadeFilesOptions {
|
||||
/** Probe interval (ms) for `isOnline`. Default 30_000. Disable with 0. */
|
||||
probeIntervalMs?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get a typed `FileClient` for `peer`. Memoizes per (shade, peer) so
|
||||
* multiple components sharing the same peer reuse the same client.
|
||||
*/
|
||||
export function useShadeFiles(
|
||||
peer: string,
|
||||
opts: UseShadeFilesOptions = {},
|
||||
): UseShadeFilesResult {
|
||||
const { files } = useShadeFilesContext();
|
||||
const [fs, setFs] = useState<FileClient | null>(null);
|
||||
const [error, setError] = useState<unknown>(null);
|
||||
const [isOnline, setIsOnline] = useState(false);
|
||||
const [probeTick, setProbeTick] = useState(0);
|
||||
const cancelledRef = useRef(false);
|
||||
|
||||
useEffect(() => {
|
||||
cancelledRef.current = false;
|
||||
let mounted = true;
|
||||
setError(null);
|
||||
setFs(null);
|
||||
files
|
||||
.client(peer)
|
||||
.then((client) => {
|
||||
if (!mounted) {
|
||||
client.close();
|
||||
return;
|
||||
}
|
||||
setFs(client);
|
||||
})
|
||||
.catch((err) => {
|
||||
if (mounted) setError(err);
|
||||
});
|
||||
return () => {
|
||||
mounted = false;
|
||||
cancelledRef.current = true;
|
||||
};
|
||||
}, [files, peer]);
|
||||
|
||||
// Periodic probe: stat('/') as a cheap reachability check.
|
||||
useEffect(() => {
|
||||
if (fs === null) return undefined;
|
||||
const intervalMs = opts.probeIntervalMs ?? 30_000;
|
||||
let cancelled = false;
|
||||
const probe = (): void => {
|
||||
fs.stat('/')
|
||||
.then(() => {
|
||||
if (!cancelled) setIsOnline(true);
|
||||
})
|
||||
.catch(() => {
|
||||
if (!cancelled) setIsOnline(false);
|
||||
});
|
||||
};
|
||||
probe();
|
||||
if (intervalMs <= 0) return () => undefined;
|
||||
const timer = setInterval(probe, intervalMs);
|
||||
return () => {
|
||||
cancelled = true;
|
||||
clearInterval(timer);
|
||||
};
|
||||
}, [fs, opts.probeIntervalMs, probeTick]);
|
||||
|
||||
return {
|
||||
fs,
|
||||
isLoading: fs === null && error === null,
|
||||
error,
|
||||
isOnline,
|
||||
refresh: () => setProbeTick((t) => t + 1),
|
||||
};
|
||||
}
|
||||
107
packages/shade-files/src/rpc/channel.ts
Normal file
107
packages/shade-files/src/rpc/channel.ts
Normal file
@@ -0,0 +1,107 @@
|
||||
import type { Shade } from '@shade/sdk';
|
||||
import {
|
||||
encodeEnvelope,
|
||||
looksLikeFileEnvelope,
|
||||
tryParseEnvelope,
|
||||
type ClassifiedEnvelope,
|
||||
} from '../protocol/envelope-codec.js';
|
||||
import type { RpcEnvelope } from '../schemas/envelope.js';
|
||||
|
||||
/**
|
||||
* Routing hooks called by the channel when it receives different envelope
|
||||
* classes. M-Files-1 ships the skeleton; M-Files-2 wires up real handlers
|
||||
* (server-side dispatcher, client-side response routing).
|
||||
*/
|
||||
export interface RpcChannelHooks {
|
||||
onRequest?: (from: string, env: ClassifiedEnvelope & { kind: 'request' }) => Promise<void>;
|
||||
onResponse?: (from: string, env: ClassifiedEnvelope & { kind: 'response' }) => void;
|
||||
onError?: (from: string, env: ClassifiedEnvelope & { kind: 'error' }) => void;
|
||||
onCancel?: (from: string, env: ClassifiedEnvelope & { kind: 'cancel' }) => void;
|
||||
}
|
||||
|
||||
/**
|
||||
* `ShadeFileRpcChannel` rides on top of `Shade.send`/`Shade.onMessage`
|
||||
* and routes `shade.fs.*` JSON envelopes to registered hooks.
|
||||
*
|
||||
* Counterpart to `ShadeControlChannel` from `@shade/sdk` — but with
|
||||
* request-response semantics instead of fire-and-forget.
|
||||
*
|
||||
* One channel per `Shade` instance; the SDK's `files` getter memoizes it.
|
||||
* Both server and client roles share the same channel: the same Shade can
|
||||
* both serve files and consume them from peers.
|
||||
*/
|
||||
export class ShadeFileRpcChannel {
|
||||
private hooks: RpcChannelHooks = {};
|
||||
private readonly unsubscribe: () => void;
|
||||
private destroyed = false;
|
||||
|
||||
constructor(private readonly shade: Shade) {
|
||||
this.unsubscribe = shade.onMessage(async (from, plaintext) => {
|
||||
if (!looksLikeFileEnvelope(plaintext)) return;
|
||||
const classified = tryParseEnvelope(plaintext);
|
||||
if (classified === null) return;
|
||||
await this.dispatch(from, classified);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Merge hooks into the current set. Returns the previous values for the
|
||||
* keys being set so callers can restore them on cleanup. Other hook
|
||||
* slots are left unchanged — the same channel can simultaneously serve
|
||||
* (`onRequest`+`onCancel`) and consume (`onResponse`+`onError`).
|
||||
*/
|
||||
setHooks(hooks: RpcChannelHooks): RpcChannelHooks {
|
||||
if (this.destroyed) throw new Error('ShadeFileRpcChannel: destroyed');
|
||||
const prev: RpcChannelHooks = {};
|
||||
const keys = ['onRequest', 'onResponse', 'onError', 'onCancel'] as const;
|
||||
for (const key of keys) {
|
||||
if (key in hooks) {
|
||||
prev[key] = this.hooks[key] as never;
|
||||
this.hooks[key] = hooks[key] as never;
|
||||
}
|
||||
}
|
||||
return prev;
|
||||
}
|
||||
|
||||
/**
|
||||
* Send an envelope to a peer. Encrypts via `Shade.send` (Double Ratchet)
|
||||
* then delivers the ratchet envelope to the peer's `transferRoute()`
|
||||
* control endpoint via the SDK's configured envelope transport.
|
||||
*/
|
||||
async send(peerAddress: string, envelope: RpcEnvelope): Promise<void> {
|
||||
if (this.destroyed) throw new Error('ShadeFileRpcChannel: destroyed');
|
||||
const plaintext = encodeEnvelope(envelope);
|
||||
const ratchetEnvelope = await this.shade.send(peerAddress, plaintext);
|
||||
await this.shade.deliverControlEnvelope(peerAddress, ratchetEnvelope);
|
||||
}
|
||||
|
||||
destroy(): void {
|
||||
if (this.destroyed) return;
|
||||
this.destroyed = true;
|
||||
this.unsubscribe();
|
||||
this.hooks = {};
|
||||
}
|
||||
|
||||
private async dispatch(from: string, classified: ClassifiedEnvelope): Promise<void> {
|
||||
try {
|
||||
switch (classified.kind) {
|
||||
case 'request':
|
||||
if (this.hooks.onRequest !== undefined) {
|
||||
await this.hooks.onRequest(from, classified);
|
||||
}
|
||||
return;
|
||||
case 'response':
|
||||
this.hooks.onResponse?.(from, classified);
|
||||
return;
|
||||
case 'error':
|
||||
this.hooks.onError?.(from, classified);
|
||||
return;
|
||||
case 'cancel':
|
||||
this.hooks.onCancel?.(from, classified);
|
||||
return;
|
||||
}
|
||||
} catch (err) {
|
||||
console.error('[ShadeFileRpcChannel] dispatch error:', err);
|
||||
}
|
||||
}
|
||||
}
|
||||
113
packages/shade-files/src/rpc/pending.ts
Normal file
113
packages/shade-files/src/rpc/pending.ts
Normal file
@@ -0,0 +1,113 @@
|
||||
import { CancelledError, OperationTimeoutError, fileErrorFromPayload } from '../schemas/errors.js';
|
||||
import type { RpcError, RpcResponse } from '../schemas/envelope.js';
|
||||
|
||||
interface PendingEntry {
|
||||
resolve: (result: unknown) => void;
|
||||
reject: (err: unknown) => void;
|
||||
timer: ReturnType<typeof setTimeout> | null;
|
||||
abortListener: (() => void) | null;
|
||||
signal?: AbortSignal | undefined;
|
||||
}
|
||||
|
||||
export interface RegisterOptions {
|
||||
timeoutMs?: number;
|
||||
signal?: AbortSignal;
|
||||
/** Called when client cancels (signal aborted or timeout); transport can ship an `RpcCancel`. */
|
||||
onCancel?: (reason: 'timeout' | 'signal') => void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Client-side tracker for in-flight RPC calls. Routes incoming
|
||||
* `RpcResponse`/`RpcError` to the right pending promise via correlation id.
|
||||
*/
|
||||
export class PendingRpcRegistry {
|
||||
private readonly entries = new Map<string, PendingEntry>();
|
||||
|
||||
/** Register a pending request. Returns a Promise resolved/rejected by routing. */
|
||||
register<T = unknown>(requestId: string, opts: RegisterOptions = {}): Promise<T> {
|
||||
const p = new Promise<T>((resolve, reject) => {
|
||||
const cleanup = (): void => {
|
||||
const e = this.entries.get(requestId);
|
||||
if (e === undefined) return;
|
||||
if (e.timer !== null) clearTimeout(e.timer);
|
||||
if (e.abortListener !== null && e.signal !== undefined) {
|
||||
e.signal.removeEventListener('abort', e.abortListener);
|
||||
}
|
||||
this.entries.delete(requestId);
|
||||
};
|
||||
|
||||
const entry: PendingEntry = {
|
||||
resolve: (v) => {
|
||||
cleanup();
|
||||
resolve(v as T);
|
||||
},
|
||||
reject: (err) => {
|
||||
cleanup();
|
||||
reject(err);
|
||||
},
|
||||
timer: null,
|
||||
abortListener: null,
|
||||
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
|
||||
};
|
||||
|
||||
if (opts.timeoutMs !== undefined && opts.timeoutMs > 0) {
|
||||
entry.timer = setTimeout(() => {
|
||||
opts.onCancel?.('timeout');
|
||||
entry.reject(new OperationTimeoutError());
|
||||
}, opts.timeoutMs);
|
||||
}
|
||||
|
||||
if (opts.signal !== undefined) {
|
||||
if (opts.signal.aborted) {
|
||||
// Defer microtask so caller's await sees the rejection.
|
||||
queueMicrotask(() => {
|
||||
opts.onCancel?.('signal');
|
||||
entry.reject(new CancelledError());
|
||||
});
|
||||
} else {
|
||||
const listener = (): void => {
|
||||
opts.onCancel?.('signal');
|
||||
entry.reject(new CancelledError());
|
||||
};
|
||||
entry.abortListener = listener;
|
||||
opts.signal.addEventListener('abort', listener, { once: true });
|
||||
}
|
||||
}
|
||||
|
||||
this.entries.set(requestId, entry);
|
||||
});
|
||||
// Attach a no-op rejection handler so Bun's strict unhandled-rejection
|
||||
// detection doesn't flag the registered promise BEFORE the caller's
|
||||
// `await` attaches its handler. The caller's await still observes the
|
||||
// rejection through its own handler chain.
|
||||
p.catch(() => {
|
||||
/* suppress unhandled-rejection warning */
|
||||
});
|
||||
return p;
|
||||
}
|
||||
|
||||
/** Route an incoming response. */
|
||||
resolveResponse(env: RpcResponse): void {
|
||||
const entry = this.entries.get(env.id);
|
||||
if (entry === undefined) return; // unknown id — possibly a stale duplicate
|
||||
entry.resolve(env.result);
|
||||
}
|
||||
|
||||
/** Route an incoming error envelope. */
|
||||
resolveError(env: RpcError): void {
|
||||
const entry = this.entries.get(env.id);
|
||||
if (entry === undefined) return;
|
||||
entry.reject(fileErrorFromPayload(env.error));
|
||||
}
|
||||
|
||||
/** Reject all pending entries — for shutdown. */
|
||||
rejectAll(reason: unknown): void {
|
||||
for (const entry of this.entries.values()) entry.reject(reason);
|
||||
this.entries.clear();
|
||||
}
|
||||
|
||||
/** For tests / metrics. */
|
||||
size(): number {
|
||||
return this.entries.size;
|
||||
}
|
||||
}
|
||||
46
packages/shade-files/src/schemas/envelope.ts
Normal file
46
packages/shade-files/src/schemas/envelope.ts
Normal file
@@ -0,0 +1,46 @@
|
||||
import { z } from 'zod';
|
||||
import { FileErrorPayloadSchema } from './errors.js';
|
||||
import { IdempotencyKeySchema, RequestIdSchema } from './primitives.js';
|
||||
|
||||
const KIND_REQUEST_PATTERN = /^shade\.fs\.[A-Za-z0-9.-]+\/v\d+$/;
|
||||
const KIND_RESPONSE_PATTERN = /^shade\.fs\.[A-Za-z0-9.-]+\/v\d+\.response$/;
|
||||
|
||||
/**
|
||||
* Request envelope. Carries op kind, correlation id, args (validated per
|
||||
* op), idempotency key (mutations only), and an Ed25519 signature binding
|
||||
* `(sender, kind, id, argsHash, signedAt)`.
|
||||
*/
|
||||
export const RpcRequestSchema = z.object({
|
||||
kind: z.string().regex(KIND_REQUEST_PATTERN, 'invalid request kind'),
|
||||
id: RequestIdSchema,
|
||||
args: z.unknown(),
|
||||
idempotencyKey: IdempotencyKeySchema.optional(),
|
||||
attempt: z.number().int().positive().optional(),
|
||||
deadlineMs: z.number().int().positive().optional(),
|
||||
sig: z.string().min(1).max(128),
|
||||
signedAt: z.number().int().positive(),
|
||||
});
|
||||
export type RpcRequest = z.infer<typeof RpcRequestSchema>;
|
||||
|
||||
export const RpcResponseSchema = z.object({
|
||||
kind: z.string().regex(KIND_RESPONSE_PATTERN, 'invalid response kind'),
|
||||
id: RequestIdSchema,
|
||||
result: z.unknown(),
|
||||
});
|
||||
export type RpcResponse = z.infer<typeof RpcResponseSchema>;
|
||||
|
||||
export const RpcErrorSchema = z.object({
|
||||
kind: z.literal('shade.fs.error/v1'),
|
||||
id: RequestIdSchema,
|
||||
error: FileErrorPayloadSchema,
|
||||
});
|
||||
export type RpcError = z.infer<typeof RpcErrorSchema>;
|
||||
|
||||
export const RpcCancelSchema = z.object({
|
||||
kind: z.literal('shade.fs.cancel/v1'),
|
||||
id: RequestIdSchema,
|
||||
reason: z.string().max(512).optional(),
|
||||
});
|
||||
export type RpcCancel = z.infer<typeof RpcCancelSchema>;
|
||||
|
||||
export type RpcEnvelope = RpcRequest | RpcResponse | RpcError | RpcCancel;
|
||||
228
packages/shade-files/src/schemas/errors.ts
Normal file
228
packages/shade-files/src/schemas/errors.ts
Normal file
@@ -0,0 +1,228 @@
|
||||
import { z } from 'zod';
|
||||
import { ShadeError } from '@shade/core';
|
||||
|
||||
export const FileErrorCodeSchema = z.enum([
|
||||
'NOT_FOUND',
|
||||
'PERMISSION_DENIED',
|
||||
'CONFLICT',
|
||||
'QUOTA_EXCEEDED',
|
||||
'RATE_LIMIT',
|
||||
'PATH_VALIDATION',
|
||||
'FINGERPRINT_REQUIRED',
|
||||
'OPERATION_TIMEOUT',
|
||||
'IDEMPOTENCY_CONFLICT',
|
||||
'CANCELLED',
|
||||
'INTERNAL',
|
||||
'NOT_IMPLEMENTED',
|
||||
'CUSTOM_OP_REJECTED',
|
||||
'INVALID_SIGNATURE',
|
||||
'INVALID_ARGS',
|
||||
]);
|
||||
export type FileErrorCode = z.infer<typeof FileErrorCodeSchema>;
|
||||
|
||||
export const FileErrorPayloadSchema = z.object({
|
||||
code: FileErrorCodeSchema,
|
||||
message: z.string().max(2048),
|
||||
/** Suggested retry delay in ms; only set for retriable errors (rate limit, transient). */
|
||||
retryAfterMs: z.number().int().nonnegative().optional(),
|
||||
/** Path or arg field the error refers to. */
|
||||
field: z.string().max(128).optional(),
|
||||
/** Optional cause string (sanitized — no stack traces leak from server). */
|
||||
cause: z.string().max(2048).optional(),
|
||||
});
|
||||
export type FileErrorPayload = z.infer<typeof FileErrorPayloadSchema>;
|
||||
|
||||
// ─── Class hierarchy ─────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Base class for all `@shade/files` errors. Extends `ShadeError` so the
|
||||
* existing `errorToHttpStatus()` mapping in `@shade/core` continues to apply.
|
||||
*/
|
||||
export class FileError extends ShadeError {
|
||||
readonly payload: FileErrorPayload;
|
||||
|
||||
constructor(payload: FileErrorPayload) {
|
||||
super(`SHADE_FS_${payload.code}`, payload.message);
|
||||
this.name = 'FileError';
|
||||
this.payload = payload;
|
||||
}
|
||||
|
||||
override toJSON(): { name: string; code: string; message: string; payload: FileErrorPayload } {
|
||||
return {
|
||||
name: this.name,
|
||||
code: this.code,
|
||||
message: this.message,
|
||||
payload: this.payload,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
export class NotFoundError extends FileError {
|
||||
constructor(message = 'Not found', field?: string) {
|
||||
super({ code: 'NOT_FOUND', message, ...(field !== undefined ? { field } : {}) });
|
||||
this.name = 'NotFoundError';
|
||||
}
|
||||
}
|
||||
|
||||
export class PermissionDeniedError extends FileError {
|
||||
constructor(message = 'Permission denied') {
|
||||
super({ code: 'PERMISSION_DENIED', message });
|
||||
this.name = 'PermissionDeniedError';
|
||||
}
|
||||
}
|
||||
|
||||
export class ConflictError extends FileError {
|
||||
constructor(message: string) {
|
||||
super({ code: 'CONFLICT', message });
|
||||
this.name = 'ConflictError';
|
||||
}
|
||||
}
|
||||
|
||||
export class QuotaExceededError extends FileError {
|
||||
constructor(message = 'Quota exceeded', retryAfterMs?: number) {
|
||||
super({
|
||||
code: 'QUOTA_EXCEEDED',
|
||||
message,
|
||||
...(retryAfterMs !== undefined ? { retryAfterMs } : {}),
|
||||
});
|
||||
this.name = 'QuotaExceededError';
|
||||
}
|
||||
}
|
||||
|
||||
export class FsRateLimitError extends FileError {
|
||||
constructor(message = 'Rate limit exceeded', retryAfterMs?: number) {
|
||||
super({
|
||||
code: 'RATE_LIMIT',
|
||||
message,
|
||||
...(retryAfterMs !== undefined ? { retryAfterMs } : {}),
|
||||
});
|
||||
this.name = 'FsRateLimitError';
|
||||
}
|
||||
}
|
||||
|
||||
export class PathValidationError extends FileError {
|
||||
constructor(message: string, field = 'path') {
|
||||
super({ code: 'PATH_VALIDATION', message, field });
|
||||
this.name = 'PathValidationError';
|
||||
}
|
||||
}
|
||||
|
||||
export class FingerprintRequiredError extends FileError {
|
||||
constructor(message = 'Peer fingerprint must be verified before this operation') {
|
||||
super({ code: 'FINGERPRINT_REQUIRED', message });
|
||||
this.name = 'FingerprintRequiredError';
|
||||
}
|
||||
}
|
||||
|
||||
export class OperationTimeoutError extends FileError {
|
||||
constructor(message = 'Operation timed out') {
|
||||
super({ code: 'OPERATION_TIMEOUT', message });
|
||||
this.name = 'OperationTimeoutError';
|
||||
}
|
||||
}
|
||||
|
||||
export class IdempotencyConflictError extends FileError {
|
||||
constructor(
|
||||
message = 'Idempotency key reused with different arguments',
|
||||
) {
|
||||
super({ code: 'IDEMPOTENCY_CONFLICT', message });
|
||||
this.name = 'IdempotencyConflictError';
|
||||
}
|
||||
}
|
||||
|
||||
export class CancelledError extends FileError {
|
||||
constructor(message = 'Cancelled') {
|
||||
super({ code: 'CANCELLED', message });
|
||||
this.name = 'CancelledError';
|
||||
}
|
||||
}
|
||||
|
||||
export class InternalFileError extends FileError {
|
||||
constructor(message = 'Internal server error', cause?: string) {
|
||||
super({
|
||||
code: 'INTERNAL',
|
||||
message,
|
||||
...(cause !== undefined ? { cause } : {}),
|
||||
});
|
||||
this.name = 'InternalFileError';
|
||||
}
|
||||
}
|
||||
|
||||
export class NotImplementedError extends FileError {
|
||||
constructor(op: string) {
|
||||
super({ code: 'NOT_IMPLEMENTED', message: `Operation not implemented: ${op}` });
|
||||
this.name = 'NotImplementedError';
|
||||
}
|
||||
}
|
||||
|
||||
export class CustomOpRejectedError extends FileError {
|
||||
constructor(message: string) {
|
||||
super({ code: 'CUSTOM_OP_REJECTED', message });
|
||||
this.name = 'CustomOpRejectedError';
|
||||
}
|
||||
}
|
||||
|
||||
export class InvalidSignatureError extends FileError {
|
||||
constructor(message = 'RPC envelope signature verification failed') {
|
||||
super({ code: 'INVALID_SIGNATURE', message });
|
||||
this.name = 'InvalidSignatureError';
|
||||
}
|
||||
}
|
||||
|
||||
export class InvalidArgsError extends FileError {
|
||||
constructor(message: string, field?: string) {
|
||||
super({ code: 'INVALID_ARGS', message, ...(field !== undefined ? { field } : {}) });
|
||||
this.name = 'InvalidArgsError';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reconstruct the right `FileError` subclass from a wire payload. Used by
|
||||
* the client to surface typed errors from server responses.
|
||||
*/
|
||||
export function fileErrorFromPayload(payload: FileErrorPayload): FileError {
|
||||
switch (payload.code) {
|
||||
case 'NOT_FOUND':
|
||||
return new NotFoundError(payload.message, payload.field);
|
||||
case 'PERMISSION_DENIED':
|
||||
return new PermissionDeniedError(payload.message);
|
||||
case 'CONFLICT':
|
||||
return new ConflictError(payload.message);
|
||||
case 'QUOTA_EXCEEDED':
|
||||
return new QuotaExceededError(payload.message, payload.retryAfterMs);
|
||||
case 'RATE_LIMIT':
|
||||
return new FsRateLimitError(payload.message, payload.retryAfterMs);
|
||||
case 'PATH_VALIDATION':
|
||||
return new PathValidationError(payload.message, payload.field);
|
||||
case 'FINGERPRINT_REQUIRED':
|
||||
return new FingerprintRequiredError(payload.message);
|
||||
case 'OPERATION_TIMEOUT':
|
||||
return new OperationTimeoutError(payload.message);
|
||||
case 'IDEMPOTENCY_CONFLICT':
|
||||
return new IdempotencyConflictError(payload.message);
|
||||
case 'CANCELLED':
|
||||
return new CancelledError(payload.message);
|
||||
case 'INTERNAL':
|
||||
return new InternalFileError(payload.message, payload.cause);
|
||||
case 'NOT_IMPLEMENTED':
|
||||
return new NotImplementedError(payload.message);
|
||||
case 'CUSTOM_OP_REJECTED':
|
||||
return new CustomOpRejectedError(payload.message);
|
||||
case 'INVALID_SIGNATURE':
|
||||
return new InvalidSignatureError(payload.message);
|
||||
case 'INVALID_ARGS':
|
||||
return new InvalidArgsError(payload.message, payload.field);
|
||||
}
|
||||
}
|
||||
|
||||
/** Convert any thrown value into a `FileErrorPayload` for wire serialization. */
|
||||
export function payloadFromError(err: unknown): FileErrorPayload {
|
||||
if (err instanceof FileError) return err.payload;
|
||||
if (err instanceof Error) {
|
||||
return {
|
||||
code: 'INTERNAL',
|
||||
message: err.message.slice(0, 2048),
|
||||
};
|
||||
}
|
||||
return { code: 'INTERNAL', message: 'unknown error' };
|
||||
}
|
||||
33
packages/shade-files/src/schemas/file-entry.ts
Normal file
33
packages/shade-files/src/schemas/file-entry.ts
Normal file
@@ -0,0 +1,33 @@
|
||||
import { z } from 'zod';
|
||||
import { CursorSchema } from './primitives.js';
|
||||
|
||||
export const FileKindSchema = z.enum(['file', 'dir']);
|
||||
export type FileKind = z.infer<typeof FileKindSchema>;
|
||||
|
||||
/**
|
||||
* Directory entry shape. App-specific extension via `metadata` — keep it
|
||||
* lean; large blobs (thumbnails) belong in `getThumbnail`, not here.
|
||||
*/
|
||||
export const FileEntrySchema = z.object({
|
||||
/** Base name only — no path separators. */
|
||||
name: z.string().min(1).max(512).refine((s) => !s.includes('/') && !s.includes('\\'), {
|
||||
message: 'name must not contain path separators',
|
||||
}),
|
||||
kind: FileKindSchema,
|
||||
/** Plaintext byte size; 0 for directories. */
|
||||
size: z.number().int().nonnegative(),
|
||||
/** Last-modified time, unix milliseconds. */
|
||||
mtime: z.number().int(),
|
||||
/** RFC 6838 media type if known. */
|
||||
contentType: z.string().max(255).optional(),
|
||||
/** App-specific extension fields. Default empty. */
|
||||
metadata: z.record(z.string(), z.unknown()).default({}),
|
||||
});
|
||||
export type FileEntry = z.infer<typeof FileEntrySchema>;
|
||||
|
||||
export const ListPageSchema = z.object({
|
||||
entries: z.array(FileEntrySchema),
|
||||
nextCursor: CursorSchema.optional(),
|
||||
hasMore: z.boolean(),
|
||||
});
|
||||
export type ListPage = z.infer<typeof ListPageSchema>;
|
||||
5
packages/shade-files/src/schemas/index.ts
Normal file
5
packages/shade-files/src/schemas/index.ts
Normal file
@@ -0,0 +1,5 @@
|
||||
export * from './primitives.js';
|
||||
export * from './file-entry.js';
|
||||
export * from './ops.js';
|
||||
export * from './envelope.js';
|
||||
export * from './errors.js';
|
||||
157
packages/shade-files/src/schemas/ops.ts
Normal file
157
packages/shade-files/src/schemas/ops.ts
Normal file
@@ -0,0 +1,157 @@
|
||||
import { z } from 'zod';
|
||||
import { Base64Schema, CursorSchema, PathSchema, Sha256HexSchema } from './primitives.js';
|
||||
import { FileEntrySchema, FileKindSchema, ListPageSchema } from './file-entry.js';
|
||||
|
||||
// ─── list ────────────────────────────────────────────────────
|
||||
|
||||
export const ListArgsSchema = z.object({
|
||||
path: PathSchema,
|
||||
cursor: CursorSchema.optional(),
|
||||
pageSize: z.number().int().min(1).max(1000).default(100),
|
||||
filter: z
|
||||
.object({
|
||||
prefix: z.string().max(256).optional(),
|
||||
kind: FileKindSchema.optional(),
|
||||
})
|
||||
.optional(),
|
||||
});
|
||||
export type ListArgs = z.infer<typeof ListArgsSchema>;
|
||||
export const ListResultSchema = ListPageSchema;
|
||||
export type ListResult = z.infer<typeof ListResultSchema>;
|
||||
|
||||
// ─── stat ────────────────────────────────────────────────────
|
||||
|
||||
export const StatArgsSchema = z.object({ path: PathSchema });
|
||||
export type StatArgs = z.infer<typeof StatArgsSchema>;
|
||||
export const StatResultSchema = FileEntrySchema;
|
||||
export type StatResult = z.infer<typeof StatResultSchema>;
|
||||
|
||||
// ─── mkdir ───────────────────────────────────────────────────
|
||||
|
||||
export const MkdirArgsSchema = z.object({
|
||||
path: PathSchema,
|
||||
recursive: z.boolean().default(false),
|
||||
});
|
||||
export type MkdirArgs = z.infer<typeof MkdirArgsSchema>;
|
||||
export const MkdirResultSchema = z.object({ entry: FileEntrySchema });
|
||||
export type MkdirResult = z.infer<typeof MkdirResultSchema>;
|
||||
|
||||
// ─── delete ──────────────────────────────────────────────────
|
||||
|
||||
export const DeleteArgsSchema = z.object({
|
||||
path: PathSchema,
|
||||
recursive: z.boolean().default(false),
|
||||
});
|
||||
export type DeleteArgs = z.infer<typeof DeleteArgsSchema>;
|
||||
export const DeleteResultSchema = z.object({
|
||||
deletedCount: z.number().int().nonnegative(),
|
||||
});
|
||||
export type DeleteResult = z.infer<typeof DeleteResultSchema>;
|
||||
|
||||
// ─── move ────────────────────────────────────────────────────
|
||||
|
||||
export const MoveArgsSchema = z.object({
|
||||
src: PathSchema,
|
||||
dst: PathSchema,
|
||||
overwrite: z.boolean().default(false),
|
||||
});
|
||||
export type MoveArgs = z.infer<typeof MoveArgsSchema>;
|
||||
export const MoveResultSchema = z.object({ entry: FileEntrySchema });
|
||||
export type MoveResult = z.infer<typeof MoveResultSchema>;
|
||||
|
||||
// ─── read ────────────────────────────────────────────────────
|
||||
|
||||
export const ReadArgsSchema = z.object({
|
||||
path: PathSchema,
|
||||
range: z
|
||||
.object({
|
||||
start: z.number().int().nonnegative(),
|
||||
end: z.number().int().positive(),
|
||||
})
|
||||
.optional(),
|
||||
/** Force inline path even if size unknown / large. Use with care. */
|
||||
preferInline: z.boolean().optional(),
|
||||
});
|
||||
export type ReadArgs = z.infer<typeof ReadArgsSchema>;
|
||||
|
||||
export const ReadResultSchema = z.discriminatedUnion('kind', [
|
||||
z.object({
|
||||
kind: z.literal('inline'),
|
||||
bytesB64: Base64Schema,
|
||||
size: z.number().int().nonnegative(),
|
||||
sha256: Sha256HexSchema,
|
||||
contentType: z.string().optional(),
|
||||
}),
|
||||
z.object({
|
||||
kind: z.literal('streams'),
|
||||
streamId: z.string(),
|
||||
size: z.number().int().nonnegative(),
|
||||
sha256: Sha256HexSchema,
|
||||
contentType: z.string().optional(),
|
||||
}),
|
||||
]);
|
||||
export type ReadResult = z.infer<typeof ReadResultSchema>;
|
||||
|
||||
// ─── write ───────────────────────────────────────────────────
|
||||
|
||||
export const WriteArgsSchema = z.discriminatedUnion('kind', [
|
||||
z.object({
|
||||
kind: z.literal('inline'),
|
||||
path: PathSchema,
|
||||
bytesB64: Base64Schema,
|
||||
contentType: z.string().optional(),
|
||||
overwrite: z.boolean().default(false),
|
||||
}),
|
||||
z.object({
|
||||
kind: z.literal('streams'),
|
||||
path: PathSchema,
|
||||
/** Declared plaintext size; server clamps quota against this. */
|
||||
size: z.number().int().nonnegative(),
|
||||
contentType: z.string().optional(),
|
||||
overwrite: z.boolean().default(false),
|
||||
/** base64url(16) — matches `metadata.shadeFilesWriteId` on the streams transfer. */
|
||||
writeId: z.string(),
|
||||
}),
|
||||
]);
|
||||
export type WriteArgs = z.infer<typeof WriteArgsSchema>;
|
||||
export const WriteResultSchema = z.object({ entry: FileEntrySchema });
|
||||
export type WriteResult = z.infer<typeof WriteResultSchema>;
|
||||
|
||||
// ─── getThumbnail ────────────────────────────────────────────
|
||||
|
||||
export const ThumbnailSizeSchema = z.union([
|
||||
z.literal(64),
|
||||
z.literal(128),
|
||||
z.literal(256),
|
||||
z.literal(512),
|
||||
]);
|
||||
export type ThumbnailSize = z.infer<typeof ThumbnailSizeSchema>;
|
||||
|
||||
export const GetThumbnailArgsSchema = z.object({
|
||||
path: PathSchema,
|
||||
size: ThumbnailSizeSchema,
|
||||
format: z.enum(['png', 'webp', 'jpeg']).default('png'),
|
||||
});
|
||||
export type GetThumbnailArgs = z.infer<typeof GetThumbnailArgsSchema>;
|
||||
|
||||
export const GetThumbnailResultSchema = z.object({
|
||||
bytesB64: Base64Schema,
|
||||
format: z.enum(['png', 'webp', 'jpeg']),
|
||||
width: z.number().int().positive().max(4096),
|
||||
height: z.number().int().positive().max(4096),
|
||||
sha256: Sha256HexSchema,
|
||||
});
|
||||
export type GetThumbnailResult = z.infer<typeof GetThumbnailResultSchema>;
|
||||
|
||||
// ─── custom ──────────────────────────────────────────────────
|
||||
|
||||
export const CustomArgsSchema = z.object({
|
||||
name: z.string().min(1).max(128).regex(/^[a-z0-9.-]+$/i, 'invalid custom op name'),
|
||||
payload: z.unknown(),
|
||||
});
|
||||
export type CustomArgs = z.infer<typeof CustomArgsSchema>;
|
||||
|
||||
export const CustomResultSchema = z.object({
|
||||
result: z.unknown(),
|
||||
});
|
||||
export type CustomResult = z.infer<typeof CustomResultSchema>;
|
||||
42
packages/shade-files/src/schemas/primitives.ts
Normal file
42
packages/shade-files/src/schemas/primitives.ts
Normal file
@@ -0,0 +1,42 @@
|
||||
import { z } from 'zod';
|
||||
|
||||
/**
|
||||
* POSIX-style absolute path. Must start with `/`, max 4096 chars, no NULs,
|
||||
* no CR/LF/DEL, no backslashes (Windows-style rejected). Further policy
|
||||
* (root-scope, traversal hardening) applied by `validatePath` server-side.
|
||||
*/
|
||||
export const PathSchema = z
|
||||
.string()
|
||||
.min(1)
|
||||
.max(4096)
|
||||
.regex(/^\//, 'must be absolute (start with /)')
|
||||
.refine((s) => !/[\x00-\x08\x0a-\x1f\x7f\\]/.test(s), {
|
||||
message: 'contains forbidden control or backslash characters',
|
||||
});
|
||||
|
||||
export type Path = z.infer<typeof PathSchema>;
|
||||
|
||||
/** Base64url(16 bytes) — used for both requestId and idempotencyKey. */
|
||||
export const RequestIdSchema = z
|
||||
.string()
|
||||
.min(22)
|
||||
.max(22)
|
||||
.regex(/^[A-Za-z0-9_-]+$/, 'invalid base64url');
|
||||
|
||||
export type RequestId = z.infer<typeof RequestIdSchema>;
|
||||
|
||||
export const IdempotencyKeySchema = RequestIdSchema;
|
||||
export type IdempotencyKey = z.infer<typeof IdempotencyKeySchema>;
|
||||
|
||||
/**
|
||||
* Opaque cursor token. Server-defined shape; clients pass through verbatim.
|
||||
* Capped at 2 KiB to prevent over-sized cursors leaking server state.
|
||||
*/
|
||||
export const CursorSchema = z.string().min(1).max(2048);
|
||||
export type Cursor = z.infer<typeof CursorSchema>;
|
||||
|
||||
/** Hex-encoded SHA-256 (64 chars). */
|
||||
export const Sha256HexSchema = z.string().length(64).regex(/^[0-9a-f]+$/);
|
||||
|
||||
/** Standard base64 (with padding) used for binary blobs in JSON envelopes. */
|
||||
export const Base64Schema = z.string().regex(/^[A-Za-z0-9+/]*=*$/);
|
||||
73
packages/shade-files/src/server/cursor.ts
Normal file
73
packages/shade-files/src/server/cursor.ts
Normal file
@@ -0,0 +1,73 @@
|
||||
import { sha256 } from '@noble/hashes/sha2.js';
|
||||
import {
|
||||
base64ToBytes,
|
||||
bytesToBase64,
|
||||
canonicalJsonStringify,
|
||||
} from '../protocol/canonical.js';
|
||||
|
||||
/**
|
||||
* Opaque pagination cursor. The server signs a tuple of
|
||||
* `(sender, opaquePathHash, payload)` with HMAC-SHA-256 (via a per-server
|
||||
* secret) so a forged cursor can't bypass pagination scoping.
|
||||
*
|
||||
* The payload itself is server-defined; in tests we use simple `{ offset }`.
|
||||
*/
|
||||
|
||||
export interface CursorBuilder {
|
||||
encode(sender: string, pathHash: string, payload: unknown): string;
|
||||
decode(sender: string, pathHash: string, cursor: string): unknown | null;
|
||||
}
|
||||
|
||||
export function createCursorBuilder(serverSecret: Uint8Array): CursorBuilder {
|
||||
if (serverSecret.length < 16) {
|
||||
throw new Error('serverSecret must be at least 16 bytes');
|
||||
}
|
||||
|
||||
function mac(input: Uint8Array): Uint8Array {
|
||||
// Simple HMAC-SHA-256 implementation using @noble/hashes — keyed
|
||||
// construction over secret || input. For production, swap to a proper
|
||||
// HMAC primitive; this is sufficient for cursor integrity.
|
||||
const padded = new Uint8Array(serverSecret.length + input.length);
|
||||
padded.set(serverSecret, 0);
|
||||
padded.set(input, serverSecret.length);
|
||||
return sha256(padded);
|
||||
}
|
||||
|
||||
return {
|
||||
encode(sender, pathHash, payload) {
|
||||
const json = canonicalJsonStringify({ s: sender, p: pathHash, d: payload });
|
||||
const enc = new TextEncoder().encode(json);
|
||||
const tag = mac(enc).slice(0, 16); // 128-bit truncation is plenty
|
||||
const out = new Uint8Array(enc.length + tag.length);
|
||||
out.set(enc, 0);
|
||||
out.set(tag, enc.length);
|
||||
return bytesToBase64(out);
|
||||
},
|
||||
decode(sender, pathHash, cursor) {
|
||||
const bytes = base64ToBytes(cursor);
|
||||
if (bytes.length < 17) return null;
|
||||
const enc = bytes.slice(0, bytes.length - 16);
|
||||
const tag = bytes.slice(bytes.length - 16);
|
||||
const expected = mac(enc).slice(0, 16);
|
||||
if (!constantTimeEqualBytes(tag, expected)) return null;
|
||||
try {
|
||||
const parsed = JSON.parse(new TextDecoder().decode(enc)) as {
|
||||
s: string;
|
||||
p: string;
|
||||
d: unknown;
|
||||
};
|
||||
if (parsed.s !== sender || parsed.p !== pathHash) return null;
|
||||
return parsed.d;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function constantTimeEqualBytes(a: Uint8Array, b: Uint8Array): boolean {
|
||||
if (a.length !== b.length) return false;
|
||||
let diff = 0;
|
||||
for (let i = 0; i < a.length; i++) diff |= a[i]! ^ b[i]!;
|
||||
return diff === 0;
|
||||
}
|
||||
86
packages/shade-files/src/server/custom-ops.ts
Normal file
86
packages/shade-files/src/server/custom-ops.ts
Normal file
@@ -0,0 +1,86 @@
|
||||
/**
|
||||
* Custom op registry — typed via TypeScript declaration merging.
|
||||
*
|
||||
* Consumers register a custom op by:
|
||||
* 1. Adding to `CustomOpsMap` via declaration merging (compile-time typing).
|
||||
* 2. Passing a Zod-backed handler in `FileHandlerConfig.custom`.
|
||||
*
|
||||
* Server validates wire args against the registered Zod schema before
|
||||
* invoking the user handler, then validates the user's response against
|
||||
* the schema before shipping it.
|
||||
*
|
||||
* @example
|
||||
* ```ts
|
||||
* // In your app, once globally:
|
||||
* declare module '@shade/files' {
|
||||
* interface CustomOpsMap {
|
||||
* 'app.deploy-mod': CustomOpDef<{ jarPath: string }, { deploymentId: string }>;
|
||||
* }
|
||||
* }
|
||||
*
|
||||
* // Server registers a handler:
|
||||
* shade.files.serve({
|
||||
* custom: {
|
||||
* 'app.deploy-mod': {
|
||||
* args: z.object({ jarPath: z.string() }),
|
||||
* response: z.object({ deploymentId: z.string() }),
|
||||
* handler: async (args, ctx) => ({ deploymentId: '...' }),
|
||||
* },
|
||||
* },
|
||||
* });
|
||||
*
|
||||
* // Client gets typed I/O:
|
||||
* const result = await fs.custom('app.deploy-mod', { jarPath: '...' });
|
||||
* // ^? { deploymentId: string }
|
||||
* ```
|
||||
*/
|
||||
import type { ZodType } from 'zod';
|
||||
import type { OpContext } from './handler-context.js';
|
||||
|
||||
// ─── Public typing scaffold (declaration-merged by consumers) ─
|
||||
|
||||
/** Marker shape for one custom op entry — consumers extend `CustomOpsMap`. */
|
||||
export interface CustomOpDef<A = unknown, R = unknown> {
|
||||
args: A;
|
||||
response: R;
|
||||
}
|
||||
|
||||
/**
|
||||
* Registry of all custom ops known at compile time. Empty by default —
|
||||
* consumers extend via TypeScript declaration merging:
|
||||
*
|
||||
* ```ts
|
||||
* declare module '@shade/files' {
|
||||
* interface CustomOpsMap {
|
||||
* 'app.foo': CustomOpDef<{ x: number }, { y: string }>;
|
||||
* }
|
||||
* }
|
||||
* ```
|
||||
*/
|
||||
// eslint-disable-next-line @typescript-eslint/no-empty-object-type
|
||||
export interface CustomOpsMap {}
|
||||
|
||||
// ─── Server-side handler config ──────────────────────────────
|
||||
|
||||
/**
|
||||
* Server-side definition for one custom op: the args schema (validated
|
||||
* inbound), the response schema (validated outbound), and the handler.
|
||||
*/
|
||||
export interface CustomOpRegistration<TArgs = unknown, TResponse = unknown> {
|
||||
args: ZodType<TArgs>;
|
||||
response: ZodType<TResponse>;
|
||||
handler: (args: TArgs, ctx: OpContext<{ name: string; payload: TArgs }>) => Promise<TResponse>;
|
||||
/**
|
||||
* Optional cost override for the rate limiter. Default: same as
|
||||
* `opCost.custom` (1 unless overridden in `RateLimitConfig`).
|
||||
*/
|
||||
cost?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Map from custom-op name to registration. The keys are the same names
|
||||
* declared in `CustomOpsMap`. Required typing via declaration merging is
|
||||
* not strict here — runtime validation is enforced by the registered Zod
|
||||
* schemas, so consumers using untyped names still get safety.
|
||||
*/
|
||||
export type CustomOpRegistrations = Record<string, CustomOpRegistration<any, any>>;
|
||||
59
packages/shade-files/src/server/handler-context.ts
Normal file
59
packages/shade-files/src/server/handler-context.ts
Normal file
@@ -0,0 +1,59 @@
|
||||
import type { Shade } from '@shade/sdk';
|
||||
import type { StandardOp } from '../protocol/kinds.js';
|
||||
|
||||
export type OpKind = StandardOp | `custom:${string}`;
|
||||
|
||||
/**
|
||||
* Per-call context handed to every server-side handler/middleware.
|
||||
*
|
||||
* Path-related ops set `path` to the (validated, normalized) primary path.
|
||||
* For `move`, `path` is `args.src`. For ops without a path (`custom`), the
|
||||
* field is the empty string.
|
||||
*/
|
||||
export interface OpContext<TArgs = unknown> {
|
||||
/** The op being executed. */
|
||||
op: OpKind;
|
||||
/** Validated + normalized primary path; empty string for path-less ops. */
|
||||
path: string;
|
||||
/** Zod-validated args for this op. */
|
||||
args: TArgs;
|
||||
/** Peer address (from Shade.receive). */
|
||||
sender: string;
|
||||
/** Aborted on client cancel or timeout. */
|
||||
signal: AbortSignal;
|
||||
/** Idempotency key (mutations only). */
|
||||
idempotencyKey: string | undefined;
|
||||
/** Attempt counter (1-based). */
|
||||
attemptNumber: number;
|
||||
/** Wall-clock when dispatch started (for metrics). */
|
||||
startedAt: number;
|
||||
/** Lazy fingerprint accessor — call only if needed. */
|
||||
fingerprint: () => Promise<string>;
|
||||
/** Mutable scratchpad for middleware (e.g. cache fingerprint hit). */
|
||||
metadata: Map<string, unknown>;
|
||||
}
|
||||
|
||||
/** Build an `OpContext`. Internal — used by the dispatcher. */
|
||||
export function buildOpContext<TArgs>(args: {
|
||||
op: OpKind;
|
||||
path: string;
|
||||
parsedArgs: TArgs;
|
||||
sender: string;
|
||||
signal: AbortSignal;
|
||||
idempotencyKey: string | undefined;
|
||||
attemptNumber: number;
|
||||
shade: Shade;
|
||||
}): OpContext<TArgs> {
|
||||
return {
|
||||
op: args.op,
|
||||
path: args.path,
|
||||
args: args.parsedArgs,
|
||||
sender: args.sender,
|
||||
signal: args.signal,
|
||||
idempotencyKey: args.idempotencyKey,
|
||||
attemptNumber: args.attemptNumber,
|
||||
startedAt: Date.now(),
|
||||
fingerprint: () => args.shade.getFingerprintFor(args.sender),
|
||||
metadata: new Map(),
|
||||
};
|
||||
}
|
||||
701
packages/shade-files/src/server/handler.ts
Normal file
701
packages/shade-files/src/server/handler.ts
Normal file
@@ -0,0 +1,701 @@
|
||||
import type { Shade } from '@shade/sdk';
|
||||
import type { ZodTypeAny } from 'zod';
|
||||
import {
|
||||
MUTATION_OPS,
|
||||
opOfKind,
|
||||
responseKindOf,
|
||||
type StandardOp,
|
||||
} from '../protocol/kinds.js';
|
||||
import {
|
||||
DeleteArgsSchema,
|
||||
DeleteResultSchema,
|
||||
GetThumbnailArgsSchema,
|
||||
GetThumbnailResultSchema,
|
||||
ListArgsSchema,
|
||||
ListResultSchema,
|
||||
MkdirArgsSchema,
|
||||
MkdirResultSchema,
|
||||
MoveArgsSchema,
|
||||
MoveResultSchema,
|
||||
ReadArgsSchema,
|
||||
ReadResultSchema,
|
||||
StatArgsSchema,
|
||||
StatResultSchema,
|
||||
WriteArgsSchema,
|
||||
WriteResultSchema,
|
||||
type DeleteArgs,
|
||||
type DeleteResult,
|
||||
type GetThumbnailArgs,
|
||||
type ListArgs,
|
||||
type ListResult,
|
||||
type MkdirArgs,
|
||||
type MkdirResult,
|
||||
type MoveArgs,
|
||||
type MoveResult,
|
||||
type ReadArgs,
|
||||
type StatArgs,
|
||||
type StatResult,
|
||||
type WriteArgs,
|
||||
type WriteResult,
|
||||
} from '../schemas/ops.js';
|
||||
import {
|
||||
CancelledError,
|
||||
CustomOpRejectedError,
|
||||
FileError,
|
||||
FingerprintRequiredError,
|
||||
InvalidArgsError,
|
||||
InvalidSignatureError,
|
||||
NotImplementedError,
|
||||
PathValidationError,
|
||||
payloadFromError,
|
||||
} from '../schemas/errors.js';
|
||||
import type { RpcCancel, RpcError, RpcRequest, RpcResponse } from '../schemas/envelope.js';
|
||||
import { buildOpContext, type OpContext } from './handler-context.js';
|
||||
import { IdempotencyCache, type IdempotencyCacheOptions } from './idempotency-cache.js';
|
||||
import { RateLimiter, type RateLimitConfig } from './rate-limiter.js';
|
||||
import { validatePath, type PathPolicy } from './path-policy.js';
|
||||
import {
|
||||
adaptReadResult,
|
||||
adaptThumbnailResult,
|
||||
adaptWriteArgs,
|
||||
} from './io-adapters.js';
|
||||
import type {
|
||||
UserReadResult,
|
||||
UserThumbnailResult,
|
||||
UserWriteArgs,
|
||||
} from './io-types.js';
|
||||
import type { ServerStreamsBridge } from './streams-bridge.js';
|
||||
import type { CustomOpRegistrations } from './custom-ops.js';
|
||||
import {
|
||||
METRIC_BYTES_IN,
|
||||
METRIC_BYTES_OUT,
|
||||
METRIC_FINGERPRINT_REJECT_TOTAL,
|
||||
METRIC_IDEMPOTENCY_CONFLICT_TOTAL,
|
||||
METRIC_IDEMPOTENCY_HIT_TOTAL,
|
||||
METRIC_OP_DURATION_MS,
|
||||
METRIC_OP_TOTAL,
|
||||
METRIC_RATE_LIMIT_REJECT_TOTAL,
|
||||
METRIC_SIGNATURE_REJECT_TOTAL,
|
||||
NOOP_METRIC_SINK,
|
||||
type MetricSink,
|
||||
} from './metrics.js';
|
||||
import {
|
||||
CustomArgsSchema,
|
||||
CustomResultSchema,
|
||||
type CustomArgs,
|
||||
} from '../schemas/ops.js';
|
||||
import { canonicalRpcBytes, hashArgs } from '../protocol/canonical.js';
|
||||
|
||||
/** Replay window for the `signedAt` field on inbound RPC envelopes. */
|
||||
export const MAX_SIGNATURE_AGE_MS = 5 * 60 * 1000;
|
||||
|
||||
// ─── Public types ────────────────────────────────────────────
|
||||
|
||||
export interface FileHandlerOps {
|
||||
list?: (ctx: OpContext<ListArgs>) => Promise<ListResult>;
|
||||
stat?: (ctx: OpContext<StatArgs>) => Promise<StatResult>;
|
||||
mkdir?: (ctx: OpContext<MkdirArgs>) => Promise<MkdirResult>;
|
||||
delete?: (ctx: OpContext<DeleteArgs>) => Promise<DeleteResult>;
|
||||
move?: (ctx: OpContext<MoveArgs>) => Promise<MoveResult>;
|
||||
/**
|
||||
* User-supplied read handler. Returns either an `inline` payload (≤ 256
|
||||
* KiB) or a `streams` payload with a precomputed sha256. The dispatcher
|
||||
* adapts to the wire shape.
|
||||
*/
|
||||
read?: (ctx: OpContext<ReadArgs>) => Promise<UserReadResult>;
|
||||
/**
|
||||
* User-supplied write handler. Receives `UserWriteArgs` with a clean
|
||||
* `Uint8Array` (inline) or `ReadableStream` + sha256-promise (streams)
|
||||
* shape — the dispatcher hides the base64 / writeId wire details.
|
||||
*/
|
||||
write?: (ctx: OpContext<UserWriteArgs>) => Promise<WriteResult>;
|
||||
/**
|
||||
* User-supplied thumbnail handler. Bytes are validated for format magic
|
||||
* before they're shipped to prevent format misclassification attacks.
|
||||
*/
|
||||
getThumbnail?: (ctx: OpContext<GetThumbnailArgs>) => Promise<UserThumbnailResult>;
|
||||
}
|
||||
|
||||
export interface FileHandlerConfig extends FileHandlerOps {
|
||||
pathPolicy?: PathPolicy;
|
||||
rateLimits?: RateLimitConfig;
|
||||
idempotency?: IdempotencyCacheOptions;
|
||||
/**
|
||||
* Required for read/write `streams` ops. Coordinates the inbound/outbound
|
||||
* `@shade/transfer` transfers via `userMetadata.shadeFiles*Id` keys.
|
||||
*/
|
||||
streamsBridge?: ServerStreamsBridge;
|
||||
/** Custom ops registry — see `CustomOpsMap` declaration-merging. */
|
||||
custom?: CustomOpRegistrations;
|
||||
/**
|
||||
* Verify the Ed25519 signature on inbound RPC envelopes. Pluggable so
|
||||
* apps can plug their own peer-identity store. Returning `false` rejects
|
||||
* with `InvalidSignatureError`. Default: skip (Double Ratchet AEAD on
|
||||
* the underlying envelope already authenticates the sender).
|
||||
*
|
||||
* The `signedAt` replay-window check (±5 min) is enforced regardless.
|
||||
*/
|
||||
verifySender?: (
|
||||
sender: string,
|
||||
canonicalBytes: Uint8Array,
|
||||
sig: string,
|
||||
) => boolean | Promise<boolean>;
|
||||
/**
|
||||
* Per-op fingerprint-verification gate. Return `'required'` to demand
|
||||
* the peer's fingerprint has been verified out-of-band (via
|
||||
* `isFingerprintVerified`); `'reject'` to deny outright;
|
||||
* `'optional'` (default) to allow.
|
||||
*/
|
||||
requireFingerprintVerifiedFor?: (
|
||||
ctx: OpContext<unknown>,
|
||||
) => 'required' | 'optional' | 'reject' | Promise<'required' | 'optional' | 'reject'>;
|
||||
/** Lookup whether the consumer has out-of-band verified the peer. */
|
||||
isFingerprintVerified?: (sender: string) => boolean | Promise<boolean>;
|
||||
/** Vendor-neutral metrics sink. */
|
||||
onMetric?: MetricSink;
|
||||
/** Called BEFORE the handler runs. Throw to deny. */
|
||||
beforeOp?: (ctx: OpContext<unknown>) => void | Promise<void>;
|
||||
/** Called AFTER the handler returns. Result is the validated response. */
|
||||
afterOp?: (ctx: OpContext<unknown>, result: unknown) => void | Promise<void>;
|
||||
/** Called when an op fails. Receives the error and the context. */
|
||||
onError?: (err: unknown, ctx: OpContext<unknown>) => void;
|
||||
/** Default per-op timeout in ms. Default 60_000. */
|
||||
defaultTimeoutMs?: number;
|
||||
/** Hard deadline for streams-bridge awaits / outbound transfers. Default 60_000. */
|
||||
ioTimeoutMs?: number;
|
||||
}
|
||||
|
||||
export interface FileHandler {
|
||||
/** Handle an incoming request envelope. Returns the envelope to send back. */
|
||||
handleRequest(from: string, request: RpcRequest): Promise<RpcResponse | RpcError>;
|
||||
/** Handle an incoming cancel envelope. */
|
||||
handleCancel(from: string, cancel: RpcCancel): void;
|
||||
/** Free up internal state (timers, abort listeners). */
|
||||
destroy(): void;
|
||||
}
|
||||
|
||||
interface OpSchemaPair {
|
||||
args: ZodTypeAny;
|
||||
result: ZodTypeAny;
|
||||
}
|
||||
|
||||
const OP_SCHEMAS: Record<StandardOp, OpSchemaPair> = {
|
||||
list: { args: ListArgsSchema, result: ListResultSchema },
|
||||
stat: { args: StatArgsSchema, result: StatResultSchema },
|
||||
mkdir: { args: MkdirArgsSchema, result: MkdirResultSchema },
|
||||
delete: { args: DeleteArgsSchema, result: DeleteResultSchema },
|
||||
move: { args: MoveArgsSchema, result: MoveResultSchema },
|
||||
read: { args: ReadArgsSchema, result: ReadResultSchema },
|
||||
write: { args: WriteArgsSchema, result: WriteResultSchema },
|
||||
getThumbnail: { args: GetThumbnailArgsSchema, result: GetThumbnailResultSchema },
|
||||
};
|
||||
|
||||
// ─── createFileHandler ───────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Build a `FileHandler` for the server side. The returned object is
|
||||
* registered with `ShadeFileRpcChannel.setHooks({ onRequest })` (typically
|
||||
* via `Shade.files.serve(...)` in the SDK).
|
||||
*/
|
||||
export function createFileHandler(
|
||||
shade: Shade,
|
||||
config: FileHandlerConfig,
|
||||
): FileHandler {
|
||||
const idempotency = new IdempotencyCache(config.idempotency);
|
||||
const rateLimiter = new RateLimiter(config.rateLimits);
|
||||
const inflightCancellers = new Map<string, AbortController>();
|
||||
const defaultTimeoutMs = config.defaultTimeoutMs ?? 60_000;
|
||||
const ioTimeoutMs = config.ioTimeoutMs ?? 60_000;
|
||||
const metrics: MetricSink = config.onMetric ?? NOOP_METRIC_SINK;
|
||||
const customRegistrations = config.custom ?? {};
|
||||
const isCustomKind = (kind: string): boolean => kind === 'shade.fs.custom/v1';
|
||||
|
||||
async function handleRequest(
|
||||
from: string,
|
||||
request: RpcRequest,
|
||||
): Promise<RpcResponse | RpcError> {
|
||||
// 0. Replay-window check (independent of sig — defends against
|
||||
// intercept-and-resend even when sig verification is disabled).
|
||||
const skewMs = Math.abs(Date.now() - request.signedAt);
|
||||
if (skewMs > MAX_SIGNATURE_AGE_MS) {
|
||||
metrics(METRIC_SIGNATURE_REJECT_TOTAL, 1, { reason: 'skew' });
|
||||
return makeErrorEnvelope(
|
||||
request,
|
||||
new InvalidSignatureError(
|
||||
`signedAt is ${skewMs}ms outside the ±${MAX_SIGNATURE_AGE_MS}ms replay window`,
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
// 0b. Optional Ed25519 sig verification — pluggable. The canonical
|
||||
// bytes bind sender + kind + id + signedAt + sha256(args).
|
||||
if (config.verifySender !== undefined) {
|
||||
const argsHashBytes = hashArgs(request.args);
|
||||
const canonical = canonicalRpcBytes({
|
||||
address: from,
|
||||
signedAt: request.signedAt,
|
||||
kind: request.kind,
|
||||
id: request.id,
|
||||
argsHash: argsHashBytes,
|
||||
});
|
||||
let ok = false;
|
||||
try {
|
||||
ok = await config.verifySender(from, canonical, request.sig);
|
||||
} catch (err) {
|
||||
metrics(METRIC_SIGNATURE_REJECT_TOTAL, 1, { reason: 'throw' });
|
||||
return makeErrorEnvelope(request, err);
|
||||
}
|
||||
if (!ok) {
|
||||
metrics(METRIC_SIGNATURE_REJECT_TOTAL, 1, { reason: 'mismatch' });
|
||||
return makeErrorEnvelope(request, new InvalidSignatureError());
|
||||
}
|
||||
}
|
||||
|
||||
// 1. Resolve op + handler. Custom ops route through `config.custom`.
|
||||
let op: StandardOp | 'custom';
|
||||
let argSchema: ZodTypeAny;
|
||||
let resultSchema: ZodTypeAny;
|
||||
let customHandler: CustomOpRegistrations[string] | undefined;
|
||||
|
||||
if (isCustomKind(request.kind)) {
|
||||
op = 'custom';
|
||||
argSchema = CustomArgsSchema;
|
||||
resultSchema = CustomResultSchema;
|
||||
} else {
|
||||
const std = opOfKind(request.kind);
|
||||
if (std === null) {
|
||||
return makeErrorEnvelope(request, new NotImplementedError(request.kind));
|
||||
}
|
||||
op = std;
|
||||
argSchema = OP_SCHEMAS[std].args;
|
||||
resultSchema = OP_SCHEMAS[std].result;
|
||||
const handler = config[std];
|
||||
if (handler === undefined) {
|
||||
return makeErrorEnvelope(request, new NotImplementedError(std));
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Args validation (wire shape)
|
||||
const argParse = argSchema.safeParse(request.args);
|
||||
if (!argParse.success) {
|
||||
const issue = argParse.error.issues[0];
|
||||
return makeErrorEnvelope(
|
||||
request,
|
||||
new InvalidArgsError(
|
||||
issue?.message ?? 'invalid arguments',
|
||||
issue?.path.join('.') || undefined,
|
||||
),
|
||||
);
|
||||
}
|
||||
const parsedArgs = argParse.data as unknown;
|
||||
|
||||
// 2b. Custom op resolution
|
||||
let resolvedOpKind: string = op;
|
||||
if (op === 'custom') {
|
||||
const customArgs = parsedArgs as CustomArgs;
|
||||
customHandler = customRegistrations[customArgs.name];
|
||||
if (customHandler === undefined) {
|
||||
return makeErrorEnvelope(
|
||||
request,
|
||||
new NotImplementedError(`custom:${customArgs.name}`),
|
||||
);
|
||||
}
|
||||
resolvedOpKind = `custom:${customArgs.name}`;
|
||||
// Validate inner payload against the custom op's args schema
|
||||
const payloadParse = customHandler.args.safeParse(customArgs.payload);
|
||||
if (!payloadParse.success) {
|
||||
const issue = payloadParse.error.issues[0];
|
||||
return makeErrorEnvelope(
|
||||
request,
|
||||
new InvalidArgsError(
|
||||
issue?.message ?? 'invalid custom-op payload',
|
||||
issue?.path.join('.') || undefined,
|
||||
),
|
||||
);
|
||||
}
|
||||
// Replace payload with validated value (Zod may apply defaults).
|
||||
(parsedArgs as CustomArgs).payload = payloadParse.data;
|
||||
}
|
||||
|
||||
// 3. Path validation (skip ops without a path)
|
||||
let primaryPath = '';
|
||||
if (op === 'move') {
|
||||
primaryPath = (parsedArgs as MoveArgs).src;
|
||||
} else if (op !== 'custom' && 'path' in (parsedArgs as object)) {
|
||||
primaryPath = (parsedArgs as { path: string }).path;
|
||||
}
|
||||
let normalizedPath = primaryPath;
|
||||
if (primaryPath !== '') {
|
||||
const validated = validatePath(primaryPath, config.pathPolicy);
|
||||
if (!validated.ok) {
|
||||
return makeErrorEnvelope(request, new PathValidationError(validated.reason));
|
||||
}
|
||||
normalizedPath = validated.normalized;
|
||||
if (op === 'move') {
|
||||
const dstValid = validatePath((parsedArgs as MoveArgs).dst, config.pathPolicy);
|
||||
if (!dstValid.ok) {
|
||||
return makeErrorEnvelope(
|
||||
request,
|
||||
new PathValidationError(dstValid.reason, 'dst'),
|
||||
);
|
||||
}
|
||||
(parsedArgs as MoveArgs).src = normalizedPath;
|
||||
(parsedArgs as MoveArgs).dst = dstValid.normalized;
|
||||
} else {
|
||||
(parsedArgs as { path: string }).path = normalizedPath;
|
||||
}
|
||||
}
|
||||
|
||||
// 4. Rate-limit acquire
|
||||
const opCostKey = op === 'custom' ? 'custom' : op;
|
||||
const customCost = customHandler?.cost;
|
||||
const estimatedBytes = estimateBytes(op === 'custom' ? 'custom' : op, parsedArgs);
|
||||
try {
|
||||
if (customCost !== undefined) {
|
||||
// Custom op-specific cost: acquire that many op-tokens manually.
|
||||
// Fall back to standard `custom` bucket cost for non-overridden.
|
||||
rateLimiter.acquire(from, opCostKey, estimatedBytes);
|
||||
// For overridden cost > 1, deduct extra tokens from the same bucket.
|
||||
// The simplest route: re-acquire (cost - 1) more.
|
||||
for (let i = 1; i < customCost; i++) {
|
||||
rateLimiter.acquire(from, opCostKey, 0);
|
||||
}
|
||||
} else {
|
||||
rateLimiter.acquire(from, opCostKey, estimatedBytes);
|
||||
}
|
||||
} catch (err) {
|
||||
metrics(METRIC_RATE_LIMIT_REJECT_TOTAL, 1, { op: resolvedOpKind });
|
||||
return makeErrorEnvelope(request, err);
|
||||
}
|
||||
|
||||
// 5. Idempotency (mutations only)
|
||||
const isMutation = MUTATION_OPS.has(opCostKey);
|
||||
let commitIdem: ((response: unknown) => void) | null = null;
|
||||
let abandonIdem: (() => void) | null = null;
|
||||
if (isMutation && request.idempotencyKey !== undefined) {
|
||||
try {
|
||||
const begin = idempotency.begin(from, request.idempotencyKey, parsedArgs);
|
||||
if (begin.status === 'replay') {
|
||||
rateLimiter.release(from, opCostKey, estimatedBytes);
|
||||
metrics(METRIC_IDEMPOTENCY_HIT_TOTAL, 1, { op: resolvedOpKind });
|
||||
return makeResponseEnvelope(request, begin.response);
|
||||
}
|
||||
if (begin.status === 'wait') {
|
||||
const cached = await begin.promise;
|
||||
rateLimiter.release(from, opCostKey, estimatedBytes);
|
||||
metrics(METRIC_IDEMPOTENCY_HIT_TOTAL, 1, { op: resolvedOpKind });
|
||||
return makeResponseEnvelope(request, cached);
|
||||
}
|
||||
commitIdem = begin.commit;
|
||||
abandonIdem = begin.abandon;
|
||||
} catch (err) {
|
||||
rateLimiter.release(from, opCostKey, estimatedBytes);
|
||||
if (
|
||||
err !== null &&
|
||||
typeof err === 'object' &&
|
||||
(err as { code?: string }).code === 'SHADE_FS_IDEMPOTENCY_CONFLICT'
|
||||
) {
|
||||
metrics(METRIC_IDEMPOTENCY_CONFLICT_TOTAL, 1, { op: resolvedOpKind });
|
||||
}
|
||||
return makeErrorEnvelope(request, err);
|
||||
}
|
||||
}
|
||||
|
||||
// 6. Build context + abort controller
|
||||
const controller = new AbortController();
|
||||
inflightCancellers.set(request.id, controller);
|
||||
const ctx = buildOpContext({
|
||||
op: op === 'custom' ? (resolvedOpKind as `custom:${string}`) : op,
|
||||
path: normalizedPath,
|
||||
parsedArgs,
|
||||
sender: from,
|
||||
signal: controller.signal,
|
||||
idempotencyKey: request.idempotencyKey,
|
||||
attemptNumber: request.attempt ?? 1,
|
||||
shade,
|
||||
});
|
||||
|
||||
// 7. Fingerprint gate
|
||||
if (config.requireFingerprintVerifiedFor !== undefined) {
|
||||
let gate: 'required' | 'optional' | 'reject';
|
||||
try {
|
||||
gate = await config.requireFingerprintVerifiedFor(ctx as OpContext<unknown>);
|
||||
} catch (err) {
|
||||
cleanup({ release: true });
|
||||
return makeErrorEnvelope(request, err);
|
||||
}
|
||||
if (gate === 'reject') {
|
||||
cleanup({ release: true });
|
||||
metrics(METRIC_FINGERPRINT_REJECT_TOTAL, 1, { op: resolvedOpKind, gate: 'reject' });
|
||||
return makeErrorEnvelope(
|
||||
request,
|
||||
new FingerprintRequiredError('operation rejected by fingerprint policy'),
|
||||
);
|
||||
}
|
||||
if (gate === 'required') {
|
||||
let verified = false;
|
||||
try {
|
||||
verified = config.isFingerprintVerified !== undefined
|
||||
? Boolean(await config.isFingerprintVerified(from))
|
||||
: false;
|
||||
} catch (err) {
|
||||
cleanup({ release: true });
|
||||
return makeErrorEnvelope(request, err);
|
||||
}
|
||||
if (!verified) {
|
||||
cleanup({ release: true });
|
||||
metrics(METRIC_FINGERPRINT_REJECT_TOTAL, 1, { op: resolvedOpKind, gate: 'required' });
|
||||
return makeErrorEnvelope(request, new FingerprintRequiredError());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 8. beforeOp
|
||||
try {
|
||||
if (config.beforeOp !== undefined) {
|
||||
await config.beforeOp(ctx as OpContext<unknown>);
|
||||
}
|
||||
} catch (err) {
|
||||
cleanup({ release: true });
|
||||
return makeErrorEnvelope(request, err);
|
||||
}
|
||||
|
||||
// 9. Run handler with timeout race — adapting I/O ops as needed.
|
||||
const timeoutMs = Math.min(
|
||||
request.deadlineMs ?? defaultTimeoutMs,
|
||||
defaultTimeoutMs,
|
||||
);
|
||||
|
||||
let wireResult: unknown;
|
||||
const startedAt = Date.now();
|
||||
try {
|
||||
wireResult = await runWithTimeout(
|
||||
() => invokeOpHandler({
|
||||
op,
|
||||
stdHandler: op === 'custom' ? undefined : (config[op] as unknown),
|
||||
customHandler,
|
||||
ctx: ctx as OpContext<unknown>,
|
||||
parsedArgs,
|
||||
sender: from,
|
||||
signal: controller.signal,
|
||||
streamsBridge: config.streamsBridge,
|
||||
ioTimeoutMs,
|
||||
}),
|
||||
timeoutMs,
|
||||
controller,
|
||||
);
|
||||
} catch (err) {
|
||||
const durationMs = Date.now() - startedAt;
|
||||
metrics(METRIC_OP_DURATION_MS, durationMs, { op: resolvedOpKind, result: 'error' });
|
||||
metrics(METRIC_OP_TOTAL, 1, { op: resolvedOpKind, result: 'error' });
|
||||
cleanup({ release: true });
|
||||
if (config.onError !== undefined) {
|
||||
try {
|
||||
config.onError(err, ctx as OpContext<unknown>);
|
||||
} catch (hookErr) {
|
||||
console.error('[FileHandler] onError hook threw:', hookErr);
|
||||
}
|
||||
}
|
||||
return makeErrorEnvelope(request, err);
|
||||
}
|
||||
|
||||
// 10. Custom-op response validation against the registered schema.
|
||||
if (op === 'custom' && customHandler !== undefined) {
|
||||
const innerParse = customHandler.response.safeParse(wireResult);
|
||||
if (!innerParse.success) {
|
||||
cleanup({ release: true });
|
||||
return makeErrorEnvelope(
|
||||
request,
|
||||
new CustomOpRejectedError(
|
||||
`custom-op response shape rejected by registered schema: ${innerParse.error.issues[0]?.message ?? 'unknown'}`,
|
||||
),
|
||||
);
|
||||
}
|
||||
wireResult = { result: innerParse.data };
|
||||
}
|
||||
|
||||
// 11. Defensive response validation against the wire schema.
|
||||
const resultParse = resultSchema.safeParse(wireResult);
|
||||
if (!resultParse.success) {
|
||||
cleanup({ release: true });
|
||||
return makeErrorEnvelope(
|
||||
request,
|
||||
new InvalidArgsError(
|
||||
`handler for ${resolvedOpKind} returned invalid response shape`,
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
// 12. afterOp
|
||||
try {
|
||||
if (config.afterOp !== undefined) {
|
||||
await config.afterOp(ctx as OpContext<unknown>, resultParse.data);
|
||||
}
|
||||
} catch (err) {
|
||||
cleanup({ release: true });
|
||||
return makeErrorEnvelope(request, err);
|
||||
}
|
||||
|
||||
// 13. Commit idempotency + emit metrics
|
||||
commitIdem?.(resultParse.data);
|
||||
cleanup({ release: false });
|
||||
|
||||
const durationMs = Date.now() - startedAt;
|
||||
metrics(METRIC_OP_DURATION_MS, durationMs, { op: resolvedOpKind, result: 'ok' });
|
||||
metrics(METRIC_OP_TOTAL, 1, { op: resolvedOpKind, result: 'ok' });
|
||||
if (estimatedBytes > 0) {
|
||||
// Inbound bytes (write) vs outbound (read) — both reuse the same
|
||||
// pre-call `estimatedBytes`, since post-execution reconciliation
|
||||
// would require deeper plumbing.
|
||||
const direction = op === 'write' ? METRIC_BYTES_IN : op === 'read' ? METRIC_BYTES_OUT : null;
|
||||
if (direction !== null) {
|
||||
metrics(direction, estimatedBytes, { op: resolvedOpKind });
|
||||
}
|
||||
}
|
||||
|
||||
return makeResponseEnvelope(request, resultParse.data);
|
||||
|
||||
function cleanup(opts: { release: boolean }): void {
|
||||
inflightCancellers.delete(request.id);
|
||||
if (opts.release) {
|
||||
abandonIdem?.();
|
||||
rateLimiter.release(from, opCostKey, estimatedBytes);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function handleCancel(_from: string, cancel: RpcCancel): void {
|
||||
const controller = inflightCancellers.get(cancel.id);
|
||||
if (controller !== undefined) {
|
||||
controller.abort(new CancelledError(cancel.reason ?? 'cancelled by sender'));
|
||||
inflightCancellers.delete(cancel.id);
|
||||
}
|
||||
}
|
||||
|
||||
function destroy(): void {
|
||||
for (const c of inflightCancellers.values()) {
|
||||
c.abort(new CancelledError('handler destroyed'));
|
||||
}
|
||||
inflightCancellers.clear();
|
||||
}
|
||||
|
||||
return Object.assign({ handleRequest, handleCancel, destroy }, {
|
||||
[INTERNAL_SYMBOL]: { idempotency, rateLimiter },
|
||||
});
|
||||
}
|
||||
|
||||
export const INTERNAL_SYMBOL = Symbol.for('@shade/files/internal');
|
||||
|
||||
// ─── Op invoker (handles I/O adapters) ───────────────────────
|
||||
|
||||
interface InvokeArgs {
|
||||
op: StandardOp | 'custom';
|
||||
stdHandler: unknown;
|
||||
customHandler: CustomOpRegistrations[string] | undefined;
|
||||
ctx: OpContext<unknown>;
|
||||
parsedArgs: unknown;
|
||||
sender: string;
|
||||
signal: AbortSignal;
|
||||
streamsBridge: ServerStreamsBridge | undefined;
|
||||
ioTimeoutMs: number;
|
||||
}
|
||||
|
||||
async function invokeOpHandler(args: InvokeArgs): Promise<unknown> {
|
||||
const { op, stdHandler, customHandler, ctx, parsedArgs, sender, signal, streamsBridge, ioTimeoutMs } = args;
|
||||
const adapterDeps = { streamsBridge, sender, signal, ioTimeoutMs };
|
||||
|
||||
switch (op) {
|
||||
case 'write': {
|
||||
const wireArgs = parsedArgs as WriteArgs;
|
||||
const { userArgs, awaitTransferDone } = await adaptWriteArgs(wireArgs, adapterDeps);
|
||||
const userCtx = { ...ctx, args: userArgs } as OpContext<UserWriteArgs>;
|
||||
const userResult = await (stdHandler as (c: OpContext<UserWriteArgs>) => Promise<WriteResult>)(userCtx);
|
||||
await awaitTransferDone();
|
||||
return userResult;
|
||||
}
|
||||
|
||||
case 'read': {
|
||||
const readArgs = parsedArgs as ReadArgs;
|
||||
const userResult = await (stdHandler as (c: OpContext<ReadArgs>) => Promise<UserReadResult>)(ctx as OpContext<ReadArgs>);
|
||||
return await adaptReadResult(userResult, readArgs, adapterDeps);
|
||||
}
|
||||
|
||||
case 'getThumbnail': {
|
||||
const userResult = await (stdHandler as (c: OpContext<GetThumbnailArgs>) => Promise<UserThumbnailResult>)(ctx as OpContext<GetThumbnailArgs>);
|
||||
return adaptThumbnailResult(userResult);
|
||||
}
|
||||
|
||||
case 'custom': {
|
||||
if (customHandler === undefined) {
|
||||
throw new NotImplementedError('custom op without registration');
|
||||
}
|
||||
const customArgs = parsedArgs as CustomArgs;
|
||||
const innerCtx = { ...ctx, args: customArgs } as OpContext<{ name: string; payload: unknown }>;
|
||||
// Pass the validated inner payload as the first arg, the OpContext as the second.
|
||||
return await customHandler.handler(
|
||||
customArgs.payload,
|
||||
innerCtx as OpContext<{ name: string; payload: unknown }>,
|
||||
);
|
||||
}
|
||||
|
||||
default:
|
||||
// Pass-through for list/stat/mkdir/delete/move.
|
||||
return await (stdHandler as (c: OpContext<unknown>) => Promise<unknown>)(ctx);
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Helpers ─────────────────────────────────────────────────
|
||||
|
||||
function makeResponseEnvelope(req: RpcRequest, result: unknown): RpcResponse {
|
||||
return {
|
||||
kind: responseKindOf(req.kind),
|
||||
id: req.id,
|
||||
result,
|
||||
};
|
||||
}
|
||||
|
||||
function makeErrorEnvelope(req: RpcRequest, err: unknown): RpcError {
|
||||
return {
|
||||
kind: 'shade.fs.error/v1',
|
||||
id: req.id,
|
||||
error: payloadFromError(err),
|
||||
};
|
||||
}
|
||||
|
||||
function estimateBytes(op: StandardOp | 'custom', args: unknown): number {
|
||||
if (op === 'write') {
|
||||
const w = args as { kind: 'inline' | 'streams'; bytesB64?: string; size?: number };
|
||||
if (w.kind === 'inline' && typeof w.bytesB64 === 'string') {
|
||||
return Math.floor((w.bytesB64.length * 3) / 4);
|
||||
}
|
||||
return w.size ?? 0;
|
||||
}
|
||||
if (op === 'read') {
|
||||
const r = args as { range?: { start: number; end: number } };
|
||||
if (r.range !== undefined) return r.range.end - r.range.start;
|
||||
return 0;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
async function runWithTimeout<T>(
|
||||
fn: () => Promise<T>,
|
||||
timeoutMs: number,
|
||||
controller: AbortController,
|
||||
): Promise<T> {
|
||||
let timer: ReturnType<typeof setTimeout> | null = null;
|
||||
const timeout = new Promise<never>((_, reject) => {
|
||||
timer = setTimeout(() => {
|
||||
controller.abort(new Error('timeout'));
|
||||
reject(new (FileError as unknown as { new (p: { code: string; message: string }): FileError })({
|
||||
code: 'OPERATION_TIMEOUT',
|
||||
message: `operation timed out after ${timeoutMs}ms`,
|
||||
}));
|
||||
}, timeoutMs);
|
||||
});
|
||||
try {
|
||||
return await Promise.race([fn(), timeout]);
|
||||
} finally {
|
||||
if (timer !== null) clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
160
packages/shade-files/src/server/idempotency-cache.ts
Normal file
160
packages/shade-files/src/server/idempotency-cache.ts
Normal file
@@ -0,0 +1,160 @@
|
||||
import { hashArgs, bytesToHex } from '../protocol/canonical.js';
|
||||
import { IdempotencyConflictError } from '../schemas/errors.js';
|
||||
|
||||
interface Entry {
|
||||
argsHash: string;
|
||||
/** `undefined` while inflight; resolved value once handler completes. */
|
||||
status: 'inflight' | 'done';
|
||||
promise?: Promise<unknown>;
|
||||
response?: unknown;
|
||||
insertedAt: number;
|
||||
lastAccessAt: number;
|
||||
}
|
||||
|
||||
export interface IdempotencyCacheOptions {
|
||||
/** Time in ms after which an entry is eligible for eviction. Default 1h. */
|
||||
ttlMs?: number;
|
||||
/** Per-sender hard cap on cached entries; oldest-by-insert evicted first. */
|
||||
maxEntriesPerSender?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-sender LRU + TTL cache for idempotent RPC calls.
|
||||
*
|
||||
* - On lookup, returns cached response if `argsHash` matches the original
|
||||
* call. Throws `IdempotencyConflictError` on argsHash mismatch.
|
||||
* - On concurrent retry of the same key (still inflight), the second
|
||||
* caller awaits the first's promise — no double-execution.
|
||||
* - LRU eviction caps memory at `maxEntriesPerSender` per sender.
|
||||
*
|
||||
* Keyed on `(senderAddress, idempotencyKey)`. Internally a 2-level map.
|
||||
*/
|
||||
export class IdempotencyCache {
|
||||
private readonly ttlMs: number;
|
||||
private readonly maxPerSender: number;
|
||||
private readonly bySender = new Map<string, Map<string, Entry>>();
|
||||
|
||||
constructor(opts: IdempotencyCacheOptions = {}) {
|
||||
this.ttlMs = opts.ttlMs ?? 60 * 60 * 1000;
|
||||
this.maxPerSender = opts.maxEntriesPerSender ?? 10_000;
|
||||
}
|
||||
|
||||
/**
|
||||
* Begin (or rejoin) an idempotent operation. Returns:
|
||||
* - `{ status: 'fresh', commit }` — handler should run, then call `commit(response)`.
|
||||
* - `{ status: 'wait', promise }` — concurrent retry; await the in-flight promise.
|
||||
* - `{ status: 'replay', response }` — completed earlier; return cached.
|
||||
*
|
||||
* Mismatched argsHash throws `IdempotencyConflictError` immediately.
|
||||
*/
|
||||
begin(
|
||||
sender: string,
|
||||
key: string,
|
||||
args: unknown,
|
||||
):
|
||||
| { status: 'fresh'; commit: (response: unknown) => void; abandon: () => void }
|
||||
| { status: 'wait'; promise: Promise<unknown> }
|
||||
| { status: 'replay'; response: unknown } {
|
||||
const argsHash = bytesToHex(hashArgs(args));
|
||||
const senderMap = this.getOrCreate(sender);
|
||||
this.evictExpired(senderMap);
|
||||
const existing = senderMap.get(key);
|
||||
if (existing !== undefined) {
|
||||
if (existing.argsHash !== argsHash) {
|
||||
throw new IdempotencyConflictError();
|
||||
}
|
||||
existing.lastAccessAt = Date.now();
|
||||
if (existing.status === 'done') {
|
||||
return { status: 'replay', response: existing.response };
|
||||
}
|
||||
// Inflight — return the same promise so concurrent retries de-dupe.
|
||||
return { status: 'wait', promise: existing.promise! };
|
||||
}
|
||||
|
||||
let resolveInflight: (value: unknown) => void = () => {};
|
||||
let rejectInflight: (err: unknown) => void = () => {};
|
||||
const inflightPromise = new Promise<unknown>((resolve, reject) => {
|
||||
resolveInflight = resolve;
|
||||
rejectInflight = reject;
|
||||
});
|
||||
// Suppress unhandled-rejection warnings when no concurrent retry is
|
||||
// awaiting this promise. Real waiters attach their own catch via
|
||||
// `await begin().promise` and still see the rejection.
|
||||
inflightPromise.catch(() => {
|
||||
/* swallow — see comment above */
|
||||
});
|
||||
const entry: Entry = {
|
||||
argsHash,
|
||||
status: 'inflight',
|
||||
promise: inflightPromise,
|
||||
insertedAt: Date.now(),
|
||||
lastAccessAt: Date.now(),
|
||||
};
|
||||
senderMap.set(key, entry);
|
||||
this.evictOverflow(senderMap);
|
||||
|
||||
const commit = (response: unknown): void => {
|
||||
entry.status = 'done';
|
||||
entry.response = response;
|
||||
entry.lastAccessAt = Date.now();
|
||||
resolveInflight(response);
|
||||
};
|
||||
const abandon = (): void => {
|
||||
// Failed/cancelled call — remove from cache so retries can proceed.
|
||||
senderMap.delete(key);
|
||||
rejectInflight(new Error('idempotency abandoned'));
|
||||
};
|
||||
|
||||
return { status: 'fresh', commit, abandon };
|
||||
}
|
||||
|
||||
/** Manual prune; called by the periodic retention job. */
|
||||
prune(now: number = Date.now()): number {
|
||||
let removed = 0;
|
||||
for (const [sender, senderMap] of this.bySender) {
|
||||
for (const [key, entry] of senderMap) {
|
||||
if (now - entry.insertedAt > this.ttlMs) {
|
||||
senderMap.delete(key);
|
||||
removed++;
|
||||
}
|
||||
}
|
||||
if (senderMap.size === 0) this.bySender.delete(sender);
|
||||
}
|
||||
return removed;
|
||||
}
|
||||
|
||||
/** Total cached entries across all senders (mainly for tests/metrics). */
|
||||
size(): number {
|
||||
let total = 0;
|
||||
for (const m of this.bySender.values()) total += m.size;
|
||||
return total;
|
||||
}
|
||||
|
||||
private getOrCreate(sender: string): Map<string, Entry> {
|
||||
let m = this.bySender.get(sender);
|
||||
if (m === undefined) {
|
||||
m = new Map();
|
||||
this.bySender.set(sender, m);
|
||||
}
|
||||
return m;
|
||||
}
|
||||
|
||||
private evictExpired(senderMap: Map<string, Entry>): void {
|
||||
const now = Date.now();
|
||||
for (const [key, entry] of senderMap) {
|
||||
if (now - entry.insertedAt > this.ttlMs) senderMap.delete(key);
|
||||
}
|
||||
}
|
||||
|
||||
private evictOverflow(senderMap: Map<string, Entry>): void {
|
||||
if (senderMap.size <= this.maxPerSender) return;
|
||||
// Map iteration order = insertion order; first key is oldest.
|
||||
const overflow = senderMap.size - this.maxPerSender;
|
||||
let i = 0;
|
||||
for (const key of senderMap.keys()) {
|
||||
if (i >= overflow) break;
|
||||
senderMap.delete(key);
|
||||
i++;
|
||||
}
|
||||
}
|
||||
}
|
||||
229
packages/shade-files/src/server/io-adapters.ts
Normal file
229
packages/shade-files/src/server/io-adapters.ts
Normal file
@@ -0,0 +1,229 @@
|
||||
/**
|
||||
* Adapt between wire-shaped I/O ops and user-handler-shaped I/O ops.
|
||||
*
|
||||
* client → server (write): WriteArgs (wire) → UserWriteArgs (handler)
|
||||
* server → client (read): UserReadResult (handler) → ReadResult (wire)
|
||||
* server → client (thumb): UserThumbnailResult → GetThumbnailResult
|
||||
*
|
||||
* The streams-bridge is consulted whenever we're crossing the > 256 KiB
|
||||
* boundary: outbound reads kick a new transfer; inbound writes await one.
|
||||
*/
|
||||
|
||||
import { sha256 } from '@noble/hashes/sha2.js';
|
||||
import type {
|
||||
GetThumbnailResult,
|
||||
ReadArgs,
|
||||
ReadResult,
|
||||
WriteArgs,
|
||||
} from '../schemas/ops.js';
|
||||
import { base64ToBytes, bytesToBase64, bytesToHex } from '../protocol/canonical.js';
|
||||
import { INLINE_THRESHOLD } from '../client/inline-threshold.js';
|
||||
import {
|
||||
ConflictError,
|
||||
InternalFileError,
|
||||
InvalidArgsError,
|
||||
} from '../schemas/errors.js';
|
||||
import type {
|
||||
UserReadResult,
|
||||
UserThumbnailResult,
|
||||
UserWriteArgs,
|
||||
} from './io-types.js';
|
||||
import type { ServerStreamsBridge } from './streams-bridge.js';
|
||||
|
||||
export interface IoAdapterDeps {
|
||||
streamsBridge: ServerStreamsBridge | undefined;
|
||||
sender: string;
|
||||
signal: AbortSignal;
|
||||
/** Hard deadline for streams-bridge awaits / outbound transfers. */
|
||||
ioTimeoutMs: number;
|
||||
}
|
||||
|
||||
// ─── write: wire args → user args ────────────────────────────
|
||||
|
||||
/**
|
||||
* Convert validated `WriteArgs` (wire shape) into `UserWriteArgs`. For
|
||||
* streams writes, this awaits the matching inbound transfer via the
|
||||
* streams-bridge and synthesizes a `ReadableStream` for the user handler.
|
||||
*/
|
||||
export async function adaptWriteArgs(
|
||||
wireArgs: WriteArgs,
|
||||
deps: IoAdapterDeps,
|
||||
): Promise<{
|
||||
userArgs: UserWriteArgs;
|
||||
/** Resolves once the inbound transfer (if any) has completed. */
|
||||
awaitTransferDone: () => Promise<void>;
|
||||
}> {
|
||||
if (wireArgs.kind === 'inline') {
|
||||
const bytes = base64ToBytes(wireArgs.bytesB64);
|
||||
if (bytes.byteLength > INLINE_THRESHOLD) {
|
||||
throw new InvalidArgsError(
|
||||
`inline write exceeds ${INLINE_THRESHOLD}-byte threshold (got ${bytes.byteLength})`,
|
||||
'bytesB64',
|
||||
);
|
||||
}
|
||||
const hashHex = bytesToHex(sha256(bytes));
|
||||
const userArgs: UserWriteArgs = {
|
||||
path: wireArgs.path,
|
||||
overwrite: wireArgs.overwrite,
|
||||
...(wireArgs.contentType !== undefined ? { contentType: wireArgs.contentType } : {}),
|
||||
content: {
|
||||
kind: 'inline',
|
||||
bytes,
|
||||
size: bytes.byteLength,
|
||||
sha256: hashHex,
|
||||
},
|
||||
};
|
||||
return { userArgs, awaitTransferDone: async () => undefined };
|
||||
}
|
||||
|
||||
// Streams write — must have a streams-bridge configured.
|
||||
if (deps.streamsBridge === undefined) {
|
||||
throw new InternalFileError(
|
||||
'streams-bridge not configured: cannot accept streamed write',
|
||||
);
|
||||
}
|
||||
const parked = await deps.streamsBridge.awaitWrite(wireArgs.writeId, {
|
||||
expectedFrom: deps.sender,
|
||||
signal: deps.signal,
|
||||
timeoutMs: deps.ioTimeoutMs,
|
||||
});
|
||||
|
||||
// sha256 from the inbound transfer's done() — user handler can `await`
|
||||
// it immediately after draining the readable.
|
||||
const sha256Promise = parked.done.then((r) => r.sha256);
|
||||
sha256Promise.catch(() => {
|
||||
/* swallow until consumer awaits */
|
||||
});
|
||||
|
||||
// Cancellation: if the user handler aborts (or the dispatcher times out),
|
||||
// also tear down the inbound transfer.
|
||||
const onAbort = (): void => {
|
||||
void parked.handle.abort('user-cancel').catch(() => undefined);
|
||||
};
|
||||
if (deps.signal.aborted) onAbort();
|
||||
else deps.signal.addEventListener('abort', onAbort, { once: true });
|
||||
|
||||
// The dispatcher calls this AFTER the user handler resolves to ensure
|
||||
// the transfer is fully drained (so any wire-level integrity errors
|
||||
// surface here before we send the RPC ack to the client).
|
||||
const awaitTransferDone = async (): Promise<void> => {
|
||||
try {
|
||||
await parked.done;
|
||||
} finally {
|
||||
deps.signal.removeEventListener('abort', onAbort);
|
||||
}
|
||||
};
|
||||
|
||||
const userArgs: UserWriteArgs = {
|
||||
path: wireArgs.path,
|
||||
overwrite: wireArgs.overwrite,
|
||||
...(wireArgs.contentType !== undefined ? { contentType: wireArgs.contentType } : {}),
|
||||
content: {
|
||||
kind: 'streams',
|
||||
stream: parked.readable,
|
||||
size: wireArgs.size,
|
||||
sha256: sha256Promise,
|
||||
},
|
||||
};
|
||||
return { userArgs, awaitTransferDone };
|
||||
}
|
||||
|
||||
// ─── read: user result → wire result ─────────────────────────
|
||||
|
||||
/**
|
||||
* Convert a user-supplied `UserReadResult` into a wire `ReadResult`. For
|
||||
* streams results, this kicks an outbound `@shade/transfer` and embeds the
|
||||
* resulting `streamId` in the response so the client-side bridge can
|
||||
* subscribe.
|
||||
*/
|
||||
export async function adaptReadResult(
|
||||
userResult: UserReadResult,
|
||||
args: ReadArgs,
|
||||
deps: IoAdapterDeps,
|
||||
): Promise<ReadResult> {
|
||||
if (userResult.kind === 'inline') {
|
||||
if (userResult.bytes.byteLength > INLINE_THRESHOLD) {
|
||||
throw new InternalFileError(
|
||||
`inline read result exceeds ${INLINE_THRESHOLD} bytes (got ${userResult.bytes.byteLength})`,
|
||||
);
|
||||
}
|
||||
const sha256Hex = userResult.sha256 ?? bytesToHex(sha256(userResult.bytes));
|
||||
return {
|
||||
kind: 'inline',
|
||||
bytesB64: bytesToBase64(userResult.bytes),
|
||||
size: userResult.bytes.byteLength,
|
||||
sha256: sha256Hex,
|
||||
...(userResult.contentType !== undefined ? { contentType: userResult.contentType } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
if (deps.streamsBridge === undefined) {
|
||||
throw new InternalFileError(
|
||||
'streams-bridge not configured: cannot ship streamed read result',
|
||||
);
|
||||
}
|
||||
const { readStreamId } = await deps.streamsBridge.initiateRead({
|
||||
peer: deps.sender,
|
||||
stream: userResult.stream,
|
||||
size: userResult.size,
|
||||
...(userResult.contentType !== undefined ? { contentType: userResult.contentType } : {}),
|
||||
name: args.path,
|
||||
signal: deps.signal,
|
||||
});
|
||||
return {
|
||||
kind: 'streams',
|
||||
streamId: readStreamId,
|
||||
size: userResult.size,
|
||||
sha256: userResult.sha256,
|
||||
...(userResult.contentType !== undefined ? { contentType: userResult.contentType } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
// ─── getThumbnail: user result → wire result ─────────────────
|
||||
|
||||
const FORMAT_MAGIC: Record<string, Uint8Array> = {
|
||||
png: new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
|
||||
jpeg: new Uint8Array([0xff, 0xd8, 0xff]),
|
||||
webp: new Uint8Array([0x52, 0x49, 0x46, 0x46]), // 'RIFF', then size, then 'WEBP'
|
||||
};
|
||||
const WEBP_HEAD = new Uint8Array([0x57, 0x45, 0x42, 0x50]); // 'WEBP' at offset 8
|
||||
|
||||
/**
|
||||
* Validate a user thumbnail result for format magic-bytes match. The user
|
||||
* MAY produce any image format internally, but this guards against
|
||||
* misclassification (e.g. claiming PNG with JPEG bytes), which can confuse
|
||||
* downstream rendering and security scanners.
|
||||
*/
|
||||
function checkFormatMagic(bytes: Uint8Array, format: 'png' | 'webp' | 'jpeg'): boolean {
|
||||
const magic = FORMAT_MAGIC[format];
|
||||
if (magic === undefined) return false;
|
||||
if (bytes.byteLength < magic.byteLength) return false;
|
||||
for (let i = 0; i < magic.byteLength; i++) {
|
||||
if (bytes[i] !== magic[i]) return false;
|
||||
}
|
||||
if (format === 'webp') {
|
||||
if (bytes.byteLength < 12) return false;
|
||||
for (let i = 0; i < 4; i++) {
|
||||
if (bytes[8 + i] !== WEBP_HEAD[i]) return false;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
export function adaptThumbnailResult(
|
||||
userResult: UserThumbnailResult,
|
||||
): GetThumbnailResult {
|
||||
if (!checkFormatMagic(userResult.bytes, userResult.format)) {
|
||||
throw new ConflictError(
|
||||
`thumbnail bytes do not match declared format=${userResult.format}`,
|
||||
);
|
||||
}
|
||||
const hash = userResult.sha256 ?? bytesToHex(sha256(userResult.bytes));
|
||||
return {
|
||||
bytesB64: bytesToBase64(userResult.bytes),
|
||||
format: userResult.format,
|
||||
width: userResult.width,
|
||||
height: userResult.height,
|
||||
sha256: hash,
|
||||
};
|
||||
}
|
||||
102
packages/shade-files/src/server/io-types.ts
Normal file
102
packages/shade-files/src/server/io-types.ts
Normal file
@@ -0,0 +1,102 @@
|
||||
/**
|
||||
* Server-facing types for the I/O ops (`read`, `write`, `getThumbnail`).
|
||||
*
|
||||
* These differ from the wire schemas in `schemas/ops.ts` so user-supplied
|
||||
* handlers see clean, typed values (`Uint8Array`, `ReadableStream`) instead
|
||||
* of base64 strings and opaque transfer ids. The dispatcher in
|
||||
* `server/handler.ts` adapts between the two.
|
||||
*/
|
||||
|
||||
import type { ThumbnailSize, WriteResult } from '../schemas/ops.js';
|
||||
|
||||
// ─── read handler types ──────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Value returned by a user-supplied `read` handler.
|
||||
*
|
||||
* - `inline`: the bytes are returned in-band; the dispatcher base64-encodes
|
||||
* them and computes the sha256 if the user did not provide one. Inline
|
||||
* reads MUST be ≤ 256 KiB plaintext.
|
||||
* - `streams`: the user provides a `ReadableStream`, the declared `size`,
|
||||
* and a precomputed `sha256` (e.g. cached on the file's metadata row).
|
||||
* The dispatcher initiates an outbound `@shade/transfer` transfer and
|
||||
* echoes the transfer's `streamId` back to the client over RPC so the
|
||||
* client-side bridge can subscribe.
|
||||
*/
|
||||
export type UserReadResult =
|
||||
| UserReadResultInline
|
||||
| UserReadResultStreams;
|
||||
|
||||
export interface UserReadResultInline {
|
||||
kind: 'inline';
|
||||
bytes: Uint8Array;
|
||||
/** Optional — dispatcher computes if absent. */
|
||||
sha256?: string;
|
||||
contentType?: string;
|
||||
}
|
||||
|
||||
export interface UserReadResultStreams {
|
||||
kind: 'streams';
|
||||
stream: ReadableStream<Uint8Array>;
|
||||
size: number;
|
||||
/** REQUIRED — the dispatcher cannot synchronously hash a stream. */
|
||||
sha256: string;
|
||||
contentType?: string;
|
||||
}
|
||||
|
||||
// ─── write handler types ─────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Value passed to a user-supplied `write` handler.
|
||||
*
|
||||
* The dispatcher decodes the wire `WriteArgs` and (for `streams`) parks
|
||||
* the inbound transfer via the streams-bridge, then synthesizes this
|
||||
* cleaner shape for the user.
|
||||
*/
|
||||
export interface UserWriteArgs {
|
||||
path: string;
|
||||
contentType?: string;
|
||||
overwrite: boolean;
|
||||
content: UserWriteContent;
|
||||
}
|
||||
|
||||
export type UserWriteContent =
|
||||
| UserWriteContentInline
|
||||
| UserWriteContentStreams;
|
||||
|
||||
export interface UserWriteContentInline {
|
||||
kind: 'inline';
|
||||
bytes: Uint8Array;
|
||||
size: number;
|
||||
/** sha256 of `bytes`, computed by the dispatcher. */
|
||||
sha256: string;
|
||||
}
|
||||
|
||||
export interface UserWriteContentStreams {
|
||||
kind: 'streams';
|
||||
/** Plaintext stream. Closes when the inbound `@shade/transfer` completes. */
|
||||
stream: ReadableStream<Uint8Array>;
|
||||
/** Declared plaintext size from the client's RPC args. */
|
||||
size: number;
|
||||
/**
|
||||
* Promise that resolves with the transfer-verified sha256 once the entire
|
||||
* stream has been received. Use this to attach a content hash to the
|
||||
* user's storage record.
|
||||
*/
|
||||
sha256: Promise<string>;
|
||||
}
|
||||
|
||||
// ─── getThumbnail handler types ──────────────────────────────
|
||||
|
||||
/** Value returned by a user-supplied `getThumbnail` handler. */
|
||||
export interface UserThumbnailResult {
|
||||
bytes: Uint8Array;
|
||||
format: 'png' | 'webp' | 'jpeg';
|
||||
width: number;
|
||||
height: number;
|
||||
/** Optional — dispatcher computes if absent. */
|
||||
sha256?: string;
|
||||
}
|
||||
|
||||
// ─── Re-exports ──────────────────────────────────────────────
|
||||
export type { ThumbnailSize, WriteResult };
|
||||
25
packages/shade-files/src/server/metrics.ts
Normal file
25
packages/shade-files/src/server/metrics.ts
Normal file
@@ -0,0 +1,25 @@
|
||||
/**
|
||||
* Standard metric names emitted by the dispatcher.
|
||||
*
|
||||
* The user-supplied `onMetric(name, value, tags)` callback is vendor-neutral —
|
||||
* pipe to Prometheus, OpenTelemetry, statsd, or just append to a log file.
|
||||
*/
|
||||
|
||||
export const METRIC_OP_DURATION_MS = 'shade_files_op_duration_ms';
|
||||
export const METRIC_OP_TOTAL = 'shade_files_op_total';
|
||||
export const METRIC_BYTES_IN = 'shade_files_bytes_in';
|
||||
export const METRIC_BYTES_OUT = 'shade_files_bytes_out';
|
||||
export const METRIC_IDEMPOTENCY_HIT_TOTAL = 'shade_files_idempotency_hit_total';
|
||||
export const METRIC_IDEMPOTENCY_CONFLICT_TOTAL = 'shade_files_idempotency_conflict_total';
|
||||
export const METRIC_RATE_LIMIT_REJECT_TOTAL = 'shade_files_rate_limit_reject_total';
|
||||
export const METRIC_FINGERPRINT_REJECT_TOTAL = 'shade_files_fingerprint_reject_total';
|
||||
export const METRIC_SIGNATURE_REJECT_TOTAL = 'shade_files_signature_reject_total';
|
||||
|
||||
/** Tags attached to every metric event. */
|
||||
export type MetricTags = Record<string, string | number | boolean>;
|
||||
|
||||
/** User-supplied metric sink. Synchronous to keep hot-path fast. */
|
||||
export type MetricSink = (name: string, value: number, tags: MetricTags) => void;
|
||||
|
||||
/** No-op sink used when the user doesn't supply one. */
|
||||
export const NOOP_METRIC_SINK: MetricSink = () => undefined;
|
||||
95
packages/shade-files/src/server/path-policy.ts
Normal file
95
packages/shade-files/src/server/path-policy.ts
Normal file
@@ -0,0 +1,95 @@
|
||||
import {
|
||||
decodePercentEscapes,
|
||||
isPathInside,
|
||||
posixNormalize,
|
||||
} from '../utils/path.js';
|
||||
|
||||
export interface PathPolicy {
|
||||
/** Chroot-style absolute root. All paths must lie inside this directory. */
|
||||
rootScope?: string;
|
||||
/** Reject `..` segments after normalization. Default true. */
|
||||
forbidTraversal?: boolean;
|
||||
/** Hard cap on raw-input length (post-decode). Default 4096. */
|
||||
maxLength?: number;
|
||||
/** Allow symlink components. Default false (consumer enforces realpath if true). */
|
||||
allowSymlinks?: boolean;
|
||||
/** App-specific extra check. Returns 'allow' | 'reject'. */
|
||||
extra?: (normalizedPath: string) => 'allow' | 'reject';
|
||||
}
|
||||
|
||||
export type PathValidationResult =
|
||||
| { ok: true; normalized: string }
|
||||
| { ok: false; reason: string };
|
||||
|
||||
const DEFAULT_MAX_LENGTH = 4096;
|
||||
/** Bytes that must never appear in any path. */
|
||||
const FORBIDDEN_RE = /[\x00-\x08\x0a-\x1f\x7f\\]/;
|
||||
|
||||
/**
|
||||
* Validate and normalize a path against the configured policy.
|
||||
*
|
||||
* Performs (in order):
|
||||
* 1. Length check on raw input.
|
||||
* 2. Forbidden-bytes check on raw input (NUL/CR/LF/DEL/backslash).
|
||||
* 3. Percent-decode (defense against `%2e%2e` smuggling).
|
||||
* 4. Forbidden-bytes check on decoded form.
|
||||
* 5. POSIX normalization.
|
||||
* 6. `..` traversal rejection.
|
||||
* 7. Root-scope boundary check.
|
||||
* 8. Consumer-supplied `extra` predicate.
|
||||
*
|
||||
* Returns the normalized path on success. The handler MUST use the
|
||||
* normalized form, not the raw input — the raw could trip TOCTOU on
|
||||
* the caller's filesystem.
|
||||
*/
|
||||
export function validatePath(
|
||||
rawPath: string,
|
||||
policy: PathPolicy = {},
|
||||
): PathValidationResult {
|
||||
const maxLength = policy.maxLength ?? DEFAULT_MAX_LENGTH;
|
||||
const forbidTraversal = policy.forbidTraversal ?? true;
|
||||
|
||||
if (typeof rawPath !== 'string') {
|
||||
return { ok: false, reason: 'path must be a string' };
|
||||
}
|
||||
if (rawPath.length === 0) {
|
||||
return { ok: false, reason: 'path is empty' };
|
||||
}
|
||||
if (rawPath.length > maxLength) {
|
||||
return { ok: false, reason: `path exceeds ${maxLength} characters` };
|
||||
}
|
||||
if (FORBIDDEN_RE.test(rawPath)) {
|
||||
return { ok: false, reason: 'path contains forbidden control or backslash characters' };
|
||||
}
|
||||
|
||||
const decoded = decodePercentEscapes(rawPath);
|
||||
if (FORBIDDEN_RE.test(decoded)) {
|
||||
return { ok: false, reason: 'percent-decoded path contains forbidden characters' };
|
||||
}
|
||||
if (!decoded.startsWith('/')) {
|
||||
return { ok: false, reason: 'path must be absolute' };
|
||||
}
|
||||
|
||||
if (forbidTraversal) {
|
||||
// Detect `..` segments BEFORE normalization (otherwise they'd be silently
|
||||
// collapsed). Handles both raw and decoded forms.
|
||||
if (/(^|\/)\.\.(\/|$)/.test(rawPath) || /(^|\/)\.\.(\/|$)/.test(decoded)) {
|
||||
return { ok: false, reason: 'path contains `..` traversal segment' };
|
||||
}
|
||||
}
|
||||
|
||||
const normalized = posixNormalize(decoded);
|
||||
|
||||
if (policy.rootScope !== undefined) {
|
||||
const root = posixNormalize(policy.rootScope);
|
||||
if (!isPathInside(normalized, root)) {
|
||||
return { ok: false, reason: 'path is outside root scope' };
|
||||
}
|
||||
}
|
||||
|
||||
if (policy.extra !== undefined && policy.extra(normalized) === 'reject') {
|
||||
return { ok: false, reason: 'path rejected by policy.extra' };
|
||||
}
|
||||
|
||||
return { ok: true, normalized };
|
||||
}
|
||||
157
packages/shade-files/src/server/rate-limiter.ts
Normal file
157
packages/shade-files/src/server/rate-limiter.ts
Normal file
@@ -0,0 +1,157 @@
|
||||
import { FsRateLimitError, QuotaExceededError } from '../schemas/errors.js';
|
||||
|
||||
export interface RateLimitConfig {
|
||||
/** Cap on op-tokens per minute per sender. Default 600. */
|
||||
maxOpsPerMinutePerSender?: number;
|
||||
/** Cap on byte-tokens per hour per sender. Default 10 GiB. */
|
||||
maxBytesPerHourPerSender?: number;
|
||||
/** Per-op token cost. Default 1 per op except write=5, delete=3. */
|
||||
opCost?: Partial<Record<string, number>> & { default?: number };
|
||||
}
|
||||
|
||||
interface Bucket {
|
||||
tokens: number;
|
||||
capacity: number;
|
||||
/** Tokens replenished per millisecond. */
|
||||
refillRate: number;
|
||||
lastRefill: number;
|
||||
}
|
||||
|
||||
interface SenderBuckets {
|
||||
ops: Bucket;
|
||||
bytes: Bucket;
|
||||
}
|
||||
|
||||
/**
|
||||
* Two-bucket per-sender rate limiter. Each operation consumes:
|
||||
* - `opCost(op)` from the ops bucket.
|
||||
* - `estimatedBytes` from the bytes bucket (post-call reconciliation possible).
|
||||
*
|
||||
* Rejection throws either `FsRateLimitError` (op-bucket) or `QuotaExceededError`
|
||||
* (bytes-bucket), each with a `retryAfterMs` hint.
|
||||
*
|
||||
* Cancellation MUST call `release()` to return reserved tokens.
|
||||
*/
|
||||
export class RateLimiter {
|
||||
private readonly opsCapacity: number;
|
||||
private readonly opsRefillPerMs: number;
|
||||
private readonly bytesCapacity: number;
|
||||
private readonly bytesRefillPerMs: number;
|
||||
private readonly opCost: Required<NonNullable<RateLimitConfig['opCost']>>;
|
||||
private readonly bySender = new Map<string, SenderBuckets>();
|
||||
|
||||
constructor(config: RateLimitConfig = {}) {
|
||||
this.opsCapacity = config.maxOpsPerMinutePerSender ?? 600;
|
||||
this.opsRefillPerMs = this.opsCapacity / 60_000;
|
||||
this.bytesCapacity = config.maxBytesPerHourPerSender ?? 10 * 1024 * 1024 * 1024;
|
||||
this.bytesRefillPerMs = this.bytesCapacity / 3_600_000;
|
||||
this.opCost = {
|
||||
list: config.opCost?.list ?? 1,
|
||||
stat: config.opCost?.stat ?? 1,
|
||||
mkdir: config.opCost?.mkdir ?? 2,
|
||||
delete: config.opCost?.delete ?? 3,
|
||||
move: config.opCost?.move ?? 2,
|
||||
read: config.opCost?.read ?? 1,
|
||||
write: config.opCost?.write ?? 5,
|
||||
getThumbnail: config.opCost?.getThumbnail ?? 2,
|
||||
custom: config.opCost?.custom ?? 1,
|
||||
default: config.opCost?.default ?? 1,
|
||||
} as Required<NonNullable<RateLimitConfig['opCost']>>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Acquire tokens for an operation. Throws if either bucket is empty;
|
||||
* otherwise atomically deducts from both. `estimatedBytes` may be 0 for
|
||||
* non-I/O ops.
|
||||
*/
|
||||
acquire(sender: string, op: string, estimatedBytes: number = 0): void {
|
||||
const buckets = this.getBuckets(sender);
|
||||
this.refill(buckets, Date.now());
|
||||
const cost = this.costFor(op);
|
||||
|
||||
if (buckets.ops.tokens < cost) {
|
||||
const need = cost - buckets.ops.tokens;
|
||||
const retryMs = Math.ceil(need / this.opsRefillPerMs);
|
||||
throw new FsRateLimitError(`op rate limit exceeded`, retryMs);
|
||||
}
|
||||
if (estimatedBytes > 0 && buckets.bytes.tokens < estimatedBytes) {
|
||||
const need = estimatedBytes - buckets.bytes.tokens;
|
||||
const retryMs = Math.ceil(need / this.bytesRefillPerMs);
|
||||
throw new QuotaExceededError(`byte quota exceeded`, retryMs);
|
||||
}
|
||||
buckets.ops.tokens -= cost;
|
||||
if (estimatedBytes > 0) buckets.bytes.tokens -= estimatedBytes;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reconcile after the actual byte cost is known. `actualBytes` may be
|
||||
* higher or lower than the estimate; caps to capacity on release.
|
||||
*/
|
||||
reconcile(sender: string, estimatedBytes: number, actualBytes: number): void {
|
||||
if (estimatedBytes === actualBytes) return;
|
||||
const buckets = this.getBuckets(sender);
|
||||
const delta = estimatedBytes - actualBytes;
|
||||
if (delta > 0) {
|
||||
buckets.bytes.tokens = Math.min(buckets.bytes.capacity, buckets.bytes.tokens + delta);
|
||||
} else {
|
||||
buckets.bytes.tokens = Math.max(0, buckets.bytes.tokens + delta);
|
||||
}
|
||||
}
|
||||
|
||||
/** Return tokens to both buckets (cancellation cleanup). */
|
||||
release(sender: string, op: string, bytesReserved: number = 0): void {
|
||||
const buckets = this.bySender.get(sender);
|
||||
if (buckets === undefined) return;
|
||||
const cost = this.costFor(op);
|
||||
buckets.ops.tokens = Math.min(buckets.ops.capacity, buckets.ops.tokens + cost);
|
||||
if (bytesReserved > 0) {
|
||||
buckets.bytes.tokens = Math.min(buckets.bytes.capacity, buckets.bytes.tokens + bytesReserved);
|
||||
}
|
||||
}
|
||||
|
||||
/** Available token snapshot — for tests/metrics. */
|
||||
snapshot(sender: string): { ops: number; bytes: number } | null {
|
||||
const buckets = this.bySender.get(sender);
|
||||
if (buckets === undefined) return null;
|
||||
this.refill(buckets, Date.now());
|
||||
return { ops: buckets.ops.tokens, bytes: buckets.bytes.tokens };
|
||||
}
|
||||
|
||||
private costFor(op: string): number {
|
||||
const explicit = (this.opCost as Record<string, number>)[op];
|
||||
return explicit ?? this.opCost.default;
|
||||
}
|
||||
|
||||
private getBuckets(sender: string): SenderBuckets {
|
||||
let b = this.bySender.get(sender);
|
||||
if (b !== undefined) return b;
|
||||
const now = Date.now();
|
||||
b = {
|
||||
ops: {
|
||||
tokens: this.opsCapacity,
|
||||
capacity: this.opsCapacity,
|
||||
refillRate: this.opsRefillPerMs,
|
||||
lastRefill: now,
|
||||
},
|
||||
bytes: {
|
||||
tokens: this.bytesCapacity,
|
||||
capacity: this.bytesCapacity,
|
||||
refillRate: this.bytesRefillPerMs,
|
||||
lastRefill: now,
|
||||
},
|
||||
};
|
||||
this.bySender.set(sender, b);
|
||||
return b;
|
||||
}
|
||||
|
||||
private refill(buckets: SenderBuckets, now: number): void {
|
||||
const refillBucket = (b: Bucket): void => {
|
||||
const elapsed = now - b.lastRefill;
|
||||
if (elapsed <= 0) return;
|
||||
b.tokens = Math.min(b.capacity, b.tokens + elapsed * b.refillRate);
|
||||
b.lastRefill = now;
|
||||
};
|
||||
refillBucket(buckets.ops);
|
||||
refillBucket(buckets.bytes);
|
||||
}
|
||||
}
|
||||
289
packages/shade-files/src/server/streams-bridge.ts
Normal file
289
packages/shade-files/src/server/streams-bridge.ts
Normal file
@@ -0,0 +1,289 @@
|
||||
/**
|
||||
* Server-side bridge between `@shade/files` content RPC ops and the
|
||||
* `@shade/transfer` engine.
|
||||
*
|
||||
* Two responsibilities:
|
||||
* 1. **Inbound writes (client → server, > 256 KiB).** When a client kicks
|
||||
* `shade.upload(...)` with `userMetadata.shadeFilesWriteId = <id>`, this
|
||||
* bridge intercepts the corresponding `IncomingTransfer` from
|
||||
* `shade.onIncomingTransfer`, **immediately** calls `accept(...)` (the
|
||||
* transfer engine requires synchronous accept — it rejects chunks that
|
||||
* arrive before accept), pipes the plaintext into a TransformStream,
|
||||
* and parks the readable side. The `write` RPC handler then awaits
|
||||
* `awaitWrite(writeId, ...)` to pick up the readable, hand it to the
|
||||
* user handler, and observe the transfer's done() result.
|
||||
*
|
||||
* 2. **Outbound reads (server → client, > 256 KiB).** When the user-supplied
|
||||
* `read` handler returns `{ kind: 'streams', stream, size, sha256 }`,
|
||||
* this bridge calls `shade.upload(...)` with
|
||||
* `userMetadata.shadeFilesReadStreamId = <id>`. The id is then echoed
|
||||
* back to the client in the `read` RPC response so the client-side
|
||||
* bridge can subscribe to the matching incoming transfer.
|
||||
*/
|
||||
import type { IncomingTransfer, TransferHandle, TransferProgress } from '@shade/transfer';
|
||||
import { generateRequestId } from '../protocol/correlate.js';
|
||||
import { OperationTimeoutError } from '../schemas/errors.js';
|
||||
|
||||
/** Metadata key carried on inbound (client → server) write streams. */
|
||||
export const META_KEY_WRITE_ID = 'shadeFilesWriteId';
|
||||
/** Metadata key carried on outbound (server → client) read streams. */
|
||||
export const META_KEY_READ_STREAM_ID = 'shadeFilesReadStreamId';
|
||||
|
||||
/**
|
||||
* Subset of `Shade` we depend on. Lets us unit-test without a real Shade
|
||||
* runtime and keeps the bridge framework-agnostic.
|
||||
*/
|
||||
export interface StreamsBridgeShade {
|
||||
upload(opts: import('@shade/transfer').TransferOptions): Promise<TransferHandle>;
|
||||
onIncomingTransfer(
|
||||
handler: (incoming: IncomingTransfer) => void | Promise<void>,
|
||||
): Promise<() => void>;
|
||||
}
|
||||
|
||||
export interface ParkedWrite {
|
||||
/** Sender address from the transfer engine. */
|
||||
from: string;
|
||||
/** Plaintext stream. The bridge already accepted the transfer; just consume. */
|
||||
readable: ReadableStream<Uint8Array>;
|
||||
/** Resolves when the transfer fully completes (sha256 available). */
|
||||
done: Promise<{ sha256: string; bytesSent: number }>;
|
||||
/** Underlying transfer handle — for abort propagation. */
|
||||
handle: TransferHandle;
|
||||
/** Unix epoch ms when the transfer arrived. Used for stale-eviction. */
|
||||
arrivedAt: number;
|
||||
}
|
||||
|
||||
export interface AwaitWriteOptions {
|
||||
/** Sender address — must match `parked.from` for delivery. */
|
||||
expectedFrom: string;
|
||||
/** AbortSignal from the OpContext (cancellation/timeout). */
|
||||
signal: AbortSignal;
|
||||
/** Hard deadline (ms from now). Default 60_000. */
|
||||
timeoutMs?: number;
|
||||
}
|
||||
|
||||
export interface ServerStreamsBridge {
|
||||
/**
|
||||
* Wait for an inbound transfer carrying `userMetadata.shadeFilesWriteId
|
||||
* === writeId` from `expectedFrom`. Resolves with the parked entry whose
|
||||
* `readable` can be consumed by the user handler.
|
||||
*
|
||||
* If the transfer has already arrived (race: stream-init beat the RPC),
|
||||
* resolves immediately. Otherwise awaits up to `timeoutMs`.
|
||||
*/
|
||||
awaitWrite(writeId: string, opts: AwaitWriteOptions): Promise<ParkedWrite>;
|
||||
|
||||
/**
|
||||
* Initiate an outbound transfer to `peer` carrying the given stream.
|
||||
* Stamps `userMetadata.shadeFilesReadStreamId = <streamId>`. Returns the
|
||||
* new streamId so the caller can echo it back to the client over RPC.
|
||||
*/
|
||||
initiateRead(opts: {
|
||||
peer: string;
|
||||
stream: ReadableStream<Uint8Array>;
|
||||
size: number;
|
||||
contentType?: string;
|
||||
name?: string;
|
||||
signal?: AbortSignal;
|
||||
onProgress?: (p: TransferProgress) => void;
|
||||
}): Promise<{ readStreamId: string; handle: TransferHandle }>;
|
||||
|
||||
/** Tear down the incoming-transfer subscription and reject all pending awaits. */
|
||||
destroy(): Promise<void>;
|
||||
}
|
||||
|
||||
interface PendingWaiter {
|
||||
resolve: (parked: ParkedWrite) => void;
|
||||
reject: (err: unknown) => void;
|
||||
expectedFrom: string;
|
||||
timer: ReturnType<typeof setTimeout> | null;
|
||||
abortListener: (() => void) | null;
|
||||
signal: AbortSignal;
|
||||
}
|
||||
|
||||
export interface CreateServerStreamsBridgeOptions {
|
||||
/** Default deadline for `awaitWrite` if the caller doesn't supply one. */
|
||||
defaultAwaitWriteTimeoutMs?: number;
|
||||
/** How long to retain a parked transfer waiting for its RPC. Default 60_000. */
|
||||
parkedWriteTtlMs?: number;
|
||||
}
|
||||
|
||||
export async function createServerStreamsBridge(
|
||||
shade: StreamsBridgeShade,
|
||||
options: CreateServerStreamsBridgeOptions = {},
|
||||
): Promise<ServerStreamsBridge> {
|
||||
const parkedWriteTtlMs = options.parkedWriteTtlMs ?? 60_000;
|
||||
const defaultAwaitTimeoutMs = options.defaultAwaitWriteTimeoutMs ?? 60_000;
|
||||
|
||||
const parked = new Map<string, ParkedWrite>();
|
||||
const waiters = new Map<string, PendingWaiter>();
|
||||
let destroyed = false;
|
||||
|
||||
const unsubscribeIncoming = await shade.onIncomingTransfer(async (incoming) => {
|
||||
const writeId = incoming.metadata.userMetadata?.[META_KEY_WRITE_ID];
|
||||
if (writeId === undefined) return; // not a files-bridge transfer; ignore
|
||||
|
||||
// Accept synchronously into a TransformStream so the engine can deliver
|
||||
// chunks immediately. The readable side is parked for the matching RPC.
|
||||
const ts = new TransformStream<Uint8Array, Uint8Array>();
|
||||
let handle: TransferHandle;
|
||||
try {
|
||||
handle = await incoming.accept({
|
||||
output: { kind: 'pipe', pipeTo: ts.writable },
|
||||
});
|
||||
} catch (err) {
|
||||
// Accept failed — likely double-init or shutdown. Drop the parked
|
||||
// entry; the awaiting RPC will time out.
|
||||
console.error('[shade-files streams-bridge] accept failed:', err);
|
||||
return;
|
||||
}
|
||||
|
||||
const arrived: ParkedWrite = {
|
||||
from: incoming.from,
|
||||
readable: ts.readable,
|
||||
done: handle.done().then((r) => ({ sha256: r.sha256, bytesSent: r.bytesSent })),
|
||||
handle,
|
||||
arrivedAt: Date.now(),
|
||||
};
|
||||
arrived.done.catch(() => {
|
||||
/* swallow until consumer awaits */
|
||||
});
|
||||
|
||||
const waiter = waiters.get(writeId);
|
||||
if (waiter !== undefined) {
|
||||
waiters.delete(writeId);
|
||||
cleanupWaiter(waiter);
|
||||
if (incoming.from !== waiter.expectedFrom) {
|
||||
void handle.abort('sender-mismatch').catch(() => undefined);
|
||||
waiter.reject(
|
||||
new Error(
|
||||
`streams-bridge: writeId=${writeId} delivered by ${incoming.from}, expected ${waiter.expectedFrom}`,
|
||||
),
|
||||
);
|
||||
return;
|
||||
}
|
||||
waiter.resolve(arrived);
|
||||
return;
|
||||
}
|
||||
|
||||
// No waiter yet — park.
|
||||
parked.set(writeId, arrived);
|
||||
setTimeout(() => {
|
||||
const stale = parked.get(writeId);
|
||||
if (stale === arrived) {
|
||||
parked.delete(writeId);
|
||||
void handle.abort('rpc-timeout').catch(() => undefined);
|
||||
}
|
||||
}, parkedWriteTtlMs).unref?.();
|
||||
});
|
||||
|
||||
function cleanupWaiter(w: PendingWaiter): void {
|
||||
if (w.timer !== null) clearTimeout(w.timer);
|
||||
if (w.abortListener !== null) {
|
||||
w.signal.removeEventListener('abort', w.abortListener);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
async awaitWrite(writeId, opts) {
|
||||
if (destroyed) throw new Error('streams-bridge: destroyed');
|
||||
// 1. Already parked?
|
||||
const ready = parked.get(writeId);
|
||||
if (ready !== undefined) {
|
||||
parked.delete(writeId);
|
||||
if (ready.from !== opts.expectedFrom) {
|
||||
void ready.handle.abort('sender-mismatch').catch(() => undefined);
|
||||
throw new Error(
|
||||
`streams-bridge: writeId=${writeId} delivered by ${ready.from}, expected ${opts.expectedFrom}`,
|
||||
);
|
||||
}
|
||||
return ready;
|
||||
}
|
||||
|
||||
if (waiters.has(writeId)) {
|
||||
throw new Error(`streams-bridge: writeId=${writeId} already awaited`);
|
||||
}
|
||||
|
||||
// 2. Park a waiter
|
||||
const timeoutMs = opts.timeoutMs ?? defaultAwaitTimeoutMs;
|
||||
return await new Promise<ParkedWrite>((resolve, reject) => {
|
||||
const w: PendingWaiter = {
|
||||
resolve,
|
||||
reject,
|
||||
expectedFrom: opts.expectedFrom,
|
||||
timer: null,
|
||||
abortListener: null,
|
||||
signal: opts.signal,
|
||||
};
|
||||
w.timer = setTimeout(() => {
|
||||
if (waiters.get(writeId) === w) {
|
||||
waiters.delete(writeId);
|
||||
cleanupWaiter(w);
|
||||
reject(new OperationTimeoutError(`streams-bridge: writeId=${writeId} timed out after ${timeoutMs}ms`));
|
||||
}
|
||||
}, timeoutMs);
|
||||
if (opts.signal.aborted) {
|
||||
cleanupWaiter(w);
|
||||
reject(opts.signal.reason ?? new Error('aborted before await'));
|
||||
return;
|
||||
}
|
||||
const onAbort = (): void => {
|
||||
if (waiters.get(writeId) === w) {
|
||||
waiters.delete(writeId);
|
||||
cleanupWaiter(w);
|
||||
reject(opts.signal.reason ?? new Error('aborted'));
|
||||
}
|
||||
};
|
||||
w.abortListener = onAbort;
|
||||
opts.signal.addEventListener('abort', onAbort, { once: true });
|
||||
waiters.set(writeId, w);
|
||||
});
|
||||
},
|
||||
|
||||
async initiateRead(opts) {
|
||||
if (destroyed) throw new Error('streams-bridge: destroyed');
|
||||
const readStreamId = generateRequestId();
|
||||
const transferOpts: import('@shade/transfer').TransferOptions = {
|
||||
to: opts.peer,
|
||||
input: opts.stream,
|
||||
metadata: {
|
||||
...(opts.name !== undefined ? { name: opts.name } : {}),
|
||||
...(opts.contentType !== undefined ? { contentType: opts.contentType } : {}),
|
||||
sizeBytes: opts.size,
|
||||
userMetadata: { [META_KEY_READ_STREAM_ID]: readStreamId },
|
||||
},
|
||||
...(opts.signal !== undefined ? { signal: opts.signal } : {}),
|
||||
...(opts.onProgress !== undefined ? { onProgress: opts.onProgress } : {}),
|
||||
};
|
||||
const handle = await shade.upload(transferOpts);
|
||||
// The dispatcher returns the response immediately and doesn't itself
|
||||
// await this handle — the receiver awaits its own handle.done() on
|
||||
// the inbound side. Attach a no-op catch so engine teardown during
|
||||
// an in-flight outbound transfer doesn't surface as an unhandled
|
||||
// rejection in test environments.
|
||||
handle.done().catch(() => undefined);
|
||||
return { readStreamId, handle };
|
||||
},
|
||||
|
||||
async destroy() {
|
||||
if (destroyed) return;
|
||||
destroyed = true;
|
||||
unsubscribeIncoming();
|
||||
// Reject all waiting RPCs
|
||||
for (const w of waiters.values()) {
|
||||
cleanupWaiter(w);
|
||||
w.reject(new Error('streams-bridge: destroyed'));
|
||||
}
|
||||
waiters.clear();
|
||||
// Abort all parked transfers
|
||||
for (const p of parked.values()) {
|
||||
try {
|
||||
await p.handle.abort('bridge-destroyed');
|
||||
} catch {
|
||||
/* swallow */
|
||||
}
|
||||
}
|
||||
parked.clear();
|
||||
},
|
||||
};
|
||||
}
|
||||
61
packages/shade-files/src/server/thumbnail.ts
Normal file
61
packages/shade-files/src/server/thumbnail.ts
Normal file
@@ -0,0 +1,61 @@
|
||||
/**
|
||||
* Thumbnail format-hardening helper.
|
||||
*
|
||||
* Magic-byte verification for the three formats `@shade/files` advertises
|
||||
* over the wire (`png`, `webp`, `jpeg`). Apps producing thumbnails outside
|
||||
* the dispatcher (e.g. precomputing on disk) can call `assertThumbnailFormat`
|
||||
* to fail-fast on mislabelled bytes before they reach the network.
|
||||
*
|
||||
* The same magic-byte table is used inside `io-adapters.ts:adaptThumbnailResult`
|
||||
* to enforce the invariant on every outbound thumbnail.
|
||||
*/
|
||||
|
||||
import { ConflictError } from '../schemas/errors.js';
|
||||
|
||||
const PNG_MAGIC = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
|
||||
const JPEG_MAGIC = new Uint8Array([0xff, 0xd8, 0xff]);
|
||||
const WEBP_RIFF = new Uint8Array([0x52, 0x49, 0x46, 0x46]);
|
||||
const WEBP_HEAD = new Uint8Array([0x57, 0x45, 0x42, 0x50]);
|
||||
|
||||
export type ThumbnailFormat = 'png' | 'webp' | 'jpeg';
|
||||
|
||||
/**
|
||||
* Verify the bytes look like the declared format. Returns boolean — does
|
||||
* not throw. For a throwing variant, use `assertThumbnailFormat`.
|
||||
*/
|
||||
export function isThumbnailFormat(bytes: Uint8Array, format: ThumbnailFormat): boolean {
|
||||
switch (format) {
|
||||
case 'png':
|
||||
return startsWith(bytes, PNG_MAGIC);
|
||||
case 'jpeg':
|
||||
return startsWith(bytes, JPEG_MAGIC);
|
||||
case 'webp': {
|
||||
if (bytes.byteLength < 12) return false;
|
||||
if (!startsWith(bytes, WEBP_RIFF)) return false;
|
||||
for (let i = 0; i < 4; i++) {
|
||||
if (bytes[8 + i] !== WEBP_HEAD[i]) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Throws `ConflictError` if `bytes` doesn't match the declared `format`. */
|
||||
export function assertThumbnailFormat(
|
||||
bytes: Uint8Array,
|
||||
format: ThumbnailFormat,
|
||||
): void {
|
||||
if (!isThumbnailFormat(bytes, format)) {
|
||||
throw new ConflictError(
|
||||
`thumbnail bytes do not match declared format=${format}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function startsWith(bytes: Uint8Array, prefix: Uint8Array): boolean {
|
||||
if (bytes.byteLength < prefix.byteLength) return false;
|
||||
for (let i = 0; i < prefix.byteLength; i++) {
|
||||
if (bytes[i] !== prefix[i]) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
86
packages/shade-files/src/utils/path.ts
Normal file
86
packages/shade-files/src/utils/path.ts
Normal file
@@ -0,0 +1,86 @@
|
||||
/**
|
||||
* Minimal POSIX path utilities — works in browser and Bun without depending
|
||||
* on Node's built-in `node:path`. Only handles forward-slash absolute paths.
|
||||
*/
|
||||
|
||||
/**
|
||||
* Normalize a POSIX absolute path: collapse `//`, resolve `.` and `..`,
|
||||
* preserve trailing slash semantics. Returns `'/'` for `''`/`'/'` input.
|
||||
*
|
||||
* Throws nothing on invalid input — sanitization is the caller's job
|
||||
* (`validatePath` in `server/path-policy.ts` does that).
|
||||
*/
|
||||
export function posixNormalize(path: string): string {
|
||||
if (path === '' || path === '/') return '/';
|
||||
const isAbs = path.startsWith('/');
|
||||
const segments = path.split('/');
|
||||
const stack: string[] = [];
|
||||
for (const seg of segments) {
|
||||
if (seg === '' || seg === '.') continue;
|
||||
if (seg === '..') {
|
||||
if (stack.length > 0 && stack[stack.length - 1] !== '..') {
|
||||
stack.pop();
|
||||
} else if (!isAbs) {
|
||||
stack.push('..');
|
||||
}
|
||||
// For absolute paths, `..` past root is silently dropped.
|
||||
continue;
|
||||
}
|
||||
stack.push(seg);
|
||||
}
|
||||
const joined = stack.join('/');
|
||||
if (isAbs) return '/' + joined;
|
||||
return joined === '' ? '.' : joined;
|
||||
}
|
||||
|
||||
/** Join path segments with POSIX semantics. */
|
||||
export function posixJoin(...parts: string[]): string {
|
||||
if (parts.length === 0) return '.';
|
||||
const joined = parts.filter((p) => p !== '').join('/');
|
||||
return posixNormalize(joined);
|
||||
}
|
||||
|
||||
/** Return the directory portion of a path, or `'/'` for root. */
|
||||
export function posixDirname(path: string): string {
|
||||
const normalized = posixNormalize(path);
|
||||
if (normalized === '/') return '/';
|
||||
const lastSlash = normalized.lastIndexOf('/');
|
||||
if (lastSlash === 0) return '/';
|
||||
if (lastSlash === -1) return '.';
|
||||
return normalized.slice(0, lastSlash);
|
||||
}
|
||||
|
||||
/** Return the file/dir name (last component). */
|
||||
export function posixBasename(path: string): string {
|
||||
const normalized = posixNormalize(path);
|
||||
if (normalized === '/') return '';
|
||||
const lastSlash = normalized.lastIndexOf('/');
|
||||
return lastSlash === -1 ? normalized : normalized.slice(lastSlash + 1);
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode percent-escapes BEFORE security checks so attackers can't
|
||||
* smuggle `..` via `%2e%2e`. Decodes UTF-8 sequences where possible;
|
||||
* leaves invalid sequences untouched (the caller still rejects them).
|
||||
*/
|
||||
export function decodePercentEscapes(s: string): string {
|
||||
if (!s.includes('%')) return s;
|
||||
try {
|
||||
return decodeURIComponent(s);
|
||||
} catch {
|
||||
// Invalid percent-encoding — leave as-is so policy rejects on the raw form.
|
||||
return s;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Predicate: does `child` lie under `root` after both are normalized?
|
||||
* Both must be absolute. `root` is treated as a directory boundary.
|
||||
*/
|
||||
export function isPathInside(child: string, root: string): boolean {
|
||||
const c = posixNormalize(child);
|
||||
const r = posixNormalize(root);
|
||||
if (r === '/') return c.startsWith('/');
|
||||
// Ensure boundary alignment so `/foo` is NOT inside `/foobar`.
|
||||
return c === r || c.startsWith(r + '/');
|
||||
}
|
||||
77
packages/shade-files/tests/integration/custom-op.test.ts
Normal file
77
packages/shade-files/tests/integration/custom-op.test.ts
Normal file
@@ -0,0 +1,77 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { z } from 'zod';
|
||||
import { setupFileRig, type FileTestRig } from './helpers/rig.js';
|
||||
import { CustomOpRejectedError, NotImplementedError } from '../../src/index.js';
|
||||
|
||||
// Module augmentation: register a typed custom op for the test.
|
||||
declare module '../../src/index.js' {
|
||||
interface CustomOpsMap {
|
||||
'test.echo': { args: { message: string }; response: { echoed: string } };
|
||||
'test.add': { args: { a: number; b: number }; response: { sum: number } };
|
||||
}
|
||||
}
|
||||
|
||||
describe('Custom ops — registry + Zod validation + typed I/O', () => {
|
||||
let rig: FileTestRig;
|
||||
const callLog: string[] = [];
|
||||
|
||||
beforeAll(async () => {
|
||||
rig = await setupFileRig({
|
||||
custom: {
|
||||
'test.echo': {
|
||||
args: z.object({ message: z.string().min(1).max(64) }),
|
||||
response: z.object({ echoed: z.string() }),
|
||||
handler: async (args, ctx) => {
|
||||
callLog.push(`echo:${ctx.sender}:${args.message}`);
|
||||
return { echoed: args.message.toUpperCase() };
|
||||
},
|
||||
},
|
||||
'test.add': {
|
||||
args: z.object({ a: z.number(), b: z.number() }),
|
||||
response: z.object({ sum: z.number() }),
|
||||
handler: async (args) => ({ sum: args.a + args.b }),
|
||||
},
|
||||
'test.bad-response': {
|
||||
args: z.object({}),
|
||||
response: z.object({ x: z.number() }),
|
||||
// Returns wrong shape on purpose.
|
||||
handler: async () => ({ y: 'not-a-number' }) as unknown as { x: number },
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await rig.teardown();
|
||||
});
|
||||
|
||||
test('typed echo round-trips through registered Zod schemas', async () => {
|
||||
const result = await rig.fs.custom('test.echo', { message: 'hello' });
|
||||
expect(result.echoed).toBe('HELLO');
|
||||
expect(callLog).toContain('echo:alice:hello');
|
||||
});
|
||||
|
||||
test('typed add', async () => {
|
||||
const result = await rig.fs.custom('test.add', { a: 3, b: 4 });
|
||||
expect(result.sum).toBe(7);
|
||||
});
|
||||
|
||||
test('invalid args (Zod-rejected payload) → InvalidArgsError', async () => {
|
||||
await expect(
|
||||
// message: '' violates min(1) — TypeScript still allows it since string
|
||||
rig.fs.custom('test.echo', { message: '' }),
|
||||
).rejects.toThrow();
|
||||
});
|
||||
|
||||
test('unknown custom op name → NotImplementedError', async () => {
|
||||
await expect(
|
||||
rig.fs.custom('test.unknown' as never, {} as never),
|
||||
).rejects.toBeInstanceOf(NotImplementedError);
|
||||
});
|
||||
|
||||
test('handler returns wrong shape → CustomOpRejectedError', async () => {
|
||||
await expect(
|
||||
rig.fs.custom('test.bad-response' as never, {} as never),
|
||||
).rejects.toBeInstanceOf(CustomOpRejectedError);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,210 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { sha256 } from '@noble/hashes/sha2.js';
|
||||
import {
|
||||
NotFoundError,
|
||||
downloadDirectory,
|
||||
createMemoryDirectory,
|
||||
walk,
|
||||
type DirectoryHandleLike,
|
||||
type FileEntry,
|
||||
} from '../../src/index.js';
|
||||
import { setupFileRig, type FileTestRig } from './helpers/rig.js';
|
||||
|
||||
interface StoredFile {
|
||||
bytes: Uint8Array;
|
||||
contentType?: string;
|
||||
sha256: string;
|
||||
}
|
||||
|
||||
function bytesToHex(b: Uint8Array): string {
|
||||
return Array.from(b, (x) => x.toString(16).padStart(2, '0')).join('');
|
||||
}
|
||||
|
||||
async function streamToBytes(s: ReadableStream<Uint8Array>): Promise<Uint8Array> {
|
||||
const reader = s.getReader();
|
||||
const parts: Uint8Array[] = [];
|
||||
let total = 0;
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (value === undefined) continue;
|
||||
parts.push(value);
|
||||
total += value.byteLength;
|
||||
}
|
||||
reader.releaseLock();
|
||||
const out = new Uint8Array(total);
|
||||
let offset = 0;
|
||||
for (const p of parts) {
|
||||
out.set(p, offset);
|
||||
offset += p.byteLength;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function streamFromBytes(bytes: Uint8Array): ReadableStream<Uint8Array> {
|
||||
return new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
controller.enqueue(bytes);
|
||||
controller.close();
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
describe('downloadDirectory — bulk download from remote', () => {
|
||||
let rig: FileTestRig;
|
||||
const blobs = new Map<string, StoredFile>();
|
||||
const dirs = new Set<string>();
|
||||
|
||||
beforeAll(async () => {
|
||||
blobs.clear();
|
||||
dirs.clear();
|
||||
// Build remote tree:
|
||||
// /src/
|
||||
// ├── small.txt ('hello world\n', 12 bytes)
|
||||
// ├── img.bin (50 KiB random)
|
||||
// └── nested/
|
||||
// ├── big.bin (400 KiB random)
|
||||
// └── tiny.bin (3 bytes)
|
||||
dirs.add('/');
|
||||
dirs.add('/src');
|
||||
dirs.add('/src/nested');
|
||||
const small = new TextEncoder().encode('hello world\n');
|
||||
const mid = new Uint8Array(50 * 1024); crypto.getRandomValues(mid);
|
||||
const big = new Uint8Array(400 * 1024); crypto.getRandomValues(big);
|
||||
const tiny = new Uint8Array([1, 2, 3]);
|
||||
blobs.set('/src/small.txt', { bytes: small, sha256: bytesToHex(sha256(small)), contentType: 'text/plain' });
|
||||
blobs.set('/src/img.bin', { bytes: mid, sha256: bytesToHex(sha256(mid)) });
|
||||
blobs.set('/src/nested/big.bin', { bytes: big, sha256: bytesToHex(sha256(big)) });
|
||||
blobs.set('/src/nested/tiny.bin', { bytes: tiny, sha256: bytesToHex(sha256(tiny)) });
|
||||
|
||||
rig = await setupFileRig({
|
||||
list: async (ctx) => {
|
||||
if (!dirs.has(ctx.path)) throw new NotFoundError(ctx.path);
|
||||
const entries: FileEntry[] = [];
|
||||
const dirPrefix = ctx.path === '/' ? '/' : ctx.path + '/';
|
||||
// Subdirs
|
||||
for (const d of dirs) {
|
||||
if (d === ctx.path) continue;
|
||||
if (!d.startsWith(dirPrefix)) continue;
|
||||
const rest = d.slice(dirPrefix.length);
|
||||
if (rest.includes('/')) continue;
|
||||
entries.push({ name: rest, kind: 'dir', size: 0, mtime: 0, metadata: {} });
|
||||
}
|
||||
// Files
|
||||
for (const [path, blob] of blobs) {
|
||||
if (!path.startsWith(dirPrefix)) continue;
|
||||
const rest = path.slice(dirPrefix.length);
|
||||
if (rest.includes('/')) continue;
|
||||
entries.push({
|
||||
name: rest,
|
||||
kind: 'file',
|
||||
size: blob.bytes.byteLength,
|
||||
mtime: 0,
|
||||
...(blob.contentType !== undefined ? { contentType: blob.contentType } : {}),
|
||||
metadata: {},
|
||||
});
|
||||
}
|
||||
return { entries, hasMore: false };
|
||||
},
|
||||
read: async (ctx) => {
|
||||
const blob = blobs.get(ctx.path);
|
||||
if (blob === undefined) throw new NotFoundError(ctx.path);
|
||||
if (blob.bytes.byteLength > 256 * 1024) {
|
||||
return blob.contentType !== undefined
|
||||
? {
|
||||
kind: 'streams',
|
||||
stream: streamFromBytes(blob.bytes),
|
||||
size: blob.bytes.byteLength,
|
||||
sha256: blob.sha256,
|
||||
contentType: blob.contentType,
|
||||
}
|
||||
: {
|
||||
kind: 'streams',
|
||||
stream: streamFromBytes(blob.bytes),
|
||||
size: blob.bytes.byteLength,
|
||||
sha256: blob.sha256,
|
||||
};
|
||||
}
|
||||
return blob.contentType !== undefined
|
||||
? { kind: 'inline', bytes: blob.bytes, sha256: blob.sha256, contentType: blob.contentType }
|
||||
: { kind: 'inline', bytes: blob.bytes, sha256: blob.sha256 };
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await rig.teardown();
|
||||
});
|
||||
|
||||
test('downloads entire tree, sha256 matches per file', async () => {
|
||||
const local = createMemoryDirectory('local');
|
||||
const handle = downloadDirectory(rig.fs, '/src', local);
|
||||
const result = await handle.done();
|
||||
expect(result.filesDone).toBe(4);
|
||||
expect(result.bytesDone).toBe(12 + 50 * 1024 + 400 * 1024 + 3);
|
||||
|
||||
// Verify local tree contents
|
||||
const downloadedFiles = new Map<string, Uint8Array>();
|
||||
async function dump(dir: DirectoryHandleLike, prefix: string): Promise<void> {
|
||||
for await (const [name, child] of dir.entries()) {
|
||||
const path = prefix === '' ? name : `${prefix}/${name}`;
|
||||
if (child.kind === 'directory') {
|
||||
await dump(child as DirectoryHandleLike, path);
|
||||
} else {
|
||||
const file = await (child as { getFile: () => Promise<{ arrayBuffer: () => Promise<ArrayBuffer> }> }).getFile();
|
||||
downloadedFiles.set(path, new Uint8Array(await file.arrayBuffer()));
|
||||
}
|
||||
}
|
||||
}
|
||||
await dump(local, '');
|
||||
expect(downloadedFiles.size).toBe(4);
|
||||
expect(downloadedFiles.has('small.txt')).toBe(true);
|
||||
expect(downloadedFiles.has('img.bin')).toBe(true);
|
||||
expect(downloadedFiles.has('nested/big.bin')).toBe(true);
|
||||
expect(downloadedFiles.has('nested/tiny.bin')).toBe(true);
|
||||
|
||||
expect(bytesToHex(sha256(downloadedFiles.get('nested/big.bin')!))).toBe(
|
||||
blobs.get('/src/nested/big.bin')!.sha256,
|
||||
);
|
||||
expect(bytesToHex(sha256(downloadedFiles.get('img.bin')!))).toBe(
|
||||
blobs.get('/src/img.bin')!.sha256,
|
||||
);
|
||||
});
|
||||
|
||||
test('aggregated progress events fire monotonically', async () => {
|
||||
const local = createMemoryDirectory('local');
|
||||
const handle = downloadDirectory(rig.fs, '/src', local);
|
||||
const progresses: { filesDone: number; bytesDone: number }[] = [];
|
||||
(async () => {
|
||||
for await (const ev of handle.events) {
|
||||
if (ev.type === 'progress') {
|
||||
progresses.push({ filesDone: ev.filesDone, bytesDone: ev.bytesDone });
|
||||
}
|
||||
}
|
||||
})().catch(() => undefined);
|
||||
await handle.done();
|
||||
await new Promise((r) => setTimeout(r, 30));
|
||||
for (let i = 1; i < progresses.length; i++) {
|
||||
expect(progresses[i]!.filesDone).toBeGreaterThanOrEqual(progresses[i - 1]!.filesDone);
|
||||
expect(progresses[i]!.bytesDone).toBeGreaterThanOrEqual(progresses[i - 1]!.bytesDone);
|
||||
}
|
||||
});
|
||||
|
||||
test('aborts via handle.abort()', async () => {
|
||||
const local = createMemoryDirectory('local');
|
||||
const handle = downloadDirectory(rig.fs, '/src', local);
|
||||
setTimeout(() => void handle.abort('test-cancel'), 5);
|
||||
await expect(handle.done()).rejects.toThrow();
|
||||
});
|
||||
|
||||
test('walk + downloadDirectory are consistent', async () => {
|
||||
const local = createMemoryDirectory('local');
|
||||
const remoteFiles: string[] = [];
|
||||
for await (const item of walk(rig.fs, '/src')) {
|
||||
if (item.entry.kind === 'file') remoteFiles.push(item.relativePath);
|
||||
}
|
||||
const handle = downloadDirectory(rig.fs, '/src', local);
|
||||
const result = await handle.done();
|
||||
expect(result.filesDone).toBe(remoteFiles.length);
|
||||
});
|
||||
});
|
||||
142
packages/shade-files/tests/integration/helpers/rig.ts
Normal file
142
packages/shade-files/tests/integration/helpers/rig.ts
Normal file
@@ -0,0 +1,142 @@
|
||||
import { createShade, type Shade } from '@shade/sdk';
|
||||
import {
|
||||
createPrekeyServer,
|
||||
MemoryPrekeyStore,
|
||||
PrekeyServerEvents,
|
||||
} from '@shade/server';
|
||||
import { SubtleCryptoProvider } from '@shade/crypto-web';
|
||||
import {
|
||||
PendingRpcRegistry,
|
||||
ShadeFileRpcChannel,
|
||||
attachClientRouting,
|
||||
attachFileHandler,
|
||||
createClientStreamsBridge,
|
||||
createFileClient,
|
||||
createFileHandler,
|
||||
createServerStreamsBridge,
|
||||
type ClientStreamsBridge,
|
||||
type FileClient,
|
||||
type FileHandler,
|
||||
type FileHandlerConfig,
|
||||
type ServerStreamsBridge,
|
||||
} from '../../../src/index.js';
|
||||
|
||||
const crypto = new SubtleCryptoProvider();
|
||||
|
||||
export interface FileTestRig {
|
||||
alice: Shade;
|
||||
bob: Shade;
|
||||
fs: FileClient;
|
||||
bobHandler: FileHandler;
|
||||
/** Server-side streams bridge (Bob). */
|
||||
bobStreamsBridge: ServerStreamsBridge;
|
||||
/** Client-side streams bridge (Alice). */
|
||||
aliceStreamsBridge: ClientStreamsBridge;
|
||||
/** Tear everything down (kills servers, shuts down shades). */
|
||||
teardown(): Promise<void>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Setup options.
|
||||
*
|
||||
* Defaults to wiring streams-bridges on both sides so content I/O tests
|
||||
* (`read-write-streams.test.ts`) work transparently. Pass `withStreams: false`
|
||||
* to skip — useful for the legacy `std-ops` tests that don't need them.
|
||||
*/
|
||||
export interface SetupRigOptions {
|
||||
withStreams?: boolean;
|
||||
}
|
||||
|
||||
export async function setupFileRig(
|
||||
bobConfig: Omit<FileHandlerConfig, 'streamsBridge'>,
|
||||
options: SetupRigOptions = {},
|
||||
): Promise<FileTestRig> {
|
||||
const withStreams = options.withStreams ?? true;
|
||||
// 1. Prekey server
|
||||
const prekeyEvents = new PrekeyServerEvents();
|
||||
const prekey = createPrekeyServer({
|
||||
crypto,
|
||||
store: new MemoryPrekeyStore(),
|
||||
disableRateLimit: true,
|
||||
events: prekeyEvents,
|
||||
});
|
||||
const prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
|
||||
const prekeyUrl = `http://localhost:${prekeyServer.port}`;
|
||||
|
||||
// 2. Two Shades
|
||||
const alice = await createShade({ prekeyServer: prekeyUrl, address: 'alice' });
|
||||
const bob = await createShade({ prekeyServer: prekeyUrl, address: 'bob' });
|
||||
|
||||
// 3. BOTH sides need a transferRoute mounted because file-RPC is
|
||||
// request/response — Bob's reply must reach Alice's HTTP endpoint
|
||||
// just as Alice's request reached Bob's.
|
||||
let bobBaseUrl = '';
|
||||
let aliceBaseUrl = '';
|
||||
|
||||
alice.configureTransfers({
|
||||
resolveBaseUrl: async (peer) => {
|
||||
if (peer === 'bob') return bobBaseUrl;
|
||||
throw new Error(`alice: unknown peer ${peer}`);
|
||||
},
|
||||
});
|
||||
bob.configureTransfers({
|
||||
resolveBaseUrl: async (peer) => {
|
||||
if (peer === 'alice') return aliceBaseUrl;
|
||||
throw new Error(`bob: unknown peer ${peer}`);
|
||||
},
|
||||
});
|
||||
|
||||
const bobApp = await bob.transferRoute();
|
||||
const aliceApp = await alice.transferRoute();
|
||||
const bobServer = Bun.serve({ port: 0, fetch: bobApp.fetch });
|
||||
const aliceServer = Bun.serve({ port: 0, fetch: aliceApp.fetch });
|
||||
bobBaseUrl = `http://localhost:${bobServer.port}`;
|
||||
aliceBaseUrl = `http://localhost:${aliceServer.port}`;
|
||||
|
||||
// 4. Streams bridges (both sides) — required for content I/O > 256 KiB.
|
||||
let bobStreamsBridge: ServerStreamsBridge | undefined;
|
||||
let aliceStreamsBridge: ClientStreamsBridge | undefined;
|
||||
if (withStreams) {
|
||||
bobStreamsBridge = await createServerStreamsBridge(bob);
|
||||
aliceStreamsBridge = await createClientStreamsBridge(alice);
|
||||
}
|
||||
|
||||
// 5. Bob: file handler + channel
|
||||
const bobChannel = new ShadeFileRpcChannel(bob);
|
||||
const fullBobConfig: FileHandlerConfig = {
|
||||
...bobConfig,
|
||||
...(bobStreamsBridge !== undefined ? { streamsBridge: bobStreamsBridge } : {}),
|
||||
};
|
||||
const bobHandler = createFileHandler(bob, fullBobConfig);
|
||||
attachFileHandler(bobChannel, bobHandler);
|
||||
|
||||
// 6. Alice: client + channel + pending registry
|
||||
const aliceChannel = new ShadeFileRpcChannel(alice);
|
||||
const alicePending = new PendingRpcRegistry();
|
||||
attachClientRouting(aliceChannel, alicePending);
|
||||
const fs = createFileClient(alice, aliceChannel, alicePending, 'bob', {
|
||||
defaultTimeoutMs: 5000,
|
||||
...(aliceStreamsBridge !== undefined ? { streamsBridge: aliceStreamsBridge } : {}),
|
||||
});
|
||||
|
||||
return {
|
||||
alice,
|
||||
bob,
|
||||
fs,
|
||||
bobHandler,
|
||||
bobStreamsBridge: bobStreamsBridge as ServerStreamsBridge,
|
||||
aliceStreamsBridge: aliceStreamsBridge as ClientStreamsBridge,
|
||||
async teardown() {
|
||||
bobChannel.destroy();
|
||||
aliceChannel.destroy();
|
||||
bobHandler.destroy();
|
||||
if (bobStreamsBridge !== undefined) await bobStreamsBridge.destroy();
|
||||
if (aliceStreamsBridge !== undefined) await aliceStreamsBridge.destroy();
|
||||
await alice.shutdown();
|
||||
await bob.shutdown();
|
||||
bobServer.stop();
|
||||
aliceServer.stop();
|
||||
prekeyServer.stop();
|
||||
},
|
||||
};
|
||||
}
|
||||
72
packages/shade-files/tests/integration/metrics.test.ts
Normal file
72
packages/shade-files/tests/integration/metrics.test.ts
Normal file
@@ -0,0 +1,72 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { setupFileRig, type FileTestRig } from './helpers/rig.js';
|
||||
import {
|
||||
METRIC_OP_DURATION_MS,
|
||||
METRIC_OP_TOTAL,
|
||||
METRIC_RATE_LIMIT_REJECT_TOTAL,
|
||||
type MetricSink,
|
||||
type MetricTags,
|
||||
type FileEntry,
|
||||
} from '../../src/index.js';
|
||||
|
||||
interface MetricEvent {
|
||||
name: string;
|
||||
value: number;
|
||||
tags: MetricTags;
|
||||
}
|
||||
|
||||
describe('Metrics — onMetric on success', () => {
|
||||
let rig: FileTestRig;
|
||||
const events: MetricEvent[] = [];
|
||||
const sink: MetricSink = (name, value, tags) => events.push({ name, value, tags });
|
||||
|
||||
beforeAll(async () => {
|
||||
rig = await setupFileRig({
|
||||
onMetric: sink,
|
||||
list: async () => ({ entries: [], hasMore: false }),
|
||||
});
|
||||
});
|
||||
afterAll(async () => { await rig.teardown(); });
|
||||
|
||||
test('emits op_total + op_duration_ms with result=ok on success', async () => {
|
||||
events.length = 0;
|
||||
await rig.fs.list('/');
|
||||
const totals = events.filter((e) => e.name === METRIC_OP_TOTAL);
|
||||
expect(totals.length).toBeGreaterThanOrEqual(1);
|
||||
expect(totals[0]!.tags.result).toBe('ok');
|
||||
expect(totals[0]!.tags.op).toBe('list');
|
||||
const durations = events.filter((e) => e.name === METRIC_OP_DURATION_MS);
|
||||
expect(durations.length).toBeGreaterThanOrEqual(1);
|
||||
expect(durations[0]!.value).toBeGreaterThanOrEqual(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Metrics — onMetric rate-limit reject', () => {
|
||||
let rig: FileTestRig;
|
||||
const events: MetricEvent[] = [];
|
||||
const sink: MetricSink = (name, value, tags) => events.push({ name, value, tags });
|
||||
|
||||
beforeAll(async () => {
|
||||
rig = await setupFileRig({
|
||||
onMetric: sink,
|
||||
rateLimits: { maxOpsPerMinutePerSender: 3 },
|
||||
stat: async (_ctx) => {
|
||||
const e: FileEntry = { name: 'x', kind: 'file', size: 1, mtime: 0, metadata: {} };
|
||||
return e;
|
||||
},
|
||||
});
|
||||
});
|
||||
afterAll(async () => { await rig.teardown(); });
|
||||
|
||||
test('emits rate_limit_reject_total when capacity exhausted', async () => {
|
||||
events.length = 0;
|
||||
// Cap is 3; first 3 stats succeed, 4th rejected.
|
||||
await rig.fs.stat('/x');
|
||||
await rig.fs.stat('/x');
|
||||
await rig.fs.stat('/x');
|
||||
await expect(rig.fs.stat('/x')).rejects.toThrow();
|
||||
const rejects = events.filter((e) => e.name === METRIC_RATE_LIMIT_REJECT_TOTAL);
|
||||
expect(rejects.length).toBeGreaterThanOrEqual(1);
|
||||
expect(rejects[0]!.tags.op).toBe('stat');
|
||||
});
|
||||
});
|
||||
138
packages/shade-files/tests/integration/read-write-inline.test.ts
Normal file
138
packages/shade-files/tests/integration/read-write-inline.test.ts
Normal file
@@ -0,0 +1,138 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { sha256 } from '@noble/hashes/sha2.js';
|
||||
import { NotFoundError, type FileEntry } from '../../src/index.js';
|
||||
import type { UserReadResult, UserWriteArgs, WriteResult } from '../../src/server/io-types.js';
|
||||
import type { OpContext } from '../../src/server/handler-context.js';
|
||||
import { setupFileRig, type FileTestRig } from './helpers/rig.js';
|
||||
|
||||
interface StoredBlob {
|
||||
bytes: Uint8Array;
|
||||
contentType?: string;
|
||||
sha256: string;
|
||||
mtime: number;
|
||||
}
|
||||
|
||||
function bytesToHex(b: Uint8Array): string {
|
||||
return Array.from(b, (x) => x.toString(16).padStart(2, '0')).join('');
|
||||
}
|
||||
|
||||
describe('Content I/O — inline read/write E2E', () => {
|
||||
let rig: FileTestRig;
|
||||
const blobs = new Map<string, StoredBlob>();
|
||||
|
||||
beforeAll(async () => {
|
||||
blobs.clear();
|
||||
rig = await setupFileRig({
|
||||
read: async (ctx: OpContext<{ path: string }>): Promise<UserReadResult> => {
|
||||
const blob = blobs.get(ctx.path);
|
||||
if (blob === undefined) throw new NotFoundError(ctx.path);
|
||||
return blob.contentType !== undefined
|
||||
? { kind: 'inline', bytes: blob.bytes, sha256: blob.sha256, contentType: blob.contentType }
|
||||
: { kind: 'inline', bytes: blob.bytes, sha256: blob.sha256 };
|
||||
},
|
||||
write: async (ctx: OpContext<UserWriteArgs>): Promise<WriteResult> => {
|
||||
const args = ctx.args;
|
||||
if (args.content.kind !== 'inline') {
|
||||
throw new Error('expected inline content for this test');
|
||||
}
|
||||
if (blobs.has(args.path) && !args.overwrite) {
|
||||
throw new Error('exists');
|
||||
}
|
||||
const stored: StoredBlob = {
|
||||
bytes: args.content.bytes,
|
||||
...(args.contentType !== undefined ? { contentType: args.contentType } : {}),
|
||||
sha256: args.content.sha256,
|
||||
mtime: Date.now(),
|
||||
};
|
||||
blobs.set(args.path, stored);
|
||||
const entry: FileEntry = {
|
||||
name: args.path.split('/').filter(Boolean).pop() ?? '',
|
||||
kind: 'file',
|
||||
size: args.content.size,
|
||||
mtime: stored.mtime,
|
||||
...(args.contentType !== undefined ? { contentType: args.contentType } : {}),
|
||||
metadata: { sha256: args.content.sha256 },
|
||||
};
|
||||
return { entry };
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await rig.teardown();
|
||||
});
|
||||
|
||||
test('write 1 KiB inline → read it back, sha256 matches', async () => {
|
||||
const data = new Uint8Array(1024);
|
||||
for (let i = 0; i < data.length; i++) data[i] = (i * 7) & 0xff;
|
||||
const expectedSha = bytesToHex(sha256(data));
|
||||
|
||||
const writeResult = await rig.fs.write('/small.bin', data, { contentType: 'application/octet-stream' });
|
||||
expect(writeResult.entry.size).toBe(1024);
|
||||
expect(writeResult.entry.metadata.sha256).toBe(expectedSha);
|
||||
|
||||
const readResult = await rig.fs.read('/small.bin');
|
||||
expect(readResult.kind).toBe('inline');
|
||||
if (readResult.kind === 'inline') {
|
||||
expect(readResult.bytes.byteLength).toBe(1024);
|
||||
expect(readResult.sha256).toBe(expectedSha);
|
||||
expect(readResult.contentType).toBe('application/octet-stream');
|
||||
expect(Array.from(readResult.bytes)).toEqual(Array.from(data));
|
||||
}
|
||||
});
|
||||
|
||||
test('write 100 KiB inline → read it back, sha256 matches', async () => {
|
||||
const data = new Uint8Array(100 * 1024);
|
||||
crypto.getRandomValues(data);
|
||||
const expectedSha = bytesToHex(sha256(data));
|
||||
|
||||
await rig.fs.write('/big.bin', data);
|
||||
const readResult = await rig.fs.read('/big.bin');
|
||||
expect(readResult.kind).toBe('inline');
|
||||
if (readResult.kind === 'inline') {
|
||||
expect(readResult.bytes.byteLength).toBe(100 * 1024);
|
||||
expect(readResult.sha256).toBe(expectedSha);
|
||||
expect(Array.from(readResult.bytes)).toEqual(Array.from(data));
|
||||
}
|
||||
});
|
||||
|
||||
test('overwrite without flag → server-defined error; with flag → succeeds', async () => {
|
||||
const a = new Uint8Array([1, 2, 3]);
|
||||
const b = new Uint8Array([4, 5, 6]);
|
||||
await rig.fs.write('/dup.bin', a);
|
||||
await expect(rig.fs.write('/dup.bin', b)).rejects.toThrow();
|
||||
await rig.fs.write('/dup.bin', b, { overwrite: true });
|
||||
const out = await rig.fs.read('/dup.bin');
|
||||
expect(out.kind).toBe('inline');
|
||||
if (out.kind === 'inline') {
|
||||
expect(Array.from(out.bytes)).toEqual([4, 5, 6]);
|
||||
}
|
||||
});
|
||||
|
||||
test('write Blob input → inferred contentType, round-trips', async () => {
|
||||
const blob = new Blob([new Uint8Array([0xde, 0xad, 0xbe, 0xef])], { type: 'image/png' });
|
||||
await rig.fs.write('/blobby.png', blob);
|
||||
const out = await rig.fs.read('/blobby.png');
|
||||
expect(out.kind).toBe('inline');
|
||||
if (out.kind === 'inline') {
|
||||
expect(out.contentType).toBe('image/png');
|
||||
expect(Array.from(out.bytes)).toEqual([0xde, 0xad, 0xbe, 0xef]);
|
||||
}
|
||||
});
|
||||
|
||||
test('read non-existent → NotFoundError', async () => {
|
||||
await expect(rig.fs.read('/missing.bin')).rejects.toThrow();
|
||||
});
|
||||
|
||||
test('inline path also handles 256 KiB exactly (boundary)', async () => {
|
||||
const data = new Uint8Array(256 * 1024);
|
||||
crypto.getRandomValues(data);
|
||||
const expectedSha = bytesToHex(sha256(data));
|
||||
await rig.fs.write('/boundary.bin', data);
|
||||
const out = await rig.fs.read('/boundary.bin');
|
||||
expect(out.kind).toBe('inline');
|
||||
if (out.kind === 'inline') {
|
||||
expect(out.sha256).toBe(expectedSha);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,175 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { sha256 } from '@noble/hashes/sha2.js';
|
||||
import { NotFoundError, type FileEntry } from '../../src/index.js';
|
||||
import { setupFileRig, type FileTestRig } from './helpers/rig.js';
|
||||
|
||||
interface StoredBlob {
|
||||
bytes: Uint8Array;
|
||||
contentType?: string;
|
||||
sha256: string;
|
||||
}
|
||||
|
||||
function bytesToHex(b: Uint8Array): string {
|
||||
return Array.from(b, (x) => x.toString(16).padStart(2, '0')).join('');
|
||||
}
|
||||
|
||||
async function streamToBytes(s: ReadableStream<Uint8Array>): Promise<Uint8Array> {
|
||||
const reader = s.getReader();
|
||||
const parts: Uint8Array[] = [];
|
||||
let total = 0;
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (value === undefined) continue;
|
||||
parts.push(value);
|
||||
total += value.byteLength;
|
||||
}
|
||||
reader.releaseLock();
|
||||
const out = new Uint8Array(total);
|
||||
let offset = 0;
|
||||
for (const p of parts) {
|
||||
out.set(p, offset);
|
||||
offset += p.byteLength;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function streamFromBytes(bytes: Uint8Array): ReadableStream<Uint8Array> {
|
||||
return new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
controller.enqueue(bytes);
|
||||
controller.close();
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
describe('Content I/O — streamed read/write E2E (>256 KiB)', () => {
|
||||
let rig: FileTestRig;
|
||||
const blobs = new Map<string, StoredBlob>();
|
||||
|
||||
beforeAll(async () => {
|
||||
blobs.clear();
|
||||
rig = await setupFileRig({
|
||||
read: async (ctx) => {
|
||||
const blob = blobs.get(ctx.path);
|
||||
if (blob === undefined) throw new NotFoundError(ctx.path);
|
||||
// Return as streams when blob ≥ 256 KiB.
|
||||
if (blob.bytes.byteLength > 256 * 1024) {
|
||||
return blob.contentType !== undefined
|
||||
? {
|
||||
kind: 'streams',
|
||||
stream: streamFromBytes(blob.bytes),
|
||||
size: blob.bytes.byteLength,
|
||||
sha256: blob.sha256,
|
||||
contentType: blob.contentType,
|
||||
}
|
||||
: {
|
||||
kind: 'streams',
|
||||
stream: streamFromBytes(blob.bytes),
|
||||
size: blob.bytes.byteLength,
|
||||
sha256: blob.sha256,
|
||||
};
|
||||
}
|
||||
return blob.contentType !== undefined
|
||||
? { kind: 'inline', bytes: blob.bytes, sha256: blob.sha256, contentType: blob.contentType }
|
||||
: { kind: 'inline', bytes: blob.bytes, sha256: blob.sha256 };
|
||||
},
|
||||
write: async (ctx) => {
|
||||
const args = ctx.args;
|
||||
let bytes: Uint8Array;
|
||||
let resolvedSha: string;
|
||||
if (args.content.kind === 'inline') {
|
||||
bytes = args.content.bytes;
|
||||
resolvedSha = args.content.sha256;
|
||||
} else {
|
||||
bytes = await streamToBytes(args.content.stream);
|
||||
resolvedSha = await args.content.sha256;
|
||||
if (bytes.byteLength !== args.content.size) {
|
||||
throw new Error(`stream produced ${bytes.byteLength} bytes; declared ${args.content.size}`);
|
||||
}
|
||||
}
|
||||
const stored: StoredBlob = {
|
||||
bytes,
|
||||
...(args.contentType !== undefined ? { contentType: args.contentType } : {}),
|
||||
sha256: resolvedSha,
|
||||
};
|
||||
blobs.set(args.path, stored);
|
||||
const entry: FileEntry = {
|
||||
name: args.path.split('/').filter(Boolean).pop() ?? '',
|
||||
kind: 'file',
|
||||
size: bytes.byteLength,
|
||||
mtime: Date.now(),
|
||||
...(args.contentType !== undefined ? { contentType: args.contentType } : {}),
|
||||
metadata: { sha256: resolvedSha },
|
||||
};
|
||||
return { entry };
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await rig.teardown();
|
||||
});
|
||||
|
||||
test('write 1 MiB streamed → server sees streams content + sha256 matches', async () => {
|
||||
const data = new Uint8Array(1024 * 1024);
|
||||
crypto.getRandomValues(data);
|
||||
const expectedSha = bytesToHex(sha256(data));
|
||||
|
||||
const writeResult = await rig.fs.write('/big1mb.bin', data);
|
||||
expect(writeResult.entry.size).toBe(data.byteLength);
|
||||
expect(writeResult.entry.metadata.sha256).toBe(expectedSha);
|
||||
});
|
||||
|
||||
test('read 1 MiB streamed → client gets streams output, drains correctly', async () => {
|
||||
const out = await rig.fs.read('/big1mb.bin');
|
||||
expect(out.kind).toBe('streams');
|
||||
if (out.kind === 'streams') {
|
||||
expect(out.size).toBe(1024 * 1024);
|
||||
const drained = await streamToBytes(out.stream);
|
||||
expect(drained.byteLength).toBe(1024 * 1024);
|
||||
expect(bytesToHex(sha256(drained))).toBe(out.sha256);
|
||||
await out.done();
|
||||
}
|
||||
});
|
||||
|
||||
test('boundary 256 KiB + 1 → streams write + read round-trip', async () => {
|
||||
const data = new Uint8Array(256 * 1024 + 1);
|
||||
for (let i = 0; i < data.length; i++) data[i] = (i * 31) & 0xff;
|
||||
const expectedSha = bytesToHex(sha256(data));
|
||||
|
||||
await rig.fs.write('/boundary-plus-one.bin', data);
|
||||
const out = await rig.fs.read('/boundary-plus-one.bin');
|
||||
expect(out.kind).toBe('streams');
|
||||
if (out.kind === 'streams') {
|
||||
expect(out.sha256).toBe(expectedSha);
|
||||
const drained = await streamToBytes(out.stream);
|
||||
expect(drained.byteLength).toBe(256 * 1024 + 1);
|
||||
expect(Array.from(drained.slice(0, 4))).toEqual(Array.from(data.slice(0, 4)));
|
||||
expect(Array.from(drained.slice(-4))).toEqual(Array.from(data.slice(-4)));
|
||||
await out.done();
|
||||
}
|
||||
});
|
||||
|
||||
test('write streams via { stream, size } wrapper → ok', async () => {
|
||||
const data = new Uint8Array(500 * 1024);
|
||||
crypto.getRandomValues(data);
|
||||
const expectedSha = bytesToHex(sha256(data));
|
||||
const wrapped = { stream: streamFromBytes(data), size: data.byteLength, contentType: 'application/octet-stream' };
|
||||
const result = await rig.fs.write('/wrapped.bin', wrapped);
|
||||
expect(result.entry.size).toBe(data.byteLength);
|
||||
expect(result.entry.metadata.sha256).toBe(expectedSha);
|
||||
expect(result.entry.contentType).toBe('application/octet-stream');
|
||||
});
|
||||
|
||||
test('write streams from Blob > 256 KiB → ok', async () => {
|
||||
const data = new Uint8Array(400 * 1024);
|
||||
crypto.getRandomValues(data);
|
||||
const expectedSha = bytesToHex(sha256(data));
|
||||
const blob = new Blob([data], { type: 'image/png' });
|
||||
const result = await rig.fs.write('/blobby-big.png', blob);
|
||||
expect(result.entry.size).toBe(data.byteLength);
|
||||
expect(result.entry.metadata.sha256).toBe(expectedSha);
|
||||
expect(result.entry.contentType).toBe('image/png');
|
||||
});
|
||||
});
|
||||
100
packages/shade-files/tests/integration/sdk-namespace.test.ts
Normal file
100
packages/shade-files/tests/integration/sdk-namespace.test.ts
Normal file
@@ -0,0 +1,100 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { createShade, type Shade } from '@shade/sdk';
|
||||
import { createPrekeyServer, MemoryPrekeyStore, PrekeyServerEvents } from '@shade/server';
|
||||
import { SubtleCryptoProvider } from '@shade/crypto-web';
|
||||
import type { FileEntry } from '../../src/index.js';
|
||||
|
||||
const crypto = new SubtleCryptoProvider();
|
||||
|
||||
/**
|
||||
* End-to-end test of `Shade.files` — the high-level SDK entrypoint.
|
||||
* Verifies that `shade.files.serve(...)` and `shade.files.client(peer)`
|
||||
* compose correctly and share a single channel + bridges per Shade.
|
||||
*/
|
||||
describe('Shade.files namespace — end-to-end via SDK getter', () => {
|
||||
let alice: Shade;
|
||||
let bob: Shade;
|
||||
let prekeyServer: { stop(): void };
|
||||
let aliceServer: { stop(): void };
|
||||
let bobServer: { stop(): void };
|
||||
let stopBobFiles: (() => Promise<void>) | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
const prekey = createPrekeyServer({
|
||||
crypto,
|
||||
store: new MemoryPrekeyStore(),
|
||||
disableRateLimit: true,
|
||||
events: new PrekeyServerEvents(),
|
||||
});
|
||||
prekeyServer = Bun.serve({ port: 0, fetch: prekey.fetch });
|
||||
const prekeyUrl = `http://localhost:${(prekeyServer as unknown as { port: number }).port}`;
|
||||
|
||||
alice = await createShade({ prekeyServer: prekeyUrl, address: 'alice' });
|
||||
bob = await createShade({ prekeyServer: prekeyUrl, address: 'bob' });
|
||||
|
||||
let aliceUrl = '';
|
||||
let bobUrl = '';
|
||||
alice.configureTransfers({
|
||||
resolveBaseUrl: async (peer) => {
|
||||
if (peer === 'bob') return bobUrl;
|
||||
throw new Error(`unknown peer: ${peer}`);
|
||||
},
|
||||
});
|
||||
bob.configureTransfers({
|
||||
resolveBaseUrl: async (peer) => {
|
||||
if (peer === 'alice') return aliceUrl;
|
||||
throw new Error(`unknown peer: ${peer}`);
|
||||
},
|
||||
});
|
||||
const aliceApp = await alice.transferRoute();
|
||||
const bobApp = await bob.transferRoute();
|
||||
aliceServer = Bun.serve({ port: 0, fetch: aliceApp.fetch });
|
||||
bobServer = Bun.serve({ port: 0, fetch: bobApp.fetch });
|
||||
aliceUrl = `http://localhost:${(aliceServer as unknown as { port: number }).port}`;
|
||||
bobUrl = `http://localhost:${(bobServer as unknown as { port: number }).port}`;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (stopBobFiles !== null) await stopBobFiles();
|
||||
await alice.files.destroy();
|
||||
await bob.files.destroy();
|
||||
await alice.shutdown();
|
||||
await bob.shutdown();
|
||||
aliceServer.stop();
|
||||
bobServer.stop();
|
||||
prekeyServer.stop();
|
||||
});
|
||||
|
||||
test('shade.files getter is memoized', () => {
|
||||
const a = bob.files;
|
||||
const b = bob.files;
|
||||
expect(a).toBe(b);
|
||||
});
|
||||
|
||||
test('serve() + client() round-trip stat through the SDK', async () => {
|
||||
stopBobFiles = await bob.files.serve({
|
||||
stat: async (ctx) => {
|
||||
const e: FileEntry = {
|
||||
name: ctx.path.split('/').filter(Boolean).pop() ?? '',
|
||||
kind: 'file',
|
||||
size: 42,
|
||||
mtime: 1234,
|
||||
metadata: {},
|
||||
};
|
||||
return e;
|
||||
},
|
||||
});
|
||||
|
||||
const fs = await alice.files.client('bob');
|
||||
const result = await fs.stat('/answer.txt');
|
||||
expect(result.name).toBe('answer.txt');
|
||||
expect(result.size).toBe(42);
|
||||
expect(result.mtime).toBe(1234);
|
||||
});
|
||||
|
||||
test('second serve() throws (one handler per Shade)', async () => {
|
||||
await expect(
|
||||
bob.files.serve({ stat: async () => ({ name: 'x', kind: 'file', size: 0, mtime: 0, metadata: {} }) }),
|
||||
).rejects.toThrow(/handler is already registered/);
|
||||
});
|
||||
});
|
||||
199
packages/shade-files/tests/integration/std-ops.test.ts
Normal file
199
packages/shade-files/tests/integration/std-ops.test.ts
Normal file
@@ -0,0 +1,199 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import {
|
||||
ConflictError,
|
||||
IdempotencyConflictError,
|
||||
NotFoundError,
|
||||
PermissionDeniedError,
|
||||
type FileEntry,
|
||||
} from '../../src/index.js';
|
||||
import { setupFileRig, type FileTestRig } from './helpers/rig.js';
|
||||
|
||||
describe('Standard ops — list/stat/mkdir/delete/move E2E', () => {
|
||||
let rig: FileTestRig;
|
||||
// Simple in-memory backing store on Bob.
|
||||
const tree = new Map<string, FileEntry>();
|
||||
|
||||
beforeAll(async () => {
|
||||
tree.clear();
|
||||
tree.set('/', { name: '', kind: 'dir', size: 0, mtime: 0, metadata: {} });
|
||||
tree.set('/foo', { name: 'foo', kind: 'dir', size: 0, mtime: 100, metadata: {} });
|
||||
tree.set('/foo/bar.txt', {
|
||||
name: 'bar.txt',
|
||||
kind: 'file',
|
||||
size: 12,
|
||||
mtime: 200,
|
||||
contentType: 'text/plain',
|
||||
metadata: {},
|
||||
});
|
||||
tree.set('/foo/baz.txt', {
|
||||
name: 'baz.txt',
|
||||
kind: 'file',
|
||||
size: 5,
|
||||
mtime: 300,
|
||||
metadata: {},
|
||||
});
|
||||
|
||||
rig = await setupFileRig({
|
||||
list: async (ctx) => {
|
||||
const dir = ctx.path;
|
||||
const entries: FileEntry[] = [];
|
||||
for (const [path, entry] of tree) {
|
||||
if (path === dir) continue;
|
||||
if (!path.startsWith(dir === '/' ? '/' : dir + '/')) continue;
|
||||
const rest = path.slice(dir === '/' ? 1 : dir.length + 1);
|
||||
if (rest.includes('/')) continue;
|
||||
entries.push(entry);
|
||||
}
|
||||
return { entries, hasMore: false };
|
||||
},
|
||||
stat: async (ctx) => {
|
||||
const e = tree.get(ctx.path);
|
||||
if (e === undefined) throw new NotFoundError(`${ctx.path} not found`);
|
||||
return e;
|
||||
},
|
||||
mkdir: async (ctx) => {
|
||||
if (tree.has(ctx.path)) throw new ConflictError(`${ctx.path} exists`);
|
||||
const name = ctx.path.split('/').filter(Boolean).pop() ?? '';
|
||||
const entry: FileEntry = {
|
||||
name,
|
||||
kind: 'dir',
|
||||
size: 0,
|
||||
mtime: Date.now(),
|
||||
metadata: {},
|
||||
};
|
||||
tree.set(ctx.path, entry);
|
||||
return { entry };
|
||||
},
|
||||
delete: async (ctx) => {
|
||||
if (!tree.has(ctx.path)) throw new NotFoundError(ctx.path);
|
||||
let count = 0;
|
||||
if (ctx.args.recursive) {
|
||||
for (const path of [...tree.keys()]) {
|
||||
if (path === ctx.path || path.startsWith(ctx.path + '/')) {
|
||||
tree.delete(path);
|
||||
count++;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
tree.delete(ctx.path);
|
||||
count = 1;
|
||||
}
|
||||
return { deletedCount: count };
|
||||
},
|
||||
move: async (ctx) => {
|
||||
const src = ctx.args.src;
|
||||
const dst = ctx.args.dst;
|
||||
const e = tree.get(src);
|
||||
if (e === undefined) throw new NotFoundError(src);
|
||||
if (tree.has(dst) && !ctx.args.overwrite) {
|
||||
throw new ConflictError(`${dst} exists`);
|
||||
}
|
||||
const newName = dst.split('/').filter(Boolean).pop() ?? e.name;
|
||||
tree.delete(src);
|
||||
tree.set(dst, { ...e, name: newName });
|
||||
return { entry: tree.get(dst)! };
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await rig.teardown();
|
||||
});
|
||||
|
||||
test('list /', async () => {
|
||||
const page = await rig.fs.list('/');
|
||||
expect(page.hasMore).toBe(false);
|
||||
const names = page.entries.map((e) => e.name).sort();
|
||||
expect(names).toEqual(['foo']);
|
||||
});
|
||||
|
||||
test('list /foo', async () => {
|
||||
const page = await rig.fs.list('/foo');
|
||||
const names = page.entries.map((e) => e.name).sort();
|
||||
expect(names).toEqual(['bar.txt', 'baz.txt']);
|
||||
});
|
||||
|
||||
test('stat existing file', async () => {
|
||||
const e = await rig.fs.stat('/foo/bar.txt');
|
||||
expect(e.size).toBe(12);
|
||||
expect(e.contentType).toBe('text/plain');
|
||||
});
|
||||
|
||||
test('stat missing → NotFoundError', async () => {
|
||||
let caught: unknown = null;
|
||||
try {
|
||||
await rig.fs.stat('/no/such/file');
|
||||
} catch (err) {
|
||||
caught = err;
|
||||
}
|
||||
expect(caught instanceof NotFoundError).toBe(true);
|
||||
expect((caught as NotFoundError).payload.code).toBe('NOT_FOUND');
|
||||
});
|
||||
|
||||
test('mkdir creates a new directory', async () => {
|
||||
const result = await rig.fs.mkdir('/created');
|
||||
expect(result.entry.kind).toBe('dir');
|
||||
expect(tree.has('/created')).toBe(true);
|
||||
});
|
||||
|
||||
test('mkdir on existing path → ConflictError', async () => {
|
||||
await expect(rig.fs.mkdir('/foo')).rejects.toBeInstanceOf(ConflictError);
|
||||
});
|
||||
|
||||
test('delete file', async () => {
|
||||
tree.set('/temp.txt', { name: 'temp.txt', kind: 'file', size: 0, mtime: 0, metadata: {} });
|
||||
const r = await rig.fs.delete('/temp.txt');
|
||||
expect(r.deletedCount).toBe(1);
|
||||
expect(tree.has('/temp.txt')).toBe(false);
|
||||
});
|
||||
|
||||
test('delete missing → NotFoundError', async () => {
|
||||
await expect(rig.fs.delete('/doesnt-exist')).rejects.toBeInstanceOf(NotFoundError);
|
||||
});
|
||||
|
||||
test('move file', async () => {
|
||||
tree.set('/x.txt', { name: 'x.txt', kind: 'file', size: 0, mtime: 0, metadata: {} });
|
||||
const r = await rig.fs.move('/x.txt', '/y.txt');
|
||||
expect(r.entry.name).toBe('y.txt');
|
||||
expect(tree.has('/x.txt')).toBe(false);
|
||||
expect(tree.has('/y.txt')).toBe(true);
|
||||
});
|
||||
|
||||
test('move overwrite=false → ConflictError on collision', async () => {
|
||||
tree.set('/a.txt', { name: 'a.txt', kind: 'file', size: 0, mtime: 0, metadata: {} });
|
||||
tree.set('/b.txt', { name: 'b.txt', kind: 'file', size: 0, mtime: 0, metadata: {} });
|
||||
await expect(rig.fs.move('/a.txt', '/b.txt')).rejects.toBeInstanceOf(ConflictError);
|
||||
});
|
||||
|
||||
test('idempotent retry: same key + same args → cached response', async () => {
|
||||
tree.set('/idem.txt', { name: 'idem.txt', kind: 'file', size: 0, mtime: 0, metadata: {} });
|
||||
const key = 'IdemKey1AaBbCcDdEeFfGg'; // 22 chars
|
||||
const r1 = await rig.fs.delete('/idem.txt', { idempotencyKey: key });
|
||||
expect(r1.deletedCount).toBe(1);
|
||||
// 2nd call with same key → same result without throwing NotFound
|
||||
const r2 = await rig.fs.delete('/idem.txt', { idempotencyKey: key });
|
||||
expect(r2.deletedCount).toBe(1);
|
||||
});
|
||||
|
||||
test('idempotency conflict: same key + different args', async () => {
|
||||
tree.set('/c1.txt', { name: 'c1.txt', kind: 'file', size: 0, mtime: 0, metadata: {} });
|
||||
tree.set('/c2.txt', { name: 'c2.txt', kind: 'file', size: 0, mtime: 0, metadata: {} });
|
||||
const key = 'ConflictKeyAaBbCcDdEeF'; // 22 chars
|
||||
await rig.fs.delete('/c1.txt', { idempotencyKey: key });
|
||||
await expect(
|
||||
rig.fs.delete('/c2.txt', { idempotencyKey: key }),
|
||||
).rejects.toBeInstanceOf(IdempotencyConflictError);
|
||||
});
|
||||
|
||||
test('path validation: traversal rejected by server', async () => {
|
||||
await expect(rig.fs.list('/foo')).resolves.toBeDefined();
|
||||
// Client-side schema rejects traversal too — bypass via direct bad path:
|
||||
await expect(rig.fs.stat('/foo/../etc')).rejects.toThrow();
|
||||
});
|
||||
|
||||
// We don't test PermissionDenied here (no beforeOp gate configured),
|
||||
// but the type is referenced to ensure it's exported properly.
|
||||
test('PermissionDeniedError is exported', () => {
|
||||
expect(PermissionDeniedError).toBeDefined();
|
||||
});
|
||||
});
|
||||
123
packages/shade-files/tests/integration/thumbnail.test.ts
Normal file
123
packages/shade-files/tests/integration/thumbnail.test.ts
Normal file
@@ -0,0 +1,123 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { sha256 } from '@noble/hashes/sha2.js';
|
||||
import { ConflictError, NotFoundError } from '../../src/index.js';
|
||||
import { setupFileRig, type FileTestRig } from './helpers/rig.js';
|
||||
|
||||
function bytesToHex(b: Uint8Array): string {
|
||||
return Array.from(b, (x) => x.toString(16).padStart(2, '0')).join('');
|
||||
}
|
||||
|
||||
// Minimal valid PNG: 8-byte signature + IHDR + zero-byte IDAT + IEND. We
|
||||
// use realistic magic bytes so the format-hardening check accepts it.
|
||||
function tinyPng(): Uint8Array {
|
||||
const sig = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
|
||||
const filler = new Uint8Array(20);
|
||||
const out = new Uint8Array(sig.length + filler.length);
|
||||
out.set(sig, 0);
|
||||
out.set(filler, sig.length);
|
||||
return out;
|
||||
}
|
||||
|
||||
function tinyJpeg(): Uint8Array {
|
||||
const head = new Uint8Array([0xff, 0xd8, 0xff, 0xe0]);
|
||||
const filler = new Uint8Array(12);
|
||||
const out = new Uint8Array(head.length + filler.length);
|
||||
out.set(head, 0);
|
||||
out.set(filler, head.length);
|
||||
return out;
|
||||
}
|
||||
|
||||
function tinyWebp(): Uint8Array {
|
||||
// 'RIFF' + sizeLE + 'WEBP' + 'VP8 ' (or 'VP8L' / 'VP8X')
|
||||
const out = new Uint8Array(20);
|
||||
out.set([0x52, 0x49, 0x46, 0x46], 0); // 'RIFF'
|
||||
out.set([0x0c, 0x00, 0x00, 0x00], 4); // size little-endian (12 bytes follow)
|
||||
out.set([0x57, 0x45, 0x42, 0x50], 8); // 'WEBP'
|
||||
out.set([0x56, 0x50, 0x38, 0x20], 12); // 'VP8 '
|
||||
return out;
|
||||
}
|
||||
|
||||
describe('Content I/O — getThumbnail E2E with format hardening', () => {
|
||||
let rig: FileTestRig;
|
||||
// Map: path → (size → { bytes, format })
|
||||
const thumbnails = new Map<string, Map<number, { bytes: Uint8Array; format: 'png' | 'webp' | 'jpeg' }>>();
|
||||
|
||||
beforeAll(async () => {
|
||||
thumbnails.clear();
|
||||
thumbnails.set(
|
||||
'/photo.png',
|
||||
new Map([
|
||||
[64, { bytes: tinyPng(), format: 'png' }],
|
||||
[128, { bytes: tinyPng(), format: 'png' }],
|
||||
]),
|
||||
);
|
||||
thumbnails.set(
|
||||
'/holiday.jpg',
|
||||
new Map([[256, { bytes: tinyJpeg(), format: 'jpeg' }]]),
|
||||
);
|
||||
thumbnails.set(
|
||||
'/icon.webp',
|
||||
new Map([[64, { bytes: tinyWebp(), format: 'webp' }]]),
|
||||
);
|
||||
// Mismatched format: server returns PNG bytes but claims JPEG.
|
||||
thumbnails.set('/lying.png', new Map([[64, { bytes: tinyPng(), format: 'jpeg' }]]));
|
||||
|
||||
rig = await setupFileRig({
|
||||
getThumbnail: async (ctx) => {
|
||||
const sizes = thumbnails.get(ctx.path);
|
||||
if (sizes === undefined) throw new NotFoundError(ctx.path);
|
||||
const entry = sizes.get(ctx.args.size);
|
||||
if (entry === undefined) throw new NotFoundError(`${ctx.path}@${ctx.args.size}`);
|
||||
return {
|
||||
bytes: entry.bytes,
|
||||
format: entry.format,
|
||||
width: ctx.args.size,
|
||||
height: ctx.args.size,
|
||||
};
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await rig.teardown();
|
||||
});
|
||||
|
||||
test('PNG thumbnail round-trips with computed sha256', async () => {
|
||||
const result = await rig.fs.getThumbnail('/photo.png', 64);
|
||||
expect(result.format).toBe('png');
|
||||
expect(result.width).toBe(64);
|
||||
expect(result.height).toBe(64);
|
||||
expect(result.sha256).toBe(bytesToHex(sha256(result.bytes)));
|
||||
// Verify magic bytes survived
|
||||
expect(Array.from(result.bytes.slice(0, 8))).toEqual([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
|
||||
});
|
||||
|
||||
test('JPEG thumbnail round-trips', async () => {
|
||||
const result = await rig.fs.getThumbnail('/holiday.jpg', 256);
|
||||
expect(result.format).toBe('jpeg');
|
||||
expect(result.width).toBe(256);
|
||||
expect(Array.from(result.bytes.slice(0, 3))).toEqual([0xff, 0xd8, 0xff]);
|
||||
});
|
||||
|
||||
test('WebP thumbnail round-trips', async () => {
|
||||
const result = await rig.fs.getThumbnail('/icon.webp', 64);
|
||||
expect(result.format).toBe('webp');
|
||||
expect(Array.from(result.bytes.slice(0, 4))).toEqual([0x52, 0x49, 0x46, 0x46]);
|
||||
expect(Array.from(result.bytes.slice(8, 12))).toEqual([0x57, 0x45, 0x42, 0x50]);
|
||||
});
|
||||
|
||||
test('PNG bytes claimed as JPEG → format-hardening rejects', async () => {
|
||||
await expect(rig.fs.getThumbnail('/lying.png', 64)).rejects.toBeInstanceOf(ConflictError);
|
||||
});
|
||||
|
||||
test('non-existent path → NotFoundError', async () => {
|
||||
await expect(rig.fs.getThumbnail('/missing.png', 64)).rejects.toBeInstanceOf(NotFoundError);
|
||||
});
|
||||
|
||||
test('different sizes return different thumbnails', async () => {
|
||||
const small = await rig.fs.getThumbnail('/photo.png', 64);
|
||||
const big = await rig.fs.getThumbnail('/photo.png', 128);
|
||||
expect(small.width).toBe(64);
|
||||
expect(big.width).toBe(128);
|
||||
});
|
||||
});
|
||||
238
packages/shade-files/tests/integration/upload-directory.test.ts
Normal file
238
packages/shade-files/tests/integration/upload-directory.test.ts
Normal file
@@ -0,0 +1,238 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { sha256 } from '@noble/hashes/sha2.js';
|
||||
import {
|
||||
ConflictError,
|
||||
NotFoundError,
|
||||
uploadDirectory,
|
||||
createMemoryDirectory,
|
||||
type FileEntry,
|
||||
type BulkTransferEvent,
|
||||
} from '../../src/index.js';
|
||||
import { setupFileRig, type FileTestRig } from './helpers/rig.js';
|
||||
|
||||
interface StoredFile {
|
||||
bytes: Uint8Array;
|
||||
contentType?: string;
|
||||
sha256: string;
|
||||
}
|
||||
|
||||
function bytesToHex(b: Uint8Array): string {
|
||||
return Array.from(b, (x) => x.toString(16).padStart(2, '0')).join('');
|
||||
}
|
||||
|
||||
async function streamToBytes(s: ReadableStream<Uint8Array>): Promise<Uint8Array> {
|
||||
const reader = s.getReader();
|
||||
const parts: Uint8Array[] = [];
|
||||
let total = 0;
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (value === undefined) continue;
|
||||
parts.push(value);
|
||||
total += value.byteLength;
|
||||
}
|
||||
reader.releaseLock();
|
||||
const out = new Uint8Array(total);
|
||||
let offset = 0;
|
||||
for (const p of parts) {
|
||||
out.set(p, offset);
|
||||
offset += p.byteLength;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
describe('uploadDirectory — bulk upload to remote', () => {
|
||||
let rig: FileTestRig;
|
||||
const blobs = new Map<string, StoredFile>();
|
||||
const dirs = new Set<string>(['/']);
|
||||
|
||||
beforeAll(async () => {
|
||||
blobs.clear();
|
||||
dirs.clear();
|
||||
dirs.add('/');
|
||||
|
||||
rig = await setupFileRig({
|
||||
mkdir: async (ctx) => {
|
||||
const path = ctx.path;
|
||||
if (dirs.has(path)) {
|
||||
if (!ctx.args.recursive) throw new ConflictError('exists');
|
||||
// Idempotent for recursive
|
||||
}
|
||||
// Recursive: add ancestors
|
||||
if (ctx.args.recursive) {
|
||||
const segments = path.split('/').filter(Boolean);
|
||||
let acc = '';
|
||||
for (const seg of segments) {
|
||||
acc += '/' + seg;
|
||||
dirs.add(acc);
|
||||
}
|
||||
} else {
|
||||
dirs.add(path);
|
||||
}
|
||||
return {
|
||||
entry: {
|
||||
name: path.split('/').filter(Boolean).pop() ?? '',
|
||||
kind: 'dir',
|
||||
size: 0,
|
||||
mtime: Date.now(),
|
||||
metadata: {},
|
||||
},
|
||||
};
|
||||
},
|
||||
write: async (ctx) => {
|
||||
const args = ctx.args;
|
||||
let bytes: Uint8Array;
|
||||
let storedSha: string;
|
||||
if (args.content.kind === 'inline') {
|
||||
bytes = args.content.bytes;
|
||||
storedSha = args.content.sha256;
|
||||
} else {
|
||||
bytes = await streamToBytes(args.content.stream);
|
||||
storedSha = await args.content.sha256;
|
||||
}
|
||||
if (blobs.has(args.path) && !args.overwrite) {
|
||||
throw new ConflictError(`${args.path} exists`);
|
||||
}
|
||||
blobs.set(args.path, {
|
||||
bytes,
|
||||
...(args.contentType !== undefined ? { contentType: args.contentType } : {}),
|
||||
sha256: storedSha,
|
||||
});
|
||||
const entry: FileEntry = {
|
||||
name: args.path.split('/').filter(Boolean).pop() ?? '',
|
||||
kind: 'file',
|
||||
size: bytes.byteLength,
|
||||
mtime: Date.now(),
|
||||
...(args.contentType !== undefined ? { contentType: args.contentType } : {}),
|
||||
metadata: { sha256: storedSha },
|
||||
};
|
||||
return { entry };
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await rig.teardown();
|
||||
});
|
||||
|
||||
test('uploads a small tree with mixed inline + streams', async () => {
|
||||
const local = createMemoryDirectory('local');
|
||||
const sub = local.addDir('sub');
|
||||
const small = new Uint8Array(100); for (let i = 0; i < small.length; i++) small[i] = i & 0xff;
|
||||
const big = new Uint8Array(400 * 1024); crypto.getRandomValues(big);
|
||||
local.addFile('hello.txt', new TextEncoder().encode('hello world'), 'text/plain');
|
||||
local.addFile('small.bin', small);
|
||||
sub.addFile('big.bin', big, 'application/octet-stream');
|
||||
|
||||
const handle = uploadDirectory(rig.fs, local, '/upload-target');
|
||||
|
||||
const events: BulkTransferEvent[] = [];
|
||||
(async () => {
|
||||
for await (const ev of handle.events) events.push(ev);
|
||||
})().catch(() => undefined);
|
||||
|
||||
const result = await handle.done();
|
||||
expect(result.filesDone).toBe(3);
|
||||
expect(result.bytesDone).toBe(11 + 100 + 400 * 1024);
|
||||
|
||||
// Verify remote tree
|
||||
expect(blobs.has('/upload-target/hello.txt')).toBe(true);
|
||||
expect(blobs.has('/upload-target/small.bin')).toBe(true);
|
||||
expect(blobs.has('/upload-target/sub/big.bin')).toBe(true);
|
||||
expect(dirs.has('/upload-target/sub')).toBe(true);
|
||||
|
||||
// Sha256 paritet for streamed file
|
||||
expect(blobs.get('/upload-target/sub/big.bin')!.sha256).toBe(bytesToHex(sha256(big)));
|
||||
expect(blobs.get('/upload-target/hello.txt')!.contentType).toBe('text/plain');
|
||||
|
||||
// Wait a tick for events to flush
|
||||
await new Promise((r) => setTimeout(r, 30));
|
||||
const planEvent = events.find((e) => e.type === 'plan');
|
||||
expect(planEvent).toBeDefined();
|
||||
if (planEvent && planEvent.type === 'plan') {
|
||||
expect(planEvent.totalFiles).toBe(3);
|
||||
expect(planEvent.totalBytes).toBe(11 + 100 + 400 * 1024);
|
||||
}
|
||||
const completes = events.filter((e) => e.type === 'complete');
|
||||
expect(completes.length).toBe(1);
|
||||
});
|
||||
|
||||
test('aggregated progress is monotonically non-decreasing', async () => {
|
||||
const local = createMemoryDirectory('local');
|
||||
for (let i = 0; i < 10; i++) {
|
||||
local.addFile(`f${i}.bin`, new Uint8Array(50));
|
||||
}
|
||||
const handle = uploadDirectory(rig.fs, local, '/progress');
|
||||
const progresses: { filesDone: number; bytesDone: number }[] = [];
|
||||
(async () => {
|
||||
for await (const ev of handle.events) {
|
||||
if (ev.type === 'progress') {
|
||||
progresses.push({ filesDone: ev.filesDone, bytesDone: ev.bytesDone });
|
||||
}
|
||||
}
|
||||
})().catch(() => undefined);
|
||||
await handle.done();
|
||||
await new Promise((r) => setTimeout(r, 30));
|
||||
for (let i = 1; i < progresses.length; i++) {
|
||||
expect(progresses[i]!.filesDone).toBeGreaterThanOrEqual(progresses[i - 1]!.filesDone);
|
||||
expect(progresses[i]!.bytesDone).toBeGreaterThanOrEqual(progresses[i - 1]!.bytesDone);
|
||||
}
|
||||
expect(progresses[progresses.length - 1]!.filesDone).toBe(10);
|
||||
});
|
||||
|
||||
test('fail-fast: first error aborts the bulk', async () => {
|
||||
const local = createMemoryDirectory('local');
|
||||
for (let i = 0; i < 5; i++) {
|
||||
local.addFile(`x${i}.bin`, new Uint8Array(10));
|
||||
}
|
||||
// Pre-create a conflicting file at /conflict/x0.bin so the first write fails.
|
||||
blobs.set('/conflict/x0.bin', { bytes: new Uint8Array(0), sha256: 'x' });
|
||||
|
||||
const handle = uploadDirectory(rig.fs, local, '/conflict', { concurrency: 1 });
|
||||
await expect(handle.done()).rejects.toThrow();
|
||||
});
|
||||
|
||||
test('continueOnError: completes despite per-file errors', async () => {
|
||||
const local = createMemoryDirectory('local');
|
||||
for (let i = 0; i < 5; i++) {
|
||||
local.addFile(`y${i}.bin`, new Uint8Array(10));
|
||||
}
|
||||
blobs.set('/cont/y2.bin', { bytes: new Uint8Array(0), sha256: 'x' });
|
||||
|
||||
const handle = uploadDirectory(rig.fs, local, '/cont', {
|
||||
concurrency: 1,
|
||||
continueOnError: true,
|
||||
});
|
||||
const errors: string[] = [];
|
||||
(async () => {
|
||||
for await (const ev of handle.events) {
|
||||
if (ev.type === 'file-error') errors.push(ev.path);
|
||||
}
|
||||
})().catch(() => undefined);
|
||||
const result = await handle.done();
|
||||
await new Promise((r) => setTimeout(r, 30));
|
||||
expect(errors).toEqual(['y2.bin']);
|
||||
expect(result.filesDone).toBe(4);
|
||||
});
|
||||
|
||||
test('concurrency cap respected', async () => {
|
||||
const local = createMemoryDirectory('local');
|
||||
for (let i = 0; i < 30; i++) {
|
||||
local.addFile(`z${i}.bin`, new Uint8Array(10));
|
||||
}
|
||||
// Concurrency above MAX (16) should be clamped.
|
||||
const handle = uploadDirectory(rig.fs, local, '/cap', { concurrency: 100 });
|
||||
const result = await handle.done();
|
||||
expect(result.filesDone).toBe(30);
|
||||
});
|
||||
|
||||
test('aborts mid-flight via handle.abort()', async () => {
|
||||
const local = createMemoryDirectory('local');
|
||||
for (let i = 0; i < 50; i++) {
|
||||
local.addFile(`q${i}.bin`, new Uint8Array(50 * 1024)); // 50 KiB each
|
||||
}
|
||||
const handle = uploadDirectory(rig.fs, local, '/abort');
|
||||
setTimeout(() => void handle.abort('test-cancel'), 20);
|
||||
await expect(handle.done()).rejects.toThrow();
|
||||
});
|
||||
});
|
||||
126
packages/shade-files/tests/integration/walk.test.ts
Normal file
126
packages/shade-files/tests/integration/walk.test.ts
Normal file
@@ -0,0 +1,126 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { walk, type FileEntry, NotFoundError } from '../../src/index.js';
|
||||
import { setupFileRig, type FileTestRig } from './helpers/rig.js';
|
||||
|
||||
describe('walk — async-iterable depth-first directory traversal', () => {
|
||||
let rig: FileTestRig;
|
||||
// In-memory tree:
|
||||
// /
|
||||
// ├── a/
|
||||
// │ ├── 1.txt
|
||||
// │ └── b/
|
||||
// │ └── 2.txt
|
||||
// ├── c/
|
||||
// │ └── 3.txt
|
||||
// └── 4.txt
|
||||
const tree = new Map<string, FileEntry>();
|
||||
|
||||
function setEntry(path: string, kind: 'file' | 'dir'): void {
|
||||
const name = path === '/' ? '' : path.split('/').filter(Boolean).pop() ?? '';
|
||||
tree.set(path, { name, kind, size: kind === 'file' ? 10 : 0, mtime: 0, metadata: {} });
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
tree.clear();
|
||||
setEntry('/', 'dir');
|
||||
setEntry('/a', 'dir');
|
||||
setEntry('/a/1.txt', 'file');
|
||||
setEntry('/a/b', 'dir');
|
||||
setEntry('/a/b/2.txt', 'file');
|
||||
setEntry('/c', 'dir');
|
||||
setEntry('/c/3.txt', 'file');
|
||||
setEntry('/4.txt', 'file');
|
||||
|
||||
rig = await setupFileRig({
|
||||
list: async (ctx) => {
|
||||
if (!tree.has(ctx.path)) throw new NotFoundError(ctx.path);
|
||||
const entries: FileEntry[] = [];
|
||||
for (const [path, entry] of tree) {
|
||||
if (path === ctx.path) continue;
|
||||
if (!path.startsWith(ctx.path === '/' ? '/' : ctx.path + '/')) continue;
|
||||
const rest = path.slice(ctx.path === '/' ? 1 : ctx.path.length + 1);
|
||||
if (rest.includes('/')) continue;
|
||||
entries.push(entry);
|
||||
}
|
||||
return { entries, hasMore: false };
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await rig.teardown();
|
||||
});
|
||||
|
||||
test('walks the entire tree depth-first', async () => {
|
||||
const items: { path: string; depth: number }[] = [];
|
||||
for await (const item of walk(rig.fs, '/')) {
|
||||
items.push({ path: item.relativePath, depth: item.depth });
|
||||
}
|
||||
// Depth-first order: visit a, then descend into a/* (1.txt + a/b → a/b/2.txt), then c, c/3.txt, then 4.txt
|
||||
expect(items.map((i) => i.path)).toEqual([
|
||||
'a',
|
||||
'a/1.txt',
|
||||
'a/b',
|
||||
'a/b/2.txt',
|
||||
'c',
|
||||
'c/3.txt',
|
||||
'4.txt',
|
||||
]);
|
||||
// Depth values
|
||||
expect(items.find((i) => i.path === 'a')?.depth).toBe(1);
|
||||
expect(items.find((i) => i.path === 'a/1.txt')?.depth).toBe(2);
|
||||
expect(items.find((i) => i.path === 'a/b/2.txt')?.depth).toBe(3);
|
||||
});
|
||||
|
||||
test('respects maxDepth', async () => {
|
||||
const items: string[] = [];
|
||||
for await (const item of walk(rig.fs, '/', { maxDepth: 1 })) {
|
||||
items.push(item.relativePath);
|
||||
}
|
||||
// Only direct children — no descent into /a or /c.
|
||||
expect(items.sort()).toEqual(['4.txt', 'a', 'c']);
|
||||
});
|
||||
|
||||
test('breaks cleanly when consumer stops iterating', async () => {
|
||||
const items: string[] = [];
|
||||
for await (const item of walk(rig.fs, '/')) {
|
||||
items.push(item.relativePath);
|
||||
if (items.length === 2) break;
|
||||
}
|
||||
expect(items).toEqual(['a', 'a/1.txt']);
|
||||
});
|
||||
|
||||
test('filter callback skips entries (and excludes their subtree)', async () => {
|
||||
const items: string[] = [];
|
||||
for await (const item of walk(rig.fs, '/', {
|
||||
filter: (entry, _rel) => entry.name !== 'a',
|
||||
})) {
|
||||
items.push(item.relativePath);
|
||||
}
|
||||
expect(items.sort()).toEqual(['4.txt', 'c', 'c/3.txt']);
|
||||
});
|
||||
|
||||
test('aborts via signal mid-walk', async () => {
|
||||
const ctrl = new AbortController();
|
||||
const items: string[] = [];
|
||||
setTimeout(() => ctrl.abort(), 5);
|
||||
let threw = false;
|
||||
try {
|
||||
for await (const item of walk(rig.fs, '/', { signal: ctrl.signal })) {
|
||||
items.push(item.relativePath);
|
||||
await new Promise((r) => setTimeout(r, 10));
|
||||
}
|
||||
} catch {
|
||||
threw = true;
|
||||
}
|
||||
expect(threw).toBe(true);
|
||||
});
|
||||
|
||||
test('walking non-existent path → throws on first list', async () => {
|
||||
await expect(async () => {
|
||||
for await (const _item of walk(rig.fs, '/nope')) {
|
||||
/* unreachable */
|
||||
}
|
||||
}).toThrow();
|
||||
});
|
||||
});
|
||||
86
packages/shade-files/tests/security/fingerprint-gate.test.ts
Normal file
86
packages/shade-files/tests/security/fingerprint-gate.test.ts
Normal file
@@ -0,0 +1,86 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { setupFileRig, type FileTestRig } from '../integration/helpers/rig.js';
|
||||
import { FingerprintRequiredError, NotFoundError, type FileEntry } from '../../src/index.js';
|
||||
|
||||
describe('Fingerprint gate', () => {
|
||||
let rig: FileTestRig;
|
||||
const verifiedSet = new Set<string>();
|
||||
|
||||
beforeAll(async () => {
|
||||
verifiedSet.clear();
|
||||
rig = await setupFileRig({
|
||||
requireFingerprintVerifiedFor: (ctx) => {
|
||||
// Mutations require verification; reads are optional.
|
||||
const op = ctx.op;
|
||||
if (op === 'mkdir' || op === 'delete' || op === 'move' || op === 'write') return 'required';
|
||||
if (op === 'list') return 'optional';
|
||||
return 'optional';
|
||||
},
|
||||
isFingerprintVerified: (sender) => verifiedSet.has(sender),
|
||||
stat: async (ctx) => {
|
||||
const e: FileEntry = {
|
||||
name: ctx.path.split('/').filter(Boolean).pop() ?? '',
|
||||
kind: 'file', size: 1, mtime: 0, metadata: {},
|
||||
};
|
||||
return e;
|
||||
},
|
||||
mkdir: async (ctx) => {
|
||||
const e: FileEntry = {
|
||||
name: ctx.path.split('/').filter(Boolean).pop() ?? '',
|
||||
kind: 'dir', size: 0, mtime: 0, metadata: {},
|
||||
};
|
||||
return { entry: e };
|
||||
},
|
||||
delete: async () => ({ deletedCount: 1 }),
|
||||
list: async () => ({ entries: [], hasMore: false }),
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await rig.teardown();
|
||||
});
|
||||
|
||||
test('mutation without verification → FingerprintRequiredError', async () => {
|
||||
verifiedSet.delete('alice');
|
||||
await expect(rig.fs.mkdir('/locked')).rejects.toBeInstanceOf(FingerprintRequiredError);
|
||||
await expect(rig.fs.delete('/locked')).rejects.toBeInstanceOf(FingerprintRequiredError);
|
||||
});
|
||||
|
||||
test('mutation after marking peer verified → succeeds', async () => {
|
||||
verifiedSet.add('alice');
|
||||
const result = await rig.fs.mkdir('/verified-dir');
|
||||
expect(result.entry.name).toBe('verified-dir');
|
||||
});
|
||||
|
||||
test('optional ops (stat, list) work without verification', async () => {
|
||||
verifiedSet.delete('alice');
|
||||
const stat = await rig.fs.stat('/anything');
|
||||
expect(stat.name).toBe('anything');
|
||||
const list = await rig.fs.list('/anything');
|
||||
expect(list.entries).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Fingerprint gate with reject policy', () => {
|
||||
let rig: FileTestRig;
|
||||
beforeAll(async () => {
|
||||
rig = await setupFileRig({
|
||||
requireFingerprintVerifiedFor: () => 'reject',
|
||||
stat: async (ctx) => {
|
||||
const e: FileEntry = {
|
||||
name: 'x',
|
||||
kind: 'file',
|
||||
size: 1,
|
||||
mtime: 0,
|
||||
metadata: {},
|
||||
};
|
||||
return e;
|
||||
},
|
||||
});
|
||||
});
|
||||
afterAll(async () => { await rig.teardown(); });
|
||||
|
||||
test('all ops rejected outright', async () => {
|
||||
await expect(rig.fs.stat('/x')).rejects.toBeInstanceOf(FingerprintRequiredError);
|
||||
});
|
||||
});
|
||||
55
packages/shade-files/tests/security/quota.test.ts
Normal file
55
packages/shade-files/tests/security/quota.test.ts
Normal file
@@ -0,0 +1,55 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { setupFileRig, type FileTestRig } from '../integration/helpers/rig.js';
|
||||
import { FsRateLimitError, QuotaExceededError, type FileEntry } from '../../src/index.js';
|
||||
|
||||
describe('Op rate limit', () => {
|
||||
let rig: FileTestRig;
|
||||
let listCount = 0;
|
||||
beforeAll(async () => {
|
||||
listCount = 0;
|
||||
rig = await setupFileRig({
|
||||
rateLimits: { maxOpsPerMinutePerSender: 5 },
|
||||
list: async () => {
|
||||
listCount++;
|
||||
return { entries: [], hasMore: false };
|
||||
},
|
||||
});
|
||||
});
|
||||
afterAll(async () => { await rig.teardown(); });
|
||||
|
||||
test('op rate-limit kicks in after capacity', async () => {
|
||||
listCount = 0;
|
||||
for (let i = 0; i < 5; i++) await rig.fs.list('/');
|
||||
expect(listCount).toBe(5);
|
||||
await expect(rig.fs.list('/')).rejects.toBeInstanceOf(FsRateLimitError);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Byte quota', () => {
|
||||
let rig: FileTestRig;
|
||||
beforeAll(async () => {
|
||||
rig = await setupFileRig({
|
||||
rateLimits: {
|
||||
// Plenty of ops, but tight byte cap for the quota test.
|
||||
maxOpsPerMinutePerSender: 100,
|
||||
maxBytesPerHourPerSender: 1024,
|
||||
},
|
||||
write: async (ctx) => {
|
||||
const e: FileEntry = {
|
||||
name: ctx.args.path.split('/').filter(Boolean).pop() ?? '',
|
||||
kind: 'file',
|
||||
size: ctx.args.content.kind === 'inline' ? ctx.args.content.bytes.byteLength : ctx.args.content.size,
|
||||
mtime: 0,
|
||||
metadata: {},
|
||||
};
|
||||
return { entry: e };
|
||||
},
|
||||
});
|
||||
});
|
||||
afterAll(async () => { await rig.teardown(); });
|
||||
|
||||
test('write 2 KiB inline → exceeds 1 KiB/hour cap', async () => {
|
||||
const big = new Uint8Array(2048);
|
||||
await expect(rig.fs.write('/big.bin', big)).rejects.toBeInstanceOf(QuotaExceededError);
|
||||
});
|
||||
});
|
||||
119
packages/shade-files/tests/security/replay.test.ts
Normal file
119
packages/shade-files/tests/security/replay.test.ts
Normal file
@@ -0,0 +1,119 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { setupFileRig, type FileTestRig } from '../integration/helpers/rig.js';
|
||||
import { ConflictError, NotFoundError, type FileEntry } from '../../src/index.js';
|
||||
|
||||
/**
|
||||
* Replay-window: the dispatcher rejects requests where `signedAt` is more
|
||||
* than ±5 min from the server clock. Plus: idempotent retries on the same
|
||||
* mutation key produce a single side-effect even with stale signedAt.
|
||||
*/
|
||||
|
||||
describe('Replay window + idempotent retry', () => {
|
||||
let rig: FileTestRig;
|
||||
let writeCount = 0;
|
||||
const blobs = new Map<string, Uint8Array>();
|
||||
|
||||
beforeAll(async () => {
|
||||
rig = await setupFileRig({
|
||||
mkdir: async (ctx) => {
|
||||
if (blobs.has(ctx.path)) throw new ConflictError('exists');
|
||||
blobs.set(ctx.path, new Uint8Array(0));
|
||||
const e: FileEntry = {
|
||||
name: ctx.path.split('/').filter(Boolean).pop() ?? '',
|
||||
kind: 'dir', size: 0, mtime: Date.now(), metadata: {},
|
||||
};
|
||||
return { entry: e };
|
||||
},
|
||||
write: async (ctx) => {
|
||||
writeCount++;
|
||||
if (ctx.args.content.kind !== 'inline') throw new Error('inline expected');
|
||||
blobs.set(ctx.args.path, ctx.args.content.bytes);
|
||||
const e: FileEntry = {
|
||||
name: ctx.args.path.split('/').filter(Boolean).pop() ?? '',
|
||||
kind: 'file',
|
||||
size: ctx.args.content.bytes.byteLength,
|
||||
mtime: Date.now(),
|
||||
metadata: { sha256: ctx.args.content.sha256 },
|
||||
};
|
||||
return { entry: e };
|
||||
},
|
||||
stat: async (ctx) => {
|
||||
if (!blobs.has(ctx.path)) throw new NotFoundError(ctx.path);
|
||||
return {
|
||||
name: ctx.path.split('/').filter(Boolean).pop() ?? '',
|
||||
kind: 'file', size: 0, mtime: 0, metadata: {},
|
||||
};
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await rig.teardown();
|
||||
});
|
||||
|
||||
test('idempotent retry: same key + same args → single side-effect', async () => {
|
||||
writeCount = 0;
|
||||
// Idempotency keys are exactly 22 chars, base64url alphabet.
|
||||
const key = 'replay_key_1234567890A';
|
||||
const data = new Uint8Array([1, 2, 3]);
|
||||
const r1 = await rig.fs.write('/replay.bin', data, { idempotencyKey: key });
|
||||
const r2 = await rig.fs.write('/replay.bin', data, { idempotencyKey: key });
|
||||
const r3 = await rig.fs.write('/replay.bin', data, { idempotencyKey: key });
|
||||
expect(writeCount).toBe(1);
|
||||
expect(r1.entry.size).toBe(3);
|
||||
expect(r2.entry.size).toBe(3);
|
||||
expect(r3.entry.size).toBe(3);
|
||||
});
|
||||
|
||||
test('out-of-window signedAt → InvalidSignatureError (skew rejection)', async () => {
|
||||
// Build a custom client that LIES about signedAt. We hand-craft an
|
||||
// RpcRequest envelope and ship it via the underlying channel.
|
||||
const {
|
||||
ShadeFileRpcChannel,
|
||||
PendingRpcRegistry,
|
||||
attachClientRouting,
|
||||
KIND_STAT_V1,
|
||||
generateRequestId,
|
||||
} = await import('../../src/index.js');
|
||||
const aliceChannel = new ShadeFileRpcChannel(rig.alice);
|
||||
const alicePending = new PendingRpcRegistry();
|
||||
attachClientRouting(aliceChannel, alicePending);
|
||||
|
||||
const requestId = generateRequestId();
|
||||
const stalePromise = alicePending.register<unknown>(requestId, { timeoutMs: 3000 });
|
||||
await aliceChannel.send('bob', {
|
||||
kind: KIND_STAT_V1,
|
||||
id: requestId,
|
||||
args: { path: '/replay.bin' },
|
||||
sig: 'unsigned',
|
||||
signedAt: Date.now() - 10 * 60 * 1000, // 10 min in the past — outside ±5 min window
|
||||
});
|
||||
await expect(stalePromise).rejects.toThrow(/replay window|signature/i);
|
||||
aliceChannel.destroy();
|
||||
});
|
||||
|
||||
test('signedAt far in the future → also rejected', async () => {
|
||||
const {
|
||||
ShadeFileRpcChannel,
|
||||
PendingRpcRegistry,
|
||||
attachClientRouting,
|
||||
KIND_STAT_V1,
|
||||
generateRequestId,
|
||||
} = await import('../../src/index.js');
|
||||
const aliceChannel = new ShadeFileRpcChannel(rig.alice);
|
||||
const alicePending = new PendingRpcRegistry();
|
||||
attachClientRouting(aliceChannel, alicePending);
|
||||
|
||||
const requestId = generateRequestId();
|
||||
const promise = alicePending.register<unknown>(requestId, { timeoutMs: 3000 });
|
||||
await aliceChannel.send('bob', {
|
||||
kind: KIND_STAT_V1,
|
||||
id: requestId,
|
||||
args: { path: '/replay.bin' },
|
||||
sig: 'unsigned',
|
||||
signedAt: Date.now() + 10 * 60 * 1000,
|
||||
});
|
||||
await expect(promise).rejects.toThrow(/replay window|signature/i);
|
||||
aliceChannel.destroy();
|
||||
});
|
||||
});
|
||||
115
packages/shade-files/tests/security/tampered-envelope.test.ts
Normal file
115
packages/shade-files/tests/security/tampered-envelope.test.ts
Normal file
@@ -0,0 +1,115 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { ed25519 } from '@noble/curves/ed25519.js';
|
||||
import { setupFileRig, type FileTestRig } from '../integration/helpers/rig.js';
|
||||
import {
|
||||
bytesToBase64,
|
||||
canonicalRpcBytes,
|
||||
hashArgs,
|
||||
InvalidSignatureError,
|
||||
type FileEntry,
|
||||
NotFoundError,
|
||||
} from '../../src/index.js';
|
||||
|
||||
/**
|
||||
* The dispatcher's `verifySender` callback gets the canonical bytes the
|
||||
* client claims they signed. By plugging a real Ed25519 verify in tests,
|
||||
* we can demonstrate that:
|
||||
* - A valid sig over the canonical bytes is accepted.
|
||||
* - Tampering ANY bound field (kind, args, signedAt, sender) breaks
|
||||
* verification → InvalidSignatureError.
|
||||
*/
|
||||
|
||||
describe('Tampered envelope — Ed25519 sig verification', () => {
|
||||
let rig: FileTestRig;
|
||||
// Generate a stable Ed25519 keypair for Alice. Bob will pin it.
|
||||
const alicePriv = ed25519.utils.randomSecretKey();
|
||||
const alicePub = ed25519.getPublicKey(alicePriv);
|
||||
|
||||
beforeAll(async () => {
|
||||
rig = await setupFileRig({
|
||||
verifySender: (sender, canonical, sigBase64) => {
|
||||
// We only know Alice's key for this test. Bob's pub key would be
|
||||
// looked up similarly in a real app.
|
||||
if (sender !== 'alice') return false;
|
||||
// Decode base64 sig
|
||||
try {
|
||||
const bin = atob(sigBase64);
|
||||
const sigBytes = new Uint8Array(bin.length);
|
||||
for (let i = 0; i < bin.length; i++) sigBytes[i] = bin.charCodeAt(i);
|
||||
return ed25519.verify(sigBytes, canonical, alicePub);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
},
|
||||
stat: async (ctx) => {
|
||||
if (ctx.path !== '/exists.txt') throw new NotFoundError(ctx.path);
|
||||
const e: FileEntry = { name: 'exists.txt', kind: 'file', size: 1, mtime: 0, metadata: {} };
|
||||
return e;
|
||||
},
|
||||
});
|
||||
// Re-create the client with a real signRequest hook.
|
||||
// (Rig's default fs has signRequest=undefined; we replace it.)
|
||||
const { createFileClient, ShadeFileRpcChannel, PendingRpcRegistry, attachClientRouting } = await import('../../src/index.js');
|
||||
const aliceChannel = new ShadeFileRpcChannel(rig.alice);
|
||||
const alicePending = new PendingRpcRegistry();
|
||||
attachClientRouting(aliceChannel, alicePending);
|
||||
rig.fs = createFileClient(rig.alice, aliceChannel, alicePending, 'bob', {
|
||||
defaultTimeoutMs: 5000,
|
||||
signRequest: async (canonical) => {
|
||||
const sig = ed25519.sign(canonical, alicePriv);
|
||||
return bytesToBase64(sig);
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await rig.teardown();
|
||||
});
|
||||
|
||||
test('valid signature → request succeeds', async () => {
|
||||
const result = await rig.fs.stat('/exists.txt');
|
||||
expect(result.name).toBe('exists.txt');
|
||||
});
|
||||
|
||||
test('tampered args → InvalidSignatureError', async () => {
|
||||
// Craft a request manually: sign over '/a' but ship '/b'.
|
||||
const { createFileClient, ShadeFileRpcChannel, PendingRpcRegistry, attachClientRouting } = await import('../../src/index.js');
|
||||
const aliceChannel = new ShadeFileRpcChannel(rig.alice);
|
||||
const alicePending = new PendingRpcRegistry();
|
||||
attachClientRouting(aliceChannel, alicePending);
|
||||
const tamperedFs = createFileClient(rig.alice, aliceChannel, alicePending, 'bob', {
|
||||
defaultTimeoutMs: 3000,
|
||||
signRequest: async (_canonical) => {
|
||||
// Sign over a DIFFERENT canonical (different argsHash), so the
|
||||
// server's recomputation won't match.
|
||||
const fake = canonicalRpcBytes({
|
||||
address: 'alice',
|
||||
signedAt: 0,
|
||||
kind: 'shade.fs.list/v1',
|
||||
id: 'AAAAAAAAAAAAAAAAAAAAAA',
|
||||
argsHash: hashArgs({ tampered: true }),
|
||||
});
|
||||
const sig = ed25519.sign(fake, alicePriv);
|
||||
return bytesToBase64(sig);
|
||||
},
|
||||
});
|
||||
await expect(tamperedFs.stat('/exists.txt')).rejects.toBeInstanceOf(InvalidSignatureError);
|
||||
});
|
||||
|
||||
test('valid signature from unknown signer → InvalidSignatureError', async () => {
|
||||
const { createFileClient, ShadeFileRpcChannel, PendingRpcRegistry, attachClientRouting } = await import('../../src/index.js');
|
||||
const aliceChannel = new ShadeFileRpcChannel(rig.alice);
|
||||
const alicePending = new PendingRpcRegistry();
|
||||
attachClientRouting(aliceChannel, alicePending);
|
||||
|
||||
const otherPriv = ed25519.utils.randomSecretKey();
|
||||
const wrongFs = createFileClient(rig.alice, aliceChannel, alicePending, 'bob', {
|
||||
defaultTimeoutMs: 3000,
|
||||
signRequest: async (canonical) => {
|
||||
const sig = ed25519.sign(canonical, otherPriv);
|
||||
return bytesToBase64(sig);
|
||||
},
|
||||
});
|
||||
await expect(wrongFs.stat('/exists.txt')).rejects.toBeInstanceOf(InvalidSignatureError);
|
||||
});
|
||||
});
|
||||
135
packages/shade-files/tests/unit/canonical.test.ts
Normal file
135
packages/shade-files/tests/unit/canonical.test.ts
Normal file
@@ -0,0 +1,135 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
canonicalRpcBytes,
|
||||
canonicalJsonStringify,
|
||||
hashArgs,
|
||||
bytesToHex,
|
||||
bytesToBase64,
|
||||
base64ToBytes,
|
||||
} from '../../src/index.js';
|
||||
|
||||
describe('canonicalJsonStringify', () => {
|
||||
test('sorts object keys', () => {
|
||||
expect(canonicalJsonStringify({ b: 1, a: 2 })).toBe('{"a":2,"b":1}');
|
||||
});
|
||||
|
||||
test('skips undefined values', () => {
|
||||
expect(canonicalJsonStringify({ a: 1, b: undefined, c: 2 })).toBe('{"a":1,"c":2}');
|
||||
});
|
||||
|
||||
test('preserves array order', () => {
|
||||
expect(canonicalJsonStringify([3, 1, 2])).toBe('[3,1,2]');
|
||||
});
|
||||
|
||||
test('recursive sorting', () => {
|
||||
const a = canonicalJsonStringify({ outer: { y: 1, x: 2 } });
|
||||
const b = canonicalJsonStringify({ outer: { x: 2, y: 1 } });
|
||||
expect(a).toBe(b);
|
||||
expect(a).toBe('{"outer":{"x":2,"y":1}}');
|
||||
});
|
||||
|
||||
test('handles primitives', () => {
|
||||
expect(canonicalJsonStringify(null)).toBe('null');
|
||||
expect(canonicalJsonStringify(true)).toBe('true');
|
||||
expect(canonicalJsonStringify(42)).toBe('42');
|
||||
expect(canonicalJsonStringify('hi')).toBe('"hi"');
|
||||
});
|
||||
|
||||
test('different argument orders hash identically', () => {
|
||||
const h1 = hashArgs({ b: 1, a: 2 });
|
||||
const h2 = hashArgs({ a: 2, b: 1 });
|
||||
expect(bytesToHex(h1)).toBe(bytesToHex(h2));
|
||||
});
|
||||
|
||||
test('different values hash differently', () => {
|
||||
expect(bytesToHex(hashArgs({ a: 1 }))).not.toBe(bytesToHex(hashArgs({ a: 2 })));
|
||||
});
|
||||
});
|
||||
|
||||
describe('canonicalRpcBytes', () => {
|
||||
test('deterministic for same input (known vector)', () => {
|
||||
const args = {
|
||||
address: 'alice',
|
||||
signedAt: 1730000000000,
|
||||
kind: 'shade.fs.list/v1',
|
||||
id: 'AbCdEfGhIjKlMnOpQrStUv',
|
||||
argsHash: new Uint8Array(32).fill(0xab),
|
||||
};
|
||||
const a = canonicalRpcBytes(args);
|
||||
const b = canonicalRpcBytes(args);
|
||||
expect(a).toEqual(b);
|
||||
});
|
||||
|
||||
test('changes when address differs', () => {
|
||||
const base = {
|
||||
signedAt: 1,
|
||||
kind: 'shade.fs.list/v1',
|
||||
id: 'AbCdEfGhIjKlMnOpQrStUv',
|
||||
argsHash: new Uint8Array(32),
|
||||
};
|
||||
const a = canonicalRpcBytes({ ...base, address: 'alice' });
|
||||
const b = canonicalRpcBytes({ ...base, address: 'bob' });
|
||||
expect(a).not.toEqual(b);
|
||||
});
|
||||
|
||||
test('changes when signedAt differs', () => {
|
||||
const base = {
|
||||
address: 'alice',
|
||||
kind: 'shade.fs.list/v1',
|
||||
id: 'AbCdEfGhIjKlMnOpQrStUv',
|
||||
argsHash: new Uint8Array(32),
|
||||
};
|
||||
expect(canonicalRpcBytes({ ...base, signedAt: 1 })).not.toEqual(
|
||||
canonicalRpcBytes({ ...base, signedAt: 2 }),
|
||||
);
|
||||
});
|
||||
|
||||
test('changes when kind differs', () => {
|
||||
const base = {
|
||||
address: 'alice',
|
||||
signedAt: 1,
|
||||
id: 'AbCdEfGhIjKlMnOpQrStUv',
|
||||
argsHash: new Uint8Array(32),
|
||||
};
|
||||
expect(canonicalRpcBytes({ ...base, kind: 'shade.fs.list/v1' })).not.toEqual(
|
||||
canonicalRpcBytes({ ...base, kind: 'shade.fs.stat/v1' }),
|
||||
);
|
||||
});
|
||||
|
||||
test('changes when argsHash differs', () => {
|
||||
const base = {
|
||||
address: 'alice',
|
||||
signedAt: 1,
|
||||
kind: 'shade.fs.list/v1',
|
||||
id: 'AbCdEfGhIjKlMnOpQrStUv',
|
||||
};
|
||||
expect(
|
||||
canonicalRpcBytes({ ...base, argsHash: new Uint8Array(32) }),
|
||||
).not.toEqual(
|
||||
canonicalRpcBytes({ ...base, argsHash: new Uint8Array(32).fill(1) }),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('hashArgs', () => {
|
||||
test('produces 32-byte digest', () => {
|
||||
expect(hashArgs({ a: 1 }).length).toBe(32);
|
||||
});
|
||||
|
||||
test('null input is fine', () => {
|
||||
expect(hashArgs(null).length).toBe(32);
|
||||
});
|
||||
});
|
||||
|
||||
describe('bytesToHex / base64', () => {
|
||||
test('hex roundtrip', () => {
|
||||
const b = new Uint8Array([0x01, 0xab, 0xff, 0x00]);
|
||||
expect(bytesToHex(b)).toBe('01abff00');
|
||||
});
|
||||
|
||||
test('base64 roundtrip', () => {
|
||||
const b = new Uint8Array([1, 2, 3, 4]);
|
||||
const enc = bytesToBase64(b);
|
||||
expect(base64ToBytes(enc)).toEqual(b);
|
||||
});
|
||||
});
|
||||
90
packages/shade-files/tests/unit/concurrency.test.ts
Normal file
90
packages/shade-files/tests/unit/concurrency.test.ts
Normal file
@@ -0,0 +1,90 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { runWithConcurrency } from '../../src/client/concurrency.js';
|
||||
|
||||
async function* range(n: number): AsyncIterable<number> {
|
||||
for (let i = 0; i < n; i++) yield i;
|
||||
}
|
||||
|
||||
function delay(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
describe('runWithConcurrency', () => {
|
||||
test('runs all items', async () => {
|
||||
const seen: number[] = [];
|
||||
await runWithConcurrency(range(10), async (i) => {
|
||||
seen.push(i);
|
||||
}, { concurrency: 4 });
|
||||
expect(seen.sort((a, b) => a - b)).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]);
|
||||
});
|
||||
|
||||
test('respects concurrency cap (never exceeds N inflight)', async () => {
|
||||
let inflight = 0;
|
||||
let peak = 0;
|
||||
await runWithConcurrency(range(50), async () => {
|
||||
inflight++;
|
||||
peak = Math.max(peak, inflight);
|
||||
await delay(5);
|
||||
inflight--;
|
||||
}, { concurrency: 4 });
|
||||
expect(peak).toBeLessThanOrEqual(4);
|
||||
expect(peak).toBeGreaterThanOrEqual(2);
|
||||
});
|
||||
|
||||
test('throws on first error by default (fail-fast)', async () => {
|
||||
let processed = 0;
|
||||
await expect(
|
||||
runWithConcurrency(range(20), async (i) => {
|
||||
await delay(1);
|
||||
if (i === 3) throw new Error('boom');
|
||||
processed++;
|
||||
}, { concurrency: 2 }),
|
||||
).rejects.toThrow('boom');
|
||||
// We don't process all 20; bounded by fail-fast.
|
||||
expect(processed).toBeLessThan(20);
|
||||
});
|
||||
|
||||
test('continueOnError reports each + drains', async () => {
|
||||
const errors: number[] = [];
|
||||
let processed = 0;
|
||||
await runWithConcurrency(range(10), async (i) => {
|
||||
await delay(1);
|
||||
if (i % 3 === 0) throw new Error(`bad-${i}`);
|
||||
processed++;
|
||||
}, {
|
||||
concurrency: 3,
|
||||
continueOnError: true,
|
||||
onError: (item) => errors.push(item),
|
||||
});
|
||||
expect(processed).toBe(6); // i = 1,2,4,5,7,8
|
||||
expect(errors.sort((a, b) => a - b)).toEqual([0, 3, 6, 9]);
|
||||
});
|
||||
|
||||
test('aborts via signal', async () => {
|
||||
const ctrl = new AbortController();
|
||||
let processed = 0;
|
||||
setTimeout(() => ctrl.abort(), 20);
|
||||
await expect(
|
||||
runWithConcurrency(range(100), async () => {
|
||||
await delay(5);
|
||||
processed++;
|
||||
}, { concurrency: 4, signal: ctrl.signal }),
|
||||
).rejects.toThrow();
|
||||
expect(processed).toBeLessThan(100);
|
||||
});
|
||||
|
||||
test('concurrency=1 is sequential', async () => {
|
||||
const order: number[] = [];
|
||||
await runWithConcurrency(range(5), async (i) => {
|
||||
order.push(i);
|
||||
await delay(1);
|
||||
}, { concurrency: 1 });
|
||||
expect(order).toEqual([0, 1, 2, 3, 4]);
|
||||
});
|
||||
|
||||
test('throws on concurrency < 1', () => {
|
||||
expect(() =>
|
||||
runWithConcurrency(range(0), async () => undefined, { concurrency: 0 }),
|
||||
).toThrow('concurrency must be ≥ 1');
|
||||
});
|
||||
});
|
||||
64
packages/shade-files/tests/unit/correlate.test.ts
Normal file
64
packages/shade-files/tests/unit/correlate.test.ts
Normal file
@@ -0,0 +1,64 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import * as fc from 'fast-check';
|
||||
import {
|
||||
generateRequestId,
|
||||
generateIdempotencyKey,
|
||||
base64UrlEncode,
|
||||
base64UrlDecode,
|
||||
RequestIdSchema,
|
||||
} from '../../src/index.js';
|
||||
|
||||
describe('generateRequestId', () => {
|
||||
test('produces 22-char base64url string', () => {
|
||||
const id = generateRequestId();
|
||||
expect(id.length).toBe(22);
|
||||
expect(RequestIdSchema.safeParse(id).success).toBe(true);
|
||||
});
|
||||
|
||||
test('1e5 generated IDs are all unique', () => {
|
||||
const seen = new Set<string>();
|
||||
for (let i = 0; i < 100_000; i++) {
|
||||
const id = generateRequestId();
|
||||
expect(seen.has(id)).toBe(false);
|
||||
seen.add(id);
|
||||
}
|
||||
expect(seen.size).toBe(100_000);
|
||||
});
|
||||
|
||||
test('generateIdempotencyKey returns the same shape', () => {
|
||||
expect(generateIdempotencyKey().length).toBe(22);
|
||||
});
|
||||
});
|
||||
|
||||
describe('base64url encode/decode', () => {
|
||||
test('roundtrip arbitrary bytes (property-based)', () => {
|
||||
fc.assert(
|
||||
fc.property(fc.uint8Array({ minLength: 0, maxLength: 64 }), (bytes) => {
|
||||
const decoded = base64UrlDecode(base64UrlEncode(bytes));
|
||||
expect(decoded).toEqual(bytes);
|
||||
}),
|
||||
{ numRuns: 500 },
|
||||
);
|
||||
});
|
||||
|
||||
test('produces URL-safe alphabet only', () => {
|
||||
fc.assert(
|
||||
fc.property(fc.uint8Array({ minLength: 1, maxLength: 64 }), (bytes) => {
|
||||
const enc = base64UrlEncode(bytes);
|
||||
expect(enc).not.toMatch(/[+/=]/);
|
||||
}),
|
||||
{ numRuns: 200 },
|
||||
);
|
||||
});
|
||||
|
||||
test('handles empty input', () => {
|
||||
expect(base64UrlEncode(new Uint8Array(0))).toBe('');
|
||||
expect(base64UrlDecode('')).toEqual(new Uint8Array(0));
|
||||
});
|
||||
|
||||
test('decodes inputs without padding correctly', () => {
|
||||
expect(base64UrlDecode('YQ')).toEqual(new Uint8Array([0x61]));
|
||||
expect(base64UrlDecode('YWI')).toEqual(new Uint8Array([0x61, 0x62]));
|
||||
expect(base64UrlDecode('YWJj')).toEqual(new Uint8Array([0x61, 0x62, 0x63]));
|
||||
});
|
||||
});
|
||||
149
packages/shade-files/tests/unit/envelope-codec.test.ts
Normal file
149
packages/shade-files/tests/unit/envelope-codec.test.ts
Normal file
@@ -0,0 +1,149 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
encodeEnvelope,
|
||||
looksLikeFileEnvelope,
|
||||
tryParseEnvelope,
|
||||
classify,
|
||||
KIND_LIST_V1,
|
||||
KIND_ERROR_V1,
|
||||
KIND_CANCEL_V1,
|
||||
responseKindOf,
|
||||
} from '../../src/index.js';
|
||||
import type { RpcRequest, RpcResponse, RpcError, RpcCancel } from '../../src/index.js';
|
||||
|
||||
const ID = 'AbCdEfGhIjKlMnOpQrStUv';
|
||||
|
||||
describe('looksLikeFileEnvelope', () => {
|
||||
test('matches plaintext containing shade.fs', () => {
|
||||
expect(looksLikeFileEnvelope('{"kind":"shade.fs.list/v1"}')).toBe(true);
|
||||
});
|
||||
test('rejects unrelated', () => {
|
||||
expect(looksLikeFileEnvelope('hello world')).toBe(false);
|
||||
expect(looksLikeFileEnvelope('{"kind":"shade.stream-init/v1"}')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('tryParseEnvelope', () => {
|
||||
test('returns null on non-JSON', () => {
|
||||
expect(tryParseEnvelope('not json {{')).toBeNull();
|
||||
});
|
||||
|
||||
test('returns null on JSON that does not match any envelope', () => {
|
||||
expect(tryParseEnvelope('{"kind":"unknown"}')).toBeNull();
|
||||
expect(tryParseEnvelope('{"foo":"bar"}')).toBeNull();
|
||||
});
|
||||
|
||||
test('classifies request', () => {
|
||||
const req: RpcRequest = {
|
||||
kind: KIND_LIST_V1,
|
||||
id: ID,
|
||||
args: { path: '/' },
|
||||
sig: 'abc',
|
||||
signedAt: 1,
|
||||
};
|
||||
const c = tryParseEnvelope(encodeEnvelope(req));
|
||||
expect(c?.kind).toBe('request');
|
||||
});
|
||||
|
||||
test('classifies response', () => {
|
||||
const resp: RpcResponse = {
|
||||
kind: responseKindOf(KIND_LIST_V1),
|
||||
id: ID,
|
||||
result: { entries: [], hasMore: false },
|
||||
};
|
||||
const c = tryParseEnvelope(encodeEnvelope(resp));
|
||||
expect(c?.kind).toBe('response');
|
||||
});
|
||||
|
||||
test('classifies error', () => {
|
||||
const err: RpcError = {
|
||||
kind: KIND_ERROR_V1,
|
||||
id: ID,
|
||||
error: { code: 'NOT_FOUND', message: 'missing' },
|
||||
};
|
||||
const c = tryParseEnvelope(encodeEnvelope(err));
|
||||
expect(c?.kind).toBe('error');
|
||||
});
|
||||
|
||||
test('classifies cancel', () => {
|
||||
const cancel: RpcCancel = {
|
||||
kind: KIND_CANCEL_V1,
|
||||
id: ID,
|
||||
reason: 'user-cancel',
|
||||
};
|
||||
const c = tryParseEnvelope(encodeEnvelope(cancel));
|
||||
expect(c?.kind).toBe('cancel');
|
||||
});
|
||||
|
||||
test('rejects request with missing fields', () => {
|
||||
expect(tryParseEnvelope(JSON.stringify({ kind: KIND_LIST_V1 }))).toBeNull();
|
||||
expect(
|
||||
tryParseEnvelope(
|
||||
JSON.stringify({ kind: KIND_LIST_V1, id: ID, args: {}, sig: 'x' }),
|
||||
),
|
||||
).toBeNull(); // missing signedAt
|
||||
});
|
||||
|
||||
test('rejects response shape with non-.response suffix and no signature', () => {
|
||||
// Missing both `.response` suffix (so not a response) AND missing
|
||||
// `sig`/`signedAt` (so not a valid request) → no schema matches.
|
||||
expect(
|
||||
tryParseEnvelope(
|
||||
JSON.stringify({ kind: 'shade.fs.list/v1', id: ID, result: {} }),
|
||||
),
|
||||
).toBeNull();
|
||||
});
|
||||
|
||||
test('rejects malformed id length', () => {
|
||||
const req = {
|
||||
kind: KIND_LIST_V1,
|
||||
id: 'short',
|
||||
args: {},
|
||||
sig: 'x',
|
||||
signedAt: 1,
|
||||
};
|
||||
expect(tryParseEnvelope(JSON.stringify(req))).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('classify on already-validated envelopes', () => {
|
||||
test('correct discriminator on each branch', () => {
|
||||
const req: RpcRequest = {
|
||||
kind: KIND_LIST_V1, id: ID, args: {}, sig: 'x', signedAt: 1,
|
||||
};
|
||||
expect(classify(req).kind).toBe('request');
|
||||
|
||||
const resp: RpcResponse = {
|
||||
kind: responseKindOf(KIND_LIST_V1), id: ID, result: {},
|
||||
};
|
||||
expect(classify(resp).kind).toBe('response');
|
||||
|
||||
const err: RpcError = {
|
||||
kind: KIND_ERROR_V1, id: ID, error: { code: 'NOT_FOUND', message: '' },
|
||||
};
|
||||
expect(classify(err).kind).toBe('error');
|
||||
|
||||
const cancel: RpcCancel = { kind: KIND_CANCEL_V1, id: ID };
|
||||
expect(classify(cancel).kind).toBe('cancel');
|
||||
});
|
||||
});
|
||||
|
||||
describe('encodeEnvelope', () => {
|
||||
test('roundtrips request envelope', () => {
|
||||
const req: RpcRequest = {
|
||||
kind: KIND_LIST_V1,
|
||||
id: ID,
|
||||
args: { path: '/foo' },
|
||||
idempotencyKey: 'IdemKeyAaBbCcDdEeFfGgH',
|
||||
attempt: 1,
|
||||
sig: 'sig',
|
||||
signedAt: 1730000000000,
|
||||
};
|
||||
const c = tryParseEnvelope(encodeEnvelope(req));
|
||||
expect(c?.kind).toBe('request');
|
||||
if (c?.kind === 'request') {
|
||||
expect(c.envelope.idempotencyKey).toBe('IdemKeyAaBbCcDdEeFfGgH');
|
||||
expect(c.envelope.attempt).toBe(1);
|
||||
}
|
||||
});
|
||||
});
|
||||
108
packages/shade-files/tests/unit/idempotency-cache.test.ts
Normal file
108
packages/shade-files/tests/unit/idempotency-cache.test.ts
Normal file
@@ -0,0 +1,108 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { IdempotencyCache, IdempotencyConflictError } from '../../src/index.js';
|
||||
|
||||
describe('IdempotencyCache.begin', () => {
|
||||
test('first call returns fresh', () => {
|
||||
const cache = new IdempotencyCache();
|
||||
const result = cache.begin('alice', 'key1', { path: '/foo' });
|
||||
expect(result.status).toBe('fresh');
|
||||
});
|
||||
|
||||
test('replay returns cached response', () => {
|
||||
const cache = new IdempotencyCache();
|
||||
const a = cache.begin('alice', 'key1', { path: '/foo' });
|
||||
if (a.status !== 'fresh') throw new Error('expected fresh');
|
||||
a.commit({ ok: true });
|
||||
|
||||
const b = cache.begin('alice', 'key1', { path: '/foo' });
|
||||
expect(b.status).toBe('replay');
|
||||
if (b.status === 'replay') {
|
||||
expect(b.response).toEqual({ ok: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('argsHash mismatch throws IdempotencyConflictError', () => {
|
||||
const cache = new IdempotencyCache();
|
||||
const a = cache.begin('alice', 'key1', { path: '/foo' });
|
||||
if (a.status !== 'fresh') throw new Error('expected fresh');
|
||||
a.commit({ ok: true });
|
||||
|
||||
expect(() =>
|
||||
cache.begin('alice', 'key1', { path: '/different' }),
|
||||
).toThrow(IdempotencyConflictError);
|
||||
});
|
||||
|
||||
test('inflight retry returns wait-promise', async () => {
|
||||
const cache = new IdempotencyCache();
|
||||
const a = cache.begin('alice', 'key1', { x: 1 });
|
||||
if (a.status !== 'fresh') throw new Error('expected fresh');
|
||||
|
||||
const b = cache.begin('alice', 'key1', { x: 1 });
|
||||
expect(b.status).toBe('wait');
|
||||
|
||||
if (b.status === 'wait') {
|
||||
a.commit({ result: 42 });
|
||||
const v = await b.promise;
|
||||
expect(v).toEqual({ result: 42 });
|
||||
}
|
||||
});
|
||||
|
||||
test('different senders are isolated', () => {
|
||||
const cache = new IdempotencyCache();
|
||||
const a = cache.begin('alice', 'k', { x: 1 });
|
||||
if (a.status !== 'fresh') throw new Error('expected fresh');
|
||||
a.commit({ side: 'alice' });
|
||||
|
||||
const b = cache.begin('bob', 'k', { x: 1 });
|
||||
expect(b.status).toBe('fresh');
|
||||
});
|
||||
|
||||
test('abandon removes the entry so retries proceed fresh', () => {
|
||||
const cache = new IdempotencyCache();
|
||||
const a = cache.begin('alice', 'key', { p: 1 });
|
||||
if (a.status !== 'fresh') throw new Error('expected fresh');
|
||||
a.abandon();
|
||||
|
||||
const b = cache.begin('alice', 'key', { p: 1 });
|
||||
expect(b.status).toBe('fresh');
|
||||
});
|
||||
});
|
||||
|
||||
describe('IdempotencyCache TTL + LRU', () => {
|
||||
test('expired entries are evicted on next access', async () => {
|
||||
const cache = new IdempotencyCache({ ttlMs: 5 });
|
||||
const a = cache.begin('alice', 'k', { x: 1 });
|
||||
if (a.status !== 'fresh') throw new Error('expected fresh');
|
||||
a.commit('done');
|
||||
|
||||
await new Promise((r) => setTimeout(r, 15));
|
||||
|
||||
const b = cache.begin('alice', 'k', { x: 1 });
|
||||
expect(b.status).toBe('fresh'); // expired → fresh again
|
||||
});
|
||||
|
||||
test('LRU caps per-sender entries', () => {
|
||||
const cache = new IdempotencyCache({ maxEntriesPerSender: 3 });
|
||||
for (let i = 0; i < 5; i++) {
|
||||
const r = cache.begin('alice', `key${i}`, { i });
|
||||
if (r.status === 'fresh') r.commit(i);
|
||||
}
|
||||
expect(cache.size()).toBe(3); // first two evicted
|
||||
});
|
||||
});
|
||||
|
||||
describe('IdempotencyCache.prune', () => {
|
||||
test('removes only TTL-expired entries', async () => {
|
||||
// Two senders so begin() on one doesn't auto-evict the other's expired entry.
|
||||
const cache = new IdempotencyCache({ ttlMs: 5 });
|
||||
const a = cache.begin('alice', 'old', { x: 1 });
|
||||
if (a.status === 'fresh') a.commit(1);
|
||||
await new Promise((r) => setTimeout(r, 15));
|
||||
const b = cache.begin('bob', 'new', { x: 2 });
|
||||
if (b.status === 'fresh') b.commit(2);
|
||||
|
||||
const removed = cache.prune();
|
||||
expect(removed).toBe(1); // alice/old expired; bob/new fresh
|
||||
expect(cache.size()).toBe(1);
|
||||
});
|
||||
});
|
||||
161
packages/shade-files/tests/unit/inline-threshold.test.ts
Normal file
161
packages/shade-files/tests/unit/inline-threshold.test.ts
Normal file
@@ -0,0 +1,161 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { decideInline, INLINE_THRESHOLD } from '../../src/client/inline-threshold.js';
|
||||
|
||||
const KIB = 1024;
|
||||
|
||||
function streamOf(...chunks: Uint8Array[]): ReadableStream<Uint8Array> {
|
||||
let i = 0;
|
||||
return new ReadableStream<Uint8Array>({
|
||||
pull(controller) {
|
||||
if (i < chunks.length) {
|
||||
controller.enqueue(chunks[i]!);
|
||||
i++;
|
||||
} else {
|
||||
controller.close();
|
||||
}
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
async function drainStream(s: ReadableStream<Uint8Array>): Promise<Uint8Array> {
|
||||
const reader = s.getReader();
|
||||
const parts: Uint8Array[] = [];
|
||||
let total = 0;
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (value === undefined) continue;
|
||||
parts.push(value);
|
||||
total += value.byteLength;
|
||||
}
|
||||
reader.releaseLock();
|
||||
const out = new Uint8Array(total);
|
||||
let offset = 0;
|
||||
for (const p of parts) {
|
||||
out.set(p, offset);
|
||||
offset += p.byteLength;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
describe('decideInline (Uint8Array)', () => {
|
||||
test('1 KiB → inline', async () => {
|
||||
const bytes = new Uint8Array(KIB).fill(0xab);
|
||||
const decision = await decideInline(bytes);
|
||||
expect(decision.kind).toBe('inline');
|
||||
if (decision.kind === 'inline') {
|
||||
expect(decision.bytes.byteLength).toBe(KIB);
|
||||
}
|
||||
});
|
||||
|
||||
test('exactly 256 KiB → inline (boundary)', async () => {
|
||||
const bytes = new Uint8Array(INLINE_THRESHOLD).fill(0xcd);
|
||||
const decision = await decideInline(bytes);
|
||||
expect(decision.kind).toBe('inline');
|
||||
});
|
||||
|
||||
test('256 KiB + 1 → streams (boundary +1)', async () => {
|
||||
const bytes = new Uint8Array(INLINE_THRESHOLD + 1).fill(0xef);
|
||||
const decision = await decideInline(bytes);
|
||||
expect(decision.kind).toBe('streams');
|
||||
if (decision.kind === 'streams') {
|
||||
expect(decision.size).toBe(INLINE_THRESHOLD + 1);
|
||||
const drained = await drainStream(decision.stream);
|
||||
expect(drained.byteLength).toBe(INLINE_THRESHOLD + 1);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('decideInline (Blob)', () => {
|
||||
test('small Blob → inline + propagates contentType', async () => {
|
||||
const blob = new Blob([new Uint8Array(100).fill(7)], { type: 'application/octet-stream' });
|
||||
const decision = await decideInline(blob);
|
||||
expect(decision.kind).toBe('inline');
|
||||
if (decision.kind === 'inline') {
|
||||
expect(decision.contentType).toBe('application/octet-stream');
|
||||
expect(decision.bytes.byteLength).toBe(100);
|
||||
}
|
||||
});
|
||||
|
||||
test('large Blob → streams + propagates size + contentType', async () => {
|
||||
const big = new Uint8Array(INLINE_THRESHOLD + KIB).fill(1);
|
||||
const blob = new Blob([big], { type: 'image/png' });
|
||||
const decision = await decideInline(blob);
|
||||
expect(decision.kind).toBe('streams');
|
||||
if (decision.kind === 'streams') {
|
||||
expect(decision.size).toBe(big.byteLength);
|
||||
expect(decision.contentType).toBe('image/png');
|
||||
}
|
||||
});
|
||||
|
||||
test('empty Blob.type → no contentType', async () => {
|
||||
const blob = new Blob([new Uint8Array(10)]);
|
||||
const decision = await decideInline(blob);
|
||||
expect(decision.kind).toBe('inline');
|
||||
if (decision.kind === 'inline') {
|
||||
expect(decision.contentType).toBeUndefined();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('decideInline (ReadableStream — bare)', () => {
|
||||
test('EOF before threshold → inline', async () => {
|
||||
const stream = streamOf(new Uint8Array(100), new Uint8Array(200));
|
||||
const decision = await decideInline(stream);
|
||||
expect(decision.kind).toBe('inline');
|
||||
if (decision.kind === 'inline') {
|
||||
expect(decision.bytes.byteLength).toBe(300);
|
||||
}
|
||||
});
|
||||
|
||||
test('crosses threshold mid-chunk → streams + remainder available', async () => {
|
||||
// 256 KiB + 1 byte across two chunks
|
||||
const a = new Uint8Array(200 * KIB).fill(0x10);
|
||||
const b = new Uint8Array(100 * KIB).fill(0x20);
|
||||
const stream = streamOf(a, b);
|
||||
const decision = await decideInline(stream);
|
||||
expect(decision.kind).toBe('streams');
|
||||
if (decision.kind === 'streams') {
|
||||
const drained = await drainStream(decision.stream);
|
||||
expect(drained.byteLength).toBe(a.byteLength + b.byteLength);
|
||||
expect(drained[0]).toBe(0x10);
|
||||
expect(drained[drained.length - 1]).toBe(0x20);
|
||||
}
|
||||
});
|
||||
|
||||
test('empty stream → inline (zero bytes)', async () => {
|
||||
const stream = streamOf();
|
||||
const decision = await decideInline(stream);
|
||||
expect(decision.kind).toBe('inline');
|
||||
if (decision.kind === 'inline') {
|
||||
expect(decision.bytes.byteLength).toBe(0);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('decideInline ({ stream, size })', () => {
|
||||
test('declared size ≤ threshold → inline', async () => {
|
||||
const bytes = new Uint8Array(KIB).fill(9);
|
||||
const decision = await decideInline({ stream: streamOf(bytes), size: KIB, contentType: 'text/plain' });
|
||||
expect(decision.kind).toBe('inline');
|
||||
if (decision.kind === 'inline') {
|
||||
expect(decision.bytes.byteLength).toBe(KIB);
|
||||
expect(decision.contentType).toBe('text/plain');
|
||||
}
|
||||
});
|
||||
|
||||
test('declared size > threshold → streams', async () => {
|
||||
const big = new Uint8Array(500 * KIB).fill(2);
|
||||
const decision = await decideInline({ stream: streamOf(big), size: big.byteLength });
|
||||
expect(decision.kind).toBe('streams');
|
||||
if (decision.kind === 'streams') {
|
||||
expect(decision.size).toBe(big.byteLength);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('decideInline (errors)', () => {
|
||||
test('unsupported input throws TypeError', async () => {
|
||||
await expect(decideInline(42 as unknown as Uint8Array)).rejects.toThrow(TypeError);
|
||||
});
|
||||
});
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user