feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
Some checks failed
Test / test (push) Has been cancelled
Some checks failed
Test / test (push) Has been cancelled
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).
@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
+ per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
(> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
/ shadeFilesReadStreamId correlation. Server-side TransformStream
bridges accept inbound transfers immediately (engine rejects chunks
that arrive before accept) and park the readable for the matching
RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
uploadDirectory()/downloadDirectory() with bounded concurrency pool
(default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
policy (traversal + percent-decode hardening), fingerprint gate
(required/optional/reject), pluggable Ed25519 sig verification with
±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.
Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
u16 to u32. The previous u16 silently truncated payloads above
64 KiB — a hard correctness ceiling that blocked inline file ops
up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
only. Cross-platform Kotlin port (android/shade-android) updated to
match; test-vectors/wire-format.json regenerated.
Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
Concurrent decryptions of the same peer raced ratchet state
(manifested as sporadic "Failed to decrypt — wrong key or tampered
data" under load — surfaced once concurrent uploadDirectory pumped
many writes in flight). Encrypt was already serialized via
Shade.send's encryptChains; decrypt is now serialized at the
manager layer too.
@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
application-level key/value pairs that round-trip verbatim through
stream-init plaintext. Used by @shade/files for write/read
correlation; available to any consumer.
@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
BackgroundTasks.setHook(name, fn) for runtime hook registration.
Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
shade-sdk streams-bridge + shade-widgets transfer hooks were
uncommitted prior to this session. Including them keeps the
workspace consistent at 0.3.0 since @shade/files depends on them.
Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
3× stable). Coverage spans unit (inline-threshold + concurrency),
integration (read-write inline + streams up to 1 MiB, walk +
upload/download directory, custom-op, metrics, SDK namespace
end-to-end), and security (tampered-envelope sig verification,
replay window, fingerprint gate, rate-limit + quota).
Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
201
docs/files.md
Normal file
201
docs/files.md
Normal file
@@ -0,0 +1,201 @@
|
||||
# `@shade/files` — E2EE filesystem RPC
|
||||
|
||||
`@shade/files` exposes a typed, end-to-end-encrypted RPC surface for
|
||||
filesystem-style operations between two Shade peers. Both sides ride a
|
||||
single Double Ratchet session for control envelopes; large-file content
|
||||
(`> 256 KiB`) flows through `@shade/transfer` lanes, correlated back to
|
||||
the RPC by an opaque `userMetadata.shadeFilesWriteId` /
|
||||
`shadeFilesReadStreamId`.
|
||||
|
||||
The package is a **primitive**, not a UI: it ships hooks and clients, not
|
||||
file managers. Consumers (e.g. Dispatch, Mail, Drive-style apps) keep
|
||||
their own UI and plug `@shade/files` underneath.
|
||||
|
||||
## Quick start
|
||||
|
||||
### Server (Bob exposes a filesystem)
|
||||
|
||||
```ts
|
||||
import { createShade } from '@shade/sdk';
|
||||
|
||||
const bob = await createShade({ prekeyServer, address: 'bob' });
|
||||
bob.configureTransfers({ resolveBaseUrl: ... });
|
||||
Bun.serve({ port: 8080, fetch: (await bob.transferRoute()).fetch });
|
||||
|
||||
const stop = await bob.files.serve({
|
||||
list: async (ctx) => ({ entries: await readdirAt(ctx.path), hasMore: false }),
|
||||
stat: async (ctx) => statAt(ctx.path),
|
||||
mkdir: async (ctx) => mkdirAt(ctx.path, ctx.args.recursive),
|
||||
delete: async (ctx) => deleteAt(ctx.path, ctx.args.recursive),
|
||||
move: async (ctx) => moveAt(ctx.args.src, ctx.args.dst, ctx.args.overwrite),
|
||||
read: async (ctx) => readAt(ctx.path), // returns inline | streams
|
||||
write: async (ctx) => writeAt(ctx.args), // receives inline | streams
|
||||
getThumbnail: async (ctx) => thumbnailAt(ctx.path, ctx.args.size, ctx.args.format),
|
||||
});
|
||||
|
||||
// Later: stop the handler.
|
||||
await stop();
|
||||
```
|
||||
|
||||
### Client (Alice consumes Bob's filesystem)
|
||||
|
||||
```ts
|
||||
const alice = await createShade({ prekeyServer, address: 'alice' });
|
||||
alice.configureTransfers({ resolveBaseUrl: ... });
|
||||
Bun.serve({ port: 8081, fetch: (await alice.transferRoute()).fetch });
|
||||
|
||||
const fs = await alice.files.client('bob');
|
||||
|
||||
const page = await fs.list('/photos');
|
||||
await fs.write('/photos/cover.png', new Uint8Array(...)); // auto inline/streams
|
||||
const result = await fs.read('/photos/cover.png');
|
||||
if (result.kind === 'inline') console.log(result.bytes.byteLength);
|
||||
else for await (const _chunk of /* result.stream */) { /* ... */ }
|
||||
```
|
||||
|
||||
## Op surface
|
||||
|
||||
| Op | Args | Result |
|
||||
|----------------|------------------------------------------|-----------------------------------------|
|
||||
| `list` | `path`, `cursor?`, `pageSize?`, `filter?`| `{ entries, hasMore, nextCursor? }` |
|
||||
| `stat` | `path` | `FileEntry` |
|
||||
| `mkdir` | `path`, `recursive?` | `{ entry: FileEntry }` |
|
||||
| `delete` | `path`, `recursive?` | `{ deletedCount }` |
|
||||
| `move` | `src`, `dst`, `overwrite?` | `{ entry: FileEntry }` |
|
||||
| `read` | `path`, `range?`, `preferInline?` | inline `{ bytes, sha256, size }` or streams `{ stream, size, sha256 }` |
|
||||
| `write` | `path`, `Uint8Array \| Blob \| Stream` | `{ entry: FileEntry }` |
|
||||
| `getThumbnail` | `path`, `size: 64\|128\|256\|512`, `format?` | `{ bytes, format, width, height, sha256 }` |
|
||||
| `custom<K>` | typed via `CustomOpsMap[K]` | typed via `CustomOpsMap[K]` |
|
||||
|
||||
`MUTATION_OPS = { mkdir, delete, move, write, custom }` — these auto-generate
|
||||
an idempotency key per logical call so transparent retries under the SDK
|
||||
don't produce duplicates.
|
||||
|
||||
## Inline vs streams
|
||||
|
||||
The threshold is `INLINE_THRESHOLD = 256 * 1024` bytes (plaintext). The
|
||||
client's `decideInline()` runs at `write` time:
|
||||
|
||||
* `Uint8Array` / `Blob` with known size → direct comparison.
|
||||
* Bare `ReadableStream` → peek the first chunks; promote to streams as
|
||||
soon as the buffered prefix exceeds the threshold.
|
||||
|
||||
Streams paths kick `shade.upload(...)` with `userMetadata.shadeFilesWriteId`
|
||||
in parallel with the RPC envelope. The server's streams-bridge accepts the
|
||||
inbound transfer immediately into a `TransformStream` and parks the
|
||||
readable side until the matching `write` RPC arrives.
|
||||
|
||||
## Custom ops
|
||||
|
||||
Augment `CustomOpsMap` once globally for type-safe consumer-defined ops:
|
||||
|
||||
```ts
|
||||
declare module '@shade/files' {
|
||||
interface CustomOpsMap {
|
||||
'app.deploy': CustomOpDef<{ jarPath: string }, { deploymentId: string }>;
|
||||
}
|
||||
}
|
||||
|
||||
// Server registers a Zod-backed handler:
|
||||
await shade.files.serve({
|
||||
custom: {
|
||||
'app.deploy': {
|
||||
args: z.object({ jarPath: z.string() }),
|
||||
response: z.object({ deploymentId: z.string() }),
|
||||
handler: async (args, ctx) => ({ deploymentId: deploy(args.jarPath) }),
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
// Client gets typed I/O:
|
||||
const result = await fs.custom('app.deploy', { jarPath: '/mods/foo.jar' });
|
||||
// ^? { deploymentId: string }
|
||||
```
|
||||
|
||||
## Production hooks
|
||||
|
||||
All hooks are callbacks the consumer plugs in — `@shade/files` enforces
|
||||
the *mechanism*; the app owns the *policy*.
|
||||
|
||||
```ts
|
||||
await shade.files.serve({
|
||||
pathPolicy: { rootScope: '/srv/shade-root', forbidTraversal: true },
|
||||
rateLimits: {
|
||||
maxOpsPerMinutePerSender: 600,
|
||||
maxBytesPerHourPerSender: 10 * 1024 ** 3,
|
||||
opCost: { read: 1, write: 5, delete: 3, default: 1 },
|
||||
},
|
||||
idempotency: { ttlMs: 60 * 60 * 1000, maxEntriesPerSender: 10_000 },
|
||||
requireFingerprintVerifiedFor: (ctx) =>
|
||||
['delete', 'write', 'mkdir'].includes(String(ctx.op)) ? 'required' : 'optional',
|
||||
isFingerprintVerified: (sender) => verifiedSet.has(sender),
|
||||
verifySender: async (sender, canonical, sig) => {
|
||||
return ed25519.verify(base64ToBytes(sig), canonical, getPubKey(sender));
|
||||
},
|
||||
onMetric: (name, value, tags) => prometheus.observe(name, value, tags),
|
||||
onError: (err, ctx) => logger.error({ op: ctx.op, sender: ctx.sender }, err),
|
||||
});
|
||||
```
|
||||
|
||||
## React
|
||||
|
||||
```tsx
|
||||
import { ShadeFilesProvider, useFileList } from '@shade/files/react';
|
||||
|
||||
function FileBrowser({ shade, peer }: { shade: Shade; peer: string }) {
|
||||
return (
|
||||
<ShadeFilesProvider shade={shade}>
|
||||
<Listing peer={peer} path="/" />
|
||||
</ShadeFilesProvider>
|
||||
);
|
||||
}
|
||||
|
||||
function Listing({ peer, path }) {
|
||||
const { entries, isLoading, hasMore, loadMore, refresh } = useFileList(peer, path);
|
||||
if (isLoading) return <Spinner />;
|
||||
return (
|
||||
<ul>
|
||||
{entries.map((e) => <li key={e.name}>{e.name} ({e.kind})</li>)}
|
||||
{hasMore && <button onClick={loadMore}>More</button>}
|
||||
</ul>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
`useFileTransfer` exposes a generic state machine for `BulkTransferHandle`s
|
||||
returned by `uploadDirectory()` / `downloadDirectory()`:
|
||||
|
||||
```tsx
|
||||
const { start, abort, state, filesDone, filesTotal, bytesDone } = useFileUpload();
|
||||
const handle = uploadDirectory(fs, localDir, '/uploaded');
|
||||
useEffect(() => { start(handle); return () => void abort(); }, []);
|
||||
```
|
||||
|
||||
## Path safety
|
||||
|
||||
The dispatcher applies `validatePath()` before invoking the user handler:
|
||||
|
||||
1. Length check on raw input.
|
||||
2. Forbidden-bytes check (NUL/CR/LF/DEL/backslash).
|
||||
3. Percent-decode (defends against `%2e%2e` smuggling).
|
||||
4. POSIX normalization.
|
||||
5. `..` traversal rejection.
|
||||
6. Root-scope boundary check.
|
||||
7. Consumer-supplied `extra` predicate.
|
||||
|
||||
The user handler receives the *normalized* path; using `args.path` raw is a
|
||||
TOCTOU bug.
|
||||
|
||||
## Wire compatibility
|
||||
|
||||
`@shade/files` 0.3.0 requires `@shade/proto` 0.3.0+. The proto layer's wire
|
||||
VERSION was bumped from `0x01` to `0x02` to lift the 64 KiB length-prefix
|
||||
ceiling that previously capped ratchet payloads. **Sessions are
|
||||
incompatible across the bump**; both peers must run 0.3.0+.
|
||||
|
||||
## Related modules
|
||||
|
||||
* `@shade/streams` — chunk encryption, lane key derivation. Indirect dep.
|
||||
* `@shade/transfer` — multi-lane transport with HTTP / WS fallback.
|
||||
* `@shade/sdk` — `Shade.files` getter; `BackgroundHooks.onPruneFiles` for
|
||||
retention.
|
||||
Reference in New Issue
Block a user