Shade/docs/files.md

# `@shade/files` — E2EE filesystem RPC

`@shade/files` exposes a typed, end-to-end-encrypted RPC surface for
filesystem-style operations between two Shade peers. Both sides ride a
single Double Ratchet session for control envelopes; large-file content
(`> 256 KiB`) flows through `@shade/transfer` lanes, correlated back to
the RPC by an opaque `userMetadata.shadeFilesWriteId` /
`shadeFilesReadStreamId`.

The package is a **primitive**, not a UI: it ships hooks and clients, not
file managers. Consumers (e.g. Dispatch, Mail, Drive-style apps) keep
their own UI and plug `@shade/files` underneath.

## Quick start

### Server (Bob exposes a filesystem)

```ts
import { createShade } from '@shade/sdk';

const bob = await createShade({ prekeyServer, address: 'bob' });
bob.configureTransfers({ resolveBaseUrl: ... });
Bun.serve({ port: 8080, fetch: (await bob.transferRoute()).fetch });

const stop = await bob.files.serve({
  list: async (ctx) => ({ entries: await readdirAt(ctx.path), hasMore: false }),
  stat: async (ctx) => statAt(ctx.path),
  mkdir: async (ctx) => mkdirAt(ctx.path, ctx.args.recursive),
  delete: async (ctx) => deleteAt(ctx.path, ctx.args.recursive),
  move: async (ctx) => moveAt(ctx.args.src, ctx.args.dst, ctx.args.overwrite),
  read: async (ctx) => readAt(ctx.path),     // returns inline | streams
  write: async (ctx) => writeAt(ctx.args),   // receives inline | streams
  getThumbnail: async (ctx) => thumbnailAt(ctx.path, ctx.args.size, ctx.args.format),
});

// Later: stop the handler.
await stop();
```

### Client (Alice consumes Bob's filesystem)

```ts
const alice = await createShade({ prekeyServer, address: 'alice' });
alice.configureTransfers({ resolveBaseUrl: ... });
Bun.serve({ port: 8081, fetch: (await alice.transferRoute()).fetch });

const fs = await alice.files.client('bob');

const page = await fs.list('/photos');
await fs.write('/photos/cover.png', new Uint8Array(...));   // auto inline/streams
const result = await fs.read('/photos/cover.png');
if (result.kind === 'inline') console.log(result.bytes.byteLength);
else for await (const _chunk of /* result.stream */) { /* ... */ }
```

## HTTP RPC — browser-friendly request-response (4.1+)

The default `shade.files.client(peer)` requires both peers to be mutually
addressable over HTTP — the response to a `list`/`read` etc. round-trips
through `Shade.deliverControlEnvelope`, which POSTs to the peer's
`/v1/transfer/control` endpoint. **That doesn't work for browsers** —
a tab can't host an HTTP server, so the server cannot call back outbound.

`@shade/files` 4.1 ships a parallel **request-response** transport that
lets browser-style clients fully consume the file-RPC surface without
any inbound channel. It mirrors the way `@shade/server`'s
`shade-auth-middleware` works: one POST per RPC, encrypted envelope in
the request body, encrypted response in the same HTTP response.

### Server side — mount the RPC route

```ts
// 1. Register the file handler. `inlineOnly: true` skips the
//    streams-bridge (which would require @shade/transfer).
await shade.files.serve(handlerConfig, { inlineOnly: true });

// 2. Mount the route on your Hono app under any base path.
app.route('/api/v1/shade-files', shade.files.rpcRoute());
//                                         ^^^^^^^^^^^^^^
//                                         POST <base>/rpc
```

`rpcRoute()` accepts:

| Option              | Default | Purpose                                                                                            |
|---------------------|---------|----------------------------------------------------------------------------------------------------|
| `maxBodyBytes`      | 1 MiB   | Max request body. The protocol caps inline payloads at 256 KiB; the headroom is for base64 inflation + custom-op envelopes. |
| `acceptFirstMessage`| `false` | Accept `0x01` PreKeyMessage envelopes — required when the RPC route also doubles as the X3DH handshake (browser's first-ever request). |

### Browser client

```ts
import { createShade } from '@shade/sdk';

const shade = await createShade({
  prekeyServer: 'https://shade.example.com',
  storage: 'memory',
  address: 'alice@example.com',
});

const fs = shade.files.httpClient('bob@example.com', {
  rpcUrl: 'https://dispatch.example.com/api/v1/shade-files/rpc',
  // Optional: thread CSRF / auth tokens, override fetch, etc.
  headers: { 'X-CSRF-Token': csrfToken },
});

await fs.mkdir('/photos');
await fs.write('/photos/cover.png', new Uint8Array([/* ... */]), {
  contentType: 'image/png',
});
const result = await fs.read('/photos/cover.png');
```

### What works in HTTP-RPC mode

- `list`, `stat`, `mkdir`, `delete`, `move`, `getThumbnail`, `custom<K>` — full parity.
- `write` — **inline only** (≤ 256 KiB plaintext). Larger inputs throw `ConflictError`.
- `read` — **inline only**. If the server returns a streamed `read` result, the client throws `InternalFileError` directing callers to the stateful pathway.

### Wire contract

```
POST /rpc HTTP/1.1
Content-Type: application/octet-stream
X-Shade-Sender-Address: alice@example.com

<wire-encoded ShadeEnvelope (0x01 first-time, 0x02 after) wrapping
 JSON-encoded RpcRequest>

────

200 OK
Content-Type: application/octet-stream

<wire-encoded ShadeEnvelope (0x02) wrapping JSON-encoded RpcResponse | RpcError>
```

Transport-level failures (no session, undecryptable envelope, body too
big) return JSON `{ "error": "..." }` with appropriate 4xx status.
Application-level failures (file not found, permission denied) ship
encrypted `RpcError` envelopes — the client maps them back to typed
`FileError` subclasses (`NotFoundError`, `ConflictError`, etc.).

### Symmetry with `@shade/server`

The shape mirrors `@shade/server`'s shade-auth-middleware: encrypted
envelope rides the request body, server decrypts via the existing
ratchet session, performs the protected operation, returns an encrypted
envelope in the response. No bidirectional channel required, no
WebSocket, no SSE.

### When to use which

| Setup                                         | Use                                           |
|-----------------------------------------------|-----------------------------------------------|
| Browser client ↔ Bun/Hono server              | `httpClient()` + `rpcRoute()`                 |
| Server ↔ server (both can host HTTP)          | `client()` (default) — supports streams       |
| Service-worker / extension ↔ server           | `httpClient()` (no inbound listener)          |
| CLI / daemon ↔ daemon                         | Either; `client()` if you need streams        |

## Op surface

| Op             | Args                                     | Result                                  |
|----------------|------------------------------------------|-----------------------------------------|
| `list`         | `path`, `cursor?`, `pageSize?`, `filter?`| `{ entries, hasMore, nextCursor? }`     |
| `stat`         | `path`                                   | `FileEntry`                             |
| `mkdir`        | `path`, `recursive?`                     | `{ entry: FileEntry }`                  |
| `delete`       | `path`, `recursive?`                     | `{ deletedCount }`                      |
| `move`         | `src`, `dst`, `overwrite?`               | `{ entry: FileEntry }`                  |
| `read`         | `path`, `range?`, `preferInline?`        | inline `{ bytes, sha256, size }` or streams `{ stream, size, sha256 }` |
| `write`        | `path`, `Uint8Array \| Blob \| Stream`   | `{ entry: FileEntry }`                  |
| `getThumbnail` | `path`, `size: 64\|128\|256\|512`, `format?` | `{ bytes, format, width, height, sha256 }` |
| `custom<K>`    | typed via `CustomOpsMap[K]`              | typed via `CustomOpsMap[K]`             |

`MUTATION_OPS = { mkdir, delete, move, write, custom }` — these auto-generate
an idempotency key per logical call so transparent retries under the SDK
don't produce duplicates.

## Inline vs streams

The threshold is `INLINE_THRESHOLD = 256 * 1024` bytes (plaintext). The
client's `decideInline()` runs at `write` time:

* `Uint8Array` / `Blob` with known size → direct comparison.
* Bare `ReadableStream` → peek the first chunks; promote to streams as
  soon as the buffered prefix exceeds the threshold.

Streams paths kick `shade.upload(...)` with `userMetadata.shadeFilesWriteId`
in parallel with the RPC envelope. The server's streams-bridge accepts the
inbound transfer immediately into a `TransformStream` and parks the
readable side until the matching `write` RPC arrives.

## Custom ops

Augment `CustomOpsMap` once globally for type-safe consumer-defined ops:

```ts
declare module '@shade/files' {
  interface CustomOpsMap {
    'app.deploy': CustomOpDef<{ jarPath: string }, { deploymentId: string }>;
  }
}

// Server registers a Zod-backed handler:
await shade.files.serve({
  custom: {
    'app.deploy': {
      args: z.object({ jarPath: z.string() }),
      response: z.object({ deploymentId: z.string() }),
      handler: async (args, ctx) => ({ deploymentId: deploy(args.jarPath) }),
    },
  },
});

// Client gets typed I/O:
const result = await fs.custom('app.deploy', { jarPath: '/mods/foo.jar' });
//    ^? { deploymentId: string }
```

## Production hooks

All hooks are callbacks the consumer plugs in — `@shade/files` enforces
the *mechanism*; the app owns the *policy*.

```ts
await shade.files.serve({
  pathPolicy: { rootScope: '/srv/shade-root', forbidTraversal: true },
  rateLimits: {
    maxOpsPerMinutePerSender: 600,
    maxBytesPerHourPerSender: 10 * 1024 ** 3,
    opCost: { read: 1, write: 5, delete: 3, default: 1 },
  },
  idempotency: { ttlMs: 60 * 60 * 1000, maxEntriesPerSender: 10_000 },
  requireFingerprintVerifiedFor: (ctx) =>
    ['delete', 'write', 'mkdir'].includes(String(ctx.op)) ? 'required' : 'optional',
  isFingerprintVerified: (sender) => verifiedSet.has(sender),
  verifySender: async (sender, canonical, sig) => {
    return ed25519.verify(base64ToBytes(sig), canonical, getPubKey(sender));
  },
  onMetric: (name, value, tags) => prometheus.observe(name, value, tags),
  onError: (err, ctx) => logger.error({ op: ctx.op, sender: ctx.sender }, err),
});
```

## React

```tsx
import { ShadeFilesProvider, useFileList } from '@shade/files/react';

function FileBrowser({ shade, peer }: { shade: Shade; peer: string }) {
  return (
    <ShadeFilesProvider shade={shade}>
      <Listing peer={peer} path="/" />
    </ShadeFilesProvider>
  );
}

function Listing({ peer, path }) {
  const { entries, isLoading, hasMore, loadMore, refresh } = useFileList(peer, path);
  if (isLoading) return <Spinner />;
  return (
    <ul>
      {entries.map((e) => <li key={e.name}>{e.name} ({e.kind})</li>)}
      {hasMore && <button onClick={loadMore}>More</button>}
    </ul>
  );
}
```

`useFileTransfer` exposes a generic state machine for `BulkTransferHandle`s
returned by `uploadDirectory()` / `downloadDirectory()`:

```tsx
const { start, abort, state, filesDone, filesTotal, bytesDone } = useFileUpload();
const handle = uploadDirectory(fs, localDir, '/uploaded');
useEffect(() => { start(handle); return () => void abort(); }, []);
```

## Path safety

The dispatcher applies `validatePath()` before invoking the user handler:

1. Length check on raw input.
2. Forbidden-bytes check (NUL/CR/LF/DEL/backslash).
3. Percent-decode (defends against `%2e%2e` smuggling).
4. POSIX normalization.
5. `..` traversal rejection.
6. Root-scope boundary check.
7. Consumer-supplied `extra` predicate.

The user handler receives the *normalized* path; using `args.path` raw is a
TOCTOU bug.

## Wire compatibility

`@shade/files` 0.3.0 requires `@shade/proto` 0.3.0+. The proto layer's wire
VERSION was bumped from `0x01` to `0x02` to lift the 64 KiB length-prefix
ceiling that previously capped ratchet payloads. **Sessions are
incompatible across the bump**; both peers must run 0.3.0+.

## Rich file metadata + previews (V3.9)

`stream-init` carries optional E2EE `fileMetadata` (filename, MIME,
thumbnail-stream pointer). `@shade/files` consumers see this on the
incoming-transfer side and can render previews via `<TransferRow
showThumbnail />`. The thumbnail itself rides as a separate AEAD
stream — server never sees preview pixels in plaintext.

See [streams.md § Rich file metadata + previews](streams.md#rich-file-metadata--previews-v39)
for the wire format, format-hardening rules, and renderer trust
model. The pattern integrates seamlessly with `@shade/files`'s own
write/read RPCs — pass `fileMetadata` in the underlying
`shade.upload` and the same `ShadeThumbnailCache` powers previews
across all transfer surfaces.

## Related modules

* `@shade/streams` — chunk encryption, lane key derivation. Indirect dep.
* `@shade/transfer` — multi-lane transport with HTTP / WS fallback.
* `@shade/transport-webrtc` (V3.11, optional) — direct P2P chunk
  delivery via `RTCDataChannel`; large `read`/`write` payloads
  automatically prefer WebRTC when both peers have called
  `shade.configureWebRTC()`.
* `@shade/sdk` — `Shade.files` getter; `BackgroundHooks.onPruneFiles` for
  retention.