Files
Shade/docs/shade-overview.html
Sterister fa770d3063
Some checks failed
Test / test (push) Has been cancelled
feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive
M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to
ship. Apps keep their own UI; this layer ships the typed RPC, the
streams bridge for content I/O, and production hooks (rate limit,
retention, fingerprint gate, metrics).

@shade/files (NEW)
- Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with
  Zod-validated wire schemas + clean user-handler types.
- Custom ops: typed via TypeScript declaration merging on CustomOpsMap
  + per-op Zod schemas; client.custom('app.foo', {...}) is fully typed.
- Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams
  (> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId
  / shadeFilesReadStreamId correlation. Server-side TransformStream
  bridges accept inbound transfers immediately (engine rejects chunks
  that arrive before accept) and park the readable for the matching
  RPC.
- Directory ops: walk(path, opts) async-iterable depth-first walker;
  uploadDirectory()/downloadDirectory() with bounded concurrency pool
  (default 4, cap 16), aggregated progress, abort.
- Production hooks (callback-based, vendor-neutral): rate-limit (op +
  byte), idempotency cache (LRU + TTL + in-flight de-dupe), path
  policy (traversal + percent-decode hardening), fingerprint gate
  (required/optional/reject), pluggable Ed25519 sig verification with
  ±5 min replay window, onMetric sink (standard names).
- React hooks (subpath @shade/files/react): ShadeFilesProvider,
  useShadeFiles, useFileList, useFileTransfer/Upload/Download.
- Shade.files.serve(handler) + Shade.files.client(peer) high-level
  entrypoint in @shade/sdk; lazy + memoized; one handler per Shade.

Wire format bump
- @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from
  u16 to u32. The previous u16 silently truncated payloads above
  64 KiB — a hard correctness ceiling that blocked inline file ops
  up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions
  only. Cross-platform Kotlin port (android/shade-android) updated to
  match; test-vectors/wire-format.json regenerated.

Concurrency safety
- ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex.
  Concurrent decryptions of the same peer raced ratchet state
  (manifested as sporadic "Failed to decrypt — wrong key or tampered
  data" under load — surfaced once concurrent uploadDirectory pumped
  many writes in flight). Encrypt was already serialized via
  Shade.send's encryptChains; decrypt is now serialized at the
  manager layer too.

@shade/streams extension
- StreamMetadata.userMetadata?: Record<string, string> for
  application-level key/value pairs that round-trip verbatim through
  stream-init plaintext. Used by @shade/files for write/read
  correlation; available to any consumer.

@shade/sdk extension
- Shade.files getter (lazy + memoized).
- BackgroundHooks.onPruneFiles + periodic timer (default 5 min) +
  BackgroundTasks.setHook(name, fn) for runtime hook registration.

Bundles in-flight 0.2.0 work
- packages/shade-streams/, packages/shade-transfer/, related
  shade-sdk streams-bridge + shade-widgets transfer hooks were
  uncommitted prior to this session. Including them keeps the
  workspace consistent at 0.3.0 since @shade/files depends on them.

Tests
- 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail;
  3× stable). Coverage spans unit (inline-threshold + concurrency),
  integration (read-write inline + streams up to 1 MiB, walk +
  upload/download directory, custom-op, metrics, SDK namespace
  end-to-end), and security (tampered-envelope sig verification,
  replay window, fingerprint gate, rate-limit + quota).

Release artifacts
- All packages bumped to 0.3.0 via scripts/bump-version.ts.
- scripts/publish-all.ts PACKAGES updated with shade-files in
  topological order (after shade-transfer, before shade-sdk).
- bun run publish:dry clean (14 packed, 0 failed).
- examples/08-files-browser/ — three-process CLI demo (prekey + Bob
  server + Alice CLI) covering list/stat/mkdir/delete/upload/download.
- docs/files.md — full API + design doc.
- CHANGELOG.md 0.3.0 entry.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00

619 lines
21 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Shade — end-to-end encryption as a module</title>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link href="https://fonts.googleapis.com/css2?family=Fraunces:ital,opsz,wght@0,9..144,400;0,9..144,600;0,9..144,700;1,9..144,400&family=Source+Serif+4:ital,opsz,wght@0,8..60,400;0,8..60,600;1,8..60,400&display=swap" rel="stylesheet" />
<style>
:root {
--bg: #0f1210;
--bg-elevated: #181d1a;
--surface: #1f2622;
--border: #2d3832;
--text: #e8ebe6;
--muted: #8a9a8f;
--accent: #6ee7a8;
--accent-dim: #3d8f63;
--warning: #f0c674;
--font-display: "Fraunces", Georgia, serif;
--font-body: "Source Serif 4", Georgia, serif;
--radius: 10px;
--shadow: 0 12px 40px rgba(0, 0, 0, 0.45);
}
* {
box-sizing: border-box;
}
html {
scroll-behavior: smooth;
}
body {
margin: 0;
min-height: 100vh;
font-family: var(--font-body);
font-size: 1.05rem;
line-height: 1.65;
color: var(--text);
background:
radial-gradient(ellipse 120% 80% at 50% -20%, rgba(110, 231, 168, 0.08), transparent 50%),
var(--bg);
}
.wrap {
max-width: 720px;
margin: 0 auto;
padding: 2.5rem 1.25rem 4rem;
}
header {
margin-bottom: 2.5rem;
padding-bottom: 2rem;
border-bottom: 1px solid var(--border);
}
h1 {
font-family: var(--font-display);
font-size: clamp(2rem, 5vw, 2.75rem);
font-weight: 700;
letter-spacing: -0.02em;
margin: 0 0 0.75rem;
line-height: 1.15;
}
.lede {
font-size: 1.15rem;
color: var(--muted);
margin: 0;
max-width: 38ch;
}
.badge-row {
display: flex;
flex-wrap: wrap;
gap: 0.5rem;
margin-top: 1.25rem;
}
.badge {
font-family: var(--font-display);
font-size: 0.75rem;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.06em;
padding: 0.35rem 0.65rem;
border-radius: 999px;
background: var(--surface);
border: 1px solid var(--border);
color: var(--accent);
}
h2 {
font-family: var(--font-display);
font-size: 1.35rem;
font-weight: 600;
margin: 2.25rem 0 0.75rem;
color: var(--text);
}
h3 {
font-family: var(--font-display);
font-size: 1.05rem;
font-weight: 600;
margin: 1.5rem 0 0.5rem;
}
p {
margin: 0 0 1rem;
}
a {
color: var(--accent);
text-decoration-thickness: 1px;
text-underline-offset: 3px;
}
a:hover {
color: #9af0c4;
}
.callout {
background: var(--bg-elevated);
border: 1px solid var(--border);
border-radius: var(--radius);
padding: 1rem 1.15rem;
margin: 1.25rem 0;
box-shadow: var(--shadow);
}
.callout strong {
color: var(--accent);
font-weight: 600;
}
/* Tabs */
.tabs {
display: flex;
flex-wrap: wrap;
gap: 0.35rem;
margin: 1.5rem 0 0;
border-bottom: 1px solid var(--border);
padding-bottom: 0;
}
.tab-btn {
font-family: var(--font-display);
font-size: 0.9rem;
padding: 0.6rem 1rem;
border: 1px solid transparent;
border-bottom: none;
border-radius: var(--radius) var(--radius) 0 0;
background: transparent;
color: var(--muted);
cursor: pointer;
transition: color 0.15s, background 0.15s, border-color 0.15s;
}
.tab-btn:hover {
color: var(--text);
background: rgba(110, 231, 168, 0.06);
}
.tab-btn[aria-selected="true"] {
color: var(--text);
background: var(--surface);
border-color: var(--border);
border-bottom-color: var(--surface);
margin-bottom: -1px;
position: relative;
z-index: 1;
}
.tab-panel {
display: none;
background: var(--surface);
border: 1px solid var(--border);
border-top: none;
border-radius: 0 0 var(--radius) var(--radius);
padding: 1.25rem 1.25rem 1.1rem;
margin-bottom: 1.5rem;
}
.tab-panel.active {
display: block;
}
.tab-panel ul {
margin: 0.5rem 0 0;
padding-left: 1.2rem;
}
.tab-panel li {
margin-bottom: 0.35rem;
}
code, .mono {
font-family: ui-monospace, "Cascadia Code", "Source Code Pro", monospace;
font-size: 0.88em;
background: rgba(0, 0, 0, 0.35);
padding: 0.12em 0.35em;
border-radius: 4px;
}
/* Accordion */
.accordion {
margin: 1.5rem 0;
border: 1px solid var(--border);
border-radius: var(--radius);
overflow: hidden;
background: var(--bg-elevated);
}
.acc-item + .acc-item {
border-top: 1px solid var(--border);
}
.acc-trigger {
width: 100%;
text-align: left;
font-family: var(--font-display);
font-size: 1rem;
font-weight: 600;
padding: 1rem 1.15rem;
background: transparent;
color: var(--text);
border: none;
cursor: pointer;
display: flex;
align-items: center;
justify-content: space-between;
gap: 0.75rem;
}
.acc-trigger:hover {
background: rgba(110, 231, 168, 0.05);
}
.acc-trigger::after {
content: "▼";
font-size: 0.55rem;
color: var(--accent);
transition: transform 0.2s;
}
.acc-trigger[aria-expanded="false"]::after {
transform: rotate(-90deg);
}
.acc-panel {
padding: 0 1.15rem 1rem;
color: var(--muted);
}
.acc-panel[hidden] {
display: none;
}
/* Flow simulator */
.flow {
margin: 2rem 0;
padding: 1.25rem;
background: var(--surface);
border-radius: var(--radius);
border: 1px solid var(--border);
}
.flow h3 {
margin-top: 0;
font-family: var(--font-display);
}
.flow-steps {
display: grid;
gap: 0.65rem;
margin: 1rem 0;
}
.flow-step {
display: grid;
grid-template-columns: auto 1fr;
gap: 0.75rem;
align-items: start;
padding: 0.65rem 0.75rem;
border-radius: 8px;
background: var(--bg);
border: 1px solid transparent;
opacity: 0.35;
transition: opacity 0.25s, border-color 0.25s, box-shadow 0.25s;
}
.flow-step.active {
opacity: 1;
border-color: var(--accent-dim);
box-shadow: 0 0 0 1px rgba(110, 231, 168, 0.15);
}
.flow-num {
font-family: var(--font-display);
font-weight: 700;
font-size: 0.85rem;
width: 1.75rem;
height: 1.75rem;
display: flex;
align-items: center;
justify-content: center;
border-radius: 50%;
background: var(--accent-dim);
color: var(--bg);
}
.flow-step.active .flow-num {
background: var(--accent);
color: var(--bg);
}
.flow-actions {
display: flex;
flex-wrap: wrap;
gap: 0.5rem;
margin-top: 1rem;
}
.btn {
font-family: var(--font-display);
font-size: 0.9rem;
font-weight: 600;
padding: 0.55rem 1.1rem;
border-radius: 8px;
border: 1px solid var(--accent-dim);
background: rgba(110, 231, 168, 0.12);
color: var(--accent);
cursor: pointer;
}
.btn:hover {
background: rgba(110, 231, 168, 0.22);
}
.btn-secondary {
border-color: var(--border);
background: transparent;
color: var(--muted);
}
.btn-secondary:hover {
color: var(--text);
border-color: var(--muted);
}
footer {
margin-top: 3rem;
padding-top: 1.5rem;
border-top: 1px solid var(--border);
font-size: 0.9rem;
color: var(--muted);
}
</style>
</head>
<body>
<div class="wrap">
<header>
<h1>Shade</h1>
<p class="lede">
A reusable module for <strong style="color: var(--text); font-weight: 600;">end-to-end encrypted</strong> communication in your own apps — using the same kind of protocol as Signal.
</p>
<div class="badge-row">
<span class="badge">X3DH</span>
<span class="badge">Double Ratchet</span>
<span class="badge">TypeScript</span>
<span class="badge">Platform-agnostic crypto</span>
</div>
</header>
<section id="what">
<h2>What does the project do?</h2>
<p>
<strong>Shade</strong> is a monorepo that implements secure messaging encryption between two parties (for example browser and server, or two clients). Messages are encrypted so the transport layer (HTTP, WebSocket, etc.) only sees opaque bytes — not the content.
</p>
<div class="callout">
<strong>Core idea:</strong> Embed <code>ShadeSessionManager</code> (from <code>@shade/core</code>) together with a <code>CryptoProvider</code> (e.g. Web Crypto in the browser/Bun) and storage. Then call <code>encrypt</code> / <code>decrypt</code> per peer, just like in the demo code <code>demo.ts</code>.
</div>
<p>
The first message to someone new performs key agreement (X3DH). After that each message uses the <em>Double Ratchet</em>: fresh message keys and periodic DH steps provide <strong>forward secrecy</strong> (past messages do not survive key compromise) and <strong>post-compromise security</strong> (the system “recovers” over time after a compromise).
</p>
</section>
<section id="packages">
<h2>Packages (how they fit)</h2>
<div class="tabs" role="tablist" aria-label="Package overview">
<button type="button" class="tab-btn" role="tab" id="tab-core" aria-selected="true" aria-controls="panel-core">shade-core</button>
<button type="button" class="tab-btn" role="tab" id="tab-crypto" aria-selected="false" aria-controls="panel-crypto">shade-crypto-web</button>
<button type="button" class="tab-btn" role="tab" id="tab-proto" aria-selected="false" aria-controls="panel-proto">shade-proto</button>
<button type="button" class="tab-btn" role="tab" id="tab-transport" aria-selected="false" aria-controls="panel-transport">shade-transport</button>
<button type="button" class="tab-btn" role="tab" id="tab-server" aria-selected="false" aria-controls="panel-server">shade-server</button>
</div>
<div id="panel-core" class="tab-panel active" role="tabpanel" aria-labelledby="tab-core">
<p style="margin-top:0"><strong>The protocol.</strong> X3DH, Double Ratchet, session shapes, errors. No platform crypto here — only the <code>CryptoProvider</code> interface.</p>
<ul>
<li><code>ShadeSessionManager</code> — high-level API: <code>initialize</code>, <code>createPreKeyBundle</code>, <code>initSessionFromBundle</code>, <code>encrypt</code>, <code>decrypt</code></li>
<li>Symmetric encryption: <strong>AES-256-GCM</strong> with AAD from the ratchet header</li>
</ul>
</div>
<div id="panel-crypto" class="tab-panel" role="tabpanel" aria-labelledby="tab-crypto" hidden>
<p style="margin-top:0"><strong>Crypto implementation for web/Bun/Node</strong> via SubtleCrypto — X25519, Ed25519, HKDF, HMAC, random bytes.</p>
<ul>
<li>Lets you use <code>shade-core</code> in the browser and on servers that support Web Crypto</li>
<li>Comments in source point toward future Android (e.g. Tink) as a separate provider</li>
</ul>
</div>
<div id="panel-proto" class="tab-panel" role="tabpanel" aria-labelledby="tab-proto" hidden>
<p style="margin-top:0"><strong>Binary wire format</strong> for messages: version + type + length-prefixed fields (big-endian).</p>
<ul>
<li>Type <code>0x01</code> = PreKeyMessage, <code>0x02</code> = RatchetMessage</li>
<li>Use when you want to serialize <code>ShadeEnvelope</code> efficiently on the wire</li>
</ul>
</div>
<div id="panel-transport" class="tab-panel" role="tabpanel" aria-labelledby="tab-transport" hidden>
<p style="margin-top:0"><strong>Transport adapters</strong> — not encryption itself, but how you move bytes (e.g. fetch or WebSocket).</p>
<ul>
<li>Hooks your application to the channel you already use</li>
</ul>
</div>
<div id="panel-server" class="tab-panel" role="tabpanel" aria-labelledby="tab-server" hidden>
<p style="margin-top:0"><strong>Prekey server (Hono)</strong> — stores public keys so Alice can start a conversation while Bob is “offline”.</p>
<ul>
<li><code>POST /v1/keys/register</code> — register identity + bundle</li>
<li><code>GET /v1/keys/bundle/:address</code> — fetch bundle (consumes one-time prekey when available)</li>
<li><code>POST /v1/keys/replenish</code> — replenish one-time prekeys</li>
</ul>
</div>
</section>
<section id="keys-in-brief">
<h2>Keys at a glance</h2>
<div class="accordion" id="key-acc">
<div class="acc-item">
<button type="button" class="acc-trigger" aria-expanded="true" aria-controls="acc-identity" id="btn-identity">
Identity key (long-term)
</button>
<div class="acc-panel" id="acc-identity" role="region" aria-labelledby="btn-identity">
<strong>Ed25519</strong> signs the “signed prekey”. <strong>X25519</strong> is used in DiffieHellman in X3DH and in the ratchet. One identity per device/user is typical.
</div>
</div>
<div class="acc-item">
<button type="button" class="acc-trigger" aria-expanded="false" aria-controls="acc-spk" id="btn-spk">
Signed prekey (medium-lived, rotated)
</button>
<div class="acc-panel" id="acc-spk" role="region" aria-labelledby="btn-spk" hidden>
An X25519 key that is published and signed by the identity. The recipient verifies the signature before DH. The codebase recommends rotation on the order of 17 days.
</div>
</div>
<div class="acc-item">
<button type="button" class="acc-trigger" aria-expanded="false" aria-controls="acc-otpk" id="btn-otpk">
One-time prekeys
</button>
<div class="acc-panel" id="acc-otpk" role="region" aria-labelledby="btn-otpk" hidden>
Optional but important for stronger security: each fetched bundle can include one one-time key that is removed after use (<code>processPreKeyMessage</code> clears it from storage). Improves protection against certain attacks when many clients connect to the same recipient.
</div>
</div>
</div>
</section>
<section id="flow-demo">
<h2>Interactive flow: zero to encrypted message</h2>
<p>
Click “Next step” to walk through the sequence as Shade builds it. This mirrors <code>ShadeSessionManager</code> and the demo in the repo.
</p>
<div class="flow">
<h3>Sessions and messages</h3>
<div class="flow-steps" id="flow-steps"></div>
<div class="flow-actions">
<button type="button" class="btn" id="flow-next">Next step</button>
<button type="button" class="btn btn-secondary" id="flow-reset">Start over</button>
</div>
</div>
</section>
<section id="x3dh-ratchet">
<h2>X3DH and Double Ratchet (brief)</h2>
<p>
<strong>X3DH</strong> solves “I want to talk to Bob now, but Bob may not reply until later”. Bob publishes a <em>prekey bundle</em> on the server. Alice fetches it, runs 3 or 4 DH operations (depending on whether a one-time key is used), and derives a shared root key both parties can reconstruct — without the server learning the secret.
</p>
<p>
<strong>Double Ratchet</strong> uses that root as a starting point. For each message (or when new DH keys spin), keys are derived; on the wire payloads are AES-GCM with authentication (AAD binds ciphertext to the ratchet header). The protocol also tolerates messages arriving out of order within limits (<code>MAX_SKIP</code>).
</p>
<p>
Signal specifications (English): <a href="https://signal.org/docs/specifications/x3dh/" target="_blank" rel="noopener">X3DH</a> · <a href="https://signal.org/docs/specifications/doubleratchet/" target="_blank" rel="noopener">Double Ratchet</a>.
</p>
</section>
<section id="reuse">
<h2>Using Shade across projects</h2>
<p>
Treat Shade as three layers you combine as needed:
</p>
<ol>
<li><strong>Core + crypto provider + storage</strong> — the E2EE engine itself (runs in a client or a server process that must decrypt).</li>
<li><strong>Proto</strong> — when you want compact binary serialization.</li>
<li><strong>Transport + prekey server</strong> — when you want standardized key discovery and channels.</li>
</ol>
<p>
Reference path: <code class="mono">bun demo.ts</code> from the repo root shows a frontend/backend flow with memory storage and real crypto primitives.
</p>
</section>
<footer>
<p>Shade — overview written as static HTML in <code class="mono">docs/shade-overview.html</code>. Open the file directly in the browser or serve it as static assets.</p>
</footer>
</div>
<script>
(function () {
// Tabs
var tabBtns = document.querySelectorAll(".tab-btn");
var panels = document.querySelectorAll(".tab-panel");
tabBtns.forEach(function (btn) {
btn.addEventListener("click", function () {
var id = btn.getAttribute("aria-controls");
tabBtns.forEach(function (b) {
b.setAttribute("aria-selected", b === btn ? "true" : "false");
});
panels.forEach(function (p) {
var on = p.id === id;
p.classList.toggle("active", on);
p.hidden = !on;
});
});
});
// Accordion (single-open for clarity)
var triggers = document.querySelectorAll(".acc-trigger");
triggers.forEach(function (t) {
t.addEventListener("click", function () {
var exp = t.getAttribute("aria-expanded") === "true";
var panelId = t.getAttribute("aria-controls");
var panel = document.getElementById(panelId);
triggers.forEach(function (other) {
var oid = other.getAttribute("aria-controls");
var op = document.getElementById(oid);
other.setAttribute("aria-expanded", "false");
if (op) {
op.hidden = true;
}
});
if (!exp) {
t.setAttribute("aria-expanded", "true");
if (panel) panel.hidden = false;
} else {
t.setAttribute("aria-expanded", "false");
if (panel) panel.hidden = true;
}
});
});
// Flow steps
var steps = [
{
title: "Initialize client",
body: "Call initialize(): load or generate the identity keys (Ed25519 + X25519), registrationId, and signed prekey.",
},
{
title: "Publish prekey bundle",
body: "Build a bundle with createPreKeyBundle() / generateOneTimePreKeys() and register it on the prekey server (or share out-of-band for a demo).",
},
{
title: "Start session with peer",
body: "Fetch the peer bundle, run initSessionFromBundle(address, bundle). X3DH runs and the ratchet session is stored in StorageProvider.",
},
{
title: "First encrypt",
body: "encrypt() returns a ShadeEnvelope of type 'prekey': includes ephemeral keys, prekey IDs, and the first RatchetMessage (AES-GCM).",
},
{
title: "Peer decrypt",
body: "decrypt() on PreKeyMessage: restores the same root key, initReceiverSession, ratchetDecrypt — plaintext out.",
},
{
title: "Further messages",
body: "The next encrypt() calls yield type 'ratchet'. DH ratchet steps rotate chains and improve security over time.",
},
];
var container = document.getElementById("flow-steps");
var idx = 0;
function renderFlow() {
container.innerHTML = "";
steps.forEach(function (s, i) {
var row = document.createElement("div");
row.className = "flow-step" + (i <= idx ? " active" : "");
row.innerHTML =
'<span class="flow-num">' + (i + 1) + "</span>" +
"<div><strong style=\"color:var(--text)\">" + s.title + "</strong><br />" + s.body + "</div>";
container.appendChild(row);
});
}
document.getElementById("flow-next").addEventListener("click", function () {
idx = Math.min(idx + 1, steps.length - 1);
renderFlow();
});
document.getElementById("flow-reset").addEventListener("click", function () {
idx = 0;
renderFlow();
});
renderFlow();
})();
</script>
</body>
</html>