Files
Shade/packages/shade-sdk/src/backup.ts

235 lines
7.2 KiB
TypeScript
Raw Normal View History

feat(files): @shade/files 0.3.0 — E2EE filesystem RPC primitive M-Files-1..6 land the full files-RPC layer + everything 0.3.0 needs to ship. Apps keep their own UI; this layer ships the typed RPC, the streams bridge for content I/O, and production hooks (rate limit, retention, fingerprint gate, metrics). @shade/files (NEW) - Standard ops: list/stat/mkdir/delete/move/read/write/getThumbnail with Zod-validated wire schemas + clean user-handler types. - Custom ops: typed via TypeScript declaration merging on CustomOpsMap + per-op Zod schemas; client.custom('app.foo', {...}) is fully typed. - Content I/O: inline (≤ 256 KiB plaintext) base64-in-RPC; streams (> 256 KiB) ride @shade/transfer via userMetadata.shadeFilesWriteId / shadeFilesReadStreamId correlation. Server-side TransformStream bridges accept inbound transfers immediately (engine rejects chunks that arrive before accept) and park the readable for the matching RPC. - Directory ops: walk(path, opts) async-iterable depth-first walker; uploadDirectory()/downloadDirectory() with bounded concurrency pool (default 4, cap 16), aggregated progress, abort. - Production hooks (callback-based, vendor-neutral): rate-limit (op + byte), idempotency cache (LRU + TTL + in-flight de-dupe), path policy (traversal + percent-decode hardening), fingerprint gate (required/optional/reject), pluggable Ed25519 sig verification with ±5 min replay window, onMetric sink (standard names). - React hooks (subpath @shade/files/react): ShadeFilesProvider, useShadeFiles, useFileList, useFileTransfer/Upload/Download. - Shade.files.serve(handler) + Shade.files.client(peer) high-level entrypoint in @shade/sdk; lazy + memoized; one handler per Shade. Wire format bump - @shade/proto wire VERSION 0x01 → 0x02. Length prefixes changed from u16 to u32. The previous u16 silently truncated payloads above 64 KiB — a hard correctness ceiling that blocked inline file ops up to 256 KiB. Wire-incompatible with 0.2.x peers; new sessions only. Cross-platform Kotlin port (android/shade-android) updated to match; test-vectors/wire-format.json regenerated. Concurrency safety - ShadeSessionManager.encrypt/.decrypt now run under per-peer mutex. Concurrent decryptions of the same peer raced ratchet state (manifested as sporadic "Failed to decrypt — wrong key or tampered data" under load — surfaced once concurrent uploadDirectory pumped many writes in flight). Encrypt was already serialized via Shade.send's encryptChains; decrypt is now serialized at the manager layer too. @shade/streams extension - StreamMetadata.userMetadata?: Record<string, string> for application-level key/value pairs that round-trip verbatim through stream-init plaintext. Used by @shade/files for write/read correlation; available to any consumer. @shade/sdk extension - Shade.files getter (lazy + memoized). - BackgroundHooks.onPruneFiles + periodic timer (default 5 min) + BackgroundTasks.setHook(name, fn) for runtime hook registration. Bundles in-flight 0.2.0 work - packages/shade-streams/, packages/shade-transfer/, related shade-sdk streams-bridge + shade-widgets transfer hooks were uncommitted prior to this session. Including them keeps the workspace consistent at 0.3.0 since @shade/files depends on them. Tests - 74 new tests in @shade/files (572 → 646 workspace pass; 0 fail; 3× stable). Coverage spans unit (inline-threshold + concurrency), integration (read-write inline + streams up to 1 MiB, walk + upload/download directory, custom-op, metrics, SDK namespace end-to-end), and security (tampered-envelope sig verification, replay window, fingerprint gate, rate-limit + quota). Release artifacts - All packages bumped to 0.3.0 via scripts/bump-version.ts. - scripts/publish-all.ts PACKAGES updated with shade-files in topological order (after shade-transfer, before shade-sdk). - bun run publish:dry clean (14 packed, 0 failed). - examples/08-files-browser/ — three-process CLI demo (prekey + Bob server + Alice CLI) covering list/stat/mkdir/delete/upload/download. - docs/files.md — full API + design doc. - CHANGELOG.md 0.3.0 entry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 14:00:01 +02:00
import type { CryptoProvider, StorageProvider } from '@shade/core';
feat(advanced): M-Adv 1-3 — multi-device, backup/restore, group messaging Phase D complete. Shade is now at parity with Signal libsignal's core feature set. M-Adv 1: Multi-device support (simplified Sesame) - DeviceListManager tracks per-user device lists ("user:deviceId" addresses) - fanOutEncrypt() sends one message to all known devices via independent 1:1 Double Ratchet sessions - observeIncoming() auto-registers new devices from received messages - JSON serialization for persistence - userOfDevice/deviceIdOf address parsers M-Adv 2: Backup and restore - @shade/sdk exports BackupBlob format: version + salt + nonce + ciphertext - Passphrase-derived key via HKDF (note: upgrade path to Argon2id documented) - exportBackup()/importBackup() handle identity, prekeys, sessions, trust - backupToString/backupFromString for single-string transport (copy/paste, QR) - shade.exportBackup()/importBackup() convenience methods on SDK - CLI: shade backup export <file> / shade backup restore <file> - Rebuilds manager + transport after restore so ratchet state is consistent M-Adv 3: Group messaging (Sender Keys) - Per-sender chain key + Ed25519 signing key per group - createSenderKey / buildDistribution / installDistribution for key distribution - senderKeyEncrypt advances chain and signs ciphertext+header - senderKeyDecrypt verifies signature then advances the sender's chain - Out-of-order handling with bounded skip - O(1) per message (once distributions are installed) - Defensive ByteArray copies in distribution to prevent zeroize-across-refs 276 tests passing, 0 failures. All 13 SDK/tooling/platform/advanced milestones complete. Shade is feature-complete for v2.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:51:34 +02:00
import {
toBase64,
fromBase64,
serializeIdentityKeyPair,
deserializeIdentityKeyPair,
serializeSignedPreKey,
deserializeSignedPreKey,
serializeOneTimePreKey,
deserializeOneTimePreKey,
serializeSessionState,
deserializeSessionState,
} from '@shade/core';
/**
* Shade backup format v1.
*
* A passphrase-encrypted blob containing the entire local Shade state:
* identity, signed prekeys, one-time prekeys, sessions, trusted identities.
*
* Encryption: AES-256-GCM. Key derivation from passphrase:
* HKDF-SHA256(passphrase, randomSalt, "ShadeBackupKey", 32)
*
* Note: HKDF is NOT a proper password KDF. For a real password-based backup
* you should use Argon2id. This implementation uses HKDF as a simple
* placeholder because CryptoProvider doesn't expose Argon2. Upgrade path
* is to add `argon2id` to CryptoProvider and swap it in here. For now,
* DOCUMENT THIS and require the user to choose a high-entropy passphrase.
*/
const BACKUP_VERSION = 1;
const BACKUP_INFO = new TextEncoder().encode('ShadeBackupKey');
const SALT_SIZE = 32;
export interface BackupPayload {
version: number;
identity: string | null;
registrationId: number;
signedPreKeys: Array<{ keyId: number; data: string }>;
oneTimePreKeys: Array<{ keyId: number; data: string }>;
sessions: Array<{ address: string; state: string }>;
trustedIdentities: Array<{ address: string; key: string }>;
}
export interface BackupBlob {
version: number;
salt: string; // base64
nonce: string; // base64
ciphertext: string; // base64
}
/**
* Export the entire Shade state as an encrypted backup blob.
*
* @param crypto CryptoProvider instance
* @param storage Storage to read from
* @param passphrase User-chosen passphrase (HIGH ENTROPY REQUIRED)
* @param knownAddresses Addresses whose sessions should be included.
* Pass an empty list to back up identity + prekeys only (no sessions).
*/
export async function exportBackup(
crypto: CryptoProvider,
storage: StorageProvider,
passphrase: string,
knownAddresses: string[] = [],
): Promise<BackupBlob> {
if (passphrase.length < 12) {
throw new Error('Passphrase must be at least 12 characters');
}
// Gather state
const identity = await storage.getIdentityKeyPair();
const registrationId = await storage.getLocalRegistrationId();
const signedPreKeys: Array<{ keyId: number; data: string }> = [];
// SignedPreKeys: we can't enumerate them from the interface; try IDs 1-100
for (let i = 1; i <= 100; i++) {
const spk = await storage.getSignedPreKey(i);
if (spk) signedPreKeys.push({ keyId: i, data: serializeSignedPreKey(spk) });
}
const oneTimePreKeys: Array<{ keyId: number; data: string }> = [];
const otpkCount = await storage.getOneTimePreKeyCount();
// Try a reasonable ID range
for (let i = 1; i <= otpkCount + 1000; i++) {
const otpk = await storage.getOneTimePreKey(i);
if (otpk) oneTimePreKeys.push({ keyId: i, data: serializeOneTimePreKey(otpk) });
if (oneTimePreKeys.length >= otpkCount) break;
}
const sessions: Array<{ address: string; state: string }> = [];
for (const address of knownAddresses) {
const state = await storage.getSession(address);
if (state) sessions.push({ address, state: serializeSessionState(state) });
}
// Trusted identities: can't enumerate; caller must provide addresses
const trustedIdentities: Array<{ address: string; key: string }> = [];
const payload: BackupPayload = {
version: BACKUP_VERSION,
identity: identity ? serializeIdentityKeyPair(identity) : null,
registrationId,
signedPreKeys,
oneTimePreKeys,
sessions,
trustedIdentities,
};
const plaintext = new TextEncoder().encode(JSON.stringify(payload));
// Derive encryption key from passphrase via HKDF
const salt = crypto.randomBytes(SALT_SIZE);
const key = await crypto.hkdf(
new TextEncoder().encode(passphrase),
salt,
BACKUP_INFO,
32,
);
const { ciphertext, nonce } = await crypto.aesGcmEncrypt(key, plaintext);
// Zero the derived key after use
crypto.zeroize(key);
return {
version: BACKUP_VERSION,
salt: toBase64(salt),
nonce: toBase64(nonce),
ciphertext: toBase64(ciphertext),
};
}
/**
* Decrypt a backup blob and return its payload without touching storage.
* Useful for peeking at the embedded identity (e.g. to fingerprint it for
* a verification gate) before any state is overwritten.
feat(advanced): M-Adv 1-3 — multi-device, backup/restore, group messaging Phase D complete. Shade is now at parity with Signal libsignal's core feature set. M-Adv 1: Multi-device support (simplified Sesame) - DeviceListManager tracks per-user device lists ("user:deviceId" addresses) - fanOutEncrypt() sends one message to all known devices via independent 1:1 Double Ratchet sessions - observeIncoming() auto-registers new devices from received messages - JSON serialization for persistence - userOfDevice/deviceIdOf address parsers M-Adv 2: Backup and restore - @shade/sdk exports BackupBlob format: version + salt + nonce + ciphertext - Passphrase-derived key via HKDF (note: upgrade path to Argon2id documented) - exportBackup()/importBackup() handle identity, prekeys, sessions, trust - backupToString/backupFromString for single-string transport (copy/paste, QR) - shade.exportBackup()/importBackup() convenience methods on SDK - CLI: shade backup export <file> / shade backup restore <file> - Rebuilds manager + transport after restore so ratchet state is consistent M-Adv 3: Group messaging (Sender Keys) - Per-sender chain key + Ed25519 signing key per group - createSenderKey / buildDistribution / installDistribution for key distribution - senderKeyEncrypt advances chain and signs ciphertext+header - senderKeyDecrypt verifies signature then advances the sender's chain - Out-of-order handling with bounded skip - O(1) per message (once distributions are installed) - Defensive ByteArray copies in distribution to prevent zeroize-across-refs 276 tests passing, 0 failures. All 13 SDK/tooling/platform/advanced milestones complete. Shade is feature-complete for v2.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:51:34 +02:00
*/
export async function decryptBackup(
feat(advanced): M-Adv 1-3 — multi-device, backup/restore, group messaging Phase D complete. Shade is now at parity with Signal libsignal's core feature set. M-Adv 1: Multi-device support (simplified Sesame) - DeviceListManager tracks per-user device lists ("user:deviceId" addresses) - fanOutEncrypt() sends one message to all known devices via independent 1:1 Double Ratchet sessions - observeIncoming() auto-registers new devices from received messages - JSON serialization for persistence - userOfDevice/deviceIdOf address parsers M-Adv 2: Backup and restore - @shade/sdk exports BackupBlob format: version + salt + nonce + ciphertext - Passphrase-derived key via HKDF (note: upgrade path to Argon2id documented) - exportBackup()/importBackup() handle identity, prekeys, sessions, trust - backupToString/backupFromString for single-string transport (copy/paste, QR) - shade.exportBackup()/importBackup() convenience methods on SDK - CLI: shade backup export <file> / shade backup restore <file> - Rebuilds manager + transport after restore so ratchet state is consistent M-Adv 3: Group messaging (Sender Keys) - Per-sender chain key + Ed25519 signing key per group - createSenderKey / buildDistribution / installDistribution for key distribution - senderKeyEncrypt advances chain and signs ciphertext+header - senderKeyDecrypt verifies signature then advances the sender's chain - Out-of-order handling with bounded skip - O(1) per message (once distributions are installed) - Defensive ByteArray copies in distribution to prevent zeroize-across-refs 276 tests passing, 0 failures. All 13 SDK/tooling/platform/advanced milestones complete. Shade is feature-complete for v2.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:51:34 +02:00
crypto: CryptoProvider,
blob: BackupBlob,
passphrase: string,
): Promise<BackupPayload> {
feat(advanced): M-Adv 1-3 — multi-device, backup/restore, group messaging Phase D complete. Shade is now at parity with Signal libsignal's core feature set. M-Adv 1: Multi-device support (simplified Sesame) - DeviceListManager tracks per-user device lists ("user:deviceId" addresses) - fanOutEncrypt() sends one message to all known devices via independent 1:1 Double Ratchet sessions - observeIncoming() auto-registers new devices from received messages - JSON serialization for persistence - userOfDevice/deviceIdOf address parsers M-Adv 2: Backup and restore - @shade/sdk exports BackupBlob format: version + salt + nonce + ciphertext - Passphrase-derived key via HKDF (note: upgrade path to Argon2id documented) - exportBackup()/importBackup() handle identity, prekeys, sessions, trust - backupToString/backupFromString for single-string transport (copy/paste, QR) - shade.exportBackup()/importBackup() convenience methods on SDK - CLI: shade backup export <file> / shade backup restore <file> - Rebuilds manager + transport after restore so ratchet state is consistent M-Adv 3: Group messaging (Sender Keys) - Per-sender chain key + Ed25519 signing key per group - createSenderKey / buildDistribution / installDistribution for key distribution - senderKeyEncrypt advances chain and signs ciphertext+header - senderKeyDecrypt verifies signature then advances the sender's chain - Out-of-order handling with bounded skip - O(1) per message (once distributions are installed) - Defensive ByteArray copies in distribution to prevent zeroize-across-refs 276 tests passing, 0 failures. All 13 SDK/tooling/platform/advanced milestones complete. Shade is feature-complete for v2.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:51:34 +02:00
if (blob.version !== BACKUP_VERSION) {
throw new Error(`Unsupported backup version: ${blob.version}`);
}
const salt = fromBase64(blob.salt);
const nonce = fromBase64(blob.nonce);
const ciphertext = fromBase64(blob.ciphertext);
const key = await crypto.hkdf(
new TextEncoder().encode(passphrase),
salt,
BACKUP_INFO,
32,
);
let plaintext: Uint8Array;
try {
plaintext = await crypto.aesGcmDecrypt(key, ciphertext, nonce);
} catch {
crypto.zeroize(key);
throw new Error('Wrong passphrase or corrupted backup');
}
crypto.zeroize(key);
return JSON.parse(new TextDecoder().decode(plaintext)) as BackupPayload;
}
feat(advanced): M-Adv 1-3 — multi-device, backup/restore, group messaging Phase D complete. Shade is now at parity with Signal libsignal's core feature set. M-Adv 1: Multi-device support (simplified Sesame) - DeviceListManager tracks per-user device lists ("user:deviceId" addresses) - fanOutEncrypt() sends one message to all known devices via independent 1:1 Double Ratchet sessions - observeIncoming() auto-registers new devices from received messages - JSON serialization for persistence - userOfDevice/deviceIdOf address parsers M-Adv 2: Backup and restore - @shade/sdk exports BackupBlob format: version + salt + nonce + ciphertext - Passphrase-derived key via HKDF (note: upgrade path to Argon2id documented) - exportBackup()/importBackup() handle identity, prekeys, sessions, trust - backupToString/backupFromString for single-string transport (copy/paste, QR) - shade.exportBackup()/importBackup() convenience methods on SDK - CLI: shade backup export <file> / shade backup restore <file> - Rebuilds manager + transport after restore so ratchet state is consistent M-Adv 3: Group messaging (Sender Keys) - Per-sender chain key + Ed25519 signing key per group - createSenderKey / buildDistribution / installDistribution for key distribution - senderKeyEncrypt advances chain and signs ciphertext+header - senderKeyDecrypt verifies signature then advances the sender's chain - Out-of-order handling with bounded skip - O(1) per message (once distributions are installed) - Defensive ByteArray copies in distribution to prevent zeroize-across-refs 276 tests passing, 0 failures. All 13 SDK/tooling/platform/advanced milestones complete. Shade is feature-complete for v2.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:51:34 +02:00
/**
* Write a previously-decrypted backup payload to storage. Splitting this
* from `decryptBackup` lets callers run an interstitial verification gate
* (V3.3) before any pinned-trust entries are overwritten.
*/
export async function applyBackupPayload(
storage: StorageProvider,
payload: BackupPayload,
): Promise<void> {
feat(advanced): M-Adv 1-3 — multi-device, backup/restore, group messaging Phase D complete. Shade is now at parity with Signal libsignal's core feature set. M-Adv 1: Multi-device support (simplified Sesame) - DeviceListManager tracks per-user device lists ("user:deviceId" addresses) - fanOutEncrypt() sends one message to all known devices via independent 1:1 Double Ratchet sessions - observeIncoming() auto-registers new devices from received messages - JSON serialization for persistence - userOfDevice/deviceIdOf address parsers M-Adv 2: Backup and restore - @shade/sdk exports BackupBlob format: version + salt + nonce + ciphertext - Passphrase-derived key via HKDF (note: upgrade path to Argon2id documented) - exportBackup()/importBackup() handle identity, prekeys, sessions, trust - backupToString/backupFromString for single-string transport (copy/paste, QR) - shade.exportBackup()/importBackup() convenience methods on SDK - CLI: shade backup export <file> / shade backup restore <file> - Rebuilds manager + transport after restore so ratchet state is consistent M-Adv 3: Group messaging (Sender Keys) - Per-sender chain key + Ed25519 signing key per group - createSenderKey / buildDistribution / installDistribution for key distribution - senderKeyEncrypt advances chain and signs ciphertext+header - senderKeyDecrypt verifies signature then advances the sender's chain - Out-of-order handling with bounded skip - O(1) per message (once distributions are installed) - Defensive ByteArray copies in distribution to prevent zeroize-across-refs 276 tests passing, 0 failures. All 13 SDK/tooling/platform/advanced milestones complete. Shade is feature-complete for v2.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:51:34 +02:00
if (payload.identity) {
await storage.saveIdentityKeyPair(deserializeIdentityKeyPair(payload.identity));
}
await storage.saveLocalRegistrationId(payload.registrationId);
for (const spk of payload.signedPreKeys) {
await storage.saveSignedPreKey(deserializeSignedPreKey(spk.data));
}
for (const otpk of payload.oneTimePreKeys) {
await storage.saveOneTimePreKey(deserializeOneTimePreKey(otpk.data));
}
for (const s of payload.sessions) {
await storage.saveSession(s.address, deserializeSessionState(s.state));
}
for (const t of payload.trustedIdentities) {
await storage.saveTrustedIdentity(t.address, fromBase64(t.key));
}
}
/**
* Import a backup blob, decrypt with the passphrase, and write all state
* to the given storage (overwriting existing state).
*/
export async function importBackup(
crypto: CryptoProvider,
storage: StorageProvider,
blob: BackupBlob,
passphrase: string,
): Promise<void> {
const payload = await decryptBackup(crypto, blob, passphrase);
await applyBackupPayload(storage, payload);
}
feat(advanced): M-Adv 1-3 — multi-device, backup/restore, group messaging Phase D complete. Shade is now at parity with Signal libsignal's core feature set. M-Adv 1: Multi-device support (simplified Sesame) - DeviceListManager tracks per-user device lists ("user:deviceId" addresses) - fanOutEncrypt() sends one message to all known devices via independent 1:1 Double Ratchet sessions - observeIncoming() auto-registers new devices from received messages - JSON serialization for persistence - userOfDevice/deviceIdOf address parsers M-Adv 2: Backup and restore - @shade/sdk exports BackupBlob format: version + salt + nonce + ciphertext - Passphrase-derived key via HKDF (note: upgrade path to Argon2id documented) - exportBackup()/importBackup() handle identity, prekeys, sessions, trust - backupToString/backupFromString for single-string transport (copy/paste, QR) - shade.exportBackup()/importBackup() convenience methods on SDK - CLI: shade backup export <file> / shade backup restore <file> - Rebuilds manager + transport after restore so ratchet state is consistent M-Adv 3: Group messaging (Sender Keys) - Per-sender chain key + Ed25519 signing key per group - createSenderKey / buildDistribution / installDistribution for key distribution - senderKeyEncrypt advances chain and signs ciphertext+header - senderKeyDecrypt verifies signature then advances the sender's chain - Out-of-order handling with bounded skip - O(1) per message (once distributions are installed) - Defensive ByteArray copies in distribution to prevent zeroize-across-refs 276 tests passing, 0 failures. All 13 SDK/tooling/platform/advanced milestones complete. Shade is feature-complete for v2.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 00:51:34 +02:00
/** Serialize a backup blob to a compact single-string form (for copy/paste or QR). */
export function backupToString(blob: BackupBlob): string {
return `shade-backup:v${blob.version}:${blob.salt}:${blob.nonce}:${blob.ciphertext}`;
}
/** Parse a backup string back into a BackupBlob. */
export function backupFromString(str: string): BackupBlob {
const parts = str.split(':');
if (parts.length !== 5 || parts[0] !== 'shade-backup') {
throw new Error('Invalid backup string format');
}
const version = parseInt(parts[1]!.replace('v', ''), 10);
return {
version,
salt: parts[2]!,
nonce: parts[3]!,
ciphertext: parts[4]!,
};
}