Shade/docs/observability.md

Observability v2 — OpenTelemetry tracing

Shade ships an opt-in OpenTelemetry layer that wraps TransferEngine, ShadeSessionManager, the prekey HTTP routes, and @shade/files op-handlers in distributed spans. The layer is off by default and PII-safe by construction — span attributes never include peer addresses, plaintext payloads, or exact byte counts.

This complements the always-on Prometheus metrics exposed by @shade/server and the structural events emitted by @shade/core. Use metrics for aggregate counters and histograms, tracing for per-request causality and tail-latency hunting.

---

Quick start

import { trace } from '@opentelemetry/api';
import { withTracer } from '@shade/observability';
import { createShade } from '@shade/sdk';

// Use the OTel SDK of your choice (NodeSDK + OTLP exporter, Honeycomb,
// Sentry's OTel adapter, …) to register a tracer provider on the
// `@opentelemetry/api` global. Then:
const tracer = trace.getTracer('my-app');

const shade = await createShade({
  prekeyServer: 'https://shade.example.com',
  storage: 'sqlite:/data/shade.db',
  observability: withTracer(tracer, { sample: 0.1 }),
});

The hook propagates automatically to:

• ShadeSessionManager.encrypt / .decrypt (per-peer mutex acquisition, ratchet step).
• TransferEngine.upload / accepted incoming downloads (lane count, retry count, partition mode).
• @shade/files op-handlers (per request, with op + result).

For the prekey server pass the hook to createPrekeyRoutes:

import { createPrekeyRoutes } from '@shade/server';
import { withTracer } from '@shade/observability';

const app = createPrekeyRoutes(store, crypto, {
  observability: withTracer(tracer),
});

---

Off-by-default semantics

withTracer() returns a no-op hook — the SDK never starts spans — when any of the following are true:

1. The tracer argument is undefined/null.
2. The SHADE_OTEL_ENABLED env-var is not set to 1 or true. Override with withTracer(tracer, { force: true }), or override the var name with withTracer(tracer, { envVar: 'MY_VAR' }).
3. The configured sample rate is 0.

Per-span sampling (sample: 0.1 = 10 %) keeps trace volume bounded in production. Default is 1 (sample everything when the hook is active).

---

PII policy — what is safe to log, and what isn't

Category	Status	Why
Peer hash (shade.peer.hash)	✅ allowed	8-hex-char pseudonym derived via SHA-256. Stable across spans for a given address but does not expose the address itself.
Bytes bin (shade.bytes.bin)	✅ allowed	One of ≤4KB, 4–64KB, 64KB–1MB, 1–10MB, 10–100MB, 100MB–1GB, ≥1GB. Coarse enough to mask file-size fingerprinting.
Lane count (shade.lane.count)	✅ allowed	Snapped to {1, 4, 16, 64}.
Retry count (shade.retry.count)	✅ allowed	Integer.
Error code (shade.error.code)	✅ allowed	SHADE_* stable string code — never the full message, which may interpolate user input.
Op kind (shade.op)	✅ allowed	list, read, write, custom:foo, etc.
Route template (shade.route)	✅ allowed	/v1/keys/bundle/:address — the template, never the resolved path.
HTTP status (shade.http.status)	✅ allowed	Integer status code.
Partition mode (shade.partition)	✅ allowed	range or round-robin.
Direction (shade.direction)	✅ allowed	upload or download.
Plaintext peer addresses	❌ forbidden	Use peerHash().
Plaintext message/file payloads	❌ forbidden	Encryption boundary — never log.
Exact byte counts	❌ forbidden	Use bytesBin().
User identifiers (email, DID, device:UUID)	❌ forbidden	Treat as PII.

The full attribute-key allow-list is exported from @shade/observability as ATTR_* constants. Plug-in authors who want to attach their own tags should pass each (key, value) through safeAttribute(), which throws UnsafeAttributeError for any key/value pair that looks like the forbidden categories above (heuristics: @, device:, did:, key fragments such as peer.address / bytes.exact, oversized strings).

---

Span surface

shade.session.encrypt / shade.session.decrypt

Wraps each per-peer encrypt/decrypt call. Includes the time spent waiting on the per-peer mutex (shade.lock.wait_ms) — handy for diagnosing ratchet contention under load.

shade.transfer.upload / shade.transfer.upload.resume

Wraps an outbound stream transfer end-to-end. Attributes: peer.hash, bytes.bin, lane.count, partition, retry.count, result, error.code.

shade.transfer.download

Started when the consumer calls incoming.accept(...), ended when the transfer completes, aborts, or fails an integrity check. Same attribute set as upload.

shade.prekey.request

One span per HTTP request handled by @shade/server's prekey routes. Attributes: route (the template), http.status, error.code on failure. The address path-parameter is never placed on the span.

shade.files.op

One span per @shade/files RPC. Attributes: peer.hash, op (the resolved op kind, e.g. read or custom:foo), bytes.bin (estimated plaintext size, binned), result, error.code.

---

Recording & testing

@shade/observability ships a deterministic in-memory recorder for unit tests:

import { createRecorder } from '@shade/observability';

const rec = createRecorder();
const shade = await createShade({ ..., observability: rec });

// … exercise code under test …

const hits = rec.scanForPII(['alice@example.com', 'plaintext-secret']);
expect(hits).toHaveLength(0);

The Shade test suite runs this recorder over every documented entry point — see packages/shade-observability/tests/integration-pii.test.ts and packages/shade-transfer/tests/observability.test.ts. Any new instrumentation must keep the suite green.

---

Performance characteristics

• With OTel off (default): every Shade hook resolves to the shared NOOP_HOOK instance. The cost is one function call + an object allocation that V8 hoists out in the steady state — measured at < 1 % overhead vs the pre-V3.4 baseline in the upload roundtrip benchmark.
• With OTel on: cost depends entirely on the configured exporter. Use sample: 0.1 (or smaller) on hot paths in production.

---

Adding new instrumentation

1. Identify a logical operation worth a span — typically anything that crosses a network/disk boundary or contends on a lock.
2. Add an observability?: ObservabilityHook to the relevant config surface, default to NOOP_HOOK.
3. Name the span shade.<area>.<op> to keep cardinality bounded.
4. Set attributes via the ATTR_* constants from @shade/observability. Never introduce a new attribute key without a PII review — if you must, run the value through safeAttribute().
5. Add a test that exercises the new instrumentation under the createRecorder() recorder and asserts no PII leaks.

---

Migration

Previous versions had no tracing — only Prometheus metrics. Adding the observability field to existing configs is fully backwards-compatible and never required. The SHADE_OTEL_ENABLED gate ensures forgetting to flip the env-var in production won't surprise anyone with unexpected overhead.