release(v4.0.0): Shade GA — V3.x consolidation + audit prep
Some checks failed
Test / test (push) Has been cancelled
Cross-platform vectors / TypeScript vectors (bun) (push) Has been cancelled
Cross-platform vectors / Kotlin vectors (gradle) (push) Has been cancelled
Docker build and publish / docker (push) Has been cancelled
Publish / publish (push) Has been cancelled

V3.1 → V3.12 consolidated and tagged for the first GA release. Wire
format unchanged from 0.4.x — 4.0 peers interoperate with 0.4.x peers
byte-for-byte. The version bump is semantic: audit-cycle complete,
opt-in surface fully exposed, threat model refreshed for every new
surface.

Highlights:
- All 24 @shade/* packages bumped to 4.0.0 in lockstep.
- CHANGELOG 4.0.0 section is the canonical manifest of what landed.
- THREAT-MODEL extended (§10 fingerprint gates, §11 WebRTC P2P, §12
  Web-Worker boundary) + residual-risks table refreshed.
- OpenAPI now covers all 27 routes: prekey, transfer, KT, inbox,
  bridge, observer, /metrics, /healthz, /ready.
- MIGRATION 0.3.x → 4.0 documented + smoke-tested against
  shade migrate-storage on a real SQLite DB.
- docs/audit/REVIEW-BUNDLE.md + SCOPE.md ready for external reviewer.
- scripts/soak.ts harness for the GA-stable 2-week soak window.
- All V*.md plans archived under docs/archive/ with Status: Done.
- Voice/Video carved out into V5.0; 4.0 audit focuses on the frozen
  non-realtime stack.

Tests: TS 1000/1000 + Kotlin 11/11 cross-platform vectors green.
Docker: gt.zyon.no/stian/shade-prekey:4.0.0 builds and reports
  version 4.0.0 on /health.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-03 18:35:35 +02:00
parent 8b055912b7
commit e6fdf31b49
298 changed files with 37909 additions and 256 deletions

View File

@@ -4,7 +4,15 @@ Kotlin implementation of the Shade E2EE protocol for Android apps. Byte-for-byte
## Status
**Milestone M-Cross 1 — initial scaffold.** The protocol implementation is being ported. Cross-platform test vectors in `test-vectors/` verify that Kotlin and TypeScript produce identical output for every step (identity gen → HKDF X3DH → ratchet → fingerprint → wire format).
**M-Cross 1 ✅** — keys + HKDF + X3DH + fingerprint.
**M-Cross 2 ✅** — full ratchet step (encrypt + decrypt roundtrip) + wire 0x02 (RatchetMessage and PreKeyMessage with/without OTPK).
**M-Cross 3 ✅** — streams 0x11 (KDF labels with embedded NULs, deterministic chunk nonce/AAD, wire 0x11 encode/decode).
**M-Cross 4 ✅** — backup-key HKDF + AES-GCM, group sender-key step (`kdfChainKey` + Ed25519 sign over `aad ‖ ct`), storage HKDF (storageKey/fieldKey/rowNonce). Pending: scrypt master-key, argon2id swap, Android KeystoreStorage (sibling module).
Cross-platform test vectors in `/test-vectors/` are loaded by both the TS
and Kotlin test suites; any byte-divergence fails CI within 60 s. See
`ROADMAP-ANDROID.md` for the parity-checkpoint matrix and
`/docs/cross-platform.md` for how to add a new vector.
## Usage (target API)
@@ -40,13 +48,18 @@ Backed by Google Tink:
## Building
Requires Android SDK 35 and JDK 17.
Requires JDK 17. The module compiles as a pure-JVM Kotlin library so the
parity gate runs without an Android SDK install. The Android-specific
storage adapter (Keystore + EncryptedSharedPreferences) will land as a
sibling Gradle module in M-Cross 4.
```bash
./gradlew :shade-android:assembleDebug
cd android
./gradlew :shade-android:test
```
The Gradle wrapper downloads Gradle 8.10.2 on first run.
## Compatibility
The Kotlin implementation must produce byte-identical output to `@shade/core` for:

View File

@@ -0,0 +1,137 @@
# Shade Android — Roadmap & Parity Status
This document tracks the M-Cross milestones from `docs/V3.5.md` and the
status of every cross-platform parity sjekkpunkt. The Kotlin port must be
**byte-for-byte compatible** with the TypeScript implementation; this is
verified continuously by `test-vectors/*.json` consumed by both runners.
> **No "production" label** is allowed on Android until M-Cross 2 is green
> (ratchet + wire 0x02 + storage encryption) and M-Cross 3 is green
> (streams 0x11). See `docs/V3.5.md` §Akseptansekriterier.
## Milestones
### M-Cross 1 — Scaffold ✅
Foundation primitives. All passing in CI.
| Sjekkpunkt | Vector | TS test | Kotlin test |
|---|---|---|---|
| 1. KDF chain (root + chain ratchet) | `kdf-chain.json` | ✅ | ✅ |
| 2. HKDF labels | `hkdf.json` | ✅ | ✅ |
| 3. X3DH initial root key (3 + 4 DH outputs) | `x3dh.json` | ✅ | ✅ |
| 5. Fingerprint (60-digit safety number) | `fingerprint.json` | ✅ | ✅ |
### M-Cross 2 — Ratchet & Wire 0x02 ✅
Full ratchet step + binary envelope encoding for both message types.
| Sjekkpunkt | Vector | TS test | Kotlin test |
|---|---|---|---|
| 4. Ratchet step (encrypt deterministic) | `ratchet-step.json` | ✅ | ✅ |
| 4. Ratchet step (decrypt roundtrip) | `ratchet-step.json` | ✅ | ✅ |
| 6. Wire 0x02 RatchetMessage | `wire-format.json` | ✅ | ✅ |
| 6. Wire 0x02 PreKeyMessage (with OTPK) | `wire-format.json` | ✅ | ✅ |
| 6. Wire 0x02 PreKeyMessage (no OTPK, 0xFFFFFFFF marker) | `wire-format.json` | ✅ | ✅ |
The ratchet-step vector exercises every layer that contributes to a
ratchet message's wire bytes: `kdfRootKey``kdfChainKey` → 40-byte header
AAD → AES-256-GCM with deterministic nonce. Both implementations recompute
each layer and compare against the recorded hex. The decrypt half feeds
the recorded ciphertext back through `aesGcmDecrypt(messageKey, nonce, aad)`
and checks the plaintext recovers — proving the AEAD agrees in both
directions.
### M-Cross 3 — Streams 0x11 ✅
Multi-lane chunk encryption (`@shade/streams`) ported. KDF labels with
embedded NULs match TS byte-for-byte; deterministic
`(laneId, seq)`-derived nonces and the 29-byte chunk AAD agree across
runners; wire 0x11 encode/decode is roundtrip-verified.
| Sjekkpunkt | Vector | TS test | Kotlin test |
|---|---|---|---|
| `deriveStreamKey` (HKDF, info `shade-stream/v1\0master`) | `streams.json` | ✅ | ✅ |
| `deriveLaneKey` (HKDF, info `shade-stream/v1\0lane\0` ‖ u32_be laneId) — incl. laneId 0xFFFFFFFF | `streams.json` | ✅ | ✅ |
| `buildChunkNonce(laneId, seq)` — incl. seq = 2^64 - 2 | `streams.json` | ✅ | ✅ |
| `buildChunkAad(streamId, laneId, seq, isLast)` | `streams.json` | ✅ | ✅ |
| Chunk AES-256-GCM encrypt + decrypt (deterministic nonce + AAD) | `streams.json` | ✅ | ✅ |
| Wire 0x11 envelope encode + decode + type-tag inspector | `streams.json` | ✅ | ✅ |
Sequence numbers are unsigned u64 on the wire; the Kotlin port accepts
them as `Long` for the bit pattern (negative-signed-long for values past
2^63 - 1) — this matches the JVM `ByteBuffer.putLong` behavior and the
`java.lang.Long.parseUnsignedLong` JSON-decoder used in tests.
Pending end-to-end interop test (TS server → Kotlin client over an actual
socket) — not gated by vectors but recommended before flipping the
"production" label.
### M-Cross 4 — Backup, Group, Storage HKDF ✅ (cryptographic layer)
The cryptographic primitives that Kotlin needs to share with TS are now
covered. The remaining work is the high-level glue (BackupBlob JSON
schema, full SenderKey/GroupSession state-tracking, Android-Keystore
storage adapter, scrypt password-KDF) — all per-platform plumbing that
doesn't gate vector parity.
| Sjekkpunkt | Vector | TS test | Kotlin test |
|---|---|---|---|
| 7. Backup v1 HKDF (`info="ShadeBackupKey"`) | `backup.json` | ✅ | ✅ |
| 7. Backup v1 AES-GCM roundtrip (no AAD) | `backup.json` | ✅ | ✅ |
| Group sender header AAD (u16/u16/u32 length prefixes) | `group.json` | ✅ | ✅ |
| Group sender-key step: `kdfChainKey` + AES-GCM + Ed25519 sign(aad ‖ ct) | `group.json` | ✅ | ✅ |
| Storage HKDF: `storageKey` (`info="shade-storage-v1"`) | `storage-hkdf.json` | ✅ | ✅ |
| Storage HKDF: `fieldKey` (`info="shade-field-v1:{table}:{column}"`) | `storage-hkdf.json` | ✅ | ✅ |
| Storage HKDF: `rowNonce` (`info="shade-row-nonce-v1:{table}:{pk}"`) | `storage-hkdf.json` | ✅ | ✅ |
Pending sub-tasks (don't gate vector parity):
- **scrypt master-key derivation**: `test-vectors/storage-encryption.json`
pins `scrypt(N=1024, r=8, p=1, dkLen=32)` for unit-test config; Tink
doesn't ship scrypt. Add Bouncy Castle (`org.bouncycastle:bcprov-jdk18on`)
to the Kotlin module, wrap as `CryptoProvider.scrypt(...)`, then a follow-up
vector consumes the full storage-encryption.json end to end.
- **argon2id**: Both backup.ts and the threat-model docs flag HKDF as a
placeholder for a real password KDF. When `argon2id` is added to
`CryptoProvider`, both ports swap together and the backup vector gets
re-pinned.
- **Android KeystoreStorage adapter**: lives in a sibling Android Library
Gradle module that depends on this JVM module. Binds Tink to the Android
Keystore + EncryptedSharedPreferences.
## Build & Test
This module compiles as a **pure-JVM** Kotlin library (`kotlin("jvm")`)
so the parity gate can run without an Android SDK installation in CI.
The protocol code uses `tink:1.15.0` (JVM JAR), `java.nio.ByteBuffer`,
and `javax.crypto` — no `android.*` imports.
The Android-specific storage adapter (KeystoreStorage,
EncryptedSharedPreferences) will land as a sibling Gradle module
(`shade-android-keystore`) in M-Cross 4 and depend on this one.
```bash
# From repo root
cd android
./gradlew :shade-android:test
```
Requires JDK 17. The Gradle wrapper downloads Gradle 8.10.2 on first run.
## Compatibility contract
The Kotlin implementation must produce byte-identical output to the TS
reference for:
- KDF chain derivations (root key ratchet, chain key ratchet)
- X3DH shared secrets (3- and 4-DH variants)
- Ratchet message keys + AES-GCM ciphertext (given the same key/plaintext/AAD/nonce)
- Header AAD encoding (40 bytes: `dhPublicKey(32) || u32_be(prevCounter) || u32_be(counter)`)
- Fingerprints (12 × 5-digit groups)
- Binary wire format 0x02 (RatchetMessage + PreKeyMessage)
- Binary wire format 0x11 (StreamChunk) — M-Cross 3
- Storage encryption KDF chain — M-Cross 4
Each is covered by a vector file in `/test-vectors/`. Adding a new
sjekkpunkt: see `docs/cross-platform.md`.

View File

@@ -1,50 +1,51 @@
plugins {
id("com.android.library")
kotlin("android")
kotlin("jvm")
`java-library`
}
android {
namespace = "no.zyon.shade"
compileSdk = 35
// V3.5 — Cross-platform parity gate.
//
// This module compiles as a pure-JVM Kotlin library so CI can run the
// cross-platform vector tests without an Android SDK. The protocol code
// is JVM-safe (no `android.*` imports); only Tink + java.* are used.
//
// When KeystoreStorage and EncryptedSharedPreferences-backed adapters land
// (M-Cross 4 + V3.5 §Storage), they will live in a sibling Android Library
// module that depends on this one.
defaultConfig {
minSdk = 26
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
}
java {
sourceCompatibility = JavaVersion.VERSION_17
targetCompatibility = JavaVersion.VERSION_17
}
buildFeatures {
buildConfig = true
}
compileOptions {
sourceCompatibility = JavaVersion.VERSION_17
targetCompatibility = JavaVersion.VERSION_17
}
kotlinOptions {
jvmTarget = "17"
kotlin {
compilerOptions {
jvmTarget.set(org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17)
}
}
dependencies {
// Google Tink for X25519, Ed25519, AES-GCM, HMAC, HKDF
implementation("com.google.crypto.tink:tink-android:1.15.0")
// Google Tink for X25519, Ed25519, AES-GCM, HMAC, HKDF (JVM build).
// The same `subtle.*` API as `tink-android` so the source compiles unchanged.
implementation("com.google.crypto.tink:tink:1.15.0")
// Android Keystore + EncryptedSharedPreferences
implementation("androidx.security:security-crypto:1.1.0-alpha06")
// JSON serialization for session state
// JSON serialization (session state + test-vector loader on JVM).
implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.7.3")
// Coroutines for async interop
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.9.0")
// Coroutines (StorageProvider uses `suspend` functions).
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.9.0")
// SQLite for session storage (optional; can also use EncryptedSharedPreferences only)
implementation("androidx.sqlite:sqlite:2.4.0")
// OkHttp for transport
implementation("com.squareup.okhttp3:okhttp:4.12.0")
// org.json — bundled with Android but not present on the JVM classpath.
implementation("org.json:json:20240303")
testImplementation("junit:junit:4.13.2")
testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.9.0")
}
tasks.withType<Test>().configureEach {
useJUnit()
testLogging {
events("passed", "failed", "skipped")
showStandardStreams = false
}
}

View File

@@ -0,0 +1,36 @@
package no.zyon.shade.backup
import no.zyon.shade.crypto.CryptoProvider
/**
* Backup format v1 — passphrase-derived AES-256-GCM blob.
* Mirror @shade/sdk/backup.ts.
*
* backupKey = HKDF(passphrase_utf8, salt_random_32, info="ShadeBackupKey", 32)
* blob = AES-256-GCM(backupKey, plaintext, no AAD)
*
* The stored on-disk form is `{ version, salt(b64), nonce(b64), ciphertext(b64) }`.
* This file ships only the cryptographic primitives — payload schema and JSON
* serialization live alongside the high-level SDK and don't need a Kotlin port
* for vector parity (each platform builds the BackupBlob in its native idiom).
*
* NOTE: HKDF is NOT a proper password KDF. The TS SDK acknowledges this and
* warns users to choose a high-entropy passphrase. When `argon2id` lands in
* `CryptoProvider`, both ports swap together. Until then, byte-parity for the
* HKDF + AEAD layer is what V3.5 §sjekkpunkt 8 gates.
*/
private val BACKUP_INFO: ByteArray = "ShadeBackupKey".toByteArray(Charsets.UTF_8)
const val BACKUP_KEY_BYTES = 32
const val BACKUP_VERSION = 1
fun deriveBackupKey(crypto: CryptoProvider, passphrase: String, salt: ByteArray): ByteArray {
require(passphrase.length >= 12) { "Passphrase must be at least 12 characters" }
require(salt.size >= 16) { "salt must be at least 16 bytes" }
return crypto.hkdf(
passphrase.toByteArray(Charsets.UTF_8),
salt,
BACKUP_INFO,
BACKUP_KEY_BYTES,
)
}

View File

@@ -0,0 +1,77 @@
package no.zyon.shade.group
import no.zyon.shade.crypto.CryptoProvider
import no.zyon.shade.protocol.kdfChainKey
import java.nio.ByteBuffer
/**
* Group sender-keys (Sesame). Mirror @shade/core/sender-keys.ts.
*
* Each sender maintains a chain key that ratchets forward with `kdfChainKey`
* — same primitive the Double Ratchet uses for its symmetric chain. Per-message
* AEAD AAD binds (groupId, senderAddress, iteration) so a captured ciphertext
* cannot be replayed under a different sender or group:
*
* aad = u16_be(groupIdLen) || groupId || u16_be(senderAddrLen) || senderAddr || u32_be(iteration)
*
* Each ciphertext is signed by the sender's Ed25519 key over `aad || ciphertext`,
* which is what receivers verify before advancing their chain.
*/
data class SenderKeyMessage(
val senderAddress: String,
val iteration: Int,
val ciphertext: ByteArray,
val nonce: ByteArray,
val signature: ByteArray,
)
fun encodeSenderHeader(groupId: String, senderAddress: String, iteration: Int): ByteArray {
val gBytes = groupId.toByteArray(Charsets.UTF_8)
val sBytes = senderAddress.toByteArray(Charsets.UTF_8)
require(gBytes.size <= 0xFFFF) { "groupId too long (>65535 UTF-8 bytes)" }
require(sBytes.size <= 0xFFFF) { "senderAddress too long (>65535 UTF-8 bytes)" }
val out = ByteArray(2 + gBytes.size + 2 + sBytes.size + 4)
val buf = ByteBuffer.wrap(out)
buf.putShort(gBytes.size.toShort())
buf.put(gBytes)
buf.putShort(sBytes.size.toShort())
buf.put(sBytes)
buf.putInt(iteration)
return out
}
/**
* Compute (newChainKey, messageKey, aad) for the next group message.
* Pure function; caller is responsible for state advancement and the AEAD/sign
* steps (which need access to the signing private key not exposed here).
*/
data class SenderStepResult(
val newChainKey: ByteArray,
val messageKey: ByteArray,
val aad: ByteArray,
)
fun senderKeyStep(
crypto: CryptoProvider,
chainKey: ByteArray,
groupId: String,
senderAddress: String,
iteration: Int,
): SenderStepResult {
val r = kdfChainKey(crypto, chainKey)
val aad = encodeSenderHeader(groupId, senderAddress, iteration)
return SenderStepResult(newChainKey = r.newChainKey, messageKey = r.messageKey, aad = aad)
}
/**
* Concatenate `aad || ciphertext` — the byte string the sender signs and the
* receiver verifies. Exposed as a helper so vector parity can pin both sides.
*/
fun senderSignedBytes(aad: ByteArray, ciphertext: ByteArray): ByteArray {
val out = ByteArray(aad.size + ciphertext.size)
aad.copyInto(out, 0)
ciphertext.copyInto(out, aad.size)
return out
}

View File

@@ -0,0 +1,145 @@
package no.zyon.shade.serialization
import no.zyon.shade.streams.StreamConstants
import java.nio.ByteBuffer
/**
* Wire-decoded stream-chunk envelope (type 0x11).
*
* Mirror @shade/proto/wire.ts `StreamChunkWire`. The nonce is deterministic
* (derived from `(laneId, seq)` on both sides) but is also serialized over
* the wire for self-description and validated by the receiver.
*
* `seq` is unsigned-u64 on the wire; on the JVM we keep it as Long. The
* encode/decode helpers operate on the raw 8-byte big-endian representation,
* so values past Long.MAX_VALUE roundtrip via `Long.toULong()`.
*/
data class StreamChunkWire(
val streamId: ByteArray,
val laneId: Long,
val seq: Long,
val isLast: Boolean,
val nonce: ByteArray,
val aad: ByteArray,
val ciphertext: ByteArray,
) {
override fun equals(other: Any?): Boolean {
if (this === other) return true
if (other !is StreamChunkWire) return false
return streamId.contentEquals(other.streamId) &&
laneId == other.laneId &&
seq == other.seq &&
isLast == other.isLast &&
nonce.contentEquals(other.nonce) &&
aad.contentEquals(other.aad) &&
ciphertext.contentEquals(other.ciphertext)
}
override fun hashCode(): Int {
var result = streamId.contentHashCode()
result = 31 * result + laneId.hashCode()
result = 31 * result + seq.hashCode()
result = 31 * result + isLast.hashCode()
result = 31 * result + nonce.contentHashCode()
result = 31 * result + aad.contentHashCode()
result = 31 * result + ciphertext.contentHashCode()
return result
}
}
/** Stream-chunk wire codec. Mirror @shade/proto/wire.ts `encodeStreamChunk`/`decodeStreamChunk`. */
object StreamChunkWireFormat {
private const val VERSION: Byte = 0x02
const val TYPE_STREAM_CHUNK: Byte = 0x11
fun encodeStreamChunk(c: StreamChunkWire): ByteArray {
require(c.streamId.size == StreamConstants.STREAM_ID_BYTES) {
"streamId must be ${StreamConstants.STREAM_ID_BYTES} bytes"
}
require(c.nonce.size == StreamConstants.STREAM_NONCE_BYTES) {
"nonce must be ${StreamConstants.STREAM_NONCE_BYTES} bytes"
}
require(c.laneId in 0L..0xFFFFFFFFL) { "laneId out of u32 range: ${c.laneId}" }
// c.seq is unsigned-u64; negative signed longs encode as the high half
// of the u64 range. ByteBuffer.putLong writes the raw 8-byte pattern.
val headerSize =
1 + 1 +
StreamConstants.STREAM_ID_BYTES +
4 + 8 + 1 +
StreamConstants.STREAM_NONCE_BYTES +
4 + c.aad.size +
4
val out = ByteArray(headerSize + c.ciphertext.size)
val buf = ByteBuffer.wrap(out)
buf.put(VERSION)
buf.put(TYPE_STREAM_CHUNK)
buf.put(c.streamId)
buf.putInt(c.laneId.toInt())
buf.putLong(c.seq)
buf.put(if (c.isLast) 0x01.toByte() else 0x00.toByte())
buf.put(c.nonce)
buf.putInt(c.aad.size)
buf.put(c.aad)
buf.putInt(c.ciphertext.size)
buf.put(c.ciphertext)
return out
}
fun decodeStreamChunk(data: ByteArray): StreamChunkWire {
val minHeaderSize = 2 +
StreamConstants.STREAM_ID_BYTES +
4 + 8 + 1 +
StreamConstants.STREAM_NONCE_BYTES +
4 + 4
require(data.size >= minHeaderSize) {
"stream-chunk too short: ${data.size} < $minHeaderSize"
}
require(data[0] == VERSION) { "Unknown version: ${data[0]}" }
require(data[1] == TYPE_STREAM_CHUNK) { "Not a stream-chunk: type=${data[1]}" }
val buf = ByteBuffer.wrap(data)
buf.position(2)
val streamId = ByteArray(StreamConstants.STREAM_ID_BYTES)
buf.get(streamId)
val laneId = buf.int.toLong() and 0xFFFFFFFFL
val seq = buf.long
val isLast = buf.get() == 0x01.toByte()
val nonce = ByteArray(StreamConstants.STREAM_NONCE_BYTES)
buf.get(nonce)
val aadLen = buf.int
require(buf.position() + aadLen + 4 <= data.size) {
"stream-chunk truncated in aad/ctLen"
}
val aad = ByteArray(aadLen)
buf.get(aad)
val ctLen = buf.int
require(buf.position() + ctLen == data.size) {
"stream-chunk length mismatch: declared ${buf.position() + ctLen}, actual ${data.size}"
}
val ciphertext = ByteArray(ctLen)
buf.get(ciphertext)
return StreamChunkWire(streamId, laneId, seq, isLast, nonce, aad, ciphertext)
}
/** Inspect the type tag without full parsing. Mirror @shade/proto/wire.ts. */
enum class EnvelopeKind { PREKEY, RATCHET, STREAM_CHUNK, UNKNOWN }
fun inspectEnvelopeType(data: ByteArray): EnvelopeKind {
if (data.size < 2 || data[0] != VERSION) return EnvelopeKind.UNKNOWN
return when (data[1]) {
0x01.toByte() -> EnvelopeKind.PREKEY
0x02.toByte() -> EnvelopeKind.RATCHET
TYPE_STREAM_CHUNK -> EnvelopeKind.STREAM_CHUNK
else -> EnvelopeKind.UNKNOWN
}
}
}

View File

@@ -0,0 +1,48 @@
package no.zyon.shade.streams
import javax.crypto.Cipher
import javax.crypto.spec.GCMParameterSpec
import javax.crypto.spec.SecretKeySpec
/**
* AES-256-GCM with caller-supplied nonce. Mirror @shade/streams/aead.ts.
*
* Unlike `CryptoProvider.aesGcmEncrypt` (which generates a random nonce
* internally), streams require deterministic nonces derived from
* `(laneId, seq)`. Returns the ciphertext concatenated with the 16-byte
* authentication tag — same layout SubtleCrypto produces.
*/
const val AEAD_TAG_BYTES = 16
fun aesGcmEncryptWithNonce(
key: ByteArray,
nonce: ByteArray,
plaintext: ByteArray,
aad: ByteArray,
): ByteArray {
require(nonce.size == StreamConstants.STREAM_NONCE_BYTES) {
"AES-GCM nonce must be ${StreamConstants.STREAM_NONCE_BYTES} bytes"
}
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val spec = GCMParameterSpec(128, nonce)
cipher.init(Cipher.ENCRYPT_MODE, SecretKeySpec(key, "AES"), spec)
cipher.updateAAD(aad)
return cipher.doFinal(plaintext)
}
fun aesGcmDecryptWithNonce(
key: ByteArray,
nonce: ByteArray,
ciphertext: ByteArray,
aad: ByteArray,
): ByteArray {
require(nonce.size == StreamConstants.STREAM_NONCE_BYTES) {
"AES-GCM nonce must be ${StreamConstants.STREAM_NONCE_BYTES} bytes"
}
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val spec = GCMParameterSpec(128, nonce)
cipher.init(Cipher.DECRYPT_MODE, SecretKeySpec(key, "AES"), spec)
cipher.updateAAD(aad)
return cipher.doFinal(ciphertext)
}

View File

@@ -0,0 +1,50 @@
package no.zyon.shade.streams
import java.nio.ByteBuffer
/**
* Deterministic AEAD nonce + AAD construction for stream chunks.
* Mirror @shade/streams/nonce.ts.
*
* nonce[0..4] = u32_be(laneId)
* nonce[4..12] = u64_be(seq)
*
* aad = streamId(16) || u32_be(laneId) || u64_be(seq) || u8(isLast)
*
* `seq` is unsigned-u64 on the wire. Kotlin's `Long` is signed; we accept it
* for the bit pattern (same as TS `BigInt` would write), so values past
* `Long.MAX_VALUE` arrive here as negative signed longs. `ByteBuffer.putLong`
* writes the raw 8 bytes regardless of sign — that's what we want.
*
* Use `java.lang.Long.parseUnsignedLong("…")` to decode JSON strings
* representing u64 values larger than 2^63 - 1.
*/
fun buildChunkNonce(laneId: Long, seq: Long): ByteArray {
require(laneId in 0L..0xFFFFFFFFL) { "laneId must fit in u32: $laneId" }
val out = ByteArray(StreamConstants.STREAM_NONCE_BYTES)
val buf = ByteBuffer.wrap(out)
buf.putInt(laneId.toInt())
buf.putLong(seq)
return out
}
fun buildChunkAad(
streamId: ByteArray,
laneId: Long,
seq: Long,
isLast: Boolean,
): ByteArray {
require(streamId.size == StreamConstants.STREAM_ID_BYTES) {
"streamId must be ${StreamConstants.STREAM_ID_BYTES} bytes"
}
require(laneId in 0L..0xFFFFFFFFL) { "laneId must fit in u32: $laneId" }
val out = ByteArray(StreamConstants.STREAM_ID_BYTES + 4 + 8 + 1)
streamId.copyInto(out, 0)
val buf = ByteBuffer.wrap(out, StreamConstants.STREAM_ID_BYTES, 4 + 8 + 1)
buf.putInt(laneId.toInt())
buf.putLong(seq)
out[out.size - 1] = if (isLast) 0x01 else 0x00
return out
}

View File

@@ -1,18 +1,36 @@
package no.zyon.shade
import no.zyon.shade.backup.deriveBackupKey
import no.zyon.shade.crypto.TinkProvider
import no.zyon.shade.fingerprint.computeFingerprint
import no.zyon.shade.group.encodeSenderHeader
import no.zyon.shade.group.senderKeyStep
import no.zyon.shade.group.senderSignedBytes
import no.zyon.shade.protocol.deriveInitialRootKey
import no.zyon.shade.protocol.kdfChainKey
import no.zyon.shade.protocol.kdfRootKey
import no.zyon.shade.serialization.StreamChunkWire
import no.zyon.shade.serialization.StreamChunkWireFormat
import no.zyon.shade.serialization.WireFormat
import no.zyon.shade.streams.aesGcmDecryptWithNonce
import no.zyon.shade.streams.aesGcmEncryptWithNonce
import no.zyon.shade.streams.buildChunkAad
import no.zyon.shade.streams.buildChunkNonce
import no.zyon.shade.streams.deriveLaneKey
import no.zyon.shade.streams.deriveStreamKey
import no.zyon.shade.types.PreKeyMessage
import no.zyon.shade.types.RatchetMessage
import no.zyon.shade.types.ShadeEnvelope
import org.json.JSONArray
import org.json.JSONObject
import org.junit.Assert.assertEquals
import org.junit.Assert.assertTrue
import org.junit.Test
import java.io.File
import org.json.JSONObject
import org.json.JSONArray
import java.nio.ByteBuffer
import javax.crypto.Cipher
import javax.crypto.spec.GCMParameterSpec
import javax.crypto.spec.SecretKeySpec
/**
* Cross-platform test vectors. MUST match the TypeScript implementation
@@ -25,6 +43,7 @@ class CrossPlatformVectorTest {
private val crypto = TinkProvider()
private val vectorsDir = File("../../test-vectors")
private val expectedVersion = 2
private fun fromHex(str: String): ByteArray {
val bytes = ByteArray(str.length / 2)
@@ -39,10 +58,40 @@ class CrossPlatformVectorTest {
return bytes.joinToString("") { "%02x".format(it) }
}
private data class VectorFile(val version: Int, val vectors: JSONArray)
private fun loadVectors(name: String): JSONArray {
val file = File(vectorsDir, name)
val content = file.readText()
return JSONObject(content).getJSONArray("vectors")
val obj = JSONObject(content)
val version = obj.getInt("version")
assertEquals("Unexpected vector schema version in $name", expectedVersion, version)
return obj.getJSONArray("vectors")
}
private fun encodeRatchetHeader(
dhPublicKey: ByteArray,
previousCounter: Int,
counter: Int,
): ByteArray {
val buf = ByteBuffer.allocate(40)
buf.put(dhPublicKey)
buf.putInt(previousCounter)
buf.putInt(counter)
return buf.array()
}
private fun aesGcmEncryptDeterministic(
key: ByteArray,
nonce: ByteArray,
plaintext: ByteArray,
aad: ByteArray,
): ByteArray {
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val spec = GCMParameterSpec(128, nonce)
cipher.init(Cipher.ENCRYPT_MODE, SecretKeySpec(key, "AES"), spec)
cipher.updateAAD(aad)
return cipher.doFinal(plaintext)
}
@Test
@@ -106,31 +155,347 @@ class CrossPlatformVectorTest {
}
@Test
fun wireFormatVectorsMatch() {
fun wireFormatRatchetVectorsMatch() {
val vectors = loadVectors("wire-format.json")
val v = vectors.getJSONObject(0)
val m = v.getJSONObject("message")
var found = false
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
if (v.optString("kind") != "ratchet") continue
found = true
val m = v.getJSONObject("message")
val msg = RatchetMessage(
dhPublicKey = fromHex(m.getString("dhPublicKey")),
previousCounter = m.getInt("previousCounter"),
counter = m.getInt("counter"),
ciphertext = fromHex(m.getString("ciphertext")),
nonce = fromHex(m.getString("nonce")),
)
val envelope = ShadeEnvelope(
type = ShadeEnvelope.EnvelopeType.RATCHET,
content = msg,
timestamp = 0,
senderAddress = "",
)
val encoded = WireFormat.encodeEnvelope(envelope)
assertEquals(v.getString("encoded"), hex(encoded))
val msg = RatchetMessage(
dhPublicKey = fromHex(m.getString("dhPublicKey")),
previousCounter = m.getInt("previousCounter"),
counter = m.getInt("counter"),
ciphertext = fromHex(m.getString("ciphertext")),
nonce = fromHex(m.getString("nonce")),
)
val envelope = ShadeEnvelope(
type = ShadeEnvelope.EnvelopeType.RATCHET,
content = msg,
timestamp = 0,
senderAddress = "",
)
val encoded = WireFormat.encodeEnvelope(envelope)
assertEquals(v.getString("encoded"), hex(encoded))
// Roundtrip decode
val decoded = WireFormat.decodeEnvelope(encoded)
assertEquals(ShadeEnvelope.EnvelopeType.RATCHET, decoded.type)
val rm = decoded.content as RatchetMessage
assertEquals(msg.counter, rm.counter)
val decoded = WireFormat.decodeEnvelope(encoded)
assertEquals(ShadeEnvelope.EnvelopeType.RATCHET, decoded.type)
val rm = decoded.content as RatchetMessage
assertEquals(msg.counter, rm.counter)
assertEquals(hex(msg.ciphertext), hex(rm.ciphertext))
}
assertTrue("No ratchet wire vectors found", found)
}
@Test
fun wireFormatPreKeyVectorsMatch() {
val vectors = loadVectors("wire-format.json")
var matched = 0
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
if (v.optString("kind") != "prekey") continue
matched++
val m = v.getJSONObject("message")
val inner = m.getJSONObject("inner")
val innerMsg = RatchetMessage(
dhPublicKey = fromHex(inner.getString("dhPublicKey")),
previousCounter = inner.getInt("previousCounter"),
counter = inner.getInt("counter"),
ciphertext = fromHex(inner.getString("ciphertext")),
nonce = fromHex(inner.getString("nonce")),
)
val preKeyId: Int? = if (m.isNull("preKeyId")) null else m.getInt("preKeyId")
val pre = PreKeyMessage(
registrationId = m.getInt("registrationId"),
preKeyId = preKeyId,
signedPreKeyId = m.getInt("signedPreKeyId"),
ephemeralKey = fromHex(m.getString("ephemeralKey")),
identityDHKey = fromHex(m.getString("identityDHKey")),
message = innerMsg,
)
val envelope = ShadeEnvelope(
type = ShadeEnvelope.EnvelopeType.PREKEY,
content = pre,
timestamp = 0,
senderAddress = "",
)
val encoded = WireFormat.encodeEnvelope(envelope)
assertEquals(v.getString("encoded"), hex(encoded))
val decoded = WireFormat.decodeEnvelope(encoded)
assertEquals(ShadeEnvelope.EnvelopeType.PREKEY, decoded.type)
val dm = decoded.content as PreKeyMessage
assertEquals(pre.registrationId, dm.registrationId)
assertEquals(pre.preKeyId, dm.preKeyId)
assertEquals(pre.signedPreKeyId, dm.signedPreKeyId)
assertEquals(hex(pre.ephemeralKey), hex(dm.ephemeralKey))
assertEquals(hex(innerMsg.ciphertext), hex(dm.message.ciphertext))
}
assertTrue("Expected at least 2 prekey vectors", matched >= 2)
}
private fun findVector(arr: JSONArray, prefix: String): JSONObject {
for (i in 0 until arr.length()) {
val o = arr.getJSONObject(i)
if (o.getString("description").startsWith(prefix)) return o
}
throw AssertionError("Vector with description prefix '$prefix' not found")
}
@Test
fun streamsVectorsMatch() {
val vectors = loadVectors("streams.json")
// 1. deriveStreamKey
val sk = findVector(vectors, "deriveStreamKey")
val streamSecret = fromHex(sk.getString("streamSecret"))
val streamId = fromHex(sk.getString("streamId"))
val streamKey = deriveStreamKey(crypto, streamSecret, streamId)
assertEquals(sk.getString("streamKey"), hex(streamKey))
// 2. deriveLaneKey
val lk = findVector(vectors, "deriveLaneKey")
val lkStreamKey = fromHex(lk.getString("streamKey"))
val lkStreamId = fromHex(lk.getString("streamId"))
val lanes = lk.getJSONArray("lanes")
for (i in 0 until lanes.length()) {
val lane = lanes.getJSONObject(i)
val laneId = lane.getLong("laneId")
val k = deriveLaneKey(crypto, lkStreamKey, lkStreamId, laneId)
assertEquals(lane.getString("laneKey"), hex(k))
}
// 3. buildChunkNonce
val nv = findVector(vectors, "buildChunkNonce")
val nonces = nv.getJSONArray("nonces")
for (i in 0 until nonces.length()) {
val n = nonces.getJSONObject(i)
val laneId = n.getLong("laneId")
val seq = java.lang.Long.parseUnsignedLong(n.getString("seq"))
val out = buildChunkNonce(laneId, seq)
assertEquals(n.getString("nonce"), hex(out))
}
// 4. buildChunkAad
val av = findVector(vectors, "buildChunkAad")
val avStreamId = fromHex(av.getString("streamId"))
val cases = av.getJSONArray("cases")
for (i in 0 until cases.length()) {
val c = cases.getJSONObject(i)
val laneId = c.getLong("laneId")
val seq = java.lang.Long.parseUnsignedLong(c.getString("seq"))
val isLast = c.getBoolean("isLast")
val out = buildChunkAad(avStreamId, laneId, seq, isLast)
assertEquals(c.getString("aad"), hex(out))
}
// 5. End-to-end chunk encrypt + decrypt
val ev = findVector(vectors, "End-to-end chunk encrypt")
val laneKey = fromHex(ev.getString("laneKey"))
val nonce = fromHex(ev.getString("nonce"))
val aad = fromHex(ev.getString("aad"))
val plaintext = fromHex(ev.getString("plaintext"))
val ct = aesGcmEncryptWithNonce(laneKey, nonce, plaintext, aad)
assertEquals(ev.getString("ciphertext"), hex(ct))
val pt = aesGcmDecryptWithNonce(laneKey, nonce, fromHex(ev.getString("ciphertext")), aad)
assertEquals(ev.getString("plaintext"), hex(pt))
// 6. Wire 0x11 envelope encode/decode
val wv = findVector(vectors, "Wire 0x11")
val wire = StreamChunkWire(
streamId = fromHex(wv.getString("streamId")),
laneId = wv.getLong("laneId"),
seq = java.lang.Long.parseUnsignedLong(wv.getString("seq")),
isLast = wv.getBoolean("isLast"),
nonce = fromHex(wv.getString("nonce")),
aad = fromHex(wv.getString("extraAad")),
ciphertext = fromHex(wv.getString("ciphertext")),
)
val encoded = StreamChunkWireFormat.encodeStreamChunk(wire)
assertEquals(wv.getString("encoded"), hex(encoded))
val decoded = StreamChunkWireFormat.decodeStreamChunk(encoded)
assertEquals(hex(wire.streamId), hex(decoded.streamId))
assertEquals(wire.laneId, decoded.laneId)
assertEquals(wire.seq, decoded.seq)
assertEquals(wire.isLast, decoded.isLast)
assertEquals(hex(wire.nonce), hex(decoded.nonce))
assertEquals(hex(wire.ciphertext), hex(decoded.ciphertext))
// 7. Envelope-type inspector
assertEquals(
StreamChunkWireFormat.EnvelopeKind.STREAM_CHUNK,
StreamChunkWireFormat.inspectEnvelopeType(encoded),
)
}
@Test
fun backupVectorsMatch() {
val vectors = loadVectors("backup.json")
val kv = findVector(vectors, "Backup v1: HKDF")
val backupKey = deriveBackupKey(crypto, kv.getString("passphrase"), fromHex(kv.getString("salt")))
assertEquals(kv.getString("backupKey"), hex(backupKey))
val ev = findVector(vectors, "Backup v1: AES-256-GCM")
val ct = aesGcmEncryptDeterministic(
fromHex(ev.getString("backupKey")),
fromHex(ev.getString("nonce")),
fromHex(ev.getString("plaintext")),
ByteArray(0),
)
assertEquals(ev.getString("ciphertext"), hex(ct))
val pt = crypto.aesGcmDecrypt(
fromHex(ev.getString("backupKey")),
fromHex(ev.getString("ciphertext")),
fromHex(ev.getString("nonce")),
null,
)
assertEquals(ev.getString("plaintext"), hex(pt))
}
@Test
fun groupSenderKeyVectorsMatch() {
val vectors = loadVectors("group.json")
// 1. Header AAD
val hv = findVector(vectors, "Sender header AAD")
val aad = encodeSenderHeader(
hv.getString("groupId"),
hv.getString("senderAddress"),
hv.getInt("iteration"),
)
assertEquals(hv.getString("aad"), hex(aad))
// 2. Sender-key step
val sv = findVector(vectors, "Sender-key step")
val step = senderKeyStep(
crypto,
fromHex(sv.getString("chainKey")),
sv.getString("groupId"),
sv.getString("senderAddress"),
sv.getInt("iteration"),
)
assertEquals(sv.getString("newChainKey"), hex(step.newChainKey))
assertEquals(sv.getString("messageKey"), hex(step.messageKey))
assertEquals(sv.getString("aad"), hex(step.aad))
val ct = aesGcmEncryptDeterministic(
step.messageKey,
fromHex(sv.getString("nonce")),
fromHex(sv.getString("plaintext")),
step.aad,
)
assertEquals(sv.getString("ciphertext"), hex(ct))
// 3. Ed25519 verify on the recorded signature
val signed = senderSignedBytes(step.aad, ct)
val ok = crypto.verify(
fromHex(sv.getString("signingPublicKey")),
signed,
fromHex(sv.getString("signature")),
)
assertTrue("Sender-key signature verification failed", ok)
// 4. Decrypt roundtrip
val pt = crypto.aesGcmDecrypt(
step.messageKey,
fromHex(sv.getString("ciphertext")),
fromHex(sv.getString("nonce")),
step.aad,
)
assertEquals(sv.getString("plaintext"), hex(pt))
}
@Test
fun storageHkdfVectorsMatch() {
val vectors = loadVectors("storage-hkdf.json")
val sv = findVector(vectors, "Storage HKDF: storageKey")
val storageKey = crypto.hkdf(
fromHex(sv.getString("masterKey")),
ByteArray(0),
"shade-storage-v1".toByteArray(Charsets.UTF_8),
32,
)
assertEquals(sv.getString("storageKey"), hex(storageKey))
val fv = findVector(vectors, "Storage HKDF: fieldKey")
val fStorageKey = fromHex(fv.getString("storageKey"))
val fields = fv.getJSONArray("fields")
for (i in 0 until fields.length()) {
val f = fields.getJSONObject(i)
val info = "shade-field-v1:${f.getString("table")}:${f.getString("column")}"
.toByteArray(Charsets.UTF_8)
val k = crypto.hkdf(fStorageKey, ByteArray(0), info, 32)
assertEquals(f.getString("fieldKey"), hex(k))
}
val nv = findVector(vectors, "Storage HKDF: rowNonce")
val rowKey = fromHex(nv.getString("rowKey"))
val nonces = nv.getJSONArray("nonces")
for (i in 0 until nonces.length()) {
val n = nonces.getJSONObject(i)
val info = "shade-row-nonce-v1:${n.getString("table")}:${n.getString("pk")}"
.toByteArray(Charsets.UTF_8)
val out = crypto.hkdf(rowKey, ByteArray(0), info, 12)
assertEquals(n.getString("nonce"), hex(out))
}
}
@Test
fun ratchetStepRoundtripMatches() {
val vectors = loadVectors("ratchet-step.json")
assertTrue("ratchet-step vectors expected", vectors.length() > 0)
for (i in 0 until vectors.length()) {
val v = vectors.getJSONObject(i)
val inputs = v.getJSONObject("inputs")
val derived = v.getJSONObject("derived")
val rootKey = fromHex(inputs.getString("rootKey"))
val dhSendPriv = fromHex(inputs.getString("dhSendPrivateKey"))
val dhSendPub = fromHex(inputs.getString("dhSendPublicKey"))
val dhRemotePub = fromHex(inputs.getString("dhRemotePublicKey"))
val plaintext = fromHex(inputs.getString("plaintext"))
val nonce = fromHex(inputs.getString("nonce"))
val previousCounter = inputs.getInt("previousCounter")
val counter = inputs.getInt("counter")
// 1. DH
val dhOutput = crypto.x25519(dhSendPriv, dhRemotePub)
assertEquals(derived.getString("dhOutput"), hex(dhOutput))
// 2. kdfRootKey
val root = kdfRootKey(crypto, rootKey, dhOutput)
assertEquals(derived.getString("newRootKey"), hex(root.newRootKey))
assertEquals(derived.getString("chainKey"), hex(root.chainKey))
// 3. kdfChainKey
val chain = kdfChainKey(crypto, root.chainKey)
assertEquals(derived.getString("newChainKey"), hex(chain.newChainKey))
assertEquals(derived.getString("messageKey"), hex(chain.messageKey))
// 4. Header AAD
val aad = encodeRatchetHeader(dhSendPub, previousCounter, counter)
assertEquals(derived.getString("aad"), hex(aad))
// 5. AES-GCM encrypt with fixed nonce
val ciphertext = aesGcmEncryptDeterministic(chain.messageKey, nonce, plaintext, aad)
assertEquals(v.getString("ciphertext"), hex(ciphertext))
// 6. Roundtrip decrypt
val recovered = crypto.aesGcmDecrypt(
chain.messageKey,
fromHex(v.getString("ciphertext")),
nonce,
aad,
)
assertEquals(inputs.getString("plaintext"), hex(recovered))
}
}
}