75 shipped systems projects across 5 engineering tracks, every one a working implementation with committed benchmark reports. Each page below reports the results its own measurement harness produced — including the findings that changed my mind about the technique.

LLM inference and serving, agent reliability, multi-agent protocols, AI security, retrieval and memory, cost and observability, and data and post-training. Every page reports the results its own measurement harness produced — including the findings that changed my mind about the technique.

Inference and serving 3

PageServe

Inference engine

A mini LLM serving engine implementing continuous batching and a paged KV cache, verified token-for-token against Hugging Face at temperature 0 — delivering a 3.14x throughput gain over a sequential baseline and a 3.07x capacity gain from paging under heavy-tailed load.

• 3.07x capacity gain from paging on heavy-tailed load
• 3.14x throughput vs sequential baseline
• exact token-for-token match vs Hugging Face
• 31% peak KV memory cut by prefix sharing

Python PyTorch Qwen2 Apple M4 Pro

KVStudy

Inference study

A controlled study of KV-cache compression — quantization down to 2 bits, four eviction policies, and their combinations — establishing 4x memory reduction at no measured quality loss, and showing which metric detects the damage the others miss.

• 0.25x KV memory at no measured quality loss
• identical perplexity while accuracy fell 67 points
• 14 configurations in the matrix
• 25 tests

Python PyTorch Qwen2.5-0.5B Apple M4 Pro

SpecDec

Inference study

Speculative decoding with the exact rejection-sampling correction: verified token-for-token at temperature 0, verified distributionally above it, and benchmarked against an audited cost model that identifies the draft-to-target depth ratio as the CPU latency ceiling.

• 4.92 tokens per target pass (code, k=8)
• 90.9% peak draft acceptance rate
• 1.05x wall speedup, code at k=8
• 0.40 measured draft cost ratio

Python PyTorch Transformers Qwen2.5 Apple M4 Pro

Agent reliability 2

Trajeval

Agent evaluation

A regression-testing harness for AI agents: pass@k / pass^k with bootstrap confidence intervals, step-level trajectory scoring, and a CI gate that statistically distinguishes real regressions from run-to-run noise — with every estimator validated against agents whose true success rate is known.

• 6.0% measured A/A false-alarm rate
• 100% gate detection of 20-point regressions
• 1e-12 estimator agreement with closed form
• 31/31 tests passing

Python numpy SQLite Gemini Ollama GitHub Actions

Computer-Use Agent Sandbox

Agent evaluation

A vision-language-model browser agent measured end to end on a self-hosted replica shopping site with programmatic completion checkers: 36 tasks, 72 episodes, a scripted control that solves 36 / 36, and a DOM-vs-screenshot ablation that puts the DOM arm at 94% with Wilson intervals and an auditable failure taxonomy.

• 94% vs 83% DOM beats screenshot
• 36 / 36 scripted control solves every task
• [82%–98%] Wilson CI on the DOM arm
• 72 episodes across both modes

Python Playwright Ollama gemma4 31B FastAPI

Multi-agent systems 2

Durable Agents

Multi-agent systems

A supervisor/worker multi-agent system on LangGraph with crash-proof state, human approval gates, and a 30-task evaluation suite scored by programmatic checkers. Kill the process with `kill -9` at any point and the run resumes at the exact step it died on, with every side effect exactly once.

• kill -9 mid-run, resumes exactly-once
• 30/30 local Gemma 4 eval, vs 28/30 on API
• 150/150 offline suite, 30 tasks × 5 runs
• 13/13 tests against real Postgres

Python LangGraph Postgres 16 Ollama Gemini Typer

A2A Interop

Protocol engineering

Two agents on genuinely different frameworks — a hand-rolled loop and LangGraph — cooperating through an A2A-style protocol built from scratch, with chaos injection driving a nine-condition failure matrix in which every condition lands in a defined, bounded outcome. The harness pins down six specific places the protocol shape under-specifies itself.

• 9 / 9 chaos conditions land in a defined outcome
• 6 protocol gaps found by implementing
• 23 / 23 tests passing
• 0 shared code between the two agents

Python FastAPI LangGraph httpx SSE

AI security 2

InjectGuard

AI security

A prompt-injection defense where the protection is architectural: untrusted content can shrink an agent's capabilities but never expand them. Takes worst-case attack success from 100% to 0% at zero utility cost, evaluated on 12 attacks and 8 benign tasks across three model backends.

• 100% → 0% attack success, at zero utility cost
• 0/12 attacks land with the detector switched off
• 8/8 benign utility preserved in every arm
• 65 tests

Python Ollama Gemma 4 31B Gemini Typer

MCPGate

AI security

A governed MCP server: OAuth 2.1 with PKCE, declarative tool scoping, row-level data filters, per-identity rate limits, and a tamper-evident audit trail — refusing all 25 abuse cases with zero policy violations at 69 µs of median guard overhead.

• 25/25 abuse cases refused, zero policy violations
• 69 µs median guard-pipeline overhead
• 40/40 tests passing
• 1:1 audit events per call, allowed or denied

Python MCP SDK OAuth 2.1 PyJWT SQLite Typer

Retrieval and memory 2

RAGStudy

Retrieval study

A controlled comparison of static hybrid RAG against an agentic retrieval loop over one shared index, stratified into single-hop, multi-hop and unanswerable. The harness localises the entire gain to multi-hop (+0.267 / +0.333, replicated on two independent models) and meters its price at 2.2x the calls — exactly the evidence needed to route by question shape.

• +0.333 multi-hop gain (local model)
• +0.400 multi-hop gain vs context-matched arm
• ×2.19 metered LLM-call cost of the loop
• 2 models independent replication

Python Gemini Ollama BM25 embeddings bootstrap CIs

Memlayer

Agent memory

A long-term memory layer for agents: raw episodes consolidated into durable semantic facts with provenance and recency, retrieved by embedding similarity behind a precision floor, and corrections that supersede old facts while retaining them as auditable history. A three-arm harness (memory-off / on / oracle) isolates the layer's contribution from the model's own ability and attributes each probe to a named component, measuring a +92% memory contribution on a local model.

• +92% memory contribution over the off arm
• 86% / 92% retrieval precision / recall on the local model
• 4 / 4 correction scenarios with intact history
• 0% off-arm leak check, passing by construction

Python SQLite embeddings Ollama Gemini

Cost and observability 2

CostRouter

Routing study

A cost-aware LLM router over three tiers, evaluated on 180 public-benchmark items against the strictest available null: random routing at a matched escalation rate. The learned router leads that null across the budget grid, its margin is statistically significant from 40% escalation upward, and 13 interpretable surface features outperform 768-dimensional embeddings.

• 0.8496 frontier area (random 0.8276, oracle 0.9307)
• 13 > 768 surface features beat embeddings
• +5.8% over random at 50% escalation (p=0.0002)
• +35 pts tier gap on reasoning vs +6.7 creative

Python scikit-learn Gemini Ollama MMLU-Pro IFEval

GenAI Observability

Observability

An instrumented RAG agent emitting OpenTelemetry GenAI-convention telemetry through a collector into Jaeger/Prometheus/Grafana: 356 spans validated against the vendored spec model with zero violations, cost attribution per feature and tenant, tail sampling that keeps every error and every quality-regressed trace, and a staged prompt regression diagnosed from telemetry alone.

• 356 / 0 spans validated / violations
• 20 / 20 error traces kept by the sampler
• 6 / 6 regressed traces kept by the quality policy
• 2.8x sampling correction applied to reported spend

Python OpenTelemetry OTel Collector Jaeger Prometheus Grafana Ollama Gemini

Data and post-training 2

Synth Pipeline

Data pipeline

A synthetic data pipeline built around measured quality filtering: persona-conditioned generation, a validated four-stage filter stack, and a three-way training comparison in which the filtered arm reached the best held-out accuracy on less than half the unfiltered arm's training data.

• 33.3% filtered synthetic — best of three arms
• 126 → 38 candidates surviving the filter stack
• 74 vs 162 training rows for the best arm vs unfiltered
• 81 / 81 tests passing

Python embeddings SQLite Ollama Gemini

Toolcall DPO

Post-training

Post-training a small model for tool calling: DPO and GRPO written from scratch with verifier-graded rewards, decomposed evaluation and pair-quality ablations. DPO lifts held-out accuracy 16.1% → 21.5% with the second trained skill held at 100%, and the decomposed evaluation isolates GRPO reward hacking that a validity-only metric would have scored as a win.

• 16.1% → 21.5% held-out composite after DPO
• reward ↑, exactness ↓ GRPO reward hacking, isolated by decomposed eval
• 48 > 92 near-miss pairs beat twice as many mixed
• 17 / 17 tests passing

Python PyTorch DPO GRPO TRL export

The work of making AI agents function inside real organizations: wrapping legacy systems in governed tool layers, and giving operators the audit, replay, and kill-switch controls they ask for on day one.

Agent deployment 2

Enterprise Agent Deployment

Agent deployment

Three business workflows executed by LangGraph agents over a simulated customer's CRM, ticketing and document systems, each wrapped as a governed MCP server with human approval gated on effects rather than tool names. The trap that beat a per-tool gate is refused on all three routes with zero writes, and every action is provable after the fact against a tamper-evident audit chain of 4,200+ events.

• 99 tests, real Postgres + MCP over HTTP
• 3/3 evasion routes refused by the effect gate, zero writes
• 108/108 deterministic control arm, proving the harness
• 0 unlogged, phantom, ungranted writes in audit reconciliation

Python LangGraph MCP Postgres 16 Ollama FastAPI

Two-Week Prototypes

Engagement kit

One reusable engagement kit built once (1,455 lines), then three agent-powered apps for three customers — grounded RAG, a human-in-the-loop quoting agent, and governed MCP tools — each on a two-week clock with a measured before/after. The quoting agent scores 8/8 where the flat-fee baseline scores 0/8, and a rogue read-only token is denied on all 15 mutation attempts.

• 86 offline tests, kit + three engagements
• 8/8 vs 0/8 Frontdesk quotes vs flat-fee baseline
• 15/15 rogue read-only calls denied, 0 mutations
• 13/15 held-out generalization probe

Python MCP SQLite Ollama Gemini GitHub Actions

Legacy modernization 2

Legacy MCP

Integration layer

A governed MCP tool layer over an authentically awful legacy stand-in — a SOAP service fronting cryptic stored procedures — where a scripted generic agent completes 0/6 realistic tasks against the raw interface and 6/6 through the wrapper, protected by a rate cap and circuit breaker whose behavior is measured against a live degraded backend.

• 0/6 → 6/6 task success, raw interface vs wrapped, end-state checkers
• 4 calls total backend contact while circuit-breaking a 2s-slow backend
• 0/25 vs 20/25 SYSTEM BUSY faults, paced wrapper vs unthrottled caller
• 33/33 tests passing

Python MCP SDK FastAPI httpx SQLite

SchemaMap

Data integration

LLM-assisted schema mapping with human confirmation. Proposes source-to-target column mappings from name, type and value-distribution evidence, calibrates its own confidence on a held-out split, and flags proposals where its evidence disagrees with itself. Top-1 accuracy is 51/54 against 8 planted name traps, all 3 errors land inside the flagged set, 0 of 29 auto-accepts are wrong, and reviewer effort drops 43%.

• 51/54 top-1 accuracy, vs 8 planted name traps
• 0/29 wrong mappings auto-accepted
• 43% fewer column inspections, 504 to 289
• 0.0105 ECE calibrated, vs 0.2429 uncalibrated

Python SQLite Typer Ollama Gemini

Agent operations 3

Governance Console

Operations console

A governance console for a fleet of heterogeneous agents: three frameworks with unrelated native event formats normalized into one audit timeline with exact cost attribution, session replay, and capability-token revocation whose mid-flight halt bounds are measured — 26ms mean online, and TTL-bounded (with up to 7 leaked effects quantified) behind an offline verifier.

• 26ms / 50ms mean / max mid-flight halt, online gateway, 10/10 within bound
• 7 effects max leakage behind an offline verifier before TTL expiry — quantified
• exact cost reconciliation across per-agent and per-tenant rollups
• 33/33 tests passing

Python LangGraph SQLite HMAC capability tokens

Voice Agent Ops

Voice operations

A phone-line voice agent for a service shop — booking, rescheduling, cancelling, status — evaluated by a simulated-caller harness that drives real 8 kHz audio through the full stack under seven failure conditions, with every claimed outcome verified against the backend database. Confirming every high-stakes slot takes wrong bookings from 17% to 0%, and p95 turn latency holds at 929 ms.

• 17% -> 0% wrong bookings, no-confirm vs confirm-everything
• 929 ms p95 turn latency, 0 of 559 turns over budget
• 98 ms barge-in overlap, vs 1,971 ms unhandled
• 593 tests in ~72 s, fully offline

Python NumPy SQLite Typer Rich

CUA vs RPA

Automation study

A computer-use agent, a scripted RPA bot, and a script-first/agent-on-failure hybrid, run head-to-head on the same invoice-entry workflow across five levels of UI drift — 750 measured episodes scored by reading the app's stored state. The script holds 100% through cosmetic change and falls to 0% the moment structure drifts; the agent and the hybrid hold 100% at every level.

• 750 episodes, 0 harness faults
• 100% agent success at every drift level, one code path
• 0% script success at L2–L4; breaks, never miswrites
• 334x margin on the hybrid-beats-script conclusion

Python Playwright Chromium uv GitHub Actions

Documents and field data 2

DocPipe

Document pipeline

A self-verifying document pipeline: arithmetic and cross-document rules plus dual-strategy disagreement route only genuine uncertainty to humans, and a correction loop lifts frozen-holdout accuracy from 91.79% to 97.92% while a frozen control arm stays flat. Straight-through processing rises from 6.86% to 45.10% with an escaped-error rate held at 0.00%.

• 91.8% -> 97.9% frozen-holdout accuracy after 2 weeks of corrections
• +38.24 pts straight-through processing, 6.86% to 45.10%
• 0.00% escaped-error rate at the operating threshold
• 18/18 induced vendor priors match hidden ground truth

Python SQLite Typer Rich GitHub Actions

FieldSync

Field data sync

Offline-first field data capture with sync: per-field merge policies over a causal oplog, resumable media upload, and schema-version negotiation. Across 200 randomised offline-edit schedules this design loses nothing, while three control designs replaying the identical op log lose 1,255–1,340 pieces of inspection content and silently settle 279–436 professional disagreements by whichever clock ran fast.

• 0 / 0 / 0 content lost, corrections reverted, disagreements swallowed
• 1,255–1,340 pieces lost by the control designs, same op log
• 13/13 interrupted uploads resume byte-exact, 0 chunks wasted
• 1.6 s 20-device storm convergence, 1,500/1,500 ops durable

Python FastAPI SQLite IndexedDB Hypothesis Chromium

Analysis copilots 4

Ops Copilot

Analytics copilot

A copilot that answers plain-English questions about a business warehouse by choosing a metric from a curated semantic layer — never by writing SQL. It scores 100/100 on a 100-question benchmark (70 answered, 30 refused with the right reason), every answer ships with lineage, and the modelling traps the layer prevents are measured: a fluent query that joins shipments to order lines overstates revenue by 9.9%.

• 100/100 benchmark — 70/70 answered, 30/30 refused
• 30/30 refusal reasons correct, across 9 classes
• +9.9% revenue overstatement from the fan-out trap, measured
• 0 string literals in 177 compiled query shapes

Python SQLite YAML Ollama Rich GitHub Actions

FinOps Agent

Cost analysis agent

A cloud cost analysis agent that reads two months of billing, finds waste across four classes at 1.00 precision and recall on planted scenarios, and drafts a reviewable remediation for each finding — never auto-applied, graded by execution on a mock cloud. A savings-verification loop grades its own projections against the next month's bill: 11/12 verified, portfolio 4.3% off, and the one engineered shortfall flagged at a 61% miss.

• 1.00 / 1.00 detection precision and recall, 4 waste classes
• 4.3% portfolio projection error, $1,462.97 vs $1,399.52
• 11/12 savings verified; the 12th flagged at a 61% miss, by design
• 12/12 remediations reach the intended end state on the mock cloud

Python SQLite Bash Terraform Ollama

Compliance Evidence Agent

Compliance agent

An agent that collects SOC 2 evidence for 15 controls out of four company systems over scoped MCP tools, stores every artifact in a hash-chained ledger, and refuses to call a control done when the evidence is stale, partial, misaimed, or unattested. It finds all six planted deficiencies with the right reason for each — the same evidence through an existence-only checker reports 15 of 15 green, including through a credential revocation that blinded it completely.

• 6/6 planted deficiencies found, each with the correct reason
• 0 vs 10 silent gaps under credential revocation, this checker vs existence-only
• 8/8 tamper classes caught by the expected detector
• 20/20 mutation-check guards caught

Python MCP SQLite YAML Typer GitHub Actions

TwinSim

Digital twin

A digital twin of a fulfilment operation that must pass a pre-registered backtest on sealed weeks before any what-if is trusted: shipped-throughput MAE 1.9% against a 5.4% budget, with 90% predictive intervals covering the actual in 32/32 week-by-metric cells. A guarded, metered proposal agent then optimises against the twin — greedy confirms +3.28 pts on-time on fresh seeds vs +2.56 for equal-budget random, and the winner stays positive in 15/15 sensitivity rows.

• 1.9% holdout MAE vs a 5.4% pre-registered budget
• 32/32 predictive-interval coverage, week x metric
• +3.28 pts confirmed on-time gain, fresh seeds, 90% CI [+0.83, +5.72]
• 15/15 sensitivity rows positive; worst row +1.55

Python FastAPI Typer Rich qwen3:8b GitHub Actions

Governed data access 2

Permission-Aware RAG

Governed retrieval

Enterprise RAG where the access decision happens inside retrieval and is re-verified against the source of truth before any text reaches a generator. Zero leaks across 20 adversarial probes and three leak surfaces while all 17 entitled questions are answered — and the suite mutates permissions mid-flight, because two of the four unsafe control designs are byte-identical to the correct one until something changes.

• 0 leaks across 20 probes and 3 surfaces
• 17/17 positive controls answered, citing the gold chunk
• 0 vs 6 stale-version citations, with vs without the freshness guard
• 13.1 ms p95 permission propagation at sweep cadence 1

Python SQLite BM25 Ollama GitHub Actions

Multi-Org Data Sharing

Data sharing

Two organizations share a governed slice of one Postgres database, where the caller is often a partner's AI agent acting on a human's delegated authority. Row and column limits are enforced by Postgres RLS from grant tables, credentials are short-lived signed claims the database verifies for itself, and 30 adversarial attacks across four families pass with zero bypasses while revocation halts an agent in 7.9 ms mean.

• 30/30 adversarial attacks held, zero bypasses
• 7.9 ms mean revocation halt, 0 rows after commit
• 4 bypasses in the app-filter control arm, all masked-column channels
• 12/12 mutation-check mutants caught

Python Postgres 16 plpgsql pgcrypto uv

Developer infrastructure for the agent era — sandboxed execution, repo-scale context, durable workflows, and plugin isolation.

Sandboxing and isolation 2

Sandboxd

Execution service

A code-execution service for AI agents, built twice — on gVisor and on Firecracker microVMs — so the isolation and latency claims are measured and compared rather than asserted. 16/16 hostile payloads contained on both backends, and a warm pool that cuts Firecracker latency 44x, from 3.7 s cold to 83 ms.

• 16/16 hostile payloads contained, both backends
• 44x Firecracker: 3665 ms cold to 83 ms warm
• 23.5 ms warm-pool p50 on gVisor
• ~60 ms warm cost of a hardware boundary

Rust tokio axum gVisor Firecracker cgroup v2

CapBox

Plugin sandbox

A WASM plugin sandbox on Wasmtime where plugins hold explicit capability grants and nothing else — proven by a 12-payload hostile suite in which every containment claim is falsifiable, and priced by an overhead benchmark against a native plugin interface. The warm call boundary costs ~80 ns.

• 12/12 hostile payloads contained, all falsifiable
• ~80 ns warm call boundary vs native
• ~13 µs per-call isolation, fresh store + instance
• 1.4–1.7x fuel-metering tax on guest compute

Rust Wasmtime WebAssembly Go WAT

Coding-agent infrastructure 4

RepoCtx

Context engine

A repository context engine for coding agents: tree-sitter symbol chunking, hybrid BM25 + dense + call-graph retrieval, and a hard token-budget packer — evaluated on 50 tasks mined from real merged PRs. 0.907 recall@16k vs 0.825 single-strategy, and the same 8B agent's file recall jumps 3.4x when the engine fills the window.

• 0.907 recall@16k, hybrid + graph, vs 0.825 BM25
• 3.4x agent file recall, 0.703 vs 0.205 without
• 86% task hit rate, vs 32% without engine
• 63 ms p50 full query path, p95 91 ms

Python tree-sitter BM25 Ollama nomic-embed-text uv

ASTReview

Code review bot

An AST-aware AI code-review bot: tree-sitter semantic diffing classifies each change at the symbol level, rules catch the mechanical bug classes for zero tokens, and the LLM reviews only hunks that can contain a behavior bug. On the labeled corpus it scores 1.000 precision and 1.000 recall vs 0.833 and 0.500 for a naive whole-diff baseline, with fewer LLM calls.

• 10/0 findings TP/FP, vs 5/1 for the baseline
• 1.000 precision, vs 0.833 whole-diff baseline
• 2x recall, 1.000 vs 0.500 baseline
• 8 vs 12 LLM calls — triage skips a third of spend

Python tree-sitter httpx uv pytest

SpecForge

Dev toolchain

A spec-driven dev toolchain: machine-readable feature specs compile deterministically to pytest suites that gate sandboxed coding agents, plus an ambiguity linter. On a 20-feature benchmark with a local 31B model, detailed specs solved 20/20 on the first attempt; terse prose solved 17/20 at 1.7x the attempts and 1.7x the wall time.

• 20/20 detailed specs solved, 100% first-attempt
• 17/20 terse prose, at 1.70 mean attempts
• 308 s mean wall per feature, vs 515 s terse
• 109 tests, all 20 benchmark features CI-verified

Python pytest YAML Ollama uv

Retrace

Replay debugger

A record/replay/diff debugger for AI agents: 19/19 committed sessions replay byte-identically with every boundary mocked, and a validated trajectory aligner (2000/2000 planted-divergence trials) pinpoints the first divergence — including run pairs where both give the right answer while the drifted one silently ignores its own tools.

• 19/19 sessions replay byte-identically
• 2000/2000 planted divergences recovered, kind + index
• 4/4 negative controls correctly fail
• ~75 B dedup break-even, measured by sweep

Python Typer Ollama http.server GitHub Actions

Governed tool layers 2

MCPGov

Governed MCP server

A governed MCP server over Postgres: FORCE-RLS tenant isolation, exactly-once mutations under 16-thread retry storms, a loop breaker with 100% recall and 0 false breaks (median 8 wasted calls), and a fork-proof tamper-evident audit chain. 100 Postgres-backed tests plus a 13-act adversarial HTTP demo run in CI.

• 100% runaway loops broken, median 8 wasted calls
• 0 false breaks on legitimate workloads
• 16 threads retry storm, every mutation exactly-once
• 13-act adversarial HTTP demo, asserted in CI

Python Postgres MCP JWT uv

ToolGate

Policy engine

IAM for coding agents: declarative YAML policies compile to decision functions that gate every tool call outside the model, with an escalation queue and a hash-chained audit log. The balanced policy blocks 93/93 bypass attempts while allowing 38/38 benign operations, deciding in p50 210 µs against a 5 ms budget.

• 93/93 bypass attempts blocked, balanced policy
• 38/38 benign operations allowed, same policy
• 210 µs p50 decision latency, p99 0.67 ms
• 306 tests; bypass suite gates CI

Python YAML JSONL uv pytest

Durable and from-scratch systems 3

MiniTemporal

Durable execution

A durable-execution engine (mini-Temporal) in ~2k lines of zero-dependency Go: workflows survive SIGKILL at any instant and resume at the exact step. A chaos suite killed the worker 30 times across 100 executions and audited zero lost and zero duplicated external effects, with deterministic replay at ~200k events/s.

• 30 SIGKILLs across 100 chaos workflows
• 0 / 0 lost / duplicated external effects
• ~200k events/s deterministic replay
• 7 duplicate deliveries suppressed by idempotency

Go JSONL CRC32 fsync Bash

Syncd

Sync engine

A local-first sync engine — server-authoritative mutation log with client rebase and permission-scoped partial replication — in ~1.8k lines of zero-dependency TypeScript. Property-tested, not asserted: 400/400 concurrent-edit schedules converge byte-for-byte, and 11,144 replicated rows audited against an independent permission oracle show 0 leaks.

• 400/400 concurrent-edit schedules converge byte-for-byte
• 0 leaks 11,144 replicated rows audited vs oracle
• 10.6 ms p50 cross-client sync latency
• 4.6x fewer pokes under a 100-client storm

TypeScript Bun SSE HTTP localStorage

HNSWdb

Vector database

A vector database from scratch in zero-dependency Go: HNSW index plus filtered search with cost-based strategy switching, benchmarked against exact ground truth on SIFT1M. 0.994 recall@10 at 1.8 ms p50, ~100x faster than brute force at 0.97 recall, and an auto filter planner 2,500x faster than fixed graph traversal at 0.1% selectivity.

• 0.994 recall@10 at 1.8 ms p50, SIFT1M
• ~100x vs exact scan (~81 ms) at 0.97 recall
• 2,500x auto planner vs fixed traversal at 0.1% selectivity
• 127 s to build 1M vectors, ~7,900 vec/s

Go HNSW HTTP SIFT1M Make

Delivery, evaluation, and caching 3

EvalGate

Evaluation CI

Evaluation-driven CI for AI features: tiered golden sets, paired bootstrap significance on per-case score deltas, judges validated against programmatic truth, and a PASS / BLOCK / INCONCLUSIVE merge gate that escalates tiers only when it cannot decide. On the 20-scenario matrix it blocks 12/12 true regressions with 0/8 false blocks.

• 12/12 true regressions blocked, 0 missed
• 0/8 false blocks on not-worse changes
• 87% false-pass rate of the naive judge, vs 0% strict
• 4/4 live prompt variants, verdict matches oracle

Python Typer Ollama GitHub Actions uv

Tripwire

Progressive delivery

Progressive delivery for prompts and models: sticky canary routing plus an always-valid mSPRT sequential test driving automatic promote and rollback. It peeks after every request without inflating the false-rollback rate — 0/2000 false rollbacks on A/A traffic where a naive peeked interval fires 234/2000 times.

• 0/2000 A/A false rollbacks, vs 234/2000 naive
• 2.3x how far the naive peeked test overshoots alpha
• 400/400 -0.20 regressions caught, median 131 labels
• 24/24 to 0/24 live prompt regression, auto-rolled back

Python Typer Ollama http.server GitHub Actions

SemCached

Caching proxy

A semantic caching proxy for LLM APIs that treats the wrong-hit rate as a first-class, measured number: on human-labeled replay traffic the shipped 0.92 threshold buys a 33.1% hit rate at a 6.0% wrong-hit rate and a 32.4% cost reduction, with tenant and prompt isolation that is structural rather than probabilistic.

• 6.0% wrong-hit rate at the shipped 0.92 threshold
• 32.4% cost reduction on the QQP replay
• 33.1% overall hit rate at 0.92
• 1.4 ms exact-hit p50; semantic hit 7.3 ms

Python FastAPI ONNX Runtime bge-small-en-v1.5 uvicorn

Supply-chain security 1

BomSquad

Supply-chain security

A supply-chain security toolkit for npm and PyPI: CycloneDX SBOMs from lockfiles, sigstore/SLSA provenance verified down to the lockfile digest, and a typosquat plus hallucinated-import scanner. Name rules detect 28/54 documented historical attacks, and two-stage exoneration leaves 2 flags across 6,517 packages from real clean trees — a 0.031% false-positive rate.

• 28/54 documented historical squat attacks detected
• 0.031% FP rate — 2 flags in 6,517 real packages
• 98.4% PyPI malware census caught with registry state
• 106/479 real direct deps cryptographically verified

Python CycloneDX sigstore in-toto pypi-attestations

Latency engineering and market microstructure — measured in microseconds, verified by fuzzing and formal methods.

Tick-to-trade and matching 3

Tick2Trade

Trading pipeline

A Rust tick-to-trade pipeline with a published per-hop latency budget: 625 ns p99 internal in replay on an idle host, 583 ns under load, 6.3 µs p99 against the live venue, and zero hot-path allocations proven by a counting allocator across 3.5 million traversals. The budget identified HMAC signing as 80% of the traversal; a one-line dependency change bought a 1.5x pipeline speedup.

• 625 ns internal p99 in replay, 3.5M traversals
• 6.3 µs internal p99 live, same hops, gap published
• 0 hot-path allocations, proven by counting allocator
• 1.5x pipeline speedup from a one-line HMAC fix

Rust rustls WebSocket HDR histograms Criterion Binance bookTicker

Matchbook

Matching engine

A zero-unsafe Rust limit-order-book matching engine verified against a BTreeMap model and two C++ reference builds of the same written contract, plus 2.49 billion fuzz executions with every invariant checked after every command. Throughput lands within 1.01-1.09x of the pointer-based C++ build; the four-way diff found a real defect the fuzzers structurally could not.

• 2.49B fuzz executions, 0 crashes
• 1.01-1.09x of pointer-based C++ throughput, 4 workloads
• 4-way byte-identical agreement, gated before timing
• 29% smaller order slot than the C++ pointer build

Rust C++ cargo-fuzz libFuzzer proptest

ITCH-RTL

Hardware feed parser

A SystemVerilog ITCH 5.0 feed parser verified bit-identically against a software golden model over 5,000,000 messages — 0 mismatching bytes — at a fixed 1-cycle latency for all five message types. Yosys estimates 447 LUTs and 832 flip-flops on Xilinx 7-series; a 400-fault malformed-input suite checks that every rejection fires by the exact expected reject code.

• 5M messages diffed, 0 mismatching bytes
• 1 cycle fixed latency, min equals max for all 5 types
• 447/832 LUTs and FFs, Yosys 7-series estimate
• 400/400 injected faults caught by the exact reject code

SystemVerilog Verilator Yosys C++ Python

Execution and routing 2

Shortfall

RL execution study

RL for optimal execution, measured honestly: PPO beats tuned TWAP and Almgren-Chriss baselines by 0.86 bps (9% of all-in cost, paired bootstrap CI) over 200,000 common-random-number episodes and captures 94% of an exact dynamic-programming oracle ceiling. Frozen-weight OOD stress includes the scenario where the edge is designed to vanish — and it does, degrading to TWAP, not blowing up.

• 0.86 bps edge vs tuned TWAP and AC, CI clear of zero
• 94% of the exact-DP oracle ceiling captured
• 200k paired common-random-number episodes
• +0.03 bps when the signal is flattened — fails safe to TWAP

Python PyTorch NumPy pytest uv

Latency-Aware SOR

Order router

A latency-aware smart order router replayed over recorded multi-venue crypto book data with honest arrival modeling: +4.68 bps of fee-adjusted execution price versus best-displayed routing, decomposed into 4.26 bps fees, 0.20 staleness, 0.22 splitting. The win holds at 4.58-4.75 bps across a 4x sweep of latency-model error, and drops to -0.16 bps with all fees zeroed — most of the win is fees, measured.

• +4.68 bps vs best-displayed routing, fees included
• 4.26/0.20/0.22 bps from fees, staleness, splitting — residual zero
• 4.58-4.75 bps held across a 4x latency-model-error sweep
• -0.16 bps with every fee zeroed — the honest counterfactual

Python Decimal arithmetic YAML pytest uv

Prediction and research 4

LOBGap

Prediction study

DeepLOB on FI-2010, evaluated twice: 75.7% accuracy under the published split versus 73.4% under purged per-instrument walk-forward, with the +2.2-point gap counted leak by leak — 10 footprint overlaps, 396 train and 1,386 test windows mixing spliced instruments. The tape's own oracle bound settles the economics: -13.15 bps net per trade at any classifier accuracy.

• +2.2 pts accuracy bought by the published protocol's leakage
• 73.4% purged walk-forward accuracy, the believable number
• -13.15 bps oracle net per trade after costs — uneconomic at any accuracy
• 1,386 test windows mixing spliced instruments, counted

Python PyTorch FI-2010 NumPy uv

PitSignal

Signal pipeline

A point-in-time LLM signal pipeline over 255 real SEC 8-K earnings filings where PIT discipline is enforced by the type system and every evaluation is bracketed by controls: a planted arm that must fire (t = 15.8) and a shuffled arm that must stay silent (t = 0.55). The naive filing-date join places 94 of 255 fills before the document existed; the honest event study, clustered by issuer, says the naive text signal is not significant.

• t = 15.8 planted control fires — the study can see signal
• t = 0.55 shuffled control silent — no manufactured significance
• 94/255 naive-join fills placed before publication
• 34/255 wrong EDGAR timestamps caught, 4-5 hours early

Python Ollama SEC EDGAR Yahoo Finance pytest uv

TickStack

Research stack

The modern tick research stack — Arrow in memory, Parquet on disk, DuckDB for queries — benchmarked against a fair pandas baseline on the five canonical quant queries, with wins from 2.8x to 19.5x. Every timing is gated by a correctness check that reconstructs the order book from the market-by-price stream and matches ground truth at 1,048,318 of 1,048,318 quote timestamps.

• 2.8-19.5x DuckDB over fair pandas, all five queries
• 1.05M/1.05M book timestamps matched against ground truth
• ~200x harness distortion found and fixed before recording
• 207 ms per-symbol partitioning at 30 days vs 14 ms single file

DuckDB Apache Arrow Parquet pandas Python pytest

GBDT2C

ML inference

A GBDT trading signal compiled to branch-lean C++, with bit-exactness as a first-class deliverable: 1,000,000 of 1,000,000 held-out book events reproduce XGBoost's own margins bit for bit, features included. The end-to-end hot path runs at 500 ns p50 and 1.58 µs p99 — 228x XGBoost's native single-row predict and 6.6x ONNX Runtime — with the honest negatives kept: XGBoost wins batch scoring 4.7x, and the quantized variant is slower than inlined branches.

• 1M/1M margins bit-exact vs XGBoost, features included
• 500 ns event-to-margin p50, incl. 16 features; p99 1.58 µs
• 228x vs native XGBoost single-row predict
• 6.6x vs ONNX Runtime single-sample, 1 thread

C++ Python XGBoost ONNX Runtime clang Binance L1 data

Market simulation 1

PitSim

Market simulator

An agent-based adversarial market simulator whose baseline LOB population quantitatively reproduces five stylized facts of real markets (excess kurtosis 4.03, Hill index 4.49, volatility clustering), plus a bandit adversary that learns quote-matching erases a fixed-clock market maker's edge (-26 ticks/h paired) at +322/h profit while manipulation loses 100x what it extracts. The intuitive clock-jitter defense fails; faster refresh restores the baseline.

• 5/5 stylized facts reproduced, 30 seeds x 4 hours
• +322/h adversary profit from the learned quote-matching arm
• -26 ticks/h victim edge erased, paired against same-seed baseline
• 100x momentum ignition costs more than it extracts

Go Go stdlib only Python matplotlib

Risk and surveillance 3

Deadman

Kill switch

A provably safe trading kill switch: a two-phase halt state machine model-checked in TLA+ so no order passes once the stop completes, and a Go core in one atomic word mirroring the spec action-for-action. Conformance tests replay all 23,240 model event sequences against the implementation, and the order-path enabled-check costs 0.39 ns.

• 23,240 event sequences replayed from the model's own graph
• 0.39 ns enabled-check, a single atomic load
• 4 states for TLC to refute the naive one-phase design
• 54/54 labeled model edges covered by conformance tests

Go TLA+ TLC model checker Go race detector

Broker Rails

Risk rails

An MCP trading server with non-bypassable pre-trade risk rails: enforcement runs in the SDK dispatcher, so a refused order never reaches the handler. 21 adversarial attempts to exceed a limit produce zero rail violations, a 7-day unattended session holds every limit against a deliberately greedy agent, and the whole session rebuilds from a tamper-evident audit log of 1,710 records.

• 21/21 adversarial attacks refused by the expected rail
• 0 rail violations across direct, protocol, injection channels
• 7 days unattended, 233 refusals, every limit held
• 1,710 audit records — chain verified, session rebuilt

Python MCP YAML Alpaca paper API pytest uv

Surveil

Surveillance

Streaming market-manipulation surveillance — spoofing, layering, wash trading, momentum ignition — measured against injected ground truth on a 4.6M-event stream: the learned scorer detects 100% of 200 injected episodes at 2.15 false positives per hour versus 81% at 5.85 for hand-tuned rules. The gap is widest on subtle manipulation, where rules catch 42.5% and the learned model catches all of it.

• 100% detection, 200/200 injected episodes
• 2.15/hr false positives vs 5.85 for the rule baseline
• 381 ms p50 detection latency vs 1126 ms for rules
• 4.6M events, 6.5-hour session, single core

Python pytest discrete-event simulation streaming features

Funding and payments 2

PerpCarry

Arb monitor

A cross-venue perp funding-rate arbitrage monitor normalizing 4 venues and 3 funding conventions into one no-lookahead expected-carry model with all-in costs and isolated-margin liquidation on both legs. A 30-day paper replay over real data delivers the honest verdict: $143.96 of funding collected, -$29.23 net after $206.22 of execution costs, with the model-vs-realized gap decomposed exactly per trade.

• -$29.23 net after costs — the honest 30-day verdict
• $206.22 execution costs vs $143.96 funding collected
• 4 venues 3 funding conventions normalized, no lookahead
• 10/11 trades liquidated at 20x; zero at 5x or below

Go Bybit API Gate API Hyperliquid API Deribit API

PayPer

Machine payments

An x402 machine-payable API with exactly-once payment accounting: 5,289 settlements reconcile exactly across 631 server kills — 0 lost, 0 double-charged, 0 double-refunded. The crypto (Keccak-256, secp256k1, EIP-712) is built from scratch in zero-dependency Go and verified byte-for-byte against eth-account ground truth.

• 5,289 settlements reconcile exactly, 0 pending
• 631 SIGKILLs mid-payment, zero money lost
• 0 double-settles on chain, 0 discrepancies
• 21.9 ms p50 full paid call, probe to sign to settle to serve

Go x402 EIP-3009 EIP-712 secp256k1 Keccak-256

The service layer AI products stand on — gateways, rate limits, metering, and zero-downtime operations.

Serving AI traffic 3

LLM Gateway

LLM gateway

One OpenAI-compatible API in front of many providers, in zero-dependency Go. Killing a provider under 120k requests of live load produced zero client-visible errors, the cost ledger reconciles exactly against the providers' own logs (120,901 of 120,901 rows, to the picodollar), and the gateway adds 1.84 ms p99 of its own latency at ~19.5k req/s on one laptop.

• 1.84 ms p99 added latency, self-timed
• 120,901/120,901 ledger rows reconcile exactly
• 0 client errors, provider killed under 120k reqs
• 4.8× median cache speedup, 100% hit on repeat

Go OpenAI API Anthropic API SSE Prometheus k6

TPMGate

Rate limiter

A distributed token-based rate limiter for AI APIs in zero-dependency Go on Redis, including a hand-rolled RESP client. Admission spends an estimate inside an atomic Lua script and settles against the true cost: three independent nodes firing 20,000 concurrent requests at a bucket that fits 2,000 admit exactly 2,000, and the decision ledger reconciles 120 of 120 rows against the provider log with no tolerance term.

• 2,000/2,000 admitted from 20k concurrent, 3 nodes
• 0 over-admissions, zero errors
• 120/120 ledger rows reconcile, no tolerance
• <0.02 ms enforcement cost above a bare Redis PING

Go Redis Lua RESP2/RESP3 Prometheus

TokenWire

Streaming backend

A streaming inference backend in zero-dependency Go — SSE plus a hand-rolled RFC 6455 WebSocket — with bounded per-connection buffering. 200 streams force-disconnected six times each resumed exactly: 40,000 tokens verified token-for-token with 0 lost and 0 duplicated, and cancellation reaches the model backend in about a millisecond, saving 97.5% of the token spend.

• 0 / 0 tokens lost / duplicated over 1,200 disconnects
• 40,000 tokens verified against the model's own log
• ~1 ms cancellation propagation, 97.5% spend saved
• 1,000 concurrent streams, +12.7 ms added ITL p99

Go SSE RFC 6455 WebSocket Prometheus

Metering, billing, and payments 2

Meterd

Metering and billing

Usage metering and billing for AI products in zero-dependency Go: an idempotent event ledger, windowed aggregation with an explicit late-event policy, versioned plans with credits and caps, and ledger-replay reconciliation. A chaos harness delivered 292,792 events — duplicates, replays, global shuffle, 5% beyond grace, 1% dropped — and every cell, invoice, and gap reconciled exactly.

• 292,792 chaos deliveries reconcile exactly
• 19/19 billing scenarios vs hand-computed invoices
• 50/50 corrupted cells caught by drift detection
• 26,620 usage cells match ground truth both ways

Go HTTP JSONL CRC32-framed WAL

X402Gate

Payment infrastructure

Pay-per-request API infrastructure on the x402 model in zero-dependency Go: challenge, verify, settle, serve, with each payment bound to one response through a crash-safe write-ahead ledger. A failure-injection suite drove 3,000 payments through 6,432 attempts with 653 post-payment crashes; an independent auditor found zero double-charges, zero unpaid serves, and reconciliation that closes with drift 0.

• 0 / 0 double-charges / unpaid serves, chain-audited
• 653 post-payment crashes injected, all recovered
• drift 0 reconciliation closes exactly
• ~609× verify cost cut by caching, 10.955 to 0.018 ms p50

Go x402 EIP-712 EIP-3009 secp256k1 keccak256

Durable execution and sandboxing 2

Flowd

Durable execution

A durable-execution backend for agent workflows in zero-dependency Go: an event-sourced control plane with task leases, durable timers, and human signals, plus stateless replay workers. A chaos suite ran 300 executions through 165 worker SIGKILLs, 20 server SIGKILLs, and 10 rolling deploys mid-execution with zero lost and zero duplicated external effects.

• 0 / 0 lost / duplicated effects under chaos
• 165 + 20 worker and server SIGKILLs survived
• 300/300 histories byte-consistent under strict replay
• 10 rolling v1→v2 deploys with executions in flight

Go Event sourcing JSONL WAL HTTP Deterministic replay

Execd

Execution API

A multi-tenant sandboxed code-execution API in zero-dependency Go, in the shape of what E2B and Modal sell to AI teams. Tenant-tagged warm pools with a structural cross-tenant reboot gate showed 0 leaks across 7 attack categories, warm exec runs at p50 0.16 ms against 192 ms cold boot, and 288 ledger entries reconcile exactly against the independent usage log.

• 0 leaks 7 attack categories, reboot gate
• 0.16 ms warm exec p50 vs 192 ms cold boot
• 288/288 ledger vs usage log, exact reconcile
• 1 of 30 flood requests admitted at cap 1, victims unharmed

Go Python net/http JSONL Process sandboxing

Postgres as a platform 3

PGLimits

Limits study

An evidence-based limits study of "just use Postgres for everything": queue, full-text search, pub/sub, and vector workloads on one tuned Postgres 16 versus Redis, Elasticsearch, and Qdrant on identical data. SKIP LOCKED sustained 45-47k jobs/s against Redis's 8-11k, GIN beat Elasticsearch 22.9k to 15.2k qps, and interference was instrumented down to the autovacuum cycle.

• 45-47k jobs/s SKIP LOCKED, vs 8-11k Redis pattern
• 22.9k FTS qps vs 15.2k Elasticsearch
• 0.949 pgvector recall ceiling; Qdrant hit 0.994
• p99 328 ms episodic autovacuum interference spike, root-caused

Postgres 16 Go Redis Elasticsearch Qdrant pgvector

PGSieve

Sync backend

A permission-scoped sync backend on Postgres logical replication: clients subscribe as a user and get a snapshot plus a live stream containing only rows that user may see, with grants and revokes enforced in the replication path itself. A 19-test leak suite including 30-cycle revocation races converged exactly to SQL ground truth, and fan-out runs at 1.61 ms p50 to 100 subscribers.

• 0 leaks 19-test suite under -race, incl. revocation races
• 13,028 events in the race hammer, all authorized
• 1.61 ms p50 write-to-client, 100 subscribers
• 75k/s peak deliveries at 2,500 subscribers

Go Postgres 16 Logical replication pgoutput SSE

PGShift

Schema migration

Zero-downtime PostgreSQL schema migration in zero-dependency Go: expand, backfill, verify, cutover, contract, with cross-node verification fenced on xmin. It widens a write-hot 2M-row integer column to bigint holding its lock 3-4 ms for a 2.6-8.5 ms write stall, where the naive ALTER stalls every writer for 1,145 ms — and the fenced verifier avoided 20,000 lag-induced false alarms while catching all three injected faults.

• 3-4 ms lock held at cutover, vs 1,145 ms naive stall
• 20,000 lag false alarms fenced on a correct migration
• 3/3 injected dual-write faults caught
• 4/4 pre-contract phases roll back, catalog-verified

Go PostgreSQL 16 Streaming replication SCRAM-SHA-256 PL/pgSQL

Multi-tenant data systems 2

TenantVec

Vector search

Multi-tenant vector search over object storage in the style of turbopuffer: object storage is the source of truth, disk and RAM are caches, and namespaces move between tiers by use. 240 tenants and 840k vectors served ~4,750 queries/s with zero errors and zero cross-tenant leaks in a 12,000-result scan, at 0.08 ms hot / 17.4 ms warm / 30.9 ms cold server p50.

• 0.08/17/31 ms hot/warm/cold tier server p50
• 240 / 840k tenants / vectors at ~4.7k qps, 0 errors
• 0 leaks 12,000-result cross-tenant scan
• 128 MiB hot-tier cap, ~a third of fleet index bytes

Go hnswdb MinIO/S3 SigV4 Object storage WAL

Tailpipe

Telemetry pipeline

An OTLP trace collector for GenAI workloads in zero-dependency Go. Tail sampling kept 100.0000% of every must-keep class — error, high-cost, slow, flagged — verified per-trace over 800k+ traces while cutting stored wire bytes 89.7%, and per-tenant AI spend reconciles exactly against the generator's ledger, to the picodollar, while absorbing 35,043 injected duplicate deliveries.

• 100.0000% must-keep retention, verified over 800k traces
• 89.7% wire-byte reduction (98.4% on disk)
• exact per-tenant spend, to the picodollar
• 45,644/s unthrottled trace capacity (~167k spans/s)

Go OTLP/HTTP OpenTelemetry GenAI semconv JSONL gzip Prometheus

Platform studies 2

EdgeBench

Platform study

Edge vs serverless vs container, measured honestly: the same API (byte-identical responses, parity-tested) under workerd isolates, a Lambda-semantics host, and a long-lived container. Isolate cold starts land at p50 17.7 ms versus 78.5-99.1 ms of process boot, warm latency ties within ~1 ms, and a cost model verified against the vendors' own pricing examples shows a 6.5x spread on an LLM proxy at 100M requests/month.

• 17.7 ms isolate cold start p50, vs 79/99 ms process boot
• ~1 ms warm-latency spread across all three platforms
• 207 rps serverless LLM-proxy ceiling, a real platform limit
• $37 vs $242 Workers vs Lambda, LLM proxy at 100M req/mo

Node.js workerd Go Postgres 16 k6-style harness Python cost model

BotShop

API design study

Agent-facing API design, measured instead of asserted: one commerce store, two API surfaces of identical functionality, and a 20-task agent evaluation judged only by final store state. The same 8B model nearly doubles task success on the agent-optimized surface (63% vs 35%), recovers 9/9 fault-injected tasks, and produces zero duplicate purchases thanks to required idempotency keys.

• 63% vs 35% task success, agent surface vs baseline
• 9/9 fault-injected tasks recovered, vs 4/9 baseline
• 0 duplicate purchases under response loss
• 0 cap violations in all 9 capped runs

Go Ollama qwen3:8b MCP JSON Schema REST

Realtime collaboration 1

RTRoom

Multiplayer backend

A multiplayer backend in zero-dependency Go: room-sharded WebSocket servers with a Redis backplane, per-connection coalescing that cuts wire bytes and backplane publishes 3.0x at a measured +30 ms p50, and presence that converges 1.83 s after a node is killed with no cleanup. Every durable update dropped under back-pressure — 759 of them — was accounted for by a resync; none was lost silently.

• 3.0× fewer wire bytes and publishes, at +30 ms p50
• 1.83 s presence convergence after node kill, 2.2 s bound
• 759/759 durable drops accounted for by resync
• 27/27 backplane conformance tests, both implementations

Go RFC 6455 WebSocket Redis RESP2 Lua Prometheus