- All
- Riteangle 17
- Decision systems 8
- Guardrails 6
- Operations 6
- Agent architecture 4
- Agent evals 4
- Advertising 2
- Agentic architecture 2
- Context engineering 2
- Marketing 2
- Data platform 1
Agent evals — 4 posts
A replay harness gating an agent's edits at zero divergence, and the assumption ontology that explains each one
Letting an agent edit a production configuration from a plain-English request is reckless unless something can answer what the change would actually do — offline, thousands of times, without touching live traffic. So a shadow reimplementation exists, and it earns the right to speak by reproducing every routing decision production has already made: 5,501 of 5,501, zero divergences, against a threshold set in advance. The same harness then makes a 7,900-line refactor reviewable, because behaviour preservation is demonstrated rather than argued.
7 min read
Agent replay with writes suppressed at the boundary, and the side-effect ledger that proves nothing escaped
To debug an agent you need the prompt it actually saw, and the only way to get that is to run it — which sends messages to real people. This harness is the production code path with the writes cut at the boundary, and every suppressed effect recorded in plain language rather than silently skipped. It is very good at showing you one turn and completely unable to tell you whether that turn was any good, which is the state most eval work is actually in.
8 min read
NULLS FIRST and a client insert default corrupted an agent's context window for nine turns running
An agent's context window is a query result, and it is the one artifact nobody audits — assembled at runtime, used once, discarded. Two separately correct, separately documented library defaults met in the middle and pinned a single old message to the top of every window, so the agent read it as the newest thing said and escalated accordingly. This is what that failure looked like, why every natural diagnosis pointed at the prompt instead, and how you test for a bug that lives in the gap between two libraries.
8 min read
LLM-as-judge in the send path, and the human-label calibration that gives its score an error bar
Once a system generates text faster than anyone can read it, the choice is between shipping unreviewed output and building something that reviews it — and human review does not scale to every message while sampling misses exactly the rare failures worth catching. A small model grading a large one costs a fraction of the thing it guards, which is what makes total coverage affordable rather than aspirational. The catch is that a judge nobody has scored against human labels produces a number with no error bar, and a confidently wrong judge looks exactly like a working one.
21 min read