Some work can only be judged on live outcomes: a plan has to run in sim, a label has to match a human. Those get a traffic split. The post-review flow grades every arm, and the winner is promoted to the route when the result clears 95% confidence.
| Arm | Split | Score | vs. control | p50 | $/1K calls | Daily score |
|---|---|---|---|---|---|---|
Claude Sonnet 5 Control | 50% | 94.2% | — | 1,180ms | $8.80 | |
DeepSeek V4 Pro (Vast) Challenger | 50% | 92.4% | 98.1% | 920ms | $1.21−86% |
Read so far: Challenger is 98.1% of control at a fraction of the cost. Needs 4 more days to clear 95% confidence.
| Arm | Split | Score | vs. control | p50 | $/1K calls | Daily score |
|---|---|---|---|---|---|---|
Gemini 3.1 Pro Control | 80% | 87.4% | — | 3,900ms | $44.20 | |
Claude Sonnet 5 Arm B | 10% | 88.9% | 101.7% | 3,400ms | $40.00−10% | |
Kimi K3 Arm C | 10% | 84.1% | 96.2% | 4,300ms | $53.06+20% |
Read so far: Sonnet 5 is ahead on success rate and latency at similar cost. Kimi K3 is cheaper but trails by 3 points.
| Arm | Split | Score | vs. control | p50 | $/1K calls | Daily score |
|---|---|---|---|---|---|---|
Jev v1 thresholds Control · 0.38 / 0.74 | 50% | 61.1% | — | 2,600ms | $42.00 | |
Jev v2 thresholds Challenger · 0.45 / 0.74 | 50% | 60.4% | 98.9% | 2,200ms | $36.12−14% |
Read so far: Too early. v2 sends 9% more traffic to the low-cost tier; acceptance is flat so far.
| Arm | Split | Score | vs. control | p50 | $/1K calls | Daily score |
|---|---|---|---|---|---|---|
Qwen3.8 27B Incumbent route | 50% | 98.7% | — | 610ms | $0.31 | |
Gemma 3 12B Challenger | 50% | 95.8% | 97.1% | 380ms | $0.10−67% |
Outcome: Kept Qwen3.8 27B. Gemma is 68% cheaper but missed the 97% parity bar on urgency labels. It gets retested when the next Gemma ships.
| Arm | Split | Score | vs. control | p50 | $/1K calls | Daily score |
|---|---|---|---|---|---|---|
Jev v1.2 Control | 50% | 81.2% | — | 2,900ms | $29.16 | |
Jev v1.3winner Challenger | 50% | 81.6% | 100.5% | 2,600ms | $27.00−7% |
Outcome: Promoted 12 days ago. The retrained scorer sends 7% fewer calls to Opus with no drop in resolution rate.
| Ended | Workload | Test | Outcome | Impact | Note |
|---|---|---|---|---|---|
| Aug 30 | Fleet ops copilot | Jev retrain v1.3 vs v1.2 | promoted | −$1.90K/mo | 7% fewer calls to Opus |
| Aug 24 | Support ticket triage | Qwen3.8 27B vs Gemma 3 12B | kept incumbent | — | Gemma 95.8% parity, below bar |