Evals Platform
Nobody can read every conversation an AI assistant has. LLM judges can grade replies at scale, but a judge only counts if it agrees with careful humans. So the real work is grading the graders, and that starts with how well people can judge.
PD
AI
EVALS
1 · LABEL INDEPENDENTLY
Expert judge A
Expert judge B
Expert judge C
Blind to each other and to the model’s verdict.
2 · COMPARE AGREEMENT
Where do experts diverge?
Disagreement is a signal about what “good” means, not noise to average away.
3 · TRUSTED GROUND TRUTH
Judges graded against a measured standard
Variance is measured instead of assumed, so teams can own their own evals.
WHAT I BUILT
Working with an ML scientist, I designed the interface for human judges: how expert evaluators review real conversations, label them blind to the judge’s verdict, and feed those labels back into a held-out test.
THE BAR
A judge is trusted only when it clears a high bar on both pass and fail precision and recall, measured on labels it was never tuned on.
WHAT’S NEXT
Experts disagree about what good looks like, and most eval tooling assumes one expert’s labels are the golden set. I’m designing a platform that compares multiple expert judges against each other, so variance becomes something you can measure, and domain teams can manage their own evals.