Adversarial LLM-judge framework
A single model asked "which answer is better?" is a weak instrument. It drifts, it favours whichever answer it read first, and it gives you a number with no reasoning you can audit. This framework replaces that with a structured argument: advocates make the case for each answer, a judge presses on the weak points, and a jury votes.
- Type
- Personal project, open source
- Stack
- Python · OpenAI API · multi-agent orchestration
- Benchmarks
- MT-Bench · AUTO-J · AlignBench
- Basis
- Implements arXiv:2410.04663
The problem
LLM-as-judge is now the default way to evaluate model outputs at scale, and it has three failure modes that matter. It is position-biased: the same pair of answers scored in reverse order often produces the opposite verdict. It is opaque: a 7/10 tells you nothing you can act on. And it is unbounded: a naive evaluation harness will happily spend real money re-reading the same context hundreds of times.
I wanted an evaluator where disagreement is visible rather than averaged away, and where the cost of a verdict is a number I choose in advance.
How it works
Two candidate answers enter. Advocates argue for their assigned answer, a judge probes the arguments, and five jurors with distinct personas vote independently. A strict majority decides; there is no tiebreaking by the judge, so a 3–2 verdict is visibly a close call rather than a confident one.
Two protocols share this shape: a single-round mode with three parallel advocates, and an iterative mode that runs additional rounds until the jury converges.
Decisions that mattered
Jurors vote independently, and a strict majority decides.
Letting jurors see each other's votes collapses them onto the first opinion expressed. Independence is what makes a 3–2 split meaningful information rather than noise to be smoothed over.
Jurors are given distinct personas rather than five copies of one rubric.
Five identical evaluators are one evaluator with extra latency. Diversity is what lets the panel catch a failure mode any single lens would miss.
Every evaluation carries a token budget and cost tracker.
Debate protocols multiply calls quickly: advocates, rounds, and five jurors compound. Without a hard ceiling the harness is not safe to point at a full benchmark.
Position-swap testing is built in, not bolted on.
Running the same pair with the answers reversed measures the ordering bias directly, so the framework reports how much of its own verdict came from presentation order.
What I would do differently
The framework currently treats the jury size as fixed. Panel size should scale with disagreement: a unanimous first three votes rarely needs two more, while a genuine split deserves more evidence. That would cut cost on easy cases and spend it where the decision is actually close.
It also has no persistent store: results are written per run rather than accumulated into a dataset you can query across model versions. Wiring the verdicts into a golden dataset would turn it from an evaluation script into a regression suite.