4 papers
Evaluation & Benchmarks
Judging agents honestly: benchmarks, LLM-as-judge, and outcome measurement.
- Papers
- 4
- Read time
- 1.2h
- Avg length
- 18m
Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
Instead of asking an LLM to rate an output 1–5, ask it a checklist of small yes/no questions and count the yeses — you get a score that matches humans better *and* tells you exactly what was wrong, which you can then feed back to fix either the judge or the generator.
- Read
- 18 min
- Source
- arXiv:2606.27226
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Using a strong LLM (GPT-4) as an automated judge of chatbot answers agrees with human preferences over 80% of the time — the same rate humans agree with each other — so you can replace slow, expensive human eval with a prompt, if you account for a handful of predictable biases.
- Read
- 16 min
- Source
- arXiv:2306.05685
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
OSWorld is a real, scriptable virtual-machine playground plus 369-task benchmark that lets you turn any LLM/VLM into a computer-using agent and grade it on actual desktop work — and it exposes that the best 2024 models finish only ~12% of tasks while humans hit ~72%.
- Read
- 22 min
- Source
- arXiv:2404.07972
Outcome Primitives: A Framework for Measuring AGI Value in the World
Stop asking "can the model pass the test?" and start asking "did a real person get real work done?" — VCF gives that question a coordinate system: what kind of outcome, how big, and how sure we are it actually happened.
- Read
- 14 min