BlackArc6 papers
Evaluation & Benchmarks
Judging agents honestly: benchmarks, LLM-as-judge, and outcome measurement.
- Papers
- 6
- Read time
- 1.7h
- Avg length
- 17m
Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
Instead of asking an LLM to rate an output 1–5, ask it a checklist of small yes/no questions and count the yeses — you get a score that matches humans better *and* tells you exactly what was wrong, which you can then feed back to fix either the judge or the generator.
- Read
- 18 min
- Source
- arXiv:2606.27226
InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback
InterCode turns "write code from instructions" into a proper reinforcement-learning environment — code is the action, a Docker container's output is the observation, and a coding agent gets to try, fail, and retry instead of generating one shot and stopping.
- Read
- 14 min
- Source
- arXiv:2306.14898
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Using a strong LLM (GPT-4) as an automated judge of chatbot answers agrees with human preferences over 80% of the time — the same rate humans agree with each other — so you can replace slow, expensive human eval with a prompt, if you account for a handful of predictable biases.
- Read
- 16 min
- Source
- arXiv:2306.05685
Measuring AI Ability to Complete Long Software Tasks
METR measured, in one interpretable unit, how long a task a human would need to do it in — instead of raw benchmark percentages — and found that the length of task frontier AI can reliably finish has been doubling roughly every seven months since 2019, on pace to hit month-long autonomous work around 2028–2030.
- Read
- 20 min
- Source
- arXiv:2503.14499
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
OSWorld is a real, scriptable virtual-machine playground plus 369-task benchmark that lets you turn any LLM/VLM into a computer-using agent and grade it on actual desktop work — and it exposes that the best 2024 models finish only ~12% of tasks while humans hit ~72%.
- Read
- 22 min
- Source
- arXiv:2404.07972
Outcome Primitives: A Framework for Measuring AGI Value in the World
Stop asking "can the model pass the test?" and start asking "did a real person get real work done?" — VCF gives that question a coordinate system: what kind of outcome, how big, and how sure we are it actually happened.
- Read
- 14 min