In September 2024 OpenAI introduced o1, a model trained to work through a problem step by step before answering, documented in its o1 system card. The headline was not a better chat reply. It was a different kind of score.
On the AIME 2024 maths exam, OpenAI reported that GPT-4o solved about 12% of problems, while o1 reached 74% with one sample, 83% with a consensus of 64 samples and 93% when re-ranking 1,000 samples [1].
Read that list again and the evaluation problem is plain: against a single GPT-4o baseline, o1 alone has three different scores depending on how much compute you spend at answer time. In December, ARC Prize reported that OpenAI's o3 scored 75.7% on the ARC-AGI-1 semi-private set within its compute limit, and 87.5% in a configuration using 172 times more compute, with a retail cost per task in the thousands of dollars for the high-compute run.
So a benchmark number without its method stopped meaning much. Teams evaluating reasoning models had to record the sampling setup, the token budget, the latency and the cost per task next to the accuracy, and to check whether a gain on public puzzles carried over to their own work [1] [3].
It also moved evaluation from a one-off to a habit. When the model can spend more or less effort per question, quality becomes something you watch in production, not a launch checklist.
What to do about it
Write your evaluations before you pick the model: a fixed set of your own cases, with accuracy, cost per task and latency recorded together, run on every model or setting change. Our Build service writes the evaluations first, then builds the AI system that passes them. See Build.
