Most AI benchmark numbers you see are published by the company that built the model, on the day it launches, on tasks it chose. That is not dishonest, but it is not independent either.
Heldout Labs exists to answer a narrower question: when you actually run these tools on realistic work, what happens?
What we do differently
- We run the tests ourselves, with the same tasks and settings for every contender.
- We publish everything: methodology, dates, exact versions and raw data.
- We re-run. Models change every month. Our results pages are living documents with a changelog, not one-off posts.
What we will not do
We will not accept payment from companies we evaluate, and we will not hide the limitations of a test. See the methodology for the full standards.