Methodology

The standards every benchmark and comparison on this site follows.

How we test

• Exact versions and dates. Every result names the model or product version and the date it was run.
• Repeated runs. We report the number of runs and the spread, not a single lucky sample.
• Same conditions. Same prompts, tasks, hardware and settings for every contender, unless stated.
• Cost and time measured. We record what a run actually cost and how long it took.
• Limitations stated. Each page lists what the test does not show.

What we publish

• Raw results as CSV or JSON, licensed CC BY 4.0.
• Prompts, task sets and scripts needed to reproduce, where licensing allows.
• A dated changelog when a benchmark is re-run or corrected.

Independence

We do not accept payment from companies we evaluate, and we do not let vendors review results before publication. Conflicts of interest are disclosed on the page they affect.

Corrections

If a result is wrong, we fix it, mark the page as updated and log what changed. Send corrections to the address on theabout page.