ResearchHow we test Otto, and what we find
Research
How Otto handles evidence, and how it compares with general-purpose AI.
One scale for Otto, Claude and open models
Every benchmark run on one scale, by Bayesian item response theory: Otto on Claude Haiku 5.5 scores 87.6 out of 100, against 78.2 for the same model given the full paper, with open models and the ChatGPT app placed alongside.
Read the postAll research
2 posts
TitleDescriptionTopicDate
One scale for Otto, Claude and open modelsEvery benchmark run on one scale, by Bayesian item response theory: Otto on Claude Haiku 5.5 scores 87.6 out of 100, against 78.2 for the same model given the full paper, with open models and the ChatGPT app placed alongside.Evaluation, item response theoryGrounded, not guessed: how Otto handles research evidenceAgainst the same AI model working from search results, Otto made 7.1× fewer unsupported claims and found 1.8× more of the evidence experts marked across a 25-paper library.Evaluation