A research assistant is only as useful as the sentences you can trust. We put Otto against the same AI model used the way people use a chatbot, on research questions it had never seen. Working from search results, the model made 7.1× as many unsupported claims as Otto. Given the whole paper it was about as careful, though Otto’s citations held up more often. Across a library, Otto found more of the evidence experts marked.
At a glance
Every comparison runs the same model inside Otto and outside it, so the difference is what Otto does around the model.1
Otto, in multiples
Fewer unsupported claims than Claude working from search resultsClaude Haiku 5.5 with the 5 passages a search finds, 180 held-out questions
Fewer unsupported claims than the ChatGPT appChatGPT app with the PDF uploaded vs Otto, on the same 12 held-out questions; a small sample2
Fewer misquotes than Claude given the whole paperquotes not word for word in the paper, 180 held-out questions; 1.7% against 5.1% of quotes
More of the evidence experts marked, found across 25 papersagainst Claude working from search, 60 held-out claims
Fewer citations that fail to back their sentenceagainst Claude Haiku 5.5 given the whole paper, 180 held-out questions
More expert-marked evidence than Claude given every abstract60 held-out claims, 25 papers each
Otto against Claude on research tasks
| Otto | Claude + paper | Claude + search | Claude alone | |
|---|---|---|---|---|
| Answering questions about a paper | 84.2%79–89 | 81.8%76–87 | 43.7%36–51 | 19.8%14–26 |
| Claims the paper supports | 95.5%94–97 | 95.7%94–97 | 68.0%63–73 | 29.9%19–41 |
| Citations that check out | 94.3%93–96 | 89.9%87–93 | 94.0%91–96 | 2.0%0–7 |
| Finding evidence across a library | 89.5%82–96 | 65.7%53–79 | 49.2%37–62 | — |
| Highlighting the sentence that shows it | 92.5%85–98 | 86.4%76–95 | 84.8%72–96 | — |
| Doing what was asked in the app | 100.0%88–100 | — | — | — |
Answering from a paper
Most chatbots answer from fragments: the passages a search returns, or what fits in the window. Working that way, 32% of Claude’s claims were not supported by the paper. Otto’s were 4.5%, and it answered 84% of questions correctly against 44%.
Even against the model given the entire paper, Otto’s citations held up more often (94% against 90%) and fewer of its quotes strayed from the paper’s wording (1.7% against 5.1%). On getting the answer right the two were level within the margin of error (84% against 82%).
Claims the paper does not support
Evidence across a library
Real questions span many papers. Given a claim and a project of 25 papers, Otto found the papers experts marked, with the right stance, 90% of the time, against 66% for Claude given every abstract and 49% working from search. It quoted the very sentence the experts marked 92% of the time, and 99% of its quotes were word for word in the paper it named, so every finding can be checked in a click.
Finding evidence across a library
OttoClaude + paperClaude + search
How Otto does it
- Reads the whole source. Otto reads the full paper, or every paper in the project, before answering, instead of the few passages a search happens to find.
- Cites the exact words. Every quote is checked in code against what Otto read. A quote that is not word for word is replaced by the sentence it paraphrases, or marked unverified.
- Names a paper only on its own words. Across a library, a paper is listed only when its text states the claim or its opposite.
- Acts, and shows what it did. In the app it did every one of 30 workspace tasks as asked, and never reported a task done that was not.
In the app, this is Settings → Otto → Effort: High.
Researchers stay at the centre of the work. Otto reads everything, finds what bears on the question and shows exactly where it came from, so the time goes on thinking about the evidence rather than hunting for it or checking it twice.