Every benchmark run so far, on every model and every version of Otto, answered a different slice of the same
question bank. Item response theory puts them on one scale: it learns how hard each question is and how sharply it separates
strong from weak, then reads each configuration's ability from the questions it actually faced.
87.6General Capability Score of Otto · High (I, latest) on Claude Haiku 5.5, the top of the field
+9.5points Otto adds to Claude Haiku 5.5 over the same model given the full paper (P > 0 = 1.00)
74.8best open model as a chatbot with the paper: Qwen 3.6 27B
The General Capability Score is the share of a balanced battery a configuration is expected to get right: half questions about
one paper (QASPER), half claims whose evidence must be found across a library (SciFact). It is on a 0 to 100 scale, comes with a
90% credible interval, and does not depend on which questions a run happened to face.
Figure 1 · General Capability Score
OttoChatbot given the paperSearch passages / closed book
Expected share of a balanced battery answered correctly (half paper questions, half library evidence claims), with 90% credible intervals. Filled dots were measured on both halves; open rings on paper questions only, their library half inferred from the model, so their intervals are wider. Otto · High (I, latest) on Claude Haiku 5.5 leads at 87.6; the same model as a chatbot with the paper scores 78.2.
Model or harness
The model separates what a model brings from what the harness around it brings. Moving from Claude Haiku 5.5 to Sonnet 5.5 is
worth -0.04 logits (90% -1.80 to +1.62); the newest Otto harness is worth +1.75
logits (90% -0.02 to +3.56) over a chatbot given the paper, on whatever model it runs. On the score itself, Otto comes out ahead on 5 of the 5 models it was run on; the data are clear (P > 0.95) only for Claude Haiku 5.5, as the rest were run on fewer questions.
Figure 2 · What Otto adds to each model
Chatbot + paperInside Otto
Each row is one model, first as a chatbot given the full paper, then inside Otto (the newest Otto harness run on that model). Points are General Capability Scores; the label on the right is the lift with the posterior probability that it is above zero.
Figure 3 · Model and harness, separated
The model splits each configuration's ability into what the model brings and what the harness brings (plus a small interaction). One logit is about 20 points of accuracy near the middle of the scale. Harness effects for Otto are estimated across every model Otto was run on.
Where the benchmark measures
A test measures well only near the difficulty of its items. A person-item map puts runs and questions on the same line.
Figure 4 · Where the benchmark can tell configurations apart
OttoChatbot + paperOtherItems by difficultyTest information
Above the line, each configuration at its ability; below it, how many items sit at each difficulty, on the same logit scale. The curve is the test information: where the benchmark measures most sharply. On paper questions it peaks near θ = +1.5, while Otto configurations sit around θ = +5.2: the top of the field is measured with fewer items, so harder questions would separate it better.
The items
Not all questions are equally useful. Some everyone answers, some nobody does, and a minority carries most of the ranking.
Clustering the item bank by difficulty, discrimination and harness effect gives 4 groups.
Every paper question and library claim at its estimated difficulty and discrimination, coloured by cluster (k-means on difficulty, discrimination and the item's own harness effect; k = 4 chosen by silhouette). Library claims are drawn slightly larger.
Figure 6 · Otto against the same model, by type of question
OttoClaude Haiku 5.5 + paper
Question types as the datasets define them. Otto at High effort on Claude Haiku 5.5 against the same model given the paper, on the same questions, each question counted once. The number beside each type is how many questions both answered.
Figure 7 · Questions where the harness changes the answer
Otto does better than its ability predictsOtto does worse
An item-level term in the model asks whether Otto does better or worse on this exact item than its overall ability predicts. 0 items lean clearly towards Otto (blue) and 8 clearly away (sand), with 90% intervals as dark lines. Every other item behaves as ability alone predicts.
How certain the ranking is
Figure 8 · Probability that the row scores higher than the column
Read across a row: the posterior probability that the row's configuration has the higher General Capability Score than the column's. Darker is more certain. Values near 0.5 mean the data cannot yet order the pair.
Otto on the workspace (operating the app: commands, planning, retrieval across projects) is measured only for Otto, so it is
kept out of the general score. Expected workspace success: Otto · High (I, latest) 97.0%, Otto · High (H, ships) 96.9%, Otto · High (F) 96.8%.
Checks
The model was fitted with Hamiltonian Monte Carlo, 4 chains of 1,500 draws. The largest split
R̂ over the abilities reported is 1.022 and their smallest effective sample size 271
(across a sweep of all raw parameters, R̂ ≤ 1.028 and effective sample size ≥ 155). Discrimination matters: the two-parameter model beats the
Rasch model by 688 WAIC points, and on responses it never saw it
predicts better than simple accuracy baselines.
Predicting 10% of responses held out
AUC ↑
Log loss ↓
Brier ↓
Run accuracy only
0.791
0.440
0.140
Item accuracy only
0.730
0.504
0.163
Run + item (additive logit)
0.891
0.373
0.115
Rasch, hierarchical
0.902
0.327
0.098
2PL + DIF, hierarchical
0.905
0.317
0.094
Figure 9 · Does the model fit?
Otto runsOther runs
Left: predicted against observed, in ten bins. The curve runs a little steeper than the diagonal: averaged over its own uncertainty the model hedges, so outcomes are slightly more extreme than predicted (largest gap 18 points). Right: each of the 60 runs' observed accuracy against what the model replays for it; 100% fall inside their 90% predictive interval (open rings fall outside).
Method
Each response is correct or not. The chance of a correct answer is a logistic function of the respondent's ability minus the
item's difficulty, scaled by the item's discrimination, plus an item-level term for Otto (differential item functioning).
Ability is built from parts: the model, the harness, their interaction, a domain offset, the configuration's own profile across
domains, and a run-to-run term that absorbs differences between repeated runs. QASPER answer types (extractive, abstractive,
yes/no, unanswerable) shift item difficulty. Priors are weakly informative; everything is sampled jointly.
The ChatGPT app was given each paper's PDF, so it is counted as a chatbot with the paper; it answered 11 questions, so its interval is wide.
Runs re-scored under another label were removed as copies (4). Workspace runs carry their harness in their label (ops-h → H, ops-i → I).
Exploratory: judgements are by a model judge without human validation, and several configurations faced only the development split.
Otto is not another frontier model. It is an agent that makes whichever model it runs on more capable, and the measure here is built to say by how much, and with how much certainty.