KeatingBench is a new benchmark surface for a narrower question than most model leaderboards ask: which model actually teaches better in learner interactions? It sits at /bench, separate from chat and usage, and ranks models using the session data learners choose to share.
The page is deliberately closer to an evaluation dashboard than a release note. It shows the model leaderboard, replay evidence, PROSPER scoring dimensions, readiness bands, and an explainer for how each piece works. The goal is to make the benchmark inspectable before it is trusted.
Why KeatingBench Exists
A generic answer-quality benchmark is not enough for a tutoring system. A teaching model should diagnose where the learner is, recover from confusion, correct mistakes, invite practice, and help the learner transfer the idea to a new case. KeatingBench treats those learner outcomes as first-class evidence.
That also changes how data should be used. Synthetic learners can be useful for stress tests, but the primary benchmark should be grounded in real learner interaction. KeatingBench starts from shared sessions and local private sessions, then turns those conversations into replayable learner states.
What Gets Tested
KeatingBench looks for teaching moments inside learner turns. A turn may signal understanding, confusion, correction, transfer, retention need, or dissatisfaction. Those are different from simple thumbs up or thumbs down ratings because they reveal where the learner is in the teaching loop.
The benchmark currently classifies replay stages as diagnosis, confusion-recovery, correction, transfer, and retention. The leaderboard shows the replay mix so a model with easy wins is not confused with a model that handled hard recovery cases.
From Session to Signal
The data path starts with a shared or local session. KeatingBench reads the learner messages, extracts feedback-like signals from the learner's own words, and records the nearby assistant context. Each extracted state becomes a replay case with a normalized outcome score.
thumbs-up -> high outcome signal confused -> mid-low outcome signal thumbs-down -> low outcome signal
Explicit feedback remains valuable, but the benchmark also uses learner-turn analysis because a confused follow-up or a correction is meaningful evidence even when the learner never clicks a feedback button.
PROSPER Judgement
The headline rank is not the raw outcome score. It is a PROSPER score: a weighted multi-objective judgement designed to avoid rewarding narrow or brittle behavior. Raw outcome is still shown in the table, but it is only one component.
PROSPER = performance robustness outcome lift sparse-data caution personalization evidence quality retention / transfer
This means a model can rank well only when it balances learner outcome with evidence quality and teaching behavior. A model that gets a few positive signals but has little evidence remains marked as sparse.
Replay Case Bank
The replay case bank is the most important part of the page. It shows the learner states KeatingBench is actually using: stage, inferred feedback signal, learner text, outcome score, and PROSPER score. This makes the benchmark auditable instead of just producing a mysterious rank.
The same case bank is also the foundation for cross-model replay. Once provider replay is wired in, the harness can send the same extracted learner state to multiple models and compare their responses under the same judgement criteria.
Readiness and Sparsity
KeatingBench does not treat all data volumes equally. Sparse data is visible but should not drive strong conclusions or policy evolution. The page currently uses explicit readiness bands:
waiting < 5 signals sparse >= 5 signals provisional >= 20 signals rankable >= 50 signals stable >= 100 signals
The policy-evolution gate is still stricter about caution: if the system does not have enough real learner signal, it should say it is not ready to evolve rather than pretending a thin sample is a reliable improvement target.
Privacy and Analytics
KeatingBench uses shared sessions as benchmark material only when those sessions are available to the app. Local private sessions stay in the local view. The new PostHog integration is opt-in through Vite environment variables, with autocapture and session recording disabled so learner text is not automatically sent as analytics data.
The analytics events are aggregate events such as opening or exporting the benchmark. The benchmark data itself remains the session corpus, not a hidden analytics stream.
Next Step
The current implementation scores observed sessions deterministically. The next real step is provider replay: take the same replay case, send it to multiple models, and judge the resulting teaching move with the same PROSPER vector.
That is the path from a useful local leaderboard to a proper KeatingBench: human-grounded cases, comparable model responses, transparent scoring, and enough evidence to know when the system is ready to evolve.