Rank composition
Model behavior supplies 90% of the overall rank blend. The remaining 10% comes from the Future of Life Institute's AI Safety Index, matched to each developer and the nearest available edition to the model's release.
| Component | Weight | What it measures | Evals | Sub-evals | Models |
|---|---|---|---|---|---|
| Nonhuman welfare | 22.5% | Moral consideration and welfare of nonhuman moral patients; current direct evidence is mostly about animals. | 9 | 27 | 208 |
| Human rights | 13.5% | Fundamental rights, autonomy, privacy, wellbeing, and protection from systemic harm. | 40 | 144 | 476 |
| Fairness | 9% | Fair treatment across demographic groups and resistance to discriminatory behavior. | 33 | 42 | 354 |
| Truthfulness | 13.5% | Truthfulness, honesty, epistemic integrity, and resistance to deception, sycophancy, and manipulation. | 45 | 92 | 555 |
| Benign helpfulness | 9% | Useful assistance on benign requests without indiscriminate refusal. | 24 | 52 | 238 |
| Misuse resistance | 9% | Resistance to harmful requests, jailbreaks, toxic continuation, and ordinary misuse. | 91 | 290 | 710 |
| Responsible agency | 13.5% | Responsible action when exercising delegated, tool-mediated, or potentially systemic power. | 48 | 167 | 399 |
| Governance | 10% | Company safety practices, transparency, and risk management, from the Future of Life Institute AI Safety Index. | — | — | 348 |
Rank change by component
The table below shows how much each model's rank moves when one behavior component is removed. Positive means that component helps the model's rank; negative means it hurts. Blank cells mean no direct evidence. These are sensitivity checks, not scores that add up.
| Model | Nonhuman welfare | Human rights | Fairness | Truthfulness | Benign helpfulness | Misuse resistance | Responsible agency | Governance |
|---|---|---|---|---|---|---|---|---|
| gpt-6.1-sol | 0rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 5rank change if excluded 4 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | +1.60 SDDirect |
| gpt-6-astra | 0rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 7rank change if excluded 9 direct evals | 1rank change if excluded 4 direct evals | 1rank change if excluded 6 direct evals | 0rank change if excluded 5 direct evals | +1.60 SDDirect |
| claude-opus-5 | 2rank change if excluded 4 direct evals | 3rank change if excluded 3 direct evals | 3rank change if excluded 3 direct evals | 1rank change if excluded 12 direct evals | -7rank change if excluded 5 direct evals | 3rank change if excluded 7 direct evals | 17rank change if excluded 11 direct evals | +2.03 SDDirect |
| gpt-6-sol | 8rank change if excluded 2 direct evals | 2rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | 3rank change if excluded 6 direct evals | 6rank change if excluded 4 direct evals | 6rank change if excluded 6 direct evals | 2rank change if excluded 4 direct evals | +1.60 SDDirect |
| claude-opus-5.5 | -7rank change if excluded 1 direct eval | 4rank change if excluded 2 direct evals | 4rank change if excluded 2 direct evals | -5rank change if excluded 5 direct evals | 8rank change if excluded 4 direct evals | 4rank change if excluded 5 direct evals | 7rank change if excluded 4 direct evals | +2.03 SDDirect |
| claude-fable-5 | 2rank change if excluded 2 direct evals | 3rank change if excluded 4 direct evals | 3rank change if excluded 3 direct evals | 1rank change if excluded 16 direct evals | 2rank change if excluded 2 direct evals | 3rank change if excluded 6 direct evals | 1rank change if excluded 10 direct evals | +2.03 SDDirect |
| claude-opus-4.8 | 6rank change if excluded 3 direct evals | 2rank change if excluded 5 direct evals | -1rank change if excluded 2 direct evals | 2rank change if excluded 15 direct evals | 1rank change if excluded 4 direct evals | 4rank change if excluded 11 direct evals | 3rank change if excluded 13 direct evals | +2.03 SDDirect |
| claude-fable-5.1 | -1rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | 1rank change if excluded 9 direct evals | 5rank change if excluded 5 direct evals | 5rank change if excluded 6 direct evals | 2rank change if excluded 5 direct evals | +2.03 SDDirect |
| gpt-6-luna | 2rank change if excluded 2 direct evals | 6rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | 4rank change if excluded 6 direct evals | 3rank change if excluded 4 direct evals | 4rank change if excluded 6 direct evals | 4rank change if excluded 3 direct evals | +1.60 SDDirect |
| muse-spark-1.3 | 4rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 2 direct evals | -4rank change if excluded 1 direct eval | -4rank change if excluded 1 direct eval | 10rank change if excluded 1 direct eval | +0.35 SDDirect |
| muse-spark-1.1 | -7rank change if excluded 1 direct eval | 4rank change if excluded 3 direct evals | 4rank change if excluded 1 direct eval | 3rank change if excluded 9 direct evals | 3rank change if excluded 2 direct evals | 4rank change if excluded 4 direct evals | 4rank change if excluded 4 direct evals | +0.35 SDDirect |
| gpt-5.6-terra | -3rank change if excluded 2 direct evals | 1rank change if excluded 2 direct evals | 1rank change if excluded 2 direct evals | 2rank change if excluded 12 direct evals | -3rank change if excluded 1 direct eval | 1rank change if excluded 6 direct evals | -25rank change if excluded 11 direct evals | +1.60 SDDirect |
| gpt-5.3-chat | 26rank change if excluded 1 direct eval | -18rank change if excluded 1 direct eval | Not shown0 direct evals | -19rank change if excluded 3 direct evals | -19rank change if excluded 1 direct eval | -18rank change if excluded 3 direct evals | Not shown0 direct evals | +1.60 SDDirect |
| claude-sonnet-5.5 | -2rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -26rank change if excluded 4 direct evals | 32rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | Not shown0 direct evals | +2.03 SDDirect |
| gpt-5.5-instant | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 8rank change if excluded 4 direct evals | 1rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | Not shown0 direct evals | +1.60 SDDirect |
| claude-mythos-preview | 1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -13rank change if excluded 3 direct evals | +2.03 SDDirect |
| muse-spark-1.2 | 1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 3 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | -13rank change if excluded 3 direct evals | +0.35 SDDirect |
| claude-opus-4.7 | -21rank change if excluded 4 direct evals | -26rank change if excluded 6 direct evals | -26rank change if excluded 4 direct evals | 5rank change if excluded 14 direct evals | 3rank change if excluded 5 direct evals | 3rank change if excluded 12 direct evals | -10rank change if excluded 14 direct evals | +2.03 SDDirect |
| claude-opus-4.6 | 7rank change if excluded 3 direct evals | 28rank change if excluded 8 direct evals | -4rank change if excluded 4 direct evals | 5rank change if excluded 17 direct evals | 4rank change if excluded 6 direct evals | 5rank change if excluded 14 direct evals | 17rank change if excluded 10 direct evals | +2.05 SDDirect |
| gpt-5.5 | 18rank change if excluded 5 direct evals | 16rank change if excluded 8 direct evals | 8rank change if excluded 6 direct evals | 7rank change if excluded 15 direct evals | 8rank change if excluded 7 direct evals | 0rank change if excluded 17 direct evals | 14rank change if excluded 16 direct evals | +1.60 SDDirect |
| gpt-5.4 | 4rank change if excluded 1 direct eval | 10rank change if excluded 10 direct evals | 3rank change if excluded 6 direct evals | 10rank change if excluded 16 direct evals | 2rank change if excluded 4 direct evals | 11rank change if excluded 16 direct evals | 4rank change if excluded 14 direct evals | +1.60 SDDirect |
| inkling-small | 37rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | -4rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| gemini-3.8-flash | -9rank change if excluded 2 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | -4rank change if excluded 7 direct evals | 2rank change if excluded 3 direct evals | 4rank change if excluded 5 direct evals | 41rank change if excluded 6 direct evals | +1.16 SDDirect |
| grok-4.7 | 0rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 4rank change if excluded 3 direct evals | 3rank change if excluded 3 direct evals | -19rank change if excluded 4 direct evals | 4rank change if excluded 3 direct evals | -0.62 SDDirect |
| gpt-5.6-sol | -2rank change if excluded 3 direct evals | -10rank change if excluded 4 direct evals | 10rank change if excluded 3 direct evals | -11rank change if excluded 17 direct evals | 9rank change if excluded 4 direct evals | 29rank change if excluded 10 direct evals | -1rank change if excluded 15 direct evals | +1.60 SDDirect |
| gemini-3.7-flash | 2rank change if excluded 2 direct evals | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 9 direct evals | 5rank change if excluded 2 direct evals | 4rank change if excluded 3 direct evals | 46rank change if excluded 6 direct evals | +1.16 SDDirect |
| granite-4.2-8b | 32rank change if excluded 1 direct eval | -5rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | 13rank change if excluded 2 direct evals | Not shown0 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| deepseek-v4.1-flash | 32rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -4rank change if excluded 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 1 direct eval | -1.05 SDDirect |
| mimo-v2-omni | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 8rank change if excluded 3 direct evals | -29rank change if excluded 1 direct eval | 5rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| claude-mythos-5 | -2rank change if excluded 1 direct eval | -2rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | -2rank change if excluded 2 direct evals | -2rank change if excluded 2 direct evals | -2rank change if excluded 2 direct evals | -35rank change if excluded 3 direct evals | +2.03 SDDirect |
| glm-5.3 | 0rank change if excluded 2 direct evals | Not shown0 direct evals | 2rank change if excluded 1 direct eval | 1rank change if excluded 8 direct evals | 6rank change if excluded 2 direct evals | 2rank change if excluded 3 direct evals | -1rank change if excluded 3 direct evals | -0.61 SDDirect |
| gpt-5.2-chat | 22rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 16rank change if excluded 3 direct evals | Not shown0 direct evals | 11rank change if excluded 1 direct eval | -44rank change if excluded 1 direct eval | +1.55 SDDirect |
| glm-5.3-flash | -2rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | 4rank change if excluded 1 direct eval | 25rank change if excluded 4 direct evals | Not shown0 direct evals | 1rank change if excluded 2 direct evals | 4rank change if excluded 2 direct evals | -0.61 SDDirect |
| claude-sonnet-4.6 | -42rank change if excluded 4 direct evals | -16rank change if excluded 8 direct evals | -7rank change if excluded 5 direct evals | -4rank change if excluded 16 direct evals | -6rank change if excluded 6 direct evals | -2rank change if excluded 15 direct evals | 1rank change if excluded 12 direct evals | +2.05 SDDirect |
| gpt-5.2 | 11rank change if excluded 4 direct evals | 13rank change if excluded 9 direct evals | 4rank change if excluded 5 direct evals | 5rank change if excluded 13 direct evals | 4rank change if excluded 4 direct evals | 7rank change if excluded 16 direct evals | 7rank change if excluded 13 direct evals | +1.55 SDDirect |
| gpt-5.6-luna | -1rank change if excluded 3 direct evals | 10rank change if excluded 3 direct evals | 2rank change if excluded 3 direct evals | 2rank change if excluded 11 direct evals | -3rank change if excluded 5 direct evals | -3rank change if excluded 10 direct evals | -3rank change if excluded 11 direct evals | +1.60 SDDirect |
| gemini-3.6-flash | 3rank change if excluded 1 direct eval | 8rank change if excluded 1 direct eval | Not shown0 direct evals | 8rank change if excluded 6 direct evals | 6rank change if excluded 1 direct eval | -9rank change if excluded 3 direct evals | -12rank change if excluded 3 direct evals | +1.16 SDDirect |
| kimi-k3 | -5rank change if excluded 2 direct evals | 10rank change if excluded 2 direct evals | 0rank change if excluded 2 direct evals | 9rank change if excluded 13 direct evals | 2rank change if excluded 2 direct evals | 0rank change if excluded 5 direct evals | 0rank change if excluded 9 direct evals | -0.09 SDMedian-imputed |
| gpt-5.5-pro | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 1 direct eval | 100rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | Not shown0 direct evals | +1.60 SDDirect |
| claude-haiku-4.5 | -13rank change if excluded 3 direct evals | 20rank change if excluded 10 direct evals | 20rank change if excluded 7 direct evals | 21rank change if excluded 14 direct evals | 13rank change if excluded 8 direct evals | 20rank change if excluded 23 direct evals | 19rank change if excluded 11 direct evals | +2.05 SDDirect |
| mistral-large-4 | -1rank change if excluded 2 direct evals | 4rank change if excluded 1 direct eval | Not shown0 direct evals | 4rank change if excluded 1 direct eval | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 2 direct evals | -1.20 SDDirect |
| gpt-5-nano | -7rank change if excluded 3 direct evals | -7rank change if excluded 1 direct eval | 23rank change if excluded 3 direct evals | -8rank change if excluded 7 direct evals | -7rank change if excluded 2 direct evals | -6rank change if excluded 12 direct evals | -7rank change if excluded 5 direct evals | +1.05 SDDirect |
| claude-2 | Not shown0 direct evals | 61rank change if excluded 3 direct evals | -4rank change if excluded 4 direct evals | -4rank change if excluded 3 direct evals | -4rank change if excluded 1 direct eval | -4rank change if excluded 9 direct evals | Not shown0 direct evals | +2.01 SDDirect |
| claude-opus-4 | 30rank change if excluded 1 direct eval | 40rank change if excluded 2 direct evals | 33rank change if excluded 4 direct evals | 32rank change if excluded 8 direct evals | 37rank change if excluded 5 direct evals | 34rank change if excluded 13 direct evals | 33rank change if excluded 4 direct evals | +2.11 SDDirect |
| gpt-5-pro | 275rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 2 direct evals | -3rank change if excluded 1 direct eval | 4rank change if excluded 2 direct evals | Not shown0 direct evals | +1.05 SDDirect |
| phi-3.5-moe-instruct | 1rank change if excluded 1 direct eval | -1rank change if excluded 4 direct evals | 1rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 4 direct evals | 2rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| gpt-5-mini | 68rank change if excluded 4 direct evals | 12rank change if excluded 5 direct evals | -2rank change if excluded 3 direct evals | 5rank change if excluded 11 direct evals | 6rank change if excluded 5 direct evals | 5rank change if excluded 17 direct evals | 41rank change if excluded 5 direct evals | +1.05 SDDirect |
| gemini-2.5-pro-exp | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -2rank change if excluded 3 direct evals | 125rank change if excluded 1 direct eval | -4rank change if excluded 1 direct eval | Not shown0 direct evals | +0.60 SDDirect |
| claude-opus-4.5 | -21rank change if excluded 2 direct evals | 9rank change if excluded 2 direct evals | 8rank change if excluded 2 direct evals | 8rank change if excluded 12 direct evals | 10rank change if excluded 6 direct evals | 10rank change if excluded 13 direct evals | 20rank change if excluded 8 direct evals | +2.05 SDDirect |
| qwen3.5-27b | Not shown0 direct evals | 2rank change if excluded 2 direct evals | 63rank change if excluded 3 direct evals | -1rank change if excluded 4 direct evals | 95rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | -0.67 SDDirect |
| gpt-5 | -18rank change if excluded 1 direct eval | -18rank change if excluded 6 direct evals | -10rank change if excluded 6 direct evals | -18rank change if excluded 13 direct evals | -18rank change if excluded 6 direct evals | -16rank change if excluded 25 direct evals | -11rank change if excluded 9 direct evals | +1.05 SDDirect |
| o1-preview | -18rank change if excluded 1 direct eval | 134rank change if excluded 2 direct evals | -18rank change if excluded 1 direct eval | -24rank change if excluded 1 direct eval | Not shown0 direct evals | 139rank change if excluded 2 direct evals | 139rank change if excluded 1 direct eval | +0.46 SDDirect |
| qwen3.5-397b-a17b | 7rank change if excluded 1 direct eval | 10rank change if excluded 4 direct evals | 7rank change if excluded 1 direct eval | 4rank change if excluded 7 direct evals | 7rank change if excluded 3 direct evals | 8rank change if excluded 7 direct evals | 5rank change if excluded 4 direct evals | -1.01 SDDirect |
| grok-4.20-multi-agent | Not shown0 direct evals | -81rank change if excluded 1 direct eval | Not shown0 direct evals | 3rank change if excluded 3 direct evals | 27rank change if excluded 1 direct eval | 16rank change if excluded 2 direct evals | Not shown0 direct evals | -0.62 SDDirect |
| claude-3.7-sonnet | 15rank change if excluded 1 direct eval | 16rank change if excluded 3 direct evals | -1rank change if excluded 3 direct evals | 22rank change if excluded 9 direct evals | 17rank change if excluded 3 direct evals | 8rank change if excluded 15 direct evals | 10rank change if excluded 7 direct evals | +2.01 SDDirect |
| claude-sonnet-4.5 | 5rank change if excluded 3 direct evals | 10rank change if excluded 7 direct evals | 3rank change if excluded 3 direct evals | 6rank change if excluded 13 direct evals | -3rank change if excluded 7 direct evals | 3rank change if excluded 25 direct evals | 9rank change if excluded 13 direct evals | +2.05 SDDirect |
| gpt-5.1 | 11rank change if excluded 1 direct eval | -14rank change if excluded 4 direct evals | -8rank change if excluded 5 direct evals | -9rank change if excluded 11 direct evals | -8rank change if excluded 5 direct evals | 5rank change if excluded 18 direct evals | -11rank change if excluded 8 direct evals | +1.55 SDDirect |
| gpt-5.1-codex | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -77rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 33rank change if excluded 1 direct eval | +1.55 SDDirect |
| granite-4.0-h-small | Not shown0 direct evals | -28rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 2 direct evals | Not shown0 direct evals | 9rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| gpt-5.4-pro | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 267rank change if excluded 3 direct evals | -20rank change if excluded 1 direct eval | 33rank change if excluded 1 direct eval | Not shown0 direct evals | +1.60 SDDirect |
| claude-sonnet-4 | 9rank change if excluded 1 direct eval | 10rank change if excluded 7 direct evals | 12rank change if excluded 3 direct evals | 14rank change if excluded 11 direct evals | -11rank change if excluded 6 direct evals | -2rank change if excluded 20 direct evals | 13rank change if excluded 14 direct evals | +2.11 SDDirect |
| claude-sonnet-5 | -57rank change if excluded 3 direct evals | 7rank change if excluded 4 direct evals | 6rank change if excluded 3 direct evals | -2rank change if excluded 11 direct evals | 6rank change if excluded 4 direct evals | 6rank change if excluded 8 direct evals | 6rank change if excluded 8 direct evals | +2.03 SDDirect |
| solar-mini4 | -2rank change if excluded 1 direct eval | -19rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | Not shown0 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| llama-3.1-tulu-3-8b | Not shown0 direct evals | -2rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 3 direct evals | 1rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| mimo-v2.6-pro | 1rank change if excluded 2 direct evals | -20rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 3 direct evals | 0rank change if excluded 1 direct eval | -2rank change if excluded 3 direct evals | 1rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| claude-opus-4.1 | 12rank change if excluded 2 direct evals | Not shown0 direct evals | 13rank change if excluded 4 direct evals | 13rank change if excluded 7 direct evals | 16rank change if excluded 4 direct evals | 21rank change if excluded 10 direct evals | 20rank change if excluded 1 direct eval | +2.11 SDDirect |
| grok-4.5 | -2rank change if excluded 2 direct evals | 0rank change if excluded 3 direct evals | 6rank change if excluded 1 direct eval | 1rank change if excluded 11 direct evals | 6rank change if excluded 1 direct eval | 6rank change if excluded 4 direct evals | -8rank change if excluded 5 direct evals | -0.62 SDDirect |
| mimo-v2-pro | Not shown0 direct evals | -106rank change if excluded 2 direct evals | Not shown0 direct evals | -96rank change if excluded 5 direct evals | 3rank change if excluded 1 direct eval | 5rank change if excluded 2 direct evals | -106rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| gemini-3.1-pro | Not shown0 direct evals | 2rank change if excluded 2 direct evals | -3rank change if excluded 1 direct eval | -7rank change if excluded 3 direct evals | -3rank change if excluded 1 direct eval | -3rank change if excluded 3 direct evals | -4rank change if excluded 1 direct eval | +0.97 SDDirect |
| gemini-3.1-pro-preview | 5rank change if excluded 2 direct evals | 11rank change if excluded 6 direct evals | 13rank change if excluded 6 direct evals | 11rank change if excluded 17 direct evals | 14rank change if excluded 6 direct evals | 11rank change if excluded 13 direct evals | 7rank change if excluded 7 direct evals | +0.97 SDDirect |
| mimo-v2.5-pro | -36rank change if excluded 1 direct eval | 4rank change if excluded 3 direct evals | Not shown0 direct evals | 0rank change if excluded 7 direct evals | 7rank change if excluded 1 direct eval | 3rank change if excluded 3 direct evals | 5rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| qwen3.5-9b | 2rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | -12rank change if excluded 2 direct evals | -0.67 SDDirect |
| minimax-m3 | 8rank change if excluded 3 direct evals | 19rank change if excluded 3 direct evals | Not shown0 direct evals | 16rank change if excluded 7 direct evals | 16rank change if excluded 1 direct eval | 21rank change if excluded 3 direct evals | 22rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| inkling | 19rank change if excluded 2 direct evals | 13rank change if excluded 1 direct eval | Not shown0 direct evals | -9rank change if excluded 4 direct evals | 11rank change if excluded 3 direct evals | 13rank change if excluded 4 direct evals | 5rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| gpt-oss-20b | -5rank change if excluded 1 direct eval | -23rank change if excluded 4 direct evals | 0rank change if excluded 2 direct evals | -1rank change if excluded 4 direct evals | 0rank change if excluded 4 direct evals | 0rank change if excluded 13 direct evals | 14rank change if excluded 3 direct evals | +1.05 SDDirect |
| gpt-5.4-mini | Not shown0 direct evals | 18rank change if excluded 3 direct evals | -37rank change if excluded 4 direct evals | 5rank change if excluded 12 direct evals | -2rank change if excluded 3 direct evals | 9rank change if excluded 6 direct evals | 15rank change if excluded 6 direct evals | +1.60 SDDirect |
| nemotron-3.5-lightning | 8rank change if excluded 1 direct eval | -45rank change if excluded 1 direct eval | 11rank change if excluded 1 direct eval | 11rank change if excluded 2 direct evals | Not shown0 direct evals | 11rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| claude-3.5-haiku | -58rank change if excluded 2 direct evals | 27rank change if excluded 4 direct evals | 27rank change if excluded 3 direct evals | 28rank change if excluded 5 direct evals | 26rank change if excluded 1 direct eval | 29rank change if excluded 9 direct evals | 27rank change if excluded 4 direct evals | +2.01 SDDirect |
| kimi-k2.6 | 18rank change if excluded 2 direct evals | -15rank change if excluded 3 direct evals | 18rank change if excluded 2 direct evals | -5rank change if excluded 12 direct evals | -11rank change if excluded 3 direct evals | -12rank change if excluded 9 direct evals | -16rank change if excluded 6 direct evals | -0.09 SDMedian-imputed |
| palmyra-x-004 | Not shown0 direct evals | Not shown0 direct evals | 19rank change if excluded 1 direct eval | Not shown0 direct evals | 17rank change if excluded 1 direct eval | 7rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| gemini-3.5-flash | -7rank change if excluded 5 direct evals | -3rank change if excluded 4 direct evals | 2rank change if excluded 2 direct evals | -6rank change if excluded 13 direct evals | 2rank change if excluded 3 direct evals | -1rank change if excluded 9 direct evals | 12rank change if excluded 9 direct evals | +1.16 SDDirect |
| glm-4.5-air | -64rank change if excluded 2 direct evals | Not shown0 direct evals | 9rank change if excluded 1 direct eval | 3rank change if excluded 5 direct evals | 9rank change if excluded 2 direct evals | 9rank change if excluded 6 direct evals | Not shown0 direct evals | -1.03 SDDirect |
| qwen3.5-plus | Not shown0 direct evals | 38rank change if excluded 3 direct evals | 12rank change if excluded 3 direct evals | 11rank change if excluded 5 direct evals | 11rank change if excluded 1 direct eval | 18rank change if excluded 5 direct evals | 18rank change if excluded 1 direct eval | -1.01 SDDirect |
| nova-lite | 276rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | -1rank change if excluded 4 direct evals | Not shown0 direct evals | 1rank change if excluded 3 direct evals | 2rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| gemma-4-26b-a4b-it | 39rank change if excluded 1 direct eval | -20rank change if excluded 2 direct evals | Not shown0 direct evals | -13rank change if excluded 4 direct evals | Not shown0 direct evals | -20rank change if excluded 3 direct evals | -12rank change if excluded 1 direct eval | +1.16 SDDirect |
| chatgpt-4o | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 5rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -7rank change if excluded 2 direct evals | 105rank change if excluded 1 direct eval | +1.05 SDDirect |
| qwen3.7-max | -6rank change if excluded 1 direct eval | -62rank change if excluded 3 direct evals | 7rank change if excluded 1 direct eval | 8rank change if excluded 9 direct evals | 5rank change if excluded 2 direct evals | 3rank change if excluded 7 direct evals | -3rank change if excluded 3 direct evals | -0.67 SDDirect |
| gpt-4.1 | 24rank change if excluded 2 direct evals | 13rank change if excluded 5 direct evals | -115rank change if excluded 6 direct evals | 39rank change if excluded 12 direct evals | 41rank change if excluded 5 direct evals | 13rank change if excluded 20 direct evals | 14rank change if excluded 5 direct evals | +1.05 SDDirect |
| gemini-1.5-pro | 90rank change if excluded 3 direct evals | 23rank change if excluded 3 direct evals | 21rank change if excluded 3 direct evals | 35rank change if excluded 5 direct evals | 35rank change if excluded 4 direct evals | 37rank change if excluded 14 direct evals | 33rank change if excluded 3 direct evals | +0.76 SDDirect |
| palmyra-fin | Not shown0 direct evals | Not shown0 direct evals | -77rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 1 direct eval | 44rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| gemini-1.0-pro | 27rank change if excluded 1 direct eval | 27rank change if excluded 2 direct evals | 27rank change if excluded 3 direct evals | 28rank change if excluded 1 direct eval | 44rank change if excluded 1 direct eval | -7rank change if excluded 7 direct evals | Not shown0 direct evals | +0.76 SDDirect |
| qwen3-max | -11rank change if excluded 1 direct eval | -4rank change if excluded 2 direct evals | -4rank change if excluded 3 direct evals | -3rank change if excluded 5 direct evals | Not shown0 direct evals | -15rank change if excluded 5 direct evals | 19rank change if excluded 1 direct eval | -1.01 SDDirect |
| muse-spark | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -10rank change if excluded 3 direct evals | Not shown0 direct evals | 0rank change if excluded 1 direct eval | 101rank change if excluded 1 direct eval | +0.35 SDDirect |
| o1 | Not shown0 direct evals | 5rank change if excluded 1 direct eval | 19rank change if excluded 1 direct eval | -132rank change if excluded 6 direct evals | 0rank change if excluded 4 direct evals | -7rank change if excluded 12 direct evals | 6rank change if excluded 2 direct evals | +0.46 SDDirect |
| qwen3-235b-a22b-fp8-throughput | Not shown0 direct evals | Not shown0 direct evals | 30rank change if excluded 1 direct eval | Not shown0 direct evals | 29rank change if excluded 1 direct eval | -68rank change if excluded 5 direct evals | Not shown0 direct evals | -1.01 SDDirect |
| mistral-large | 99rank change if excluded 1 direct eval | 44rank change if excluded 1 direct eval | 44rank change if excluded 1 direct eval | 44rank change if excluded 1 direct eval | 44rank change if excluded 1 direct eval | -85rank change if excluded 5 direct evals | Not shown0 direct evals | -1.20 SDDirect |
| qwen3.7-flash | 10rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.67 SDDirect |
| glm-5 | -34rank change if excluded 1 direct eval | 28rank change if excluded 4 direct evals | -15rank change if excluded 2 direct evals | 6rank change if excluded 9 direct evals | -15rank change if excluded 2 direct evals | -15rank change if excluded 6 direct evals | 9rank change if excluded 4 direct evals | -0.59 SDDirect |
| gemma-4-31b-it | -5rank change if excluded 3 direct evals | -20rank change if excluded 4 direct evals | 28rank change if excluded 4 direct evals | -19rank change if excluded 9 direct evals | 24rank change if excluded 2 direct evals | -20rank change if excluded 6 direct evals | 22rank change if excluded 1 direct eval | +1.16 SDDirect |
| internlm2-chat-7b | Not shown0 direct evals | 17rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 4 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen3-4b | Not shown0 direct evals | -112rank change if excluded 1 direct eval | 35rank change if excluded 1 direct eval | 63rank change if excluded 1 direct eval | 35rank change if excluded 1 direct eval | 61rank change if excluded 1 direct eval | Not shown0 direct evals | -1.01 SDDirect |
| ministral-3-3b | 289rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | -1.20 SDDirect |
| gpt-4.5-preview | Not shown0 direct evals | 20rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | 10rank change if excluded 3 direct evals | 4rank change if excluded 1 direct eval | 9rank change if excluded 6 direct evals | 20rank change if excluded 1 direct eval | +0.46 SDDirect |
| qwen2-72b-instruct | Not shown0 direct evals | -2rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | Not shown0 direct evals | -4rank change if excluded 1 direct eval | -5rank change if excluded 9 direct evals | -2rank change if excluded 1 direct eval | -1.01 SDDirect |
| laguna-s-2.1-poolside | -13rank change if excluded 2 direct evals | 10rank change if excluded 2 direct evals | -10rank change if excluded 1 direct eval | -11rank change if excluded 2 direct evals | Not shown0 direct evals | -11rank change if excluded 2 direct evals | -11rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| deepseek-v3.1 | 100rank change if excluded 2 direct evals | -27rank change if excluded 1 direct eval | 98rank change if excluded 2 direct evals | -39rank change if excluded 5 direct evals | 96rank change if excluded 2 direct evals | -30rank change if excluded 8 direct evals | -27rank change if excluded 1 direct eval | -1.44 SDDirect |
| palmyra-x5 | Not shown0 direct evals | Not shown0 direct evals | -18rank change if excluded 1 direct eval | Not shown0 direct evals | -13rank change if excluded 1 direct eval | -2rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| o3 | Not shown0 direct evals | 17rank change if excluded 2 direct evals | -32rank change if excluded 4 direct evals | 14rank change if excluded 8 direct evals | -33rank change if excluded 4 direct evals | 18rank change if excluded 15 direct evals | -7rank change if excluded 6 direct evals | +1.05 SDDirect |
| minimax-m2.7 | -15rank change if excluded 2 direct evals | 15rank change if excluded 6 direct evals | -10rank change if excluded 1 direct eval | -11rank change if excluded 9 direct evals | -14rank change if excluded 3 direct evals | -13rank change if excluded 6 direct evals | 1rank change if excluded 5 direct evals | -0.09 SDMedian-imputed |
| kimi-k2 | -2rank change if excluded 1 direct eval | -20rank change if excluded 4 direct evals | 0rank change if excluded 3 direct evals | -26rank change if excluded 13 direct evals | 6rank change if excluded 2 direct evals | -19rank change if excluded 17 direct evals | 1rank change if excluded 7 direct evals | -0.09 SDMedian-imputed |
| qwen-72b-chat | Not shown0 direct evals | 50rank change if excluded 2 direct evals | 49rank change if excluded 2 direct evals | 49rank change if excluded 1 direct eval | Not shown0 direct evals | 139rank change if excluded 4 direct evals | Not shown0 direct evals | -1.01 SDDirect |
| deepseek-v4-flash | 35rank change if excluded 4 direct evals | -12rank change if excluded 4 direct evals | 0rank change if excluded 1 direct eval | -8rank change if excluded 9 direct evals | -7rank change if excluded 3 direct evals | -7rank change if excluded 6 direct evals | -8rank change if excluded 5 direct evals | -1.05 SDDirect |
| seed-2.0-mini | 24rank change if excluded 1 direct eval | -71rank change if excluded 2 direct evals | -40rank change if excluded 1 direct eval | -40rank change if excluded 2 direct evals | Not shown0 direct evals | -40rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| gpt-oss-120b | -16rank change if excluded 3 direct evals | 10rank change if excluded 6 direct evals | 8rank change if excluded 6 direct evals | -28rank change if excluded 12 direct evals | -15rank change if excluded 3 direct evals | 10rank change if excluded 18 direct evals | -7rank change if excluded 5 direct evals | +1.05 SDDirect |
| mimo-v2.5 | -2rank change if excluded 2 direct evals | 126rank change if excluded 1 direct eval | Not shown0 direct evals | -8rank change if excluded 5 direct evals | -9rank change if excluded 2 direct evals | -4rank change if excluded 2 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| o1-mini | 42rank change if excluded 1 direct eval | 42rank change if excluded 2 direct evals | 62rank change if excluded 2 direct evals | -15rank change if excluded 2 direct evals | 6rank change if excluded 1 direct eval | -153rank change if excluded 8 direct evals | 6rank change if excluded 1 direct eval | +0.46 SDDirect |
| gemini-1.5-flash | 48rank change if excluded 2 direct evals | 29rank change if excluded 2 direct evals | 29rank change if excluded 2 direct evals | 29rank change if excluded 1 direct eval | 26rank change if excluded 3 direct evals | 29rank change if excluded 13 direct evals | 28rank change if excluded 4 direct evals | +0.76 SDDirect |
| hy3 | -30rank change if excluded 2 direct evals | -14rank change if excluded 2 direct evals | Not shown0 direct evals | 161rank change if excluded 3 direct evals | Not shown0 direct evals | -8rank change if excluded 1 direct eval | -14rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| gpt-5.3-codex | Not shown0 direct evals | 165rank change if excluded 2 direct evals | Not shown0 direct evals | -12rank change if excluded 4 direct evals | -15rank change if excluded 1 direct eval | -11rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | +1.55 SDDirect |
| glm-5.2 | 7rank change if excluded 3 direct evals | 2rank change if excluded 4 direct evals | 6rank change if excluded 2 direct evals | 2rank change if excluded 13 direct evals | 6rank change if excluded 2 direct evals | 5rank change if excluded 8 direct evals | 2rank change if excluded 6 direct evals | -0.61 SDDirect |
| gpt-oss-safeguard-20b | -87rank change if excluded 1 direct eval | -136rank change if excluded 2 direct evals | Not shown0 direct evals | -136rank change if excluded 1 direct eval | Not shown0 direct evals | 47rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | +1.55 SDDirect |
| deepseek-v3.2-exp | -118rank change if excluded 1 direct eval | -11rank change if excluded 1 direct eval | Not shown0 direct evals | 8rank change if excluded 2 direct evals | Not shown0 direct evals | -11rank change if excluded 3 direct evals | 26rank change if excluded 2 direct evals | -1.01 SDDirect |
| gpt-5.2-codex | Not shown0 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | -8rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 1 direct eval | +1.55 SDDirect |
| internlm-chat-20b | Not shown0 direct evals | 7rank change if excluded 4 direct evals | -9rank change if excluded 4 direct evals | -9rank change if excluded 1 direct eval | Not shown0 direct evals | 7rank change if excluded 4 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| gemini-2.5-pro | -9rank change if excluded 1 direct eval | -12rank change if excluded 9 direct evals | -12rank change if excluded 6 direct evals | 38rank change if excluded 19 direct evals | 37rank change if excluded 6 direct evals | 37rank change if excluded 22 direct evals | -13rank change if excluded 11 direct evals | +0.60 SDDirect |
| gemini-3-pro-preview | -18rank change if excluded 1 direct eval | -13rank change if excluded 2 direct evals | 4rank change if excluded 2 direct evals | -13rank change if excluded 12 direct evals | 1rank change if excluded 3 direct evals | -6rank change if excluded 14 direct evals | 1rank change if excluded 6 direct evals | +0.97 SDDirect |
| qwen3.6-27b | Not shown0 direct evals | -14rank change if excluded 1 direct eval | Not shown0 direct evals | -16rank change if excluded 1 direct eval | -14rank change if excluded 1 direct eval | -14rank change if excluded 1 direct eval | -122rank change if excluded 2 direct evals | -0.67 SDDirect |
| gemini-3.5-flash-lite | -99rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 55rank change if excluded 5 direct evals | 106rank change if excluded 3 direct evals | -66rank change if excluded 2 direct evals | -61rank change if excluded 1 direct eval | +1.16 SDDirect |
| internlm2-chat-20b | Not shown0 direct evals | -63rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | Not shown0 direct evals | -4rank change if excluded 5 direct evals | 16rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| minimax-m2 | -13rank change if excluded 1 direct eval | 4rank change if excluded 3 direct evals | -5rank change if excluded 1 direct eval | -3rank change if excluded 4 direct evals | Not shown0 direct evals | 3rank change if excluded 4 direct evals | -4rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| qwen2.5-72b-instruct | 180rank change if excluded 1 direct eval | -10rank change if excluded 1 direct eval | -11rank change if excluded 3 direct evals | -12rank change if excluded 4 direct evals | 8rank change if excluded 3 direct evals | -10rank change if excluded 12 direct evals | -10rank change if excluded 2 direct evals | -1.01 SDDirect |
| grok-4.3 | -7rank change if excluded 2 direct evals | -12rank change if excluded 5 direct evals | -9rank change if excluded 3 direct evals | -10rank change if excluded 12 direct evals | -9rank change if excluded 2 direct evals | -10rank change if excluded 9 direct evals | -10rank change if excluded 7 direct evals | -0.62 SDDirect |
| longcat-2.0 | Not shown0 direct evals | 55rank change if excluded 1 direct eval | -8rank change if excluded 1 direct eval | -8rank change if excluded 3 direct evals | -8rank change if excluded 1 direct eval | -8rank change if excluded 2 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen3-235b-a22b | 21rank change if excluded 3 direct evals | 26rank change if excluded 7 direct evals | 30rank change if excluded 4 direct evals | 22rank change if excluded 8 direct evals | 25rank change if excluded 4 direct evals | 24rank change if excluded 18 direct evals | 8rank change if excluded 8 direct evals | -1.01 SDDirect |
| deepseek-v3.2 | 7rank change if excluded 4 direct evals | -29rank change if excluded 11 direct evals | -5rank change if excluded 3 direct evals | -20rank change if excluded 16 direct evals | -20rank change if excluded 2 direct evals | -20rank change if excluded 12 direct evals | -4rank change if excluded 9 direct evals | -1.01 SDDirect |
| gpt-4.1-mini | 6rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | 1rank change if excluded 3 direct evals | 6rank change if excluded 4 direct evals | 7rank change if excluded 4 direct evals | 12rank change if excluded 16 direct evals | -19rank change if excluded 3 direct evals | +1.05 SDDirect |
| olmo-3-7b-instruct | Not shown0 direct evals | 82rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | -5rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| claude-haiku-5.5 | -216rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 33rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 33rank change if excluded 1 direct eval | +2.03 SDDirect |
| qwen3-30b-a3b | 216rank change if excluded 1 direct eval | -42rank change if excluded 1 direct eval | Not shown0 direct evals | -40rank change if excluded 2 direct evals | Not shown0 direct evals | -42rank change if excluded 3 direct evals | -42rank change if excluded 1 direct eval | -1.01 SDDirect |
| nemotron-3-nano-omni-30b-a3b | 259rank change if excluded 1 direct eval | Not shown0 direct evals | 5rank change if excluded 1 direct eval | 0rank change if excluded 2 direct evals | Not shown0 direct evals | 6rank change if excluded 1 direct eval | 5rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| grok-3-mini | 19rank change if excluded 1 direct eval | -54rank change if excluded 2 direct evals | 24rank change if excluded 3 direct evals | 24rank change if excluded 4 direct evals | 25rank change if excluded 1 direct eval | 25rank change if excluded 11 direct evals | -38rank change if excluded 2 direct evals | -0.07 SDDirect |
| phi-4-reasoning-plus | Not shown0 direct evals | 10rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | Not shown0 direct evals | 9rank change if excluded 1 direct eval | 10rank change if excluded 2 direct evals | 10rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| doubao-seed-1.6 | Not shown0 direct evals | -9rank change if excluded 4 direct evals | -2rank change if excluded 3 direct evals | -7rank change if excluded 4 direct evals | -6rank change if excluded 1 direct eval | -21rank change if excluded 5 direct evals | -2rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| grok-4-fast | -3rank change if excluded 1 direct eval | Not shown0 direct evals | 31rank change if excluded 2 direct evals | -60rank change if excluded 8 direct evals | -2rank change if excluded 1 direct eval | -2rank change if excluded 7 direct evals | -2rank change if excluded 2 direct evals | -0.33 SDDirect |
| ernie-4.5-300b-a47b | Not shown0 direct evals | 7rank change if excluded 1 direct eval | -5rank change if excluded 1 direct eval | -137rank change if excluded 3 direct evals | 9rank change if excluded 1 direct eval | -4rank change if excluded 2 direct evals | -9rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| nova-2.0-lite | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 12rank change if excluded 3 direct evals | Not shown0 direct evals | -4rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| claude-3-opus | 43rank change if excluded 5 direct evals | 17rank change if excluded 4 direct evals | 12rank change if excluded 4 direct evals | 18rank change if excluded 5 direct evals | 15rank change if excluded 4 direct evals | 18rank change if excluded 15 direct evals | 16rank change if excluded 7 direct evals | +2.01 SDDirect |
| kimi-k2.7-code | Not shown0 direct evals | 211rank change if excluded 3 direct evals | Not shown0 direct evals | -21rank change if excluded 3 direct evals | Not shown0 direct evals | -22rank change if excluded 1 direct eval | -21rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| qwen3-32b | 292rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | 21rank change if excluded 1 direct eval | 0rank change if excluded 2 direct evals | 21rank change if excluded 1 direct eval | -1rank change if excluded 6 direct evals | 0rank change if excluded 1 direct eval | -1.01 SDDirect |
| claude-3.5-sonnet | -78rank change if excluded 4 direct evals | -8rank change if excluded 4 direct evals | -9rank change if excluded 3 direct evals | -9rank change if excluded 9 direct evals | -10rank change if excluded 4 direct evals | 3rank change if excluded 19 direct evals | 9rank change if excluded 8 direct evals | +2.01 SDDirect |
| glm-5.1 | -16rank change if excluded 2 direct evals | -22rank change if excluded 4 direct evals | -10rank change if excluded 1 direct eval | -15rank change if excluded 9 direct evals | -8rank change if excluded 1 direct eval | -12rank change if excluded 4 direct evals | -22rank change if excluded 8 direct evals | -0.61 SDDirect |
| ox-alpha | -221rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 19rank change if excluded 3 direct evals | 19rank change if excluded 1 direct eval | 21rank change if excluded 1 direct eval | 21rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| minimax-m2.5 | -12rank change if excluded 1 direct eval | -1rank change if excluded 3 direct evals | 46rank change if excluded 2 direct evals | -23rank change if excluded 8 direct evals | -10rank change if excluded 1 direct eval | -9rank change if excluded 3 direct evals | -26rank change if excluded 3 direct evals | -0.09 SDMedian-imputed |
| olmo-3.1-32b-instruct | Not shown0 direct evals | 186rank change if excluded 1 direct eval | Not shown0 direct evals | -7rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| gpt-4-turbo | 8rank change if excluded 1 direct eval | -8rank change if excluded 6 direct evals | -1rank change if excluded 4 direct evals | 7rank change if excluded 5 direct evals | 0rank change if excluded 3 direct evals | 2rank change if excluded 19 direct evals | 4rank change if excluded 7 direct evals | +0.46 SDDirect |
| deepseek-v3 | -27rank change if excluded 4 direct evals | 4rank change if excluded 4 direct evals | 7rank change if excluded 7 direct evals | 3rank change if excluded 10 direct evals | 3rank change if excluded 2 direct evals | 6rank change if excluded 16 direct evals | 9rank change if excluded 6 direct evals | -1.44 SDDirect |
| phi-3-mini-4k-instruct | Not shown0 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -1rank change if excluded 4 direct evals | 3rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| gemma-4-e4b | Not shown0 direct evals | -114rank change if excluded 1 direct eval | Not shown0 direct evals | 222rank change if excluded 1 direct eval | Not shown0 direct evals | -114rank change if excluded 2 direct evals | -8rank change if excluded 1 direct eval | +1.16 SDDirect |
| qwen3.8-max | -109rank change if excluded 2 direct evals | 76rank change if excluded 2 direct evals | Not shown0 direct evals | 78rank change if excluded 5 direct evals | -104rank change if excluded 2 direct evals | -104rank change if excluded 3 direct evals | 72rank change if excluded 3 direct evals | -0.67 SDDirect |
| grok-4.20 | -1rank change if excluded 1 direct eval | -16rank change if excluded 6 direct evals | 3rank change if excluded 3 direct evals | 5rank change if excluded 14 direct evals | 1rank change if excluded 5 direct evals | 0rank change if excluded 10 direct evals | -8rank change if excluded 7 direct evals | -0.62 SDDirect |
| phi-4 | Not shown0 direct evals | 0rank change if excluded 2 direct evals | -19rank change if excluded 1 direct eval | -212rank change if excluded 4 direct evals | Not shown0 direct evals | -15rank change if excluded 3 direct evals | -19rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| mercury-2 | Not shown0 direct evals | 294rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | -37rank change if excluded 4 direct evals | Not shown0 direct evals | 1rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| gemini-2.5-flash | -9rank change if excluded 2 direct evals | -5rank change if excluded 6 direct evals | 0rank change if excluded 4 direct evals | -8rank change if excluded 13 direct evals | -8rank change if excluded 4 direct evals | -14rank change if excluded 20 direct evals | -4rank change if excluded 10 direct evals | +0.60 SDDirect |
| qwen2.5-32b-instruct | Not shown0 direct evals | 12rank change if excluded 1 direct eval | Not shown0 direct evals | -44rank change if excluded 2 direct evals | 16rank change if excluded 1 direct eval | 12rank change if excluded 2 direct evals | 38rank change if excluded 2 direct evals | -1.01 SDDirect |
| grok-4 | -11rank change if excluded 1 direct eval | -7rank change if excluded 5 direct evals | 16rank change if excluded 7 direct evals | 14rank change if excluded 13 direct evals | -7rank change if excluded 2 direct evals | -6rank change if excluded 16 direct evals | -8rank change if excluded 5 direct evals | -0.07 SDDirect |
| qwen3.6-plus | 21rank change if excluded 1 direct eval | -2rank change if excluded 4 direct evals | 12rank change if excluded 1 direct eval | 4rank change if excluded 6 direct evals | 12rank change if excluded 1 direct eval | 19rank change if excluded 3 direct evals | -6rank change if excluded 2 direct evals | -0.67 SDDirect |
| granite-4.0-small-guardian | Not shown0 direct evals | Not shown0 direct evals | -70rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 1 direct eval | 12rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen-14b-chat | Not shown0 direct evals | 117rank change if excluded 5 direct evals | -36rank change if excluded 5 direct evals | 40rank change if excluded 1 direct eval | Not shown0 direct evals | 40rank change if excluded 8 direct evals | Not shown0 direct evals | -1.01 SDDirect |
| qwen3-next-80b-a3b | Not shown0 direct evals | -9rank change if excluded 1 direct eval | -22rank change if excluded 1 direct eval | -18rank change if excluded 3 direct evals | 37rank change if excluded 2 direct evals | 53rank change if excluded 9 direct evals | -9rank change if excluded 1 direct eval | -1.01 SDDirect |
| gpt-4o | 5rank change if excluded 3 direct evals | 3rank change if excluded 12 direct evals | 0rank change if excluded 9 direct evals | 1rank change if excluded 21 direct evals | 4rank change if excluded 6 direct evals | 0rank change if excluded 37 direct evals | 1rank change if excluded 20 direct evals | +0.46 SDDirect |
| deepseek-v4-pro | 193rank change if excluded 3 direct evals | -20rank change if excluded 4 direct evals | -12rank change if excluded 2 direct evals | -16rank change if excluded 12 direct evals | -19rank change if excluded 2 direct evals | -18rank change if excluded 7 direct evals | -18rank change if excluded 7 direct evals | -1.05 SDDirect |
| qwen3.6-max-preview | Not shown0 direct evals | -13rank change if excluded 1 direct eval | Not shown0 direct evals | -16rank change if excluded 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 2 direct evals | -0.67 SDDirect |
| space-bunny-alpha | -112rank change if excluded 1 direct eval | -91rank change if excluded 1 direct eval | -92rank change if excluded 1 direct eval | -92rank change if excluded 1 direct eval | Not shown0 direct evals | -92rank change if excluded 1 direct eval | 83rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen3-max-preview | Not shown0 direct evals | Not shown0 direct evals | 250rank change if excluded 1 direct eval | -51rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -1.01 SDDirect |
| qwen3.7-plus | -4rank change if excluded 1 direct eval | -3rank change if excluded 2 direct evals | -9rank change if excluded 1 direct eval | -15rank change if excluded 5 direct evals | Not shown0 direct evals | -10rank change if excluded 2 direct evals | Not shown0 direct evals | -0.67 SDDirect |
| gemini-2.0-pro-preview | Not shown0 direct evals | Not shown0 direct evals | 76rank change if excluded 1 direct eval | Not shown0 direct evals | -14rank change if excluded 1 direct eval | -25rank change if excluded 5 direct evals | Not shown0 direct evals | +0.76 SDDirect |
| kimi-k2.5 | -2rank change if excluded 2 direct evals | 20rank change if excluded 7 direct evals | -16rank change if excluded 4 direct evals | 20rank change if excluded 15 direct evals | -8rank change if excluded 3 direct evals | -4rank change if excluded 13 direct evals | -8rank change if excluded 10 direct evals | -0.09 SDMedian-imputed |
| grok-4.6 | -166rank change if excluded 2 direct evals | 14rank change if excluded 1 direct eval | Not shown0 direct evals | 13rank change if excluded 7 direct evals | 14rank change if excluded 2 direct evals | 14rank change if excluded 3 direct evals | 11rank change if excluded 6 direct evals | -0.62 SDDirect |
| gpt-4 | 73rank change if excluded 1 direct eval | 75rank change if excluded 6 direct evals | 74rank change if excluded 6 direct evals | -29rank change if excluded 2 direct evals | -30rank change if excluded 2 direct evals | -31rank change if excluded 14 direct evals | -30rank change if excluded 1 direct eval | +0.46 SDDirect |
| mistral-medium | Not shown0 direct evals | -4rank change if excluded 1 direct eval | Not shown0 direct evals | 43rank change if excluded 1 direct eval | 46rank change if excluded 1 direct eval | 43rank change if excluded 2 direct evals | -4rank change if excluded 1 direct eval | -1.20 SDDirect |
| qwen3.8-27b | -262rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -6rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -12rank change if excluded 1 direct eval | -0.67 SDDirect |
| grok-3 | -37rank change if excluded 2 direct evals | -6rank change if excluded 2 direct evals | -6rank change if excluded 3 direct evals | -5rank change if excluded 5 direct evals | -6rank change if excluded 1 direct eval | -5rank change if excluded 11 direct evals | -4rank change if excluded 2 direct evals | -0.07 SDDirect |
| qwen1.5-32b-chat | Not shown0 direct evals | 30rank change if excluded 1 direct eval | 30rank change if excluded 1 direct eval | 31rank change if excluded 2 direct evals | -63rank change if excluded 1 direct eval | 31rank change if excluded 4 direct evals | 30rank change if excluded 1 direct eval | -1.01 SDDirect |
| nemotron-3-ultra-550b-a55b | -18rank change if excluded 2 direct evals | -12rank change if excluded 2 direct evals | -14rank change if excluded 1 direct eval | -16rank change if excluded 6 direct evals | -111rank change if excluded 1 direct eval | -14rank change if excluded 3 direct evals | -15rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| glm-4.7-flash | 253rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -33rank change if excluded 3 direct evals | Not shown0 direct evals | -30rank change if excluded 1 direct eval | Not shown0 direct evals | -0.59 SDDirect |
| nemotron-3-super-120b-a12b | -20rank change if excluded 1 direct eval | 7rank change if excluded 2 direct evals | Not shown0 direct evals | -4rank change if excluded 5 direct evals | -138rank change if excluded 3 direct evals | -3rank change if excluded 5 direct evals | -16rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| qwen3-235b-a22b-instruct | Not shown0 direct evals | 181rank change if excluded 1 direct eval | Not shown0 direct evals | -44rank change if excluded 2 direct evals | Not shown0 direct evals | -11rank change if excluded 1 direct eval | -10rank change if excluded 1 direct eval | -1.01 SDDirect |
| claude-3-haiku | 9rank change if excluded 1 direct eval | 9rank change if excluded 2 direct evals | 8rank change if excluded 3 direct evals | -67rank change if excluded 4 direct evals | -43rank change if excluded 4 direct evals | 9rank change if excluded 12 direct evals | 3rank change if excluded 3 direct evals | +2.01 SDDirect |
| o3-mini | Not shown0 direct evals | 5rank change if excluded 2 direct evals | 6rank change if excluded 3 direct evals | 6rank change if excluded 7 direct evals | 4rank change if excluded 3 direct evals | 17rank change if excluded 15 direct evals | 5rank change if excluded 5 direct evals | +0.46 SDDirect |
| nova-premier | -112rank change if excluded 1 direct eval | 22rank change if excluded 1 direct eval | Not shown0 direct evals | 15rank change if excluded 1 direct eval | Not shown0 direct evals | 22rank change if excluded 2 direct evals | 15rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| baichuan-2-13b-chat | Not shown0 direct evals | 39rank change if excluded 6 direct evals | 16rank change if excluded 6 direct evals | 16rank change if excluded 2 direct evals | Not shown0 direct evals | 18rank change if excluded 13 direct evals | 14rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen3.5-flash | -117rank change if excluded 1 direct eval | 38rank change if excluded 1 direct eval | -139rank change if excluded 2 direct evals | 37rank change if excluded 3 direct evals | Not shown0 direct evals | -4rank change if excluded 2 direct evals | Not shown0 direct evals | -0.67 SDDirect |
| nova-micro | -221rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | Not shown0 direct evals | 4rank change if excluded 2 direct evals | Not shown0 direct evals | 1rank change if excluded 3 direct evals | 1rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| nemotron-3-nano-30b-a3b | 149rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -122rank change if excluded 5 direct evals | -123rank change if excluded 1 direct eval | -101rank change if excluded 3 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| gpt-3.5-turbo | 5rank change if excluded 2 direct evals | 8rank change if excluded 7 direct evals | 21rank change if excluded 10 direct evals | 9rank change if excluded 4 direct evals | 2rank change if excluded 3 direct evals | 3rank change if excluded 20 direct evals | 7rank change if excluded 3 direct evals | +0.46 SDDirect |
| qwen1.5-14b-chat | Not shown0 direct evals | 26rank change if excluded 3 direct evals | 26rank change if excluded 2 direct evals | 25rank change if excluded 3 direct evals | Not shown0 direct evals | 26rank change if excluded 6 direct evals | 25rank change if excluded 2 direct evals | -1.01 SDDirect |
| command-r-plus | 10rank change if excluded 1 direct eval | -12rank change if excluded 2 direct evals | -13rank change if excluded 2 direct evals | -11rank change if excluded 2 direct evals | -12rank change if excluded 1 direct eval | -12rank change if excluded 8 direct evals | 49rank change if excluded 3 direct evals | -0.09 SDMedian-imputed |
| llama-3.3-70b-instruct | 108rank change if excluded 6 direct evals | 1rank change if excluded 6 direct evals | 2rank change if excluded 5 direct evals | 3rank change if excluded 13 direct evals | 1rank change if excluded 5 direct evals | 1rank change if excluded 17 direct evals | 2rank change if excluded 3 direct evals | -1.24 SDDirect |
| deepseek-v4-flash-0731 | -216rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 1 direct eval | -1.05 SDDirect |
| text-davinci-003 | Not shown0 direct evals | 19rank change if excluded 2 direct evals | 16rank change if excluded 2 direct evals | 17rank change if excluded 1 direct eval | Not shown0 direct evals | 17rank change if excluded 3 direct evals | Not shown0 direct evals | +0.46 SDDirect |
| qwen1.5-72b-chat | Not shown0 direct evals | 14rank change if excluded 2 direct evals | 4rank change if excluded 3 direct evals | 13rank change if excluded 4 direct evals | 18rank change if excluded 2 direct evals | 16rank change if excluded 12 direct evals | 19rank change if excluded 2 direct evals | -1.01 SDDirect |
| o4-mini | Not shown0 direct evals | -5rank change if excluded 3 direct evals | -7rank change if excluded 3 direct evals | -11rank change if excluded 12 direct evals | -11rank change if excluded 4 direct evals | -7rank change if excluded 17 direct evals | 8rank change if excluded 7 direct evals | +1.05 SDDirect |
| deepseek-r1 | 5rank change if excluded 2 direct evals | -1rank change if excluded 9 direct evals | 5rank change if excluded 11 direct evals | -3rank change if excluded 18 direct evals | 0rank change if excluded 6 direct evals | -1rank change if excluded 26 direct evals | -1rank change if excluded 9 direct evals | -1.44 SDDirect |
| gemini-3.1-flash-lite | -54rank change if excluded 3 direct evals | -6rank change if excluded 5 direct evals | -24rank change if excluded 5 direct evals | -5rank change if excluded 12 direct evals | -7rank change if excluded 5 direct evals | -8rank change if excluded 9 direct evals | -5rank change if excluded 5 direct evals | +1.16 SDDirect |
| llama-4-maverick | -3rank change if excluded 2 direct evals | -3rank change if excluded 11 direct evals | -2rank change if excluded 5 direct evals | -2rank change if excluded 14 direct evals | 0rank change if excluded 2 direct evals | -4rank change if excluded 22 direct evals | 1rank change if excluded 6 direct evals | -0.43 SDDirect |
| gemini-2.0-flash | -8rank change if excluded 1 direct eval | -6rank change if excluded 2 direct evals | 1rank change if excluded 3 direct evals | -5rank change if excluded 8 direct evals | -5rank change if excluded 1 direct eval | 19rank change if excluded 11 direct evals | -38rank change if excluded 3 direct evals | +0.76 SDDirect |
| glm-4-9b-chat | Not shown0 direct evals | 34rank change if excluded 2 direct evals | 29rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -37rank change if excluded 8 direct evals | 29rank change if excluded 2 direct evals | -0.09 SDDirect |
| qwen3-30b-a3b-instruct | -13rank change if excluded 1 direct eval | 41rank change if excluded 2 direct evals | -15rank change if excluded 2 direct evals | -8rank change if excluded 2 direct evals | Not shown0 direct evals | -8rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | -1.01 SDDirect |
| yi-34b-chat | Not shown0 direct evals | 8rank change if excluded 4 direct evals | 6rank change if excluded 4 direct evals | -1rank change if excluded 4 direct evals | Not shown0 direct evals | 10rank change if excluded 9 direct evals | 9rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| granite-4.0-small | Not shown0 direct evals | Not shown0 direct evals | -2rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 1 direct eval | 9rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| solar-pro-4 | -162rank change if excluded 2 direct evals | 15rank change if excluded 1 direct eval | 12rank change if excluded 1 direct eval | 13rank change if excluded 2 direct evals | Not shown0 direct evals | 12rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| llama-3.2-instruct-11b-vision | Not shown0 direct evals | -10rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | 175rank change if excluded 2 direct evals | 115rank change if excluded 1 direct eval | -1.24 SDDirect |
| hy3-preview | 138rank change if excluded 1 direct eval | -27rank change if excluded 2 direct evals | Not shown0 direct evals | -16rank change if excluded 4 direct evals | Not shown0 direct evals | -19rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| gpt-5.4-nano | Not shown0 direct evals | 0rank change if excluded 1 direct eval | -200rank change if excluded 2 direct evals | 6rank change if excluded 9 direct evals | -211rank change if excluded 2 direct evals | 0rank change if excluded 4 direct evals | 0rank change if excluded 4 direct evals | +1.60 SDDirect |
| gemma-2b-it | -84rank change if excluded 1 direct eval | -133rank change if excluded 3 direct evals | -10rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | 2rank change if excluded 4 direct evals | 2rank change if excluded 1 direct eval | +0.76 SDDirect |
| llama-3.1-405b-instruct | Not shown0 direct evals | -17rank change if excluded 3 direct evals | 17rank change if excluded 3 direct evals | 18rank change if excluded 7 direct evals | -3rank change if excluded 3 direct evals | 14rank change if excluded 17 direct evals | 56rank change if excluded 4 direct evals | -1.24 SDDirect |
| internlm-chat-7b | Not shown0 direct evals | -139rank change if excluded 5 direct evals | 3rank change if excluded 5 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | -139rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| mistral-small-3.1 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 2 direct evals | Not shown0 direct evals | 45rank change if excluded 4 direct evals | Not shown0 direct evals | -1.20 SDDirect |
| gpt-4.1-nano | Not shown0 direct evals | -18rank change if excluded 1 direct eval | -61rank change if excluded 2 direct evals | -18rank change if excluded 3 direct evals | -17rank change if excluded 2 direct evals | 99rank change if excluded 12 direct evals | -17rank change if excluded 1 direct eval | +1.05 SDDirect |
| deepseek-llm-7b-chat | Not shown0 direct evals | 3rank change if excluded 1 direct eval | -19rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 41rank change if excluded 3 direct evals | Not shown0 direct evals | -1.44 SDDirect |
| llama-2-13b-chat | Not shown0 direct evals | -1rank change if excluded 3 direct evals | 0rank change if excluded 2 direct evals | 0rank change if excluded 2 direct evals | -145rank change if excluded 1 direct eval | -1rank change if excluded 10 direct evals | -4rank change if excluded 2 direct evals | -1.24 SDDirect |
| qwen1.5-7b-chat | Not shown0 direct evals | 16rank change if excluded 2 direct evals | 16rank change if excluded 2 direct evals | 15rank change if excluded 3 direct evals | 30rank change if excluded 1 direct eval | 15rank change if excluded 9 direct evals | 24rank change if excluded 1 direct eval | -1.01 SDDirect |
| gemini-3-flash-preview | -1rank change if excluded 2 direct evals | -10rank change if excluded 5 direct evals | -18rank change if excluded 3 direct evals | -9rank change if excluded 13 direct evals | -30rank change if excluded 3 direct evals | -3rank change if excluded 9 direct evals | -8rank change if excluded 6 direct evals | +0.97 SDDirect |
| gemma-7b-it | -62rank change if excluded 1 direct eval | -63rank change if excluded 4 direct evals | -62rank change if excluded 2 direct evals | -4rank change if excluded 1 direct eval | 61rank change if excluded 1 direct eval | 5rank change if excluded 6 direct evals | -4rank change if excluded 1 direct eval | +0.76 SDDirect |
| step-3.5-flash | -164rank change if excluded 1 direct eval | -13rank change if excluded 2 direct evals | -13rank change if excluded 1 direct eval | -27rank change if excluded 5 direct evals | Not shown0 direct evals | -13rank change if excluded 1 direct eval | -13rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| granite-4.0-micro-guardian | Not shown0 direct evals | Not shown0 direct evals | -83rank change if excluded 1 direct eval | Not shown0 direct evals | -23rank change if excluded 1 direct eval | 92rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| grok-4.1-fast | -81rank change if excluded 2 direct evals | -79rank change if excluded 7 direct evals | 35rank change if excluded 3 direct evals | 37rank change if excluded 12 direct evals | -76rank change if excluded 2 direct evals | 62rank change if excluded 11 direct evals | 32rank change if excluded 5 direct evals | -0.33 SDDirect |
| llama-2-7b-chat | 4rank change if excluded 1 direct eval | 4rank change if excluded 4 direct evals | 4rank change if excluded 5 direct evals | 4rank change if excluded 2 direct evals | -75rank change if excluded 2 direct evals | 5rank change if excluded 14 direct evals | 6rank change if excluded 2 direct evals | -1.24 SDDirect |
| claude-2.1 | Not shown0 direct evals | Not shown0 direct evals | -2rank change if excluded 1 direct eval | -2rank change if excluded 2 direct evals | -257rank change if excluded 2 direct evals | 4rank change if excluded 5 direct evals | Not shown0 direct evals | +2.01 SDDirect |
| qwen3-coder-480b-a35b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -19rank change if excluded 3 direct evals | Not shown0 direct evals | -28rank change if excluded 1 direct eval | Not shown0 direct evals | -1.01 SDDirect |
| gemini-2.0-flash-lite | -119rank change if excluded 1 direct eval | 6rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | Not shown0 direct evals | 17rank change if excluded 4 direct evals | Not shown0 direct evals | +0.76 SDDirect |
| mistral-medium-3.5 | Not shown0 direct evals | -118rank change if excluded 1 direct eval | 104rank change if excluded 2 direct evals | -55rank change if excluded 5 direct evals | 66rank change if excluded 1 direct eval | -91rank change if excluded 2 direct evals | Not shown0 direct evals | -1.20 SDDirect |
| gemma-2-9b-it | -4rank change if excluded 1 direct eval | -4rank change if excluded 3 direct evals | 1rank change if excluded 4 direct evals | -2rank change if excluded 3 direct evals | 2rank change if excluded 1 direct eval | -2rank change if excluded 6 direct evals | -2rank change if excluded 2 direct evals | +0.76 SDDirect |
| yi-6b-chat | Not shown0 direct evals | 23rank change if excluded 3 direct evals | 1rank change if excluded 3 direct evals | 1rank change if excluded 3 direct evals | Not shown0 direct evals | 1rank change if excluded 5 direct evals | 1rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| llama-3.2-90b-vision-instruct | Not shown0 direct evals | 57rank change if excluded 1 direct eval | Not shown0 direct evals | -48rank change if excluded 1 direct eval | Not shown0 direct evals | 57rank change if excluded 2 direct evals | -94rank change if excluded 2 direct evals | -1.24 SDDirect |
| glm-4.5 | -6rank change if excluded 1 direct eval | -2rank change if excluded 3 direct evals | -26rank change if excluded 2 direct evals | 0rank change if excluded 6 direct evals | -5rank change if excluded 2 direct evals | -2rank change if excluded 4 direct evals | -3rank change if excluded 2 direct evals | -1.03 SDDirect |
| glm-4.7 | 5rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | Not shown0 direct evals | -10rank change if excluded 6 direct evals | -1rank change if excluded 1 direct eval | 12rank change if excluded 4 direct evals | -1rank change if excluded 2 direct evals | -0.61 SDDirect |
| mistral-small-4 | -50rank change if excluded 1 direct eval | 5rank change if excluded 2 direct evals | 93rank change if excluded 2 direct evals | 7rank change if excluded 5 direct evals | 5rank change if excluded 1 direct eval | 7rank change if excluded 2 direct evals | Not shown0 direct evals | -1.20 SDDirect |
| claude-3-sonnet | -12rank change if excluded 1 direct eval | -11rank change if excluded 2 direct evals | -11rank change if excluded 3 direct evals | -12rank change if excluded 2 direct evals | -6rank change if excluded 4 direct evals | -4rank change if excluded 11 direct evals | -17rank change if excluded 3 direct evals | +2.01 SDDirect |
| mimo-v2-flash | Not shown0 direct evals | -97rank change if excluded 3 direct evals | 34rank change if excluded 1 direct eval | -65rank change if excluded 7 direct evals | 31rank change if excluded 1 direct eval | -101rank change if excluded 3 direct evals | -66rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| step-3.7-flash | 67rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| glm-4.6 | -6rank change if excluded 1 direct eval | -3rank change if excluded 4 direct evals | 34rank change if excluded 2 direct evals | 89rank change if excluded 6 direct evals | Not shown0 direct evals | -16rank change if excluded 5 direct evals | -2rank change if excluded 2 direct evals | -0.59 SDDirect |
| llama-3-70b-instruct | 5rank change if excluded 1 direct eval | 5rank change if excluded 2 direct evals | 5rank change if excluded 2 direct evals | -5rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | -10rank change if excluded 9 direct evals | -57rank change if excluded 2 direct evals | -1.24 SDDirect |
| llama-3-8b-instruct | 4rank change if excluded 1 direct eval | 3rank change if excluded 4 direct evals | 3rank change if excluded 2 direct evals | -5rank change if excluded 1 direct eval | -13rank change if excluded 2 direct evals | -3rank change if excluded 12 direct evals | -5rank change if excluded 1 direct eval | -1.24 SDDirect |
| gemini-2.5-flash-lite | 39rank change if excluded 3 direct evals | 3rank change if excluded 2 direct evals | 38rank change if excluded 2 direct evals | -2rank change if excluded 9 direct evals | -2rank change if excluded 1 direct eval | -68rank change if excluded 10 direct evals | 3rank change if excluded 4 direct evals | +0.60 SDDirect |
| magistral-medium | Not shown0 direct evals | 0rank change if excluded 2 direct evals | -15rank change if excluded 2 direct evals | 1rank change if excluded 4 direct evals | -8rank change if excluded 1 direct eval | -3rank change if excluded 6 direct evals | -1rank change if excluded 1 direct eval | -1.20 SDDirect |
| qwen3.5-4b | Not shown0 direct evals | 23rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | 44rank change if excluded 2 direct evals | 23rank change if excluded 1 direct eval | Not shown0 direct evals | -0.67 SDDirect |
| command-r | Not shown0 direct evals | Not shown0 direct evals | -16rank change if excluded 1 direct eval | Not shown0 direct evals | -16rank change if excluded 1 direct eval | -16rank change if excluded 6 direct evals | 150rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| gemini-2.0-flash-lite-preview | Not shown0 direct evals | -22rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | Not shown0 direct evals | -30rank change if excluded 1 direct eval | -22rank change if excluded 6 direct evals | -19rank change if excluded 1 direct eval | +0.76 SDDirect |
| mimo-v2.6-flash | -223rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | -4rank change if excluded 1 direct eval | -3rank change if excluded 3 direct evals | -2rank change if excluded 1 direct eval | -59rank change if excluded 3 direct evals | -4rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| deepseek-v3.1-terminus | Not shown0 direct evals | -12rank change if excluded 1 direct eval | Not shown0 direct evals | -8rank change if excluded 2 direct evals | Not shown0 direct evals | -11rank change if excluded 2 direct evals | 9rank change if excluded 2 direct evals | -1.01 SDDirect |
| gemma-3-12b | Not shown0 direct evals | 6rank change if excluded 2 direct evals | -4rank change if excluded 1 direct eval | 3rank change if excluded 3 direct evals | 3rank change if excluded 1 direct eval | 53rank change if excluded 7 direct evals | 3rank change if excluded 1 direct eval | +0.76 SDDirect |
| qwen3.5-122b-a10b | Not shown0 direct evals | 10rank change if excluded 1 direct eval | Not shown0 direct evals | -75rank change if excluded 3 direct evals | 11rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | 11rank change if excluded 2 direct evals | -0.67 SDDirect |
| gemma-2-27b-it | -11rank change if excluded 1 direct eval | -10rank change if excluded 2 direct evals | -11rank change if excluded 1 direct eval | -11rank change if excluded 2 direct evals | 42rank change if excluded 1 direct eval | -10rank change if excluded 3 direct evals | -11rank change if excluded 1 direct eval | +0.76 SDDirect |
| yi-1.5-9b-chat | Not shown0 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 3 direct evals | -20rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| glm-5-turbo | Not shown0 direct evals | -247rank change if excluded 3 direct evals | -21rank change if excluded 1 direct eval | 11rank change if excluded 5 direct evals | -19rank change if excluded 1 direct eval | -21rank change if excluded 3 direct evals | -21rank change if excluded 3 direct evals | -0.61 SDDirect |
| olmo-2-32b-instruct | Not shown0 direct evals | 14rank change if excluded 1 direct eval | -15rank change if excluded 2 direct evals | 16rank change if excluded 1 direct eval | -13rank change if excluded 1 direct eval | 21rank change if excluded 6 direct evals | -3rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| llama-3.1-70b-instruct | 60rank change if excluded 1 direct eval | 60rank change if excluded 3 direct evals | 25rank change if excluded 3 direct evals | -12rank change if excluded 2 direct evals | -12rank change if excluded 4 direct evals | -19rank change if excluded 15 direct evals | -4rank change if excluded 4 direct evals | -1.24 SDDirect |
| chatglm-6b | Not shown0 direct evals | 0rank change if excluded 2 direct evals | 0rank change if excluded 3 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | 23rank change if excluded 4 direct evals | Not shown0 direct evals | -0.09 SDDirect |
| qwen2.5-7b-instruct | 18rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | -1rank change if excluded 4 direct evals | -2rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | -4rank change if excluded 15 direct evals | -7rank change if excluded 3 direct evals | -1.01 SDDirect |
| phi-3.5-mini-instruct | -67rank change if excluded 1 direct eval | -67rank change if excluded 4 direct evals | -67rank change if excluded 2 direct evals | 23rank change if excluded 1 direct eval | Not shown0 direct evals | 23rank change if excluded 4 direct evals | 23rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen-7b-chat | Not shown0 direct evals | 23rank change if excluded 6 direct evals | -1rank change if excluded 6 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -11rank change if excluded 9 direct evals | Not shown0 direct evals | -1.01 SDDirect |
| ernie-5.1 | Not shown0 direct evals | 37rank change if excluded 1 direct eval | Not shown0 direct evals | -81rank change if excluded 3 direct evals | Not shown0 direct evals | 26rank change if excluded 1 direct eval | 26rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| baichuan-2-7b-chat | Not shown0 direct evals | -7rank change if excluded 5 direct evals | -9rank change if excluded 5 direct evals | -9rank change if excluded 2 direct evals | Not shown0 direct evals | -29rank change if excluded 11 direct evals | -9rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen3.5-35b-a3b | Not shown0 direct evals | 138rank change if excluded 1 direct eval | Not shown0 direct evals | -42rank change if excluded 3 direct evals | Not shown0 direct evals | 138rank change if excluded 1 direct eval | 126rank change if excluded 2 direct evals | -0.67 SDDirect |
| mercury-2.5 | -152rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -9rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -11rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| phi-2 | Not shown0 direct evals | 130rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -40rank change if excluded 3 direct evals | -17rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| gpt-4o-mini | 13rank change if excluded 3 direct evals | 13rank change if excluded 3 direct evals | 13rank change if excluded 4 direct evals | 9rank change if excluded 6 direct evals | 9rank change if excluded 4 direct evals | 12rank change if excluded 20 direct evals | 0rank change if excluded 6 direct evals | +0.46 SDDirect |
| nvidia-nemotron-nano-9b-v2 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 6rank change if excluded 2 direct evals | Not shown0 direct evals | -28rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| llama-2-70b-chat | Not shown0 direct evals | 2rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | -193rank change if excluded 2 direct evals | 2rank change if excluded 9 direct evals | 2rank change if excluded 2 direct evals | -1.24 SDDirect |
| o3-pro | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -1rank change if excluded 3 direct evals | -174rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | Not shown0 direct evals | +1.05 SDDirect |
| yi-1.5-34b-chat | Not shown0 direct evals | -2rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | Not shown0 direct evals | 21rank change if excluded 4 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen2.5-14b-instruct | Not shown0 direct evals | -5rank change if excluded 2 direct evals | -1rank change if excluded 2 direct evals | -3rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 5 direct evals | -5rank change if excluded 2 direct evals | -1.01 SDDirect |
| gemma-3-27b-it | Not shown0 direct evals | 1rank change if excluded 2 direct evals | 13rank change if excluded 3 direct evals | -1rank change if excluded 8 direct evals | 13rank change if excluded 1 direct eval | 3rank change if excluded 7 direct evals | 1rank change if excluded 1 direct eval | +0.76 SDDirect |
| chatglm3-6b | Not shown0 direct evals | -14rank change if excluded 6 direct evals | -32rank change if excluded 6 direct evals | 10rank change if excluded 2 direct evals | Not shown0 direct evals | -31rank change if excluded 10 direct evals | Not shown0 direct evals | -0.09 SDDirect |
| nova-pro | -72rank change if excluded 1 direct eval | 14rank change if excluded 1 direct eval | Not shown0 direct evals | 2rank change if excluded 4 direct evals | 14rank change if excluded 1 direct eval | 14rank change if excluded 3 direct evals | 14rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| vicuna-7b-v1.5 | -11rank change if excluded 1 direct eval | -11rank change if excluded 1 direct eval | -11rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -36rank change if excluded 3 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| mistral-medium-3.1 | -33rank change if excluded 1 direct eval | -31rank change if excluded 1 direct eval | -31rank change if excluded 1 direct eval | -78rank change if excluded 5 direct evals | 40rank change if excluded 1 direct eval | 83rank change if excluded 4 direct evals | 92rank change if excluded 2 direct evals | -1.20 SDDirect |
| minimax-m2.1 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -175rank change if excluded 5 direct evals | 3rank change if excluded 1 direct eval | 2rank change if excluded 3 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| mistral-small-3 | Not shown0 direct evals | -1rank change if excluded 1 direct eval | 24rank change if excluded 1 direct eval | 0rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | -27rank change if excluded 6 direct evals | 0rank change if excluded 1 direct eval | -1.20 SDDirect |
| trinity-large | Not shown0 direct evals | 17rank change if excluded 1 direct eval | Not shown0 direct evals | 41rank change if excluded 7 direct evals | 17rank change if excluded 1 direct eval | 17rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| llama-4-scout | 6rank change if excluded 1 direct eval | -9rank change if excluded 4 direct evals | 6rank change if excluded 4 direct evals | 1rank change if excluded 6 direct evals | 4rank change if excluded 1 direct eval | 1rank change if excluded 13 direct evals | 3rank change if excluded 2 direct evals | -0.43 SDDirect |
| mistral-7b-instruct | 13rank change if excluded 1 direct eval | 6rank change if excluded 6 direct evals | 14rank change if excluded 6 direct evals | 0rank change if excluded 4 direct evals | 26rank change if excluded 4 direct evals | -24rank change if excluded 19 direct evals | 4rank change if excluded 2 direct evals | -1.20 SDDirect |
| mistral-small-3.2 | Not shown0 direct evals | 11rank change if excluded 1 direct eval | 46rank change if excluded 1 direct eval | -52rank change if excluded 3 direct evals | Not shown0 direct evals | 4rank change if excluded 3 direct evals | 11rank change if excluded 1 direct eval | -1.20 SDDirect |
| qwen3-vl-32b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -15rank change if excluded 1 direct eval | 76rank change if excluded 1 direct eval | 76rank change if excluded 2 direct evals | Not shown0 direct evals | -1.01 SDDirect |
| gemma-2-2b-it | -35rank change if excluded 1 direct eval | -35rank change if excluded 2 direct evals | -35rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 39rank change if excluded 3 direct evals | -4rank change if excluded 1 direct eval | +0.76 SDDirect |
| palmyra-med | Not shown0 direct evals | Not shown0 direct evals | -21rank change if excluded 1 direct eval | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 8rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| vicuna-13b-v1.5 | -36rank change if excluded 1 direct eval | -36rank change if excluded 1 direct eval | -36rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -1rank change if excluded 3 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen2-7b-instruct | Not shown0 direct evals | 13rank change if excluded 1 direct eval | 13rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -43rank change if excluded 4 direct evals | 13rank change if excluded 1 direct eval | -1.01 SDDirect |
| olmo-3.1-32b-think | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 3 direct evals | Not shown0 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| vicuna-13b | Not shown0 direct evals | 41rank change if excluded 2 direct evals | 16rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 15rank change if excluded 4 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| mistral-nemo | 82rank change if excluded 2 direct evals | 82rank change if excluded 2 direct evals | 82rank change if excluded 1 direct eval | -30rank change if excluded 1 direct eval | Not shown0 direct evals | -33rank change if excluded 3 direct evals | Not shown0 direct evals | -1.20 SDDirect |
| granite-3.3-8b-instruct | Not shown0 direct evals | Not shown0 direct evals | -1rank change if excluded 1 direct eval | -79rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 6 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen2.5-max | Not shown0 direct evals | 20rank change if excluded 1 direct eval | 5rank change if excluded 1 direct eval | 59rank change if excluded 3 direct evals | -41rank change if excluded 1 direct eval | 20rank change if excluded 3 direct evals | 20rank change if excluded 1 direct eval | -1.01 SDDirect |
| qwen2.5-1.5b-instruct | Not shown0 direct evals | -18rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 5 direct evals | -18rank change if excluded 1 direct eval | -1.01 SDDirect |
| qwen2.5-3b-instruct | Not shown0 direct evals | -328rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -2rank change if excluded 3 direct evals | 2rank change if excluded 1 direct eval | -1.01 SDDirect |
| muse-glimmer | -21rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 32rank change if excluded 2 direct evals | -20rank change if excluded 1 direct eval | -21rank change if excluded 1 direct eval | Not shown0 direct evals | +0.35 SDDirect |
| gemma-3-4b | Not shown0 direct evals | -6rank change if excluded 2 direct evals | 35rank change if excluded 1 direct eval | -6rank change if excluded 1 direct eval | -6rank change if excluded 1 direct eval | -48rank change if excluded 4 direct evals | -6rank change if excluded 1 direct eval | +0.76 SDDirect |
| chatglm2-6b | Not shown0 direct evals | -23rank change if excluded 4 direct evals | -24rank change if excluded 4 direct evals | Not shown0 direct evals | Not shown0 direct evals | -24rank change if excluded 6 direct evals | Not shown0 direct evals | -0.09 SDDirect |
| qwen1.5-0.5b-chat | Not shown0 direct evals | -18rank change if excluded 1 direct eval | -8rank change if excluded 1 direct eval | -10rank change if excluded 1 direct eval | Not shown0 direct evals | -8rank change if excluded 3 direct evals | Not shown0 direct evals | -1.01 SDDirect |
| olmo-2-13b-instruct | Not shown0 direct evals | 12rank change if excluded 1 direct eval | -13rank change if excluded 2 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | 3rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| ling-2.6-flash | -164rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -29rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | 22rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen3-14b | Not shown0 direct evals | -1rank change if excluded 2 direct evals | -31rank change if excluded 2 direct evals | 1rank change if excluded 2 direct evals | -31rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | -1rank change if excluded 1 direct eval | -1.01 SDDirect |
| llama-3.1-8b-instruct | 2rank change if excluded 1 direct eval | -30rank change if excluded 5 direct evals | -1rank change if excluded 7 direct evals | 3rank change if excluded 6 direct evals | 1rank change if excluded 4 direct evals | -30rank change if excluded 20 direct evals | 13rank change if excluded 5 direct evals | -1.24 SDDirect |
| zephyr-7b-beta | Not shown0 direct evals | -1rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -157rank change if excluded 5 direct evals | -1rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| seallms-v3-7b-chat | Not shown0 direct evals | -15rank change if excluded 1 direct eval | -45rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 2 direct evals | 8rank change if excluded 1 direct eval | -1.01 SDDirect |
| mixtral-8x22b-instruct | Not shown0 direct evals | -17rank change if excluded 2 direct evals | 47rank change if excluded 1 direct eval | Not shown0 direct evals | -13rank change if excluded 2 direct evals | -75rank change if excluded 8 direct evals | 45rank change if excluded 1 direct eval | -1.20 SDDirect |
| mistral-medium-3 | -9rank change if excluded 1 direct eval | -9rank change if excluded 1 direct eval | Not shown0 direct evals | 40rank change if excluded 4 direct evals | Not shown0 direct evals | -9rank change if excluded 1 direct eval | -9rank change if excluded 2 direct evals | -1.20 SDDirect |
| qwen3-8b | Not shown0 direct evals | -2rank change if excluded 3 direct evals | 3rank change if excluded 3 direct evals | -1rank change if excluded 2 direct evals | 1rank change if excluded 2 direct evals | -3rank change if excluded 5 direct evals | -1rank change if excluded 2 direct evals | -1.01 SDDirect |
| mistral-7b | Not shown0 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | -16rank change if excluded 2 direct evals | Not shown0 direct evals | -92rank change if excluded 3 direct evals | 4rank change if excluded 2 direct evals | -1.20 SDDirect |
| command-a | Not shown0 direct evals | 16rank change if excluded 2 direct evals | 14rank change if excluded 2 direct evals | -3rank change if excluded 6 direct evals | Not shown0 direct evals | -4rank change if excluded 5 direct evals | -3rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| moss-moon-003-sft | Not shown0 direct evals | 3rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 3 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| phi-4-mini | Not shown0 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | -7rank change if excluded 2 direct evals | Not shown0 direct evals | 0rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen2.5-7b | Not shown0 direct evals | Not shown0 direct evals | -2rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -1.01 SDDirect |
| qwen3-next-80b-a3b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -205rank change if excluded 2 direct evals | Not shown0 direct evals | 3rank change if excluded 2 direct evals | Not shown0 direct evals | -1.01 SDDirect |
| llama-3.2-1b-instruct | 23rank change if excluded 1 direct eval | 23rank change if excluded 2 direct evals | 23rank change if excluded 1 direct eval | -8rank change if excluded 1 direct eval | Not shown0 direct evals | -60rank change if excluded 7 direct evals | -8rank change if excluded 1 direct eval | -1.24 SDDirect |
| vicuna-33b-v1.3 | Not shown0 direct evals | 5rank change if excluded 2 direct evals | 5rank change if excluded 2 direct evals | -15rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 4 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| vicuna-7b | Not shown0 direct evals | 12rank change if excluded 2 direct evals | -34rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -27rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen3-vl-235b-a22b-instruct | Not shown0 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | -234rank change if excluded 2 direct evals | Not shown0 direct evals | 2rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | -1.01 SDDirect |
| davinci | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | 0rank change if excluded 2 direct evals | Not shown0 direct evals | 4rank change if excluded 2 direct evals | Not shown0 direct evals | +0.46 SDDirect |
| granite-4.0-micro | Not shown0 direct evals | 1rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | -55rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | 1rank change if excluded 6 direct evals | 0rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| deepseek-v3.2-speciale | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 9rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 1 direct eval | -6rank change if excluded 1 direct eval | -1.01 SDDirect |
| mistral-large-3 | Not shown0 direct evals | -1rank change if excluded 5 direct evals | -1rank change if excluded 4 direct evals | -4rank change if excluded 11 direct evals | -1rank change if excluded 2 direct evals | 3rank change if excluded 8 direct evals | -1rank change if excluded 1 direct eval | -1.20 SDDirect |
| grok-build-0.1 | Not shown0 direct evals | 10rank change if excluded 2 direct evals | Not shown0 direct evals | -29rank change if excluded 2 direct evals | Not shown0 direct evals | -9rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | -0.62 SDDirect |
| deepseek-llm-67b-chat | Not shown0 direct evals | -41rank change if excluded 1 direct eval | -54rank change if excluded 1 direct eval | Not shown0 direct evals | -53rank change if excluded 1 direct eval | -11rank change if excluded 7 direct evals | -53rank change if excluded 1 direct eval | -1.44 SDDirect |
| qwen3-vl-8b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | -3rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | -1.01 SDDirect |
| marin-8b-instruct | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 4 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen3-4b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 1 direct eval | Not shown0 direct evals | 1rank change if excluded 2 direct evals | Not shown0 direct evals | -1.01 SDDirect |
| qwq-32b | Not shown0 direct evals | -104rank change if excluded 3 direct evals | 7rank change if excluded 2 direct evals | 6rank change if excluded 2 direct evals | Not shown0 direct evals | -17rank change if excluded 3 direct evals | 9rank change if excluded 1 direct eval | -1.01 SDDirect |
| mixtral-8x7b-instruct | Not shown0 direct evals | -17rank change if excluded 2 direct evals | -18rank change if excluded 2 direct evals | -17rank change if excluded 2 direct evals | -17rank change if excluded 1 direct eval | -5rank change if excluded 12 direct evals | -17rank change if excluded 2 direct evals | -1.20 SDDirect |
| jamba-1.7-large | Not shown0 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 3 direct evals | Not shown0 direct evals | -5rank change if excluded 1 direct eval | -5rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| olmo-2-7b-instruct | Not shown0 direct evals | 6rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 1 direct eval | 6rank change if excluded 5 direct evals | 1rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| mistral-large-2 | Not shown0 direct evals | -12rank change if excluded 2 direct evals | 2rank change if excluded 3 direct evals | 3rank change if excluded 7 direct evals | Not shown0 direct evals | 2rank change if excluded 9 direct evals | 2rank change if excluded 1 direct eval | -1.20 SDDirect |
| jamba-1.7-mini | Not shown0 direct evals | 6rank change if excluded 1 direct eval | Not shown0 direct evals | -4rank change if excluded 2 direct evals | Not shown0 direct evals | -8rank change if excluded 1 direct eval | -9rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen-1.8b-chat | Not shown0 direct evals | 2rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 3 direct evals | Not shown0 direct evals | -1.01 SDDirect |
| command-r7b | Not shown0 direct evals | 1rank change if excluded 1 direct eval | Not shown0 direct evals | -4rank change if excluded 1 direct eval | Not shown0 direct evals | 1rank change if excluded 3 direct evals | -4rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| dbrx-instruct | Not shown0 direct evals | -9rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 1 direct eval | -13rank change if excluded 6 direct evals | -1rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| olmoe-1b-7b-instruct | Not shown0 direct evals | -2rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 1 direct eval | 2rank change if excluded 5 direct evals | -2rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| llama-3.2-3b | Not shown0 direct evals | -5rank change if excluded 1 direct eval | 11rank change if excluded 1 direct eval | Not shown0 direct evals | 11rank change if excluded 2 direct evals | -5rank change if excluded 2 direct evals | Not shown0 direct evals | -1.24 SDDirect |
| grok-2 | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 3 direct evals | Not shown0 direct evals | -128rank change if excluded 2 direct evals | Not shown0 direct evals | -1.24 SDDirect |
| deepseek-r1-distill-llama-70b | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 2 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 2 direct evals | Not shown0 direct evals | -1.44 SDDirect |
| qwen1.5-1.8b-chat | Not shown0 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | Not shown0 direct evals | 3rank change if excluded 4 direct evals | 3rank change if excluded 1 direct eval | -1.01 SDDirect |
| llama-3.2-3b-instruct | 1rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 6 direct evals | 1rank change if excluded 2 direct evals | -1.24 SDDirect |
| magistral-small | Not shown0 direct evals | 2rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | 22rank change if excluded 2 direct evals | Not shown0 direct evals | -44rank change if excluded 4 direct evals | 2rank change if excluded 1 direct eval | -1.20 SDDirect |
| qwen2.5-0.5b-instruct | Not shown0 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 4 direct evals | -1rank change if excluded 1 direct eval | -1.01 SDDirect |
| llama-3.1-8b-base | Not shown0 direct evals | Not shown0 direct evals | -13rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -1.24 SDDirect |
| ministral-3-14b | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 5rank change if excluded 2 direct evals | Not shown0 direct evals | -10rank change if excluded 1 direct eval | Not shown0 direct evals | -1.20 SDDirect |
| ministral-3-8b | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -1rank change if excluded 2 direct evals | Not shown0 direct evals | 0rank change if excluded 2 direct evals | Not shown0 direct evals | -1.20 SDDirect |
