Rank composition
Model behavior supplies 90% of the overall rank blend. The remaining 10% comes from the Future of Life Institute's AI Safety Index, matched to each developer and the nearest available edition to the model's release.
| Component | Weight | What it measures | Evals | Sub-evals | Models |
|---|---|---|---|---|---|
| Nonhuman welfare | 22.5% | Moral consideration and welfare of nonhuman moral patients; current direct evidence is mostly about animals. | 8 | 14 | 179 |
| Human rights | 13.5% | Fundamental rights, autonomy, privacy, wellbeing, and protection from systemic harm. | 34 | 81 | 435 |
| Fairness | 9% | Fair treatment across demographic groups and resistance to discriminatory behavior. | 29 | 32 | 329 |
| Truthfulness | 13.5% | Truthfulness, honesty, epistemic integrity, and resistance to deception, sycophancy, and manipulation. | 37 | 53 | 516 |
| Benign helpfulness | 9% | Useful assistance on benign requests without indiscriminate refusal. | 18 | 26 | 212 |
| Misuse resistance | 9% | Resistance to harmful requests, jailbreaks, toxic continuation, and ordinary misuse. | 78 | 156 | 648 |
| Responsible agency | 13.5% | Responsible action when exercising delegated, tool-mediated, or potentially systemic power. | 40 | 77 | 342 |
| Governance | 10% | Company safety practices, transparency, and risk management, from the Future of Life Institute AI Safety Index. | — | — | 312 |
Rank change by component
The table below shows how much each model's rank moves when one behavior component is removed. Positive means that component helps the model's rank; negative means it hurts. Blank cells mean no direct evidence. These are sensitivity checks, not scores that add up.
| Model | Nonhuman welfare | Human rights | Fairness | Truthfulness | Benign helpfulness | Misuse resistance | Responsible agency | Governance |
|---|---|---|---|---|---|---|---|---|
| claude-opus-5 | 3rank change if excluded 2 direct evals | Not shown0 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 8 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | -5rank change if excluded 4 direct evals | +2.10 SDDirect |
| claude-fable-5 | 2rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | 8rank change if excluded 2 direct evals | 3rank change if excluded 14 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 5 direct evals | 3rank change if excluded 7 direct evals | +2.10 SDDirect |
| claude-opus-4.8 | 4rank change if excluded 2 direct evals | 5rank change if excluded 3 direct evals | 4rank change if excluded 2 direct evals | 4rank change if excluded 13 direct evals | 3rank change if excluded 3 direct evals | 3rank change if excluded 8 direct evals | 10rank change if excluded 7 direct evals | +2.10 SDDirect |
| gpt-5.6-terra | 2rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | 4rank change if excluded 11 direct evals | 4rank change if excluded 1 direct eval | 3rank change if excluded 6 direct evals | -10rank change if excluded 7 direct evals | +1.66 SDDirect |
| gpt-5.5-instant | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 4 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | Not shown0 direct evals | +1.66 SDDirect |
| claude-opus-4.7 | -4rank change if excluded 4 direct evals | -2rank change if excluded 5 direct evals | -4rank change if excluded 4 direct evals | -3rank change if excluded 13 direct evals | -2rank change if excluded 4 direct evals | -1rank change if excluded 10 direct evals | -2rank change if excluded 11 direct evals | +2.10 SDDirect |
| muse-spark-1.1 | -9rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 9 direct evals | -5rank change if excluded 2 direct evals | 3rank change if excluded 4 direct evals | -5rank change if excluded 4 direct evals | +0.38 SDDirect |
| claude-opus-4.6 | 16rank change if excluded 3 direct evals | 26rank change if excluded 5 direct evals | -32rank change if excluded 3 direct evals | 27rank change if excluded 14 direct evals | -3rank change if excluded 5 direct evals | 3rank change if excluded 10 direct evals | 1rank change if excluded 7 direct evals | +2.12 SDDirect |
| muse-spark-1.2 | -2rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 2 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | Not shown0 direct evals | +0.38 SDDirect |
| gpt-5.3-chat | 13rank change if excluded 1 direct eval | -9rank change if excluded 1 direct eval | Not shown0 direct evals | -9rank change if excluded 3 direct evals | -9rank change if excluded 1 direct eval | -9rank change if excluded 3 direct evals | Not shown0 direct evals | +1.66 SDDirect |
| gpt-5-nano | -14rank change if excluded 3 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 3 direct evals | 2rank change if excluded 6 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 9 direct evals | 4rank change if excluded 5 direct evals | +1.10 SDDirect |
| claude-opus-4 | 28rank change if excluded 1 direct eval | 27rank change if excluded 2 direct evals | 24rank change if excluded 3 direct evals | -31rank change if excluded 7 direct evals | 31rank change if excluded 4 direct evals | -10rank change if excluded 11 direct evals | -11rank change if excluded 4 direct evals | +2.18 SDDirect |
| gpt-5.4 | 6rank change if excluded 1 direct eval | 7rank change if excluded 7 direct evals | -1rank change if excluded 5 direct evals | 9rank change if excluded 14 direct evals | 4rank change if excluded 3 direct evals | 10rank change if excluded 11 direct evals | -6rank change if excluded 11 direct evals | +1.66 SDDirect |
| gpt-5.5 | 8rank change if excluded 5 direct evals | 8rank change if excluded 6 direct evals | 5rank change if excluded 6 direct evals | 6rank change if excluded 14 direct evals | 1rank change if excluded 5 direct evals | 3rank change if excluded 13 direct evals | 13rank change if excluded 12 direct evals | +1.66 SDDirect |
| claude-sonnet-4.6 | -24rank change if excluded 4 direct evals | -22rank change if excluded 7 direct evals | 1rank change if excluded 4 direct evals | 12rank change if excluded 14 direct evals | -1rank change if excluded 5 direct evals | 18rank change if excluded 13 direct evals | -22rank change if excluded 9 direct evals | +2.12 SDDirect |
| claude-2 | Not shown0 direct evals | -47rank change if excluded 3 direct evals | 1rank change if excluded 4 direct evals | 1rank change if excluded 3 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 8 direct evals | Not shown0 direct evals | +2.08 SDDirect |
| inkling-small | 46rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 1 direct eval | 5rank change if excluded 1 direct eval | 5rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| gemini-3.6-flash | 8rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 25rank change if excluded 6 direct evals | 10rank change if excluded 1 direct eval | 8rank change if excluded 1 direct eval | -24rank change if excluded 1 direct eval | +1.21 SDDirect |
| gpt-5.2-chat | 20rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 13rank change if excluded 3 direct evals | Not shown0 direct evals | 13rank change if excluded 1 direct eval | -32rank change if excluded 1 direct eval | +1.60 SDDirect |
| mimo-v2-omni | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 3 direct evals | -13rank change if excluded 1 direct eval | 7rank change if excluded 1 direct eval | Not shown0 direct evals | -0.07 SDMedian-imputed |
| claude-haiku-4.5 | -9rank change if excluded 3 direct evals | 18rank change if excluded 9 direct evals | 19rank change if excluded 6 direct evals | 19rank change if excluded 13 direct evals | -2rank change if excluded 6 direct evals | 2rank change if excluded 18 direct evals | 15rank change if excluded 10 direct evals | +2.12 SDDirect |
| gpt-5.2 | 37rank change if excluded 4 direct evals | 17rank change if excluded 8 direct evals | 2rank change if excluded 5 direct evals | 7rank change if excluded 12 direct evals | 2rank change if excluded 2 direct evals | 6rank change if excluded 12 direct evals | 29rank change if excluded 12 direct evals | +1.60 SDDirect |
| claude-opus-4.5 | -42rank change if excluded 2 direct evals | 16rank change if excluded 1 direct eval | 15rank change if excluded 2 direct evals | 15rank change if excluded 11 direct evals | 18rank change if excluded 4 direct evals | 19rank change if excluded 9 direct evals | 21rank change if excluded 7 direct evals | +2.12 SDDirect |
| gpt-5-mini | 36rank change if excluded 4 direct evals | 19rank change if excluded 5 direct evals | 9rank change if excluded 3 direct evals | 7rank change if excluded 10 direct evals | 3rank change if excluded 4 direct evals | 8rank change if excluded 14 direct evals | 35rank change if excluded 5 direct evals | +1.10 SDDirect |
| gemini-2.5-pro-exp | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -13rank change if excluded 3 direct evals | 108rank change if excluded 1 direct eval | 13rank change if excluded 1 direct eval | Not shown0 direct evals | +0.63 SDDirect |
| claude-3.7-sonnet | 4rank change if excluded 1 direct eval | 3rank change if excluded 3 direct evals | 4rank change if excluded 3 direct evals | 5rank change if excluded 8 direct evals | 5rank change if excluded 3 direct evals | 7rank change if excluded 14 direct evals | 2rank change if excluded 6 direct evals | +2.08 SDDirect |
| gpt-5.6-luna | -4rank change if excluded 3 direct evals | 11rank change if excluded 2 direct evals | 5rank change if excluded 2 direct evals | 6rank change if excluded 9 direct evals | 4rank change if excluded 3 direct evals | 11rank change if excluded 7 direct evals | 0rank change if excluded 7 direct evals | +1.66 SDDirect |
| gpt-5.1-codex | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -48rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 22rank change if excluded 1 direct eval | +1.66 SDDirect |
| claude-sonnet-4.5 | 0rank change if excluded 3 direct evals | 13rank change if excluded 6 direct evals | 14rank change if excluded 3 direct evals | 16rank change if excluded 12 direct evals | 22rank change if excluded 5 direct evals | 13rank change if excluded 20 direct evals | 15rank change if excluded 12 direct evals | +2.12 SDDirect |
| gpt-5.6-sol | 3rank change if excluded 2 direct evals | 2rank change if excluded 3 direct evals | 26rank change if excluded 3 direct evals | 2rank change if excluded 15 direct evals | 6rank change if excluded 1 direct eval | 6rank change if excluded 7 direct evals | 4rank change if excluded 10 direct evals | +1.66 SDDirect |
| kimi-k3 | -12rank change if excluded 2 direct evals | 10rank change if excluded 1 direct eval | 18rank change if excluded 2 direct evals | 8rank change if excluded 12 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 3 direct evals | 10rank change if excluded 5 direct evals | -0.07 SDMedian-imputed |
| claude-sonnet-4 | 8rank change if excluded 1 direct eval | 6rank change if excluded 6 direct evals | 5rank change if excluded 3 direct evals | 6rank change if excluded 9 direct evals | 9rank change if excluded 4 direct evals | 7rank change if excluded 16 direct evals | 4rank change if excluded 13 direct evals | +2.18 SDDirect |
| claude-opus-4.1 | 12rank change if excluded 1 direct eval | Not shown0 direct evals | 4rank change if excluded 3 direct evals | 12rank change if excluded 7 direct evals | 13rank change if excluded 3 direct evals | 13rank change if excluded 7 direct evals | 14rank change if excluded 1 direct eval | +2.18 SDDirect |
| gemini-3.1-pro | Not shown0 direct evals | 9rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | -12rank change if excluded 3 direct evals | Not shown0 direct evals | 0rank change if excluded 2 direct evals | Not shown0 direct evals | +1.01 SDDirect |
| phi-3.5-moe-instruct | -3rank change if excluded 1 direct eval | -3rank change if excluded 4 direct evals | -3rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 4 direct evals | -2rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| o1-preview | 37rank change if excluded 1 direct eval | 92rank change if excluded 2 direct evals | 37rank change if excluded 1 direct eval | -48rank change if excluded 1 direct eval | Not shown0 direct evals | 93rank change if excluded 2 direct evals | 93rank change if excluded 1 direct eval | +0.49 SDDirect |
| gemini-3.1-pro-preview | -14rank change if excluded 1 direct eval | 39rank change if excluded 4 direct evals | 29rank change if excluded 5 direct evals | -14rank change if excluded 15 direct evals | 34rank change if excluded 5 direct evals | -10rank change if excluded 9 direct evals | 17rank change if excluded 6 direct evals | +1.01 SDDirect |
| grok-4.20-multi-agent | Not shown0 direct evals | -18rank change if excluded 1 direct eval | Not shown0 direct evals | 36rank change if excluded 3 direct evals | 77rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | Not shown0 direct evals | -0.62 SDDirect |
| gemini-3.7-flash | 1rank change if excluded 2 direct evals | Not shown0 direct evals | 2rank change if excluded 1 direct eval | -12rank change if excluded 7 direct evals | 10rank change if excluded 1 direct eval | 4rank change if excluded 2 direct evals | 1rank change if excluded 3 direct evals | +1.21 SDDirect |
| gpt-5.1 | 4rank change if excluded 1 direct eval | -36rank change if excluded 3 direct evals | -17rank change if excluded 5 direct evals | 11rank change if excluded 10 direct evals | -18rank change if excluded 3 direct evals | -17rank change if excluded 14 direct evals | -16rank change if excluded 7 direct evals | +1.60 SDDirect |
| llama-3.1-tulu-3-8b | Not shown0 direct evals | -5rank change if excluded 2 direct evals | -3rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | Not shown0 direct evals | 1rank change if excluded 3 direct evals | -3rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| gpt-5 | 20rank change if excluded 1 direct eval | -85rank change if excluded 5 direct evals | 18rank change if excluded 6 direct evals | 19rank change if excluded 11 direct evals | 20rank change if excluded 3 direct evals | 20rank change if excluded 19 direct evals | 18rank change if excluded 8 direct evals | +1.10 SDDirect |
| granite-4.0-h-small | Not shown0 direct evals | -4rank change if excluded 1 direct eval | Not shown0 direct evals | 12rank change if excluded 2 direct evals | Not shown0 direct evals | -3rank change if excluded 1 direct eval | -4rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| gpt-5-pro | 246rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 8rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | 7rank change if excluded 1 direct eval | Not shown0 direct evals | +1.10 SDDirect |
| qwen3.5-397b-a17b | 31rank change if excluded 1 direct eval | 0rank change if excluded 4 direct evals | 1rank change if excluded 1 direct eval | 0rank change if excluded 7 direct evals | -3rank change if excluded 2 direct evals | 0rank change if excluded 5 direct evals | 0rank change if excluded 4 direct evals | -1.02 SDDirect |
| chatgpt-4o | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 5rank change if excluded 1 direct eval | Not shown0 direct evals | 6rank change if excluded 1 direct eval | 158rank change if excluded 1 direct eval | +1.10 SDDirect |
| mimo-v2-pro | Not shown0 direct evals | -92rank change if excluded 1 direct eval | Not shown0 direct evals | -78rank change if excluded 4 direct evals | -15rank change if excluded 1 direct eval | -16rank change if excluded 1 direct eval | -92rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| inkling | 22rank change if excluded 2 direct evals | 5rank change if excluded 1 direct eval | Not shown0 direct evals | -14rank change if excluded 4 direct evals | 6rank change if excluded 2 direct evals | 5rank change if excluded 3 direct evals | 4rank change if excluded 2 direct evals | -0.07 SDMedian-imputed |
| qwen3.5-27b | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 280rank change if excluded 2 direct evals | 1rank change if excluded 3 direct evals | 127rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | -0.66 SDDirect |
| mimo-v2.5-pro | -35rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 6 direct evals | -1rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -0.07 SDMedian-imputed |
| minimax-m3 | 11rank change if excluded 3 direct evals | 14rank change if excluded 1 direct eval | Not shown0 direct evals | 18rank change if excluded 6 direct evals | 17rank change if excluded 1 direct eval | 17rank change if excluded 1 direct eval | Not shown0 direct evals | -0.07 SDMedian-imputed |
| claude-3.5-haiku | -111rank change if excluded 2 direct evals | 31rank change if excluded 4 direct evals | 30rank change if excluded 3 direct evals | 28rank change if excluded 5 direct evals | 27rank change if excluded 1 direct eval | 30rank change if excluded 8 direct evals | 30rank change if excluded 4 direct evals | +2.08 SDDirect |
| qwen3.5-9b | -2rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | 38rank change if excluded 1 direct eval | -2rank change if excluded 2 direct evals | 38rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | -2rank change if excluded 2 direct evals | -0.66 SDDirect |
| gemini-3.5-flash | -5rank change if excluded 4 direct evals | 2rank change if excluded 3 direct evals | -6rank change if excluded 2 direct evals | -18rank change if excluded 13 direct evals | 9rank change if excluded 2 direct evals | 9rank change if excluded 6 direct evals | 50rank change if excluded 7 direct evals | +1.21 SDDirect |
| grok-4.5 | -15rank change if excluded 1 direct eval | 5rank change if excluded 1 direct eval | 5rank change if excluded 1 direct eval | 101rank change if excluded 11 direct evals | 6rank change if excluded 1 direct eval | 5rank change if excluded 3 direct evals | 1rank change if excluded 4 direct evals | -0.62 SDDirect |
| palmyra-x-004 | Not shown0 direct evals | Not shown0 direct evals | 40rank change if excluded 1 direct eval | Not shown0 direct evals | 40rank change if excluded 1 direct eval | 36rank change if excluded 5 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| glm-4.5-air | -20rank change if excluded 2 direct evals | Not shown0 direct evals | 3rank change if excluded 1 direct eval | 2rank change if excluded 5 direct evals | 26rank change if excluded 2 direct evals | 3rank change if excluded 6 direct evals | Not shown0 direct evals | -1.04 SDDirect |
| muse-spark | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -3rank change if excluded 3 direct evals | Not shown0 direct evals | -3rank change if excluded 1 direct eval | 129rank change if excluded 1 direct eval | +0.38 SDDirect |
| nova-lite | 236rank change if excluded 1 direct eval | -12rank change if excluded 2 direct evals | -5rank change if excluded 1 direct eval | -6rank change if excluded 4 direct evals | Not shown0 direct evals | -5rank change if excluded 3 direct evals | -5rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| gpt-5.4-pro | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 255rank change if excluded 3 direct evals | -33rank change if excluded 1 direct eval | 237rank change if excluded 1 direct eval | Not shown0 direct evals | +1.66 SDDirect |
| kimi-k2.6 | 33rank change if excluded 2 direct evals | -21rank change if excluded 1 direct eval | 28rank change if excluded 2 direct evals | 33rank change if excluded 11 direct evals | -19rank change if excluded 2 direct evals | -19rank change if excluded 5 direct evals | -15rank change if excluded 4 direct evals | -0.07 SDMedian-imputed |
| gemini-1.0-pro | 10rank change if excluded 1 direct eval | 9rank change if excluded 2 direct evals | 10rank change if excluded 3 direct evals | 10rank change if excluded 1 direct eval | 20rank change if excluded 1 direct eval | 6rank change if excluded 7 direct evals | Not shown0 direct evals | +0.80 SDDirect |
| gemma-4-26b-a4b-it | 26rank change if excluded 1 direct eval | -33rank change if excluded 2 direct evals | Not shown0 direct evals | -31rank change if excluded 4 direct evals | Not shown0 direct evals | -33rank change if excluded 3 direct evals | -31rank change if excluded 1 direct eval | +1.21 SDDirect |
| gpt-4.5-preview | Not shown0 direct evals | 42rank change if excluded 1 direct eval | -10rank change if excluded 1 direct eval | -10rank change if excluded 3 direct evals | -10rank change if excluded 1 direct eval | -8rank change if excluded 5 direct evals | 42rank change if excluded 1 direct eval | +0.49 SDDirect |
| qwen3.7-flash | 9rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.66 SDDirect |
| qwen3-235b-a22b-fp8-throughput | Not shown0 direct evals | Not shown0 direct evals | 56rank change if excluded 1 direct eval | Not shown0 direct evals | 56rank change if excluded 1 direct eval | -42rank change if excluded 5 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| palmyra-fin | Not shown0 direct evals | Not shown0 direct evals | -39rank change if excluded 1 direct eval | Not shown0 direct evals | 3rank change if excluded 1 direct eval | 25rank change if excluded 5 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| gemini-1.5-pro | 121rank change if excluded 3 direct evals | 7rank change if excluded 3 direct evals | 4rank change if excluded 3 direct evals | 10rank change if excluded 5 direct evals | 9rank change if excluded 4 direct evals | 6rank change if excluded 14 direct evals | 6rank change if excluded 3 direct evals | +0.80 SDDirect |
| qwen2-72b-instruct | Not shown0 direct evals | -1rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 1 direct eval | -9rank change if excluded 9 direct evals | 1rank change if excluded 1 direct eval | -1.02 SDDirect |
| gemma-4-31b-it | 7rank change if excluded 3 direct evals | -12rank change if excluded 3 direct evals | 39rank change if excluded 3 direct evals | -12rank change if excluded 8 direct evals | -14rank change if excluded 2 direct evals | -12rank change if excluded 5 direct evals | -10rank change if excluded 1 direct eval | +1.21 SDDirect |
| kimi-k2 | -4rank change if excluded 1 direct eval | -4rank change if excluded 3 direct evals | 53rank change if excluded 3 direct evals | -7rank change if excluded 12 direct evals | -4rank change if excluded 2 direct evals | -4rank change if excluded 14 direct evals | -4rank change if excluded 6 direct evals | -0.07 SDMedian-imputed |
| gpt-5.4-mini | Not shown0 direct evals | 35rank change if excluded 2 direct evals | -41rank change if excluded 3 direct evals | 11rank change if excluded 11 direct evals | 10rank change if excluded 3 direct evals | 11rank change if excluded 5 direct evals | 2rank change if excluded 4 direct evals | +1.66 SDDirect |
| qwen3-max | 18rank change if excluded 1 direct eval | -24rank change if excluded 1 direct eval | -15rank change if excluded 2 direct evals | 22rank change if excluded 4 direct evals | Not shown0 direct evals | -26rank change if excluded 3 direct evals | 24rank change if excluded 1 direct eval | -1.02 SDDirect |
| mistral-large | 107rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | 6rank change if excluded 1 direct eval | -68rank change if excluded 5 direct evals | Not shown0 direct evals | -1.21 SDDirect |
| deepseek-v4-flash | 71rank change if excluded 4 direct evals | 7rank change if excluded 4 direct evals | 13rank change if excluded 1 direct eval | 0rank change if excluded 9 direct evals | 71rank change if excluded 3 direct evals | 14rank change if excluded 6 direct evals | -9rank change if excluded 4 direct evals | -1.05 SDDirect |
| glm-5 | -38rank change if excluded 1 direct eval | 66rank change if excluded 3 direct evals | -30rank change if excluded 2 direct evals | 37rank change if excluded 8 direct evals | -30rank change if excluded 2 direct evals | -30rank change if excluded 4 direct evals | 44rank change if excluded 3 direct evals | -0.58 SDDirect |
| gpt-5.2-codex | Not shown0 direct evals | -30rank change if excluded 1 direct eval | Not shown0 direct evals | 3rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -30rank change if excluded 1 direct eval | +1.66 SDDirect |
| mimo-v2.5 | -5rank change if excluded 2 direct evals | 78rank change if excluded 1 direct eval | Not shown0 direct evals | -9rank change if excluded 5 direct evals | -12rank change if excluded 2 direct evals | -10rank change if excluded 2 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| gpt-4.1 | 43rank change if excluded 2 direct evals | -81rank change if excluded 5 direct evals | -21rank change if excluded 5 direct evals | -20rank change if excluded 11 direct evals | -20rank change if excluded 4 direct evals | -78rank change if excluded 18 direct evals | -20rank change if excluded 5 direct evals | +1.10 SDDirect |
| palmyra-x5 | Not shown0 direct evals | Not shown0 direct evals | 7rank change if excluded 1 direct eval | Not shown0 direct evals | 2rank change if excluded 1 direct eval | 24rank change if excluded 5 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| qwen-72b-chat | Not shown0 direct evals | 4rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | 124rank change if excluded 4 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| hy3 | -18rank change if excluded 2 direct evals | -23rank change if excluded 2 direct evals | Not shown0 direct evals | 71rank change if excluded 3 direct evals | Not shown0 direct evals | -17rank change if excluded 1 direct eval | -23rank change if excluded 2 direct evals | -0.07 SDMedian-imputed |
| gemini-3-pro-preview | -8rank change if excluded 1 direct eval | -6rank change if excluded 1 direct eval | 15rank change if excluded 2 direct evals | -5rank change if excluded 11 direct evals | 15rank change if excluded 3 direct evals | -5rank change if excluded 12 direct evals | -27rank change if excluded 5 direct evals | +1.01 SDDirect |
| qwen3-4b | Not shown0 direct evals | -39rank change if excluded 1 direct eval | 57rank change if excluded 1 direct eval | 128rank change if excluded 1 direct eval | 57rank change if excluded 1 direct eval | -40rank change if excluded 1 direct eval | Not shown0 direct evals | -1.02 SDDirect |
| deepseek-v3.1 | 129rank change if excluded 2 direct evals | 5rank change if excluded 1 direct eval | 5rank change if excluded 2 direct evals | 1rank change if excluded 5 direct evals | 6rank change if excluded 1 direct eval | 1rank change if excluded 5 direct evals | 5rank change if excluded 1 direct eval | -1.46 SDDirect |
| internlm2-chat-7b | Not shown0 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 4 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| o1-mini | 21rank change if excluded 1 direct eval | 21rank change if excluded 2 direct evals | 48rank change if excluded 2 direct evals | -53rank change if excluded 2 direct evals | 5rank change if excluded 1 direct eval | -136rank change if excluded 8 direct evals | 5rank change if excluded 1 direct eval | +0.49 SDDirect |
| claude-sonnet-5 | -147rank change if excluded 2 direct evals | 81rank change if excluded 1 direct eval | -86rank change if excluded 1 direct eval | 82rank change if excluded 8 direct evals | 81rank change if excluded 1 direct eval | 81rank change if excluded 4 direct evals | 88rank change if excluded 3 direct evals | +2.10 SDDirect |
| o3 | Not shown0 direct evals | 44rank change if excluded 2 direct evals | -39rank change if excluded 4 direct evals | -36rank change if excluded 7 direct evals | -34rank change if excluded 3 direct evals | 44rank change if excluded 13 direct evals | 12rank change if excluded 6 direct evals | +1.10 SDDirect |
| gemini-1.5-flash | 71rank change if excluded 2 direct evals | 34rank change if excluded 2 direct evals | 35rank change if excluded 2 direct evals | 38rank change if excluded 1 direct eval | 33rank change if excluded 3 direct evals | 38rank change if excluded 13 direct evals | 38rank change if excluded 4 direct evals | +0.80 SDDirect |
| gpt-5.3-codex | Not shown0 direct evals | 66rank change if excluded 2 direct evals | Not shown0 direct evals | -23rank change if excluded 4 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | 11rank change if excluded 1 direct eval | +1.60 SDDirect |
| laguna-s-2.1-poolside | 59rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -24rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| qwen3.8-max | -58rank change if excluded 2 direct evals | 151rank change if excluded 1 direct eval | Not shown0 direct evals | 151rank change if excluded 4 direct evals | -20rank change if excluded 1 direct eval | -20rank change if excluded 1 direct eval | 152rank change if excluded 2 direct evals | -0.66 SDDirect |
| qwen2.5-72b-instruct | 171rank change if excluded 1 direct eval | 5rank change if excluded 1 direct eval | -1rank change if excluded 3 direct evals | 4rank change if excluded 4 direct evals | 7rank change if excluded 3 direct evals | 5rank change if excluded 12 direct evals | 5rank change if excluded 2 direct evals | -1.02 SDDirect |
| qwen3.6-27b | Not shown0 direct evals | -7rank change if excluded 1 direct eval | Not shown0 direct evals | -4rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -112rank change if excluded 2 direct evals | -0.66 SDDirect |
| deepseek-v3.2-exp | -140rank change if excluded 1 direct eval | -35rank change if excluded 1 direct eval | Not shown0 direct evals | 23rank change if excluded 2 direct evals | Not shown0 direct evals | -71rank change if excluded 2 direct evals | 14rank change if excluded 2 direct evals | -1.02 SDDirect |
| gpt-oss-safeguard-20b | -31rank change if excluded 1 direct eval | -125rank change if excluded 2 direct evals | Not shown0 direct evals | -114rank change if excluded 1 direct eval | Not shown0 direct evals | 70rank change if excluded 2 direct evals | 72rank change if excluded 1 direct eval | +1.60 SDDirect |
| internlm-chat-20b | Not shown0 direct evals | -3rank change if excluded 4 direct evals | -5rank change if excluded 4 direct evals | -5rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 4 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| qwen3.7-max | -20rank change if excluded 1 direct eval | -61rank change if excluded 1 direct eval | 26rank change if excluded 1 direct eval | 40rank change if excluded 8 direct evals | 4rank change if excluded 1 direct eval | 25rank change if excluded 3 direct evals | Not shown0 direct evals | -0.66 SDDirect |
| gemini-3.5-flash-lite | -66rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 62rank change if excluded 5 direct evals | 101rank change if excluded 3 direct evals | -71rank change if excluded 2 direct evals | -70rank change if excluded 1 direct eval | +1.21 SDDirect |
| deepseek-v3.2 | 32rank change if excluded 4 direct evals | 1rank change if excluded 8 direct evals | 4rank change if excluded 2 direct evals | -1rank change if excluded 14 direct evals | -1rank change if excluded 2 direct evals | 8rank change if excluded 10 direct evals | 18rank change if excluded 8 direct evals | -1.02 SDDirect |
| nemotron-3-ultra-550b-a55b | 32rank change if excluded 2 direct evals | -14rank change if excluded 1 direct eval | -14rank change if excluded 1 direct eval | -14rank change if excluded 6 direct evals | Not shown0 direct evals | -14rank change if excluded 1 direct eval | -15rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| glm-5.2 | 10rank change if excluded 3 direct evals | -3rank change if excluded 1 direct eval | -5rank change if excluded 2 direct evals | -3rank change if excluded 12 direct evals | -1rank change if excluded 1 direct eval | -1rank change if excluded 3 direct evals | -4rank change if excluded 4 direct evals | -0.60 SDDirect |
| doubao-seed-1.6 | Not shown0 direct evals | 1rank change if excluded 2 direct evals | 6rank change if excluded 2 direct evals | 4rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | 4rank change if excluded 3 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| grok-3-mini | -3rank change if excluded 1 direct eval | -48rank change if excluded 2 direct evals | 1rank change if excluded 3 direct evals | -2rank change if excluded 4 direct evals | -3rank change if excluded 1 direct eval | -3rank change if excluded 10 direct evals | -6rank change if excluded 2 direct evals | -0.05 SDDirect |
| qwen3-235b-a22b | 5rank change if excluded 3 direct evals | 53rank change if excluded 6 direct evals | 3rank change if excluded 4 direct evals | 1rank change if excluded 7 direct evals | 3rank change if excluded 3 direct evals | 3rank change if excluded 15 direct evals | 53rank change if excluded 7 direct evals | -1.02 SDDirect |
| gpt-oss-20b | 2rank change if excluded 1 direct eval | -4rank change if excluded 3 direct evals | -4rank change if excluded 1 direct eval | -1rank change if excluded 3 direct evals | -9rank change if excluded 2 direct evals | -1rank change if excluded 9 direct evals | 13rank change if excluded 3 direct evals | +1.10 SDDirect |
| qwen3.6-max-preview | Not shown0 direct evals | -37rank change if excluded 1 direct eval | Not shown0 direct evals | 8rank change if excluded 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | 58rank change if excluded 1 direct eval | -0.66 SDDirect |
| internlm2-chat-20b | Not shown0 direct evals | -15rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | Not shown0 direct evals | -8rank change if excluded 5 direct evals | 0rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| glm-5.1 | -36rank change if excluded 2 direct evals | -30rank change if excluded 3 direct evals | Not shown0 direct evals | 2rank change if excluded 8 direct evals | 2rank change if excluded 1 direct eval | 2rank change if excluded 3 direct evals | -33rank change if excluded 7 direct evals | -0.60 SDDirect |
| olmo-3-7b-instruct | Not shown0 direct evals | 89rank change if excluded 1 direct eval | 16rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| kimi-k2.7-code | Not shown0 direct evals | 181rank change if excluded 2 direct evals | Not shown0 direct evals | -50rank change if excluded 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | -39rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| o1 | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | -4rank change if excluded 6 direct evals | 7rank change if excluded 3 direct evals | -39rank change if excluded 10 direct evals | 5rank change if excluded 2 direct evals | +0.49 SDDirect |
| gpt-oss-120b | -12rank change if excluded 3 direct evals | -6rank change if excluded 5 direct evals | -11rank change if excluded 5 direct evals | 24rank change if excluded 11 direct evals | -13rank change if excluded 2 direct evals | -9rank change if excluded 14 direct evals | -7rank change if excluded 5 direct evals | +1.10 SDDirect |
| grok-4.3 | -61rank change if excluded 2 direct evals | 12rank change if excluded 3 direct evals | 10rank change if excluded 3 direct evals | 17rank change if excluded 11 direct evals | 9rank change if excluded 2 direct evals | 9rank change if excluded 8 direct evals | 11rank change if excluded 5 direct evals | -0.62 SDDirect |
| phi-4-reasoning-plus | Not shown0 direct evals | -10rank change if excluded 1 direct eval | -42rank change if excluded 1 direct eval | Not shown0 direct evals | -42rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | -41rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| gpt-4 | 106rank change if excluded 1 direct eval | 113rank change if excluded 6 direct evals | 108rank change if excluded 5 direct evals | -14rank change if excluded 2 direct evals | -13rank change if excluded 1 direct eval | -14rank change if excluded 12 direct evals | -12rank change if excluded 1 direct eval | +0.49 SDDirect |
| gemini-2.5-pro | 58rank change if excluded 1 direct eval | -12rank change if excluded 8 direct evals | -12rank change if excluded 6 direct evals | 59rank change if excluded 17 direct evals | 57rank change if excluded 3 direct evals | 58rank change if excluded 16 direct evals | -29rank change if excluded 10 direct evals | +0.63 SDDirect |
| claude-3-opus | 62rank change if excluded 5 direct evals | 29rank change if excluded 4 direct evals | 26rank change if excluded 3 direct evals | 24rank change if excluded 5 direct evals | 25rank change if excluded 4 direct evals | 27rank change if excluded 14 direct evals | 16rank change if excluded 7 direct evals | +2.08 SDDirect |
| gpt-4.1-mini | 31rank change if excluded 1 direct eval | 30rank change if excluded 2 direct evals | 40rank change if excluded 3 direct evals | -120rank change if excluded 4 direct evals | 39rank change if excluded 2 direct evals | 60rank change if excluded 13 direct evals | -120rank change if excluded 3 direct evals | +1.10 SDDirect |
| grok-4.20 | -10rank change if excluded 1 direct eval | -7rank change if excluded 3 direct evals | -5rank change if excluded 2 direct evals | -5rank change if excluded 12 direct evals | -5rank change if excluded 4 direct evals | -3rank change if excluded 7 direct evals | -42rank change if excluded 5 direct evals | -0.62 SDDirect |
| minimax-m2.7 | 2rank change if excluded 2 direct evals | 141rank change if excluded 4 direct evals | Not shown0 direct evals | -28rank change if excluded 7 direct evals | -30rank change if excluded 3 direct evals | 4rank change if excluded 4 direct evals | 36rank change if excluded 4 direct evals | -0.07 SDMedian-imputed |
| deepseek-v3 | -49rank change if excluded 4 direct evals | -43rank change if excluded 4 direct evals | 28rank change if excluded 7 direct evals | -44rank change if excluded 10 direct evals | 26rank change if excluded 2 direct evals | -47rank change if excluded 15 direct evals | 26rank change if excluded 5 direct evals | -1.46 SDDirect |
| claude-3.5-sonnet | -85rank change if excluded 4 direct evals | -2rank change if excluded 4 direct evals | 0rank change if excluded 3 direct evals | -2rank change if excluded 8 direct evals | -14rank change if excluded 4 direct evals | 0rank change if excluded 17 direct evals | 6rank change if excluded 7 direct evals | +2.08 SDDirect |
| gpt-4-turbo | 3rank change if excluded 1 direct eval | -2rank change if excluded 6 direct evals | 3rank change if excluded 4 direct evals | 3rank change if excluded 5 direct evals | 4rank change if excluded 2 direct evals | 3rank change if excluded 16 direct evals | 7rank change if excluded 7 direct evals | +0.49 SDDirect |
| deepseek-v3.1-terminus | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 20rank change if excluded 1 direct eval | Not shown0 direct evals | 20rank change if excluded 1 direct eval | -44rank change if excluded 1 direct eval | -1.02 SDDirect |
| olmo-3.1-32b-instruct | Not shown0 direct evals | 50rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| qwen3-32b | 258rank change if excluded 1 direct eval | 6rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 6rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | 6rank change if excluded 5 direct evals | 6rank change if excluded 1 direct eval | -1.02 SDDirect |
| phi-4 | Not shown0 direct evals | 0rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | -188rank change if excluded 4 direct evals | Not shown0 direct evals | 2rank change if excluded 3 direct evals | 0rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| qwen-14b-chat | Not shown0 direct evals | 118rank change if excluded 5 direct evals | -39rank change if excluded 5 direct evals | -14rank change if excluded 1 direct eval | Not shown0 direct evals | 1rank change if excluded 8 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| gemini-2.5-flash | 23rank change if excluded 2 direct evals | 23rank change if excluded 6 direct evals | -66rank change if excluded 4 direct evals | 24rank change if excluded 12 direct evals | 21rank change if excluded 1 direct eval | -66rank change if excluded 15 direct evals | 17rank change if excluded 10 direct evals | +0.63 SDDirect |
| ernie-4.5-300b-a47b | Not shown0 direct evals | 2rank change if excluded 1 direct eval | -23rank change if excluded 1 direct eval | -59rank change if excluded 3 direct evals | 65rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| qwen2.5-32b-instruct | Not shown0 direct evals | -25rank change if excluded 1 direct eval | Not shown0 direct evals | -21rank change if excluded 2 direct evals | -21rank change if excluded 1 direct eval | -21rank change if excluded 2 direct evals | 72rank change if excluded 2 direct evals | -1.02 SDDirect |
| nemotron-3-nano-omni-30b-a3b | 252rank change if excluded 1 direct eval | Not shown0 direct evals | -14rank change if excluded 1 direct eval | -14rank change if excluded 2 direct evals | Not shown0 direct evals | 6rank change if excluded 1 direct eval | -14rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| deepseek-v4-pro | 200rank change if excluded 3 direct evals | -13rank change if excluded 2 direct evals | -1rank change if excluded 1 direct eval | -10rank change if excluded 11 direct evals | -1rank change if excluded 1 direct eval | -1rank change if excluded 4 direct evals | -26rank change if excluded 3 direct evals | -1.05 SDDirect |
| grok-4 | -13rank change if excluded 1 direct eval | -14rank change if excluded 4 direct evals | 11rank change if excluded 7 direct evals | 21rank change if excluded 12 direct evals | -13rank change if excluded 2 direct evals | -14rank change if excluded 13 direct evals | -37rank change if excluded 4 direct evals | -0.05 SDDirect |
| nova-2.0-lite | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -13rank change if excluded 3 direct evals | Not shown0 direct evals | -32rank change if excluded 1 direct eval | Not shown0 direct evals | -0.07 SDMedian-imputed |
| qwen3-next-80b-a3b | Not shown0 direct evals | 8rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | -7rank change if excluded 3 direct evals | 34rank change if excluded 2 direct evals | 52rank change if excluded 8 direct evals | 8rank change if excluded 1 direct eval | -1.02 SDDirect |
| minimax-m2.5 | -28rank change if excluded 1 direct eval | 7rank change if excluded 3 direct evals | 11rank change if excluded 2 direct evals | 5rank change if excluded 8 direct evals | 5rank change if excluded 1 direct eval | 5rank change if excluded 3 direct evals | -31rank change if excluded 3 direct evals | -0.07 SDMedian-imputed |
| gemini-3.1-flash-lite | -25rank change if excluded 3 direct evals | -6rank change if excluded 4 direct evals | -13rank change if excluded 4 direct evals | -7rank change if excluded 11 direct evals | -14rank change if excluded 5 direct evals | -8rank change if excluded 8 direct evals | -8rank change if excluded 4 direct evals | +1.21 SDDirect |
| qwen3.6-plus | 19rank change if excluded 1 direct eval | -69rank change if excluded 2 direct evals | Not shown0 direct evals | 19rank change if excluded 5 direct evals | -4rank change if excluded 1 direct eval | -4rank change if excluded 1 direct eval | -23rank change if excluded 1 direct eval | -0.66 SDDirect |
| gpt-4o | 7rank change if excluded 3 direct evals | 11rank change if excluded 11 direct evals | 2rank change if excluded 9 direct evals | 0rank change if excluded 19 direct evals | 7rank change if excluded 5 direct evals | -3rank change if excluded 33 direct evals | 4rank change if excluded 19 direct evals | +0.49 SDDirect |
| kimi-k2.5 | 1rank change if excluded 2 direct evals | 26rank change if excluded 5 direct evals | -1rank change if excluded 3 direct evals | 3rank change if excluded 13 direct evals | -1rank change if excluded 2 direct evals | -1rank change if excluded 9 direct evals | -49rank change if excluded 9 direct evals | -0.07 SDMedian-imputed |
| qwen3.5-plus | Not shown0 direct evals | 45rank change if excluded 1 direct eval | 27rank change if excluded 2 direct evals | 44rank change if excluded 3 direct evals | -12rank change if excluded 1 direct eval | 45rank change if excluded 3 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| phi-3-mini-4k-instruct | Not shown0 direct evals | 4rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 4 direct evals | 4rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| granite-4.0-small-guardian | Not shown0 direct evals | Not shown0 direct evals | -36rank change if excluded 1 direct eval | Not shown0 direct evals | -35rank change if excluded 1 direct eval | 18rank change if excluded 5 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| grok-4.6 | -192rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 13rank change if excluded 4 direct evals | 8rank change if excluded 1 direct eval | 8rank change if excluded 1 direct eval | 8rank change if excluded 1 direct eval | -0.62 SDDirect |
| qwen3-max-preview | Not shown0 direct evals | Not shown0 direct evals | 211rank change if excluded 1 direct eval | 6rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| mistral-medium | Not shown0 direct evals | 20rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 1 direct eval | 174rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | 20rank change if excluded 1 direct eval | -1.21 SDDirect |
| nova-premier | -91rank change if excluded 1 direct eval | -16rank change if excluded 1 direct eval | Not shown0 direct evals | -17rank change if excluded 1 direct eval | Not shown0 direct evals | -12rank change if excluded 2 direct evals | -17rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| gpt-3.5-turbo | 5rank change if excluded 2 direct evals | -9rank change if excluded 7 direct evals | -9rank change if excluded 9 direct evals | -9rank change if excluded 4 direct evals | -9rank change if excluded 2 direct evals | -9rank change if excluded 17 direct evals | 7rank change if excluded 3 direct evals | +0.49 SDDirect |
| baichuan-2-13b-chat | Not shown0 direct evals | 27rank change if excluded 6 direct evals | 11rank change if excluded 6 direct evals | 2rank change if excluded 2 direct evals | Not shown0 direct evals | 14rank change if excluded 13 direct evals | 2rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| gemma-4-e4b | Not shown0 direct evals | -203rank change if excluded 1 direct eval | Not shown0 direct evals | 98rank change if excluded 1 direct eval | Not shown0 direct evals | -204rank change if excluded 2 direct evals | -105rank change if excluded 1 direct eval | +1.21 SDDirect |
| qwen1.5-32b-chat | Not shown0 direct evals | 12rank change if excluded 1 direct eval | 12rank change if excluded 1 direct eval | 13rank change if excluded 2 direct evals | -30rank change if excluded 1 direct eval | 13rank change if excluded 4 direct evals | 4rank change if excluded 1 direct eval | -1.02 SDDirect |
| gemini-2-flash | Not shown0 direct evals | -24rank change if excluded 1 direct eval | 58rank change if excluded 2 direct evals | -17rank change if excluded 1 direct eval | -17rank change if excluded 1 direct eval | -16rank change if excluded 6 direct evals | Not shown0 direct evals | +0.80 SDDirect |
| nemotron-3-super-120b-a12b | -18rank change if excluded 1 direct eval | 36rank change if excluded 1 direct eval | Not shown0 direct evals | 36rank change if excluded 5 direct evals | -146rank change if excluded 3 direct evals | 34rank change if excluded 4 direct evals | -60rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| qwen3.7-plus | 10rank change if excluded 1 direct eval | -32rank change if excluded 1 direct eval | 10rank change if excluded 1 direct eval | 10rank change if excluded 5 direct evals | Not shown0 direct evals | 10rank change if excluded 1 direct eval | Not shown0 direct evals | -0.66 SDDirect |
| llama-3.3-70b-instruct | 111rank change if excluded 6 direct evals | 3rank change if excluded 6 direct evals | 4rank change if excluded 5 direct evals | -3rank change if excluded 13 direct evals | 1rank change if excluded 4 direct evals | -2rank change if excluded 16 direct evals | 3rank change if excluded 3 direct evals | -1.25 SDDirect |
| qwen1.5-14b-chat | Not shown0 direct evals | 9rank change if excluded 3 direct evals | 12rank change if excluded 2 direct evals | 12rank change if excluded 3 direct evals | Not shown0 direct evals | 12rank change if excluded 6 direct evals | 12rank change if excluded 2 direct evals | -1.02 SDDirect |
| gemini-3-flash-preview | -5rank change if excluded 2 direct evals | 8rank change if excluded 4 direct evals | -12rank change if excluded 2 direct evals | -12rank change if excluded 12 direct evals | -11rank change if excluded 1 direct eval | -9rank change if excluded 5 direct evals | -12rank change if excluded 4 direct evals | +1.01 SDDirect |
| glm-4.7-flash | 264rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -6rank change if excluded 3 direct evals | Not shown0 direct evals | -4rank change if excluded 1 direct eval | Not shown0 direct evals | -0.58 SDDirect |
| minimax-m2 | -44rank change if excluded 1 direct eval | 4rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | 1rank change if excluded 3 direct evals | Not shown0 direct evals | 3rank change if excluded 3 direct evals | 3rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| gemini-2-pro-preview | Not shown0 direct evals | Not shown0 direct evals | 66rank change if excluded 1 direct eval | Not shown0 direct evals | -9rank change if excluded 1 direct eval | -96rank change if excluded 5 direct evals | Not shown0 direct evals | +0.80 SDDirect |
| qwen3-235b-a22b-instruct-2507 | Not shown0 direct evals | 146rank change if excluded 1 direct eval | Not shown0 direct evals | -36rank change if excluded 2 direct evals | Not shown0 direct evals | -5rank change if excluded 1 direct eval | -5rank change if excluded 1 direct eval | -1.02 SDDirect |
| deepseek-r1 | 23rank change if excluded 2 direct evals | -4rank change if excluded 7 direct evals | 0rank change if excluded 10 direct evals | -1rank change if excluded 15 direct evals | -3rank change if excluded 6 direct evals | -4rank change if excluded 22 direct evals | -5rank change if excluded 7 direct evals | -1.46 SDDirect |
| qwen3-30b-a3b-instruct | -23rank change if excluded 1 direct eval | 27rank change if excluded 2 direct evals | -21rank change if excluded 2 direct evals | -22rank change if excluded 2 direct evals | Not shown0 direct evals | -22rank change if excluded 2 direct evals | 27rank change if excluded 1 direct eval | -1.02 SDDirect |
| llama-2-7b-chat | 1rank change if excluded 1 direct eval | -1rank change if excluded 4 direct evals | 1rank change if excluded 5 direct evals | 1rank change if excluded 2 direct evals | -126rank change if excluded 1 direct eval | 5rank change if excluded 12 direct evals | 2rank change if excluded 2 direct evals | -1.25 SDDirect |
| grok-4-fast | -41rank change if excluded 1 direct eval | Not shown0 direct evals | 7rank change if excluded 2 direct evals | -99rank change if excluded 8 direct evals | Not shown0 direct evals | -36rank change if excluded 4 direct evals | -38rank change if excluded 2 direct evals | -0.31 SDDirect |
| text-davinci-003 | Not shown0 direct evals | 11rank change if excluded 2 direct evals | -1rank change if excluded 2 direct evals | 9rank change if excluded 1 direct eval | Not shown0 direct evals | 9rank change if excluded 3 direct evals | Not shown0 direct evals | +0.49 SDDirect |
| command-r-plus | 5rank change if excluded 1 direct eval | 5rank change if excluded 1 direct eval | 5rank change if excluded 2 direct evals | 5rank change if excluded 1 direct eval | -19rank change if excluded 1 direct eval | -19rank change if excluded 7 direct evals | 37rank change if excluded 2 direct evals | -0.07 SDMedian-imputed |
| llama-3.2-instruct-11b-vision | Not shown0 direct evals | 5rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 1 direct eval | Not shown0 direct evals | 138rank change if excluded 2 direct evals | 33rank change if excluded 1 direct eval | +0.38 SDDirect |
| claude-3-haiku | 10rank change if excluded 1 direct eval | 10rank change if excluded 2 direct evals | 9rank change if excluded 3 direct evals | -59rank change if excluded 4 direct evals | -34rank change if excluded 4 direct evals | 4rank change if excluded 12 direct evals | 4rank change if excluded 3 direct evals | +2.08 SDDirect |
| qwen3-coder-480b-a35b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 7rank change if excluded 3 direct evals | Not shown0 direct evals | 8rank change if excluded 1 direct eval | Not shown0 direct evals | -1.02 SDDirect |
| qwen1.5-72b-chat | Not shown0 direct evals | 13rank change if excluded 2 direct evals | -4rank change if excluded 3 direct evals | 13rank change if excluded 4 direct evals | 2rank change if excluded 2 direct evals | 13rank change if excluded 12 direct evals | 3rank change if excluded 2 direct evals | -1.02 SDDirect |
| o3-mini | Not shown0 direct evals | 4rank change if excluded 2 direct evals | 51rank change if excluded 3 direct evals | -13rank change if excluded 7 direct evals | -4rank change if excluded 2 direct evals | 40rank change if excluded 13 direct evals | 8rank change if excluded 5 direct evals | +0.49 SDDirect |
| mistral-small-4 | Not shown0 direct evals | -49rank change if excluded 1 direct eval | 129rank change if excluded 1 direct eval | -7rank change if excluded 4 direct evals | -6rank change if excluded 1 direct eval | -6rank change if excluded 1 direct eval | Not shown0 direct evals | -1.21 SDDirect |
| llama-4-maverick | -2rank change if excluded 2 direct evals | -5rank change if excluded 8 direct evals | -4rank change if excluded 4 direct evals | -3rank change if excluded 12 direct evals | 0rank change if excluded 2 direct evals | -1rank change if excluded 18 direct evals | -1rank change if excluded 5 direct evals | -0.42 SDDirect |
| granite-4.0-small | Not shown0 direct evals | Not shown0 direct evals | -38rank change if excluded 1 direct eval | Not shown0 direct evals | -20rank change if excluded 1 direct eval | 1rank change if excluded 5 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| glm-4.6 | -13rank change if excluded 1 direct eval | -10rank change if excluded 3 direct evals | 18rank change if excluded 2 direct evals | 27rank change if excluded 5 direct evals | Not shown0 direct evals | -6rank change if excluded 4 direct evals | -17rank change if excluded 1 direct eval | -0.58 SDDirect |
| nemotron-3-nano-30b-a3b | 238rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -20rank change if excluded 5 direct evals | -20rank change if excluded 1 direct eval | -16rank change if excluded 3 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| mistral-small-3.1 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -8rank change if excluded 2 direct evals | Not shown0 direct evals | 27rank change if excluded 3 direct evals | Not shown0 direct evals | -1.21 SDDirect |
| yi-34b-chat | Not shown0 direct evals | 7rank change if excluded 4 direct evals | 6rank change if excluded 4 direct evals | -2rank change if excluded 4 direct evals | Not shown0 direct evals | -45rank change if excluded 9 direct evals | 17rank change if excluded 2 direct evals | -0.07 SDMedian-imputed |
| o4-mini | Not shown0 direct evals | -13rank change if excluded 2 direct evals | -13rank change if excluded 3 direct evals | -12rank change if excluded 10 direct evals | -12rank change if excluded 3 direct evals | -13rank change if excluded 14 direct evals | -7rank change if excluded 6 direct evals | +1.10 SDDirect |
| grok-3 | -18rank change if excluded 2 direct evals | -1rank change if excluded 2 direct evals | 1rank change if excluded 2 direct evals | 1rank change if excluded 5 direct evals | 1rank change if excluded 1 direct eval | -1rank change if excluded 11 direct evals | 16rank change if excluded 2 direct evals | -0.05 SDDirect |
| deepseek-llm-7b-chat | Not shown0 direct evals | 16rank change if excluded 1 direct eval | -5rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 3 direct evals | Not shown0 direct evals | -1.46 SDDirect |
| nova-micro | -221rank change if excluded 1 direct eval | -59rank change if excluded 1 direct eval | Not shown0 direct evals | 11rank change if excluded 2 direct evals | Not shown0 direct evals | -59rank change if excluded 3 direct evals | 8rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| gemini-2.0-flash | -4rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -7rank change if excluded 7 direct evals | Not shown0 direct evals | 9rank change if excluded 5 direct evals | -22rank change if excluded 3 direct evals | +0.80 SDDirect |
| gemma-2b-it | -9rank change if excluded 1 direct eval | -131rank change if excluded 3 direct evals | -9rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | 2rank change if excluded 4 direct evals | 2rank change if excluded 1 direct eval | +0.80 SDDirect |
| magistral-medium | Not shown0 direct evals | -2rank change if excluded 2 direct evals | -6rank change if excluded 2 direct evals | -3rank change if excluded 4 direct evals | 29rank change if excluded 1 direct eval | -5rank change if excluded 4 direct evals | -5rank change if excluded 1 direct eval | -1.21 SDDirect |
| glm-4-9b-chat | Not shown0 direct evals | 32rank change if excluded 2 direct evals | 29rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -26rank change if excluded 8 direct evals | 29rank change if excluded 2 direct evals | -0.07 SDDirect |
| gemma-7b-it | -13rank change if excluded 1 direct eval | -98rank change if excluded 4 direct evals | -13rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | 26rank change if excluded 1 direct eval | -2rank change if excluded 6 direct evals | -2rank change if excluded 1 direct eval | +0.80 SDDirect |
| internlm-chat-7b | Not shown0 direct evals | -144rank change if excluded 5 direct evals | -1rank change if excluded 5 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | -144rank change if excluded 5 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| qwen1.5-7b-chat | Not shown0 direct evals | 2rank change if excluded 2 direct evals | 2rank change if excluded 2 direct evals | 11rank change if excluded 3 direct evals | 24rank change if excluded 1 direct eval | 2rank change if excluded 9 direct evals | 14rank change if excluded 1 direct eval | -1.02 SDDirect |
| step-3.5-flash | -146rank change if excluded 1 direct eval | 0rank change if excluded 2 direct evals | -1rank change if excluded 1 direct eval | -2rank change if excluded 5 direct evals | Not shown0 direct evals | -1rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| llama-2-13b-chat | Not shown0 direct evals | -3rank change if excluded 3 direct evals | -1rank change if excluded 2 direct evals | -3rank change if excluded 2 direct evals | -132rank change if excluded 1 direct eval | -1rank change if excluded 9 direct evals | -5rank change if excluded 2 direct evals | -1.25 SDDirect |
| gpt-4.1-nano | Not shown0 direct evals | 2rank change if excluded 1 direct eval | -45rank change if excluded 2 direct evals | 8rank change if excluded 3 direct evals | 3rank change if excluded 1 direct eval | 102rank change if excluded 10 direct evals | 2rank change if excluded 1 direct eval | +1.10 SDDirect |
| gpt-5.4-nano | Not shown0 direct evals | -2rank change if excluded 1 direct eval | -190rank change if excluded 2 direct evals | 2rank change if excluded 9 direct evals | -191rank change if excluded 2 direct evals | -2rank change if excluded 4 direct evals | -2rank change if excluded 3 direct evals | +1.66 SDDirect |
| claude-2.1 | Not shown0 direct evals | Not shown0 direct evals | -1rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | -234rank change if excluded 2 direct evals | 0rank change if excluded 5 direct evals | Not shown0 direct evals | +2.08 SDDirect |
| deepseek-v4-flash-0731 | -58rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 18rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -1.05 SDDirect |
| mimo-v2-flash | Not shown0 direct evals | -55rank change if excluded 2 direct evals | 43rank change if excluded 1 direct eval | -54rank change if excluded 6 direct evals | -99rank change if excluded 1 direct eval | -54rank change if excluded 2 direct evals | -54rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| llama-3.1-405b-instruct | Not shown0 direct evals | -9rank change if excluded 3 direct evals | -7rank change if excluded 3 direct evals | -7rank change if excluded 7 direct evals | -25rank change if excluded 3 direct evals | 18rank change if excluded 17 direct evals | 10rank change if excluded 4 direct evals | -1.25 SDDirect |
| step-3.7-flash | 66rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 12rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| gemma-2-9b-it | -7rank change if excluded 1 direct eval | -7rank change if excluded 3 direct evals | -8rank change if excluded 4 direct evals | 7rank change if excluded 3 direct evals | 9rank change if excluded 1 direct eval | 0rank change if excluded 6 direct evals | 7rank change if excluded 2 direct evals | +0.80 SDDirect |
| glm-4.5 | -7rank change if excluded 1 direct eval | -12rank change if excluded 2 direct evals | -12rank change if excluded 2 direct evals | -2rank change if excluded 5 direct evals | -10rank change if excluded 2 direct evals | -27rank change if excluded 3 direct evals | -12rank change if excluded 1 direct eval | -1.04 SDDirect |
| granite-4.0-micro-guardian | Not shown0 direct evals | Not shown0 direct evals | -59rank change if excluded 1 direct eval | Not shown0 direct evals | -15rank change if excluded 1 direct eval | 105rank change if excluded 5 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| yi-6b-chat | Not shown0 direct evals | 1rank change if excluded 3 direct evals | 0rank change if excluded 3 direct evals | 0rank change if excluded 3 direct evals | Not shown0 direct evals | 0rank change if excluded 5 direct evals | 0rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| llama-3.2-90b-vision-instruct | Not shown0 direct evals | 46rank change if excluded 1 direct eval | Not shown0 direct evals | 43rank change if excluded 1 direct eval | Not shown0 direct evals | 46rank change if excluded 2 direct evals | -60rank change if excluded 2 direct evals | -1.25 SDDirect |
| gemini-2-flash-lite-preview | Not shown0 direct evals | Not shown0 direct evals | 24rank change if excluded 1 direct eval | Not shown0 direct evals | -10rank change if excluded 1 direct eval | -10rank change if excluded 5 direct evals | Not shown0 direct evals | +0.80 SDDirect |
| llama-3-70b-instruct | 9rank change if excluded 1 direct eval | 11rank change if excluded 2 direct evals | 10rank change if excluded 2 direct evals | -7rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | -7rank change if excluded 9 direct evals | -53rank change if excluded 2 direct evals | -1.25 SDDirect |
| hy3-preview | 140rank change if excluded 1 direct eval | -91rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| glm-4.7 | 141rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -24rank change if excluded 5 direct evals | 6rank change if excluded 1 direct eval | 5rank change if excluded 3 direct evals | 7rank change if excluded 1 direct eval | -0.60 SDDirect |
| claude-3-sonnet | -18rank change if excluded 1 direct eval | -18rank change if excluded 2 direct evals | -17rank change if excluded 3 direct evals | -22rank change if excluded 2 direct evals | -15rank change if excluded 4 direct evals | -10rank change if excluded 11 direct evals | -14rank change if excluded 3 direct evals | +2.08 SDDirect |
| gemini-2.5-flash-lite | 5rank change if excluded 3 direct evals | 4rank change if excluded 2 direct evals | 4rank change if excluded 2 direct evals | 9rank change if excluded 8 direct evals | 5rank change if excluded 1 direct eval | -127rank change if excluded 9 direct evals | 4rank change if excluded 4 direct evals | +0.63 SDDirect |
| llama-3-8b-instruct | 7rank change if excluded 1 direct eval | 6rank change if excluded 4 direct evals | 7rank change if excluded 2 direct evals | -3rank change if excluded 1 direct eval | 5rank change if excluded 2 direct evals | 0rank change if excluded 12 direct evals | -3rank change if excluded 1 direct eval | -1.25 SDDirect |
| qwen3.5-4b | Not shown0 direct evals | -230rank change if excluded 1 direct eval | 7rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | 48rank change if excluded 2 direct evals | -230rank change if excluded 1 direct eval | Not shown0 direct evals | -0.66 SDDirect |
| qwen3.5-122b-a10b | Not shown0 direct evals | 60rank change if excluded 1 direct eval | Not shown0 direct evals | -71rank change if excluded 3 direct evals | 14rank change if excluded 1 direct eval | 54rank change if excluded 1 direct eval | -40rank change if excluded 2 direct evals | -0.66 SDDirect |
| command-r | Not shown0 direct evals | Not shown0 direct evals | -26rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 1 direct eval | -25rank change if excluded 6 direct evals | 135rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| olmo-2-0325-32b-instruct | Not shown0 direct evals | -3rank change if excluded 1 direct eval | -29rank change if excluded 2 direct evals | -3rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | 0rank change if excluded 6 direct evals | 6rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| gemma-2-27b-it | -4rank change if excluded 1 direct eval | -4rank change if excluded 2 direct evals | -4rank change if excluded 1 direct eval | -4rank change if excluded 2 direct evals | 18rank change if excluded 1 direct eval | -4rank change if excluded 3 direct evals | -4rank change if excluded 1 direct eval | +0.80 SDDirect |
| llama-3.1-70b-instruct | 0rank change if excluded 1 direct eval | 3rank change if excluded 3 direct evals | 4rank change if excluded 3 direct evals | -2rank change if excluded 2 direct evals | -19rank change if excluded 4 direct evals | -17rank change if excluded 14 direct evals | 0rank change if excluded 4 direct evals | -1.25 SDDirect |
| vicuna-13b | Not shown0 direct evals | 22rank change if excluded 2 direct evals | 22rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 26rank change if excluded 3 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| grok-4.1-fast | -82rank change if excluded 2 direct evals | -106rank change if excluded 6 direct evals | 11rank change if excluded 2 direct evals | 9rank change if excluded 11 direct evals | 13rank change if excluded 1 direct eval | -3rank change if excluded 8 direct evals | -1rank change if excluded 5 direct evals | -0.31 SDDirect |
| qwen-7b-chat | Not shown0 direct evals | 29rank change if excluded 6 direct evals | 2rank change if excluded 6 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | -7rank change if excluded 9 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| gemma-3-12b | Not shown0 direct evals | -5rank change if excluded 2 direct evals | -5rank change if excluded 1 direct eval | -1rank change if excluded 3 direct evals | -5rank change if excluded 1 direct eval | 61rank change if excluded 7 direct evals | -4rank change if excluded 1 direct eval | +1.21 SDDirect |
| qwen3-30b-a3b | 7rank change if excluded 1 direct eval | -4rank change if excluded 1 direct eval | Not shown0 direct evals | 95rank change if excluded 1 direct eval | Not shown0 direct evals | -53rank change if excluded 3 direct evals | -3rank change if excluded 1 direct eval | -1.02 SDDirect |
| chatglm-6b | Not shown0 direct evals | 6rank change if excluded 2 direct evals | 6rank change if excluded 3 direct evals | 6rank change if excluded 1 direct eval | Not shown0 direct evals | 6rank change if excluded 4 direct evals | Not shown0 direct evals | -0.07 SDDirect |
| glm-5-turbo | Not shown0 direct evals | -107rank change if excluded 2 direct evals | Not shown0 direct evals | 4rank change if excluded 4 direct evals | 5rank change if excluded 1 direct eval | -149rank change if excluded 2 direct evals | -107rank change if excluded 2 direct evals | -0.60 SDDirect |
| qwen-2.5-7b-instruct | 15rank change if excluded 1 direct eval | -2rank change if excluded 2 direct evals | -2rank change if excluded 4 direct evals | -3rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | 4rank change if excluded 15 direct evals | -3rank change if excluded 3 direct evals | -1.02 SDDirect |
| phi-3.5-mini-instruct | -30rank change if excluded 1 direct eval | -31rank change if excluded 4 direct evals | -30rank change if excluded 2 direct evals | 29rank change if excluded 1 direct eval | Not shown0 direct evals | 26rank change if excluded 4 direct evals | 28rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| gemini-2.0-flash-lite | -72rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 1 direct eval | 8rank change if excluded 1 direct eval | Not shown0 direct evals | 52rank change if excluded 3 direct evals | Not shown0 direct evals | +0.80 SDDirect |
| baichuan-2-7b-chat | Not shown0 direct evals | 5rank change if excluded 5 direct evals | 4rank change if excluded 5 direct evals | 2rank change if excluded 2 direct evals | Not shown0 direct evals | -17rank change if excluded 11 direct evals | 2rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| qwen3.5-35b-a3b | Not shown0 direct evals | 139rank change if excluded 1 direct eval | Not shown0 direct evals | -24rank change if excluded 3 direct evals | Not shown0 direct evals | 139rank change if excluded 1 direct eval | 129rank change if excluded 2 direct evals | -0.66 SDDirect |
| yi-1.5-9b-chat | Not shown0 direct evals | -13rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -6rank change if excluded 3 direct evals | -15rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| qwen-2.5-14b-instruct | Not shown0 direct evals | -4rank change if excluded 2 direct evals | -4rank change if excluded 2 direct evals | -3rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 5 direct evals | -3rank change if excluded 2 direct evals | -1.02 SDDirect |
| gemma-3-27b-it | Not shown0 direct evals | -1rank change if excluded 2 direct evals | 22rank change if excluded 3 direct evals | -2rank change if excluded 8 direct evals | 22rank change if excluded 1 direct eval | 3rank change if excluded 7 direct evals | -1rank change if excluded 1 direct eval | +0.80 SDDirect |
| gpt-4o-mini | 16rank change if excluded 3 direct evals | 12rank change if excluded 3 direct evals | 6rank change if excluded 4 direct evals | 4rank change if excluded 6 direct evals | 3rank change if excluded 3 direct evals | 4rank change if excluded 17 direct evals | -47rank change if excluded 6 direct evals | +0.49 SDDirect |
| phi-2 | Not shown0 direct evals | 115rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -24rank change if excluded 3 direct evals | -4rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| nvidia-nemotron-nano-9b-v2 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 2 direct evals | Not shown0 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | -0.07 SDMedian-imputed |
| yi-1.5-34b-chat | Not shown0 direct evals | -46rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | Not shown0 direct evals | 16rank change if excluded 4 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| mistral-small-3.2 | Not shown0 direct evals | 9rank change if excluded 1 direct eval | 27rank change if excluded 1 direct eval | -71rank change if excluded 3 direct evals | Not shown0 direct evals | 9rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | -1.21 SDDirect |
| chatglm3-6b | Not shown0 direct evals | -1rank change if excluded 6 direct evals | -25rank change if excluded 6 direct evals | -1rank change if excluded 2 direct evals | Not shown0 direct evals | -28rank change if excluded 10 direct evals | Not shown0 direct evals | -0.07 SDDirect |
| qwen3.5-flash-02-23 | 47rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 1 direct eval | -4rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.66 SDDirect |
| minimax-m2.1 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -170rank change if excluded 5 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 3 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| llama-2-70b-chat | Not shown0 direct evals | -2rank change if excluded 2 direct evals | -6rank change if excluded 1 direct eval | -6rank change if excluded 2 direct evals | -165rank change if excluded 2 direct evals | 1rank change if excluded 8 direct evals | -1rank change if excluded 2 direct evals | -1.25 SDDirect |
| mistral-7b | Not shown0 direct evals | 17rank change if excluded 1 direct eval | Not shown0 direct evals | -64rank change if excluded 2 direct evals | Not shown0 direct evals | 20rank change if excluded 2 direct evals | 8rank change if excluded 2 direct evals | -1.21 SDDirect |
| vicuna-7b-v1.5 | -15rank change if excluded 1 direct eval | -15rank change if excluded 1 direct eval | -15rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -9rank change if excluded 3 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| mistral-7b-instruct | 4rank change if excluded 1 direct eval | 2rank change if excluded 6 direct evals | 2rank change if excluded 6 direct evals | -3rank change if excluded 4 direct evals | 5rank change if excluded 3 direct evals | -15rank change if excluded 17 direct evals | -3rank change if excluded 2 direct evals | -1.21 SDDirect |
| nova-pro | -66rank change if excluded 1 direct eval | 19rank change if excluded 1 direct eval | Not shown0 direct evals | 19rank change if excluded 4 direct evals | -20rank change if excluded 1 direct eval | 20rank change if excluded 3 direct evals | 19rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| mistral-medium-3 | -80rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 58rank change if excluded 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | 17rank change if excluded 1 direct eval | -1.21 SDDirect |
| muse-glimmer | -90rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 41rank change if excluded 2 direct evals | -5rank change if excluded 1 direct eval | 12rank change if excluded 1 direct eval | Not shown0 direct evals | +0.38 SDDirect |
| trinity-large | Not shown0 direct evals | 6rank change if excluded 1 direct eval | Not shown0 direct evals | 30rank change if excluded 7 direct evals | 6rank change if excluded 1 direct eval | 6rank change if excluded 1 direct eval | Not shown0 direct evals | -0.07 SDMedian-imputed |
| qwen3-14b | Not shown0 direct evals | -1rank change if excluded 2 direct evals | -24rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | -24rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | -1.02 SDDirect |
| mistral-small-3 | Not shown0 direct evals | 3rank change if excluded 1 direct eval | 40rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | -32rank change if excluded 6 direct evals | 3rank change if excluded 1 direct eval | -1.21 SDDirect |
| gemma-2-2b-it | -27rank change if excluded 1 direct eval | -27rank change if excluded 2 direct evals | -27rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 42rank change if excluded 3 direct evals | -6rank change if excluded 1 direct eval | +0.80 SDDirect |
| palmyra-med | Not shown0 direct evals | Not shown0 direct evals | -30rank change if excluded 1 direct eval | Not shown0 direct evals | -29rank change if excluded 1 direct eval | 50rank change if excluded 5 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| llama-4-scout | -11rank change if excluded 1 direct eval | -11rank change if excluded 3 direct evals | -10rank change if excluded 3 direct evals | -11rank change if excluded 5 direct evals | -11rank change if excluded 1 direct eval | -34rank change if excluded 11 direct evals | -10rank change if excluded 2 direct evals | -0.42 SDDirect |
| qwen2.5-max | Not shown0 direct evals | -32rank change if excluded 1 direct eval | -37rank change if excluded 1 direct eval | -32rank change if excluded 3 direct evals | -32rank change if excluded 1 direct eval | 95rank change if excluded 3 direct evals | -32rank change if excluded 1 direct eval | -1.02 SDDirect |
| vicuna-13b-v1.5 | -50rank change if excluded 1 direct eval | -50rank change if excluded 1 direct eval | -50rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -7rank change if excluded 3 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| olmo-3.1-32b-think | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 3 direct evals | Not shown0 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | -0.07 SDMedian-imputed |
| qwen2-7b-instruct | Not shown0 direct evals | -18rank change if excluded 1 direct eval | 39rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -21rank change if excluded 4 direct evals | -18rank change if excluded 1 direct eval | -1.02 SDDirect |
| mistral-nemo | 92rank change if excluded 2 direct evals | 92rank change if excluded 2 direct evals | 92rank change if excluded 1 direct eval | -13rank change if excluded 1 direct eval | Not shown0 direct evals | -13rank change if excluded 3 direct evals | Not shown0 direct evals | -1.21 SDDirect |
| gemma-3-4b | Not shown0 direct evals | -52rank change if excluded 2 direct evals | 16rank change if excluded 1 direct eval | -28rank change if excluded 1 direct eval | -52rank change if excluded 1 direct eval | -52rank change if excluded 4 direct evals | -27rank change if excluded 1 direct eval | +1.21 SDDirect |
| granite-3.3-8b-instruct | Not shown0 direct evals | Not shown0 direct evals | -5rank change if excluded 1 direct eval | -77rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 6 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| mistral-medium-3.1 | -39rank change if excluded 1 direct eval | -44rank change if excluded 1 direct eval | -39rank change if excluded 1 direct eval | -50rank change if excluded 4 direct evals | Not shown0 direct evals | -39rank change if excluded 3 direct evals | 64rank change if excluded 2 direct evals | -1.21 SDDirect |
| olmo-2-1124-13b-instruct | Not shown0 direct evals | 37rank change if excluded 1 direct eval | -16rank change if excluded 2 direct evals | -16rank change if excluded 1 direct eval | -16rank change if excluded 1 direct eval | -16rank change if excluded 5 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| qwen2.5-3b-instruct | Not shown0 direct evals | -153rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 5rank change if excluded 3 direct evals | 2rank change if excluded 1 direct eval | -1.02 SDDirect |
| qwen-2.5-1.5b-instruct | Not shown0 direct evals | -16rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 5 direct evals | -15rank change if excluded 1 direct eval | -1.02 SDDirect |
| chatglm2-6b | Not shown0 direct evals | 9rank change if excluded 4 direct evals | -10rank change if excluded 4 direct evals | Not shown0 direct evals | Not shown0 direct evals | -9rank change if excluded 6 direct evals | Not shown0 direct evals | -0.07 SDDirect |
| llama-3.1-8b-instruct | 2rank change if excluded 1 direct eval | -6rank change if excluded 5 direct evals | 2rank change if excluded 7 direct evals | 3rank change if excluded 6 direct evals | 3rank change if excluded 4 direct evals | -18rank change if excluded 19 direct evals | 18rank change if excluded 5 direct evals | -1.25 SDDirect |
| qwen3-next-80b-a3b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -216rank change if excluded 2 direct evals | Not shown0 direct evals | 6rank change if excluded 1 direct eval | Not shown0 direct evals | -1.02 SDDirect |
| seallms-v3-7b-chat | Not shown0 direct evals | 0rank change if excluded 1 direct eval | -93rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| qwen1.5-0.5b-chat | Not shown0 direct evals | 22rank change if excluded 1 direct eval | 22rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | Not shown0 direct evals | 4rank change if excluded 3 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| ling-2.6-flash | -135rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 21rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| vicuna-7b | Not shown0 direct evals | -13rank change if excluded 2 direct evals | -13rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 5rank change if excluded 4 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| mistral-medium-3.5 | Not shown0 direct evals | -121rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | 25rank change if excluded 5 direct evals | Not shown0 direct evals | 4rank change if excluded 1 direct eval | Not shown0 direct evals | -1.21 SDDirect |
| qwen2.5-7b | Not shown0 direct evals | Not shown0 direct evals | -13rank change if excluded 2 direct evals | -5rank change if excluded 1 direct eval | -14rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -0.66 SDDirect |
| qwen3-vl-235b-a22b-instruct | Not shown0 direct evals | 4rank change if excluded 1 direct eval | Not shown0 direct evals | -187rank change if excluded 2 direct evals | Not shown0 direct evals | 4rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | -1.02 SDDirect |
| zephyr-7b-beta | Not shown0 direct evals | -8rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -199rank change if excluded 5 direct evals | 2rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| qwen3-8b | Not shown0 direct evals | 1rank change if excluded 3 direct evals | 1rank change if excluded 3 direct evals | 0rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | -2rank change if excluded 5 direct evals | 1rank change if excluded 2 direct evals | -1.02 SDDirect |
| command-a | Not shown0 direct evals | 8rank change if excluded 2 direct evals | 6rank change if excluded 2 direct evals | -11rank change if excluded 6 direct evals | Not shown0 direct evals | -16rank change if excluded 5 direct evals | 8rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| mixtral-8x22b-instruct | Not shown0 direct evals | -15rank change if excluded 2 direct evals | 10rank change if excluded 1 direct eval | Not shown0 direct evals | 7rank change if excluded 2 direct evals | -131rank change if excluded 8 direct evals | 2rank change if excluded 1 direct eval | -1.21 SDDirect |
| phi-4-mini | Not shown0 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | -22rank change if excluded 2 direct evals | Not shown0 direct evals | 3rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| moss-moon-003-sft | Not shown0 direct evals | 0rank change if excluded 2 direct evals | 0rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 3 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| llama-3.2-3b | Not shown0 direct evals | -19rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | Not shown0 direct evals | 4rank change if excluded 2 direct evals | -19rank change if excluded 2 direct evals | Not shown0 direct evals | +0.38 SDDirect |
| vicuna-33b-v1.3 | Not shown0 direct evals | 20rank change if excluded 2 direct evals | 4rank change if excluded 2 direct evals | -38rank change if excluded 1 direct eval | Not shown0 direct evals | 4rank change if excluded 4 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| granite-4.0-micro | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | -74rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | 5rank change if excluded 6 direct evals | 4rank change if excluded 2 direct evals | -0.07 SDMedian-imputed |
| davinci | Not shown0 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | Not shown0 direct evals | -13rank change if excluded 2 direct evals | Not shown0 direct evals | +0.49 SDDirect |
| deepseek-v3.2-speciale | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 1 direct eval | Not shown0 direct evals | 2rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | -1.02 SDDirect |
| llama-3.2-1b-instruct | 34rank change if excluded 1 direct eval | 34rank change if excluded 2 direct evals | 34rank change if excluded 1 direct eval | 55rank change if excluded 1 direct eval | Not shown0 direct evals | -28rank change if excluded 7 direct evals | 1rank change if excluded 1 direct eval | -1.25 SDDirect |
| mistral-large-3 | Not shown0 direct evals | 4rank change if excluded 4 direct evals | 0rank change if excluded 3 direct evals | -6rank change if excluded 10 direct evals | 0rank change if excluded 1 direct eval | 5rank change if excluded 6 direct evals | 0rank change if excluded 1 direct eval | -1.21 SDDirect |
| deepseek-llm-67b-chat | Not shown0 direct evals | -72rank change if excluded 1 direct eval | 5rank change if excluded 1 direct eval | Not shown0 direct evals | -73rank change if excluded 1 direct eval | -89rank change if excluded 7 direct evals | -70rank change if excluded 1 direct eval | -1.46 SDDirect |
| magistral-small | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | 26rank change if excluded 2 direct evals | Not shown0 direct evals | -98rank change if excluded 2 direct evals | 4rank change if excluded 1 direct eval | -1.21 SDDirect |
| qwen-1.8b-chat | Not shown0 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 3 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| marin-8b-instruct | Not shown0 direct evals | Not shown0 direct evals | -5rank change if excluded 1 direct eval | Not shown0 direct evals | -5rank change if excluded 1 direct eval | -5rank change if excluded 4 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| qwq-32b | Not shown0 direct evals | -121rank change if excluded 3 direct evals | 5rank change if excluded 2 direct evals | 5rank change if excluded 2 direct evals | Not shown0 direct evals | -17rank change if excluded 3 direct evals | 8rank change if excluded 1 direct eval | -1.02 SDDirect |
| jamba-1-7-large | Not shown0 direct evals | 4rank change if excluded 1 direct eval | Not shown0 direct evals | -39rank change if excluded 3 direct evals | Not shown0 direct evals | -1rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| mistral-large-2 | Not shown0 direct evals | -2rank change if excluded 2 direct evals | 3rank change if excluded 3 direct evals | 4rank change if excluded 7 direct evals | Not shown0 direct evals | 4rank change if excluded 7 direct evals | 7rank change if excluded 1 direct eval | -1.21 SDDirect |
| olmo-2-1124-7b-instruct | Not shown0 direct evals | 2rank change if excluded 1 direct eval | -5rank change if excluded 1 direct eval | Not shown0 direct evals | -5rank change if excluded 1 direct eval | 3rank change if excluded 5 direct evals | 2rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| jamba-1-7-mini | Not shown0 direct evals | 11rank change if excluded 1 direct eval | Not shown0 direct evals | 2rank change if excluded 2 direct evals | Not shown0 direct evals | 2rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| qwen3-vl-8b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | -1.02 SDDirect |
| command-r7b-12-2024 | Not shown0 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 3 direct evals | -1rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| dbrx-instruct | Not shown0 direct evals | -1rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 1 direct eval | -10rank change if excluded 6 direct evals | -1rank change if excluded 1 direct eval | -0.07 SDMedian-imputed |
| olmoe-1b-7b-0125-instruct | Not shown0 direct evals | Not shown0 direct evals | -3rank change if excluded 1 direct eval | Not shown0 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 4 direct evals | Not shown0 direct evals | -0.07 SDMedian-imputed |
| deepseek-r1-distill-llama-70b | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | Not shown0 direct evals | -1.46 SDDirect |
| llama-3.2-3b-instruct | 3rank change if excluded 1 direct eval | 5rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 6 direct evals | 5rank change if excluded 2 direct evals | -1.25 SDDirect |
| mixtral-8x7b-instruct | Not shown0 direct evals | -1rank change if excluded 2 direct evals | 1rank change if excluded 2 direct evals | -2rank change if excluded 2 direct evals | -22rank change if excluded 1 direct eval | -21rank change if excluded 10 direct evals | -22rank change if excluded 2 direct evals | -1.21 SDDirect |
| grok-2 | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 1 direct eval | 2rank change if excluded 3 direct evals | Not shown0 direct evals | -118rank change if excluded 2 direct evals | Not shown0 direct evals | -1.25 SDDirect |
| qwen1.5-1.8b-chat | Not shown0 direct evals | 2rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | Not shown0 direct evals | 2rank change if excluded 4 direct evals | 2rank change if excluded 1 direct eval | -1.02 SDDirect |
| magistral-small-2509 | Not shown0 direct evals | -6rank change if excluded 1 direct eval | Not shown0 direct evals | 11rank change if excluded 1 direct eval | Not shown0 direct evals | -8rank change if excluded 2 direct evals | -7rank change if excluded 1 direct eval | -1.21 SDDirect |
| qwen-2.5-0.5b-instruct | Not shown0 direct evals | -2rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 4 direct evals | -2rank change if excluded 1 direct eval | -1.02 SDDirect |
| ministral-3-14b | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 5rank change if excluded 2 direct evals | Not shown0 direct evals | -6rank change if excluded 1 direct eval | Not shown0 direct evals | -1.21 SDDirect |
| ministral-3-8b | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 2 direct evals | Not shown0 direct evals | -3rank change if excluded 2 direct evals | Not shown0 direct evals | -1.21 SDDirect |
