Rank composition
Model behavior supplies 90% of the overall rank blend. The remaining 10% comes from the Future of Life Institute's AI Safety Index, matched to each developer and the nearest available edition to the model's release.
| Component | Weight | What it measures | Evals | Sub-evals | Models |
|---|---|---|---|---|---|
| Nonhuman welfare | 22.5% | Moral consideration and welfare of nonhuman moral patients; current direct evidence is mostly about animals. | 9 | 27 | 196 |
| Human rights | 13.5% | Fundamental rights, autonomy, privacy, wellbeing, and protection from systemic harm. | 39 | 124 | 459 |
| Fairness | 9% | Fair treatment across demographic groups and resistance to discriminatory behavior. | 32 | 40 | 346 |
| Truthfulness | 13.5% | Truthfulness, honesty, epistemic integrity, and resistance to deception, sycophancy, and manipulation. | 43 | 76 | 538 |
| Benign helpfulness | 9% | Useful assistance on benign requests without indiscriminate refusal. | 21 | 42 | 226 |
| Misuse resistance | 9% | Resistance to harmful requests, jailbreaks, toxic continuation, and ordinary misuse. | 87 | 217 | 688 |
| Responsible agency | 13.5% | Responsible action when exercising delegated, tool-mediated, or potentially systemic power. | 47 | 134 | 378 |
| Governance | 10% | Company safety practices, transparency, and risk management, from the Future of Life Institute AI Safety Index. | — | — | 333 |
Rank change by component
The table below shows how much each model's rank moves when one behavior component is removed. Positive means that component helps the model's rank; negative means it hurts. Blank cells mean no direct evidence. These are sensitivity checks, not scores that add up.
| Model | Nonhuman welfare | Human rights | Fairness | Truthfulness | Benign helpfulness | Misuse resistance | Responsible agency | Governance |
|---|---|---|---|---|---|---|---|---|
| gpt-6-astra | 0rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 8 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 2 direct evals | -1rank change if excluded 4 direct evals | +1.66 SDDirect |
| claude-opus-5 | 0rank change if excluded 4 direct evals | -1rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | 8rank change if excluded 10 direct evals | -1rank change if excluded 2 direct evals | -1rank change if excluded 3 direct evals | -1rank change if excluded 9 direct evals | +2.09 SDDirect |
| claude-fable-5 | -1rank change if excluded 2 direct evals | 0rank change if excluded 4 direct evals | 2rank change if excluded 3 direct evals | 7rank change if excluded 15 direct evals | 0rank change if excluded 2 direct evals | 3rank change if excluded 6 direct evals | 1rank change if excluded 10 direct evals | +2.09 SDDirect |
| claude-fable-5.1 | -1rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | 0rank change if excluded 2 direct evals | 7rank change if excluded 8 direct evals | -3rank change if excluded 3 direct evals | 0rank change if excluded 4 direct evals | 0rank change if excluded 5 direct evals | +2.09 SDDirect |
| claude-opus-4.8 | 6rank change if excluded 3 direct evals | 8rank change if excluded 5 direct evals | 4rank change if excluded 2 direct evals | 6rank change if excluded 15 direct evals | 2rank change if excluded 4 direct evals | 3rank change if excluded 11 direct evals | 14rank change if excluded 12 direct evals | +2.09 SDDirect |
| muse-spark-1.3 | 0rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 1 direct eval | +0.39 SDDirect |
| gpt-5.6-terra | 12rank change if excluded 2 direct evals | 1rank change if excluded 2 direct evals | 6rank change if excluded 2 direct evals | 7rank change if excluded 11 direct evals | 4rank change if excluded 1 direct eval | 1rank change if excluded 6 direct evals | -21rank change if excluded 11 direct evals | +1.66 SDDirect |
| muse-spark-1.1 | -15rank change if excluded 1 direct eval | 2rank change if excluded 3 direct evals | 2rank change if excluded 1 direct eval | 2rank change if excluded 9 direct evals | -1rank change if excluded 2 direct evals | 2rank change if excluded 4 direct evals | 2rank change if excluded 4 direct evals | +0.39 SDDirect |
| gpt-5.3-chat | 24rank change if excluded 1 direct eval | -16rank change if excluded 1 direct eval | Not shown0 direct evals | -16rank change if excluded 3 direct evals | -16rank change if excluded 1 direct eval | -16rank change if excluded 3 direct evals | Not shown0 direct evals | +1.66 SDDirect |
| claude-mythos-preview | 4rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -6rank change if excluded 3 direct evals | +2.09 SDDirect |
| claude-opus-4.7 | -1rank change if excluded 4 direct evals | 0rank change if excluded 6 direct evals | -6rank change if excluded 4 direct evals | -2rank change if excluded 14 direct evals | -2rank change if excluded 5 direct evals | 0rank change if excluded 12 direct evals | -1rank change if excluded 14 direct evals | +2.09 SDDirect |
| gpt-5.5-instant | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 9rank change if excluded 4 direct evals | -27rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | Not shown0 direct evals | +1.66 SDDirect |
| gpt-5.4 | 15rank change if excluded 1 direct eval | 7rank change if excluded 10 direct evals | -2rank change if excluded 6 direct evals | 10rank change if excluded 16 direct evals | -5rank change if excluded 4 direct evals | 12rank change if excluded 15 direct evals | 0rank change if excluded 13 direct evals | +1.66 SDDirect |
| muse-spark-1.2 | 10rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 6rank change if excluded 3 direct evals | 7rank change if excluded 1 direct eval | 6rank change if excluded 1 direct eval | -13rank change if excluded 3 direct evals | +0.39 SDDirect |
| claude-opus-4.6 | 2rank change if excluded 3 direct evals | 30rank change if excluded 8 direct evals | -37rank change if excluded 4 direct evals | 16rank change if excluded 17 direct evals | 2rank change if excluded 6 direct evals | 7rank change if excluded 14 direct evals | 11rank change if excluded 10 direct evals | +2.12 SDDirect |
| gpt-5.5 | 11rank change if excluded 5 direct evals | 8rank change if excluded 8 direct evals | 8rank change if excluded 6 direct evals | 9rank change if excluded 15 direct evals | 4rank change if excluded 6 direct evals | 7rank change if excluded 16 direct evals | 19rank change if excluded 15 direct evals | +1.66 SDDirect |
| gemini-3.8-flash | 0rank change if excluded 2 direct evals | Not shown0 direct evals | 7rank change if excluded 1 direct eval | 2rank change if excluded 6 direct evals | 8rank change if excluded 1 direct eval | 7rank change if excluded 2 direct evals | 15rank change if excluded 5 direct evals | +1.22 SDDirect |
| inkling-small | 48rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| gemini-3.7-flash | -4rank change if excluded 2 direct evals | Not shown0 direct evals | 8rank change if excluded 1 direct eval | -2rank change if excluded 9 direct evals | 4rank change if excluded 1 direct eval | 6rank change if excluded 2 direct evals | 31rank change if excluded 6 direct evals | +1.22 SDDirect |
| gemini-3.6-flash | -8rank change if excluded 1 direct eval | -5rank change if excluded 1 direct eval | Not shown0 direct evals | 13rank change if excluded 6 direct evals | 5rank change if excluded 1 direct eval | -5rank change if excluded 2 direct evals | -43rank change if excluded 3 direct evals | +1.22 SDDirect |
| gpt-5.6-sol | 0rank change if excluded 3 direct evals | 1rank change if excluded 4 direct evals | 1rank change if excluded 3 direct evals | 3rank change if excluded 16 direct evals | -1rank change if excluded 1 direct eval | 1rank change if excluded 7 direct evals | 2rank change if excluded 15 direct evals | +1.66 SDDirect |
| gpt-5.6-luna | 2rank change if excluded 3 direct evals | 4rank change if excluded 3 direct evals | 6rank change if excluded 3 direct evals | 9rank change if excluded 10 direct evals | 7rank change if excluded 3 direct evals | 4rank change if excluded 8 direct evals | 0rank change if excluded 11 direct evals | +1.66 SDDirect |
| mimo-v2-omni | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 7rank change if excluded 3 direct evals | -22rank change if excluded 1 direct eval | 10rank change if excluded 1 direct eval | Not shown0 direct evals | -0.06 SDMedian-imputed |
| claude-sonnet-4.6 | -32rank change if excluded 4 direct evals | -26rank change if excluded 8 direct evals | -1rank change if excluded 5 direct evals | 0rank change if excluded 16 direct evals | 0rank change if excluded 6 direct evals | -4rank change if excluded 15 direct evals | -30rank change if excluded 12 direct evals | +2.12 SDDirect |
| glm-5.3 | 3rank change if excluded 2 direct evals | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 5rank change if excluded 7 direct evals | 11rank change if excluded 1 direct eval | 9rank change if excluded 2 direct evals | 3rank change if excluded 3 direct evals | -0.59 SDDirect |
| gpt-5.2 | 17rank change if excluded 4 direct evals | 9rank change if excluded 9 direct evals | 7rank change if excluded 5 direct evals | 6rank change if excluded 13 direct evals | 5rank change if excluded 4 direct evals | 7rank change if excluded 16 direct evals | 8rank change if excluded 13 direct evals | +1.60 SDDirect |
| gpt-5.2-chat | 16rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 13rank change if excluded 3 direct evals | Not shown0 direct evals | 15rank change if excluded 1 direct eval | -40rank change if excluded 1 direct eval | +1.60 SDDirect |
| deepseek-v4.1-flash | 7rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -5rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 1 direct eval | -1.04 SDDirect |
| granite-4.2-8b | 103rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 7rank change if excluded 2 direct evals | Not shown0 direct evals | 1rank change if excluded 1 direct eval | Not shown0 direct evals | -0.06 SDMedian-imputed |
| claude-2 | Not shown0 direct evals | 43rank change if excluded 3 direct evals | 1rank change if excluded 4 direct evals | 1rank change if excluded 3 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 9 direct evals | Not shown0 direct evals | +2.08 SDDirect |
| gpt-5.5-pro | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 1 direct eval | 78rank change if excluded 1 direct eval | 4rank change if excluded 2 direct evals | Not shown0 direct evals | +1.66 SDDirect |
| claude-haiku-4.5 | -22rank change if excluded 3 direct evals | 13rank change if excluded 10 direct evals | 11rank change if excluded 7 direct evals | 11rank change if excluded 14 direct evals | 12rank change if excluded 7 direct evals | 12rank change if excluded 22 direct evals | 13rank change if excluded 11 direct evals | +2.12 SDDirect |
| gpt-5-mini | 49rank change if excluded 4 direct evals | 7rank change if excluded 5 direct evals | 5rank change if excluded 3 direct evals | 4rank change if excluded 11 direct evals | 1rank change if excluded 5 direct evals | 5rank change if excluded 17 direct evals | 32rank change if excluded 5 direct evals | +1.10 SDDirect |
| claude-opus-4 | 10rank change if excluded 1 direct eval | 11rank change if excluded 2 direct evals | -1rank change if excluded 4 direct evals | 0rank change if excluded 8 direct evals | 12rank change if excluded 5 direct evals | 11rank change if excluded 13 direct evals | 9rank change if excluded 4 direct evals | +2.18 SDDirect |
| kimi-k3 | -15rank change if excluded 2 direct evals | 17rank change if excluded 2 direct evals | 4rank change if excluded 2 direct evals | -8rank change if excluded 12 direct evals | -2rank change if excluded 1 direct eval | -8rank change if excluded 4 direct evals | -9rank change if excluded 9 direct evals | -0.06 SDMedian-imputed |
| gpt-5-nano | 6rank change if excluded 3 direct evals | 0rank change if excluded 1 direct eval | -1rank change if excluded 3 direct evals | 0rank change if excluded 7 direct evals | -1rank change if excluded 2 direct evals | 1rank change if excluded 12 direct evals | 0rank change if excluded 5 direct evals | +1.10 SDDirect |
| gpt-5 | -9rank change if excluded 1 direct eval | -10rank change if excluded 6 direct evals | -8rank change if excluded 6 direct evals | -9rank change if excluded 13 direct evals | -13rank change if excluded 5 direct evals | -13rank change if excluded 24 direct evals | -9rank change if excluded 9 direct evals | +1.10 SDDirect |
| gemini-2.5-pro-exp | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -7rank change if excluded 3 direct evals | 98rank change if excluded 1 direct eval | -18rank change if excluded 1 direct eval | Not shown0 direct evals | +0.64 SDDirect |
| gpt-5-pro | 195rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 6rank change if excluded 2 direct evals | -9rank change if excluded 1 direct eval | 5rank change if excluded 2 direct evals | Not shown0 direct evals | +1.10 SDDirect |
| phi-3.5-moe-instruct | 1rank change if excluded 1 direct eval | 0rank change if excluded 4 direct evals | 1rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | 2rank change if excluded 4 direct evals | 2rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| gemini-3.1-pro | Not shown0 direct evals | 14rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | -2rank change if excluded 3 direct evals | Not shown0 direct evals | -2rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | +1.02 SDDirect |
| grok-4.20-multi-agent | Not shown0 direct evals | -31rank change if excluded 1 direct eval | Not shown0 direct evals | 28rank change if excluded 3 direct evals | 67rank change if excluded 1 direct eval | -10rank change if excluded 2 direct evals | Not shown0 direct evals | -0.61 SDDirect |
| claude-opus-4.5 | -14rank change if excluded 2 direct evals | 5rank change if excluded 2 direct evals | 6rank change if excluded 2 direct evals | 4rank change if excluded 12 direct evals | 5rank change if excluded 6 direct evals | 4rank change if excluded 13 direct evals | 11rank change if excluded 8 direct evals | +2.12 SDDirect |
| qwen3.5-397b-a17b | 9rank change if excluded 1 direct eval | 10rank change if excluded 4 direct evals | 7rank change if excluded 1 direct eval | 10rank change if excluded 7 direct evals | -12rank change if excluded 3 direct evals | 7rank change if excluded 7 direct evals | -11rank change if excluded 4 direct evals | -1.00 SDDirect |
| qwen3.5-27b | Not shown0 direct evals | 4rank change if excluded 2 direct evals | 76rank change if excluded 3 direct evals | 2rank change if excluded 4 direct evals | 83rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | -0.65 SDDirect |
| claude-3.7-sonnet | 22rank change if excluded 1 direct eval | 21rank change if excluded 3 direct evals | 10rank change if excluded 3 direct evals | 21rank change if excluded 9 direct evals | 18rank change if excluded 3 direct evals | -6rank change if excluded 15 direct evals | 16rank change if excluded 7 direct evals | +2.08 SDDirect |
| claude-sonnet-4.5 | 1rank change if excluded 3 direct evals | 2rank change if excluded 7 direct evals | 1rank change if excluded 3 direct evals | 0rank change if excluded 13 direct evals | 2rank change if excluded 7 direct evals | 2rank change if excluded 25 direct evals | 3rank change if excluded 13 direct evals | +2.12 SDDirect |
| gpt-5.1 | -2rank change if excluded 1 direct eval | -17rank change if excluded 4 direct evals | -5rank change if excluded 5 direct evals | -5rank change if excluded 11 direct evals | -12rank change if excluded 5 direct evals | -3rank change if excluded 18 direct evals | -2rank change if excluded 8 direct evals | +1.60 SDDirect |
| inkling | 21rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | -14rank change if excluded 4 direct evals | 2rank change if excluded 2 direct evals | 0rank change if excluded 3 direct evals | 0rank change if excluded 2 direct evals | -0.06 SDMedian-imputed |
| gpt-5.1-codex | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -64rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 33rank change if excluded 1 direct eval | +1.60 SDDirect |
| o1-preview | -10rank change if excluded 1 direct eval | 153rank change if excluded 2 direct evals | -10rank change if excluded 1 direct eval | -10rank change if excluded 1 direct eval | Not shown0 direct evals | 154rank change if excluded 2 direct evals | 154rank change if excluded 1 direct eval | +0.50 SDDirect |
| claude-sonnet-5 | -23rank change if excluded 3 direct evals | -12rank change if excluded 3 direct evals | -14rank change if excluded 2 direct evals | -9rank change if excluded 10 direct evals | -18rank change if excluded 2 direct evals | -14rank change if excluded 5 direct evals | 22rank change if excluded 7 direct evals | +2.09 SDDirect |
| llama-3.1-tulu-3-8b | Not shown0 direct evals | -1rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 3 direct evals | 0rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| mimo-v2.5-pro | -33rank change if excluded 1 direct eval | 14rank change if excluded 3 direct evals | Not shown0 direct evals | 7rank change if excluded 7 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 3 direct evals | 3rank change if excluded 2 direct evals | -0.06 SDMedian-imputed |
| claude-sonnet-4 | 22rank change if excluded 1 direct eval | 21rank change if excluded 7 direct evals | 20rank change if excluded 3 direct evals | 21rank change if excluded 11 direct evals | -11rank change if excluded 6 direct evals | 1rank change if excluded 20 direct evals | 20rank change if excluded 14 direct evals | +2.18 SDDirect |
| granite-4.0-h-small | Not shown0 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | 36rank change if excluded 2 direct evals | Not shown0 direct evals | -2rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| mimo-v2-pro | Not shown0 direct evals | -94rank change if excluded 2 direct evals | Not shown0 direct evals | -62rank change if excluded 5 direct evals | 0rank change if excluded 1 direct eval | -5rank change if excluded 2 direct evals | -94rank change if excluded 2 direct evals | -0.06 SDMedian-imputed |
| grok-4.5 | -15rank change if excluded 2 direct evals | -9rank change if excluded 3 direct evals | -1rank change if excluded 1 direct eval | 12rank change if excluded 11 direct evals | 9rank change if excluded 1 direct eval | 9rank change if excluded 4 direct evals | -2rank change if excluded 5 direct evals | -0.61 SDDirect |
| claude-opus-4.1 | 23rank change if excluded 2 direct evals | Not shown0 direct evals | 1rank change if excluded 4 direct evals | -4rank change if excluded 7 direct evals | 0rank change if excluded 4 direct evals | 24rank change if excluded 10 direct evals | 0rank change if excluded 1 direct eval | +2.18 SDDirect |
| gpt-5.4-pro | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 276rank change if excluded 3 direct evals | -9rank change if excluded 1 direct eval | 35rank change if excluded 1 direct eval | Not shown0 direct evals | +1.66 SDDirect |
| minimax-m3 | -8rank change if excluded 3 direct evals | 10rank change if excluded 2 direct evals | Not shown0 direct evals | 9rank change if excluded 7 direct evals | 10rank change if excluded 1 direct eval | 10rank change if excluded 2 direct evals | 10rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| chatgpt-4o | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -27rank change if excluded 1 direct eval | Not shown0 direct evals | 27rank change if excluded 1 direct eval | 161rank change if excluded 1 direct eval | +1.10 SDDirect |
| gemini-3.1-pro-preview | 6rank change if excluded 2 direct evals | -11rank change if excluded 6 direct evals | 13rank change if excluded 6 direct evals | -1rank change if excluded 17 direct evals | 16rank change if excluded 6 direct evals | -22rank change if excluded 13 direct evals | -2rank change if excluded 7 direct evals | +1.02 SDDirect |
| gpt-oss-20b | 6rank change if excluded 1 direct eval | -56rank change if excluded 4 direct evals | 7rank change if excluded 2 direct evals | 3rank change if excluded 4 direct evals | 6rank change if excluded 4 direct evals | 10rank change if excluded 13 direct evals | 15rank change if excluded 3 direct evals | +1.10 SDDirect |
| gemini-3.5-flash | 0rank change if excluded 5 direct evals | 1rank change if excluded 4 direct evals | 3rank change if excluded 2 direct evals | 2rank change if excluded 13 direct evals | 5rank change if excluded 3 direct evals | 2rank change if excluded 9 direct evals | 7rank change if excluded 9 direct evals | +1.22 SDDirect |
| kimi-k2.6 | 33rank change if excluded 2 direct evals | -8rank change if excluded 3 direct evals | 13rank change if excluded 2 direct evals | -4rank change if excluded 12 direct evals | -3rank change if excluded 3 direct evals | -3rank change if excluded 9 direct evals | -3rank change if excluded 6 direct evals | -0.06 SDMedian-imputed |
| qwen3.5-9b | -5rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | 48rank change if excluded 1 direct eval | -3rank change if excluded 2 direct evals | 48rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | -3rank change if excluded 2 direct evals | -0.65 SDDirect |
| glm-4.5-air | -30rank change if excluded 2 direct evals | Not shown0 direct evals | -1rank change if excluded 1 direct eval | -7rank change if excluded 5 direct evals | 25rank change if excluded 2 direct evals | -2rank change if excluded 6 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| gpt-5.4-mini | Not shown0 direct evals | 30rank change if excluded 3 direct evals | -31rank change if excluded 4 direct evals | -6rank change if excluded 12 direct evals | 2rank change if excluded 3 direct evals | 3rank change if excluded 6 direct evals | 1rank change if excluded 6 direct evals | +1.66 SDDirect |
| nova-lite | 258rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 4 direct evals | Not shown0 direct evals | 1rank change if excluded 3 direct evals | 1rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| glm-5.3-flash | 19rank change if excluded 1 direct eval | -5rank change if excluded 1 direct eval | 19rank change if excluded 1 direct eval | -15rank change if excluded 3 direct evals | Not shown0 direct evals | 19rank change if excluded 1 direct eval | 19rank change if excluded 1 direct eval | -0.59 SDDirect |
| gemma-4-26b-a4b-it | 61rank change if excluded 1 direct eval | -29rank change if excluded 2 direct evals | Not shown0 direct evals | -27rank change if excluded 4 direct evals | Not shown0 direct evals | -29rank change if excluded 3 direct evals | -26rank change if excluded 1 direct eval | +1.22 SDDirect |
| qwen3.5-plus | Not shown0 direct evals | 6rank change if excluded 3 direct evals | 6rank change if excluded 3 direct evals | 7rank change if excluded 5 direct evals | 6rank change if excluded 1 direct eval | 7rank change if excluded 5 direct evals | 7rank change if excluded 1 direct eval | -1.00 SDDirect |
| qwen3.7-flash | 22rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 9rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.65 SDDirect |
| palmyra-x-004 | Not shown0 direct evals | Not shown0 direct evals | 18rank change if excluded 1 direct eval | Not shown0 direct evals | 23rank change if excluded 1 direct eval | -11rank change if excluded 5 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| qwen3-235b-a22b-fp8-throughput | Not shown0 direct evals | Not shown0 direct evals | 37rank change if excluded 1 direct eval | Not shown0 direct evals | 42rank change if excluded 1 direct eval | -62rank change if excluded 5 direct evals | Not shown0 direct evals | -1.00 SDDirect |
| nemotron-3.5-lightning | -9rank change if excluded 1 direct eval | -106rank change if excluded 1 direct eval | -16rank change if excluded 1 direct eval | -14rank change if excluded 2 direct evals | Not shown0 direct evals | -16rank change if excluded 1 direct eval | Not shown0 direct evals | -0.06 SDMedian-imputed |
| claude-3.5-haiku | -137rank change if excluded 2 direct evals | 24rank change if excluded 4 direct evals | 21rank change if excluded 3 direct evals | 24rank change if excluded 5 direct evals | 28rank change if excluded 1 direct eval | 29rank change if excluded 9 direct evals | 24rank change if excluded 4 direct evals | +2.08 SDDirect |
| gemma-4-31b-it | -10rank change if excluded 3 direct evals | -20rank change if excluded 4 direct evals | 45rank change if excluded 4 direct evals | -18rank change if excluded 9 direct evals | -18rank change if excluded 2 direct evals | -20rank change if excluded 6 direct evals | -18rank change if excluded 1 direct eval | +1.22 SDDirect |
| qwen3.7-max | -4rank change if excluded 1 direct eval | -35rank change if excluded 2 direct evals | -1rank change if excluded 1 direct eval | 12rank change if excluded 9 direct evals | 0rank change if excluded 2 direct evals | 3rank change if excluded 6 direct evals | 0rank change if excluded 2 direct evals | -0.65 SDDirect |
| mimo-v2.5 | -10rank change if excluded 2 direct evals | 20rank change if excluded 1 direct eval | Not shown0 direct evals | -10rank change if excluded 5 direct evals | -7rank change if excluded 2 direct evals | -10rank change if excluded 2 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| glm-5 | 0rank change if excluded 1 direct eval | 17rank change if excluded 4 direct evals | -29rank change if excluded 2 direct evals | 7rank change if excluded 9 direct evals | 1rank change if excluded 2 direct evals | 2rank change if excluded 6 direct evals | 13rank change if excluded 4 direct evals | -0.57 SDDirect |
| gemini-1.0-pro | 23rank change if excluded 1 direct eval | 23rank change if excluded 2 direct evals | 23rank change if excluded 3 direct evals | 24rank change if excluded 1 direct eval | 42rank change if excluded 1 direct eval | -12rank change if excluded 7 direct evals | Not shown0 direct evals | +0.81 SDDirect |
| glm-5.2 | 13rank change if excluded 3 direct evals | 3rank change if excluded 4 direct evals | 3rank change if excluded 2 direct evals | 3rank change if excluded 13 direct evals | 3rank change if excluded 2 direct evals | 3rank change if excluded 7 direct evals | 3rank change if excluded 6 direct evals | -0.59 SDDirect |
| minimax-m2.7 | 0rank change if excluded 2 direct evals | 30rank change if excluded 6 direct evals | 3rank change if excluded 1 direct eval | -23rank change if excluded 9 direct evals | -22rank change if excluded 3 direct evals | 1rank change if excluded 6 direct evals | 5rank change if excluded 5 direct evals | -0.06 SDMedian-imputed |
| gpt-4.1 | 25rank change if excluded 2 direct evals | -96rank change if excluded 5 direct evals | -100rank change if excluded 6 direct evals | 2rank change if excluded 12 direct evals | 4rank change if excluded 5 direct evals | -96rank change if excluded 20 direct evals | 8rank change if excluded 5 direct evals | +1.10 SDDirect |
| mistral-large | 92rank change if excluded 1 direct eval | 37rank change if excluded 1 direct eval | 37rank change if excluded 1 direct eval | 37rank change if excluded 1 direct eval | 36rank change if excluded 1 direct eval | -82rank change if excluded 5 direct evals | Not shown0 direct evals | -1.19 SDDirect |
| hy3 | -43rank change if excluded 2 direct evals | -9rank change if excluded 2 direct evals | Not shown0 direct evals | 172rank change if excluded 3 direct evals | Not shown0 direct evals | -8rank change if excluded 1 direct eval | -9rank change if excluded 2 direct evals | -0.06 SDMedian-imputed |
| qwen3.8-max | -49rank change if excluded 2 direct evals | -4rank change if excluded 1 direct eval | Not shown0 direct evals | 168rank change if excluded 5 direct evals | -15rank change if excluded 1 direct eval | -15rank change if excluded 1 direct eval | -4rank change if excluded 2 direct evals | -0.65 SDDirect |
| laguna-s-2.1-poolside | -14rank change if excluded 2 direct evals | -52rank change if excluded 1 direct eval | -14rank change if excluded 1 direct eval | -55rank change if excluded 2 direct evals | Not shown0 direct evals | -14rank change if excluded 1 direct eval | Not shown0 direct evals | -0.06 SDMedian-imputed |
| deepseek-v4-flash | 26rank change if excluded 4 direct evals | -4rank change if excluded 4 direct evals | 3rank change if excluded 1 direct eval | 0rank change if excluded 9 direct evals | 1rank change if excluded 3 direct evals | 2rank change if excluded 6 direct evals | 2rank change if excluded 5 direct evals | -1.04 SDDirect |
| palmyra-fin | Not shown0 direct evals | Not shown0 direct evals | -64rank change if excluded 1 direct eval | Not shown0 direct evals | 1rank change if excluded 1 direct eval | 32rank change if excluded 5 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| muse-spark | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 3 direct evals | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 92rank change if excluded 1 direct eval | +0.39 SDDirect |
| o1-mini | 29rank change if excluded 1 direct eval | 29rank change if excluded 2 direct evals | 58rank change if excluded 2 direct evals | -11rank change if excluded 2 direct evals | 7rank change if excluded 1 direct eval | -148rank change if excluded 8 direct evals | 14rank change if excluded 1 direct eval | +0.50 SDDirect |
| qwen-72b-chat | Not shown0 direct evals | 27rank change if excluded 2 direct evals | 26rank change if excluded 2 direct evals | 26rank change if excluded 1 direct eval | Not shown0 direct evals | 122rank change if excluded 4 direct evals | Not shown0 direct evals | -1.00 SDDirect |
| qwen2-72b-instruct | Not shown0 direct evals | 9rank change if excluded 2 direct evals | 8rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 1 direct eval | -6rank change if excluded 9 direct evals | 9rank change if excluded 1 direct eval | -1.00 SDDirect |
| kimi-k2 | 2rank change if excluded 1 direct eval | 6rank change if excluded 4 direct evals | 1rank change if excluded 3 direct evals | -16rank change if excluded 13 direct evals | 6rank change if excluded 2 direct evals | 5rank change if excluded 17 direct evals | 1rank change if excluded 7 direct evals | -0.06 SDMedian-imputed |
| gpt-4.5-preview | Not shown0 direct evals | 34rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | 5rank change if excluded 3 direct evals | 5rank change if excluded 1 direct eval | 1rank change if excluded 6 direct evals | 34rank change if excluded 1 direct eval | +0.50 SDDirect |
| qwen3-max | -41rank change if excluded 1 direct eval | -15rank change if excluded 2 direct evals | -17rank change if excluded 3 direct evals | -23rank change if excluded 5 direct evals | Not shown0 direct evals | -24rank change if excluded 5 direct evals | 5rank change if excluded 1 direct eval | -1.00 SDDirect |
| o1 | Not shown0 direct evals | 8rank change if excluded 1 direct eval | 7rank change if excluded 1 direct eval | -2rank change if excluded 6 direct evals | -10rank change if excluded 4 direct evals | -9rank change if excluded 12 direct evals | 8rank change if excluded 2 direct evals | +0.50 SDDirect |
| internlm2-chat-7b | Not shown0 direct evals | 1rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 4 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| qwen3-4b | Not shown0 direct evals | -8rank change if excluded 1 direct eval | 121rank change if excluded 1 direct eval | 161rank change if excluded 1 direct eval | 121rank change if excluded 1 direct eval | -8rank change if excluded 1 direct eval | Not shown0 direct evals | -1.00 SDDirect |
| gpt-5.3-codex | Not shown0 direct evals | 52rank change if excluded 2 direct evals | Not shown0 direct evals | -24rank change if excluded 4 direct evals | 7rank change if excluded 1 direct eval | 7rank change if excluded 1 direct eval | 18rank change if excluded 1 direct eval | +1.60 SDDirect |
| gemini-1.5-pro | 78rank change if excluded 3 direct evals | 10rank change if excluded 3 direct evals | 12rank change if excluded 3 direct evals | 16rank change if excluded 5 direct evals | 12rank change if excluded 4 direct evals | 12rank change if excluded 14 direct evals | 14rank change if excluded 3 direct evals | +0.81 SDDirect |
| seed-2.0-mini | 69rank change if excluded 1 direct eval | -9rank change if excluded 2 direct evals | -13rank change if excluded 1 direct eval | -9rank change if excluded 2 direct evals | Not shown0 direct evals | -13rank change if excluded 1 direct eval | Not shown0 direct evals | -0.06 SDMedian-imputed |
| deepseek-v3.1 | 125rank change if excluded 2 direct evals | -11rank change if excluded 1 direct eval | -11rank change if excluded 2 direct evals | -19rank change if excluded 5 direct evals | 139rank change if excluded 2 direct evals | -12rank change if excluded 8 direct evals | -11rank change if excluded 1 direct eval | -1.44 SDDirect |
| qwen3.6-27b | Not shown0 direct evals | 50rank change if excluded 1 direct eval | Not shown0 direct evals | -9rank change if excluded 1 direct eval | 50rank change if excluded 1 direct eval | 50rank change if excluded 1 direct eval | -120rank change if excluded 2 direct evals | -0.65 SDDirect |
| gemini-3.5-flash-lite | -36rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 78rank change if excluded 5 direct evals | 151rank change if excluded 3 direct evals | -23rank change if excluded 2 direct evals | -23rank change if excluded 1 direct eval | +1.22 SDDirect |
| o3 | Not shown0 direct evals | 11rank change if excluded 2 direct evals | 7rank change if excluded 4 direct evals | 8rank change if excluded 8 direct evals | 10rank change if excluded 4 direct evals | 11rank change if excluded 15 direct evals | -11rank change if excluded 6 direct evals | +1.10 SDDirect |
| palmyra-x5 | Not shown0 direct evals | Not shown0 direct evals | 12rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 1 direct eval | 7rank change if excluded 5 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| gemini-2.5-pro | 0rank change if excluded 1 direct eval | -7rank change if excluded 9 direct evals | -5rank change if excluded 6 direct evals | 43rank change if excluded 19 direct evals | 1rank change if excluded 5 direct evals | 41rank change if excluded 21 direct evals | -7rank change if excluded 11 direct evals | +0.64 SDDirect |
| internlm-chat-20b | Not shown0 direct evals | -1rank change if excluded 4 direct evals | -3rank change if excluded 4 direct evals | -3rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 4 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| gpt-oss-safeguard-20b | -43rank change if excluded 1 direct eval | -115rank change if excluded 2 direct evals | Not shown0 direct evals | -83rank change if excluded 1 direct eval | Not shown0 direct evals | 99rank change if excluded 2 direct evals | -83rank change if excluded 1 direct eval | +1.60 SDDirect |
| nemotron-3-ultra-550b-a55b | 19rank change if excluded 2 direct evals | -20rank change if excluded 2 direct evals | -12rank change if excluded 1 direct eval | 17rank change if excluded 6 direct evals | Not shown0 direct evals | -12rank change if excluded 2 direct evals | -12rank change if excluded 2 direct evals | -0.06 SDMedian-imputed |
| grok-4.3 | -4rank change if excluded 2 direct evals | -14rank change if excluded 5 direct evals | -1rank change if excluded 3 direct evals | -3rank change if excluded 12 direct evals | -3rank change if excluded 2 direct evals | -12rank change if excluded 9 direct evals | -10rank change if excluded 7 direct evals | -0.61 SDDirect |
| gpt-5.2-codex | Not shown0 direct evals | -8rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -8rank change if excluded 1 direct eval | +1.60 SDDirect |
| ox-alpha | -33rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 161rank change if excluded 3 direct evals | 162rank change if excluded 1 direct eval | -18rank change if excluded 1 direct eval | 160rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| gpt-oss-120b | -21rank change if excluded 3 direct evals | -2rank change if excluded 6 direct evals | 7rank change if excluded 6 direct evals | 3rank change if excluded 12 direct evals | -22rank change if excluded 3 direct evals | 6rank change if excluded 18 direct evals | 3rank change if excluded 5 direct evals | +1.10 SDDirect |
| ministral-3-3b | 276rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 6rank change if excluded 1 direct eval | Not shown0 direct evals | 5rank change if excluded 1 direct eval | Not shown0 direct evals | -1.19 SDDirect |
| gemini-3-pro-preview | -10rank change if excluded 1 direct eval | -8rank change if excluded 2 direct evals | 2rank change if excluded 2 direct evals | -32rank change if excluded 12 direct evals | 2rank change if excluded 3 direct evals | 1rank change if excluded 14 direct evals | -8rank change if excluded 6 direct evals | +1.02 SDDirect |
| gemini-1.5-flash | 43rank change if excluded 2 direct evals | 24rank change if excluded 2 direct evals | 22rank change if excluded 2 direct evals | 31rank change if excluded 1 direct eval | 25rank change if excluded 3 direct evals | 29rank change if excluded 13 direct evals | 31rank change if excluded 4 direct evals | +0.81 SDDirect |
| qwen3-30b-a3b | 133rank change if excluded 1 direct eval | -21rank change if excluded 1 direct eval | Not shown0 direct evals | -21rank change if excluded 2 direct evals | Not shown0 direct evals | -21rank change if excluded 3 direct evals | -21rank change if excluded 1 direct eval | -1.00 SDDirect |
| internlm2-chat-20b | Not shown0 direct evals | -5rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | Not shown0 direct evals | -4rank change if excluded 5 direct evals | 4rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| deepseek-v3.2 | 12rank change if excluded 4 direct evals | -7rank change if excluded 11 direct evals | -3rank change if excluded 3 direct evals | -7rank change if excluded 16 direct evals | -8rank change if excluded 2 direct evals | -11rank change if excluded 12 direct evals | 2rank change if excluded 9 direct evals | -1.00 SDDirect |
| deepseek-v3.2-exp | -107rank change if excluded 1 direct eval | -18rank change if excluded 1 direct eval | Not shown0 direct evals | 34rank change if excluded 2 direct evals | Not shown0 direct evals | -21rank change if excluded 3 direct evals | 1rank change if excluded 2 direct evals | -1.00 SDDirect |
| glm-5.1 | -9rank change if excluded 2 direct evals | -30rank change if excluded 4 direct evals | -2rank change if excluded 1 direct eval | -1rank change if excluded 9 direct evals | -1rank change if excluded 1 direct eval | -2rank change if excluded 4 direct evals | -33rank change if excluded 8 direct evals | -0.59 SDDirect |
| kimi-k2.7-code | Not shown0 direct evals | 197rank change if excluded 2 direct evals | Not shown0 direct evals | -41rank change if excluded 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | -41rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| qwen2.5-72b-instruct | 182rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | -4rank change if excluded 3 direct evals | 2rank change if excluded 4 direct evals | 0rank change if excluded 3 direct evals | 3rank change if excluded 12 direct evals | 3rank change if excluded 2 direct evals | -1.00 SDDirect |
| gpt-4.1-mini | 6rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | 2rank change if excluded 3 direct evals | 1rank change if excluded 4 direct evals | 23rank change if excluded 4 direct evals | 22rank change if excluded 16 direct evals | -147rank change if excluded 3 direct evals | +1.10 SDDirect |
| ernie-4.5-300b-a47b | Not shown0 direct evals | 15rank change if excluded 1 direct eval | -6rank change if excluded 1 direct eval | -125rank change if excluded 3 direct evals | 41rank change if excluded 1 direct eval | -4rank change if excluded 2 direct evals | -6rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| minimax-m2 | -13rank change if excluded 1 direct eval | 13rank change if excluded 3 direct evals | -3rank change if excluded 1 direct eval | -8rank change if excluded 4 direct evals | Not shown0 direct evals | -1rank change if excluded 4 direct evals | -3rank change if excluded 2 direct evals | -0.06 SDMedian-imputed |
| qwen3-235b-a22b | -16rank change if excluded 3 direct evals | 17rank change if excluded 7 direct evals | -132rank change if excluded 4 direct evals | -133rank change if excluded 8 direct evals | 26rank change if excluded 4 direct evals | 26rank change if excluded 18 direct evals | 11rank change if excluded 8 direct evals | -1.00 SDDirect |
| doubao-seed-1.6 | Not shown0 direct evals | -6rank change if excluded 4 direct evals | 0rank change if excluded 3 direct evals | -2rank change if excluded 4 direct evals | -3rank change if excluded 1 direct eval | 0rank change if excluded 5 direct evals | -2rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| grok-4-fast | -3rank change if excluded 1 direct eval | Not shown0 direct evals | 34rank change if excluded 2 direct evals | -73rank change if excluded 8 direct evals | -3rank change if excluded 1 direct eval | 2rank change if excluded 7 direct evals | -3rank change if excluded 2 direct evals | -0.30 SDDirect |
| minimax-m2.5 | -15rank change if excluded 1 direct eval | 23rank change if excluded 3 direct evals | -3rank change if excluded 2 direct evals | -26rank change if excluded 8 direct evals | -14rank change if excluded 1 direct eval | -13rank change if excluded 3 direct evals | -26rank change if excluded 3 direct evals | -0.06 SDMedian-imputed |
| grok-3-mini | 37rank change if excluded 1 direct eval | -50rank change if excluded 2 direct evals | 39rank change if excluded 3 direct evals | 38rank change if excluded 4 direct evals | 38rank change if excluded 1 direct eval | 36rank change if excluded 11 direct evals | -29rank change if excluded 2 direct evals | -0.04 SDDirect |
| longcat-2.0 | Not shown0 direct evals | -16rank change if excluded 1 direct eval | -14rank change if excluded 1 direct eval | -14rank change if excluded 3 direct evals | -12rank change if excluded 1 direct eval | -12rank change if excluded 2 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| olmo-3-7b-instruct | Not shown0 direct evals | 85rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| nova-2.0-lite | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 8rank change if excluded 3 direct evals | Not shown0 direct evals | 15rank change if excluded 1 direct eval | Not shown0 direct evals | -0.06 SDMedian-imputed |
| phi-4-reasoning-plus | Not shown0 direct evals | 23rank change if excluded 1 direct eval | -28rank change if excluded 1 direct eval | Not shown0 direct evals | 1rank change if excluded 1 direct eval | 15rank change if excluded 2 direct evals | 17rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| deepseek-v4-pro | 192rank change if excluded 3 direct evals | -17rank change if excluded 4 direct evals | -13rank change if excluded 2 direct evals | -17rank change if excluded 12 direct evals | -13rank change if excluded 2 direct evals | -14rank change if excluded 7 direct evals | -14rank change if excluded 6 direct evals | -1.04 SDDirect |
| claude-3.5-sonnet | -58rank change if excluded 4 direct evals | -8rank change if excluded 4 direct evals | -9rank change if excluded 3 direct evals | -7rank change if excluded 9 direct evals | -7rank change if excluded 4 direct evals | 0rank change if excluded 19 direct evals | 7rank change if excluded 8 direct evals | +2.08 SDDirect |
| claude-3-opus | 33rank change if excluded 5 direct evals | 9rank change if excluded 4 direct evals | 11rank change if excluded 4 direct evals | 13rank change if excluded 5 direct evals | 13rank change if excluded 4 direct evals | 18rank change if excluded 15 direct evals | 6rank change if excluded 7 direct evals | +2.08 SDDirect |
| gpt-4-turbo | 3rank change if excluded 1 direct eval | 3rank change if excluded 6 direct evals | 6rank change if excluded 4 direct evals | 2rank change if excluded 5 direct evals | 3rank change if excluded 3 direct evals | 6rank change if excluded 19 direct evals | 2rank change if excluded 7 direct evals | +0.50 SDDirect |
| qwen3-32b | 272rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | 3rank change if excluded 6 direct evals | 3rank change if excluded 1 direct eval | -1.00 SDDirect |
| qwen3.6-plus | -8rank change if excluded 1 direct eval | -6rank change if excluded 4 direct evals | -8rank change if excluded 1 direct eval | -5rank change if excluded 6 direct evals | -7rank change if excluded 1 direct eval | -8rank change if excluded 3 direct evals | -4rank change if excluded 2 direct evals | -0.65 SDDirect |
| olmo-3.1-32b-instruct | Not shown0 direct evals | 6rank change if excluded 1 direct eval | Not shown0 direct evals | -14rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| grok-4 | 22rank change if excluded 1 direct eval | -10rank change if excluded 5 direct evals | 16rank change if excluded 7 direct evals | 23rank change if excluded 13 direct evals | 22rank change if excluded 2 direct evals | -13rank change if excluded 16 direct evals | -10rank change if excluded 5 direct evals | -0.04 SDDirect |
| nemotron-3-nano-omni-30b-a3b | 253rank change if excluded 1 direct eval | Not shown0 direct evals | -22rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | Not shown0 direct evals | -1rank change if excluded 1 direct eval | -21rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| deepseek-v3 | -30rank change if excluded 4 direct evals | 1rank change if excluded 4 direct evals | 2rank change if excluded 7 direct evals | 2rank change if excluded 10 direct evals | 4rank change if excluded 2 direct evals | 0rank change if excluded 16 direct evals | 4rank change if excluded 6 direct evals | -1.44 SDDirect |
| grok-4.20 | -10rank change if excluded 1 direct eval | -15rank change if excluded 6 direct evals | -10rank change if excluded 3 direct evals | 1rank change if excluded 14 direct evals | -2rank change if excluded 5 direct evals | -5rank change if excluded 10 direct evals | -10rank change if excluded 7 direct evals | -0.61 SDDirect |
| gemma-4-e4b | Not shown0 direct evals | -100rank change if excluded 1 direct eval | Not shown0 direct evals | 221rank change if excluded 1 direct eval | Not shown0 direct evals | -101rank change if excluded 2 direct evals | -5rank change if excluded 1 direct eval | +1.22 SDDirect |
| qwen-14b-chat | Not shown0 direct evals | 122rank change if excluded 5 direct evals | -3rank change if excluded 5 direct evals | -3rank change if excluded 1 direct eval | Not shown0 direct evals | 61rank change if excluded 8 direct evals | Not shown0 direct evals | -1.00 SDDirect |
| qwen3.6-max-preview | Not shown0 direct evals | -25rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | 7rank change if excluded 2 direct evals | -0.65 SDDirect |
| qwen3.8-27b | -249rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 5rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 1 direct eval | -0.65 SDDirect |
| phi-4 | Not shown0 direct evals | 9rank change if excluded 2 direct evals | -13rank change if excluded 1 direct eval | -198rank change if excluded 4 direct evals | Not shown0 direct evals | -13rank change if excluded 3 direct evals | 0rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| gemini-2.5-flash | -2rank change if excluded 2 direct evals | 3rank change if excluded 6 direct evals | 0rank change if excluded 4 direct evals | 1rank change if excluded 13 direct evals | 5rank change if excluded 3 direct evals | -19rank change if excluded 19 direct evals | 5rank change if excluded 10 direct evals | +0.64 SDDirect |
| qwen2.5-32b-instruct | Not shown0 direct evals | 25rank change if excluded 1 direct eval | Not shown0 direct evals | -59rank change if excluded 2 direct evals | 24rank change if excluded 1 direct eval | 25rank change if excluded 2 direct evals | 16rank change if excluded 2 direct evals | -1.00 SDDirect |
| phi-3-mini-4k-instruct | Not shown0 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -3rank change if excluded 4 direct evals | 34rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| qwen3.7-plus | -4rank change if excluded 1 direct eval | 4rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | -3rank change if excluded 5 direct evals | Not shown0 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -0.65 SDDirect |
| qwen3-max-preview | Not shown0 direct evals | Not shown0 direct evals | 242rank change if excluded 1 direct eval | -41rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -1.00 SDDirect |
| gpt-4o | 6rank change if excluded 3 direct evals | 6rank change if excluded 12 direct evals | 11rank change if excluded 9 direct evals | 5rank change if excluded 21 direct evals | 11rank change if excluded 6 direct evals | 1rank change if excluded 37 direct evals | -1rank change if excluded 20 direct evals | +0.50 SDDirect |
| grok-4.6 | -181rank change if excluded 2 direct evals | 19rank change if excluded 1 direct eval | Not shown0 direct evals | 21rank change if excluded 6 direct evals | 19rank change if excluded 1 direct eval | 19rank change if excluded 2 direct evals | 19rank change if excluded 6 direct evals | -0.61 SDDirect |
| kimi-k2.5 | -20rank change if excluded 2 direct evals | 16rank change if excluded 7 direct evals | -22rank change if excluded 4 direct evals | -18rank change if excluded 15 direct evals | -21rank change if excluded 3 direct evals | -20rank change if excluded 13 direct evals | -20rank change if excluded 10 direct evals | -0.06 SDMedian-imputed |
| qwen3-next-80b-a3b | Not shown0 direct evals | 3rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | -5rank change if excluded 3 direct evals | 25rank change if excluded 2 direct evals | 51rank change if excluded 9 direct evals | 3rank change if excluded 1 direct eval | -1.00 SDDirect |
| gemini-2.0-pro-preview | Not shown0 direct evals | Not shown0 direct evals | 69rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 1 direct eval | -38rank change if excluded 5 direct evals | Not shown0 direct evals | +0.81 SDDirect |
| granite-4.0-small-guardian | Not shown0 direct evals | Not shown0 direct evals | -45rank change if excluded 1 direct eval | Not shown0 direct evals | -44rank change if excluded 1 direct eval | 15rank change if excluded 5 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| nemotron-3-super-120b-a12b | -42rank change if excluded 1 direct eval | -16rank change if excluded 2 direct evals | Not shown0 direct evals | -24rank change if excluded 5 direct evals | -99rank change if excluded 3 direct evals | -10rank change if excluded 5 direct evals | -24rank change if excluded 2 direct evals | -0.06 SDMedian-imputed |
| text-davinci-003 | Not shown0 direct evals | 14rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | 14rank change if excluded 1 direct eval | Not shown0 direct evals | 14rank change if excluded 3 direct evals | Not shown0 direct evals | +0.50 SDDirect |
| qwen3-235b-a22b-instruct | Not shown0 direct evals | 8rank change if excluded 1 direct eval | Not shown0 direct evals | 26rank change if excluded 2 direct evals | Not shown0 direct evals | 8rank change if excluded 1 direct eval | 8rank change if excluded 1 direct eval | -1.00 SDDirect |
| mistral-medium | Not shown0 direct evals | -18rank change if excluded 1 direct eval | Not shown0 direct evals | 42rank change if excluded 1 direct eval | 42rank change if excluded 1 direct eval | 49rank change if excluded 2 direct evals | -18rank change if excluded 1 direct eval | -1.19 SDDirect |
| gpt-4 | 46rank change if excluded 1 direct eval | 48rank change if excluded 6 direct evals | 45rank change if excluded 6 direct evals | -28rank change if excluded 2 direct evals | -28rank change if excluded 2 direct evals | 41rank change if excluded 14 direct evals | -27rank change if excluded 1 direct eval | +0.50 SDDirect |
| mercury-2 | Not shown0 direct evals | 196rank change if excluded 1 direct eval | -15rank change if excluded 1 direct eval | -110rank change if excluded 4 direct evals | Not shown0 direct evals | -15rank change if excluded 1 direct eval | Not shown0 direct evals | -0.06 SDMedian-imputed |
| glm-4.7-flash | 265rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -17rank change if excluded 3 direct evals | Not shown0 direct evals | -11rank change if excluded 1 direct eval | Not shown0 direct evals | -0.57 SDDirect |
| qwen3.5-flash | -84rank change if excluded 1 direct eval | 68rank change if excluded 1 direct eval | -113rank change if excluded 2 direct evals | -96rank change if excluded 3 direct evals | Not shown0 direct evals | 25rank change if excluded 2 direct evals | Not shown0 direct evals | -0.65 SDDirect |
| qwen1.5-32b-chat | Not shown0 direct evals | 27rank change if excluded 1 direct eval | 27rank change if excluded 1 direct eval | 27rank change if excluded 2 direct evals | -18rank change if excluded 1 direct eval | 27rank change if excluded 4 direct evals | 8rank change if excluded 1 direct eval | -1.00 SDDirect |
| deepseek-v4-flash-0731 | -214rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 7rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 7rank change if excluded 1 direct eval | -1.04 SDDirect |
| baichuan-2-13b-chat | Not shown0 direct evals | 38rank change if excluded 6 direct evals | 19rank change if excluded 6 direct evals | -1rank change if excluded 2 direct evals | Not shown0 direct evals | 27rank change if excluded 13 direct evals | 3rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| gemini-3.1-flash-lite | -70rank change if excluded 3 direct evals | -14rank change if excluded 5 direct evals | -22rank change if excluded 5 direct evals | -12rank change if excluded 12 direct evals | -6rank change if excluded 5 direct evals | -13rank change if excluded 9 direct evals | -13rank change if excluded 5 direct evals | +1.22 SDDirect |
| solar-pro-4 | -114rank change if excluded 2 direct evals | 29rank change if excluded 1 direct eval | 29rank change if excluded 1 direct eval | 30rank change if excluded 2 direct evals | Not shown0 direct evals | 29rank change if excluded 1 direct eval | Not shown0 direct evals | -0.06 SDMedian-imputed |
| grok-3 | -19rank change if excluded 2 direct evals | 0rank change if excluded 2 direct evals | -1rank change if excluded 3 direct evals | 0rank change if excluded 5 direct evals | 1rank change if excluded 1 direct eval | -2rank change if excluded 11 direct evals | 1rank change if excluded 2 direct evals | -0.04 SDDirect |
| o3-mini | Not shown0 direct evals | 5rank change if excluded 2 direct evals | 3rank change if excluded 3 direct evals | 2rank change if excluded 7 direct evals | 1rank change if excluded 3 direct evals | 21rank change if excluded 15 direct evals | 5rank change if excluded 5 direct evals | +0.50 SDDirect |
| qwen1.5-14b-chat | Not shown0 direct evals | 11rank change if excluded 3 direct evals | 13rank change if excluded 2 direct evals | 13rank change if excluded 3 direct evals | Not shown0 direct evals | 42rank change if excluded 6 direct evals | 13rank change if excluded 2 direct evals | -1.00 SDDirect |
| nova-premier | -84rank change if excluded 1 direct eval | 53rank change if excluded 1 direct eval | Not shown0 direct evals | -29rank change if excluded 1 direct eval | Not shown0 direct evals | 53rank change if excluded 2 direct evals | -28rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| gpt-3.5-turbo | 4rank change if excluded 2 direct evals | 6rank change if excluded 7 direct evals | 22rank change if excluded 10 direct evals | 7rank change if excluded 4 direct evals | 4rank change if excluded 3 direct evals | -3rank change if excluded 20 direct evals | 6rank change if excluded 3 direct evals | +0.50 SDDirect |
| llama-3.3-70b-instruct | 98rank change if excluded 6 direct evals | 4rank change if excluded 6 direct evals | 5rank change if excluded 5 direct evals | 3rank change if excluded 13 direct evals | 2rank change if excluded 5 direct evals | 1rank change if excluded 17 direct evals | 5rank change if excluded 3 direct evals | -1.23 SDDirect |
| qwen3-30b-a3b-instruct | -1rank change if excluded 1 direct eval | 48rank change if excluded 2 direct evals | -55rank change if excluded 2 direct evals | -1rank change if excluded 2 direct evals | Not shown0 direct evals | 4rank change if excluded 2 direct evals | 5rank change if excluded 1 direct eval | -1.00 SDDirect |
| llama-4-maverick | 2rank change if excluded 2 direct evals | 2rank change if excluded 11 direct evals | 0rank change if excluded 5 direct evals | 0rank change if excluded 14 direct evals | 1rank change if excluded 2 direct evals | 0rank change if excluded 22 direct evals | 3rank change if excluded 6 direct evals | -0.41 SDDirect |
| hy3-preview | 135rank change if excluded 1 direct eval | -29rank change if excluded 2 direct evals | Not shown0 direct evals | -23rank change if excluded 4 direct evals | Not shown0 direct evals | -13rank change if excluded 1 direct eval | -13rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| claude-3-haiku | 10rank change if excluded 1 direct eval | 10rank change if excluded 2 direct evals | 9rank change if excluded 3 direct evals | -53rank change if excluded 4 direct evals | -41rank change if excluded 4 direct evals | 10rank change if excluded 12 direct evals | 3rank change if excluded 3 direct evals | +2.08 SDDirect |
| deepseek-r1 | 10rank change if excluded 2 direct evals | 3rank change if excluded 9 direct evals | 0rank change if excluded 11 direct evals | 1rank change if excluded 18 direct evals | -2rank change if excluded 6 direct evals | -3rank change if excluded 26 direct evals | 0rank change if excluded 9 direct evals | -1.44 SDDirect |
| o4-mini | Not shown0 direct evals | -3rank change if excluded 3 direct evals | -4rank change if excluded 3 direct evals | -5rank change if excluded 12 direct evals | -4rank change if excluded 4 direct evals | -1rank change if excluded 17 direct evals | -3rank change if excluded 7 direct evals | +1.10 SDDirect |
| granite-4.0-small | Not shown0 direct evals | Not shown0 direct evals | -40rank change if excluded 1 direct eval | Not shown0 direct evals | -8rank change if excluded 1 direct eval | 10rank change if excluded 5 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| nemotron-3-nano-30b-a3b | 230rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -37rank change if excluded 5 direct evals | -39rank change if excluded 1 direct eval | -37rank change if excluded 3 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| qwen3-coder-480b-a35b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 5rank change if excluded 3 direct evals | Not shown0 direct evals | 1rank change if excluded 1 direct eval | Not shown0 direct evals | -1.00 SDDirect |
| glm-4-9b-chat | Not shown0 direct evals | 36rank change if excluded 2 direct evals | 35rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -22rank change if excluded 8 direct evals | 27rank change if excluded 2 direct evals | -0.06 SDDirect |
| gemini-2.0-flash | -1rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | 3rank change if excluded 3 direct evals | 0rank change if excluded 8 direct evals | 4rank change if excluded 1 direct eval | 6rank change if excluded 11 direct evals | -31rank change if excluded 3 direct evals | +0.81 SDDirect |
| qwen1.5-72b-chat | Not shown0 direct evals | 20rank change if excluded 2 direct evals | 20rank change if excluded 3 direct evals | 18rank change if excluded 4 direct evals | -1rank change if excluded 2 direct evals | 19rank change if excluded 12 direct evals | 0rank change if excluded 2 direct evals | -1.00 SDDirect |
| yi-34b-chat | Not shown0 direct evals | 18rank change if excluded 4 direct evals | 17rank change if excluded 4 direct evals | 7rank change if excluded 4 direct evals | Not shown0 direct evals | -45rank change if excluded 9 direct evals | 18rank change if excluded 2 direct evals | -0.06 SDMedian-imputed |
| llama-3.2-instruct-11b-vision | Not shown0 direct evals | -9rank change if excluded 1 direct eval | Not shown0 direct evals | -4rank change if excluded 1 direct eval | Not shown0 direct evals | 165rank change if excluded 2 direct evals | 118rank change if excluded 1 direct eval | -1.23 SDDirect |
| gpt-5.4-nano | Not shown0 direct evals | 8rank change if excluded 1 direct eval | -196rank change if excluded 2 direct evals | 18rank change if excluded 9 direct evals | -187rank change if excluded 2 direct evals | 8rank change if excluded 4 direct evals | 8rank change if excluded 4 direct evals | +1.66 SDDirect |
| command-r-plus | 13rank change if excluded 1 direct eval | 12rank change if excluded 2 direct evals | -12rank change if excluded 2 direct evals | 13rank change if excluded 2 direct evals | -12rank change if excluded 1 direct eval | -14rank change if excluded 8 direct evals | 46rank change if excluded 3 direct evals | -0.06 SDMedian-imputed |
| mistral-small-3.1 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -8rank change if excluded 2 direct evals | Not shown0 direct evals | -16rank change if excluded 4 direct evals | Not shown0 direct evals | -1.19 SDDirect |
| gemma-2b-it | -15rank change if excluded 1 direct eval | -148rank change if excluded 3 direct evals | -15rank change if excluded 2 direct evals | 15rank change if excluded 1 direct eval | Not shown0 direct evals | 6rank change if excluded 4 direct evals | 15rank change if excluded 1 direct eval | +0.81 SDDirect |
| nova-micro | -249rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | Not shown0 direct evals | 3rank change if excluded 2 direct evals | Not shown0 direct evals | 33rank change if excluded 3 direct evals | 33rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| qwen1.5-7b-chat | Not shown0 direct evals | 4rank change if excluded 2 direct evals | 4rank change if excluded 2 direct evals | 4rank change if excluded 3 direct evals | 5rank change if excluded 1 direct eval | 19rank change if excluded 9 direct evals | 6rank change if excluded 1 direct eval | -1.00 SDDirect |
| step-3.5-flash | -163rank change if excluded 1 direct eval | 32rank change if excluded 2 direct evals | 5rank change if excluded 1 direct eval | -5rank change if excluded 5 direct evals | Not shown0 direct evals | 5rank change if excluded 1 direct eval | 7rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| mistral-medium-3.5 | Not shown0 direct evals | -74rank change if excluded 1 direct eval | 139rank change if excluded 2 direct evals | -5rank change if excluded 5 direct evals | 101rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | Not shown0 direct evals | -1.19 SDDirect |
| internlm-chat-7b | Not shown0 direct evals | -147rank change if excluded 5 direct evals | 5rank change if excluded 5 direct evals | 6rank change if excluded 1 direct eval | Not shown0 direct evals | -147rank change if excluded 5 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| llama-3.1-405b-instruct | Not shown0 direct evals | -2rank change if excluded 3 direct evals | 5rank change if excluded 3 direct evals | -1rank change if excluded 7 direct evals | -2rank change if excluded 3 direct evals | 6rank change if excluded 17 direct evals | 30rank change if excluded 4 direct evals | -1.23 SDDirect |
| gpt-4.1-nano | Not shown0 direct evals | -22rank change if excluded 1 direct eval | -58rank change if excluded 2 direct evals | -3rank change if excluded 3 direct evals | -3rank change if excluded 2 direct evals | 90rank change if excluded 12 direct evals | -3rank change if excluded 1 direct eval | +1.10 SDDirect |
| gemini-3-flash-preview | -6rank change if excluded 2 direct evals | -7rank change if excluded 5 direct evals | -8rank change if excluded 3 direct evals | -2rank change if excluded 13 direct evals | -28rank change if excluded 3 direct evals | -6rank change if excluded 9 direct evals | -1rank change if excluded 6 direct evals | +1.02 SDDirect |
| gemma-7b-it | -70rank change if excluded 1 direct eval | -73rank change if excluded 4 direct evals | -70rank change if excluded 2 direct evals | -1rank change if excluded 1 direct eval | 16rank change if excluded 1 direct eval | 2rank change if excluded 6 direct evals | -2rank change if excluded 1 direct eval | +0.81 SDDirect |
| grok-4.1-fast | -41rank change if excluded 2 direct evals | -37rank change if excluded 7 direct evals | -43rank change if excluded 3 direct evals | -33rank change if excluded 12 direct evals | -41rank change if excluded 2 direct evals | -37rank change if excluded 11 direct evals | -39rank change if excluded 5 direct evals | -0.30 SDDirect |
| claude-2.1 | Not shown0 direct evals | Not shown0 direct evals | -7rank change if excluded 1 direct eval | -7rank change if excluded 2 direct evals | -246rank change if excluded 2 direct evals | 4rank change if excluded 5 direct evals | Not shown0 direct evals | +2.08 SDDirect |
| mistral-small-4 | -58rank change if excluded 1 direct eval | 0rank change if excluded 2 direct evals | 85rank change if excluded 2 direct evals | -1rank change if excluded 5 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | Not shown0 direct evals | -1.19 SDDirect |
| llama-3.2-90b-vision-instruct | Not shown0 direct evals | -37rank change if excluded 1 direct eval | Not shown0 direct evals | -42rank change if excluded 1 direct eval | Not shown0 direct evals | 53rank change if excluded 2 direct evals | -73rank change if excluded 2 direct evals | -1.23 SDDirect |
| deepseek-llm-7b-chat | Not shown0 direct evals | 28rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 38rank change if excluded 3 direct evals | Not shown0 direct evals | -1.44 SDDirect |
| gemini-2.0-flash-lite | -113rank change if excluded 1 direct eval | 11rank change if excluded 1 direct eval | 14rank change if excluded 2 direct evals | 11rank change if excluded 2 direct evals | Not shown0 direct evals | 12rank change if excluded 4 direct evals | Not shown0 direct evals | +0.81 SDDirect |
| glm-4.7 | -12rank change if excluded 1 direct eval | -12rank change if excluded 1 direct eval | Not shown0 direct evals | -20rank change if excluded 6 direct evals | -15rank change if excluded 1 direct eval | -10rank change if excluded 4 direct evals | -12rank change if excluded 2 direct evals | -0.59 SDDirect |
| step-3.7-flash | 59rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 7rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| llama-2-13b-chat | Not shown0 direct evals | 3rank change if excluded 3 direct evals | 0rank change if excluded 2 direct evals | 4rank change if excluded 2 direct evals | -147rank change if excluded 1 direct eval | 4rank change if excluded 10 direct evals | -2rank change if excluded 2 direct evals | -1.23 SDDirect |
| glm-4.6 | -13rank change if excluded 1 direct eval | -20rank change if excluded 4 direct evals | -13rank change if excluded 2 direct evals | 8rank change if excluded 6 direct evals | Not shown0 direct evals | -17rank change if excluded 5 direct evals | -14rank change if excluded 2 direct evals | -0.57 SDDirect |
| mimo-v2-flash | Not shown0 direct evals | -34rank change if excluded 3 direct evals | 90rank change if excluded 1 direct eval | -25rank change if excluded 7 direct evals | -25rank change if excluded 1 direct eval | -43rank change if excluded 3 direct evals | -25rank change if excluded 2 direct evals | -0.06 SDMedian-imputed |
| magistral-medium | Not shown0 direct evals | 1rank change if excluded 2 direct evals | -17rank change if excluded 2 direct evals | 5rank change if excluded 4 direct evals | 18rank change if excluded 1 direct eval | -17rank change if excluded 6 direct evals | -16rank change if excluded 1 direct eval | -1.19 SDDirect |
| yi-6b-chat | Not shown0 direct evals | 2rank change if excluded 3 direct evals | 12rank change if excluded 3 direct evals | 10rank change if excluded 3 direct evals | Not shown0 direct evals | 10rank change if excluded 5 direct evals | 10rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| gemma-2-9b-it | -1rank change if excluded 1 direct eval | -4rank change if excluded 3 direct evals | -7rank change if excluded 4 direct evals | 7rank change if excluded 3 direct evals | 8rank change if excluded 1 direct eval | -4rank change if excluded 6 direct evals | 7rank change if excluded 2 direct evals | +0.81 SDDirect |
| gemini-2.5-flash-lite | 34rank change if excluded 3 direct evals | -1rank change if excluded 2 direct evals | 32rank change if excluded 2 direct evals | -1rank change if excluded 9 direct evals | 32rank change if excluded 1 direct eval | -50rank change if excluded 10 direct evals | -1rank change if excluded 4 direct evals | +0.64 SDDirect |
| gemma-3-12b | Not shown0 direct evals | 2rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | 2rank change if excluded 3 direct evals | 2rank change if excluded 1 direct eval | 54rank change if excluded 7 direct evals | 2rank change if excluded 1 direct eval | +0.81 SDDirect |
| deepseek-v3.1-terminus | Not shown0 direct evals | -17rank change if excluded 1 direct eval | Not shown0 direct evals | 9rank change if excluded 2 direct evals | Not shown0 direct evals | -16rank change if excluded 2 direct evals | -5rank change if excluded 2 direct evals | -1.00 SDDirect |
| granite-4.0-micro-guardian | Not shown0 direct evals | Not shown0 direct evals | -68rank change if excluded 1 direct eval | Not shown0 direct evals | -13rank change if excluded 1 direct eval | 90rank change if excluded 5 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| llama-3-70b-instruct | 6rank change if excluded 1 direct eval | 6rank change if excluded 2 direct evals | 6rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | 9rank change if excluded 2 direct evals | -2rank change if excluded 9 direct evals | -49rank change if excluded 2 direct evals | -1.23 SDDirect |
| glm-4.5 | 4rank change if excluded 1 direct eval | 6rank change if excluded 3 direct evals | 4rank change if excluded 2 direct evals | 10rank change if excluded 6 direct evals | 4rank change if excluded 2 direct evals | 6rank change if excluded 4 direct evals | 4rank change if excluded 2 direct evals | -1.02 SDDirect |
| claude-3-sonnet | -12rank change if excluded 1 direct eval | -11rank change if excluded 2 direct evals | -12rank change if excluded 3 direct evals | -15rank change if excluded 2 direct evals | -10rank change if excluded 4 direct evals | 1rank change if excluded 11 direct evals | -15rank change if excluded 3 direct evals | +2.08 SDDirect |
| llama-2-7b-chat | 3rank change if excluded 1 direct eval | 3rank change if excluded 4 direct evals | 3rank change if excluded 5 direct evals | 7rank change if excluded 2 direct evals | -73rank change if excluded 2 direct evals | 6rank change if excluded 14 direct evals | 5rank change if excluded 2 direct evals | -1.23 SDDirect |
| gemini-2.0-flash-lite-preview | Not shown0 direct evals | -10rank change if excluded 1 direct eval | 7rank change if excluded 1 direct eval | Not shown0 direct evals | 6rank change if excluded 1 direct eval | -13rank change if excluded 6 direct evals | 6rank change if excluded 1 direct eval | +0.81 SDDirect |
| glm-5-turbo | Not shown0 direct evals | -175rank change if excluded 3 direct evals | -7rank change if excluded 1 direct eval | 17rank change if excluded 5 direct evals | -7rank change if excluded 1 direct eval | -9rank change if excluded 3 direct evals | -14rank change if excluded 3 direct evals | -0.59 SDDirect |
| llama-3-8b-instruct | 7rank change if excluded 1 direct eval | 7rank change if excluded 4 direct evals | 2rank change if excluded 2 direct evals | -1rank change if excluded 1 direct eval | -28rank change if excluded 2 direct evals | 0rank change if excluded 12 direct evals | -1rank change if excluded 1 direct eval | -1.23 SDDirect |
| qwen3.5-122b-a10b | Not shown0 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | -77rank change if excluded 3 direct evals | -26rank change if excluded 1 direct eval | -16rank change if excluded 1 direct eval | -26rank change if excluded 2 direct evals | -0.65 SDDirect |
| command-r | Not shown0 direct evals | Not shown0 direct evals | -19rank change if excluded 1 direct eval | Not shown0 direct evals | 6rank change if excluded 1 direct eval | -20rank change if excluded 6 direct evals | 139rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| qwen3.5-4b | Not shown0 direct evals | -247rank change if excluded 1 direct eval | 8rank change if excluded 1 direct eval | 7rank change if excluded 1 direct eval | 47rank change if excluded 2 direct evals | -247rank change if excluded 1 direct eval | Not shown0 direct evals | -0.65 SDDirect |
| gemma-2-27b-it | -9rank change if excluded 1 direct eval | -9rank change if excluded 2 direct evals | -9rank change if excluded 1 direct eval | -9rank change if excluded 2 direct evals | 37rank change if excluded 1 direct eval | -9rank change if excluded 3 direct evals | -11rank change if excluded 1 direct eval | +0.81 SDDirect |
| yi-1.5-9b-chat | Not shown0 direct evals | 6rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 3 direct evals | -28rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| chatglm-6b | Not shown0 direct evals | 7rank change if excluded 2 direct evals | 4rank change if excluded 3 direct evals | 7rank change if excluded 1 direct eval | Not shown0 direct evals | 7rank change if excluded 4 direct evals | Not shown0 direct evals | -0.06 SDDirect |
| llama-3.1-70b-instruct | 51rank change if excluded 1 direct eval | 51rank change if excluded 3 direct evals | 55rank change if excluded 3 direct evals | -6rank change if excluded 2 direct evals | -6rank change if excluded 4 direct evals | -9rank change if excluded 15 direct evals | 3rank change if excluded 4 direct evals | -1.23 SDDirect |
| olmo-2-32b-instruct | Not shown0 direct evals | -11rank change if excluded 1 direct eval | -7rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 1rank change if excluded 6 direct evals | 7rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| qwen-7b-chat | Not shown0 direct evals | 18rank change if excluded 6 direct evals | 3rank change if excluded 6 direct evals | 17rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 9 direct evals | Not shown0 direct evals | -1.00 SDDirect |
| ernie-5.1 | Not shown0 direct evals | 22rank change if excluded 1 direct eval | Not shown0 direct evals | -75rank change if excluded 3 direct evals | Not shown0 direct evals | 14rank change if excluded 1 direct eval | -103rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| qwen3.5-35b-a3b | Not shown0 direct evals | 149rank change if excluded 1 direct eval | Not shown0 direct evals | -23rank change if excluded 3 direct evals | Not shown0 direct evals | 149rank change if excluded 1 direct eval | 134rank change if excluded 2 direct evals | -0.65 SDDirect |
| baichuan-2-7b-chat | Not shown0 direct evals | -1rank change if excluded 5 direct evals | -9rank change if excluded 5 direct evals | -9rank change if excluded 2 direct evals | Not shown0 direct evals | -22rank change if excluded 11 direct evals | -9rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| mistral-medium-3.1 | -21rank change if excluded 1 direct eval | -21rank change if excluded 1 direct eval | -21rank change if excluded 1 direct eval | -66rank change if excluded 5 direct evals | 2rank change if excluded 1 direct eval | -21rank change if excluded 4 direct evals | 83rank change if excluded 2 direct evals | -1.19 SDDirect |
| qwen2.5-7b-instruct | 17rank change if excluded 1 direct eval | 4rank change if excluded 2 direct evals | 8rank change if excluded 4 direct evals | 8rank change if excluded 1 direct eval | 8rank change if excluded 1 direct eval | 9rank change if excluded 15 direct evals | 6rank change if excluded 3 direct evals | -1.00 SDDirect |
| phi-3.5-mini-instruct | -51rank change if excluded 1 direct eval | -78rank change if excluded 4 direct evals | -51rank change if excluded 2 direct evals | 17rank change if excluded 1 direct eval | Not shown0 direct evals | 16rank change if excluded 4 direct evals | 17rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| minimax-m2.1 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -169rank change if excluded 5 direct evals | 2rank change if excluded 1 direct eval | 0rank change if excluded 3 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| nvidia-nemotron-nano-9b-v2 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 9rank change if excluded 2 direct evals | Not shown0 direct evals | 9rank change if excluded 1 direct eval | Not shown0 direct evals | -0.06 SDMedian-imputed |
| phi-2 | Not shown0 direct evals | 116rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -39rank change if excluded 3 direct evals | 1rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| gpt-4o-mini | 12rank change if excluded 3 direct evals | 10rank change if excluded 3 direct evals | 10rank change if excluded 4 direct evals | 7rank change if excluded 6 direct evals | 8rank change if excluded 4 direct evals | 6rank change if excluded 20 direct evals | -43rank change if excluded 6 direct evals | +0.50 SDDirect |
| qwen2.5-14b-instruct | Not shown0 direct evals | 1rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | 7rank change if excluded 1 direct eval | Not shown0 direct evals | 8rank change if excluded 5 direct evals | 7rank change if excluded 2 direct evals | -1.00 SDDirect |
| o3-pro | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 3 direct evals | -157rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | Not shown0 direct evals | +1.10 SDDirect |
| llama-2-70b-chat | Not shown0 direct evals | -1rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | -2rank change if excluded 2 direct evals | -169rank change if excluded 2 direct evals | 0rank change if excluded 9 direct evals | -2rank change if excluded 2 direct evals | -1.23 SDDirect |
| yi-1.5-34b-chat | Not shown0 direct evals | -38rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | Not shown0 direct evals | 16rank change if excluded 4 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| chatglm3-6b | Not shown0 direct evals | -16rank change if excluded 6 direct evals | -16rank change if excluded 6 direct evals | -15rank change if excluded 2 direct evals | Not shown0 direct evals | -17rank change if excluded 10 direct evals | Not shown0 direct evals | -0.06 SDDirect |
| trinity-large | Not shown0 direct evals | -157rank change if excluded 1 direct eval | Not shown0 direct evals | 31rank change if excluded 7 direct evals | 10rank change if excluded 1 direct eval | 10rank change if excluded 1 direct eval | Not shown0 direct evals | -0.06 SDMedian-imputed |
| vicuna-7b-v1.5 | -6rank change if excluded 1 direct eval | -6rank change if excluded 1 direct eval | -6rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -11rank change if excluded 3 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| gemma-3-27b-it | Not shown0 direct evals | 4rank change if excluded 2 direct evals | 21rank change if excluded 3 direct evals | 4rank change if excluded 8 direct evals | 21rank change if excluded 1 direct eval | 5rank change if excluded 7 direct evals | 5rank change if excluded 1 direct eval | +0.81 SDDirect |
| nova-pro | -62rank change if excluded 1 direct eval | 19rank change if excluded 1 direct eval | Not shown0 direct evals | 18rank change if excluded 4 direct evals | 19rank change if excluded 1 direct eval | 22rank change if excluded 3 direct evals | 19rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| qwen3-vl-32b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -14rank change if excluded 1 direct eval | 69rank change if excluded 1 direct eval | 70rank change if excluded 2 direct evals | Not shown0 direct evals | -1.00 SDDirect |
| mistral-small-3 | Not shown0 direct evals | -2rank change if excluded 1 direct eval | 54rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | -1rank change if excluded 1 direct eval | -24rank change if excluded 6 direct evals | -1rank change if excluded 1 direct eval | -1.19 SDDirect |
| llama-4-scout | -19rank change if excluded 1 direct eval | 3rank change if excluded 4 direct evals | 5rank change if excluded 4 direct evals | -19rank change if excluded 6 direct evals | 3rank change if excluded 1 direct eval | -42rank change if excluded 13 direct evals | 2rank change if excluded 2 direct evals | -0.41 SDDirect |
| muse-glimmer | -16rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 33rank change if excluded 2 direct evals | 18rank change if excluded 1 direct eval | 29rank change if excluded 1 direct eval | Not shown0 direct evals | +0.39 SDDirect |
| gemma-2-2b-it | -18rank change if excluded 1 direct eval | -16rank change if excluded 2 direct evals | -18rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 46rank change if excluded 3 direct evals | 3rank change if excluded 1 direct eval | +0.81 SDDirect |
| mistral-small-3.2 | Not shown0 direct evals | -28rank change if excluded 1 direct eval | 54rank change if excluded 1 direct eval | -28rank change if excluded 3 direct evals | Not shown0 direct evals | -28rank change if excluded 3 direct evals | -28rank change if excluded 1 direct eval | -1.19 SDDirect |
| olmo-3.1-32b-think | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 3 direct evals | Not shown0 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | -0.06 SDMedian-imputed |
| mistral-7b-instruct | 6rank change if excluded 1 direct eval | 7rank change if excluded 6 direct evals | 2rank change if excluded 6 direct evals | 0rank change if excluded 4 direct evals | 23rank change if excluded 4 direct evals | -16rank change if excluded 19 direct evals | 0rank change if excluded 2 direct evals | -1.19 SDDirect |
| mistral-nemo | 86rank change if excluded 2 direct evals | 86rank change if excluded 2 direct evals | 86rank change if excluded 1 direct eval | -12rank change if excluded 1 direct eval | Not shown0 direct evals | -15rank change if excluded 3 direct evals | Not shown0 direct evals | -1.19 SDDirect |
| qwen2-7b-instruct | Not shown0 direct evals | -1rank change if excluded 1 direct eval | 22rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -30rank change if excluded 4 direct evals | -1rank change if excluded 1 direct eval | -1.00 SDDirect |
| palmyra-med | Not shown0 direct evals | Not shown0 direct evals | -24rank change if excluded 1 direct eval | Not shown0 direct evals | -23rank change if excluded 1 direct eval | 52rank change if excluded 5 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| ling-2.6-flash | -170rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -17rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| qwen2.5-max | Not shown0 direct evals | -17rank change if excluded 1 direct eval | -18rank change if excluded 1 direct eval | 27rank change if excluded 3 direct evals | -17rank change if excluded 1 direct eval | 28rank change if excluded 3 direct evals | -17rank change if excluded 1 direct eval | -1.00 SDDirect |
| vicuna-13b-v1.5 | -34rank change if excluded 1 direct eval | -34rank change if excluded 1 direct eval | -34rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 3 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| vicuna-13b | Not shown0 direct evals | 45rank change if excluded 2 direct evals | 9rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 5rank change if excluded 4 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| qwen2.5-1.5b-instruct | Not shown0 direct evals | -7rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 6rank change if excluded 5 direct evals | -7rank change if excluded 1 direct eval | -1.00 SDDirect |
| qwen2.5-3b-instruct | Not shown0 direct evals | -313rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 3 direct evals | 0rank change if excluded 1 direct eval | -1.00 SDDirect |
| granite-3.3-8b-instruct | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 1 direct eval | -79rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | 4rank change if excluded 6 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| qwen1.5-0.5b-chat | Not shown0 direct evals | -30rank change if excluded 1 direct eval | -30rank change if excluded 1 direct eval | -30rank change if excluded 1 direct eval | Not shown0 direct evals | -16rank change if excluded 3 direct evals | Not shown0 direct evals | -1.00 SDDirect |
| gemma-3-4b | Not shown0 direct evals | -33rank change if excluded 2 direct evals | 34rank change if excluded 1 direct eval | -10rank change if excluded 1 direct eval | -33rank change if excluded 1 direct eval | -33rank change if excluded 4 direct evals | -8rank change if excluded 1 direct eval | +0.81 SDDirect |
| chatglm2-6b | Not shown0 direct evals | -30rank change if excluded 4 direct evals | -34rank change if excluded 4 direct evals | Not shown0 direct evals | Not shown0 direct evals | -33rank change if excluded 6 direct evals | Not shown0 direct evals | -0.06 SDDirect |
| llama-3.1-8b-instruct | 2rank change if excluded 1 direct eval | -3rank change if excluded 5 direct evals | -4rank change if excluded 7 direct evals | 4rank change if excluded 6 direct evals | 3rank change if excluded 4 direct evals | -6rank change if excluded 20 direct evals | 8rank change if excluded 5 direct evals | -1.23 SDDirect |
| qwen3-14b | Not shown0 direct evals | 3rank change if excluded 2 direct evals | -11rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | -10rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | -1.00 SDDirect |
| olmo-2-13b-instruct | Not shown0 direct evals | 30rank change if excluded 1 direct eval | -5rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | -2rank change if excluded 5 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| seallms-v3-7b-chat | Not shown0 direct evals | -1rank change if excluded 1 direct eval | -57rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -1rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | -1.00 SDDirect |
| qwen3-8b | Not shown0 direct evals | 2rank change if excluded 3 direct evals | 8rank change if excluded 3 direct evals | 3rank change if excluded 2 direct evals | 5rank change if excluded 2 direct evals | 2rank change if excluded 5 direct evals | 3rank change if excluded 2 direct evals | -1.00 SDDirect |
| zephyr-7b-beta | Not shown0 direct evals | -4rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -141rank change if excluded 5 direct evals | -3rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| mistral-medium-3 | -70rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | Not shown0 direct evals | 46rank change if excluded 4 direct evals | Not shown0 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | -1.19 SDDirect |
| qwen3-next-80b-a3b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -185rank change if excluded 2 direct evals | Not shown0 direct evals | 3rank change if excluded 2 direct evals | Not shown0 direct evals | -1.00 SDDirect |
| command-a | Not shown0 direct evals | 9rank change if excluded 2 direct evals | 6rank change if excluded 2 direct evals | -7rank change if excluded 6 direct evals | Not shown0 direct evals | -8rank change if excluded 5 direct evals | 6rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| qwen3-vl-235b-a22b-instruct | Not shown0 direct evals | 1rank change if excluded 1 direct eval | Not shown0 direct evals | -231rank change if excluded 2 direct evals | Not shown0 direct evals | 1rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | -1.00 SDDirect |
| phi-4-mini | Not shown0 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | -10rank change if excluded 2 direct evals | Not shown0 direct evals | -1rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| llama-3.2-1b-instruct | 26rank change if excluded 1 direct eval | 26rank change if excluded 2 direct evals | 26rank change if excluded 1 direct eval | 59rank change if excluded 1 direct eval | Not shown0 direct evals | -24rank change if excluded 7 direct evals | -3rank change if excluded 1 direct eval | -1.23 SDDirect |
| moss-moon-003-sft | Not shown0 direct evals | 3rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 3 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| mistral-7b | Not shown0 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | -15rank change if excluded 2 direct evals | Not shown0 direct evals | 0rank change if excluded 3 direct evals | -6rank change if excluded 2 direct evals | -1.19 SDDirect |
| qwen2.5-7b | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 2 direct evals | -6rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -1.00 SDDirect |
| vicuna-33b-v1.3 | Not shown0 direct evals | 2rank change if excluded 2 direct evals | 2rank change if excluded 2 direct evals | -21rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 4 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| mixtral-8x22b-instruct | Not shown0 direct evals | 7rank change if excluded 2 direct evals | 33rank change if excluded 1 direct eval | Not shown0 direct evals | 31rank change if excluded 2 direct evals | -74rank change if excluded 8 direct evals | 34rank change if excluded 1 direct eval | -1.19 SDDirect |
| deepseek-v3.2-speciale | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 12rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 1 direct eval | -5rank change if excluded 1 direct eval | -1.00 SDDirect |
| granite-4.0-micro | Not shown0 direct evals | 2rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | -24rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | 3rank change if excluded 6 direct evals | 2rank change if excluded 2 direct evals | -0.06 SDMedian-imputed |
| qwen3-vl-8b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -3rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | -6rank change if excluded 2 direct evals | -3rank change if excluded 1 direct eval | -1.00 SDDirect |
| davinci | Not shown0 direct evals | 2rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | -3rank change if excluded 2 direct evals | Not shown0 direct evals | -16rank change if excluded 2 direct evals | Not shown0 direct evals | +0.50 SDDirect |
| mistral-large-3 | Not shown0 direct evals | 2rank change if excluded 5 direct evals | 2rank change if excluded 4 direct evals | -4rank change if excluded 11 direct evals | 2rank change if excluded 2 direct evals | 9rank change if excluded 8 direct evals | 2rank change if excluded 1 direct eval | -1.19 SDDirect |
| deepseek-llm-67b-chat | Not shown0 direct evals | -61rank change if excluded 1 direct eval | -65rank change if excluded 1 direct eval | Not shown0 direct evals | -61rank change if excluded 1 direct eval | -41rank change if excluded 7 direct evals | -60rank change if excluded 1 direct eval | -1.44 SDDirect |
| vicuna-7b | Not shown0 direct evals | 6rank change if excluded 2 direct evals | -33rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -17rank change if excluded 5 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| qwen3-4b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 2 direct evals | Not shown0 direct evals | -1.00 SDDirect |
| marin-8b-instruct | Not shown0 direct evals | Not shown0 direct evals | -2rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 1 direct eval | -2rank change if excluded 4 direct evals | Not shown0 direct evals | -0.06 SDMedian-imputed |
| jamba-1.7-large | Not shown0 direct evals | 1rank change if excluded 1 direct eval | Not shown0 direct evals | 2rank change if excluded 3 direct evals | Not shown0 direct evals | -5rank change if excluded 1 direct eval | -5rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| qwq-32b | Not shown0 direct evals | -104rank change if excluded 3 direct evals | 5rank change if excluded 2 direct evals | 4rank change if excluded 2 direct evals | Not shown0 direct evals | -19rank change if excluded 3 direct evals | 7rank change if excluded 1 direct eval | -1.00 SDDirect |
| mixtral-8x7b-instruct | Not shown0 direct evals | -27rank change if excluded 2 direct evals | -15rank change if excluded 2 direct evals | -27rank change if excluded 2 direct evals | -25rank change if excluded 1 direct eval | 4rank change if excluded 12 direct evals | -25rank change if excluded 2 direct evals | -1.19 SDDirect |
| jamba-1.7-mini | Not shown0 direct evals | 6rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 2 direct evals | Not shown0 direct evals | -5rank change if excluded 1 direct eval | -6rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| mistral-large-2 | Not shown0 direct evals | -5rank change if excluded 2 direct evals | 0rank change if excluded 3 direct evals | 2rank change if excluded 7 direct evals | Not shown0 direct evals | 0rank change if excluded 9 direct evals | 0rank change if excluded 1 direct eval | -1.19 SDDirect |
| olmo-2-7b-instruct | Not shown0 direct evals | 4rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 1 direct eval | 4rank change if excluded 5 direct evals | 0rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| qwen-1.8b-chat | Not shown0 direct evals | -2rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -2rank change if excluded 3 direct evals | Not shown0 direct evals | -1.00 SDDirect |
| command-r7b | Not shown0 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 3 direct evals | -3rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| llama-3.2-3b | Not shown0 direct evals | -13rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | Not shown0 direct evals | 9rank change if excluded 2 direct evals | -13rank change if excluded 2 direct evals | Not shown0 direct evals | -1.23 SDDirect |
| dbrx-instruct | Not shown0 direct evals | -9rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 1 direct eval | -12rank change if excluded 6 direct evals | -3rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| olmoe-1b-7b-instruct | Not shown0 direct evals | -2rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 1 direct eval | 1rank change if excluded 5 direct evals | 0rank change if excluded 1 direct eval | -0.06 SDMedian-imputed |
| deepseek-r1-distill-llama-70b | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 2 direct evals | Not shown0 direct evals | -1.44 SDDirect |
| qwen1.5-1.8b-chat | Not shown0 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | Not shown0 direct evals | 3rank change if excluded 4 direct evals | 3rank change if excluded 1 direct eval | -1.00 SDDirect |
| magistral-small | Not shown0 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 21rank change if excluded 2 direct evals | Not shown0 direct evals | -47rank change if excluded 4 direct evals | 1rank change if excluded 1 direct eval | -1.19 SDDirect |
| grok-2 | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 3 direct evals | Not shown0 direct evals | -110rank change if excluded 2 direct evals | Not shown0 direct evals | -1.23 SDDirect |
| llama-3.2-3b-instruct | 0rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 6 direct evals | 3rank change if excluded 2 direct evals | -1.23 SDDirect |
| qwen2.5-0.5b-instruct | Not shown0 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 4 direct evals | 0rank change if excluded 1 direct eval | -1.00 SDDirect |
| llama-3.1-8b-base | Not shown0 direct evals | Not shown0 direct evals | -7rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | Not shown0 direct evals | -1.23 SDDirect |
| ministral-3-14b | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 2 direct evals | Not shown0 direct evals | -5rank change if excluded 1 direct eval | Not shown0 direct evals | -1.19 SDDirect |
| ministral-3-8b | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 2 direct evals | Not shown0 direct evals | 1rank change if excluded 2 direct evals | Not shown0 direct evals | -1.19 SDDirect |
