Evaluation profile
BullshitBench v2
1sub-evals
1.48%Safety weight
0%Freedom weight
1components
Weights below are portfolio-specific global index weights.
Model score (higher is better)Predicted score
About this eval
Detection and explanation of plausible-sounding nonsense without refusal.
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction by ranking | Safety weight | Freedom weight |
|---|---|---|---|---|
| clear_pushback_ratebullshitbench-v2/bullshitbench-v2.csv:clear_pushback_rateMeasures whether the model identifies plausible-sounding nonsense and clearly explains that it is unsupported instead of answering as if it were true. | Safety: truthfulness_honesty:1.000bullshitbench-v2 | Safety: higher | 1.48% | — |
clear_pushback_rate
Measures whether the model identifies plausible-sounding nonsense and clearly explains that it is unsupported instead of answering as if it were true.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-opus-4.8 | 0.945 | official | |
| 2 | claude-sonnet-4.6 | 0.9 | official | |
| 3 | claude-opus-4.6 | 0.85 | official | |
| 4 | claude-opus-4.5 | 0.845 | official | |
| 5 | claude-sonnet-5 | 0.79 | official | |
| 6 | claude-opus-4.7 | 0.785 | official | |
| 7 | claude-sonnet-4.5 | 0.765 | official | |
| 8 | claude-haiku-4.5 | 0.74 | official | |
| 9 | qwen3.5-397b-a17b | 0.735 | official | |
| 10 | kimi-k3 | 0.72 | official | |
| 11 | claude-opus-5 | 0.715 | official | |
| 12 | glm-5.3 | 0.68 | official | |
| 13 | grok-4.20-multi-agent | 0.655 | official | |
| 13 | qwen3.6-plus | 0.655 | official | |
| 15 | qwen3.7-max | 0.635 | official | |
| 16 | minimax-m3 | 0.625 | official | |
| 17 | gemini-3.5-flash-lite | 0.62 | official | |
| 18 | grok-4.6 | 0.6 | official | |
| 18 | qwen3.8-max | 0.6 | official | |
| 20 | ox-alpha | 0.585 | official | |
| 21 | kimi-k2.6 | 0.575 | official | |
| 22 | grok-4.20 | 0.55 | official | |
| 23 | qwen3-max | 0.545 | official | |
| 24 | grok-4.5 | 0.54 | official | |
| 25 | qwen3.8-27b | 0.53 | official | |
| 26 | claude-3.5-haiku | 0.5 | official | |
| 26 | mimo-v2.5-pro | 0.5 | official | |
| 28 | gpt-5.6-terra | 0.495 | official | |
| 28 | muse-spark-1.2 | 0.495 | official | |
| 30 | claude-fable-5 | 0.49 | official | |
| 31 | grok-4.3 | 0.48 | official | |
| 32 | gpt-5.6-sol | 0.465 | official | |
| 33 | claude-3.7-sonnet | 0.46 | official | |
| 34 | gpt-5.5 | 0.4567 | official | |
| 35 | claude-3.5-sonnet | 0.45 | official | |
| 35 | gpt-5.4 | 0.45 | official | |
| 37 | nemotron-3-ultra-550b-a55b | 0.445 | official | |
| 38 | nemotron-3-super-120b-a12b | 0.4375 | official | |
| 39 | claude-opus-4.1 | 0.425 | official | |
| 40 | gemini-3-pro-preview | 0.42 | official | |
| 41 | kimi-k2.5 | 0.415 | official | |
| 42 | gpt-5.2-codex | 0.4033 | official | |
| 43 | gpt-5.3-chat | 0.4 | official | |
| 44 | gpt-5-codex | 0.39 | official | |
| 44 | hunter-alpha | 0.39 | official | |
| 46 | gpt-5.6-luna | 0.38 | official | |
| 47 | gpt-5.1 | 0.35 | official | |
| 47 | gpt-5.5-pro | 0.35 | official | |
| 49 | claude-opus-4 | 0.34 | official | |
| 49 | gemini-3.1-pro-preview | 0.34 | official | |
| 49 | gpt-5.5-instant | 0.34 | official | |
| 52 | deepseek-v4-pro-0813 | 0.335 | official | |
| 52 | gemini-3.6-flash | 0.335 | official | |
| 54 | gemini-3.7-flash | 0.33 | official | |
| 54 | gpt-5.2 | 0.33 | official | |
| 56 | gpt-5.1-codex | 0.32 | official | |
| 57 | gpt-5.4-mini | 0.3167 | official | |
| 58 | healer-alpha | 0.315 | official | |
| 59 | claude-sonnet-4 | 0.295 | official | |
| 59 | mimo-v2.5 | 0.295 | official | |
| 61 | deepseek-v4-flash-0731 | 0.29 | official | |
| 62 | llama-4-maverick | 0.28 | official | |
| 63 | glm-5-turbo | 0.27 | official | |
| 63 | gpt-5.2-chat | 0.27 | official | |
| 65 | o3 | 0.26 | official | |
| 66 | glm-5 | 0.24 | official | |
| 66 | glm-5.2 | 0.24 | official | |
| 68 | gemma-4-31b-it | 0.225 | official | |
| 69 | gpt-5.3-codex | 0.21 | official | |
| 69 | laguna-s-2.1-poolside | 0.21 | official | |
| 71 | gemini-2.5-pro | 0.2 | official | |
| 71 | glm-5.1 | 0.2 | official | |
| 71 | qwen3-coder-480b-a35b-instruct | 0.2 | official | |
| 74 | gemini-3.5-flash | 0.195 | official | |
| 74 | gpt-5 | 0.195 | official | |
| 76 | gemini-2.5-flash | 0.19 | official | |
| 76 | llama-4-scout | 0.19 | official | |
| 76 | nemotron-3-nano-30b-a3b | 0.19 | official | |
| 79 | step-3.5-flash | 0.175 | official | |
| 80 | deepseek-v4-flash | 0.16 | official | |
| 80 | gemma-4-26b-a4b-it | 0.16 | official | |
| 82 | gemini-2.0-flash | 0.15 | official | |
| 82 | trinity-large | 0.15 | official | |
| 84 | grok-4.1-fast | 0.145 | official | |
| 84 | mimo-v2-flash | 0.145 | official | |
| 86 | deepseek-v4-pro | 0.14 | official | |
| 86 | gpt-4.1 | 0.14 | official | |
| 86 | llama-3.1-8b-instruct | 0.14 | official | |
| 89 | gpt-5.4-nano | 0.1233 | official | |
| 90 | gpt-4o | 0.12 | official | |
| 90 | hy3-preview | 0.12 | official | |
| 92 | deepseek-v3.2 | 0.115 | official | |
| 93 | gemini-3.1-flash-lite | 0.11 | official | |
| 94 | claude-3-haiku | 0.1 | official | |
| 94 | gemini-3-flash-preview | 0.1 | official | |
| 94 | kimi-k2 | 0.1 | official | |
| 97 | doubao-seed-1.6 | 0.09 | official | |
| 98 | minimax-m2.5 | 0.085 | official | |
| 99 | gpt-oss-120b | 0.08 | official | |
| 100 | deepseek-r1 | 0.075 | official | |
| 100 | minimax-m2.7 | 0.075 | official | |
| 102 | ernie-4.5-vl-424b-a47b | 0.07 | official | |
| 102 | glm-4.5 | 0.07 | official | |
| 104 | intellect-3 | 0.06 | official | |
| 104 | o4-mini | 0.06 | official | |
| 106 | jamba-1.7-large | 0.05 | official | |
| 106 | mistral-small-4 | 0.05 | official | |
| 108 | qwen3-235b-a22b | 0.045 | official | |
| 109 | deepseek-v3 | 0.04 | official | |
| 109 | ernie-4.5-300b-a47b | 0.04 | official | |
| 109 | nvidia-nemotron-nano-9b-v2 | 0.04 | official | |
| 112 | gemma-3-27b-it | 0.03 | official | |
| 113 | gpt-4o-mini | 0.02 | official | |
| 113 | mistral-large-3 | 0.02 | official |