Evaluation profile

BullshitBench v2

1sub-evals

1.47%Safety weight

0%Freedom weight

1components

Weights below are portfolio-specific global index weights.

Model score (higher is better)Predicted score

About this eval

Detection and explanation of plausible-sounding nonsense without refusal.

Included in the behavior ranking.

Sub-evals

Measure	Component	Direction by ranking	Safety weight	Freedom weight
clear_pushback_ratebullshitbench-v2/bullshitbench-v2.csv:clear_pushback_rateMeasures whether the model identifies plausible-sounding nonsense and clearly explains that it is unsupported instead of answering as if it were true.	Safety: truthfulness_honesty:1.000bullshitbench-v2	Safety: higher	1.47%	—

clear_pushback_rate

Measures whether the model identifies plausible-sounding nonsense and clearly explains that it is unsupported instead of answering as if it were true.

Rank	Model	Value	Provenance
1	claude-opus-4.8	0.945	official
2	claude-sonnet-4.6	0.9	official
3	claude-opus-4.6	0.85	official
4	claude-opus-4.5	0.845	official
5	claude-sonnet-5	0.79	official
6	claude-opus-4.7	0.785	official
7	claude-sonnet-4.5	0.765	official
8	claude-haiku-4.5	0.74	official
9	qwen3.5-397b-a17b	0.735	official
10	kimi-k3	0.73	official
11	claude-opus-5	0.715	official
12	claude-fable-5.1	0.705	official
13	glm-5.3	0.69	official
14	gpt-6-astra	0.665	official
15	grok-4.20-multi-agent	0.655	official
15	qwen3.6-plus	0.655	official
17	qwen3.7-max	0.645	official
18	minimax-m3	0.635	official
19	gemini-3.5-flash-lite	0.63	official
20	grok-4.6	0.61	official
20	qwen3.8-max	0.61	official
22	ox-alpha	0.595	official
23	kimi-k2.6	0.575	official
24	deepseek-v4.1-flash	0.57	official
25	grok-4.20	0.55	official
26	grok-4.5	0.545	official
26	qwen3-max	0.545	official
28	qwen3.8-27b	0.54	official
29	claude-fable-5	0.505	official
29	gpt-5.6-terra	0.505	official
31	claude-3.5-haiku	0.5	official
31	mimo-v2.5-pro	0.5	official
33	muse-spark-1.2	0.495	official
34	grok-4.3	0.48	official
35	gpt-5.6-sol	0.475	official
36	claude-3.7-sonnet	0.465	official
37	claude-3.5-sonnet	0.46	official
38	gpt-5.5	0.4567	official
39	gpt-5.4	0.45	official
40	nemotron-3-ultra-550b-a55b	0.445	official
41	nemotron-3-super-120b-a12b	0.4375	official
42	claude-opus-4.1	0.425	official
43	gemini-3-pro-preview	0.42	official
44	kimi-k2.5	0.415	official
45	gpt-5.2-codex	0.4033	official
46	gpt-5.3-chat	0.4	official
47	gpt-5-codex	0.39	official
47	gpt-5.6-luna	0.39	official
47	hunter-alpha	0.39	official
50	gpt-5.1	0.35	official
50	gpt-5.5-pro	0.35	official
52	claude-opus-4	0.34	official
52	gemini-3.1-pro-preview	0.34	official
52	gpt-5.5-instant	0.34	official
55	deepseek-v4-pro-0813	0.335	official
55	gemini-3.6-flash	0.335	official
57	gemini-3.7-flash	0.33	official
57	gpt-5.2	0.33	official
59	gpt-5.1-codex	0.32	official
60	gpt-5.4-mini	0.3167	official
61	healer-alpha	0.315	official
62	claude-sonnet-4	0.295	official
62	deepseek-v4-flash-0731	0.295	official
62	mimo-v2.5	0.295	official
65	llama-4-maverick	0.29	official
66	glm-5-turbo	0.27	official
66	gpt-5.2-chat	0.27	official
68	o3	0.26	official
69	glm-5	0.24	official
69	glm-5.2	0.24	official
71	gemma-4-31b-it	0.225	official
72	laguna-s-2.1-poolside	0.215	official
73	gpt-5.3-codex	0.21	official
74	gemini-2.5-pro	0.2	official
74	glm-5.1	0.2	official
74	llama-4-scout	0.2	official
74	qwen3-coder-480b-a35b-instruct	0.2	official
78	gemini-3.5-flash	0.195	official
78	gpt-5	0.195	official
80	gemini-2.5-flash	0.19	official
80	nemotron-3-nano-30b-a3b	0.19	official
82	step-3.5-flash	0.175	official
83	deepseek-v4-flash	0.16	official
83	gemma-4-26b-a4b-it	0.16	official
85	gemini-2.0-flash	0.15	official
85	trinity-large	0.15	official
87	grok-4.1-fast	0.145	official
87	mimo-v2-flash	0.145	official
89	deepseek-v4-pro	0.14	official
89	gpt-4.1	0.14	official
89	llama-3.1-8b-instruct	0.14	official
92	gpt-5.4-nano	0.1267	official
93	gpt-4o	0.12	official
93	hy3-preview	0.12	official
95	deepseek-v3.2	0.115	official
96	gemini-3.1-flash-lite	0.11	official
97	claude-3-haiku	0.1	official
97	gemini-3-flash-preview	0.1	official
97	kimi-k2	0.1	official
100	doubao-seed-1.6	0.09	official
101	gpt-oss-120b	0.085	official
101	minimax-m2.5	0.085	official
103	deepseek-r1	0.075	official
103	minimax-m2.7	0.075	official
105	ernie-4.5-vl-424b-a47b	0.07	official
105	glm-4.5	0.07	official
107	intellect-3	0.06	official
107	o4-mini	0.06	official
109	jamba-1.7-large	0.05	official
109	mistral-small-4	0.05	official
111	qwen3-235b-a22b	0.045	official
112	deepseek-v3	0.04	official
112	ernie-4.5-300b-a47b	0.04	official
112	nvidia-nemotron-nano-9b-v2	0.04	official
115	gemma-3-27b-it	0.03	official
116	gpt-4o-mini	0.02	official
116	mistral-large-3	0.02	official