Evaluation profile

HELM Safety

5sub-evals

3.08%Safety weight

9.41%Freedom weight

3components

Weights below are portfolio-specific global index weights.

Model score (higher is better)Predicted score

About this eval

Standardized HELM safety metrics spanning harmful compliance, bias, red-teaming, and over-refusal.

Included in the behavior ranking.

Sub-evals

Measure	Component	Direction by ranking	Safety weight	Freedom weight
anthropic_red_teamhelm/helm.csv:anthropic_red_teamMeasures whether the model resists adversarial conversations intended to provoke harmful, dishonest, or abusive responses.	Safety: ordinary_harm_misuse_resistance:1.000 · Freedom: user_direction_following:1.000helm-anthropic-red-team	Safety: higher · Freedom: lower	0.445%	3.19%
bbqhelm/helm.csv:bbqMeasures whether the model avoids choosing stereotyped answers when questions about protected groups are ambiguous.	Safety: fairness_nondiscrimination:1.000helm-bbq	Safety: higher	1.6%	—
harmbenchhelm/helm.csv:harmbenchMeasures whether the model avoids complying with requests designed to elicit harmful behavior.	Safety: ordinary_harm_misuse_resistance:1.000 · Freedom: user_direction_following:1.000harmbench	Safety: higher · Freedom: lower	0.0927%	0.638%
simple_safety_testshelm/helm.csv:simple_safety_testsMeasures whether the model gives safe responses to straightforward prompts about dangerous or harmful activities.	Safety: ordinary_harm_misuse_resistance:1.000 · Freedom: user_direction_following:1.000helm-simple-safety-tests	Safety: higher · Freedom: lower	0.445%	3.19%
xstesthelm/helm.csv:xstestMeasures whether the model answers harmless requests instead of refusing them merely because they contain superficially sensitive words.	Safety: benign_helpfulness_non_overrefusal:0.556\|ordinary_harm_misuse_resistance:0.444 · Freedom: user_direction_following:1.000xstest	Safety: higher · Freedom: higher	0.5%	2.39%

anthropic_red_team

Measures whether the model resists adversarial conversations intended to provoke harmful, dishonest, or abusive responses.

Rank	Model	Value	Provenance
1	claude-3-haiku	1	official
2	gemini-1.5-flash	0.999	official
2	gemini-1.5-pro	0.999	official
2	granite-4.0-small-guardian	0.999	official
2	qwen3-235b-a22b	0.999	official
2	qwen3-next-80b-a3b	0.999	official
7	claude-3-opus	0.998	official
7	claude-3-sonnet	0.998	official
7	claude-3.5-sonnet	0.998	official
10	claude-3.7-sonnet	0.997	official
10	gpt-4-turbo	0.997	official
10	gpt-oss-20b	0.997	official
13	gpt-4.1-nano	0.996	official
13	gpt-5-nano	0.996	official
13	granite-4.0-micro-guardian	0.996	official
13	granite-4.0-small	0.996	official
13	qwen2.5-72b-instruct	0.996	official
18	gemini-2.5-pro	0.995	official
18	gpt-5.1	0.995	official
18	gpt-oss-120b	0.995	official
18	grok-3-mini	0.995	official
18	palmyra-fin	0.995	official
23	deepseek-llm-67b-chat	0.994	official
23	gemini-2.0-flash	0.994	official
23	gemini-2.0-flash-lite-preview	0.994	official
23	gpt-4.5-preview	0.994	official
27	gpt-4.1	0.993	official
27	gpt-4.1-mini	0.993	official
27	kimi-k2	0.993	official
27	olmo-2-32b-instruct	0.993	official
27	palmyra-x5	0.993	official
32	gpt-4o	0.991	official
32	gpt-5	0.991	official
32	gpt-5-mini	0.991	official
32	granite-3.3-8b-instruct	0.991	official
32	qwen2-72b-instruct	0.991	official
37	glm-4.5-air	0.99	official
37	qwen1.5-72b-chat	0.99	official
37	qwen3-235b-a22b-fp8-throughput	0.99	official
40	claude-opus-4	0.989	official
41	claude-sonnet-4	0.988	official
41	gemini-2.5-flash	0.988	official
41	llama-3-8b-instruct	0.988	official
44	gemini-2.0-pro-preview	0.987	official
44	gemini-2.5-flash-lite	0.987	official
44	granite-4.0-micro	0.987	official
47	o3-mini	0.986	official
47	palmyra-x-004	0.986	official
49	qwen2.5-7b-instruct	0.985	official
50	gpt-4o-mini	0.983	official
50	o1	0.983	official
50	o1-mini	0.983	official
50	o3	0.983	official
54	claude-sonnet-4.5	0.982	official
54	llama-4-maverick	0.982	official
54	o4-mini	0.982	official
57	gpt-3.5-turbo	0.981	official
58	command-r-plus	0.98	official
59	palmyra-med	0.978	official
60	deepseek-r1	0.975	official
61	deepseek-v3	0.971	official
61	gemini-3-pro-preview	0.971	official
63	marin-8b-instruct	0.97	official
64	claude-haiku-4.5	0.969	official
65	llama-3.1-8b-instruct	0.968	official
66	llama-3-70b-instruct	0.967	official
67	llama-3.1-405b-instruct	0.965	official
67	llama-4-scout	0.965	official
69	mistral-small-3	0.964	official
70	mixtral-8x22b-instruct	0.962	official
71	grok-4	0.957	official
72	grok-3	0.955	official
73	command-r	0.937	official
74	llama-3.1-70b-instruct	0.932	official
75	mixtral-8x7b-instruct	0.928	official
76	olmo-2-13b-instruct	0.907	official
77	olmoe-1b-7b-instruct	0.889	official
78	olmo-2-7b-instruct	0.88	official
79	mistral-7b-instruct	0.806	official
80	dbrx-instruct	0.766	official

bbq

Measures whether the model avoids choosing stereotyped answers when questions about protected groups are ambiguous.

Rank	Model	Value	Provenance
1	claude-sonnet-4.5	0.989	official
2	gpt-oss-120b	0.985	official
3	gemini-3-pro-preview	0.984	official
4	claude-opus-4	0.982	official
5	o3	0.979	official
6	glm-4.5-air	0.978	official
7	gemini-2.5-flash	0.977	official
8	gpt-5-nano	0.976	official
9	o1	0.973	official
10	claude-sonnet-4	0.9725	official
11	o1-mini	0.969	official
12	gpt-5	0.968	official
13	deepseek-v3	0.967	official
13	gpt-oss-20b	0.967	official
13	grok-3-mini	0.967	official
16	qwen3-235b-a22b-fp8-throughput	0.966	official
17	deepseek-r1	0.9657	official
18	gemini-2.5-pro	0.964	official
19	gpt-5-mini	0.963	official
20	qwen3-235b-a22b	0.962	official
21	gemini-2.0-pro-preview	0.956	official
22	palmyra-x-004	0.955	official
23	gemini-2.0-flash	0.954	official
23	llama-3.1-70b-instruct	0.954	official
23	qwen2.5-72b-instruct	0.954	official
26	gpt-4o	0.951	official
26	qwen2-72b-instruct	0.951	official
28	claude-3.5-sonnet	0.949	official
28	gemini-2.5-flash-lite	0.949	official
28	kimi-k2	0.949	official
31	palmyra-x5	0.948	official
32	gemini-1.5-flash	0.947	official
33	gemini-1.5-pro	0.945	official
33	llama-3.1-405b-instruct	0.945	official
35	palmyra-fin	0.942	official
36	gpt-4-turbo	0.941	official
36	o3-mini	0.941	official
38	claude-3-opus	0.94	official
38	o4-mini	0.94	official
40	grok-4	0.937	official
41	grok-3	0.936	official
42	mistral-small-3	0.933	official
43	llama-4-maverick	0.93	official
44	claude-haiku-4.5	0.928	official
45	gpt-4.1	0.926	official
46	granite-4.0-small	0.925	official
47	claude-3.7-sonnet	0.921	official
47	gpt-4.1-mini	0.921	official
49	gemini-2.0-flash-lite-preview	0.92	official
49	gpt-4.5-preview	0.92	official
51	qwen3-next-80b-a3b	0.917	official
52	mixtral-8x22b-instruct	0.916	official
53	llama-3-70b-instruct	0.91	official
54	qwen2.5-7b-instruct	0.906	official
55	claude-3-sonnet	0.9	official
55	granite-4.0-small-guardian	0.9	official
57	command-r-plus	0.899	official
58	gpt-5.1	0.887	official
59	gpt-4o-mini	0.882	official
60	gpt-4.1-nano	0.875	official
60	llama-4-scout	0.875	official
62	deepseek-llm-67b-chat	0.862	official
63	mixtral-8x7b-instruct	0.857	official
64	qwen1.5-72b-chat	0.846	official
65	granite-3.3-8b-instruct	0.84	official
66	palmyra-med	0.801	official
67	dbrx-instruct	0.792	official
68	llama-3.1-8b-instruct	0.785	official
69	granite-4.0-micro	0.779	official
70	llama-3-8b-instruct	0.765	official
71	marin-8b-instruct	0.755	official
72	granite-4.0-micro-guardian	0.752	official
73	command-r	0.724	official
74	olmo-2-32b-instruct	0.714	official
75	olmo-2-13b-instruct	0.704	official
76	gpt-3.5-turbo	0.6513	official
77	claude-3-haiku	0.625	official
78	mistral-7b-instruct	0.6165	official
79	olmo-2-7b-instruct	0.525	official
80	olmoe-1b-7b-instruct	0.419	official

harmbench

Measures whether the model avoids complying with requests designed to elicit harmful behavior.

Rank	Model	Value	Provenance
1	gpt-oss-120b	1	official
2	gpt-oss-20b	0.987	official
3	gpt-5-nano	0.984	official
3	o3	0.984	official
5	claude-3.5-sonnet	0.981	official
6	gpt-5	0.976	official
6	gpt-5.1	0.976	official
8	claude-3-opus	0.974	official
8	kimi-k2	0.974	official
10	gpt-5-mini	0.971	official
11	o4-mini	0.97	official
12	o1	0.963	official
13	claude-sonnet-4	0.9605	official
14	claude-haiku-4.5	0.959	official
15	claude-3-sonnet	0.958	official
15	gpt-4.5-preview	0.958	official
17	o3-mini	0.952	official
18	claude-sonnet-4.5	0.92	official
19	gpt-4.1	0.917	official
20	claude-3-haiku	0.913	official
21	claude-opus-4	0.904	official
22	gpt-4-turbo	0.898	official
23	o1-mini	0.885	official
24	gpt-4.1-nano	0.868	official
25	granite-4.0-small-guardian	0.864	official
26	granite-4.0-micro-guardian	0.863	official
27	gpt-4.1-mini	0.856	official
28	gpt-4o-mini	0.849	official
29	qwen3-next-80b-a3b	0.846	official
30	claude-3.7-sonnet	0.843	official
31	olmo-2-32b-instruct	0.841	official
31	palmyra-fin	0.841	official
33	gpt-4o	0.829	official
34	granite-3.3-8b-instruct	0.812	official
35	gemini-1.5-flash	0.8	official
36	gemini-1.5-pro	0.799	official
37	qwen3-235b-a22b	0.796	official
38	qwen2-72b-instruct	0.768	official
39	marin-8b-instruct	0.751	official
40	palmyra-x-004	0.738	official
40	palmyra-x5	0.738	official
42	qwen2.5-72b-instruct	0.728	official
43	llama-3-8b-instruct	0.727	official
44	gemini-3-pro-preview	0.725	official
45	gemini-2.0-flash-lite-preview	0.722	official
46	granite-4.0-small	0.711	official
47	granite-4.0-micro	0.695	official
48	qwen2.5-7b-instruct	0.677	official
49	gemini-2.5-flash-lite	0.67	official
50	gpt-3.5-turbo	0.6663	official
51	gemini-2.0-flash	0.662	official
52	llama-4-maverick	0.661	official
53	gemini-2.5-pro	0.654	official
54	gemini-2.0-pro-preview	0.653	official
55	deepseek-llm-67b-chat	0.649	official
56	qwen1.5-72b-chat	0.648	official
57	llama-3-70b-instruct	0.64	official
58	olmoe-1b-7b-instruct	0.629	official
59	llama-3.1-405b-instruct	0.627	official
60	gemini-2.5-flash	0.626	official
61	llama-3.1-8b-instruct	0.616	official
62	llama-4-scout	0.6	official
63	olmo-2-7b-instruct	0.578	official
64	grok-3-mini	0.572	official
65	qwen3-235b-a22b-fp8-throughput	0.569	official
66	glm-4.5-air	0.561	official
67	palmyra-med	0.556	official
68	mixtral-8x22b-instruct	0.521	official
69	olmo-2-13b-instruct	0.519	official
70	command-r	0.504	official
71	deepseek-r1	0.4987	official
72	deepseek-v3	0.497	official
73	command-r-plus	0.485	official
74	llama-3.1-70b-instruct	0.469	official
75	grok-3	0.453	official
76	mistral-small-3	0.452	official
77	mixtral-8x7b-instruct	0.451	official
78	grok-4	0.397	official
79	mistral-7b-instruct	0.2885	official
80	dbrx-instruct	0.271	official

simple_safety_tests

Measures whether the model gives safe responses to straightforward prompts about dangerous or harmful activities.

Rank	Model	Value	Provenance
1	claude-3-haiku	1	official
1	claude-3-opus	1	official
1	claude-3-sonnet	1	official
1	claude-3.5-sonnet	1	official
1	claude-3.7-sonnet	1	official
1	claude-sonnet-4.5	1	official
1	command-r-plus	1	official
1	gpt-4.1	1	official
1	gpt-4.1-mini	1	official
1	gpt-4.5-preview	1	official
1	gpt-5-mini	1	official
1	gpt-5-nano	1	official
1	gpt-oss-120b	1	official
1	gpt-oss-20b	1	official
1	granite-4.0-micro-guardian	1	official
1	granite-4.0-small-guardian	1	official
1	kimi-k2	1	official
1	o4-mini	1	official
1	palmyra-fin	1	official
1	palmyra-x-004	1	official
1	palmyra-x5	1	official
1	qwen2.5-72b-instruct	1	official
1	qwen3-235b-a22b	1	official
24	gpt-5	0.998	official
24	gpt-5.1	0.998	official
26	claude-opus-4	0.9975	official
26	claude-sonnet-4	0.9975	official
28	qwen3-next-80b-a3b	0.995	official
29	grok-3-mini	0.993	official
29	llama-3-8b-instruct	0.993	official
29	llama-4-maverick	0.993	official
32	glm-4.5-air	0.99	official
32	gpt-4-turbo	0.99	official
32	gpt-4.1-nano	0.99	official
32	llama-3-70b-instruct	0.99	official
32	o1	0.99	official
32	o3	0.99	official
32	o3-mini	0.99	official
32	qwen1.5-72b-chat	0.99	official
40	claude-haiku-4.5	0.988	official
40	llama-3.1-405b-instruct	0.988	official
40	llama-3.1-8b-instruct	0.988	official
43	gemini-2.0-flash	0.985	official
43	gpt-4o	0.985	official
43	palmyra-med	0.985	official
43	qwen2-72b-instruct	0.985	official
43	qwen3-235b-a22b-fp8-throughput	0.985	official
48	gemini-2.5-flash	0.98	official
48	granite-4.0-small	0.98	official
48	olmo-2-32b-instruct	0.98	official
51	deepseek-r1	0.9793	official
52	gpt-4o-mini	0.978	official
53	gemini-2.0-flash-lite-preview	0.977	official
54	gemini-1.5-pro	0.975	official
54	gemini-2.0-pro-preview	0.975	official
54	gemini-3-pro-preview	0.975	official
54	granite-3.3-8b-instruct	0.975	official
58	gemini-1.5-flash	0.97	official
58	gemini-2.5-pro	0.97	official
58	llama-4-scout	0.97	official
58	o1-mini	0.97	official
62	deepseek-llm-67b-chat	0.968	official
62	grok-3	0.968	official
64	gemini-2.5-flash-lite	0.965	official
65	qwen2.5-7b-instruct	0.96	official
66	marin-8b-instruct	0.958	official
67	deepseek-v3	0.953	official
68	granite-4.0-micro	0.945	official
69	command-r	0.943	official
70	gpt-3.5-turbo	0.9337	official
71	mistral-small-3	0.933	official
72	llama-3.1-70b-instruct	0.925	official
73	grok-4	0.922	official
74	mixtral-8x22b-instruct	0.915	official
75	mixtral-8x7b-instruct	0.905	official
76	olmo-2-13b-instruct	0.81	official
77	olmo-2-7b-instruct	0.775	official
78	olmoe-1b-7b-instruct	0.725	official
79	mistral-7b-instruct	0.6215	official
80	dbrx-instruct	0.535	official

xstest

Measures whether the model answers harmless requests instead of refusing them merely because they contain superficially sensitive words.

Rank	Model	Value	Provenance
1	gemini-2.5-flash	0.988	official
1	qwen3-235b-a22b-fp8-throughput	0.988	official
3	gemini-2.5-pro	0.987	official
4	glm-4.5-air	0.986	official
4	qwen3-235b-a22b	0.986	official
6	grok-3-mini	0.984	official
6	palmyra-x-004	0.984	official
8	qwen3-next-80b-a3b	0.983	official
9	gpt-5.1	0.982	official
9	kimi-k2	0.982	official
11	gpt-4.1	0.979	official
11	qwen2.5-72b-instruct	0.979	official
13	gemini-2.5-flash-lite	0.978	official
14	gpt-4-turbo	0.977	official
14	gpt-5-mini	0.977	official
16	gpt-5-nano	0.976	official
17	gpt-4.1-mini	0.974	official
17	o4-mini	0.974	official
19	gemini-3-pro-preview	0.973	official
19	gpt-4o	0.973	official
19	o3	0.973	official
22	deepseek-v3	0.971	official
22	gpt-5	0.971	official
24	claude-opus-4	0.97	official
24	o1	0.97	official
24	o1-mini	0.97	official
27	claude-sonnet-4	0.969	official
27	qwen2-72b-instruct	0.969	official
29	llama-3-70b-instruct	0.968	official
30	grok-4	0.966	official
30	qwen2.5-7b-instruct	0.966	official
32	llama-4-maverick	0.965	official
33	claude-3.7-sonnet	0.964	official
33	claude-sonnet-4.5	0.964	official
33	grok-3	0.964	official
36	palmyra-med	0.963	official
37	palmyra-fin	0.962	official
38	deepseek-r1	0.9617	official
39	gpt-4.1-nano	0.96	official
39	gpt-4o-mini	0.96	official
41	llama-3.1-405b-instruct	0.959	official
42	qwen1.5-72b-chat	0.957	official
43	claude-3.5-sonnet	0.956	official
43	llama-3-8b-instruct	0.956	official
43	llama-4-scout	0.956	official
46	gemini-2.0-pro-preview	0.954	official
46	olmo-2-32b-instruct	0.954	official
48	gemini-2.0-flash	0.953	official
48	llama-3.1-8b-instruct	0.953	official
48	mistral-small-3	0.953	official
48	palmyra-x5	0.953	official
52	gpt-4.5-preview	0.951	official
53	granite-4.0-small	0.946	official
54	llama-3.1-70b-instruct	0.945	official
55	mixtral-8x22b-instruct	0.941	official
55	o3-mini	0.941	official
57	command-r	0.939	official
58	command-r-plus	0.938	official
59	gpt-3.5-turbo	0.9373	official
60	claude-haiku-4.5	0.932	official
61	mixtral-8x7b-instruct	0.931	official
62	gemini-2.0-flash-lite-preview	0.927	official
62	gpt-oss-120b	0.927	official
64	olmo-2-13b-instruct	0.926	official
65	claude-3-opus	0.925	official
66	gemini-1.5-flash	0.921	official
67	marin-8b-instruct	0.917	official
68	granite-3.3-8b-instruct	0.907	official
69	gemini-1.5-pro	0.904	official
70	deepseek-llm-67b-chat	0.889	official
71	olmo-2-7b-instruct	0.888	official
72	gpt-oss-20b	0.872	official
73	granite-4.0-micro	0.869	official
74	claude-3-sonnet	0.858	official
75	claude-3-haiku	0.853	official
76	olmoe-1b-7b-instruct	0.843	official
77	mistral-7b-instruct	0.8065	official
78	granite-4.0-small-guardian	0.806	official
79	dbrx-instruct	0.774	official
80	granite-4.0-micro-guardian	0.754	official