Evaluation profile

HELM Classic RealToxicityPrompts

2sub-evals

0.157%Safety weight

1.13%Freedom weight

1components

Weights below are portfolio-specific global index weights.

Model score (lower is better)Predicted score

About this eval

Toxic continuation generation under toxic and non-toxic prompt strata.

Included in the behavior ranking.

Measure	Component	Direction by ranking	Safety weight	Freedom weight
non_toxic_prompt_toxic_fractionrealtoxicityprompts/realtoxicityprompts.csv:non_toxic_prompt_toxic_fractionMeasures whether the model generates toxic continuations when the prompt does not contain toxic language.	Safety: ordinary_harm_misuse_resistance:1.000 · Freedom: user_direction_following:1.000realtoxicityprompts	Safety: lower · Freedom: higher	0.0786%	0.564%
toxic_prompt_toxic_fractionrealtoxicityprompts/realtoxicityprompts.csv:toxic_prompt_toxic_fractionMeasures whether the model generates toxic continuations when the prompt itself contains toxic language.	Safety: ordinary_harm_misuse_resistance:1.000 · Freedom: user_direction_following:1.000realtoxicityprompts	Safety: lower · Freedom: higher	0.0786%	0.564%

Measures whether the model generates toxic continuations when the prompt does not contain toxic language.

Rank	Model	Value	Provenance
1	command-medium-beta	0.002317	official
1	xlarge	0.002317	official
3	medium	0.00296	official
4	large	0.003089	official
4	small	0.003089	official
6	palmyra-x	0.003089	official
7	command-xlarge-beta	0.003604	official
8	t5-11b	0.004118	official
9	t0pp	0.009266	official
10	text-davinci-003	0.01004	official
11	luminous-base	0.01055	official
12	luminous-extended	0.01313	official
13	ul2	0.01364	official
14	text-ada-001	0.01519	official
15	j1-large	0.01725	official
16	j1-grande	0.0175	official
17	gpt-j-6b	0.01905	official
18	bloom	0.01905	official
19	text-babbage-001	0.01931	official
20	ada	0.01982	official
21	babbage	0.02059	official
22	j2-large	0.02059	official
23	j1-jumbo	0.02111	official
24	tnlgv2-7b	0.02136	official
25	glm	0.02188	official
26	gpt-neox-20b	0.02188	official
27	text-curie-001	0.02214	official
28	curie	0.02291	official
29	j1-grande-v2-beta	0.02342	official
30	opt-66b	0.02368	official
31	opt-175b	0.0242	official
32	j2-jumbo	0.0242	official
33	stanford-online-all-v4-s3	0.02445	official
34	palmyra-instruct-30	0.02471	official
34	text-davinci-002	0.02471	official
36	j2-grande	0.02523	official
37	tnlgv2-530b	0.02523	official
38	luminous-supreme	0.02728	official
39	yalm	0.02754	official
40	davinci	0.03449	official

Measures whether the model generates toxic continuations when the prompt itself contains toxic language.

Rank	Model	Value	Provenance
1	palmyra-x	0.008072	official
2	t5-11b	0.01435	official
3	medium	0.01525	official
4	command-xlarge-beta	0.01614	official
5	small	0.01704	official
6	command-medium-beta	0.01794	official
7	large	0.01794	official
8	xlarge	0.02018	official
9	luminous-base	0.04126	official
10	luminous-extended	0.04305	official
11	text-ada-001	0.06457	official
12	text-davinci-003	0.06726	official
13	palmyra-instruct-30	0.07265	official
13	ul2	0.07265	official
15	gpt-neox-20b	0.07803	official
16	opt-175b	0.08072	official
17	gpt-j-6b	0.0852	official
18	luminous-supreme	0.0852	official
19	babbage	0.0861	official
20	j1-large	0.0861	official
21	ada	0.08789	official
22	curie	0.08969	official
23	tnlgv2-530b	0.09148	official
24	j1-jumbo	0.09148	official
25	opt-66b	0.09238	official
26	stanford-online-all-v4-s3	0.09327	official
27	j1-grande	0.09417	official
28	bloom	0.09507	official
29	glm	0.09507	official
30	tnlgv2-7b	0.09596	official
31	j2-large	0.09865	official
32	j2-jumbo	0.09955	official
33	text-davinci-002	0.1013	official
34	text-babbage-001	0.1045	official
35	davinci	0.1049	official
35	j1-grande-v2-beta	0.1049	official
37	text-curie-001	0.1067	official
38	j2-grande	0.1094	official
39	t0pp	0.1184	official
40	yalm	0.157	official