Evaluation profile
SafetyBench
7sub-evals
0.552%total index weight
3components
Within-component eval weight: Misuse resistance 0.628% · Human rights 2.57% · Fairness 1.04%.
Model score (higher is better)Predicted score
About this eval
Multiple-choice safety understanding across Chinese and English prompts.
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction | Total index weight | Within-component weight |
|---|---|---|---|---|
| EMsafetybench/safetybench.csv:EMMeasures whether the model selects safe answers to bilingual multiple-choice questions about ethics and morality. | ordinary_harm_misuse_resistance:1.000safetybench/safetybench.csv | Higher is better | 0.0209% | Misuse resistance 0.209% |
| IAsafetybench/safetybench.csv:IAMeasures whether the model selects safe answers to bilingual multiple-choice questions about crimes and illegal activities. | ordinary_harm_misuse_resistance:1.000safetybench/safetybench.csv | Higher is better | 0.0209% | Misuse resistance 0.209% |
| MHsafetybench/safetybench.csv:MHMeasures whether the model selects safe answers to bilingual multiple-choice questions about mental health and emotional harm. | human_rights_systemic_harm:1.000safetybench/safetybench.csv | Higher is better | 0.128% | Human rights 0.856% |
| OFFsafetybench/safetybench.csv:OFFMeasures whether the model selects safe answers to bilingual multiple-choice questions about offensive language and content. | ordinary_harm_misuse_resistance:1.000safetybench/safetybench.csv | Higher is better | 0.0209% | Misuse resistance 0.209% |
| PHsafetybench/safetybench.csv:PHMeasures whether the model selects safe answers to bilingual multiple-choice questions about physical health and bodily harm. | human_rights_systemic_harm:1.000safetybench/safetybench.csv | Higher is better | 0.128% | Human rights 0.856% |
| PPsafetybench/safetybench.csv:PPMeasures whether the model selects safe answers to bilingual multiple-choice questions about privacy and property. | human_rights_systemic_harm:1.000safetybench/safetybench.csv | Higher is better | 0.128% | Human rights 0.856% |
| UBsafetybench/safetybench.csv:UBMeasures whether the model selects safe answers to bilingual multiple-choice questions about unfairness and social bias. | fairness_nondiscrimination:1.000safetybench/safetybench.csv | Higher is better | 0.104% | Fairness 1.04% |
EM
Measures whether the model selects safe answers to bilingual multiple-choice questions about ethics and morality.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-4 | 92.25 | official | |
| 2 | internlm-chat-7b-v1.1 | 78.8 | official | |
| 3 | chatglm2-lite | 78.05 | official | |
| 4 | gpt-3.5-turbo | 77.75 | official | |
| 5 | baichuan-2-13b-chat | 75.75 | official | |
| 6 | internlm-chat-7b | 75.4 | official | |
| 7 | text-davinci-003 | 72.95 | official | |
| 8 | qwen-7b-chat | 71.4 | official | |
| 9 | flan-t5-xxl | 69.5 | official | |
| 10 | chatglm2-6b | 69.4 | official | |
| 11 | baichuan-13b-chat | 68.4 | official | |
| 12 | wizardlm-13b | 68.1 | official | |
| 13 | vicuna-33b | 66.4 | official | |
| 14 | vicuna-13b | 65.9 | official | |
| 15 | vicuna-7b | 59.8 | official | |
| 16 | openchat-13b | 56.6 | official | |
| 17 | llama-2-13b-chat | 54.6 | official | |
| 18 | llama-2-chinese-13b-chat | 52.3 | official | |
| 19 | wizardlm-7b | 51.2 | official | |
| 20 | llama-2-7b-chat | 49.8 | official | |
| 21 | llama-2-chinese-7b-chat | 43.4 | official |
IA
Measures whether the model selects safe answers to bilingual multiple-choice questions about crimes and illegal activities.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-4 | 92.35 | official | |
| 2 | chatglm2-lite | 85.05 | official | |
| 3 | gpt-3.5-turbo | 85 | official | |
| 4 | baichuan-2-13b-chat | 82.65 | official | |
| 4 | internlm-chat-7b-v1.1 | 82.65 | official | |
| 6 | text-davinci-003 | 81.8 | official | |
| 7 | qwen-7b-chat | 80.1 | official | |
| 8 | internlm-chat-7b | 79.5 | official | |
| 9 | baichuan-13b-chat | 78.65 | official | |
| 10 | chatglm2-6b | 78.2 | official | |
| 10 | flan-t5-xxl | 78.2 | official | |
| 12 | wizardlm-13b | 72.3 | official | |
| 13 | vicuna-13b | 71.4 | official | |
| 14 | vicuna-33b | 70.8 | official | |
| 15 | llama-2-13b-chat | 68.5 | official | |
| 16 | llama-2-chinese-13b-chat | 66.9 | official | |
| 17 | openchat-13b | 66.6 | official | |
| 18 | vicuna-7b | 65.1 | official | |
| 19 | llama-2-7b-chat | 62.4 | official | |
| 20 | wizardlm-7b | 55.4 | official | |
| 21 | llama-2-chinese-7b-chat | 53.5 | official |
MH
Measures whether the model selects safe answers to bilingual multiple-choice questions about mental health and emotional harm.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-4 | 92.8 | official | |
| 2 | gpt-3.5-turbo | 87.75 | official | |
| 3 | chatglm2-lite | 87.65 | official | |
| 4 | internlm-chat-7b-v1.1 | 85.5 | official | |
| 5 | qwen-7b-chat | 84.45 | official | |
| 6 | internlm-chat-7b | 84.3 | official | |
| 7 | baichuan-2-13b-chat | 83.65 | official | |
| 8 | baichuan-13b-chat | 83.15 | official | |
| 9 | text-davinci-003 | 82.35 | official | |
| 10 | chatglm2-6b | 82 | official | |
| 11 | vicuna-33b | 79.7 | official | |
| 12 | wizardlm-13b | 79.4 | official | |
| 13 | flan-t5-xxl | 77.9 | official | |
| 14 | vicuna-13b | 77.5 | official | |
| 15 | llama-2-13b-chat | 73.6 | official | |
| 16 | openchat-13b | 73.1 | official | |
| 16 | vicuna-7b | 73.1 | official | |
| 18 | llama-2-7b-chat | 70.2 | official | |
| 19 | llama-2-chinese-13b-chat | 69.4 | official | |
| 20 | llama-2-chinese-7b-chat | 61.7 | official | |
| 21 | wizardlm-7b | 60.7 | official |
OFF
Measures whether the model selects safe answers to bilingual multiple-choice questions about offensive language and content.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-4 | 86.15 | official | |
| 2 | flan-t5-xxl | 79.2 | official | |
| 3 | gpt-3.5-turbo | 77.4 | official | |
| 4 | text-davinci-003 | 73.2 | official | |
| 5 | chatglm2-lite | 70.7 | official | |
| 6 | baichuan-2-13b-chat | 69.25 | official | |
| 7 | qwen-7b-chat | 69.1 | official | |
| 8 | vicuna-13b | 68.4 | official | |
| 9 | wizardlm-13b | 68.3 | official | |
| 10 | chatglm2-6b | 68.1 | official | |
| 11 | internlm-chat-7b-v1.1 | 67.35 | official | |
| 12 | internlm-chat-7b | 67.2 | official | |
| 13 | vicuna-33b | 66.7 | official | |
| 14 | vicuna-7b | 65.1 | official | |
| 15 | baichuan-13b-chat | 59.25 | official | |
| 16 | openchat-13b | 52.6 | official | |
| 16 | wizardlm-7b | 52.6 | official | |
| 18 | llama-2-7b-chat | 48.9 | official | |
| 18 | llama-2-chinese-7b-chat | 48.9 | official | |
| 20 | llama-2-13b-chat | 48.4 | official | |
| 21 | llama-2-chinese-13b-chat | 48.1 | official |
PH
Measures whether the model selects safe answers to bilingual multiple-choice questions about physical health and bodily harm.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-4 | 94.35 | official | |
| 2 | chatglm2-lite | 79.65 | official | |
| 2 | gpt-3.5-turbo | 79.65 | official | |
| 4 | internlm-chat-7b-v1.1 | 76.65 | official | |
| 5 | baichuan-2-13b-chat | 76.35 | official | |
| 6 | text-davinci-003 | 74.8 | official | |
| 7 | internlm-chat-7b | 74.15 | official | |
| 8 | vicuna-33b | 73 | official | |
| 9 | qwen-7b-chat | 70.4 | official | |
| 10 | wizardlm-13b | 69.4 | official | |
| 11 | baichuan-13b-chat | 68.2 | official | |
| 12 | chatglm2-6b | 67.9 | official | |
| 13 | flan-t5-xxl | 67 | official | |
| 14 | vicuna-13b | 65.3 | official | |
| 15 | vicuna-7b | 60.9 | official | |
| 16 | llama-2-13b-chat | 60.7 | official | |
| 17 | openchat-13b | 59.9 | official | |
| 18 | llama-2-7b-chat | 54.5 | official | |
| 19 | wizardlm-7b | 52.4 | official | |
| 20 | llama-2-chinese-13b-chat | 49.7 | official | |
| 21 | llama-2-chinese-7b-chat | 43 | official |
PP
Measures whether the model selects safe answers to bilingual multiple-choice questions about privacy and property.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-4 | 91 | official | |
| 2 | gpt-3.5-turbo | 85.65 | official | |
| 3 | chatglm2-lite | 82.65 | official | |
| 4 | baichuan-2-13b-chat | 82.05 | official | |
| 5 | internlm-chat-7b-v1.1 | 80.7 | official | |
| 6 | text-davinci-003 | 80.2 | official | |
| 7 | internlm-chat-7b | 78.7 | official | |
| 8 | qwen-7b-chat | 77.2 | official | |
| 9 | baichuan-13b-chat | 77 | official | |
| 10 | flan-t5-xxl | 76.4 | official | |
| 11 | chatglm2-6b | 76 | official | |
| 12 | vicuna-13b | 75.4 | official | |
| 13 | wizardlm-13b | 75 | official | |
| 14 | openchat-13b | 71.1 | official | |
| 14 | vicuna-33b | 71.1 | official | |
| 16 | llama-2-13b-chat | 70.1 | official | |
| 17 | vicuna-7b | 68.4 | official | |
| 18 | llama-2-7b-chat | 65 | official | |
| 19 | llama-2-chinese-13b-chat | 64.7 | official | |
| 20 | llama-2-chinese-7b-chat | 57.6 | official | |
| 21 | wizardlm-7b | 55.8 | official |
UB
Measures whether the model selects safe answers to bilingual multiple-choice questions about unfairness and social bias.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-4 | 77.9 | official | |
| 2 | flan-t5-xxl | 70.2 | official | |
| 3 | wizardlm-13b | 69.6 | official | |
| 4 | gpt-3.5-turbo | 67.9 | official | |
| 5 | internlm-chat-7b-v1.1 | 66.3 | official | |
| 5 | llama-2-13b-chat | 66.3 | official | |
| 7 | qwen-7b-chat | 65.9 | official | |
| 8 | internlm-chat-7b | 64.75 | official | |
| 9 | llama-2-7b-chat | 63.2 | official | |
| 10 | baichuan-13b-chat | 62.65 | official | |
| 11 | openchat-13b | 62.6 | official | |
| 12 | chatglm2-6b | 61.6 | official | |
| 13 | llama-2-chinese-7b-chat | 61.3 | official | |
| 14 | text-davinci-003 | 60.45 | official | |
| 15 | chatglm2-lite | 59.15 | official | |
| 16 | vicuna-33b | 56.8 | official | |
| 17 | llama-2-chinese-13b-chat | 54.4 | official | |
| 18 | vicuna-13b | 53 | official | |
| 19 | vicuna-7b | 52.7 | official | |
| 20 | baichuan-2-13b-chat | 49.2 | official | |
| 21 | wizardlm-7b | 48.8 | official |