Evaluation profile
SALAD-Bench
18sub-evals
1.54%total index weight
4components
Within-component eval weight: Human rights 6.26% · Misuse resistance 0.919% · Truthfulness 1.37% · Fairness 3.04%.
Model score (higher is better)Predicted score
About this eval
Broad safety behavior, adversarial robustness, and safety knowledge across a six-domain, 16-task, 66-category taxonomy.
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction | Total index weight | Within-component weight |
|---|---|---|---|---|
| attack_enhanced_human_autonomy_integritysalad-bench/salad-bench.csv:attack_enhanced_human_autonomy_integrityMeasures content that undermines human autonomy or integrity under the attack-enhanced protocol. | human_rights_systemic_harm:1.000salad-bench/salad-bench.csv | Higher is better | 0.15% | Human rights 1% |
| attack_enhanced_information_safety_harmssalad-bench/salad-bench.csv:attack_enhanced_information_safety_harmsMeasures privacy, information-safety, and security-related harmful content under the attack-enhanced protocol. | human_rights_systemic_harm:0.500|ordinary_harm_misuse_resistance:0.500salad-bench/salad-bench.csv | Higher is better | 0.0873% | Human rights 0.501% · Misuse resistance 0.122% |
| attack_enhanced_malicious_usesalad-bench/salad-bench.csv:attack_enhanced_malicious_useMeasures malicious-use harmful content under the attack-enhanced protocol. | ordinary_harm_misuse_resistance:1.000salad-bench/salad-bench.csv | Higher is better | 0.0245% | Misuse resistance 0.245% |
| attack_enhanced_misinformation_harmssalad-bench/salad-bench.csv:attack_enhanced_misinformation_harmsMeasures misinformation-related harmful content under the attack-enhanced protocol. | truthfulness_honesty:1.000salad-bench/salad-bench.csv | Higher is better | 0.0823% | Truthfulness 0.549% |
| attack_enhanced_representation_toxicitysalad-bench/salad-bench.csv:attack_enhanced_representation_toxicityMeasures toxic or unfair representational content under the attack-enhanced protocol. | fairness_nondiscrimination:1.000salad-bench/salad-bench.csv | Higher is better | 0.122% | Fairness 1.22% |
| attack_enhanced_socioeconomic_harmssalad-bench/salad-bench.csv:attack_enhanced_socioeconomic_harmsMeasures socioeconomic harmful content under the attack-enhanced protocol. | human_rights_systemic_harm:1.000salad-bench/salad-bench.csv | Higher is better | 0.15% | Human rights 1% |
| base_human_autonomy_integritysalad-bench/salad-bench.csv:base_human_autonomy_integrityMeasures content that undermines human autonomy or integrity under the base protocol. | human_rights_systemic_harm:1.000salad-bench/salad-bench.csv | Higher is better | 0.15% | Human rights 1% |
| base_information_safety_harmssalad-bench/salad-bench.csv:base_information_safety_harmsMeasures privacy, information-safety, and security-related harmful content under the base protocol. | human_rights_systemic_harm:0.500|ordinary_harm_misuse_resistance:0.500salad-bench/salad-bench.csv | Higher is better | 0.0873% | Human rights 0.501% · Misuse resistance 0.122% |
| base_malicious_usesalad-bench/salad-bench.csv:base_malicious_useMeasures malicious-use harmful content under the base protocol. | ordinary_harm_misuse_resistance:1.000salad-bench/salad-bench.csv | Higher is better | 0.0245% | Misuse resistance 0.245% |
| base_misinformation_harmssalad-bench/salad-bench.csv:base_misinformation_harmsMeasures misinformation-related harmful content under the base protocol. | truthfulness_honesty:1.000salad-bench/salad-bench.csv | Higher is better | 0.0823% | Truthfulness 0.549% |
| base_representation_toxicitysalad-bench/salad-bench.csv:base_representation_toxicityMeasures toxic or unfair representational content under the base protocol. | fairness_nondiscrimination:1.000salad-bench/salad-bench.csv | Higher is better | 0.122% | Fairness 1.22% |
| base_socioeconomic_harmssalad-bench/salad-bench.csv:base_socioeconomic_harmsMeasures socioeconomic harmful content under the base protocol. | human_rights_systemic_harm:1.000salad-bench/salad-bench.csv | Higher is better | 0.15% | Human rights 1% |
| mcq_human_autonomy_integritysalad-bench/salad-bench.csv:mcq_human_autonomy_integrityMeasures content that undermines human autonomy or integrity under the mcq protocol. | human_rights_systemic_harm:1.000salad-bench/salad-bench.csv | Higher is better | 0.0751% | Human rights 0.501% |
| mcq_information_safety_harmssalad-bench/salad-bench.csv:mcq_information_safety_harmsMeasures privacy, information-safety, and security-related harmful content under the mcq protocol. | human_rights_systemic_harm:0.500|ordinary_harm_misuse_resistance:0.500salad-bench/salad-bench.csv | Higher is better | 0.0437% | Human rights 0.25% · Misuse resistance 0.0612% |
| mcq_malicious_usesalad-bench/salad-bench.csv:mcq_malicious_useMeasures malicious-use harmful content under the mcq protocol. | ordinary_harm_misuse_resistance:1.000salad-bench/salad-bench.csv | Higher is better | 0.0122% | Misuse resistance 0.122% |
| mcq_misinformation_harmssalad-bench/salad-bench.csv:mcq_misinformation_harmsMeasures misinformation-related harmful content under the mcq protocol. | truthfulness_honesty:1.000salad-bench/salad-bench.csv | Higher is better | 0.0412% | Truthfulness 0.274% |
| mcq_representation_toxicitysalad-bench/salad-bench.csv:mcq_representation_toxicityMeasures toxic or unfair representational content under the mcq protocol. | fairness_nondiscrimination:1.000salad-bench/salad-bench.csv | Higher is better | 0.0608% | Fairness 0.608% |
| mcq_socioeconomic_harmssalad-bench/salad-bench.csv:mcq_socioeconomic_harmsMeasures socioeconomic harmful content under the mcq protocol. | human_rights_systemic_harm:1.000salad-bench/salad-bench.csv | Higher is better | 0.0751% | Human rights 0.501% |
attack_enhanced_human_autonomy_integrity
Measures content that undermines human autonomy or integrity under the attack-enhanced protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-2 | 86.64 | official | |
| 2 | gpt-4-turbo | 73.49 | official | |
| 3 | gpt-3.5-turbo | 72.41 | official | |
| 4 | llama-2-13b-chat | 62.72 | official | |
| 5 | llama-2-70b-chat | 62.28 | official | |
| 6 | gemma-2b-it | 56.47 | official | |
| 7 | internlm-chat-20b | 29.74 | official | |
| 8 | qwen1.5-4b-chat | 26.29 | official | |
| 9 | yi-34b-chat | 24.14 | official | |
| 10 | yi-6b-chat | 23.71 | official | |
| 11 | internlm-chat-7b | 21.12 | official | |
| 11 | internlm2-chat-7b | 21.12 | official | |
| 13 | llama-2-7b-chat | 20.47 | official | |
| 13 | qwen1.5-72b-chat | 20.47 | official | |
| 15 | internlm2-chat-20b | 15.52 | official | |
| 16 | gemini-1.0-pro | 13.36 | official | |
| 16 | gemma-7b-it | 13.36 | official | |
| 18 | chatglm3-6b | 12.72 | official | |
| 19 | mixtral-8x7b-instruct | 11.85 | official | |
| 20 | qwen1.5-0.5b-chat | 11.42 | official | |
| 21 | qwen-72b-chat | 11.21 | official | |
| 22 | qwen-14b-chat | 9.91 | official | |
| 22 | qwen1.5-7b-chat | 9.91 | official | |
| 24 | tulu-2-dpo-70b | 8.19 | official | |
| 25 | qwen-7b-chat | 7.76 | official | |
| 26 | qwen1.5-14b-chat | 7.11 | official | |
| 27 | tulu-2-dpo-7b | 6.47 | official | |
| 28 | mistral-7b-instruct | 6.035 | official | |
| 29 | qwen1.5-1.8b-chat | 6.03 | official | |
| 30 | vicuna-7b-v1.5-16k | 4.96 | official | |
| 31 | tulu-2-dpo-13b | 4.53 | official | |
| 31 | vicuna-13b-v1.5-16k | 4.53 | official | |
| 33 | vicuna-33b-v1.3 | 4.31 | official |
attack_enhanced_information_safety_harms
Measures privacy, information-safety, and security-related harmful content under the attack-enhanced protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-2 | 93.49 | official | |
| 2 | gpt-4-turbo | 85.67 | official | |
| 3 | gpt-3.5-turbo | 72.64 | official | |
| 4 | llama-2-13b-chat | 71.01 | official | |
| 5 | llama-2-70b-chat | 68.4 | official | |
| 6 | gemma-2b-it | 38.76 | official | |
| 7 | yi-34b-chat | 27.36 | official | |
| 8 | yi-6b-chat | 27.04 | official | |
| 9 | internlm-chat-20b | 23.45 | official | |
| 10 | qwen1.5-4b-chat | 23.13 | official | |
| 11 | gemma-7b-it | 22.8 | official | |
| 12 | gemini-1.0-pro | 20.85 | official | |
| 13 | qwen1.5-72b-chat | 17.92 | official | |
| 14 | internlm2-chat-7b | 17.59 | official | |
| 14 | llama-2-7b-chat | 17.59 | official | |
| 16 | internlm-chat-7b | 16.61 | official | |
| 17 | chatglm3-6b | 12.05 | official | |
| 18 | qwen-72b-chat | 9.45 | official | |
| 19 | tulu-2-dpo-70b | 8.14 | official | |
| 20 | qwen1.5-1.8b-chat | 7.82 | official | |
| 21 | internlm2-chat-20b | 7.17 | official | |
| 21 | qwen1.5-7b-chat | 7.17 | official | |
| 23 | qwen-7b-chat | 6.84 | official | |
| 24 | qwen-14b-chat | 6.51 | official | |
| 25 | mixtral-8x7b-instruct | 5.21 | official | |
| 25 | qwen1.5-0.5b-chat | 5.21 | official | |
| 25 | tulu-2-dpo-7b | 5.21 | official | |
| 28 | tulu-2-dpo-13b | 4.56 | official | |
| 29 | vicuna-33b-v1.3 | 4.23 | official | |
| 30 | qwen1.5-14b-chat | 3.91 | official | |
| 30 | vicuna-7b-v1.5-16k | 3.91 | official | |
| 32 | mistral-7b-instruct | 3.095 | official | |
| 33 | vicuna-13b-v1.5-16k | 2.93 | official |
attack_enhanced_malicious_use
Measures malicious-use harmful content under the attack-enhanced protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-2 | 87.03 | official | |
| 2 | gpt-4-turbo | 75.53 | official | |
| 3 | gpt-3.5-turbo | 71.86 | official | |
| 4 | llama-2-70b-chat | 66.15 | official | |
| 5 | llama-2-13b-chat | 62.56 | official | |
| 6 | gemma-2b-it | 53.59 | official | |
| 7 | internlm-chat-20b | 29.36 | official | |
| 8 | internlm2-chat-7b | 24.31 | official | |
| 9 | yi-6b-chat | 23.82 | official | |
| 10 | qwen1.5-4b-chat | 22.84 | official | |
| 11 | yi-34b-chat | 22.76 | official | |
| 12 | internlm-chat-7b | 22.51 | official | |
| 13 | qwen1.5-72b-chat | 17.05 | official | |
| 14 | llama-2-7b-chat | 16.31 | official | |
| 15 | internlm2-chat-20b | 12.56 | official | |
| 16 | gemini-1.0-pro | 12.23 | official | |
| 17 | chatglm3-6b | 11.01 | official | |
| 18 | qwen-14b-chat | 10.44 | official | |
| 19 | gemma-7b-it | 9.95 | official | |
| 20 | qwen-72b-chat | 9.14 | official | |
| 21 | qwen-7b-chat | 8.4 | official | |
| 21 | qwen1.5-0.5b-chat | 8.4 | official | |
| 23 | qwen1.5-7b-chat | 8.32 | official | |
| 24 | tulu-2-dpo-70b | 8.16 | official | |
| 25 | mixtral-8x7b-instruct | 7.67 | official | |
| 26 | qwen1.5-14b-chat | 6.53 | official | |
| 27 | tulu-2-dpo-13b | 6.12 | official | |
| 28 | qwen1.5-1.8b-chat | 4.98 | official | |
| 29 | mistral-7b-instruct | 4.65 | official | |
| 30 | tulu-2-dpo-7b | 4.57 | official | |
| 31 | vicuna-13b-v1.5-16k | 4.24 | official | |
| 32 | vicuna-33b-v1.3 | 4 | official | |
| 33 | vicuna-7b-v1.5-16k | 3.83 | official |
attack_enhanced_misinformation_harms
Measures misinformation-related harmful content under the attack-enhanced protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-2 | 91.61 | official | |
| 2 | gpt-4-turbo | 84.7 | official | |
| 3 | gpt-3.5-turbo | 71.38 | official | |
| 4 | llama-2-13b-chat | 70.23 | official | |
| 5 | llama-2-70b-chat | 62.17 | official | |
| 6 | gemma-2b-it | 52.3 | official | |
| 7 | qwen1.5-4b-chat | 28.29 | official | |
| 8 | internlm-chat-20b | 27.96 | official | |
| 9 | yi-34b-chat | 26.81 | official | |
| 10 | yi-6b-chat | 24.84 | official | |
| 11 | internlm-chat-7b | 19.9 | official | |
| 12 | gemma-7b-it | 18.91 | official | |
| 13 | llama-2-7b-chat | 18.75 | official | |
| 14 | qwen1.5-72b-chat | 18.42 | official | |
| 15 | gemini-1.0-pro | 17.93 | official | |
| 16 | internlm2-chat-7b | 16.45 | official | |
| 17 | chatglm3-6b | 13.16 | official | |
| 18 | mixtral-8x7b-instruct | 9.54 | official | |
| 19 | internlm2-chat-20b | 9.38 | official | |
| 19 | qwen1.5-7b-chat | 9.38 | official | |
| 21 | qwen-72b-chat | 9.05 | official | |
| 22 | qwen-7b-chat | 8.72 | official | |
| 23 | qwen-14b-chat | 8.39 | official | |
| 24 | tulu-2-dpo-70b | 8.22 | official | |
| 25 | qwen1.5-0.5b-chat | 6.91 | official | |
| 26 | qwen1.5-1.8b-chat | 6.74 | official | |
| 27 | vicuna-7b-v1.5-16k | 6.09 | official | |
| 28 | tulu-2-dpo-13b | 5.76 | official | |
| 29 | tulu-2-dpo-7b | 5.59 | official | |
| 30 | vicuna-33b-v1.3 | 4.61 | official | |
| 31 | qwen1.5-14b-chat | 4.28 | official | |
| 32 | mistral-7b-instruct | 4.11 | official | |
| 33 | vicuna-13b-v1.5-16k | 3.29 | official |
attack_enhanced_representation_toxicity
Measures toxic or unfair representational content under the attack-enhanced protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-2 | 87.15 | official | |
| 2 | gpt-4-turbo | 82.44 | official | |
| 3 | gpt-3.5-turbo | 75.65 | official | |
| 4 | llama-2-70b-chat | 68.62 | official | |
| 5 | llama-2-13b-chat | 65.85 | official | |
| 6 | gemma-2b-it | 48.11 | official | |
| 7 | internlm-chat-20b | 31.79 | official | |
| 8 | gemini-1.0-pro | 27.13 | official | |
| 9 | internlm-chat-7b | 23.71 | official | |
| 10 | yi-34b-chat | 22.6 | official | |
| 11 | qwen1.5-4b-chat | 22.09 | official | |
| 12 | yi-6b-chat | 20.24 | official | |
| 13 | internlm2-chat-7b | 19.27 | official | |
| 14 | llama-2-7b-chat | 18.21 | official | |
| 15 | gemma-7b-it | 17.56 | official | |
| 16 | qwen1.5-72b-chat | 14.19 | official | |
| 17 | chatglm3-6b | 12.57 | official | |
| 18 | qwen-72b-chat | 11.97 | official | |
| 19 | mixtral-8x7b-instruct | 10.35 | official | |
| 20 | internlm2-chat-20b | 10.12 | official | |
| 21 | qwen1.5-1.8b-chat | 8.18 | official | |
| 22 | qwen-14b-chat | 7.44 | official | |
| 23 | qwen1.5-7b-chat | 7.39 | official | |
| 24 | tulu-2-dpo-70b | 7.35 | official | |
| 25 | qwen1.5-0.5b-chat | 6.79 | official | |
| 26 | qwen1.5-14b-chat | 6.19 | official | |
| 27 | qwen-7b-chat | 5.91 | official | |
| 28 | tulu-2-dpo-13b | 5.59 | official | |
| 29 | mistral-7b-instruct | 4.09 | official | |
| 30 | tulu-2-dpo-7b | 3.56 | official | |
| 31 | vicuna-7b-v1.5-16k | 3.33 | official | |
| 32 | vicuna-13b-v1.5-16k | 3.19 | official | |
| 33 | vicuna-33b-v1.3 | 3.14 | official |
attack_enhanced_socioeconomic_harms
Measures socioeconomic harmful content under the attack-enhanced protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-2 | 88.31 | official | |
| 2 | gpt-4-turbo | 80.09 | official | |
| 3 | gpt-3.5-turbo | 68.83 | official | |
| 4 | llama-2-13b-chat | 68.4 | official | |
| 5 | llama-2-70b-chat | 60.17 | official | |
| 6 | gemma-2b-it | 51.95 | official | |
| 7 | internlm-chat-20b | 27.27 | official | |
| 8 | llama-2-7b-chat | 23.81 | official | |
| 8 | yi-34b-chat | 23.81 | official | |
| 10 | yi-6b-chat | 23.38 | official | |
| 11 | internlm-chat-7b | 22.08 | official | |
| 12 | internlm2-chat-7b | 20.35 | official | |
| 13 | qwen1.5-4b-chat | 19.05 | official | |
| 14 | chatglm3-6b | 17.75 | official | |
| 15 | qwen1.5-72b-chat | 14.29 | official | |
| 16 | gemini-1.0-pro | 12.99 | official | |
| 17 | internlm2-chat-20b | 12.55 | official | |
| 18 | gemma-7b-it | 12.12 | official | |
| 18 | qwen1.5-0.5b-chat | 12.12 | official | |
| 20 | tulu-2-dpo-70b | 10.82 | official | |
| 21 | mixtral-8x7b-instruct | 9.52 | official | |
| 22 | qwen-72b-chat | 9.09 | official | |
| 22 | qwen1.5-14b-chat | 9.09 | official | |
| 24 | qwen-14b-chat | 7.79 | official | |
| 24 | qwen-7b-chat | 7.79 | official | |
| 24 | tulu-2-dpo-7b | 7.79 | official | |
| 27 | qwen1.5-1.8b-chat | 7.36 | official | |
| 27 | qwen1.5-7b-chat | 7.36 | official | |
| 29 | tulu-2-dpo-13b | 6.93 | official | |
| 29 | vicuna-7b-v1.5-16k | 6.93 | official | |
| 31 | mistral-7b-instruct | 5.41 | official | |
| 32 | vicuna-13b-v1.5-16k | 5.19 | official | |
| 33 | vicuna-33b-v1.3 | 3.9 | official |
base_human_autonomy_integrity
Measures content that undermines human autonomy or integrity under the base protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-2 | 99.88 | official | |
| 2 | internlm-chat-20b | 98.89 | official | |
| 3 | internlm2-chat-20b | 98.54 | official | |
| 4 | llama-2-13b-chat | 98.43 | official | |
| 5 | llama-2-7b-chat | 98.37 | official | |
| 6 | llama-2-70b-chat | 98.25 | official | |
| 7 | internlm2-chat-7b | 98.02 | official | |
| 8 | qwen-14b-chat | 97.32 | official | |
| 9 | gemma-2b-it | 97.03 | official | |
| 10 | internlm-chat-7b | 96.85 | official | |
| 11 | qwen1.5-4b-chat | 96.62 | official | |
| 12 | qwen-72b-chat | 96.39 | official | |
| 13 | gpt-4-turbo | 96.21 | official | |
| 14 | qwen1.5-14b-chat | 96.1 | official | |
| 14 | qwen1.5-72b-chat | 96.1 | official | |
| 16 | qwen1.5-7b-chat | 95.17 | official | |
| 17 | gemma-7b-it | 94.82 | official | |
| 18 | qwen-7b-chat | 94.35 | official | |
| 18 | tulu-2-dpo-70b | 94.35 | official | |
| 20 | chatglm3-6b | 92.55 | official | |
| 21 | yi-34b-chat | 91.73 | official | |
| 22 | gemini-1.0-pro | 91.09 | official | |
| 23 | tulu-2-dpo-13b | 89.52 | official | |
| 24 | gpt-3.5-turbo | 89.4 | official | |
| 25 | yi-6b-chat | 86.78 | official | |
| 26 | tulu-2-dpo-7b | 85.73 | official | |
| 27 | qwen1.5-0.5b-chat | 83.23 | official | |
| 28 | mixtral-8x7b-instruct | 76 | official | |
| 29 | qwen1.5-1.8b-chat | 68.32 | official | |
| 30 | mistral-7b-instruct | 67.21 | official | |
| 31 | vicuna-33b-v1.3 | 58.12 | official | |
| 32 | vicuna-13b-v1.5-16k | 47.35 | official | |
| 33 | vicuna-7b-v1.5-16k | 46.42 | official |
base_information_safety_harms
Measures privacy, information-safety, and security-related harmful content under the base protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | internlm-chat-20b | 99.8 | official | |
| 2 | claude-2 | 99.66 | official | |
| 3 | llama-2-7b-chat | 99.26 | official | |
| 4 | llama-2-70b-chat | 99.19 | official | |
| 5 | gemma-2b-it | 98.38 | official | |
| 5 | gpt-4-turbo | 98.38 | official | |
| 7 | qwen-72b-chat | 97.9 | official | |
| 8 | qwen1.5-14b-chat | 97.77 | official | |
| 9 | gemma-7b-it | 97.49 | official | |
| 10 | qwen1.5-72b-chat | 96.89 | official | |
| 11 | internlm2-chat-20b | 96.68 | official | |
| 12 | qwen-14b-chat | 96.34 | official | |
| 13 | qwen1.5-4b-chat | 96.14 | official | |
| 14 | qwen1.5-7b-chat | 95.67 | official | |
| 15 | internlm-chat-7b | 95.13 | official | |
| 16 | llama-2-13b-chat | 94.92 | official | |
| 17 | tulu-2-dpo-70b | 94.65 | official | |
| 18 | internlm2-chat-7b | 94.45 | official | |
| 19 | qwen-7b-chat | 93.84 | official | |
| 20 | yi-6b-chat | 93.77 | official | |
| 21 | yi-34b-chat | 93.23 | official | |
| 22 | tulu-2-dpo-13b | 92.82 | official | |
| 23 | chatglm3-6b | 92.21 | official | |
| 24 | gemini-1.0-pro | 91.4 | official | |
| 25 | gpt-3.5-turbo | 90.79 | official | |
| 26 | tulu-2-dpo-7b | 89.3 | official | |
| 27 | mixtral-8x7b-instruct | 86.8 | official | |
| 28 | qwen1.5-0.5b-chat | 84.9 | official | |
| 29 | mistral-7b-instruct | 78.74 | official | |
| 30 | qwen1.5-1.8b-chat | 70.21 | official | |
| 31 | vicuna-33b-v1.3 | 70.07 | official | |
| 32 | vicuna-7b-v1.5-16k | 63.51 | official | |
| 33 | vicuna-13b-v1.5-16k | 57.28 | official |
base_malicious_use
Measures malicious-use harmful content under the base protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-2 | 99.97 | official | |
| 2 | internlm2-chat-20b | 98.88 | official | |
| 3 | internlm2-chat-7b | 98.63 | official | |
| 4 | llama-2-7b-chat | 98.54 | official | |
| 5 | internlm-chat-20b | 98.44 | official | |
| 6 | llama-2-70b-chat | 98.17 | official | |
| 7 | llama-2-13b-chat | 97.98 | official | |
| 8 | qwen-14b-chat | 97.33 | official | |
| 9 | qwen1.5-4b-chat | 96.93 | official | |
| 10 | qwen1.5-14b-chat | 96.87 | official | |
| 11 | qwen-72b-chat | 96.47 | official | |
| 12 | internlm-chat-7b | 96.28 | official | |
| 13 | gemma-2b-it | 95.98 | official | |
| 14 | gpt-4-turbo | 95.83 | official | |
| 15 | qwen1.5-72b-chat | 95.2 | official | |
| 16 | qwen1.5-7b-chat | 94.12 | official | |
| 17 | gemma-7b-it | 93.54 | official | |
| 18 | qwen-7b-chat | 93.24 | official | |
| 19 | tulu-2-dpo-70b | 92.75 | official | |
| 20 | chatglm3-6b | 91.15 | official | |
| 21 | yi-34b-chat | 89.36 | official | |
| 22 | gpt-3.5-turbo | 88.74 | official | |
| 23 | gemini-1.0-pro | 87.27 | official | |
| 24 | tulu-2-dpo-13b | 85.52 | official | |
| 25 | tulu-2-dpo-7b | 83.26 | official | |
| 26 | yi-6b-chat | 82.67 | official | |
| 27 | qwen1.5-0.5b-chat | 80.92 | official | |
| 28 | mixtral-8x7b-instruct | 67.65 | official | |
| 29 | mistral-7b-instruct | 57.35 | official | |
| 30 | qwen1.5-1.8b-chat | 56.36 | official | |
| 31 | vicuna-33b-v1.3 | 46.84 | official | |
| 32 | vicuna-13b-v1.5-16k | 34.59 | official | |
| 33 | vicuna-7b-v1.5-16k | 33.49 | official |
base_misinformation_harms
Measures misinformation-related harmful content under the base protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-2 | 99.7 | official | |
| 2 | internlm2-chat-20b | 98.67 | official | |
| 2 | internlm2-chat-7b | 98.67 | official | |
| 4 | llama-2-13b-chat | 97.64 | official | |
| 5 | internlm-chat-7b | 97.05 | official | |
| 6 | internlm-chat-20b | 97 | official | |
| 7 | qwen1.5-4b-chat | 96.45 | official | |
| 8 | gemma-2b-it | 96.41 | official | |
| 9 | llama-2-7b-chat | 96.26 | official | |
| 10 | qwen1.5-14b-chat | 95.72 | official | |
| 11 | llama-2-70b-chat | 95.67 | official | |
| 12 | gemma-7b-it | 95.57 | official | |
| 13 | qwen-14b-chat | 95.42 | official | |
| 14 | qwen-72b-chat | 94.04 | official | |
| 14 | qwen1.5-7b-chat | 94.04 | official | |
| 16 | gemini-1.0-pro | 93.75 | official | |
| 17 | qwen1.5-72b-chat | 93.65 | official | |
| 18 | gpt-4-turbo | 93.35 | official | |
| 19 | qwen-7b-chat | 93.11 | official | |
| 20 | tulu-2-dpo-70b | 92.96 | official | |
| 21 | gpt-3.5-turbo | 92.61 | official | |
| 22 | chatglm3-6b | 91.38 | official | |
| 23 | tulu-2-dpo-7b | 89.81 | official | |
| 24 | tulu-2-dpo-13b | 88.48 | official | |
| 25 | qwen1.5-0.5b-chat | 88.08 | official | |
| 26 | yi-34b-chat | 87.74 | official | |
| 27 | yi-6b-chat | 86.12 | official | |
| 28 | mixtral-8x7b-instruct | 84.39 | official | |
| 29 | mistral-7b-instruct | 75.97 | official | |
| 30 | qwen1.5-1.8b-chat | 73.46 | official | |
| 31 | vicuna-33b-v1.3 | 69.97 | official | |
| 32 | vicuna-13b-v1.5-16k | 62.78 | official | |
| 33 | vicuna-7b-v1.5-16k | 62.04 | official |
base_representation_toxicity
Measures toxic or unfair representational content under the base protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-2 | 99.58 | official | |
| 2 | internlm2-chat-20b | 97.53 | official | |
| 3 | internlm2-chat-7b | 97.19 | official | |
| 4 | llama-2-13b-chat | 95.71 | official | |
| 5 | gemma-2b-it | 95.56 | official | |
| 6 | gemma-7b-it | 94.42 | official | |
| 7 | internlm-chat-7b | 94.37 | official | |
| 8 | internlm-chat-20b | 93.39 | official | |
| 9 | llama-2-7b-chat | 93.06 | official | |
| 10 | qwen1.5-4b-chat | 93.05 | official | |
| 11 | llama-2-70b-chat | 92.71 | official | |
| 11 | qwen1.5-14b-chat | 92.71 | official | |
| 13 | qwen-14b-chat | 92.21 | official | |
| 14 | qwen-72b-chat | 90.69 | official | |
| 15 | qwen1.5-7b-chat | 90.23 | official | |
| 16 | qwen1.5-72b-chat | 90.06 | official | |
| 17 | tulu-2-dpo-70b | 88.87 | official | |
| 18 | gpt-4-turbo | 88.74 | official | |
| 19 | chatglm3-6b | 88.73 | official | |
| 20 | qwen-7b-chat | 88.39 | official | |
| 21 | gpt-3.5-turbo | 87.99 | official | |
| 22 | gemini-1.0-pro | 87.37 | official | |
| 23 | tulu-2-dpo-13b | 85.28 | official | |
| 24 | tulu-2-dpo-7b | 84.49 | official | |
| 25 | mixtral-8x7b-instruct | 82.05 | official | |
| 26 | yi-34b-chat | 81.07 | official | |
| 27 | yi-6b-chat | 78.37 | official | |
| 28 | qwen1.5-0.5b-chat | 75.3 | official | |
| 29 | mistral-7b-instruct | 74.01 | official | |
| 30 | qwen1.5-1.8b-chat | 64.05 | official | |
| 31 | vicuna-33b-v1.3 | 52.78 | official | |
| 32 | vicuna-13b-v1.5-16k | 51.39 | official | |
| 33 | vicuna-7b-v1.5-16k | 47.27 | official |
base_socioeconomic_harms
Measures socioeconomic harmful content under the base protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-2 | 99.41 | official | |
| 2 | internlm-chat-20b | 96.36 | official | |
| 3 | internlm2-chat-20b | 95.77 | official | |
| 4 | llama-2-70b-chat | 94.83 | official | |
| 5 | internlm2-chat-7b | 94.71 | official | |
| 6 | llama-2-7b-chat | 94.12 | official | |
| 7 | qwen1.5-4b-chat | 94.01 | official | |
| 8 | qwen1.5-14b-chat | 93.65 | official | |
| 9 | qwen-14b-chat | 93.07 | official | |
| 10 | qwen-72b-chat | 92.24 | official | |
| 11 | gpt-4-turbo | 92.01 | official | |
| 12 | qwen1.5-72b-chat | 91.89 | official | |
| 13 | llama-2-13b-chat | 91.19 | official | |
| 13 | qwen1.5-7b-chat | 91.19 | official | |
| 15 | internlm-chat-7b | 90.95 | official | |
| 16 | gemma-2b-it | 89.78 | official | |
| 17 | yi-34b-chat | 89.19 | official | |
| 18 | qwen-7b-chat | 88.48 | official | |
| 19 | chatglm3-6b | 86.96 | official | |
| 20 | yi-6b-chat | 86.72 | official | |
| 21 | tulu-2-dpo-70b | 86.49 | official | |
| 22 | gemma-7b-it | 86.13 | official | |
| 23 | tulu-2-dpo-13b | 84.49 | official | |
| 24 | gemini-1.0-pro | 82.26 | official | |
| 25 | qwen1.5-0.5b-chat | 81.2 | official | |
| 26 | tulu-2-dpo-7b | 81.08 | official | |
| 27 | mixtral-8x7b-instruct | 80.85 | official | |
| 28 | gpt-3.5-turbo | 77.32 | official | |
| 29 | mistral-7b-instruct | 74.14 | official | |
| 30 | qwen1.5-1.8b-chat | 74.03 | official | |
| 31 | vicuna-33b-v1.3 | 65.8 | official | |
| 32 | vicuna-13b-v1.5-16k | 57.58 | official | |
| 33 | vicuna-7b-v1.5-16k | 57.11 | official |
mcq_human_autonomy_integrity
Measures content that undermines human autonomy or integrity under the mcq protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-4-turbo | 90.56 | official | |
| 2 | qwen1.5-72b-chat | 83.89 | official | |
| 3 | tulu-2-dpo-70b | 73.61 | official | |
| 4 | qwen-72b-chat | 69.72 | official | |
| 5 | internlm2-chat-7b | 63.89 | official | |
| 6 | qwen1.5-14b-chat | 62.22 | official | |
| 7 | qwen-14b-chat | 61.94 | official | |
| 8 | internlm2-chat-20b | 61.11 | official | |
| 9 | gpt-3.5-turbo | 51.39 | official | |
| 9 | mixtral-8x7b-instruct | 51.39 | official | |
| 11 | gemini-1.0-pro | 45.28 | official | |
| 12 | qwen-7b-chat | 42.5 | official | |
| 12 | vicuna-33b-v1.3 | 42.5 | official | |
| 14 | tulu-2-dpo-13b | 41.67 | official | |
| 15 | gemma-7b-it | 40.56 | official | |
| 16 | vicuna-13b-v1.5-16k | 40.28 | official | |
| 17 | llama-2-70b-chat | 33.61 | official | |
| 18 | mistral-7b-instruct | 32.92 | official | |
| 19 | tulu-2-dpo-7b | 27.78 | official | |
| 20 | vicuna-7b-v1.5-16k | 27.5 | official | |
| 21 | yi-34b-chat | 25 | official | |
| 22 | claude-2 | 21.67 | official | |
| 23 | gemma-2b-it | 20.83 | official | |
| 24 | qwen1.5-7b-chat | 19.17 | official | |
| 25 | chatglm3-6b | 17.78 | official | |
| 26 | llama-2-13b-chat | 10.28 | official | |
| 27 | llama-2-7b-chat | 6.389 | official | |
| 28 | yi-6b-chat | 5.833 | official | |
| 29 | internlm-chat-20b | 3.333 | official | |
| 30 | internlm-chat-7b | 0 | official | |
| 30 | qwen1.5-0.5b-chat | 0 | official | |
| 30 | qwen1.5-1.8b-chat | 0 | official | |
| 30 | qwen1.5-4b-chat | 0 | official |
mcq_information_safety_harms
Measures privacy, information-safety, and security-related harmful content under the mcq protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-4-turbo | 88.33 | official | |
| 2 | qwen1.5-72b-chat | 80.28 | official | |
| 3 | qwen-72b-chat | 65 | official | |
| 4 | tulu-2-dpo-70b | 64.72 | official | |
| 5 | internlm2-chat-7b | 57.5 | official | |
| 6 | mixtral-8x7b-instruct | 51.94 | official | |
| 7 | internlm2-chat-20b | 51.11 | official | |
| 7 | qwen-14b-chat | 51.11 | official | |
| 9 | qwen1.5-14b-chat | 49.72 | official | |
| 10 | gpt-3.5-turbo | 43.89 | official | |
| 11 | gemini-1.0-pro | 43.33 | official | |
| 12 | gemma-7b-it | 40.56 | official | |
| 13 | vicuna-13b-v1.5-16k | 37.5 | official | |
| 14 | qwen-7b-chat | 37.22 | official | |
| 15 | vicuna-33b-v1.3 | 36.94 | official | |
| 16 | tulu-2-dpo-13b | 34.44 | official | |
| 17 | mistral-7b-instruct | 32.5 | official | |
| 18 | yi-34b-chat | 31.67 | official | |
| 19 | llama-2-70b-chat | 30.83 | official | |
| 20 | claude-2 | 28.61 | official | |
| 21 | chatglm3-6b | 20.56 | official | |
| 22 | tulu-2-dpo-7b | 19.72 | official | |
| 23 | llama-2-13b-chat | 17.5 | official | |
| 24 | vicuna-7b-v1.5-16k | 17.22 | official | |
| 25 | gemma-2b-it | 14.17 | official | |
| 26 | qwen1.5-7b-chat | 11.94 | official | |
| 27 | internlm-chat-20b | 4.167 | official | |
| 28 | yi-6b-chat | 3.889 | official | |
| 29 | llama-2-7b-chat | 3.333 | official | |
| 30 | internlm-chat-7b | 0 | official | |
| 30 | qwen1.5-0.5b-chat | 0 | official | |
| 30 | qwen1.5-1.8b-chat | 0 | official | |
| 30 | qwen1.5-4b-chat | 0 | official |
mcq_malicious_use
Measures malicious-use harmful content under the mcq protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-4-turbo | 90.71 | official | |
| 2 | qwen1.5-72b-chat | 84.81 | official | |
| 3 | tulu-2-dpo-70b | 75.45 | official | |
| 4 | qwen-72b-chat | 70 | official | |
| 5 | internlm2-chat-7b | 62.24 | official | |
| 6 | internlm2-chat-20b | 57.24 | official | |
| 7 | qwen1.5-14b-chat | 55.77 | official | |
| 8 | qwen-14b-chat | 54.74 | official | |
| 9 | mixtral-8x7b-instruct | 53.27 | official | |
| 10 | gpt-3.5-turbo | 47.82 | official | |
| 11 | gemini-1.0-pro | 46.6 | official | |
| 12 | vicuna-33b-v1.3 | 42.37 | official | |
| 13 | qwen-7b-chat | 41.35 | official | |
| 14 | gemma-7b-it | 40.38 | official | |
| 15 | tulu-2-dpo-13b | 40.13 | official | |
| 16 | vicuna-13b-v1.5-16k | 38.21 | official | |
| 17 | mistral-7b-instruct | 34.36 | official | |
| 18 | yi-34b-chat | 27.76 | official | |
| 19 | llama-2-70b-chat | 27.24 | official | |
| 20 | tulu-2-dpo-7b | 26.86 | official | |
| 21 | vicuna-7b-v1.5-16k | 23.01 | official | |
| 22 | chatglm3-6b | 19.55 | official | |
| 23 | claude-2 | 19.17 | official | |
| 24 | qwen1.5-7b-chat | 16.35 | official | |
| 25 | gemma-2b-it | 16.15 | official | |
| 26 | llama-2-13b-chat | 7.885 | official | |
| 27 | yi-6b-chat | 5.962 | official | |
| 28 | llama-2-7b-chat | 4.551 | official | |
| 29 | internlm-chat-20b | 3.974 | official | |
| 30 | internlm-chat-7b | 0.0641 | official | |
| 31 | qwen1.5-0.5b-chat | 0 | official | |
| 31 | qwen1.5-1.8b-chat | 0 | official | |
| 31 | qwen1.5-4b-chat | 0 | official |
mcq_misinformation_harms
Measures misinformation-related harmful content under the mcq protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-4-turbo | 88.57 | official | |
| 2 | qwen1.5-72b-chat | 80 | official | |
| 3 | tulu-2-dpo-70b | 68.1 | official | |
| 4 | qwen-72b-chat | 66.43 | official | |
| 5 | internlm2-chat-7b | 62.62 | official | |
| 6 | internlm2-chat-20b | 59.05 | official | |
| 7 | qwen-14b-chat | 55 | official | |
| 8 | qwen1.5-14b-chat | 54.52 | official | |
| 9 | mixtral-8x7b-instruct | 52.86 | official | |
| 10 | gpt-3.5-turbo | 49.05 | official | |
| 11 | gemini-1.0-pro | 44.76 | official | |
| 12 | tulu-2-dpo-13b | 40.71 | official | |
| 13 | qwen-7b-chat | 40.24 | official | |
| 13 | vicuna-13b-v1.5-16k | 40.24 | official | |
| 15 | vicuna-33b-v1.3 | 39.76 | official | |
| 16 | gemma-7b-it | 38.1 | official | |
| 17 | mistral-7b-instruct | 34.76 | official | |
| 18 | llama-2-70b-chat | 30.95 | official | |
| 19 | tulu-2-dpo-7b | 28.33 | official | |
| 20 | yi-34b-chat | 26.43 | official | |
| 21 | vicuna-7b-v1.5-16k | 24.29 | official | |
| 22 | chatglm3-6b | 20.95 | official | |
| 23 | claude-2 | 20.71 | official | |
| 24 | qwen1.5-7b-chat | 18.81 | official | |
| 25 | gemma-2b-it | 17.86 | official | |
| 26 | llama-2-13b-chat | 10.95 | official | |
| 27 | llama-2-7b-chat | 6.19 | official | |
| 28 | yi-6b-chat | 5.238 | official | |
| 29 | internlm-chat-20b | 3.333 | official | |
| 30 | internlm-chat-7b | 0 | official | |
| 30 | qwen1.5-0.5b-chat | 0 | official | |
| 30 | qwen1.5-1.8b-chat | 0 | official | |
| 30 | qwen1.5-4b-chat | 0 | official |
mcq_representation_toxicity
Measures toxic or unfair representational content under the mcq protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-4-turbo | 86.88 | official | |
| 2 | qwen1.5-72b-chat | 79.27 | official | |
| 3 | tulu-2-dpo-70b | 68.96 | official | |
| 4 | qwen-72b-chat | 67.92 | official | |
| 5 | internlm2-chat-7b | 61.25 | official | |
| 6 | internlm2-chat-20b | 57.92 | official | |
| 7 | qwen1.5-14b-chat | 57.71 | official | |
| 8 | qwen-14b-chat | 55.42 | official | |
| 9 | mixtral-8x7b-instruct | 52.08 | official | |
| 10 | gpt-3.5-turbo | 46.77 | official | |
| 11 | vicuna-33b-v1.3 | 42.29 | official | |
| 12 | tulu-2-dpo-13b | 40.31 | official | |
| 13 | vicuna-13b-v1.5-16k | 40.21 | official | |
| 14 | qwen-7b-chat | 40.1 | official | |
| 15 | gemini-1.0-pro | 39.9 | official | |
| 16 | gemma-7b-it | 38.85 | official | |
| 17 | mistral-7b-instruct | 31.3 | official | |
| 18 | tulu-2-dpo-7b | 28.65 | official | |
| 19 | llama-2-70b-chat | 27.71 | official | |
| 20 | yi-34b-chat | 26.98 | official | |
| 21 | claude-2 | 24.38 | official | |
| 22 | vicuna-7b-v1.5-16k | 23.65 | official | |
| 23 | gemma-2b-it | 18.65 | official | |
| 24 | chatglm3-6b | 17.81 | official | |
| 25 | qwen1.5-7b-chat | 15.62 | official | |
| 26 | llama-2-13b-chat | 8.229 | official | |
| 27 | yi-6b-chat | 4.792 | official | |
| 28 | llama-2-7b-chat | 4.271 | official | |
| 29 | internlm-chat-20b | 3.229 | official | |
| 30 | internlm-chat-7b | 0.1042 | official | |
| 31 | qwen1.5-0.5b-chat | 0 | official | |
| 31 | qwen1.5-1.8b-chat | 0 | official | |
| 31 | qwen1.5-4b-chat | 0 | official |
mcq_socioeconomic_harms
Measures socioeconomic harmful content under the mcq protocol.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-4-turbo | 83.89 | official | |
| 2 | qwen1.5-72b-chat | 78.89 | official | |
| 3 | qwen-72b-chat | 66.67 | official | |
| 3 | tulu-2-dpo-70b | 66.67 | official | |
| 5 | internlm2-chat-7b | 56.11 | official | |
| 6 | qwen-14b-chat | 55.56 | official | |
| 7 | qwen1.5-14b-chat | 52.78 | official | |
| 8 | mixtral-8x7b-instruct | 48.89 | official | |
| 9 | gpt-3.5-turbo | 46.67 | official | |
| 10 | internlm2-chat-20b | 46.11 | official | |
| 11 | gemini-1.0-pro | 44.44 | official | |
| 12 | vicuna-13b-v1.5-16k | 40.56 | official | |
| 13 | vicuna-33b-v1.3 | 40 | official | |
| 14 | gemma-7b-it | 38.89 | official | |
| 14 | tulu-2-dpo-13b | 38.89 | official | |
| 16 | qwen-7b-chat | 37.78 | official | |
| 17 | mistral-7b-instruct | 31.94 | official | |
| 18 | llama-2-70b-chat | 31.67 | official | |
| 18 | yi-34b-chat | 31.67 | official | |
| 20 | claude-2 | 29.44 | official | |
| 21 | tulu-2-dpo-7b | 25.56 | official | |
| 22 | chatglm3-6b | 22.78 | official | |
| 23 | vicuna-7b-v1.5-16k | 21.11 | official | |
| 24 | qwen1.5-7b-chat | 17.78 | official | |
| 25 | gemma-2b-it | 16.67 | official | |
| 26 | llama-2-13b-chat | 12.78 | official | |
| 27 | internlm-chat-20b | 7.778 | official | |
| 28 | yi-6b-chat | 6.667 | official | |
| 29 | llama-2-7b-chat | 1.667 | official | |
| 30 | internlm-chat-7b | 0 | official | |
| 30 | qwen1.5-0.5b-chat | 0 | official | |
| 30 | qwen1.5-1.8b-chat | 0 | official | |
| 30 | qwen1.5-4b-chat | 0 | official |