Evaluation profile
AILuminate General Purpose AI Chat
12sub-evals
1.17%Safety weight
2.02%Freedom weight
4components
Weights below are portfolio-specific global index weights.
Model score (lower is better)Predicted score
About this eval
General-purpose AI chat safety across MLCommons hazard categories.
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction by ranking | Safety weight | Freedom weight |
|---|---|---|---|---|
| child_sexual_exploitationailuminate/ailuminate.csv:child_sexual_exploitationMeasures whether the model avoids policy-violating responses involving child sexual exploitation. | Safety: human_rights_systemic_harm:1.000 · Freedom: user_direction_following:1.000ailuminate/ailuminate.csv | Safety: lower · Freedom: higher | 0.148% | 0.168% |
| defamationailuminate/ailuminate.csv:defamationMeasures whether the model avoids policy-violating responses involving defamation. | Safety: human_rights_systemic_harm:0.500|truthfulness_honesty:0.500 · Freedom: user_direction_following:1.000ailuminate/ailuminate.csv | Safety: lower · Freedom: higher | 0.0655% | 0.168% |
| hateailuminate/ailuminate.csv:hateMeasures whether the model avoids policy-violating responses involving hate. | Safety: fairness_nondiscrimination:1.000 · Freedom: user_direction_following:1.000ailuminate/ailuminate.csv | Safety: lower · Freedom: higher | 0.123% | 0.168% |
| indiscriminate_weapons_cbrneailuminate/ailuminate.csv:indiscriminate_weapons_cbrneMeasures whether the model avoids policy-violating responses involving indiscriminate weapons chemical, biological, radiological, nuclear, or explosive weapons. | Safety: ordinary_harm_misuse_resistance:1.000 · Freedom: user_direction_following:1.000ailuminate/ailuminate.csv | Safety: lower · Freedom: higher | 0.0256% | 0.168% |
| intellectual_propertyailuminate/ailuminate.csv:intellectual_propertyMeasures whether the model avoids policy-violating responses involving intellectual property. | Safety: human_rights_systemic_harm:1.000 · Freedom: user_direction_following:1.000ailuminate/ailuminate.csv | Safety: lower · Freedom: higher | 0.148% | 0.168% |
| non_violent_crimesailuminate/ailuminate.csv:non_violent_crimesMeasures whether the model avoids policy-violating responses involving non violent crimes. | Safety: ordinary_harm_misuse_resistance:1.000 · Freedom: user_direction_following:1.000ailuminate/ailuminate.csv | Safety: lower · Freedom: higher | 0.0256% | 0.168% |
| privacyailuminate/ailuminate.csv:privacyMeasures whether the model avoids policy-violating responses involving privacy. | Safety: human_rights_systemic_harm:1.000 · Freedom: user_direction_following:1.000ailuminate/ailuminate.csv | Safety: lower · Freedom: higher | 0.148% | 0.168% |
| sex_related_crimesailuminate/ailuminate.csv:sex_related_crimesMeasures whether the model avoids policy-violating responses involving sex related crimes. | Safety: human_rights_systemic_harm:1.000 · Freedom: user_direction_following:1.000ailuminate/ailuminate.csv | Safety: lower · Freedom: higher | 0.148% | 0.168% |
| sexual_contentailuminate/ailuminate.csv:sexual_contentMeasures whether the model avoids policy-violating responses involving sexual content. | Safety: ordinary_harm_misuse_resistance:1.000 · Freedom: user_direction_following:1.000ailuminate/ailuminate.csv | Safety: lower · Freedom: higher | 0.0102% | 0.168% |
| specialized_adviceailuminate/ailuminate.csv:specialized_adviceMeasures whether the model avoids policy-violating responses involving specialized advice. | Safety: ordinary_harm_misuse_resistance:1.000 · Freedom: user_direction_following:1.000ailuminate/ailuminate.csv | Safety: lower · Freedom: higher | 0.0256% | 0.168% |
| suicide_self_harmailuminate/ailuminate.csv:suicide_self_harmMeasures whether the model avoids policy-violating responses involving suicide self harm. | Safety: human_rights_systemic_harm:1.000 · Freedom: user_direction_following:1.000ailuminate/ailuminate.csv | Safety: lower · Freedom: higher | 0.148% | 0.168% |
| violent_crimesailuminate/ailuminate.csv:violent_crimesMeasures whether the model avoids policy-violating responses involving violent crimes. | Safety: human_rights_systemic_harm:1.000 · Freedom: user_direction_following:1.000ailuminate/ailuminate.csv | Safety: lower · Freedom: higher | 0.148% | 0.168% |
child_sexual_exploitation
Measures whether the model avoids policy-violating responses involving child sexual exploitation.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-3.5-sonnet | 1 | official | |
| 2 | claude-3.5-haiku | 1.8 | official | |
| 3 | gemini-2.0-flash-lite | 3 | official | |
| 4 | gemma-2-9b-it | 6.7 | official | |
| 5 | phi-3.5-moe-instruct | 8.2 | official | |
| 6 | olmo-2-32b-instruct | 9.1 | official | |
| 7 | phi-4 | 10.3 | official | |
| 8 | llama-3.1-tulu-3-8b | 11.1 | official | |
| 9 | gpt-4o | 12 | official | |
| 10 | athene-v2-chat | 12.88 | official | |
| 11 | qwen1.5-110b-chat | 13.1 | official | |
| 12 | olmo-2-13b-instruct | 14.2 | official | |
| 13 | mistral-large-moderated | 14.9 | official | |
| 14 | gpt-4o-mini | 17.6 | official | |
| 15 | llama-3.1-8b-instruct | 18.9 | official | |
| 16 | llama-3.1-405b-instruct | 20.4 | official | |
| 17 | mistral-large | 21.5 | official | |
| 18 | command-a | 22.1 | official | |
| 19 | aya-expanse-8b | 24.62 | official | |
| 20 | ministral-8b-moderated | 25 | official | |
| 21 | nova-lite | 25.7 | official | |
| 22 | llama-3.3-70b-instruct | 26 | official | |
| 23 | phi-3.5-mini-instruct | 26.1 | official | |
| 24 | gemma-3-27b-it | 27 | official | |
| 25 | gemini-2.0-flash | 27.2 | official | |
| 26 | yi-1.5-34b-chat | 30.3 | official | |
| 27 | gemini-1.5-pro | 34.7 | official | |
| 28 | mistral-large-2 | 35.9 | official | |
| 29 | ai21-jamba-large-1.5-azure | 36 | official | |
| 30 | ministral-8b | 44 | official | |
| 31 | qwq-32b | 55.1 | official | |
| 32 | olmo-7b-instruct | 81.2 | official |
defamation
Measures whether the model avoids policy-violating responses involving defamation.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | athene-v2-chat | 0 | official | |
| 2 | claude-3.5-sonnet | 1.3 | official | |
| 3 | claude-3.5-haiku | 2.5 | official | |
| 4 | phi-4 | 3.5 | official | |
| 5 | phi-3.5-moe-instruct | 3.9 | official | |
| 6 | llama-3.1-tulu-3-8b | 7.2 | official | |
| 7 | gemma-2-9b-it | 8.5 | official | |
| 8 | mistral-large-moderated | 9.3 | official | |
| 9 | phi-3.5-mini-instruct | 10.3 | official | |
| 10 | olmo-2-32b-instruct | 10.6 | official | |
| 11 | olmo-2-13b-instruct | 11 | official | |
| 12 | qwen1.5-110b-chat | 12.2 | official | |
| 13 | gemini-2.0-flash-lite | 13 | official | |
| 14 | ministral-8b-moderated | 13.7 | official | |
| 15 | llama-3.1-8b-instruct | 15.8 | official | |
| 16 | gpt-4o | 15.86 | official | |
| 17 | aya-expanse-8b | 16.8 | official | |
| 18 | gemini-2.0-flash | 17.2 | official | |
| 19 | gpt-4o-mini | 17.4 | official | |
| 20 | command-a | 20.5 | official | |
| 21 | llama-3.1-405b-instruct | 20.6 | official | |
| 22 | gemini-1.5-pro | 21.6 | official | |
| 23 | nova-lite | 23.9 | official | |
| 24 | mistral-large | 26.3 | official | |
| 25 | llama-3.3-70b-instruct | 26.8 | official | |
| 26 | yi-1.5-34b-chat | 29.3 | official | |
| 27 | gemma-3-27b-it | 38 | official | |
| 28 | ai21-jamba-large-1.5-azure | 38.6 | official | |
| 29 | mistral-large-2 | 39.2 | official | |
| 30 | ministral-8b | 40.7 | official | |
| 31 | qwq-32b | 56 | official | |
| 32 | olmo-7b-instruct | 74 | official |
hate
Measures whether the model avoids policy-violating responses involving hate.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | athene-v2-chat | 0 | official | |
| 2 | claude-3.5-sonnet | 0.8 | official | |
| 3 | claude-3.5-haiku | 1 | official | |
| 4 | phi-4 | 1.9 | official | |
| 5 | phi-3.5-moe-instruct | 2.5 | official | |
| 6 | llama-3.1-tulu-3-8b | 2.6 | official | |
| 7 | gemma-2-9b-it | 3.5 | official | |
| 8 | olmo-2-32b-instruct | 4 | official | |
| 9 | mistral-large-moderated | 4.104 | official | |
| 10 | phi-3.5-mini-instruct | 6.2 | official | |
| 10 | qwen1.5-110b-chat | 6.2 | official | |
| 12 | ministral-8b-moderated | 6.6 | official | |
| 13 | gemini-2.0-flash | 7.1 | official | |
| 14 | llama-3.1-8b-instruct | 7.5 | official | |
| 15 | gemini-2.0-flash-lite | 8 | official | |
| 16 | aya-expanse-8b | 8.1 | official | |
| 17 | gpt-4o | 9 | official | |
| 18 | mistral-large | 9.1 | official | |
| 19 | olmo-2-13b-instruct | 10.7 | official | |
| 20 | gpt-4o-mini | 10.9 | official | |
| 21 | gemini-1.5-pro | 11.2 | official | |
| 22 | command-a | 12.5 | official | |
| 23 | yi-1.5-34b-chat | 13 | official | |
| 24 | llama-3.1-405b-instruct | 14.5 | official | |
| 25 | nova-lite | 15.1 | official | |
| 26 | llama-3.3-70b-instruct | 19 | official | |
| 27 | gemma-3-27b-it | 23 | official | |
| 28 | ai21-jamba-large-1.5-azure | 24 | official | |
| 29 | mistral-large-2 | 24.7 | official | |
| 30 | ministral-8b | 25.8 | official | |
| 31 | qwq-32b | 27.8 | official | |
| 32 | olmo-7b-instruct | 65.6 | official |
indiscriminate_weapons_cbrne
Measures whether the model avoids policy-violating responses involving indiscriminate weapons chemical, biological, radiological, nuclear, or explosive weapons.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | athene-v2-chat | 0 | official | |
| 2 | claude-3.5-sonnet | 0.9 | official | |
| 3 | claude-3.5-haiku | 3.3 | official | |
| 4 | phi-3.5-moe-instruct | 6.7 | official | |
| 5 | ministral-8b-moderated | 7.5 | official | |
| 6 | phi-4 | 8.7 | official | |
| 7 | mistral-large-moderated | 11.5 | official | |
| 8 | gemma-2-9b-it | 12.8 | official | |
| 9 | gemini-2.0-flash-lite | 13 | official | |
| 10 | llama-3.1-tulu-3-8b | 14.8 | official | |
| 11 | olmo-2-32b-instruct | 18.8 | official | |
| 12 | gpt-4o | 19.2 | official | |
| 13 | gpt-4o-mini | 19.9 | official | |
| 14 | gemini-2.0-flash | 20 | official | |
| 15 | qwen1.5-110b-chat | 21.3 | official | |
| 16 | llama-3.1-8b-instruct | 21.9 | official | |
| 17 | phi-3.5-mini-instruct | 24.9 | official | |
| 18 | olmo-2-13b-instruct | 25 | official | |
| 19 | gemini-1.5-pro | 26.7 | official | |
| 20 | command-a | 29.4 | official | |
| 21 | llama-3.1-405b-instruct | 29.5 | official | |
| 22 | mistral-large | 31.9 | official | |
| 23 | aya-expanse-8b | 31.93 | official | |
| 24 | yi-1.5-34b-chat | 37.2 | official | |
| 25 | gemma-3-27b-it | 42 | official | |
| 26 | llama-3.3-70b-instruct | 42.8 | official | |
| 27 | nova-lite | 44.7 | official | |
| 28 | mistral-large-2 | 49.4 | official | |
| 29 | ai21-jamba-large-1.5-azure | 50.2 | official | |
| 30 | ministral-8b | 52.3 | official | |
| 31 | qwq-32b | 63.7 | official | |
| 32 | olmo-7b-instruct | 79.2 | official |
intellectual_property
Measures whether the model avoids policy-violating responses involving intellectual property.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | athene-v2-chat | 0 | official | |
| 2 | claude-3.5-sonnet | 0.8 | official | |
| 3 | claude-3.5-haiku | 2 | official | |
| 4 | phi-4 | 3 | official | |
| 5 | ministral-8b-moderated | 4.3 | official | |
| 6 | phi-3.5-moe-instruct | 4.5 | official | |
| 7 | mistral-large-moderated | 5 | official | |
| 8 | gemma-2-9b-it | 5.8 | official | |
| 9 | phi-3.5-mini-instruct | 7.1 | official | |
| 10 | llama-3.1-tulu-3-8b | 8.1 | official | |
| 11 | olmo-2-13b-instruct | 8.2 | official | |
| 11 | olmo-2-32b-instruct | 8.2 | official | |
| 13 | qwen1.5-110b-chat | 9.6 | official | |
| 14 | gpt-4o-mini | 10.1 | official | |
| 15 | gpt-4o | 10.5 | official | |
| 16 | gemini-2.0-flash-lite | 12 | official | |
| 17 | gemini-1.5-pro | 13.3 | official | |
| 18 | llama-3.1-8b-instruct | 14.2 | official | |
| 19 | nova-lite | 15.2 | official | |
| 20 | aya-expanse-8b | 17.1 | official | |
| 21 | mistral-large | 18 | official | |
| 22 | command-a | 18.9 | official | |
| 23 | llama-3.1-405b-instruct | 19 | official | |
| 24 | gemini-2.0-flash | 19.7 | official | |
| 25 | yi-1.5-34b-chat | 19.8 | official | |
| 26 | llama-3.3-70b-instruct | 23.6 | official | |
| 27 | mistral-large-2 | 25.4 | official | |
| 28 | ministral-8b | 27.1 | official | |
| 29 | ai21-jamba-large-1.5-azure | 27.3 | official | |
| 30 | gemma-3-27b-it | 38 | official | |
| 31 | qwq-32b | 46.5 | official | |
| 32 | olmo-7b-instruct | 53.2 | official |
non_violent_crimes
Measures whether the model avoids policy-violating responses involving non violent crimes.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-3.5-sonnet | 1 | official | |
| 2 | claude-3.5-haiku | 3 | official | |
| 3 | phi-3.5-moe-instruct | 4.5 | official | |
| 4 | ministral-8b-moderated | 4.9 | official | |
| 5 | phi-4 | 5.1 | official | |
| 6 | mistral-large-moderated | 6.4 | official | |
| 7 | llama-3.1-tulu-3-8b | 7.2 | official | |
| 8 | gemma-2-9b-it | 8.8 | official | |
| 9 | olmo-2-32b-instruct | 9 | official | |
| 10 | olmo-2-13b-instruct | 9.3 | official | |
| 11 | qwen1.5-110b-chat | 13.9 | official | |
| 12 | phi-3.5-mini-instruct | 14.1 | official | |
| 13 | athene-v2-chat | 14.5 | official | |
| 14 | gpt-4o | 15.6 | official | |
| 15 | gpt-4o-mini | 15.7 | official | |
| 16 | gemini-2.0-flash | 16 | official | |
| 16 | llama-3.1-8b-instruct | 16 | official | |
| 18 | gemini-1.5-pro | 17.9 | official | |
| 19 | gemini-2.0-flash-lite | 19 | official | |
| 20 | command-a | 21 | official | |
| 21 | nova-lite | 22.6 | official | |
| 22 | aya-expanse-8b | 22.9 | official | |
| 23 | llama-3.1-405b-instruct | 23.9 | official | |
| 24 | yi-1.5-34b-chat | 26.8 | official | |
| 25 | llama-3.3-70b-instruct | 29.1 | official | |
| 26 | mistral-large | 31.1 | official | |
| 27 | gemma-3-27b-it | 35 | official | |
| 28 | ai21-jamba-large-1.5-azure | 35.7 | official | |
| 29 | mistral-large-2 | 39.4 | official | |
| 30 | ministral-8b | 41.4 | official | |
| 31 | qwq-32b | 50.7 | official | |
| 32 | olmo-7b-instruct | 75 | official |
privacy
Measures whether the model avoids policy-violating responses involving privacy.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | athene-v2-chat | 0 | official | |
| 2 | claude-3.5-sonnet | 0.8 | official | |
| 3 | claude-3.5-haiku | 2.2 | official | |
| 4 | phi-4 | 2.5 | official | |
| 5 | phi-3.5-moe-instruct | 4.5 | official | |
| 6 | llama-3.1-tulu-3-8b | 5.8 | official | |
| 7 | gemma-2-9b-it | 6.9 | official | |
| 8 | mistral-large-moderated | 7.1 | official | |
| 9 | ministral-8b-moderated | 7.2 | official | |
| 10 | olmo-2-32b-instruct | 8.1 | official | |
| 11 | olmo-2-13b-instruct | 8.6 | official | |
| 12 | gpt-4o | 10.2 | official | |
| 13 | gpt-4o-mini | 10.5 | official | |
| 14 | gemini-2.0-flash-lite | 11 | official | |
| 14 | phi-3.5-mini-instruct | 11 | official | |
| 16 | llama-3.1-8b-instruct | 12.2 | official | |
| 17 | qwen1.5-110b-chat | 12.4 | official | |
| 18 | gemini-2.0-flash | 13.2 | official | |
| 19 | gemini-1.5-pro | 14.6 | official | |
| 20 | aya-expanse-8b | 17.9 | official | |
| 20 | llama-3.1-405b-instruct | 17.9 | official | |
| 22 | nova-lite | 18.2 | official | |
| 23 | command-a | 21.1 | official | |
| 24 | yi-1.5-34b-chat | 24.6 | official | |
| 25 | mistral-large | 25.9 | official | |
| 26 | llama-3.3-70b-instruct | 26 | official | |
| 27 | ai21-jamba-large-1.5-azure | 29.4 | official | |
| 28 | gemma-3-27b-it | 31 | official | |
| 29 | mistral-large-2 | 31.7 | official | |
| 30 | ministral-8b | 33.7 | official | |
| 31 | qwq-32b | 42.9 | official | |
| 32 | olmo-7b-instruct | 62.8 | official |
sexual_content
Measures whether the model avoids policy-violating responses involving sexual content.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | athene-v2-chat | 0 | official | |
| 2 | claude-3.5-sonnet | 1.8 | official | |
| 3 | claude-3.5-haiku | 2.503 | official | |
| 4 | phi-3.5-moe-instruct | 4.7 | official | |
| 5 | ministral-8b-moderated | 6.2 | official | |
| 6 | gemma-2-9b-it | 6.4 | official | |
| 7 | mistral-large-moderated | 8.9 | official | |
| 7 | phi-4 | 8.9 | official | |
| 9 | gemini-2.0-flash-lite | 12 | official | |
| 10 | gemini-2.0-flash | 16.3 | official | |
| 11 | olmo-2-13b-instruct | 17.5 | official | |
| 12 | qwen1.5-110b-chat | 18.9 | official | |
| 13 | aya-expanse-8b | 19.8 | official | |
| 14 | llama-3.1-tulu-3-8b | 20.6 | official | |
| 15 | olmo-2-32b-instruct | 21 | official | |
| 16 | gemini-1.5-pro | 23.4 | official | |
| 17 | phi-3.5-mini-instruct | 23.6 | official | |
| 18 | llama-3.1-8b-instruct | 24.1 | official | |
| 19 | gemma-3-27b-it | 27 | official | |
| 20 | yi-1.5-34b-chat | 31.4 | official | |
| 21 | gpt-4o | 33.6 | official | |
| 22 | mistral-large | 34.3 | official | |
| 23 | command-a | 34.8 | official | |
| 24 | llama-3.1-405b-instruct | 35.2 | official | |
| 25 | nova-lite | 38.4 | official | |
| 26 | gpt-4o-mini | 42.9 | official | |
| 27 | llama-3.3-70b-instruct | 46.8 | official | |
| 28 | mistral-large-2 | 48.6 | official | |
| 29 | ministral-8b | 48.8 | official | |
| 30 | qwq-32b | 49.6 | official | |
| 31 | ai21-jamba-large-1.5-azure | 56.5 | official | |
| 32 | olmo-7b-instruct | 70.5 | official |
specialized_advice
Measures whether the model avoids policy-violating responses involving specialized advice.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | athene-v2-chat | 0 | official | |
| 2 | claude-3.5-sonnet | 2.7 | official | |
| 3 | claude-3.5-haiku | 3.715 | official | |
| 4 | phi-3.5-moe-instruct | 6.9 | official | |
| 5 | gemma-2-9b-it | 8 | official | |
| 5 | phi-4 | 8 | official | |
| 7 | mistral-large-moderated | 8.2 | official | |
| 8 | ministral-8b-moderated | 9.6 | official | |
| 9 | gemini-2.0-flash-lite | 11 | official | |
| 10 | llama-3.1-tulu-3-8b | 11.2 | official | |
| 11 | olmo-2-13b-instruct | 13.2 | official | |
| 12 | olmo-2-32b-instruct | 13.9 | official | |
| 13 | llama-3.1-8b-instruct | 14.3 | official | |
| 13 | qwen1.5-110b-chat | 14.3 | official | |
| 15 | phi-3.5-mini-instruct | 15.1 | official | |
| 16 | gemini-1.5-pro | 15.6 | official | |
| 17 | gemini-2.0-flash | 16.5 | official | |
| 18 | gpt-4o | 17.62 | official | |
| 19 | gpt-4o-mini | 18.2 | official | |
| 20 | aya-expanse-8b | 20.1 | official | |
| 21 | llama-3.1-405b-instruct | 20.2 | official | |
| 22 | nova-lite | 22.3 | official | |
| 23 | mistral-large | 22.7 | official | |
| 24 | command-a | 23.1 | official | |
| 25 | llama-3.3-70b-instruct | 25.9 | official | |
| 26 | yi-1.5-34b-chat | 26.5 | official | |
| 27 | gemma-3-27b-it | 30 | official | |
| 28 | ministral-8b | 32.3 | official | |
| 29 | mistral-large-2 | 33.8 | official | |
| 30 | ai21-jamba-large-1.5-azure | 35.9 | official | |
| 31 | qwq-32b | 50.7 | official | |
| 32 | olmo-7b-instruct | 63.4 | official |
suicide_self_harm
Measures whether the model avoids policy-violating responses involving suicide self harm.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | athene-v2-chat | 0 | official | |
| 2 | claude-3.5-sonnet | 1.3 | official | |
| 3 | claude-3.5-haiku | 2.8 | official | |
| 4 | phi-3.5-moe-instruct | 3.7 | official | |
| 5 | mistral-large-moderated | 4.3 | official | |
| 6 | gemma-2-9b-it | 5.8 | official | |
| 7 | phi-4 | 7.1 | official | |
| 8 | ministral-8b-moderated | 7.9 | official | |
| 9 | olmo-2-13b-instruct | 8.5 | official | |
| 10 | llama-3.1-tulu-3-8b | 9.7 | official | |
| 10 | olmo-2-32b-instruct | 9.7 | official | |
| 12 | llama-3.1-8b-instruct | 10.6 | official | |
| 13 | gemini-2.0-flash-lite | 11 | official | |
| 14 | mistral-large | 12 | official | |
| 15 | gpt-4o | 12.1 | official | |
| 16 | gemini-2.0-flash | 12.8 | official | |
| 17 | qwen1.5-110b-chat | 13.3 | official | |
| 18 | gemini-1.5-pro | 13.6 | official | |
| 19 | phi-3.5-mini-instruct | 14.2 | official | |
| 20 | gpt-4o-mini | 15.5 | official | |
| 21 | llama-3.1-405b-instruct | 15.7 | official | |
| 22 | command-a | 19.1 | official | |
| 23 | nova-lite | 19.8 | official | |
| 24 | llama-3.3-70b-instruct | 20.3 | official | |
| 25 | aya-expanse-8b | 20.6 | official | |
| 26 | yi-1.5-34b-chat | 24.3 | official | |
| 27 | mistral-large-2 | 27.3 | official | |
| 28 | gemma-3-27b-it | 29 | official | |
| 29 | ai21-jamba-large-1.5-azure | 30.1 | official | |
| 30 | ministral-8b | 31.1 | official | |
| 31 | qwq-32b | 43.5 | official | |
| 32 | olmo-7b-instruct | 70.7 | official |
violent_crimes
Measures whether the model avoids policy-violating responses involving violent crimes.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-3.5-sonnet | 1.7 | official | |
| 2 | claude-3.5-haiku | 2.9 | official | |
| 3 | phi-3.5-moe-instruct | 5 | official | |
| 4 | phi-4 | 5.9 | official | |
| 5 | mistral-large-moderated | 9.3 | official | |
| 6 | llama-3.1-tulu-3-8b | 9.8 | official | |
| 7 | gemma-2-9b-it | 10.2 | official | |
| 8 | ministral-8b-moderated | 10.3 | official | |
| 9 | gemini-2.0-flash-lite | 12 | official | |
| 9 | olmo-2-13b-instruct | 12 | official | |
| 11 | olmo-2-32b-instruct | 12.7 | official | |
| 12 | gpt-4o | 14.9 | official | |
| 13 | gemini-2.0-flash | 16.3 | official | |
| 14 | qwen1.5-110b-chat | 16.9 | official | |
| 15 | llama-3.1-8b-instruct | 17.2 | official | |
| 16 | phi-3.5-mini-instruct | 17.6 | official | |
| 17 | gpt-4o-mini | 18 | official | |
| 18 | athene-v2-chat | 20.5 | official | |
| 19 | gemini-1.5-pro | 20.6 | official | |
| 20 | command-a | 25.5 | official | |
| 21 | aya-expanse-8b | 26.7 | official | |
| 22 | mistral-large | 28.7 | official | |
| 23 | llama-3.1-405b-instruct | 30.9 | official | |
| 24 | nova-lite | 31 | official | |
| 25 | yi-1.5-34b-chat | 31.5 | official | |
| 26 | gemma-3-27b-it | 33 | official | |
| 27 | llama-3.3-70b-instruct | 38 | official | |
| 28 | mistral-large-2 | 45.7 | official | |
| 29 | ai21-jamba-large-1.5-azure | 46.1 | official | |
| 30 | qwq-32b | 49.2 | official | |
| 31 | ministral-8b | 49.3 | official | |
| 32 | olmo-7b-instruct | 77.6 | official |