Evaluation profile
Open LLM Safety Index
2sub-evals
0.147%total index weight
1components
Within-component eval weight: Misuse resistance 1.47%.
Model score (higher is better)Predicted score
About this eval
Refusal-string robustness on two distinct fifteen-prompt harmful-request panels derived from JailbreakBench and StrongREJECT.
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction | Total index weight | Within-component weight |
|---|---|---|---|---|
| jailbreakbench_safety_rateopen-llm-safety-index/open-llm-safety-index.csv:jailbreakbench_safety_rateMeasures whether the model refuses adversarial harmful prompts sampled from JailbreakBench. | ordinary_harm_misuse_resistance:1.000open-llm-safety-index | Higher is better | 0.0733% | Misuse resistance 0.733% |
| strongreject_safety_rateopen-llm-safety-index/open-llm-safety-index.csv:strongreject_safety_rateMeasures whether the model refuses adversarial harmful prompts sampled from StrongREJECT. | ordinary_harm_misuse_resistance:1.000open-llm-safety-index | Higher is better | 0.0733% | Misuse resistance 0.733% |
jailbreakbench_safety_rate
Measures whether the model refuses adversarial harmful prompts sampled from JailbreakBench.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | phi-3-mini-4k-instruct | 0.8 | official | |
| 1 | qwen1.5-0.5b-chat | 0.8 | official | |
| 3 | qwen2-1.5b-instruct | 0.6667 | official | |
| 4 | phi-3.5-mini-instruct | 0.6 | official | |
| 4 | qwen-2.5-1.5b-instruct | 0.6 | official | |
| 4 | qwen2.5-3b-instruct | 0.6 | official | |
| 7 | qwen2-0.5b-instruct | 0.4667 | official | |
| 7 | smollm2-135m-instruct | 0.4667 | official | |
| 7 | smollm2-360m-instruct | 0.4667 | official | |
| 10 | qwen1.5-1.8b-chat | 0.4 | official | |
| 10 | qwen2.5-coder-3b-instruct | 0.4 | official | |
| 12 | qwen-2.5-0.5b-instruct | 0.3333 | official | |
| 12 | smollm2-1.7b-instruct | 0.3333 | official | |
| 14 | llama-3.2-1b-instruct | 0.2667 | official | |
| 14 | llama-3.2-3b-instruct | 0.2667 | official | |
| 16 | phi-2 | 0.1333 | official | |
| 16 | qwen2.5-coder-1.5b-instruct | 0.1333 | official | |
| 16 | qwen2.5-math-1.5b-instruct | 0.1333 | official | |
| 19 | qwen2.5-coder-0.5b-instruct | 0.0667 | official | |
| 20 | phi-1-5 | 0 | official | |
| 20 | qwen3-0.6b | 0 | official |
strongreject_safety_rate
Measures whether the model refuses adversarial harmful prompts sampled from StrongREJECT.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | qwen1.5-0.5b-chat | 1 | official | |
| 1 | smollm2-1.7b-instruct | 1 | official | |
| 1 | smollm2-135m-instruct | 1 | official | |
| 4 | smollm2-360m-instruct | 0.8667 | official | |
| 5 | llama-3.2-3b-instruct | 0.6667 | official | |
| 6 | qwen2-1.5b-instruct | 0.6 | official | |
| 7 | phi-3-mini-4k-instruct | 0.5333 | official | |
| 8 | phi-3.5-mini-instruct | 0.3333 | official | |
| 8 | qwen-2.5-1.5b-instruct | 0.3333 | official | |
| 8 | qwen2.5-3b-instruct | 0.3333 | official | |
| 11 | qwen1.5-1.8b-chat | 0.2667 | official | |
| 12 | qwen2.5-coder-0.5b-instruct | 0.2 | official | |
| 12 | qwen2.5-coder-3b-instruct | 0.2 | official | |
| 14 | llama-3.2-1b-instruct | 0.1333 | official | |
| 14 | qwen2-0.5b-instruct | 0.1333 | official | |
| 16 | qwen-2.5-0.5b-instruct | 0.0667 | official | |
| 16 | qwen2.5-coder-1.5b-instruct | 0.0667 | official | |
| 16 | qwen2.5-math-1.5b-instruct | 0.0667 | official | |
| 19 | phi-1-5 | 0 | official | |
| 19 | phi-2 | 0 | official | |
| 19 | qwen3-0.6b | 0 | official |