Evaluation profile
Microsoft Phi Safety Panels
4sub-evals
0.253%Safety weight
0.846%Freedom weight
2components
Weights below are portfolio-specific global index weights.
Model score (lower is better)Predicted score
About this eval
Harmful-content and jailbreak defect rates
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction by ranking | Safety weight | Freedom weight |
|---|---|---|---|---|
| harmful_continuationphi-system-card/phi-system-card.csv:harmful_continuationMeasures harmful-content continuation defect rate. | Safety: ordinary_harm_misuse_resistance:1.000 · Freedom: user_direction_following:1.000microsoft-phi-safety-panels | Safety: lower · Freedom: higher | 0.0281% | 0.212% |
| harmful_summarizationphi-system-card/phi-system-card.csv:harmful_summarizationMeasures harmful-content summarization defect rate. | Safety: ordinary_harm_misuse_resistance:1.000 · Freedom: user_direction_following:1.000microsoft-phi-safety-panels | Safety: lower · Freedom: higher | 0.0281% | 0.212% |
| jailbreakphi-system-card/phi-system-card.csv:jailbreakMeasures jailbreak defect rate. | Safety: ordinary_harm_misuse_resistance:1.000 · Freedom: user_direction_following:1.000microsoft-phi-safety-panels | Safety: lower · Freedom: higher | 0.0337% | 0.212% |
| third_party_harmphi-system-card/phi-system-card.csv:third_party_harmMeasures third-party harm defect rate. | Safety: human_rights_systemic_harm:1.000 · Freedom: user_direction_following:1.000microsoft-phi-safety-panels | Safety: lower · Freedom: higher | 0.163% | 0.212% |
harmful_continuation
Measures harmful-content continuation defect rate.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | phi-3-small | 0.003 | official | |
| 2 | phi-3.5-moe-instruct | 0.004 | official | |
| 3 | phi-3.5-mini-instruct | 0.006 | official | |
| 4 | phi-3-mini | 0.007 | official | |
| 5 | phi-3-medium | 0.01 | official | |
| 6 | gemma-7b-it | 0.013 | official | |
| 6 | llama-3-8b-instruct | 0.013 | official | |
| 8 | mistral-nemo | 0.025 | official | |
| 9 | mistral-7b-instruct | 0.026 | official | |
| 10 | phi-2 | 0.029 | official |
harmful_summarization
Measures harmful-content summarization defect rate.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | llama-3-8b-instruct | 0.082 | official | |
| 2 | phi-3-mini | 0.1 | official | |
| 3 | gemma-7b-it | 0.103 | official | |
| 4 | mistral-nemo | 0.11 | official | |
| 4 | phi-3-small | 0.11 | official | |
| 6 | phi-3-medium | 0.112 | official | |
| 7 | phi-3.5-mini-instruct | 0.119 | official | |
| 8 | phi-3.5-moe-instruct | 0.12 | official | |
| 9 | phi-2 | 0.144 | official | |
| 10 | mistral-7b-instruct | 0.223 | official |
jailbreak
Measures jailbreak defect rate.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | phi-3-small | 0.107 | official | |
| 2 | phi-3.5-moe-instruct | 0.11 | official | |
| 3 | phi-3-medium | 0.111 | official | |
| 4 | gemma-7b-it | 0.114 | official | |
| 5 | phi-3.5-mini-instruct | 0.119 | official | |
| 6 | phi-3-mini | 0.123 | official | |
| 7 | llama-3-8b-instruct | 0.13 | official | |
| 8 | phi-2 | 0.15 | official | |
| 9 | mistral-7b-instruct | 0.156 | official | |
| 10 | mistral-nemo | 0.174 | official |
third_party_harm
Measures third-party harm defect rate.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | phi-3.5-moe-instruct | 0.152 | official | |
| 2 | phi-2 | 0.24 | official | |
| 2 | phi-3-mini | 0.24 | official | |
| 4 | phi-3.5-mini-instruct | 0.243 | official | |
| 5 | phi-3-medium | 0.251 | official | |
| 6 | phi-3-small | 0.253 | official | |
| 7 | llama-3-8b-instruct | 0.373 | official | |
| 8 | gemma-7b-it | 0.383 | official | |
| 9 | mistral-7b-instruct | 0.562 | official | |
| 10 | mistral-nemo | 0.599 | official |