Evaluation profile

MedSafetyBench

1sub-evals

0.306%total index weight

1components

Within-component eval weight: Misuse resistance 3.06%.

Model score (higher is better)Predicted score

About this eval

Refusal or safe redirection rather than willingness to comply with harmful medical requests across eleven AMA-derived ethical categories.

Included in the behavior ranking.

Measure	Component	Direction	Total index weight	Within-component weight
medical_safety_scoremedsafetybench/medsafetybench.csv:medical_safety_scoreMeasures whether a model refuses or safely redirects harmful medical requests under the benchmark's documented safety prompt.	ordinary_harm_misuse_resistance:1.000medsafetybench	Higher is better	0.306%	Misuse resistance 3.06%

Measures whether a model refuses or safely redirects harmful medical requests under the benchmark's documented safety prompt.

Rank	Model	Value	Provenance
1	llama-2-7b-chat	99.81	official
2	llama-2-70b-chat	99.5	official
3	llama-2-13b-chat	99.25	official
4	gpt-3.5-turbo	98.94	official
5	gpt-4-turbo	96.53	official
6	gpt-4o	96.5	official
7	llama-3.1-70b-instruct	96.39	official
8	gpt-4o-mini	94.89	official
9	mixtral-8x7b-instruct	94	official
10	llama-3.1-8b-instruct	93.28	official
11	llama-2-13b	89.11	official
12	llama-2-70b	85.14	official
13	mistral-7b-instruct	83.81	official
14	meditron-70b	80.44	official
15	llama-2-7b	80.14	official
16	med42-70b	77.79	official
17	pythia-12b-deduped	77.36	official
18	clinicalcamel-70b	71.25	official
19	medalpaca-7b	69	official
20	medalpaca-13b	65.13	official
21	pythia-6.9b-deduped	63.85	official
22	llama-3.1-70b-base	61.47	official
23	meditron-7b	61.39	official
24	llama-7b	58.29	official
25	llama-13b	56.62	official
26	llama-3.1-8b-base	54.35	official
27	mistral-7b	53.19	official
28	mixtral-8x7b-base	49.53	official
29	vicuna-7b	46.8	official
30	vicuna-13b	45.8	official