Evaluation profile
COMPL-AI LLM RuLES Multi-Turn Rule Following
1sub-evals
0.0325%Safety weight
0.706%Freedom weight
1components
Weights below are portfolio-specific global index weights.
Model score (higher is better)Predicted score
About this eval
Multi-turn adversarial rule following.
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction by ranking | Safety weight | Freedom weight |
|---|---|---|---|---|
| scorecompl-ai/compl-ai-llm-rules.csv:scoreMeasures rule adherence across six adversarial multi-turn scenarios. | Safety: ordinary_harm_misuse_resistance:1.000 · Freedom: user_direction_following:1.000compl-ai-selected | Safety: higher · Freedom: lower | 0.0325% | 0.706% |
score
Measures rule adherence across six adversarial multi-turn scenarios.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-4-turbo | 0.8827 | official | |
| 2 | claude-3-opus | 0.7557 | official | |
| 3 | gpt-3.5-turbo | 0.655 | official | |
| 4 | yi-34b-chat | 0.5829 | official | |
| 5 | gemini-1.5-flash | 0.5169 | official | |
| 6 | gemma-2-9b-it | 0.4996 | official | |
| 7 | qwen1.5-72b-chat | 0.4856 | official | |
| 8 | llama-2-70b-chat | 0.3822 | official | |
| 9 | llama-2-13b-chat | 0.3652 | official | |
| 10 | bielik-11b-v2.3-instruct | 0.3431 | official | |
| 11 | mistral-7b | 0.2931 | official | |
| 12 | llama-2-7b-chat | 0.2699 | official | |
| 13 | mixtral-8x7b-instruct | 0.2561 | official | |
| 14 | mistral-7b-instruct | 0.1723 | official |