Evaluation profile
SYCON Bench
2sub-evals
0.662%total index weight
2components
Within-component eval weight: Truthfulness 1.78% · Fairness 3.95%.
Model score (higher is better)Predicted score
About this eval
Sycophancy and resistance to false or unethical user pressure.
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction | Total index weight | Within-component weight |
|---|---|---|---|---|
| false_presupposition_tofsycon/sycon.csv:false_presupposition_tofMeasures how many conversational turns the model continues correcting a user’s false presupposition under repeated pressure. | truthfulness_honesty:1.000sycon/sycon.csv | Higher is better | 0.267% | Truthfulness 1.78% |
| unethical_queries_tofsycon/sycon.csv:unethical_queries_tofMeasures how many conversational turns the model continues resisting an unethical request under repeated user pressure. | fairness_nondiscrimination:1.000sycon/sycon.csv | Higher is better | 0.395% | Fairness 3.95% |
false_presupposition_tof
Measures how many conversational turns the model continues correcting a user’s false presupposition under repeated pressure.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | deepseek-r1 | 3.21 | official | |
| 2 | o3-mini | 2.98 | official | |
| 3 | claude-3.7-sonnet | 2.92 | official | |
| 3 | gpt-4o | 2.92 | official | |
| 5 | deepseek-v3 | 2.88 | official | |
| 6 | qwen2.5-72b-instruct | 2.35 | official | |
| 7 | qwen-2.5-14b-instruct | 2.31 | official | |
| 8 | qwen-2.5-7b-instruct | 1.93 | official | |
| 9 | llama-3.3-70b-instruct | 1.9 | official | |
| 10 | gemma-2-9b-it | 1.86 | official | |
| 11 | llama-3.1-8b-instruct | 1.45 | official |
unethical_queries_tof
Measures how many conversational turns the model continues resisting an unethical request under repeated user pressure.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-3.7-sonnet | 2.73 | official | |
| 2 | deepseek-r1 | 2.72 | official | |
| 3 | gemma-2-9b-it | 2.36 | official | |
| 4 | o3-mini | 2.31 | official | |
| 5 | deepseek-v3 | 1.99 | official | |
| 6 | llama-3.3-70b-instruct | 1.39 | official | |
| 7 | qwen2.5-72b-instruct | 1.32 | official | |
| 8 | gpt-4o | 1.23 | official | |
| 9 | qwen-2.5-14b-instruct | 1.14 | official | |
| 10 | llama-3.1-8b-instruct | 0.85 | official | |
| 11 | qwen-2.5-7b-instruct | 0.72 | official |