Evaluation profile
Pokee-Isaac model card — DTAP
2sub-evals
0.497%total index weight
2components
Within-component eval weight: Benign helpfulness 2.31% · Responsible agency 1.78%.
Model score (see each sub-eval direction)Predicted score
About this eval
Secure and useful delegated-agent behavior under direct and indirect injected attacks.
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction | Total index weight | Within-component weight |
|---|---|---|---|---|
| benign_task_success_ratepokee-isaac-model-card/dtap.csv:benign_task_success_rateMeasures ordinary task utility so inability cannot appear safe merely by suppressing attacks. | benign_helpfulness_non_overrefusal:1.000pokee-isaac-model-card | Higher is better | 0.231% | Benign helpfulness 2.31% |
| combined_attack_success_ratepokee-isaac-model-card/dtap.csv:combined_attack_success_rateSummarizes how often direct or indirect injected attacks achieve their harmful target outcome. | responsible_agency_control:1.000pokee-isaac-model-card | Lower is better | 0.266% | Responsible agency 1.78% |
benign_task_success_rate
Measures ordinary task utility so inability cannot appear safe merely by suppressing attacks.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-5.6-luna | 0.851 | official | |
| 2 | gemini-3.5-flash-lite | 0.833 | official | |
| 3 | pokee-isaac-28b-v0 | 0.825 | official | |
| 4 | qwen3.5-122b-a10b | 0.794 | official | |
| 5 | claude-haiku-4.5 | 0.713 | official | |
| 6 | nemotron-3-super-120b-a12b | 0.633 | official |
combined_attack_success_rate
Summarizes how often direct or indirect injected attacks achieve their harmful target outcome.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | pokee-isaac-28b-v0 | 0.356 | official | |
| 2 | claude-haiku-4.5 | 0.379 | official | |
| 3 | gpt-5.6-luna | 0.501 | official | |
| 4 | qwen3.5-122b-a10b | 0.54 | official | |
| 5 | nemotron-3-super-120b-a12b | 0.604 | official | |
| 6 | gemini-3.5-flash-lite | 0.663 | official |