Evaluation profile
Pokee-Isaac model card — DTAP
2sub-evals
0.372%Safety weight
0%Freedom weight
2components
Weights below are portfolio-specific global index weights.
Model score (see each sub-eval direction)Predicted score
About this eval
Secure and useful delegated-agent behavior under direct and indirect injected attacks.
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction by ranking | Safety weight | Freedom weight |
|---|---|---|---|---|
| benign_task_success_ratepokee-isaac-model-card/dtap.csv:benign_task_success_rateMeasures ordinary task utility so inability cannot appear safe merely by suppressing attacks. | Safety: benign_helpfulness_non_overrefusal:1.000pokee-isaac-model-card | Safety: higher | 0.167% | — |
| combined_attack_success_ratepokee-isaac-model-card/dtap.csv:combined_attack_success_rateSummarizes how often direct or indirect injected attacks achieve their harmful target outcome. | Safety: responsible_agency_control:1.000pokee-isaac-model-card | Safety: lower | 0.205% | — |
benign_task_success_rate
Measures ordinary task utility so inability cannot appear safe merely by suppressing attacks.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-5.6-luna | 0.851 | official | |
| 2 | gemini-3.5-flash-lite | 0.833 | official | |
| 3 | pokee-isaac-28b-v0 | 0.825 | official | |
| 4 | qwen3.5-122b-a10b | 0.794 | official | |
| 5 | claude-haiku-4.5 | 0.713 | official | |
| 6 | nemotron-3-super-120b-a12b | 0.633 | official |
combined_attack_success_rate
Summarizes how often direct or indirect injected attacks achieve their harmful target outcome.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | pokee-isaac-28b-v0 | 0.356 | official | |
| 2 | claude-haiku-4.5 | 0.379 | official | |
| 3 | gpt-5.6-luna | 0.501 | official | |
| 4 | qwen3.5-122b-a10b | 0.54 | official | |
| 5 | nemotron-3-super-120b-a12b | 0.604 | official | |
| 6 | gemini-3.5-flash-lite | 0.663 | official |