Evaluation profile
SpeechMap model completion
1sub-evals
0%Safety weight
6%Freedom weight
1components
Weights below are portfolio-specific global index weights.
Model score (higher is better)Predicted score
About this eval
Completion rather than evasion, denial, or provider error on sensitive and controversial user requests.
Not included: not yet admitted by the scoring-source policy
Sub-evals
| Measure | Component | Direction by ranking | Safety weight | Freedom weight |
|---|---|---|---|---|
| complete_pctspeechmap/speechmap.csv:complete_pctMeasures how often the deployed model configuration gives a complete answer instead of evading, denying, or returning a provider error on sensitive requests. | Freedom: user_direction_following:1.000speechmap | Freedom: higher | — | 6% |
complete_pct
Measures how often the deployed model configuration gives a complete answer instead of evading, denying, or returning a provider error on sensitive requests.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | grok-4 | 98.3 | official | |
| 2 | grok-4.1-fast | 97.9 | official | |
| 3 | grok-3-beta | 96.1 | official | |
| 4 | grok-2 | 95.9 | official | |
| 5 | gpt-4 | 95.2 | official | |
| 6 | gpt-3.5-turbo | 94.5 | official | |
| 7 | grok-4.5 | 93.1 | official | |
| 8 | mistral-large-2 | 92 | official | |
| 9 | deepseek-r1-zero | 91.6 | official | |
| 10 | mistral-small-4 | 90.9 | official | |
| 11 | deepseek-chat | 90 | official | |
| 12 | qwen3-32b | 89.8 | official | |
| 13 | deepseek-v3.2-speciale | 89.1 | official | |
| 14 | gemini-3.1-flash-lite | 88.4 | official | |
| 15 | deepseek-v3 | 87.9 | official | |
| 16 | gemini-2.5-flash | 87.7 | official | |
| 17 | gemini-3.5-flash | 87.5 | official | |
| 18 | qwen3-30b-a3b | 87.4 | official | |
| 19 | gemini-3.6-flash | 87.2 | official | |
| 20 | reka-flash-3 | 86.4 | official | |
| 21 | qwen3-14b | 86.1 | official | |
| 22 | glm-4.7-flash | 85.9 | official | |
| 23 | ministral-3-3b | 85.1 | official | |
| 24 | gemini-3.5-flash-lite | 84.9 | official | |
| 25 | gemma-4-31b-it | 84.8 | official | |
| 26 | gemini-2.5-flash-lite | 84.6 | official | |
| 27 | gpt-4o-mini | 84.5 | official | |
| 28 | qwq-32b | 83.9 | official | |
| 29 | grok-4.6 | 83.6 | official | |
| 30 | glm-4.7 | 80.3 | official | |
| 31 | muse-glimmer | 80.1 | official | |
| 32 | grok-4.3 | 79.1 | official | |
| 33 | glm-4.6 | 79 | official | |
| 34 | glm-4.5-air | 78.9 | official | |
| 35 | gpt-4-turbo | 78.4 | official | |
| 36 | mistral-7b-instruct | 78.3 | official | |
| 37 | glm-5.1 | 78.1 | official | |
| 38 | glm-5.2 | 77.8 | official | |
| 39 | qwen2.5-7b-instruct | 77.6 | official | |
| 40 | gemini-3-pro-preview | 77 | official | |
| 41 | muse-spark-1.1 | 76.8 | official | |
| 42 | gemma-4-26b-a4b-it | 76.3 | official | |
| 42 | hy3-preview | 76.3 | official | |
| 44 | nemotron-3.5-lightning | 75 | official | |
| 44 | phi-3-medium-128k-instruct | 75 | official | |
| 46 | gemma-3-27b-it | 74.5 | official | |
| 47 | glm-5 | 74.2 | official | |
| 48 | qwen2.5-72b-instruct | 73.3 | official | |
| 49 | gemini-2.5-pro | 72.6 | official | |
| 49 | gpt-4.1-mini | 72.6 | official | |
| 51 | deepseek-v4-flash-vision | 71.7 | official | |
| 52 | claude-opus-4.7 | 71.5 | official | |
| 52 | glm-5.3 | 71.5 | official | |
| 54 | olmo-3-7b-think | 71.3 | official | |
| 55 | llama-3-70b-instruct | 70.8 | official | |
| 56 | gpt-5 | 70.1 | official | |
| 56 | llama-4-scout | 70.1 | official | |
| 58 | gpt-5.2-chat | 69.6 | official | |
| 59 | o3-mini | 69.3 | official | |
| 60 | intellect-3 | 69.2 | official | |
| 61 | deepseek-v3.1-terminus | 68.6 | official | |
| 61 | glm-5-turbo | 68.6 | official | |
| 63 | gpt-4.1 | 68.4 | official | |
| 64 | glm-4.5 | 68.3 | official | |
| 65 | nemotron-3-nano-30b-a3b | 67.7 | official | |
| 66 | o1 | 67.5 | official | |
| 67 | phi-4-reasoning-plus | 67.3 | official | |
| 68 | deepseek-v3.2 | 67.2 | official | |
| 69 | qwen2.5-max | 66.8 | official | |
| 70 | kimi-k3 | 65.9 | official | |
| 71 | trinity-large | 65.5 | official | |
| 72 | glm-5.3-flash | 65.4 | official | |
| 73 | claude-fable-5 | 64.8 | official | |
| 74 | qwen3.8-max | 64.7 | official | |
| 75 | gpt-4.1-nano | 64.5 | official | |
| 76 | phi-3-mini-128k-instruct | 64.3 | official | |
| 77 | deepseek-v3.2-exp | 63.6 | official | |
| 78 | gpt-5.3-chat | 63.3 | official | |
| 79 | grok-4.20 | 62.6 | official | |
| 80 | deepseek-r1 | 62.5 | official | |
| 81 | glm-4.6v | 61.7 | official | |
| 81 | llama-3.1-70b-instruct | 61.7 | official | |
| 81 | ring-2.6-1t | 61.7 | official | |
| 84 | llama-3.2-90b-vision-instruct | 61.6 | official | |
| 85 | o4-mini | 60.9 | official | |
| 86 | claude-3.7-sonnet | 60.8 | official | |
| 87 | llama-3.3-70b-instruct | 60.7 | official | |
| 88 | longcat-flash-chat | 60.4 | official | |
| 89 | o3 | 60.2 | official | |
| 90 | gpt-4o | 60.1 | official | |
| 91 | gemini-3.1-pro-preview | 59.9 | official | |
| 92 | claude-opus-5 | 59.1 | official | |
| 93 | llama-4-maverick | 58.3 | official | |
| 94 | mercury-2 | 57 | official | |
| 95 | glm-5v-turbo | 56.9 | official | |
| 96 | gpt-5-nano | 56 | official | |
| 97 | gpt-5-mini | 55.6 | official | |
| 98 | olmo-3-32b-think | 55.2 | official | |
| 99 | phi-4-multimodal-instruct | 54.3 | official | |
| 100 | minimax-m2.1 | 54 | official | |
| 101 | gpt-4.5-preview | 53.3 | official | |
| 102 | kimi-k2 | 52.5 | official | |
| 103 | gpt-5.6-luna | 52.4 | official | |
| 104 | llama-3.1-405b-instruct | 51.7 | official | |
| 105 | claude-opus-4 | 51.1 | official | |
| 106 | claude-opus-4.5 | 50.8 | official | |
| 107 | qwen3.7-max | 50.6 | official | |
| 108 | mimo-v2-flash | 50.4 | official | |
| 109 | claude-opus-4.6 | 49.8 | official | |
| 110 | minimax-m2.7 | 49.2 | official | |
| 110 | phi-4 | 49.2 | official | |
| 112 | qwen3-next-80b-a3b-instruct | 49.1 | official | |
| 113 | longcat-2.0 | 48.9 | official | |
| 114 | gemini-2.0-flash | 48.7 | official | |
| 115 | healer-alpha | 48.6 | official | |
| 115 | step-3.5-flash | 48.6 | official | |
| 117 | laguna-s-2.1-poolside | 48.1 | official | |
| 118 | nemotron-3-ultra-550b-a55b | 47.7 | official | |
| 119 | qwen3-235b-a22b | 47.6 | official | |
| 120 | gemma-2-27b-it | 47.4 | official | |
| 121 | deepseek-v4-flash-0731 | 46.8 | official | |
| 121 | deepseek-v4-pro | 46.8 | official | |
| 123 | ling-2.6-1t | 46.7 | official | |
| 124 | gpt-5.6-sol | 45.8 | official | |
| 125 | aurora-alpha | 45.6 | official | |
| 126 | step-3.7-flash | 45.4 | official | |
| 127 | minimax-m2.5 | 45.1 | official | |
| 128 | gemma-3n-e4b-it | 44.9 | official | |
| 129 | ling-3.0-flash | 44.4 | official | |
| 130 | qwen3.5-35b-a3b | 44.3 | official | |
| 131 | deepseek-v4-flash | 44.2 | official | |
| 131 | gpt-5.6-terra | 44.2 | official | |
| 131 | qwen3.5-flash | 44.2 | official | |
| 134 | claude-sonnet-4 | 43.9 | official | |
| 135 | claude-opus-4.8 | 43.4 | official | |
| 136 | nemotron-3-super-120b-a12b | 42.5 | official | |
| 137 | claude-opus-4.1 | 42.2 | official | |
| 138 | claude-sonnet-5 | 40.9 | official | |
| 139 | gpt-5.1-chat | 40.8 | official | |
| 139 | qwen3.5-122b-a10b | 40.8 | official | |
| 139 | qwen3.7-plus | 40.8 | official | |
| 142 | olmo-3-7b-instruct | 40.7 | official | |
| 143 | ernie-4.5-300b-a47b | 40.3 | official | |
| 144 | gpt-5.5 | 40.1 | official | |
| 145 | qwen3-max | 39.4 | official | |
| 146 | qwen3.5-27b | 38.9 | official | |
| 147 | qwen3.8-27b | 38.5 | official | |
| 148 | mimo-v2-omni | 38.3 | official | |
| 149 | qwen3.5-397b-a17b | 37.4 | official | |
| 150 | inkling-small | 36.9 | official | |
| 151 | gpt-oss-120b | 36.3 | official | |
| 151 | kimi-k2.5 | 36.3 | official | |
| 153 | o1-preview | 35.8 | official | |
| 154 | nemotron-3-nano-omni-30b-a3b | 35.7 | official | |
| 155 | gpt-5.1 | 35.5 | official | |
| 156 | owl-alpha | 35.1 | official | |
| 157 | llama-3-8b-instruct | 34.4 | official | |
| 157 | mimo-v2.5 | 34.4 | official | |
| 157 | qwen3.6-plus | 34.4 | official | |
| 160 | hy3 | 34.2 | official | |
| 161 | o1-mini | 33.8 | official | |
| 162 | kimi-k2.6 | 33 | official | |
| 163 | olmo-3.1-32b-think | 32.5 | official | |
| 164 | mimo-v2.5-pro | 32.4 | official | |
| 164 | qwen3-next-80b-a3b | 32.4 | official | |
| 166 | claude-sonnet-4.5 | 32.3 | official | |
| 167 | gpt-5.4-nano | 32.2 | official | |
| 168 | seed-2.0-mini | 31.6 | official | |
| 169 | gemma-2-9b-it | 30 | official | |
| 170 | gpt-5.4-mini | 29.4 | official | |
| 171 | claude-sonnet-4.6 | 29.1 | official | |
| 172 | gpt-5.4 | 28.6 | official | |
| 173 | qwen3.7-flash | 28.3 | official | |
| 174 | minimax-m3 | 27 | official | |
| 175 | gpt-oss-20b | 26.8 | official | |
| 176 | gpt-5.2 | 26.1 | official | |
| 176 | hunter-alpha | 26.1 | official | |
| 178 | mimo-v2-pro | 25.4 | official | |
| 179 | claude-haiku-4.5 | 20.8 | official | |
| 180 | llama-3.1-8b-instruct | 16.4 | official | |
| 181 | claude-3.5-haiku | 3.8 | official |