Evaluation profile

AA-Omniscience

1sub-evals

2.54%Safety weight

0%Freedom weight

1components

Weights below are portfolio-specific global index weights.

Model score (lower is better)Predicted score

About this eval

Hallucination propensity on difficult closed-book factual questions under explicit encouragement to abstain when uncertain.

Included in the behavior ranking.

Sub-evals

Measure	Component	Direction by ranking	Safety weight	Freedom weight
hallucination_rateaa-omniscience/aa-omniscience.csv:hallucination_rateMeasures whether the model avoids confidently giving incorrect answers across six thousand factual questions.	Safety: truthfulness_honesty:1.000aa-omniscience	Safety: lower	2.54%	—

hallucination_rate

Measures whether the model avoids confidently giving incorrect answers across six thousand factual questions.

Rank	Model	Value	Provenance
1	minicpm5-1b	0.009009	official
2	g9v3-3b	0.1166	official
3	g9v3-39a5b	0.1304	official
4	command-a-plus	0.1416	official
5	lfm2.5-2.6b	0.16	official
6	grok-4.3	0.1694	official
7	qwen3.8-27b	0.1809	official
8	minimax-m3	0.1843	official
9	grok-4.20	0.1992	official
10	quasar-438b	0.2144	official
11	minicpm5-2b	0.2191	official
12	k-exaone-2.0	0.226	official
13	grok-4.6	0.24	official
14	solar-pro-4	0.244	official
15	mimo-v2.5-pro	0.247	official
16	solar-open2-250b	0.2538	official
17	granite-4.2-30b	0.2558	official
18	qwen3.7-max	0.2563	official
19	claude-haiku-4.5	0.2571	official
20	grok-3-mini	0.2582	official
21	k2-horizon-375b-a23b	0.2585	official
22	glm-5.2	0.263	official
23	granite-4.2-3b	0.2634	official
24	glm-5.3-flash	0.2763	official
25	qwen3.7-plus	0.2766	official
26	claude-sonnet-4	0.2912	official
27	glm-5.3	0.2955	official
28	nemotron-3-ultra-550b-a55b	0.2969	official
29	glm-5.1	0.2995	official
30	mimo-v2-pro	0.2998	official
31	gemma-3-270m	0.3042	official
32	ling-3.0-tiny	0.3046	official
33	gemma-4-e4b	0.3094	official
34	muse-spark-1.3	0.3146	official
35	mimo-v2.5	0.3193	official
36	granite-4.2-8b	0.3197	official
37	gemma-4-e2b	0.3236	official
38	agnes-2.5-pro-beta	0.328	official
39	a-x-k2	0.3299	official
40	muse-spark-1.2	0.3329	official
41	gemini-3.5-flash-lite	0.3436	official
42	qwen3.6-plus	0.3464	official
43	glm-5	0.3532	official
44	minimax-m2.7	0.3556	official
45	qwen3.5-omni-plus	0.37	official
46	nemotron-3.5-lightning	0.3757	official
47	gpt-4o	0.3792	official
48	motif-3	0.3877	official
49	qwen3.8-max	0.3918	official
50	claude-opus-4.8	0.3925	official
51	claude-sonnet-5	0.3937	official
52	claude-3.7-sonnet	0.3997	official
53	mimo-v2-omni	0.4005	official
54	kimi-k2.6	0.405	official
55	claude-3.5-haiku	0.4115	official
56	claude-opus-4.7	0.4229	official
57	llama-3.1-8b-instruct	0.4304	official
58	jt-4.1-flash-236b-a21b	0.4367	official
59	ling-3.0-flash	0.4405	official
60	gpt-5-mini	0.4432	official
61	gpt-6-astra	0.4477	official
62	qwen3-coder-480b-a35b-instruct	0.4493	official
63	qwen3.8-flash-next	0.4532	official
64	qwen3.6-max-preview	0.4619	official
65	lfm2.5-8b-a1b	0.4691	official
66	mimo-v2-flash	0.4841	official
67	claude-sonnet-4.6	0.4845	official
68	claude-sonnet-4.5	0.4908	official
69	qwen3.6-27b	0.4927	official
70	kimi-k2.5	0.4996	official
71	muse-spark-1.1	0.4998	official
72	qwen3.6-35b-a3b	0.505	official
73	gemini-3.1-pro-preview	0.5087	official
74	gpt-5.4-nano	0.5113	official
75	glm-4.6v	0.5146	official
76	gpt-5.1-codex-mini	0.5158	official
77	gpt-5.1	0.5189	official
78	gemini-3.8-flash	0.519	official
79	llama-3.1-405b-instruct	0.5244	official
80	gpt-5-nano	0.5262	official
81	kimi-k3	0.532	official
82	grok-4.5	0.5415	official
83	gemini-3.6-flash	0.5563	official
84	kat-coder-pro-v2	0.5598	official
85	nova-lite	0.5761	official
86	nanbeige4.1-3b	0.581	official
87	claude-opus-5	0.5955	official
88	magistral-medium	0.5966	official
89	nvidia-nemotron-nano-9b-v2	0.6049	official
90	mistral-medium-3	0.609	official
91	claude-opus-4.5	0.61	official
92	qwen3.5-0.8b	0.6135	official
93	gpt-5.2	0.616	official
94	gemini-3.5-flash	0.6176	official
95	devstral	0.6219	official
96	glm-5-turbo	0.6264	official
97	claude-opus-4.6	0.6284	official
98	lfm2.2-6b	0.6289	official
99	inkling-small	0.6299	official
100	claude-fable-5	0.6364	official
101	jt-35b-flash	0.6384	official
102	grok-4	0.6449	official
103	gemini-3.7-flash	0.6453	official
104	nova-micro	0.6508	official
105	gpt-5.5-instant	0.653	official
106	claude-fable-5.1	0.6563	official
107	olmo-3.1-32b-think	0.6629	official
108	llama-3.2-1b-instruct	0.6634	official
109	mistral-small-4	0.665	official
110	ernie-4.5-300b-a47b	0.6693	official
111	llama-nemotron-super-49b-v1.5	0.6726	official
112	kat-coder-pro-v1	0.6736	official
113	glm-4.6	0.6757	official
114	inkling	0.6766	official
115	mistral-large-2	0.6768	official
116	grok-4-fast	0.6827	official
117	minimax-m2.1	0.6854	official
118	glm-5v-turbo	0.688	official
119	lfm2-24b-a2b	0.6903	official
120	jamba-reasoning-3b	0.6904	official
121	gpt-4	0.6941	official
122	nova-2.0-pro	0.6952	official
123	o1	0.6963	official
124	glm-4.5	0.7015	official
125	k2-think-v2	0.7117	official
126	llama-3.1-nemotron-instruct-70b	0.7126	official
127	gemini-2.5-flash-lite	0.7171	official
128	nova-premier	0.7192	official
129	kimi-k2	0.7267	official
130	olmo-3.1-32b-instruct	0.7324	official
131	gpt-5.2-codex	0.7336	official
132	gemma-4-12b	0.7341	official
133	grok-4.1-fast	0.7341	official
134	llama-3.3-nemotron-super-49b	0.7369	official
135	hy3	0.7413	official
136	qwen3.5-2b	0.7424	official
137	deepseek-v3.1-terminus	0.7483	official
138	gpt-5.6-luna	0.75	official
139	qwen3.5-27b	0.7513	official
140	longcat-2.0	0.7539	official
141	gpt-5-codex	0.7542	official
142	qwen3-14b-instruct	0.7582	official
143	k2-v2	0.7604	official
144	granite-4.0-350m	0.7613	official
145	mistral-small-3.2	0.7681	official
146	gpt-5.1-codex	0.772	official
147	command-a	0.7732	official
148	qwen3-235b-a22b-instruct	0.774	official
149	mistral-small-3.1	0.7756	official
150	nova-pro	0.7771	official
151	phi-4-mini	0.7799	official
152	llama-3.1-70b-instruct	0.7821	official
153	apodex-1.1	0.7843	official
154	gpt-5	0.7902	official
155	grok-code-fast-1	0.7929	official
156	llama-4-scout	0.7935	official
157	exaone-4.5-33b	0.7992	official
158	doubao-seed-code	0.7999	official
159	cogito-v2.1	0.8016	official
160	ministral-3-3b	0.8019	official
161	hermes-4-llama-3.1-405b	0.8023	official
162	qwen3-coder-30b-a3b-instruct	0.803	official
163	o4-mini	0.8045	official
164	claude-3-haiku	0.8045	official
165	tri-21b-think-preview	0.8053	official
166	qwen3-30b-a3b-instruct	0.8073	official
167	o3-mini	0.8108	official
168	deepseek-r1-distill-llama-70b	0.8109	official
169	hyperclova-x-seed-think-32b	0.8119	official
170	phi-4	0.8119	official
171	llama-3.1-nemotron-ultra-253b-v1	0.8123	official
172	mistral-7b-instruct	0.8126	official
173	lfm2.1-2b	0.8138	official
174	mistral-medium-3.5	0.8159	official
175	devstral-small	0.8162	official
176	granite-4.0-h-nano-1b	0.8172	official
177	qwen3-4b-instruct	0.8174	official
178	llama-3.2-instruct-11b-vision	0.8188	official
179	muse-glimmer	0.8194	official
180	gemma-4-31b-it	0.8195	official
181	exaone-4.0-32b	0.821	official
182	qwen3-32b-instruct	0.8212	official
183	gemma-3-1b	0.8229	official
184	hypernova-60b	0.8233	official
185	kimi-k2.7-code	0.8238	official
186	deepseek-v3.1	0.8251	official
187	gpt-4.1-nano	0.8256	official
188	gpt-5.4	0.8258	official
189	gemini-2.5-flash	0.826	official
190	gemini-3.1-flash-lite	0.8266	official
191	hermes-4-llama-3.1-70b	0.8268	official
192	qwen3.5-397b-a17b	0.8269	official
193	apertus-70b-instruct	0.8282	official
194	north-mini-code	0.8321	official
195	nemotron-3-nano-30b-a3b	0.8327	official
196	step-3-vl-10b	0.8332	official
197	magistral-small	0.8345	official
198	deepseek-r1	0.8345	official
199	qwen3.5-9b	0.8361	official
200	qwen3-vl-32b	0.8368	official
201	muse-spark	0.8419	official
202	mistral-medium-3.1	0.8458	official
203	nova-2.0-omni	0.847	official
204	glm-4.5v	0.8478	official
205	step-3.7-flash	0.8498	official
206	qwen3-vl-235b-a22b	0.8515	official
207	deepseek-v3.2	0.8518	official
208	devstral-2	0.8527	official
209	apriel-v1.5-15b-thinker	0.8528	official
210	ring-2.6-1t	0.8533	official
211	nemotron-cascade-2-30b-a3b	0.8536	official
212	tri-21b-think-v0.5	0.8542	official
213	qwen3.5-35b-a3b	0.8543	official
214	qwen2.5-72b-instruct	0.8546	official
215	nemotron-3-nano-omni-30b-a3b	0.8565	official
216	olmo-3-32b-think	0.8567	official
217	deepseek-v3	0.8586	official
218	nova-2.0-lite	0.8587	official
219	trinity-large	0.8589	official
220	mistral-large-3	0.8596	official
221	gemini-2.0-flash	0.8624	official
222	deepseek-r1-qwen3-8b	0.8635	official
223	seed-oss-36b-instruct	0.8639	official
224	gemma-4-26b-a4b-it	0.8644	official
225	qwen3-30b-a3b	0.8654	official
226	qwen3.5-4b	0.8655	official
227	gpt-oss-20b	0.8671	official
228	nemotron-3-super-120b-a12b	0.8696	official
229	ernie-5.0-thinking-preview	0.8701	official
230	qwen3-next-80b-a3b	0.8701	official
231	qwen3.5-122b-a10b	0.8709	official
232	granite-4.1-8b	0.8713	official
233	devstral-small-2	0.8729	official
234	grok-3	0.8732	official
235	hy3-preview	0.874	official
236	llama-3-70b-instruct	0.8754	official
237	intellect-3	0.8759	official
238	granite-4.0-h-small	0.8785	official
239	deepseek-v4-pro	0.8785	official
240	gpt-5.6-terra	0.8788	official
241	gpt-5.5	0.8804	official
242	granite-4.0-h-350m	0.8806	official
243	agnes-2.5-pro-alpha	0.8807	official
244	o3	0.8809	official
245	minimax-m2.5	0.8812	official
246	solar-pro-3	0.8822	official
247	step-3.5-flash	0.8849	official
248	nvidia-nemotron-3-nano-4b	0.8857	official
249	falcon-h1r-7b	0.8877	official
250	k-exaone	0.8884	official
251	llama-4-maverick	0.8888	official
252	deepseek-v4-flash	0.8897	official
253	qwen3-omni-30b-a3b	0.8899	official
254	deepseek-v3.2-speciale	0.8915	official
255	gpt-5.3-codex	0.8918	official
256	gpt-5.6-sol	0.8942	official
257	ling-flash-2.0	0.8948	official
258	solar-open-100b	0.8965	official
259	lfm2.5-1.2b-instruct	0.8973	official
260	ring-1t	0.8975	official
261	qwen3-vl-30b-a3b	0.8977	official
262	llama-3-8b-instruct	0.8979	official
263	gpt-5.4-mini	0.8985	official
264	qwen3-max	0.8992	official
265	qwen3-max-preview	0.8993	official
266	gemini-3-pro-preview	0.9	official
267	ring-flash-2.0	0.9007	official
268	llama-3.3-70b-instruct	0.9021	official
269	motif-2-12.7b	0.9059	official
270	sarvam-m	0.9064	official
271	qwen3-8b-instruct	0.9074	official
272	gpt-oss-120b	0.9082	official
273	diffusiongemma-26b-a4b	0.9089	official
274	gemini-2.5-pro	0.9089	official
275	minimax-m2	0.909	official
276	qwen3-0.6b-instruct	0.9094	official
277	qwen3-vl-8b-instruct	0.9101	official
278	molmo2-8b	0.911	official
279	olmo-3-7b-instruct	0.9122	official
280	mercury-2	0.9122	official
281	reka-flash-3	0.9127	official
282	jt-mini	0.913	official
283	qwen3-vl-8b	0.9141	official
284	deepseek-v4-flash-vision	0.9148	official
285	minimax-m1-80k	0.9153	official
286	qwen3-vl-235b-a22b-instruct	0.9155	official
287	deepseek-v4-flash-0731	0.917	official
288	exaone-4.0-1.2b	0.9174	official
289	qwen3-vl-4b	0.9182	official
290	qwen3-vl-32b-instruct	0.9182	official
291	apriel-v1.6-15b-thinker	0.9193	official
292	gemma-3-27b-it	0.9209	official
293	lfm2-8b-a1b	0.923	official
294	nvidia-nemotron-nano-12b-v2-vl	0.9235	official
295	gemini-3-flash-preview	0.9241	official
296	ministral-3-14b	0.9252	official
297	gpt-4.1-mini	0.9272	official
298	qwen3-next-80b-a3b-instruct	0.9275	official
299	celeris-1	0.9275	official
300	ling-2.6-1t	0.9283	official
301	glm-4.5-air	0.9288	official
302	glm-4.7	0.9291	official
303	solar-pro-2	0.9303	official
304	grok-build-0.1	0.9312	official
305	olmo-3-7b-think	0.9321	official
306	gpt-4.1	0.9331	official
307	gemma-3n-e2b	0.9338	official
308	qwen3.5-omni-flash	0.934	official
309	sarvam-105b	0.9343	official
310	granite-4.0-nano-1b	0.9352	official
311	qwen3-vl-30b-a3b-instruct	0.9366	official
312	qwen3-coder-next	0.9368	official
313	glm-4.7-flash	0.9394	official
314	ministral-3-8b	0.9416	official
315	mi-dm-k-2.5-pro	0.9427	official
316	ling-1t	0.9473	official
317	qwen3-1.7b-instruct	0.9481	official
318	deepseek-v4-pro-0813	0.9483	official
319	nex-n2-pro	0.9491	official
320	granite-4.1-30b	0.9498	official
321	jamba-1.7-large	0.9507	official
322	granite-4.1-3b	0.9508	official
323	jamba-1.7-mini	0.951	official
324	lfm2.5-vl-1.6b	0.9574	official
325	granite-4.0-micro	0.9577	official
326	ling-mini-2.0	0.9587	official
327	minicpm-v4.6-1.3b	0.9607	official
328	apertus-8b-instruct	0.9614	official
329	longcat-flash-lite	0.9616	official
330	tiny-aya-global	0.962	official
331	granite-3.3-8b-instruct	0.9622	official
332	sarvam-30b	0.9628	official
333	ling-2.6-flash	0.9672	official
334	gemma-3n-e4b-it	0.9707	official
335	gemma-3-12b	0.974	official
336	qwen3-vl-4b-instruct	0.9742	official
337	qwen3-omni-30b-a3b-instruct	0.9759	official
338	gemma-3-4b	0.9821	official