Evaluation profile

AA-Omniscience

1sub-evals

2.53%Safety weight

0%Freedom weight

1components

Weights below are portfolio-specific global index weights.

Model score (lower is better)Predicted score

About this eval

Hallucination propensity on difficult closed-book factual questions under explicit encouragement to abstain when uncertain.

Included in the behavior ranking.

Sub-evals

Measure	Component	Direction by ranking	Safety weight	Freedom weight
hallucination_rateaa-omniscience/aa-omniscience.csv:hallucination_rateMeasures whether the model avoids confidently giving incorrect answers across six thousand factual questions.	Safety: truthfulness_honesty:1.000aa-omniscience	Safety: lower	2.53%	—

hallucination_rate

Measures whether the model avoids confidently giving incorrect answers across six thousand factual questions.

Rank	Model	Value	Provenance
1	minicpm5-1b	0.009009	official
2	g9v3-3b	0.1166	official
3	g9v3-39a5b	0.1304	official
4	command-a-plus	0.1416	official
5	lfm2.5-2.6b	0.16	official
6	grok-4.3	0.1694	official
7	qwen3.8-27b	0.1809	official
8	minimax-m3	0.1843	official
9	grok-4.20	0.1992	official
10	quasar-438b	0.2144	official
11	minicpm5-2b	0.2191	official
12	ling-3.0-flash-vl	0.2205	official
13	k-exaone-2.0	0.226	official
14	grok-4.6	0.24	official
15	solar-pro-4	0.244	official
16	mimo-v2.5-pro	0.247	official
17	solar-open2-250b	0.2538	official
18	granite-4.2-30b	0.2558	official
19	qwen3.7-max	0.2563	official
20	claude-haiku-4.5	0.2571	official
21	grok-3-mini	0.2582	official
22	k2-horizon-375b-a23b	0.2585	official
23	glm-5.2	0.263	official
24	granite-4.2-3b	0.2634	official
25	glm-5.3-flash	0.2763	official
26	qwen3.7-plus	0.2766	official
27	claude-sonnet-4	0.2912	official
28	glm-5.3	0.2955	official
29	nemotron-3-ultra-550b-a55b	0.2969	official
30	glm-5.1	0.2995	official
31	mimo-v2-pro	0.2998	official
32	gemma-3-270m	0.3042	official
33	ling-3.0-tiny	0.3046	official
34	k2-horizon-mova-36b-a4b	0.3053	official
35	gemma-4-e4b	0.3094	official
36	muse-spark-1.3	0.3146	official
37	mimo-v2.5	0.3193	official
38	granite-4.2-8b	0.3197	official
39	gemma-4-e2b	0.3236	official
40	agnes-2.5-pro-beta	0.328	official
41	a-x-k2	0.3299	official
42	muse-spark-1.2	0.3329	official
43	k2-horizon-7b	0.3363	official
44	gemini-3.5-flash-lite	0.3436	official
45	qwen3.6-plus	0.3464	official
46	glm-5	0.3532	official
47	minimax-m2.7	0.3556	official
48	qwen3.5-omni-plus	0.37	official
49	nemotron-3.5-lightning	0.3757	official
50	gpt-4o	0.3792	official
51	motif-3	0.3877	official
52	qwen3.8-max	0.3918	official
53	claude-opus-4.8	0.3925	official
54	claude-sonnet-5	0.3937	official
55	claude-3.7-sonnet	0.3997	official
56	mimo-v2-omni	0.4005	official
57	kimi-k2.6	0.405	official
58	claude-3.5-haiku	0.4115	official
59	claude-opus-4.7	0.4229	official
60	llama-3.1-8b-instruct	0.4304	official
61	jt-4.1-flash-236b-a21b	0.4367	official
62	ling-3.0-flash	0.4405	official
63	k2-horizon-3.7b	0.4414	official
64	gpt-5-mini	0.4432	official
65	gpt-6-astra	0.4477	official
66	qwen3-coder-480b-a35b-instruct	0.4493	official
67	qwen3.8-flash-next	0.4532	official
68	qwen3.6-max-preview	0.4619	official
69	lfm2.5-8b-a1b	0.4691	official
70	agnes-3.0-flash	0.4838	official
71	mimo-v2-flash	0.4841	official
72	claude-sonnet-4.6	0.4845	official
73	claude-sonnet-4.5	0.4908	official
74	qwen3.6-27b	0.4927	official
75	kimi-k2.5	0.4996	official
76	muse-spark-1.1	0.4998	official
77	qwen3.6-35b-a3b	0.505	official
78	gemini-3.1-pro-preview	0.5087	official
79	gpt-5.4-nano	0.5113	official
80	glm-4.6v	0.5146	official
81	gpt-5.1-codex-mini	0.5158	official
82	gpt-5.1	0.5189	official
83	gemini-3.8-flash	0.519	official
84	llama-3.1-405b-instruct	0.5244	official
85	gpt-5-nano	0.5262	official
86	kimi-k3	0.532	official
87	grok-4.5	0.5415	official
88	gemini-3.6-flash	0.5563	official
89	kat-coder-pro-v2	0.5598	official
90	nova-lite	0.5761	official
91	nanbeige4.1-3b	0.581	official
92	claude-opus-5	0.5955	official
93	magistral-medium	0.5966	official
94	nvidia-nemotron-nano-9b-v2	0.6049	official
95	mistral-medium-3	0.609	official
96	claude-opus-4.5	0.61	official
97	qwen3.5-0.8b	0.6135	official
98	gpt-5.2	0.616	official
99	gemini-3.5-flash	0.6176	official
100	devstral	0.6219	official
101	glm-5-turbo	0.6264	official
102	claude-opus-4.6	0.6284	official
103	lfm2.2-6b	0.6289	official
104	inkling-small	0.6299	official
105	claude-fable-5	0.6364	official
106	jt-35b-flash	0.6384	official
107	grok-4	0.6449	official
108	gemini-3.7-flash	0.6453	official
109	nova-micro	0.6508	official
110	gpt-5.5-instant	0.653	official
111	claude-fable-5.1	0.6563	official
112	olmo-3.1-32b-think	0.6629	official
113	llama-3.2-1b-instruct	0.6634	official
114	mistral-small-4	0.665	official
115	ernie-4.5-300b-a47b	0.6693	official
116	llama-nemotron-super-49b-v1.5	0.6726	official
117	kat-coder-pro-v1	0.6736	official
118	glm-4.6	0.6757	official
119	inkling	0.6766	official
120	mistral-large-2	0.6768	official
121	grok-4-fast	0.6827	official
122	minimax-m2.1	0.6854	official
123	glm-5v-turbo	0.688	official
124	lfm2-24b-a2b	0.6903	official
125	jamba-reasoning-3b	0.6904	official
126	gpt-4	0.6941	official
127	nova-2.0-pro	0.6952	official
128	o1	0.6963	official
129	glm-4.5	0.7015	official
130	k2-think-v2	0.7117	official
131	llama-3.1-nemotron-instruct-70b	0.7126	official
132	gemini-2.5-flash-lite	0.7171	official
133	nova-premier	0.7192	official
134	kimi-k2	0.7267	official
135	olmo-3.1-32b-instruct	0.7324	official
136	gpt-5.2-codex	0.7336	official
137	gemma-4-12b	0.7341	official
138	grok-4.1-fast	0.7341	official
139	llama-3.3-nemotron-super-49b	0.7369	official
140	hy3	0.7413	official
141	qwen3.5-2b	0.7424	official
142	deepseek-v3.1-terminus	0.7483	official
143	gpt-5.6-luna	0.75	official
144	qwen3.5-27b	0.7513	official
145	longcat-2.0	0.7539	official
146	gpt-5-codex	0.7542	official
147	qwen3-14b-instruct	0.7582	official
148	k2-v2	0.7604	official
149	granite-4.0-350m	0.7613	official
150	mistral-small-3.2	0.7681	official
151	gpt-5.1-codex	0.772	official
152	command-a	0.7732	official
153	qwen3-235b-a22b-instruct	0.774	official
154	mistral-small-3.1	0.7756	official
155	nova-pro	0.7771	official
156	phi-4-mini	0.7799	official
157	llama-3.1-70b-instruct	0.7821	official
158	apodex-1.1	0.7843	official
159	gpt-5	0.7902	official
160	grok-code-fast-1	0.7929	official
161	llama-4-scout	0.7935	official
162	exaone-4.5-33b	0.7992	official
163	doubao-seed-code	0.7999	official
164	cogito-v2.1	0.8016	official
165	ministral-3-3b	0.8019	official
166	hermes-4-llama-3.1-405b	0.8023	official
167	qwen3-coder-30b-a3b-instruct	0.803	official
168	o4-mini	0.8045	official
169	claude-3-haiku	0.8045	official
170	tri-21b-think-preview	0.8053	official
171	qwen3-30b-a3b-instruct	0.8073	official
172	o3-mini	0.8108	official
173	deepseek-r1-distill-llama-70b	0.8109	official
174	hyperclova-x-seed-think-32b	0.8119	official
175	phi-4	0.8119	official
176	llama-3.1-nemotron-ultra-253b-v1	0.8123	official
177	mistral-7b-instruct	0.8126	official
178	lfm2.1-2b	0.8138	official
179	mistral-medium-3.5	0.8159	official
180	devstral-small	0.8162	official
181	granite-4.0-h-nano-1b	0.8172	official
182	qwen3-4b-instruct	0.8174	official
183	llama-3.2-instruct-11b-vision	0.8188	official
184	muse-glimmer	0.8194	official
185	gemma-4-31b-it	0.8195	official
186	exaone-4.0-32b	0.821	official
187	qwen3-32b-instruct	0.8212	official
188	gemma-3-1b	0.8229	official
189	hypernova-60b	0.8233	official
190	kimi-k2.7-code	0.8238	official
191	deepseek-v3.1	0.8251	official
192	gpt-4.1-nano	0.8256	official
193	gpt-5.4	0.8258	official
194	gemini-2.5-flash	0.826	official
195	gemini-3.1-flash-lite	0.8266	official
196	hermes-4-llama-3.1-70b	0.8268	official
197	qwen3.5-397b-a17b	0.8269	official
198	apertus-70b-instruct	0.8282	official
199	north-mini-code	0.8321	official
200	nemotron-3-nano-30b-a3b	0.8327	official
201	step-3-vl-10b	0.8332	official
202	magistral-small	0.8345	official
203	deepseek-r1	0.8345	official
204	qwen3.5-9b	0.8361	official
205	qwen3-vl-32b	0.8368	official
206	muse-spark	0.8419	official
207	mistral-medium-3.1	0.8458	official
208	nova-2.0-omni	0.847	official
209	glm-4.5v	0.8478	official
210	step-3.7-flash	0.8498	official
211	qwen3-vl-235b-a22b	0.8515	official
212	deepseek-v3.2	0.8518	official
213	devstral-2	0.8527	official
214	apriel-v1.5-15b-thinker	0.8528	official
215	ring-2.6-1t	0.8533	official
216	nemotron-cascade-2-30b-a3b	0.8536	official
217	tri-21b-think-v0.5	0.8542	official
218	qwen3.5-35b-a3b	0.8543	official
219	qwen2.5-72b-instruct	0.8546	official
220	nemotron-3-nano-omni-30b-a3b	0.8565	official
221	olmo-3-32b-think	0.8567	official
222	k2-horizon-0.9b	0.8569	official
223	deepseek-v3	0.8586	official
224	nova-2.0-lite	0.8587	official
225	trinity-large	0.8589	official
226	mistral-large-3	0.8596	official
227	gemini-2.0-flash	0.8624	official
228	deepseek-r1-qwen3-8b	0.8635	official
229	seed-oss-36b-instruct	0.8639	official
230	gemma-4-26b-a4b-it	0.8644	official
231	qwen3-30b-a3b	0.8654	official
232	qwen3.5-4b	0.8655	official
233	gpt-oss-20b	0.8671	official
234	nemotron-3-super-120b-a12b	0.8696	official
235	ernie-5.0-thinking-preview	0.8701	official
236	qwen3-next-80b-a3b	0.8701	official
237	qwen3.5-122b-a10b	0.8709	official
238	granite-4.1-8b	0.8713	official
239	devstral-small-2	0.8729	official
240	grok-3	0.8732	official
241	hy3-preview	0.874	official
242	llama-3-70b-instruct	0.8754	official
243	intellect-3	0.8759	official
244	granite-4.0-h-small	0.8785	official
245	deepseek-v4-pro	0.8785	official
246	gpt-5.6-terra	0.8788	official
247	gpt-5.5	0.8804	official
248	granite-4.0-h-350m	0.8806	official
249	agnes-2.5-pro-alpha	0.8807	official
250	o3	0.8809	official
251	minimax-m2.5	0.8812	official
252	solar-pro-3	0.8822	official
253	step-3.5-flash	0.8849	official
254	nvidia-nemotron-3-nano-4b	0.8857	official
255	falcon-h1r-7b	0.8877	official
256	k-exaone	0.8884	official
257	llama-4-maverick	0.8888	official
258	deepseek-v4-flash	0.8897	official
259	qwen3-omni-30b-a3b	0.8899	official
260	deepseek-v3.2-speciale	0.8915	official
261	gpt-5.3-codex	0.8918	official
262	gpt-5.6-sol	0.8942	official
263	ling-flash-2.0	0.8948	official
264	solar-open-100b	0.8965	official
265	lfm2.5-1.2b-instruct	0.8973	official
266	ring-1t	0.8975	official
267	qwen3-vl-30b-a3b	0.8977	official
268	llama-3-8b-instruct	0.8979	official
269	gpt-5.4-mini	0.8985	official
270	qwen3-max	0.8992	official
271	qwen3-max-preview	0.8993	official
272	gemini-3-pro-preview	0.9	official
273	ring-flash-2.0	0.9007	official
274	llama-3.3-70b-instruct	0.9021	official
275	motif-2-12.7b	0.9059	official
276	sarvam-m	0.9064	official
277	qwen3-8b-instruct	0.9074	official
278	gpt-oss-120b	0.9082	official
279	diffusiongemma-26b-a4b	0.9089	official
280	gemini-2.5-pro	0.9089	official
281	minimax-m2	0.909	official
282	qwen3-0.6b-instruct	0.9094	official
283	qwen3-vl-8b-instruct	0.9101	official
284	molmo2-8b	0.911	official
285	olmo-3-7b-instruct	0.9122	official
286	mercury-2	0.9122	official
287	reka-flash-3	0.9127	official
288	jt-mini	0.913	official
289	qwen3-vl-8b	0.9141	official
290	deepseek-v4-flash-vision	0.9148	official
291	minimax-m1-80k	0.9153	official
292	qwen3-vl-235b-a22b-instruct	0.9155	official
293	deepseek-v4-flash-0731	0.917	official
294	exaone-4.0-1.2b	0.9174	official
295	qwen3-vl-4b	0.9182	official
296	qwen3-vl-32b-instruct	0.9182	official
297	apriel-v1.6-15b-thinker	0.9193	official
298	gemma-3-27b-it	0.9209	official
299	lfm2-8b-a1b	0.923	official
300	nvidia-nemotron-nano-12b-v2-vl	0.9235	official
301	gemini-3-flash-preview	0.9241	official
302	ministral-3-14b	0.9252	official
303	gpt-4.1-mini	0.9272	official
304	qwen3-next-80b-a3b-instruct	0.9275	official
305	celeris-1	0.9275	official
306	ling-2.6-1t	0.9283	official
307	glm-4.5-air	0.9288	official
308	glm-4.7	0.9291	official
309	solar-pro-2	0.9303	official
310	grok-build-0.1	0.9312	official
311	olmo-3-7b-think	0.9321	official
312	gpt-4.1	0.9331	official
313	gemma-3n-e2b	0.9338	official
314	qwen3.5-omni-flash	0.934	official
315	sarvam-105b	0.9343	official
316	granite-4.0-nano-1b	0.9352	official
317	qwen3-vl-30b-a3b-instruct	0.9366	official
318	qwen3-coder-next	0.9368	official
319	glm-4.7-flash	0.9394	official
320	ministral-3-8b	0.9416	official
321	mi-dm-k-2.5-pro	0.9427	official
322	ling-1t	0.9473	official
323	qwen3-1.7b-instruct	0.9481	official
324	deepseek-v4-pro-0813	0.9483	official
325	nex-n2-pro	0.9491	official
326	granite-4.1-30b	0.9498	official
327	jamba-1.7-large	0.9507	official
328	granite-4.1-3b	0.9508	official
329	jamba-1.7-mini	0.951	official
330	lfm2.5-vl-1.6b	0.9574	official
331	granite-4.0-micro	0.9577	official
332	ling-mini-2.0	0.9587	official
333	minicpm-v4.6-1.3b	0.9607	official
334	apertus-8b-instruct	0.9614	official
335	longcat-flash-lite	0.9616	official
336	tiny-aya-global	0.962	official
337	granite-3.3-8b-instruct	0.9622	official
338	sarvam-30b	0.9628	official
339	deepseek-v4.1-flash	0.9646	official
340	ling-2.6-flash	0.9672	official
341	gemma-3n-e4b-it	0.9707	official
342	gemma-3-12b	0.974	official
343	qwen3-vl-4b-instruct	0.9742	official
344	qwen3-omni-30b-a3b-instruct	0.9759	official
345	gemma-3-4b	0.9821	official