Evaluation profile

AA-Omniscience

1sub-evals

3.79%total index weight

1components

Within-component eval weight: Truthfulness 25.3%.

Model score (lower is better)Predicted score

About this eval

Hallucination propensity on difficult closed-book factual questions under explicit encouragement to abstain when uncertain.

Included in the behavior ranking.

Sub-evals

Measure	Component	Direction	Total index weight	Within-component weight
hallucination_rateaa-omniscience/aa-omniscience.csv:hallucination_rateMeasures whether the model avoids confidently giving incorrect answers across six thousand factual questions.	truthfulness_honesty:1.000aa-omniscience	Lower is better	3.79%	Truthfulness 25.3%

hallucination_rate

Measures whether the model avoids confidently giving incorrect answers across six thousand factual questions.

Rank	Model	Value	Provenance
1	minicpm5-1b	0.008506	official
2	g9v3-3b	0.1211	official
3	command-a-plus	0.1408	official
4	grok-4.3	0.1603	official
5	minimax-m3	0.1608	official
6	grok-4.20	0.193	official
7	qwen3.7-max	0.2294	official
8	mimo-v2.5-pro	0.2455	official
9	claude-haiku-4.5	0.2471	official
10	grok-3-mini	0.2535	official
11	qwen3.7-plus	0.2549	official
12	glm-5.2	0.2813	official
13	claude-sonnet-4	0.2852	official
14	nemotron-3-ultra-550b-a55b	0.2853	official
15	glm-5.1	0.2938	official
16	mimo-v2-pro	0.2992	official
17	gemma-4-e4b	0.3006	official
18	qwen3.6-plus	0.3197	official
19	mimo-v2.5	0.3215	official
20	gemma-3-270m	0.3246	official
21	gemma-4-e2b	0.3308	official
22	gemini-3.5-flash-lite	0.3352	official
23	glm-5	0.34	official
24	minimax-m2.7	0.3442	official
25	claude-opus-4.8	0.3589	official
26	qwen3.5-omni-plus	0.3598	official
27	claude-opus-4.7	0.3618	official
28	claude-sonnet-5	0.3725	official
29	gpt-4o	0.3789	official
30	muse-spark-1.1	0.3809	official
31	claude-3.7-sonnet	0.3851	official
32	kimi-k2.6	0.3926	official
33	mimo-v2-omni	0.3987	official
34	qwen3.5-0-8b	0.4006	official
35	claude-3.5-haiku	0.4154	official
36	llama-3.1-8b-instruct	0.4177	official
37	gpt-5-mini	0.424	official
38	jt-4-1-flash-236b-a21b	0.4326	official
39	qwen3-coder-480b-a35b-instruct	0.4329	official
40	qwen3.6-max-preview	0.4418	official
41	claude-sonnet-4.6	0.4614	official
42	qwen3.5-2b	0.4709	official
43	claude-sonnet-4.5	0.474	official
44	kimi-k2.5	0.4786	official
45	qwen3.6-27b	0.4828	official
46	mimo-v2-flash	0.4839	official
47	motif-0714	0.4852	official
48	gpt-5-nano	0.4885	official
49	qwen3.6-35b-a3b	0.497	official
50	gemini-3.1-pro-preview	0.4987	official
51	glm-4.6v	0.4989	official
52	claude-opus-5	0.5007	official
53	gpt-5.4-nano	0.504	official
54	gpt-5.1-codex-mini	0.5077	official
55	kimi-k3	0.5094	official
56	llama-3.1-405b-instruct	0.5103	official
57	gpt-5.1	0.5132	official
58	lfm2.5-8b-a1b	0.5284	official
59	grok-4.5	0.5353	official
60	gemini-3.6-flash	0.5355	official
61	claude-fable-5	0.5485	official
62	nova-lite	0.5556	official
63	kat-coder-pro-v2	0.5583	official
64	inkling-small	0.5686	official
65	nanbeige4-1-3b	0.5698	official
66	llama-3.2-3b-instruct	0.5753	official
67	k2-think-v2	0.5889	official
68	magistral-medium	0.5947	official
69	claude-opus-4.5	0.5978	official
70	gemini-3.5-flash	0.6028	official
71	gpt-5.2	0.6057	official
72	mistral-medium-3	0.6091	official
73	nvidia-nemotron-nano-9b-v2	0.6101	official
74	claude-opus-4.6	0.6133	official
75	glm-5-turbo	0.6217	official
76	devstral	0.6259	official
77	inkling	0.6311	official
78	jt-35b-flash	0.6328	official
79	grok-4	0.642	official
80	gpt-5.5-instant	0.6443	official
81	nova-micro	0.6497	official
82	llama-nemotron-super-49b-v1-5	0.6587	official
83	grok-4-fast	0.6596	official
84	glm-4.6	0.6613	official
85	gemini-2.5-flash-lite-preview-09-2025	0.6644	official
86	kat-coder-pro-v1	0.6655	official
87	mistral-small-4	0.6679	official
88	olmo-3.1-32b-think	0.668	official
89	minimax-m2.1	0.6702	official
90	ernie-4.5-300b-a47b	0.671	official
91	glm-4.5	0.6716	official
92	mistral-large-2	0.6784	official
93	glm-5v-turbo	0.679	official
94	nova-premier	0.6847	official
95	llama-3.1-nemotron-instruct-70b	0.6882	official
96	o1	0.6925	official
97	lfm2-24b-a2b	0.6997	official
98	tri-21b-think-preview	0.7014	official
99	olmo-3.1-32b-instruct	0.7039	official
100	gemini-2.5-flash	0.7066	official
101	qwen3-235b-a22b-instruct-2507	0.7076	official
102	kimi-k2	0.7088	official
103	qwen3.5-4b	0.7153	official
104	grok-4.1-fast	0.7241	official
105	deepseek-v3.1-terminus	0.725	official
106	gpt-5.2-codex	0.7282	official
107	llama-3.3-nemotron-super-49b	0.7298	official
108	hy3	0.7299	official
109	muse-spark	0.7319	official
110	gpt-5.6-luna	0.7349	official
111	gemma-4-12b	0.7349	official
112	gpt-5-codex	0.7408	official
113	qwen3.5-27b	0.7415	official
114	gpt-5.1-codex	0.7437	official
115	granite-4.0-micro	0.7447	official
116	longcat-2.0	0.7469	official
117	llama-3.1-70b-instruct	0.7472	official
118	gemini-2.5-flash-lite	0.7542	official
119	qwen3-14b-instruct	0.7547	official
120	magistral-small	0.7548	official
121	phi-4-mini	0.7553	official
122	gpt-5	0.7584	official
123	command-a	0.7615	official
124	mistral-small-3.2	0.7627	official
125	k2-v2	0.7644	official
126	nova-2.0-pro	0.7654	official
127	deepseek-v3	0.7675	official
128	mistral-small-3.1	0.7684	official
129	devstral-small	0.7713	official
130	qwen3-235b-a22b-instruct	0.7719	official
131	nova-pro	0.7788	official
132	qwen3-4b-2507-instruct	0.78	official
133	gpt-oss-120b	0.7815	official
134	claude-3-haiku	0.7825	official
135	llama-4-scout	0.7831	official
136	grok-code-fast-1	0.7852	official
137	apertus-70b-instruct	0.7862	official
138	mistral-7b-instruct	0.7896	official
139	o3-mini	0.7898	official
140	cogito-v2-1	0.7906	official
141	hyperclova-x-seed-think-32b	0.7907	official
142	qwen3-coder-30b-a3b-instruct	0.7923	official
143	lfm2.1-2b	0.793	official
144	doubao-seed-code	0.794	official
145	nova-2.0-lite	0.7956	official
146	gpt-4.1	0.7962	official
147	qwen3.5-397b-a17b	0.7985	official
148	hermes-4-llama-3-1-70b	0.8012	official
149	o4-mini	0.8012	official
150	qwen3-30b-a3b-instruct	0.8029	official
151	kimi-k2.7-code	0.8029	official
152	deepseek-v3.1	0.8033	official
153	hermes-4-llama-3-1-405b	0.8038	official
154	gpt-4.1-nano	0.8041	official
155	qwen3-vl-32b	0.8042	official
156	phi-4	0.8051	official
157	exaone-4.5-33b	0.807	official
158	ministral-3-3b	0.8082	official
159	glm-4.5v	0.8085	official
160	gemma-4-26b-a4b-it	0.809	official
161	exaone-4.0-32b	0.8104	official
162	gpt-5.4	0.8109	official
163	mistral-medium-3.1	0.813	official
164	qwen3.5-9b	0.8135	official
165	deepseek-r1-distill-llama-70b	0.8151	official
166	gemma-4-31b-it	0.8158	official
167	llama-3.2-instruct-11b-vision	0.8161	official
168	gemini-3.1-flash-lite	0.8163	official
169	llama-3.1-nemotron-ultra-253b-v1	0.8167	official
170	qwen3-32b-instruct	0.8195	official
171	gpt-4.1-mini	0.8198	official
172	mistral-medium-3.5	0.82	official
173	step-3-vl-10b	0.8217	official
174	qwen3-next-80b-a3b	0.8221	official
175	hypernova-60b	0.8222	official
176	qwen3-vl-235b-a22b	0.8252	official
177	nemotron-3-nano-30b-a3b	0.8295	official
178	north-mini-code	0.8305	official
179	nemotron-3-nano-omni-30b-a3b	0.8314	official
180	seed-oss-36b-instruct	0.834	official
181	k-exaone	0.8353	official
182	granite-4.0-350m	0.8367	official
183	mistral-large-3	0.8375	official
184	ring-2.6-1t	0.8383	official
185	gemma-3-1b	0.8385	official
186	nova-2.0-omni	0.8395	official
187	qwen3.5-35b-a3b	0.8401	official
188	deepseek-r1	0.8405	official
189	glm-4.7-flash	0.8432	official
190	deepseek-v3.2	0.8435	official
191	ring-1t	0.8437	official
192	step-3.7-flash	0.8438	official
193	ernie-5.0-thinking-preview	0.8459	official
194	grok-3	0.8461	official
195	qwen2.5-72b-instruct	0.8469	official
196	gemini-3-pro-preview	0.8486	official
197	nemotron-cascade-2-30b-a3b	0.8503	official
198	devstral-2	0.8506	official
199	llama-3.3-70b-instruct	0.8511	official
200	devstral-small-2	0.8512	official
201	olmo-3-32b-think	0.8515	official
202	gpt-5.6-terra	0.8519	official
203	qwen3.5-122b-a10b	0.8545	official
204	tri-21b-think-v0.5	0.855	official
205	gpt-5.5	0.8553	official
206	apriel-v1-5-15b-thinker	0.8568	official
207	jamba-reasoning-3b	0.8596	official
208	deepseek-r1-0528-qwen3-8b	0.865	official
209	llama-3-70b-instruct	0.865	official
210	gemini-2.0-flash	0.8654	official
211	devstral-small-2505	0.8655	official
212	gpt-oss-20b	0.866	official
213	trinity-large	0.8665	official
214	agnes-2-5-pro-alpha	0.8665	official
215	minimax-m1-80k	0.868	official
216	gpt-5.3-codex	0.869	official
217	hy3-preview	0.8695	official
218	intellect-3	0.8698	official
219	nemotron-3-super-120b-a12b	0.8701	official
220	gpt-5.6-sol	0.8711	official
221	o3	0.8712	official
222	granite-4.1-8b	0.8726	official
223	qwen3-30b-a3b-2507	0.8727	official
224	llama-4-maverick	0.8732	official
225	gemini-2.5-pro	0.8738	official
226	lfm2.5-1-2b-instruct	0.8752	official
227	granite-4.0-h-nano-1b	0.8753	official
228	solar-pro-3	0.8781	official
229	deepseek-v4-pro	0.8789	official
230	granite-4.0-h-small	0.8791	official
231	llama-3-8b-instruct	0.8797	official
232	lfm2-8b-a1b	0.8798	official
233	solar-open-100b	0.8812	official
234	gemini-2.5-flash-preview-09-2025	0.883	official
235	step-3.5-flash	0.884	official
236	minimax-m2	0.8842	official
237	qwen3-max-preview	0.8849	official
238	falcon-h1r-7b	0.8865	official
239	qwen3-omni-30b-a3b	0.8867	official
240	qwen3-vl-235b-a22b-instruct	0.8873	official
241	mi-dm-k-2-5-pro-dec28	0.8891	official
242	deepseek-v3.2-speciale	0.8894	official
243	ministral-3-8b	0.8897	official
244	gpt-5.4-mini	0.8916	official
245	qwen3-vl-30b-a3b	0.892	official
246	qwen3-8b-instruct	0.8926	official
247	minimax-m2.5	0.893	official
248	qwen3-max	0.8939	official
249	nvidia-nemotron-3-nano-4b	0.8943	official
250	gemma-3-27b-it	0.8954	official
251	ring-flash-2.0	0.8957	official
252	deepseek-v4-flash	0.8968	official
253	ling-flash-2.0	0.8996	official
254	reka-flash-3	0.9011	official
255	gemini-3-flash-preview	0.9018	official
256	ministral-3-14b	0.9021	official
257	glm-4.7	0.9029	official
258	motif-2-12-7b	0.9041	official
259	qwen3-vl-8b-instruct	0.9042	official
260	grok-build-0.1	0.9047	official
261	sarvam-m	0.9051	official
262	nvidia-nemotron-nano-12b-v2-vl	0.9066	official
263	magistral-small-2509	0.907	official
264	molmo2-8b	0.907	official
265	qwen3-vl-8b	0.9072	official
266	apriel-v1-6-15b-thinker	0.9089	official
267	qwen3-coder-next	0.9091	official
268	diffusiongemma-26b-a4b	0.9115	official
269	jt-mini	0.9123	official
270	qwen3-vl-32b-instruct	0.9125	official
271	lfm2.2-6b	0.9142	official
272	exaone-4.0-1-2b	0.9149	official
273	solar-pro-2	0.9149	official
274	mercury-2	0.9151	official
275	llama-3.2-1b-instruct	0.9155	official
276	granite-4.0-h-350m	0.9171	official
277	olmo-3-7b-instruct	0.9182	official
278	ling-2.6-1t	0.9223	official
279	glm-4.5-air	0.9233	official
280	gemma-3n-e2b	0.9238	official
281	qwen3-0.6b-instruct	0.924	official
282	qwen3-vl-4b	0.9281	official
283	qwen3-vl-30b-a3b-instruct	0.9294	official
284	midm-250-pro-rsnsft	0.9307	official
285	granite-3.3-8b-instruct	0.9309	official
286	qwen3-next-80b-a3b-instruct	0.9316	official
287	qwen3-1.7b-instruct	0.9345	official
288	granite-4.0-nano-1b	0.9352	official
289	sarvam-105b	0.9353	official
290	qwen3.5-omni-flash	0.9361	official
291	jamba-1-7-large	0.9375	official
292	lfm2.5-vl-1-6b	0.9395	official
293	ling-1t	0.9399	official
294	nex-n2-pro	0.9454	official
295	apertus-8b-instruct	0.9467	official
296	olmo-3-7b-think	0.9475	official
297	granite-4.1-30b	0.9506	official
298	granite-4.1-3b	0.9546	official
299	gemma-3n-e4b-it	0.9556	official
300	longcat-flash-lite	0.9574	official
301	tiny-aya-global	0.9578	official
302	ling-mini-2.0	0.958	official
303	ling-2.6-flash	0.958	official
304	qwen3-vl-4b-instruct	0.9611	official
305	jamba-1-7-mini	0.9667	official
306	minicpm-v4-6-1-3b	0.9684	official
307	lfm2.5-1.2b	0.9689	official
308	sarvam-30b	0.97	official
309	gemma-3-12b	0.9712	official
310	qwen3-omni-30b-a3b-instruct	0.9768	official
311	gemma-3-4b	0.9883	official