Evaluation profile

AA-Omniscience

1sub-evals

2.51%Safety weight

0%Freedom weight

1components

Weights below are portfolio-specific global index weights.

Model score (lower is better)Predicted score

About this eval

Hallucination propensity on difficult closed-book factual questions under explicit encouragement to abstain when uncertain.

Included in the behavior ranking.

Sub-evals

Measure	Component	Direction by ranking	Safety weight	Freedom weight
hallucination_rateaa-omniscience/aa-omniscience.csv:hallucination_rateMeasures whether the model avoids confidently giving incorrect answers across six thousand factual questions.	Safety: truthfulness_honesty:1.000aa-omniscience	Safety: lower	2.51%	—

hallucination_rate

Measures whether the model avoids confidently giving incorrect answers across six thousand factual questions.

Rank	Model	Value	Provenance
1	minicpm5-1b	0.009009	official
2	g9v3-3b	0.1166	official
3	g9v3-39a5b	0.1304	official
4	command-a-plus	0.1416	official
5	gemini-4-argon	0.1507	official
6	lfm2.5-2.6b	0.16	official
7	grok-4.3	0.1694	official
8	qwen3.8-27b	0.1809	official
9	minimax-m3	0.1843	official
10	grok-4.20	0.1992	official
11	quasar-438b	0.2144	official
12	minicpm5-2b	0.2191	official
13	ling-3.0-flash-vl	0.2205	official
14	k-exaone-2.0	0.226	official
15	grok-4.6	0.24	official
16	solar-pro-4	0.244	official
17	mimo-v2.5-pro	0.247	official
18	solar-open2-250b	0.2538	official
19	granite-4.2-30b	0.2558	official
20	qwen3.7-max	0.2563	official
21	claude-haiku-4.5	0.2571	official
22	grok-3-mini	0.2582	official
23	k2-horizon-375b-a23b	0.2585	official
24	glm-5.2	0.263	official
25	granite-4.2-3b	0.2634	official
26	glm-5.3-flash	0.2763	official
27	qwen3.7-plus	0.2766	official
28	qwen3.8-max	0.2884	official
29	claude-sonnet-4	0.2912	official
30	grok-4.7	0.2934	official
31	glm-5.3	0.2955	official
32	nemotron-3-ultra-550b-a55b	0.2969	official
33	glm-5.1	0.2995	official
34	mimo-v2-pro	0.2998	official
35	gemma-3-270m	0.3042	official
36	ling-3.0-tiny	0.3046	official
37	k2-horizon-mova-36b-a4b	0.3053	official
38	gemma-4-e4b	0.3094	official
39	muse-spark-1.3	0.3146	official
40	mimo-v2.5	0.3193	official
41	granite-4.2-8b	0.3197	official
42	gemma-4-e2b	0.3236	official
43	agnes-2.5-pro-beta	0.328	official
44	a-x-k2	0.3299	official
45	muse-spark-1.2	0.3329	official
46	k2-horizon-7b	0.3363	official
47	gemini-3.5-flash-lite	0.3436	official
48	qwen3.6-plus	0.3464	official
49	glm-5	0.3532	official
50	minimax-m2.7	0.3556	official
51	solar-mini4	0.3583	official
52	qwen3.5-omni-plus	0.37	official
53	nemotron-3.5-lightning	0.3757	official
54	ling-3.1-flash	0.3788	official
55	gpt-4o	0.3792	official
56	motif-3	0.3877	official
57	claude-opus-4.8	0.3925	official
58	claude-sonnet-5	0.3937	official
59	ling-3.0-flash-fin	0.3951	official
60	claude-3.7-sonnet	0.3997	official
61	mimo-v2-omni	0.4005	official
62	claude-haiku-5.5	0.4042	official
63	kimi-k2.6	0.405	official
64	mimo-v2.6-pro	0.4062	official
65	claude-3.5-haiku	0.4115	official
66	mistral-large-4	0.4195	official
67	claude-opus-4.7	0.4229	official
68	step-5-preview	0.4302	official
69	llama-3.1-8b-instruct	0.4304	official
70	jt-4.1-flash-236b-a21b	0.4367	official
71	ling-3.0-flash	0.4405	official
72	k2-horizon-3.7b	0.4414	official
73	gpt-5-mini	0.4432	official
74	gpt-6-astra	0.4477	official
75	qwen3-coder-480b-a35b-instruct	0.4493	official
76	qwen3.8-flash-next	0.4532	official
77	qwen3.6-max-preview	0.4619	official
78	lfm2.5-8b-a1b	0.4691	official
79	claude-sonnet-5.5	0.4701	official
80	agnes-3.0-flash	0.4838	official
81	mimo-v2-flash	0.4841	official
82	claude-sonnet-4.6	0.4845	official
83	claude-sonnet-4.5	0.4908	official
84	qwen3.6-27b	0.4927	official
85	gpt-6.1-sol	0.4936	official
86	kimi-k2.5	0.4996	official
87	muse-spark-1.1	0.4998	official
88	qwen3.6-35b-a3b	0.505	official
89	gpt-6-sol	0.5074	official
90	gemini-3.1-pro-preview	0.5087	official
91	gpt-5.4-nano	0.5113	official
92	glm-4.6v	0.5146	official
93	gpt-5.1-codex-mini	0.5158	official
94	gpt-5.1	0.5189	official
95	gemini-3.8-flash	0.519	official
96	llama-3.1-405b-instruct	0.5244	official
97	gpt-5-nano	0.5262	official
98	kimi-k3	0.532	official
99	deepseek-v4.1-flash	0.5399	official
100	grok-4.5	0.5415	official
101	mimo-v2.6-flash	0.5436	official
102	gemini-3.6-flash	0.5563	official
103	kat-coder-pro-v2	0.5598	official
104	nova-lite	0.5761	official
105	nanbeige4.1-3b	0.581	official
106	claude-opus-5.5	0.5861	official
107	claude-opus-5	0.5955	official
108	magistral-medium	0.5966	official
109	nvidia-nemotron-nano-9b-v2	0.6049	official
110	mistral-medium-3	0.609	official
111	claude-opus-4.5	0.61	official
112	qwen3.5-0.8b	0.6135	official
113	gpt-5.2	0.616	official
114	gemini-3.5-flash	0.6176	official
115	devstral	0.6219	official
116	glm-5-turbo	0.6264	official
117	claude-opus-4.6	0.6284	official
118	lfm2.2-6b	0.6289	official
119	inkling-small	0.6299	official
120	claude-fable-5	0.6364	official
121	jt-35b-flash	0.6384	official
122	grok-4	0.6449	official
123	gemini-3.7-flash	0.6453	official
124	nova-micro	0.6508	official
125	gpt-5.5-instant	0.653	official
126	claude-fable-5.1	0.6563	official
127	olmo-3.1-32b-think	0.6629	official
128	llama-3.2-1b-instruct	0.6634	official
129	mistral-small-4	0.665	official
130	ernie-4.5-300b-a47b	0.6693	official
131	llama-nemotron-super-49b-v1.5	0.6726	official
132	kat-coder-pro-v1	0.6736	official
133	glm-4.6	0.6757	official
134	inkling	0.6766	official
135	mistral-large-2	0.6768	official
136	grok-4-fast	0.6827	official
137	minimax-m2.1	0.6854	official
138	glm-5v-turbo	0.688	official
139	lfm2-24b-a2b	0.6903	official
140	jamba-reasoning-3b	0.6904	official
141	gpt-4	0.6941	official
142	o1	0.6963	official
143	glm-4.5	0.7015	official
144	k2-think-v2	0.7117	official
145	llama-3.1-nemotron-instruct-70b	0.7126	official
146	gemini-2.5-flash-lite	0.7171	official
147	nova-premier	0.7192	official
148	kimi-k2	0.7267	official
149	olmo-3.1-32b-instruct	0.7324	official
150	gpt-5.2-codex	0.7336	official
151	gemma-4-12b	0.7341	official
152	grok-4.1-fast	0.7341	official
153	llama-3.3-nemotron-super-49b	0.7369	official
154	hy3	0.7413	official
155	qwen3.5-2b	0.7424	official
156	deepseek-v3.1-terminus	0.7483	official
157	gpt-5.6-luna	0.75	official
158	qwen3.5-27b	0.7513	official
159	longcat-2.0	0.7539	official
160	gpt-5-codex	0.7542	official
161	qwen3-14b-instruct	0.7582	official
162	k2-v2	0.7604	official
163	granite-4.0-350m	0.7613	official
164	gpt-6-luna	0.7673	official
165	mistral-small-3.2	0.7681	official
166	gpt-5.1-codex	0.772	official
167	command-a	0.7732	official
168	qwen3-235b-a22b-instruct	0.774	official
169	mistral-small-3.1	0.7756	official
170	nova-pro	0.7771	official
171	phi-4-mini	0.7799	official
172	llama-3.1-70b-instruct	0.7821	official
173	apodex-1.1	0.7843	official
174	gpt-5	0.7902	official
175	grok-code-fast-1	0.7929	official
176	llama-4-scout	0.7935	official
177	exaone-4.5-33b	0.7992	official
178	nova-2.0-pro	0.7994	official
179	doubao-seed-code	0.7999	official
180	cogito-v2.1	0.8016	official
181	ministral-3-3b	0.8019	official
182	hermes-4-llama-3.1-405b	0.8023	official
183	qwen3-coder-30b-a3b-instruct	0.803	official
184	mercury-2.5	0.8033	official
185	o4-mini	0.8045	official
186	claude-3-haiku	0.8045	official
187	tri-21b-think-preview	0.8053	official
188	qwen3-30b-a3b-instruct	0.8073	official
189	o3-mini	0.8108	official
190	deepseek-r1-distill-llama-70b	0.8109	official
191	hyperclova-x-seed-think-32b	0.8119	official
192	phi-4	0.8119	official
193	llama-3.1-nemotron-ultra-253b-v1	0.8123	official
194	mistral-7b-instruct	0.8126	official
195	lfm2.1-2b	0.8138	official
196	mistral-medium-3.5	0.8159	official
197	devstral-small	0.8162	official
198	granite-4.0-h-nano-1b	0.8172	official
199	qwen3-4b-instruct	0.8174	official
200	llama-3.2-instruct-11b-vision	0.8188	official
201	muse-glimmer	0.8194	official
202	gemma-4-31b-it	0.8195	official
203	exaone-4.0-32b	0.821	official
204	qwen3-32b-instruct	0.8212	official
205	gemma-3-1b	0.8229	official
206	hypernova-60b	0.8233	official
207	kimi-k2.7-code	0.8238	official
208	deepseek-v3.1	0.8251	official
209	gpt-4.1-nano	0.8256	official
210	gpt-5.4	0.8258	official
211	gemini-2.5-flash	0.826	official
212	gemini-3.1-flash-lite	0.8266	official
213	hermes-4-llama-3.1-70b	0.8268	official
214	qwen3.5-397b-a17b	0.8269	official
215	apertus-70b-instruct	0.8282	official
216	north-mini-code	0.8321	official
217	nemotron-3-nano-30b-a3b	0.8327	official
218	step-3-vl-10b	0.8332	official
219	magistral-small	0.8345	official
220	deepseek-r1	0.8345	official
221	qwen3.5-9b	0.8361	official
222	qwen3-vl-32b	0.8368	official
223	muse-spark	0.8419	official
224	mistral-medium-3.1	0.8458	official
225	nova-2.0-omni	0.847	official
226	glm-4.5v	0.8478	official
227	step-3.7-flash	0.8498	official
228	qwen3-vl-235b-a22b	0.8515	official
229	deepseek-v3.2	0.8518	official
230	devstral-2	0.8527	official
231	apriel-v1.5-15b-thinker	0.8528	official
232	ring-2.6-1t	0.8533	official
233	nemotron-cascade-2-30b-a3b	0.8536	official
234	tri-21b-think-v0.5	0.8542	official
235	qwen3.5-35b-a3b	0.8543	official
236	qwen2.5-72b-instruct	0.8546	official
237	nemotron-3-nano-omni-30b-a3b	0.8565	official
238	olmo-3-32b-think	0.8567	official
239	k2-horizon-0.9b	0.8569	official
240	deepseek-v3	0.8586	official
241	nova-2.0-lite	0.8587	official
242	trinity-large	0.8589	official
243	mistral-large-3	0.8596	official
244	gemini-2.0-flash	0.8624	official
245	deepseek-r1-qwen3-8b	0.8635	official
246	seed-oss-36b-instruct	0.8639	official
247	gemma-4-26b-a4b-it	0.8644	official
248	qwen3-30b-a3b	0.8654	official
249	qwen3.5-4b	0.8655	official
250	gpt-oss-20b	0.8671	official
251	nemotron-3-super-120b-a12b	0.8696	official
252	ernie-5.0-thinking-preview	0.8701	official
253	qwen3-next-80b-a3b	0.8701	official
254	qwen3.5-122b-a10b	0.8709	official
255	granite-4.1-8b	0.8713	official
256	devstral-small-2	0.8729	official
257	grok-3	0.8732	official
258	hy3-preview	0.874	official
259	llama-3-70b-instruct	0.8754	official
260	intellect-3	0.8759	official
261	granite-4.0-h-small	0.8785	official
262	deepseek-v4-pro	0.8785	official
263	gpt-5.6-terra	0.8788	official
264	gpt-5.5	0.8804	official
265	granite-4.0-h-350m	0.8806	official
266	agnes-2.5-pro-alpha	0.8807	official
267	o3	0.8809	official
268	minimax-m2.5	0.8812	official
269	solar-pro-3	0.8822	official
270	step-3.5-flash	0.8849	official
271	nvidia-nemotron-3-nano-4b	0.8857	official
272	falcon-h1r-7b	0.8877	official
273	k-exaone	0.8884	official
274	llama-4-maverick	0.8888	official
275	deepseek-v4-flash	0.8897	official
276	qwen3-omni-30b-a3b	0.8899	official
277	deepseek-v3.2-speciale	0.8915	official
278	gpt-5.3-codex	0.8918	official
279	gpt-5.6-sol	0.8942	official
280	ling-flash-2.0	0.8948	official
281	solar-open-100b	0.8965	official
282	lfm2.5-1.2b-instruct	0.8973	official
283	ring-1t	0.8975	official
284	qwen3-vl-30b-a3b	0.8977	official
285	llama-3-8b-instruct	0.8979	official
286	gpt-5.4-mini	0.8985	official
287	qwen3-max	0.8992	official
288	qwen3-max-preview	0.8993	official
289	gemini-3-pro-preview	0.9	official
290	ring-flash-2.0	0.9007	official
291	llama-3.3-70b-instruct	0.9021	official
292	motif-2-12.7b	0.9059	official
293	sarvam-m	0.9064	official
294	qwen3-8b-instruct	0.9074	official
295	gpt-oss-120b	0.9082	official
296	diffusiongemma-26b-a4b	0.9089	official
297	gemini-2.5-pro	0.9089	official
298	minimax-m2	0.909	official
299	qwen3-0.6b-instruct	0.9094	official
300	qwen3-vl-8b-instruct	0.9101	official
301	molmo2-8b	0.911	official
302	olmo-3-7b-instruct	0.9122	official
303	mercury-2	0.9122	official
304	reka-flash-3	0.9127	official
305	jt-mini	0.913	official
306	qwen3-vl-8b	0.9141	official
307	deepseek-v4-flash-vision	0.9148	official
308	minimax-m1-80k	0.9153	official
309	qwen3-vl-235b-a22b-instruct	0.9155	official
310	deepseek-v4-flash-0731	0.917	official
311	exaone-4.0-1.2b	0.9174	official
312	qwen3-vl-4b	0.9182	official
313	qwen3-vl-32b-instruct	0.9182	official
314	apriel-v1.6-15b-thinker	0.9193	official
315	deepseek-v4-pro-0813	0.9194	official
316	gemma-3-27b-it	0.9209	official
317	lfm2-8b-a1b	0.923	official
318	nvidia-nemotron-nano-12b-v2-vl	0.9235	official
319	gemini-3-flash-preview	0.9241	official
320	ministral-3-14b	0.9252	official
321	gpt-4.1-mini	0.9272	official
322	qwen3-next-80b-a3b-instruct	0.9275	official
323	celeris-1	0.9275	official
324	ling-2.6-1t	0.9283	official
325	glm-4.5-air	0.9288	official
326	glm-4.7	0.9291	official
327	solar-pro-2	0.9303	official
328	grok-build-0.1	0.9312	official
329	olmo-3-7b-think	0.9321	official
330	gpt-4.1	0.9331	official
331	gemma-3n-e2b	0.9338	official
332	qwen3.5-omni-flash	0.934	official
333	sarvam-105b	0.9343	official
334	granite-4.0-nano-1b	0.9352	official
335	qwen3-vl-30b-a3b-instruct	0.9366	official
336	qwen3-coder-next	0.9368	official
337	glm-4.7-flash	0.9394	official
338	ministral-3-8b	0.9416	official
339	mi-dm-k-2.5-pro	0.9427	official
340	ling-1t	0.9473	official
341	qwen3-1.7b-instruct	0.9481	official
342	nex-n2-pro	0.9491	official
343	granite-4.1-30b	0.9498	official
344	jamba-1.7-large	0.9507	official
345	granite-4.1-3b	0.9508	official
346	jamba-1.7-mini	0.951	official
347	lfm2.5-vl-1.6b	0.9574	official
348	granite-4.0-micro	0.9577	official
349	ling-mini-2.0	0.9587	official
350	minicpm-v4.6-1.3b	0.9607	official
351	apertus-8b-instruct	0.9614	official
352	longcat-flash-lite	0.9616	official
353	tiny-aya-global	0.962	official
354	granite-3.3-8b-instruct	0.9622	official
355	sarvam-30b	0.9628	official
356	ling-2.6-flash	0.9672	official
357	gemma-3n-e4b-it	0.9707	official
358	gemma-3-12b	0.974	official
359	qwen3-vl-4b-instruct	0.9742	official
360	qwen3-omni-30b-a3b-instruct	0.9759	official
361	gemma-3-4b	0.9821	official