Evaluation profile
Vectara HHEM Factual Consistency
1sub-evals
1.02%total index weight
1components
Within-component eval weight: Truthfulness 6.8%.
Model score (higher is better)Predicted score
About this eval
Grounded summarization factual consistency.
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction | Total index weight | Within-component weight |
|---|---|---|---|---|
| factual_consistency_ratevectara-hhem/vectara-hhem.csv:factual_consistency_rateMeasures whether grounded summaries remain supported by their source context. | truthfulness_honesty:1.000vectara-hhem | Higher is better | 1.02% | Truthfulness 6.8% |
factual_consistency_rate
Measures whether grounded summaries remain supported by their source context.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | finix-s1-32b | 98.2 | official | |
| 2 | gpt-5.4-nano | 96.9 | official | |
| 3 | gemini-2.5-flash-lite | 96.7 | official | |
| 4 | llama-3.3-70b-instruct | 95.9 | official | |
| 5 | gemma-3-12b | 95.6 | official | |
| 6 | mistral-large-2 | 95.5 | official | |
| 7 | qwen3-8b | 95.2 | official | |
| 8 | mistral-small-3 | 94.9 | official | |
| 8 | nova-2.0-lite | 94.9 | official | |
| 8 | nova-pro | 94.9 | official | |
| 11 | gemma-4-26b-a4b-it | 94.8 | official | |
| 11 | granite-4.0-h-small | 94.8 | official | |
| 13 | deepseek-v3.2-exp | 94.7 | official | |
| 13 | jamba-mini-2 | 94.7 | official | |
| 15 | qwen3-14b | 94.6 | official | |
| 16 | deepseek-v3.1 | 94.5 | official | |
| 16 | gpt-5.4-mini | 94.5 | official | |
| 16 | nova-micro | 94.5 | official | |
| 19 | gpt-4.1 | 94.4 | official | |
| 20 | qwen3-4b | 94.3 | official | |
| 21 | grok-3 | 94.2 | official | |
| 22 | qwen3-32b | 94.1 | official | |
| 23 | deepseek-v3 | 93.9 | official | |
| 23 | nova-lite | 93.9 | official | |
| 25 | deepseek-v3.2 | 93.7 | official | |
| 26 | command-r-plus-08-2024 | 93.1 | official | |
| 26 | trinity-large | 93.1 | official | |
| 28 | gemini-2.5-pro | 93 | official | |
| 28 | gpt-5.4 | 93 | official | |
| 30 | ministral-3b | 92.7 | official | |
| 31 | gemma-3-27b-it | 92.6 | official | |
| 31 | gemma-4-31b-it | 92.6 | official | |
| 31 | ministral-8b-24.10 | 92.6 | official | |
| 34 | llama-4-scout | 92.3 | official | |
| 35 | gemini-2.5-flash | 92.2 | official | |
| 36 | gemini-3.1-flash-lite | 91.8 | official | |
| 36 | llama-4-maverick | 91.8 | official | |
| 38 | gpt-5.4-pro | 91.7 | official | |
| 39 | deepseek-v4-pro | 91.4 | official | |
| 40 | minimax-m2.5 | 90.9 | official | |
| 41 | command-a | 90.7 | official | |
| 41 | glm-4.5-air | 90.7 | official | |
| 41 | glm-4.7-flash | 90.7 | official | |
| 41 | gpt-5.5 | 90.7 | official | |
| 41 | qwen3-235b-a22b | 90.7 | official | |
| 41 | qwen3-next-80b-a3b | 90.7 | official | |
| 47 | glm-4.6 | 90.5 | official | |
| 48 | gpt-4o | 90.4 | official | |
| 48 | gpt-5.2 | 90.4 | official | |
| 48 | nemotron-3-nano-30b-a3b | 90.4 | official | |
| 51 | jamba-1-7-large | 90.3 | official | |
| 52 | claude-haiku-4.5 | 90.2 | official | |
| 53 | glm-5 | 89.9 | official | |
| 54 | claude-sonnet-4 | 89.7 | official | |
| 55 | gemini-3.1-pro-preview | 89.6 | official | |
| 56 | gpt-5-nano | 89.5 | official | |
| 56 | qwen3.5-35b-a3b | 89.5 | official | |
| 56 | qwen3.5-flash-02-23 | 89.5 | official | |
| 59 | claude-sonnet-4.6 | 89.4 | official | |
| 59 | granite-3.3-8b-instruct | 89.4 | official | |
| 61 | qwen3.5-plus | 89.3 | official | |
| 62 | kimi-k2.6 | 89.2 | official | |
| 63 | c4ai-aya-expanse-32b | 89.1 | official | |
| 63 | claude-opus-4.5 | 89.1 | official | |
| 65 | qwen3.5-122b-a10b | 88.8 | official | |
| 66 | deepseek-r1 | 88.7 | official | |
| 67 | gpt-5.1 | 88.5 | official | |
| 68 | glm-4.7 | 88.3 | official | |
| 69 | claude-opus-4.1 | 88.2 | official | |
| 69 | minimax-m2.1 | 88.2 | official | |
| 71 | claude-opus-4 | 88 | official | |
| 71 | claude-opus-4.7 | 88 | official | |
| 71 | claude-sonnet-4.5 | 88 | official | |
| 74 | qwen3.5-27b | 87.9 | official | |
| 75 | claude-opus-4.6 | 87.8 | official | |
| 76 | mercury-2 | 87.7 | official | |
| 77 | gpt-5-mini | 87.1 | official | |
| 77 | minimax-m2.7 | 87.1 | official | |
| 79 | gemini-3-flash-preview | 86.5 | official | |
| 80 | gemini-3-pro-preview | 86.4 | official | |
| 81 | gpt-oss-120b | 85.8 | official | |
| 81 | kimi-k2.5 | 85.8 | official | |
| 83 | mistral-large-3 | 85.5 | official | |
| 84 | jamba-1-7-mini | 85.3 | official | |
| 85 | gpt-5 | 85.1 | official | |
| 86 | kimi-k2 | 82.1 | official | |
| 87 | grok-4.1-fast | 81.5 | official | |
| 88 | o4-mini | 81.4 | official | |
| 89 | ministral-3-14b | 80.6 | official | |
| 90 | grok-4-fast | 80.05 | official | |
| 91 | ministral-3-8b | 78.3 | official | |
| 92 | mistral-medium-3.1 | 77.3 | official | |
| 93 | o3-pro | 76.7 | official | |
| 94 | phi-4-mini | 76.5 | official |