Arena 增加事实评分

meng shao (@shao__meng) 转发了 Sumanth (@Sumanth_077) 的帖子： The model humans prefer the most is not always the most accurate one! Arena just added factuality scoring to their leaderboard alongside human preference. The idea: human preference measures whether a response felt good to read. Factuality measures the correctness of claims in a model's response. These are two different things. Most benchmarks test models on fixed, predetermined questions in a lab setting. What Arena is doing differently: measuring factuality in real, open-ended user conversations at scale. Over 2 million claims extracted from actual conversations, verified against the web, across 130k Text Arena battles and 40k Search Arena battles. The way it works: they randomly sample battles, extract web-verifiable claims from each model response, verify them against the web, and calculate a factuality score per model. The final ranking is a weighted combination of human preference and factuality score. When factuality weight is enabled, the shifts are significant. In Text Arena, GPT-5.5 jumps 13 spots. Muse Spark drops 13. Claude Fable 5 moves down slightly. In Search Arena, GPT-5.5-search takes the top spot while Gemini grounding drops from 7 to 13. I've shared the full methodology blog in the replies!  > **引用原帖 Arena.ai (@arena):** > Introducing factuality in the Arena: a new ranking of models according to a weighted combination of human preference and factuality. > Model rankings are now viewable according to a weighted combination of human preference and factuality. Factuality is live in our Text and Search Arenas as a non-default toggle. > We audit model responses by randomly sampling battles and extracting web-verifiable claims. We then verify these claims and compare the average correctness between model responses. > To power these rankings, we’ve labeled over 2 million claims made by LLMs in real-world conversations, 1.3+ million from Text Arena, and 700k+ from Search Arena. > Notable highlights with factuality enabled in the Text Arena: > - Claude Fable 5 moves down slightly to spot #2 > - GPT-5.5 saw the largest increase, moving up 13 spots into the #7 spot > - Muse Spark dropped the most from #7 to #20 (-13pt) > By labs, Meta saw the largest drop from #2 to #5, while Anthropic overall held the #1 spot. Looking at only open model providers, Xiaomi saw the largest improvement, jumping from #9 to #6. > Learn more about the findings and methodology in this thread. > https://x.com/arena/status/2077432293023678685