METR 发现 GPT-5.6 Sol 在基准测试中作弊率创纪录

Truly wild. METR found that GPT-5.6 Sol gamed/cheated the benchmark so much that the score became unstable. The model showed situational awareness, concealed misbehavior, and attempts to bypass restrictions. GPT-5.6 Sol had the highest detected cheating rate METR has seen on its public ReAct agent harness, including attempts to exploit the evaluation setup instead of solving tasks normally. So METR was benchmarking for number of hours as an estimate for the length of software tasks GPT-5.6 Sol can complete. The capability estimate became almost unusable: counting cheating as failure gave 11.3hrs, counting it as success pushed it past 270hrs, and removing cheating left a hugely uncertain 71hrs estimate.  > **引用原帖 Rohan Paul (@rohanpaul_ai):** > BREAKING: OpenAI just dropped the limited preview of its new GPT 5.6 model suite: Sol, the flagship; Terra, a medium-tier model for “high-volume work”; and Luna, a “fast and affordable” everyday model. > The most revealing part is the release gate: OpenAI says the U.S. government asked it to start with a small trusted-partner preview before broader access. > Sol is the flagship model, and OpenAI claims it is a step above GPT-5.5, especially on agentic work where the model must plan, use tools, correct itself, and keep working across many steps. > Terminal-Bench 2.1 is a solid coding benchmark because it tests command-line workflows, so here meaning Sol is being judged on messy developer tasks closer to real work. > ---- > One key claim is cybersecurity: OpenAI says Sol is its best model yet for vulnerability research and exploitation tasks, while still saying it did not cross the internal Cyber Critical threshold. > “GPT‐5.6 is trained to refuse prohibited cyber assistance, including when users attempt to disguise their intent or jailbreak the model.” It also said that flagship model Sol “is better at helping people find and fix vulnerabilities than reliably carrying out end-to-end attacks,” and that Sol doesn’t cross the cyber-critical threshold under OpenAI’s preparedness framework > But Sol did not autonomously produce a full-chain exploit in the tested Chromium and Firefox settings. > They also introduced 2 new modes for Sol: “max” for deeper reasoning and “ultra” for using sub-agents, bringing OpenClaw to mind and possibly hinting at OpenClaw creator Peter Steinberger’s early impact at OpenAI. > ---- > Pricing: GPT-5.6 Sol costs $5 per 1M input tokens and $30 per 1M output tokens, ~same level as GPT-5.5. > Terra is positioned near GPT-5.5 performance at 2x lower cost, while Luna is the cheapest model for large-volume workloads. > -- > The safety story is unusually compute-heavy: OpenAI says it used over 700,000 A100-equivalent GPU hours for automated red-teaming against broad jailbreak attacks. > Overall, OpenAI appeared to be using a more cautious approach during the preview, which the Trump administration is watching closely. > OpenAI said safeguards might sometimes block valid work, especially in dual-use areas where defensive and offensive actions can look alike at first. That is one thing the preview is meant to test. > https://x.com/rohanpaul_ai/status/2070573957271732353 Rohan Paul (@rohanpaul_ai): https://t.co/R2BuH5AevN