PACE: A Proxy for Agentic Capability Evaluation

PACE: A Proxy for Agentic Capability Evaluation > **引用原帖 Yueqi Song (@yueqi_song):** > 🚀Excited to release PACE: A Proxy for Agentic Capability Evaluation! > Evaluating LLM agents on benchmarks like SWE-Bench and GAIA is expensive, slow, and infrastructure-heavy, often costing $$$ and taking hours or days per model. > ❓But do we always need to run full agentic evaluations? > In PACE, we show that agentic benchmark performance can be accurately predicted from a small, carefully selected set of cheap non-agentic benchmark instances. > PACE automatically selects proxy instances from existing benchmarks covering skills like instruction following, planning, tool use, reasoning, coding, retrieval, and multimodal understanding. > Across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks, PACE-BENCH achieves: > ✅ 3.80% MAE for absolute score prediction > ✅ 0.81 Spearman correlation for model ranking > ✅ ~84% pairwise preference accuracy > ✅ ~100× lower cost than target benchmark sampling > Beyond prediction, PACE also reveals what capabilities different agentic benchmarks actually require, e.g., planning, verification, long-context aggregation, and instruction following. > We hope PACE makes agentic evaluation cheaper, faster, and more accessible for model development, model selection, and routing :) > 📃 Paper: https://t.co/I6iYUCADov > 💻 Code: https://t.co/tLd7HrSgEA > I'm incredibly grateful to have worked with @lintangsutawika, @Jiarui_Liu_, @lltjuatja, @JiayiiGeng, @lrzneedresearch, @daniel_js_lee, @Aditya_Soni_8, @Vincent92965015, @xiangyue96, and @gneubig . > https://x.com/yueqi_song/status/2074180763302670648