AITOPAI热报
PapersSelected73°

ReFigBench shows harness choice changes coding agent results

Impressive paper showing how much the harness changes a coding agent's results. Harnesses do play a...

elvis@omarsar0
Post
View the post
Why we picked it

The same model can score differently depending on the harness around it, so benchmark numbers reflect the full stack rather than the model alone. Teams evaluating coding agents should rerun comparisons under their own harness before drawing conclusions. ReFigBench adds a new evaluation axis, and details are on arXiv (2609.18844) alongside a chat interface from dair.ai.

A new benchmark called ReFigBench asks coding agents to rebuild real arXiv overview figures as editable PowerPoint slides that preserve text, layout and connections. GPT-5.5 was run inside both Claude Code and Codex on the same 1,000 tasks, and a specialized PowerPoint workflow improved results in one harness while degrading them in the other, even with identical prompts. The benchmark covers ten configurations across the GPT, Claude, MiMo and MiniMax families, scored by artifact checks, two families of LLM judges and blinded human comparisons. Perception remains the main bottleneck, and the specialized workflow removed native connectors in every configuration even though human judges still preferred its renderings in most matchups.

Original · elvis

Impressive paper showing how much the harness changes a coding agent's results. Harnesses do play a...

Impressive paper showing how much the harness changes a coding agent's results. Harnesses do play a huge role in what you are getting out of the models. GPT-5.5 was run inside both Claude Code and Codex on the same 1,000 tasks. With a specialized PowerPoint workflow, it improved inside one harness and got worse inside the other. The harness also changed scores when the prompt was identical. ReFigBench asks coding agents to rebuild real arXiv overview figures as editable PowerPoint slides that keep the text, layout and connections. It covers ten configurations across the GPT, Claude, MiMo and MiniMax families, scored by artifact checks, two families of LLM judges and blinded human comparisons. Perception is still the main bottleneck. The specialized workflow removed native connectors in every configuration, yet human judges still preferred its renderings in most matchups. Paper: arxiv.org/abs/2609.18844 Chat with Paper: academy.dair.ai/papers/refigbe… 💬 5 🔄 3 ❤️ 24 👀 1769 📊 8 ⚡

#ReFigBench#GPT-5.5#Claude Code#Codex#coding assistants

Keep reading

Recent stories with the same tag or section
PapersBuildersSelected03:53
Yann LeCun Explains Why JEPA Research Papers Differ From Meta Models

The thread lays out a taxonomy of SSL methods — predictor presence, anti-collapse criterion, augmentation scheme — that helps readers place any new 'JEPA' paper in the right context. Researchers deciding which papers to cite or reproduce will find it a concise map of what academic work can and cannot demonstrate. The core distinction between method discovery and resource-heavy scaling applies far beyond the JEPA family.

#JEPA#DINO#V-JEPA#Meta-FAIR
官方账号Yann LeCun@ylecunSource
PapersBuildersSelected02:51
Oxford paper finds frontier LLM agents fail cross-vendor coordination

The practical implication is that agent interoperability breaks down not from capability gaps but from unstated conventions, so inter-agent protocols may need explicit shared norms rather than smarter models. Teams building multi-agent systems that span different vendors should add instructions telling agents to avoid unstated conventions and prefer actions any partner can read correctly. The finding applies to anyone chaining GPT-5.2 and Gemini 3 Pro agents together today.

#GPT-5.2#Gemini 3 Pro#agents#AI safety
rohanpaul_ai@rohanpaul_aiSource