AITOPAI热报
PapersSelected

SF Research Launches EvoHarnessBench for Evolving Agent Harnesses

官方账号Salesforce AI Research@SFResearch
Post
View the post
Why we picked it

Most agent benchmarks score a fixed toolset, so they miss the stability-versus-plasticity problem that surfaces in production, where harnesses keep growing and stored artifacts go stale. Teams building or selecting agent frameworks can run the benchmark via the live demo or plug in their own agent to see where their setup fails as capabilities expand. It suits engineering leads evaluating agent reliability over time rather than one-shot benchmark numbers.

SF Research announced EvoHarnessBench, a benchmark that tests agents as their harness of tools, skills and specialist agents expands over time. The benchmark spans 17 streams and 802 tasks, evaluating agents stage by stage as new capabilities arrive. Its key finding is a recurring trade-off: methods that preserve earlier competence often fail to exploit newly introduced capabilities, and vice versa. The benchmark therefore measures retention and adaptation separately rather than relying on aggregate scores. Results point toward bi-level adaptation, combining inner updates from current experience with an outer check on whether those updates still hold when the harness changes again.

Original · Salesforce AI Research

EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness? https://t.co/LQUhfKpAQB

Agents work through a harness of tools, skills, and specialist agents that keeps expanding as new capabilities arrive. EvoHarnessBench tests what happens as that harness grows, stage by stage, across 17 streams and 802 tasks.

A recurring pattern: the methods that best preserve earlier competence often struggle to use newly introduced capabilities, and vice versa. Aggregate scores can hide this, so we measure retention and adaptation separately.

Our results point to bi-level adaptation: inner updates from current experience, plus an outer check on whether those updates still hold once the harness moves again, including spotting when stored artifacts have gone stale.

Try the live demo or bring your own agent.

#AgenticAI #EnterpriseAI

#EvoHarnessBench#SF Research#agents#benchmarks

Keep reading

Recent stories with the same tag or section
PapersBuildersSelected13:32
Meta and Michigan State paper finds LLM-built reasoning graphs unreliable

The non-obvious point is that most reasoning-graph pipelines assume the LLM's decomposition is correct without ever verifying it, so confidence scores inherit hidden graph errors. Builders of agent workflows and RAG-style decomposition systems should add graph-level validation before trusting computed confidence. The Qwen3-32B versus Qwen3-8B result is a concrete warning that scaling a model does not guarantee better sub-question answering.

#Qwen3#Meta#reasoning models#AI safety
rohanpaul_ai@rohanpaul_aiSource
PapersBuildersSelected12:03
Google Paper: OO-Harness Specialist Harnesses Outperform One Global Agent Harness

The object-oriented structure explains why single-harness tuning keeps trading one task type's gains for another's losses, and shows a router can avoid that cost on out-of-distribution tasks. Teams building agentic pipelines can adopt the pattern directly: keep the base harness as fallback and route to specialists only when runs match known failure clusters. The AutomationBench comparison against Meta-Harness gives a concrete baseline for evaluating similar routing setups.

#Google#agents#OO-Harness#AutomationBench
rohanpaul_ai@rohanpaul_aiSource
PapersBuildersSelected74°11:03
AI Discovers A-greement, a Faster Consensus Protocol

The non-obvious point is that the agent harness itself, AI-greement, found bugs in established protocols while verifying new ones — a reusable recipe for applying coding agents to distributed-systems research. Distributed-systems engineers and researchers can study the harness design to run their own autonomous protocol search. It complements classics like Paxos and Raft with measurable latency gains in geo-replicated settings.

#A-greement#consensus protocols#AI agents#Lean
arXiv cs.AI@Clément Burgelin et al. (8)Source
PapersBuildersSelected10:41
arXiv Paper Finds Formatting Cues Lost in Agent Document Ingestion

The reliability bottleneck for document-driven agents sits in the ingestion layer, not the reasoning model, so swapping Claude for GPT-5.6 will not fix it. Agent builders working on compliance or policy workflows can instrument conversion pipelines to preserve highlights and colour, and typing the cue explicitly adds a further +0.085 F1 over passive preservation. Since no mainstream converter or parsing benchmark currently scores colour fidelity, teams can gain an edge by auditing their own ingestion against OOXML properties.

#docling#Claude#GPT-5.6#RAG
arXiv: OpenAI@Yikai Zhao et al. (3)Source
PapersBuildersSelected02:05
AKV: Agents Manage Their Own KV Cache to Cut LLM Costs

This reframes context engineering as a systems problem: instead of editing the visible prompt, the agent rewrites its own cache, avoiding the re-prefill penalty that summarization triggers. Teams building long-running agents with multi-turn tool calls can cut serving cost measurably without changing what the model sees. The gains apply to any stack serving LLMs with prefix caching, such as vLLM-based deployments.

#KV cache#agents#context engineering#reasoning models
DAIR.AI@dair_aiSource
PapersBuildersSelected00:19
PyTorchCon 2026 Keynote Maps Five Layers of the Agentic Stack

The bidirectional design idea turns agent evaluation data into training rewards, a concrete recipe for closing the loop between deployment and model improvement. Researchers and framework builders can experiment with reward signals derived from real agent traces rather than static benchmarks. The framing also clarifies where projects like PyTorch and the Agentic AI Foundation sit relative to each other in the stack.

#PyTorch#agents#Agentic AI Foundation#fine-tuning
官方账号PyTorch@PyTorchSource