PRISM tool-boundary optimization paper
Product builders can improve reliability by focusing on tool-boundary edits rather than model changes, with PRISM showing 14.2 pp lift on BFCL benchmark.
Arky Yang's new paper (arXiv, 4 Sep 2026) introduces PRISM for optimizing LLM tool-agent harnesses. The research uses fixed inner model gpt-5.4-mini and outer proposer claude-opus-4.7 on BFCL and tau2 benchmarks. PRISM-MW achieved mean lifts of 14.2/14.9/10.1 pp with positive RelLift95 on all three tasks. The paper shows harness optimization outperforms model improvement for agent performance.
I like this framing a lot: agent improvement is harness improvement, not model improvement The interesting interventions are often at the tool boundary: - what context you pass - when tools are provided - how you recover from failure - what gets measured afterward That’s the loop we’re building around deepagents (the orchestration logic) + LangSmith (how you measure) Arky Yang @arkyyang Beyond Prompts: Measuring and Optimizing LLM Tool-Agent Harnesses — new paper (arXiv, 4 Sep 2026; accepted EMNLP 2026). arxiv.org/abs/2609.05736 Treats harness optimization around a fixed model as budgeted selection: edits are guarded intercepts at the tool boundary, not rewrites of execution logic. PRISM clusters failures, routes each repair to a prompt, middleware, or joint surface, and picks candidates by gate pass-rate AND reliability (RelLift95), not score alone. 3 distinct takeaways for product builders: - Report selected-harness reliability, not just mean lift. BH found the best individual harnesses but had negative reliable lift on both tau-bench domains — a strong explorer, an unreliable selector. - Constrain middleware to tool-boundary edit patterns (silent correction, error blocking, prerequisite blocking). Unconstrained middleware (BH-MW) fell below its own prompt-only variant on Retail; the constraint, not the surface, is what makes middleware safely searchable. - Route each failure to the right surface: prompts for global policy, middleware for locally-checkable execution invariants. Removing routing collapsed lift first-order on all three tasks; prompts alone miss runtime failures, unconstrained middleware adds selection risk. Experiment setup (from the paper): fixed inner execution model gpt-5.4-mini, outer proposer claude-opus-4.7. Benchmarks: BFCL multi-round (600-case scorecard) and tau2-Retail / tau2-Telecom (74-case scorecard each of 114 tasks). Baselines BH, GEPA, MIPROv2 + PRISM; disjoint repair/gate/scorecard splits, 4 seeds, up to 16 runs. Baseline = native Deep Agents (v0.5.6). Metrics: mean held-out lift (pp), WorstLift, repeatability, RelLift95(B) by 5,000 bootstrap. PRISM-MW mean lifts 14.2 / 14.9 / 10.1 pp and positive RelLift95 on all three. 🔗 View Quoted Tweet 💬 4 🔄 2 ❤️ 9 👀 1665 📊 4 ⚡