DeFiFlowBench benchmark for safe DeFi workflow synthesis
DeFiFlowBench: Benchmarking and Improving Safe Executability in Natural-Language DeFi Workflow Synthesis
This research provides a new benchmark and model for ensuring safety in DeFi workflow synthesis, which is crucial for preventing costly and unsafe trades in decentralized finance applications.
A new benchmark called DeFiFlowBench tests 207 prompts for natural-language DeFi workflow synthesis. It measures graph coverage and safety predicates, finding that direct prompting leads to 14-19 unsafe executions per configuration under a 5% price-impact cap. A proposed model, Koan-Safe, achieves a 0.67 score on a static safety proxy, with no unsafe executions on the benchmark, compared to 0.33 for the best baseline.
DeFiFlowBench: Benchmarking and Improving Safe Executability in Natural-Language DeFi Workflow Synthesis
A structurally valid DeFi workflow can still authorize a costly trade. We introduce DeFiFlowBench, a benchmark of 207 team-authored prompts for natural-language DeFi workflow synthesis. It measures graph coverage, configuration completeness, and declared safety predicates, then tests supported trade configurations on a local EVM. Direct, constrained, and few-shot prompting produce 14-19 unsafe held-out executions per configuration under a fixed 5% price-impact cap. A slippage bound derived from a quote does not prevent the price impact of the order itself. We propose Koan-Safe, which combines a prompt-only intent parser, a replaceable generator, and structural repair with default safety parameters. On 75 held-out workflow prompts, its hybrid variant scores 0.67 on the static safety proxy, compared with 0.33 for the best baseline. Koan-Safe records no unsafe executions on the saved benchmark outputs. A matched-candidate ablation produces 14-17 unsafe executions when enforcement is disabled. Additional tests expose the limits of default injection: permissive existing thresholds can still authorize unsafe trades. A separately evaluated policy cap addresses this failure on a 36-case diagnostic grid. These results support explicit trade protections and execution-based evaluation, while distinguishing declared safety from a general guarantee.
