LLM Budget Allocation Under Uncertainty
Which LLM for Which Work? Budgeted Model Allocation under Uncertain Evaluation
For companies in ASEAN with limited AI budgets, this research provides a method to optimize LLM allocation when performance data is uncertain, potentially saving significant costs.
The paper addresses a company's challenge of allocating fixed AI budgets across different LLMs for recurring workloads. The core difficulty is estimating model performance without complete quality tables, as models are rarely compared on the same work and recorded scores are often proxies for actual outcomes. The authors propose CASE (causal active sequential experimentation), which targets evaluation to specific model-workload pairs where evidence can matter most. Experiments show that better model quality information yields more savings than optimizing assignments on existing estimates.
Which LLM for Which Work? Budgeted Model Allocation under Uncertain Evaluation
A company with a fixed artificial intelligence (AI) budget must decide which large language model (LLM) handles each recurring workload. What it lacks is the quality table, how well each model performs on each workload. Given that table, the decision is a multiple-choice knapsack problem and is routine to solve, so estimating it is the difficulty, and that estimation fails in two ways. Models are rarely compared on the same work, and the recorded score is usually a proxy rather than the outcome the company values. Causal and off-policy methods repair the first but condition on the second, while evaluator-validation methods estimate the second but stop short of the decision. Worse, buying more re-evaluation cannot settle the second: randomization governs which requests are scored, not how a score is produced, so the table stays uncertain however much evaluation is purchased. Yet the deployment decision may still be determined even when the table is not. We therefore ask whether one assignment stays optimal across every quality table consistent with the evidence. For the fixed-budget problem, this admits an exact two-solve certificate: solve once at the estimated table and once at a least-favourable table. Agreement certifies the assignment; disagreement identifies the model-workload pairs where further evidence can matter. We propose CASE (causal active sequential experimentation), which targets evaluation to those pairs and repeats the test as evidence accumulates. On a production log, the measurement failure is the larger of the two: correcting assignment exactly still leaves most of the loss, and randomized re-evaluation does not remove it. In our experiments, the available evidence often does not determine the assignment. On paid software tasks, better information about model quality yields more savings than further optimization of the assignment on the same estimates.
