PapersSelected

LLMs Program Tabular Foundation Models Without Gradient Optimization

Weights & Caches: LLMs Can Program Tabular Foundation Models

Why we picked it

Steering a pretrained model through cache edits rather than retraining points to cheaper, more surgical model customization than fine-tuning. Teams maintaining tabular prediction pipelines, and researchers studying model internals, gain a concrete recipe: ask an LLM to repair or redirect TFM behaviour and validate on held-out data. The 0.849 vs 0.876 ROC AUC gap against the pretrained baseline shows the approach is close to, but not yet matching, standard training.

A new arXiv paper (2611.20174) demonstrates that GPT-6 Astra can directly program statistical computations in tabular foundation models (TFMs) by manipulating weights and caches, without any gradient-based optimization. In the first of three settings, Astra programs nanoTabPFN from zero-initialized weights, reaching a mean ROC AUC of 0.849 across 20 TabArena datasets, compared to 0.876 for the pretrained baseline.

Body

The second setting shows Astra diagnosing and repairing a signed-product regression failure in pretrained TabPFN-3.5, restoring near-perfect predictions on held-out synthetic datasets while keeping performance on TabArena-Lite, at a cost of only 0.08% average relative error increase. Third, the model writes KV-cache-editing programs that steer TabPFN-3.5's predictions on unseen synthetic datasets, reversing linear trends and converting sine waves into approximate square waves without touching the model weights.

The work builds on recent weight-space programming research showing LLMs can reason about neural-network parameters directly. The authors argue LLM-driven programming could complement gradient-based optimization across TFM development and usage, and offers a new method for investigating and intervening in a model's internal computations.

Original · arXiv cs.LG

Weights & Caches: LLMs Can Program Tabular Foundation Models

Neural networks are typically programmed through optimization, rather than by explicitly specifying their computations. Recent work on weight-space programming suggests that large language models (LLMs) can instead reason about and manipulate neural-network weights directly. In this work, we show that LLMs can directly program statistical computations in tabular foundation models (TFMs), whose parameters enable inference across different datasets. We investigate three complementary settings using GPT-6 Astra, without gradient-based optimization. First, Astra directly programs nanoTabPFN from zero-initialized weights, achieving a mean ROC AUC of 0.849 across 20 TabArena datasets, compared to 0.876 for a pretrained baseline. Second, Astra diagnoses and repairs a signed-product regression failure in pretrained TabPFN-3.5, restoring near-perfect predictions on held-out synthetic datasets while preserving performance on TabArena-Lite, with only a 0.08% average relative increase in error. Third, it writes KV-cache-editing programs that steer TabPFN-3.5's predictions on unseen synthetic datasets, demonstrating transformations such as reversing linear trends and converting sine waves into approximate square waves, without modifying model weights. Together, our findings suggest that LLM-driven programming could complement gradient-based optimization across TFMs' development and usage, while offering a new way to investigate and intervene in their internal computations.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include TabPFN, GPT-6 Astra, KV cache, TabArena.

Source attribution: curated from arXiv cs.LG (@Shi Bin Hoo, Magnus Buehler, Omar Swelam, Frank Hutter, Lennart Purucker).