PapersSelected11月6日 00:54

Stanford paper maps the in-context learning circuit in small LLMs

rohanpaul_ai@rohanpaul_ai

Post

Why we picked it

The practical implication is that few-shot prompting works by label lookup, not feature learning, so prompt design should repeat labels several times and use classes the model can already distinguish. Builders tuning small models for agent pipelines or classification tasks can use this to cut prompt size and failure cases. The finding also explains why tiny models fail on tasks needing unseen features.

A Stanford paper traces how small language models solve few-shot tasks in Qwen3-4B, SmolLM3-3B and Llama-3.2-3B. Each label token stores the class of its sentence, and about 90 later attention heads match the new sentence against those stored classes. The lookup only activates when a label repeats: with 1 example per class the model was right about 20% of the time, rising to 96% with 6. The authors conclude that examples cannot teach a pattern the model does not already notice, such as the third letter of a word.

Original · rohanpaul_ai

New Stanford paper finds that showing a small AI language model more examples cannot teach it a pattern it does not already notice, such as the third letter of a word.

finds that LLMs learn from few-shot examples by looking up stored labels, and only switch that lookup on when labels repeat.

An early signal opens this lookup only when some label repeats. With 1 example per class the model was right about 20% of the time, and with 6 it was right 96%.

Give a model a dozen sentences tagged with made-up labels like foo and bar, and it learns the mapping without weight updates. Earlier work found parts of this, but not how they fit together.

They traced the full path in Qwen3-4B, SmolLM3-3B and Llama-3.2-3B. Each label token stores the class of its sentence, and about 90 later heads match the new sentence against those stored classes.

For few-shot prompts, repeat every label several times and pick classes the model can already tell apart, since examples cannot teach a feature it lacks.

– arxiv. org/abs/2611.05344

Title: "A Circuit for In-Context Learning"

#Qwen3 #Llama-3.2 #SmolLM3 #prompt engineering #reasoning models

Keep reading

Recent stories with the same tag or section

Papers BuildersSelected14:04

Training Data and Routing Choices Push Open Models on ARC-AGI-2

The clearest lesson is that most of the gain for the 27B model came from the serving-side inference procedure rather than fine-tuning data, which changes how practitioners should budget effort on reasoning benchmarks. Builders running small-model inference on constrained hardware can reuse the expert-pruning and KV-cache techniques directly. The study's 189-run methodology on the ARC Prize Foundation's semi-private set also offers a rare reproducible comparison for offline ARC-AGI-2 solvers.

#Qwen3 #ARC-AGI-2 #reasoning models #fine-tuning

arXiv cs.AI@Nguyen Minh Thang, Trinh Thi PhuongSource

Papers BuildersMulti-source11:09

Chinese contentD3D：异步投机解码框架，Qwen3-8B 上实现 5 倍加速

复旦团队新出的 D3D，让扩散草稿不用等目标模型的隐藏状态，解码比 DFlash 快一截，GSM8K 上能到 7.48 倍，做推理加速的可以看看。

#D3D #Qwen3 #投机解码 #扩散模型

Event

arXiv cs.LG@Baoyou Chen et al. (7)5 sourcesSource

Papers BuildersMulti-source11:08

Chinese contentGIST：面向离线 RLVR 的分层训练数据选择方法

一篇教你离线 RLVR 该挑哪些回答、哪些 token 来训的论文，在 Qwen3-0.6B 的 GSM8K 上比七个基线都强，还省了三分之二的监督 token。

#RLVR #Qwen3 #GSM8K #数据选择

Event

arXiv cs.LG@Xuliang Huang et al. (4)5 sourcesSource

Papers Builders13:11

Chinese contentPEARL 方法抑制语言模型不良行为，训练后不易复发

研究者把模型“越训越坏”的问题做成了可测量的框架，还给出 PEARL 这个 GRPO 改进版，做对齐或安全微调的可以看看怎么压制不良行为不反弹。

#PEARL #GRPO #Qwen3 #GPT-OSS

arXiv cs.LG@Luxi He et al. (5)Source

Papers BuildersMulti-sourceSelected14:19

DecisionRAG: Cornell paper replaces retrievers with a decision model over document trees

The interesting shift is that retrieval becomes a tree-traversal decision problem instead of an embedding-similarity problem, so latency scales with tree depth rather than corpus size, with billion-paragraph corpora theoretically needing around five decisions. Teams paying for a reranker stage, or struggling with weak first-stage recall on help centres and document corpora, can test the tree-plus-decision-model setup; on NQ a dense retrieval stage can optionally be mixed in to recover over-pruned evidence. It competes with Qwen3-Reranker-8B pipelines at lower cost.

#DecisionRAG #Jev #RAG #Qwen3

Event