AITOPAI热报
PapersSelected

Untied Ulysses 让 Llama 3B 在单节点训练 3M 上下文

Official accountTogether AI@togethercompute
Post
View the post
Why we picked it

长上下文训练一直是显存大户，Untied Ulysses 让单节点就能跑 3M token，做 LLM 训练和推理优化的团队值得关注，能省下不少 GPU 预算。

Together AI 团队提出 Untied Ulysses 方法，解决了长上下文训练中的显存瓶颈。传统方法在单节点 8xH100 上训练 Llama 3B 模型时，仅模型参数就会耗尽显存，无法支持 3M token 的上下文长度。

Body

新方法通过优化注意力机制，在 8B 和 32B 规模下实现了比先前实现长 25% 的序列训练。这项研究让大模型长上下文训练变得更可行，降低了硬件门槛。

Original · Together AI

Training a Llama 3B model with a 3M token context on a single 8xH100 node fails because model parameters alone exhaust GPU memory. @m_ryabinin explains how Untied Ulysses, his team's latest research, pushes past that wall, training at 8B and 32B scale with 25% longer sequences than prior implementations.

https://t.co/nm0sjLUSUL

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Key entities and technical topics under continuous tracking include 长上下文, 显存优化, 注意力机制, Together AI.

Source attribution: curated from Together AI (@togethercompute).

#长上下文#显存优化#注意力机制#Together AI#训练效率

Keep reading

Recent stories with the same tag or section
PapersBuilders10:53
Chinese contentMPPA：免训练方法将扩散语言模型 LLaDA-8B 上下文扩展到 2 倍窗口

扩散语言模型长上下文外推一直是难点，这篇论文没做任何训练就把 LLaDA-8B 的 8192 token 困惑度从 19.51 降到 3.12，比窗口内还低。思路是利用解码过程中的掩码密度给 RoPE 加动态偏移，和自回归那边静态的 NTK 缩放完全不同。做 dLLM 推理或长文本任务的工程师可以直接看方法细节。

#LLaDA#dLLM#RoPE#长上下文
arXiv cs.LG@Gaurav Sarkar, Syed Affan DaimiSource
IndustryBuildersSelected02:46
Open LLM Architectures Shift From Full Attention to Hybrids

The notable point is that full attention has disappeared entirely from new open models, not merely become rarer — a structural shift that directly lowers inference cost at long context. Developers building agents or handling large codebases should weigh hybrid-architecture models for memory and latency savings, while being aware that fixed-state recurrent layers trade some retrieval fidelity. Teams comparing open-weight options for long-context serving will find this trend data useful for shortlisting candidates.

#open-source models#attention architectures#Mamba-2#long context
Official accountSebastian Raschka@rasbtSource
PapersBuildersSelected23:24
Microsoft harness lifts Llama3-8B long-context reasoning without training

The key insight is that evaluation and inference-time scaffolding can substitute for expensive data synthesis and fine-tuning on long-context tasks. Teams serving small open models like Llama3-8B or Qwen2.5-3B can apply the approach to extend context handling at under half the cost of training-based methods like NExtLong or PolicyLong. Engineers building local or low-budget deployments should test harness transfer before commissioning post-training runs.

#Llama3-8B#Qwen2.5-3B#Microsoft#long-context
Event
elvis@omarsar0Source
ModelsBuildersSelected21:53
i14 Labs launches Sugar Glider O(1) long-context LLM architecture

A fixed-size KV cache changes serving economics more than raw benchmark scores: it turns long-context deployment from a memory problem into a configuration choice. Teams building agents, RAG pipelines or document-heavy workflows should watch the waitlist and independent reproductions of the 1B-token retrieval claims. Compared with existing long-context approaches that accumulate KV cache linearly, the 32x throughput figure at 1M tokens is the number to verify.

#Sugar Glider#i14 Labs#long context#KV cache
elvis@omarsar0Source