How-toSelected

Sebastian Raschka calls for a modular LLM building-block API

Why we picked it

The practical gap here is interoperability, not capability: open-source architecture variants already exist but are locked into incompatible codebases. Researchers running ablations, engineers evaluating architectures like Mamba2 versus attention, and labs publishing open-weight models would all gain from standardised, swappable components. Existing libraries such as Megatron Core and OLMo-core cover their own model families, so cross-family comparison still means manual porting today.

AI researcher Sebastian Raschka argued in a post on X that the open-source LLM ecosystem lacks a shared building-block API that would let researchers swap architectural components without rewriting model code. He highlighted the high switching costs when running experiments or ablation studies in PyTorch or raw CUDA, such as replacing GQA with MLA, dense FFNs with MoE, attention layers with Mamba2 or Gated DeltaNet, or standard residual connections with attention residuals. He noted the process stays messy even with LLM-assisted coding.

Body

Raschka suggested the API could sit on top of PyTorch or CUDA directly, but stressed it would need careful long-term engineering and maintenance rather than a weekend project, plus enough efficiency that practitioners trust it in production. He acknowledged existing efforts including torchtitan, torchtune, Megatron Core and OLMo-core, but said none reach the right level of abstraction. He also flagged the adoption chicken-and-egg problem: the library needs popularity to gain users and users to gain popularity.

The proposal effectively asks a large, well-funded lab to back the effort, since sustaining production-grade infrastructure is costly. For researchers comparing architectures across open-weight models from different labs, a common component API would cut experiment setup from days to hours and make ablation results easier to reproduce and compare.

Original · Sebastian Raschka

The open-source LLM ecosystem could really benefit from an LLM building-block API.

We have so many LLM components across different models now. Whether you are using PyTorch or C++ and CUDA code directly, there is a major switching cost when you want to run experiments or ablation studies, say swapping GQA with MLA, dense FFNs with MoE, attention layers with Mamba2 or Gated DeltaNet, or standard residual connections with attention residuals.

Currently, it’s very messy even with LLM-assisted coding.

It could maybe be on top of PyTorch or CUDA directly, but it needs to be carefully engineered and maintained (versus quickly vibe-coded on a weekend) and efficient enough that people trust it to use in production (which is really a requirement to move beyond toy examples).

Maybe some food for thought for a big, well-funded lab :)

(There are some efforts that go into that direction, like torchtitan, torchtune, Megatron Core, OLMo-core, but they are not quite that level of abstraction, I'd say.

And, as always, there’s the chicken-and-egg problem: it needs to be popular to get widely adopted, and it needs to get widely adopted to become popular.)

Industry Coordinates & Source Verification

This report is categorized under AITOP's "How-to" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include PyTorch, Megatron Core, Mamba2, MoE.

Source attribution: curated from Sebastian Raschka (@rasbt).