RiLM: Efficient Language Modeling via Geodesic Decoding
RiLM: Parameter-Efficient Language Modeling via Geodesic Decoding
HypRiLM achieves 2x better perplexity than strong baselines at under 300k parameters, making it ideal for edge deployment and domain adaptation.
RiLM removes the output matrix layer entirely, using geodesic distance on a Riemannian manifold for next-token probabilities.
HypRiLM achieves 54.2 validation perplexity on WikiText-2, outperforming LSTM, Transformer, and SSM controls at 113-147 PPL.
The model transfers across corpora and larger vocabularies, with hyperbolic curvature providing selective benefits.
RiLM: Parameter-Efficient Language Modeling via Geodesic Decoding
Language models under one million parameters matter for edge deployment, domain adaptation, and reproducible research, yet a two-layer LSTM or Transformer at embedding width d = 128 still spends roughly one third of its capacity on the output matrix W_out in R^(d x |V|). We propose Riemannian Language Models (RiLM), which remove that layer entirely: context unfolds as a trajectory on a Riemannian manifold, and next-token probabilities arise from squared geodesic distance between the current state and vocabulary embeddings. The same embedding map serves input and output -- decoding is geometry. We instantiate the framework on flat R^d (Flat RiLM) and the Poincare ball H^d (HypRiLM) with a shared MLP composition map phi (~290k parameters, d = 128, |V| = 2000). Across five seeds on WikiText-2, HypRiLM reaches 54.2 +/- 0.2 validation perplexity versus 87.6 +/- 0.6 for Flat RiLM; tied and matched LSTM, Transformer, and SSM controls remain at 113-147 PPL on WT-2 -- HypRiLM leads by roughly 2x over the strongest tied recurrent baseline (SSM, 113.0 +/- 3.8). Penn Treebank and a 10k-vocabulary stress test confirm that geodesic decoding transfers across corpora and larger |V|, while hyperbolic curvature helps selectively. We also characterize boundary collapse in naive hyperbolic recurrence and show how Mobius stabilization restores trainability. Claims are scoped to controlled small-model comparisons, not full-vocabulary state of the art.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include RiLM, language models, parameter efficiency.
Source attribution: curated from arXiv cs.AI (@Fang Li).