AdamX: Cosine similarity meets gradient descent
This is a new optimizer that could improve training efficiency for AI models.
We introduce AdamX, a first-order optimizer that incorporates cosine similarity as an adaptive mechanism for controlling update magnitudes.
The proposed method is scalable, model-agnostic, and straightforward to integrate into existing training pipelines.
We further introduce a variance rectification scheme that promotes smoother optimization during the early stages of training.
Overall, we provide empirical evidence that AdamX achieves competitive convergence rates across a range of benchmark datasets and architectures.
Performance is evaluated in terms of the number of epochs required to reach predefined performance thresholds under a fixed hyperparameter budget.
We introduce AdamX, a first-order optimizer that incorporates cosine similarity as an adaptive mechanism for controlling update magnitudes. The proposed method is scalable, model-agnostic, and straightforward to integrate into existing training pipelines. We further introduce a variance rectification scheme that promotes smoother optimization during the early stages of training. Overall, we provide empirical evidence that AdamX achieves competitive convergence rates across a range of benchmark datasets and architectures. Performance is evaluated in terms of the number of epochs required to reach predefined performance thresholds under a fixed hyperparameter budget. Code and Experiments available at: https://github.com/FranciscoCaldas/adamX.
This report is categorized under AITOP's "Models" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include AdamX, optimizer, gradient descent, machine learning.
Source attribution: curated from arXiv cs.LG (@Francisco Caldas, Ruben Belo, Cláudia Soares).
