Bias-Induced Crossover in Dense Associative Memory Capacity
Bias-Induced Crossover in Absolute Capacity of Dense Associative Memory
This paper provides a theoretical analysis of bias effects on associative memory capacity, which is relevant for understanding the fundamental limits of neural network models used in AI applications.
This academic paper analyzes the effect of bias on the absolute capacity of dense associative memory systems.
It examines centered binary patterns under the Krotov-Hopfield single-site criterion where the probability of error is 1/N.
For polynomial interactions of order n, the analysis shows that for fixed q < 1/2, the capacity is O(N^(n/2)) for even n ≥ 4 and O(N^((n+1)/2)) for odd n ≥ 5.
Asymptotic matching predicts a bias-induced crossover in the region 1-2q = O(ln N/N^(⌊n/2⌋-1)).Computer simulations are compared with finite-size conditioned-Gaussian predictions, and an activity-dependent control potential is proposed to restore the N^(n-1)/ln N capacity for fixed 0 < q < 1/2 within the conditioned-Gaussian approximation.
Bias-Induced Crossover in Absolute Capacity of Dense Associative Memory
The absolute capacity of dense associative memory has mainly been analyzed for unbiased patterns. Here we examine the effect of bias in centered binary patterns under the Krotov-Hopfield single-site criterion $P_{\mathrm{error}}=1/N$, where $P_{\mathrm{error}}$ is the probability that a single-site flip lowers the energy of a stored pattern and $N$ is the number of neurons. Each pattern component takes $1-q$ with probability $q$ and $-q$ otherwise, where $0<q\le1/2$. For polynomial interactions of order $n$, a signal-to-noise analysis gives an absolute capacity of order $N^{n-1}/\ln N$ at $q=1/2$. For fixed $q<1/2$, however, the capacity is $O(N^{n/2})$ for even $n\ge4$ and $O(N^{(n+1)/2})$ for odd $n\ge5$. For $n=3$, both the unbiased and fixed-bias capacities remain $O(N^2/\ln N)$. For $n\ge4$, these different asymptotic forms imply a nonuniform large-$N$ limit near $q=1/2$. Asymptotic matching predicts a bias-induced crossover in the region $1-2q=O(\ln N/N^{\lfloor n/2\rfloor-1})$. The crossover originates from a bias-dependent crosstalk mean that reduces the stability of sites carrying the more frequent value $-q$. Computer simulations are compared with the finite-size conditioned-Gaussian predictions. An activity-dependent control potential that cancels the conditional crosstalk mean restores the $N^{n-1}/\ln N$ capacity for fixed $0<q<1/2$ within the conditioned-Gaussian approximation.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Key entities and technical topics under continuous tracking include Bias-Induced Crossover, Dense Associative Memory, Krotov-Hopfield.
Source attribution: curated from arXiv cs.LG (@Yuto Sakurai, Takeaki Shimokawa, Kazunori Iwata, Kazushi Mimura).