Karpathy maps the inference chain from 1970s sci-fi to LLMs
The post reframes today's AI as the end point of a contingent chain rather than an obvious trajectory, which sharpens how readers judge current forecasts. Practitioners and analysts studying AI's historical drivers get a compact map linking computing, networking, data, Moore's law and scaling economics. It also clarifies why capital and capability now concentrate in a handful of labs that train and deploy copies of a few models.
Andrej Karpathy argued in a post on X that predicting today's AI from around 1970 would have required chaining several hard inferences. His list runs from digital knowledge work and a global internet (echoing Licklider's 1963 'Intergalactic Computer Network') to big data, statistical learning based on Shannon's 1948 n-grams and Rosenblatt's 1958 Perceptron, then LLMs as a high-fidelity ELIZA (1966), and finally scaling laws concentrating training in a few organisations. He noted counter-examples like HAL but said most pre-2000 sci-fi missed this path. He added that such a prediction would have to 'short' the idea that robots could be programmed by hand.
@shostekofsky There are a few notable counter-examples (e.g. HAL) but this is basically right for a lot of pre-2000 sci-fi. I think getting AI right from e.g. somewhere in the ~1970s would require a fairly complicated set of inferences, probably the biggest things would be: Image the world in 50 years. 1. "Digital knowledge work / computing". The vast majority of knowledge work (which makes up a very large fraction of the economy) goes digital. As computers get cheaper, much of society’s work happens on screens, over billions of machines in homes and offices. 2. "Internet". Similar to ARPANET (and see Licklider’s “Intergalactic Computer Network” (1963)), this creates demand for a global network of computers where people work, argue, teach, joke, and socialize. 3. "Big data". This creates a very large, compounding repository of digital information, with traces of how people reason, how they create, what they know etc. 4. "Statistical learning". Trying to program intelligence by hand is extremely difficult. But Shannon's n-gram approximations of English (1948) show examples of how language has statistical structure that can be "learned". You could imagine doing so over extensions of cybernetic computing (e.g. in the style of Rosenblatt's Perceptron 1958). So instead of programming thought, we could fit a statistical model to the record. 5. "LLMs". This might lead to something along the lines of ELIZA (1966), but with enough data (per 1-3), and enough compute (see Moore's law 1965), the imitation might become of extreme hight fidelity to the point that it's economically extremely useful. 6. "Scaling laws". These networks may demand a huge amount of data / computing resources and expertise to train. It's plausible that the more of each you have, the better your ELIZA gets. The economics concentrate, where a few organizations build a few minds and then deploy copies of them digitally. Something like that. Basically, it's a lot of leaps. You have to start with computers, to internet, to big data, to big compute (Moore's law), which then allow large-scale statistical learning approach based on neural networks, and then posit scaling laws, and then imagine the form factor of language models as a lot more sophisticated ELIZA. And you'd have to short everything else, most predominantly I think the idea that we'd figure out how to program robots by hand. 💬 5 🔄 12 ❤️ 76 👀 4018 📊 17 ⚡
