做 AI 应用或代理的团队，如果被 token 成本压得喘不过气，这个思路值得一试——用检索让小模型干大模型的活，省 20 倍成本不是梦。
Exa CEO Will Bryk 在 a16z 的访谈中提出，不应为所有任务使用巨型模型，而应通过检索增强让小模型表现如大模型。他称，大模型负责决策，向小模型分配任务，小模型借助检索可更准确可靠。Exa 通过高效筛选网络信息，能为客户节省高达 20 倍的成本。Andrej Karpathy 也评论认为，模型大小竞争正逆向发展，未来会出现极小的、能“思考”的模型，但前提是先用大模型帮助生成理想训练数据。
Exa CEO Will Bryk explains why retrieval can help solve the tokenpocalypse: "We should not be using...
Exa CEO Will Bryk explains why retrieval can help solve the tokenpocalypse: "We should not be using gigantic models for every task." "You should use a family of models of different sizes. The big model decides what to do, and it dishes out commands to the small models, and those small models can be way more accurate and reliable if they're using retrieval." "Retrieval helps small models act like big models... We do save our customers a huge amount of tokens because they can use smaller models and use retrieval." "We could save 20x on cost for customers compared to other providers by being very efficient in what information from the web the agent actually sees." @WilliamBryk with @sarahdingwang Your browser does not support the video tag. 🔗 View on Twitter Andrej Karpathy @karpathy LLM model size competition is intensifying… backwards! My bet is that we'll see models that "think" very well and reliably that are very very small. There is most likely a setting even of GPT-2 parameters for which most people will consider GPT-2 "smart". The reason current models are so large is because we're still being very wasteful during training - we're asking them to memorize the internet and, remarkably, they do and can e.g. recite SHA hashes of common numbers, or recall really esoteric facts. (Actually LLMs are really good at memorization, qualitatively a lot better than humans, sometimes needing just a single update to remember a lot of detail for a long time). But imagine if you were going to be tested, closed book, on reciting arbitrary passages of the internet given the first few words. This is the standard (pre)training objective for models today. The reason doing better is hard is because demonstrations of thinking are "entangled" with knowledge, in the training data. Therefore, the models have to first get larger before they can get smaller, because we need their (automated) help to refactor and mold the training data into ideal, synthetic formats. It's a staircase of improvement - of one model helping to generate the training data for next, until we're left with "perfect training set". When you train GPT-2 on it, it will be a really strong / smart model by today's standards. Maybe the MMLU will be a bit lower because it won't remember all of its chemistry perfectly. Maybe it needs to look something up once in a while to make sure. 🔗 View Quoted Tweet 💬 3 🔄 3 ❤️ 18 👀 3336 📊 5 ⚡