想自己搭本地AI玩？这篇告诉你M5 Macbook Pro够不够用、Qwen 35b和9b差多远，别花冤枉钱。
作者分享在homelab运行本地AI模型的体验：消费级硬件（如M5 Macbook Pro 32/48GB、DGX Spark）可运行30-100B参数的量化模型（如Qwen 35b），但与1万亿参数的前沿云模型差距约18年。9B模型效果差、速度慢，需30-50 tok/s才能可用。本地模型适合简单查询，不适合编程。作者使用2台DGX Spark和5090 eGPU，建议公司配发MBP时选大内存。
some of you know that i've been tinkering with local AI models on my homelab setup. It's fun but her...
some of you know that i've been tinkering with local AI models on my homelab setup. It's fun but here's the reality today: - nothing you can run on consumer hardware is even close to what you can buy with a monthly subscription. The gap is ~18 years when you compare frontier to frontier, but you're actually comparing frontier cloud LLMs (1 trillion parameter models) to what you can run locally which might be 30-100 billion. This is a hobby! - best hardware off the shelf is probably a new macbook pro with lots of RAM or a gaming PC. Or if you want, a DGX Spark from Nvidia. A new M5 Macbook Pro with 32/48GB is probably the minimum spec for most. Most people's machines don't have enough memory though, and small models are not that good - this is because you need to run, say, a quantized version of qwen 35b and above to get good/coherent answers back from your prompts. If you run a 9b parameter model bc that's what fits, you won't like the results. Slow and bad for most use cases - you need 30-50 tok/s for it to be usable from a UX perspective. This is determined mostly by memory bandwidth which is determined by the type of GPU. So again, macbook pro or a gaming PC - normie prompts work pretty well now, the kind of stuff you'd plug into google. ("what is the population..." "what's the best..."). You don't want to code using local models - there's a huge huge gap however even with all these drawbacks, you learn a ton! And it's fun to experiment. I have 2x DGX sparks and a 5090 eGPU in my homelab and it's great. And you can always run GLM or whatever open weight frontier model on openrouter as a backup also. Def interesting to tinker with context window sizes, new tech like DSpark/etc. Over the next few years, a good number of white collar workers will run "good enough" models on their macbook pros issued at work and that'll unlock a lot of interesting new use cases. So when you get a new MBP from work, ask for more memory! 💬 11 🔄 0 ❤️ 13 👀 2288 📊 11 ⚡