hunch builds a real-time learning local semantic decision system on Google's EmbeddingGemma 2
The real shift is architectural: routing routine agent decisions to a continuously learning local embedding model cuts both latency and token spend without retraining loops. Developers building agent monitoring, log analysis or task routers can replicate this with EmbeddingGemma 2 on commodity CPUs. Against cloud-only routing stacks, the sub-100ms local inference makes real-time agent loop control practical.
A new open-source project called hunch uses Google's EmbeddingGemma 2 with a lightweight classifier and teacher-model distillation to build a local semantic decision system that learns in real time. It runs on CPU, producing a text judgment every 22 milliseconds and an image judgment in about 100 milliseconds, at under US$0.03 per million tokens. A large model acts as the teacher, labeling decisions while the small local model progressively takes over high-frequency judgments such as tool-call failure detection, log anomaly classification, user intent routing and agent routing. Complex questions stay with the large model, while simple, frequent and stable tasks are handled locally.
又有人把谷歌的EmbeddingGemma 2玩出了新花样！做成了一个能实时学习的本地语义决策系统，CPU上就能实现毫秒级判断
文本22毫秒出一次判断，图片约100毫秒，每百万token成本不到3美分，本地运行
项目叫hunch，让大模型当老师，让本地小模型边用边学，逐渐接管大模型的高频判断任务
架构：EmbeddingGemma 2+轻量分类器+教师模型蒸馏
像工具调用是否失败、日志是否异常、用户意图属于哪一类、应该路由给哪个Agent，这些判断每次都调用大模型，既有延迟也会持续消耗Token
大模型先提供判断、充当老师，本地小模型持续学习，把重复任务接过去，复杂问题继续交给大模型，简单、高频、稳定的任务交给本地轻量模型
让AI的高频判断形成一个持续学习、逐渐降低成本的闭环，让小模型也能承担更有实际价值的任务
对Agent日志分析、工具调用监控、实时分类和任务路由，这种架构比较有潜力
#hunch #EmbeddingGemma2 #语义决策系统
