FlyOCR AI model trained on fruit fly brain
For developers interested in novel neural architectures, this model demonstrates a creative approach to document parsing using biological-inspired neural networks, offering a new perspective on AI model design beyond standard transformer-based VLMs.
FlyOCR is an AI model that uses a full MaleCNS v1.0 fruit fly brain to read PDFs. It splits PDF images into glyphs, maps pixels to neural activations, and runs simplified neural dynamics across 166k neurons and 25m edges. On a Microsoft 10-K document, it correctly identifies ~86% of balance sheet headings and 87% of sampled characters and digits. The model's architecture is inspired by doomfly and aims to replace existing document parsing VLMs.
Introducing FlyOCR 🪰 - I trained a fly brain to read a PDF It uses the full MaleCNS v1.0 fruit fly connectcome. The architecture is inspired by doomfly by @nftechie_ The fly splits a pdf image into individual glyphs, maps pixels into receptor activations, runs simplified current-based dynamics across the 166k neurons and 25m edges in the circuit, applies a compact readout model on the downstream spikes, and concatenates everything into the parsed output. On reading an actual Microsoft 10-k, the fly gets ~86% over the balance sheet heading, but is largely able to read the numeric values correctly. Over 1.7k+ sampled glyphs (chars+digits) it gets 87% accuracy. With enough training it might match some of the latter-generation MNIST models! Maybe eventually we’ll replace our doc parsing VLMs with flies. Full video below. Repo with full code + report: github.com/jerryjliu/fly_… f Your browser does not support the video tag. 🔗 View on Twitter 💬 3 🔄 0 ❤️ 5 👀 374 📊 4 ⚡
