Developer gets a $10 ESP32 microcontroller to run a tiny language model

Developer SlvDev demonstrated that a low-cost ESP32-S3 microcontroller can host a functional language model despite its modest 520 KB SRAM and 8 MB PSRAM. By selecting Microsoft Research’s TinyStories model and applying 4-bit quantization, the memory requirement dropped from 60 MB to about 14.9 MB. Further savings came from per-layer-embedding, which moved roughly 12 MB of weights to the board’s flash, leaving only 2 MB of active data in RAM.

This configuration delivers close to 10 tokens per second, enough for short story generation but insufficient for complex tasks like chat or code generation. SlvDev posted the full process on GitHub and a walkthrough on the Better Stack YouTube channel. While the achievement is mainly a proof-of-concept, it underscores how aggressive model compression can enable AI on ultra-low-power edge devices.