别再以为跑AI需要几千块的显卡了!几十元的ESP32-S3,本地跑28M大模型Forget about expensive GPUs for AI! Run a 28M model locally on a $50 ESP32-S3.


AI free token/AI免费积分

本文来源:https://mp.weixin.qq.com/s/UrdeA8TUT2RYO02-Z-LxRw
若有版权问题,请来信告知:13951431913#139.com 或在右方提交评论。

一个ESP32-S3芯片,几十块就能买得到。上面跑了一个28.9M参数的大语言模型,完全离线,生成速度9.5 tok/s。

别再以为跑AI需要几千块的显卡了!几十元的ESP32-S3,本地跑28M大模型

你没看错。一包烟钱不到的微控制器,跑大模型。

之前能在这种芯片上跑的语言模型,最大只有26万参数。这次直接翻了100倍。

这玩意是怎么塞进去的?

先说硬件:ESP32-S3 有 512KB 的 SRAM(快速内存)、8MB PSRAM(扩展内存)、16MB 闪存。

别再以为跑AI需要几千块的显卡了!几十元的ESP32-S3,本地跑28M大模型

难题来了——通常整个模型必须放在快速内存里才能跑。512KB 能干啥?所以过去在微控制器上跑的语言模型被死死限制在几十万参数级别,实用性约等于零。

这次的做法很聪明:大部分参数不放在 SRAM,放在闪存里。

大语言模型的参数,大部分集中在嵌入表(embedding table)里——一个超大的查找表,模型只读它,不用在上面做复杂的数学计算。所以你可以把这张 2500 万行的表直接放闪存里,CPU 需要的时候按行读取就行。

别再以为跑AI需要几千块的显卡了!几十元的ESP32-S3,本地跑28M大模型

这个思路来自 Google Gemma 模型的 Per-Layer Embeddings(逐层嵌入)工艺。

结果就是:28.9M 参数,14.9MB 模型文件(4-bit 量化),全塞进一个 几十元的芯片。

跑起来什么样?

生成速度大约 9.5 tok/s——不算快,但够用。每个字符被写到连接在芯片上的一个小显示屏。

完全本地运行。没有 Wi-Fi 连接,不依赖任何服务器,所有推理都在 ESP32-S3 内部完成。

功耗?ESP32-S3 的典型功耗在几十 mW 级别,比你的手机充电器漏的电还少。

谁做的,怎么复现?

开发者 slvDev 把整个项目开源了,GitHub 地址:github.com/slvDev/esp32-ai

需要的东西:

• 一块 ESP32-S3 开发板(淘宝几十块人民币)

• 一个 1.8 寸 TFT 表明屏(选配,用来表明输出文字)

• ESP-IDF 开发环境

• 把模型烧进去,上电就能跑

具体步骤项目 README 写得很清楚,跟着走就行。

别再以为跑AI需要几千块的显卡了!几十元的ESP32-S3,本地跑28M大模型

这到底有什么用?

说实话,9.5 tok/s 的速度和 28M 参数的能力,干不了 ChatGPT 的活。

但它的意义不在表现。

第一,边缘 AI 的可行性验证。 如果 几十元的芯片能跑 28M 参数的模型,那更贵的 MCU(例如乐鑫新出的 ESP32-P4)能跑多少?10 亿参数的模型在嵌入式设备上跑,大概率就这一两年的事。

第二,离线推理的情形。 很多情形不需要连网——智能家居控制面板、工厂设备的本地语音助手、甚至可穿戴设备。如果你能把这些情形的 AI 推理做到芯片上,不需要每句话都去云端绕一圈,延迟、隐私、成本全部应对。

第三,给所有硬件发烧友的玩具。 花几十块钱自己焊一个跑大模型的设备,写在简历上、摆在家里、或者纯粹为了"我能在 ESP32 上跑 LLM"这句话的成就感。

别再以为跑AI需要几千块的显卡了!几十元的ESP32-S3,本地跑28M大模型

最后说一句

几十元的芯片能跑 28M 参数的 LLM。不是让你用它写论文,是让你知道——AI 硬件的门槛,已经低到这种程度了。

转给那个还在犹豫要不要买显卡跑模型的朋友。

项目地址: github.com/slvDev/esp32-ai

核心芯片: ESP32-S3(约 五十元)

模型大小: 28.9M 参数,14.9MB(4-bit)

生成速度: ~9.5 tok/s

Source: https://mp.weixin.qq.com/s/UrdeA8TUT2RYO02-Z-LxRw If there are copyright issues, please contact us at: 13951431913#139.com or submit a comment on the right.

An ESP32-S3 chip, which costs less than a pack of cigarettes. It runs a large language model with 28.9 million parameters, fully offline, at a generation speed of 9.5 tokens/second.

Forget about expensive GPUs for AI! Run a 28M model locally on a $50 ESP32-S3.

You read that correctly. A microcontroller for less than the price of a cigarette pack, running a large model.

Previously, the largest language model that could run on such chips was only 260,000 parameters. This time, it's jumped by 100 times.

How is this even possible?

Let's start with the hardware: The ESP32-S3 has 512KB of SRAM (fast memory), 8MB of PSRAM (extended memory), and 16MB of flash memory.

Forget about expensive GPUs for AI! Run a 28M model locally on a $50 ESP32-S3.

Here's the challenge: Normally, the entire model needs to be placed in fast memory (SRAM) to run. What can you do with 512KB? That's why language models on microcontrollers were previously limited to hundreds of thousands of parameters, rendering them essentially useless.

The approach this time is quite clever: Do not place the majority of parameters in SRAM. Place them in flash memory.

Most of the parameters in a large language model are concentrated in the embedding table — a massive lookup table that the model reads but does not perform complex mathematical operations on. Therefore, you can store this 25 million-row table directly in flash memory, and the CPU can read it row-by-row as needed.

Forget about expensive GPUs for AI! Run a 28M model locally on a $50 ESP32-S3.

This idea originates from Google Gemma's "Per-Layer Embeddings" technique.

The result is: 28.9 million parameters, a 14.9MB model file (4-bit quantized), all stuffed into a chip that costs a few dozen RMB.

What does it look like when it runs?

The generation speed is approximately 9.5 tokens/second — not fast, but usable. Each character is written to a small display screen connected to the chip.

It runs completely locally. There is no Wi-Fi connection, no dependence on servers. All inference is performed internally on the ESP32-S3.

What about power consumption? The ESP32-S3 has a typical power consumption of a few dozen milliwatts, which is less than the leakage current from your phone charger.

Who did this, and how can I replicate it?

The developer, slvDev, has open-sourced the entire project. The GitHub repository is at github.com/slvDev/esp32-ai.

What you need:

• An ESP32-S3 development board (approx. 50 RMB on Taobao)

• A 1.8-inch TFT display (optional, for visualizing text output)

• The ESP-IDF development environment

• Burn the model onto the chip, and you're ready to run.

The specific steps are detailed in the project's README. Just follow along.

Forget about expensive GPUs for AI! Run a 28M model locally on a $50 ESP32-S3.

What is the practical use of this?

To be honest, with a speed of 9.5 tokens/second and only 28 million parameters, this device cannot perform the tasks of ChatGPT.

However, its significance lies not in its performance.

First, it validates the feasibility of edge AI. If a chip costing a few dozen RMB can run a 28 million parameter model, how much can more expensive MCUs (like the new ESP32-P4) handle? Running billion-parameter models on embedded devices is likely just a year or two away.

Second, there are scenarios for offline inference. Many use cases do not require internet connectivity — such as smart home control panels, local voice assistants for factory equipment, or even wearable devices. If you can perform AI inference locally without routing every query to the cloud, you significantly reduce latency, improve privacy, and cut costs.

Third, it serves as a toy for all hardware enthusiasts. Spending a few dozen RMB to solder together a device that runs a large model is a great addition to a resume, a cool desk ornament, or simply a source of pride with the title "I ran an LLM on an ESP32."

Forget about expensive GPUs for AI! Run a 28M model locally on a $50 ESP32-S3.

In conclusion

A chip costing a few dozen RMB can run a 28 million parameter LLM. This isn't meant for writing theses, but rather to demonstrate that the threshold for AI hardware has dropped this low.

Share this with the friend who is still hesitating whether to buy a graphics card to run a model.

Project link: github.com/slvDev/esp32-ai

Core chip: ESP32-S3 (approx. 50 RMB)

Model size: 28.9 million parameters, 14.9 MB (4-bit)

Generation speed: ~9.5 tokens/second

暂无评论,快来发表第一条评论吧!No comments yet. Be the first!

📮 需求咨询📮 Inquiry