前两篇完成了监控屏展示和语音唤醒控制。ESP32 BOX能听见声音、能锁屏调音量,但说到底还只是个遥控器,只能执行预定义的有限指令。
这篇就给 ESP32 BOX 接上大脑。
接上之后,你说"今天北京天气怎么样",它会调天气工具查数据,然后读给你听。你说"帮我搜一下 Rust 嵌入式开发",它打开浏览器跳到搜索页。你说"给我讲个笑话",它还真讲一个,虽然不一定好笑。
这个大脑叫 ZeroClaw。
ZeroClaw 是什么
ZeroClaw 是一个用 Rust 写的 AI Agent 框架,二进制文件 3.4MB,内存不到 5MB,启动不到 10ms。同类框架 OpenClaw 用 TypeScript 写,依赖 Node.js 运行时,28MB 的二进制还得带上近 400MB 的 Node.js 环境,内存 1GB+,在 0.8GHz 边缘设备上启动超过 500 秒。
ZeroClaw 是 OpenClaw 的高性能 Rust 实现,做的是同样的事情,就是理解自然语言指令并调用工具执行任务,但不绑定任何模型厂商,编译产物可以部署到任何地方。它支持 22+ 个 AI 供应商,OpenAI、Anthropic、DeepSeek、Ollama 本地模型都在列。对外暴露标准的 WebSocket 和 HTTP 接口,任何能发网络请求的程序都能对接。
在本项目中,ZeroClaw 部署在 Mac 上。hub-server 通过 WebSocket 把语音识别的文本转发过去,ZeroClaw 理解意图、调工具、生成回复,hub-server 再把回复合成语音播报给用户。
整体架构:两条通道
接上 ZeroClaw 后,语音指令变成两层漏斗。
第一层是本地指令路由器,上篇实现的锁屏、静音、音量调节就在这里。这些高频指令命中后直接执行,不经网络,零延迟。
第二层是 ZeroClaw。本地路由器匹配不上的指令,全部转发给它。这一层能调天气查询、浏览器操作、搜索引擎等外部工具,代价是必须承受大模型推理的网络与计算延迟。
设计思路就是简单直接:喊一声"静音"等三秒才执行,这个体验肯定很差。但"帮我查明天北京什么天气",多等几秒是可以接受的。
代码里的分流在 process_utterance() 函数里:
pubasyncfnprocess_utterance( &self, wav_path: String, net: std::sync::Arc>,) -> Result<()> {// 1. STT 转写:本地 Whisper 或远程 APIlet text = ifself.use_internal && self.local_stt.is_some() {self.process_utterance_internally(&wav_path)? } else {self.process_utterance_via_api(&wav_path).await? };// 2. 唤醒词检测 + 指令提取(精确匹配 + 拼音模糊匹配)// 3. 8 秒待命窗口状态机// ...(上篇已详述,此处省略)// 4. 优先尝试本地指令路由(无网络延迟)if Self::try_local_command(&final_command) { Self::send_local_done_cue(&net).await;returnOk(()); }// 5. 本地无匹配 → 转发 ZeroClawmatchself.zc_relay_mode { ZcRelayMode::WsChat => {self.relay_zeroclaw_ws_chat(&final_command, &net).await?; }// ... Webhook / OpenAI / Auto 其他模式 }Ok(())} try_local_command() 命中就直接发完成提示音,没命中才走下游 ZeroClaw 处理。
WebSocket 对接:两个 Rust 程序怎么对话
hub-server 和 ZeroClaw 之间走的是 WebSocket 而不是 HTTP,因为 ZeroClaw 处理复杂指令不是立刻就完成,它可能先想一会儿,调个工具,拿到结果再继续想,最后才给你答案。WebSocket 的双向流式能力刚好匹配这种多轮交互。
建立连接
ZeroClaw 启动后监听本地端口(默认 42617),对外暴露 ws/chat 端点。hub-server 先带上鉴权信息连上去:
fnbuild_zeroclaw_ws_url(&self) -> Result {letmut u = url::Url::parse(self.zc_ws_chat_url.trim()) .context("ZEROCLAW_WS_CHAT_URL must be a valid URL")?; {letmut q = u.query_pairs_mut();// 把 API Key 作为 query 参数带上 q.append_pair("token", self.zc_api_key.trim());// 可选:指定会话 ID,用于多轮对话上下文ifletSome(ref sid) = self.zc_ws_session_id {if !sid.trim().is_empty() { q.append_pair("session_id", sid.trim()); } } }Ok(u.to_string())} 最终 URL 长这样:ws://127.0.0.1:42617/ws/chat?token=<你的token>&session_id=<会话ID>。
这里的 token 不是 OpenAI 的 API Key,而是 ZeroClaw Gateway 自己的配对令牌。
获取方式:在 ZeroClaw 所在机器上生成一个 6 位配对码,再用配对码换 token。
发送消息
连上之后,ZeroClaw 先发一帧 session_start 表示会话就绪。hub-server 收到后才能发用户指令:
// 构造 JSON 消息let outgoing = serde_json::json!({"type": "message","content": message}).to_string();// 发送给 ZeroClawwrite.send(Message::Text(outgoing.into())).await?;消息体两个字段:type 固定 message,content 是语音识别出来的指令文本。比如"今天北京天气怎么样"。
接收响应:按事件类型分流
消息发出去之后,ZeroClaw 不会一次返回给你最终答案。它按事件类型逐帧推送,代码里根据 type 字段分别处理:
let ty = v.get("type").and_then(|x| x.as_str());match ty {// 工具调用:ZeroClaw 决定调用某个工具Some("tool_call") => {let name = v.get("name").and_then(|x| x.as_str()).unwrap_or("?"); info!("[ZeroClaw/tool] call `{}` args {}", name, v.get("args").map(|a| a.to_string()).unwrap_or_default() ); }// 工具结果:工具执行完毕,返回数据Some("tool_result") => {let name = v.get("name").and_then(|x| x.as_str()).unwrap_or("?");let out = v.get("output").map(|o| o.to_string()).unwrap_or_default(); info!("[ZeroClaw/tool] result `{}`: {}", name, out); }// 最终答案:这一轮对话的完整回复Some("done") => {let full = v.get("full_response") .and_then(|x| x.as_str()) .unwrap_or("");if full.is_empty() { warn!("[ZC] ws `done` with empty full_response"); } else { reply_text = Some(full.to_string()); } success = true;break; }// 流式中间输出(推理过程、思考链等)Some("chunk") | Some("thinking") => { /* debug 日志 */ }// 会话生命周期事件Some("session_start" | "connected" | "chunk_reset" | "agent_start" | "agent_end") => {}Some("error") => { /* 错误处理 */ } _ => {}}整个流程就是一个 Agent Loop:用户说话 → ZeroClaw 判断要不要用工具 → 调工具 → 拿结果 → 生成最终回复。每个环节是一帧独立事件。
hub-server 只关心 done 帧里的 full_response,拿到结果后执行 TTS 回传。
同时,连接的可靠性也考虑到了。WebSocket 的 Ping/Pong 心跳自动回应,整轮对话 300 秒超时兜底:
let (ok, reply_text) = match timeout(Duration::from_secs(300), turn).await {Ok(v) => v,Err(_) => { error!("[ZC] ws agent turn timed out (300s)"); (false, None) }};300 秒对正常工具调用绰绰有余。ZeroClaw 那边出问题时,超时后 hub-server 不会卡死,直接返回失败并发完成提示音给设备。
TTS 回传:让盒子开口说话
ZeroClaw 返回文本,但用户对着盒子说话,期待的是语音回复。所以 hub-server 拿到 full_response 之后要把文字转成语音,再发回给盒子播放。
服务端合成
TTS 合成支持两种后端,配置里设 auto 会自动选择。
macOS 上优先走系统自带的 say 命令,支持中文语音(Tingting 等),已经足够日常使用了,不用再装额外东西。流程是 say 输出 AIFF,再用 ffmpeg 转成 16kHz 单声道 16-bit PCM:
asyncfnsynth_mac_say(text: &str, voice: &str, rate: u32) -> Option> {// 1. 用 macOS say 命令合成语音let say = Command::new("say") .arg("-v").arg(voice) // 语音:Tingting .arg("-r").arg(rate.to_string()) // 语速:180 .arg("-o").arg(&aiff) // 输出 AIFF 格式 .arg(text) .output().await;// 2. 用 ffmpeg 转成 s16le 16kHz 单声道 PCMlet ff = Command::new("ffmpeg") .arg("-y").arg("-loglevel").arg("error") .arg("-i").arg(&aiff) .arg("-f").arg("s16le") // 16-bit PCM .arg("-ar").arg("16000") // 16kHz 采样率 .arg("-ac").arg("1") // 单声道 .arg(&pcm) .output().await;} 不在 macOS 上或者想要更好的语音质量,可以换 Piper TTS。开源的离线神经网络 TTS,多语言多音色,效果比 say 好不少,但要下载语音模型。
合成逻辑不关心后端是什么,synthesize_tts_pcm() 统一返回 PCM 字节数据,调用方只处理一种格式。
下发到设备
PCM 数据准备好之后,分片后通过 TCP 发给 ESP32。复用第二篇定义的 MSG_FEEDBACK 帧类型,每片 1024 字节:
asyncfnsend_feedback_pcm_chunks( net: &std::sync::Arc>, mono_s16le: &[u8],) -> Result<()> {const CHUNK: usize = 1024;letmut w = net.lock().await;for chunk in mono_s16le.chunks(CHUNK) {let len = (chunk.len() asu16).to_le_bytes();let header = [ crate::protocol::MAGIC_HEADER, // 0x5A crate::protocol::MSG_FEEDBACK, // 0x20 len[0], len[1], ]; w.write_all(&header).await?; w.write_all(chunk).await?; }Ok(())} 设备端收到 MSG_FEEDBACK 后把 payload 塞进播放队列。payload 长度为零就是完成提示音,播放预置的"叮"声。有数据就按 PCM 流直接播放。
ZeroClaw 的回复可能很长,讲笑话可能好几句话。考虑到语音交互的特性,回答越短越好,代码里将回复硬性截断在 180 个字符内,避免用户在漫长的合成期内误以为设备死机。
TTS 合成失败时(文本为空或者后端不可用),发空帧给设备,让它播完成提示音:
asyncfnspeak_reply_and_feedback( &self, reply_text: &str, net: &std::sync::Arc>,) -> Result<()> {let trimmed = reply_text.trim();// 空回复或 TTS 未启用,直接发完成提示音if trimmed.is_empty() || !self.enable_tts_feedback { Self::send_local_done_cue(net).await;returnOk(()); }// 合成 TTS 并下发ifletSome(pcm) = synthesize_tts_pcm(&req).await { Self::send_feedback_pcm_chunks(net, &pcm).await?; } else { Self::send_local_done_cue(net).await; }Ok(())} 配置与部署
需要配置的东西不多。hub-server 的 .env 文件:
# ZeroClaw Gateway 地址ZEROCLAW_URL=http://127.0.0.1:42617/v1# 配对后获得的 API KeyZEROCLAW_API_KEY=<你的配对Token># 转发模式:ws(WebSocket,支持工具调用)推荐ZEROCLAW_MODE=wsZEROCLAW_WS_CHAT_URL=ws://127.0.0.1:42617/ws/chat# 会话 ID:不设置则每次启动自动生成新会话# ZEROCLAW_WS_SESSION_ID=# TTS 语音反馈ENABLE_TTS_FEEDBACK=trueTTS_BACKEND=autoTTS_VOICE=TingtingTTS_RATE=180获取 ZEROCLAW_API_KEY 两步搞定。先在 ZeroClaw 机器上生成配对码,再拿配对码换 token:
# 生成配对码zeroclaw gateway get-paircode --new# 用配对码换 tokencurl -sS -X POST -H "X-Pairing-Code: <配对码>" http://127.0.0.1:42617/pair响应里有个 token 字段,填进 .env 就可以了。
避坑指南
session_id 不能随便复用
ws/chat 按 session_id 恢复历史会话上下文。固定一个 session_id,会继承原有的对话行为。
我们调试时踩过这个坑:说"查询天气",ZeroClaw 只回一句"我将帮你查询天气",不真正调工具。
日志里没有任何 [ZeroClaw/tool] call 和 [ZeroClaw/tool] result,说明这轮根本没触发工具调用。换个新 session_id,立刻恢复正常。
根因是旧会话上下文污染了模型行为。代码里已经处理了:不设 ZEROCLAW_WS_SESSION_ID 时,服务端每次启动自动生成新会话 ID。只有需要长期多轮记忆时才手动固定。
首次连接要容忍 Ping 帧
ZeroClaw 的 WebSocket 建连后,不一定会先发 session_start 文本帧,有时先来几个 Ping 保活帧。代码只等一帧的话,遇到 Ping 就会误判连接失败。
我们的做法是循环读最多 8 帧,遇到 Ping 回 Pong,遇到文本帧才算连接就绪:
letmut got_server_text = false;for _ in0..8u8 {match timeout(Duration::from_secs(10), read.next()).await {Ok(Some(Ok(Message::Ping(p)))) => {let _ = write.send(Message::Pong(p)).await; }Ok(Some(Ok(Message::Text(t)))) => { got_server_text = true;break; }// ... 其他分支 }}TTS 文本要截断
ZeroClaw 可能回很长的内容,让它讲故事或解释复杂概念时尤其明显。say 合成 500 字要好几秒,这段时间扬声器不会有声音,用户会以为出了问题。
tts_max_chars 设 180,保证 3 秒内播完。语音交互里,宁可少说两句,别让人对着沉默的盒子发呆。
done 帧的 full_response 可能为空
不是每轮都能拿到有意义的回复。ZeroClaw 内部出错、工具超时、模型拒绝请求,done 帧的 full_response 都可能是空字符串。这种情况不能拿去 TTS 合成,要直接发完成提示音:
Some("done") => {let full = v.get("full_response") .and_then(|x| x.as_str()) .unwrap_or("");if full.is_empty() { warn!("[ZC] ws `done` with empty full_response"); } else { reply_text = Some(full.to_string()); } success = true;break;}结语
从翻转锁屏到语音唤醒,从本地指令路由到 ZeroClaw 兜底,这个桌面助手从一个只能执行固定指令的遥控器,变成了能理解自然语言的终端。
整条链路就是:设备采集声音,服务端转成文字,能直接处理的指令直接处理,不能处理的就交给 ZeroClaw, ZeroClaw 处理完了再让盒子说出来。
Rust 贯穿每一层。设备端 embedded Rust 驱动硬件,服务端 Rust 做协议解析、STT 推理、WebSocket 通信和 TTS 合成。ZeroClaw 本身也是 Rust。三个 Rust 程序通过 TCP 和 WebSocket 串起来,跑通了一个端到端的语音 AI 助手。
Cyber-Hub 代码已开源,想给桌上的 ESP32 接上 AI 大脑,欢迎来 GitHub Star和Fork。
代码仓库地址:https://github.com/Doomking/rust-zhixingshe-examples/tree/main/esp32/esp32-s3/box-3/cyber-hub
关注 Rust 知行社,一起学习 Rust × 硬件/AI 的硬核实战内容。
The first two sections covered the display of the monitoring screen and voice wake-up control. The ESP32 BOX is capable of hearing sound and locking the screen to adjust the volume, but fundamentally, it is still just a remote control that can only execute a limited set of predefined commands.
This section connects the ESP32 BOX to a brain.
Once connected, if you ask "What's the weather like in Beijing today?", it will call the weather service to check the data and then read it to you. If you say "Help me search for Rust embedded development", it will open the browser and navigate to the search page. If you say "Tell me a joke", it will actually tell one, though it may not be funny.
The name of this brain is ZeroClaw.
What is ZeroClaw?
ZeroClaw is an AI Agent framework written in Rust. The binary file is 3.4 MB, uses less than 5 MB of memory, and starts up in under 10 ms. In comparison, a similar framework, OpenClaw, written in TypeScript, depends on the Node.js runtime, has a 28 MB binary plus nearly 400 MB of Node.js environment, requires over 1 GB of memory, and takes over 500 seconds to start on a 0.8 GHz edge device.
ZeroClaw is a high-performance Rust implementation of the same functions as OpenClaw: understanding natural language instructions and calling tools to execute tasks. However, it is not bound to any specific model vendor and can be deployed anywhere once compiled. It supports over 22 AI providers, including OpenAI, Anthropic, DeepSeek, and local Ollama models. It exposes standard WebSocket and HTTP interfaces, allowing any program with network capabilities to connect.
In this project, ZeroClaw is deployed on a Mac. The hub-server forwards the text from the voice recognition via WebSocket, ZeroClaw interprets the intent, calls the tools, generates a reply, and the hub-server converts this reply into speech to broadcast to the user.
Overall Architecture: Two Channels
Connecting to ZeroClaw transforms the voice command processing into a two-layer funnel.
The first layer is the local instruction router, which handles the quick responses like locking the screen, muting, and adjusting the volume. These frequent commands are executed immediately without network delays.
The second layer is ZeroClaw. Any instructions that don't match the local router are forwarded here. This layer can call external tools like weather queries, browser operations, and search engines, which introduces the latency of network communication and model inference.
The design philosophy is efficiency: waiting three seconds for a simple "mute" command is unacceptable. However, waiting a few extra seconds for a complex query like "Check tomorrow's weather in Beijing" is reasonable.
This diversion is implemented in the process_utterance() function. If try_local_command() succeeds, the completion sound is played immediately; otherwise, the command is passed to ZeroClaw for further processing.
WebSocket Connection: Inter-process Communication
The hub-server communicates with ZeroClaw via WebSocket rather than HTTP because ZeroClaw requires a dialogue to process complex instructions. It may think for a while, query a tool, receive the result, and then continue before returning a final answer. WebSocket's duplex streaming capability is perfect for this multi-turn interaction.
ZeroClaw listens on the local port (default 42617) and exposes the ws/chat endpoint. The hub-server connects with authentication information:
The final URL looks like this: ws://127.0.0.1:42617/ws/chat?token=
The token here is the pairing token for ZeroClaw Gateway, not an OpenAI API key.
Acquiring the Token: Generate a 6-digit pairing code on the ZeroClaw machine and exchange it for the token.
Sending Messages
After connecting, ZeroClaw sends a session_start frame indicating the session is ready. The hub-server can then send the user command:
The message body contains two fields: type is fixed as "message", and content is the recognized text of the voice command, such as "What's the weather like in Beijing today?".
Receiving Responses: Event-Based Parsing
ZeroClaw does not return a single answer immediately. Instead, it pushes frames based on event types, which the code parses accordingly:
The entire process follows an Agent Loop: the user speaks → ZeroClaw decides if tools are needed → tools are called → results are retrieved → the final reply is generated. Each step is represented by a distinct event frame.
The hub-server is only interested in the full_response field of the done frame, which it then uses to perform TTS for playback.
Reliability is ensured by WebSocket ping/pong heartbeats and a global timeout of 300 seconds for the conversation. If ZeroClaw fails within this window, the hub-server can gracefully return an error and play the completion sound.
TTS Playback: Making the Box Speak
Once ZeroClaw returns the textual response, the hub-server must convert it to speech so the user can hear the reply through the device.
Server-Side Synthesis
TTS synthesis supports two backends, selected automatically based on the configuration.
On macOS, the system's built-in say command is used, supporting Chinese voices (like Tingting). The audio is converted to 16 kHz mono 16-bit PCM.
Alternatively, Piper TTS can be used for higher quality. Piper is an offline neural network TTS engine that requires downloading specific voice models.
Regardless of the backend, the function returns raw PCM data, ensuring consistency in how playback is handled.
Delivering to the Device
The PCM data is segmented and sent to the ESP32 via TCP using the MSG_FEEDBACK frame type. Each segment is 1024 bytes.
If the payload is empty, the device plays a standard completion sound. If data is present, it is queued and played as audio.
ZeroClaw's responses can be lengthy — long enough to cause a perceived delay in the response. To improve the user experience, the code truncates the TTS input to the first 180 characters to ensure replies are short and succinct.
In cases of failure (empty text or backend issues), the system sends an empty frame to prompt the completion sound:
Configuration and Deployment
The setup is straightforward. The hub-server's .env file requires the following settings:
The ZEROCLAW_API_KEY is obtained in two steps: generate a pairing code on the ZeroClaw machine and use it to fetch the token.
Troubleshooting
Session ID Reuse
The session_id parameter is used to restore the context of previous conversations. Reusing a session ID can lead to unexpected behavior where the model forgets its capabilities (e.g., refusing to call tools). To avoid this, the code generates a new session ID on every server start unless explicitly configured otherwise.
Initial Connection Handling
ZeroClaw may send ping frames before the session_start frame. The code must be robust enough to handle these early pings without misinterpreting the connection status.
Text Length Limitations
As noted, responses should be limited in length to prevent excessive wait times. The tts_max_chars setting ensures that the device remains responsive.
Conclusion
This integration transforms the desktop assistant from a simple remote control into a device capable of natural language understanding. The workflow is clear: the device captures audio, the server transcribes it, local commands are executed immediately, and complex queries are delegated to ZeroClaw for a spoken response. This demonstrates the power of combining embedded devices with cloud-based AI services via well-defined protocols.
首次连接要容忍 Ping 帧
After establishing a WebSocket connection with ZeroClaw, you won't necessarily receive a session_start text frame immediately — sometimes, a few Ping frames arrive first to keep the connection alive. If the code waits for only one frame, it could misinterpret a Ping as failure.
Our approach is to read up to 8 frames in a loop: respond with Pong to any Ping, and treat only the first text frame as indicating connection readiness.
TTS 文本要截断
ZeroClaw can return very long responses, especially when asked to tell stories or explain complex concepts. Generating 500 characters of speech takes several seconds, during which the speaker remains silent — this confuses users into thinking something is wrong.
We set tts_max_chars to 180, ensuring each response finishes within 3 seconds. In voice interactions, it's better to say less clearly than to keep the user waiting in silence.
done 帧的 full_response 可能为空
Not every round yields a meaningful reply. When ZeroClaw encounters internal errors, tool timeouts, or request rejections, the done frame's full_response may be an empty string — unusable for TTS. In such cases, we directly output a completion tone (e.g., "thought" sound).
结语
This desktop assistant evolves from a simple remote control executing fixed commands to a natural language-understanding terminal. It handles tasks from flipping screen orientation to voice activation, local command routing to ZeroClaw fallback.
The full pipeline: the device captures sound, the server translates it to text, executes directly-processable commands, and passes complex queries to ZeroClaw. When ZeroClaw replies, the box synthesizes and speaks the response.
Rust drives every layer: embedded Rust for hardware control, server Rust for protocol parsing, STT/TTS inference, and WebSocket communication. ZeroClaw itself is also Rust. The three Rust components connect via TCP and WebSocket to form a complete end-to-end voice AI system.
The Cyber-Hub code is open-source — if you want to turn your ESP32 into an AI-powered desktop assistant, check it out on GitHub.
代码仓库地址:https://github.com/Doomking/rust-zhixingshe-examples/tree/main/esp32/esp32-s3/box-3/cyber-hub
Follow Rust ZhiXingShe for more hard-core tutorials on Rust × Hardware/AI.


