ElatoAI:在 Arduino ESP32 上运行实时语音 AI 模型ElatoAI: Running Real-Time Voice AI Models on Arduino ESP32


AI free token/AI免费积分

ElatoAI 让你在 ESP32 上运行由 100 多种语音 AI 模型驱动的实时语音系统,并通过安全 WebSocket 与边缘函数实现全球范围内 20 分钟以上的不间断对话。

📱 应用设计

通过 ElatoAI Web 应用,你可以直接在手机上控制自己的 ESP32 AI 设备。

图片

🌟 完整功能列表

  1. 实时语音转语音:由 OpenAI Realtime API、Gemini Live API、xAI Grok Voice Agent API、Eleven Labs Conversational AI Agents 和 Hume AI EVI4 驱动的即时语音转换。
  2. 创建自定义 AI 智能体:创建具有不同人格和声音的 AI 智能体。
  3. 可自定义语音:从多种声音和人格配置中进行选择。
  4. 安全 WebSocket:可靠且加密的 WebSocket 通信。
  5. 服务端 VAD 轮次检测:智能对话轮次处理,让交互更自然。
  6. Opus 音频压缩:以极低带宽实现高质量音频流传输。
  7. 全球边缘性能:低延迟的 Deno Edge Functions 确保全球范围内的顺畅对话。
  8. ESP32 Arduino 框架:经过优化且易于集成的硬件方案。
  9. 对话历史:查看历史对话记录。
  10. 设备管理与认证:注册并管理你的设备。
  11. 用户认证:安全的用户身份认证与授权。
  12. 通过 WebRTC 和 WebSocket 对话:在 NextJS Web 应用中用 WebRTC 与 AI 对话,在 ESP32 上通过 WebSocket 对话。
  13. 音量控制:通过 NextJS Web 应用控制 ESP32 扬声器音量。
  14. 实时转录:对话实时转录结果存储在 Supabase 数据库中。
  15. OTA 更新:支持 ESP32 固件空中更新。
  16. 通过 captive portal 管理 Wi-Fi:直接在 ESP32 设备上连接 Wi-Fi 或热点。
  17. 恢复出厂设置:通过 NextJS Web 应用对 ESP32 设备执行恢复出厂设置。
  18. 按钮和触摸支持:可以通过按钮或触摸传感器控制 ESP32 设备。
  19. 无需 PSRAM:设备无需 PSRAM 即可运行语音转语音 AI。
  20. Web 客户端 OAuth:让用户管理自己的 AI 角色和设备。
  21. 音高控制:在 NextJS Web 应用中调整 AI 声音音高,做出更卡通化的声音。
  22. 工具调用:从 ESP32 设备调用边缘函数中的工具和函数,构建完整的语音 AI 智能体。
  23. 轻触唤醒:轻触触摸板即可从休眠中唤醒。
  24. 部署到 Cloudflare:借助 Cloudflare Voice Agents 和 Durable Objects 连接任意 LLM、TTS、STT 服务。

项目架构

ElatoAI 由三个主要组件组成:

  1. 前端客户端(部署在 Vercel 上的 Next.js)- 用于创建并与 AI 智能体对话,并将其“发送”到你的 ESP32 设备
  2. 边缘服务函数(Deno Edge 或 Cloudflare Workers)- 用于处理来自 ESP32 设备的 WebSocket 连接以及对模型提供商 API 的调用
  3. ESP32 IoT 客户端(PlatformIO/Arduino)- 接收来自边缘服务函数的 WebSocket 连接,并通过 Deno 边缘服务器或 Cloudflare Durable Objects 把音频发送给模型提供商

📊 关键指标

  • 延迟:全球往返延迟小于 2 秒
  • 音频质量:12kbps Opus 编码(高清晰度)+ 24kHz 采样率
  • 不间断对话:全球范围内最长可达 20 分钟连续对话
  • 全球可用性:通过边缘计算优化

🛡 安全性

  • 使用安全 WebSocket(WSS)进行加密数据传输
  • 可选:使用 256 位 AES 加密 API Key
  • 使用 Supabase DB 进行安全认证
  • 所有表均采用 Postgres RLS

🚫 限制

  • 连接边缘服务器时有 3-4 秒冷启动时间
  • 已测试最长连续对话约为 17 分钟
  • 超过 wall clock time 后边缘服务器会停止
  • ESP32 上尚未支持语音打断检测

Github:https://github.com/akdeb/ElatoAI

 

ElatoAI lets you run a real-time voice system powered by over 100 voice AI models on ESP32, enabling uninterrupted conversations of 20+ minutes globally via secure WebSocket with edge functions.

📱 App Design

Via the ElatoAI web app, you can directly control your ESP32 AI device from your phone.

图片

🌟 Full Feature List

Real-time voice-to-voice: Instant voice conversion powered by OpenAI Realtime API, Gemini Live API, xAI Grok Voice Agent API, Eleven Labs Conversational AI Agents, and Hume AI EVI4.

Create custom AI agents: Build AI agents with different personalities and voices.

Customizable voices: Choose from a variety of voice and personality configurations.

Secure WebSocket: Reliable and encrypted WebSocket communication.

Server-side VAD turn detection: Smart conversation turn handling for more natural interactions.

Opus audio compression: High-quality audio streaming at extremely low bandwidth.

Global edge performance: Low-latency Deno Edge Functions ensure smooth conversations worldwide.

ESP32 Arduino framework: Optimized and easy-to-integrate hardware solution.

Conversation history: View past conversation records.

Device management and authentication: Register and manage your devices.

User authentication: Secure user identity verification and authorization.

WebRTC and WebSocket conversations: Use WebRTC to converse with AI in the NextJS web app, and WebSocket on the ESP32.

Volume control: Control the ESP32 speaker volume through the NextJS web app.

Real-time transcription: Conversation transcription results are stored in a Supabase database.

OTA updates: Over-the-air firmware updates supported for ESP32.

Wi-Fi management via captive portal: Connect to Wi-Fi or hotspot directly on the ESP32 device.

Factory reset: Perform a factory reset on the ESP32 device via the NextJS web app.

Button and touch support: Control the ESP32 device via buttons or touch sensors.

No PSRAM required: The device runs voice-to-voice AI without PSRAM.

Web client OAuth: Let users manage their own AI personas and devices.

Pitch control: Adjust the AI voice pitch in the NextJS web app for more cartoon-like voices.

Tool calling: Call tools and functions from edge functions on the ESP32 device to build full-featured voice AI agents.

Touch wakeup: Lightly tap the touchpad to wake from sleep.

Deploy to Cloudflare: Connect any LLM, TTS, or STT service using Cloudflare Voice Agents and Durable Objects.

Project Architecture

ElatoAI consists of three main components:

Frontend client (Next.js deployed on Vercel) — for creating and conversing with AI agents and "sending" them to your ESP32 device

Edge service functions (Deno Edge or Cloudflare Workers) — for handling WebSocket connections from ESP32 devices and API calls to model providers

ESP32 IoT client (PlatformIO/Arduino) — receives WebSocket connections from edge service functions and sends audio to model providers via Deno edge servers or Cloudflare Durable Objects

📊 Key Metrics

Latency: Round-trip latency under 2 seconds globally

Audio quality: 12kbps Opus encoding (HD) + 24kHz sample rate

Uninterrupted conversations: Up to 20 minutes of continuous conversation globally

Global availability: Optimized through edge computing

🛡 Security

Encrypted data transmission using secure WebSocket (WSS)

Optional: 256-bit AES encryption for API keys

Secure authentication using Supabase DB

All tables use Postgres RLS

🚫 Limitations

3–4 second cold start time when connecting to edge servers

Maximum tested continuous conversation is approximately 17 minutes

Edge server will stop after wall clock time expires

Voice interrupt detection not yet supported on ESP32

Github: https://github.com/akdeb/ElatoAI

暂无评论,快来发表第一条评论吧!No comments yet. Be the first!

📮 需求咨询📮 Inquiry