这款 ESP32 让 AI 操控真实 iPhoneThis ESP32 Lets AI Control A Real iPhone


AI free token/AI免费积分
这款 ESP32 让 AI 操控真实 iPhone

这款 ESP32 让 AI 操控真实 iPhone

这款 ESP32 让 AI 控制一部真实的 iPhone

一个开源项目利用低成本的ESP32-S3开发板,让AI agent无需越狱即可点击、滑动和导航iPhone。

一款低成本的 ESP32-S3 开发板可以让一个人工智能(AI)代理操作一部真实的 iPhone,根据自然语言指令点击按钮、导航菜单并输入文本。名为 ESP32 iPhone Agent 的开源项目结合屏幕镜像、USB 鼠标和键盘接口以及视觉模型,以在无需在手机上安装应用或进行越狱的情况下自动化任务。

该项目使用一块 Seeed Studio XIAO ESP32S3 开发板,其开发者估价约为 US$7。然而,AI 模型并未在微控制器上运行。相反,由一台独立的计算机处理手机屏幕画面、规划操作,并将指令发送至 ESP32-S3,再由 ESP32-S3 通过 USB 连接来执行。

该方法为依赖于 Apple 开发工具或设备上已安装的应用程序的移动自动化方法提供了一种替代方案。该项目基于 MIT licence 提供,允许开发者在许可证条款下检视、修改并再分发其代码。

硬件设置需要一块具备原生 USB 支持的 ESP32-S3 development board、一部 iPhone,以及一台运行 macOS 或 Linux 的计算机。配备 USB-C 接口的 iPhone 可直接连接,而配备 Lightning 接口的机型则需要使用 Lightning to USB Camera Adapter。A USB-C hub with power pass-through 可以在运行过程中保持手机充电。

计算机必须运行代理的服务器软件和 AirPlay 接收器,且手机和计算机需连接到同一网络。用户还需要通过 OpenAI-compatible API 访问视觉模型,可以是来自云服务或本地托管的模型。尽管开发板价格低廉,但总体成本取决于计算机、任何额外硬件以及所选的 AI 服务。

该系统通过结合屏幕捕获、AI决策和物理输入来运行。iPhone 通过 AirPlay 将其显示屏画面镜像到 UxPlay,这是运行在计算机上的一款开源 AirPlay 接收器。一个 GStreamer 流水线将接收到的视频流转换为供视觉模型分析的 JPEG 图像。

该软件将任务在 planner 和 actor 之间分配。planner 制定一系列步骤以完成所请求的任务,而 actor 则查看最新的截图并选择下一个动作。系统可以为这两个角色使用同一个模型,也可以配置不同的模型。

ESP32-S3 充当代理的“手”。通过 USB 连接到 iPhone 时,它会呈现为标准人机接口设备(HID)——充当鼠标和键盘。Apple 的无障碍特性 AssistiveTouch 使手机显示一个指针,该指针可移动并点击,用于与屏幕上的控件进行交互。

计算机通过 Wi-Fi 向开发板发送指令。由于鼠标移动是基于相对位置而非绝对屏幕坐标,服务器会跟踪指针位置并在点击前校正对准。随后,它会检查屏幕以确定操作是否产生了预期结果,从而帮助 Agent 识别漏触并从错误中恢复。

构建系统时需要使用 arduino-cli 以及 ESP32 Arduino 核心库的 3.x 版本来烧录固件。开发者在运行代码仓库硬件目录中提供的烧录脚本之前,必须在固件设置中配置 Wi-Fi 凭证和计算机的 IP 地址。

在计算机上必须安装 UxPlay 和所需的 GStreamer 组件。服务器软件在 Python 虚拟环境中进行安装配置,并通过 .env 文件配置模型详细信息和 API 凭据。服务器启动后,用户可以通过浏览器访问默认端口为 8890 的网页界面。

iPhone 要求在辅助功能设置下启用 AssistiveTouch。连接开发板后,用户从控制中心屏幕镜像菜单中选择名为 iPhone Agent 的接收器。一个校准步骤将指针与镜像显示屏对齐。然后可以通过 Web 界面输入任务,并提供手动控制和紧急停止功能来帮助监督操作。

固件与服务器代码已在项目的 GitHub 仓库中以 MIT licence 发布。模型配置支持 OpenAI-compatible endpoints,包括使用 vLLM、SGLang、llama.cpp 和 Ollama 的兼容部署,前提是所选模型支持图像输入及所需 API。这使开发者能够试验基于云的推理或本地模型托管。

据作者介绍,在使用两台 NVIDIA DGX Spark 系统本地部署 NVIDIA Qwen3.8-Flash-Next-NVFP4 视觉模型(禁用模型的思考模式)的环境下,每个 Action 的耗时约为 1.3 秒,每个 Planning step 的耗时约为 2 秒。上述结果仅适用于该特定配置,不应视作通用的性能数据。该项目未公布整体任务成功率。

该系统有几个限制。iOS 不会镜像某些屏幕,包括密码输入、Apple Pay 和 DRM保护的视频,这会阻止代理看到或与它们进行交互。屏幕镜像在手机锁定时也会停止,因此会话期间需要禁用自动锁定。键盘输入目前仅限于 ASCII 字符,这限制了印地语和其他印度文字的直接文本输入。也不支持横屏应用程序,更改 AssistiveTouch 跟踪速度需要重新校准。

本文来源:https://lite.dailyadvent.com/detail/7f627306261015en_ng
若有版权问题,请来信告知:13951431913#139.com 或在右方提交评论。

This ESP32 Lets AI Control A Real iPhone

An open-source project uses a low-cost ESP32-S3 board to let an AI agent tap, swipe and navigate an iPhone without a jailbreak.

A low-cost ESP32-S3 development board can enable an artificial intelligence (AI) agent to operate a real iPhone, tapping buttons, navigating menus and entering text based on natural-language instructions. The open-source project, called ESP32 iPhone Agent, combines screen mirroring, a USB mouse-and-keyboard interface and a vision model to automate tasks without installing an app on the phone or jailbreaking it.

The project uses a Seeed Studio XIAO ESP32S3 board, which its author prices at about US$7. However, the AI model does not run on the microcontroller. Instead, a separate computer processes the phone's screen, plans actions and sends commands to the ESP32-S3, which executes them through a USB connection.

The approach offers an alternative to mobile automation methods that depend on Apple's development tools or applications installed on the device. The project is available under the MIT licence, allowing developers to inspect, modify and redistribute its code under the licence's terms.

The hardware setup requires an ESP32-S3 development board with native USB support, an iPhone and a computer running macOS or Linux. USB-C iPhones can connect directly, while Lightning-equipped models require a Lightning to USB Camera Adapter. A USB-C hub with power pass-through can help keep the phone charging during operation.

The computer must run the agent's server software and an AirPlay receiver, with both the phone and computer connected to the same network. Users also need access to a vision model through an OpenAI-compatible API, either from a cloud service or a locally hosted model. Although the development board is inexpensive, the overall cost depends on the computer, any additional hardware and the chosen AI service.

The system works by combining screen capture, AI decision-making and physical input. The iPhone mirrors its display over AirPlay to UxPlay, an open-source AirPlay receiver running on the computer. A GStreamer pipeline converts the incoming video stream into JPEG images that the vision model can analyse.

The software divides the task between a planner and an actor. The planner develops a sequence of steps to complete the requested task, while the actor examines the latest screenshot and selects the next action. The system can use the same model for both roles or configure separate models.

The ESP32-S3 acts as the agent's hands. Connected to the iPhone over USB, it presents itself as a standard human interface device (HID), functioning as a mouse and keyboard. Apple's AssistiveTouch accessibility feature enables the phone to display a pointer that can be moved and clicked to interact with on-screen controls.

The computer sends commands to the board over Wi-Fi. Because mouse movement is relative rather than based on absolute screen coordinates, the server tracks the pointer's position and corrects alignment before clicking. It then checks the screen to determine whether the action produced the expected result, helping the agent identify missed taps and recover from errors.

Building the system involves flashing the firmware using arduino-cli and version 3.x of the ESP32 Arduino core. Developers must configure the Wi-Fi credentials and the computer's IP address in the firmware settings before running the flashing script provided in the repository's hardware directory.

On the computer, UxPlay and the required GStreamer components must be installed. The server software is set up in a Python virtual environment, with model details and API credentials configured through a .env file. Once the server starts, users can access the web interface through a browser on port 8890 by default.

The iPhone requires AssistiveTouch to be enabled under its accessibility settings. After connecting the board, users select the receiver named iPhone Agent from the Screen Mirroring menu in Control Centre. A calibration step aligns the pointer with the mirrored display. Tasks can then be entered through the web interface, with manual controls and an emergency stop available to help supervise operation.

The firmware and server code are available in the project's GitHub repository under the MIT licence. Model configuration supports OpenAI-compatible endpoints, including compatible deployments using vLLM, SGLang, llama.cpp and Ollama, provided the selected model supports image inputs and the required API. This allows developers to experiment with cloud-based inference or local model hosting.

According to the author, a local setup using NVIDIA's Qwen3.8-Flash-Next-NVFP4 vision model across two NVIDIA DGX Spark systems achieved approximately 1.3 seconds per action and two seconds per planning step, with the model's thinking mode disabled. These results apply to that specific configuration and should not be treated as typical performance figures. The project does not report an overall task success rate.

The system has several limitations. iOS does not mirror certain screens, including passcode entry, Apple Pay and DRM-protected video, preventing the agent from seeing or interacting with them. Screen mirroring also stops when the phone locks, requiring automatic locking to be disabled during a session. Keyboard input is currently limited to ASCII characters, restricting direct text entry in Hindi and other Indian scripts. Landscape applications are also unsupported, and changing the AssistiveTouch tracking speed requires recalibration.

暂无评论,快来发表第一条评论吧!No comments yet. Be the first!

📮 需求咨询📮 Inquiry