ESP32 iPhone Agent goes open-source, turning a low-cost ESP32 board into an AI controller for iPhone

Developer Henry has open-sourced ESP32 iPhone Agent, a project that uses a XIAO ESP32S3 development board priced at roughly RMB 40 to 50 together with a mobile app to let AI operate a real iPhone. The setup can open apps, tap buttons, and swipe the screen without requiring a jailbreak or Apple XCTest testing privileges. Instead, the ESP32 creates its own Wi-Fi hotspot and acts as an AirPlay receiver. The iPhone mirrors its display to the board, which then sends compressed video back to an Agent app on the same phone over Bluetooth rather than decoding the stream itself. The app uses a vision model to read the interface, decides the next action, and sends commands back over Bluetooth so the ESP32 can emulate a USB mouse and keyboard. The system also calibrates the mouse pointer to reduce tap-position drift. The project runs on a standard ESP32-S3 and does not require dedicated video capture hardware such as NanoKVM Go, though it still has setup requirements and several technical limits.