Developer Henry has open-sourced ESP32 iPhone Agent, a project that uses a XIAO ESP32S3 development board costing about RMB 40 to 50 and a mobile app to let AI control a real iPhone. It can open other apps, tap buttons, and swipe across the screen.
No jailbreak and no XCTest access required
The setup does not require a jailbreak, and it does not rely on Apple XCTest testing permissions. Instead, the ESP32 creates its own Wi-Fi hotspot and serves as an AirPlay screen-mirroring receiver. The iPhone sends its display to the board, but the board does not decode the stream. It sends compressed video back over Bluetooth to the Agent app running on the same phone.
Vision recognition and input emulation
The Agent app calls a vision model to identify what is on screen and decide the next step. It then sends commands over Bluetooth, allowing the ESP32 to emulate a USB mouse and keyboard to operate the phone. To keep tap positions accurate, the system also calibrates the mouse pointer automatically and can correct the position based on the screen image when needed.
This design combines screen reception and input emulation on a standard ESP32-S3, without requiring dedicated video capture hardware such as NanoKVM Go. The Agent runs directly on the iPhone, while model inference depends on an external API or a self-hosted server. Because the phone’s Wi-Fi is occupied by the board’s hotspot, model access uses cellular data by default.
Current limitations
Initial firmware flashing, along with compiling and installing the iPhone app, still requires a Mac and Xcode. Bluetooth bandwidth limits the returned video feed to 480×1040 resolution at up to 5 frames per second, and each operation usually takes a few seconds.
At this stage, touch coordinates only work with portrait interfaces, keyboard input only supports ASCII characters, and screen mirroring stops once the phone is locked.