ESP32 iPhone Agent

Turn a seven-dollar ESP32-S3 into a device that lets an AI actually use a real iPhone. The board opens its own Wi-Fi hotspot and shows up in the iOS Screen Mirroring list as ESP32 Agent; it keeps the H.264 stream without decoding it on-device and forwards it over BLE to an app running on the same iPhone. The app decodes the frames, hands screenshots to any OpenAI-compatible vision model, and runs a planner plus an actor that emits tap, swipe, draw, type, key press, home and finish actions. The same board is also a standard USB HID mouse and keyboard: with AssistiveTouch on, its relative pointer makes a click a tap and a drag a swipe, and the app re-homes the pointer in a corner and wiggles it before precise taps to correct drift, waiting for the screen to settle before reporting back to the model. No computer, no jailbreak, and no paid developer account required (a free Apple ID signature must be renewed every 7 days; one-tap hotspot join needs the paid-account Hotspot Configuration capability, otherwise the app shows the password to join manually). Requirements and limits: the AirPlay source must be an iPhone (macOS mirroring is unsupported), iOS 18 or later, BLE carries roughly 300-500 kbps so the mirror is about 480x1040 at up to 5 fps, locking the phone stops mirroring, passcode entry, Apple Pay and DRM video are not mirrored, the board hotspot has no internet so the model is reached over cellular, keyboard input is ASCII only and landscape apps are not supported yet. License: MIT, except the AirPlay code ported from UxPlay, so the firmware binary that links it ships under GPL-3.0.

Project Info

Author Info

Henry Yuan

View Profile

Related Links

Keep exploring