Control your iPhonewith AI agents

Features

• Screenshots over USB, PNG or JPEG, about 0.1s each once the capture is open
• Tap, swipe, key, type through a Bluetooth mouse and keyboard served by the Mac
• Screen points by default: screenshots, OCR boxes and taps share one small coordinate space, sized for LLM vision
• Screen text with on-device OCR (Vision), boxes in screenshot coordinates
• Unlock a locked iPhone with its passcode, no touch; lock state in list
• Lock over the Bluetooth keyboard, verified over USB
• Several iPhones at once, each controlled on its own
• Leases: one agent per iPhone at a time; others wait or get a clear "leased by" error
• HTTP API described by OpenAPI, usable from any language
• Remote daemon over TCP with a bearer token and HTTPS

Install

123npx ghosthand list        # run without installing
bunx ghosthand list       # same, with Bun
npm install -g ghosthand  # install the `ghosthand` command

Setup (once per iPhone)

1. Plug in and trust

1ghosthand list

2. Pair the Mac as a mouse and keyboard

1ghosthand key home

• The Mac is discoverable only while a command waits for pairing.
• After Pair, the iPhone opens the connection itself. The command continues by itself.
• Later runs reconnect the paired iPhone from the Mac in about 2 s.
• After a report map change, forget the Mac on the iPhone and pair again: the iPhone keeps the map it read at pairing.
• Pairing fails, or the Mac is gone from the list: forget on both sides, then pair again. See Pairing no longer works.

3. AssistiveTouch (automatic)

12ghosthand assistivetouch      # on / off
ghosthand assistivetouch off  # give the phone back to a person; next tap turns it on

1ghosthand key home  # Home Screen (HID consumer Menu key; tested on iOS 15.8)

Use it with an agent

1npx -y skills add https://ghosthand.dev

1Use ghosthand to open Messages on my iPhone and reply "on my way" to Anna.

123ghosthand screentext
ghosthand tap --text "Anna" --screenshot s.png --screentext
ghosthand type "on my way" --enter --screentext

Agent feedback loop

      ┌─────────────────────────────────────────────────────────────────┐
      v                                                                 │
 1. see the screen           screentext (text + boxes, no image)        │
                             or screenshot -o s.png (~500 tokens)       │
 2. act, and see the result  tap / swipe / key / type                   │
                             + --screenshot s.png --screentext          │
 3. read the printed text, and s.png if the text is not enough          │
 4. decide the next action toward the user's task  ─────────────────────┘

123456export GHOSTHAND_LANGUAGE=it-IT                        # phone language for OCR, same as --language
ghosthand list
ghosthand screentext
ghosthand tap --text "Tommaso De Rossi" --screenshot s.png --screentext
ghosthand tap 200 610 --screenshot s.png --screentext  # x,y read from s.png
ghosthand type "ciao" --enter --screentext

• Text first, image when needed. --screentext output is cheap and often enough to decide (a new title, a sent message). Read s.png for icons, layout, or text OCR misses.
• Both flags use one frame: the daemon waits for a still screen (up to 2 s after frames start). When both flags are set, the CLI gets a native PNG, runs OCR on it, and resizes that same image for saving. No second capture or public frame ID. A blank or unchanged image is not proof that input failed.
• Check focus before you type. type sends keys to the field that has focus, and you cannot see which one from the command. Read a screenshot first: the cursor must be in the field you want. In Messages, Enter in the To: field adds the recipient but keeps focus there, so type "ciao" --enter put "ciao" into To: (tested on iOS 18.7). Tap the message field, check the screenshot, then type.

  123ghosthand type "3336052139" --enter         # To: field, adds the recipient
  ghosthand tap 200 562 --screenshot s.png    # check focus in the image
  ghosthand type "ciao" --enter --screentext  # Enter sends

• Tap text, not pixels: tap --text "..." saves a screenshot read and coordinate guessing. On the Home Screen, app labels do not open apps (tested on iOS 15): tap the icon, about 40 points above the label.
• No clever loops. Do not script loops that wait for a text you guess ("Open", "Done"). The next screen is hard to predict: iOS shows sheets and alerts at any time (the App Store asks to confirm Install, then can say "requires iOS 16 or later"), and a loop that waits for one word waits forever (this happened while installing Reddit on the 6s). Use a loop only when a skill or document you follow asks for it explicitly. Otherwise use the plain feedback loop: one action, read the whole screen, decide the next action.
• Failed capture after an action: the error says action completed. Do not repeat the action blindly; take a screenshot first.
• No frames: screenshot and screentext reject a locked iPhone quickly. If list reports unlocked but capture still gets no frame, see USB screen capture stops.

1ghosthand tap 207 100 --screenshot s.png --screentext   # empty top area: full screen returns

CLI

Secret text from stdin

1secret-provider | ghosthand type --stdin

Mute

12ghosthand mute  # muted iPhone XR Tommy
ghosthand unmute

Mac ──BT Mute key──> SpringBoard ──FullMute──> audiomxd
 ^                       │
 └──USB syslog_relay─────┘  "Updated fullyMuted to true"

Bounded text verification

1ghosthand tap --text "General" --wait-text "About" --timeout-ms 5000

Leases: one agent per iPhone

123ghosthand acquire -d 7EF219AD-69B4-423E-B8C4-4F4A6140822F --label ses_123 --reason "reply to Anna"  # prints 42
ghosthand tap -l 42 207 448    # runs on the lease's iPhone; no -d
ghosthand release -l 42

agent A: acquire ──> lease 42 ──> tap -l 42, type -l 42 ────────> release -l 42
agent B:       acquire ──> error: in use by "ses_123" for "reply to Anna":
                           acquired 2m3s ago, last command 4s ago, ends after 9m56s idle
agent B:       decides: give up / tell the user, or
               wait --reason "..." --timeout 600 ─────── waits ──────> lease 43

• Always pass --reason with the task. A busy acquire fails at once and shows the label, its reason, when it was acquired and its last command, so the next agent can decide to give up instead of waiting.
• wait is a separate command on purpose: an agent must read who holds the phone before it waits. It retries every second (no queue order). On timeout it shows the current label and reason.
• Idle timeout: a lease ends 10 minutes after its last command (--idle-timeout), so a crashed agent never blocks the phone.
• --force takes it from a lease whose holder is gone; that holder's next command fails with lease-invalid.
• After wait or --force, take a screenshot first. Someone else used the phone.
• Pass the ID explicitly with -l. Tool-call environments may not persist; do not rely on exporting GHOSTHAND_LEASE.
• IDs increase in SQLite and are not reused after expiry, force, release or a daemon restart. Do not delete or restore state.db.
• --label is only display text, e.g. your agent session id, so others can find who holds the phone. It is not checked, not unique, and an acquire with the same label does not resume an active lease.
• Lease numbers are coordination IDs, not authentication. TCP access still requires the daemon bearer token. Anyone with daemon access can acquire or force leases.
• Leases live in the lease table of state.db, so they survive a daemon restart (a rebuild restarts it).

Coordinates: screen points

12ghosthand screenshot -o full.png --scale 1  # native pixels, 828x1792
ghosthand tap 414 896 --scale 1             # native pixel coordinates

Why this instead of Appium?

1. USB: connect the iPhone, unlock it, and accept Trust This Computer.
2. Bluetooth: pair the Mac as a mouse and keyboard from the iPhone's Settings.

No developer account or weekly signing renewal

Reliability

Alternatives

More iPhones

12345678910ghosthand list
# 7EF219AD-...  Morse's iPhone  iPhone XR (iPhone11,8) iOS 18.7  (bluetooth connected, absolute pointer)
# 42B874EC-...  Test iPhone     iPhone 6s (iPhone8,1) iOS 15.8.4  (bluetooth connected, relative pointer)

ghosthand acquire -d 7EF219AD-69B4-423E-B8C4-4F4A6140822F  # prints e.g. 42
ghosthand acquire -d "Test iPhone"                         # prints e.g. 43
ghosthand screenshot -l 42 -o a.png
ghosthand tap -l 43 206 332
ghosthand release -l 42
ghosthand release -l 43

capture id 7EF219AD-... ── saved once ──> UDID 00008020-... ──> lock state, BT address
   first match: the only new iPhone on USB, or else the only one with that name

12ghosthand rename "iPhone XR"  # on the lease's iPhone
# renamed "iPhone" to "iPhone XR". The id stays 7EF219AD-69B4-423E-B8C4-4F4A6140822F

• list and acquire -d "iPhone XR" use the new name at once. The Bluetooth link stays connected (tested on iOS 18.7).
• macOS screen capture (QuickTime, daemon logs) keeps the old name until you unplug and plug in the cable.

How many iPhones per Mac

Several Macs: give each a unique name

123# System Settings > General > About > Name, or:
sudo scutil --set ComputerName "Lab Mac 1"
sudo scutil --set LocalHostName "lab-mac-1"   # optional: network name, no spaces

• The Bluetooth radio uses the computer name. There is one name per Mac; ghosthand cannot show its own name next to it.
• The pairing steps print the name to tap: tap "Lab Mac 1", then Pair.
• An iPhone that paired before the rename can keep showing the old name. Forget the Mac on the iPhone (Settings > Bluetooth > (i) > Forget This Device) and pair again.
• Apple guide: change your Mac's name.

Devices in list

127EF219AD-69B4-423E-B8C4-4F4A6140822F  Morse’s iPhone (2)  iPhone XR (iPhone11,8) iOS 18.7.10  (bluetooth connected, absolute pointer)
   USB capture ID                        name                model and iOS (USB)                   Bluetooth link, pointer mode

12error: No iPhone in macOS screen capture. USB check:
- iPhone di Carla (2): on USB but not trusted (AMDeviceStartSession 0xe8000025). Unlock it and tap Trust on the iPhone; ...

Unlock

click ──> screen on ──> two fast swipes up ──> check USB lock state for 4 s
                                                 ├─ unlocked by Face ID ──> done
                                                 └─ still locked
                                                      └──> type passcode once
                                                             └──> check lock state

• Keys cannot wake the screen. iOS treats our keys as coming from a mouse and drops them on the lock screen. A click usually wakes it.
• Face ID runs first. The first swipe can show notifications; the second opens the passcode flow. unlock polls the USB lock state for up to 4 s, returns if Face ID already unlocked the phone, then types the passcode once if it is still locked. It cannot confirm which screen is shown before typing.
• Lock state comes over USB from lockdown (PasswordProtected) through the private MobileDevice.framework. list shows [locked]; unlock checks it before and after.
• A wrong passcode is never retried. Failed tries count toward the iPhone lockout. The passcode is not stored.
• Locked screenshots fail immediately. USB capture does not return frames from this phone while locked; screenshot and screentext check the lock state first.
• OCR is for the agent loop after unlock, not for the locked-screen sequence. For example, screentext can find a button label and its pixel box; the agent chooses the next tap based on that result.

HTTP API

12345678910S=~/.ghosthand/daemon.sock
curl --unix-socket $S http://localhost/v1/devices
curl --unix-socket $S -H 'content-type: application/json' \
  -d '{"device":"default","label":"curl","reason":"reply to Anna"}' http://localhost/v1/leases   # {"id":"42",…}
L=http://localhost/v1/leases/42
curl --unix-socket $S $L/screenshot -o s.png
curl --unix-socket $S -H 'content-type: application/json' \
  -d '{"x":207,"y":448}' $L/tap                    # screen points, like the screenshot
curl --unix-socket $S $L/assistive-touch            # {"enabled":true}
curl --unix-socket $S -X DELETE $L                  # release

• Screenshots and input live under /v1/leases/{lease}/ and run on the lease's iPhone. Unknown, expired, released or taken with force: lease-invalid (403). Acquire while held: device-leased (409); the detail shows the label, reason, acquire time and last call. Retry to wait.
• Input routes return JSON. Send Accept: text/event-stream to get progress as server-sent events.
• Errors are application/problem+json with a stable code. In event-stream mode the status is 200 and the problem arrives as the final error event.

Remote daemon

123456# on the Mac with the iPhones
GHOSTHAND_TOKEN=secret ghosthand daemon start --port 7420 --host 0.0.0.0 \
  --tls-cert cert.pem --tls-key key.pem

# anywhere else
ghosthand --url https://mac.local:7420 --token secret --ca cert.pem list

Security

who can reach the phone              how
local user who owns ~/.ghosthand ──> unix socket, file mode 0600
remote client ─────────────────────> TCP with --port: Bearer token, TLS off localhost

1. USB cable plugged in, and the phone trusts this Mac (Trust prompt + passcode).
2. Bluetooth pairing accepted on the phone in Settings > Bluetooth.
3. The daemon running.

• The passcode is never stored. unlock gets it per call; prefer a direct pipe to unlock --stdin, not an argument.
• Apple Pay still needs you. It asks for a double press of the side button, which the Mac cannot press.

Limits

• AssistiveTouch must be on for tap and swipe. ghosthand turns it on over USB; its floating button can be hidden (see AssistiveTouch).
• Keyboard without AssistiveTouch is not tested.
• cmd+space and cmd+h did nothing in tests.
• No verified global Back command. key home goes Home; key esc sends Escape. Neither is a promise of Back in Settings or Safari. AC Back (Consumer usage 0x0224) is not exposed because reliable iOS behavior has not been verified. Descriptors are unchanged.
• Screen off: no frames. unlock tries to wake a locked iPhone; USB capture may still return no frame. An unlocked one with the screen off needs a tap from you.
• Several new iPhones with one name, plugged in together, cannot be matched to their USB data. Plug them in one at a time once (see More iPhones).
• Many fast daemon restarts can hide the iPhones for about a minute. macOS throttles the screen capture helper.
• About 7 iPhones per Mac, set by Bluetooth, not by ghosthand. Only 2 tested. See How many iPhones per Mac.
• The cable must stay in. Screenshots only come over USB; a phone on Bluetooth or Wi-Fi alone is not listed.
• Typing is US-ASCII only: no accents or emoji. There is no length limit.

Pairing no longer works: forget on both sides

1. iPhone: Settings > Bluetooth > the Mac > (i) > Forget This Device.
2. Mac: ghosthand forget (with a lease on that iPhone). It removes the Mac's bond (same as System Settings > Bluetooth > the iPhone > (i) > Forget This Device).
3. Run any input command, for example ghosthand key home. It makes the Mac discoverable and waits.
4. iPhone: Settings > Bluetooth, tap the Mac under Other Devices, then Pair.

USB screen capture stops (iOS bug): restart the daemon, then the iPhone

1error: No frame from Morse’s iPhone (2) after 8s (session running, 0 frames since open). ...

• First try ghosthand daemon restart. It fixed a stuck capture that failed with Recording Stopped (AVFoundationErrorDomain -11808) (iOS 18.7).
• If frames still do not arrive, restart the iPhone. Replugging the cable does not help, and in some cases the daemon restart did not help either (tested on iOS 18.7).
• This is an iOS bug, not ghosthand. A plain AVFoundation program gets no frames either, and QuickTime Player crashed when it opened the same iPhone screen.
• Known Apple reports with the same symptom: FB21608780 (after "Stop Mirroring" in Control Center, fixed only by a reboot) and FB22281473 (screen capture trust revoked).
• After the restart, unlock the iPhone by hand once. Before the first unlock, iOS ignores the Bluetooth keyboard, so ghosthand unlock fails.

1. ghosthand daemon stop
2. QuickTime Player > File > New Movie Recording
3. Click the ⌄ arrow next to the record button. Under Camera:

• Screen entry shows the live screen: capture works; the problem is on the Mac side.
• Screen entry is black, missing, or QuickTime crashes: iOS stopped serving the screen. Restart the iPhone.
• Two iPhones with the same name both show up as ... Camera. Only the one on USB has a screen entry.

How the Mac becomes a mouse

Mac (daemon)                                          iPhone
┌──────────────────────────────────┐                 ┌─────────────────────────────┐
│ SDP record: HID, report map      │ <──SDP query─── │ Settings > Bluetooth > Pair │
│ L2CAP 0x11 control  (handshakes) │ <──opens─────── │                             │
│ L2CAP 0x13 interrupt             │ ──reports─────> │ AssistiveTouch              │
│  report 1: mouse (absolute)      │                 │  pointer + clicks = touches │
│  report 7: mouse (relative)      │                 │                             │
│  report 2: keyboard, 6: media    │                 │                             │
└──────────────────────────────────┘                 └─────────────────────────────┘

check:  corner ──> move (80,120) pt ──> screenshot P1
        corner ──> move (160,120) pt ──> screenshot P2
        pointer found in P1 and P2 ──> gain = measured / computed (saved in state.db)
tap:    computed reports from the last known spot ──> 100 ms pause ──> click

How a phone is found

USB lockdown  BluetoothAddress ──── EC:CE:D7:A4:9B:C1 ────> Bluetooth HID link

State database

phone (USB)                                  bluetooth_pointer (Bluetooth)
┌───────────────────────────────────┐        ┌──────────────────────────────────────┐
│ usb_udid           PK             │<───────│ usb_udid   PK, FK, ON DELETE CASCADE │
│ screen_capture_id  UNIQUE         │        │ mode       absolute | relative       │
│ capture_matched_by / _at          │        │ gain, screen_width_pt, _height_pt    │
└───────────────────────────────────┘        │ report_map, found_at                 │
                                             └──────────────────────────────────────┘

12345sqlite3 -header -column ~/.ghosthand/state.db 'SELECT * FROM phone; SELECT * FROM bluetooth_pointer'
# usb_udid                   screen_capture_id                     capture_matched_by  capture_matched_at
# 00008020-000405C214BB002E  7EF219AD-69B4-423E-B8C4-4F4A6140822F  only-new-phone      1790773434
# usb_udid                   mode      gain  screen_width_pt  screen_height_pt  report_map                found_at
# 00008020-000405C214BB002E  absolute                                           classic-a85efc997e025d8f  1790773436

• Not stored, because USB lockdown gives them on every command: name, Bluetooth address, lock state, AssistiveTouch.
• Bluetooth pairing is not here. macOS keeps the bond. A phone paired while the daemon was stopped, or with no state.db, reconnects on the next command (tested).
• A saved match is never changed by itself, even while its phone is missing from one list during a plug-in. If one is wrong, delete its phone row (below).
• Schema changes preserve the lease counter. Cache tables can be recreated, but IDs must keep increasing.

123DB=~/.ghosthand/state.db
sqlite3 $DB "PRAGMA foreign_keys = ON; DELETE FROM phone WHERE usb_udid = '00008020-000405C214BB002E'"   # one iPhone
sqlite3 $DB 'DELETE FROM bluetooth_pointer'                                                             # all pointer modes

• Screen from one phone, input to another: delete its phone row. Plug in same-named new phones one at a time.
• Pointer lands in the wrong spot: delete its bluetooth_pointer row. ghosthand forget also does it, but it removes the Bluetooth bond too, so you must pair again.
• Never delete or restore state.db. Resetting its counter can give a new lease an old ID.

iOS versions

• The mode is found by itself on the first tap: iOS 17 and later use absolute; older versions are checked by moving the pointer. list shows it: (bluetooth connected, relative pointer).
• Apple does not document absolute pointers for AssistiveTouch, and no changelog mentions them. The iOS 15 behavior was found by testing over BLE: the phone read report 1 but the pointer stayed still.
• In relative mode the daemon knows exactly how far each report moves the pointer, from Apple's own acceleration table. See Relative pointer.

Which iPhones get which pointer

• The "absolute from iOS 17" claim comes from a vendor of ESP32 BLE HID firmware, not from Apple. Bluetooth Classic HID may behave differently, so older iOS is checked, not assumed.

Relative pointer (iOS 15)

tap 300,500 with the pointer at 40,120 (move +260,+380 pt) on a 6s:
  (+169,+248) (+170,+247) ── speed 300, past the flat end of the table ──> 214.9 pt each
  (+16,+24)               ── speed 28 ─────────────────────────────────> 30.0 pt
  (0,-1)                  ── speed 1: 0.2 pt, lands within 0.2 pt
  100 ms pause, then click                                   ~0.45 s in total

• Measured on an iPhone 6s (iOS 15.8.4): 15 report sizes, diagonals, and bursts vs 100 ms gaps all matched the table to 0.5 pt. 40 taps in a row landed 0.2 pt from the target on average, with no drift. Details: docs/relative-pointer.md.
• The daemon remembers where the pointer is. Only when it does not know (daemon start, Bluetooth reconnect, rotation, 60 s idle) does it push the pointer into the top-left corner first (+0.3 s).
• Tracking Speed: the pointer check measures a gain (1.000 at the default speed) and scales the table by it. Other speeds are not tested.
• Swipes follow --duration: reports are spread over it and sized so the drag ends exactly at the target.

Build from source

1234git clone https://github.com/remorses/ghosthand
cd ghosthand
swift build -c release
cp .build/release/ghosthand /usr/local/bin/

Publish to npm

12cd npm
pnpm publish   # prepublishOnly cleans dist and runs the build

CLI / curl / any language
   │ HTTP over ~/.ghosthand/daemon.sock (mode 0600, no token)
   │ HTTP or HTTPS over TCP with --port (Bearer token)
   v
Hummingbird ──> generated API (openapi.yaml) ──> own thread ──> Controller
                                                                  │
      ┌───────────────────────────────────────┬───────────────────┤
      v                                       v                   v
  FrameStream per iPhone (AVFoundation)   Lockdown lock state     HIDDevice: BT Classic
  OCR (Vision)                            (MobileDevice)          HID, CBClassicManager