Advanced

Game screens, voice-room mic seats, map tiles, 3D viewers, design canvases (Figma / Modao) — these UIs paint everything onto a <canvas>, with no DOM nodes and no accessibility nodes behind them. snapshot / find / eval querySelector will never get their @ref. This isn't a missing feature — it's a hard limit. This page teaches a different playbook: look at pixels, act by coordinates.

Why canvas has no ref

A <canvas> / WebGL hit area, and a closed shadow root, expose no DOM / AX node for the controls they paint. So no matter how snapshot scans, it cannot fabricate a @ref — coordinates are the only handle. When snapshot detects such a page it prints a hint.

Reading the screen: canvas capture and screenshot

On a canvas page, eval / get text mostly return nothing useful, so looking is your only read path. Enumerate the canvases first, grab the rendered pixels, then let you interpret them:

# Enumerate the <canvas> elements on the page (size, type)
$ chrome-use canvas list

# Save the canvas "rendered pixels" as PNG — toDataURL gets the full
# backing-store resolution (e.g. Figma's 2522x1904); for WebGL without
# preserveDrawingBuffer / tainted canvases it falls back to screenshot
$ chrome-use canvas capture out.png

# Or just grab a region shot to "see" the current state
$ chrome-use screenshot /tmp/s.png

Symptom: text shows in get text, but no @ref

Voice-room mic seats (Zego / Agora), prototype canvases (Modao / modao), game HUDs, and some Web Components paint their controls — so the text appears in get text / read_page (e.g. a string of "Add Add Add…"), but snapshot -i lists nothing and querySelectorAll returns 0. With no addressable node, @ref / find can't reach it. Drive by position:

# Try first: if it's just a "closed" shadow root (not canvas),
# this reads through it — cheap, worth trying first
$ chrome-use get text --pierce

# Otherwise "look" for where the control is
$ chrome-use screenshot /tmp/s.png

# Click that pixel (bare numbers = coordinates)
$ chrome-use click <x> <y>

Interact by coordinate: click and press --hold

On canvas, everything runs on coordinates. Compute the target point and click x y directly; when you first need a container's CSS-px box, use box @ref to get its position and convert. Coordinates are the right tool here — the snapshot-first rule explicitly makes this exception for canvas.

# Interact by viewport coordinates
$ chrome-use click 640 360

# Hold-move with precise timing (timed inside the daemon —
# not keydown + shell-sleep + keyup, which adds ~250ms of
# jitter on every round trip)
$ chrome-use press d --hold 800

# Discrete actions (jump / attack / confirm)
$ chrome-use press Space

batch: run a timed sequence in one round trip

Don't drive frame-by-frame with one CLI call per action — that's the slowest, lowest-fidelity approach, since every call is a process launch + round trip. Use batch to pack a timed sequence into a single round trip: it sends each step to the already-running daemon, and press --hold and wait both block inside the daemon, so the timing is accurate.

$ chrome-use batch "press d --hold 900" "press j" "press j" "wait 200" "press d --hold 500"

Read engine globals instead of guessing pixels

Rather than guessing from a screenshot, read the real state directly. eval runs in the page's main world, so for framework / engine-based games you can often reach straight into its global object — a Phaser / PIXI / Three instance, a store, or window.__GAME__ — and read position / score directly, far better than guessing at a screenshot.

# eval runs in the main world, so it can read the engine instance on window
$ chrome-use eval "window.__GAME__.player.position"

Real-time driving: the WebSocket stream

For truly real-time driving, drop the CLI entirely and use a WebSocket. chrome-use stream enable opens a bidirectional WS (stream status prints ws://127.0.0.1:<port>). Connect once and you get both a ~60fps live screencast and the ability to send input on the same socket — no per-action process launch, no round trips, and it works across the extension relay.

$ chrome-use stream enable
$ chrome-use stream status   # prints ws://127.0.0.1:<port>

// node (global WebSocket): live frames + locally-timed input
const ws = new WebSocket("ws://127.0.0.1:PORT")
ws.onmessage = e => { const m = JSON.parse(e.data); if (m.type==="frame") {/* base64 jpeg */} }
const k = (eventType,key,code,vk) => ws.send(JSON.stringify({type:"input_keyboard",eventType,key,code,windowsVirtualKeyCode:vk}))
k("keyDown"," ","Space",32); setTimeout(()=>k("keyUp"," ","Space",32), 80)   // one jump
// mouse works the same: {type:"input_mouse",eventType:"mousePressed",x,y,button:"left",clickCount:1}

Virtualized rich editors: probe before you commit

Unlike canvas, Google Docs/Sheets, Notion, Figma, and Lark/Feishu do have a DOM — but it lies. The editing surface is a virtualized layer, and the DOM around it is littered with decoys: a hidden <textarea> mirror, an offscreen input, a toolbar/search box, a title field. fill / type on a ref plucked from snapshot -i often lands your text in one of those decoys — the title bar, a find box — not the document. Snapshot-first still holds for navigation (menus, buttons, dialogs); but for the main editing surface, prove where your keystrokes go before you commit a paragraph.

# 1. WRITE PROBE — one throwaway token via real keystrokes, don't dump the payload yet
$ chrome-use click 520 300              # click into the document body by coordinate
$ chrome-use keyboard type "zzprobe"    # real keydown/keypress/keyup, not fill/insertText
# 2. VERIFY it landed in the document — not the title/toolbar/a hidden input
$ chrome-use screenshot /tmp/probe.png  # SEE where "zzprobe" appeared (or read it back via the app's export/API)
# 3a. landed in the doc  → continue with real keystrokes
# 3b. landed in the wrong field → stop using DOM/fill here: click the body by coordinate, real keyboard only

Decision cheat sheet