Core Usage

Interacting

Once you've read the page, it's time to act. This page covers every interaction verb for driving a real browser—click, type, check, select, upload, scroll, drag. The first argument to most commands is the @ref you got from snapshot -i (e.g. @e42); first find the element, then act on it with the commands here.

Clicking & pointer

The most-used group of verbs, all acting on a single @ref. click auto-scrolls the element into the viewport, and if the coordinate click is occluded it falls back to the DOM's .click().

click / hover / focus

$ chrome-use click @e1                   # click
$ chrome-use click @e1 --new-tab         # open the link in a new tab instead of navigating the current one
$ chrome-use dblclick @e1                # double-click
$ chrome-use hover @e1                   # hover
$ chrome-use focus @e1                   # focus (often used before keyboard input)

✅ Click succeeds but "nothing happens"?

A classic case: an autocomplete/menu <li> closes the moment the input loses focus. Try again with a DOM-dispatched click: AGENT_BROWSER_CLICK_MODE=dom chrome-use click @e1, or just use eval to select that item in the page.

ℹ️ DOM-dispatched clicks move focus too

Over the extension relay a left click is dispatched through the DOM. It now hands focus to the clicked element (or its nearest focusable ancestor / a label's control) like a real click, unless the click handler already moved focus. So click <input> followed by press Meta+a lands on that input, not on the previously focused field. A plain <li> has no focusable target, so the autocomplete trick above still holds.

Typing text

fill clears first and then types, while type appends after the existing content. The default path is insertText (fast), but many fields with autocomplete only react to real keystrokes—that's when you add --key-events.

fill / type

$ chrome-use fill @e2 "hello"            # clear, then type
$ chrome-use type @e2 " world"          # don't clear, just append
$ chrome-use type @e2 "hello" --clear   # type can clear the field first too
$ chrome-use type @e2 "hello" --delay 80  # 80ms between keystrokes

# real keystrokes (not insertText)—for autocomplete/combobox fields that
# only respond to key events, e.g. auto-filling city from a postcode (Google Places, etc.)
$ chrome-use type @e5 "201-0001" --key-events

# real keystrokes plus Enter to submit the candidate (--enter implies --key-events)—
# for async autocomplete / tag controls, e.g. Juejin's "add tag"
$ chrome-use type @e6 "ChatGPT" --enter

ℹ️ --key-events or --enter?

If you just want to confirm one tag and plain typing doesn't pop a dropdown, use --enter in a single step. If you'd rather pick one from a candidate list, first run type … --key-events to trigger the dropdown, then snapshot -i to find the candidate and click @ref.

✅ --clear and --delay

--clear lets type clear the field first too (no need to switch back to fill); --delay <ms> inserts a fixed gap between keystrokes to mimic a slower, more human typing cadence. Both are dedicated flags — neither leaks into the text that actually gets typed.

ℹ️ Page rewrote the input? type warns, fill errors

After type the field is read back: if it does not contain the typed text, a ⚠ warning quotes what was written and what the field now holds (exit 0, JSON gains readBack). A fill verification failure quotes both values too, e.g. read back "" after writing "狛江市"; when every non-ASCII character vanished while ASCII survived, it says the page filtered non-Latin input (a Latin-only / masked field), so re-typing will not help. Both deliver CJK fine on normal fields; these messages are about pages that rewrite input.

Editing inside a field, and pasting with a MIME type

fill replaces a whole value and type appends. Changing one phrase inside written text, or placing the caret somewhere and carrying on, is neither — that is what select-text is for.

select-text

$ chrome-use select-text @e3 "confirm" --prefix "please "  # selects `confirm`, not `please confirm`
$ chrome-use select-text @e3 "Hi Sam," --cursor-after       # a caret, not a selection
$ chrome-use type @e3 " quick note:"                       # continues at the caret

⚠️ A repeated phrase is refused, not resolved to the first one

The prefix and suffix are context for finding the match, not part of the selection. "Not found", "found but not with that context" and "matches N places" are three different messages because they have three different fixes. Monaco and CodeMirror keep their own selection model and are refused by name: a DOM selection there looks applied and does nothing.

In a rich-text editor, typing and pasting produce different documents: type "<b>bold</b>" gives you those eleven characters, paste --format html gives you bold text. Newlines are the other reason — type sends Enter, which submits or splits a block in most editors, while a paste inserts the line break.

paste

$ chrome-use paste $'line one\nline two' --selector "#notes"  # $'...' sends a real newline
$ chrome-use paste "<b>bold</b> text" --format html --selector "#editor"
$ chrome-use paste "# Heading" --format md            # Markdown source as plain text

✅ Your real clipboard is never touched

The content rides on a synthetic ClipboardEvent — no navigator.clipboard call, no Ctrl+V. Such an event is untrusted and has no default action, so an editor that listens gets it through its own handler and one that ignores it gets a real insert; the reply names which path ran. If neither takes, the command fails rather than printing a ✓.

Keyboard

press presses a single key on the current focus (down + up), and also supports key combos. keydown/keyup split the press and release apart, and used as a pair they let you "hold".

press / keydown / keyup

$ chrome-use press Enter                 # press a key on the current focus
$ chrome-use press Control+a             # key combo
$ chrome-use keydown d                   # hold a key down (no auto release)
$ chrome-use keyup d                     # release—use as a pair to "hold to move"
                                       # in a game: keydown d; sleep; keyup d

⚠️ A warning when the key provably did nothing

For keys whose only effect depends on page JavaScript (Arrow/Home/End/PageUp/PageDown on a text field, Escape, Enter on a bare input outside a form), press probes for keydown/keyup/keypress listeners on the focused element, its ancestors, document and window. With none found it still exits 0 but prints a ⚠ warning that the page cannot react to the key and suggests clicking the option instead; the JSON gains keyListeners: <count>. Chords (Ctrl/Meta+key), Tab, Backspace, printable keys and native defaults are not probed.

ℹ️ More precise holds in games / Canvas

For canvas / WebGL apps, press d --hold 800 is better—the hold duration is timed inside the daemon, with ~250ms less jitter per cycle than keydown + shell sleep + keyup. See Canvas & games.

Check, select and dropdowns

Checkboxes use check/uncheck. For dropdowns, both a native <select> and a custom combobox (react-select / ARIA, etc.) work with select — since v1.5.72 select falls through to portal-aware logic on a non-native control (open → wait for options → click by visible text); pick is the explicit alias for the same behavior. Native selects use the platform value/selected setters and dispatch both input and change, so React/Vue controlled forms commit the new state.

check / select / pick

$ chrome-use check @e3                   # check a checkbox
$ chrome-use uncheck @e3                 # uncheck

$ chrome-use select @e4 "Pageview"       # native <select> OR a custom combobox (v1.5.72+)
$ chrome-use select @e4 "a" "b"          # multi-select

# any combobox: open it, wait for the menu to appear (including portal-rendered),
# match by visible text, dispatch the right events; errors out if the option never
# appears (never silently no-ops)
$ chrome-use pick @e4 --option "Europe"

✅ v1.5.72: native commands survive shadow-DOM-heavy apps (#105)

select used to "return ✓ but change nothing" on custom dropdowns (a silent false success) — now fixed: non-native controls go through the portal-aware path and error loudly (listing the visible options) when the option never appears, never silently. fill is hardened the same way: it resolves Monaco through its global or AMD model API, performs one atomic setValue, and verifies the exact model readback before reporting success. When an app hides that API, it performs one trusted editor paste, verifies the selected model through the editor's own copy handler, and restores the browser clipboard. If those operations cannot be verified or the value differs, it errors loudly. Controlled inputs whose @ref anchored a wrapper still retarget to the nested editable.

CSP-restricted pages

Cloudflare Zaraz and pages with a strict Content Security Policy must not rely on eval. Use the native select and fill commands so the interaction works without requiring unsafe-eval.

Actions beyond a click

Some controls do more than click: a disclosure expands, a menu button opens a popup, a spinbutton or slider steps through a range. actions tells you what this element supports right now; do performs one of exactly those.

actions / do

$ chrome-use actions @e15
@e15 DisclosureTriangle "Disclosure summary"
  expand

$ chrome-use do @e15 expand
✓ expand on @e15
now: collapse

The set is read live, not carried over from the snapshot that minted the ref — what an element supports is state, not identity. A disclosure that was collapsed when you snapshotted may be open now, and offering expand would send you to do the opposite of what you asked.

An action outside the set is refused with the supported list rather than attempted. After acting it reports the set again, so a control that did not move cannot read as a success.

ℹ️ What is supported

expand / collapse (disclosures), showMenu (controls that open one), increment / decrement (value ranges), and toggle (pressable controls) — all derived from the element's accessibility properties. A disabled element offers none.

Uploading files

upload feeds a file to a file <input> or to a drop/paste-style editor (like X's composer). It also works over the extension relay. If a React dropzone consumes the files and immediately clears or replaces its hidden input, the command succeeds with a warning instead of falsely claiming that the page rejected the upload.

upload

$ chrome-use upload @e5 file1.pdf        # upload a file

ℹ️ Why upload works in relay mode too

chrome.debugger forbids setFileInputFiles, so the file bytes are streamed into the page and reconstructed there into a File (chunked to native-messaging's 1 MiB cap). That's why both file <input> elements and drop/paste editors receive it.

Scrolling

scroll scrolls the page in a direction; scrollintoview scrolls a specific element into the viewport. For content inside cross-origin iframes (payment / checkout / KYC widgets), ordinary page scrolling can't reach it—use --at x,y to wheel at a pixel point, or --frame n to target a frame.

scroll / scrollintoview

$ chrome-use scroll down 500              # scroll the page (up/down/left/right)
$ chrome-use scroll down 700 --at 640,400  # wheel at a pixel point—reaches cross-origin iframes
                                        # (Stripe/checkout/KYC) that ordinary page scrolling can't
$ chrome-use scroll down 700 --frame 2    # scroll frame #2 from `chrome-use frames`
$ chrome-use scrollintoview @e1           # scroll the element into the viewport

Drag & slider CAPTCHAs

drag can both drop between two refs and drag a handle by a pixel offset (sliders / canvas). For the NetEase Yidun slider CAPTCHA common to unattended logins, use solve-slider to pass it automatically.

drag / solve-slider

$ chrome-use drag @e1 @e2                # drag and drop
$ chrome-use drag @e1 60                 # drag the handle +60px (slider/canvas); `+60,-3` means dx,dy
$ chrome-use solve-slider                # auto-solve the NetEase Yidun slider CAPTCHA on the page
$ chrome-use solve-slider 5              # …retry up to 5 times (each failure refreshes the puzzle)

ℹ️ How solve-slider passes

It fetches the CAPTCHA's own background image + puzzle-piece slice by URL (no screenshot), locates the gap offline via edge + mask cross-correlation, then drags the handle into the gap with a humanized, self-calibrating closed-loop trajectory—it's exactly this human-like motion that gets past Yidun's behavioral detection. Both float (inline) and popup (modal, as on Zhihu) modes are supported; just run it right after the submit that triggers the slider. This drag force-enables the humanize trajectory regardless of the global AGENT_BROWSER_HUMANIZE.

⚠️ CAPTCHAs not yet covered

Yidun's enhanced slider (icon-shaped puzzle pieces + decoys) and click-select (click in order) are harder challenges and aren't handled yet.

Downloads

With ab-connect 0.5.13 or newer, HTTP(S) links are downloaded through Chrome's downloads API. chrome-use resolves a link's href before acting, including dynamically-created anchors listed by snapshot -i. This prevents a cross-origin download link from navigating the active tab.

bash

$ chrome-use download @e2 ./video.mp4
$ chrome-use download-url https://example.com/report.pdf ./report.pdf
$ chrome-use downloads --limit 10 --json
$ chrome-use downloads --clear

Omitting the destination from download-url keeps Chrome's normal download location. downloads --clear removes history entries only and never deletes files.

Cross-origin iframes: always act by ref

Embedded payment / checkout / KYC widgets (Google Payments, Stripe, etc.) are cross-process out-of-process iframes. Drive them by ref, never by screenshot. snapshot -i pierces these iframes and lists the elements inside them by @ref (including input values); get text --all-frames reads their text. Once you have the ref, act on it as usual.

act by ref inside an iframe

# snapshot -i lists elements inside cross-origin iframes, then:
$ chrome-use click @e
$ chrome-use type @e "…"
$ chrome-use hover @e
$ chrome-use dblclick @e
$ chrome-use drag @a @b

# postcode/autocomplete field inside the frame—use real keystrokes:
$ chrome-use type @e "201-0001" --key-events

⚠️ Never target the <iframe> element itself

focus and press on an Iframe ref land on the container in the parent document, so the keystroke goes to the parent page instead of the field inside. Those commands now return a warning naming the frame boundary rather than a plain ✓ on the wrong target. Go through the frame: chrome-use frames to list them, then chrome-use frame <id> and act on a ref inside the frame, or chrome-use eval --frame <id> "…".

✅ Why by ref, not coordinates

Over the extension relay, these actions are dispatched via DOM in the element's own frame, so they precisely hit the right element in the right tab. Coordinate clicks/scrolls can drift onto whatever other tab is currently in the foreground—so prefer refs. When you need to scroll to content further down inside a frame, use scroll down N --at x,y (a pixel point on the frame) or --frame n.

⚠️ find can't reach closed shadow roots or cross-origin iframes

find/selectors match the page DOM (querySelectorAll), so for these two kinds of elements they report "Element not found"—even though snapshot -i can list them (it goes through the CDP accessibility tree, which pierces both) and get text can read them. In that case locate by the @ref from the snapshot instead of find; box @ref gives coordinates as a fallback. Also, verify the exact label text before concluding an element "doesn't exist": LinkedIn's "Save" button is actually labeled 收藏, not 保存, and snapshot -i has always shown button "收藏" [ref=eN]—just click @eN.

Interacting is just one link in the core loop; read it alongside these pages:

🔎 Finding elements

When refs aren't enough, switch to find role/text/label, CSS or coordinates.

⏳ Waiting

Wait for the page to settle before and after actions, so you don't hit unrendered elements.

🎮 Canvas & games

Canvas apps with no accessibility tree—drive by coordinates and WebSocket.