Core Usage

Interacting

Custom comboboxes scroll instantly into view before trusted pointer input opens and selects an option. pick returns verified: false with a warning when the choice is unconfirmed; hidden form changes are reported as evidence, not proof, when selection clears the search input. Page/container scrolling is instant and reports scrolled: false with a warning when the position is unchanged.

Once you've read the page, it's time to act. This page covers every interaction verb for driving a real browser—click, type, check, select, upload, scroll, drag. The first argument to most commands is the @ref you got from snapshot -i (e.g. @e42); first find the element, then act on it with the commands here.

Clicking & pointer

Semantic clicks report the covering element with a bounded tag, id/class and text description. If the hit-test cannot confirm the target, the click fails without guessing a blocker or dispatching input.

The most-used group of verbs, all acting on a single @ref. click auto-scrolls the element into the viewport and refuses a covered target by default. --allow-dom explicitly opts into a DOM click.

click / hover / focus

$ chrome-use click @e1                   # click
$ chrome-use click @e1 --new-tab         # open the link in a new tab instead of navigating the current one
$ chrome-use dblclick @e1                # double-click
$ chrome-use hover @e1                   # hover (a real pointer move, confirmed on the element)
$ chrome-use focus @e1                   # focus (often used before keyboard input)

ℹ️ hover reports only what the page confirms

hover scrolls the element into view, finds a point where the element itself is on top, moves the real pointer there, then checks that a trusted pointer event landed on the element (it does not inspect CSS styles). Only a confirmed hover prints ✓ Hovered … (a trusted pointer event landed on it). A target covered by a banner or backdrop is refused (hover refused) and nothing is sent; a move whose events miss the element is an error (did not take effect). Inside a cross-origin iframe no real pointer can be placed, so only synthetic events (isTrusted=false) go out, CSS :hover never applies, and the output says the hover is unconfirmed (whether the page reacted is unknown). A frame covered by a parent overlay, or one that is scaled, rotated or zoomed, is refused before anything is sent.

✅ Click succeeds but "nothing happens"?

A classic case: an autocomplete/menu <li> closes the moment the input loses focus. Try again with a DOM-dispatched click: AGENT_BROWSER_CLICK_MODE=dom chrome-use click @e1, or just use eval to select that item in the page.

ℹ️ DOM-dispatched clicks move focus too

Over the extension relay a left click is dispatched through the DOM. It now hands focus to the clicked element (or its nearest focusable ancestor / a label's control) like a real click, unless the click handler already moved focus. So click <input> followed by press Meta+a lands on that input, not on the previously focused field. A plain <li> has no focusable target, so the autocomplete trick above still holds.

Typing text

For Ant Design / rc-picker date ranges, fill the start bound and then the end bound. fill confirms each bound through the component's trusted keyboard or visible confirmation control and reads it back, instead of blurring and refocusing a draft. Focus follows the picker. Invalid, read-only or disabled controls fail without a synthetic overwrite; an unverified popup requires explicit picker interaction. Verify the final range before running the query.

fill clears first and then types, while type appends after the existing content. The default path is insertText (fast), but many fields with autocomplete only react to real keystrokes—that's when you add --key-events.

fill / type

$ chrome-use fill @e2 "hello"            # clear, then type
$ chrome-use type @e2 " world"          # don't clear, just append
$ chrome-use type @e2 "hello" --clear   # type can clear the field first too
$ chrome-use type @e2 "hello" --delay 80  # 80ms between keystrokes

# real keystrokes (not insertText)—for autocomplete/combobox fields that
# only respond to key events, e.g. auto-filling city from a postcode (Google Places, etc.)
$ chrome-use type @e5 "100-0001" --key-events

# real keystrokes plus Enter to submit the candidate (--enter implies --key-events)—
# for async autocomplete / tag controls, e.g. Juejin's "add tag"
$ chrome-use type @e6 "ChatGPT" --enter

ℹ️ --key-events or --enter?

If you just want to confirm one tag and plain typing doesn't pop a dropdown, use --enter in a single step. If you'd rather pick one from a candidate list, first run type … --key-events to trigger the dropdown, then snapshot -i to find the candidate and click @ref.

✅ --clear and --delay

--clear lets type clear the field first too (no need to switch back to fill); --delay <ms> inserts a fixed gap between keystrokes to mimic a slower, more human typing cadence. Both are dedicated flags — neither leaks into the text that actually gets typed.

ℹ️ Page rewrote the input? type warns, fill errors

After type the field is read back: if it does not contain the typed text, a ⚠ warning quotes what was written and what the field now holds (exit 0, JSON gains readBack). A fill verification failure quotes both values too, e.g. read back "" after writing "千代田区"; when every non-ASCII character vanished while ASCII survived, it says the page filtered non-Latin input (a Latin-only / masked field), so re-typing will not help. Both deliver CJK fine on normal fields; these messages are about pages that rewrite input.

⚠️ The field shows the value, but the page never took it (#358)

fill and type --clear enter text the way a user does: select the old value, then write with trusted events (isTrusted: true beforeinput / input, and a trusted change on blur), so pages that only honour real input register it. fill warns when the field's form or dialog had a disabled Save / Submit before the fill and still has it after (JSON gains commitControl); when the trusted write cannot produce the value it falls back to the native setter, reports engine: input-synthetic and warns. click refuses a :disabled control and reports dispatch: dom when it had to fall back to element.click(). keyboard type reports the focused element and its read-back, and fails when a focused field did not change. For a Save button, use click @ref --observe to see whether a request went out.

If the page discards the element handle part-way through a fill (CDP "Could not find object with given id", seen on the extension relay), fill re-resolves the element once and reads it first: a field that already holds the value is reported with a ⚠ note and not typed again; one that does not is filled once more and says so. A field that cannot be read back at all is unknown, not different: the fill fails with "value is unknown" and writes nothing; check it with get value <ref> first. Do not repeat the fill yourself after any of these.

Also: button "简体中文" [toggles=checkbox(checked=true)] in a snapshot is a switch, not a link. In LinkedIn's "edit profile language" dialog, clicking it deletes that language's profile. Never use it to switch tabs.

Editing inside a field, and pasting with a MIME type

fill replaces a whole value and type appends. Changing one phrase inside written text, or placing the caret somewhere and carrying on, is neither — that is what select-text is for.

select-text

$ chrome-use select-text @e3 "confirm" --prefix "please "  # selects `confirm`, not `please confirm`
$ chrome-use select-text @e3 "Hi Sam," --cursor-after       # a caret, not a selection
$ chrome-use type @e3 " quick note:"                       # continues at the caret

⚠️ A repeated phrase is refused, not resolved to the first one

The prefix and suffix are context for finding the match, not part of the selection. "Not found", "found but not with that context" and "matches N places" are three different messages because they have three different fixes. Monaco and CodeMirror keep their own selection model and are refused by name: a DOM selection there looks applied and does nothing.

In a rich-text editor, typing and pasting produce different documents: type "<b>bold</b>" gives you those eleven characters, paste --format html gives you bold text. Newlines are the other reason — type sends Enter, which submits or splits a block in most editors, while a paste inserts the line break.

paste

$ chrome-use paste $'line one\nline two' --selector "#notes"  # $'...' sends a real newline
$ chrome-use paste "<b>bold</b> text" --format html --selector "#editor"
$ chrome-use paste "# Heading" --format md            # Markdown source as plain text

✅ Your real clipboard is never touched

The content rides on a synthetic ClipboardEvent — no navigator.clipboard call, no Ctrl+V. Such an event is untrusted and has no default action, so an editor that listens gets it through its own handler and one that ignores it gets a real insert; the reply names which path ran. If neither takes, the command fails rather than printing a ✓.

Keyboard

press presses a single key on the current focus (down + up), and also supports key combos. keydown/keyup split the press and release apart, and used as a pair they let you "hold".

press / keydown / keyup

$ chrome-use press Enter                 # press a key on the current focus
$ chrome-use press Control+a             # key combo
$ chrome-use keydown d                   # hold a key down (no auto release)
$ chrome-use keyup d                     # release—use as a pair to "hold to move"
                                       # in a game: keydown d; sleep; keyup d

⚠️ A warning when the key provably did nothing

For keys whose only effect depends on page JavaScript (Arrow/Home/End/PageUp/PageDown on a text field, Escape, Enter on a bare input outside a form), press probes for keydown/keyup/keypress listeners on the focused element, its ancestors, document and window. With none found it still exits 0 but prints a ⚠ warning that the page cannot react to the key and suggests clicking the option instead; the JSON gains keyListeners: <count>. Chords (Ctrl/Meta+key), Tab, Backspace, printable keys and native defaults are not probed.

ℹ️ More precise holds in games / Canvas

For canvas / WebGL apps, press d --hold 800 is better—the hold duration is timed inside the daemon, with ~250ms less jitter per cycle than keydown + shell sleep + keyup. See Canvas & games.

Check, select and dropdowns

Checkboxes use check/uncheck. For dropdowns, both a native <select> and a custom combobox (react-select / ARIA, etc.) work with select — since v1.5.72 select falls through to portal-aware logic on a non-native control (open → wait for options → click by visible text); pick is the explicit alias for the same behavior. Native selects use the platform value/selected setters and dispatch both input and change, so React/Vue controlled forms commit the new state.

check / select / pick

$ chrome-use check @e3                   # check a checkbox
$ chrome-use uncheck @e3                 # uncheck

$ chrome-use select @e4 "Pageview"       # native <select> OR a custom combobox (v1.5.72+)
$ chrome-use select @e4 "a" "b"          # multi-select

# any combobox: open it, wait for the menu to appear (including portal-rendered),
# match by visible text, dispatch the right events; errors out if the option never
# appears (never silently no-ops)
$ chrome-use pick @e4 --option "Europe" --observe  # returns the delta it caused, like click

✅ v1.5.72: native commands survive shadow-DOM-heavy apps (#105)

select used to "return ✓ but change nothing" on custom dropdowns (a silent false success) — now fixed: non-native controls go through the portal-aware path and error loudly (listing the visible options) when the option never appears, never silently. fill is hardened the same way: it resolves Monaco through its global or AMD model API, performs one atomic setValue, and verifies the exact model readback before reporting success. When an app hides that API, it performs one trusted editor paste, verifies the selected model through the editor's own copy handler, and restores the browser clipboard. If those operations cannot be verified or the value differs, it errors loudly. Controlled inputs whose @ref anchored a wrapper still retarget to the nested editable.

CSP-restricted pages

Cloudflare Zaraz and pages with a strict Content Security Policy must not rely on eval. Use the native select and fill commands so the interaction works without requiring unsafe-eval.

Actions beyond a click

Some controls do more than click: a disclosure expands, a menu button opens a popup, a spinbutton or slider steps through a range. actions tells you what this element supports right now; do performs one of exactly those.

actions / do

$ chrome-use actions @e15
@e15 DisclosureTriangle "Disclosure summary"
  expand

$ chrome-use do @e15 expand
✓ expand on @e15
now: collapse

The set is read live, not carried over from the snapshot that minted the ref — what an element supports is state, not identity. A disclosure that was collapsed when you snapshotted may be open now, and offering expand would send you to do the opposite of what you asked.

An action outside the set is refused with the supported list rather than attempted. After acting it reports the set again, so a control that did not move cannot read as a success.

ℹ️ What is supported

expand / collapse (disclosures), showMenu (controls that open one), increment / decrement (value ranges), and toggle (pressable controls) — all derived from the element's accessibility properties. A disabled element offers none.

Uploading files

upload feeds a file to a file <input> or to a drop/paste-style editor (like X's composer). It also works over the extension relay. If a React dropzone consumes the files and immediately clears or replaces its hidden input, the command succeeds with a warning instead of falsely claiming that the page rejected the upload.

upload

$ chrome-use upload @e5 file1.pdf        # upload a file

ℹ️ Why upload works in relay mode too

chrome.debugger forbids setFileInputFiles unless the extension has Allow access to file URLs (off for a Web Store install), so the file bytes are streamed into the page and reconstructed there into a File, in chunks well under native-messaging's 1 MiB cap. Every chunk costs the same whatever the file size (a 200 MB video takes about 15 s), and uploads to one browser run one at a time: a second one waits its turn and the reply's delivery.queuedMs says how long. That's why both file <input> elements and drop/paste editors receive it. Turning on Allow access to file URLs for the chrome-use extension lets Chrome read the file directly instead.

Scrolling

scroll scrolls the page in a direction; scrollintoview scrolls a specific element into the viewport. For content inside cross-origin iframes (payment / checkout / KYC widgets), ordinary page scrolling can't reach it—use --at x,y to wheel at a pixel point, or --frame n to target a frame.

scroll / scrollintoview

$ chrome-use scroll down 500              # scroll the page (up/down/left/right)
$ chrome-use scroll down 700 --at 640,400  # wheel at a pixel point—reaches cross-origin iframes
                                        # (Stripe/checkout/KYC) that ordinary page scrolling can't
$ chrome-use scroll down 700 --frame 2    # scroll frame #2 from `chrome-use frames`
$ chrome-use scroll down --until "#comments" # step until the target is in view (also @ref, text=, --until-text)
                                        # not found: exit 1 saying how far it scrolled; cap with --max-steps / --timeout
$ chrome-use scrollintoview @e1           # scroll the element into the viewport

Drag & slider CAPTCHAs

drag can both drop between two refs and drag a handle by a pixel offset (sliders / canvas). For the NetEase Yidun slider CAPTCHA common to unattended logins, use solve-slider to pass it automatically.

drag / solve-slider

$ chrome-use drag @e1 @e2                # drag and drop
$ chrome-use drag @e1 60                 # drag the handle +60px (slider/canvas); `+60,-3` means dx,dy
$ chrome-use solve-slider                # auto-solve the NetEase Yidun slider CAPTCHA on the page
$ chrome-use solve-slider 5              # …retry up to 5 times (each failure refreshes the puzzle)

ℹ️ How solve-slider passes

It fetches the CAPTCHA's own background image + puzzle-piece slice by URL (no screenshot), locates the gap offline via edge + mask cross-correlation, then drags the handle into the gap with a humanized, self-calibrating closed-loop trajectory—it's exactly this human-like motion that gets past Yidun's behavioral detection. Both float (inline) and popup (modal, as on Zhihu) modes are supported; just run it right after the submit that triggers the slider. This drag force-enables the humanize trajectory regardless of the global AGENT_BROWSER_HUMANIZE.

Sliders and ordered clicks

solve-slider handles ordinary puzzles and rotating icon-shaped pieces: it measures linear/angular motion, matches the main silhouette, corrects the actual CSS position and verifies the current challenge. Ordered clicks still use the agent's image understanding, coordinate conversion and ordinary clicks. Do not run the slider solver on an ordered-click challenge.

Attempt the visible challenge first

In an authorized login or browser task, identify the current challenge and try its normal controls. A CAPTCHA label or observed.humanCheck warning does not prove personal presence is required. Use solve-slider 1 for an ordinary Yidun puzzle. For ordered icons, view a screenshot, identify the instruction and targets, convert image coordinates to viewport CSS pixels and click in order. Measure screenshot and viewport dimensions; do not simply divide by devicePixelRatio. Activate the intended tab before capturing and measuring: activation can change the layout.

Load chrome-use skills get core/captcha for the complete workflow. Use at most three fresh attempts, including solver retries. Reread refreshed challenges and do not resend SMS to retry them. A Done receipt does not establish success: read the site's result, then continue the authorized task. Exhausted solve-slider attempts return an error and a nonzero exit code. Hand off only after failed attempts, ambiguous targets or a step requiring personal presence, and resume after user confirmation. This workflow does not guarantee every provider or challenge can be solved; do not disable checks or fabricate verification results. A frozen countdown or provider success does not establish site acceptance; verify the site response too.

Downloads

With ab-connect 0.5.13 or newer, HTTP(S) links are downloaded through Chrome's downloads API. chrome-use resolves a link's href before acting, including dynamically-created anchors listed by snapshot -i. This prevents a cross-origin download link from navigating the active tab.

bash

$ chrome-use download @e2 ./video.mp4
$ chrome-use download-url https://example.com/report.pdf ./report.pdf
$ chrome-use downloads --limit 10 --json
$ chrome-use downloads --clear

Omitting the destination from download-url keeps Chrome's normal download location. downloads --clear removes history entries only and never deletes files.

Cross-origin iframes: always act by ref

Embedded payment / checkout / KYC widgets (Google Payments, Stripe, etc.) are cross-process out-of-process iframes. Drive them by ref, never by screenshot. snapshot -i pierces these iframes and lists the elements inside them by @ref (including input values); get text --all-frames reads their text. Once you have the ref, act on it as usual.

act by ref inside an iframe

# snapshot -i lists elements inside cross-origin iframes, then:
$ chrome-use click @e
$ chrome-use type @e "…"
$ chrome-use hover @e          # synthetic events only in the frame: CSS :hover does not apply, reported unconfirmed
$ chrome-use dblclick @e
$ chrome-use drag @a @b

# postcode/autocomplete field inside the frame—use real keystrokes:
$ chrome-use type @e "100-0001" --key-events

⚠️ Never target the <iframe> element itself

focus and press on an Iframe ref land on the container in the parent document, so the keystroke goes to the parent page instead of the field inside. Those commands now return a warning naming the frame boundary rather than a plain ✓ on the wrong target. Go through the frame: chrome-use frames to list them, then chrome-use frame <id> and act on a ref inside the frame, or chrome-use eval --frame <id> "…".

✅ Why by ref, not coordinates

Over the extension relay, these actions are dispatched via DOM in the element's own frame, so they precisely hit the right element in the right tab. Coordinate clicks/scrolls can drift onto whatever other tab is currently in the foreground—so prefer refs. When you need to scroll to content further down inside a frame, use scroll down N --at x,y (a pixel point on the frame) or --frame n.

⚠️ find can't reach closed shadow roots or cross-origin iframes

find/selectors match the page DOM (querySelectorAll), so for these two kinds of elements they report "Element not found"—even though snapshot -i can list them (it goes through the CDP accessibility tree, which pierces both) and get text can read them. In that case locate by the @ref from the snapshot instead of find; box @ref gives coordinates as a fallback. Also, verify the exact label text before concluding an element "doesn't exist": LinkedIn's "Save" button is actually labeled 收藏, not 保存, and snapshot -i has always shown button "收藏" [ref=eN]—just click @eN.

Fields that reformat what you type

Stripe shows an expiry 1234 as 12 / 34 and a card number in groups of four. fill accepts a field that only added spaces and separators and says so with a ⚠ note; a value that differs in any other way is still an error, and a card or password value is never echoed in it.

Rich-text editors (contenteditable) are compared as visible text: zero-width characters (the U+200B an editor pads lines and topic nodes with), NBSP and whitespace runs do not count. A write the editor only formatted succeeds with an "editor formatted the text" note; missing or changed characters are still an error.

What a click caused

With --observe, observed.announced lists the text of toasts and alert/status/aria-live regions that appeared while the click ran and the page settled, even one gone before the reply ("video under review, not saved"). A clickable <div> named by its text (generic "Post") keeps working as a @ref right after the snapshot; if the page reuses that node for other content the ref is refused. A tab opened from a tab the session does not own (--tab … --force, the side panel on your tab) stays yours: the click reports it with followed: false, and tab adopt <url> takes it.

Interacting is just one link in the core loop; read it alongside these pages:

🔎 Finding elements

When refs aren't enough, switch to find role/text/label, CSS or coordinates.

⏳ Waiting

Wait for the page to settle before and after actions, so you don't hit unrendered elements.

🎮 Canvas & games

Canvas apps with no accessibility tree—drive by coordinates and WebSocket.

Semantic find role/text/label/placeholder/alt/title/testid requires exactly one visible match, including locate-only queries. Ambiguity returns up to eight candidates with visible state and selector/context hints; no action is dispatched and input values are omitted. Narrow --name/--exact, or use --within <CSS|@ref> to restrict the query to exactly one container. A scope must belong to the active tab's main document; cross-frame refs and an explicitly selected iframe are refused; use direct frame refs or `frame main`. find first/last/nth explicitly selects an order and keeps its existing behavior. A plain CSS selector or XPath given to an action (click, fill, type, hover, check, select, drag …) must match exactly one element too: one that matches several is refused with code selector_ambiguous, listing the first five matches with their text and position, and nothing is done. Narrow it, use a @ref, or choose by index with find nth <index> <selector> <action>; AGENT_BROWSER_STRICT_SELECTORS=0 (read when the session's daemon starts) restores first-match for old scripts. Roles and label names use Chrome accessibility data, including aria-labelledby; a semantic query re-resolves on each call. A detached target before dispatch fails safely; uncertain actions are never replayed.

Discover a scope from the page first: find query "Beta account" returns candidate selector anchors; choose the article/container corresponding to that heading, then use find role button click --name Save --exact --within "<returned-selector>". Inspect an ambiguous scope with snapshot -s "<selector>" and narrow it rather than accepting the first container. Unknown semantic-find options and duplicate --within are refused; use find label Email fill -- --name to enter literal flag-looking text. --exact false requests substring matching.

The existing chrome_use_find tool in mcp --tools all accepts within with the same unique-scope rules; tool count is unchanged. text for fill/type is passed literally, including --name --observe, rather than parsed as CLI options.

Semantic find retains the existing --observe unsupported warning; use one batch containing the scoped find action and a task-specific get text receipt when both are known. The MCP find tool does not advertise an observe field.