Accessibility snapshots and targets for LLM agents

The six forms

Form	Example	Use it when
Snapshot ref	`@e7`	You have a snapshot and want the exact element it showed.
Role and name	`button "Next"`	You know what the control says, even before seeing the page.
Bare role	`textbox`	There is one of its kind, or in `each` you mean all of them.
Point	`420,260`	Canvas widgets and controls with no name, read off a screenshot.
CSS selector	`#email`, `.balance`, `#rows tr`	You know the markup.
Current item	`@item`	Inside an `each` that clicks its items.

Role and name, bare roles and CSS selectors wait up to 5 seconds for the element to appear and have a size, then fail with no element matches <target>. A point needs no element and acts at once.

Snapshot refs

A snapshot gives each interactive element a ref. Refs are numbered per page and never reused, so a ref always means the element it was given to.

okyra snapshot -i
page "Bank login" file:///…/login.html
scroll 0 of 819
form:
  textbox "Email" @e1
  textbox "Password" @e2 (password)
  button "Log in" @e3

okyra type @e1 ana@example.com
okyra click @e3

When the element is gone, for example after the page navigated, the ref fails at once instead of clicking something else: @e1 is no longer on the page; take a new snapshot.

Role and name

Write the target the way a snapshot line reads: a role, a space, and the name in double quotes. This form works before any snapshot, so one plan can fill a form, click Next and fill the next step.

okyra type 'textbox "Email"' ana@example.com
okyra click 'button "Log in"'
okyra click 'radio "Yearly"'
okyra text 'text "Order number"'

The name matches exactly first, then as a part. The placeholder and the title attribute count as names too.
The given role is tried first, then any interactive role, since a page may draw a link where you expected a button.
Matching ignores case.
The role text finds the innermost visible element whose text contains the words, for reading values that are not controls.
While a modal dialog is open, targets resolve inside it.

Roles with refs: button, link, checkbox, radio, tab, menuitem, menuitemcheckbox, menuitemradio, switch, option, combobox, textbox, searchbox, slider, spinbutton, treeitem.

Bare role

A role with no name, such as textbox or link, means the first visible element with that role. In each and extract it means every one of them. A real CSS tag that matches wins, so button still selects <button> elements.

okyra type textbox "first field"     # the first visible text field
okyra extract link --limit 3          # the text of every link, first three

Points

Two numbers, x,y, in CSS pixels of the viewport. Screenshots are taken at one image pixel per CSS pixel, also on a Retina screen, so a point read off the image is a valid target as it is. --annotate lists every ref with its centre point.

okyra screenshot --annotate /tmp/order.png    # "@e3 radio \"Medium\" at 36,138", ...
okyra click 36,138

CSS selectors

Anything that is not one of the forms above is a CSS selector. It also reaches into open shadow roots. For extract and each, every visible match counts; for the other actions, the first. A ref or a role and name always means one element, in extract too.

okyra text '#hello'
okyra extract '#rows tr'

The current item: @item

Inside an each step with "open": "click", @item is the item it is on now. Use it to hover or read the item itself, such as every point of a chart:

{ "action": "each", "target": "circle.dot", "open": "click", "name": "points",
  "steps": [{ "action": "hover", "target": "@item", "name": "tip" }] }

The full example is on the plans page.