Kuang Grade Mark Eleven: Selector & Set-of-Marks Recipes

1. Golden Operational Sequence (Preferential Tool Hierarchy)

To maximize execution speed, minimize token burn, and eliminate bot shadowbans, ALWAYS follow this 3-tier sequence:

┌────────────────────────────────────────────────────────────────────────┐
│ TIER 1 (FIRST CHOICE): Grounded Markdown (`observe(as="markdown")`)    │
│ • Downsamples page to clean text with inline [#ID] marks.              │
│ • You can immediately call `act(id)` (click) or `act(id, action="type", text="…")`.│
│ • ZERO image tokens burned, 100% hardware trusted (isTrusted: true).    │
│ • Pages large output via `budget` + `next_cursor`. See partition 4.      │
│   Legacy `get_page_markdown()` is a hidden alias of this projection.    │
├────────────────────────────────────────────────────────────────────────┤
│ TIER 2 (FALLBACK): Set-of-Marks (`take_screenshot(marks:true)`)        │
│ • Use ONLY when visual layout, complex maps, or CAPTCHAs require vision.│
│ • Downscaled 640px JPEG (~15x lighter than full PNG).                  │
│ • Mark IDs match the markdown marks identically.                       │
├────────────────────────────────────────────────────────────────────────┤
│ TIER 3 (USE DELIBERATELY): Script Inspection (`evaluate_javascript`)   │
│ • A powerful tool: arbitrary page JS that can read AND mutate.         │
│   Reach for it deliberately — observe/extract cover most reads         │
│   and act covers clicks and typing.                                    │
│ • Heavy injected JS is a bot-detection signal (untrusted events),      │
│   so keep it purposeful, not routine.                                  │
└────────────────────────────────────────────────────────────────────────┘

2. Set-of-Marks (SoM) Visual Grounding

Mark Eleven features a multi-layered Set-of-Marks visual overlay system.

Color Coding

Cyan Badges (#06b6d4): Static structural anchors (navbars, search inputs, headers, primary buttons). These IDs are session-sticky and never scramble on layout shifts.
Neon Yellow Badges (#eab308): Dynamic page content (search results, live table rows, product cards).
Muted Gray Badges (#a1a1aa): Disabled elements (do not click).

Inspection Tools

observe(as="markdown"): Page text with inline [#ID] marks stamped directly on buttons, links, and inputs.
take_screenshot(marks:true, detail="economy"): Downscaled 640px JPEG Set-of-Marks screenshot (~15x lighter than PNG). Recommended for vision.
take_screenshot(marks:true, detail="full"): Full-resolution PNG with numbered badges + marks lookup table.
observe(as="som") (or take_screenshot(marks:true, detail="markdown")): Marks table only with NO image. Ideal for fast coordinate checks or text-only models (~15ms, 0 image tokens).

Interaction

Call act(mark_id, action="click"): Dispatches human-like Bézier cursor movement and hardware click (isTrusted: true). Coordinates are dynamically re-calculated from the live DOM node, tracking layout shifts automatically.
Call act(mark_id, action="type", text="…"): Clicks the input badge and types with realistic Gaussian cadence (isTrusted: true).
No badge? The same tool addresses a target three ways: act(selector="…") or act(x=…, y=…). Mark ids are preferred; selectors and coordinates are escape hatches.

Marks are Refs

The badge index is also the ref consumed by the primitive layer: find(...) returns rows carrying the mark id, and act(mark_id=…) accepts the same number (it re-resolves the live centre and can attach post-conditions). See partition 4 — Orthogonal Primitives & Workflow Recipes for act, wait_for, expect, find, extract, collect_all, page_state/diff, export_har and the context_* profiles.

Off-screen Marks

On long or virtualized lists pass include_offscreen=true to take_screenshot(marks:true) / observe(...) so below-the-fold rows are numbered too — act scrolls the element into view before clicking. To harvest the whole list in one call use collect_all(..., scroll="window").