Hyperbolic Gateway
Markdown · View source on GitHub

Shadow DOM & iframes — what Hyperbolic Gateway can see and click

Two different problems that get confused because both "hide" content:

surface open shadow DOM closed shadow DOM iframe (same-site and cross-site)
observe(markdown) ✅ composed DOM ❌ ❌ main frame only
Set-of-Marks / find ✅ marks inside roots ✅ find(sealed:true) (coordinates, no mark id) ✅ find(sealed:true) + frames
act(mark_id / selector) ✅ resolves the inner control ❌ (sealed results carry coordinates) ✅ 100+ frame mark ids (find(include_frames:true))
act(target_text) ✅ (composed text) ❌ ❌
act(x, y) ✅ ✅ (pixels are pixels) ✅
frames — text ✅ controls in frames ✅ via find(sealed:true) ✅ frame as content
frames — screenshots ✅ ✅ (pixels are pixels) ✅
focus_walk (Tab) — ✅ trusted entry + AX confirmation —
component signature ✅ find/observe (host surface) ❌ sealed results ❌ frame results
evaluate ✅ ❌ ❌
evaluate {frames:true} ✅ per frame ✅ when the demonstrated effect is inside the root ✅

A control sealed inside a closed root or an iframe is still addressable by pixels. frames computes iframe pixels; find(sealed:true) computes closed-root pixels through the accessibility tree. Everything else is main-document only, on purpose.

Shadow DOM

Discovery

The marks scanner does not use document.querySelectorAll on its own. It collects roots first:

const collectRoots = (root) => {
  roots.push(root);
  for (const el of root.querySelectorAll('*')) {
    if (el.shadowRoot) collectRoots(el.shadowRoot);   // open roots only
  }
};

Every interactive selector then runs in each root, so a <button> inside a web component is a first-class candidate with its own mark id (src/main/marks/marks-overlay.ts).

Resolution

kqDeep(selector) searches the document and every open root. When a selector matches a custom-element host, targetBoxScript resolves it to the control a mouse would actually hit — the inner <input>, not the wrapper.

This matters for typing: focusing a host either moves focus off the inner input or does nothing, and the keystrokes then go nowhere. That failure is documented in FIELD_NOTES_2026-10-20_SIGNUP.md; act now focuses the resolved control and reports what actually received focus.

Hit-testing

document.elementFromPoint() returns the shadow host, never the inner element. A naive occlusion check therefore marks every shadow control as "covered" and drops it. kqAtPoint() walks down through open roots to the deepest hit, and containment checks use kqContainsComposed() — because el.contains() does not cross a shadow boundary.

Reading

observe(as:"markdown") renders the composed DOM: open roots, live form values, passwords as a length only. The envelope reports shadow_roots so you can tell how much of the page came from components.

POST /dom/facts {deep: true} (and the frames tool) use the same root walk, so a web component's inner control appears in control facts and not just in marks. Deep mode also fixes the two things that break inside a root: hit-testing descends via shadowRoot.elementFromPoint, and containment uses a composed walk instead of el.contains().

Closed roots (1.4.91)

find {sealed:true} adds a sealed array to the normal result. It contains closed-root controls with a role, name, rect and actions — no mark id and no input value:

// POST /find {"sealed": true, "contains": "Pay"}
{
  "marks": [ /* open/light DOM refs as before */ ],
  "sealed": [{
    "role": "button",
    "name": "Sealed Pay",
    "rect": { "x": 18, "y": 8, "width": 89, "height": 21 },
    "actions": ["click"],
    "source": "ax",
    "frameId": "…",
    "frameUrl": "https://pay.example/checkout",
    "inViewport": true
  }],
  "sealedTotal": 1,
  "sealedCached": false
}

How it stays honest:

Entering a closed root (focus_walk)

focus_walk presses real Tab/Shift+Tab keys from a light-DOM host and reads the accessibility tree after each step to see which inner node actually has focus:

// POST /focus_walk {"selector": "#checkout-host", "max_steps": 6}
{
  "started": true,
  "landed": true,
  "reason": "left-host",
  "steps": [
    { "index": 1, "role": "button", "name": "Pay now", "rect": { "x": 18, "y": 8, "width": 89, "height": 21 }, "gapMs": 213 },
    { "index": 2, "role": "textbox", "name": "Card number", "rect": { … }, "gapMs": 198 }
  ],
  "medianStepGapMs": 205,
  "restored": false
}

Component signatures (1.4.94)

A mark from find/observe(as:"components") carries the nearest custom-element host's exported surface. The same surface is attached to sealed controls (read from the light-DOM host through CDP) and to frames/frame-mark controls (read in the frame):

{
  "tag": "pay-widget",
  "defined": true,
  "part": ["card"],
  "exportparts": ["button:pay-button", "label:pay-label"],
  "cssVars": ["--brand", "--radius"],
  "signature": "pay-widget:3640e0c6"
}

part, exportparts and the custom-property names are sorted and normalised before the hash, so attribute order and re-renders do not change the signature. A renamed part or a different exported surface does.

Recipes store the signature on a step. Replay verifies it before acting:

Limits: signatures are read from the host, so they cover page-visible hosts. Sealed and iframe controls carry the same surface. partHighlight and cdpStyleMatch are described below.

Limits

Iframes

The boundary

webContents.executeJavaScript runs in the top frame only. An iframe is a separate document with its own globals; a cross-origin one cannot be reached from the parent at all.

evaluateInFrames

evaluateInFrames(wc, expression, { maxFrames?, timeoutMs?, includeBlank? })
  -> Array<{ frameId, url, name, ok, value?, error? }>

Walks the CDP frame tree and evaluates the expression in each frame's own isolated world (src/main/browser/frame-eval.ts). Bounds: 8 frames by default (max 32), 1500 ms per command, isolated-world contexts cached and invalidated on did-start-loading, blank frames skipped, failures reported per frame instead of thrown.

Measured: ~20 ms of frame traversal for a page with one child.

The frame as content

Knowing that an iframe exists is not the same as being able to do anything with it. POST /frames (MCP frames) answers the three questions a blank box cannot:

  1. Where is it? — content-box rect in top-level viewport pixels.
  2. What is inside it? — its controls, described by the same control-facts expression as marks, with their rects already offset into the page.
  3. What does it look like? — optionally, a cropped screenshot per visible frame.
// POST /frames {"include_controls": true, "screenshots": true}
{
  "viewport": { "width": 1280, "height": 601 },
  "frames": [
    { "depth": 0, "url": "https://shop.example/checkout", "rect": { "x": 0, "y": 0, "width": 1280, "height": 601 } },
    {
      "depth": 1,
      "url": "https://pay.example/frame",
      "crossOrigin": true,
      "rect": { "x": 323, "y": 142, "width": 400, "height": 200 },
      "controls": [
        { "tag": "button", "name": "Pay now", "actions": ["click", "hover"],
          "rect": { "x": 331, "y": 222, "width": 89, "height": 21 }, "visible": true, "occluded": false }
      ]
    }
  ],
  "images": [{ "url": "https://pay.example/frame", "width": 600, "height": 300, "mimeType": "image/jpeg" }]
}

The control's rect is a page coordinate, not a frame coordinate: act(x: 376, y: 233) clicks it, and that is verified in the live test — the child document records the click.

How the geometry is derived. CDP gives the frame tree and each frame's owner element (DOM.getFrameOwner). The owner is resolved in its parent's document and measured with getBoundingClientRect(); the parent's own origin is added. That is why it works cross-origin: the iframe is unreachable, but the <iframe> element belongs to the parent, which is not. Because parents come before children in the tree, nested frames compose instead of falling back to "in-frame activation".

Cross-site frames (1.4.90). Chromium omits an out-of-process iframe (a different site, not merely a different origin) from the parent's Page.getFrameTree; it appears only as an iframe target. src/main/browser/frame-tree.ts attaches those targets (Target.attachToTarget({flatten:true})), reads each target's own frame tree, keeps only targets whose parent is a frame in this tab, and detaches the sessions in finally. FrameInfo.oopif marks the frame and targetId names the target; evaluation creates its isolated world on that session. Geometry is still measured through the parent's owner element, so the top-level rect is identical to a same-site frame. The live suite serves the child from 127.0.0.2 while the page stays on 127.0.0.1, keeping the OOPIF path covered.

How the screenshots are made. One viewport capture, then nativeImage.crop() per visible frame (clamped to the viewport). The device-pixel ratio is derived from the captured image rather than assumed, so a 1.5× display yields a 600×300 crop of a 400×200 frame. One capture for N frames is cheaper than N CDP clips, and it avoids the page-vs-viewport coordinate ambiguity in Page.captureScreenshot.

Cost. Opt-in, four CDP commands per frame plus one evaluate. screenshots adds one capture and one crop per visible frame. Both are bounded: 6 frames by default, 16 max.

What uses it

Consent is the first consumer, in stages:

  1. top-document scan (one evaluate — the common case)
  2. only if that finds nothing and the page embeds an <iframe>, the same scan runs in each frame
  3. the match is clicked at top-level coordinates (iframe rect + local centre), so it remains a trusted CDP click
  4. if the offset cannot be derived (nested frame, dynamic src), the control is activated in-frame and the result says delivery: "in-frame" rather than pretending

The isolated-world gotcha

An isolated world shares the DOM but not page globals. window.__anything set by the page is invisible inside it; attributes, text and computed styles are readable. Tests and features must assert on DOM state, not on page variables — this cost a debugging round during Phase 3.

Frame control marks (1.4.97)

find {include_frames:true} stamps and returns child-frame controls with 100+ mark ids (100 + child_index * 100, 99 ids per frame). act(mark_id) and type_mark resolve them live: the element is found by data-mark-id inside its frame, scrolled into view, measured, and the frame's top-level rect is added before the humanized click.

Visual / style provenance (1.4.101)

Limits

What to do today

need approach
read text inside an iframe POST /evaluate {frames:true}
see what an iframe looks like frames {screenshots:true} — one image per frame
list and click a control inside an iframe frames → act(x, y) at the reported control centre
type into an iframe control act(x, y, action:"type", text:…) — the click focuses the frame's field, then keys go there
answer a consent wall in an iframe dismiss_consent — it does the offset math
automate a whole flow inside an iframe frames + coordinate acts; marks are main-frame only

What remains

Remaining work is the Clone Lab proposal (TODO.md item 18), parked by decision — the read/observe/extract/target paths now cover main documents, open/closed roots, same-site frames and OOPIFs.

Force-open is the one detectably risky path: off by default, opt-in per run, confirmed in Settings, and called out by a red chip. See SHADOW_WALK.md.

Coordinates stay the contract for sealed content: an id would imply a DOM handle that does not exist. The delivery plan, the settings levers, and the risk tiers live in SHADOW_WALK.md.

Tests

npm run test:tabwalk is the keyboard fixture:

npm run test:shadow (live) and npm run test:shadow:unit cover the settings gate: defaults, clamping, corrupt-file fallback, the KUANG_SHADOW_WALK=off kill switch, every structured refusal, the master switch, output suppression and cap enforcement.

npm run test:dom covers the general control/frame surface:

npm run test:sealed is the closed-root fixture:

Settings gate

The Shadow Walk levers are enforced before work starts, not after:

surface disabled result
find {sealed:true} 400 SEALED_DISABLED (or SHADOW_WALK_DISABLED when the master is off)
focus_walk 400 TAB_WALK_DISABLED
/frames, frame geometry 400 FRAMES_DISABLED
/dom/facts {deep:true} 400 DEEP_OPEN_ROOTS_DISABLED
marks / find / observe deepOpenRoots:false keeps them in the light DOM (no error)

The panel lives in Settings → SHADOW WALK; writes go over IPC to data/shadow-walk.json. /health returns a shadowWalk block with the active policy, so an agent can check before asking. Turning frame controls or screenshots off returns frames without those fields and a notes line, rather than silently ignoring the request.

See also