============================ New Features (2026-05) ============================ Twenty-three additions covering smarter locators, deeper IDE / ops tooling, two new platforms, and a couple of fresh integrations. Every feature ships with a headless Python API, an ``AC_*`` executor command, an ``ac_*`` MCP tool, and (where it makes sense) a Qt GUI tab — same pattern as the rest of the framework. .. contents:: :local: :depth: 2 Locator + selector intelligence =============================== Self-healing locator -------------------- ``image_template → VLM fallback`` with a JSON-lines audit log so flaky locators can be tuned over time:: from je_auto_control import self_heal_click outcome = self_heal_click( template_path="submit.png", description="the green Submit button", ) Executor: ``AC_self_heal_locate / _click / _log_list / _log_clear``. MCP: ``ac_self_heal_*``. GUI: **Self-Healing** tab. Anchor-based locator -------------------- Find element B by spatial relation to anchor A. Anchor + target can use different backends — pick the cheapest one that uniquely identifies each part:: from je_auto_control import ( anchor_locate, image_locator, ocr_locator, ) outcome = anchor_locate( anchor=ocr_locator("Username"), target=image_locator("submit_green.png"), relation="below", ) Relations: ``above``, ``below``, ``left_of``, ``right_of``, ``near``. Executor: ``AC_anchor_locate / _click``. OCR with structured output -------------------------- Cluster raw OCR matches into rows, tables (sets of rows that share column alignment), and form-field ``label:value`` pairs:: from je_auto_control import ocr_read_structure result = ocr_read_structure(region=[0, 0, 1280, 800]) for field in result.fields: print(field.label, "=", field.value) Executor: ``AC_ocr_read_structure``. Smart waits ----------- Frame-diff replacements for ``time.sleep``:: from je_auto_control import wait_until_screen_stable wait_until_screen_stable(timeout_s=10.0, stable_for_s=0.5) Helpers: ``wait_until_screen_stable``, ``wait_until_pixel_changes``, ``wait_until_region_idle``. Executor: ``AC_wait_screen_stable``, ``AC_wait_pixel_changes``, ``AC_wait_region_idle``. A/B locator framework --------------------- Race N strategies for the same target and recommend the historically best one:: from je_auto_control import ab_locate, ab_best_strategy outcome = ab_locate( target_id="submit_button", strategies={ "image": image_locator("submit.png"), "ocr": ocr_locator("Submit"), "vlm": vlm_locator("the green Submit button"), }, ) print("historical best:", ab_best_strategy("submit_button")) Persistent ledger under ``~/.je_auto_control/ab_locator_stats.json``. Executor: ``AC_ab_locate / _report / _best_strategy / _clear``. Operations + observability ========================== Cost telemetry -------------- Per-call LLM token + USD log with day / model / provider roll-up:: from je_auto_control import record_llm_call, summarise_llm_costs record_llm_call( provider="anthropic", model="claude-opus-4-7", input_tokens=512, output_tokens=128, label="vlm_locate", ) summary = summarise_llm_costs() print(summary.total_usd, summary.by_model) Pricing table covers Claude 4.x and OpenAI; override per-call. Executor: ``AC_costs_record / _summary / _list / _clear``. Trace replay UI --------------- Scrubbable timeline over the existing time-travel recordings — load a directory containing ``manifest.json`` + ``actions.jsonl`` and step backwards through frames with the per-step action list alongside. ``TraceReplayController`` ships as a pure-Python class for non-GUI use; the **Trace Replay** GUI tab is a thin shell on top. Failure → ticket automation --------------------------- Fan a failure report out to Jira / Linear / GitHub Issues when a scheduled run, trigger, or REST job blows up:: from je_auto_control import ( FailureReport, GitHubBackend, default_failure_hook_manager, ) default_failure_hook_manager.register( GitHubBackend(owner="acme", repo="ops", token=os.environ["GH_TOKEN"]), ) Executor: ``AC_failure_hook_fire / _list / _clear``. Container CI templates ---------------------- * ``.github/workflows/docker.yml`` — builds the image, runs the headless pytest suite inside it under Xvfb, smoke-tests the REST entrypoint. * ``ci_templates/.gitlab-ci.yml`` — equivalent pipeline for GitLab via Docker-in-Docker. * ``docker/Dockerfile.xfce`` — XFCE4 desktop + x11vnc variant for flows that need a real WM. See ``docs/source/getting_started/run_in_ci.rst`` for the full guide. Cross-host DAG orchestrator --------------------------- Run a DAG where each node carries ``(host, actions | action_file, depends_on)``. Local nodes execute in-process; remote nodes go through the admin-console REST clients. Failures cascade — every downstream node is reported as ``skipped`` instead of attempted:: je_auto_control.run_dag({ "nodes": [ {"id": "step1", "host": "local", "actions": [...]}, {"id": "step2", "host": "machine-a", "action_file": "x.json", "depends_on": ["step1"]}, ], }) Executor: ``AC_run_dag``. GUI: **DAG Runner** tab. Multi-viewer presence --------------------- Roster + controller / observer roles for the multi-viewer remote desktop. Pure-Python ``PresenceRegistry`` ships independently so input-dispatch gating can be unit-tested without aiortc. Executor: ``AC_presence_register / _unregister / _update_cursor / _set_role / _list / _clear``. GUI: **Viewer Roster** tab. Agent + integrations ==================== Computer-use high-level API --------------------------- Wraps :class:`ComputerUseAgentBackend` + :class:`AgentLoop` so a single call drives Anthropic's official ``computer_20250124`` tool:: from je_auto_control import run_computer_use result = run_computer_use( "open Calculator, compute 12 * 7, screenshot the result", max_steps=15, wall_seconds=120.0, ) Auto-detects display size; takes ``max_steps`` + ``wall_seconds`` budgets so a runaway loop can't drain the API. Executor: ``AC_computer_use``. GUI: **Computer Use** tab. WebRunner executor + MCP integration ------------------------------------ Brand-new convenience commands on top of the existing ``je_web_runner`` bridge:: je_auto_control.web_open("https://example.com") je_auto_control.web_screenshot("loaded.png") je_auto_control.web_quit() Executor: ``AC_web_open / _quit / _screenshot / _current_url`` (joining the existing ``AC_web_run``). MCP exposes the same surface as ``ac_web_*``. GUI: **WebRunner** tab. Chat-ops bot ------------ Transport-agnostic ``CommandRouter`` plus a polling Slack adapter so ``/run