English Documentation

Comprehensive guides for all AutoControl features.

User Guide

• Installation
  • Install from PyPI
  • Requirements
  • With GUI Support
  • Linux Prerequisites
  • Raspberry Pi
  • Development Environment
• Mouse Control
  • Getting Mouse Button Table
  • Press and Release
  • Click
  • Position
  • Scroll
• Keyboard Control
  • Getting Key Tables
  • Press and Release
  • Type a Single Key
  • Check Key State
  • Type a String
  • Hotkey Combinations
• Screen Operations
  • Screenshot
  • Screen Size
  • Get Pixel Color
• Image Recognition
  • Locate All Matches
  • Locate Image Center
  • Locate and Click
  • Parameters
• Recording & Playback
  • Record and Replay
  • How It Works
  • Editing a recording
• Keywords & Executor
  • Keyword Format
  • Available Action Commands
  • Executing a JSON File
  • Executing All JSON Files in a Directory
  • Extending the Executor
• Report Generation
  • Setup
  • Generating Reports
  • Getting Report Content as String
  • Report Contents
• Callback Executor
  • Basic Usage
  • How It Works
  • Parameters
  • Extending the Callback Executor
• Scheduler
  • Basic Example
  • Blocking vs Non-Blocking
  • Interval Scheduling
  • Cron Scheduling
  • Removing Jobs
  • Shutting Down
• Socket Server (Remote API)
  • Starting the Server
  • Sending Commands (Client)
  • Protocol Details
• MCP Server (Use AutoControl from Claude)
  • Tool catalogue
  • Resources, prompts, sampling
  • Logging notifications, progress, cancellation
  • Starting the server (programmatic)
  • Starting the server (command line)
  • CLI inspection flags
  • Registering with Claude Desktop
  • Registering with Claude Code
  • HTTP transport (with SSE, auth, TLS)
  • Read-only / safe mode
  • Confirmation prompts (elicitation)
  • Audit log
  • Auto-screenshot on tool error
  • Rate limiting
  • Plugin hot-reload
  • CI smoke tests with the fake backend
  • Security notes
• Critical Exit
  • Enabling Critical Exit
  • Changing the Hotkey
  • Example: Recovering from Runaway Mouse
• Command-Line Interface
  • Subcommand CLI (python -m je_auto_control.cli)
  • Legacy flag-style CLI (python -m je_auto_control)
• Project Management
  • Creating a Project
  • Generated Structure
• New Features (2026-04)
  • Clipboard
  • Dry-run / step-debug executor
  • Global hotkey daemon (Windows)
  • Event triggers
  • Cron scheduling
  • Plugin loader
  • REST API server
  • CLI runner
  • Multi-language GUI (i18n)
  • Closable tabs + menu bar
  • OCR (text on screen)
  • Accessibility element finder
  • VLM (AI) element locator
  • Run history + error-snapshot artifacts
  • OCR — region dump and regex search
  • Runtime variables and data-driven control flow
  • LLM action planner
  • Remote desktop (host + viewer)
  • Remote desktop — Quick Connect + Phase 4/5 hardening
  • Remote desktop — secure transports, audio, clipboard, file transfer
  • Remote desktop — AnyDesk-style popout window
  • Remote desktop — responsive sub-tab sizing
  • Remote desktop — MCP tool surface
  • Driver-level input backends — drive games that ignore SendInput / XTest
  • Per-action profiler
  • Run history timeline + failure thumbnails
  • Encrypted secret manager
  • Webhook (HTTP push) trigger
  • IMAP email trigger
• New Features (2026-05)
  • Locator + selector intelligence
  • Operations + observability
  • Agent + integrations
  • Platform coverage
  • Developer experience
• New Features (2026-06) — QA Layer
  • Assertions
  • Data-driven execution
  • Flaky-test detection & quarantine
  • QA suite runner + CI reports
  • Accessibility & i18n audit
  • Mobile device matrix
• New Features (2026-06-17) — Automation Toolkit
  • Human-like input
  • Vision
  • Flow control & variables
  • Triggers & smart waits
  • Window management
  • File security & safety
  • Reporting & notifications
  • GUI
• New Features (2026-06-18) — CLI & Integrations
  • Command-line interface
  • Code generation
  • HTTP / API
  • SQL
  • Email (SMTP)
  • PDF
  • Smart waits
  • Security
• New Features (2026-06-19) — Visual Regression & FSM
  • Visual regression (golden images)
  • Finite-state machine
• New Features (2026-06-19) — Native UI Control
  • Reading and setting values
  • Invoking and toggling
  • Reading tables / grids
  • Targeting controls
  • Platforms
• New Features (2026-06-19) — Popup Watchdog
  • Quick start
  • Custom rules
  • Executor commands
• New Features (2026-06-19) — Unattended Reliability
  • OTP / TOTP for 2FA logins
  • Native file dialogs
  • Locked-session guard
• New Features (2026-06-19) — Transactional Queue
  • Dispatcher / performer
  • Failure semantics
  • Executor commands
• New Features (2026-06-19) — Test & Tooling Batch
  • Synthetic test data
  • MCP registry manifest
  • Risk-based test selection
• New Features (2026-06-19) — Authoring & Debugging
  • Element repository
  • Step debugger and tracer
• New Features (2026-06-19) — Agent Toolkit
  • Skill / playbook library
  • Prompt-injection guardrail
  • A2A agent card
• New Features (2026-06-19) — Office I/O
  • Excel
  • Word
  • PowerPoint
• New Features (2026-06-19) — Memory & Determinism
  • Agent episodic memory
  • Deterministic run
• New Features (2026-06-19) — WCAG 2.2 Audit Engine
  • Conformance audit
  • Target Size rule
• New Features (2026-06-19) — Reactive Observer
  • Python API
  • Executor / MCP commands
• New Features (2026-06-19) — SBOM & Suite Sharding
  • CycloneDX SBOM
  • Duration-aware suite sharding
• New Features (2026-06-19) — Data Quality
  • Row schema validation
  • Field extraction
  • Row masking
• New Features (2026-06-19) — i18n / l10n Testing
  • Pseudo-localization
  • Text-overflow detection
  • Catalog completeness
• New Features (2026-06-19) — Checkpoint & Resume
  • Flow checkpoint & resume
  • py.typed marker
• New Features (2026-06-19) — Set-of-Marks Overlay
  • Numbering and the legend
  • Live “mark then click” loop
• New Features (2026-06-19) — Semantic Screen State
  • Snapshot & diff
  • Describe the screen
• New Features (2026-06-19) — Timed Input Macros
  • Timed timeline replay
  • Input-sequence DSL
• New Features (2026-06-19) — Resilience Primitives
  • RetryPolicy
  • CircuitBreaker
• New Features (2026-06-19) — CI Annotations & Clipboard History
  • CI workflow annotations
  • Clipboard history
• New Features (2026-06-19) — Heal Analytics & Secret Scan
  • Self-heal analytics
  • Secret scan
• New Features (2026-06-19) — Process-Doc (SOP) Generator
  • Usage
• New Features (2026-06-19) — Tweened Drag
  • Usage
• New Features (2026-06-19) — MCP Structured Tool Output
  • Declaring an output schema
• New Features (2026-06-19) — Plugin SDK
  • Authoring a plugin
  • Discovering & loading
• Maker-Checker Approval Gate
  • Headless API
  • Executor commands
• Just-In-Time Credential Leases
  • Headless API
  • Executor commands
• Network Egress Allowlist Guard
  • Headless API
  • Executor commands
• Approval Testing (Golden-Master Baselines)
  • Headless API
  • Executor commands
• Agent Trajectory Evaluation
  • Headless API
  • Executor command
• Compliance Control Report (SOC2 / ISO 27001)
  • Mapped controls
  • Headless API
  • Executor command
• Agent Observability (GenAI OpenTelemetry Spans)
  • Headless API
  • Executor commands
• Video Step-Overlay Report
  • Headless API
  • Executor command
• Fuzzy String Matching & Dedupe
  • Headless API
  • Executor commands
• S3-Compatible Artifact Store
  • Headless API
  • Executor commands
• Perceptual-Hash Image Dedupe
  • Headless API
  • Executor commands
• Locale-Aware Number, Currency & Date Parsing
  • Headless API
  • Executor commands
• Voice-Command Router
  • Headless API
  • Executor commands
• Coordinate-Space Mapping (Model Grid ⇄ Physical Pixels)
  • Headless API
  • Executor commands
• Stuck-Loop Guard (Agent Loop Progress Detection)
  • Headless API
  • Executor commands
• Task / Process Mining (Automation-Candidate Discovery)
  • Headless API
  • Executor command
• Environment-Scoped Typed Asset Store
  • Headless API
  • Executor commands
• Outbound CloudEvents Emitter
  • Headless API
  • Executor command
• Multi-Channel Webhook Notifications
  • Headless API
  • Executor command
• JSONPath Querying
  • Headless API
  • Executor commands
• Saga / Compensating Rollback
  • Headless API
  • Executor command
• DMN-Style Decision Tables
  • Hit policies
  • Headless API
  • Executor command
• Self-Healing Locator Write-Back
  • Headless API
  • Executor commands
• Text PII Detection & Redaction
  • Headless API
  • Executor commands
• SARIF 2.1.0 Findings Export
  • Headless API
  • Executor command
• JSON Schema Validation
  • Supported keywords
  • Headless API
  • Executor command
• Dependency Vulnerability Scanning (OSV)
  • Headless API
  • Live advisories (optional)
  • Executor command
• OpenVEX Vulnerability Triage
  • Headless API
  • Executor command
• License Policy Gate
  • Headless API
  • Executor command
• JSON Web Tokens (JWT)
  • Security
  • Headless API
  • Executor commands
• Client-Side Rate Limiting
  • Headless API
  • Executor command
• JSON Pointer, Patch & Merge Patch
  • Headless API
  • Executor commands
• Full-Text Search (BM25)
  • Headless API
  • Executor command
• Statistics & A/B Significance
  • Headless API
  • Executor commands
• Calendar Recurrence Rules (RRULE)
  • Headless API
  • Executor commands
• Text Diff, Patch & Three-Way Merge
  • Headless API
  • Executor commands
• Feature Flags
  • Headless API
  • Executor commands
• SLSA Build Provenance
  • Headless API
  • Executor commands
• JSON Contract & Snapshot Matching
  • Headless API
  • Executor commands
• Chaos Experiments
  • Headless API
  • Executor command
• Service-Level Objectives (SLO)
  • Headless API
  • Executor commands
• Streaming Latency Percentiles
  • Headless API
  • Executor command
• Bulkhead & Rate-Limit Headers
  • Headless API
  • Executor commands
• HTTP Record & Replay Cassette
  • Headless API
  • Executor command
• W3C Trace Context Propagation
  • Headless API
  • Executor commands
• Data Profiling & Schema Inference
  • Headless API
  • Executor commands
• RFC 9457 Problem Details Parsing
  • Headless API
  • Executor command
• Dotenv (.env) Parsing
  • Headless API
  • Executor commands
• Server-Sent Events (SSE) Client Parser
  • Headless API
  • Executor command
• Layered Configuration Resolver
  • Headless API
  • Executor commands
• Distribution Drift Detection
  • Headless API
  • Executor commands
• Dataset Diff (Row-Set Change Report)
  • Headless API
  • Executor commands
• W3C Baggage Propagation
  • Headless API
  • Executor commands
• URI-Scheme Value References
  • Headless API
  • Executor commands
• Referential Integrity Checks
  • Headless API
  • Executor commands
• RFC 8288 Link Header & Pagination
  • Headless API
  • Executor commands
• Secret Redaction for Config & Logs
  • Headless API
  • Executor commands
• multipart/form-data Build & Parse
  • Headless API
  • Executor commands
• HTTP Content Negotiation & Decompression
  • Headless API
  • Executor commands
• Cookie Jar (HTTP Session Carry)
  • Headless API
  • Executor commands
• Conditional HTTP Requests & Cache Validators
  • Headless API
  • Executor commands
• Canonical Log Lines & Structured Logging
  • Headless API
  • Executor command
• OTLP/JSON Span Export
  • Headless API
  • Executor command
• Typed Configuration Schema
  • Headless API
  • Executor command
• JSON-Schema Compatibility Checking
  • Headless API
  • Executor command
• Unicode Text Normalisation & Slugify
  • Headless API
  • Executor commands
• Time-Series Transforms
  • Headless API
  • Executor commands
• String-Distance Similarity Metrics
  • Headless API
  • Executor command
• Near-Duplicate Text Detection (SimHash / MinHash)
  • Headless API
  • Executor commands
• Single-Series Anomaly Detection
  • Headless API
  • Executor command
• Moving-Average Smoothing
  • Headless API
  • Executor commands
• Idempotency-Key Store
  • Headless API
  • Executor commands
• Time-Windowed Deduplication
  • Headless API
  • Executor command
• Per-Stream Sequence-Gap Detection
  • Headless API
  • Executor command
• Optimistic-Concurrency Versioned Store
  • Headless API
  • Executor commands
• Transactional Outbox
  • Headless API
  • Executor commands
• Locale-Aware String Collation
  • Headless API
  • Executor commands
• Confusable / Homoglyph Detection
  • Headless API
  • Executor commands
• Readability Scoring
  • Headless API
  • Executor commands
• Bidirectional-Text QA (Trojan-Source Scan)
  • Headless API
  • Executor commands
• Locale-Aware List Formatting
  • Headless API
  • Executor commands
• ICU-lite MessageFormat (Plural / Select)
  • Headless API
  • Executor commands
• GNU gettext Catalog I/O (.po / .mo)
  • Headless API
  • Executor commands
• Check-Digit Algorithms
  • Headless API
  • Executor commands
• Multi-Waypoint Mouse Gestures
  • Headless API
  • Executor commands
• Clear-Then-Type Field Entry
  • Headless API
  • Executor commands
• Wait Until Gone (Blocking Vanish Waits)
  • Headless API
  • Executor commands
• Hold Key / Auto-Repeat
  • Headless API
  • Executor commands
• Relative Mouse Movement
  • Headless API
  • Executor commands
• Wait for Region Colour
  • Headless API
  • Executor commands
• Unicode Text Entry (Emoji / CJK)
  • Headless API
  • Executor commands
• Held Modifiers Across an Action Group
  • Headless API
  • Executor commands
• Anchor Ordinal & Locate-All
  • Headless API
  • Executor commands
• Grid / Table Cell Addressing
  • Headless API
  • Executor commands
• Wait for Window Title (Regex)
  • Headless API
  • Executor commands
• Confidence-Returning Template Matching
  • Headless API
  • Executor commands
• Locate On-Screen Regions by Colour
  • Headless API
  • Executor commands
• Masked Template Matching (Ignore the Background)
  • Headless API
  • Executor commands
• Structural-Similarity (SSIM) Comparison
  • Headless API
  • Executor commands
• ORB Feature Matching (Rotation / Scale / Theme Robust)
  • Headless API
  • Executor command
• Locate UI Elements by Edge / Contour (No Template)
  • Headless API
  • Executor commands
• Window Tiling / Layout Geometry Planner
  • Headless API
  • Executor commands
• Arrange Multiple Windows (Grid / Cascade)
  • Headless API
  • Executor commands
• Image Pre-processing for OCR / Template Matching
  • Headless API
  • Executor command
• Multi-Monitor / Virtual-Desktop Geometry
  • Headless API
  • Executor commands
• Actionability Gate — Wait Until Ready Before Acting
  • Headless API
  • Executor command
• Fuse & Order On-Screen Element Boxes
  • Headless API
  • Executor commands
• HSV Colour-Space Segmentation
  • Headless API
  • Executor commands
• Model-Free Text-Region Detection (MSER)
  • Headless API
  • Executor commands
• Line / Grid / Separator Detection (Hough)
  • Headless API
  • Executor commands
• Retrying Value Assertions (expect.poll)
  • Headless API
  • Executor command
• Composable / Filtered Candidate Locators
  • Headless API
  • Executor command
• Rich Clipboard — HTML (CF_HTML)
  • Headless API
  • Executor commands
• Colour-Histogram Fingerprint & Change Detection
  • Headless API
  • Executor commands
• Localized Motion / Activity Detection
  • Headless API
  • Executor commands
• Window Z-Order — Always-On-Top / Front / Back
  • Headless API
  • Executor commands
• Soft Assertions — Aggregate Failures at Block End
  • Headless API
  • Executor command
• Perceptual (YIQ) Image Diff with Anti-Alias Suppression
  • Headless API
  • Executor command
• Window Client-Area Geometry
  • Headless API
  • Executor commands
• Canonical Computer-Use Action Schema
  • Headless API
  • Executor command
• Token-Budgeted A11y Text Observation
  • Headless API
  • Executor commands
• Pre-Action Grounding Guard
  • Headless API
  • Executor command
• Portable Agent-Trajectory Trace (Record & Replay)
  • Headless API
  • Executor command
• Geometry-Aware Element Diff & Stable IDs
  • Headless API
  • Executor commands
• Weighted Candidate Scoring
  • Headless API
  • Executor commands
• Barcode Decoding (1-D)
  • Headless API
  • Executor command
• Rotation- and Scale-Tolerant Template Matching
  • Headless API
  • Executor commands
• Coarse Labelled Screen Grid (VLM Grounding)
  • Headless API
  • Executor commands
• Clipboard File-Drop List (CF_HDROP)
  • Headless API
  • Executor commands
• Trust-Scored Template Matching (Ambiguity / PSR)
  • Headless API
  • Executor command
• Fill a Ruling-Line Grid With OCR Text (Addressable Tables)
  • Headless API
  • Executor command
• Token-Budgeted Observation Delta (What Changed)
  • Headless API
  • Executor command
• Auto-Thresholded Template Matching (Otsu on the Score Map)
  • Headless API
  • Executor commands
• Whitespace-Projection Columns (Borderless Tables)
  • Headless API
  • Executor commands
• Form Field Association (Multi-Direction) + Checkbox State
  • Headless API
  • Executor commands
• Action-Effect Classification (Did My Click Do Anything?)
  • Headless API
  • Executor commands
• Edge-Shape (Chamfer) Template Matching
  • Headless API
  • Executor commands
• Declarative Action Postconditions
  • Headless API
  • Executor command
• Repair-Tactic Policy for Failed / No-Effect Actions
  • Headless API
  • Executor command
• Sub-Pixel Template-Match Refinement
  • Headless API
  • Executor command
• Grounding Self-Consistency (Consensus Over Proposals)
  • Headless API
  • Executor commands
• Column-Aware Reading Order (XY-Cut)
  • Headless API
  • Executor commands
• Paragraph & List Grouping of OCR Lines
  • Headless API
  • Executor commands
• Settle Detection Over a Churn Series
  • Headless API
  • Executor command
• Heading vs Body Classification + Document Outline
  • Headless API
  • Executor commands
• Per-Step Critic Features + Rule-Based Step Scorer
  • Headless API
  • Executor commands
• Multi-Template Consensus Matching
  • Headless API
  • Executor commands
• Colour-Aware Template Matching (HSV)
  • Headless API
  • Executor commands
• Pre-Match Settle Gating + Match Persistence
  • Headless API
  • Executor commands
• Extended UIA Control Patterns (Expand / Select / Range / Scroll)
  • Headless API
  • Executor commands
• OCR backends
  • Backend selection
  • Language codes
  • Tesseract setup notes
  • EasyOCR / PaddleOCR setup notes
  • Headless usage
  • JSON action surface
  • Diagnostics
• Observability — metrics and traces
  • Quick start
  • Built-in metrics
  • Adding your own metrics
  • Tracing
  • Production deployment
  • Security note
• Operations & Admin Layer
  • Folder sync (additive mirror)
  • coturn TURN config bundle
  • Hardened REST API
  • Prometheus metrics
  • Multi-host admin console
  • Audit log hash chain
  • WebRTC packet inspector
  • USB device enumeration
  • USB hotplug events
  • System diagnostics
  • Web admin dashboard
  • OpenAPI 3.1 + Swagger UI
  • Configuration bundle
• USB Passthrough — Phase 2 Design
  • Goals
  • Non-Goals
  • Transport
  • Per-OS driver wrappers
  • Security & ACL
  • Phasing
  • Design decisions (formerly open questions)
• USB Passthrough — Phase 2e Security Review Checklist
  • Threat model
  • ACL
  • Audit
  • Protocol hardening
  • Resource bounds
  • Lifecycle
  • Per-OS requirements
  • Pen-test scenarios
  • Sign-off
• USB Passthrough — Operator Guide
  • Prerequisites
  • Driver setup (per OS)
  • Enabling the feature
  • ACL setup
  • Starting the host
  • Viewer-side: claim and transfer
  • Troubleshooting matrix
  • Headless control (AC_usb_* commands)
  • What is not shipped yet