AutoControl
AutoControl is a cross-platform Python GUI automation framework providing mouse control, keyboard input, image recognition, screen capture, action scripting, and report generation – all through a unified API that works on Windows, macOS, and Linux (X11).
Note
AutoControl supports Linux Wayland via CLI bridges (wtype + ydotool +
grim). See Running AutoControl in CI / Cloud for compositor / install
notes. Set JE_AUTOCONTROL_LINUX_DISPLAY_SERVER=x11 to force the
XWayland fallback.
Getting Started
User Guide (English)
- English Documentation
- Installation
- Mouse Control
- Keyboard Control
- Screen Operations
- Image Recognition
- Recording & Playback
- Keywords & Executor
- Report Generation
- Callback Executor
- Scheduler
- Socket Server (Remote API)
- MCP Server (Use AutoControl from Claude)
- Critical Exit
- Command-Line Interface
- Project Management
- New Features (2026-04)
- New Features (2026-05)
- New Features (2026-06) — QA Layer
- New Features (2026-06-17) — Automation Toolkit
- New Features (2026-06-18) — CLI & Integrations
- New Features (2026-06-19) — Visual Regression & FSM
- New Features (2026-06-19) — Native UI Control
- New Features (2026-06-19) — Popup Watchdog
- New Features (2026-06-19) — Unattended Reliability
- New Features (2026-06-19) — Transactional Queue
- New Features (2026-06-19) — Test & Tooling Batch
- New Features (2026-06-19) — Authoring & Debugging
- New Features (2026-06-19) — Agent Toolkit
- New Features (2026-06-19) — Office I/O
- New Features (2026-06-19) — Memory & Determinism
- New Features (2026-06-19) — WCAG 2.2 Audit Engine
- New Features (2026-06-19) — Reactive Observer
- New Features (2026-06-19) — SBOM & Suite Sharding
- New Features (2026-06-19) — Data Quality
- New Features (2026-06-19) — i18n / l10n Testing
- New Features (2026-06-19) — Checkpoint & Resume
- New Features (2026-06-19) — Set-of-Marks Overlay
- New Features (2026-06-19) — Semantic Screen State
- New Features (2026-06-19) — Timed Input Macros
- New Features (2026-06-19) — Resilience Primitives
- New Features (2026-06-19) — CI Annotations & Clipboard History
- New Features (2026-06-19) — Heal Analytics & Secret Scan
- New Features (2026-06-19) — Process-Doc (SOP) Generator
- New Features (2026-06-19) — Tweened Drag
- New Features (2026-06-19) — MCP Structured Tool Output
- New Features (2026-06-19) — Plugin SDK
- Maker-Checker Approval Gate
- Just-In-Time Credential Leases
- Network Egress Allowlist Guard
- Approval Testing (Golden-Master Baselines)
- Agent Trajectory Evaluation
- Compliance Control Report (SOC2 / ISO 27001)
- Agent Observability (GenAI OpenTelemetry Spans)
- Video Step-Overlay Report
- Fuzzy String Matching & Dedupe
- S3-Compatible Artifact Store
- Perceptual-Hash Image Dedupe
- Locale-Aware Number, Currency & Date Parsing
- Voice-Command Router
- Coordinate-Space Mapping (Model Grid ⇄ Physical Pixels)
- Stuck-Loop Guard (Agent Loop Progress Detection)
- Task / Process Mining (Automation-Candidate Discovery)
- Environment-Scoped Typed Asset Store
- Outbound CloudEvents Emitter
- Multi-Channel Webhook Notifications
- JSONPath Querying
- Saga / Compensating Rollback
- DMN-Style Decision Tables
- Self-Healing Locator Write-Back
- Text PII Detection & Redaction
- SARIF 2.1.0 Findings Export
- JSON Schema Validation
- Dependency Vulnerability Scanning (OSV)
- OpenVEX Vulnerability Triage
- License Policy Gate
- JSON Web Tokens (JWT)
- Client-Side Rate Limiting
- JSON Pointer, Patch & Merge Patch
- Full-Text Search (BM25)
- Statistics & A/B Significance
- Calendar Recurrence Rules (RRULE)
- Text Diff, Patch & Three-Way Merge
- Feature Flags
- SLSA Build Provenance
- JSON Contract & Snapshot Matching
- Chaos Experiments
- Service-Level Objectives (SLO)
- Streaming Latency Percentiles
- Bulkhead & Rate-Limit Headers
- HTTP Record & Replay Cassette
- W3C Trace Context Propagation
- Data Profiling & Schema Inference
- RFC 9457 Problem Details Parsing
- Dotenv (.env) Parsing
- Server-Sent Events (SSE) Client Parser
- Layered Configuration Resolver
- Distribution Drift Detection
- Dataset Diff (Row-Set Change Report)
- W3C Baggage Propagation
- URI-Scheme Value References
- Referential Integrity Checks
- RFC 8288 Link Header & Pagination
- Secret Redaction for Config & Logs
- multipart/form-data Build & Parse
- HTTP Content Negotiation & Decompression
- Cookie Jar (HTTP Session Carry)
- Conditional HTTP Requests & Cache Validators
- Canonical Log Lines & Structured Logging
- OTLP/JSON Span Export
- Typed Configuration Schema
- JSON-Schema Compatibility Checking
- Unicode Text Normalisation & Slugify
- Time-Series Transforms
- String-Distance Similarity Metrics
- Near-Duplicate Text Detection (SimHash / MinHash)
- Single-Series Anomaly Detection
- Moving-Average Smoothing
- Idempotency-Key Store
- Time-Windowed Deduplication
- Per-Stream Sequence-Gap Detection
- Optimistic-Concurrency Versioned Store
- Transactional Outbox
- Locale-Aware String Collation
- Confusable / Homoglyph Detection
- Readability Scoring
- Bidirectional-Text QA (Trojan-Source Scan)
- Locale-Aware List Formatting
- ICU-lite MessageFormat (Plural / Select)
- GNU gettext Catalog I/O (.po / .mo)
- Check-Digit Algorithms
- Multi-Waypoint Mouse Gestures
- Clear-Then-Type Field Entry
- Wait Until Gone (Blocking Vanish Waits)
- Hold Key / Auto-Repeat
- Relative Mouse Movement
- Wait for Region Colour
- Unicode Text Entry (Emoji / CJK)
- Held Modifiers Across an Action Group
- Anchor Ordinal & Locate-All
- Grid / Table Cell Addressing
- Wait for Window Title (Regex)
- Confidence-Returning Template Matching
- Locate On-Screen Regions by Colour
- Masked Template Matching (Ignore the Background)
- Structural-Similarity (SSIM) Comparison
- ORB Feature Matching (Rotation / Scale / Theme Robust)
- Locate UI Elements by Edge / Contour (No Template)
- Window Tiling / Layout Geometry Planner
- Arrange Multiple Windows (Grid / Cascade)
- Image Pre-processing for OCR / Template Matching
- Multi-Monitor / Virtual-Desktop Geometry
- Actionability Gate — Wait Until Ready Before Acting
- Fuse & Order On-Screen Element Boxes
- HSV Colour-Space Segmentation
- Model-Free Text-Region Detection (MSER)
- Line / Grid / Separator Detection (Hough)
- Retrying Value Assertions (expect.poll)
- Composable / Filtered Candidate Locators
- Rich Clipboard — HTML (CF_HTML)
- Colour-Histogram Fingerprint & Change Detection
- Localized Motion / Activity Detection
- Window Z-Order — Always-On-Top / Front / Back
- Soft Assertions — Aggregate Failures at Block End
- Perceptual (YIQ) Image Diff with Anti-Alias Suppression
- Window Client-Area Geometry
- Canonical Computer-Use Action Schema
- Token-Budgeted A11y Text Observation
- Pre-Action Grounding Guard
- Portable Agent-Trajectory Trace (Record & Replay)
- Geometry-Aware Element Diff & Stable IDs
- Weighted Candidate Scoring
- Barcode Decoding (1-D)
- Rotation- and Scale-Tolerant Template Matching
- Coarse Labelled Screen Grid (VLM Grounding)
- Clipboard File-Drop List (CF_HDROP)
- Trust-Scored Template Matching (Ambiguity / PSR)
- Fill a Ruling-Line Grid With OCR Text (Addressable Tables)
- Token-Budgeted Observation Delta (What Changed)
- Auto-Thresholded Template Matching (Otsu on the Score Map)
- Whitespace-Projection Columns (Borderless Tables)
- Form Field Association (Multi-Direction) + Checkbox State
- Action-Effect Classification (Did My Click Do Anything?)
- Edge-Shape (Chamfer) Template Matching
- Declarative Action Postconditions
- Repair-Tactic Policy for Failed / No-Effect Actions
- Sub-Pixel Template-Match Refinement
- Grounding Self-Consistency (Consensus Over Proposals)
- Column-Aware Reading Order (XY-Cut)
- Paragraph & List Grouping of OCR Lines
- Settle Detection Over a Churn Series
- Heading vs Body Classification + Document Outline
- Per-Step Critic Features + Rule-Based Step Scorer
- Multi-Template Consensus Matching
- Colour-Aware Template Matching (HSV)
- Pre-Match Settle Gating + Match Persistence
- Extended UIA Control Patterns (Expand / Select / Range / Scroll)
- OCR backends
- Observability — metrics and traces
- Operations & Admin Layer
- USB Passthrough — Phase 2 Design
- USB Passthrough — Phase 2e Security Review Checklist
- USB Passthrough — Operator Guide
User Guide (繁體中文)
- 繁體中文文件
- 安裝
- 滑鼠控制
- 鍵盤控制
- 螢幕操作
- 圖片辨識
- 錄製與回放
- 關鍵字與執行者
- 報告產生
- 回調函數執行器
- 排程器
- Socket 伺服器（遠端 API）
- MCP 伺服器 (讓 Claude 使用 AutoControl)
- 緊急退出
- 命令列介面
- 專案管理
- 新功能 (2026-04)
- 新功能 (2026-05)
- 新功能 (2026-06) — 測試框架層
- 新功能 (2026-06-17) — 自動化工具箱
- 新功能 (2026-06-18) — CLI 與整合
- 新功能 (2026-06-19) — 視覺回歸與狀態機
- 新功能 (2026-06-19) — 原生 UI 控制
- 新功能 (2026-06-19) — 彈窗看門狗
- 新功能 (2026-06-19) — 無人值守可靠性
- 新功能 (2026-06-19) — 交易式工作佇列
- 新功能 (2026-06-19) — 測試與工具三件套
- 新功能 (2026-06-19) — 編寫與除錯
- 新功能 (2026-06-19) — Agent 工具組
- 新功能 (2026-06-19) — Office 讀寫
- 新功能 (2026-06-19) — 記憶與決定性
- 新功能 (2026-06-19) — WCAG 2.2 稽核引擎
- 新功能 (2026-06-19) — 反應式觀察器
- 新功能 (2026-06-19) — SBOM 與測試分片
- 新功能 (2026-06-19) — 資料品質
- 新功能 (2026-06-19) — i18n / l10n 測試
- 新功能 (2026-06-19) — 檢查點與續跑
- 新功能 (2026-06-19) — Set-of-Marks 疊圖
- 新功能 (2026-06-19) — 語意螢幕狀態
- 新功能 (2026-06-19) — 計時輸入巨集
- 新功能 (2026-06-19) — 韌性原語
- 新功能 (2026-06-19) — CI 註解與剪貼簿歷史
- 新功能 (2026-06-19) — 修復分析與機密掃描
- 新功能 (2026-06-19) — 流程文件(SOP)產生器
- 新功能 (2026-06-19) — 緩動拖曳
- 新功能 (2026-06-19) — MCP 結構化工具輸出
- 新功能 (2026-06-19) — Plugin SDK
- Maker-Checker 審批閘門
- 即時(Just-In-Time)憑證租約
- 網路出口允許清單守衛
- 核准式測試(Golden-Master 基準)
- Agent 軌跡評估
- 合規控制報告(SOC2 / ISO 27001)
- Agent 可觀測性(GenAI OpenTelemetry Spans)
- 影片步驟疊加報告
- 模糊字串比對與去重
- S3 相容成品儲存
- 感知雜湊影像去重
- 區域設定感知的數字、貨幣與日期解析
- 語音指令路由器
- 座標空間對映(模型網格 ⇄ 實體像素)
- 卡迴圈守衛(Agent Loop 進度偵測)
- 任務 / 流程探勘(自動化候選發現)
- 環境範圍的具型別資產儲存
- 對外 CloudEvents 發送器
- 多通道 Webhook 通知
- JSONPath 查詢
- Saga / 補償回溯
- DMN 式決策表
- 自我修復定位器回寫
- 文字 PII 偵測與遮蔽
- SARIF 2.1.0 發現項目匯出
- JSON Schema 驗證
- 相依套件漏洞掃描(OSV)
- OpenVEX 漏洞分級
- 授權政策閘門
- JSON Web Token(JWT)
- 用戶端速率限制
- JSON Pointer、Patch 與 Merge Patch
- 全文搜尋(BM25)
- 統計與 A/B 顯著性
- 行事曆週期規則(RRULE)
- 文字 Diff、套用與三方合併
- 功能旗標(Feature Flags)
- SLSA 建置來源證明(Provenance)
- JSON 合約與快照比對
- 混沌實驗(Chaos Experiments)
- 服務等級目標(SLO)
- 串流延遲百分位
- 隔艙與速率限制標頭
- HTTP 錄製與重播卡帶
- W3C Trace Context 傳播
- 資料剖析與結構推斷
- RFC 9457 Problem Details 解析
- Dotenv(.env)解析
- Server-Sent Events(SSE)用戶端解析器
- 分層設定解析器
- 分布漂移偵測
- 資料集差異(資料列變更報告)
- W3C Baggage 傳播
- URI-Scheme 值參照
- 參照完整性檢查
- RFC 8288 Link 標頭與分頁
- 設定與日誌的機密遮蔽
- multipart/form-data 建立與解析
- HTTP 內容協商與解壓縮
- Cookie Jar(HTTP 工作階段攜帶)
- 條件式 HTTP 請求與快取驗證子
- 標準日誌行與結構化日誌
- OTLP/JSON Span 匯出
- 具型別的設定結構
- JSON-Schema 相容性檢查
- Unicode 文字正規化與 Slug
- 時間序列轉換
- 字串距離相似度量
- 近似重複文字偵測(SimHash / MinHash)
- 單序列異常偵測
- 移動平均平滑
- 冪等鍵儲存
- 時間視窗去重
- 逐串流序號間隙偵測
- 樂觀並行版本儲存
- 交易型 Outbox
- 地區感知字串排序(Collation)
- 易混淆字元 / 同形異義字偵測
- 可讀性評分
- 雙向文字 QA(Trojan-Source 掃描)
- 地區感知清單格式化
- ICU-lite MessageFormat(複數 / 選擇)
- GNU gettext 目錄 I/O(.po / .mo)
- 檢查碼演算法
- 多路徑點滑鼠手勢
- 清空再輸入欄位
- 等待消失(阻塞式 vanish 等待)
- 按住按鍵 / 自動重複
- 相對滑鼠移動
- 等待區域顏色
- Unicode 文字輸入(emoji / CJK)
- 在動作群組中持續按住修飾鍵
- 錨點序數與全部定位
- 表格 / 格線儲存格定位
- 等待視窗標題(正則)
- 具信心分數的模板比對
- 依顏色定位螢幕區域
- 遮罩模板比對(忽略背景)
- 結構相似度(SSIM)比較
- ORB 特徵比對(對旋轉 / 縮放 / 主題穩健)
- 以邊緣 / 輪廓定位 UI 元素(免模板)
- 視窗鋪排 / 版面幾何規劃器
- 排列多個視窗(網格 / 層疊)
- 影像前處理(供 OCR / 模板比對)
- 多螢幕 / 虛擬桌面幾何
- 可操作性閘門——在操作前先等待就緒
- 融合並排序螢幕元素框
- HSV 色彩空間分割
- 免模型文字區偵測(MSER)
- 線條 / 網格 / 分隔線偵測(Hough)
- 重試式數值斷言(expect.poll)
- 可串接 / 可過濾的候選定位器
- 豐富剪貼簿——HTML(CF_HTML)
- 色彩直方圖指紋與變化偵測
- 局部動態 / 活動偵測
- 視窗 Z-order——置頂 / 移到最前 / 移到最後
- 軟性斷言——區塊結束時彙整所有失敗
- 感知式(YIQ)影像比對含反鋸齒抑制
- 視窗客戶區幾何
- 標準化 Computer-Use 動作結構
- 符記預算內的無障礙文字觀測
- 動作前接地防護
- 可攜式 Agent 軌跡記錄(錄製與重播)
- 幾何感知的元素差異與穩定 ID
- 加權候選評分
- 一維條碼解碼
- 旋轉與縮放容忍的樣板比對
- 粗粒度標籤螢幕網格(VLM Grounding)
- 剪貼簿檔案拖放清單(CF_HDROP)
- 信任評分樣板比對(歧義 / PSR)
- 以 OCR 文字填入框線網格(可定址表格)
- 權杖預算化的觀測差異(變更了什麼)
- 自動門檻樣板比對(對分數圖做 Otsu)
- 留白投影欄位偵測(無框線表格)
- 表單欄位關聯(多方向)+ 核取方塊狀態
- 動作效果分類(我的點擊有沒有效果?)
- 邊緣形狀(Chamfer)樣板比對
- 宣告式動作後置條件
- 失敗 / 無效果動作的修復策略引擎
- 次像素樣板比對精修
- Grounding 自我一致性(提案共識)
- 欄位感知閱讀順序(XY-Cut)
- OCR 行的段落與清單分組
- 變化量序列的穩定偵測
- 標題與內文分類 + 文件大綱
- 逐步評審特徵 + 規則式步驟評分
- 多模板共識比對
- 色彩感知樣板比對(HSV)
- 比對前安定閘 + 命中穩定性
- 擴充 UIA 控制模式(展開 / 選取 / 範圍 / 捲動)
- OCR 後端
- 可觀測性 — Metrics 與 Traces
- 維運與管理層
- USB Passthrough — 第二階段設計
- USB Passthrough — Phase 2e 安全審查清單
- USB Passthrough — 操作員指南