English Documentation

Comprehensive guides for all AutoControl features.

User Guide

• Installation
  • Install from PyPI
  • Requirements
  • With GUI Support
  • Linux Prerequisites
  • Raspberry Pi
  • Development Environment
• Mouse Control
  • Getting Mouse Button Table
  • Press and Release
  • Click
  • Position
  • Scroll
• Keyboard Control
  • Getting Key Tables
  • Press and Release
  • Type a Single Key
  • Check Key State
  • Type a String
  • Hotkey Combinations
• Screen Operations
  • Screenshot
  • Screen Size
  • Get Pixel Color
• Image Recognition
  • Locate All Matches
  • Locate Image Center
  • Locate and Click
  • Parameters
• Recording & Playback
  • Record and Replay
  • How It Works
  • Editing a recording
• Keywords & Executor
  • Keyword Format
  • Available Action Commands
  • Executing a JSON File
  • Executing All JSON Files in a Directory
  • Extending the Executor
• Report Generation
  • Setup
  • Generating Reports
  • Getting Report Content as String
  • Report Contents
• Callback Executor
  • Basic Usage
  • How It Works
  • Parameters
  • Extending the Callback Executor
• Scheduler
  • Basic Example
  • Blocking vs Non-Blocking
  • Interval Scheduling
  • Cron Scheduling
  • Removing Jobs
  • Shutting Down
• Socket Server (Remote API)
  • Starting the Server
  • Sending Commands (Client)
  • Protocol Details
• MCP Server (Use AutoControl from Claude)
  • Tool catalogue
  • Resources, prompts, sampling
  • Logging notifications, progress, cancellation
  • Starting the server (programmatic)
  • Starting the server (command line)
  • CLI inspection flags
  • Registering with Claude Desktop
  • Registering with Claude Code
  • HTTP transport (with SSE, auth, TLS)
  • Read-only / safe mode
  • Confirmation prompts (elicitation)
  • Audit log
  • Auto-screenshot on tool error
  • Rate limiting
  • Plugin hot-reload
  • CI smoke tests with the fake backend
  • Security notes
• Critical Exit
  • Enabling Critical Exit
  • Changing the Hotkey
  • Example: Recovering from Runaway Mouse
• Command-Line Interface
  • Subcommand CLI (python -m je_auto_control.cli)
  • Legacy flag-style CLI (python -m je_auto_control)
• Project Management
  • Creating a Project
  • Generated Structure
• New Features (2026-04)
  • Clipboard
  • Dry-run / step-debug executor
  • Global hotkey daemon (Windows)
  • Event triggers
  • Cron scheduling
  • Plugin loader
  • REST API server
  • CLI runner
  • Multi-language GUI (i18n)
  • Closable tabs + menu bar
  • OCR (text on screen)
  • Accessibility element finder
  • VLM (AI) element locator
  • Run history + error-snapshot artifacts
  • OCR — region dump and regex search
  • Runtime variables and data-driven control flow
  • LLM action planner
  • Remote desktop (host + viewer)
  • Remote desktop — Quick Connect + Phase 4/5 hardening
  • Remote desktop — secure transports, audio, clipboard, file transfer
  • Remote desktop — AnyDesk-style popout window
  • Remote desktop — responsive sub-tab sizing
  • Remote desktop — MCP tool surface
  • Driver-level input backends — drive games that ignore SendInput / XTest
  • Per-action profiler
  • Run history timeline + failure thumbnails
  • Encrypted secret manager
  • Webhook (HTTP push) trigger
  • IMAP email trigger
• New Features (2026-05)
  • Locator + selector intelligence
  • Operations + observability
  • Agent + integrations
  • Platform coverage
  • Developer experience
• New Features (2026-06) — QA Layer
  • Assertions
  • Data-driven execution
  • Flaky-test detection & quarantine
  • QA suite runner + CI reports
  • Accessibility & i18n audit
  • Mobile device matrix
• OCR backends
  • Backend selection
  • Language codes
  • Tesseract setup notes
  • EasyOCR / PaddleOCR setup notes
  • Headless usage
  • JSON action surface
  • Diagnostics
• Observability — metrics and traces
  • Quick start
  • Built-in metrics
  • Adding your own metrics
  • Tracing
  • Production deployment
  • Security note
• Operations & Admin Layer
  • Folder sync (additive mirror)
  • coturn TURN config bundle
  • Hardened REST API
  • Prometheus metrics
  • Multi-host admin console
  • Audit log hash chain
  • WebRTC packet inspector
  • USB device enumeration
  • USB hotplug events
  • System diagnostics
  • Web admin dashboard
  • OpenAPI 3.1 + Swagger UI
  • Configuration bundle
• USB Passthrough — Phase 2 Design
  • Goals
  • Non-Goals
  • Transport
  • Per-OS driver wrappers
  • Security & ACL
  • Phasing
  • Design decisions (formerly open questions)
• USB Passthrough — Phase 2e Security Review Checklist
  • Threat model
  • ACL
  • Audit
  • Protocol hardening
  • Resource bounds
  • Lifecycle
  • Per-OS requirements
  • Pen-test scenarios
  • Sign-off
• USB Passthrough — Operator Guide
  • Prerequisites
  • Driver setup (per OS)
  • Enabling the feature
  • ACL setup
  • Starting the host
  • Viewer-side: claim and transfer
  • Troubleshooting matrix
  • Headless control (AC_usb_* commands)
  • What is not shipped yet