ActionReplay

Record and replay user actions in any Windows app (no API required).

ActionReplay records your actions in any Windows app and replays them later (useful for automating software without APIs).

This repo currently contains two tracks:

• v1 (screen/vision): template matching + mouse/keyboard simulation
• v2 prototype (UIA-first): uses Windows UI Automation snapshots captured at click points for more stable replay in classic desktop apps (e.g., Office/Outlook)

Docs

• 中文文档: docs/README.zh-CN.md
• English docs: docs/README.md

Install

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -U pip
python -m pip install -r requirements.txt
python -m pip install -e .

Note: do not commit .venv/ to GitHub (it is machine-specific and large).

After install, you can run either:

• actionreplay ...
• python -m actionreplay ...

Quick start

Record (desktop)

actionreplay record-desktop --out recordings/demo.json

Replay (v2 prototype)

actionreplay replay-desktop --flow recordings/demo.json --timeout 60

Useful replay options:

# Auto-open a file/path once if the target app is not running
actionreplay replay-desktop --flow recordings/demo.json --open test.xlsx --timeout 240

# Override numeric typed chunks during replay (repeatable)
actionreplay replay-desktop --flow recordings/demo.json --typed-value 2500 --typed-value 2800

# Write a failure report bundle when replay fails
actionreplay replay-desktop --flow recordings/demo.json --report-dir tmp/reports --report-monitor 0

Outlook (UIA smoke test)

actionreplay outlook-send --to someone@example.com --subject "test" --body "hello"

Excel (UIA smoke test)

actionreplay excel-smoke --file test.xlsx --value "ActionReplay smoke" --save --close

Web (Chrome / public sites) (MVP)

For websites (especially public sites with dynamic pages), UIA/coordinate replay is not stable. This repo includes a Playwright (DOM-level) MVP.

One-time setup (installs browser binaries):

python -m pip install -r requirements.txt
python -m playwright install chromium

Save login state (manual login + captcha if any):

actionreplay web-save-state --url "https://example.com" --out webstate/example.json

Record a web flow (headed):

actionreplay web-record --url "https://example.com" --storage-state webstate/example.json --out webflows/example.json

Replay the recorded flow (pauses when captcha/challenge is detected):

actionreplay web-replay --flow webflows/example.json --storage-state webstate/example.json --timeout 180

Recommended: general web record/replay (Codegen script)

For most sites, prefer Playwright's built-in Recorder/Codegen (wrapped here as web-codegen). It is much more general across popups/new tabs, iframes, rich editors, and auto-waits.

Record and generate a runnable script:

actionreplay web-codegen --url "https://example.com" --out webflows/example_codegen.py --storage-state webstate/example.json

Replay (run the generated script):

python .\webflows\example_codegen.py

Publishing articles (Toutiao / Xiaohongshu / Zhihu)

For “publish article” flows, record once with placeholders, then inject real content at replay time:

• During recording, type:
  • __AR_TITLE__ into the title field
  • __AR_BODY__ into the body editor
• During replay, provide content:
  • AR_ARTICLE_DIR pointing to a folder containing title.txt and body.md/body.txt, OR
  • AR_ARTICLE_TITLE / AR_ARTICLE_BODY

Example:

$env:AR_ARTICLE_DIR = "D:\\articles\\article1"
python .\webflows\example_codegen.py

More details: docs/web.md

Where are the smoke/test commands?

• CLI subcommands are defined in actionreplay/cli.py
• Implementations:
  • outlook-send: actionreplay/outlook_uia.py
  • excel-smoke: actionreplay/excel_smoke.py
  • web commands: actionreplay/web_state.py, actionreplay/web_recorder.py, actionreplay/web_player.py, actionreplay/web_codegen.py

Safety

This tool controls your mouse/keyboard. Start with simple flows, keep your hands on the keyboard, and be ready to stop the run if something goes off-track.

License

PolyForm Noncommercial 1.0.0. You may use/modify/redistribute for noncommercial purposes only. See LICENSE.