Python SDK for Midscene.js - AI-powered UI automation using natural language
Features | Installation | Quick Start | Documentation | 中文文档
PyMidscene is a Python port of Midscene.js - an AI-powered UI automation framework. It allows you to control web browsers using natural language instead of CSS selectors or XPath.
No more fragile selectors! Just describe what you want to click, type, or extract:
# Instead of: page.click("#submit-btn-primary") await agent.ai_click("the blue Submit button") # Instead of: page.fill("input[name='email']", "test@example.com") await agent.ai_input("email input field", "test@example.com") # Extract structured data with natural language result = await agent.ai_query({ "title": "the page title", "price": "the product price as a number" })
- Natural Language Automation - Describe elements in plain English/Chinese, no selectors needed
- Multi-Model Support - Works with Doubao, Qwen, GPT-4V, Claude, and other vision LLMs
- Playwright Integration - Seamless integration with Playwright for web automation
- Android Integration - Control real devices over ADB with
pymidscene[android](see pymidscene/android/README.md) - iOS Integration - Drive iPhones / simulators through WebDriverAgent (see pymidscene/ios/README.md)
- XPath Caching - Smart caching system compatible with Midscene.js format
- Visual Reports - Generate beautiful HTML reports for debugging and sharing
- Type-Safe - Full type hints for excellent IDE support
pip install pymidscene
# Install Playwright browsers
playwright install chromiumOr with Poetry:
poetry add pymidscene playwright install chromium
# For Doubao (recommended for Chinese users) export MIDSCENE_MODEL_NAME="doubao-seed-1-6-251015" export MIDSCENE_MODEL_API_KEY="your-api-key" export MIDSCENE_MODEL_BASE_URL="https://ark.cn-beijing.volces.com/api/v3" export MIDSCENE_MODEL_FAMILY="doubao-vision" # For Qwen export MIDSCENE_MODEL_NAME="qwen-vl-max" export MIDSCENE_MODEL_API_KEY="your-api-key" export MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" export MIDSCENE_MODEL_FAMILY="qwen2.5-vl"
import asyncio import os from playwright.async_api import async_playwright from pymidscene import PlaywrightAgent async def main(): # Configure model (or use environment variables) os.environ["MIDSCENE_MODEL_NAME"] = "doubao-seed-1-6-251015" os.environ["MIDSCENE_MODEL_API_KEY"] = "your-api-key" os.environ["MIDSCENE_MODEL_BASE_URL"] = "https://ark.cn-beijing.volces.com/api/v3" os.environ["MIDSCENE_MODEL_FAMILY"] = "doubao-vision" async with async_playwright() as p: browser = await p.chromium.launch(headless=False) page = await browser.new_page() # Create agent with optional caching agent = PlaywrightAgent(page, cache_id="my_task") # Navigate to page await page.goto("https://www.example.com") # Use natural language to interact await agent.ai_click("the search box") await agent.ai_input("search input", "Python automation") await agent.ai_click("search button") # Extract data result = await agent.ai_query({ "results_count": "number of search results", "first_title": "title of the first result" }) print(f"Found: {result}") # Assert page state await agent.ai_assert("search results are displayed") # Generate visual report report_path = agent.finish() print(f"Report saved to: {report_path}") await browser.close() if __name__ == "__main__": asyncio.run(main())
| Method | Description |
|---|---|
ai_click(description) |
Click an element described in natural language |
ai_input(description, text) |
Type text into an input field |
ai_locate(description) |
Locate an element and return its coordinates |
ai_query(schema) |
Extract structured data from the page |
ai_assert(assertion) |
Assert that a condition is true |
ai_action(task) |
Execute a complex task with AI planning loop (plan-execute-replan) |
ai_wait_for(assertion, timeout) |
Wait until a page condition is met (polling) |
ai_scroll(direction, distance) |
Scroll the page with AI assistance |
finish() |
Generate HTML report and return the path |
| Model | Family | Provider |
|---|---|---|
| doubao-seed-1-6-251015 | doubao-vision | Bytedance/Volcano |
| qwen-vl-max | qwen2.5-vl | Alibaba |
| gpt-4-vision-preview | openai | OpenAI |
| claude-3-opus | claude | Anthropic |
PyMidscene uses XPath-based caching compatible with Midscene.js:
# midscene_run/cache/my_task.cache.yaml midsceneVersion: 1.0.0 cacheId: my_task caches: - type: locate prompt: the login button cache: xpaths: - /html/body/div[1]/button[1]
This means:
- Cache files are interchangeable between JS and Python versions
- XPath-based caching works across different window sizes
- Cache invalidation happens automatically when elements move
Check out the examples/ directory:
basic_usage.py- Getting startedlogin_demo.py- Login automation with visual reportlogin_demo.html- Test page for login demo
pymidscene/
├── pymidscene/ # Main package
│ ├── core/ # Core automation logic
│ │ ├── agent/ # Agent implementation
│ │ ├── ai_model/ # AI model integration
│ │ └── dump.py # Report generation
│ ├── web_integration/ # Browser integrations
│ │ └── playwright/ # Playwright adapter
│ └── shared/ # Shared utilities
├── examples/ # Usage examples
├── tests/ # Test suite
└── docs/ # Documentation
This is the Python implementation of Midscene.js.
- Midscene.js - Original JavaScript version
- Official Documentation
- Awesome Midscene - Community projects
Contributions are welcome! Please see CONTRIBUTING.md for guidelines.
# Development setup git clone https://github.com/AIPythoner/pymidscene.git cd pymidscene pip install -e ".[dev]" # Run tests pytest # Format code black pymidscene tests
MIT License - see LICENSE file for details.
- Midscene.js - The original JavaScript framework by Bytedance
- Playwright - Browser automation library
Made with love by the PyMidscene community