appenv_demo

# Web App Environments and Cross-App RL Tasks Demo This demo features three web applications (100+ in total)—SiYuan for note-taking, SignaturePDF for PDF editing, and Docmost for online document collaboration—along with some samples of cross-app RL tasks (easy-level) designed by our team. That is, to finish each RL task, the agent must interact with multiple App instead of single one. Each task comes with accurate outcome reward to guild training and fine-grained evaluation. We evaluate these tasks using frontier LLMs under the MCPMark framework, where models interact with the apps through Playwright MCP to complete the tasks. The setup can also be readily extended to other agent harnesses, such as GUI Agent and Browser-Use agents. <video controls autoplay muted loop width="800"> <source src="./Demo.mp4" type="video/mp4"> </video> ## Installation Create a conda environment with the required system libraries (needed for Playwright/Chromium on headless servers): ```bash cd webenv_demo conda create -n webenv python=3.11 -y conda activate webenv pip install -e . playwright install ``` ## Quick Start ### 1. Configure Server Hostname & API Keys Run the one-click setup script: ```bash ./start.sh ``` This will: - Replace the `<your-server-hostname>` placeholder in all task files with your the app-deployed server's IP or just localhost. - Configure OpenRouter API proxy keys in `.mcp_env` To manually configure API keys (OpenAI, Anthropic, Google, etc.), edit `.mcp_env` directly. ### 2. Deploy Docker Environments Docker images are placed in the `docker/` directory. ```bash cd docker # SiYuan (note-taking) docker load -i mw-siyuan.tar docker run --name siyuan -p 6037:6806 -d mw-siyuan:latest # SignaturePDF (PDF editing) docker load -i mw-signaturepdf.tar docker run --name signaturepdf -p 6093:80 -d mw-signaturepdf:latest # Docmost (wiki/docs) docker load -i mw-docmost.tar docker run --name docmost -p 6001:3000 -d mw-docmost:latest ``` After a few minutes, verify the services are running: - SiYuan: `http://<your-server-hostname>:6037` - SignaturePDF: `http://<your-server-hostname>:6093` - Docmost: `http://<your-server-hostname>:6001` ### 3. Run Evaluation (with Gemini-3-Flash) Make sure the `webenv` conda environment is active (`conda activate webenv`), then: ```bash # All tasks python -m pipeline --mcp playwright_webarena --models openai/google/gemini-3-flash-preview \ --task-suite standard --exp-name test_all --k 1 ``` After the evaluation, you **MUST** run the `reset.sh` to reset the environment. ### 4. Aggregate Results ```bash python -m src.aggregators.aggregate_results --exp-name <your-exp-name> ``` ## CLI Reference | Argument | Default | Description | |---|---|---| | `--mcp` | `filesystem` | MCP service to use | | `--models` | *(required)* | Comma-separated model list | | `--agent` | `mcpmark` | Agent implementation | | `--tasks` | `all` | `"all"`, `"category"`, or `"category/task_id"` | | `--task-suite` | `standard` | `standard` or `easy` | | `--exp-name` | timestamp | Experiment name (required when `--k` > 1) | | `--k` | `4` | Number of runs for pass@k metrics | | `--timeout` | `3600` | Agent execution timeout (seconds) | | `--reasoning-effort` | `default` | `minimal` / `low` / `medium` / `high` / `default` | ## License See [LICENSE](LICENSE).