MCPMark

# MCPMark for MirrorWeb MCPMark is an automated evaluation framework for testing LLM agents on multi-step tasks across MCP (Model Context Protocol) services. ## Installation Create a conda environment with the required system libraries (needed for Playwright/Chromium on headless servers): ```bash cd mcpmark conda create -n mcpmark python=3.11 -y conda activate mcpmark pip install -e . playwright install ``` ## Quick Start ### 1. Configure Server Hostname & API Keys Run the one-click setup script: ```bash ./start.sh ``` This will: - Replace the `<your-server-hostname>` placeholder in all task files with your server's hostname or IP - (Optionally) configure Qiniu API proxy keys in `.mcp_env` To manually configure API keys (OpenAI, Anthropic, Google, etc.), edit `.mcp_env` directly. ### 2. Deploy Docker Environments Docker images can be downloaded from [ModelScope](https://modelscope.cn/datasets/KeanShi/mw-docker). Place the `.tar` files in the `docker/` directory, then: ```bash cd docker # SiYuan (note-taking) docker load -i mw-siyuan.tar docker run --name siyuan -p 6037:6806 -d mw-siyuan:latest # SignaturePDF (PDF editing) docker load -i mw-signaturepdf.tar docker run --name signaturepdf -p 6093:80 -d mw-signaturepdf:latest # Docmost (wiki/docs) docker load -i mw-docmost.tar docker run --name docmost -p 6001:3000 -d mw-docmost:latest ``` After a few minutes, verify the services are running: - SiYuan: `http://<your-server-hostname>:6037` - SignaturePDF: `http://<your-server-hostname>:6093` - Docmost: `http://<your-server-hostname>:6001` ### 3. Run Evaluation (with Gemini-3-Flash) Make sure the `mcpmark` conda environment is active (`conda activate mcpmark`), then: ```bash # Single task python -m pipeline --mcp playwright_webarena --models gemini-3-flash \ --tasks websymphony_v1/task_11_invoice_watermark \ --task-suite standard --exp-name test_task02 --k 4 # All tasks python -m pipeline --mcp playwright_webarena --models gemini-3-flash \ --task-suite standard --exp-name test_all --k 1 ``` After the evaluation, you **MUST** run the `reset.sh` to reset the environment. ### 4. Aggregate Results ```bash python -m src.aggregators.aggregate_results --exp-name <your-exp-name> ``` ## CLI Reference | Argument | Default | Description | |---|---|---| | `--mcp` | `filesystem` | MCP service to use | | `--models` | *(required)* | Comma-separated model list | | `--agent` | `mcpmark` | Agent implementation | | `--tasks` | `all` | `"all"`, `"category"`, or `"category/task_id"` | | `--task-suite` | `standard` | `standard` or `easy` | | `--exp-name` | timestamp | Experiment name (required when `--k` > 1) | | `--k` | `4` | Number of runs for pass@k metrics | | `--timeout` | `3600` | Agent execution timeout (seconds) | | `--reasoning-effort` | `default` | `minimal` / `low` / `medium` / `high` / `default` | ## License See [LICENSE](LICENSE).