playwright-mcp-testing-framework

# MCP Test Framework A framework for running automated tests using Model Context Protocol (MCP) servers and LLMs. ## Overview This framework allows you to write tests in markdown files and execute them using an LLM-powered agent through the MCP protocol. The agent interprets the test instructions and executes them using Playwright or other supported MCP servers. ## Features - Write tests in markdown format - Execute tests with LLM-powered automation - Support for Playwright MCP server - Flexible test execution from files or directories - Multi-layered fallback mechanisms for reliable element selection - Robust error handling and reporting ## Installation 1. Clone the repository 2. Install Python dependencies using requirements.txt: ``` pip install -r requirements.txt ``` 3. Install required npm packages: ``` npm install @playwright/mcp@latest ``` ## Configuration Create a `.env` file in the project root with your OpenAI API key: ``` OPENAI_API_KEY= GOOGLE_API_KEY= ANTHROPIC_API_KEY= ``` ## Usage Run a single test: ``` python main.py tests/simple_test.md ``` Run all tests in a directory: ``` python main.py /tests ``` ## Test Format Tests are written in markdown format. Here's a simple example: ```markdown # Simple Test ## Test Steps 1. Navigate to https://todomvc.com/examples/vanilla-es6/ 2. Add a todo item "Buy groceries" 3. Add another item "Finish documentation" 4. Mark "Buy groceries" as completed 5. Verify that the item is crossed out 6. Click the "Active" filter 7. Verify that only "Finish documentation" is shown ``` ## Project Structure ``` ├── src/ │ ├── agent/ # MCP agent related code │ ├── test_runner/ # Test execution logic │ └── utils/ # Utility functions ├── tests/ # Test case markdown files ├── main.py # Main entry point ├── requirements.txt # Python dependencies ├── .env # Environment variables ├── README.md # Project documentation ``` ## Future Improvements Here are some potential areas for enhancing the framework: ### Parallel Test Execution - Implement concurrent test execution to improve throughput - Add support for test sharding across multiple machines - Create a dependency system to manage tests that can't run in parallel ### Enhanced Reporting - Implement detailed HTML test reports with screenshots - Add support for video recording of test executions - Create a dashboard for monitoring test results over time ### Multi-LLM Support - Expand beyond OpenAI to support Gemini, Anthropic, and other LLMs - Implement a fallback mechanism to try alternative LLMs if the primary one fails - Add LLM response caching to reduce API costs ### Resilience Improvements - Enhance the multi-layered fallback system for element selection - Add automatic retry logic for flaky tests - Implement smart waiting strategies based on application state ### Test Management - Create a test suite organization system - Add tags/categories for selective test execution - Implement test prioritization based on historical success rates ### CI/CD Integration - Add GitHub Actions workflow templates - Create Jenkins pipeline configuration examples - Support for common CI/CD platforms ## Contributing Contributions are welcome! Please feel free to submit a Pull Request. ## License This project is licensed under the MIT License - see the LICENSE file for details.