fetch-mcp

<div align="center"> <img src="https://raw.githubusercontent.com/jae-jae/fetcher-mcp/refs/heads/main/icon.svg" width="100" height="100" alt="Fetcher MCP Icon" /> </div> # Fetcher MCP MCP server for fetch web page content using Playwright headless browser. ## Advantages - **JavaScript Support**: Unlike traditional web scrapers, Fetcher MCP uses Playwright to execute JavaScript, making it capable of handling dynamic web content and modern web applications. - **Intelligent Content Extraction**: Built-in Readability algorithm automatically extracts the main content from web pages, removing ads, navigation, and other non-essential elements. - **Flexible Output Format**: Supports both HTML and Markdown output formats, making it easy to integrate with various downstream applications. - **Parallel Processing**: The `fetch_urls` tool enables concurrent fetching of multiple URLs, significantly improving efficiency for batch operations. - **Resource Optimization**: Automatically blocks unnecessary resources (images, stylesheets, fonts, media) to reduce bandwidth usage and improve performance. - **Robust Error Handling**: Comprehensive error handling and logging ensure reliable operation even when dealing with problematic web pages. - **Configurable Parameters**: Fine-grained control over timeouts, content extraction, and output formatting to suit different use cases. ## Quick Start Run directly with npx: ```bash npx -y fetcher-mcp ``` First time setup - install the required browser by running the following command in your terminal: ```bash npx playwright install chromium ``` ### Debug Mode Run with the `--debug` option to show the browser window for debugging: ```bash npx -y fetcher-mcp --debug ``` ## Configuration MCP Configure this MCP server in Claude Desktop: On MacOS: `~/Library/Application Support/Claude/claude_desktop_config.json` On Windows: `%APPDATA%/Claude/claude_desktop_config.json` ```json { "mcpServers": { "fetcher": { "command": "npx", "args": ["-y", "fetcher-mcp"] } } } ``` ## Features - `fetch_url` - Retrieve web page content from a specified URL - Uses Playwright headless browser to parse JavaScript - Supports intelligent extraction of main content and conversion to Markdown - Supports the following parameters: - `url`: The URL of the web page to fetch (required parameter) - `timeout`: Page loading timeout in milliseconds, default is 30000 (30 seconds) - `waitUntil`: Specifies when navigation is considered complete, options: 'load', 'domcontentloaded', 'networkidle', 'commit', default is 'load' - `extractContent`: Whether to intelligently extract the main content, default is true - `maxLength`: Maximum length of returned content (in characters), default is no limit - `returnHtml`: Whether to return HTML content instead of Markdown, default is false - `waitForNavigation`: Whether to wait for additional navigation after initial page load (useful for sites with anti-bot verification), default is false - `navigationTimeout`: Maximum time to wait for additional navigation in milliseconds, default is 10000 (10 seconds) - `disableMedia`: Whether to disable media resources (images, stylesheets, fonts, media), default is true - `debug`: Whether to enable debug mode (showing browser window), overrides the --debug command line flag if specified - `fetch_urls` - Batch retrieve web page content from multiple URLs in parallel - Uses multi-tab parallel fetching for improved performance - Returns combined results with clear separation between webpages - Supports the following parameters: - `urls`: Array of URLs to fetch (required parameter) - Other parameters are the same as `fetch_url` ## Tips ### Handling Special Website Scenarios #### Dealing with Anti-Crawler Mechanisms - **Wait for Complete Loading**: For websites using CAPTCHA, redirects, or other verification mechanisms, include in your prompt: ``` Please wait for the page to fully load ``` This will use the `waitForNavigation: true` parameter. - **Increase Timeout Duration**: For websites that load slowly: ``` Please set the page loading timeout to 60 seconds ``` This adjusts both `timeout` and `navigationTimeout` parameters accordingly. #### Content Retrieval Adjustments - **Preserve Original HTML Structure**: When content extraction might fail: ``` Please preserve the original HTML content ``` Sets `extractContent: false` and `returnHtml: true`. - **Fetch Complete Page Content**: When extracted content is too limited: ``` Please fetch the complete webpage content instead of just the main content ``` Sets `extractContent: false`. - **Return Content as HTML**: When HTML format is needed instead of default Markdown: ``` Please return the content in HTML format ``` Sets `returnHtml: true`. ### Debugging and Authentication #### Enabling Debug Mode - **Dynamic Debug Activation**: To display the browser window during a specific fetch operation: ``` Please enable debug mode for this fetch operation ``` This sets `debug: true` even if the server was started without the `--debug` flag. #### Using Custom Cookies for Authentication - **Manual Login**: To login using your own credentials: ``` Please run in debug mode so I can manually log in to the website ``` Sets `debug: true` or uses the `--debug` flag, keeping the browser window open for manual login. - **Interacting with Debug Browser**: When debug mode is enabled: 1. The browser window remains open 2. You can manually log into the website using your credentials 3. After login is complete, content will be fetched with your authenticated session - **Enable Debug for Specific Requests**: Even if the server is already running, you can enable debug mode for a specific request: ``` Please enable debug mode for this authentication step ``` Sets `debug: true` for this specific request only, opening the browser window for manual login. ## Development ### Install Dependencies ```bash npm install ``` ### Install Playwright Browser Install the browsers needed for Playwright: ```bash npm run install-browser ``` ### Build the Server ```bash npm run build ``` ## Debugging Use MCP Inspector for debugging: ```bash npm run inspector ``` You can also enable visible browser mode for debugging: ```bash node build/index.js --debug ``` ## Related Projects - [g-search-mcp](https://github.com/jae-jae/g-search-mcp): A powerful MCP server for Google search that enables parallel searching with multiple keywords simultaneously. Perfect for batch search operations and data collection. ## License Licensed under the [MIT License](https://choosealicense.com/licenses/mit/)