Appearance
Web Fetch
Transform any webpage into clean, structured Markdown that AI agents can efficiently parse and understand.
Overview
Web Fetch is a built-in capability that enables AI agents to access and understand web content. When enabled on your MCP Gateway, agents can fetch any public webpage and receive it as clean, structured Markdown.
Key Features
- Intelligent Conversion: Preserves document structure, heading hierarchies, and table layouts
- JavaScript Support: Handles dynamically rendered content
- Smart Filtering: Removes ads, navigation, and irrelevant content
- Fast Processing: Millisecond conversion speeds with 15-minute cache
- Redirect Handling: Automatically follows redirects with cross-domain support
How to Enable
- Navigate to your MCP Gateway settings
- Locate Web Fetch in the Built-in Capabilities section
- Toggle the capability ON
- Save your configuration
Once enabled, any AI agent connected to your gateway can access web content through the MCP protocol.
What AI Agents Can Do
Research and Fact-Checking
- Extract information from academic papers and research sites
- Access news articles and press releases
- Read technical documentation
- Gather data from company websites and reports
Documentation Processing
- Transform API documentation into structured formats
- Extract product specifications
- Process user guides and tutorials
- Access release notes and changelogs
Content Analysis
- Compile market research from multiple sources
- Perform competitive analysis
- Create literature reviews
- Analyze trends across websites
Knowledge Gathering
- Build FAQ compilations
- Collect best practices
- Access industry standards documentation
- Retrieve regulatory compliance information
Content Transformation
Web Fetch automatically converts various HTML elements into clean Markdown:
| HTML Element | Output Format | Notes |
|---|---|---|
| Headings (h1-h6) | Markdown headings | Preserves hierarchy |
| Paragraphs | Plain text | Maintains spacing |
| Lists (ul/ol) | Markdown lists | Nested lists supported |
| Tables | Markdown tables | Complex tables simplified |
| Code blocks | Formatted code | Syntax preserved |
| Links | Markdown links | Relative URLs resolved |
| Images | Markdown images | Optional inclusion |
| Blockquotes | Quoted text | Nested quotes supported |
Performance
- Response Time: Millisecond conversion speeds
- Timeout: 30 seconds maximum
- Rate Limit: 100 requests per minute
- Cache: 15 minutes for repeated URLs
- Max Size: 200KB included, larger responses count as multiple requests
Capability Behavior
Performance Characteristics
- Responses are automatically cached for 15 minutes
- Content is optimized for AI processing
- Large sites are handled with pagination support
Automatic Error Handling
- Timeouts after 30 seconds
- Follows redirects across domains
- Handles invalid or inaccessible URLs
- Includes rate limiting protection
Content Processing
- Filters irrelevant content automatically
- Preserves authoritative information
- Recognizes paywalls and login requirements
Limitations
- JavaScript Execution: Limited to standard rendering
- Authentication: No support for logged-in content
- Media: Video and audio content not extracted
- Interactive Elements: Forms and widgets not preserved
- File Downloads: Direct file downloads not supported
Pricing
Web Fetch is included with standard MCP Gateway usage. No additional charges apply for:
- Standard web pages under 200KB
- Up to 100 requests per minute
- Cached content retrieval
Large responses over 200KB count as multiple gateway requests.
Typical Applications
Technical Documentation Access
AI agents use Web Fetch to access API documentation, developer guides, and technical specifications, converting them into structured information for analysis.
Multi-Source Research
Agents gather information from multiple web sources simultaneously for comprehensive research, competitive analysis, or content aggregation.
Change Monitoring
Agents periodically fetch web pages to detect updates, price changes, new product releases, or content modifications.
Next Steps
Related Capabilities
- Web Search - Find relevant pages before fetching
- Knowledge Retrieval - Store and search fetched content
- HTTP Client - For API endpoints and raw responses
Platform Features
- All Capabilities - Explore other built-in tools
- MCP Gateway - Configure your gateway
- Usage & Limits - Understand platform constraints