The Web Scraping screen (titled Web Scraping Settings in the CMS) lets your agent read the content of web pages and convert it to Markdown, and controls which images and paths are included. Found in the agent sidebar under Tools → Web Scraping.
Web Scraping Configuration
| |
|---|
| Enable Web Scraping | Lets the agent extract and analyze web page content in real time. The other fields are disabled while it is off. |
| Include Images | Keeps images in the Markdown output. When off, all images are removed from scraped content. |
| Timeout (seconds) | How long to wait for a page to load, from 5 to 60 seconds. Default: 30. |
Image Size Filters
Images outside this size range are filtered out.
| |
|---|
| Min Image Size (KB) | Smaller images, such as icons and decorations, are removed. From 1 KB to 10240 KB. Default: 5 KB. |
| Max Image Size (MB) | Larger images are removed to avoid heavy downloads. From 1 MB to 50 MB. Default: 10 MB. |
Ignored Paths
| |
|---|
| Path Patterns to Ignore | Paths to skip when scraping. Supports absolute paths (/icons), wildcards (*.svg) and file names (logo.png). Leave empty to include all paths. Up to 50 patterns, 200 characters each. |
| Add New Path Pattern | Adds an entry to the list. Each entry has a delete button. |
Saving
| |
|---|
| Save | Checks the fields, asks you to confirm, then saves the settings. |
Related guide: Search and Read the Web