> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Scrape

> Fetch and extract page content from URLs in your documents

![Web Scrape stage showing content extraction from URLs](https://mintcdn.com/mixpeek/TmiAqiYj-LwmWL2a/assets/retrievers/web-scrape.svg?fit=max\&auto=format\&n=TmiAqiYj-LwmWL2a\&q=85\&s=e12278d3bcf69dd335260e72367de304)

The Web Scrape stage fetches content from web URLs using the Engine's Playwright service. It handles JavaScript rendering and main-content extraction to add web content to your retrieval pipeline.

> [!NOTE]
>
> **Stage Category**: APPLY (Enriches documents with scraped content)
>
> **Transformation**: N documents → N documents (with web content added)

## When to Use

| Use Case | Description |
| - | - |
| **URL enrichment** | Extract content from URLs in documents |
| **Reference expansion** | Scrape linked references for context |
| **Content aggregation** | Pull in external content sources |
| **Real-time content** | Access current webpage content |

## When NOT to Use

| Scenario | Recommended Alternative |
| - | - |
| Searching the web | `external_web_search` (Exa) |
| Static content already indexed | Use indexed content |
| High-volume scraping | Pre-index content in Mixpeek |

## Parameters

| Parameter | Type | Default | Description |
| - | - | - | - |
| `url_field` | string | `metadata.url` | Document field containing the URL to scrape |
| `output_field` | string | `web_content` | Field that receives the extracted content |
| `strategy` | string | `auto` | `static`, `javascript`, or `auto` |
| `timeout` | integer | `10` | Request timeout in seconds (1 to 60) |
| `include_html` | boolean | `false` | Include the raw HTML in the output |
| `min_content_length` | integer | `500` | With `auto`, fall back to JavaScript when static parsing returns fewer characters |
| `when` | object | none | Only scrape documents that match this condition |
| `on_error` | string | `skip` | `skip`, `remove`, or `raise` |
| `batch_size` | integer | `5` | Concurrent fetch requests (1 to 20) |

## Configuration Examples

```json Basic URL Scraping theme={null}
{
  "stage_name": "web_scrape",
  "stage_type": "apply",
  "config": {
    "stage_id": "web_scrape",
    "parameters": {
      "url_field": "metadata.source_url",
      "output_field": "page_content"
    }
  }
}
```

```json With JavaScript Rendering theme={null}
{
  "stage_name": "web_scrape",
  "stage_type": "apply",
  "config": {
    "stage_id": "web_scrape",
    "parameters": {
      "url_field": "metadata.url",
      "output_field": "content",
      "strategy": "javascript",
      "timeout": 30
    }
  }
}
```

```json Selective Scraping theme={null}
{
  "stage_name": "web_scrape",
  "stage_type": "apply",
  "config": {
    "stage_id": "web_scrape",
    "parameters": {
      "url_field": "metadata.reference_url",
      "output_field": "reference",
      "when": {
        "AND": [
          {"field": "score", "operator": "gte", "value": 0.8}
        ]
      }
    }
  }
}
```

```json HTML Extraction theme={null}
{
  "stage_name": "web_scrape",
  "stage_type": "apply",
  "config": {
    "stage_id": "web_scrape",
    "parameters": {
      "url_field": "url",
      "output_field": "html_content",
      "include_html": true,
      "timeout": 15
    }
  }
}
```

## Output Schema

### Default Output

The extracted content is stored as an object at `output_field`.

```json theme={null}
{
  "document_id": "doc_123",
  "metadata": {
    "source_url": "https://example.com/article"
  },
  "page_content": {
    "url": "https://example.com/article",
    "final_url": "https://example.com/article",
    "title": "Article Title",
    "text": "Article content here...",
    "content_length": 1842,
    "metadata": {
      "author": "Jane Doe",
      "date": "2025-01-15",
      "description": "Meta description",
      "sitename": "Example",
      "language": "en"
    },
    "strategy_used": "static"
  }
}
```

### With Raw HTML

Set `include_html` to `true` to add the page HTML to the same object.

```json theme={null}
{
  "document_id": "doc_123",
  "html_content": {
    "url": "https://example.com/article",
    "final_url": "https://example.com/article",
    "title": "Article Title",
    "text": "Article content here...",
    "content_length": 1842,
    "metadata": {},
    "strategy_used": "static",
    "html": "<html>...</html>"
  }
}
```

### Failed Fetch

With the default `on_error: skip`, a document whose fetch fails passes through unchanged, without the output field.

## Scraping Strategies

| Strategy | Description |
| - | - |
| `static` | Fast HTML parsing without JavaScript. Best for articles and blogs |
| `javascript` | Full browser rendering. Slower, and required for single-page apps |
| `auto` | Tries `static` first and falls back to `javascript` when the extracted text is shorter than `min_content_length` |

> [!TIP]
>
> Use `strategy: "javascript"` when scraping JavaScript-heavy sites, and raise `timeout` to 20-30 seconds.

## Performance

| Metric | Value |
| - | - |
| **Latency** | 1-10s (depends on page complexity) |
| **Concurrent requests** | `batch_size`, 5 by default and up to 20 |
| **Timeout default** | 10 seconds |

> [!WARNING]
>
> Web scraping adds significant latency. Use sparingly and consider pre-indexing frequently accessed content.

## Common Pipeline Patterns

### Enrich Documents with Referenced Content

```json theme={null}
[
  {
    "stage_name": "semantic_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 10 }
        ],
        "final_top_k": 10
      }
    }
  },
  {
    "stage_name": "structured_filter",
    "stage_type": "filter",
    "config": {
      "stage_id": "attribute_filter",
      "parameters": {
        "field": "metadata.has_url",
        "operator": "eq",
        "value": true
      }
    }
  },
  {
    "stage_name": "web_scrape",
    "stage_type": "apply",
    "config": {
      "stage_id": "web_scrape",
      "parameters": {
        "url_field": "metadata.source_url",
        "output_field": "source_content"
      }
    }
  }
]
```

### Scrape and Summarize

```json theme={null}
[
  {
    "stage_name": "web_scrape",
    "stage_type": "apply",
    "config": {
      "stage_id": "web_scrape",
      "parameters": {
        "url_field": "metadata.url",
        "output_field": "page_content"
      }
    }
  },
  {
    "stage_name": "json_transform",
    "stage_type": "apply",
    "config": {
      "stage_id": "json_transform",
      "parameters": {
        "template": {
          "content": "{{ DOC.page_content.text }}",
          "source": "{{ DOC.metadata.url }}"
        }
      }
    }
  },
  {
    "stage_name": "summarize",
    "stage_type": "reduce",
    "config": {
      "stage_id": "summarize",
      "parameters": {
        "provider": "google",
        "model_name": "gemini-2.5-flash-lite",
        "prompt": "Summarize the key points from these webpages:\n\n{{DOCUMENTS}}"
      }
    }
  }
]
```

## Error Handling

| Error | Behavior |
| - | - |
| Missing or empty URL field | Follows `on_error` |
| Invalid URL, timeout, or HTTP error such as 404 or 403 | Follows `on_error` |

`on_error` accepts three values. `skip` (default) passes the document through unchanged. `remove` drops the document from the results. `raise` fails the stage.

## Rate Limits and Best Practices

1. **Batch wisely**: Limit to 5-10 URLs per pipeline run
2. **Cache results**: Consider storing scraped content
3. **Use timeouts**: Set an appropriate `timeout` (in seconds) for your use case

## Related

* [External Web Search](https://docs.mixpeek.com/docs/retrieval/stages/external-web-search.md) - Search the web (Exa)
* [API Call](https://docs.mixpeek.com/docs/retrieval/stages/api-call.md) - General HTTP enrichment
* [Document Enrich](https://docs.mixpeek.com/docs/retrieval/stages/document-enrich.md) - Collection joins


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.