> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Text Extractor

> Dense multilingual (E5-Large) text embeddings for semantic and cross-lingual search — query in one language, match content in 100+ others

> [!WARNING]
>
> Built-in extractor names are a **deprecated alias** — collections are now created by picking [features](https://docs.mixpeek.com/docs/processing/features.md). This pipeline is selected with `features: ["text_search"]`. Existing `feature_extractor` configs keep working; see the [migration guide](https://docs.mixpeek.com/docs/processing/extractor-migration.md).

> [card] [View on GitHub](https://github.com/mixpeek/mixpeek-extractors/blob/main/extractors/text_extractor/README.md)
>
> Runnable reference for this extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry.

![Text extractor pipeline showing chunking, E5-Large embedding, and optional LLM extraction](https://mintcdn.com/mixpeek/TwtTrae3Fi3EFJ72/assets/extractors/text.svg?fit=max\&auto=format\&n=TwtTrae3Fi3EFJ72\&q=85\&s=5c9f4dd60090c5941a6283f448b4da86)

The text extractor generates dense vector embeddings from text using the E5-Large multilingual model. Optimized for semantic search, RAG applications, and general-purpose text retrieval. Supports text chunking/decomposition with multiple splitting strategies. Fast (5ms/doc), cost-effective (free), and supports 100+ languages.

> [!NOTE]
>
> **Cross-lingual retrieval.** Because all 100+ languages share one embedding space, a query in one language matches content in **any** language — an English query (e.g. "foreign language", "translate what they said") retrieves Farsi, Mandarin, or Spanish text with **no translation step and no per-language index**. Note that this is *retrieval*, not translation: matched text is returned in its original language. To render it in the reader's language, add an `llm_enrich` translate stage — see the [Search Across Languages recipe](https://docs.mixpeek.com/docs/retrieval/cookbook.md#search-across-languages-cross-lingual).

> [!NOTE]
>
> View extractor details at [api.mixpeek.com/v1/collections/features/extractors/text\_extractor\_v1](https://api.mixpeek.com/v1/collections/features/extractors/text_extractor_v1) or fetch programmatically with `GET /v1/collections/features/extractors/{feature_extractor_id}`.

## Pipeline Steps

1. **Filter Dataset** (if collection\_id provided)
   * Filter to specified collection
2. **Apply Input Mappings**
   * Resolve text field from source (e.g., `transcription`, `content`, `data`)
3. **Text Chunking** (conditional: if `split_by != "none"`)
   * Split by: characters, words, sentences, paragraphs, pages, or markdown structure
   * Configure `chunk_size` and `chunk_overlap`
   * Each chunk becomes a separate document
4. **E5 Text Embedding Generation**
   * Multilingual E5-Large model (1024D)
   * L2 normalized vectors
   * Batch size: 4,096 texts
5. **Output**
   * Text documents with embeddings
   * One document per input (or per chunk if chunking enabled)

## When to Use

| Use Case | Description |
| - | - |
| **Product search** | Search products by natural language descriptions |
| **FAQ matching** | Match user questions to knowledge base articles |
| **Document retrieval** | Find relevant documents from large corpora |
| **Content discovery** | Recommend similar content based on semantic similarity |
| **RAG chunking** | Split documents into chunks for retrieval-augmented generation |
| **Multi-language / cross-lingual search** | Query in one language, match content in another (foreign-language, translate-intent queries) across 100+ languages with a single model |

## When NOT to Use

| Scenario | Recommended Alternative |
| - | - |
| Exact phrase / keyword matching (SKUs, codes, names) | Add a [lexical (BM25) search](https://docs.mixpeek.com/docs/retrieval/stages/feature-search.md#lexical-bm25-search) |
| Keyword-heavy queries (e.g. "iPhone 15 Pro Max 256GB") | Lexical (BM25) search alongside the dense one |
| Critical technical terms or short texts (1–5 words) | Lexical (BM25) search |

> [!NOTE]
>
> Dense embeddings and exact-keyword matching are complementary. Keep `text_extractor` for semantic recall and add a `lexical: true` search over a `text` index for exact tokens — fuse them with `rrf`. See [Lexical (BM25) Search](https://docs.mixpeek.com/docs/retrieval/stages/feature-search.md#lexical-bm25-search).

## Input Schema

| Field | Type | Required | Description |
| - | - | - | - |
| `text` | string | **Yes** | Text content to process. Recommended: 10-400 words for optimal quality. Maximum: 512 tokens (\~400 words), longer text is truncated. |

```json theme={null}
{
  "text": "Premium wireless Bluetooth headphones with active noise cancellation, 30-hour battery life, and premium sound quality."
}
```

**Input Examples:**

| Type | Example |
| - | - |
| Product description | "Premium wireless Bluetooth headphones with active noise cancellation" |
| FAQ question | "How do I reset my password if I forgot it?" |
| Article paragraph | "Machine learning models have revolutionized natural language processing..." |
| User query | "best restaurants near Times Square" |

## Output Schema

| Field | Type | Description |
| - | - | - |
| `text` | string | The processed text content (full text or chunk) |
| `text_extractor_v1_embedding` | float\[1024] | Dense vector embedding, L2 normalized |

```json theme={null}
{
  "text": "Premium wireless Bluetooth headphones with active noise cancellation",
  "text_extractor_v1_embedding": [0.023, -0.041, 0.018, ...]
}
```

When chunking is enabled, each chunk becomes a separate document with tracking metadata stored in `metadata` (not in the document payload):

* `chunk_index` – Position of this chunk in the original document
* `chunk_text` – The text content of this chunk
* `total_chunks` – Total number of chunks from the source

With `split_by: "markdown"`, each document also carries three structure fields:

| Field | Type | Description |
| - | - | - |
| `heading_path` | string | Headings enclosing the chunk, outermost first, joined with `" > "` |
| `chunk_kind` | string | `"section"` for prose, `"code"` for a fenced code block |
| `code_language` | string | Language from the code fence's info string, when the fence names one. Set on code chunks only. |

## Parameters

### Chunking Parameters

| Parameter | Type | Default | Description |
| - | - | - | - |
| `split_by` | string | `"none"` | Strategy for splitting text into chunks |
| `chunk_size` | integer | `1000` | Target size for each chunk (units depend on `split_by`) |
| `chunk_overlap` | integer | `0` | Number of units to overlap between consecutive chunks. Not used for `markdown`. |

#### Split Strategies

| Strategy | Description | Best For |
| - | - | - |
| `characters` | Split by character count | Uniform sizes, quick testing |
| `words` | Split by word boundaries | General text, preserves words |
| `sentences` | Split by sentence boundaries | Q\&A, precise retrieval, preserves semantic units |
| `paragraphs` | Split by paragraph (double newlines) | Articles, documentation, natural structure |
| `pages` | Split by page breaks | PDFs, paginated documents |
| `markdown` | One document per heading section and one per fenced code block | Docs sites, READMEs, any markdown with code examples |
| `none` | No splitting (default) | Short texts \< 400 words |

**Recommended chunk sizes:**

* `characters`: 500-2000
* `words`: 100-400
* `sentences`: 3-10
* `paragraphs`: 1-3
* `pages`: 1
* `markdown`: `chunk_size` counts words of prose per section chunk

#### Markdown splitting

`split_by: "markdown"` follows the structure of the page:

* Each heading section becomes one document. The chunk starts with its heading, so it reads on its own.
* A section longer than `chunk_size` words splits at paragraph boundaries.
* Each fenced code block becomes its own document. Code blocks are never split, whatever their length.
* Front matter is dropped. Its `title` becomes the outermost heading in `heading_path`.
* A `#` comment inside a code block does not count as a heading.

Use it when your text has code examples. The other strategies count characters, words or sentences, so they can cut through a code fence and lose the code's language and section.

The ColBERT extractor (`colbert_extractor`) accepts `split_by: "markdown"` too.

**Chunk overlap:** 10-20% of `chunk_size` helps preserve context across boundaries. Example: `chunk_size: 1000`, `chunk_overlap: 100-200`.

### Embedding Model

| Parameter | Type | Default | Description |
| - | - | - | - |
| `embedding_model` | string | *current TEXT default* | Override the embedding model via the [model registry](https://docs.mixpeek.com/docs/processing/model-registry.md) |

The text extractor resolves its model through the central embedding registry. Leave unset to use the current TEXT modality default (`intfloat_e5_large_instruct_v1`, 1024d) — the registry swaps hot when a new frontier text model ships, so existing collections pick it up without a code change.

> [!WARNING]
>
> Dimensions are locked at namespace creation. Switching `embedding_model` on an existing namespace requires a migration since the vector index dimensionality is fixed.

### Embedding Task

Instruction-aware embedding models (E5, Gemini) use a **task hint** to optimize the embedding for a specific downstream use case. By default, all extractors use `retrieval_document` at ingestion time, which produces embeddings optimized for asymmetric search (queries find documents).

> [!NOTE]
>
> Set `embedding_task` at the **collection level**, not on the extractor. See [Collection Embedding Task](https://docs.mixpeek.com/docs/platform/processing.md#embedding-task) for full details and examples.

| Task | Use Case | Effect on E5 | Effect on Gemini |
| - | - | - | - |
| `retrieval_document` | **Default.** Search: find documents from queries | Prepends `"passage: "` | Instructs "represent for retrieval" |
| `retrieval_query` | Rare at index time. Query-side is automatic | Prepends `"query: "` | Instructs "represent this query" |
| `semantic_similarity` | Symmetric comparison (deduplication, matching) | Prepends `"query: "` | Instructs "represent for similarity" |
| `classification` | Document categorization pipelines | Prepends `"query: "` | Instructs "represent for classification" |
| `clustering` | Grouping documents into clusters | Prepends `"query: "` | Not applied |

> [!NOTE]
>
> You almost never need to set this. The default `retrieval_document` is correct for search, and at query time Mixpeek automatically uses `retrieval_query`. Only override if your collection is primarily used for clustering, classification, or symmetric similarity — not retrieval.

> [!NOTE]
>
> Non-instruction-aware models (SigLIP, CLIP, Vertex multimodal) ignore this parameter.

### LLM Structured Extraction Parameters

| Parameter | Type | Default | Description |
| - | - | - | - |
| `response_shape` | string \| object | `null` | Define custom structured output using LLM extraction |
| `llm_provider` | string | `null` | LLM provider: `openai`, `google`, `anthropic`, `typesafe` |
| `llm_model` | string | `null` | Specific model for extraction |

#### response\_shape Modes

**Natural Language Mode (string):**

```json theme={null}
{
  "response_shape": "Extract key entities, sentiment (positive/negative/neutral), and main topics from the text"
}
```

The service automatically infers JSON schema from your description.

**JSON Schema Mode (object):**

```json theme={null}
{
  "response_shape": {
    "type": "object",
    "properties": {
      "sentiment": {
        "type": "string",
        "enum": ["positive", "negative", "neutral"]
      },
      "entities": {
        "type": "array",
        "items": { "type": "string" }
      },
      "topics": {
        "type": "array",
        "items": { "type": "string" },
        "maxItems": 5
      }
    },
    "required": ["sentiment"]
  }
}
```

#### LLM Provider & Model Options

| Provider | Example Models |
| - | - |
| `openai` | `gpt-4o-mini-2024-07-18` (cost-effective), `gpt-4o-2024-08-06` (best quality) |
| `google` | `gemini-2.5-flash` (fastest), `gemini-1.5-flash-001` |
| `anthropic` | `claude-3-5-haiku-20241022` (fast), `claude-3-5-sonnet-20241022` (best reasoning) |
| `typesafe` | `jev-latest`: labels, flags and ratings from a fixed set, with probabilities |

With `llm_provider: "typesafe"`, [Jev](https://docs.mixpeek.com/docs/processing/jev.md) answers each `response_shape` field as a typed question. `response_shape` must be a JSON schema, and every field needs a closed set of values: a `boolean`, an `enum`, a `number` from 0 to 1, or an `integer` with `minimum` and `maximum`. Natural-language `response_shape` needs a generative model. The collection is rejected at creation if a field cannot be answered, and the error names the field.

## Configuration Examples

```json Basic Embedding (No Chunking) theme={null}
{
  "feature_extractor": {
    "feature_extractor_name": "text_extractor",
    "version": "v1",
    "input_mappings": {
      "text": "description"
    },
    "field_passthrough": [
      { "source_path": "metadata.product_id" }
    ],
    "parameters": {}
  }
}
```

```json Sentence Chunking for RAG theme={null}
{
  "feature_extractor": {
    "feature_extractor_name": "text_extractor",
    "version": "v1",
    "input_mappings": {
      "text": "document_content"
    },
    "field_passthrough": [
      { "source_path": "metadata.document_id" }
    ],
    "parameters": {
      "split_by": "sentences",
      "chunk_size": 5,
      "chunk_overlap": 1
    }
  }
}
```

```json Paragraph Chunking for Articles theme={null}
{
  "feature_extractor": {
    "feature_extractor_name": "text_extractor",
    "version": "v1",
    "input_mappings": {
      "text": "article_body"
    },
    "field_passthrough": [
      { "source_path": "metadata.title" },
      { "source_path": "metadata.author" }
    ],
    "parameters": {
      "split_by": "paragraphs",
      "chunk_size": 2,
      "chunk_overlap": 0
    }
  }
}
```

```json Markdown Chunking for Docs theme={null}
{
  "feature_extractor": {
    "feature_extractor_name": "text_extractor",
    "version": "v1",
    "input_mappings": {
      "text": "page_markdown"
    },
    "parameters": {
      "split_by": "markdown",
      "chunk_size": 300
    }
  }
}
```

```json Word-Level Chunking with Overlap theme={null}
{
  "feature_extractor": {
    "feature_extractor_name": "text_extractor",
    "version": "v1",
    "input_mappings": {
      "text": "content"
    },
    "parameters": {
      "split_by": "words",
      "chunk_size": 300,
      "chunk_overlap": 50
    }
  }
}
```

```json LLM Extraction (Natural Language) theme={null}
{
  "feature_extractor": {
    "feature_extractor_name": "text_extractor",
    "version": "v1",
    "input_mappings": {
      "text": "review_text"
    },
    "parameters": {
      "response_shape": "Extract sentiment (positive/negative/neutral), key product features mentioned, and overall rating impression",
      "llm_provider": "openai",
      "llm_model": "gpt-4o-mini-2024-07-18"
    }
  }
}
```

```json LLM Extraction (JSON Schema) theme={null}
{
  "feature_extractor": {
    "feature_extractor_name": "text_extractor",
    "version": "v1",
    "input_mappings": {
      "text": "document"
    },
    "parameters": {
      "response_shape": {
        "type": "object",
        "properties": {
          "entities": {
            "type": "array",
            "items": { "type": "string" },
            "description": "Named entities mentioned"
          },
          "sentiment": {
            "type": "string",
            "enum": ["positive", "negative", "neutral"]
          },
          "topics": {
            "type": "array",
            "items": { "type": "string" },
            "maxItems": 5
          }
        },
        "required": ["sentiment"]
      },
      "llm_provider": "anthropic",
      "llm_model": "claude-3-5-haiku-20241022"
    }
  }
}
```

## Performance & Costs

| Metric | Value |
| - | - |
| **Embedding latency** | \~5ms per document (batched: \~2ms/doc) |
| **Query latency** | 5-10ms for top-100 results |
| **Cost** | Free (self-hosted E5-Large) |
| **GPU required** | No (but 5-10x faster with GPU) |
| **Memory** | \~4GB per 1M documents |
| **Index build** | \~1 hour per 10M documents |

**LLM extraction** adds cost and latency based on provider pricing. Only use when structured extraction is needed.

## Dense Embeddings vs Lexical (BM25)

`text_extractor` produces **dense** embeddings — great for semantic recall, weak on exact tokens. For exact-keyword precision, pair it with a **lexical (BM25)** search (the `lexical: true` option on a `feature_search` stage, backed by a `text` payload index — not a separate extractor).

| Dimension | Dense (`text_extractor`) | Lexical (BM25, `lexical: true`) |
| - | - | - |
| **Matches** | Meaning / paraphrase | Exact tokens, SKUs, codes, prices |
| **Semantic recall** | Excellent | Poor |
| **Exact matching** | Poor | Excellent |
| **Multi-language** | Excellent (E5-Large) | Good (token-based) |
| **Requires** | Embedding index | `text` payload index |

> [!TIP]
>
> The strongest setup is **hybrid**: run a dense search and a lexical search in the same `feature_search` stage and fuse with `rrf`. See [Lexical (BM25) Search](https://docs.mixpeek.com/docs/retrieval/stages/feature-search.md#lexical-bm25-search).

## Vector Index

| Property | Value |
| - | - |
| **Feature URI** | `mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1` |
| **Index name** | `text_extractor_v1_embedding` |
| **Dimensions** | 1024 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Datatype** | float32 |
| **Inference model** | `multilingual_e5_large_instruct_v1` |
| **Normalization** | L2 normalized |

> [!NOTE]
>
> In retrievers, reference this feature by its **Feature URI** above (the output name is `multilingual_e5_large_instruct_v1`, **not** the index name `text_extractor_v1_embedding`).

### Bring your own vectors

To store vectors you computed yourself in a `text_extractor` collection, embed the way the platform does.

| Property | Value |
| - | - |
| Model | `intfloat/multilingual-e5-large-instruct` |
| Dimensions | 1024 |
| Distance | Cosine |
| Document text | Prepend `passage: ` before you embed |
| Query text | Prepend `query: ` before you embed a query vector you pass with `input_mode: vector` |

The platform L2-normalizes its own vectors. Normalize yours the same way so they compare with the managed ones.

The model and dimensions above are the current text default. Check `index.dimensions` on the collection's `vector_indexes` entry before you embed, because the [default can change](#embedding-model).

Send each vector under a key the collection accepts. [Write vectors into a managed collection](https://docs.mixpeek.com/docs/vector-store/documents.md#write-vectors-into-a-managed-collection) lists both.

## Limitations

* **Token limit**: 512 tokens (\~400 words). Longer text is automatically truncated.
* **Exact phrases**: Cannot reliably match exact phrases or technical terms.
* **Domain jargon**: Struggles with very domain-specific jargon or acronyms.
* **Terminology variance**: May miss documents that use different terminology for the same concept.
* **Short texts**: Less effective for very short texts (1-5 words) where lexical matching is sufficient.
* **Keyword-heavy queries**: Less effective for queries like "iPhone 15 Pro Max 256GB".

## Related

* [Feature Extractors Overview](https://docs.mixpeek.com/docs/processing/feature-extractors.md)
* [Passthrough Extractor](https://docs.mixpeek.com/docs/processing/extractors/passthrough.md)
* [Multimodal Extractor](https://docs.mixpeek.com/docs/processing/extractors/multimodal.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.