Text Extractor

Dense multilingual (E5-Large) text embeddings for semantic and cross-lingual search — query in one language, match content in 100+ others

Built-in extractor names are a deprecated alias — collections are now created by picking features. This pipeline is selected with features: ["text_search"]. Existing feature_extractor configs keep working; see the migration guide.

View on GitHub

Runnable reference for this extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry.

The text extractor generates dense vector embeddings from text using the E5-Large multilingual model. Optimized for semantic search, RAG applications, and general-purpose text retrieval. Supports text chunking/decomposition with multiple splitting strategies. Fast (5ms/doc), cost-effective (free), and supports 100+ languages.

Cross-lingual retrieval. Because all 100+ languages share one embedding space, a query in one language matches content in any language — an English query (e.g. “foreign language”, “translate what they said”) retrieves Farsi, Mandarin, or Spanish text with no translation step and no per-language index. Note that this is retrieval, not translation: matched text is returned in its original language. To render it in the reader’s language, add an llm_enrich translate stage — see the Search Across Languages recipe.

View extractor details at api.mixpeek.com/v1/collections/features/extractors/text_extractor_v1 or fetch programmatically with GET /v1/collections/features/extractors/{feature_extractor_id}.

Pipeline Steps

1. Filter Dataset (if collection_id provided)
   • Filter to specified collection
2. Apply Input Mappings
   • Resolve text field from source (e.g., transcription, content, data)
3. Text Chunking (conditional: if split_by != "none")
   • Split by: characters, words, sentences, paragraphs, or pages
   • Configure chunk_size and chunk_overlap
   • Each chunk becomes a separate document
4. E5 Text Embedding Generation
   • Multilingual E5-Large model (1024D)
   • L2 normalized vectors
   • Batch size: 4,096 texts
5. Output
   • Text documents with embeddings
   • One document per input (or per chunk if chunking enabled)

When to Use

Use Case	Description
Product search	Search products by natural language descriptions
FAQ matching	Match user questions to knowledge base articles
Document retrieval	Find relevant documents from large corpora
Content discovery	Recommend similar content based on semantic similarity
RAG chunking	Split documents into chunks for retrieval-augmented generation
Multi-language / cross-lingual search	Query in one language, match content in another (foreign-language, translate-intent queries) across 100+ languages with a single model

When NOT to Use

Scenario	Recommended Alternative
Exact phrase / keyword matching (SKUs, codes, names)	Add a lexical (BM25) search
Keyword-heavy queries (e.g. “iPhone 15 Pro Max 256GB”)	Lexical (BM25) search alongside the dense one
Critical technical terms or short texts (1–5 words)	Lexical (BM25) search

Dense embeddings and exact-keyword matching are complementary. Keep text_extractor for semantic recall and add a lexical: true search over a text index for exact tokens — fuse them with rrf. See Lexical (BM25) Search.

Input Schema

Field	Type	Required	Description
text	string	Yes	Text content to process. Recommended: 10-400 words for optimal quality. Maximum: 512 tokens (~400 words), longer text is truncated.

{
  "text": "Premium wireless Bluetooth headphones with active noise cancellation, 30-hour battery life, and premium sound quality."
}

Input Examples:

Type	Example
Product description	”Premium wireless Bluetooth headphones with active noise cancellation”
FAQ question	”How do I reset my password if I forgot it?”
Article paragraph	”Machine learning models have revolutionized natural language processing…”
User query	”best restaurants near Times Square”

Output Schema

Field	Type	Description
text	string	The processed text content (full text or chunk)
text_extractor_v1_embedding	float[1024]	Dense vector embedding, L2 normalized

{
  "text": "Premium wireless Bluetooth headphones with active noise cancellation",
  "text_extractor_v1_embedding": [0.023, -0.041, 0.018, ...]
}

When chunking is enabled, each chunk becomes a separate document with tracking metadata stored in metadata (not in the document payload):

• chunk_index – Position of this chunk in the original document
• chunk_text – The text content of this chunk
• total_chunks – Total number of chunks from the source

Parameters

Chunking Parameters

Parameter	Type	Default	Description
split_by	string	"none"	Strategy for splitting text into chunks
chunk_size	integer	1000	Target size for each chunk (units depend on split_by)
chunk_overlap	integer	0	Number of units to overlap between consecutive chunks

Split Strategies

Strategy	Description	Best For
characters	Split by character count	Uniform sizes, quick testing
words	Split by word boundaries	General text, preserves words
sentences	Split by sentence boundaries	Q&A, precise retrieval, preserves semantic units
paragraphs	Split by paragraph (double newlines)	Articles, documentation, natural structure
pages	Split by page breaks	PDFs, paginated documents
none	No splitting (default)	Short texts < 400 words

Recommended chunk sizes:

• characters: 500-2000
• words: 100-400
• sentences: 3-10
• paragraphs: 1-3
• pages: 1

Chunk overlap: 10-20% of chunk_size helps preserve context across boundaries. Example: chunk_size: 1000, chunk_overlap: 100-200.

Embedding Model

Parameter	Type	Default	Description
embedding_model	string	current TEXT default	Override the embedding model via the model registry

The text extractor resolves its model through the central embedding registry. Leave unset to use the current TEXT modality default (intfloat_e5_large_instruct_v1, 1024d) — the registry swaps hot when a new frontier text model ships, so existing collections pick it up without a code change.

Dimensions are locked at namespace creation. Switching embedding_model on an existing namespace requires a migration since the vector index dimensionality is fixed.

Embedding Task

Instruction-aware embedding models (E5, Gemini) use a task hint to optimize the embedding for a specific downstream use case. By default, all extractors use retrieval_document at ingestion time, which produces embeddings optimized for asymmetric search (queries find documents).

Set embedding_task at the collection level, not on the extractor. See Collection Embedding Task for full details and examples.

Task	Use Case	Effect on E5	Effect on Gemini
retrieval_document	Default. Search: find documents from queries	Prepends "passage: "	Instructs “represent for retrieval”
retrieval_query	Rare at index time. Query-side is automatic	Prepends "query: "	Instructs “represent this query”
semantic_similarity	Symmetric comparison (deduplication, matching)	Prepends "query: "	Instructs “represent for similarity”
classification	Document categorization pipelines	Prepends "query: "	Instructs “represent for classification”
clustering	Grouping documents into clusters	Prepends "query: "	Not applied

You almost never need to set this. The default retrieval_document is correct for search, and at query time Mixpeek automatically uses retrieval_query. Only override if your collection is primarily used for clustering, classification, or symmetric similarity — not retrieval.

Non-instruction-aware models (SigLIP, CLIP, Vertex multimodal) ignore this parameter.

LLM Structured Extraction Parameters

Parameter	Type	Default	Description
response_shape	string | object	null	Define custom structured output using LLM extraction
llm_provider	string	null	LLM provider: openai, google, anthropic, typesafe
llm_model	string	null	Specific model for extraction

response_shape Modes

Natural Language Mode (string):

{
  "response_shape": "Extract key entities, sentiment (positive/negative/neutral), and main topics from the text"
}

The service automatically infers JSON schema from your description.

JSON Schema Mode (object):

{
  "response_shape": {
    "type": "object",
    "properties": {
      "sentiment": {
        "type": "string",
        "enum": ["positive", "negative", "neutral"]
      },
      "entities": {
        "type": "array",
        "items": { "type": "string" }
      },
      "topics": {
        "type": "array",
        "items": { "type": "string" },
        "maxItems": 5
      }
    },
    "required": ["sentiment"]
  }
}

LLM Provider & Model Options

Provider	Example Models
openai	gpt-4o-mini-2024-07-18 (cost-effective), gpt-4o-2024-08-06 (best quality)
google	gemini-2.5-flash (fastest), gemini-1.5-flash-001
anthropic	claude-3-5-haiku-20241022 (fast), claude-3-5-sonnet-20241022 (best reasoning)
typesafe	jev-latest: labels, flags and ratings from a fixed set, with probabilities

With llm_provider: "typesafe", Jev answers each response_shape field as a typed question. response_shape must be a JSON schema, and every field needs a closed set of values: a boolean, an enum, a number from 0 to 1, or an integer with minimum and maximum. Natural-language response_shape needs a generative model. The collection is rejected at creation if a field cannot be answered, and the error names the field.

Configuration Examples

Basic Embedding (No Chunking)

{
  "feature_extractor": {
    "feature_extractor_name": "text_extractor",
    "version": "v1",
    "input_mappings": {
      "text": "description"
    },
    "field_passthrough": [
      { "source_path": "metadata.product_id" }
    ],
    "parameters": {}
  }
}

Performance & Costs

Metric	Value
Embedding latency	~5ms per document (batched: ~2ms/doc)
Query latency	5-10ms for top-100 results
Cost	Free (self-hosted E5-Large)
GPU required	No (but 5-10x faster with GPU)
Memory	~4GB per 1M documents
Index build	~1 hour per 10M documents

LLM extraction adds cost and latency based on provider pricing. Only use when structured extraction is needed.

Dense Embeddings vs Lexical (BM25)

text_extractor produces dense embeddings — great for semantic recall, weak on exact tokens. For exact-keyword precision, pair it with a lexical (BM25) search (the lexical: true option on a feature_search stage, backed by a text payload index — not a separate extractor).

Dimension	Dense (text_extractor)	Lexical (BM25, lexical: true)
Matches	Meaning / paraphrase	Exact tokens, SKUs, codes, prices
Semantic recall	Excellent	Poor
Exact matching	Poor	Excellent
Multi-language	Excellent (E5-Large)	Good (token-based)
Requires	Embedding index	text payload index

The strongest setup is hybrid: run a dense search and a lexical search in the same feature_search stage and fuse with rrf. See Lexical (BM25) Search.

Vector Index

Property	Value
Feature URI	mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1
Index name	text_extractor_v1_embedding
Dimensions	1024
Type	Dense
Distance metric	Cosine
Datatype	float32
Inference model	multilingual_e5_large_instruct_v1
Normalization	L2 normalized

In retrievers, reference this feature by its Feature URI above (the output name is multilingual_e5_large_instruct_v1, not the index name text_extractor_v1_embedding).

Bring your own vectors

To store vectors you computed yourself in a text_extractor collection, embed the way the platform does.

Property	Value
Model	intfloat/multilingual-e5-large-instruct
Dimensions	1024
Distance	Cosine
Document text	Prepend passage: before you embed
Query text	Prepend query: before you embed a query vector you pass with input_mode: vector

The platform L2-normalizes its own vectors. Normalize yours the same way so they compare with the managed ones.

The model and dimensions above are the current text default. Check index.dimensions on the collection’s vector_indexes entry before you embed, because the default can change.

Send each vector under a key the collection accepts. Write vectors into a managed collection lists both.

Limitations

• Token limit: 512 tokens (~400 words). Longer text is automatically truncated.
• Exact phrases: Cannot reliably match exact phrases or technical terms.
• Domain jargon: Struggles with very domain-specific jargon or acronyms.
• Terminology variance: May miss documents that use different terminology for the same concept.
• Short texts: Less effective for very short texts (1-5 words) where lexical matching is sufficient.
• Keyword-heavy queries: Less effective for queries like “iPhone 15 Pro Max 256GB”.

Related

• Feature Extractors Overview
• Passthrough Extractor
• Multimodal Extractor