> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Model & Lineage

> Understand how objects transform into documents and how lineage tracks provenance

Mixpeek's data model is built around a clear transformation pipeline: **Objects → Documents → Features**. Each entity has a specific purpose, and lineage metadata ensures you can always trace results back to their source.

## Core Entities

> [card] Objects
>
> Raw inputs registered in buckets. Objects hold blobs (video, image, text, audio) and metadata but are **not processed** until added to a batch.

> [card] Documents
>
> Processed representations produced by collection pipelines (the [features](https://docs.mixpeek.com/docs/processing/features.md) you enabled). Documents live in collections and include vectors, metadata, and lineage references.

> [card] Features
>
> Extracted representations (embeddings, classifications, segments) stored as [MVS](https://mixpeek.com/mvs) vectors and referenced by feature URIs.

## Transformation Flow

![Mixpeek Transformation Flow](https://mintcdn.com/mixpeek/TwtTrae3Fi3EFJ72/assets/mixpeek-flow.svg?fit=max\&auto=format\&n=TwtTrae3Fi3EFJ72\&q=85\&s=ae6474d56d2179ffd66e0576c18e64a6)

### 1. Object Registration

Objects are created via the Buckets API and validated against the bucket's JSON schema:

```bash theme={null}
POST /v1/buckets/{bucket_id}/objects
```

**Object Structure:**

* `object_id` – unique identifier
* `bucket_id` – parent bucket reference
* `key_prefix` – logical path/grouping
* `blobs[]` – array of file references or inline data
* `metadata` – custom JSON validated against bucket schema
* `created_at` / `updated_at` – audit timestamps

Every bucket has a managed system collection, `mvs_bucket_<bucket_id>`, that
holds one document per object. Writing an object writes its document. Collections you define with
feature extractors are separate: an object becomes documents in them when a
batch processes it.

A bucket is addressable as a collection. Name a `bkt_...` id where a
`collection_id` goes and it resolves to `mvs_bucket_<bucket_id>`. See
[Bucket-backed collections](https://docs.mixpeek.com/docs/vector-store/system-collections.md#bucket-backed-collections).

### 2. Batch Processing

Batches group objects for efficient parallel processing:

```bash theme={null}
POST /v1/buckets/{bucket_id}/batches
POST /v1/buckets/{bucket_id}/batches/{batch_id}/submit
```

When submitted, the API:

1. Resolves all collections that consume the bucket
2. Generates per-extractor artifacts (manifests) and uploads to S3
3. Dispatches Ray tasks to the Engine with S3 artifact URIs

### 3. Document Creation

The Engine runs each collection's processing pipeline in parallel. For each object and each collection:

1. **Download manifest** – fetch pipeline config and input mappings
2. **Run the pipeline** – run model inference (embeddings, classifications, etc.)
3. **Write to MVS** – upsert vectors and payload with `internal_id` tenant filter
4. **Update metadata** – set `__fully_enriched`, `__pipeline_version`, `source_object_id`

**Document Structure:**

* `document_id` – MVS point ID
* `collection_id` – parent collection
* `source_object_id` – lineage back to originating object
* `root_object_id` – if object was derived, traces to original input
* `feature_refs[]` – array of feature URIs (e.g., `mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1`)
* `metadata` – passthrough fields + enrichments from taxonomies/clusters
* `__fully_enriched` – boolean flag indicating the full pipeline succeeded
* `__missing_features` – array of feature addresses that failed
* `__pipeline_version` – integer tracking collection schema version

### 4. Feature Storage

Features are stored as [MVS](https://mixpeek.com/mvs) vectors with named indexes:

**Feature URI Format:**

```
mixpeek://<extractor_name>@<version>/<output_name>
```

Example:

```
mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1
mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding
mixpeek://image_extractor@v1/google_siglip_base_v1
```

These URIs are used in:

* Retriever stage configurations (`feature_uri`)
* Taxonomy input mappings
* Join enrichment strategies

## Lineage Tracking

Every document maintains lineage metadata for auditability and debugging:

Every field below lives under `_internal.lineage` on the document.

| Field | Purpose |
| - | - |
| `root_object_id` | The bucket object at the root of the tree |
| `root_bucket_id` | The bucket that object belongs to |
| `source_type` | `bucket` for a tier-0 document, `collection` for a derived one |
| `source_object_id` | The object this document came from. On a derived document it inherits the root |
| `source_document_id` | The upstream document, `null` on a tier-0 document |
| `source_collection_id` | The upstream collection, `null` on a tier-0 document |
| `chain` | One entry per processing step, each naming the collection, document and feature extractor |

### Querying Lineage

Use the Document Lineage API to trace provenance:

```bash theme={null}
# Full lineage for one document
GET /v1/collections/{collection_id}/documents/{document_id}/lineage

# Every collection and document derived from one object
GET /v1/objects/{object_id}/decomposition-tree

# Every document derived from one object, as a normal document list
POST /v1/collections/{collection_id}/documents/list
{"filters": {"AND": [
  {"field": "_internal.lineage.root_object_id", "operator": "eq", "value": "obj_..."}
]}}
```

The same filter works on any collection in the chain, so one object id reaches
its backing document and every document derived from it.

## Multi-Level Decomposition

Some pipelines produce **multiple documents per object** (e.g., video → scenes, PDF → pages):

```
Object: video_file.mp4
  └─ Document: full_video_summary
       ├─ Document: scene_001 (0:00-0:15)
       ├─ Document: scene_002 (0:15-0:32)
       └─ Document: scene_003 (0:32-1:00)
```

Each scene document includes:

* `_internal.lineage.root_object_id` → the `video_file.mp4` object
* `_internal.lineage.source_object_id` → the same object, inherited from the root
* `_internal.lineage.source_document_id` → the `full_video_summary` document
* `parent_document_id` → the same document, at the top level
* `segment_metadata` → `{ start_time: 0.0, end_time: 15.0 }`

### Scene times

A video or audio segment document carries its times in seconds as `start_time`
and `end_time`, the names every video extractor writes. Documents from
`universal_extractor` (collections created with `features=["video_search"]`)
also carry `start_time_s` and `end_time_s`. The two names are interchangeable
in document reads, retriever results, filters and sorts, so either one finds
the time on every collection, including collections written before both names
were stored.

## Tenant Isolation

All entities are scoped by:

* **`internal_id`** (organization) – injected at API layer from `Authorization` header
* **`namespace_id`** – resolved from `X-Namespace` header

MVS namespaces map 1:1 to namespaces, and all queries/writes automatically include `internal_id` filters. This ensures hard multi-tenancy without application-level filtering.

## Schema Evolution

Collections track `__pipeline_version` to handle schema changes:

1. Update collection definition (add/remove features, change mappings)
2. Increment `pipeline_version` automatically
3. New batches write documents with updated version
4. Old documents remain queryable with legacy schema
5. Optional: Trigger reprocessing to backfill with the new pipeline

## Feature Reuse Across Collections

Processing pipelines are **reusable**. Collections that enable the same feature emit the same feature URI, even when wired to different input fields (advanced per-pipeline config shown; the extractor names are deprecated aliases):

```json theme={null}
// Collection A: Product descriptions
{
  "feature_extractor": {
    "feature_extractor_name": "text_extractor",
    "input_mappings": { "text": "product_description" }
  }
}

// Collection B: Customer reviews
{
  "feature_extractor": {
    "feature_extractor_name": "text_extractor",
    "input_mappings": { "text": "review_text" }
  }
}
```

Both produce `mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1` features, enabling **cross-collection search** via retrievers that span multiple collection IDs.

## Document Lifecycle States

| State | Meaning |
| - | - |
| `pending` | Batch submitted but Engine hasn't started processing |
| `processing` | Feature extraction in progress |
| `completed` | The full pipeline succeeded (`__fully_enriched: true`) |
| `partial` | Some features failed (`__missing_features` populated) |
| `enriching` | Materialized taxonomy/cluster enrichment running |

Query `__fully_enriched` in retriever filters to exclude incomplete documents.

## Best Practices

> [accordion] Design bucket schemas for validation, not storage
>
> Bucket schemas enforce input shape but don't constrain downstream processing. Keep them minimal and use collection `field_passthrough` to propagate only what's needed.

> [accordion] Use key\_prefix for logical grouping
>
> Organize objects with `key_prefix` (e.g., `/catalog/electronics`, `/users/avatars`) to enable bulk operations and filtering without custom metadata.

> [accordion] Batch efficiently
>
> Group 100-1000 objects per batch for optimal throughput. Smaller batches add orchestration overhead; larger batches delay feedback and complicate retries.

> [accordion] Leverage lineage for debugging
>
> When retrieval results are unexpected, use lineage APIs to trace documents back to source objects and inspect raw input data and processing history.

> [accordion] Version collections explicitly
>
> When changing features or mappings, create a new collection rather than mutating the existing one. This preserves reproducibility and simplifies rollback.

## Example: E-Commerce Product Pipeline

```json theme={null}
1. Register object in "products" bucket:
   - blobs: [{ property: "image", type: "image", url: "s3://..." }]
   - metadata: { sku: "HD-9000", category: "headphones", price: 299 }

2. Process through "product-embeddings" collection (features: ["image_search", "text_search"]):
   - image_search → image embeddings
   - text_search → description embeddings
   - Passthrough: sku, category, price

3. Resulting document in MVS:
   - document_id: "doc_abc123"
   - source_object_id: "obj_prod_001"
   - feature_refs: [
       "mixpeek://image_extractor@v1/google_siglip_base_v1",
       "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"
     ]
   - metadata: { sku: "HD-9000", category: "headphones", price: 299 }
   - __fully_enriched: true

4. Query with hybrid retriever:
   - Stage 1: feature_search on mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1
   - Stage 2: attribute_filter where category = "headphones"
   - Stage 3: sort_attribute by price ascending
```

## Next Steps

* Dive into advanced pipeline configuration with [Feature Extractors](https://docs.mixpeek.com/docs/processing/feature-extractors.md)
* Explore [Collections](https://docs.mixpeek.com/docs/ingestion/collections.md) configuration and output schemas
* Review [Document Lineage API](https://docs.mixpeek.com/docs/api-reference/document-lineage/get-document-lineage.md) for tracing provenance
* Understand [Namespaces](https://docs.mixpeek.com/docs/ingestion/namespaces.md) for tenant isolation


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.