> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Deduplication & Re-processing

> Content-hash dedup decides what work is skipped, replaced, or forced when objects are processed into collections — and lineage decides what re-runs when a source changes

Every object ingested into Mixpeek is fingerprinted with a **SHA-256 content hash**, and every document produced from it carries that hash in its system envelope alongside a full [lineage record](https://docs.mixpeek.com/docs/retrieval/lineage-traversal.md). Together they answer the two questions every pipeline eventually asks:

1. **Have I already processed this?** — the content hash.
2. **What downstream work is stale now that this changed?** — the lineage chain.

## Content hashing

The hash is computed from the source content at ingestion and stored:

* on the **object** (in the bucket), and
* on every **document** derived from it (`content_hash` in the document's system envelope).

Two objects with identical bytes produce the same hash — regardless of filename, upload time, or path. A changed file produces a new hash, which is what marks its derived documents as stale.

"Latest version" of an asset is therefore a content-hash lookup, not a version counter you maintain yourself.

## Dedup strategy

When you submit a batch, `dedup_strategy` controls how objects that were **already processed in a prior batch** are handled. Dedup is scoped to the *(bucket, collection)* pair — an object is a duplicate if the target collection already has documents produced from that same source object.

| Strategy | Behavior | Cost profile |
| - | - | - |
| `skip` *(default)* | An object is reprocessed only if its content or the collection's extractor configuration changed since it was last extracted; everything else is skipped | Free for unchanged content — no extraction, no inference |
| `replace` | Every object is reprocessed. Once its new documents are written, its earlier documents in the collection are deleted. If the batch fails, or fails on that object, its earlier documents are kept | Full extraction cost; use for a clean rebuild after the source content or the extractor configuration changed |
| `force` | Every object is reprocessed and its documents are overwritten in place. Documents a re-run no longer produces are not removed | Full extraction cost; prefer `replace`, which also removes those |

```json theme={null}
{
  "bucket_id": "bkt_abc123",
  "collection_id": "col_xyz789",
  "dedup_strategy": "skip"
}
```

> [!WARNING]
>
> `skip` reprocesses an object when its content or the extractor configuration changed since it was last extracted, and skips everything else. It does not delete documents a re-run no longer produces, for example the extra chunks left behind when a new chunking setting yields fewer of them. For a clean rebuild that removes those, use `replace`.

## The re-processing cascade

Derived assets stay in sync with their source through the combination of both mechanisms:

1. A source object's content changes → its hash changes.
2. Re-submitting with `replace` (or re-syncing the source) reprocesses it — and every derived document downstream in its [lineage](https://docs.mixpeek.com/docs/retrieval/lineage-traversal.md) (transcode → embed → tag) is rebuilt from the new source.
3. Sources whose hash **didn't** change are skipped — unchanged content is never re-paid for.

This is what keeps N derived artifacts consistent with one upstream asset without a separate file system or job-tracking database: the document *is* the record, the hash *is* the version, and lineage *is* the dependency graph.

## Related

* [Lineage traversal](https://docs.mixpeek.com/docs/retrieval/lineage-traversal.md) — querying and walking derivation chains
* [Processing pipeline](https://docs.mixpeek.com/docs/platform/processing.md) — how objects become documents
* [Batch ingestion at scale](https://docs.mixpeek.com/docs/operations/batch-ingestion-at-scale.md) — batch mechanics, tiers, and monitoring


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.