> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Batch ingestion at scale

> How to size, submit, and monitor large batch ingestions — chunking, concurrency, automatic recovery, and the limits that apply at each plan tier.

Processing a large bucket (100k–5M objects) is a batching decision, not one API call. This page gives you the sizing model the platform itself uses, so you can pick a chunk size and batch count with numbers instead of guesses.

## Two submission paths

**`POST /v1/collections/{collection_id}/trigger`** creates **one batch containing every object** in the source bucket. There is no server-side splitting. Use it for buckets up to a few thousand objects.

**`POST /v1/buckets/{bucket_id}/batches/bulk-submit`** splits the bucket into batches of `chunk_size` objects (default 1,000, maximum 50,000) and submits each one. Use it for everything larger.

> [!WARNING]
>
> Do not trigger a collection over a very large bucket. The single batch it creates
> runs as one job with a fixed worker ceiling and a 24-hour execution deadline, so
> it under-parallelizes and can time out. Bulk-submit is the large-corpus path.

```python Python theme={null}
from mixpeek import Mixpeek

mp = Mixpeek(api_key="YOUR_API_KEY")

result = mp.buckets.batches.bulk_submit(
    bucket_id="bkt_123",
    collection_ids=["col_abc"],
    chunk_size=20000,
)
print(result.batch_ids)
```

```javascript JavaScript theme={null}
import { Mixpeek } from "@mixpeek/sdk";

const mp = new Mixpeek({ apiKey: "YOUR_API_KEY" });

const result = await mp.buckets.batches.bulkSubmit({
  bucketId: "bkt_123",
  collectionIds: ["col_abc"],
  chunkSize: 20000,
});
console.log(result.batchIds);
```

```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/bkt_123/batches/bulk-submit" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "X-Namespace: your-namespace" \
  -H "Content-Type: application/json" \
  -d '{"collection_ids": ["col_abc"], "chunk_size": 20000}'
```

## Picking a chunk size

Each batch runs as its own job. A job scales its workers with object count — roughly one CPU worker per 500 objects — up to a per-job worker ceiling. A batch of 16,000–20,000 objects saturates one job's parallelism. Larger batches do not run faster; they only raise the cost of a mid-run failure.

Going too small hurts the other way. Every batch pays a cluster cold-start, and only a fixed number of batches run at once (see the tier table). 1,200 batches of 1,000 objects serialize into deep queue waves and spend a large share of wall-clock on startup.

| Corpus size | Suggested `chunk_size` | Resulting batches |
| - | - | - |
| under 5k | one trigger call | 1 |
| 5k – 100k | 5,000–10,000 | 1–20 |
| 100k – 1M | 20,000 | 5–50 |
| 1M+ | 20,000–50,000 | 20–100+ |

Calibrate before you commit: run one small batch (1,000 objects) and one at your target size, compare objects/hour and credits from the [Billing usage page](https://docs.mixpeek.com/docs/platform/billing.md), then extrapolate. Media mix changes cost more than batch size does.

## Concurrency and queueing

Submission is accept-and-queue. Batches past your concurrency limit park as `QUEUED` and promote automatically, first-in-first-out, as running batches finish. You do not pace submissions yourself.

| Plan | Max objects per batch | Concurrent running batches |
| - | - | - |
| Free | 10,000 | 1 |
| Pro | 50,000 | 5 |
| Team | 500,000 | 10 |
| Enterprise | 5,000,000 | 50 |

Submission returns `429` only when a queue is truly full. Each running job also has a 24-hour execution deadline.

## Automatic recovery

The platform heals transient failures without your involvement:

* Failed tasks retry 3 times with exponential backoff.
* A worker that dies mid-task has its work redelivered to another worker.
* Reapers sweep every 1–10 minutes for orphaned, stuck, or stalled batches. They re-drive recoverable ones (bounded retries) and mark the rest failed.
* Failures tagged safe-to-resubmit are resubmitted automatically every 30 minutes.
* Batches that failed on transient errors retry hourly, up to 3 times, with 1/2/4-hour backoff.

Progress is durable per object, not per batch. Each object is recorded in a processing ledger as it completes, so resubmitting a partially-failed batch skips finished objects and reprocesses only the remainder. You do not pay for extraction twice.

Some failures never self-heal by design: authentication errors, schema validation errors, and hard provider quota exhaustion (for example, an exhausted OpenAI balance). These end the affected batches in `FAILED` or `COMPLETED_WITH_ERRORS` and wait for you to fix the cause, then [resubmit](https://docs.mixpeek.com/docs/api-reference/bucket-batches/submit-batch-for-processing.md).

## Monitoring a run

Watch progress through the API, not by polling logs:

* `GET /v1/buckets/{bucket_id}/batches/{batch_id}` — status, per-tier progress, heartbeat freshness, `documents_written`, `failed_object_count`, and `error_summary`.
* `GET .../batches/{batch_id}/failed-documents` — every failed object with its error, for targeted retries.
* [Batch diagnostics](https://docs.mixpeek.com/docs/troubleshoot/batch-diagnostics.md) — deeper triage when a batch misbehaves.

Enable the batch [system alerts](https://docs.mixpeek.com/docs/api-reference/alerts) — `batch_failed`, `batch_stalled`, and `batch_error_rate` — before a large run. They evaluate every 5 minutes and notify in-app plus any webhook, Slack, or email channel you configure. Without them, a failed batch is visible only if you go looking for it.

> [!NOTE]
>
> A batch can finish as `COMPLETED_WITH_ERRORS`. Treat that as a work list, not a
> verdict: read `failed-documents`, fix the cause, resubmit, and the ledger skips
> everything already done.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.