> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> Monitor API, Engine, storage, and asynchronous jobs

Mixpeek provides multiple observability surfaces: health endpoints, task metadata, Ray dashboards, analytics APIs, and webhook histories. Combine them to detect regressions early and debug production issues quickly.

## Health & Status

Four health routes, and only one of them looks at your services.

| Route | Auth | What it answers |
| - | - | - |
| `GET /v1/health` | none | The process is alive and the event loop responds. Under 10ms, no external calls. |
| `GET /v1/health/liveness` | none | Same idea, shaped for container orchestration probes. |
| `GET /v1/health/beat` | none | The work scheduler fired within the last 5 minutes. |
| `GET /v1/health/deep` | **org API key** | Pings every dependent service. Cache, metadata, vector store, object storage, task queue and inference, with per-component errors. |

> [!WARNING]
>
> **`GET /v1/health` is a liveness constant. It does not check any service.** It
> returns the same `ok` whether or not MongoDB, [MVS](https://mixpeek.com/mvs),
> Redis, the work scheduler or the engine are reachable, by design and in under 10ms.
>
> **`GET /v1/health?deep=true` does nothing.** That route declares no query
> parameters, so the flag is dropped and you get the same constant back. A
> monitor built on it reports green through a total outage. The deep check is a
> different path: `GET /v1/health/deep`.
>
> If you built a dashboard on `?deep=true`, it is measuring the event loop rather
> than the services behind it. Point it at `GET /v1/health/deep`.

> [!NOTE]
>
> **`/v1/health/deep` needs an organization API key.** It is the only health
> route with auth. An unauthenticated call gets 403, and the platform private
> token gets 401 with "API key not found", so neither an anonymous probe nor the
> private token can read component health. A 401 or 403 here is the auth
> requirement, not a broken route.
>
> On that path, `?deep=true` is a real parameter: it adds data-plane
> verification on top of the service pings, reading from collections, probing
> MVS namespaces and inspecting the task queue.

* **Tasks API** – `/v1/tasks/{task_id}` and `/v1/tasks/list` expose status for batches, clustering jobs, taxonomy materialization, and migrations. All tasks use `TaskStatusEnum`.

## Who owns this alert

`GET /v1/ops/ownership` maps every alert class to the team that owns it and the
procedure for fixing it. Org-scoped, needs a bearer token.

```bash theme={null}
# the whole registry
curl -sS "$MP_API_URL/v1/ops/ownership" -H "Authorization: Bearer $MP_API_KEY"

# just the subsystem that owns one alert class
curl -sS "$MP_API_URL/v1/ops/ownership?alert_class=MvsShardDown" \
  -H "Authorization: Bearer $MP_API_KEY"
```

Filter by `alert_class` to resolve one alert, or by `subsystem` to pull one
entry. The response is `{subsystems: [...], total}`.

| Field | Meaning |
| - | - |
| `id` | Stable subsystem slug. |
| `owner` | The team that owns this subsystem. |
| `escalation` | Who to contact when the owner does not respond. |
| `runbook` | URL or path of the procedure doc. |
| `runbook_text` | Inline minimal procedure, when no doc exists yet. |
| `alert_classes` | The alert names this subsystem covers. |
| `alert_escalations` | Per-alert overrides of `escalation`, keyed by alert class. |
| `owner_confirmed` | Present when the owner agreed to this entry, with `by` and `at`, plus optional `ref` and `note`. |

Resolving the responder for one alert: use `alert_escalations[alert_class]` when
that key is present, otherwise `escalation`. Reach for an override when a single
alert inside a subsystem pulls in someone other than the subsystem default. A key
that names an alert the subsystem does not own is rejected at load time, because
an override on an unowned alert never fires and reads like routing that works.

`owner_confirmed` records that the owner agreed to the entry, and where they said
so. Absent means nobody has confirmed it. An unconfirmed entry can still be
correct, so do not read the absence as a dispute.

> [!NOTE]
>
> **Every entry names an owner and a procedure.** The loader rejects an entry
> carrying one without the other, so `runbook` or `runbook_text` is always
> populated even when the written doc does not exist yet. Read both: either can
> be the one that is filled.
>
> CI pins that every declared alert class is covered by exactly one subsystem,
> so an alert cannot be owned twice or by nobody.

> [!WARNING]
>
> **An unknown `alert_class` returns 404, and that is the useful answer.** It
> means the alert has no owner in the registry — a gap to fill, not an empty
> result to shrug at.
>
> Class names are the CamelCase names from the alerting rules, so
> `MvsShardDown`, not `mvs_shard_down`. A snake\_case name 404s the same way, and
> the error names the registry file so you can go and look.

* **Webhooks** – webhook events recorded in MongoDB provide a durable log of ingestion and enrichment milestones (`collection.documents.written`, etc.).

## Engine Monitoring

* **Ray Dashboard (port 8265)** – view worker health, task timelines, Serve deployments, resource utilization, and logs.
* **Ray logs** – pod logs (Kubernetes) or Ray CLI provide detailed extractor and clustering output (`ray logs <job_id>`).
* **Serve metrics** – per-model latency and request counts; scrape via Prometheus or Ray metrics endpoint.

## Analytics APIs

Enable analytics (`ENABLE_ANALYTICS=true`) to capture the events these endpoints read. They are kept in MongoDB time-series collections:

| Endpoint | Insight |
| - | - |
| `/v1/analytics/retrievers/{id}/performance` | Query volume, latency percentiles |
| `/v1/analytics/retrievers/{id}/stages` | Stage-level timing and candidate counts |
| `/v1/analytics/retrievers/{id}/signals` | Cache hits, rerank scores, filter reductions |
| `/v1/analytics/retrievers/{id}/cache-performance` | Hit/miss rates and latency delta |
| `/v1/analytics/retrievers/{id}/slow-queries` | Top slow queries with execution context |
| `/v1/analytics/usage/summary` | Usage and spend (billing support) |

Use these APIs to populate dashboards or feed alerting systems.

## Logging & Tracing

* **API layer** – structured JSON logs include request IDs, namespace, HTTP status, error codes, and downstream latency.
* **Work runtime** – logs task execution, retries, and webhook dispatch results.
* **Ray workers** – include extractor metrics, batch IDs, and queue stats; aggregate logs centrally for long-term retention.
* **Correlation** – propagate `x-request-id` from API to Engine jobs via `additional_data.request_id` to stitch traces together.

## Metrics to Track

| Component | Key Metrics |
| - | - |
| API | Request rate, p95 latency, error rate, rate-limit hits |
| Work runtime | Queue depth, task execution time, retry count |
| Ray | Worker utilization (CPU/GPU), job duration, Serve requests in flight |
| MongoDB | Operation latency, primary health, replication lag |
| MVS | Storage usage, search latency, vector count per namespace |
| Redis | Connection count, command latency, cache hit ratio |

Integrate with Prometheus, Datadog, or your preferred metrics stack via existing exporters or custom scrapers.

## Alerting Playbook

1. **Latency spike** → check retriever analytics, stage statistics, and Ray Serve load.
2. **Task backlog** → inspect the task queue depth, Redis health, and Ray worker availability.
3. **Failed enrichment** → query `/v1/tasks/list` for `FAILED`, inspect `error_message`, review webhook events.
4. **Storage saturation** → monitor MVS storage usage and MongoDB disk consumption; scale storage or shard by namespace.
5. **Cache regression** → view cache hit-rate endpoint; adjust TTLs or stage cache configuration.

## Dashboards to Build

* **API dashboard** – health endpoint status, request latency, error breakdown, rate-limit counters.
* **Engine dashboard** – Ray worker utilization, job runtime percentiles, extractor throughput, Serve queue depth.
* **Retrieval performance** – retriever analytics charts (latency, cache hits, slow queries).
* **Storage dashboard** – MongoDB/Redis/MVS metrics for capacity planning.
* **Task tracker** – open tasks by status, median processing times, failure rates.

## Incident Response Tips

* Keep runbooks for common failures (e.g., extractor timeouts, MVS restarts).
* Use webhook history to confirm whether ingestion completed or stalled.
* Capture Ray job IDs from task metadata to replay logs quickly.
* Snapshot retriever and collection configurations when debugging to ensure you’re reproducing the same pipeline.

With health checks, task metadata, analytics APIs, and Ray observability, you can confidently operate Mixpeek in production and catch issues before users notice.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.