Provon Telemetry

Provon telemetry is the evidence system behind Gateway trace capture, OTLP ingest, trace search, observability views, workflow triggers, billing usage, and retention. This document describes the current working model across write, read, search, summary maintenance, and reclaim paths.

The current production Worker path is Cloudflare-native:

• Cloudflare Workers expose API, OTLP, Gateway, jobs, and warehouse routing surfaces.
• Cloudflare Queues decouple OTLP request acceptance from decoding and warehouse writes.
• R2 stores staged OTLP payloads and trace attachments.
• R2 Data Catalog and Iceberg store telemetry tables.
• Cloudflare R2 SQL serves read queries.
• A Rust Cloudflare Container owns Iceberg table creation, append writes, summary row replacement,

and retention deletes.

Node and desktop deployments still use the same application interfaces, but their local storage adapter is DuckDB-backed. The Worker path is the canonical production path described below.

Core Rules

Telemetry follows a small set of rules that keep the storage model understandable and scalable:

• Telemetry is physically isolated by organization id. The organization id is used as the Iceberg

namespace and must be a SQL-compatible identifier such as orgdefault.

• Table names are canonical. Production should use the built-in table names and should not invent

per-deployment telemetry table names.

• Project isolation is a row-level field inside the organization namespace. Every projected row

carries project_id.

• Every warehouse row carries event_date as a YYYY-MM-DD UTC string. Time-bounded reads and

retention deletes must push this column so R2 SQL and Iceberg can prune partitions.

• Writes are append-oriented. Raw signal tables are append-only; read code deduplicates by stable

keys. Summary tables are rebuildable derived read models and are replaced by key.

• Worker reads default to the summaries read model. Production should not silently fall back to

direct append-only span scans when summary tables are missing.

• The API and OTLP surfaces resolve auth, project context, and queueing. The warehouse executor owns

table creation, append idempotency, Iceberg mutations, and catalog/storage execution.

Runtime Pieces

Physical Tables

Each organization namespace contains the same canonical tables:

The canonical fully qualified table name is:

<orgId>.<tableName>

For example:

orgdefault.spans
orgdefault.trace_summaries

Payload retention is controlled by PROVON_TELEMETRY_PAYLOAD_RETENTION:

• all writes both projected rows and payload rows.
• projected-only and none write projected rows without payload rows in the Worker warehouse path.

Projected rows are enough for list, search, summaries, dashboard, and usage queries. Payload rows are needed for full span, log, and metric bodies.

Write Path

Telemetry reaches the warehouse through two main write paths: external OTLP ingest and internal Gateway trace capture.

OTLP Ingest

OTLP routes are mounted only when both a blob store and an ingest queue are wired.

Accepted endpoints:

• POST /v1/traces
• POST /v1/logs
• POST /v1/metrics

The request must resolve to a project context and carry telemetry:ingest. Public API-key requests are checked for ingestion suspension before the payload is accepted.

The request path is:

1. The OTLP route validates the signal, auth context, body size, content type, and content encoding.
2. JSON and protobuf OTLP payloads are accepted. gzip, deflate, and identity transport

encodings are supported.

1. The decoded payload is streamed into the blob store under an otlp-staging key.
2. The route computes a payload hash and enqueues an IngestJob with project context, signal,

format, blob key, hash, idempotency key, byte size, and creation time.

1. The producer receives 200 only after the blob write and queue send have both succeeded.
2. If producer-side queue admission is saturated, the route returns 503 with

INGEST_QUEUE_BACKPRESSURE and an optional Retry-After.

The jobs runtime consumes ingest messages through the Cloudflare Queue handler:

1. The queue handler partitions mixed queue batches and passes ingest messages to IngestWorker.
2. IngestWorker claims a bounded batch, drops oversized jobs as invalid, and defers jobs outside

the per-tick byte budget.

1. It loads staged blobs, decodes OTLP JSON or protobuf, validates records, and groups accepted

records by (orgId, projectId, signal).

1. Trace records are normalized, trace attachments are externalized into the blob store, and dirty

trace refs are collected.

1. Groups are flushed to the trace, log, or metric store with source provenance that includes job id,

idempotency key, and record range.

1. Successful jobs are acked and their staged OTLP blobs are deleted.
2. Failed jobs are nacked with retry backoff. Exhausted jobs are dead-lettered and their staged blobs

are deleted.

1. After successful trace writes, dirty traces are enqueued onto the summary maintenance queue.

Gateway Trace Capture

Gateway calls can write telemetry without going through external OTLP staging.

The Gateway path:

1. Gateway evaluates trace-capture settings and captures request, response, attempt, usage, cost,

error, guardrail, and attachment data.

1. Captured Gateway telemetry is sent to the OTel Worker through RPC or through the

GATEWAY_TELEMETRY_QUEUE.

1. Gateway telemetry is converted to normalized trace spans.
2. Request and response body attachments are externalized into blob storage when present.
3. The spans are appended to the same trace store as OTLP spans.
4. The project is marked activated and the affected trace is marked dirty for summary maintenance.

Gateway telemetry and external OTLP telemetry therefore converge in the same organization namespace and project-scoped tables.

Warehouse Append

Worker stores do not write Iceberg tables directly. They send row batches to the internal warehouse executor through the PROVON_TELEMETRY_WAREHOUSE_EXECUTOR Service Binding.

The internal append call is:

POST /append

The request carries the organization id. The warehouse Worker uses the organization namespace to choose a Container instance. The Container then:

1. Normalizes the organization namespace.
2. Splits the request into table-specific row groups.
3. Ensures the target namespace and table exist.
4. Appends projected and payload rows into the canonical Iceberg table.
5. Coalesces append-only table writes by window, max rows, and max bytes to reduce small commits.
6. Uses source provenance as an in-memory append ledger so replayed job segments are not appended

twice in the same Container generation.

1. Replaces summary rows by key before appending rebuilt rows.

Append coalescing defaults to a short window and bounded batch size. It can be tuned with:

• PROVON_TELEMETRY_WAREHOUSE_APPEND_COALESCE_MS
• PROVON_TELEMETRY_WAREHOUSE_APPEND_COALESCE_MAX_ROWS
• PROVON_TELEMETRY_WAREHOUSE_APPEND_COALESCE_MAX_BYTES

Summary Maintenance

trace_summaries is the default derived read model. It exists so common reads do not need to scan raw append-only spans.

The default Worker dependency chain is:

spans -> trace_summaries -> query-time conversation/user/dashboard rollups

After trace writes, the ingest and Gateway paths call markTracesDirty() with trace ids and, when available, event_date. Dirty trace refs are normalized and placed on TELEMETRY_SUMMARY_QUEUE.

The jobs runtime drains summary jobs by project:

1. It groups dirty trace jobs by (orgId, projectId).
2. It projects trace_summaries from append-only spans through R2 SQL.
3. It writes trace summary rows back through the warehouse append path.
4. The warehouse executor replaces summary rows by their natural keys.
5. Jobs are acked only after compaction succeeds. Failures retry with backoff and are eventually

acknowledged with an error log after max attempts.

The Worker default is:

PROVON_TELEMETRY_READ_MODEL=summaries

When this mode is active in writer Workers, TELEMETRY_SUMMARY_QUEUE is required. If a summary table is missing, production reads should fail with a clear configuration/materialization error instead of falling back to append-only spans. PROVON_TELEMETRY_ALLOW_SUMMARY_MISSING_TABLE_FALLBACK=true is a local or test compatibility escape hatch, not a production mode.

Higher-level conversation, user, and dashboard views are aggregated from trace_summaries at query time. This keeps the Worker/Iceberg write path to one derived index instead of expanding writes into dashboard-specific gold tables.

The Node/DuckDB runtime follows the same default: dirty trace maintenance rebuilds only trace_summaries; conversation, user, and dashboard views are derived when queried.

Read Path

The public read surface is API-key and session aware. Public /v1/* routes require API-key access; project-scoped Workspace routes resolve the requested project and still require telemetry:read where API-key capabilities apply.

Important read endpoints:

Reads use the R2 SQL executor for Worker deployments. This executor:

• Inlines bound parameters into SQL literals.
• Rejects mutating SQL such as INSERT, UPDATE, DELETE, CREATE, and DROP.
• Sends read SQL to Cloudflare R2 SQL.
• Enforces a maximum response size and asks callers to narrow projections or add limits.
• Logs query fingerprints, row counts, response sizes, durations, and failures.

Trace detail reads intentionally use summary rows to locate the trace's event_date. In append-only Worker mode, a missing summary row causes trace reads to return empty detail rather than scanning all span partitions. This preserves the rule that summary materialization must exist before production trace detail reads rely on append-only spans.

Logs and metrics require an explicit time range:

• range=<named range>
• or startMs=<epoch ms>&endMs=<epoch ms>

The API rejects log and metric list calls without a range. Trace search also requires a range.

Search And Listing

Trace listing is keyset-paginated. Cursors encode the last row position instead of using offsets:

<started_at>\t<trace_id>

Supported trace search types:

• trace-id
• span-id
• user-id
• conversation-id
• span-name
• operation-name
• content
• input
• output

When a trace search query or explicit search type is present, range is required. The range is converted to both millisecond predicates and event_date predicates so the query can prune Iceberg partitions.

Trace list filters can also include:

• level filters: ok, error, unknown
• structured filter state compiled into SQL conditions
• exact trace ids for workflow/event consumers

Conversation and observed-user lists are also keyset-paginated and can use range, text search, and structured filters. They read summary tables by default.

Log and metric lists are keyset-paginated independently:

• Log cursor: <time_ms>\t<id>
• Metric cursor: <time_ms>\t<metric_name>\t<id>

Retention And Reclaim

Retention has two different reclaim concerns:

• Queue and blob cleanup for transient ingest payloads.
• Warehouse retention for persisted telemetry rows.

Staged OTLP Payload Reclaim

Staged OTLP blobs are temporary. They are deleted after:

• successful ingest job ack,
• invalid oversized job ack,
• or final failed/dead-letter handling.

If queue enqueue fails after a blob was staged, the route attempts to delete the staged blob before returning the failure.

Warehouse Data Retention

The Cloudflare jobs Worker runs scheduled maintenance and calls runDataRetentionOnce().

The Worker retention path:

1. Reads organization retention targets from the meta-store.
2. Resolves retention days from billing plan and cloud config.
3. Computes a UTC cutoff date from now - retentionDays.
4. Processes organizations in bounded batches.
5. Sends table-specific data-file delete requests through the warehouse executor.
6. Retries each organization with a per-organization timeout.
7. Accumulates rows-deleted and Iceberg maintenance stats.

Cloudflare warehouse retention prunes expired data files from:

• span_payloads
• spans
• log_payloads
• logs
• metric_payloads
• metrics
• trace_summaries

The delete predicate is:

event_date < <cutoff-date>

The Worker production path is organization-scoped because telemetry tables are physically isolated by organization namespace. The Rust executor plans matching Iceberg data files with the cutoff predicate, writes a ManifestStatus::Deleted data manifest, and commits an Operation::Delete snapshot. It does not create row-level equality delete files for warehouse retention.

Retention knobs:

• PROVON_DATA_RETENTION_INTERVAL_MS
• PROVON_DATA_RETENTION_BATCH_SIZE
• PROVON_DATA_RETENTION_DELETE_TIMEOUT_MS
• PROVON_DATA_RETENTION_MAX_RETRIES
• PROVON_DATA_RETENTION_DELETE_MAX_FILES

The executor reports maintenance stats:

• Iceberg data files deleted
• Iceberg rows deleted
• Iceberg manifest files written
• Iceberg snapshots committed
• whether the data-file delete limit was reached

Cloudflare warehouse retention currently removes expired telemetry data files from the active Iceberg snapshot. OTLP staging blobs are reclaimed by the ingest worker. Trace attachment blob retention is implemented in the DuckDB project-retention path; the Cloudflare organization-retention path should be extended separately if attachment blobs need lifecycle deletion independent of warehouse retention.

Operational Configuration

Worker telemetry depends on these binding groups:

Important ingest knobs:

• PROVON_OTLP_MAX_BYTES
• PROVON_OTLP_MAX_BATCH_BYTES
• PROVON_OTLP_MAX_JOB_BYTES
• PROVON_OTLP_WORKER_BATCH_SIZE
• PROVON_OTLP_WORKER_LOCK_MS
• PROVON_OTLP_FLUSH_MAX_ROWS
• PROVON_OTLP_FLUSH_MAX_BYTES
• PROVON_OTLP_MAX_PENDING_JOBS

Important read-model knobs:

• PROVON_TELEMETRY_READ_MODEL=summaries
• PROVON_TELEMETRY_PAYLOAD_RETENTION=all
• PROVON_TELEMETRY_ALLOW_SUMMARY_MISSING_TABLE_FALLBACK=false

Production should keep the canonical table names. Environment hooks for table-name overrides exist in some runtime types for compatibility and tests, but the current architecture relies on organization namespaces plus canonical table names as the single source of truth.

Failure Semantics

Telemetry writes are at-least-once at the queue layer and idempotent by stable row/source identity at the append layer.

Important failure behavior:

• OTLP request acceptance is durable only after both blob staging and queue enqueue succeed.
• Enqueue failure attempts to delete the staged blob.
• Queue retries use exponential backoff.
• Exhausted ingest jobs are dead-lettered.
• Ingest job failures do not block other jobs in the same batch.
• Summary compaction jobs retry independently from raw telemetry ingest.
• Summary enqueue failure is treated as part of ingest success handling so the raw job can be retried.
• R2 SQL reads are read-only and size-limited.
• Warehouse retention failures are retried per organization and then surfaced to scheduled

maintenance as failures.

Related Code

• packages/api/src/observability/otlp-route.ts
• packages/observability/src/ingest/worker.ts
• services/cloudflare-worker/src/queue-handler.ts
• services/cloudflare-worker/src/observability.ts
• services/cloudflare-worker/src/r2-data-catalog-telemetry-store.ts
• services/cloudflare-worker/src/r2-sql-executor.ts
• services/cloudflare-worker/src/telemetry-warehouse-entry.ts
• services/cloudflare-worker/src/telemetry-warehouse-executor.ts
• services/cloudflare-worker/src/telemetry-summary-compaction.ts
• services/cloudflare-worker/src/telemetry-data-retention.ts
• services/cloudflare-worker/containers/telemetry-warehouse-executor/src/append.rs
• services/cloudflare-worker/containers/telemetry-warehouse-executor/src/summaries.rs
• services/cloudflare-worker/containers/telemetry-warehouse-executor/src/retention.rs
• packages/telemetry-store/src/schema/org-tables.ts
• packages/telemetry-store/src/schema/read-model.ts
• packages/telemetry-store/src/queries/telemetry-observation-query-service.ts
• packages/telemetry-store/src/operations/iceberg-data-retention.ts