# OpenTelemetry Setup

Provon accepts standard OTLP/HTTP telemetry. Existing instrumentation can keep its SDK, semantic
conventions, and Collector while Provon adds trace views and conversation diagnosis.

Use the [Gateway](../ai-gateway/index.md) instead when Provon should route the model call and enforce
request-time policy. Read [Tracing](./index.md) first when choosing between Gateway, OTLP, and
agent transcript capture.

Use the [Tracing quickstart](./quickstart.md) to send one transparent OTLP JSON example
before adding an SDK.

Before instrumenting production workloads, define the trace boundary and evidence fields in
[Trace model and instrumentation](./trace-model.md) and the
[Tracing attribute reference](../api/tracing-attributes.md).

## Choose An Instrumentation Path

| Starting point                             | Recommended path                                      |
| ------------------------------------------ | ----------------------------------------------------- |
| Application already exports OTLP/HTTP      | Change the endpoint and authentication header         |
| Framework already emits OTel GenAI spans   | Keep its instrumentor and verify the emitted evidence |
| Application has no tracing                 | Add the standard OTel SDK and manual agent spans      |
| Team needs central policy and fan-out      | Export through an OpenTelemetry Collector             |
| Provon should own the model request path   | Use [AI Gateway](../ai-gateway/index.md) instead      |
| Local coding agent writes transcript files | Use [Agent transcript sync](./agent-transcripts.md)   |

Provon compatibility is protocol-based. Any instrumentor that emits valid OTLP/HTTP can send
telemetry, but not every framework emits the messages, tool results, participant identity, and
terminal outcome needed by diagnostic Rules. Verify the result against
[Diagnosis-ready tracing](./best-practices.md) before treating an integration as complete.

## Quick Start

Create a project and project API key in the Workbench, then configure the exporter:

```bash
export PROVON_API_KEY="your_project_api_key"
export PROVON_OTEL_URL="https://otel.provon.dev/v1"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="$PROVON_OTEL_URL/traces"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $PROVON_API_KEY"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_RESOURCE_ATTRIBUTES="service.name=support-agent,deployment.environment.name=development"
```

The hosted OTEL URL already includes `/v1`. Use signal-specific exporter endpoints so an SDK does
not append a second `/v1`. Provon accepts:

```text
POST https://otel.provon.dev/v1/traces
```

The ingest contract is:

| Setting            | Support                                   |
| ------------------ | ----------------------------------------- |
| Transport          | OTLP over HTTP                            |
| Encoding           | Protobuf or JSON                          |
| Compression        | `gzip`, `deflate`, or identity            |
| Authentication     | `Authorization: Bearer <project API key>` |
| API-key capability | `telemetry:ingest`                        |

OTLP/gRPC is not exposed.

See the [OTLP/HTTP API](../api/otlp.md) for response semantics, limits, status codes, signal
availability, and retry behavior.

For local development, set `PROVON_OTEL_URL` to `http://127.0.0.1:3000/v1`.

If the SDK requires a complete trace URL:

```bash
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://otel.provon.dev/v1/traces"
```

Run the instrumented agent, flush the exporter, then verify the new record under **Traces**. A
successful response means the payload was staged and queued; trace summaries can appear
asynchronously.

## Python

Install the OpenTelemetry SDK and OTLP/HTTP exporter:

```bash
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
```

```python
import os

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(
    resource=Resource.create({"service.name": "support-agent"}),
)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint=f"{os.environ['PROVON_OTEL_URL']}/traces",
            headers={"Authorization": f"Bearer {os.environ['PROVON_API_KEY']}"},
        )
    )
)
trace.set_tracer_provider(provider)
```

Call `provider.shutdown()` before a short-lived process exits so the final batch is exported.

## Node.js

Install the Node SDK and protobuf OTLP exporter:

```bash
pnpm add @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-proto \
  @opentelemetry/resources @opentelemetry/semantic-conventions
```

```ts
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';
import { resourceFromAttributes } from '@opentelemetry/resources';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: 'support-agent',
  }),
  traceExporter: new OTLPTraceExporter({
    url: `${process.env.PROVON_OTEL_URL}/traces`,
    headers: {
      Authorization: `Bearer ${process.env.PROVON_API_KEY}`,
    },
  }),
});

sdk.start();
```

Initialize the SDK before importing modules that register framework or model-client instrumentation.
Otherwise early calls can run before an active tracer provider exists.

Call `await sdk.shutdown()` during graceful process termination.

## OpenTelemetry Collector

Route an existing Collector to Provon when you want centralized batching, retries, sampling, and
redaction:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  batch:

exporters:
  otlphttp/provon:
    traces_endpoint: https://otel.provon.dev/v1/traces
    headers:
      Authorization: Bearer ${env:PROVON_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/provon]
```

Keep the project API key in the Collector's secret environment. Do not place it in source control.

## Framework And Agent SDK Integrations

Provon does not require a framework-specific tracing package. An integration is compatible when it
can export valid OTLP/HTTP to Provon.

Use this order:

1. Enable the framework's native OpenTelemetry instrumentation when available.
2. Point its OTLP/HTTP exporter or Collector at Provon.
3. Run one representative agent workflow.
4. Inspect the emitted operation, participant, message, tool, retrieval, usage, and error fields.
5. Add manual spans or attributes only for evidence the instrumentor does not emit.

Protocol compatibility does not imply diagnosis readiness. Some instrumentors capture only model
requests; others use legacy or framework-specific attributes. Provon preserves unknown attributes,
but built-in projections and diagnostic Rules depend on the normalized fields in the
[Tracing attribute reference](../api/tracing-attributes.md).

Avoid enabling two instrumentors for the same model call. In particular, decide whether the Gateway
record and an application span represent one duplicated operation or two intentional layers of the
execution.

## Add Diagnosis Evidence

Valid OTel spans are enough for trace visibility. Diagnostic Rules additionally need the user goal,
ordered model and tool activity, failures, recovery, and terminal answer.

At minimum:

- set a stable resource `service.name`;
- put one bounded agent run or conversation turn in each trace;
- use `gen_ai.conversation.id` across related traces;
- preserve parent span IDs and tool call IDs;
- emit GenAI message, model, usage, tool, retrieval, and error attributes.

See [Trace model and instrumentation](./trace-model.md) for boundaries and naming rules. Use the
[Tracing attribute reference](../api/tracing-attributes.md) for exact normalized fields and
structured-value handling.

## Complete Examples

The repository includes minimal OTLP/HTTP agent services:

- [TypeScript](../../examples/agent-observability-ts/README.md)
- [Python](../../examples/agent-observability-python/README.md)
- [Go](../../examples/agent-observability-go/README.md)

These examples demonstrate exporter setup, W3C context extraction, a root agent span, a child tool
span, and graceful shutdown. Extend their attributes using the trace model before treating them as
diagnosis-ready production instrumentation.

## Production

Use [Production tracing](./production.md) to choose:

- conversation-consistent sampling;
- SDK or Collector batching and retry;
- payload bounds and source-side redaction;
- raw payload and data retention;
- environment and API-key isolation.

## Troubleshooting

- `401` or `403`: verify the project API key, Bearer header, and `telemetry:ingest`.
- No traces: confirm OTLP/HTTP rather than OTLP/gRPC and flush the exporter.
- Only traces: configure log and metric pipelines separately.
- No conversation grouping: add a stable recognized conversation attribute.
- No Findings: verify trajectory evidence, Rule scheduling, thresholds, and adjudication.
- Rejected request: inspect content type, content encoding, body size, and protobuf availability.
- Accepted but missing data: inspect background ingest logs for envelope or record validation errors.

See the full [troubleshooting guide](../get-started/troubleshooting.md).

## Related Docs

- [Tracing overview](./index.md)
- [Tracing quickstart](./quickstart.md)
- [OTLP/HTTP API](../api/otlp.md)
- [Trace model and instrumentation](./trace-model.md)
- [Tracing attribute reference](../api/tracing-attributes.md)
- [Diagnosis-ready tracing](./best-practices.md)
- [Multi-agent and distributed tracing](./multi-agent.md)
- [Production tracing](./production.md)