# Tracing Quickstart

Send one diagnosis-ready agent trace to Provon over standard OTLP/HTTP, inspect it, and run the
built-in diagnostic Rules with the CLI.

This guide uses a direct OTLP JSON request so the transport and evidence are visible. For
application SDK setup, continue with [OpenTelemetry setup](./opentelemetry.md). To capture model
calls through Provon instead, use the [Gateway quickstart](./gateway-quickstart.md).

## Prerequisites

- A running Provon deployment.
- A project API key with `telemetry:ingest` and `telemetry:read`.
- `curl`, `openssl`, and a POSIX-compatible shell.
- The `provon` CLI for the inspect and diagnose steps.

Configure the deployment and project:

```bash
export PROVON_BASE_URL="http://127.0.0.1:3000"
export PROVON_API_KEY="your_project_api_key"
export PROVON_PROJECT_ID="your_project_id"
```

## 1. Send A Trace

The example creates one bounded agent run with a child tool call. It uses stable operation names,
explicit participant identity, readable input and output, and a conversation ID.

```bash
TRACE_ID="$(openssl rand -hex 16)"
CONVERSATION_ID="quickstart-$TRACE_ID"
ROOT_SPAN_ID="$(openssl rand -hex 8)"
TOOL_SPAN_ID="$(openssl rand -hex 8)"
START_NS="$(($(date +%s) * 1000000000))"

curl "$PROVON_BASE_URL/v1/traces" \
  -H "Authorization: Bearer $PROVON_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @- <<JSON
{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": { "stringValue": "quickstart-support-agent" }
          },
          {
            "key": "deployment.environment.name",
            "value": { "stringValue": "development" }
          }
        ]
      },
      "scopeSpans": [
        {
          "scope": { "name": "provon.quickstart" },
          "spans": [
            {
              "traceId": "$TRACE_ID",
              "spanId": "$ROOT_SPAN_ID",
              "name": "invoke-support-agent",
              "kind": 1,
              "startTimeUnixNano": "$START_NS",
              "endTimeUnixNano": "$((START_NS + 3000000))",
              "status": { "code": 1 },
              "attributes": [
                {
                  "key": "gen_ai.operation.name",
                  "value": { "stringValue": "invoke_agent" }
                },
                {
                  "key": "gen_ai.conversation.id",
                  "value": { "stringValue": "$CONVERSATION_ID" }
                },
                {
                  "key": "gen_ai.agent.id",
                  "value": { "stringValue": "support-agent" }
                },
                {
                  "key": "gen_ai.input.messages",
                  "value": {
                    "stringValue": "[{\"role\":\"user\",\"content\":\"Check whether order 123 can be refunded.\"}]"
                  }
                },
                {
                  "key": "gen_ai.output.messages",
                  "value": {
                    "stringValue": "[{\"role\":\"assistant\",\"content\":\"Order 123 is eligible for refund review.\"}]"
                  }
                }
              ]
            },
            {
              "traceId": "$TRACE_ID",
              "spanId": "$TOOL_SPAN_ID",
              "parentSpanId": "$ROOT_SPAN_ID",
              "name": "lookup-refund-policy",
              "kind": 1,
              "startTimeUnixNano": "$((START_NS + 1000000))",
              "endTimeUnixNano": "$((START_NS + 2000000))",
              "status": { "code": 1 },
              "attributes": [
                {
                  "key": "gen_ai.operation.name",
                  "value": { "stringValue": "execute_tool" }
                },
                {
                  "key": "gen_ai.conversation.id",
                  "value": { "stringValue": "$CONVERSATION_ID" }
                },
                {
                  "key": "gen_ai.agent.id",
                  "value": { "stringValue": "support-agent" }
                },
                {
                  "key": "gen_ai.tool.name",
                  "value": { "stringValue": "lookup_order" }
                },
                {
                  "key": "gen_ai.tool.call.id",
                  "value": { "stringValue": "call-quickstart-1" }
                },
                {
                  "key": "gen_ai.tool.call.arguments",
                  "value": { "stringValue": "{\"order_id\":\"123\"}" }
                },
                {
                  "key": "gen_ai.tool.call.result",
                  "value": {
                    "stringValue": "{\"status\":\"eligible_for_review\"}"
                  }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}
JSON

printf 'trace id: %s\n' "$TRACE_ID"
printf 'conversation id: %s\n' "$CONVERSATION_ID"
```

A successful request returns asynchronous acceptance:

```json
{
  "partialSuccess": {},
  "queued": true,
  "jobId": "ing_..."
}
```

This confirms staging and queue admission, not query readiness. See the
[OTLP/HTTP API](./otlp-api.md) for the complete delivery contract.

## 2. Verify The Evidence

Open **Traces** and search for the printed trace ID. The trace should contain:

```text
invoke-support-agent
`-- lookup-refund-policy
```

Verify:

- `service.name` is `quickstart-support-agent`;
- the tool span is a child of the agent span;
- both spans use the printed conversation ID;
- the root input and output are readable;
- the tool arguments and result are retained.

If the trace is not visible yet, wait for background ingest and summary materialization. Use
[Tracing troubleshooting](./troubleshooting.md#otlp-succeeds-but-no-trace-appears) when an accepted
payload never appears.

## 3. Inspect It From The CLI

Configure CLI authentication:

```bash
provon auth token set \
  --api-key "$PROVON_API_KEY" \
  --project "$PROVON_PROJECT_ID"
```

Search and summarize the trace:

```bash
provon traces search "$TRACE_ID" --search-type trace-id --range 1h
provon traces summarize "$TRACE_ID"
```

`traces summarize` reports telemetry facts such as duration, spans, errors, tokens, cost, and slow
operations. It does not run diagnostic Rules or create Findings.

## 4. Diagnose The Same Evidence With The CLI

Export a trace bundle, then diagnose it without uploading or persisting a Finding:

```bash
provon traces export "$TRACE_ID" --out ./quickstart-trace
provon diagnose ./quickstart-trace --deterministic-only
```

The healthy example should normally produce no Finding. That is a successful result: traces are
evidence, while Findings are supported problems.

Remove `--deterministic-only` when the calling Agent can perform the hash-bound adjudication round
trip. See [Trace diagnosis with CLI](./trace-diagnosis.md).

## Next Steps

- Replace the direct request with an [OpenTelemetry SDK or Collector](./opentelemetry.md).
- Review the [trace model](./tracing-model.md) and
  [attribute reference](./tracing-attributes.md).
- Apply the [diagnosis-ready tracing checklist](./tracing-best-practices.md).
- Learn the [Workbench](./trace-explorer.md), [CLI](./tracing-cli.md), and
  [Trace read API](./trace-api.md) investigation paths.
- Configure [production tracing](./tracing-production.md).