Skip to content

Run The Gateway In Production

A production rollout is complete only when model execution, policy enforcement, and trace evidence all work under the same project boundary. Treat the provider response and the Pro

View as Markdown Open the plain-text version of this page.
On this page Browse sections

Complete the Gateway quickstart and migration guide before moving application traffic.

1. Define The Boundary#

Use a separate Provon project API key for each application and environment boundary. The workload receives the project key; Provon stores upstream credentials in project-scoped Provider Keys.

text
application
  -> Provon project API key
  -> Gateway policy and routing
  -> Provider Key
  -> upstream provider

For production:

  • expose the Gateway over HTTPS;
  • keep Workbench sessions and application API keys separate;
  • grant the application key gateway:invoke, plus only the other capabilities it needs;
  • rotate project and provider keys independently;
  • use Specific model mappings when the target requires a strict allowlist.

2. Validate Request Compatibility#

Inventory each request shape before rollout:

  • endpoint family;
  • provider-qualified or logical model ID;
  • streaming;
  • tool calling;
  • structured output or JSON mode;
  • vision, audio, or other multimodal input;
  • maximum request and response size;
  • client timeout and cancellation behavior.

Query the exact endpoint and features:

bash
curl \
  "$PROVON_BASE_URL/v1/gateway/model-selection?endpoint=chat-completions&features=streaming,tool-calling" \
  -H "Authorization: Bearer $PROVON_API_KEY"

Do not infer compatibility from a provider name. Provon filters candidates by declared endpoint and feature support, and upstream providers may impose stricter limits.

3. Make Routing Explicit#

Start a rollout with a provider-qualified model such as openai/gpt-5-mini. This isolates Gateway transport from routing-policy changes.

Add dynamic routing only when its behavior is intentional:

Requirement Mechanism
Pin one provider <provider>/<model>
Expose a stable application model Plain model plus a project Model Policy
Select across mapped targets provon/auto plus Auto Router
Recover on transient failure Bounded retry and ordered fallback
Avoid repeatedly failing targets Circuit breaker and target health

Retries and fallback are disabled unless configured. Keep recovery inside the overall request deadline, and do not retry authentication, invalid-request, guardrail, or hard-budget failures.

4. Set Hard Resource Limits#

Attach request, token, and cost limits to the project API key before increasing traffic.

Use:

  • request rate limits for throughput boundaries;
  • token rate limits for burst control;
  • token or cost budgets for cumulative spend;
  • model-scoped limits when expensive routes need a separate boundary.

Provon reserves estimated usage before routing and settles against provider-reported usage when available. Test both the allowed path and the expected 429 rejection contract.

For Provon Cloud managed models, fund and monitor the organization Cloud AI Gateway credits balance separately. A depleted balance returns 402; BYOK requests remain on the upstream provider's billing boundary.

See Gateway governance.

5. Introduce Guardrails In Monitor Mode#

Run new guardrails in Monitor against representative traffic before enforcing them. Review:

  • match categories and false positives;
  • input versus output direction;
  • redaction or adjustment behavior;
  • blocked-request evidence;
  • client handling of the 409 block contract.

Application authorization remains the source of user permissions. Gateway tool policies are an additional request boundary, not a replacement for application checks.

6. Choose Evidence And Privacy Policy#

Keep Request traces set to All during rollout. Verify that every test request produces:

  • a root Gateway span;
  • the expected provider-attempt spans;
  • requested, Gateway, and upstream model IDs;
  • status, latency, usage, and cost when available;
  • conversation and agent context;
  • guardrail and recovery events.

Before steady state, decide:

  • whether full request and response bodies may be retained;
  • which pseudonymous user and conversation identifiers are required;
  • whether All or Filtered capture preserves enough context for diagnostic Rules;
  • the project telemetry retention period.

Filtered capture reduces volume but can fragment multi-turn conversations. If Provon should diagnose conversation-level failures, prefer complete capture and control storage with retention.

See Gateway evidence.

7. Roll Out In Stages#

A practical sequence is:

  1. Send synthetic requests through one Provider Key.
  2. Move a small production cohort with a provider-qualified model.
  3. Compare response correctness, error rate, latency, tokens, and cost.
  4. Verify trace completeness and request-ID correlation.
  5. Enable bounded same-model recovery.
  6. Introduce Model Policies or Auto Router only if required.
  7. Increase traffic after each stage passes.

Log x-provon-request-id with the application request. It connects a client-visible failure to the Gateway root span and provider attempts.

8. Prepare Rollback#

Keep rollback independent from diagnosis:

  • preserve the previous provider endpoint configuration during the rollout window;
  • make the Gateway base URL an application configuration value;
  • keep provider-qualified models available while testing policy-based routing;
  • know how to pause the project Gateway from Providers;
  • do not delete Provider Keys or mappings until the new path is stable.

Restoring the prior base URL rolls model traffic back. It does not remove Gateway traces already captured, so the failed rollout remains available as diagnostic evidence.

Release Gate#

Do not complete the rollout until all items pass:

  • Gateway endpoint uses HTTPS.
  • Project and provider credentials are separated.
  • Required endpoint and features appear in discovery.
  • Provider-qualified canary requests succeed.
  • Streaming, tools, and structured outputs are verified where used.
  • Request, token, and cost limits reject as designed.
  • Guardrails have been reviewed with representative traffic.
  • Traces contain conversation context and provider attempts.
  • Retention and payload policy match the data classification.
  • The application logs x-provon-request-id.
  • Rollback has been exercised.