Observability and agent tracing

OpenTrip uses two complementary observability surfaces:

• Cloudflare Workers Logs or Docker stdout contain newline-delimited JSON logs.
• Sentry contains request traces, errors, and AI SDK 7 OpenTelemetry spans.

Sentry is disabled when SENTRY_DSN is absent. Observability failures never disable the API or trip agent.

Configuration

Production uses 100% Sentry sampling for trip-agent routes, 10% for other API routes, and 0% for health checks. Cloudflare persists all logs and samples platform request traces at 10%.

Production must keep AI_TELEMETRY_RECORD_CONTENT=false unless an explicitly approved, time-bounded investigation requires content capture. Setting it to true sends trip conversation text, prompts, model replies, and tool arguments/results to Sentry. Authorization headers, cookies, credentials, database URLs, signed URL queries, data URLs, base64, and attachment bytes are always removed. Limit Sentry project access and set a retention policy appropriate for travel, reservation, and expense data. Set the flag to false and redeploy to stop content capture immediately.

Trace model

Every response carries x-request-id. Agent chat responses also carry x-agent-turn-id; CORS exposes both headers. A supplied request id is retained only when it matches the safe request-id format; otherwise the API generates a UUID. Agent executions additionally carry tripId, agentSessionId, turnId, messageId, suggestionId, and toolCallId when available.

requestId is the application UUID. Cloudflare's $metadata.requestId is the invocation id (often the Ray/request grouping id) and can contain several application log events. The log CLI labels and expands these separately.

The initiating user UI message id is the stable turnId. A later tool-approval request creates a new HTTP trace but keeps the same turnId and AI SDK toolCallId, so it can be found without keeping a span open across user input.

Typical explicit-chat trace:

HTTP request
└── opentrip.agent.chat
    ├── opentrip.agent.persist_message
    ├── ai.generateText / ai.streamText
    │   ├── model inference
    │   └── tool execution
    │       ├── opentrip.provider.*
    │       └── opentrip.trip.operation.apply
    └── opentrip.agent.persist_message

Ambient replies, addressed checks, operation evaluations, and suggestion responses use their own opentrip.agent.* parent spans. Deferred work is kept alive by the Worker execution context and logs the originating request and turn identifiers. Weather, geo, lodging, street-view, and attachment reads have explicit child spans beneath their AI tool spans. If the browser disconnects while the independent SSE drain finishes successfully, the chat span records opentrip.agent.client_disconnected=true without reporting a false failure.

Debugging workflow

1. Copy x-request-id from the failing browser Network response. For agent UI failures also copy a message, suggestion, or tool-call id from the payload.

2. Search Sentry Discover using the most specific available attribute:

   request.id:<request-id>
   opentrip.agent.turn_id:<turn-id>
   opentrip.agent.message_id:<message-id>
   opentrip.agent.suggestion_id:<suggestion-id>
   opentrip.trip.id:<trip-id>
   gen_ai.tool.call.id:<tool-call-id>
   opentrip.agent.ui.contract_version:2026-07-16.1
   opentrip.agent.ui.outcome:rejected
   opentrip.agent.ui.reason:<reason>

3. Open the trace waterfall. Inspect authorization/context loading, AI steps, inference finish reason and token usage, tool execution, approval, domain apply, message persistence, and stream completion in that order.

4. If Sentry has no sampled trace, query historical Workers Logs. The command first finds the matching event and then expands every Cloudflare invocation so sibling tool/provider/persistence events appear in chronological order:

   pnpm logs:cf -- --request-id <request-id> --since 1h
   pnpm logs:cf -- --turn-id <turn-id> --since 24h
   pnpm logs:cf -- --message-id <message-id>
   pnpm logs:cf -- --tool-call-id <tool-call-id>
   pnpm logs:cf -- --trip-id <trip-id> --event agent.tool.failed

   Wrangler itself exposes live tailing, not historical replay. For a live reproduction use the same repository command, which delegates to Wrangler:

   pnpm logs:cf -- --live --contains <request-or-turn-id>

5. In Docker, use the same fields:

   docker compose -f deploy/docker/compose.yaml logs -f api \
     | jq -R 'fromjson? | select(tostring | contains("<request-or-turn-id>"))'

6. Cross-check persisted state without selecting message contents unnecessarily:

   SELECT id, trip_id, role, source, trip_version, created_at
   FROM agent_messages
   WHERE id = '<message-id>';
   
   SELECT id, trip_id, message_id, status, trip_version, created_at, updated_at
   FROM agent_suggestions
   WHERE id = '<suggestion-id>';

   Compare the persisted trip_version with the mutation span's before/after versions to distinguish stale reads, concurrent claims, and failed writes.

Symptom guide

Log event reference

All application logs are one-line JSON with timestamp, level, event, and available correlation ids. Important events include:

• http.request.completed
• agent.persist_message
• agent.stream.complete
• agent.suggestion.created
• agent.addressed_check_failed
• agent.ambient_reply_failed
• agent.operation_evaluation_failed
• agent.deferred_task_failed
• worker.fetch_failed
• worker.pool_dispose_failed

Do not paste full Sentry events into public issues. Share the trace or event id with an authorized project member instead.

Historical Workers Logs setup

Create a dedicated Cloudflare API token with Workers Observability Write. Despite the operation being read-only, this is the permission currently required by the Telemetry Query API. Do not reuse the deployment token and do not add the query token to Worker secrets.

cp deploy/cloudflare/observability.example.env .env.observability
# Fill the two values, then load them into the current shell.
set -a; source .env.observability; set +a

pnpm logs:cf -- --request-id <id> --since 1h

Historical options are composable and use AND semantics:

If CLOUDFLARE_ACCOUNT_ID is absent, the command uses wrangler whoami --json only when it exposes exactly one account. Authentication/API errors go to stderr and return non-zero; zero matching events is a successful query.

Find a message from its text

agent.persist_message stores a pseudonymous messageFingerprint, never the message body. Normalization is Unicode NFKC followed by whitespace collapse and trim; the digest is lowercase SHA-256 prefixed with sha256:. File parts, tool payloads, attachments, and generated UI are excluded.

Prefer stdin so sensitive text does not enter shell history or the process list:

pbpaste | pnpm logs:cf -- --message-stdin --since 2h

The Agent message menu's Copy debug info action provides tripId, messageId, available live requestId/turnId, toolCallIds, source, createdAt, and the same fingerprint. A hash is pseudonymous rather than anonymous and can still be dictionary-tested; restrict Workers Logs access accordingly.

Street-view card runbook

For a missing or invalid deterministic card, query the copied message id and inspect these events and spans in order:

1. opentrip.provider.geo.place_search exists for a place request and records requestedLimit=5, result count, duration, request id, and turn id. It is absent for a coordinate request.
2. opentrip.provider.street_view.search records radiusMeters=100, requestedLimit=5, provider outcome, result count, duration, and the same correlation ids. Each application provider span occurs at most once.
3. street_view.provider.request_failed and street_view.provider.retry_scheduled: distinguish 401/403 configuration, 429, timeout, and provider 5xx inside the single provider operation.
4. agent.street_view_grounding.completed records only request kind, outcome, result count, duration, and correlation ids; it never records raw place text.
5. street_view.cache.hit/miss: a card immediately following a successful search should use the 15-minute metadata cache; preview bytes are cached after their first successful read.
6. agent.persist_message: confirm the sanitized assistant message and its fingerprint were written.

Production acceptance requires both provider spans for a place request, no MiniMax/model span in the street-view trace, and a persisted card image id that belongs to the same message's outcome=found grounding part. A coordinate request requires only the street-view provider span. There are no street-view tool events, generated-UI repair events, or replay telemetry. Keep AI_TELEMETRY_RECORD_CONTENT=false; provider events must not add raw place text.