> ## Documentation Index
> Fetch the complete documentation index at: https://docs.avidoai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Tracing

> Capture every step of your LLM workflow and send it to Avido for replay, evaluation, and monitoring.

When your chatbot conversation or agent run is in flight, **every action becomes an *event***.\
Bundled together they form a **trace** – a structured replay of what happened, step‑by‑step.

| Event       | When to use                                                                     |
| ----------- | ------------------------------------------------------------------------------- |
| `trace`     | The root container for a whole conversation / agent run.                        |
| `llm`       | Start and end of every LLM call.                                                |
| `tool`      | Calls to a function / external tool invoked by the model.                       |
| `retriever` | RAG queries and the chunks they return.                                         |
| `log`       | Anything else worth seeing while debugging (system prompts, branches, errors…). |

<Info>
  The full schema lives in <a href="/api-reference/ingestion" target="_blank">API ▸ Ingestion</a>.
</Info>

***

## Understanding `testId` and `traceId`

Two IDs appear on trace events — here's what each one does:

| Field     | Who sets it                      | When to include                                                                                                                                                                                     |
| --------- | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `testId`  | Avido (via [webhook](/webhooks)) | **Only** when the trace originates from an Avido test. Pass the `testId` from the webhook payload. Do **not** set it for traces that come from real user interactions.                              |
| `traceId` | You (optional)                   | Setting `traceId` client-side is **optional** — the server generates one automatically if you don't. If you do set it, use a random UUID and keep the **same value across all steps** in the trace. |

<Callout type="warning">
  Do **not** set `traceId` equal to `testId`. They serve different purposes: `testId` links the trace to an Avido test run for evaluation, while `traceId` groups events together into a single trace.
</Callout>

***

## Recommended workflow

1. **Collect events in memory** as they happen.
2. **Flush** once at the end (or on fatal error).
3. **Add a `log` event** describing the error if things blow up.
4. **Keep tracing async** – never block your user.
5. **Evaluation‑only mode?** Only ingest when the run came from an Avido test → check for `testId` from the [Webhook](/webhooks).
6. **LLM events** should contain the *raw* prompt & completion – strip provider JSON wrappers.

***

## SDK installation

<CodeGroup>
  ```bash Node theme={null}
  npm install @avidoai/sdk-node
  ```

  ```bash Python theme={null}
  pip install avido
  ```
</CodeGroup>

***

## Ingesting events

You can send events:

* **Directly via HTTP**
* **Via our SDKs** (Node: `@avidoai/sdk-node`, Python: `avido`)

<Callout type="info">
  When authenticating with an API key, include both the `x-api-key` and `x-application-id`
  headers. The application ID should match the application that owns the key so the
  request can be authorized.
</Callout>

<CodeGroup>
  ```bash cURL (default) theme={null}
  curl --request POST \
    --url https://api.avidoai.com/v0/ingest \
    --header 'Content-Type: application/json' \
    --header 'x-application-id: <application-id>' \
    --header 'x-api-key: <api-key>' \
    --data '{
    "events": [
      {
        "type": "trace",
        "timestamp": "2025-05-15T12:34:56.123455Z",
        "referenceId": "123e4567-e89b-12d3-a456-426614174000",
        "metadata": {
          "source": "chatbot"
        }
      },
      {
        "type": "llm",
        "event": "start",
        "traceId": "123e4567-e89b-12d3-a456-426614174000",
        "timestamp": "2025-05-15T12:34:56.123456Z",
        "modelId": "gpt-4o-2024-08-06",
        "params": {
          "temperature": 1.2
        },
        "input": [
          {
            "role": "user",
            "content": "Tell me a joke."
          }
        ]
      }
    ]
  }'
  ```

  ```ts Node theme={null}
  import Avido from '@avidoai/sdk-node';

  const client = new Avido({
    applicationId: process.env['AVIDO_APPLICATION_ID'],
    apiKey: process.env['AVIDO_API_KEY'],
  });

  // testId comes from the webhook payload — only include it for
  // Avido test runs, not for real user interactions.
  const ingest = await client.ingest.create({
    events: [
      {
        timestamp: '2025-05-15T12:34:56.123456Z',
        type: 'trace',
        testId: webhookBody.testId, // omit for user-originated traces
      },
      {
        timestamp: '2025-05-15T12:34:56.123456Z',
        type: 'tool',
        toolInput: 'the input to a tool call',
        toolOutput: 'the output from the tool call',
      }
    ],
  });

  console.log(ingest.data);
  ```

  ```python Python theme={null}
  import os
  from avido import Avido

  client = Avido(
      application_id=os.environ.get("AVIDO_APPLICATION_ID"),
      api_key=os.environ.get("AVIDO_API_KEY"),
  )

  # test_id comes from the webhook payload — only include it for
  # Avido test runs, not for real user interactions.
  ingest = client.ingest.create(
      events=[
          {
              "timestamp": "2025-05-15T12:34:56.123456Z",
              "type": "trace",
              "testId": webhook_body.get("testId"),  # omit for user-originated traces
          },
          {
              "timestamp": "2025-05-15T12:34:56.123456Z",
              "type": "tool",
              "toolInput": "the input to a tool call",
              "toolOutput": "the output from the tool call",
          },
      ],
  )

  print(ingest.data)
  ```
</CodeGroup>

***

## Tip: map your IDs

If you already track a conversation / run in your own DB, pass that same ID as `referenceId`.\
It makes liftover between your system and Avido effortless.

***

## Event types in depth

Every call to `client.ingest.create()` accepts an array of events.
The first event is usually a `trace` (the root container), followed by one or more step events.

### `trace` — root container

Every ingestion batch should start with a `trace` event. It groups all subsequent steps together.

<CodeGroup>
  ```ts Node theme={null}
  await client.ingest.create({
    events: [
      {
        type: 'trace',
        timestamp: new Date().toISOString(),
        referenceId: 'your-internal-conversation-id', // links to your system
        testId: webhookBody.testId,                   // links to an Avido test run
        metadata: { source: 'chatbot', userId: '42' },
      },
    ],
  });
  ```

  ```python Python theme={null}
  client.ingest.create(
      events=[
          {
              "type": "trace",
              "timestamp": datetime.now(timezone.utc).isoformat(),
              "referenceId": "your-internal-conversation-id",
              "testId": webhook_body.get("testId"),
              "metadata": {"source": "chatbot", "userId": "42"},
          },
      ],
  )
  ```
</CodeGroup>

| Field         | Required | Description                                                                                                                                                                |
| ------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type`        | Yes      | Always `"trace"`                                                                                                                                                           |
| `timestamp`   | Yes      | ISO 8601 timestamp                                                                                                                                                         |
| `referenceId` | No       | Your own ID for this conversation / run                                                                                                                                    |
| `traceId`     | No       | Client-provided UUID to identify this trace. Optional — the server generates one if omitted. If set, use the **same** random UUID for all steps in the trace.              |
| `testId`      | No       | The `testId` from a [webhook payload](/webhooks) — links the trace to an Avido test run. **Only set this for test-originated traces; omit it for real user interactions.** |
| `metadata`    | No       | Arbitrary key-value pairs for filtering and search                                                                                                                         |

### `llm` — LLM calls

Track every call to a language model. Use `event: "start"` when the call begins and `event: "end"` when it completes, or send a single event with both input and output after the call finishes.

<CodeGroup>
  ```ts Node theme={null}
  await client.ingest.create({
    events: [
      {
        type: 'trace',
        timestamp: new Date().toISOString(),
      },
      {
        type: 'llm',
        event: 'start',
        timestamp: startTime.toISOString(),
        modelId: 'gpt-4o-2024-08-06',
        params: { temperature: 0.7, maxTokens: 1024 },
        input: [
          { role: 'system', content: 'You are a helpful assistant.' },
          { role: 'user', content: 'Summarize this document.' },
        ],
      },
      {
        type: 'llm',
        event: 'end',
        timestamp: endTime.toISOString(),
        output: [
          { role: 'assistant', content: 'Here is the summary...' },
        ],
        tokenUsage: { prompt: 120, completion: 85 },
      },
    ],
  });
  ```

  ```python Python theme={null}
  client.ingest.create(
      events=[
          {
              "type": "trace",
              "timestamp": datetime.now(timezone.utc).isoformat(),
          },
          {
              "type": "llm",
              "event": "start",
              "timestamp": start_time.isoformat(),
              "modelId": "gpt-4o-2024-08-06",
              "params": {"temperature": 0.7, "maxTokens": 1024},
              "input": [
                  {"role": "system", "content": "You are a helpful assistant."},
                  {"role": "user", "content": "Summarize this document."},
              ],
          },
          {
              "type": "llm",
              "event": "end",
              "timestamp": end_time.isoformat(),
              "output": [
                  {"role": "assistant", "content": "Here is the summary..."},
              ],
              "tokenUsage": {"prompt": 120, "completion": 85},
          },
      ],
  )
  ```
</CodeGroup>

<Callout type="tip">
  Send the **raw** prompt and completion values. Strip any provider-specific JSON wrappers so Avido can display and evaluate them consistently.
</Callout>

### `tool` — function / tool calls

Log every tool or function call your model makes.

<CodeGroup>
  ```ts Node theme={null}
  {
    type: 'tool',
    timestamp: new Date().toISOString(),
    toolInput: JSON.stringify({ query: 'AAPL stock price' }),
    toolOutput: JSON.stringify({ price: 187.42, currency: 'USD' }),
  }
  ```

  ```python Python theme={null}
  {
      "type": "tool",
      "timestamp": datetime.now(timezone.utc).isoformat(),
      "toolInput": json.dumps({"query": "AAPL stock price"}),
      "toolOutput": json.dumps({"price": 187.42, "currency": "USD"}),
  }
  ```
</CodeGroup>

### `retriever` — RAG queries

Track retrieval-augmented generation lookups and the chunks they return.

<CodeGroup>
  ```ts Node theme={null}
  {
    type: 'retriever',
    timestamp: new Date().toISOString(),
    query: 'What is the refund policy?',
    result: JSON.stringify([
      { chunk: 'Refunds are available within 30 days...', score: 0.92 },
      { chunk: 'For premium plans, contact support...', score: 0.87 },
    ]),
  }
  ```

  ```python Python theme={null}
  {
      "type": "retriever",
      "timestamp": datetime.now(timezone.utc).isoformat(),
      "query": "What is the refund policy?",
      "result": json.dumps([
          {"chunk": "Refunds are available within 30 days...", "score": 0.92},
          {"chunk": "For premium plans, contact support...", "score": 0.87},
      ]),
  }
  ```
</CodeGroup>

### `log` — everything else

Use `log` events for system prompts, branching decisions, error details, or anything else worth seeing during debugging.

<CodeGroup>
  ```ts Node theme={null}
  {
    type: 'log',
    timestamp: new Date().toISOString(),
    body: 'User triggered fallback branch — no documents matched above threshold',
  }
  ```

  ```python Python theme={null}
  {
      "type": "log",
      "timestamp": datetime.now(timezone.utc).isoformat(),
      "body": "User triggered fallback branch — no documents matched above threshold",
  }
  ```
</CodeGroup>

***

## Full end-to-end example

A complete example capturing a realistic agent run with multiple event types:

<CodeGroup>
  ```ts Node theme={null}
  import Avido from '@avidoai/sdk-node';

  const client = new Avido({
    applicationId: process.env['AVIDO_APPLICATION_ID'],
    apiKey: process.env['AVIDO_API_KEY'],
  });

  async function handleRequest(prompt: string, testId?: string) {
    const events: Record<string, unknown>[] = [];
    const traceStart = new Date();

    // 1. Root trace
    events.push({
      type: 'trace',
      timestamp: traceStart.toISOString(),
      testId,
      metadata: { source: 'api' },
    });

    // 2. Retriever step
    const docs = await vectorSearch(prompt);
    events.push({
      type: 'retriever',
      timestamp: new Date().toISOString(),
      query: prompt,
      result: JSON.stringify(docs),
    });

    // 3. LLM call
    const llmStart = new Date();
    const completion = await callLLM(prompt, docs);
    events.push(
      {
        type: 'llm',
        event: 'start',
        timestamp: llmStart.toISOString(),
        modelId: 'gpt-4o-2024-08-06',
        input: [
          { role: 'system', content: 'Answer using the provided context.' },
          { role: 'user', content: prompt },
        ],
      },
      {
        type: 'llm',
        event: 'end',
        timestamp: new Date().toISOString(),
        output: [{ role: 'assistant', content: completion.text }],
        tokenUsage: {
          prompt: completion.usage.promptTokens,
          completion: completion.usage.completionTokens,
        },
      },
    );

    // 4. Flush everything in one call
    await client.ingest.create({ events });

    return completion.text;
  }
  ```

  ```python Python theme={null}
  import json
  import os
  from datetime import datetime, timezone
  from avido import Avido

  client = Avido(
      application_id=os.environ.get("AVIDO_APPLICATION_ID"),
      api_key=os.environ.get("AVIDO_API_KEY"),
  )

  def handle_request(prompt: str, test_id: str | None = None) -> str:
      events: list[dict] = []
      trace_start = datetime.now(timezone.utc)

      # 1. Root trace
      events.append({
          "type": "trace",
          "timestamp": trace_start.isoformat(),
          "testId": test_id,
          "metadata": {"source": "api"},
      })

      # 2. Retriever step
      docs = vector_search(prompt)
      events.append({
          "type": "retriever",
          "timestamp": datetime.now(timezone.utc).isoformat(),
          "query": prompt,
          "result": json.dumps(docs),
      })

      # 3. LLM call
      llm_start = datetime.now(timezone.utc)
      completion = call_llm(prompt, docs)
      events.extend([
          {
              "type": "llm",
              "event": "start",
              "timestamp": llm_start.isoformat(),
              "modelId": "gpt-4o-2024-08-06",
              "input": [
                  {"role": "system", "content": "Answer using the provided context."},
                  {"role": "user", "content": prompt},
              ],
          },
          {
              "type": "llm",
              "event": "end",
              "timestamp": datetime.now(timezone.utc).isoformat(),
              "output": [
                  {"role": "assistant", "content": completion.text},
              ],
              "tokenUsage": {
                  "prompt": completion.usage.prompt_tokens,
                  "completion": completion.usage.completion_tokens,
              },
          },
      ])

      # 4. Flush everything in one call
      client.ingest.create(events=events)

      return completion.text
  ```
</CodeGroup>

***

## Fetching traces

Use the SDK to list or retrieve traces for debugging and analysis.

<CodeGroup>
  ```ts Node theme={null}
  // List recent traces
  const traces = await client.traces.list({ limit: 10 });

  for (const trace of traces.data) {
    console.log(trace.id, trace.referenceId);
  }

  // Retrieve a specific trace by ID
  const trace = await client.traces.retrieve('trace-uuid-here');
  console.log(trace.data);
  ```

  ```python Python theme={null}
  # List recent traces
  traces = client.traces.list(limit=10)

  for trace in traces.data:
      print(trace.id, trace.reference_id)

  # Retrieve a specific trace by ID
  trace = client.traces.retrieve("trace-uuid-here")
  print(trace.data)
  ```
</CodeGroup>

***

## Error handling

Wrap your ingestion calls so a tracing failure never crashes your application.

<CodeGroup>
  ```ts Node theme={null}
  try {
    await client.ingest.create({ events });
  } catch (err) {
    // Log but don't crash — tracing is secondary to your user's request
    console.error('Avido ingestion failed:', err);
  }
  ```

  ```python Python theme={null}
  try:
      client.ingest.create(events=events)
  except Exception as e:
      # Log but don't crash — tracing is secondary to your user's request
      logger.error("Avido ingestion failed: %s", e)
  ```
</CodeGroup>

***

## Next steps

* Inspect traces in **Traces** inside the dashboard.
* Already using OpenTelemetry? See the [OpenTelemetry Integration](/opentelemetry) guide.
* Set up [Webhooks](/webhooks) to trigger tests automatically.
* Explore the full schema in the <a href="/api-reference/ingestion" target="_blank">API Reference</a>.