> ## Documentation Index > Fetch the complete documentation index at: https://docs.avidoai.com/llms.txt > Use this file to discover all available pages before exploring further. # List Traces > Retrieve traces with associated steps, filtered by application ID and optional date parameters. ## OpenAPI ````yaml https://app.stainless.com/api/spec/documented/avido/openapi.documented.yml get /v0/traces openapi: 3.1.1 info: title: Avido API description: >- Avido's API for LLM usage events, tool calls, trace management, webhook validation, and more. See each endpoint's request/response schema for details. version: 0.1.0 contact: name: Avido Support email: support@avidoai.com url: https://avidoai.com/support servers: - url: https://api.avidoai.com description: Production API security: - ApiKey: [] ApplicationId: [] paths: /v0/traces: get: tags: - Traces summary: List Traces description: >- Retrieve traces with associated steps, filtered by application ID and optional date parameters. operationId: listTraces parameters: - in: query name: skip schema: description: Number of items to skip before starting to collect the result set. example: 0 default: 0 type: integer minimum: 0 maximum: 9007199254740991 description: Number of items to skip before starting to collect the result set. - in: query name: limit schema: description: Number of items to include in the result set. example: 25 type: integer minimum: 1 maximum: 100 description: Number of items to include in the result set. - in: query name: orderBy schema: description: Field to order the result set by. example: timestamp default: timestamp type: string enum: - createdAt - modifiedAt - timestamp - totalCost - totalDurationMs - stepCount - id description: Field to order the result set by. - in: query name: orderDir schema: description: Order direction. example: desc default: desc type: string enum: - asc - desc description: Order direction. - in: query name: testId schema: description: Filter traces by test ID example: 123e4567-e89b-12d3-a456-426614174000 type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: Filter traces by test ID - in: query name: referenceId schema: description: Filter traces by reference ID example: ref-12345 type: string description: Filter traces by reference ID - in: query name: startDate schema: description: Filter traces created after this date (inclusive). example: '2024-01-01T00:00:00.000Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ description: Filter traces created after this date (inclusive). - in: query name: endDate schema: description: Filter traces created before this date (inclusive). example: '2024-12-31T23:59:59.999Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ description: Filter traces created before this date (inclusive). - in: query name: minCost schema: description: Filter traces with total cost greater than or equal to this value. example: 0.01 type: number minimum: 0 description: Filter traces with total cost greater than or equal to this value. - in: query name: maxCost schema: description: Filter traces with total cost less than or equal to this value. example: 1 type: number minimum: 0 description: Filter traces with total cost less than or equal to this value. - in: query name: minDuration schema: description: >- Filter traces with total duration (ms) greater than or equal to this value. example: 100 type: integer minimum: 0 maximum: 2147483647 description: >- Filter traces with total duration (ms) greater than or equal to this value. - in: query name: maxDuration schema: description: >- Filter traces with total duration (ms) less than or equal to this value. example: 5000 type: integer minimum: 0 maximum: 2147483647 description: >- Filter traces with total duration (ms) less than or equal to this value. - in: query name: hasError schema: description: Filter traces with or without errors. example: true type: boolean description: Filter traces with or without errors. - in: query name: metadataKey schema: description: >- Metadata key to filter on. Must be used together with metadataValue. example: userId type: string description: Metadata key to filter on. Must be used together with metadataValue. - in: query name: metadataValue schema: description: >- Metadata value to filter on. Must be used together with metadataKey. example: user-123 type: string description: Metadata value to filter on. Must be used together with metadataKey. - in: query name: search schema: description: >- Full-text search across trace step content fields (input, output, toolInput, toolOutput, content, query, result). example: error in response type: string description: >- Full-text search across trace step content fields (input, output, toolInput, toolOutput, content, query, result). - in: query name: minScore schema: description: Filter traces with eval score greater than or equal to this value. example: 0.5 type: number minimum: 0 maximum: 1 description: Filter traces with eval score greater than or equal to this value. - in: query name: maxScore schema: description: Filter traces with eval score less than or equal to this value. example: 0.9 type: number minimum: 0 maximum: 1 description: Filter traces with eval score less than or equal to this value. responses: '200': description: Successfully retrieved traces with steps content: application/json: schema: $ref: '#/components/schemas/PaginatedTraceResponse' '400': description: bad request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '401': description: unauthorized content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '403': description: forbidden content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '422': description: invalid input content: application/json: schema: $ref: '#/components/schemas/ValidationErrorResponse' '500': description: internal server error content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' x-codeSamples: - lang: JavaScript source: |- import Avido from 'avido'; const client = new Avido({ apiKey: process.env['AVIDO_API_KEY'], // This is the default and can be omitted applicationID: process.env['AVIDO_APPLICATION_ID'], // This is the default and can be omitted }); // Automatically fetches more pages as needed. for await (const trace of client.traces.list()) { console.log(trace.id); } - lang: Python source: |- import os from avido import Avido client = Avido( api_key=os.environ.get("AVIDO_API_KEY"), # This is the default and can be omitted application_id=os.environ.get("AVIDO_APPLICATION_ID"), # This is the default and can be omitted ) page = client.traces.list() page = page.data[0] print(page.id) components: schemas: PaginatedTraceResponse: type: object properties: data: type: array items: $ref: '#/components/schemas/TraceOutput' pagination: type: object properties: skip: type: integer minimum: 0 maximum: 9007199254740991 description: Number of items skipped. example: 0 limit: type: integer minimum: 1 maximum: 9007199254740991 description: Number of items per page. example: 25 total: type: integer minimum: 0 maximum: 9007199254740991 description: Total number of items available. example: 100 totalPages: type: integer minimum: 0 maximum: 9007199254740991 description: Total number of pages available. example: 4 totalCount: description: >- Total number of items without filters applied. Used to distinguish between truly empty results and empty filtered results. example: 100 type: integer minimum: 0 maximum: 9007199254740991 required: - skip - limit - total - totalPages additionalProperties: false title: PaginationResponse description: Pagination metadata returned in a paginated response. required: - data - pagination additionalProperties: false title: Paginated Traces description: Paginated response containing traces ErrorResponse: type: object properties: message: type: string description: A human-readable error message explaining the error. example: Resource not found required: - message additionalProperties: false title: ErrorResponse description: Standard error format for failed API operations. ValidationErrorResponse: type: object properties: message: type: string description: A human-readable error message indicating what went wrong. example: Invalid request data issues: type: array items: type: object properties: code: type: string message: type: string path: type: array items: type: string required: - code - message - path additionalProperties: false description: Array of detailed validation error objects. example: - code: invalid_string message: Invalid UUID path: - id required: - message - issues additionalProperties: false title: ValidationErrorResponse description: Details about validation errors in incoming requests. TraceOutput: type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: Unique Trace ID (UUID). example: 123e4567-e89b-12d3-a456-426614174000 referenceId: description: >- An optional reference ID to link the trace to an existing conversation or interaction in your own database. example: 123e4567-e89b-12d3-a456-426614174000 type: string timestamp: description: When the trace was created example: '2025-01-05T12:34:56.789123Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ metadata: description: >- Arbitrary metadata (e.g., userId, source). String inputs are parsed as JSON or wrapped in { raw: val }. example: userId: '123' source: chatbot anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} testId: description: The associated Test if this was triggered by an Avido eval type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ totalCost: description: >- Total cost of all steps in this trace (sum of step costAmount values). example: 0.0023 type: number minimum: 0 totalPromptTokens: description: >- Total number of prompt tokens used across all LLM steps in this trace. example: 150 type: integer minimum: 0 maximum: 9007199254740991 totalCompletionTokens: description: >- Total number of completion tokens used across all LLM steps in this trace. example: 200 type: integer minimum: 0 maximum: 9007199254740991 totalDurationMs: description: >- Total duration of the trace in milliseconds (sum of all step durations). example: 1500 type: integer minimum: 0 maximum: 9007199254740991 hasError: description: Whether any step in this trace has an error status. example: false type: boolean stepCount: description: Total number of steps in this trace. example: 5 type: integer minimum: 0 maximum: 9007199254740991 steps: description: The steps associated with the trace. type: array items: $ref: '#/components/schemas/StepOutput' evals: description: >- Evaluation results for this trace, joined through the associated test. type: array items: $ref: '#/components/schemas/EvalOutput' required: - id - timestamp additionalProperties: false title: Trace description: >- A trace grouping related steps (e.g. a user-agent interaction or conversation). StepOutput: oneOf: - oneOf: - $ref: '#/components/schemas/LlmStartStepOutput' - $ref: '#/components/schemas/LlmEndStepOutput' type: object discriminator: propertyName: event mapping: start: $ref: '#/components/schemas/LlmStartStepOutput' end: $ref: '#/components/schemas/LlmEndStepOutput' - $ref: '#/components/schemas/ToolStepOutput' - $ref: '#/components/schemas/RetrieverStepOutput' - type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID for the step. example: 123e4567-e89b-12d3-a456-426614174000 traceId: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID referencing the parent trace's ID. example: 610adba1-7cc0-4fa7-9e2b-8bd2fdf281b2 timestamp: description: When the step was created example: '2025-01-05T12:34:56.789123Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ type: type: string const: reranker name: description: The name of the step. example: Moderation type: string group: description: Optionally, the key for a group step to group the step with. example: inputModeration type: string params: description: Arbitrary params for the step. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} metadata: description: >- Extra metadata about this trace event. String values are parsed as JSON if possible, otherwise wrapped in { raw: val }. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} parentId: description: >- Parent step ID for hierarchical span relationships. Links this step to its parent in the trace tree. example: 456e7890-e89b-12d3-a456-426614174001 type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ endTime: description: When the step ended (for duration tracking) example: '2025-01-05T12:34:57.123456Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ durationMs: description: Duration of the step in milliseconds example: 334 type: integer minimum: 0 maximum: 9007199254740991 costAmount: description: >- Cost of the step in the configured currency (primarily for LLM steps) example: 0.0035 type: number minimum: 0 status: default: success description: >- Status of the step indicating success or failure. Defaults to 'success'. example: success type: string enum: - success - error - timeout - pending statusCode: description: >- Status code associated with the step (e.g., HTTP status code for API calls, OTEL status code for traced spans). example: '200' type: string error: description: Error message or stack trace if the step failed. example: 'TimeoutError: Request timed out after 30s' type: string required: - id - traceId - timestamp - type - status additionalProperties: false title: Reranker Step description: Track document reranking operations using the Reranker Step event. - type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID for the step. example: 123e4567-e89b-12d3-a456-426614174000 traceId: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID referencing the parent trace's ID. example: 610adba1-7cc0-4fa7-9e2b-8bd2fdf281b2 timestamp: description: When the step was created example: '2025-01-05T12:34:56.789123Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ type: type: string const: agent name: description: The name of the step. example: Moderation type: string group: description: Optionally, the key for a group step to group the step with. example: inputModeration type: string params: description: Arbitrary params for the step. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} metadata: description: >- Extra metadata about this trace event. String values are parsed as JSON if possible, otherwise wrapped in { raw: val }. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} parentId: description: >- Parent step ID for hierarchical span relationships. Links this step to its parent in the trace tree. example: 456e7890-e89b-12d3-a456-426614174001 type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ endTime: description: When the step ended (for duration tracking) example: '2025-01-05T12:34:57.123456Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ durationMs: description: Duration of the step in milliseconds example: 334 type: integer minimum: 0 maximum: 9007199254740991 costAmount: description: >- Cost of the step in the configured currency (primarily for LLM steps) example: 0.0035 type: number minimum: 0 status: default: success description: >- Status of the step indicating success or failure. Defaults to 'success'. example: success type: string enum: - success - error - timeout - pending statusCode: description: >- Status code associated with the step (e.g., HTTP status code for API calls, OTEL status code for traced spans). example: '200' type: string error: description: Error message or stack trace if the step failed. example: 'TimeoutError: Request timed out after 30s' type: string input: description: Agent input (JSON). example: '{"messages":[{"content":"What is AI?","type":"human"}]}' type: string output: description: Agent output (JSON). example: >- {"messages":[{"content":"What is AI?","type":"human"},{"content":"AI is artificial intelligence...","type":"ai"}]} type: string required: - id - traceId - timestamp - type - status additionalProperties: false title: Agent Step description: Track agent orchestration and planning using the Agent Step event. - type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID for the step. example: 123e4567-e89b-12d3-a456-426614174000 traceId: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID referencing the parent trace's ID. example: 610adba1-7cc0-4fa7-9e2b-8bd2fdf281b2 timestamp: description: When the step was created example: '2025-01-05T12:34:56.789123Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ type: type: string const: embedding name: description: The name of the step. example: Moderation type: string group: description: Optionally, the key for a group step to group the step with. example: inputModeration type: string params: description: Arbitrary params for the step. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} metadata: description: >- Extra metadata about this trace event. String values are parsed as JSON if possible, otherwise wrapped in { raw: val }. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} parentId: description: >- Parent step ID for hierarchical span relationships. Links this step to its parent in the trace tree. example: 456e7890-e89b-12d3-a456-426614174001 type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ endTime: description: When the step ended (for duration tracking) example: '2025-01-05T12:34:57.123456Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ durationMs: description: Duration of the step in milliseconds example: 334 type: integer minimum: 0 maximum: 9007199254740991 costAmount: description: >- Cost of the step in the configured currency (primarily for LLM steps) example: 0.0035 type: number minimum: 0 status: default: success description: >- Status of the step indicating success or failure. Defaults to 'success'. example: success type: string enum: - success - error - timeout - pending statusCode: description: >- Status code associated with the step (e.g., HTTP status code for API calls, OTEL status code for traced spans). example: '200' type: string error: description: Error message or stack trace if the step failed. example: 'TimeoutError: Request timed out after 30s' type: string required: - id - traceId - timestamp - type - status additionalProperties: false title: Embedding Step description: Track text embedding operations using the Embedding Step event. - type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID for the step. example: 123e4567-e89b-12d3-a456-426614174000 traceId: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID referencing the parent trace's ID. example: 610adba1-7cc0-4fa7-9e2b-8bd2fdf281b2 timestamp: description: When the step was created example: '2025-01-05T12:34:56.789123Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ type: type: string const: guardrail name: description: The name of the step. example: Moderation type: string group: description: Optionally, the key for a group step to group the step with. example: inputModeration type: string params: description: Arbitrary params for the step. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} metadata: description: >- Extra metadata about this trace event. String values are parsed as JSON if possible, otherwise wrapped in { raw: val }. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} parentId: description: >- Parent step ID for hierarchical span relationships. Links this step to its parent in the trace tree. example: 456e7890-e89b-12d3-a456-426614174001 type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ endTime: description: When the step ended (for duration tracking) example: '2025-01-05T12:34:57.123456Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ durationMs: description: Duration of the step in milliseconds example: 334 type: integer minimum: 0 maximum: 9007199254740991 costAmount: description: >- Cost of the step in the configured currency (primarily for LLM steps) example: 0.0035 type: number minimum: 0 status: default: success description: >- Status of the step indicating success or failure. Defaults to 'success'. example: success type: string enum: - success - error - timeout - pending statusCode: description: >- Status code associated with the step (e.g., HTTP status code for API calls, OTEL status code for traced spans). example: '200' type: string error: description: Error message or stack trace if the step failed. example: 'TimeoutError: Request timed out after 30s' type: string guardrailTriggered: description: >- True when the guardrail blocked or modified the operation. Sourced from the guardrail.triggered span attribute. example: true type: boolean guardrailType: description: >- Guardrail category (e.g. 'input', 'output', 'input_scope'). Sourced from guardrail.type. example: input_scope type: string guardrailAction: description: >- Action the guardrail took (e.g. 'block', 'allow', 'rewrite'). Sourced from guardrail.action. example: block type: string required: - id - traceId - timestamp - type - status additionalProperties: false title: Guardrail Step description: Track safety and policy checks using the Guardrail Step event. - type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID for the step. example: 123e4567-e89b-12d3-a456-426614174000 traceId: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID referencing the parent trace's ID. example: 610adba1-7cc0-4fa7-9e2b-8bd2fdf281b2 timestamp: description: When the step was created example: '2025-01-05T12:34:56.789123Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ type: type: string const: evaluator name: description: The name of the step. example: Moderation type: string group: description: Optionally, the key for a group step to group the step with. example: inputModeration type: string params: description: Arbitrary params for the step. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} metadata: description: >- Extra metadata about this trace event. String values are parsed as JSON if possible, otherwise wrapped in { raw: val }. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} parentId: description: >- Parent step ID for hierarchical span relationships. Links this step to its parent in the trace tree. example: 456e7890-e89b-12d3-a456-426614174001 type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ endTime: description: When the step ended (for duration tracking) example: '2025-01-05T12:34:57.123456Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ durationMs: description: Duration of the step in milliseconds example: 334 type: integer minimum: 0 maximum: 9007199254740991 costAmount: description: >- Cost of the step in the configured currency (primarily for LLM steps) example: 0.0035 type: number minimum: 0 status: default: success description: >- Status of the step indicating success or failure. Defaults to 'success'. example: success type: string enum: - success - error - timeout - pending statusCode: description: >- Status code associated with the step (e.g., HTTP status code for API calls, OTEL status code for traced spans). example: '200' type: string error: description: Error message or stack trace if the step failed. example: 'TimeoutError: Request timed out after 30s' type: string evaluatorName: description: Name of the evaluator. example: helpfulness type: string evaluationLabel: description: Classification label from the evaluation. example: helpful type: string evaluationScore: description: Numerical score from the evaluation (0-1 range). example: 0.75 type: number minimum: 0 maximum: 1 evaluationExplanation: description: Detailed explanation of the evaluation result. example: >- The response directly answers the question with clear, organized information. type: string required: - id - traceId - timestamp - type - status additionalProperties: false title: Evaluator Step description: Track evaluation operations using the Evaluator Step event. - $ref: '#/components/schemas/LogStepOutput' - $ref: '#/components/schemas/GroupStepOutput' - $ref: '#/components/schemas/ResponseStepOutput' - $ref: '#/components/schemas/RequestStepOutput' type: object EvalOutput: type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: Unique identifier of the evaluation example: 123e4567-e89b-12d3-a456-426614174000 orgId: type: string description: Organization ID that owns this evaluation example: org_123456 createdAt: description: When the evaluation was created example: '2024-01-05T12:34:56.789Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ modifiedAt: description: When the evaluation was last modified example: '2024-01-05T12:34:56.789Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ status: type: string enum: - PENDING - IN_PROGRESS - COMPLETED - FAILED - SKIPPED description: Status of the evaluation example: COMPLETED results: description: Results of the evaluation (structure depends on eval type). anyOf: - type: object properties: coherence: type: number minimum: 1 maximum: 5 engagingness: type: number minimum: 1 maximum: 5 naturalness: type: number minimum: 1 maximum: 5 relevance: type: number minimum: 1 maximum: 5 clarity: type: number minimum: 1 maximum: 5 analysis: type: string required: - coherence - engagingness - naturalness - relevance - clarity - analysis additionalProperties: false - type: object properties: score: type: number minimum: 1 maximum: 5 description: The score of the response based on the style guide from 1-5 analysis: type: string description: >- A brief explanation for your rating, referring to specific aspects of the response and the query. Make sure that the explanation is formatted as markdown, and that it is easy to read and understand. required: - score - analysis additionalProperties: false - type: object properties: ContextRelevancy: type: object properties: score: type: number error: type: string metadata: type: object properties: relevantSentences: type: array items: type: object properties: sentence: type: string reasons: type: array items: type: string required: - sentence - reasons additionalProperties: false required: - relevantSentences additionalProperties: false required: - score additionalProperties: false ContextPrecision: type: object properties: score: type: number error: type: string metadata: type: object properties: verdict: type: integer minimum: -9007199254740991 maximum: 9007199254740991 reason: type: string required: - verdict - reason additionalProperties: false required: - score additionalProperties: false Faithfulness: type: object properties: score: type: number error: type: string metadata: type: object properties: statements: type: array items: type: string faithfulness: type: array items: type: object properties: statement: type: string reason: type: string verdict: type: integer minimum: -9007199254740991 maximum: 9007199254740991 classification: description: >- Classification of the hallucination type (only present when verdict is 0) type: string enum: - UNSUPPORTED_CLAIM - CONTRADICTION - PARTIAL_HALLUCINATION - SCOPE_DRIFT required: - statement - reason - verdict additionalProperties: false required: - statements - faithfulness additionalProperties: false required: - score additionalProperties: false AnswerRelevancy: type: object properties: score: type: number error: type: string metadata: type: object properties: questions: type: array items: type: object properties: question: type: string required: - question additionalProperties: false similarity: type: array items: type: object properties: question: type: string score: type: number required: - question - score additionalProperties: false required: - questions - similarity additionalProperties: false required: - score additionalProperties: false required: - ContextRelevancy - ContextPrecision - Faithfulness - AnswerRelevancy additionalProperties: false - type: object properties: score: type: integer minimum: -9007199254740991 maximum: 9007199254740991 error: type: string metadata: type: object properties: rationale: type: string choice: type: string enum: - 'Y' - 'N' required: - rationale additionalProperties: false required: - score additionalProperties: false - type: object properties: score: type: number reason: type: string expected: type: string got: type: string expectedList: type: array items: type: string gotList: type: array items: type: string matchMode: type: string enum: - exact_unordered - contains scoreMetric: type: string enum: - f1 - precision - recall precision: type: number recall: type: number f1: type: number tp: type: number missing: type: array items: type: object properties: value: type: string need: type: number have: type: number required: - value - need - have additionalProperties: false required: - score - reason additionalProperties: false definition: $ref: '#/components/schemas/EvalDefinitionOutput' score: description: Overall score of the evaluation example: 0.9 type: number passed: description: Whether the evaluation passed example: true type: boolean message: description: Message explaining why the eval was skipped or failed type: string required: - id - orgId - createdAt - modifiedAt - status - definition additionalProperties: false title: Eval description: Complete evaluation information LlmStartStepOutput: type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID for the step. example: 123e4567-e89b-12d3-a456-426614174000 traceId: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID referencing the parent trace's ID. example: 610adba1-7cc0-4fa7-9e2b-8bd2fdf281b2 timestamp: description: When the step was created example: '2025-01-05T12:34:56.789123Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ type: type: string const: llm name: description: The name of the step. example: Moderation type: string group: description: Optionally, the key for a group step to group the step with. example: inputModeration type: string params: description: Arbitrary params for the step. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} metadata: description: >- Extra metadata about this trace event. String values are parsed as JSON if possible, otherwise wrapped in { raw: val }. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} parentId: description: >- Parent step ID for hierarchical span relationships. Links this step to its parent in the trace tree. example: 456e7890-e89b-12d3-a456-426614174001 type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ endTime: description: When the step ended (for duration tracking) example: '2025-01-05T12:34:57.123456Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ durationMs: description: Duration of the step in milliseconds example: 334 type: integer minimum: 0 maximum: 9007199254740991 costAmount: description: >- Cost of the step in the configured currency (primarily for LLM steps) example: 0.0035 type: number minimum: 0 status: default: success description: >- Status of the step indicating success or failure. Defaults to 'success'. example: success type: string enum: - success - error - timeout - pending statusCode: description: >- Status code associated with the step (e.g., HTTP status code for API calls, OTEL status code for traced spans). example: '200' type: string error: description: Error message or stack trace if the step failed. example: 'TimeoutError: Request timed out after 30s' type: string modelId: type: string description: Model ID or name used for the LLM call. example: gpt-4o-2024-08-06 event: type: string const: start input: type: string description: JSON input for this LLM trace event (e.g., the prompt). example: Tell me a joke. required: - id - traceId - timestamp - type - status - modelId - event - input additionalProperties: false title: LLM Start Trace description: Start of an LLM trace. LlmEndStepOutput: type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID for the step. example: 123e4567-e89b-12d3-a456-426614174000 traceId: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID referencing the parent trace's ID. example: 610adba1-7cc0-4fa7-9e2b-8bd2fdf281b2 timestamp: description: When the step was created example: '2025-01-05T12:34:56.789123Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ type: type: string const: llm name: description: The name of the step. example: Moderation type: string group: description: Optionally, the key for a group step to group the step with. example: inputModeration type: string params: description: Arbitrary params for the step. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} metadata: description: >- Extra metadata about this trace event. String values are parsed as JSON if possible, otherwise wrapped in { raw: val }. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} parentId: description: >- Parent step ID for hierarchical span relationships. Links this step to its parent in the trace tree. example: 456e7890-e89b-12d3-a456-426614174001 type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ endTime: description: When the step ended (for duration tracking) example: '2025-01-05T12:34:57.123456Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ durationMs: description: Duration of the step in milliseconds example: 334 type: integer minimum: 0 maximum: 9007199254740991 costAmount: description: >- Cost of the step in the configured currency (primarily for LLM steps) example: 0.0035 type: number minimum: 0 status: default: success description: >- Status of the step indicating success or failure. Defaults to 'success'. example: success type: string enum: - success - error - timeout - pending statusCode: description: >- Status code associated with the step (e.g., HTTP status code for API calls, OTEL status code for traced spans). example: '200' type: string error: description: Error message or stack trace if the step failed. example: 'TimeoutError: Request timed out after 30s' type: string modelId: type: string description: Model ID or name used for the LLM call. example: gpt-4o-2024-08-06 event: type: string const: end output: description: >- JSON describing the output. String inputs are parsed or wrapped in { message: val }. example: Why did the chicken cross the road? type: string usage: $ref: '#/components/schemas/UsageOutput' description: Number of input and output tokens used by the LLM. example: promptTokens: 100 completionTokens: 100 finishReason: description: >- The reason the LLM stopped generating, extracted from gen_ai.response.finish_reasons. example: stop type: string required: - id - traceId - timestamp - type - status - modelId - event additionalProperties: false title: LLM End Trace description: End of an LLM trace. ToolStepOutput: type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID for the step. example: 123e4567-e89b-12d3-a456-426614174000 traceId: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID referencing the parent trace's ID. example: 610adba1-7cc0-4fa7-9e2b-8bd2fdf281b2 timestamp: description: When the step was created example: '2025-01-05T12:34:56.789123Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ type: type: string const: tool name: description: The name of the step. example: Moderation type: string group: description: Optionally, the key for a group step to group the step with. example: inputModeration type: string params: description: Arbitrary params for the step. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} metadata: description: >- Extra metadata about this trace event. String values are parsed as JSON if possible, otherwise wrapped in { raw: val }. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} parentId: description: >- Parent step ID for hierarchical span relationships. Links this step to its parent in the trace tree. example: 456e7890-e89b-12d3-a456-426614174001 type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ endTime: description: When the step ended (for duration tracking) example: '2025-01-05T12:34:57.123456Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ durationMs: description: Duration of the step in milliseconds example: 334 type: integer minimum: 0 maximum: 9007199254740991 costAmount: description: >- Cost of the step in the configured currency (primarily for LLM steps) example: 0.0035 type: number minimum: 0 status: default: success description: >- Status of the step indicating success or failure. Defaults to 'success'. example: success type: string enum: - success - error - timeout - pending statusCode: description: >- Status code associated with the step (e.g., HTTP status code for API calls, OTEL status code for traced spans). example: '200' type: string error: description: Error message or stack trace if the step failed. example: 'TimeoutError: Request timed out after 30s' type: string toolCallId: description: >- Correlation ID from the LLM's tool_calls response (e.g., OpenAI's call_abc123 or Claude's toolu_01A09q). Links tool execution to the originating LLM request in parallel tool call scenarios. example: call_abc123 type: string toolInput: description: JSON input for the tool call. type: string toolOutput: description: JSON output from the tool call. type: string required: - id - traceId - timestamp - type - status additionalProperties: false title: Tool Step description: Track all tool calls using the Tool Step event RetrieverStepOutput: type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID for the step. example: 123e4567-e89b-12d3-a456-426614174000 traceId: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID referencing the parent trace's ID. example: 610adba1-7cc0-4fa7-9e2b-8bd2fdf281b2 timestamp: description: When the step was created example: '2025-01-05T12:34:56.789123Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ type: type: string const: retriever name: description: The name of the step. example: Moderation type: string group: description: Optionally, the key for a group step to group the step with. example: inputModeration type: string params: description: Arbitrary params for the step. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} metadata: description: >- Extra metadata about this trace event. String values are parsed as JSON if possible, otherwise wrapped in { raw: val }. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} parentId: description: >- Parent step ID for hierarchical span relationships. Links this step to its parent in the trace tree. example: 456e7890-e89b-12d3-a456-426614174001 type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ endTime: description: When the step ended (for duration tracking) example: '2025-01-05T12:34:57.123456Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ durationMs: description: Duration of the step in milliseconds example: 334 type: integer minimum: 0 maximum: 9007199254740991 costAmount: description: >- Cost of the step in the configured currency (primarily for LLM steps) example: 0.0035 type: number minimum: 0 status: default: success description: >- Status of the step indicating success or failure. Defaults to 'success'. example: success type: string enum: - success - error - timeout - pending statusCode: description: >- Status code associated with the step (e.g., HTTP status code for API calls, OTEL status code for traced spans). example: '200' type: string error: description: Error message or stack trace if the step failed. example: 'TimeoutError: Request timed out after 30s' type: string query: type: string description: Query used for RAG. example: What is the capital of France? result: type: string description: Retrieved text example: Paris is the capital of France. required: - id - traceId - timestamp - type - status - query - result additionalProperties: false title: Retriever Step description: Track all retriever (RAG) calls using the Retriever Step event. LogStepOutput: type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID for the step. example: 123e4567-e89b-12d3-a456-426614174000 traceId: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID referencing the parent trace's ID. example: 610adba1-7cc0-4fa7-9e2b-8bd2fdf281b2 timestamp: description: When the step was created example: '2025-01-05T12:34:56.789123Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ type: type: string const: log name: description: The name of the step. example: Moderation type: string group: description: Optionally, the key for a group step to group the step with. example: inputModeration type: string params: description: Arbitrary params for the step. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} metadata: description: >- Extra metadata about this trace event. String values are parsed as JSON if possible, otherwise wrapped in { raw: val }. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} parentId: description: >- Parent step ID for hierarchical span relationships. Links this step to its parent in the trace tree. example: 456e7890-e89b-12d3-a456-426614174001 type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ endTime: description: When the step ended (for duration tracking) example: '2025-01-05T12:34:57.123456Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ durationMs: description: Duration of the step in milliseconds example: 334 type: integer minimum: 0 maximum: 9007199254740991 costAmount: description: >- Cost of the step in the configured currency (primarily for LLM steps) example: 0.0035 type: number minimum: 0 status: default: success description: >- Status of the step indicating success or failure. Defaults to 'success'. example: success type: string enum: - success - error - timeout - pending statusCode: description: >- Status code associated with the step (e.g., HTTP status code for API calls, OTEL status code for traced spans). example: '200' type: string error: description: Error message or stack trace if the step failed. example: 'TimeoutError: Request timed out after 30s' type: string content: type: string description: The actual log message for this trace. example: Could not find the user with id 123 required: - id - traceId - timestamp - type - status - content additionalProperties: false title: Log Step description: Track all logs using the Log Step event. GroupStepOutput: type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID for the step. example: 123e4567-e89b-12d3-a456-426614174000 traceId: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID referencing the parent trace's ID. example: 610adba1-7cc0-4fa7-9e2b-8bd2fdf281b2 timestamp: description: When the step was created example: '2025-01-05T12:34:56.789123Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ type: type: string const: group name: description: The name of the step. example: Moderation type: string group: description: Optionally, the key for a group step to group the step with. example: inputModeration type: string params: description: Arbitrary params for the step. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} metadata: description: >- Extra metadata about this trace event. String values are parsed as JSON if possible, otherwise wrapped in { raw: val }. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} parentId: description: >- Parent step ID for hierarchical span relationships. Links this step to its parent in the trace tree. example: 456e7890-e89b-12d3-a456-426614174001 type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ endTime: description: When the step ended (for duration tracking) example: '2025-01-05T12:34:57.123456Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ durationMs: description: Duration of the step in milliseconds example: 334 type: integer minimum: 0 maximum: 9007199254740991 costAmount: description: >- Cost of the step in the configured currency (primarily for LLM steps) example: 0.0035 type: number minimum: 0 status: default: success description: >- Status of the step indicating success or failure. Defaults to 'success'. example: success type: string enum: - success - error - timeout - pending statusCode: description: >- Status code associated with the step (e.g., HTTP status code for API calls, OTEL status code for traced spans). example: '200' type: string error: description: Error message or stack trace if the step failed. example: 'TimeoutError: Request timed out after 30s' type: string key: type: string description: >- A unique identifier for the grouping, which must be appended to the corresponding steps example: inputModeration input: description: >- Input value attached to the grouping span (e.g. user message on an OpenInference CHAIN root). example: What is the capital of France? type: string output: description: >- Output value attached to the grouping span (e.g. final assistant message on an OpenInference CHAIN root). example: Paris is the capital of France. type: string guardrailTriggered: description: >- True when a guardrail blocked or modified the grouped operation. Sourced from the guardrail.triggered span attribute. example: true type: boolean guardrailType: description: >- Guardrail category that fired (e.g. 'input_scope', 'safety'). Sourced from guardrail.type. example: input_scope type: string guardrailAction: description: >- Action the guardrail took (e.g. 'block', 'allow', 'rewrite'). Sourced from guardrail.action. example: block type: string required: - id - traceId - timestamp - type - status - key additionalProperties: false title: Group Step description: >- Use this to group multiple steps together, for example a log, llm start, and llm end. ResponseStepOutput: type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID for the step. example: 123e4567-e89b-12d3-a456-426614174000 traceId: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID referencing the parent trace's ID. example: 610adba1-7cc0-4fa7-9e2b-8bd2fdf281b2 timestamp: description: When the step was created example: '2025-01-05T12:34:56.789123Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ type: type: string const: response name: description: The name of the step. example: Moderation type: string group: description: Optionally, the key for a group step to group the step with. example: inputModeration type: string params: description: Arbitrary params for the step. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} metadata: description: >- Extra metadata about this trace event. String values are parsed as JSON if possible, otherwise wrapped in { raw: val }. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} parentId: description: >- Parent step ID for hierarchical span relationships. Links this step to its parent in the trace tree. example: 456e7890-e89b-12d3-a456-426614174001 type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ endTime: description: When the step ended (for duration tracking) example: '2025-01-05T12:34:57.123456Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ durationMs: description: Duration of the step in milliseconds example: 334 type: integer minimum: 0 maximum: 9007199254740991 costAmount: description: >- Cost of the step in the configured currency (primarily for LLM steps) example: 0.0035 type: number minimum: 0 status: default: success description: >- Status of the step indicating success or failure. Defaults to 'success'. example: success type: string enum: - success - error - timeout - pending statusCode: description: >- Status code associated with the step (e.g., HTTP status code for API calls, OTEL status code for traced spans). example: '200' type: string error: description: Error message or stack trace if the step failed. example: 'TimeoutError: Request timed out after 30s' type: string content: type: string description: The response content from the AI to the user. example: >- Here is the answer to your question about the capital of France: Paris is the capital city of France. required: - id - traceId - timestamp - type - status - content additionalProperties: false title: Response Step description: Track AI responses to users using the Response Step event. RequestStepOutput: type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID for the step. example: 123e4567-e89b-12d3-a456-426614174000 traceId: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: UUID referencing the parent trace's ID. example: 610adba1-7cc0-4fa7-9e2b-8bd2fdf281b2 timestamp: description: When the step was created example: '2025-01-05T12:34:56.789123Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ type: type: string const: request name: description: The name of the step. example: Moderation type: string group: description: Optionally, the key for a group step to group the step with. example: inputModeration type: string params: description: Arbitrary params for the step. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} metadata: description: >- Extra metadata about this trace event. String values are parsed as JSON if possible, otherwise wrapped in { raw: val }. anyOf: - type: object propertyNames: type: string additionalProperties: {} - type: array items: {} parentId: description: >- Parent step ID for hierarchical span relationships. Links this step to its parent in the trace tree. example: 456e7890-e89b-12d3-a456-426614174001 type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ endTime: description: When the step ended (for duration tracking) example: '2025-01-05T12:34:57.123456Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ durationMs: description: Duration of the step in milliseconds example: 334 type: integer minimum: 0 maximum: 9007199254740991 costAmount: description: >- Cost of the step in the configured currency (primarily for LLM steps) example: 0.0035 type: number minimum: 0 status: default: success description: >- Status of the step indicating success or failure. Defaults to 'success'. example: success type: string enum: - success - error - timeout - pending statusCode: description: >- Status code associated with the step (e.g., HTTP status code for API calls, OTEL status code for traced spans). example: '200' type: string error: description: Error message or stack trace if the step failed. example: 'TimeoutError: Request timed out after 30s' type: string content: type: string description: The request content from the user to the AI. example: What is the capital of France? required: - id - traceId - timestamp - type - status - content additionalProperties: false title: Request Step description: Track user requests to the AI using the Request Step event. EvalDefinitionOutput: type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ createdAt: description: When the eval definition was created example: '2024-01-05T12:34:56.789Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ modifiedAt: description: When the eval definition was last modified example: '2024-01-05T12:34:56.789Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ type: $ref: '#/components/schemas/EvalType' name: type: string globalConfig: anyOf: - type: object properties: criterion: type: string minLength: 1 description: >- The criterion describes what our evaluation LLM must look for in the response. Remember that the answer to the criterion must be as a pass/fail. required: - criterion additionalProperties: false - oneOf: - $ref: '#/components/schemas/OutputMatchStringConfigOutput' - $ref: '#/components/schemas/OutputMatchListConfigOutput' type: object discriminator: propertyName: type mapping: string: $ref: '#/components/schemas/OutputMatchStringConfigOutput' list: $ref: '#/components/schemas/OutputMatchListConfigOutput' styleGuideId: anyOf: - type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ - type: 'null' application: anyOf: - $ref: '#/components/schemas/ApplicationOutput' - type: 'null' topics: description: Topic IDs this eval definition is linked to (for topic-scoped evals) example: - 456e4567-e89b-12d3-a456-426614174000 type: array items: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ required: - id - createdAt - modifiedAt - type - name additionalProperties: false UsageOutput: type: object properties: promptTokens: type: integer minimum: 0 maximum: 9007199254740991 description: Number of prompt tokens used by the LLM. example: 100 completionTokens: type: integer minimum: 0 maximum: 9007199254740991 description: Number of completion tokens used by the LLM. example: 100 required: - promptTokens - completionTokens additionalProperties: false EvalType: type: string enum: - NATURALNESS - STYLE - RECALL - CUSTOM - OUTPUT_MATCH OutputMatchStringConfigOutput: type: object properties: type: type: string const: string extract: $ref: '#/components/schemas/OutputMatchExtractConfigOutput' required: - type additionalProperties: false OutputMatchListConfigOutput: type: object properties: type: type: string const: list matchMode: type: string enum: - exact_unordered - contains scoreMetric: type: string enum: - f1 - precision - recall passThreshold: type: number minimum: 0 maximum: 1 extract: $ref: '#/components/schemas/OutputMatchExtractConfigOutput' required: - type - matchMode additionalProperties: false ApplicationOutput: type: object properties: id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ description: Unique identifier of the application example: 123e4567-e89b-12d3-a456-426614174000 orgId: type: string description: Organization ID that owns this application example: org_123456 title: type: string minLength: 1 description: Title of the application example: Customer Support Bot slug: type: string minLength: 1 description: URL-friendly slug for the application example: customer-support-bot description: type: string description: Description of the application example: AI assistant for customer support inquiries context: type: string description: Context/instructions for the application example: You are a helpful customer support assistant... language: default: en description: Language of the application. example: en type: string enum: - da - de - en - es - fr - it - nl - pt - sv type: type: string enum: - CHATBOT - AGENT description: Type of the application. Valid values are CHATBOT or AGENT. example: CHATBOT environment: default: DEV description: Environment of the application. Defaults to DEV. example: DEV type: string enum: - DEV - PROD humanAnnotationEnabled: type: boolean description: Whether human annotation is enabled for this application example: false createdAt: description: When the application was created example: '2024-01-05T12:34:56.789Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ modifiedAt: description: When the application was last modified example: '2024-01-05T12:34:56.789Z' type: string format: date-time pattern: >- ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$ required: - id - orgId - title - slug - description - context - language - type - environment - humanAnnotationEnabled - createdAt - modifiedAt additionalProperties: false title: Application description: Application configuration and metadata OutputMatchExtractConfigOutput: type: object properties: pattern: type: string minLength: 1 flags: type: string pattern: ^[gimsuy]*$ group: anyOf: - type: integer minimum: 0 maximum: 9007199254740991 - minItems: 1 type: array items: type: integer minimum: 0 maximum: 9007199254740991 required: - pattern - flags - group additionalProperties: false securitySchemes: ApiKey: type: apiKey in: header name: x-api-key description: Your unique Avido API key ApplicationId: type: apiKey in: header name: x-application-id description: Your unique Avido Application ID ````