Skip to content
AI Automation & Enterprise Systems

AI Workflow Tracing: How to Investigate a Failed or Incorrect Run

Swadeep Tripathi
• 8 min read
AI Workflow Tracing: How to Investigate a Failed or Incorrect Run, editorial image

An AI workflow can finish without a technical error and still produce the wrong result. AI workflow tracing helps a team investigate by connecting the steps of one run: the request, retrieval, model calls, tool calls and final outcome. Start with a shared run identifier and a record of each step's status and timing. Add content only when it is needed for a specific diagnostic purpose and your organization permits it.

A trace is useful for a support assistant that consults documents or an automation that updates a business system. It is less useful when it records only the final response, or when the team cannot connect a tool call to the request that caused it. The aim is to reconstruct what happened, then verify the outcome in the system that received the action.

Map one run before choosing what to record

Choose a workflow and draw its actual path. For example, a support assistant might receive a question, retrieve an internal article, ask a model to draft an answer and return that answer for review. Another workflow might look up a record and propose an update. Mark where the application calls another service, waits for a person or changes a record.

Give the overall request a trace or run ID, and associate each step with it. A trace describes the larger journey; a span describes an individual operation within it. Spans can be nested, which helps show that a tool call was part of a particular agent run. Microsoft's agent tracing overview explains these relationships and notes that steps can vary and nest within one another.

Record the workflow version and relevant configuration identifiers alongside the run. If a prompt or model changes, an investigator can then distinguish runs using different configurations. Store a reference to the support case or internal job when appropriate, rather than putting a person's details into the identifier. Use an opaque ID that support staff can search for and pass to an authorized investigator.

Choose evidence for each investigation

The table proposes a starting point for an internal trace design. The right fields depend on the workflow and on what the team is allowed to retain.

Investigation question Evidence to consider recording What still needs checking
Where did a request stop? Run ID, step name, start time, duration and status Whether a missing step reflects a real stop or missing instrumentation
Did retrieval contribute to a bad answer? Retrieval status and permitted document identifiers or versions Whether the retrieved material was relevant and supported the answer
Did a tool change a record? Tool name, request correlation ID, response status and target reference The authoritative record's actual state after the call
Did an update change behavior? Workflow, prompt and model version identifiers Comparable examples and an evaluation of output quality

Keep status meanings precise. A model asking to call a tool is a proposed action; it does not establish that the tool ran. A tool returning an acceptance response may mean that processing has started, rather than that a business record has changed. Capture separate events for proposal, execution and confirmed outcome where the integration supports them. If it does not, record the uncertainty and check the destination system.

Trace data can also be incomplete. A timeout, an uninstrumented service or an interrupted export can leave a gap. Treat a missing span as a question to investigate, not proof that an action did not occur. The related guide to AI workflow retries and duplicate actions explains why checking the destination matters before repeating a write.

Control sensitive content in traces

A diagnostic record can contain customer messages, retrieved documents or tool arguments. Before enabling detailed capture, decide who needs access, how long records should remain available and which fields can be omitted or masked. Test those choices with representative sample data and inspect what the tracing destination actually receives. Restricting access to the application alone does not necessarily restrict access to a separate observability system.

Microsoft's guidance on observability for AI systems recommends linking request and execution details while balancing investigation needs against data minimization, retention and access controls. Apply that principle to each field: if a document ID and version will answer the investigation question, retaining the full document may be unnecessary. Conversely, a masked trace may not contain enough detail to explain an incorrect answer, so define an authorized route for deeper review when needed.

Check the defaults of the specific SDK and exporter in use. OpenAI's integrations and observability guide describes tracing built into its Agents SDK and enabled by default in its normal server-side path. That is a product-specific behavior, not a rule for every framework. Review configuration and destination settings for your deployment, including other application logs, because a trace setting alone may not govern every copy of the data.

Investigate an incorrect run step by step

Begin with the reported outcome and its run ID. Open the trace and establish whether it covers the entire request. Compare timestamps and linked steps with the application and destination-system records. If the trace is partial, document the gap before drawing a conclusion.

For an unsupported answer, inspect the retrieval step first: did it run, which document versions did it return, and did those passages contain the information needed? Then examine the model step and the final response, subject to the team's access rules. A correct retrieval followed by a misleading answer suggests a different investigation from a failed retrieval. Neither observation alone proves the underlying cause.

For an unexpected update, follow the proposed tool call to its execution status and the destination record. Check whether a person approved the action and whether the approved details match the executed request. The separate article on human approval in AI workflows covers where a workflow can pause before an action. A trace helps review that pause, but it does not replace an approval control.

Finish the investigation with a reproducible test case where possible. Remove or replace sensitive content before adding it to a shared test set. Record the expected outcome and the observed failure, then use it to check the proposed fix. A trace can show the path a run took; judging whether an answer was correct still requires a reference, a reviewer or another suitable evaluation method.

Monitor patterns without mistaking activity for quality

Individual traces help explain a particular run. Aggregate monitoring helps a team decide which runs to inspect. Start with counts of completed and failed runs, tool failures, latency and missing expected steps. Segment those measures by workflow version and tool where the volume makes the comparison useful. Agree on an owner and a response for each alert before turning it on.

A low error rate does not prove that answers are useful or that actions were appropriate. Add a quality review suited to the task, such as sampling answers against source material or checking whether an approved update reached the intended record. Compare like with like: a change in the mix of requests can affect results independently of a prompt change. The existing guide to testing prompt and model changes before release describes how to build a release evaluation around representative cases.

Vendor capabilities and their availability differ. For example, Microsoft's tracing overview distinguishes generally available agent types from preview capabilities and notes that telemetry storage can incur charges. Confirm the status, retention behavior and access requirements for your own setup before relying on any specific feature.

Use this checklist before rollout

  • Map one workflow, including its model, retrieval, approval and tool boundaries.
  • Carry an opaque run ID through the application and connected services where supported.
  • Define step statuses that distinguish a proposed action from an executed and confirmed one.
  • Record version identifiers, timing and errors, and decide which content fields are necessary.
  • Inspect exported records using sample data; verify masking, access and retention settings.
  • Test a failure, a timeout and a partial trace, then confirm the destination state independently.
  • Assign an owner for alerts and a process for reviewing a sample of outcomes.

Common mistakes include collecting full prompts by default, treating a successful tool response as proof of a completed business outcome, and assuming that a missing span means no action occurred. Another is installing a dashboard before deciding who will investigate its alerts. Each leaves a different blind spot, so test the actual investigation path with the people who will use it.

Choose one recent test run as the next step. Ask an operator to find it using only its run ID, identify each major step and verify its final outcome. Note where the evidence stops, then add the smallest useful piece of instrumentation and repeat the exercise.

Frequently Asked Questions

7 Key Questions Answered

It is the practice of linking the steps in an AI workflow run so an authorized person can inspect the request, model and tool operations, timing, statuses and outcome.
A trace describes the journey of a request or workflow. A span represents one operation within that journey and can be nested under another span.
No. It can show the recorded path and evidence used in a run, but correctness still needs to be assessed against suitable source material, expected results or human review.
Not automatically. Decide what is needed for a defined investigation, review the sensitivity of each field, and verify access, masking and retention controls at the destination.
Check for instrumentation gaps, export failures and interruptions before concluding that the call never happened. For an action that could change a record, inspect the destination system before retrying.
Use an opaque run ID that can be connected to the relevant support case or job through an authorized process. Carry that ID across the workflow and connected services where supported.
Start with run completion and failure counts, tool failures, latency and missing expected steps. Add a task-specific quality review because operational metrics alone do not establish answer quality.
Tags: AI workflow tracing AI agent observability AI workflow trace AI agent debugging AI tool call monitoring AI trace data
Share this article
Related Playbooks

Explore More Insights in AI Automation & Enterprise Systems

View All Insights
Home Solutions Work Contact