How to watch an AI agent and keep an audit trail

OpenTelemetry's agent conventions are still marked Development and keep message content opt-in. Langfuse, Phoenix, Helicone and LangSmith keep traces from 14 days to 3 years. Read on each vendor's own pages, 2026-11-06.

By Mackenzie Bowes · mackenzie@mackenziebowes.com · @_mackenziebowes on X

Contents

Short answer, from the sources below: tools that record what an AI agent did (the model calls, the tool calls, the order, the cost) exist as open-source and hosted products, and they share one emerging format, OpenTelemetry's GenAI conventions, which is still marked "Development". They differ in licence, price and how long they keep a trace. None of the pages read describes tamper-evident or write-once storage, which is what the word audit trail usually implies. This page lists what each source says, who wrote it and what it leaves out.

What the standard records

Watching an agent means recording its steps. OpenTelemetry, the open standard for telemetry, publishes semantic conventions for generative AI. A Microsoft engineer's post on the OpenTelemetry blog (2026-05-14) says they "standardize how GenAI operations are recorded", including "the model being called, input and output token counts."[1]

The conventions for agents name an operation for each step: create_agent for making an agent and invoke_agent for running one, with spans for tool calls.[2] The page carries the status "Development", so the names can still change.[2]

The attributes that would hold the evidence are optional. gen_ai.input.messages (the chat history sent to the model) and gen_ai.output.messages (what the model returned) are both marked "Opt-In", and the page warns that the attribute "is likely to contain sensitive information including user/PII data."[2] A reference form, gen_ai.input.messages_ref, holds "an absolute URI identifying the input messages in external storage" so the content can live elsewhere, and the page adds that "the presence of this attribute does not indicate whether storing the content succeeded or whether the content remains available."[2]

That default is the reverse of at least one SDK. The OpenAI Agents SDK page quotes its tracing guide: "By default, trace_include_sensitive_data is True."

Four tools as each publishes itself

Prices are in U.S. dollars as listed on 2026-11-06. Langfuse's page counts "units" and LangSmith's counts traces, so the free tiers are not directly comparable.[4][7]

What a record has to hold

The European Union's AI Act is the one law found that names logging. Article 12 says "High-risk AI systems shall technically allow for the automatic recording of events (logs) over the lifetime of the system."[8] For one class of high-risk systems the minimum includes "the period of each use of the system (start date and time and end date and time of each use)" and "the identification of the natural persons involved in the verification of the results."[8] The article covers high-risk systems as the Act defines them; nothing read here says a bookkeeping agent is one.[8]

Set against that, the retention the four tools list runs from 14 days (a LangSmith base trace) through 30 and 90 days to 3 years (Langfuse Pro).[4][7] How Anthropic separates what an agent says from the state it leaves behind when testing is on Testing AI agents and hallucination rates.

Not yet proven

If you run an agent for clients and have decided how long to keep its traces, I would like to compare notes: mackenzie@mackenziebowes.com.

Sources

  1. James Newton-King, Inside the LLM Call: GenAI Observability with OpenTelemetry, OpenTelemetry blog, 2026-05-14. Read 2026-11-05. opentelemetry.io/blog/2026/genai-observability/
  2. OpenTelemetry, Semantic Conventions for GenAI agent and framework spans (status: Development). Read in Rotunda 2026-11-06. github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md
  3. Langfuse repository README. Read in Rotunda 2026-11-06. github.com/langfuse/langfuse
  4. Langfuse, Pricing. Read in Rotunda 2026-11-06. langfuse.com/pricing
  5. Arize Phoenix repository README. Read in Rotunda 2026-11-06. github.com/Arize-ai/phoenix
  6. Helicone repository README. Read in Rotunda 2026-11-06. github.com/Helicone/helicone
  7. LangChain, LangSmith Plans and Pricing. Read in Rotunda 2026-11-06. www.langchain.com/pricing
  8. European Commission AI Act Service Desk, Article 12: Record-keeping. Read in Rotunda 2026-11-06. ai-act-service-desk.ec.europa.eu/en/ai-act/article-12

Terms on this page

  • Agent observability: Being able to see what an AI agent did: which models and tools it called, in what order, and at what cost.
  • AI agent: A system where a language model decides for itself how to do a task and which tools to use.
  • LangChain: A software toolkit from LangChain for building AI agents and applications on language models.
  • Tool call: A request from an AI model to run one action, such as a search or a file write, which the surrounding program carries out.