How to watch an AI agent and keep an audit trail
OpenTelemetry's agent conventions are still marked Development and keep message content opt-in. Langfuse, Phoenix, Helicone and LangSmith keep traces from 14 days to 3 years. Read on each vendor's own pages, 2026-11-06.
Contents
Short answer, from the sources below: tools that record what an AI agent did (the model calls, the tool calls, the order, the cost) exist as open-source and hosted products, and they share one emerging format, OpenTelemetry's GenAI conventions, which is still marked "Development". They differ in licence, price and how long they keep a trace. None of the pages read describes tamper-evident or write-once storage, which is what the word audit trail usually implies. This page lists what each source says, who wrote it and what it leaves out.
What the standard records
Watching an agent means recording its steps. OpenTelemetry, the open standard for telemetry, publishes semantic conventions for generative AI. A Microsoft engineer's post on the OpenTelemetry blog (2026-05-14) says they "standardize how GenAI operations are recorded", including "the model being called, input and output token counts."[1]
The conventions for agents name an operation for each step: create_agent for making an agent and invoke_agent for running one, with spans for tool calls.[2] The page carries the status "Development", so the names can still change.[2]
The attributes that would hold the evidence are optional. gen_ai.input.messages (the chat history sent to the model) and gen_ai.output.messages (what the model returned) are both marked "Opt-In", and the page warns that the attribute "is likely to contain sensitive information including user/PII data."[2] A reference form, gen_ai.input.messages_ref, holds "an absolute URI identifying the input messages in external storage" so the content can live elsewhere, and the page adds that "the presence of this attribute does not indicate whether storing the content succeeded or whether the content remains available."[2]
That default is the reverse of at least one SDK. The OpenAI Agents SDK page quotes its tracing guide: "By default, trace_include_sensitive_data is True."
Four tools as each publishes itself
- Langfuse. Its repository says: "This repository is MIT licensed, except for the ee folders."[3] Langfuse Cloud lists Hobby at free with 50k units a month and "30 days data access", Core at $29 a month with "90 days data access", Pro at $199 a month with "3 years data access", and Enterprise at $2499 a month, the only plan card that lists "Audit Logs".[4] Self-hosting is described as Docker Compose on "your own machine or a single virtual machine".[3]
- Arize Phoenix. Its README says "Phoenix is open-source and self-hosted", is "built on top of OpenTelemetry", and that "This software is licensed under the terms of the Elastic License 2.0 (ELv2)."[5] Arize sells a managed platform, Arize AX, which uses the same instrumentation.[5]
- Helicone. The repository is marked "Apache-2.0 license". It describes itself as "an AI Gateway & LLM Observability Platform" and promises "One-line of code to log all your requests".[6]
- LangSmith (LangChain). Developer is $0 per seat a month with up to 5k base traces; Plus is $39 per seat a month with up to 10k.[7] "Base traces have a shorter retention period of 14 days. Extended traces have a longer retention period of 180 days", and a base trace can be upgraded to an extended one "for an additional fee."[7] Enterprise adds "Self-hosted and hybrid deployment options."[7]
Prices are in U.S. dollars as listed on 2026-11-06. Langfuse's page counts "units" and LangSmith's counts traces, so the free tiers are not directly comparable.[4][7]
What a record has to hold
The European Union's AI Act is the one law found that names logging. Article 12 says "High-risk AI systems shall technically allow for the automatic recording of events (logs) over the lifetime of the system."[8] For one class of high-risk systems the minimum includes "the period of each use of the system (start date and time and end date and time of each use)" and "the identification of the natural persons involved in the verification of the results."[8] The article covers high-risk systems as the Act defines them; nothing read here says a bookkeeping agent is one.[8]
Set against that, the retention the four tools list runs from 14 days (a LangSmith base trace) through 30 and 90 days to 3 years (Langfuse Pro).[4][7] How Anthropic separates what an agent says from the state it leaves behind when testing is on Testing AI agents and hallucination rates.
Not yet proven
- That any of these four tools was run. Every fact about them comes from the vendor's own repository or pricing page; no independent comparison is cited and none of the tools was installed.
- Tamper-evident storage. None of the pages read describes immutable, signed or write-once logs.
- Which retention a bookkeeping practice needs. No source read ties a trace retention period to an accounting record-keeping rule.
- Whether the OpenTelemetry names will hold. The agent conventions are marked Development.
- Tools not read. Datadog, Honeycomb, Weights & Biases Weave and the cloud providers' own agent monitoring are not covered.
- What a "unit" is in Langfuse's billing. The pricing page counts units; the definition was not read.
If you run an agent for clients and have decided how long to keep its traces, I would like to compare notes: mackenzie@mackenziebowes.com.
Sources
- James Newton-King, Inside the LLM Call: GenAI Observability with OpenTelemetry, OpenTelemetry blog, 2026-05-14. Read 2026-11-05. opentelemetry.
io/ blog/ 2026/ genai- observability/ - OpenTelemetry, Semantic Conventions for GenAI agent and framework spans (status: Development). Read in Rotunda 2026-11-06. github.
com/ open- telemetry/ semantic- conventions- genai/ blob/ main/ docs/ gen- ai/ gen- ai- agent- spans. md - Langfuse repository README. Read in Rotunda 2026-11-06. github.
com/ langfuse/ langfuse - Langfuse, Pricing. Read in Rotunda 2026-11-06. langfuse.
com/ pricing - Arize Phoenix repository README. Read in Rotunda 2026-11-06. github.
com/ Arize- ai/ phoenix - Helicone repository README. Read in Rotunda 2026-11-06. github.
com/ Helicone/ helicone - LangChain, LangSmith Plans and Pricing. Read in Rotunda 2026-11-06. www.
langchain. com/ pricing - European Commission AI Act Service Desk, Article 12: Record-keeping. Read in Rotunda 2026-11-06. ai-
act- service- desk. ec. europa. eu/ en/ ai- act/ article- 12
Terms on this page
- Agent observability: Being able to see what an AI agent did: which models and tools it called, in what order, and at what cost.
- AI agent: A system where a language model decides for itself how to do a task and which tools to use.
- LangChain: A software toolkit from LangChain for building AI agents and applications on language models.
- Tool call: A request from an AI model to run one action, such as a search or a file write, which the surrounding program carries out.