Agent profiles, part six: measuring the agents

Part four says a campaign instruction I had been shipping for months was making runs slower rather than safer, and that counting is what showed it. This part is about the counting: what is instrumented, what each layer can actually see, and the contract that decides what any of the resulting numbers is allowed to claim.

The short version of why this exists: I had a rule with a plausible story attached and no number. Plausible stories are the expensive kind of wrong, because nothing about them fails.

Three layers, answering three different questions

Nothing here is one system. There are three, they overlap almost not at all, and the overlap is the part that took the longest to work out.

The harness’s own output. Both CLIs emit OpenTelemetry. Claude Code takes CLAUDE_CODE_ENABLE_TELEMETRY and an enhanced-telemetry beta flag; Codex takes an [otel] block naming a separate OTLP HTTP endpoint for each of logs, metrics and traces. Both point at one collector on a box at home rather than straight at a vendor, and the reason is boring: the endpoint changes, the collector’s config is one file, and every profile keeps pointing at the same place. Content capture is turned all the way up on both, so user prompts, assistant responses, tool input and tool detail all land in the log store. This is the layer that answers what a session did, in the harness’s own terms.

Grafana’s Agent Observability. The Claude profiles additionally export to Agent Observability, the product view over conversations, generations and evaluation. That gets you the shape of an agent’s work rather than its raw events: which generation belonged to which conversation, what a run cost, and where evaluators disagree with it. This is the layer that answers whether a session was any good, to the extent that question can be automated at all.

The wire. Part three covers the proxy the Codex profiles route through, and the exporter that tails its archives. That layer sees OpenAI’s internal engine ids, engine queue timing, the sub-agent spawn tree and per-response token counts, none of which appears in anything either CLI prints. It is the only layer that sees the protocol rather than the client’s account of the protocol.

One configuration decision in there deserves calling out on its own: the evaluation guards run with fail-open set and a 1,500-millisecond timeout. A guard that blocks your agent because the guard service is having a slow afternoon is strictly worse than no guard: it converts an observability dependency into an availability dependency for the thing being observed. Fail-open is the right default for anything in the request path that is not itself a security control.

The Security profile has none of this. No exporter, no product export, no proxy. That is the one profile where the telemetry would have to capture the contents of a security review, and there is no version of that trade I want.

Why the content capture is defensible here and almost nowhere else

Everything above logs conversation bodies. Assistant messages, tool input, complete command output. Anything an agent printed, including a secret it happened to cat, lands verbatim in a log store.

That is only defensible because the store is mine, single-tenant, on hardware I own, and the same argument that permits committing credentials to a git server on my own network permits this and nothing beyond it. Pointed at a shared log store, or a vendor tier where support staff can read indices, it would be indefensible and I would not make the same call. The switch that controls it exists in both stacks, and mine is deliberately set the permissive way rather than accidentally.

What none of the three layers could see

For all of that, none of it could answer the question I actually had.

When a long session runs out of context, something happens to the conversation: it gets compacted natively into an encrypted item, or summarised into text, or, on one profile, thrown away and replaced with a fresh working context. The three mechanisms behave completely differently and want completely different recovery behaviour. No layer above reports which one ran. The harness emits its usage and its events; the product view sees generations; the wire sees responses. Nobody emits “this window ended, and here is how”.

Worse, the tells that look like they identify it do not. A compacted marker in a transcript says a transition happened, not which kind. A visible prose summary looks like proof of the text-summary path and is not, because retained text coexists with native encrypted compaction. So the thing I wanted to measure had no metric and an unreliable eyeball test.

The collector, and three details worth stealing

What answers it is a fourth thing: a read-only collector over the session rollout files the CLI already writes.

Read-only, and a research collector rather than a profile hook, which is the distinction that makes it safe to run at all. It watches three profiles and deliberately not the other two, opens their files read-only, and stores no prompt, no command, no tool output, no encrypted payload and no credential. File paths and thread identifiers exist in its local state because checkpointing needs them, and never leave it as metric labels. Its state directory is mode 0700 and is in none of the backup repositories, because a transcript-derived research export has no business being committed anywhere the transcripts themselves are not.

A launchd user service runs it every 300 seconds while I am logged in. Each pass stats the inventory, reads only the bytes that are new, and commits the file offset together with its aggregates in a single transaction, so a pass that dies halfway leaves a consistent checkpoint rather than a half-counted file. A partial last line waits for its next append instead of being parsed. An inode replacement, a truncation, or a 512-byte boundary hash that no longer matches replays the whole file, and globally unique receipt, call and window keys are what stop that replay double-counting anything.

Three details in it came from getting it wrong first.

It runs with the standard process type, not the background type a periodic local collector obviously wants. Background scheduling had already starved this host’s other telemetry under load, and a scheduler that deprioritises you exactly when the machine is busy loses the data from the only period you cared about.

Its metrics endpoint serves a cached copy of about 160 KiB on loopback and never scans on scrape. A scrape that triggers work makes the observer part of what is being observed, and unpicking a monitoring system’s own load out of its graphs afterwards is tedious and avoidable.

And the scan budgets, two CPU seconds and twenty elapsed seconds and 64 MiB per pass, are cooperative checks between records rather than limits the operating system enforces. Inventory, a single record and the final write can all push past them. Individual records over 16 MiB are skipped and counted rather than silently dropped, and the first bootstrap hit exactly two of those, which is why the baseline it produced is not described as exhaustive. That bootstrap read 9.24 GB in 76 seconds with a peak resident size around 256 MiB.

A separate loopback-only receiver takes the historical backfill through the same processors and exporter as the live path, so the imported series and the ongoing ones are genuinely comparable rather than two shapes of the same name.

The metrics contract is the actual deliverable

The collector was an afternoon. The document saying what its output may and may not be used to claim took considerably longer, and it is the part I would keep if I had to throw one of them away.

The unit is a context-window transition, not a session and not a day. Every observation is grouped by profile, client version, model and effort, provider, observed mechanism, root or child role, task family and policy revision, and any field that cannot be established is recorded as unknown rather than inferred. Records are deduplicated on profile and window id, across inherited transcripts and across repeated daily captures, because the same window turns up in more than one file and summing daily snapshots would count it twice.

Then, for each metric, the boundary on what it means:

• Transition count is unique window ids separated by mechanism and profile. It is not a failure count.
• Recovery read candidates is a lexical match for goal, state, protocol, overview and memory indicators in the first four top-level tool calls after a transition. It can include entirely justified reads, or a command that merely quotes an example. Calling any of it waste requires reading them.
• Recovery output bytes is the UTF-8 size of the outputs in that call window, serialised envelopes included. It is not a token count, not wall time, and not per-command attribution.
• Truncation candidates counts recognised truncation markers. A quoted warning is a false positive, and the absence of a marker does not prove nothing was hidden.
• Post-transition context is the first token count the client reports afterwards. It is a client-reported baseline, not billed usage, and a missing one stays null rather than becoming zero.
• Compaction usage receipt is a reported receipt, not an invoice.
• Thread cumulative usage is the maximum reported value per field per file, never a sum, because forked and inherited counters contaminate a total and the result looks plausible either way.
• Effective context window is the set of distinct values the client reported. It establishes nothing about the model’s public maximum or any server’s compaction threshold.

Two metrics are deliberately left empty. Compaction latency is null because event adjacency does not isolate model latency, and time to first tool call is not time to useful work unless you have read what the tool was for. Quality is null because it needs evidence a counter cannot produce.

The daily snapshots are described in the document as descriptive surveillance rather than controlled experiments, which sounds like hedging and is not. New tasks, a client upgrade, a different model or simply a longer observation window all invalidate a naive before-and-after total, and the temptation to read one anyway is exactly what the label exists to resist.

What has been proven, and what has not

The honest accounting, which is shorter than I would like.

Proven: the read pattern. Across September, one profile recorded 951 text-summary transitions over 904 distinct windows; another recorded 41 native compactions and 64 fresh-context resets. In one campaign all ten observed native compactions were followed by a goal read inside the first four tool calls, several fetching overlapping ranges of the same file. Native retention did not remove the recovery work, because the goal instructed a re-read regardless. That was enough to change the instruction, and it is the whole of what the measurement has so far established.

Not proven, and stated as such in the source document rather than quietly omitted: no monetary saving, because proxy billing is not equivalent to list pricing and the receipts are receipts rather than invoices. No compaction-count threshold beyond which quality degrades. No efficiency or quality improvement from the change itself, because the controlled comparison has not run yet.

That comparison is designed and not executed. It holds model, effort, client, tools, permissions, starting state, acceptance contract and worker topology constant and changes exactly one factor at a time: the recovery contract first, native provider support separately, experimental-reset behaviour separately. Order is counterbalanced. Root and every descendant thread id is recorded, and total usage has to come from unique inference receipts attributed to those threads, with inherited and replayed records excluded, which is why total billing is currently left as unknown rather than estimated. The thing it compares is accepted outcomes per total observed usage, not root tokens, because fewer root tokens with the work pushed into children is not an improvement and reads exactly like one.

One line in that document does more work than the rest of it together: a workflow may become faster while failing acceptance, so report both and never collapse them into a favourable composite score. The matching rule is that a model’s own claim to have completed something is not acceptance evidence. Assess the artefact.

What I would tell someone starting this

Instrument the harness, because it is nearly free and you will want the history. Expect it not to answer your actual question, because the interesting failures are in the seams between the harness, the provider and your own process, and no vendor instruments a seam they do not own.

When you do build something custom, spend the time on the contract rather than the collector. A number with no stated boundary gets read as a result by the next person who sees it, and the next person is usually you, six months later, with the context gone.