---
title: "Deduplicating and joining AI agent conversation logs"
url: "https://ryle.dev/blog/deduplicating-agent-conversation-logs/"
description: "Gateway API logs vs Claude Code and Codex transcripts: record types, human vs harness turns, dedup through caching and compaction, and joining on IDs."
author: "Ryle Goehausen"
published: 2026-10-29
---

# Deduplicating and joining AI agent conversation logs

[Ryle Goehausen](https://ryle.dev/about/) · Oct 29, 2026

Every conversation my [coding agents](https://ryle.dev/blog/agent-fleet-lessons/) have gets logged twice. The LLM gateway in front of the model APIs writes down every request and response, and the harness the agent runs in, Claude Code or Codex, keeps its own transcript. I want one record per conversation out of the two: every message stored once, every turn labeled with who produced it, and the two sources joined so a question about either can be answered in one place. I also want to be able to delete that record and rebuild it whenever I change the rules.

Storing the bytes is the easy part. The two logs use different record types, they disagree about who said what, and each one changes the conversation's history in its own way. Prompt caching rewrites old messages between requests, compaction replaces them with a summary, and resumed sessions copy them into new files. This post goes through what each format contains and the design I'd use to deduplicate and join them.

The counts come from one of my machines and the gateway that serves it: 1,269 Claude Code session files with 831,000 lines (Claude Code 2.1.259 to 2.1.282), a 3 GB sample of Codex rollout files with 713,000 lines (mostly Codex CLI 0.157 to 0.163), and 160 gateway lines from one day. Both harnesses change their formats often, so treat the type names below as a snapshot.

## The gateway's line

The gateway writes one JSON line per upstream call. Stripped down, a line looks like this:

```
{"id": "…", "model": "…", "http_status": 200, "success": true,
 "request_timestamp_ms": 1793232000000,
 "response_timestamp_ms": 1793232042000, "latency_ms": 42000,
 "request_body": {"model": "…", "system": […], "messages": […],
                  "tools": […],
                  "metadata": {"user_id": "{\"session_id\": \"…\", …}"}},
 "response_body": {"id": "msg_…", "content": […],
                   "stop_reason": "tool_use", "usage": {…}},
 "auxiliary_ids": {"…": "…"}}
```

`request_body` is what the gateway sent upstream, after its own edits. `auxiliary_ids` holds what the gateway knows about the call: the host and session it came from, which account served it, token counts and the provider's rate-limit headers.

The request body comes in two dialects in the same log. Claude Code speaks the Anthropic Messages API: a `system` list, then `messages`, whose content is either a string or a list of typed blocks (`text`, `thinking`, `tool_use`, `tool_result`). Its `metadata.user_id` is itself a JSON string that carries the session ID. Codex speaks the OpenAI Responses API: `instructions`, then `input`, a flat list of typed items. Those are `message` (role `user`, `assistant` or `developer`), `reasoning` (encrypted), `function_call` and `function_call_output`, and `custom_tool_call` and its output. Almost every item has its own ID with a type prefix (`fc_`, `rs_`, `msg_` and so on). Codex sends `store: false`, so every request carries the whole history. Its `client_metadata` holds Codex's own session, thread, turn and context-window IDs.

Responses are streamed, so the gateway has to reassemble them before logging. For the Messages API that gives back the final message: its `msg_` ID, content blocks, stop reason and usage. For the Responses API the logger has to keep the output items themselves, along with the response ID (`resp_`) and usage from the completion event. Otherwise each answer exists only where the next request resends it, and the last answer of a session doesn't exist in the log at all.

The gateway copy is the only place with the exact prompt the model read. That includes the prompts and memories the gateway [injects itself](https://ryle.dev/blog/redact-secrets-keep-prompt-cache/), along with the rendered form of everything the harness added. It also has usage, cache reads, latency, status and the serving account. It's very redundant, because each request resends everything before it.

## Claude Code's transcript

Claude Code writes one JSONL file per session. Every line has a `type`, and I found 21 of them:

| Type | Lines | What it is |
| --- | --- | --- |
| `assistant` | 264,358 | One content block of a model response |
| `attachment` | 258,423 | Context the harness injects: token budget, environment, date, hook results, changed files |
| `user` | 150,648 | Tool results, prompts and harness messages |
| `system` | 10,928 | Local events, almost none of them sent to the model |
| 17 others | 147,113 | Titles, modes, the last prompt, queued input, cost totals, file-history snapshots, pointers to forks and continuations |

Message lines carry `uuid`, `parentUuid`, `sessionId`, `timestamp`, `cwd`, `gitBranch` and the Claude Code `version`. Subagent lines set `isSidechain` and add an `agentId`. Inside `message`, the role is only ever `user` or `assistant`.

These are the parts that matter for deduplication and joining:

- **One line per content block.** A response with a thinking block, some text and two tool calls becomes four lines that share `message.id` and `requestId`, with each block's position in `apiBlockIndex`. The 264,358 assistant lines came from 138,296 responses.
- **Some assistant lines never came from a model.** 579 are error placeholders (`isApiErrorMessage`) that Claude Code writes when a request fails.
- **Attachments are stored typed.** `attachment.type` took 41 values, led by `total_tokens_reminder` (134,159 lines), `output_style` (79,269), `environment` (7,934) and `silent_turn_reminder` (6,818). Each line also stores `rendered`, the text the model receives, wrapped in `<system-reminder>` tags. In the requests my gateway logged, that context arrives as system-role messages between turns (a median of 32 per request) or as text inside a user turn. 2,800 attachment lines are marked as rendered into the person's own turn, and 8,157 have no rendered form at all.
- **Tool results are stored twice, shaped per tool.** A tool result line holds the `tool_result` block the model sees and a `toolUseResult` copy for the UI. Shell commands get `stdout`, `stderr`, `interrupted` and `isImage`, and web searches get `query`, `results` and `durationSeconds`. Edits get the old and new strings, a structured patch and a copy of the whole original file. In one 134 MB transcript, 455 of the 658 tool results of 200 characters or more were identical in both copies and the other 203 overlapped, so each copy was about 47% of the file.
- **System lines have a `subtype`.** The values were `stop_hook_summary`, `turn_duration`, `away_summary`, `compact_boundary`, `local_command`, `api_error`, `informational` and a few rarer ones. Most are local bookkeeping. `local_command` is the exception: its `<local-command-stdout>` text goes to the model as user text, and I found it word for word in the gateway's requests.
- **The parent chain is a tree.** A tool result hangs off the line holding its `tool_use`, and the next block of the same response points at that line too. Of the 2,735 places where a line has more than one child, 2,549 follow that pattern. Only 56 are different prompts sent from the same point, which is what a rewind or an edited prompt looks like.
- **Lines get copied between files.** 1,210 line UUIDs appear in two to seven files: 662 within one session's subagent files, and 548 across separate sessions, presumably where a resumed or continued session copied earlier history.

## Codex's rollout

Codex writes one rollout file per session, with lines of the form `{"timestamp", "type", "payload"}`:

| Type | Lines | What it is |
| --- | --- | --- |
| `response_item` | 304,542 | The model-visible history, item by item |
| `event_msg` | 303,404 | UI and progress events |
| `token_usage_record` | 96,933 | Usage per response, with the response ID, turn and thread |
| `inter_agent_communication_metadata` | 3,554 | Messages between agents |
| `turn_context` | 2,630 | Per-turn settings: directory, model, effort, approval and sandbox policy |
| `world_state` | 1,237 | AGENTS.md, environments and mode guidance |
| `compacted` | 753 | A compaction: the summary and the history that replaces the old one |
| `session_meta` | 326 | Session ID, directory, CLI version, base instructions, git state, parent and fork IDs |

`response_item` payloads use the Responses API's item types: `reasoning` (93,422), `custom_tool_call` and its output (about 64,000 each), `function_call` and its output (about 30,000 each), `message` (19,378: 16,689 assistant, 1,802 user and 887 developer), and `agent_message` for traffic between agents. They carry the same item IDs the gateway sees.

`event_msg` mostly repeats those items in another shape. Its `item_completed` events (199,546 lines) wrap an item kind written in CamelCase: `Reasoning`, `CommandExecution`, `AgentMessage`, `FileChange`, `UserMessage`, `ContextCompaction` and a few others. Most `CommandExecution` items hold `stdout`, `stderr`, `aggregated_output` and `formatted_output`, on top of the tool-output item the model saw, so one command's output can appear three or four times in the file.

A `compacted` line holds the summary in `message` and the new history in `replacement_history`. That's between 2 and 376 items, with a median of 11: earlier user messages, developer instructions and messages between agents, restated. In one 85 MB rollout the `compacted` lines were 25% of the bytes, and 99% of their long text had already appeared earlier in the same file. The line also carries `window_id`, `previous_window_id` and `first_window_id`, which chain the context windows together.

## Who said it

This is where the formats differ most. The APIs have only a few roles (user, assistant, and system or developer), so anything the harness generates has to arrive under one of them. In the gateway log, a tool result, a typed prompt, a background-task notification, a hook's complaint and a compaction summary are all role `user`. Tool results at least have their own block type; for the rest, only the text tells them apart. The transcripts label them much better:

| What happened | Gateway (Messages API) | Claude Code transcript | Codex rollout |
| --- | --- | --- | --- |
| Someone typed a prompt | User message, plain text | `user` line, `turnOrigin: human`, `promptSource: typed` (or `queued` if typed while the agent was busy) | User `message` item and a `UserMessage` event |
| A tool ran | User message of `tool_result` blocks | `user` line plus `toolUseResult`, a structured copy | `function_call_output` or `custom_tool_call_output` item, plus a `CommandExecution` event |
| Harness context | System-role message, or `<system-reminder>` text inside a user turn | `attachment` line with its type and rendered text | Developer message, or `<environment_context>` and AGENTS.md text in a user message |
| Background task or another agent | User message with a tag or a fixed opening line | `user` line, `turnOrigin: task_notification` or `peer`, sender in `origin` | `agent_message` item and an inter-agent record |
| Hook feedback | User message “Stop hook feedback: …” and a system message with the error | `isMeta` user line, a `hook_blocking_error` attachment and a `stop_hook_summary` system line | Not in my sample |
| Slash command | User text with `<command-name>` and `<local-command-stdout>` tags | `user` lines with the same tags, plus a `local_command` system line | Not in my sample |
| Compaction | The summary as the first user message of later requests | `compact_boundary` system line, then a user line with `isCompactSummary` | `compacted` line |
| Failed request | An error status and message | Synthetic assistant line with `isApiErrorMessage`, plus an `api_error` system line | Not in my sample |

The counts show how little of "user" is the user. Of the 150,648 user lines in the Claude Code transcripts, 141,648 are tool results. Of the other 9,000, about 1,900 are labeled as prompts someone typed or queued. The rest come from the harness or from other agents: 1,707 task notifications, 945 hook-feedback lines, 731 compaction summaries, about 600 messages from other agent sessions, 446 teammate messages, and slash-command, local-command and reminder lines. The labels also change between versions and don't always mean what they seem to. About 600 plain-text user lines carry no origin fields at all, only some harness lines set `isMeta`, and 405 of the compaction summaries carry `turnOrigin: human`, presumably from the turn they interrupted.

On the Codex side, 577 of the 2,050 text blocks in user messages were `<environment_context>` or AGENTS.md text, and its developer messages carry permissions, collaboration mode, skills and multi-agent instructions.

So who said what belongs in a derived column. Compute it from the transcript's labels where a transcript exists, and from tags in the text where only the gateway saw the call. Give the classifier a version number so it can be rerun, and never put its output in a key.

## Keys that come only from content

Every row in the derived data gets a key computed from the record's content: a version number, the dataset name and the record itself, encoded as [canonical CBOR](https://ryle.dev/blog/protobuf-to-serde-cbor/) and hashed with BLAKE3.

```
#[derive(Serialize)]
struct CaptureKey<'a, T: Serialize> {
    version: u8,
    dataset: &'a str,
    record: &'a T,
}

fn capture_key<T: Serialize>(dataset: &str, record: &T) -> blake3::Hash {
    blake3::hash(&canonical_cbor(&CaptureKey { version: 1, dataset, record }))
}
```

The timestamp in the key is the one recorded in the source, never the time the importer ran, and a row's position in its write batch and the worker that wrote it stay out. Re-running an import is then a no-op, two workers importing the same file write the same keys, and a retry after a crash can't duplicate anything.

Deduplication then works with three identities, kept apart:

- **Capture.** The key above, over a whole gateway line or transcript line. An exact replay collapses into one row; any difference keeps its own.
- **Message body.** Each message or content block, normalized, stored once under its content hash. This is where the space goes away.
- **Occurrence.** A small row saying this body appeared in this conversation, at this position, in this request or file.

Equal text never merges two occurrences. Agents repeat themselves all the time ("continue", the same `git status` before every commit), and merging on text would lose which conversations a message belonged to.

## History that changes under you

### Old messages get rewritten

The obvious way to deduplicate the gateway log is to notice that each request contains the previous one plus a few new messages, and keep only the last. On raw bytes that doesn't work. In a sample of 166 conversations, none of 22,057 requests contained the previous request in its conversation byte for byte.

Prompt caching is most of the reason. The client marks cache breakpoints with `cache_control` and moves the newest one forward on each call, which strips the marker from a block in an earlier message and rewrites that message's JSON. Across 500 consecutive request pairs it happened 465 times. Thinking and tool blocks also get reshaped, and some messages switch between a plain string and a list of blocks. In about 1% of positions an earlier message came back with a different number of blocks.

So messages get normalized before they're hashed: remove `cache_control`, turn a plain string into a single text block, and leave thinking signatures out of the identity. After that, the same message hashes the same in every request that carries it.

### Compaction replaces them

When a conversation nears the context limit, the harness summarizes older turns and carries on from the summary. In the gateway log that shows up as the message count dropping from one request to the next. In the 166-conversation sample, 45% of conversations had at least one drop. Those averaged 32.6 drops each, and on average the last one came about three-quarters of the way through.

The transcripts keep the old history. Claude Code writes a `compact_boundary` line whose `parentUuid` is empty and whose `logicalParentUuid` points at the last line before the compaction, with metadata on the trigger (automatic or manual) and the token counts before and after. Codex writes a `compacted` line with the replacement history. In the gateway log, requests after a compaction start from the summary, and the history before it exists only in the earlier requests.

That rules out keeping the longest request per conversation. In 20% of the conversations that compacted, the longest request came before the last compaction, so everything after it would be lost. The unit that keeps everything is the segment between drops: keep the longest request in each segment.

| Keep | Bytes | Smaller by |
| --- | --- | --- |
| Every request | 8.79 GB | 1× |
| Longest request per segment | 705 MB | 12.5× |
| Longest request per conversation (loses content) | 84 MB | 105× |

Two separate implementations of the segment count came out at 12.46× and 12.45×. Keeping everything costs a factor of about 8 in compression compared with the lossy rule. The unit matters most for anything priced per token: summarizing the roughly 1.65 million requests from the log's first seven weeks with a cheap hosted model would cost $9,800 to $14,400 one request at a time, and $790 to $1,160 one segment at a time.

### Branches, copies and resends

A rewind or an edited prompt makes a branch. In a Claude Code transcript it's a second child of the same parent, and in the gateway it's a request whose history stops matching the one before it. Both branches happened, so both keep their occurrences.

Copied history gets deduplicated by the line UUID together with the content hash. A resumed session's copy of an old line is the same line, but each file still records that it held it. Subagents get their own files in Claude Code and their own sessions in Codex. In the gateway they're separate runs of requests with their own system prompt, and the request metadata can name the parent session.

Calls without a session ID need grouping by content. Group them by the process and model they came from, order them by the request's own timestamp, and chain a request to an earlier one when its normalized messages extend the earlier one's. Expect many of the matches to be resends. In one sampled hour of short exchanges, exact matching grouped about two-thirds of the requests, but 43% of the matched pairs were identical.

## Joining the two

The provider's IDs do the joining, because both copies record them.

For Claude Code, the `msg_` ID in the gateway's reassembled response is the transcript's `message.id`. Sort that message's lines by `apiBlockIndex` and you get the response back: of the 26 responses in my gateway sample that also appeared in local transcripts, all 26 had the same sequence of blocks in both. `tool_use` IDs appear in both copies too, and the session ID inside `metadata.user_id` names the transcript file. Attachments match on their rendered text, with the wrapper tags removed, against the system-role messages and user text between the same two turns. Typed prompts have no provider ID, so they join through the parent chain to the response that answered them.

For Codex, the gateway's `resp_` response ID is the rollout's `token_usage_record.response_id`, and the item IDs are shared: 166 of the item IDs in my gateway sample appeared in a single local rollout. The session, thread and turn IDs in `client_metadata` match the rollout's `session_meta` and `turn_context` lines.

The join is a full outer join, so rows found in either source survive, and it only matches inside one owner, workspace and conversation. Where one source lacks a field, the other fills it in. Where they disagree, both values are kept side by side, and the original rows ride along in a carrier column so the join can be redone with better rules. Equal text and nearby timestamps never count as a match.

## Keep the originals

All of the above should be rebuildable, which means the originals have to stay byte for byte. The gateway serializes each call once and appends the line to gzipped JSONL in object storage before anything parses it. Merging small objects into bigger ones concatenates lines, and later steps copy those bytes without re-encoding them, so a call's ID can be the BLAKE3 hash of its line and comes out the same on every replay.

Transcripts go into a content-addressed archive, collected from every machine and volume an agent ran on, with a manifest per collection run. Session files only grow, so most versions are a prefix of a later one. Across my agents that came to about 440,000 file versions and 336 GB, then 94,012 distinct contents and 250 GB after hashing whole files. Of that, 166 GB was exact prefixes of a longer copy of the same file, so storing the longest version plus a length and a hash for each earlier one brings it to about 84 GB.

## Where it lives

The derived data lives in Lance files: one table per source, the message bodies, the joined occurrences, per-call metadata, and enrichment such as embeddings of each unique message body. Each part file carries [sidecar indexes](https://ryle.dev/blog/lance-index-sidecars/), and the last few days live in Postgres and get merged in on read. Embeddings and summaries are keyed by content hash plus model version, so a message resent a thousand times is embedded once, and the embeddings run under a $10 monthly budget.

The originals live in Hetzner Object Storage, in the same region as the cluster, and get copied to Backblaze B2 with `rclone copy --immutable`, which adds new objects and never overwrites or deletes one. The Lance tables stay in one region, since losing them costs a replay and nothing more. A backup counts once a restore from B2 has been checked against the primary's manifest. The originals are also the most sensitive copy, because a transcript keeps tool output exactly as printed, so access to them should be narrower than access to the screened tables built from them.

## Where I'd start

Get the originals right on day one: both copies, byte-exact, backed up offsite. Then treat everything else as derived and versioned: the normalization, the who-said-it classification, the join and the summaries. All of those will change as the harnesses change their formats, and each change is a replay.

## Related posts

- [What I learned running a fleet of AI coding agents](https://ryle.dev/blog/agent-fleet-lessons/)
- [Injecting prompts, memories and reviews at an LLM gateway without breaking the prompt cache](https://ryle.dev/blog/redact-secrets-keep-prompt-cache/)
- [Taking protobuf out of a 1,000-crate Rust monorepo](https://ryle.dev/blog/protobuf-to-serde-cbor/)
- [Indexing Lance files without the Dataset API](https://ryle.dev/blog/lance-index-sidecars/)