Deduplicating and joining AI agent conversation logs

Every conversation my coding agents have gets logged twice. The LLM gateway in front of the model APIs writes down every request and response, and the harness the agent runs in, Claude Code or Codex, keeps its own transcript. I want one record per conversation out of the two: every message stored once, every turn labeled with who produced it, and the two sources joined so a question about either can be answered in one place. I also want to be able to delete that record and rebuild it whenever I change the rules.

Storing the bytes is the easy part. The two logs use different record types, they disagree about who said what, and each one changes the conversation's history in its own way. Prompt caching rewrites old messages between requests, compaction replaces them with a summary, and resumed sessions copy them into new files. This post goes through what each format contains and the design I'd use to deduplicate and join them.

The counts come from one of my machines and the gateway that serves it: 1,269 Claude Code session files with 831,000 lines (Claude Code 2.1.259 to 2.1.282), a 3 GB sample of Codex rollout files with 713,000 lines (mostly Codex CLI 0.157 to 0.163), and 160 gateway lines from one day. Both harnesses change their formats often, so treat the type names below as a snapshot.

The gateway's line

The gateway writes one JSON line per upstream call. Stripped down, a line looks like this:

{"id": "…", "model": "…", "http_status": 200, "success": true,
 "request_timestamp_ms": 1793232000000,
 "response_timestamp_ms": 1793232042000, "latency_ms": 42000,
 "request_body": {"model": "…", "system": […], "messages": […],
                  "tools": […],
                  "metadata": {"user_id": "{\"session_id\": \"…\", …}"}},
 "response_body": {"id": "msg_…", "content": […],
                   "stop_reason": "tool_use", "usage": {…}},
 "auxiliary_ids": {"…": "…"}}

request_body is what the gateway sent upstream, after its own edits. auxiliary_ids holds what the gateway knows about the call: the host and session it came from, which account served it, token counts and the provider's rate-limit headers.

The request body comes in two dialects in the same log. Claude Code speaks the Anthropic Messages API: a system list, then messages, whose content is either a string or a list of typed blocks (text, thinking, tool_use, tool_result). Its metadata.user_id is itself a JSON string that carries the session ID. Codex speaks the OpenAI Responses API: instructions, then input, a flat list of typed items. Those are message (role user, assistant or developer), reasoning (encrypted), function_call and function_call_output, and custom_tool_call and its output. Almost every item has its own ID with a type prefix (fc_, rs_, msg_ and so on). Codex sends store: false, so every request carries the whole history. Its client_metadata holds Codex's own session, thread, turn and context-window IDs.

Responses are streamed, so the gateway has to reassemble them before logging. For the Messages API that gives back the final message: its msg_ ID, content blocks, stop reason and usage. For the Responses API the logger has to keep the output items themselves, along with the response ID (resp_) and usage from the completion event. Otherwise each answer exists only where the next request resends it, and the last answer of a session doesn't exist in the log at all.

The gateway copy is the only place with the exact prompt the model read. That includes the prompts and memories the gateway injects itself, along with the rendered form of everything the harness added. It also has usage, cache reads, latency, status and the serving account. It's very redundant, because each request resends everything before it.

Claude Code's transcript

Claude Code writes one JSONL file per session. Every line has a type, and I found 21 of them:

Message lines carry uuid, parentUuid, sessionId, timestamp, cwd, gitBranch and the Claude Code version. Subagent lines set isSidechain and add an agentId. Inside message, the role is only ever user or assistant.

These are the parts that matter for deduplication and joining:

• One line per content block. A response with a thinking block, some text and two tool calls becomes four lines that share message.id and requestId, with each block's position in apiBlockIndex. The 264,358 assistant lines came from 138,296 responses.
• Some assistant lines never came from a model. 579 are error placeholders (isApiErrorMessage) that Claude Code writes when a request fails.
• Attachments are stored typed. attachment.type took 41 values, led by total_tokens_reminder (134,159 lines), output_style (79,269), environment (7,934) and silent_turn_reminder (6,818). Each line also stores rendered, the text the model receives, wrapped in <system-reminder> tags. In the requests my gateway logged, that context arrives as system-role messages between turns (a median of 32 per request) or as text inside a user turn. 2,800 attachment lines are marked as rendered into the person's own turn, and 8,157 have no rendered form at all.
• Tool results are stored twice, shaped per tool. A tool result line holds the tool_result block the model sees and a toolUseResult copy for the UI. Shell commands get stdout, stderr, interrupted and isImage, and web searches get query, results and durationSeconds. Edits get the old and new strings, a structured patch and a copy of the whole original file. In one 134 MB transcript, 455 of the 658 tool results of 200 characters or more were identical in both copies and the other 203 overlapped, so each copy was about 47% of the file.
• System lines have a subtype. The values were stop_hook_summary, turn_duration, away_summary, compact_boundary, local_command, api_error, informational and a few rarer ones. Most are local bookkeeping. local_command is the exception: its <local-command-stdout> text goes to the model as user text, and I found it word for word in the gateway's requests.
• The parent chain is a tree. A tool result hangs off the line holding its tool_use, and the next block of the same response points at that line too. Of the 2,735 places where a line has more than one child, 2,549 follow that pattern. Only 56 are different prompts sent from the same point, which is what a rewind or an edited prompt looks like.
• Lines get copied between files. 1,210 line UUIDs appear in two to seven files: 662 within one session's subagent files, and 548 across separate sessions, presumably where a resumed or continued session copied earlier history.

Codex's rollout

Codex writes one rollout file per session, with lines of the form {"timestamp", "type", "payload"}:

response_item payloads use the Responses API's item types: reasoning (93,422), custom_tool_call and its output (about 64,000 each), function_call and its output (about 30,000 each), message (19,378: 16,689 assistant, 1,802 user and 887 developer), and agent_message for traffic between agents. They carry the same item IDs the gateway sees.

event_msg mostly repeats those items in another shape. Its item_completed events (199,546 lines) wrap an item kind written in CamelCase: Reasoning, CommandExecution, AgentMessage, FileChange, UserMessage, ContextCompaction and a few others. Most CommandExecution items hold stdout, stderr, aggregated_output and formatted_output, on top of the tool-output item the model saw, so one command's output can appear three or four times in the file.

A compacted line holds the summary in message and the new history in replacement_history. That's between 2 and 376 items, with a median of 11: earlier user messages, developer instructions and messages between agents, restated. In one 85 MB rollout the compacted lines were 25% of the bytes, and 99% of their long text had already appeared earlier in the same file. The line also carries window_id, previous_window_id and first_window_id, which chain the context windows together.

Who said it

This is where the formats differ most. The APIs have only a few roles (user, assistant, and system or developer), so anything the harness generates has to arrive under one of them. In the gateway log, a tool result, a typed prompt, a background-task notification, a hook's complaint and a compaction summary are all role user. Tool results at least have their own block type; for the rest, only the text tells them apart. The transcripts label them much better:

The counts show how little of "user" is the user. Of the 150,648 user lines in the Claude Code transcripts, 141,648 are tool results. Of the other 9,000, about 1,900 are labeled as prompts someone typed or queued. The rest come from the harness or from other agents: 1,707 task notifications, 945 hook-feedback lines, 731 compaction summaries, about 600 messages from other agent sessions, 446 teammate messages, and slash-command, local-command and reminder lines. The labels also change between versions and don't always mean what they seem to. About 600 plain-text user lines carry no origin fields at all, only some harness lines set isMeta, and 405 of the compaction summaries carry turnOrigin: human, presumably from the turn they interrupted.

On the Codex side, 577 of the 2,050 text blocks in user messages were <environment_context> or AGENTS.md text, and its developer messages carry permissions, collaboration mode, skills and multi-agent instructions.

So who said what belongs in a derived column. Compute it from the transcript's labels where a transcript exists, and from tags in the text where only the gateway saw the call. Give the classifier a version number so it can be rerun, and never put its output in a key.

Keys that come only from content

Every row in the derived data gets a key computed from the record's content: a version number, the dataset name and the record itself, encoded as canonical CBOR and hashed with BLAKE3.

#[derive(Serialize)]
struct CaptureKey<'a, T: Serialize> {
    version: u8,
    dataset: &'a str,
    record: &'a T,
}

fn capture_key<T: Serialize>(dataset: &str, record: &T) -> blake3::Hash {
    blake3::hash(&canonical_cbor(&CaptureKey { version: 1, dataset, record }))
}

The timestamp in the key is the one recorded in the source, never the time the importer ran, and a row's position in its write batch and the worker that wrote it stay out. Re-running an import is then a no-op, two workers importing the same file write the same keys, and a retry after a crash can't duplicate anything.

Deduplication then works with three identities, kept apart:

• Capture. The key above, over a whole gateway line or transcript line. An exact replay collapses into one row; any difference keeps its own.
• Message body. Each message or content block, normalized, stored once under its content hash. This is where the space goes away.
• Occurrence. A small row saying this body appeared in this conversation, at this position, in this request or file.

Equal text never merges two occurrences. Agents repeat themselves all the time ("continue", the same git status before every commit), and merging on text would lose which conversations a message belonged to.

History that changes under you

Old messages get rewritten

The obvious way to deduplicate the gateway log is to notice that each request contains the previous one plus a few new messages, and keep only the last. On raw bytes that doesn't work. In a sample of 166 conversations, none of 22,057 requests contained the previous request in its conversation byte for byte.

Prompt caching is most of the reason. The client marks cache breakpoints with cache_control and moves the newest one forward on each call, which strips the marker from a block in an earlier message and rewrites that message's JSON. Across 500 consecutive request pairs it happened 465 times. Thinking and tool blocks also get reshaped, and some messages switch between a plain string and a list of blocks. In about 1% of positions an earlier message came back with a different number of blocks.

So messages get normalized before they're hashed: remove cache_control, turn a plain string into a single text block, and leave thinking signatures out of the identity. After that, the same message hashes the same in every request that carries it.

Compaction replaces them

When a conversation nears the context limit, the harness summarizes older turns and carries on from the summary. In the gateway log that shows up as the message count dropping from one request to the next. In the 166-conversation sample, 45% of conversations had at least one drop. Those averaged 32.6 drops each, and on average the last one came about three-quarters of the way through.

The transcripts keep the old history. Claude Code writes a compact_boundary line whose parentUuid is empty and whose logicalParentUuid points at the last line before the compaction, with metadata on the trigger (automatic or manual) and the token counts before and after. Codex writes a compacted line with the replacement history. In the gateway log, requests after a compaction start from the summary, and the history before it exists only in the earlier requests.

That rules out keeping the longest request per conversation. In 20% of the conversations that compacted, the longest request came before the last compaction, so everything after it would be lost. The unit that keeps everything is the segment between drops: keep the longest request in each segment.

Two separate implementations of the segment count came out at 12.46× and 12.45×. Keeping everything costs a factor of about 8 in compression compared with the lossy rule. The unit matters most for anything priced per token: summarizing the roughly 1.65 million requests from the log's first seven weeks with a cheap hosted model would cost $9,800 to $14,400 one request at a time, and $790 to $1,160 one segment at a time.

Branches, copies and resends

A rewind or an edited prompt makes a branch. In a Claude Code transcript it's a second child of the same parent, and in the gateway it's a request whose history stops matching the one before it. Both branches happened, so both keep their occurrences.

Copied history gets deduplicated by the line UUID together with the content hash. A resumed session's copy of an old line is the same line, but each file still records that it held it. Subagents get their own files in Claude Code and their own sessions in Codex. In the gateway they're separate runs of requests with their own system prompt, and the request metadata can name the parent session.

Calls without a session ID need grouping by content. Group them by the process and model they came from, order them by the request's own timestamp, and chain a request to an earlier one when its normalized messages extend the earlier one's. Expect many of the matches to be resends. In one sampled hour of short exchanges, exact matching grouped about two-thirds of the requests, but 43% of the matched pairs were identical.

Joining the two

The provider's IDs do the joining, because both copies record them.

For Claude Code, the msg_ ID in the gateway's reassembled response is the transcript's message.id. Sort that message's lines by apiBlockIndex and you get the response back: of the 26 responses in my gateway sample that also appeared in local transcripts, all 26 had the same sequence of blocks in both. tool_use IDs appear in both copies too, and the session ID inside metadata.user_id names the transcript file. Attachments match on their rendered text, with the wrapper tags removed, against the system-role messages and user text between the same two turns. Typed prompts have no provider ID, so they join through the parent chain to the response that answered them.

For Codex, the gateway's resp_ response ID is the rollout's token_usage_record.response_id, and the item IDs are shared: 166 of the item IDs in my gateway sample appeared in a single local rollout. The session, thread and turn IDs in client_metadata match the rollout's session_meta and turn_context lines.

The join is a full outer join, so rows found in either source survive, and it only matches inside one owner, workspace and conversation. Where one source lacks a field, the other fills it in. Where they disagree, both values are kept side by side, and the original rows ride along in a carrier column so the join can be redone with better rules. Equal text and nearby timestamps never count as a match.

Keep the originals

All of the above should be rebuildable, which means the originals have to stay byte for byte. The gateway serializes each call once and appends the line to gzipped JSONL in object storage before anything parses it. Merging small objects into bigger ones concatenates lines, and later steps copy those bytes without re-encoding them, so a call's ID can be the BLAKE3 hash of its line and comes out the same on every replay.

Transcripts go into a content-addressed archive, collected from every machine and volume an agent ran on, with a manifest per collection run. Session files only grow, so most versions are a prefix of a later one. Across my agents that came to about 440,000 file versions and 336 GB, then 94,012 distinct contents and 250 GB after hashing whole files. Of that, 166 GB was exact prefixes of a longer copy of the same file, so storing the longest version plus a length and a hash for each earlier one brings it to about 84 GB.

Where it lives

The derived data lives in Lance files: one table per source, the message bodies, the joined occurrences, per-call metadata, and enrichment such as embeddings of each unique message body. Each part file carries sidecar indexes, and the last few days live in Postgres and get merged in on read. Embeddings and summaries are keyed by content hash plus model version, so a message resent a thousand times is embedded once, and the embeddings run under a $10 monthly budget.

The originals live in Hetzner Object Storage, in the same region as the cluster, and get copied to Backblaze B2 with rclone copy --immutable, which adds new objects and never overwrites or deletes one. The Lance tables stay in one region, since losing them costs a replay and nothing more. A backup counts once a restore from B2 has been checked against the primary's manifest. The originals are also the most sensitive copy, because a transcript keeps tool output exactly as printed, so access to them should be narrower than access to the screened tables built from them.

Where I'd start

Get the originals right on day one: both copies, byte-exact, backed up offsite. Then treat everything else as derived and versioned: the normalization, the who-said-it classification, the join and the summaries. All of those will change as the harnesses change their formats, and each change is a replay.

Related posts

• What I learned running a fleet of AI coding agents
• Injecting prompts, memories and reviews at an LLM gateway without breaking the prompt cache
• Taking protobuf out of a 1,000-crate Rust monorepo
• Indexing Lance files without the Dataset API