Skip to main content
Contributed by Max Kern on 2026-08-19
This cookbook shows you how to collect traces from your team’s coding agents in a single Braintrust project and use a Loop automation to periodically analyze those traces, spot recurring problems, and surface Patterns for you to review. For example, you might find that agents repeatedly resend large amounts of context, increasing model usage, or that file operations fail because of stale paths or imprecise replacement text. You might also find agents repeatedly checking whether long-running commands have finished, or tracing issues that combine unrelated work or reprocess the same sessions. This guide describes how to:
  • Create a Braintrust project for your team’s coding-agent traces.
  • Configure Claude Code, Codex, OpenCode, or Pi to send sessions to the project.
  • Use the bt CLI to add three facets and a daily Loop automation to the project.
  • Collect new traces and import useful Codex or Claude Code sessions from earlier work.
  • Run the automation manually to verify that it can review the sessions, then review any recurring problems it saves as patterns.
  • Optionally refine the facets and automation or send automation reports to Slack.

​
Prerequisites

Before you begin, you’ll need:

​
1. Create a project for your traces

One person should create a dedicated Braintrust project for your team’s coding-agent sessions. This project will contain the traces, Topics facet values, Loop automation, and patterns you create in this cookbook.
  1. Go to your organization overview.
  2. Open the project dropdown and click + Create project.
  3. Enter coding-agent-insights as the project name.
  4. Optionally add a description, then click Create.
You should now see coding-agent-insights in the project dropdown. If the project already exists, use it instead of creating another one. Before continuing, make sure every teammate whose sessions you want to collect can access the project.

​
2. Configure your coding agents

Everyone whose sessions should appear in coding-agent-insights, including you, must complete this step on each computer where they use a supported coding agent.
1

Install bt

Install the bt CLI:
After installing, run bt --version. You should see bt 0.19.0 or later.
2

Authenticate with Braintrust

Run bt status --all to check your saved logins. If you already have a valid OAuth profile that can access the organization and project where you’ll save traces, continue to the next step below. Otherwise, sign in:
Complete the browser flow. The CLI creates or updates an OAuth profile so the tracing plugins can authenticate without putting an API key in your shell history.
To authenticate bt and send coding-agent traces to a self-hosted Braintrust deployment, you need to know which URLs it uses:
  • API URL: bt uses this URL to start and refresh OAuth. Coding-agent tracing uses it to send traces. Find it under Settings > Data plane, or ask your Braintrust administrator for it.
  • Braintrust app URL: bt uses this URL to find the organizations you can access. The default is https://www.braintrust.dev.
Before you sign in, set the API URL in the environment where you start your coding agents. Run export BRAINTRUST_API_URL="<YOUR_API_URL>" in the current shell. Add the export to your shell startup file or repeat it in each shell where you start a coding agent so its tracing process inherits the URL.Then sign in:
If your organization also uses a Braintrust app URL other than https://www.braintrust.dev, include that URL:
bt saves the URLs in your OAuth profile for authentication. Keep BRAINTRUST_API_URL set whenever you start a coding agent so its tracing process sends traces to your self-hosted API. Later commands include --prefer-profile to prevent an ambient BRAINTRUST_API_KEY from taking precedence over the profile.
3

Configure your coding agents

Run the commands for the coding agents you use on this computer:
After setup:
  • Restart Claude Code to load the tracing configuration.
  • Run bt trace doctor claude.
  • Confirm that Enabled is true, Auth is ready, and the displayed organization and destination project are correct.
  • If the organization or project is incorrect, rerun bt trace setup claude with the correct --org and --project values.
4

Verify tracing

Confirm that each configured agent sends sessions to the shared project:
  • Start a new session in each configured coding agent and ask it to complete a small task. Ask participating teammates to do the same.
  • Go to Logs, select coding-agent-insights, and confirm that each new session appears as a root trace.
If a trace does not appear, run the agent’s bt trace doctor command and resolve any authentication or routing problems it reports. For Codex, also run /hooks and confirm that the Braintrust hooks are trusted. If needed, review the tracing setup for Claude Code, Codex, OpenCode, or Pi.

​
3. Configure Topics and Loop

If Topics is not already enabled, an organization owner must enable it. After that, any teammate with project-level Read, Create, and Update permissions can use bt and the prebuilt JSON file to set up the automated analysis.
1

Enable Topics

Skip this step if Topics is already enabled for coding-agent-insights. Otherwise, ask an organization owner to:
  1. In coding-agent-insights, go to Topics.
  2. Make sure Task is selected. You can leave the other built-in facets off.
  3. Choose whether to Apply to existing traces.
  4. Click Enable topics.
2

Download the configuration file

The file defines the three Topics facets and daily Loop automation that you’ll add to your project in the next step.
3

Configure your project

Pass the JSON file to bt, which can use it to configure your coding-agent-insights project.
At the confirmation prompt, verify the organization, project, three facets, one Loop automation, and Topics wiring. Then confirm the push. The command should report that it successfully pushed all four resources.
4

Review the installed resources

Confirm that the template added the facets to the project and attached them to the Topics automation:
  1. In the coding-agent-insights project, go to Topics. Confirm that cards appear for Coding agent context debt, Primary failure mode, and Skills analysis.
  2. Click Automation and select Topics. Expand Active facets and confirm that it includes the same three custom facets.
Confirm that the template created an active daily automation with the expected analysis window and pattern permissions:
  1. Go to Settings > Automations. Confirm that Coding Agent Improvements Discovery shows Every 24 hours and Active.
  2. Click Coding Agent Improvements Discovery to open it. On the Edit tab, confirm that:
    • Agent configuration shows GPT-5.6 Sol with Extra high reasoning effort.
    • Default query range shows Custom, 30 days.
    • Write tool permissions shows 2 allowed. Open it and confirm that new_pattern and update_pattern are selected.
These checks confirm that Topics will label coding-agent sessions and Loop will review the most recent 30 days for recurring problems.

​
4. Collect coding-agent traces

Accumulate traces across tasks, teammates, and coding agents. Each session gives the automation more evidence to distinguish recurring problems from one-off difficulties. You can also import useful saved sessions from earlier Codex or Claude Code work. To do this, find the session ID using the instructions for your agent, then run the import command:
  1. Use claude --resume to reopen the session you want to import.
  2. Run /status and copy its session ID.
  3. Import the session:
bt trace import supports saved sessions from Codex and Claude Code. It does not support OpenCode or Pi, so collect those sessions through live tracing.
Once a session appears in Logs, the automation can query its trace on the next scheduled or manual run. It can also use any facet values that Topics has added by then.

​
5. Verify the automation

After collecting some traces, run Coding Agent Improvements Discovery manually to confirm that it can review them:
  1. Go to Settings > Automations and click Coding Agent Improvements Discovery to open it.
  2. Click Run now. You’ll see a Starting Loop automation… notification, followed by a Loop automation started notification.
  3. In the second notification, click Open Loop. Alternatively, open Past runs and click the newest Coding Agent Improvements Discovery run. Either route opens the Loop thread, where you can watch the run.
The automation is working if the run completes and reports which sessions it reviewed. It does not need to create or update a pattern: Loop only saves problems that it finds repeatedly.
After you verify the automation, it continues running daily. When it finds a recurring problem, it creates or updates a pattern. See Review and act on patterns to inspect the evidence and close the pattern after fixing the problem.

​
6. Customize the workflow (optional)

After verifying the automation, experiment with changes in an interactive Loop thread or send its reports to Slack or a webhook.
If you’d like to refine the facets or change what the scheduled automation analyzes, start in an interactive Loop thread. You can try different analyses against recent sessions before updating the project’s facets or scheduled automation.
An interactive Loop thread uses live project data and may create or update patterns while it tests an analysis. It does not change the facets or scheduled automation unless you approve a proposed change or edit the resource yourself.
Ask Loop to suggest changes. Go to Loop and enter this prompt:
Continue the thread to refine and test its suggestions. Leave proposed changes unapproved while you experiment.Try your own analysis. If you already know what you want to try, describe the revised analysis in a new Loop thread and ask Loop to run it against recent sessions. Adjust your instructions and rerun the analysis until you’re satisfied with the results.Apply a change. When you’re satisfied, either approve a change that Loop proposed or update the automation manually. To update the automation yourself, go to Settings > Automations and click Coding Agent Improvements Discovery to open it. Make the change and click Save. Approving or saving a change updates the existing facet or automation.
To send the result of each successful automation run to Slack or a webhook, add a destination. Go to Settings > Automations and click Coding Agent Improvements Discovery to open it.
1

Connect Slack

Go to Settings > Integrations and connect your Slack workspace.
2

Add the destination

Under Destinations, click + Destination and select Send to Slack. Then choose the workspace and channel.Braintrust prepopulates a suggested Formatting prompt. Review it and adjust it as needed for your team.
Test the destination. Save the automation and click Run now. When the run finishes, confirm that the Slack channel or webhook endpoint received the formatted report.Adding a destination does not change the automation’s daily schedule or 30-day lookback window.

​
Troubleshooting

If the curl command for downloading the configuration file returns an HTTP error:
  • Confirm that the URL is https://www.braintrust.dev/docs/assets/active-observability-config.json.
  • Retry from a network that can reach www.braintrust.dev.
If bt observability template is not recognized when you push the template:
  • Run bt --version to check the installed version.
  • Install or update to bt v0.19.0 or later. If the installed version already meets this requirement, reinstall it.
  • After the installation succeeds, return to Configure your project and run the template command shown there.
If bt observability template push reports that a matching facet or Loop automation already exists, it stops before the confirmation prompt:
  • If you intend to restore the template configuration, rerun the same command with --force and confirm the replacements.
  • A new project does not need --force.
Rerunning the command with --force resets the three facets and Coding Agent Improvements Discovery to the versions in the JSON file. Any edits you made to them are lost. Slack and webhook destinations already added to the automation are preserved.
If bt trace setup codex reports a marketplace clone timeout:
  • Run the command again.
  • If it repeatedly reports fatal: early EOF, troubleshoot the network connection rather than the Braintrust project configuration.
If a test session does not appear in Logs after you configure tracing:
  • Restart the coding agent and run another test task.
  • For a self-hosted deployment, run printenv BRAINTRUST_API_URL in the shell where you start the coding agent. If it is empty or incorrect, export the API URL shown under Settings > Data plane, restart the agent from that shell, and run another test task.
  • If the trace still does not appear, run bt trace doctor <AGENT>, replacing <AGENT> with claude, codex, opencode, or pi.
  • Confirm that Enabled is true, Auth is ready, and the displayed organization and destination project are correct.
  • If Auth is not ready, run bt login --oauth, then run the doctor command again.
  • If the organization or destination project is wrong, rerun bt trace setup for that agent with the correct --org and --project values.
If bt trace import cannot find a Codex or Claude Code session:
  • Confirm that the command names the agent that created the session. Use bt trace import codex for Codex or bt trace import claude for Claude Code.
  • Run the command from the same user account and computer where that agent saved the session.
If Send to Slack does not appear in the Destination menu:
If Coding Agent Improvements Discovery fails because it cannot call gpt-5.6-sol:
  • Configure gpt-5.6-sol under AI providers, or select an available model in the automation editor.
  • Run the automation again.

​
Cleanup

Choose what to keep or remove:
  • Delete the downloaded configuration file: After pushing the template, run rm /tmp/active-observability-config.json.
  • Keep collecting traces in coding-agent-insights: No action is required.
  • Send future traces to another project: Rerun the appropriate bt trace setup command for each configured coding agent with the new --org and --project values.
  • Stop collecting coding-agent traces: Run bt trace disable <AGENT> for each configured agent, replacing <AGENT> with claude, codex, opencode, or pi. This removes the Braintrust tracing integration and its saved configuration.
  • Delete coding-agent-insights: First stop tracing or send future traces to another project. Then follow the instructions below if you no longer need the project’s data.
    Deleting the project permanently removes its traces, facets, automations, and other project data. This action cannot be undone.
    To delete the project:
    1. Go to Settings > General.
    2. Click Delete project.
    3. Enter coding-agent-insights and click Delete.

​
Recap

Your team now has a shared view of its coding-agent work in Braintrust. Three facets classify each session for common problems, and a daily Loop automation reviews the past 30 days to identify problems that recur across sessions. When it finds enough evidence, it creates or updates a pattern that your team can investigate, address, and monitor over time. As your needs change, you can refine what the facets and automation look for and send run reports to Slack or a webhook.

​
Next steps