Devin

Export Devin sessions into HoneyHive for observability and evaluation

Monitor your Devin sessions in HoneyHive to evaluate agent behavior, review conversation quality, and track usage across runs. The Devin exporter syncs sessions as structured traces, giving you full visibility into what your coding agent did and how it performed.

Quick Start

1. Clone and install

   git clone https://github.com/honeyhiveai/honeyhive-daemon.git
   cd honeyhive-daemon/devin
   pip install -r requirements.txt
   
2. Set environment variables

   export DEVIN_API_KEY=<your Devin API key>
   export HH_API_KEY=<your HoneyHive API key>
   export HH_API_URL=<your HoneyHive data plane URL>
   

   Project scope is resolved from your API key - no HH_PROJECT needed.

   For v3 keys (cog_* prefix), also set:

   export DEVIN_ORG_ID=<your Devin organization ID>
   
3. Run the exporter

   One-shot sync:

   python devin_to_honeyhive.py
   

   Daemon mode (continuous polling):

   python devin_to_honeyhive.py --daemon --interval 60
   

After syncing, open Traces to inspect your Devin sessions.

---

What Gets Traced

Each Devin session becomes a trace tree in HoneyHive:

Event name	Type	Description
session title	session	Root event with session metadata, status, PR links, and chat history.
user_message	chain	User messages (includes initial_user_message for the first prompt).
devin_message	model	Agent responses from Devin.
{category}/{event_type}	tool/model	Internal processing events (see below).
session.end	chain	Session completion with a full conversation transcript artifact.

The root session event includes metadata for linking back to Devin (devin_url, devin_status, devin_pull_requests) and metrics.acus_consumed for tracking spend (ACUs are Devin’s Agent Compute Units, their billing unit).

Internal events

For v3 API keys (cog_*), the exporter also syncs Devin’s behind-the-scenes processing events:

Category	Type	Example event names
shell	tool	shell/shell_command, shell/shell_output
git	tool	git/git_commit, git/git_push
browser	tool	browser/browser_navigate, browser/browser_click
file	tool	file/file_edit, file/file_read
message	model	message/agent_message

These are the lower-level processing steps behind the conversation-level devin_message events above. Noisy internal events (checkpoints, activity updates, terminal updates) are filtered out automatically.

Re-running the exporter is safe - it tracks sync state on disk and only processes new data.

---

Troubleshooting

Symptom	Fix
Authentication error from Devin	Verify your key prefix: apk_* for v1, cog_* for v3. Set DEVIN_ORG_ID for v3.
401 from HoneyHive	Check HH_API_URL points to the correct data plane (e.g. https://api.dp1.us.prod.honeyhive.ai).
Duplicate or stuck sessions	Delete the state file (sync_state.json) to force a full re-sync.
Need more detail	Run with --verbose for per-session sync logs and API responses.

For full configuration options (environment variables, CLI flags, incremental sync details), see the source repo README.

---

Related

Source code

Devin exporter script and documentation

Claude Code Integration

Real-time Claude Code tracing via hooks

Tracing Quickstart

Get started with HoneyHive tracing

Query Your Data

Filter and analyze traces