User Feedback

Learn how to track user feedback on your traces

Introduction

User feedback is a crucial part of understanding user experience and intent. It helps you identify areas for improvement and make informed decisions to enhance your application’s performance and user satisfaction.

Prerequisites

You have already set tracing for your code as described in our quickstart guide.

Setting

You can set on both the trace level or the span level. If the applies to the entire trace, then set it on the trace level. If the applies to a specific span, then set it on the span level. For more details, refer to the enrich traces documentation.

Python

Setting Feedback on Trace Level

In Python, you can use the enrich_session function to set on the trace level.

To pass to HoneyHive, pass it to the param in the enrich_session function. This function is used to enrich the session with additional information. Remember that enrich_session will update, not overwrite, the existing object on the trace.

Read more about the enrich_session function in the Python SDK reference.

Here’s an example of how to set on the trace level in Python:

Python

from honeyhive import HoneyHiveTracer, enrich_session

HoneyHiveTracer.init(
  api_key="my-api-key",
  project="my-project",
)

# ...

enrich_session(feedback={
  "rating": True,
  "comment": "The model hallucinated the capital of New York",
  # optionally adding reference ground truth
  "ground_truth": "The capital of New York is Albany",
  # optionally adding any arbitrary fields as you need
})

Setting Feedback on Span Level

In Python, you can use the enrich_span function to set on the span level.

To pass to HoneyHive, pass it to the param in the enrich_span function. This function is used to enrich the span with additional information. Remember that enrich_span will update, not overwrite, the existing object linked to the span.

Read more about the enrich_span function in the Python SDK reference.

Here’s an example of how to set on the span level in Python:

Python

from honeyhive import HoneyHiveTracer, trace, enrich_span

HoneyHiveTracer.init(
  api_key="my-api-key",
  project="my-project",
)

# ...

@trace
def my_function(input, user_id):
    # ...

    enrich_span(feedback={
        "rating": True,
        "comment": "The model hallucinated the capital of New York",
        "user_id": user_id,
        # optionally adding reference ground truth
        "ground_truth": "The capital of New York is Albany",
        # optionally adding any arbitrary fields as you need
    })

    # ...

    return response

# ...

When collecting user feedback in applications with separate frontend and backend components, you must propagate the sessionId to ensure feedback is associated with the correct trace:

1. In your frontend service, obtain the sessionId using HoneyHiveTracer.session_id
2. Pass this sessionId to your backend service when submitting feedback (via headers or request body)
3. In your backend service, use HoneyHiveTracer.init(session_id = <session-id>) with the received sessionId
4. Then call enrich_session() or enrich_span() with your feedback data

This ensures that user feedback is correctly associated with the original trace. For more details, see the distributed tracing docs.

TypeScript

Setting Feedback on Trace Level

In TypeScript, you can use the tracer.enrichSession function to set on the trace level.

To pass to HoneyHive, pass it to the param in the tracer.enrichSession function. This function is used to enrich the session with additional information. Remember that tracer.enrichSession will update, not overwrite, the existing object linked to the trace.

Read more about the tracer.enrichSession function in the TypeScript SDK reference.

Here’s an example of how to set on the trace level in TypeScript:

TypeScript

import { HoneyHiveTracer, enrichSession } from "honeyhive";

// Initialize tracer 
// Ensure HH_API_KEY and HH_PROJECT are set in your environment
const tracer = await HoneyHiveTracer.init({
    sessionName: "setting-feedback-session"
    // apiKey and project will be picked from environment variables
});

// Wrap the execution logic in tracer.trace()
await tracer.trace(async () => {
    // Add feedback to the entire trace session
    enrichSession({
      feedback: {
        "liked": true,
        "comment": "This was a great experience!",
        // any other custom fields and values as you need
      }
    });
    
    // ... rest of your application logic ...
    console.log("Trace session enriched with feedback.");
});

// await tracer.flush(); // If the script exits immediately

Setting Feedback on Span Level

In TypeScript, you can use the tracer.enrichSpan function to set on the span level.

To pass to HoneyHive, pass it to the param in the tracer.enrichSpan function. This function is used to enrich the span with additional information. Remember that tracer.enrichSpan will update, not overwrite, the existing object linked to the span.

Read more about the tracer.enrichSpan function in the TypeScript SDK reference.

Here’s an example of how to set on the span level in TypeScript:

TypeScript

import { HoneyHiveTracer, traceTool, enrichSpan } from "honeyhive";

// Initialize tracer 
// Ensure HH_API_KEY and HH_PROJECT are set in your environment
const tracer = await HoneyHiveTracer.init({
    sessionName: "setting-feedback-session"
s            // apiKey and project will be picked from environment variables
});

// Define the traced function using traceTool
const myTracedFunction = traceTool(function my_llm_call( // Function name is used as span name
    param1: string, 
    param2: string, 
    user_id: string | number
) {

    // Add feedback specific to this span
    enrichSpan({
      feedback: {
        "liked": true,
        "comment": "The model hallucinated the capital of New York",
        "user_id": user_id,
        // optionally adding reference ground truth
        "ground_truth": "The capital of New York is Albany",
        // any other custom fields and values as you need
      }
    });

    // Your function code here
    const response = `Result with ${param1} and ${param2}`;
    return response;
});

// --- Main Execution Logic ---
// Wrap the execution in tracer.trace() to establish context
await tracer.trace(async () => {
    const userId = "user-abc-123";
    // Execute the traced function within the trace context
    myTracedFunction("input1", "input2", userId);
});

// await tracer.flush(); // If the script exits immediately

Previously, tracing and enrichment involved calling methods directly on the tracer instance (e.g., tracer.traceFunction(), tracer.enrichSpan(), tracer.enrichSession()). While this pattern still works, it is now deprecated and will be removed in a future major version.

Please update your code to use the imported functions (traceTool, enrichSpan, enrichSession) along with the tracer.trace() wrapper as shown in the examples above. This new approach simplifies usage within nested functions by not requiring the tracer instance to be passed around.

Example of the deprecated pattern:

// OLD (DEPRECATED) PATTERN:
// const tracer = await HoneyHiveTracer.init({...});
// tracer.enrichSession({ feedback: {...} }); 
// const myFunc = tracer.traceFunction()(function(...) { ... });
// tracer.enrichSpan({ feedback: {...} }); 

When collecting user feedback in applications with separate frontend and backend components, you must propagate the sessionId to ensure feedback is associated with the correct trace:

1. In your frontend service, obtain the sessionId using tracer.sessionId
2. Pass this sessionId to your backend service when submitting feedback (via headers or request body)
3. In your backend service, use HoneyHiveTracer.init(sessionId=<session-id>) (TypeScript) with the received sessionId
4. Then call enrichSession() or enrichSpan() with your feedback data

This ensures that user feedback is correctly associated with the original trace. For more details, see the distributed tracing docs.

Concepts

What is User Feedback?

Any feedback that the end-user provides about their experience is considered user feedback. This can be of two types:

1. Explicit Feedback: Any feedback provided by the user explicitly in your app. Examples include 👍/👎, ratings (1-n), etc.
2. Implicit Feedback: Any actions within your app’s UI by your user that may indicate user experience and intent. Examples include clicking Copy, Regenerate, etc.

User Feedback vs Human Evaluator: User feedback field is used to track feedback provided by your end-users. Feedback from domain experts or other internal team members is considered a human evaluator in HoneyHive and is tracked on the metrics field.

Return Types

We accept all primary return types as user feedback. This includes:

Return Type	Available Measurements	Notes
Boolean	True/False
Number	Percentage, Sum, Avg, Median, Min, Max, P95, P98, P99
String	Any string value	Used for filters and group by

For complex data types, we allow you to drill down to the nested fields or specific positions in the array.

So, for example, if you pass feedback like:

{
  "rating": 5,
  "comment": "Great experience",
  "step1": {
    "retry": true
  },
  "edits": [
    {
      "value": "New York"
    }
  ]
}

You can chart feedback.step1.retry as a boolean field or feedback.edits.0.value as a string field.

Nesting Limitations: For complex data types like objects and arrays, we allow max 5 levels of nested objects and max 2 levels of nested arrays.

Reserved Fields

The following fields are reserved for user feedback:

ground_truth - The ground truth value for the output of a trace or span. This is used to compare the actual output with the expected output. It is rendered differently in the UI.

Learn More

Analyzing user feedback

Learn how to chart custom user feedback fields

SDK Reference

Read more about the enrich_session function in the Python SDK reference.