Discover insights with Topics

• Surface user intents: Discover what users are trying to accomplish.
• Identify friction patterns: Find where users struggle, get confused, or encounter errors.
• Track sentiment trends: Track user satisfaction and frustration across interactions.
• Analyze performance: Understand how your AI responds, where it fails, and what issues recur.

​

How it works

• The Task topic map analyzes user intents or goals.
• The Sentiment topic map analyzes emotional tone.
• The Issues topic map identifies problems with agent behavior or responses.

​

Create your first topic maps

1. In your project, go to Topics.
2. Select which built-in topic maps to create:
   • Task: Categorize user intents and goals
   • Sentiment: Classify emotional tone (POSITIVE, NEGATIVE, NEUTRAL)
   • Issues: Identify agent problems and errors
   You can create all three or choose specific ones based on your needs.
3. (Optional) Expand Settings to customize preprocessing, sampling rate, or idle timeout.
4. (Optional) Use Test on a sample log to verify extraction quality before enabling.
5. Click Create topic maps.

​

Generate topics

1. Select a topic map from the sidebar. The UI shows how many traces from the last 3 days have been summarized for the topic map.
2. (Optional) Click to customize the clustering algorithm, clustering size, and other advanced settings. Most users can use the defaults.
3. Click Generate topics. Braintrust clusters similar summaries and displays the results with auto-generated topic names and descriptions.
4. Review the results to ensure they accurately reflect your data patterns.
   • Open the topics visualization to see how summaries are grouped.
   • Review sample summaries in each topic to verify they’re semantically similar.
   • Look for topics with unclear or overlapping names.
   • Check how many traces don’t match any topic (shown in the “Noise/Unclustered” section).
   If the generated topics don’t meet your needs, see Refine your topics.
   
5. Click Save topics > Save.
   • To automatically classify new traces with these topics, enable Classify incoming traces with these topics before saving.
   • To apply classifications to existing traces, enable Process existing traces.
   Classifications only appear on traces when the corresponding toggle is enabled before saving.

​

Refine your topics

• Regenerate with different settings
  • Expand Settings under Configuration and adjust the time range or sample rate for processing existing traces.
  • Click Generate > Advanced clustering options and adjust the minimum cluster size (increase for broader topics, decrease for more granular topics), try different clustering algorithms, or change other advanced options.
  Click Generate topics again to see new results. Each generation is saved as a version, so you can experiment freely.
• Create a custom topic map Create a custom topic map when the built-in ones (Task, Sentiment, Issues) don’t capture the patterns you need. See Create custom topic maps for more details.

​

Use your topics

​

Visualize topic distributions

1. Go to Logs.
2. Select Display > Layout > Topics.

​

Build datasets from topics

1. Go to Logs and click Filter.
2. Select Classifications and choose the classification you want to filter by. Alternately, click SQL and enter a SQL filter clause. See the SQL reference for more query patterns.
   Filter by specific topic

   Filter by multiple topics

   Combine topics across topic maps

   Report incorrect code

   Copy

   Ask AI

   classifications.Task.label = "Dataset creation"
   
3. Review the filtered logs to ensure they match your criteria.
4. Select the logs you want to include.
5. Click + Dataset.
6. Choose an existing dataset or create a new one.

• Create a dataset of “Error Investigation” tasks to test your error handling
• Build a dataset of negative sentiment interactions to improve responses
• Extract “Pricing Questions” to evaluate your pricing explanations

​

Score logs based on topics

• Flag all logs with negative sentiment for review
• Score logs with specific issue types lower
• Trigger alerts when certain topics appear together

import braintrust from "braintrust";
import { z } from "zod";

const project = braintrust.projects.create({ name: "my-project" });

project.scorers.create({
  name: "Checkout experience",
  slug: "checkout-experience",
  description: "Flag traces with negative checkout experiences",
  parameters: z.object({
    trace: z.any(),
  }),
  handler: async ({ trace }) => {
    if (!trace) return { score: null };

    // Get all spans from the trace
    const spans = await trace.getSpans();

    // Find the root span which contains classifications
    const rootSpan = spans.find(
      (s) => s.span_id === s.root_span_id
    );

    if (!rootSpan) return { score: null };

    // Access classifications from the root span
    const classifications = rootSpan.classifications || {};
    const taskClassification = (classifications.Task || [{}])[0];
    const sentimentClassification = (classifications.Sentiment || [{}])[0];

    // Check if this is a negative checkout experience
    if (
      taskClassification.label === "Checkout Flow" &&
      sentimentClassification.label === "NEGATIVE"
    ) {
      return {
        score: 0,
        metadata: { reason: "Negative sentiment during checkout" },
      };
    }

    return { score: 1 };
  },
});

1. Save the code to a file (topic_scorer.ts or topic_scorer.py).
2. Push it to Braintrust:
   TypeScript

   Python

   Report incorrect code

   Copy

   Ask AI

   npx braintrust push topic_scorer.ts
   
3. Go to Settings > Project > Automations and click + Rule.
4. Select your “Checkout experience” scorer.
5. Set Scope to Trace (required for accessing classifications).
6. Configure your sampling rate and optional SQL filter
7. Click Create rule.

​

Assign topics for review

1. Go to Logs and click Filter.
2. Select Classifications and choose the classification you want to filter by. Alternately, click SQL and enter a SQL filter clause. See the SQL reference for more query patterns.
   Filter by specific topic

   Filter by multiple topics

   Combine topics across topic maps

   Report incorrect code

   Copy

   Ask AI

   classifications.Task.label = "Dataset creation"
   
3. Review the filtered logs to ensure they match your criteria.
4. Select the logs you want to include.
5. Select Assign.
6. Choose a member to assign.

​

Version topic maps

• To automatically classify new traces with this new version, enable Classify incoming traces with these topics before saving.
• To classify existing traces with this new version, enable Process existing traces. If traces had classifications from previous version, the classifications will be overwritten.

​

Create custom topic maps

• Domain-specific categories: Your logs have patterns that built-in topic maps don’t capture.
• Too many uncategorized traces: The built-in topic maps aren’t extracting relevant summaries.
• Wrong level of detail: You need more specific categorization (e.g., distinguish between different API endpoints instead of just “API request”).
• Business-specific needs: Track patterns unique to your product (e.g., “Feature requests,” “Pricing questions,” “Integration issues”).

1. Go to Topics and click + Topic map.
2. Give your topic map a name and description. Example:
   • “Churn risk”
   • “Topic map for assessing customer churn risk based on conversations”
3. Select Facet > + Custom facet.
4. Enter a descriptive name for the facet.
5. Choose a preprocessor to transform your trace data.
   • Select Preprocessor > thread (default) to format traces as conversation threads.
   • Or select + Custom preprocessor to write a JavaScript function that filters or transforms your data. Common patterns:
     Extract user messages

     Format with metadata

     Filter to errors only

     Report incorrect code

     Copy

     Ask AI

     // Extract only user messages from a conversation
     function userMessagesOnly(span) {
       const messages = span.input?.messages || [];
       const userMessages = messages
         .filter(m => m.role === 'user')
         .map(m => m.content)
         .join('\n\n');
     
       return userMessages || '(No user messages found)';
     }
     
6. Select an AI model for summary extraction.
7. Enter a prompt with clear instructions for what to extract. Example:
   Report incorrect code

   Copy

   Ask AI

   Based on this conversation, assess the churn risk for this customer.
   
   Consider:
   
   - Frustration level and language used (complaints, strong negative words)
   - Whether their issue was resolved satisfactorily
   - Mentions of competitors, alternatives, or cancellation
   - Overall satisfaction signals (thanks, happy, vs angry, disappointed)
   - Severity and recurrence of issues
   
   Classify as:
   - LOW RISK: Satisfied customer, issue resolved, positive interaction
   - MEDIUM RISK: Some frustration but issue handled, no major red flags
   - HIGH RISK: Frustrated customer, unresolved issues, or mentions of dissatisfaction
   - CRITICAL: Explicitly mentioned canceling, switching to competitor, or very angry
   
   Respond with the label followed by a colon and the key risk indicators (one sentence).
   
   Examples:
   - "LOW_RISK: User thanked the agent and confirmed their billing question was answered."
   - "HIGH_RISK: User expressed frustration about repeated API errors and said this is unacceptable."
   - "CRITICAL: User stated they are considering switching to a competitor if issues persist."
   
8. Click Test to verify extraction quality on sample traces.
9. Click Create.
10. Click Create topic map to save your custom topic map.

​

Troubleshooting

​

Next steps

• View classifications in your logs table and trace details
• Query with SQL for advanced analysis
• Monitor trends with dashboards and charts
• Create evaluation datasets from specific topics
• Set up online scoring based on classifications