Add human feedback

Review traces and provide structured scores to begin the annotation process. You can efficiently evaluate large batches with keyboard navigation, or use the kanban layout to visualize review progress across backlog, pending, and complete states.

​

Configure review scores

Review scores let you collect structured feedback on spans and label dataset rows. Configure scores in Settings > Project > Human review. See Configure human review for details on score types and options.

​

Score traces and datasets

Go to the Review page and select the type of data to review:

• Log spans: production traces and debugging sessions
• Experiment spans: Evaluation results and test runs
• Dataset rows: Test cases and examples

Then select a row and set scores. You can also add comments and tags while reviewing. When finished reviewing, click Complete review and continue to move to the next item in the queue, or use the Next row and Previous row buttons.

​

Filter review data

Select Filter to focus on specific subsets for review. Use the Basic tab for point-and-click filtering, or switch to SQL to write precise queries. Filter by scores (e.g., scores.Preference > 0.75) to find highly-rated examples or investigate low-scoring patterns. See Filter and search logs for more details.

​

Create and edit scores inline

While reviewing, create new score types or edit existing configurations without navigating to settings:

• To create a new score, click + Human review score.
• To edit an existing score, select the edit icon next to the score name.

Changes apply immediately across your project.

​

Capture production feedback

In addition to internal reviews, capture feedback directly from production users. Production feedback helps you understand real-world performance and build datasets from actual user interactions. See Capture user feedback for implementation details and Build datasets from user feedback to learn how to turn feedback into evaluation datasets. You can also use dashboards to monitor user satisfaction trends and correlate automated scores with user feedback.

​

Change the display

​

Switch the view

Each project includes default views:

• Default view: Shows all records
• Awaiting review: Shows only records flagged for review but not yet started
• Assigned to me: Shows only records assigned to you for review
• Completed: Shows only records that have finished review

Use the View menu to switch between views.

• To set the current view as default, select Manage view > Set as your default view.
• To discard unsaved changes, select Reset.

​

Create a custom view

Custom views save your table configurations including filters, column order, column visibility, and display settings. This lets you quickly switch between different ways of reviewing your data. To create a custom view:

1. Apply the filters and display settings you want.
2. Select Save as in the toolbar.
3. Enter a view name.

Custom views are accessible and configurable by any member of the organization. Views update dynamically with new rows matching saved criteria.

​

Show and hide columns

Select Display > Columns and then:

• Show or hide columns to focus on relevant data
• Reorder columns by dragging them
• Pin important columns to the left

All column settings are automatically saved when you save a view.

​

Use kanban layout

The kanban layout organizes flagged spans into three columns based on their review status:

• Backlog: Spans flagged for review but not yet started
• Pending: Spans currently being reviewed
• Complete: Spans that have finished review

To use the kanban layout:

1. On the Review page, select Display > Layout > Kanban.
2. Drag cards between columns to update review status. Changes save automatically.
3. Click any card to open the full trace for detailed review.

Each card displays the span name, creation date, assignees, and a preview of the input and output.

​

Next steps

• Add labels and corrections to categorize and tag traces
• Build datasets from reviewed logs
• Capture user feedback from production
• Run evaluations with human-reviewed datasets