Applies to:
Summary
Issue: When an online scorer and a human review column share the same name, the automated scorer overwrites human review data with its own scores. Cause: Scorers and human review columns share the same namespace when writing to scores, causing name collisions. Resolution: Use distinct names for scorers and human review columns to prevent overwriting.Resolution Steps
Immediate fix
Step 1: Rename the conflicting score
Change either the scorer name or the human review column name to make them distinct.Step 2: Verify the change
Check that human review data is no longer being overwritten by the automated scorer.Prevent future conflicts
Step 1: Use naming conventions
Adopt prefixes to distinguish score types (e.g.,auto_accuracy for automated scorers, human_quality for reviews).
Step 2: Document naming standards
Create team guidelines for scorer and human review naming to avoid namespace collisions.Technical Details
Why this happens
Both automated scorers and human review columns write to the same score storage namespace. When names match, the scorer output overwrites human review data because the system uses the score name as the storage key without distinguishing between automated and manual sources.Affected scenarios
- Online scoring rules that write to scores
- Human review columns configured with the same name as an active scorer
- Production logs where both automated and human scoring are enabled
Detection difficulty
This issue provides no warning or error message. Teams often assume someone manually scored the data, making the root cause difficult to diagnose without comparing scorer names to human review column names.Best Practices
Naming conventions
- Automated scorers: Use prefixes like
auto_,llm_, orscorer_ - Human reviews: Use prefixes like
human_,review_, ormanual_ - Specific criteria: Include the evaluation criterion in the name (e.g.,
auto_factuality,human_helpfulness)