View production metrics
The Monitor page shows custom dashboards for tracking deployed prompts and functions. For details on creating custom charts, filtering data, selecting timeframes, and configuring dashboards, see Monitor with dashboards. Production-specific metrics include:- Request count: Volume of production traffic
- Latency: Response time (total duration, time to first token)
- Token count: Prompt tokens, completion tokens, and total usage
- Cost: Estimated spend based on model pricing
- Scores: Quality metrics from online scoring
- Tools: Tool call frequency and success rates
Set up alerts
Configure alerts to notify you when metrics exceed thresholds:- Navigate to Configuration > Automations
- Click + Alert
- Define your conditions using SQL queries
- Set notification channels (email, Slack, webhooks)
- Error rate exceeds 5% for 10 minutes
- Average latency above 2 seconds
- Daily cost exceeds budget threshold
- Score drops below 0.8
Track costs
Cost charts estimate spending based on model pricing. Costs are calculated from:- Token counts (prompt and completion)
- Model pricing rates
- Provider-specific pricing tiers
Cost estimates are approximate. Actual billing from providers may vary based on rate limits, batch discounts, and other factors.
Monitor quality
Online scoring automatically evaluates production requests. View score distributions and trends in the Monitor page:- Group by score name to compare different quality metrics
- Filter by low scores to find problematic requests
- Track score changes over time to detect quality regressions
Next steps
- Score online to automatically evaluate production requests
- Set up alerts to catch issues proactively
- View logs to investigate specific requests
- Use dashboards for detailed observability