Skip to main content
Applies to:
  • Plan -
  • Deployment -

​
Summary

Goal: Query scorer LLM spend for experiments or project logs, which are excluded from the built-in estimated_cost() metric. Features: SQL span queries, span_attributes.purpose, span_attributes.type, project_logs()

​
Configuration steps

​
Step 1: Understand the default cost exclusion

The Estimated cost column on the Experiments page only covers task LLM spans. Scorer spans are filtered out by design. To get scorer costs, you must query token counts directly and calculate costs using your model pricing.

​
Step 2: Query scorer token usage

Use the SQL sandbox or API to sum tokens by model for scorer spans. For experiments, an example query is: 
SELECT
  SUM(metrics.prompt_tokens) AS scorer_prompt_tokens,
  SUM(metrics.completion_tokens) AS scorer_completion_tokens,
  metadata.model
FROM experiment('<EXPERIMENT_ID>', shape => 'spans')
WHERE span_attributes.purpose = 'scorer'
  AND span_attributes.type = 'llm'
GROUP BY metadata.model
For project logs, replace experiment() with project_logs(): 
SELECT
  SUM(metrics.prompt_tokens) AS scorer_prompt_tokens,
  SUM(metrics.completion_tokens) AS scorer_completion_tokens,
  metadata.model
FROM project_logs('<PROJECT_ID>', shape => 'spans')
WHERE span_attributes.purpose = 'scorer'
  AND span_attributes.type = 'llm'
GROUP BY metadata.model
Calculate costs by multiplying token counts by your per-model pricing (e.g., prompt_tokens * (0.04 / 1000000) + completion_tokens * (0.05 / 1000000)).

​
Step 3: Surface costs in the Monitor dashboard (optional)

In the Monitor page, create a custom chart with a span filter on span_attributes.purpose = 'scorer' and span_attributes.type = 'llm'. Use sum(metrics.prompt_tokens) as the measure, or create a cost expression like: 
sum(metrics.prompt_tokens) * (0.04 / 1000000)