Skip to main content
Applies to:


Summary Goal: Configure pass/fail thresholds for scorers in Ruby using the Braintrust API. Features: Function endpoint, scorer metadata, pass threshold configuration.

​
Configuration Steps

​
Step 1: Create the scorer registration function

POST to the /v1/function endpoint with scorer metadata. 
def register_pass_threshold(project_id, scorer_name, pass_threshold)
  return unless project_id

  api_state = Braintrust::State.global
  return unless api_state&.logged_in

  payload = {
    project_id: project_id,
    name: scorer_name,
    slug: scorer_name.to_s.downcase.gsub(/[^a-z0-9]+/, "-"),
    function_type: "scorer",
    metadata: { "__pass_threshold" => pass_threshold }
  }

  uri = URI("#{api_state.api_url}/v1/function")
  request = Net::HTTP::Post.new(uri)
  request["Authorization"] = "Bearer #{api_state.api_key}"
  request["Content-Type"] = "application/json"
  request.body = JSON.dump(payload)

  response = Braintrust::Internal::Http.with_redirects(uri, request)
  if response.code.to_i < 400
    puts "[Scorer] Registered #{scorer_name} with pass threshold: #{pass_threshold}"
  end
rescue StandardError => e
  puts "[Warning] Failed to register pass threshold for #{scorer_name}: #{e.message}"
end

​
Step 2: Call the function with your scorer details

Pass the project ID, scorer name, and desired threshold value. 
register_pass_threshold("your_project_id", "accuracy", 0.8)

​
Step 3: Verify in the Braintrust UI

The pass threshold will appear in the eval results, marking scores as pass/fail based on the threshold.

​
Notes

  • The __pass_threshold feature is not directly supported in the Ruby SDK yet
  • This workaround uses the same API endpoint that Python and TypeScript SDKs use internally
  • The slug field must be lowercase alphanumeric with hyphens only
  • Registration errors won’t fail your eval; they’re logged as warnings