Roast

​

Setup

gem "roast-ai"
gem "braintrust"

bundle install

BRAINTRUST_API_KEY=your-api-key
BRAINTRUST_DEFAULT_PROJECT=your-project-name  # Project that spans are logged to

# API key for the provider your chat cogs use (OpenAI by default)
OPENAI_API_KEY=your-openai-key

​

Auto-instrumentation

config do
  chat(:answer) { no_show_stats! }
end

execute do
  ruby(:question) { "What is the capital of France? Answer in three words or fewer." }
  chat(:answer) { ruby!(:question).value }
end

bundle exec braintrust exec -- roast execute workflow.rb

require "braintrust/setup"
require "roast"

Roast::Workflow.from_file("workflow.rb", Roast::WorkflowParams.new([], [], {}))

​

Manual instrumentation

require "braintrust"
require "roast"

Braintrust.init(default_project: "My Project", auto_instrument: false)

Braintrust.instrument!(:roast)
Braintrust.instrument!(:ruby_llm)

Roast::Workflow.from_file("workflow.rb", Roast::WorkflowParams.new([], [], {}))

​

What Braintrust traces

• Workflow spans (roast.workflow), with the workflow file name in metadata (contrib.roast.workflow.name), plus the workflow’s targets, positional arguments, and keyword arguments as input when you pass them.
• Cog spans (roast.cog.<name>), nested under the workflow span, with the cog’s type, name, and outcome (completed, failed, or skipped) in metadata. Cogs in a scope that a call cog runs nest under that cog’s span.
• Cog output for ruby cogs that return a string, and the model response for chat cogs.
• LLM call spans (ruby_llm.chat) for chat cogs, nested under the cog span, with the messages, model, response, and token usage that the RubyLLM integration records.
• Errors, recorded on the cog span when a cog fails.

​

Resources

• Roast on GitHub
• Roast tracing example
• Braintrust Ruby SDK