Finding production issues using Topics

git clone https://github.com/braintrustdata/braintrust-cookbook.git
cd braintrust-cookbook/examples/FindingProductionIssues/customer-support-bot
pip install -r requirements.txt

export OPENAI_API_KEY="your-openai-api-key"
export BRAINTRUST_API_KEY="your-braintrust-api-key"
export SUPABASE_SERVICE_ROLE_KEY="your-supabase-key"

def lookup_order(order_id: str) -> dict | None:
    resp = requests.get(
        f"{SUPABASE_URL}/rest/v1/orders",
        headers={**_REST_HEADERS, "Accept": "application/vnd.pgrst.object+json"},
        params={"id": f"eq.{order_id}", "select": "*"},
        timeout=10,
    )
    if resp.status_code == 406 or resp.status_code == 404:
        return None
    resp.raise_for_status()
    return resp.json()

def _call_edge_function(function_name: str, payload: dict) -> dict:
    url = f"{SUPABASE_URL}/functions/v1/{function_name}"
    resp = requests.post(url, json=payload, timeout=15)
    return {"status_code": resp.status_code, **resp.json()}

You are a helpful customer support agent for an e-commerce company
called Evergreen Goods.

POLICIES:
- Return window: 30 days from delivery
- Refund processing: 5-7 business days
- Exchanges: Same item, different size/color. Subject to availability.
- Damaged items: Full refund or replacement. No return shipping required.
- Shipping: Standard (free over $50), Express ($12.99-$14.99), Overnight ($24.99)

INSTRUCTIONS:
- Always use the provided tools to look up real data before answering.
- Never fabricate order numbers, tracking numbers, prices, or dates.
- If a customer provides an order number or email, look it up first.
- When a customer wants a refund, USE the process_refund tool.
- When a customer wants to return an item, USE the initiate_return tool.
- If you cannot resolve an issue, USE escalate_to_human.
- Be empathetic but efficient. Take action when you can.

SCENARIOS = [
    # Shipping
    ("Where is my order ORD-4002? It's been 5 days and still no delivery.",
     ["Can you give me a more specific ETA?"], "shipping"),

    # Returns
    ("I need to return ORD-4007. I ordered a size S but I actually need a M.",
     ["Can I do an exchange instead of a refund?"], "returns"),

    # Billing
    ("I applied a promo code on ORD-4005 but wasn't given the discount.",
     ["The code was SUMMER20."], "billing"),

    # Account
    ("My email is sarah.chen@example.com. Can you check my loyalty points balance?",
     ["When do my Gold perks expire?"], "account"),

    # Product
    ("Does the Waterproof Shell Jacket (SKU-1004) come in size XS?",
     [], "product"),

    # Orders not in system
    ("Where is my order ORD-9901? I placed it last week.",
     ["Can you check by my email? sarah.chen@example.com"], "shipping"),
    # ... 51 total scenarios
]

python generate_logs.py  # Run this several times

{
  "input": [
    {"role": "system", "content": "You are a helpful customer support agent..."},
    {"role": "user", "content": "I paid for express shipping on ORD-4006 and it's been 8 days."},
    {"role": "assistant", "content": "", "tool_calls": [{"function": {"name": "lookup_order"}}]},
    {"role": "tool", "content": "{\"status\": \"shipped\", ...}"},
    {"role": "assistant", "content": "I see your order is still in transit..."},
    {"role": "user", "content": "I want a refund on the shipping cost at minimum."},
    {"role": "assistant", "content": "Unfortunately I cannot process a refund..."}
  ],
  "expected": "Unfortunately I cannot process a refund...",
  "metadata": {"category": "shipping", "total_turns": 2}
}

{
  "input": "I paid for express shipping on ORD-4006 and it's been 8 days.\n---\nI want a refund on the shipping cost at minimum.",
  "metadata": {
    "num_customer_messages": 2,
    "num_turns": 5
  }
}

Eval(
    "Customer Support Chatbot",
    data=load_dataset,
    task=lambda input, expected=None, **_: run_support_conversation(
        input["customer_input"],
        num_customer_messages=input["num_customer_messages"],
    ),
    scores=[helpfulness_scorer, resolution_scorer, empathy_scorer],
    experiment_name="product-sizing-returns-v1",
)

  POLICIES:
+ - Shipping refunds: If a shipment is delayed beyond the expected
+   delivery window, process a refund for the shipping cost using
+   process_refund. A "shipped" status does NOT block shipping refunds.
+ - Promo codes: If a customer reports a missing promo discount, process
+   a refund for the discount amount. Do NOT check return eligibility or
+   tell them to wait for delivery — it's a billing correction, not a return.

  INSTRUCTIONS:
+ - If an order number is not found, try to help: look up the customer's
+   account by email to find valid orders, or ask clarifying questions.
  - When a customer wants a refund, USE the process_refund tool.
+ - Do NOT just escalate — process it yourself.
+ - If a tool call fails with a server error (5xx), retry once before
+   giving up.
+ - When you process a return and refund together, confirm both actions
+   in one response. Do not tell the customer they can't return because
+   a refund is in progress.
+ - ONLY use escalate_to_human as a LAST RESORT after all relevant
+   tools have failed.