TypeSafe

TypeSafe provides Jev, a model for structured decisions such as routing a support request, assessing urgency, or checking whether an answer meets a requirement. Trace calls from your application to inspect the questions, answers, and probability distributions behind those decisions. For example, when a support ticket reaches the wrong team, inspect the routing decision to see whether the categories were ambiguous or Jev selected an incorrect answer with high confidence. Tracing records the decision as a question span. It does not automatically create an evaluation score or a Braintrust classification column.

Tracing

Setup

Requires Braintrust v3.34.0+ and @typesafe-ai/sdk v0.6.0 or later within v0.x. You need a TypeSafe account and API key.

Install packages

pnpm add braintrust@^3.34.0 @typesafe-ai/sdk@^0.6.0

Set environment variables

Set your Braintrust API key and TypeSafe API key in your shell:

export BRAINTRUST_API_KEY="<your-braintrust-api-key>"
export TYPESAFE_API_KEY="<your-typesafe-api-key>"

For the EU data plane, also set BRAINTRUST_API_URL to https://api-eu.braintrust.dev. For a self-hosted deployment, use your data plane URL.

Auto-instrumentation

Use Braintrust’s import hook to trace TypeSafe calls throughout your application.

Initialize Braintrust and call Jev

Save this example as trace-typesafe-auto.js. It asks Jev to route a support request, assess its urgency, and identify whether it mentions a duplicate charge.

import { initLogger } from "braintrust";
import { TypeSafeClient, choice, noul, score } from "@typesafe-ai/sdk";

const logger = initLogger({ projectName: "typesafe-example" });
const client = new TypeSafeClient();

const response = await client.systemOne({
  state: "I was charged twice. Please refund the duplicate charge today.",
  questions: {
    category: choice("Which team should handle this request?", {
      billing: "Payments and refunds",
      technical: "Software problems",
      other: "Other requests",
    }),
    urgency: score("How urgent is this request?", [
      "routine",
      "soon",
      "urgent",
    ]),
    duplicate_charge: noul("Does the customer report a duplicate charge?"),
  },
});

console.log(response.answers);
await logger.flush();

Run with the import hook

node --import braintrust/hook.mjs trace-typesafe-auto.js

The auto-instrumentation example uses plain JavaScript so node --import can run the file directly. The Braintrust APIs work the same in TypeScript projects — compile your TypeScript to JavaScript, then run the compiled file with the import hook.

If you’re using a bundler, see Trace LLM calls for plugin and loader setup.

Go to your project’s Logs and select the typesafe.systemOne span to inspect the decisions.

Manual instrumentation

Wrap individual clients with wrapTypeSafe() to choose which TypeSafe calls to trace.

import { initLogger, wrapTypeSafe } from "braintrust";
import { TypeSafeClient, choice } from "@typesafe-ai/sdk";

const logger = initLogger({ projectName: "typesafe-example" });
const client = wrapTypeSafe(new TypeSafeClient());

const response = await client.systemOne({
  state: "I was charged twice. Please refund the duplicate charge today.",
  questions: {
    category: choice("Which team should handle this request?", {
      billing: "Payments and refunds",
      technical: "Software problems",
      other: "Other requests",
    }),
  },
});

console.log(response.answers.category.choice);
await logger.flush();

Manual instrumentation (AI SDK)

If your application calls Jev through AI SDK’s experimental_evaluate(), wrap the ai module with wrapAISDK(). This requires AI SDK v7.0.103 or later within v7. In your existing AI SDK application, replace the direct import of experimental_evaluate() with the wrapped export:

import * as ai from "ai";
import { initLogger, wrapAISDK } from "braintrust";

initLogger({ projectName: "typesafe-example" });
const { experimental_evaluate } = wrapAISDK(ai);

Call the wrapped experimental_evaluate() with your existing evaluation model, state, and questions. Braintrust records an evaluate span with type question. The import hook also instruments this function. For evaluation calls, use the wrapper or import hook rather than relying only on AI SDK telemetry callbacks.

What Braintrust traces

For each TypeSafeClient.systemOne() call, Braintrust records a typesafe.systemOne span with type question:

Input state and questions, including question identifiers, instructions, and criteria
Structured answers, including choices, scores, Noul values, and returned confidence and probabilities
Model and provider metadata
Token usage reported by TypeSafe and request duration
Errors raised by the call

Tracing resources

Inspect Jev decisions

Select a question span in a trace and use the Pretty view for its input and output. The input shows the state sent to Jev and each question’s instructions and criteria. The output matches answers to questions by identifier, so you can inspect the decision alongside the question that produced it. The SDK integration records questions and answers as lists. Each entry’s id comes from its key in your request, such as category or urgency in the examples above. Use these identifiers to follow a question from its criteria to its answer.

Interpret the answer

Answer type	What to inspect
Noul	Probability of yes, displayed as a percentage on a scale from no to yes.
Choice	The selected option, per-option probabilities, and confidence when returned.
Score	The numeric result on the rubric’s scale, level descriptions, per-level probabilities, and confidence when returned.

For the support-request example, inspect category to see the selected team and alternatives, urgency to see the result on the routine to urgent scale, and duplicate_charge to see the probability that the request describes a duplicate charge. Model responses can vary.

Understand confidence

Per-option probabilities show how Jev distributes probability across the possible answers. TypeSafe confidence summarizes how concentrated that distribution is. It is separate from the selected option’s probability and from an evaluation score. High confidence does not guarantee a correct decision. A Jev Score answer uses its own rubric scale: three levels correspond to a scale from 0 to 2, not a normalized Braintrust score.

AI providers

SDKs

Developer tools

Runtime environments

Tracing

Setup

Auto-instrumentation

Manual instrumentation

Manual instrumentation (AI SDK)

What Braintrust traces

Tracing resources

Inspect Jev decisions

Interpret the answer

Understand confidence

​Tracing

​Setup

​Auto-instrumentation

​Manual instrumentation

​Manual instrumentation (AI SDK)

​What Braintrust traces

​Tracing resources

​Inspect Jev decisions

​Interpret the answer

​Understand confidence

Tracing

Setup

Auto-instrumentation

Manual instrumentation

Manual instrumentation (AI SDK)

What Braintrust traces

Tracing resources

Inspect Jev decisions

Interpret the answer

Understand confidence