Best AI agent frameworks (2026): How to choose one and add evals

How to choose an AI agent framework in 2026

Choose the framework based on the agent's workflow shape, and keep the evaluation layer portable across frameworks.

Choosing an AI agent framework starts with the workflow your application requires. The framework should match the agent's control flow, runtime, and production requirements before you compare secondary features.

Orchestration model: LangGraph supports agents that need explicit state, branching, retries, checkpointing, or human-approval gates. CrewAI fits role-based collaboration where named agents pass work across tasks. The OpenAI Agents SDK is well-suited to linear handoff chains where one agent transfers control to another. Mastra fits TypeScript applications that combine model-driven agent loops with deterministic workflows. LlamaIndex is well-suited to retrieval-heavy agents, where document parsing, indexing, retrieval, and synthesis drive the application.

Language and runtime: Python is still the stronger choice for research tooling, niche model integrations, and retrieval infrastructure. TypeScript works better when the rest of the application already runs on Next.js, Hono, Express, or another JavaScript runtime, and when shared types span the API, agent, and UI, reducing implementation risk. LangGraph and OpenAI Agents SDK support both Python and TypeScript, CrewAI remains Python-only, LlamaIndex leads with Python and offers a TypeScript option, and Mastra is TypeScript-native by design.

Ecosystem maturity: Production-ready frameworks should provide stable releases, active maintenance, clear upgrade paths, OpenTelemetry support, checkpointing for long-running runs, MCP support for tool integration, and observability hooks that cleanly expose tool calls and state transitions. Strong community adoption helps reduce implementation risk because integrations, examples, and issue resolution become more reliable as agents move from prototype workflows into production systems.

Top 5 AI agent frameworks in 2026

1. LlamaIndex

When to pick: LlamaIndex fits retrieval-heavy agents where the main problem is reasoning over enterprise documents such as contracts, regulatory filings, technical manuals, or internal knowledge bases. The framework is strongest when retrieval quality, citation handling, and document processing shape the agent's output.

Architecture: LlamaIndex centers on indexes, retrievers, and query engines, with Workflows available for multi-step orchestration. LlamaParse and LlamaCloud add managed parsing, indexing, and extraction for teams that do not want to operate the full retrieval pipeline themselves.

Language: LlamaIndex leads with Python and also offers a TypeScript port.

Ecosystem maturity: LlamaIndex has 49K GitHub stars, and the broader LlamaIndex system includes hosted services for parsing, indexing, and extraction, which makes the framework a common fit for teams building document-heavy AI applications.

Limitation: LlamaIndex's agent and workflow primitives extend from its retrieval core. For agents that need complex branching, explicit state transitions, or multi-agent orchestration, teams often pair LlamaIndex with LangGraph, where LlamaIndex handles retrieval, and LangGraph controls the workflow.

2. Mastra

When to pick: Mastra fits TypeScript and JavaScript teams building full-stack applications where the backend already runs on Next.js, Hono, or Express. Shared Zod schemas can connect the API, agent logic, and UI, reducing coordination overhead when the same application both calls the agent and renders the result.

Architecture: Mastra is built around agents, workflows, tools, memory, RAG, and evals. Agents handle model-driven loops, workflows define deterministic step graphs, and tools wrap Zod-typed callable functions. The framework builds on the Vercel AI SDK, so streaming, tool calling, and structured outputs use abstractions that already fit many TypeScript stacks.

Language: Mastra is TypeScript-native.

Ecosystem maturity: Mastra reached stable 1.0 in January 2026 after graduating from Y Combinator W25 with $13 million in funding. The project has 22K+ GitHub stars and 300K+ weekly npm downloads, and the framework comes from the team behind Gatsby.

Limitation: Mastra is strongest when TypeScript is the firm runtime constraint. Python still leads for research tooling, niche model integrations, and experimental libraries, so teams running heavy model or retrieval experimentation may hit Python-only dependencies before they reach a Mastra limitation.

3. LangGraph

When to pick: LangGraph is best for stateful, multi-step agents that require explicit control over branching, retries, and human-approval gates. Regulated environments benefit from time-travel debugging because engineers can replay a run from any saved checkpoint.

Architecture: LangGraph represents an agent as a directed graph of nodes and edges over a typed state object. Nodes define actions, edges define transitions, and built-in checkpointing lets runs pause, resume, and replay from specific points in the graph.

Language: LangGraph ships in both Python and TypeScript, with the Python ecosystem leading on integrations.

Ecosystem maturity: LangGraph has the strongest community of any graph-based framework. The LangChain ecosystem brings 700+ integrations across databases, APIs, and enterprise tools. Stable 1.0 shipped in October 2025, and PyPI downloads exceed 38 million per month.

Limitation: LangGraph usually takes one to two weeks for engineering teams adopting the graph model from scratch. The state-machine model gives teams explicit control, but engineers new to graph-based orchestration need time to map application logic to nodes, edges, and state transitions.

4. CrewAI

When to pick: CrewAI fits multi-agent collaboration problems where roles, goals, and tasks map clearly to the work. Research-then-write pipelines, support triage with specialist agents, and content workflows work well when named agents can pass tasks through a defined process.

Architecture: CrewAI organizes agents into role-based crews, with each agent assigned a role, goal, and backstory. Tasks define the work each agent performs, and CrewAI supports sequential and hierarchical processes for moving work across the crew.

Language: CrewAI ships in Python only, with no official TypeScript port planned.

Ecosystem maturity: CrewAI has one of the larger communities in the agent-framework space, with 44.6K+ GitHub stars, native MCP support, and reported usage across 450 million monthly workflows. The framework's adoption is strongest among teams building role-based multi-agent systems and workflow prototypes.

Limitation: Conditional branching can be difficult because CrewAI is optimized for roles and tasks rather than for explicit state transitions. Logic such as "run agent A, then choose agent B or agent C based on the result" often ends up in prompts, making the behavior harder to inspect and debug as workflows become more complex.

5. OpenAI Agents SDK

When to pick: The OpenAI Agents SDK is for teams already building on the OpenAI Responses API who need a fast way to move from a provider call to a working agent. Linear handoff flows, tool calls, and guardrails are the strongest fit when the first version should stay small and easy to inspect.

Architecture: OpenAI Agents SDK builds on four primitives. An agent combines a model, instructions, and tools into a callable unit. Tool definitions wrap callable functions, handoffs transfer control from one agent to another, and guardrails enforce input and output constraints around each run.

Language: OpenAI Agents SDK ships in both Python and TypeScript.

Ecosystem maturity: OpenAI Agents SDK has been generally available since March 2025, includes first-party tracing, and now supports 100+ models through LiteLLM. OpenAI also maintains reference docs and example apps, which helps teams adopt the SDK without building the first version from scattered implementation patterns.

Limitation: The OpenAI Agents SDK is intentionally minimal, so complex routing, durable state, and long-running orchestration are usually handled in application code. That design works well for straightforward handoff chains, but teams building agents with branching logic or persistent workflow state may eventually need a fuller orchestration framework.

Best AI agent frameworks compared (2026)

Choosing the right framework

Use the guidance below to match each framework to the type of agent you are building.

Choose LlamaIndex when retrieval over enterprise documents is the core problem. The retrieval layer reduces glue code around parsing, indexing, hybrid retrieval, and citation handling, while the agent layer supports workflows built around retrieved context.

Choose Mastra when the backend is TypeScript and shared types across API, agent, and UI reduce implementation risk. Mastra is the strongest fit for full-stack JavaScript teams that need both model-driven agents and deterministic workflows.

Choose LangGraph when the agent needs explicit state, checkpointing, time-travel debugging, or human approval gates. The graph model fits workflows with branching, retries, and durability requirements that lighter frameworks often push into application code.

Choose CrewAI when the problem maps cleanly to roles, goals, and tasks. Research-then-write pipelines, support triage with specialist agents, and content workflows fit CrewAI well when collaboration between named agents is the main design pattern.

Choose OpenAI Agents SDK when the application already uses the OpenAI Responses API and the agent can be expressed through tools, guardrails, and handoffs. The SDK works well for early versions where the orchestration logic should stay small and inspectable.

Two-framework stacks are common in production. LangGraph often handles orchestration, while LlamaIndex handles retrieval, and Mastra workflows can wrap an OpenAI Agents SDK call when a TypeScript application needs a smaller, handoff-based agent within a larger workflow. The evaluation layer should support mixed stacks from the outset, because production agents rarely remain within a single framework for long.

Braintrust: The eval layer for each framework

Braintrust is the stronger choice when one application uses multiple agent frameworks, or when evaluation needs to remain consistent as orchestration choices change.

Each framework produces a different trace structure, so the evaluation layer has to understand more than final outputs. Braintrust captures framework-specific spans, connects them to scorers, and turns production behavior into reusable regression datasets across LlamaIndex, CrewAI, LangGraph, OpenAI Agents SDK, and Mastra.

Braintrust captures different trace structures from LlamaIndex, Mastra, LangGraph, CrewAI, and OpenAI Agents SDK, then connects those traces to shared datasets, scorers, and evals.

LlamaIndex: LlamaIndex traces cover queries, retrievals, retrieved chunks, reranking steps, and LLM synthesis spans. Useful scorers include retrieval relevance, faithfulness to retrieved context, citation correctness, and end-to-end QA accuracy. Braintrust captures LlamaIndex traces through auto-instrumentation: a single auto_instrument() call traces every LLM, embedding, and query engine call, and Braintrust also supports an OpenTelemetry path for LlamaIndex.

import braintrust

braintrust.auto_instrument()
braintrust.init_logger(project="my-project")

from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o-mini")
response = llm.complete("What is the capital of Australia?")
print(str(response))

Full integration reference: LlamaIndex integration.

Mastra: Mastra traces cover agent generation spans, workflow-step spans, and tool-call spans, with token metrics and structured inputs and outputs attached to each step. Useful scorers include workflow-step correctness, Zod schema validation of tool inputs, RAG faithfulness in retrieval steps, and end-to-end task quality. Braintrust integrates with Mastra through the BraintrustExporter from the @mastra/braintrust package.

const logger = initLogger({ projectName: "mastra-demo" });

const exporter = new BraintrustExporter({
  braintrustLogger: logger,
});

const mastra = new Mastra({
  agents: {
    assistant: new Agent({
      name: "Assistant",
      instructions: "You only respond in haikus.",
      model: "openai/gpt-4o-mini",
    }),
  },
  observability: new Observability({
    configs: {
      braintrust: {
        serviceName: "demo",
        exporters: [exporter],
      },
    },
  }),
});

async function main() {
  const agent = mastra.getAgent("assistant");
  const response = await agent.generate("Tell me about recursion in programming.");
  console.log(response.text);
}

main();

Full integration reference: Mastra integration.

LangGraph: LangGraph traces start with a root run, then break into nested spans for node transitions, and LLM calls through the LangChain callback handler. Useful scorers include node-level correctness, end-to-end task completion, retry counts, and error rates across stateful graph paths. Braintrust instruments LangGraph through the LangChain callback system with a single setup call.

from typing import TypedDict
import braintrust

braintrust.auto_instrument()
braintrust.init_logger(project="My Project")

from langchain_openai import ChatOpenAI
from langgraph.graph import END, START, StateGraph

class GraphState(TypedDict, total=False):
    message: str

def main():
    model = ChatOpenAI(model="gpt-5-mini")

    def say_hello(state: GraphState):
        response = model.invoke("Say hello")
        return {"message": response.content}

    def say_bye(state: GraphState):
        return {"message": f"{state.get('message', '')} Bye."}

    workflow = (
        StateGraph(state_schema=GraphState)
        .add_node("sayHello", say_hello)
        .add_node("sayBye", say_bye)
        .add_edge(START, "sayHello")
        .add_edge("sayHello", "sayBye")
        .add_edge("sayBye", END)
    )

    graph = workflow.compile()
    result = graph.invoke({})
    print(result)

if __name__ == "__main__":
    main()

Full integration reference: LangGraph integration.

CrewAI: CrewAI traces contain a crew root span with child spans for agents, tasks, and tool calls. Useful metrics include per-agent output quality, handoff fidelity between agents, and final crew output quality. Braintrust captures CrewAI traces through auto-instrumentation: call auto_instrument() before creating your crew, or setup_crewai() to instrument only CrewAI. If you already run CrewAI with OpenTelemetry, you can route those spans to Braintrust instead.

import os
import braintrust
from crewai import Agent, Crew, Task
from crewai.llm import LLM

braintrust.auto_instrument()
braintrust.init_logger(
    api_key=os.environ["BRAINTRUST_API_KEY"],
    project="crewai-example",  # Replace with your project name
)

llm = LLM(model="gpt-4o-mini")

coder = Agent(
    role="Software developer",
    goal="Write clear, concise code on demand",
    backstory="An expert coder with a keen eye for software trends.",
    llm=llm,
)

task = Task(
    description="Write a minimal HTML page with a heading that says Hello Braintrust.",
    expected_output="A concise HTML document",
    agent=coder,
)

crew = Crew(agents=[coder], tasks=[task])
result = crew.kickoff()
print(result)

Full integration reference: CrewAI integration.

OpenAI Agents SDK: OpenAI Agents SDK traces typically include a root task span for the agent run, loop-iteration spans, and child spans for tool calls, guardrails, and handoffs. Useful scorers include guardrail-trigger correctness, handoff routing accuracy, tool-call argument validation, and final-output quality. Braintrust integrates through the OpenAIAgentsTraceProcessor from @braintrust/openai-agents.

const logger = initLogger({
  projectName: "openai-agents-example", // Replace with your project name
  apiKey: process.env.BRAINTRUST_API_KEY,
});

addTraceProcessor(new OpenAIAgentsTraceProcessor({ logger }));

async function main() {
  const agent = new Agent({
    name: "Assistant",
    model: "gpt-5-mini",
    instructions: "You answer in one sentence.",
  });

  const result = await run(agent, "What is the capital of France?");
  console.log(result.finalOutput);
}

main().catch(console.error);

Full integration reference: OpenAI Agents SDK integration.

Braintrust supports agent evaluation across mixed framework stacks through native SDKs, OpenTelemetry support, agent-framework integrations, LLM Gateway instrumentation, tracing, datasets, scorers, and evals. Engineering teams can capture agent behavior across framework calls and AI providers, inspect tool use and retrieval steps, convert production traces into test cases, and run evaluations before agent changes ship. Braintrust works as the evaluation layer around the orchestration stack, connecting production behavior to repeatable quality checks across frameworks.

Start free with Braintrust to evaluate agent quality across frameworks, providers, and production traces in a single workflow.

Also read:

• Best AI agent observability tools
• AI agent evaluation: A practical framework

FAQs: Best AI agent frameworks (2026)

Do I need a framework at all to build an agent?

Many production agents start as direct provider API calls with explicit tool-calling logic and a few hundred lines of orchestration code. A framework becomes useful when state management, multi-agent coordination, retrieval pipelines, retries, or human approval steps become too complex to maintain clearly in application code.

Can I use two frameworks in one project?

Mixed framework stacks are common when each framework handles a different part of the agent system. LangGraph can control orchestration while LlamaIndex handles retrieval, and Mastra workflows can wrap an OpenAI Agents SDK call inside a TypeScript application. The main requirement is a shared tracing and evaluation layer like Braintrust, because each framework produces a different trace structure.

Should I use an OSS framework or a hosted platform?

Open-source frameworks are usually the better starting point when engineering teams want control over orchestration code, deployment, and framework portability. Hosted platforms become useful when the team needs managed checkpointing, deployments, autoscaling, or centralized operational tooling. The infrastructure decision should stay separate from the evaluation decision, because agent quality still needs to be measured against production traces and regression datasets.

If I switch frameworks later, do I lose my evals?

You lose eval continuity only when the evaluation setup is tied to one framework. A framework-agnostic eval layer, connected via SDKs or OpenTelemetry, can preserve datasets, scorers, regression tests, and trace history across framework migrations. Braintrust provides multi-framework support by capturing framework-specific spans while keeping evaluation assets reusable as orchestration code changes.

Which eval platform supports the most agent frameworks?

LangSmith is strongest for LangChain and LangGraph-heavy teams, and Langfuse fits OSS-first teams that want broad OpenTelemetry coverage. Braintrust supports native integrations with LangGraph, CrewAI, OpenAI Agents SDK, Mastra, Pydantic AI, Claude Agent SDK, and Google ADK, and provides OpenTelemetry support for LlamaIndex and other frameworks that emit OTel spans. For teams running multiple frameworks within a single agent system, Braintrust provides broader native coverage throughout the evaluation workflow.

How do I evaluate a multi-framework agent end-to-end?

Start by sending traces from each framework into the same evaluation layer through SDKs or OpenTelemetry. Then score the agent at multiple levels: retrieval relevance, tool-call correctness, node or step quality, handoff routing accuracy, and final output quality. Braintrust connects traces, scorers, datasets, and eval runs into a single workflow, so engineering teams can turn production failures into regression tests and enforce quality before agent changes ship.