How to keep audit-ready logs of every LLM call (retention, export, and compliance)

Why standard LLM logging fails an audit

Regulated teams usually discover weak LLM logging when someone asks for proof. A regulator, enterprise customer, security reviewer, or internal incident team may ask what the AI told a specific user on a specific date and why the model returned that answer. During an audit or escalation, the team needs a complete record quickly, and missing context can turn a routine evidence request into an engineering investigation.

Many LLM logging setups struggle because they were designed around debugging recent failures. Audit workflows need records that are complete, retained long enough, protected against silent changes, and easy to retrieve for a named user, a date range, or an incident.

Partial capture: Sampling can reduce storage cost, but sampled traces leave gaps in the record. If the interaction under review was dropped, the team cannot reconstruct what happened.

Short retention: Default retention windows are often measured in days or weeks. When a compliance question appears months later, the relevant prompt, response, or trace may already be gone.

Editable records: Application logs often live in systems where engineers can modify or overwrite data. A record loses evidentiary value when the team cannot show whether the record changed.

Difficult export: Even when the logs exist, producing a clean extract for a single user, session, or date range can take days of engineering effort. Audit requests usually need a faster path.

Broad access: Prompts and responses can contain personal data, protected health information, or customer-sensitive content. Giving every internal user access to every log creates a separate compliance risk.

Standard logging optimizes the process of finding and fixing software issues. Audit-ready LLM logging requires a stronger operating model around capture, retention, export, and access control. The rest of this guide explains how each control works and how to configure the controls in Braintrust.

What an audit-ready LLM log must capture

An audit-ready LLM log should preserve enough context to reconstruct a past call exactly as it happened.

An audit-ready LLM log combines the prompt, response, tool activity, versions, timestamps, identifiers, metadata, and tamper-evident storage into one complete record that can reconstruct the interaction.

The full prompt: Store the complete input sent to the model, including the system prompt, user message, and retrieved context. The response can only be reviewed fairly when the original input is available.

The full response: Keep the model output in full. Summaries and truncated previews can remove the wording that an auditor, reviewer, or customer escalation team needs to inspect.

Tool calls and returned values: Record each tool invocation, the arguments passed, and the values returned. For agentic systems, tool activity often explains why the final response looked the way it did.

Model and prompt versions: Capture the model and prompt versions that were live during the call. Version data allows a team to explain a past result after a model, prompt, or application update.

Timestamps and event order: Store start times, end times, and the ordered sequence of events. Timing data helps reviewers separate the user request, retrieval step, tool call, model response, and any follow-up action.

User and session identifiers: Tie each call to a user, account, tenant, or session. Audit requests often ask for all activity associated with a single person, customer, or incident window.

Metadata and trace identifiers: Include IDs such as trace ID, request ID, run ID, environment, application version, and tenant. Metadata makes the record easier to search, filter, export, and connect to related systems.

Integrity controls: Protect records from silent edits or deletion. A log has limited evidentiary value when a team cannot demonstrate whether the record changed after the interaction.

Completeness is also a compliance requirement. HIPAA audit controls require teams to record and review activity involving electronic protected health information, and it depends on a complete and trustworthy record of model behavior.

Braintrust captures prompts, responses, tool calls, and metadata as structured spans, which makes each LLM call easier to inspect and connect to the surrounding trace. Because prompts and responses can contain personal or health data, teams should weigh how much they log against the sensitivity of that content when setting retention windows and access rules.

LLM log retention and immutability

Capturing the right fields only helps when records are still available at the time of review. Retention should follow two schedules.

Operational logs can usually expire sooner because engineering teams use them for debugging and cost control.

Compliance logs need a longer window because audits, legal reviews, and customer escalations often happen months after the original LLM call.

Braintrust supports retention policies at the project level, with separate windows for logs, experiments, and datasets. Separate retention windows allow teams to keep production logs for audit requirements while removing short-lived development data sooner. Custom retention policies are available on the Enterprise plan. To configure a retention policy in the dashboard, follow these steps.

1. Go to Settings > Data management.
2. Click + Create automation.
3. Select the Data retention type.
4. Set the Object type to Logs, Experiments, or Datasets.
5. Set the Retention period in days.
6. Click Save.

Braintrust includes example retention patterns such as 7 days for development logs and 90 days for production logs, along with a 30-day compliance example. Treat the compliance example as a starting point for configuration. The correct retention period should be specified in the regulation, contract, or internal policy attached to the workload.

Immutability gives retained records evidentiary value. Braintrust runs deletions on the schedule the team sets. In self-hosted deployments, deleted data is soft-deleted first and then purged after a 24-hour grace period, giving teams time to recover data removed by mistake. Braintrust also writes each deletion to a delete log that can be reviewed if data is missing unexpectedly.

Teams that need stronger tamper evidence can run Braintrust with their own object storage in a self-hosted deployment. Cloud storage controls can enable bucket-level audit logging and write-once retention rules, ensuring the underlying records comply with the organization's evidence and retention requirements. Together, fixed retention windows, recorded deletion, and storage-level controls make LLM logs easier to defend during audits and incident reviews.

Exporting LLM logs for audits and incident reviews

An audit-ready log also needs a reliable export path. Auditors usually need a filtered extract for a user, an account, a session, or a date range. Engineering teams may need structured data they can load into a warehouse or incident review workflow. Planning for both outputs keeps evidence requests from becoming one-off engineering work.

Braintrust supports three export paths:

Dashboard export

Use the Logs page when a reviewer needs a one-time extract. Apply filters for the relevant user, session, project, or time range, then export the results as JSON or CSV. The dashboard export controls appear for users with Update or Delete permissions on the object; read-only users can export through the SDK or API instead.

SDK export

Use the SDK when internal review workflows need repeatable exports. Programmatic access lets teams pull logs or datasets into their own analysis scripts, review tools, or reporting jobs.

API export

Use the API when logs need to be moved to long-term storage, a data warehouse, or a compliance archive. The endpoint can return JSON or Parquet, which makes the export easier to load into downstream systems.

curl -X POST https://api.braintrust.dev/btql \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SELECT input, expected FROM project_dataset('"'<PROJECT_ID>'"', '"'<DATASET_ID>'"') WHERE expected IS NOT NULL",
    "fmt": "json"
  }'

For audit workflows, two API options are especially useful:

• The fmt parameter controls the export format, including JSON and Parquet.
• The audit_log parameter adds audit trail data on changes, helping reviewers understand who changed records and when.

For long-term archival, Braintrust can export to cloud storage on a schedule in JSON Lines or Parquet. Scheduled export gives compliance teams a standing copy in storage the organization controls, while Braintrust remains the system teams use for inspection, filtering, and review.

Access control and data isolation for LLM logs

LLM logs often contain sensitive user input, customer data, retrieved context, and protected health information. Access control decides who can read or export those records, while deployment architecture decides where sensitive log data is stored and processed.

Braintrust uses role-based access control through permission groups. The built-in groups cover the common access levels. Owners have full access; Engineers can create, update, and delete resources; and Viewers have read-only access. Custom permission groups on the Enterprise plan let teams control access at the project and object level, so only a designated group can inspect sensitive production logs.

Read access should be granted carefully. The Read permission includes data export through the API and SDK, and there is no separate export-only permission, so anyone who can read sensitive records through the API or SDK can also export them. In the Braintrust UI, download and export controls appear only for users who also have Update or Delete. Narrow project-level, organization-level, and object-level permissions remain the primary control for limiting who can read and export sensitive logs.

For stricter data isolation, Braintrust supports deployments that keep sensitive data in your own cloud, through BYOC or self-hosting. The data plane runs inside the customer's cloud account and region, where it stores sensitive data such as logs, traces, prompts, and datasets. The Braintrust-managed control plane handles the interface, authentication, and metadata such as project names, without storing or processing sensitive prompt and response content.

In these deployments, SDKs and browsers communicate directly with the customer's data plane. Prompt and response content remain within the customer's cloud boundary, supporting teams with data residency, key control, and infrastructure isolation requirements. With a self-hosted deployment, teams can manage encryption keys through their own cloud key management service, place the data plane behind a firewall or VPN, and keep LLM data in the selected region.

Braintrust also includes a control for image rendering in logs. Teams can use click-to-load or block-all modes to reduce the risk of data being sent to an external server through a malicious image URL when someone views a log.

Teams that need full environment control can self-host Braintrust using the provided infrastructure modules for AWS, Google Cloud, or Azure.

Mapping logging controls to HIPAA, SOC 2, and EU rules

Capture, retention, export, and access control align with how common compliance programs evaluate logs. The exact policy depends on the workload, jurisdiction, contract, and risk model, so compliance teams and counsel should confirm the final requirements for each system.

HIPAA

For systems that handle electronic protected health information, HIPAA's Security Rule expects safeguards for access control, audit controls, integrity, authentication, and transmission security. In LLM logging terms, teams need records that show activity involving sensitive health data, limit access to authorized users, preserve record integrity, and keep required documentation available for the required period. HHS says regulated entities must record and examine activity in systems that contain or use ePHI, and required Security Rule documentation must be retained for six years. Braintrust states that it supports HIPAA requirements and can support Business Associate Agreements.

SOC 2

SOC 2 evaluates whether a service organization's controls are designed and operating effectively over time. For LLM systems, retained logs support auditor sampling, complete records support monitoring and incident review, and access control shows that sensitive prompts and responses are handled with least privilege. Braintrust is SOC 2 Type II compliant, and the associated reports are available under a mutual NDA.

GDPR

GDPR pushes LLM logging teams to balance auditability with data minimization and storage limitation. Logs that contain personal data should capture only what the audit record requires, use retention periods that can be justified, and include identifiers that allow teams to find records tied to a data subject access or erasure request. Data residency and transfer requirements can also affect where log data is stored. Braintrust states that it supports GDPR requirements, and its BYOC and self-hosted deployment options allow teams to keep sensitive AI data within their own cloud environment.

Across HIPAA, SOC 2, and GDPR, the operating requirement is consistent. Teams should be able to reconstruct the LLM call, retain the record for the required period, produce evidence on request, and restrict access to sensitive content.

For teams handling regulated or sensitive workloads, Braintrust's enterprise plans provide the logging controls needed to prepare for audits before an evidence request arrives. Build audit-ready LLM logging on Braintrust →

FAQs: how to keep audit-ready logs of every LLM call

Is self-hosted or cloud better for compliance?

Managed cloud can work for teams with standard security requirements and lower data isolation needs. BYOC or self-hosted deployment is a better fit when compliance requirements include data residency, customer-managed keys, private networking, or stricter control over where prompts and responses are stored. Braintrust supports BYOC and self-hosted deployments for teams that need sensitive LLM data to remain within their own cloud environment.

How long should I retain LLM logs?

Retention should follow the strictest requirement attached to the workload. Engineering logs often expire sooner, while audit logs may need to remain available for months or years, depending on regulations, contract terms, and internal policies. In Braintrust, teams get project-level retention policies to manage production logs, experiments, and datasets on separate schedules.

Can I export LLM logs for an audit?

Audit exports should match the reviewer's request. A compliance reviewer may need a filtered CSV or JSON file for a specific user, session, or time range, while an internal incident team may need structured data for analysis. Braintrust gives teams dashboard exports for one-time reviews and programmatic exports for repeatable audit or incident workflows.

Do logging prompts create a PII risk?

Prompt logging can create a PII risk because users often include personal, financial, health, or customer-sensitive data in normal language. Teams should treat prompt and response logs as sensitive records, limit access to those who need it, set deliberate retention windows, and use BYOC or self-hosted deployments when sensitive LLM data must remain within the organization's environment.