> ## Documentation Index
> Fetch the complete documentation index at: https://braintrust.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Use S3 export instead of polling SQL for pipelines

export const plans_0 = "Enterprise"

export const deployments_0 = "Any"

export const data_plane_version_0 = undefined

export const use_case_0 = "Use case - External observability or data pipeline that polls SQL project_logs repeatedly and hits the 20/min rate limit"

<Note>
  **Applies to:**

  * Plan - {plans_0}
  * Deployment - {deployments_0}
  * {data_plane_version_0}
  * {use_case_0}
</Note>

## Summary

**Issue:** High-frequency SQL polling through the `/btql` API across multiple projects can exhaust the query rate limit, resulting in `429 Too Many Requests` errors.

**Cause:** Each API query counts toward the limit. Polling many projects individually, especially concurrently or on a short schedule, can exceed the available request budget.

**Resolution:** For continuous log ingestion into external systems like Datadog, use cloud storage export instead of polling. If SQL polling is required, combine project queries where possible, use cursor/high-watermark pagination, select only needed columns, and pace requests.

## Resolution steps

### If you are exporting logs to an external system (Datadog, observability pipelines)

#### Step 1: Set up S3 cloud storage export

In Braintrust, go to **Settings > Data management > Create export automation**. Configure an incremental export to an S3 bucket in your account.

This replaces interactive SQL API polling with an incremental export pipeline. It avoids the `/btql` polling rate limit and preserves full payloads depending on the export shape you choose.

#### Step 2: Ingest from S3 into Datadog

Use the [Datadog Forwarder Lambda](https://docs.datadoghq.com/serverless/libraries_integrations/forwarder/) or Firehose to pick up files from the S3 bucket. Your existing metric logic applies unchanged.

### If you still require SQL polling

#### Step 1: Batch multiple projects into one query

`project_logs()` accepts multiple project IDs. Combine projects into a single query instead of one query per project.

```sql theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
SELECT is_root, error, created
FROM project_logs('project_id_1', 'project_id_2', shape => 'traces')
WHERE created >= now() - interval '15' minute
ORDER BY created DESC
```

#### Step 2: Use a high-watermark cursor instead of a fixed time window

Store the last `created` timestamp or `_xact_id` per project after each successful query. Query only new rows on the next run to avoid re-scanning overlapping windows.

```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
WHERE created > '2024-01-01T00:00:00Z'  -- last successful timestamp
ORDER BY created ASC
```

#### Step 3: Select only required columns

Omit `input`, `output`, `expected`, and `metadata` if not strictly needed. These fields can be very large and increase backend load.

```sql theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
SELECT is_root, error, created
FROM project_logs('project_id', shape => 'traces')
WHERE created >= now() - interval '15' minute
```

#### Step 4: Pace queries with a queue

If querying many projects individually, introduce a delay between requests (e.g., one query every 3–5 seconds) to stay within the 20 requests/minute limit.

## Additional information

Rate limit increases are generally not granted when a workload-appropriate alternative exists. For multi-project log ingestion at scale, S3 export is the recommended architecture.

Self-hosted deployments can adjust the limit using the `RATELIMIT_BTQL_DEFAULT` environment variable. See [BTQL rate limits on Starter and Pro plans](/kb/btql-rate-limits-on-free-and-pro-plans) for details.