> ## Documentation Index
> Fetch the complete documentation index at: https://braintrust.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Brainstore OOM errors and retry-induced rate limits

export const plans_0 = "Enterprise"

export const deployments_0 = "Self-hosted (GCP)"

export const data_plane_version_0 = undefined

export const use_case_0 = "Use case - Self-hosted deployments on GCP experiencing rate limiting on GCS object storage and hitting 10 of 10 retries."

<Note>
  **Applies to:**

  * Plan - {plans_0}
  * Deployment - {deployments_0}
  * {data_plane_version_0}
  * {use_case_0}
</Note>

## Summary

**Issue:** Self-hosted brainstore pods restart or fail, followed by cascading rate limit failures with `429` errors on `.managed.json` objects in GCS. Multiple retry attempts per item exhaust the GCS write limit of 1 write/second per object.

**Cause:** Excessive retry attempts on individual objects trigger GCS rate limits, creating a cascading failure loop.

**Resolution:** Deploy the fix from commit SHA `40fda545aed99a12b1d04064c6205f8450d209ec` which prevents multiple per-item retry attempts.

## Resolution Steps

### Deploy the fix using commit SHA

#### Step 1: Update Helm configuration

Set the `image.tag` value in your Helm `values.yaml` to the fix commit SHA where a fix has been applied (alternatively, this will be included in a future official release):

```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
image:
    tag: "40fda545aed99a12b1d04064c6205f8450d209ec"

```

#### Step 2: Re-apply deployment

Apply the updated Helm configuration using Terraform or directly with helm upgrade.

#### Step 3: Monitor for rate limit errors

Check logs to verify `429` errors on `.managed.json` objects no longer appear.

### Alternative: Wait for official release

#### Step 1: Monitor changelog

The fix will be included in a future official data plane release.

#### Step 2: Update to latest version

Update your `image.tag` to the latest semantic version when available.