What a burst limit looks like
Burst protection is triggered when a client sends too many requests in a short interval. Common causes are a worker fan-out, a backfill started by several replicas, a webhook storm, or a retry loop that releases all jobs simultaneously. The account may still have daily capacity remaining.
Symptoms include a sudden 429 spike, recovery after traffic is slowed, and multiple workers reporting the same window. Measure requests per second, concurrent requests, endpoint distribution, and retry volume. A fixed delay in each worker can still synchronize into another spike; use jitter and a shared limiter.
What a daily limit looks like
Daily exhaustion comes from total volume across the account or application policy. A slow integration can hit it after hours of polling, repeated searches, or a full export. Waiting a few seconds does not restore the budget. The correct fix is reducing call count or waiting for the reset window.
Audit calls per workflow run, records per sync, pagination pages, property metadata refreshes, search requests, and duplicate webhook processing. A “small” call repeated for every record can dominate the budget.
Inspect the response before changing code
Log status, endpoint, method, request timestamp, retry-after value, limit headers, correlation ID, and a redacted error category. Never log the Authorization header. Compare the 429 body and headers with HubSpot’s current usage guidelines because limits and availability can vary by account, app, endpoint, and subscription.
Classify the event as burst, daily, unknown, or a provider outage. Unknown should alert and be retried conservatively, not amplified. Store the classification with the job so operators can see why work was delayed.
Build a coordinated limiter
One process-local semaphore is not enough when several replicas share the same HubSpot account. Use a distributed token bucket or a queue with a per-account concurrency key. Give interactive requests a reserved lane and backfill jobs a lower priority. Cap retries globally.
Example queue policy:
The exact value should be measured and configured, not copied from a generic example. Treat `Retry-After` as authoritative when supplied.
key = "hubspot:" + portal_id
max_in_flight(key) = conservative_value
on_429: pause(key, retry_after); requeue_with_jitter(job)
Reduce burst pressure
Batch supported operations, avoid one metadata request per record, cache property definitions with a short refresh period, and spread scheduled jobs across time. Use bounded concurrency rather than launching one promise per record. When a webhook delivers several events, enqueue them and let workers drain at a controlled rate.
Use exponential backoff with a maximum delay and randomized jitter. Reset the attempt counter only after a successful response. Do not retry 400, 401, 403, or a deterministic validation error as if it were a burst failure.
Reduce daily consumption
Prefer webhooks or journal-based change capture over frequent full scans when suitable. Store a high-water mark and request only changed records. Ask for the properties actually used. Avoid fetching the same record after every update if the write response already contains the needed values.
Deduplicate jobs by portal, object, and record. Collapse repeated updates in a short queue window. Review CRM search loops and pagination for accidental cursor resets. Check whether a monitoring probe is hitting a write endpoint or running against every customer.
Backfill safely
Estimate calls before starting. Divide the job into checkpoints by portal and record range. Pause the backfill when the account approaches a daily budget threshold. Resume from a durable checkpoint, not from the beginning. Make the write idempotent and retain a reconciliation report.
If the daily limit is exhausted, communicate the expected resume time and pause noncritical jobs. Do not rotate a token expecting the account budget to reset; a new credential can still share the same account or application limits.
Warning: retry storms are an outage multiplier. A 429 handler that immediately retries every worker can turn a short burst event into sustained throttling and exhaust the daily budget faster.
Manual QA checklist
Test one 429 with `Retry-After`, one without it, concurrent workers, several replicas, duplicate jobs, a reset after a pause, a daily-budget condition, network timeout, 401, 403, validation 400, and a successful recovery. Verify that only eligible errors retry and that retry count is bounded.
Check metrics for request rate, concurrency, 429 by classification, retry delay, queue age, daily estimate, and calls per successful record. Confirm alerts distinguish “burst throttling” from “daily budget exhausted.” Run a small backfill and kill a worker to prove the checkpoint resumes without duplicates.
Operations during an incident
When a 429 spike begins, pause backfills and low-priority polling first. Preserve a sample response and note the affected portal, app, endpoint, and start time. Lower shared concurrency before changing business logic. If the response indicates a daily budget, communicate that the account needs a volume reduction or reset window; do not promise that a token rotation will help.
After recovery, inspect the queue for jobs that were acknowledged but not completed, jobs retried more than once, and jobs whose source record changed during the pause. Reconcile by stable record ID and source timestamp. Document the number of lost, delayed, duplicated, and successfully replayed jobs.
Review the call graph quarterly. A healthy limiter can still protect an inefficient integration from burst errors while it quietly consumes the daily allowance. Track calls per business outcome, not only total calls. The best fix is often one fewer list, search, or metadata request in every record loop.
A decision table for operators
| Evidence | First action | |---|---| | `policyName` or message indicates secondly/burst | Pause concurrency and honor `Retry-After` | | Daily remaining reaches zero | Pause noncritical work until reset | | Headers are absent under OAuth | Use response policy/body plus an internal ledger | | 429 appears only on one endpoint | Inspect endpoint-specific behavior before global throttling |
This prevents a team from rotating credentials or adding workers when the account has simply exhausted its shared daily allowance.
Where these facts come from