Home / AI Architecture & System Design / Agents & LLM Applications

Agents & LLM ApplicationsSeniorQuestion 8 of 9

How do you handle rate limits from an LLM provider at scale?

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Functional requirements

Retry-with-backoff is the client-side answer. At scale the real problem is sharing a fixed capacity fairly across services, tenants and priorities.

You What are the provider's limits, and how many services share them?

Interviewer Per model: 2 million input tokens per minute, 400,000 output tokens per minute, and 5,000 requests per minute. About ten internal services share one account.

You What kinds of traffic?

Interviewer Interactive chat, an agent platform, and nightly batch jobs that summarize documents.

You Are there external customers with plans?

Interviewer Yes, multi-tenant. Enterprise customers have contractual capacity.

5–10 min Non-functional requirements and estimates

Back-of-envelope estimates:

• Which limit binds? Interactive requests average ~3,000 input and ~400 output tokens. Input: 2,000,000 / 3,000 ≈ 667 requests/min. Output: 400,000 / 400 = 1,000/min. Requests: 5,000/min. So input tokens bind first, at ~11 requests/s for this mix. Planning by request count alone would be badly wrong.
• Prompt caching on a 1,500-token static prefix often reduces how cached tokens count or cost; check the provider's rules, as this can effectively raise headroom.
• Batch jobs: 200,000 documents × 4,000 tokens = 800M input tokens. At the full real-time limit that's 400 minutes, nearly 7 hours of total capacity, which would starve everything else. They belong on the batch interface.
• Output reservation: reserving max_tokens = 4,096 for every request but averaging 400 actual output tokens means reserving ~10× too much output capacity. Reserve a realistic estimate, then reconcile.

10–18 min High-level design

You Put a central gateway in front of the provider. It estimates each request's tokens, reserves them from shared buckets (global, per tenant, per priority), queues by priority when capacity is short, and reconciles with actual usage afterwards. Batch work goes to the batch API.

Components:

• Services: call the internal LLM gateway instead of the provider directly.
• Token limiter: token buckets for requests, input tokens and output tokens, at global, per-model, per-tenant and per-priority levels.
• Priority queue: interactive > agent > batch; queued requests have deadlines and are rejected quickly if they'd miss them.
• LLM client: retries with backoff and jitter, honours Retry-After, has circuit breakers per model and region.
• Fallbacks: alternate region or model for overloaded periods, only where quality allows.
• Batch path: offline jobs submit to the provider's batch API.
• Metrics: utilization of each limit, queue wait, 429 rate, per tenant and service.

Reserve, then reconcile:

def admit(req):
    est_in  = count_tokens(req.messages)                      # exact with the tokenizer
    est_out = min(req.max_tokens, p90_output[req.feature])    # realistic, not max
    needs = {"requests": 1, "input": est_in, "output": est_out}
    for bucket in buckets_for(req.model, req.tenant, req.priority):
        if not bucket.can_take(needs):
            return enqueue(req, deadline=req.deadline)        # or reject if it can't make it
    for bucket in buckets_for(req.model, req.tenant, req.priority):
        bucket.take(needs)
    return send(req)

def on_complete(req, usage):
    for bucket in buckets_for(req.model, req.tenant, req.priority):
        bucket.refund(output=est_out - usage.output_tokens)   # or charge the overage

18–32 min Deep dive

You I'd cover the bucket design, retries and overload behaviour, and fairness between tenants.

Interviewer How do token buckets work across many gateway instances?

Deep dive 1 · Distributed token buckets

• Token bucket: capacity C refills at rate r per second; a request takes its estimated tokens; if not enough are left, it waits or is rejected. It allows short bursts up to C while holding the long-run rate at r.
• Across instances: either a shared store with atomic operations (e.g. a Redis script that refills and takes in one step), or local buckets with leases: each instance leases a slice of capacity every second and returns unused slices. Leasing avoids a network round trip per request.
• Several limits at once: a request must fit the request, input-token and output-token buckets for the model, plus its tenant and priority buckets.
• Headroom: target ~80% of the provider's limit, since estimates aren't perfect and other clients may share the account.

Interviewer What happens when the provider starts returning 429s or overloaded errors anyway?

Deep dive 2 · Retries and overload

• Retry only retryable errors: 429, 5xx and overloaded responses. Never retry 400-class errors other than 429; they're bugs.
• Exponential backoff with full jitter: wait a random time between 0 and min(cap, base × 2^attempt), and honour Retry-After when present. Jitter prevents thousands of clients retrying in lockstep.
• Retry budgets: limit retries to ~10% of traffic, so retries can't double load during an outage.
• Circuit breakers: after a burst of failures for a model and region, stop sending for a short period, then probe. Meanwhile use the fallback, or shed batch and low-priority traffic.
• Load shedding: when queues are full, reject batch and agent work first, with a clear retry_after, to keep interactive traffic healthy.

Deep dive 3 · Fairness and tenant guarantees

• Reserved capacity: enterprise tenants get guaranteed slices; unused reserved capacity can be borrowed by others but is reclaimed immediately when needed.
• Weighted fair queuing: during contention, each tenant's queue is served in proportion to its weight, so one noisy tenant can't starve the rest.
• Per-tenant daily budgets with soft and hard limits, and alerts before customers hit them.
• Priority within a tenant: a tenant's interactive traffic beats its own batch jobs.

Deep dive 4 · Capacity planning (if time allows)

• Forecast peak tokens per minute per feature, add headroom, and request provider limit increases or provisioned throughput before launches.
• Track utilization per limit (requests, input, output) and alert at ~70–80%.
• Separate accounts or projects per environment, so a load test can't eat production capacity.

32–37 min Trade-offs

Build first: the gateway with global and per-priority token buckets, retries with jitter and budgets, and metrics. Then tenant quotas and weighted queuing, then fallbacks and leasing.

37–40 min Wrap-up and likely follow-ups

Close with: "All traffic goes through a gateway that estimates tokens and reserves them from token buckets at global, model, tenant and priority levels, then reconciles with actual usage. Interactive traffic has reserved capacity and priority, batch work goes to the batch API, retries use jittered backoff within a budget, and circuit breakers and load shedding protect against provider brownouts. Input tokens usually bind first, so I plan in tokens, not requests."

Likely follow-ups:

• How accurate are token estimates? Input tokens can be counted exactly with the model's tokenizer; output must be estimated per feature from historical percentiles.
• What if a provider changes limits? Make limits configuration, read them from response headers where available, and alert on unexpected 429s.
• Multi-provider? Separate buckets per provider and model; a router chooses based on quality constraints and available capacity.

What interviewers listen for

• Working out which limit binds, with numbers.
• Central, token-aware limiting with priorities and tenant fairness.
• Backoff with jitter, retry budgets and circuit breakers.
• Moving batch work off the real-time path.

Red flags

• Only "retry with exponential backoff" as the answer.
• Rate limiting by request count when token limits are what bind.
• No plan for one tenant or batch job starving everyone else.