How should an application store and use LLM API keys?

30-second answer

Keep keys on the server in a secrets manager or environment configuration, never in client code or source control. Use separate keys per environment and service, rotate them, restrict scopes and set spending limits where possible, and proxy client requests through your backend with your own auth and rate limits.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask about the architecture (web, mobile, server-to-server), number of services and environments, and whether users bring their own keys. Framing: "An API key is a credential with spending power and data access. Keep it server-side, scoped, rotated, monitored, and have a plan for leaks."

5–15 min Core explanation: principles

Never in client code. Browsers and mobile apps can be inspected; anything shipped to them is public. Clients call your backend; your backend calls the provider.
Never in source control. Use a secrets manager (cloud secret stores, Vault) or the hosting platform's encrypted environment variables.
Separate keys per environment and service (dev, staging, prod; chat service vs. batch jobs) to limit blast radius and attribute usage.
Least privilege: use workspace/project scoping and permission restrictions the provider offers.
Rotation: scheduled rotation plus immediate rotation after any suspected exposure; support two active keys during rollover.
Spend limits and alerts at the provider and in your own monitoring.
Secret scanning in pre-commit hooks, CI and repository settings.

15–25 min Deep dive: the backend proxy

Your backend becomes the control point:

Authenticates users with your own auth (sessions, OAuth).
Applies per-user and per-tenant quotas and rate limits.
Adds server-side system prompts and guardrails users can't tamper with.
Logs usage for cost attribution.
Keeps the provider key out of reach.

Loading secrets:

api_key = secrets_client.get("llm/prod/api-key")   # fetched at startup, cached in memory
client = LLMClient(api_key=api_key)                # never logged, never returned in responses

Make sure keys can't leak through error messages, debug endpoints, logs or crash reports.

Bring-your-own-key products: encrypt user keys at rest with a KMS, restrict decryption to the service that needs it, never log them, let users revoke them, and display only the last four characters.

25–32 min Leak response playbook

Revoke the key immediately and issue a new one.
Deploy the new key via the secrets manager.
Review usage logs for unexpected volume, models or regions during the exposure window.
Find the source (commit, log, screenshot, client bundle) and remove it, including from git history if needed.
Add controls that would have prevented it (scanning, alerts).

32–37 min Production example

A developer accidentally committed a key in a test script to a public repository. Secret scanning flagged it within minutes; the key was revoked and replaced before meaningful abuse. Follow-ups: pre-commit scanning on all repos, separate low-limit keys for development, and per-key spend alerts.

37–40 min Likely follow-ups

What about mobile apps? Same answer: call your backend; optionally add app attestation to reduce abuse of your backend.
Short-lived credentials? Use them where providers or cloud platforms support workload identity; they reduce the impact of leaks.

What interviewers listen for

Server-side storage and a backend proxy as the control point.
Scoping, rotation, monitoring and an incident playbook.

Red flags

Keys in front-end code, mobile apps or repositories.