Design a multi-tenant platform where customers fine-tune and serve their own LoRA models.

30-second answer

Functional: dataset upload and validation, LoRA jobs with good defaults, loss curves, automatic evaluation against the base model, deployment by model ID, deletion and audit. Non-functional: strict tenant isolation, low idle cost, fine-tuned latency within ~10% of base. A typical 8B job (30M tokens) needs ~4N FLOPs/token, about 40 minutes on one H100. Design: tenant-keyed encrypted storage, sandboxed training containers on a fairly scheduled GPU pool, an eval and safety gate, a registry with lineage, and multi-LoRA serving that batches different adapters over shared base weights with GPU/host/object-store adapter caches and adapter-affinity routing.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Functional requirements

You Who are the tenants: internal teams or external customers?

Interviewer External customers of our AI platform. They upload data, fine-tune one of our base models and call the result through our API.

You Full fine-tuning or parameter-efficient?

Interviewer LoRA by default. Full fine-tuning might come later for big customers.

You Which base models?

Interviewer An 8B and a 70B model.

You Do customers need to see evaluation results before deploying?

Interviewer Yes, they need to know whether it got better.

#	Functional requirement	Why it matters
F1	Upload and validate training datasets (chat-format JSONL), with clear errors	Most failed jobs are data problems
F2	Launch LoRA fine-tuning jobs on supported base models with sensible defaults and limited hyperparameters	Non-experts succeed; support load stays low
F3	Job status, training and validation loss curves, cancellation	Visibility and control
F4	Automatic evaluation: held-out loss, base vs fine-tuned comparisons on customer-provided or sampled prompts	Customers decide whether to deploy
F5	Deploy a fine-tuned model to the same inference API by model ID	One API for base and fine-tuned models
F6	Versioning, deletion of data and models, audit logs	Compliance and customer trust

Out of scope: full fine-tuning, RL fine-tuning, customers bringing their own base models.

5–10 min Non-functional requirements and estimates

You How sensitive is the data?

Interviewer Very. Enterprise customers, some with strict requirements. Their data must never be used for anything else.

Requirement	Target	What it forces
Isolation	No tenant can access another's data, adapters or outputs	Per-tenant encryption keys, scoped credentials, isolated job containers
Data handling	Training data used only for that tenant's job; deletable on request	Lineage tracking, retention policies, deletion workflows
Job latency	Typical 8B job finishes within ~2 hours of submission	Queueing and capacity sized for daily demand
Serving	Fine-tuned models add ≤ 10% latency vs the base model; thousands of adapters deployable	Multi-LoRA serving on shared base pools
Safety	Fine-tuning must not produce models that bypass usage policies	Dataset screening and post-training safety evals
Cost	Serving an idle fine-tuned model costs almost nothing	No dedicated GPUs per adapter

Back-of-envelope estimates:

Demand: 500 active tenants, ~200 jobs/day; 80% on the 8B model.
Typical job: 10k examples × ~1k tokens × 3 epochs = 30M training tokens.
Compute: LoRA training costs about 4N FLOPs per token (the frozen weights need no weight gradients, saving roughly a third of 6N). For 8B: 4 × 8e9 × 30M ≈ 9.6e17 FLOPs; at ~40% of an H100's 989 TFLOP/s that's ~40 minutes on one GPU. For 70B: 8.4e18 FLOPs, ~45 minutes on 8 GPUs (FSDP or QLoRA).
Daily GPU time: 160 × 0.7 + 40 × 6 ≈ 350 GPU-hours/day, an average of ~15 GPUs, with peaks during business hours: a training pool of ~24–32 GPUs plus a queue.
Adapter size: rank 16 on all linear layers of an 8B model ≈ 42M parameters ≈ 84 MB in bf16. 5,000 deployed adapters ≈ 420 GB, so they live in object storage and are cached in host and GPU memory.

10–18 min High-level design

You Two planes. A training plane that validates data, schedules jobs on a GPU pool, trains, evaluates and registers adapters. A serving plane that hosts base models and loads adapters on demand. A shared control plane handles tenants, keys, metadata and audit.

Scroll sideways to see the whole diagram.

Multi-tenant LoRA fine-tuning platform. Training plane: validated, tenant-encrypted datasets feed a fair job scheduler; sandboxed workers train LoRA adapters, which are evaluated for quality and safety before registration. Serving plane: base-model pools batch requests for many adapters at once, loading adapters through GPU, host and object-store caches.

Components:

API and console: datasets, jobs, models, deployments; tenant authentication and quotas.
Dataset service: uploads to tenant-scoped encrypted storage; validation (schema, roles, token counts, duplicates); policy screening for disallowed content.
Job scheduler: queue with per-tenant fairness and priorities; packs jobs onto GPUs (8B jobs on single GPUs, 70B jobs on full nodes); supports preemption and resume from checkpoints.
Training workers: isolated containers per job with short-lived credentials scoped to that tenant's data; LoRA training with standard recipes; checkpoints to tenant storage.
Evaluation service: held-out loss, base vs fine-tuned comparisons with LLM-judge win rates, format checks, and a safety evaluation suite.
Model registry: adapter versions with lineage (dataset hash, base model version, hyperparameters, eval results).
Multi-LoRA serving: base-model pools with batched multi-adapter kernels and adapter caches; router with adapter affinity.

A job, end to end:

Scroll sideways to see the whole diagram.

A fine-tuning job, end to end. The dataset is validated and screened before any GPU time is spent. The job runs in a sandbox with credentials scoped to this tenant's data, checkpoints regularly, and is evaluated against the base model and a safety suite. Only adapters that pass are registered and deployable.

API sketch:

POST /v1/files                    (purpose=fine-tune) → { file_id }
POST /v1/fine_tuning/jobs         { base_model, training_file, validation_file?, hyperparameters? } → { job_id }
GET  /v1/fine_tuning/jobs/{id}    → status, metrics, eval_report, fine_tuned_model
POST /v1/chat/completions         { model: "ft:base-8b:acme:support-v3", … }
DELETE /v1/models/{id}            → adapter and derived artefacts deleted

18–32 min Deep dive

Interviewer How do you serve thousands of fine-tuned models without thousands of GPUs?

Deep dive 1 · Multi-LoRA serving

Scroll sideways to see the whole diagram.

Serving many LoRA adapters on one base model. Requests for different adapters share one batch. The base matmuls run once for the whole batch; each request's low-rank update is added with segmented kernels. Adapters are fetched on demand from host memory or object storage into a GPU cache.

Shared base weights: one base-model pool serves all adapters for that base. Each request carries an adapter ID.
Batched heterogeneous adapters: within one batch, requests can use different adapters. The base matmul runs once for the whole batch; the low-rank paths x·Aᵀ·Bᵀ are computed per adapter with segmented gather kernels (Punica/S-LoRA style). Overhead is small for ranks of 8–64.
Adapter caching tiers: GPU memory (hot adapters, e.g. 100 × 84 MB ≈ 8 GB reserved per GPU), host memory (thousands), object storage (all). Loading from host takes tens of milliseconds; from object storage, a second or more for the first request.
Routing with adapter affinity: the router sends an adapter's requests to replicas that already hold it, spreading very popular adapters across more replicas.
Large tenants: a heavily used adapter can be merged into a dedicated replica's weights for zero overhead, if the tenant pays for dedicated capacity.

Interviewer And isolation on the training side?

Deep dive 2 · Isolation and data governance

Encryption: each tenant's datasets, checkpoints and adapters are encrypted with a tenant-specific key; services get decrypt permission only for the job they're running.
Job sandboxing: one job per container with no network access except to its own storage paths; GPUs are wiped (memory cleared) between jobs; no shared filesystems across tenants.
Lineage and deletion: the registry records which dataset versions produced which adapters. Deleting a dataset can require deleting derived adapters, according to the customer's choice and contract; deletion is propagated to caches and logged.
No cross-tenant training: tenant data is never used for base-model training; access by operators requires break-glass approval and is audited.
Adapter outputs are tenant-scoped: only the owning tenant's keys can call its model IDs.

Deep dive 3 · Training quality and safety

Data validation: schema checks, token-length limits, minimum example count, duplicates, label leakage between train and validation splits, and warnings for issues like assistant turns that are all identical.
Good defaults: LR ~1e-4 for LoRA, rank 16, all linear layers, epochs chosen by dataset size, early stopping on validation loss. Most customers shouldn't need to tune hyperparameters.
Evaluation report: validation loss curve; base vs fine-tuned pairwise judge win rate on held-out prompts; format adherence; examples side by side. Warn when the fine-tune is worse than the base model.
Safety: fine-tuning on even a small amount of harmful data can remove a model's refusal behaviour. Screen training data with policy classifiers, run a safety evaluation suite on each adapter (harmful-request refusal), block deployment on failure, and keep runtime moderation on fine-tuned outputs as well.

Deep dive 4 · Scheduling and cost

Pack small jobs: an 8B LoRA job fits on one GPU; schedule by estimated GPU-hours (from token counts) with per-tenant fairness so one customer's 50 jobs don't block others.
Preemptible jobs checkpoint every ~10 minutes so the scheduler can reclaim GPUs for urgent work or serving spikes.
Use spare serving capacity at night for training when the fleets are shared.

32–37 min Trade-offs

Decision	Chosen	Alternative	Why
Fine-tuning method	LoRA by default	Full fine-tuning	Cheap training, tiny artefacts, shared serving; full FT for large shifts later
Serving	Multi-LoRA on shared base pools	Dedicated replica per model	Idle fine-tunes cost nearly nothing; small per-request overhead
Hyperparameters	Opinionated defaults, few knobs	Full control	Fewer failed jobs and support tickets
Isolation	Per-tenant keys, sandboxed jobs	Shared storage with ACLs	Stronger guarantees for enterprise customers
Safety	Pre-deploy safety evals plus runtime moderation	Trust customer data	Fine-tuning can remove safety behaviour
Evaluation	Automatic report with judge comparisons	Loss curves only	Customers need to know whether it improved their task

Build first: dataset validation, the 8B LoRA path with default hyperparameters, multi-LoRA serving, and the evaluation report. Then the 70B path, safety gating, and dedicated capacity options.

37–40 min Wrap-up and likely follow-ups

Close with: "Tenant-isolated data and jobs, LoRA training with strong defaults on a shared GPU pool, an automatic evaluation and safety report, and serving thousands of adapters from shared base-model pools with batched multi-adapter kernels and tiered adapter caches. An 8B job takes about forty minutes of one GPU, and an idle fine-tune costs almost nothing to keep deployed."

Likely follow-ups:

What happens when the base model is upgraded? Adapters are tied to a base version; offer retraining on the new base with the same dataset, and keep the old base serving for a deprecation period.
How do you support full fine-tuning? Dedicated replicas per model, priced accordingly; same pipeline otherwise.
How do you stop a customer from extracting other tenants' data? There's no shared training data or adapter state; isolation is structural, not just access control.
Can customers bring preference data? Add DPO jobs with the same pipeline; evaluation compares against the SFT adapter.

What interviewers listen for

Compute and storage estimates that drive the design.
Multi-LoRA serving mechanics and caching tiers.
Isolation and data governance treated as core requirements.
Evaluation and safety gating before deployment.

Red flags

A dedicated GPU replica per fine-tuned model.
No data validation or evaluation.
Ignoring that fine-tuning can remove safety behaviour.