Why VRAM Is the Bottleneck, Not the GPU Core
A language model only runs at full speed when its weights fit entirely inside the GPU’s memory. Those weights are billions of numbers, and generating a single token means the GPU needs all of them close at hand. The moment a model doesn’t fit and part of it spills into system RAM or onto disk, throughput doesn’t just dip — it collapses. A response that took a second now takes a minute.
That’s why picking hardware for local AI starts with video memory, not CPU cores or how much system RAM is installed. VRAM sets the ceiling: it decides which class of model you can run at all, full stop.
The Rule of Thumb: GB of VRAM ≈ Billions of Parameters
Rough rule for compressed (quantized) models: figure on roughly as many gigabytes of VRAM as the model has billions of parameters, plus some room for context. A 32-billion-parameter model in typical compression runs about 18-24 GB. The exact figure moves with compression level and context length, but that’s the ballpark you should plan around.
VRAM Requirements by Model Class
Ballpark figures for quantized models, the form they actually run in production:
| Model class | VRAM | What it handles |
|---|---|---|
| 7-9B | 6-8 GB | Chat, summarization, classification, simple knowledge-base answers. |
| 12-14B | 9-12 GB | Solid document work, emails, drafts, basic code. The practical floor for business use. |
| 27-32B | 18-24 GB | Contract review, complex instructions, vision, long context. The sweet spot for most businesses. |
| 70B | 40-48 GB | Close to top-tier cloud models. Needs two cards or a professional accelerator. |
| 100B+ (MoE) | 80 GB and up | Best-in-class reasoning. Server-grade accelerators, dedicated power. |
Beyond the Model: Vision, Speech, and Search
If the server needs to do more than answer in plain text, budget memory for the extras too:
- •Vision (documents, photos, scans) — usually the same model handles it, or budget +2-6 GB for a dedicated one,
- •Speech recognition (call transcription) — 2-10 GB, and it’s often offloaded to the CPU instead,
- •Knowledge-base search (RAG) — about 1-2 GB for the embedding model.
The Real Trap: Multiple Users at Once
The figures in the table above cover the model itself for one or two concurrent users. Every active request also eats VRAM for its own context, known as the KV cache, and with dozens of people hitting the server at once, that memory disappears a lot faster than the spec sheet suggests.
This is exactly where DIY builds fall apart: a model that flew through single-user testing grinds to a halt with fifteen employees on it. Plan memory for the multi-user case with real headroom, and configure queuing and batching on the inference engine. The rule is simple: size VRAM for peak load, not for the model alone.
Which GPU Memory Size Fits Which Job
| VRAM | Comfortable to run | Typical scenario |
|---|---|---|
| 24 GB | up to 14B freely, 27B compressed | team of 5-15, everyday tasks |
| 32 GB | 27-32B plus vision and speech | workhorse for 15-50 people |
| 48+ GB (2 cards) | 70B models, several models at once | busy departments, 50-150 people |
| 80 GB and up | 100B+ MoE, redundancy | 150+ people, data-center requirements |
Turnkey Server Configurations
To connect memory to budget, here’s a ballpark for a turnkey server: hardware, build, model setup, and configuration included.
| Configuration | VRAM | Users | Turnkey price |
|---|---|---|---|
| Starter | 1 card · 24 GB | 5-15 people | from ₽450,000 (about $5,000) |
| Workhorse | 1 card · 32 GB | 15-50 people | from ₽800,000 (about $9,000) |
| Advanced | 2 cards · 48+ GB | 50-150 people | from ₽2,000,000 (about $22,000) |
| Maximum | server-grade 80 GB+ | 150+ people | from ₽5,000,000 (about $55,000) |
For the call between a desktop tower and a rack-mount build, and what actually goes into the setup, see our guide to on-premise AI servers.
Bottom Line
Video memory is the ceiling on your local AI, full stop. For most companies, 24-32 GB is the sweet spot: enough for a 27-32B workhorse model with vision and speech built in. The mistake to avoid is sizing memory tight to the model alone — leave room for context and concurrent users, or the server will crawl the moment real traffic hits it. The right number for your headcount and use case is worth running past people who’ve actually built these servers before.
Frequently Asked Questions
How much VRAM do I need for a 30B-parameter model?
Plan on 18-24 GB in typical Q4-Q8 quantization, plus a few extra gigabytes for context length. That range is where most mid-sized businesses land: it handles contract review, document analysis, and vision tasks with room to spare.
Can a 24 GB GPU run a local LLM well?
Yes. 24 GB handles up to 14B models with headroom and runs 27B models in compressed form. It's a solid floor for a team of 5-15 people on everyday tasks, but it won't leave much room for vision, speech, or a lot of simultaneous users.
Why does a model that runs fine for one user slow down for a whole team?
Every active request adds its own context footprint in VRAM, called the KV cache, on top of the model's base memory. A card sized only for the model itself runs out of room fast once a dozen or more people are using it at once. Size the GPU for peak concurrent load, not just for the model.