What Hardware Do You Need to Run an LLM Locally?

Memory decides what you can run, and memory bandwidth decides how fast it runs. Once you understand those two facts, most hardware advice becomes much easier to judge.

What Hardware Do You Need to Run an LLM Locally? — Troiana insight cover

In short

As a rule of thumb, budget about 1.2 GB of memory per billion parameters at 4-bit quantisation, plus extra for context and 1–2 GB of headroom. So 8 GB handles small models up to about 7B parameters, 16 GB handles around 13B, 24 GB handles around 32B, and 70B needs 48 GB or more. A GPU's memory runs fastest; Apple's unified memory fits larger models at lower speed.

The two numbers that matter

Capacity determines whether a model loads. Bandwidth determines how fast it generates once loaded. That's why a large model on a machine with plenty of ordinary system memory can load fine and then crawl, and why GPUs with less memory can feel much quicker on models that fit.

Don't start by choosing a graphics card. Start by deciding which models you want to run.

The sizing formula

Memory needed is roughly:

quantised model size + context memory + 1–2 GB of headroom

Community guides commonly use around 1.2 GB per billion parameters at Q4 quantisation, as a working shortcut. A similar rule from other guides gives about 5 GB for a 7B model, 9 GB for 13B, 22 GB for 34B and 40 GB for 70B. These are estimates, so check the actual file size of the model you want, and remember that your operating system and other apps need memory as well.

Rough memory needed at Q4, before context and headroom
7B modelAbout 5 GB
13B modelAbout 9 GB
34B modelAbout 22 GB
70B modelAbout 40 GB
0 GB20 GB40 GB
The figures from one guide cited above, and they are estimates. Check the actual file size of the model you want.

Quantisation is what makes this possible. It stores model weights at lower precision, so the file is smaller, with some loss in quality. Q4 is a popular middle ground. Lower levels save more memory, while higher ones keep more quality.

Context length also uses memory. The cache that stores your conversation grows with context. One guide's rough rule is about 1 GB for every 10B parameters at 8K tokens of context, so a long context can push a model that fits over the limit. If you get an out-of-memory error, reducing context is usually the first thing to try. See what a context window is.

What fits at each memory tier

Rough guidance at Q4, leaving headroom for context:

Usable memoryModel size that's comfortableGood for
8 GBUp to about 7BSummaries, simple chat, classification, learning
16 GBAround 13BEveryday assistants, drafts, light coding help
24 GBAround 32BBetter reasoning and coding; useful context
48 GBAround 70BStrong general models with longer context
96 GB or moreLarger models, or several at onceHeavy experimentation; speed can become the limit

Models with the same parameter count can differ a lot in quality, so a newer small model may beat an older large one. Check recent benchmarks for your type of task, and test.

GPU, Apple Silicon or CPU?

A dedicated GPU generally gives the fastest generation, thanks to high memory bandwidth, as long as the whole model fits in its VRAM. Once part of the model spills over into system memory, speed can fall dramatically. NVIDIA remains the best-supported option in most software. AMD and Intel support is improving, so check the specific tools you plan to use.

Apple Silicon uses unified memory shared between CPU and GPU, which means that a high-memory Mac can load models that wouldn't fit on a typical consumer graphics card. Speed varies by chip, because Pro, Max and Ultra chips have very different memory bandwidth. It's typically slower than a high-end GPU on a model that fits both, but it's quiet, efficient and simple. See how to run an LLM locally on a Mac.

CPU only works for small models and for batch jobs where waiting doesn't matter. Fast RAM helps. For interactive use with anything but small models, it's generally frustrating.

Other parts of the machine

  • Storage: models are large, and a few can use hundreds of gigabytes. Use an SSD, and leave space.
  • Power and cooling: long agent runs keep a GPU at sustained load, so a laptop may throttle.
  • System RAM: even with a GPU, have enough system memory to load the model and run your other software.

Mistakes to avoid

Buying for a model that may be obsolete. Open models improve fast, and small models keep getting better. Buy for what you need now, with reasonable headroom, not for what you hope to run in three years.

Ignoring speed. A model that loads but generates a few words per second may be unusable for coding or chat. Look for tokens-per-second figures on hardware like yours, with the same model and quantisation.

Assuming parallel GPUs simply add up. Multiple cards can work, but setup is more complicated, and you need software and a motherboard to match.

Counting MoE models by their active size. One guide notes that mixture-of-experts models need their full parameter count loaded, even though only part is active per token.

Try before you buy

You can test without buying anything:

  1. Rent a GPU by the hour. RunPod lists an RTX 4090 from $0.34 an hour and an A100 80 GB from $1.19. An afternoon of testing costs very little.
  2. Run a small model on what you already own. Install LM Studio or Ollama, and see how far you get.
  3. Use a hosted version of the same model to judge its quality, before you pay for hardware to run it.

Is buying worth it at all?

If your main reason is saving money, do the maths first. At today's prices, budget APIs are remarkably cheap, and a GPU is expensive when it's mostly idle. Our analysis of the cheapest way to run an LLM works this through. Buy hardware when you need privacy, offline work or heavy steady use, or if you simply enjoy it. Those are perfectly good reasons.

If you're planning local AI for a team or product, book a call and we can help you scope it.

Common questions

How much RAM do I need to run an LLM locally?

As a rule of thumb, allow about 1.2 GB per billion parameters at 4-bit quantisation, plus memory for context and 1–2 GB of headroom. That makes 16 GB suitable for roughly 13B models and 48 GB for about 70B models.

Can I run an LLM without a GPU?

Yes, particularly small models on recent CPUs with fast memory, and tools like llama.cpp support this. It will be slower, so CPU use suits short tasks and batch jobs better than interactive coding or long conversations.

Is a Mac or a PC better for local LLMs?

A PC with a high-end GPU is usually faster on models that fit in its VRAM. A Mac with lots of unified memory can load larger models more simply and quietly, typically at lower speed. Test your target model on each if you can.

Why does my model run so slowly?

It probably doesn't fit fully in GPU memory, so part of it runs on the CPU. Try a smaller quantisation or model, reduce the context length, and check that GPU offload is enabled and the right backend is in use.

Have something worth building right?