Hardware · Memory

How much memory does a model need?

Pick an LLM and a context length. We add up the weights at every quantization, the KV cache that grows with context, and runtime overhead, then show the smallest systems and devices that hold it.

Model

Context length

Snowflake Arctic Instruct at 4K context

This model tops out at 4K tokens, so that is what is shown.

Quantization	Weights	KV cache + overhead	Total	Smallest hardware that holds it
`Q2_K` Low	191 GB	1.2 GB	193 GB	Apple M3 Ultra (512 GB) 12 in all
`Q3_K_M` Low	235 GB	1.2 GB	236 GB	Apple M3 Ultra (512 GB) 8 in all
`Q4_K_M` Good	292 GB	1.2 GB	293 GB	Apple M3 Ultra (512 GB) 6 in all
`Q5_K_M` High	340 GB	1.2 GB	341 GB	Apple M3 Ultra (512 GB) 6 in all
`Q6_K` High	392 GB	1.2 GB	394 GB	AMD EPYC 7002 / 7003 series NAS (512 GB) Intel Xeon D-2700 series NAS (512 GB) Intel Xeon Silver (4100 to 4300 series) NAS (512 GB) 4 in all
`Q8_0` Very high	507 GB	1.2 GB	508 GB	Nothing in the catalog holds this on one machine.
`F16` Full	957 GB	1.2 GB	958 GB	Nothing in the catalog holds this on one machine.

Hardware is listed when its usable memory is at least 8% above the total. Macs count roughly two-thirds to three-quarters of unified memory as usable. Each link opens the full “can it run” breakdown with speed.

The rule of thumb

1 · WeightsParameters × bytes eachA 70B model at 4 bits per weight is about 70 × 0.6 ≈ 42 GB. Lower quantization shrinks it and costs some quality.

2 · KV cacheGrows with contextEvery token you keep in context stores keys and values for each layer. Doubling context doubles this part, so long documents cost real memory.

3 · OverheadLeave headroomThe runtime, the operating system and other apps need memory too. Aim for a total at least 10% under the memory your GPU or device can give a model.

Estimates come from published hardware specs and our open formulas, not from running the model on your machine. Real speed varies with runtime, drivers and settings. Numbers marked “Measured” come from a linked benchmark.