Models read and write in tokens: short words, pieces of longer words, punctuation. As a rule of thumb 100 tokens is about 75 English words. Speed is quoted in tokens per second.
What the numbers feel like
Press play to watch an answer appear at each speed.
| Tokens per second | Feels like | Try it |
|---|---|---|
| 1–3 | Watching it type. Fine for a batch job you walk away from. | |
| 5–8 | About the pace people read. Usable for chat. | |
| 10–20 | Comfortable for chat and for most coding help. | |
| 30+ | Faster than you can read. Good for agents that make many calls in a row. |
On every model, hardware and “can it run” page, the play button beside a speed does the same for that model on that machine, including the wait while it reads a long prompt.
The labels on this site use the same idea. See how the Runs ratings work.
Two speeds, not one
A request has two phases with different limits:
- Prefill reads your prompt. It processes many tokens at once, so it is limited by raw compute. A long document or a coding agent’s big context makes this the part you wait for. It decides the time to first token.
- Decode writes the reply one token at a time. Each token needs the model’s weights read from memory, so it is limited by memory bandwidth. This is the tokens-per-second figure most people quote (why).
That is why a Mac can have a good tokens-per-second figure yet take a while to start answering a very long prompt, while a graphics card with more compute starts sooner.
EXO measured it with a Mac Studio and an NVIDIA DGX Spark: the Spark reads a long prompt about four times sooner, the Mac types about three times faster, and together each does the phase it is best at (clusters).
One Mac Studio6.42 s
Reads the prompt in 5.57 s, then types the answer in 0.85 s.
Spark reads, Mac types2.32 s
The DGX Spark reads the prompt in 1.47 s; the Mac types in the same 0.85 s. 2.8× sooner
If you need it faster
- Choose a smaller model or a lower quantization (quantization).
- Keep the whole model in fast memory (offloading).
- Try a mixture-of-experts model (MoE) or speculative decoding (how it works).
- Shorten the context you really need.
Check any model against your own machine with My hardware.