Troubleshooting: slow, out of memory or odd output

Most local-model problems fall into a handful of causes. Work down the list that matches your symptom.

It is very slow

1. Is the GPU being used? In the app, look for a GPU layers or offload setting and make sure it is not zero. Watch GPU memory use while it runs. A CPU-only build, or missing drivers, is the most common cause.
2. Does the model fit? If part of it runs from system RAM, speed drops sharply (offloading). Compare against your hardware on My hardware.
3. Is the context huge? Reading a very long prompt takes time, and a large cache can push the model out of fast memory (two speeds).
4. Is something else using the GPU? Browsers, games and other apps hold video memory.
5. Is the machine throttling? Laptops on battery or with poor cooling slow down under sustained load. Plug in and check temperatures.

It runs out of memory or crashes while loading

• Pick a smaller quantization or model (quantization).
• Lower the context length (why), or quantize the cache (how).
• Close other apps. On a Mac, remember the GPU can use only part of the memory (Macs).
• On a phone or tablet, the system closes an app that uses more than its share of RAM, so the app simply disappears. Pick a smaller file or model (mobile devices).
• Check the memory calculator before downloading.

It works at first, then fails in a long chat

The KV cache grows with every token. A model that fit at the start can run out as the conversation grows. Start a new chat, or reduce the context setting so older text is dropped earlier.

The output is odd

• Repeating, rambling or ignoring instructions. Check you are using an instruct model, not a base model, and that the app picked the right chat template for it. Try a slightly higher quantization.
• Forgetting the start of the conversation. The context window is too small for your input. Raise it.
• Gibberish. A corrupted download, a model newer than your app understands, or very low quantization. Update the app, re-download, and compare the file size with the source.
• Mistakes at very low bit-widths. Below 4 bits, quality drops quickly. Prefer a smaller model at 4 bits.

It will not download or load

• Some models are gated: accept the terms on Hugging Face and sign in (licenses).
• Check free disk space. Large models are tens of gigabytes.
• “Unsupported architecture” usually means the app is too old. Update it.
• Make sure the app can read the file type (formats).
• On a phone or tablet, the app needs one whole .gguf file. A file whose name ends like -00001-of-00003.gguf is one part of a split model and won’t open. If the model doesn’t show up in PocketPal’s search, download the file in the browser and add it with Add Local Model (mobile steps).

Still stuck? Start again from the first-model guide with a small model, to separate a hardware problem from a model problem.