Independent project. Not a U.S. government website.

Support our project (opens Buy Me a Coffee)
USASI

Jump to

Start hereFour ways into the catalogCompanies & LabsU.S. AI organizationsOpen Models & ToolsOpen models, software, datasets, and evaluation toolsLearnExplainers and learning pathsFor kidsAI and Super Intelligence, explained for readers 8 to 12Latest newsSourced news itemsPolicy trackerFederal AI laws, orders, and standards

Explainer · Using and running AI

Choosing a local model for your computer

Which models can my computer run, and what should I check before downloading one?

Beginner8 min readReviewed Oct 13, 2026

General information, not legal or professional advice. All explainers

How this page was made
  • Researched and written with AI assistance from primary sources, which are listed at the end with the date they were read.
  • Source-checked: a separate AI fact-check pass compared each sentence with its source and corrected what did not match (fact-check report (external site: github.com)).
  • Automated checks passed: links, structure, and formatting are validated before every publish.
  • Not individually reviewed by a person before publication. What these review levels mean

Key takeaways

  • Start from your memory: a graphics card's VRAM, an Apple silicon Mac's unified memory, or system RAM for CPU-only use, then compare a published figure for the exact file.
  • Read what a figure covers: Google's Gemma 4 table counts only the weights, not the context window or supporting software, and a longer context needs more memory.
  • Read the license for the exact release, then test speed and answers yourself; a model that fits can still run slowly, and USASI does not benchmark or rank models.
On this page
  1. Step 1: Find out which memory you have
  2. Step 2: Why memory sets the limit
  3. Step 3: What quantization trades
  4. Step 4: Read the published figure, and what it leaves out
  5. Step 5: Match a runtime to your hardware
  6. Step 6: Check the license before you use it
  7. Where the USASI finder and memory pages fit
  8. Fitting is not the same as running well
  9. Worked example: a desktop with an 8 GB graphics card
  10. What you can do next

What you can run depends first on memory: a graphics card's own memory (VRAM) on a PC, the unified memory that the processor (CPU) and graphics processor (GPU) share on an Apple silicon Mac, or system RAM if the CPU does the work alone. Find a published memory figure for the exact file you would download, including its quantization (how many bits store each weight), and note what the figure leaves out, such as the context window. Check that your runtime, the app that loads the model, supports your hardware, and read the license before you rely on the model. A model that fits can still run slowly. Speed and quality vary with hardware, settings, and task, and USASI does not benchmark, test, or rank models. This page is general information.

Step 1: Find out which memory you have

Step 2: Why memory sets the limit

The model's weights have to fit, plus working memory that grows with the conversation. Ollama's context length page (external site: docs.ollama.com) says "Setting a larger context length will increase the amount of memory required to run a model." Its default context depends on VRAM: 4k tokens below 24 GiB, 32k from 24 to 48 GiB, and 256k at 48 GiB or more. Its FAQ (external site: docs.ollama.com) says required RAM scales with the number of parallel requests times the context length.

When a model does not fit on the GPU, runtimes can split it: llama.cpp offers "CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity," and Ollama's ollama ps command shows where a model loaded: "100% GPU," "100% CPU," or a split such as "48%/52% CPU/GPU." Splitting costs speed. OpenAI's guide to running gpt-oss with Ollama (external site: developers.openai.com) says you can offload to the CPU when short on VRAM, "but expect it to run slower," and Ollama advises avoiding CPU offload for best performance.

Step 3: What quantization trades

LM Studio's download guide (external site: lmstudio.ai) calls the Q options for a model "copies of the same model, provided in varying degrees of fidelity": quantization compresses the files "while giving up some degree of quality." It suggests "a 4-bit option or higher if your machine is capable enough for running it." llama.cpp's quantize README (external site: github.com) gives sizes for Meta's Llama 3.1 8B: 14.96 GiB at 16 bits (F16), 7.95 GiB for Q8_0, and 4.58 GiB for Q4_K_M. It says quantization "may introduce some accuracy loss," and warns that requantizing an already quantized file can severely reduce quality compared with starting from 16 or 32 bits. Quantization explains the formats.

Step 4: Read the published figure, and what it leaves out

A figure is only as useful as its assumptions. Google's Gemma 4 overview (external site: ai.google.dev) (last updated July 8, 2026) gives the approximate GPU or TPU memory needed to load each model, including a "20% overhead of loading additional things." For Gemma 4 12B, it lists 26.7 GB at BF16 (16-bit), 13.4 GB at SFP8 (8-bit), and 6.7 GB at Q4_0 (4-bit). The page says these estimates "only account for the memory required to load the static model weights," without the VRAM needed for supporting software or the context window. Memory use grows with the total number of tokens in your prompt and the response, it adds, and the numbers may change with your inference tool and environment.

Other figures are framed differently:

Step 5: Match a runtime to your hardware

Runtime requirements change, so check the current page:

Hugging Face's GGUF documentation (external site: huggingface.co) describes GGUF as a file format that "is designed for use with GGML and other executors," and lists llama.cpp, LM Studio, GPT4All, and Ollama under usage with open-source tools.

Step 6: Check the license before you use it

Read the license for the exact release you download, whoever made the copy. Three examples:

For a quantized copy uploaded by someone else, Hugging Face's model card documentation (external site: huggingface.co) says the base_model field can name the original, with the relationship marked "quantized"; follow it back to the original license. A gated download (external site: huggingface.co) asks you to share your username and email address with the authors. USASI's license guide summarizes common licenses.

Where the USASI finder and memory pages fit

  • The finder checks a small set of documented configurations against what you enter, explains why each matches or not, and shows what is unverified. "Estimated to fit" means a publisher's stated figure for one exact variant is within the amount you entered, not that USASI ran it. Where no figure is published, the result says the evidence is insufficient.
  • Local AI by memory applies the same rule to fixed amounts: 8, 12, 16, and 24 GB of VRAM, and 8, 16, 24, 32, and 64 GB of Apple silicon unified memory. The VRAM pages compare only publishers' graphics-memory figures; the Apple silicon pages compare unified memory with graphics-memory and system-memory figures alike.
  • The model size calculator shows the storage arithmetic for weights at 32, 16, 8, or 4 bits each, and what it leaves out.
  • USASI does not benchmark or rank models, and no record or finder entry claims a hands-on test (review levels).

Fitting is not the same as running well

Speed depends on more than memory. llama.cpp's performance troubleshooting notes (external site: github.com) show one 4-bit, 30-billion-parameter model on one machine generating anywhere from under 0.1 to 9.1 tokens per second, depending only on its thread-count and GPU-offload settings. Quality depends on the model, the quantized file, and your task. Test with your own prompts, and record the runtime version, the exact file, the context length, where the model loaded, and the date.

Worked example: a desktop with an 8 GB graphics card

Jo is a fictional reader invented for this page, and the choices are Jo's, not a recommendation. Task Manager shows Jo's Windows desktop with 8 GB of dedicated GPU memory and 32 GB of system RAM; Jo wants a local model for drafting and summarizing. Steps are labeled Stated (by a publisher), Estimated (Jo's reasoning), or Tested (what Jo measures). Checked October 13, 2026; USASI did not run either model.

  1. First candidate (Stated). OpenAI's Ollama guide says gpt-oss-20b is best with at least 16GB of VRAM or unified memory. Jo has 8 GB.
  2. Estimated. It would probably be split between GPU and system memory, which the same guide says runs slower.
  3. Second candidate (Stated). Google's table lists 6.7 GB to load Gemma 4 12B at Q4_0, weights only. Google's Hugging Face repository (external site: huggingface.co) offers a Q4_0 GGUF of the instruction-tuned model, made with quantization-aware training: a 6,975,879,296-byte file (about 7.0 GB), plus a smaller mmproj file.
  4. Estimated. That leaves little of the 8 GB for the context window and other software, so Jo plans a short context, expects part of the model may run on the CPU, and notes Gemma 4 E4B (4.5 GB at Q4_0 in the same table) as a fallback.
  5. License (Stated). The repository's card lists Apache 2.0.
  6. Runtime (Stated). LM Studio on Windows needs AVX2 on x64 processors, so Jo confirms the processor has it.
  7. Tested. Jo runs ten typical prompts and writes down the LM Studio version, the context length, how long answers take, whether they are usable, and the date.

Jo's outcome: start with Gemma 4 12B at Q4_0 and a short context, and switch to the smaller model if answers are too slow. Another model whose published figure fits could serve as well.

What you can do next

Sources

All read on October 13, 2026.

Share or cite this page

Cite this page

APA style

USASI. (2026, October 13). Choosing a local model for your computer. United States of America Superintelligence. https://unitedstatesofamericasuperintelligence.com/learn/choosing-a-local-model/

BibTeX

@misc{usasi_learn_choosing_a_local_model,
  author = {{USASI}},
  title = {Choosing a local model for your computer},
  year = {2026},
  month = oct,
  howpublished = {\url{https://unitedstatesofamericasuperintelligence.com/learn/choosing-a-local-model/}},
  note = {United States of America Superintelligence. Last updated 2026-10-13}
}

The date is when this page was last updated. Add the date you read it if your style needs one.

Keep learning

Part of Choosing and running a model.

Support Us

Help keep USASI useful.

Find the catalog useful? Leave an optional tip to support its upkeep. Tips never affect listings, coverage, or openness assessments.

Leave a tip on Buy Me a Coffee (external site)

Optional. No USASI account required. Payment takes place on the linked provider’s website (Buy Me a Coffee).

About supporting this project