Your GPU Determines Your LLM
The Benchmark-Driven Hardware Guide to Choosing a Local Model
Figure. How hardware tiers map to the class of open-weight model you can realistically run in mid-2026.
Introduction: the only local-LLM guide you should need
I want to make a promise and then earn it. By the end of this issue you should not need another local-LLM explainer. You will know which open-weight model to run on the hardware sitting in front of you, why that is the right answer, and roughly what performance and cost to expect. That is the whole job of this article, and I have tried to write it so you can read it once and act on it.
Here is the thing almost every other guide gets wrong. They rank models by parameter count and call it a day. They tell you the 1T-parameter mixture-of-experts model is the best, which is true and useless if you have a 12GB graphics card. This guide ranks models by practical performance on real hardware, because that is the decision you are actually making.
I should be honest about where this comes from. I have not personally racked these GPUs or run every model in production. What I have done is analyze benchmark leaderboards, hardware-throughput databases, community benchmark threads, research papers, and the deployment practices that vendors and engineering teams publish. Everything below is sourced. When I give you a tokens-per-second number or a benchmark score, it traces back to a public dataset, and the references at the end let you check my work.
HOW TO READ THIS ISSUE
Find your hardware tier first, then your use case. The tier tells you which models physically fit and run at a usable speed. The use case tells you which of those models is strongest for your actual work. Everything else here is the evidence behind those two decisions.
The core thesis: there is no universally best local LLM
Let me state the argument plainly, because the entire issue revolves around it. There is no single best local LLM. The best model is a function of your constraints, and the constraints that dominate are almost always hardware, not raw quality.
The variables that actually decide the answer are your available VRAM, your system RAM, your CPU capability, the shape of your inference workload, your latency requirements, your cost ceiling, and the quality bar your application needs to clear. A model that is a joy to use on a 96GB workstation can be unusable on a 12GB GPU. The same weights, the same benchmark scores, a completely different experience.
So the goal is not to find the strongest model. The goal is to find the strongest model your hardware can run efficiently. That reframing is the most valuable thing in this article, and the local-benchmark communities back it up: hardware constraints routinely matter more than where a model sits on a leaderboard.
THE ONE-SENTENCE VERSION
Pick the highest-quality model that still hits a usable tokens-per-second on the hardware you own. Quality you cannot run at speed is not quality you have.
Why choosing a local LLM is harder than ever
Five years ago there were a handful of models and the choice made itself. In 2026 the problem inverted. The open-weight field exploded, and abundance turned into paralysis. Let me break down why selection has quietly become a hardware-optimization problem rather than a quality-ranking problem.
Too many models. Between early April and mid-May 2026 alone, Moonshot shipped Kimi K2.6, Z.ai shipped GLM-5.1, and several frontier-class open-weight models landed in roughly thirty days, according to Codersera’s tracking. The answer to “what is best” changes monthly.
Hardware fragmentation. You might be on an 8GB laptop GPU, a 24GB desktop card, a 32GB RTX 5090, a 128GB DGX Spark, or a 192GB Mac Studio. Each of those is a genuinely different deployment target with different winners.
Quantization options. Every model now ships in a dozen quantization formats. The same 70B model is a different animal at FP16, Q8, Q4_K_M, or Q3. Quantization is the single lever that decides whether a model fits your memory at all.
Benchmark confusion. MMLU is saturated and close to meaningless now. The benchmarks that matter (GPQA Diamond, SWE-bench Verified, AIME, LiveCodeBench) each crown different winners, so a single score tells you very little.
Marketing versus reality. Vendor-reported numbers run ahead of independently verified ones. Ring-2.6-1T, for example, posted strong vendor figures that no neutral third party had yet confirmed, which is exactly the kind of claim you should discount until it is reproduced.
Put those together and you can see why I keep insisting on a hardware-first frame. The model question is downstream of the silicon question.
Figure. Single-stream generation speed by hardware and model size (Q4). Notice how throughput falls off a cliff as parameters grow.
The benchmark metrics that actually matter
Before we rank anything, we need a shared vocabulary for performance. These are the numbers I trust, and the traps to avoid with each.
Tokens per second (generation speed). How fast the model produces output. A conversation feels fluid above roughly 15 tokens per second, and genuinely interactive above 30, per community benchmarking guides. Below 15 you are in batch-only territory.
First-token latency (TTFT). How long before the first word appears. This is dominated by prompt processing, which is a compute-bound step, so it behaves differently from generation speed. For agents with huge prompts, TTFT is often the real bottleneck.
Context length. How much the model can attend to at once. Long context is wonderful and expensive: a longer window uses more memory and slows generation, so a 10M-token context is a capability, not a free lunch.
Memory consumption. The gate that decides everything. If the weights plus KV cache do not fit in memory, you spill to system RAM over PCIe and your throughput collapses.
Reasoning and coding benchmarks. GPQA Diamond and AIME for reasoning and math, SWE-bench Verified, HumanEval, and LiveCodeBench for code. Use the one that matches your workload, not the one with the biggest headline number.
Cost efficiency. Tokens per dollar, all-in. This is where local inference either wins decisively or loses quietly, and we will spend a whole section on it.
Why are some benchmarks misleading? Because saturation hides differences. When five models all score in the high 80s on MMLU, the benchmark has stopped discriminating. The modern leaderboards (Vellum, Artificial Analysis, LMSYS Chatbot Arena) deliberately feature non-saturated benchmarks and blind human comparisons for exactly this reason.
Figure. Reasoning and math leaders among open-weight models. Different tasks, different champions.
Figure. Coding ability is not a single number. SWE-bench, HumanEval, and LiveCodeBench disagree about who wins.
QUANTIFIED PERFORMANCE DELTA
Moving from FP16 to Q4_K_M typically buys a 3.5x to 4.0x speedup at a 1 to 3 percent perplexity cost, which is below the noise floor on most evaluations. That single trade is what turns an unusable model into a usable one. Source: Presenc AI quantization data, 2026.
Hardware tiers that actually matter
This is the spine of the guide. I have grouped real hardware into five tiers, and for each one I will tell you what to run, what to expect, and where it breaks. The throughput numbers are midpoints of published bands; treat them as the shape of reality, not a warranty.
Tier 1: no GPU, CPU-only systems
Laptops, office PCs, mini PCs. This tier is more capable than people expect, thanks to small models built for it.
Run this. Phi-4-mini-instruct (3.8B, 128K context) is the standout entry point. Microsoft built it specifically to run on the machine you already have, with no GPU required, per the Hugging Face write-ups. Qwen3.5 small variants and compact Gemma 4 builds are also strong here.
Expected performance. A modern Ryzen 9 produces roughly 5 to 15 tokens per second on a 7B model CPU-only, according to Ollama benchmark guides. A 3.8B model at Q4 runs noticeably faster than that. Memory: a 3.8B Q4 model needs only a couple of gigabytes of RAM.
Where it breaks. Long prompts. Prompt processing on CPU is slow, so a long-document summarization that looks fine on a small input becomes painful at 32K tokens. Keep inputs tight.
Tier 2: 8GB to 12GB VRAM
RTX 3060 12GB, RTX 4060, RTX 4070. The most popular real-world tier, and a genuinely productive one.
Run this. Llama 3.3 8B, Qwen3 7B, and Mistral Small 3 at Q4_K_M. As a rule of thumb from the self-hosting leaderboards, 8GB comfortably holds 7B to 8B models at Q4.
Expected performance. A mid-range card like an RTX 3070 (8GB) typically reaches 40 to 80 tokens per second on a 7B model, per Ollama’s Windows benchmarks. That is firmly interactive.
Where it breaks. The moment you try a 13B model at anything above Q4, or push the context window long enough that the KV cache eats your remaining VRAM. You will see it as a sudden slowdown when offload kicks in.
Figure. Weights-only memory by model size and quantization, with the common VRAM ceilings marked.
Tier 3: 16GB to 24GB VRAM
RTX 4080, RTX 4090, RX 7900 XTX. This is the sweet spot where local AI stops feeling like a compromise.
Run this. Gemma 4 31B, Qwen3.6-35B-A3B, and Mistral Small 4. 24GB is the practical floor for 30B-class models, per whatllm.org. The Gemma 4 31B flagship posts 89.2 percent on AIME 2026, 84.3 percent on GPQA Diamond, and 80.0 percent on LiveCodeBench v6, which is frontier-adjacent quality on consumer hardware.
Expected performance. On an RTX 5090 (32GB) a 30B Q4 model runs around 40 to 55 tokens per second. A 24GB card lands a bit lower but stays comfortably interactive for a single user.
Where it breaks. 70B dense models. At Q4 a 70B model needs more than 32GB, so on a 24GB card it spills to RAM and throughput falls to single digits.
Tier 4: 32GB to 64GB+ workstations
RTX 5090, multi-GPU desktops, Mac Studio Ultra, enterprise workstations. Now the big mixture-of-experts models come into reach.
Run this. Llama 4 Scout (109B MoE, 10M-token context), gpt-oss 120B, and 70B dense models at Q4. The unified-memory machines shine here: a DGX Spark (128GB) runs a 70B Q4 model at 35 to 45 tokens per second because it holds the full model with no offload, per Presenc AI.
Trade-off. Raw FLOPS versus memory capacity. An RTX 5090 has more compute than a DGX Spark but only 32GB, so on a 70B Q4 model the Spark wins (35 to 45 t/s) against the 5090’s offload-penalized 14 to 22 t/s. Capacity beats compute once the model stops fitting.
Where it breaks. Multi-user serving on Apple Silicon. Unified memory does not service concurrent streams well, so a Mac that is great for one developer is a poor shared server.
Figure. Apple Silicon is decode-strong but prefill-bound. NVIDIA pulls away on big prompts.
Tier 5: extreme local AI systems
Multi-GPU servers, 128GB-plus memory systems, enterprise AI workstations. This is where the genuine frontier open-weight models live.
Run this. Kimi K2.6 (~1T total, ~32B active MoE), DeepSeek V4 Pro, GLM-5.1, Qwen3.5-397B-A17B, MiniMax M2.7, and MiMo-V2.5-Pro. On the neutral Artificial Analysis Index, Kimi K2.6 tops the open field (Index 54, ranked first open and fourth overall), and DeepSeek V4 Pro wins agentic coding while tying the closed frontier on SWE-bench, per Codersera.
What it takes. These are MoE models with enormous total parameter counts. Even with only tens of billions of active parameters per token, you need the full weight set resident, which means 128GB and up of fast memory or a multi-GPU rig. 40GB-plus is the floor once you move into 70B territory, and frontier MoE models go well beyond that.
Where it breaks. Single-GPU consumer cards. A 120B model is simply out of memory (OOM) on a 32GB RTX 5090 at Q4. This tier is not a thing you reach by quantizing harder; you reach it by buying memory.
Best local LLMs ranked by use case
Hardware tells you what fits. Use case tells you which of those models to actually pick. Here is how I would choose for each major workload, with the benchmark evidence behind it.
Figure. A quick-reference map from job to recommended open-weight model, mid-2026.
Best for coding
For agentic coding, DeepSeek V4 Pro and GLM-5 lead the open field. GLM-5 posts 77.8 percent on SWE-bench Verified, the strongest open coding result, and DeepSeek V4 Pro ties the closed frontier on the same benchmark, per ComputingForGeeks and Codersera. If you need a smaller dense coder you can actually self-host, Qwen3.6-27B under Apache-2.0 is the best practical choice.
Best for reasoning
Gemma 4 31B is the surprise leader for hardware you can own, with 89.2 percent on AIME 2026 and 84.3 percent on GPQA Diamond. Among the giants, Qwen3 235B leads open reasoning (77.2 percent GPQA Diamond, 85.7 percent AIME 2024) and DeepSeek R1 is near-perfect on MATH-500 at 97.3 percent, per ComputingForGeeks.
Best for RAG systems
Retrieval-heavy workloads stuff large amounts of fetched context into the prompt, so what you want is long, reliable context plus fast prompt processing. Llama 4 Scout owns the context-length crown at 10M tokens, and MiMo-V2.5-Pro holds strong retrieval well past 512K tokens on GraphWalks where its predecessor collapsed to zero, per BentoML. For most RAG builds, a 30B-class model with a solid 256K window (Gemma 4, Qwen3.5) is the pragmatic pick.
Best for agents
Tool use, planning, and orchestration reward breadth. GLM-5.1 performs strongly across browser-use, tool-orchestration, cybersecurity, and multi-turn business benchmarks, which makes it a versatile agent backbone rather than a single-domain specialist. DeepSeek V4 Pro and MiniMax M2.7 are the other strong agentic picks, with MiniMax M2.7 posting the highest open ELO on GDPval-AA, per BentoML.
Best for privacy-focused deployments
For offline and regulated environments, the deciding factors are licensing and footprint, not just raw quality. Gemma 4 31B and Qwen3.6-27B are both Apache-2.0 with no usage restrictions, which keeps your legal team happy, and Phi-4-mini handles fully air-gapped edge cases. Local deployment stays attractive here for the obvious reasons: data never leaves your infrastructure, you keep full control, and the marginal cost per token is electricity.
BUSINESS IMPACT
Routing high-volume commodity work to a local Gemma 4 or Qwen3.6 model while reserving a cloud API for the hardest reasoning is the hybrid pattern that consistently minimizes cost without sacrificing quality where it counts. Sources: AI Cost Check, Gridex, 2026.
Quantified performance comparisons
Architects care about measurable outcomes, so let me put the comparisons side by side. Three contrasts matter most: small versus large model, baseline versus optimized quantization, and CPU versus GPU.
Small versus large. Throughput scales roughly inversely with parameter count once memory bandwidth binds. On an RTX 5090, a 7B Q4 model runs at 130 to 150 tokens per second; a 70B Q4 model on the same card drops to 14 to 22 with offload. Doubling parameters roughly halves speed.
Baseline versus optimized. A Q4_K_M 7B model is roughly twice as fast as the same model at Q8_0 on identical hardware, with only a modest quality trade, per Ollama’s benchmarks. FP16 to Q4 is the 3.5x to 4.0x jump shown earlier.
CPU versus GPU. The single biggest performance factor. A 7B model that crawls at 5 to 15 tokens per second on a Ryzen 9 CPU runs at 40 to 80 on a mid-range GPU. GPU acceleration is not a nice-to-have; it is the difference between batch and interactive.
Figure. The quantization trade: speedup on the bars, quality cost on the line. Q4 to Q5 is the production sweet spot.
RELIABILITY INSIGHT
Q3 and below show meaningful quality loss. The temptation to quantize harder to fit a bigger model usually backfires: a clean Q4 30B beats a degraded Q2 70B for almost every real task. Stay at Q4 or Q5 unless you have measured the loss and accept it.
Scale context: numbers without scale are meaningless
A tokens-per-second figure means nothing until you anchor it to model size, memory, context, and deployment scale. So for the recommendations above, here is the scale frame I keep in my head.
Parameter count and memory. Weights-only memory is roughly two bytes per parameter at FP16 and about 0.6 bytes at Q4_K_M. A 7B Q4 model is about 4GB; a 70B Q4 model is about 40GB; a 120B model needs more than any single consumer card holds. Add 1 to 4GB for KV cache and overhead.
Context window. Bigger windows cost memory and speed. A 10M-token context (Llama 4 Scout) is a different deployment from a 256K window (Gemma 4), even at the same parameter count, because the KV cache grows with the context you actually use.
Deployment scale. Single-user and multi-user are different machines. A Mac Studio that serves one developer beautifully cannot serve fifty, because unified memory does not parallelize streams the way GPU streaming multiprocessors do.
Figure. Single-stream versus eight-stream aggregate throughput. NVIDIA silicon scales for multi-user serving; Apple does not.
The hidden cost story
This is the section that changes minds, so I want to be careful and concrete. Comparing local inference to a cloud API on per-token price alone is a trap, because the sticker price hides electricity, hardware depreciation, maintenance time, and the opportunity cost of an idle GPU.
Marginal cost per token. Running Llama 4 Scout locally on an RTX 4090 costs roughly 0.0003 dollars per million input tokens in electricity, against about 2 dollars per million for a GPT-4.1-class API, which is on the order of 6,000x cheaper per token once the hardware is paid off, per Vucense.
Electricity. An RTX 4090 at full load draws about 350W, which is roughly 30 dollars a month at the US average of 0.12 dollars per kWh if it runs 24/7, and closer to 9 dollars at a realistic 30 percent utilization, per AI Cost Check. European operators at 0.25 to 0.30 dollars per kWh roughly double that, which pushes break-even 40 to 60 percent higher in required volume, per SitePoint.
Hardware acquisition and depreciation. A capable 24GB-plus card runs 1,500 to 4,000 dollars upfront. An RTX 4090 around 1,600 dollars breaks even against a 200-dollar-a-month API bill in roughly 8 months, per Vucense, though depreciation means the asset is worth less by then.
Maintenance. Self-hosting is not set-and-forget. Plan for 2 to 5 hours a month of operational time, and remember that the most expensive hardware in your office is an idle GPU.
Figure. Cumulative cost of a local mid-range setup versus a cloud API at roughly one million tokens per month.
Figure. Marginal cost per million input tokens, log scale. After payback, local is in a different universe.
When does local become financially superior? The honest answer is volume-dependent. Below roughly 50,000 dollars a year in API spend, engineering overhead alone usually exceeds the API bill, so self-hosting costs more. Between 50,000 and 500,000 dollars, the right answer is hybrid. Above 500,000 dollars, owned infrastructure starts to dominate, per Gridex. For a moderate one-million-token-a-month workload, a two-year comparison runs about 60,000 dollars cloud versus about 5,550 dollars local, a roughly 91 percent reduction, per PepeWebTech. The break-even is not about total spend; it is about how many tokens you can route to a local model that is good enough for the task.
BUSINESS IMPACT
An idle GPU is the most expensive item in the room. The teams that win on cost are not the ones with the biggest local rig; they are the ones who keep utilization high and route only the work that genuinely needs the frontier to the cloud.
Architecture trade-offs
A premium engineering article should expose decisions rather than hide them, so here are the trade-offs I think about most, each with the measurable impact.
Qwen versus Llama. Qwen3.5 leads on multilingual coverage (201 languages) and clean Apache-2.0 licensing, and Qwen3 235B tops open reasoning. Llama 4 Scout counters with a 10M-token context that no Qwen build matches. Pick Qwen for licensing and reasoning, Llama Scout for whole-codebase and long-document work.
Mistral versus Gemma. Mistral Small models are prized for inference speed and instruction-following and make strong enterprise deployments. Gemma 4 31B wins on raw benchmark quality (89 percent AIME, 84 percent GPQA) and is now Apache-2.0. Gemma for quality on a tight footprint, Mistral for fast, predictable serving.
Small versus large. A small model you can run fast often beats a large model you can barely run. Phi-4-mini will not win hard reasoning benchmarks, but it runs on a laptop with no GPU, which is the entire point.
Quantized versus full precision. Q4 buys you 3.5x to 4x speed and roughly half the memory at 1 to 3 percent quality cost. For local inference in 2026 this is the default, not the compromise.
GPU versus CPU. GPU is the single biggest factor in speed. CPU is viable only for small models and patient workloads.
Ollama versus vLLM versus LM Studio. This is the decision people get wrong most often, so it gets its own diagram.
Figure. Match the runtime to the workload. The same GPU is fast or slow depending on the engine.
The headline numbers are stark. On identical hardware, a Red Hat serving benchmark re-validated in 2026 measured vLLM at roughly 793 tokens per second versus Ollama’s 41 at peak, with P99 latency of 80ms versus 673ms, per Codersera. Ollama is not slow at one user; it simply does not scale past a handful of concurrent requests, because it lacks PagedAttention and continuous batching. vLLM delivers roughly 16 to 20x Ollama’s concurrent throughput at the cost of a heavier setup and 20 to 30 percent more VRAM overhead.
Figure. vLLM scales with concurrency; Ollama flatlines. Under load, the engine matters more than the GPU.
ARCHITECTURE TRADE-OFF
Single user wants the lowest overhead, so reach for Ollama, llama.cpp, or MLX on Apple Silicon. Many users want throughput under load, so reach for vLLM. Choosing the wrong engine can cost you a 16x throughput difference on hardware you already paid for. Source: Codersera / Red Hat benchmark, 2026.
What breaks in production
Benchmarks are tidy; production is not. Here are the failure modes that show up repeatedly in deployment write-ups, with the cause and the mitigation for each.
VRAM exhaustion. What broke: the model loads, then crashes or slows to a crawl once context grows. Why: the KV cache grows with context and pushes total memory past the card. Mitigation: quantize the KV cache, cap context, or step down a model size. Improvement: holding a 30B Q4 model fully resident instead of spilling to RAM is the difference between roughly 45 t/s and single digits.
Context-window limits. What broke: long RAG prompts get silently truncated or balloon latency. Why: prompt processing is compute-bound and grows with input length. Mitigation: chunk and rerank retrieved context so you feed the model only what it needs.
Slow token generation. What broke: a model that benchmarked well feels sluggish. Why: it is memory-bandwidth bound, or it is a reasoning model emitting thousands of hidden thinking tokens before the visible answer. Mitigation: pick a device that holds the model in the fastest memory, or switch off extended reasoning for latency-sensitive paths.
Memory fragmentation. What broke: throughput degrades under concurrent load even though total memory looks fine. Why: naive KV-cache allocation fragments GPU memory. Mitigation: use a server with PagedAttention (vLLM), which reduces fragmentation by 50 percent or more and lifts concurrent throughput 2 to 4x, per the vLLM literature.
Model loading delays. What broke: cold starts of tens of seconds. Why: large weight files must be read and placed in memory. Mitigation: keep hot models resident and pre-warm, rather than loading on demand.
Inference bottlenecks under load. What broke: P95 latency spikes from seconds to over a minute past about ten concurrent users on a single-slot engine. Why: request queuing with no batching. Mitigation: move to a batching server; this is the core Ollama-to-vLLM migration.
Figure. Reasoning models hide most of their cost in thinking tokens. Memory speed, not headline t/s, decides wall-clock time.
The normal engineer’s guide
Let me make this fully actionable. Here is what I would tell five specific people, assuming they want the best quality their situation can sustain.
The developer with a laptop. Run Phi-4-mini or a 7B model at Q4 through Ollama or LM Studio. You get a private, capable assistant with zero cloud cost for everyday coding help and drafting. Do not chase 70B models; you will be disappointed.
The startup. Buy one 24GB-class card or rent a cloud GPU, run Gemma 4 31B or Qwen3.6-35B-A3B on vLLM, and route only the hardest reasoning to a cloud API. This hybrid keeps your burn low while you find product-market fit.
The solo founder. A single RTX 4090 or a Mac Studio with high unified memory plus Ollama is enough to ship real features. Break-even against a 200-dollar API bill is around eight months, and you keep full data control in the meantime.
The AI engineer. Standardize on vLLM for serving, keep llama.cpp or MLX for local experimentation, and treat quantization as a first-class deployment decision. Measure tokens per second, TTFT, and VRAM headroom on your actual prompts, not on a vendor’s.
The enterprise team. Self-hosting starts to make sense above roughly 8,000 conversations a day, per the TCO literature. Below that, hybrid wins. Build for high GPU utilization, choose Apache-2.0 or MIT models (Qwen, Gemma 4, GLM-5.1, DeepSeek V4) to keep legal clean, and plan capacity with the batched-throughput numbers, not the single-stream ones.
The future of local LLMs
A few directions look durable enough to plan around.
Better quantization. The Q4 sweet spot will keep moving down as formats improve. Expect Q3-class quality to approach today’s Q4, which effectively doubles the model you can run on the same card.
Edge AI and AI PCs. Models like Phi-4-mini prove that genuinely useful inference fits on commodity hardware. As NPUs ship in mainstream laptops, the no-GPU tier will get dramatically stronger.
Local agents. Throughput above 30 tokens per second for the most-deployed 7B to 30B sizes already makes local agents and RAG viable, which means a growing share of real work will run on models no cloud provider can see.
Consumer hardware acceleration. Unified-memory machines (DGX Spark, Mac Studio Ultra) brought 70B-class models into single-developer reach. The capacity-over-compute lesson will keep favoring high-bandwidth large-memory designs for local inference.
Conclusion
Let me bring it home. There is no single best local LLM because the question is incomplete without your hardware. The strongest model is the one your machine can run at a quality and speed you can actually use, and that answer changes from a laptop to a workstation to a multi-GPU server.
Hardware-first thinking matters because memory is the gate. Capacity beats compute once a model stops fitting, quantization is the lever that decides fit, and the serving engine decides whether your GPU scales or stalls under load. The biggest benchmark lesson is that no single score discriminates anymore, so you match the benchmark to the workload. The biggest deployment lesson is that an idle GPU and the wrong runtime will quietly waste hardware you already bought.
The best local LLM is not the one at the top of the leaderboard. It is the one that delivers the highest quality your hardware can sustain at the lowest cost.
















i have a dual 5090 (64gb vram, 128gb system) setup and had to build my own harness (helexa.ai) to be able to make use of qwen 3.6 27B (54gb weights before quantised down with isq). can i run the larger models? sure. but they're utterly useless on 64gb for anything more than chat or image generation. if you need tool calling for agent0 or opencode, then you need vram headroom to get anywhere near 48k max tokens, which is kinda baseline for writing code. forget about using the mainstream harnesses too if you need to split weights across 2 or more gpus. they all claim to support tensor parallelism but the reality is they fall apart on tool calling or vision as soon as you set tp=2. so the general gist of this article is correct but the reality is the benchmarks tell you about best case scenarios where you don't scrape past the surface of capabilities.
I would add that there is a category of users we have GPU with less than 8 GB VRAM, personally 💻 using Nvidia RTX 4050 with 6 GB VRAM.
Using llama.cpp I am able to run smoothly 7B model in local, with aggressive quantization able to even push and load 9B models fully onto the GPU.
Used Phi-4-Mini works good and also the new Qwen 3.5 models. I use Granite Models for focused understanding and privacy conscious as that has natively multilingual support for Arabic as well as Indian regional languages.
Earlier I was using Qwen 2.5-Coder 7B as my local coding agent. Was impressive for my light weight coding tasks.
My current coding agent models are Qwen based - Qwopus-3.5-Coder-4B (when I need more longer context) and Qwopus-3.5-Coder-9B (when need more concise capabilities).