Discussion about this post

User's avatar
rob thijssen's avatar

i have a dual 5090 (64gb vram, 128gb system) setup and had to build my own harness (helexa.ai) to be able to make use of qwen 3.6 27B (54gb weights before quantised down with isq). can i run the larger models? sure. but they're utterly useless on 64gb for anything more than chat or image generation. if you need tool calling for agent0 or opencode, then you need vram headroom to get anywhere near 48k max tokens, which is kinda baseline for writing code. forget about using the mainstream harnesses too if you need to split weights across 2 or more gpus. they all claim to support tensor parallelism but the reality is they fall apart on tool calling or vision as soon as you set tp=2. so the general gist of this article is correct but the reality is the benchmarks tell you about best case scenarios where you don't scrape past the surface of capabilities.

Timex's avatar

I would add that there is a category of users we have GPU with less than 8 GB VRAM, personally 💻 using Nvidia RTX 4050 with 6 GB VRAM.

Using llama.cpp I am able to run smoothly 7B model in local, with aggressive quantization able to even push and load 9B models fully onto the GPU.

Used Phi-4-Mini works good and also the new Qwen 3.5 models. I use Granite Models for focused understanding and privacy conscious as that has natively multilingual support for Arabic as well as Indian regional languages.

Earlier I was using Qwen 2.5-Coder 7B as my local coding agent. Was impressive for my light weight coding tasks.

My current coding agent models are Qwen based - Qwopus-3.5-Coder-4B (when I need more longer context) and Qwopus-3.5-Coder-9B (when need more concise capabilities).

2 more comments...

No posts

Ready for more?