There are so many of us with weak GPUs and these updates are massive wins. Still waiting for the 2080 moment where this 27B param model and a reasonable kv cache can exist and function at slow, but overnight-burning speeds ..
As an experiment I took, not this current quant, of Qwen 3.8 27B and had it analyze some of my writing in CPU-only mode with llamacpp. Tried a Q4 and Q5. It was dreadfully slow, but you fire that off before bed, and you can get results the next day. This was a ~6000 word markdown file, took maybe 10 minutes before it started generating anything at all, but then it just cranks away. I would need to tune down the default thinking intensity, but it otherwise did the job.
Not compatible with vibe coding, but if you want a local tool to review content, it is possible today.
Just installed Qwen 3.8 Splash tonight (doesn’t appear on linked page). Super impressed. Easily averaging 45 tps, even with thinking on. Totally acceptable. Running M4 Mac mini, 48GB, LM Studio.
Bonsai 2 does some additional tuning in addition to just quantizing (mostly around the fp16 block weights per 32 params), so they’re able to pack more in to just so few GB.
The tradeoff with unsloth seems to be that it’s more directly applicable for other off the shelf open models, whereas with Bonsai you’d have to wait for them to release another one
Not compatible with vibe coding, but if you want a local tool to review content, it is possible today.
The tradeoff with unsloth seems to be that it’s more directly applicable for other off the shelf open models, whereas with Bonsai you’d have to wait for them to release another one
Bonsai's technology would be better suited for 100s of B models.