Not to speak of the new attention scheme and the (IIRC) MLP changes.
I’m very much looking forward to ik_llama.cpp implementing it. I don’t think I can quite fit Flash on my rig (hence no Ktransformers for me) but a little quantization of the sparse layers, and it’d be perfect.
This is commonly cited, but not strictly true.
Prompt processing is completely compute limited. And at high batch sizes, where the weights are read once for many tokens generated in parallel, token generation is also quite compute limited. Obviously you want enough bandwidth to match the compute, but its very compute heavy.
You can see this for yourself. Try ~10 prompts in parallel on a CPU in llama.cpp, and it will slow to a crawl, while a GPU with a narrow bus won’t slow down much.
Training is a bit more complicated, but that’s not doable on CPUs anyway.
Now, local inference (aka a batch size of 1), past prompt processing, is heavily bandwidth limited. This is why hybrid inference works alright on CPUs. But this doesn’t really apply to servers, which process many users in parallel with each “pass”.
I’ve heard hints they sometimes give GPU farms busywork, to meet utilization targets apparently stipulated in sales contracts.
No. Not even close. Non-US models are trained (and run) on peanuts compared to big US models, because they don’t have mega GPU farms and have no other option. Deepseek in particular went all-in on software architecture efficiency.
…Ironically, the Nvidia GPU embargo was the best thing that ever happened to the Chinese devs. It made them thrifty.
Many tried to warn US regulators of this, but they had AI Bros whispering in their ears. The US tech system is just too screwed up, I guess.
…I mean, yeah? It’s obviously developed for the Chinese market.
But that’s theoretical, for now. No CPU backend I can find supports DSV4, and DeepSeek hasn’t contributed anything yet.
Just not power/cost efficiently on CPU only, is what I meant. CPUs don’t have the compute for batching (running generation requests in parallel). You need an accelerator, like Huawei’s, to be economical.
It’s fine for local inference, of course.
I just meant for mass inference serving.
Yeah, I haven’t seen much in the way of bitnet training savings yet, like regular old QAT. It does appear that Deepseek is finetuning their MoEs in a 4-bit format now, though.
What’s left unsaid is the software architecture is extremely interesting, and efficient.
Ironically, the Nvidia embargo was the best thing to ever happen to the Chinese labs (which Nvidia tried to tell the US govt). It forced them to get thrifty, unlike US labs which (allegedly) fill some GPU farms with busywork for the appearance of high utilization.
Not at scale. Even on the new architecture, one really needs some kind of accelerator to make it economical for servers.
Bitnet-like models might change the calculus, but no major trainer had tried that yet.
I mean, I use every alternative I can. Vapoursynth scripts, libraw-based projects, random GitHub repos, DaVinci…
But there are some features I just can’t get great support for outside of definitely-not-high-seas Lightroom Classic:
Good lens profiles for weird lenses.
Proper HDR PQ/HLG editing and AVIF/JXL export support.
RAW support for newer cameras, like my little R50V
I have yet to try DaVinci’s photo editing mode though. That’s very interesting.
On a technical level, that makes zero sense.
AI “agents” are basically just fancy prompts with a tool calling harness. They are infinitely replicable, at zero cost, with no intrinsic value; the cost comes from the generic CPU host, and the API calls to GPU servers, databases, or whatever else that are all centralized anyway.
Wanna hear a dirty secret?
“AI” cost is going to zero.
Model capabilities aren’t scaling, but inference efficiency is exploding, thanks to more resource-constrained labs and breakthroughs in papers. The endgame of the current bubble is mediocre but useful tools anyone can host themselves, dirt cheap. Maybe a bit more reliable and refined than what we have now, but about as “intelligent.”
And guess what?
Microsoft can’t profit off that. None of the Tech Bros can.
Point being, this exec is either delusional, or jawboning, so the world doesn’t realize that “AI” is a dumb utility/aid, and they can’t make any profit off it.
To illustrate what I mean more clearly, look at the top comments/replies for the NASA Artemis posts, as an example.
…It’s basically all conspiracy theorists, and government skeptics.
Twitter’s focusing the Artemis posts on them because it’s what they want to see, and most engaging for them.
In the EFF’s case, I’m not just talking about Musk’s influence. The algorithm will only show the EFF to users who would be highly engaged by it. E.g., angry skeptics who wouldn’t be swayed by the EFF anyway, or fans who already agree with the EFF. It’s literally not going to show the EFF to people who need to see it, as Twitter’s metrics would show it as unengaging.
This is the “false image” I keep trying to dispel. Twitter is less and less an “even spread” of exposure like people think it is, like it sort of used to be, more-and-more a hyper focused bubble of what you want to hear, and only what you want to hear. All the changes Musk is making are amplifying that. Maybe that’s fine for some orgs, but there’s no point in the EFF staying in that kind of environment, regardless of ethics.
I feel like the EFF’s messaging is just not going to get through to anyone still on Twitter.
Remember, it’s not a fair forum; it’s an algorithm. And it’s not going to show the EFF to users who need to see it.
Even not-fully-reproducible open-weights models are extremely important because they’re poison to OpenAI, and they know it. It makes what they’re trying to commodify and control effectively free and utilitarian.
But there are fully open models, too, with public training data.
It’s anticompetitiveness.
They want to squash open models, and anyone too small to comply with this.
I say this in every thread, but the real AI “battle” is open-weights ML vs OpenAI style tech bro AI. And OpenAI wants precisely no one to realize that.
Ughhh, I could go on forever, but to keep it short:
Tech bro enshittification: https://old.reddit.com/r/LocalLLaMA/comments/1p0u8hd/ollamas_enshitification_has_begun_opensource_is/
Hiding attribution to the actual open source project it’s based on: https://old.reddit.com/r/LocalLLaMA/comments/1jgh0kd/opinion_ollama_is_overhyped_and_its_unethical/
A huge support drain on llama.cpp, without a single cent, nor a notable contribution, given back.
Constant bugs and broken models from “quick and dirty” model support updates, just for hype.
Breaking standard GGUFs.
Deliberately misnaming models (like the Deepseek Qwen distills and “Deepseek”) for hype.
Horrible defaults (like ancient default models, 4096 context, really bad/lazy quantizations).
A bunch of spam, drama, and abuse on Linkedin, Twitter, Reddit and such.
Basically, the devs are Tech Bros. They’re scammer-adjacent. I’ve been in local inference for years, and wouldn’t touch ollama if you paid me to. I’d trust Gemini API over them any day.
I’d recommend base llama.cpp or ik_llama.cpp or kobold.cpp, but if you must use an “turnkey” and popular UI, LMStudio is way better.
But the problem is, if you want a performant local LLM, nothing about local inference is really turnkey. It’s just too hardware sensitive, and moves too fast.
Also, for any interested, desktop inference and quantization is my autistic interest. Ask my anything.
I don’t like Gemma 4 much so far, but if you want to try it anyway:
On Nvidia with no CPU offloading, watch this PR and run it with TabbyAPI: https://github.com/turboderp-org/exllamav3/pull/185
With CPU offloading, watch this PR and the mainline llama.cpp issues they link. Once Gemma4 inference isn’t busted, run it in IK or mainline llama.cpp: https://github.com/ikawrakow/ik_llama.cpp/issues/1572
If you’re on an AMD APU, like a Mini PC server, look at: https://github.com/lemonade-sdk/lemonade
On an AMD or Intel GPU, either use llama.cpp or kobold.cpp with the vulkan backend.
Avoid ollama like it’s the plague.
Learn chat templating and play with it in mikupad before you use a “easy” frontend, so you understand what its doing internally (and know when/how it goes wrong): https://github.com/lmg-anon/mikupad
But TBH I’d point most people to Qwen 3.5/3.6 or Step 3.5 instead. They seem big, but being sparse MoEs, they can run quite quickly on single-GPU desktops: https://huggingface.co/models?other=ik_llama.cpp&sort=modified
There’s a whole lot of interest in locally runnable ML. It was there even before ChatGPT 3.5 started the tech bro hype train, when tinkerers were messing with GPT-J 6B and GAN models.
In a nutshell, it’s basically Lemmy vs Reddit. Local and community-developed vs toxic and corporate.
They seem to have held back the “big” locally runnable model.
It’s also kinda conservative/old, architecture wise: 16-bit weights, sliding window attention interleaved with global attention. No MTP, no QAT (yet), no tightly integrated vision, no hybrid mamba like Qwen/Deepseek, nothing weird like that. It’s especially glaring since we know Google is using an exotic architecture for Gemini, and has basically infinite resources for experimentation.
It also feels kinda “deep fried” like GPT-OSS to me, see: https://github.com/ikawrakow/ik_llama.cpp/issues/1572
it is acting crazy. it can’t do anything without the proper chat template, or it goes crazy.
IMO it’s not very interesting, especially with so many other models that run really well on desktops.
It is! 143GB last I checked. I’m on 128GB RAM + 3090, 1 NUMA node, so I think it’s juuust barely too tight. But it should be perfect with a few of the “sparsest” MoEs quantized.
If KTransformers supports something like that, I may have to finally check it out, since v4 won’t need many esoteric features.