The connection was lost. Trying to connect again.

Post by Jan Wildeboer 😷:krulorange:

Earlier in the thread

Jan Wildeboer 😷:krulorange: @jwildeboer@social.wildeboer.net

(Disclaimer. I work at Red Hat, which is owned by IBM. But my opinions here are my own. So don't quote me, dear journalists)

On-Prem "AI" works and is Good Enough™ so more companies are running Open Weight LLM inference on their own bare metal. Says IBM CEO Arvind Krishna. I am not surprised. If you can save 90% of the costs and, more importantly, keep your data and prompts local, err, sovereign —it not only makes business sense, it also keeps the lawyers happy.

theregister.com/systems/2026/1

· Public · edited

Harald (oliof) Wagener @hwa@infosec.exchange

@jwildeboer
I wouldn't say you can save 90% of the cost, as unlocking performance with self hosted requires some upfront investment and at modest scale, people too. But I will say that if you have the resources and skill to self host open weights models, you can easily get factor 5-15 of what you'd pay for a subscription (that's factor 5-15 below cost covering to boot).

· Public

Jan Wildeboer 😷:krulorange: @jwildeboer@social.wildeboer.net

@hwa The focus is often on coding LLMs where the calculation between Good Enough™ and good in-house developers does get you to 90% cost savings. It of course depends on how you calculate the cost of on-prem inference. If you stick to tokens or to TCO divided by result metrics, which can be quite complicated. The main point is that on-prem inference results in fixed TCO whereas renting models is a very chaotic calculation. And compliance is growing fast in importance for (big) companies in the EU

· Public · edited

Replies

Jan Wildeboer 😷:krulorange: @jwildeboer@social.wildeboer.net

@hwa But LLM use is far broader. Call center support, dealing with internal processes etc are also use cases where the cost calculation depends mo0re on internal training on effective use. It's a growing field of practice and mistakes have been made. But the bean counters are now involved ;)

· Public