I'm the author - hello! I talk about it in the blog post - knowing what's being run, knowing where it's being run, and not having anyone else control it.

Traditional search is “free” too, but you see ads. If something looks free, then you are the product.

OpenAI has started doing ads, but for Anthropic the free tier is still just a loss leader. It's basically an ad for their pay tiers.

I figure most free AI is free as in free electricity in the coffee shop. You're welcome to use it for small reasonable loads, but try to build anything off of it and you'll soon find yourself barred from the establishment.

And that's probably good, otherwise the free ai would just be unavailable for everyone else

It's free like ads are free. Or certain kinds of advice.

Only “Free as in free beer.”

Maybe, "Free as in free WiFi?" Like WiFi, the models you can use for free online aren't the highest quality, and can be pulled any time.

The models used in TFA are halfway in between the traditional "free as in beer" software. Open weight means once you download it, it continues to work forever; and you can also do your own RL on them; but you can't really see what went into their training, nor train a new one yourself from scratch.

You are correct. In large part, the cost of something like Gemini on a very basic Google AI plan provides far more utility than local LLMs for coding assistance.

There are 2 main reasons for running local LLMS.

1. Process private data/work with uncensored models.

2. Use a large amount of inference that would quickly blow through rate limits and/or run up API costs.

The thing that is critical for 2 is that a) you have to have a sweetspot between a pretty good model, which means largest parameter counts, and fast enough token generation where you can run agentic loops. The latter is needed because you aren't going go get the "intelligence" of larger models to form shell commands and run tools to figure stuff out, so the only way around that is to have custom agentic loops to force the model into doing what you want, which results in more text processing.

From my testing, Gemma4:31b is basically the only local model that can be relied upon to produce accurate results. Qwen models chase benchmarks, which results in MoE models (thus the A3B in the model, i.e 3 billion parameters are only active during inference). In general, these are good for very specific tasks, but fail to be accurate in considering cross task data, whereas Gemma, being fully active does a much better job. If you only need to do a very specific deterministic task, those models are pretty good.

As an aside though, if your task involves pure text processing (for example take html data, make it into a markdown document), you can also additive train Gemma270M quite easily all on CPU, and on a decent CPU it gets like 50-100 tok/sec, no need for any extra hardware.

The thing with Macs is that while they can run those models and larger models no problem, the tok/sec is very slow. This limits effectively what you can do with the models. On the M4 that the poster mentioned, Gemma:31b will run about 20 tok/sec. That means that when you wants to write a whole code file or process large context, you have to wait for it to do things. Compared to workflow with larger models, where file generation often takes like <10 seconds, it takes a while to adapt.

The only benefit of using Macs is the price for Mini and cheaper studios. However, once you reach the total cost of about 2.5k (note that the M4 statedin the article us about 2k), building a gfx card rig is the way to go. You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models.

> You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models

I have a beefy Linux box with a 4090 but never took the time to set it up properly beyond simple testing; any tutorial you would recommend?

Just ollama if you are feeling lazy. With ollama, pull the model (start with https://ollama.com/library/gemma3:27b), and `ollama run gemma3:27b`

If you want to build agentic frameworks, use llama.cpp with its built in http server, and build the framework with python

Getting undeserved safety filter refusals, watching LLM accidentally reading what you might not find comfortable showing, and hitting free tier/basic subscription quotas, are not fun. Going local removes somewhat irrational anxiety that come from those.

At some point and for some tasks, predictability is important if not critical.

I’d rather use a tool where I know the limitations, over a tool where the limitations and strengths keep changing.

This way I know where in the process I ought to step in and pay attention.