It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.
That's not even remotely close to being true, even once you account for capex. You have to look at the actual usage, look at the token limits. Even if you're paying Anthropic $200k/month for scale-tier, you're going to blow through your token limits trying to run max output 24/7. Three users running Opus 4.8 at max non-stop will probably clean your monthly allowance from daddy Dario in less than a week.
With an 8x MI355x cluster at full tilt and including cooling, your power draw runs ~17kW. That's what it looks like when it's running full tilt. To be fair, hey that's pretty expensive. It does mean 8 multi-trillion parameter models unquantized running 24/7 without pause. And you get the full month like that, your monthly token limit is the time in a month. That cluster, the electrical upgrade, the cooling setup, and the electricity to run it all costs less in 2 months than your maximum affordance from Anthropic does in the same time period. Two billing cycles, and realistically it's more like two weeks. In 4 quarters you've wasted over a million. Like, what are we talking about here?
Now if you aren't using AI all that much, which is perfectly valid, and especially if you aren't using it at its absolute maximum, the story changes. Because even though at that point you're not paying nearly as much in electricity to run the cluster anymore, you still have the $300k+ capex to get the setup in the first place. But if we're not redlining it non-stop, then we're not really talking about performance anymore, are we? If your org never comes close to hitting token limits, it's probably because AI is rather marginal for you. Which again, is perfectly valid. I don't even use AI professionally.
Fact of the matter is, if your corp can justify the capex for a cluster and makes heavy use of AI, you are literally burning money by not having one in your building. The numbers are painfully obvious. Even deepseek isn't as cheap. This is before we get into things like LoRAs, custom inference pipelines, etc. which you know are kind of important if you actually care about model performance.
> With an 8x MI355x cluster at full tilt and including cooling, your power draw runs ~17kW. That's what it looks like when it's running full tilt. To be fair, hey that's pretty expensive.
Pretty expensive is an understatement. You couldn’t buy one of these if you wanted to right now. If you could it would be multiple hundreds of thousands of dollars.
> It does mean 8 multi-trillion parameter models unquantized running 24/7 without pause
You can’t even run one unquantized multi-trillion parameter (>=2T) model on 8 x MI355x with enough context for concurrent users. I don’t know how you think it’s going to run 8 of them at the same time. Did you mean 8 concurrent sessions?
Your math is way off across this post. If replacing an Anthropic subscription for a whole company was as easy as buying a box for the office and then breaking even in 2 months, it wouldn’t be some little secret that we only discover in a comment online.
> Pretty expensive is an understatement. [...] If you could it would be multiple hundreds of thousands of dollars.
Obviously, I quantified both the operating expense and the capital expense in my post. What I find curious is that you're quoting me talking about the operating expenditure, and changing the topic to be about the buy-in like these are interchangeable things. You don't think that this is a crucial and important distinction?
> You couldn’t buy one of these if you wanted to right now.
You could have spent all of 5 seconds of searching rather than just assuming[1]. You're not buying an Nvidia Superpod™.
> You can’t even run one unquantized multi-trillion parameter (>=2T) model on 8 x MI355x with enough context for concurrent users.
That's certainly fair a point. Although in the English language, especially in legal contexts, the multi- prefix is used inclusively for fractional values. That is it's strictly >1, not >=2. IE an 18 month contract is a multi-year contract, or a $1.6 million dollar asset is a "multi-million" dollar asset. But this is uninteresting semantics.
You are right, but it also doesn't matter. The gap is just that big. You can run 1 single user of Kimi K3 and still not even come remotely close to the $70k or so that a single Opus 4.8 user can burn over the course of a month on left on max. An honestly lowballed amount I know from anecdote. The per-token cost is just really expensive.
> Your math is way off across this post.
You made one technical point above, one that doesn't ever arrive at a relevant rebuttal to the substance of my post. But please, I'd love to hear you elaborate, especially because I didn't actually give much math at all.
If you want math though, here's the math. Let's say you are paying a ridiculous amount of money for electricity, a price nobody in the US pays -- $2 per kilowatt hour. That's about 5x the average rate in California, 4x as in Hawai'i. 17kW @ $2/kWh * ~8766 hours in a year puts that cluster's electrical costs at just shy of ~$298k annually assuming it takes no breaks. Let's make matters worse and round that up to $300k. It's also assuming you didn't invest in a solar hookup for your building, which I don't know why you haven't at this point, especially if you're installing a CDU for your new cluster. 12 months of Claude burning $70k a month is $840k. For a buy in of, you know what, let's call it $500k. Why not? It still doesn't matter. The operating cost is so much lower it's paid for itself plus an additional $40k in the first year. Even at a ridiculous penalty in electricity that nobody pays, even overinflating the amount of money you'd pay for the cluster and the infrastructure to get it set up, it's not even remotely close for a single user where the gap is smaller (IE, you're not wasting "a million dollars" in a year by maxing out the $200k scaling limit every month)
You can of course trot out the point that oh, in 12 months this setup will be extremely outdated! It doesn't matter. If the work it was doing today was useful, it will be useful next year too. And with the rapidly encroaching diminishing returns from parameter scaling, you're probably going to be just fine for a while. Maybe grab a quantized version of a newer Chinese model at the end, before grabbing a newer generation of AMD node. Those MI400s are looking pretty sweet after all.
> If replacing an Anthropic subscription for a whole company was as easy as buying a box for the office and then breaking even in 2 months
If you're locked in, then you're locked in. But don't pretend like you're saving money. You're not.
> it wouldn’t be some little secret that we only discover in a comment online.
Why does this have you so nasty and defensive? It's not a "little secret" that running your own infrastructure is cheaper. Of course it is. You know what else is cheaper? Owning your own office building out in the sticks, rather than leasing part of one in the city. Not everybody can make that work, there are no free lunches after all.
History repeats, these same exact lines were rolled out ad nauseum during the cloud craze. Datacenters are businesses, not charities. Frontier companies rent quite a fair amount of their infrastructure. Even if they resold that compute below cost (they don't), there's a pretty steep cliff before the economics start to look attractive.
> You could have spent all of 5 seconds of searching rather than just assuming[1].
I guarantee this will not ship to you any time soon.
The current lead time on these GPUs in measured in years. If you didn't place an order for this a long time ago, it's not coming this year.
Being able to add it to an online configurator does not mean anything right now.
> 12 months of Claude burning $70k a month is $840k
Your math is completely useless with these arbitrary numbers pulled out of the air.
If you want to begin calculating payback period you'd need to look at token costs, cost per task, utilization rates, and so on.
> The operating cost is so much lower it's paid for itself plus an additional $40k in the first year.
You went from paying back in a couple months to paying back in a year but you still haven't even talked about tokens or concurrency.
You're also neglecting the fact that hosted tokens are going down in price at a rapid rate. If someone was paying $70K per month in tokens for Opus this month, that same level of compute is going to be much cheaper 12 months from now.
> Why does this have you so nasty and defensive?
Not nasty or defensive, just tired of these armchair claims that it's easy to go out and buy an 8 X MI355X box from people who obviously have no idea what the hardware lead time is like right now, or who haven't considered the actual math on token costs and payback times. You're still making a lot of claims without a single discussion of cost per task or token.
It usually boils down to people trying to convince themselves that keeping their macs hot and with very little ram to spare only to get sub 50 tokens per second on a subpar lobotomized (quantized) model is worth it.
And I'm not even considering their time spent fiddling, fine tuning configs to adjust for ram, updating/benchmarking models, etc. Which is probably more expensive than the mac so the math is even more wrong.
> I guarantee this will not ship to you any time soon.
The assumption, the starting point, is that you have a line on the hardware. Asking around, some distributors have a 6 month lead time on Instinct GPUs, which curiously enough is about how long you'll be twiddling your thumbs waiting for the cooling loop to be put in. Yes things take time.
> Your math is completely useless with these arbitrary numbers pulled out of the air.
Your dismissal is worthless if you can't even be bothered to provide a counter-example. You've not provided a single iota of quantified reasoning beyond my original not accounting for the space used for the context of concurrent users.
> If you want to begin calculating payback period you'd need to look at token costs, cost per task, utilization rates, and so on.
Now go back and carefully reread my original post. Yes, if you are not actually redlining an LLM for a billing cycle, the capex starts to be way more relevant for this setup. Otherwise, our constraint is time and our unit of measure is $/hr.
If you want to compare token cost, it may shock you to learn that Kimi K3 without speculative decode on this setup is slightly under twice as fast as Opus 4.8 max. That's still true when fast is compared with K3 with speculative decode, and now Claude is twice as expensive as a base rate. Oops. We're already burning more money over a period of time, looking at tokens we're screaming even further ahead.
> You're also neglecting the fact that hosted tokens are going down in price at a rapid rate.
Cool. Call me when Opus 4.8 max is $0.50/million. In 4 years you could have bought the 200 acres of land down the road from your building, started a 5MW solar farm subsidiary that you'll expand over time, and as soon as your connect is up, dropped the opex of the cluster down to its maintenance costs. That subsidiary will pay the loan required to spin it up back irrespective of your primary business. When you own your own shit, you can play your own game, stack your cards deep. Have a little bit of business acumen. Fuck what The Valley is doing, that is an ecosystem fully enslaved by economic nihilism, money isn't grounded there.
> Not nasty or defensive, just tired of these armchair claims that it's easy to go out and buy an 8 X MI355X box from people who obviously have no idea what the hardware lead time is like right now
This motte-bailey routine is both nasty and defensive, particularly when you keep prosecuting a geist of numeric justification that never arrives. All I've gotten from you is vague dismissals, one borderline irrelevant technical argument, moving goalposts and missing the point. Granted, not as egregiously as other people in this chain thinking we're talking about running 100B models on a Mac, I'll give you credit for that. But this whole time, we're just talking past each other. You make realistic points and I try to bring you back to context, but you have to work with me here too.
The point was that these companies are not selling to you below cost, they're not even selling to you at-cost. Just use your head. Venture capital isn't a magic wand. Frontier companies are in the red because they're in non-stop expansion operations at massive scales. Anthropic has an operating profit of half a billion dollars[1]. They are not selling you API usage below cost.
"You can of course trot out the point that oh, in 12 months this setup will be extremely outdated! "
But interestingly still extremely valuable on the second hand market.
The capital expense isn't the amount laid out. It's the rental cost of obtaining that capital, less the depreciation on the fixed asset over the period in use.
Going back to the OP, Apple gear is well know for having good resale values, which means the capital outlay isn't anywhere near as much as some people think.
>You couldn’t buy one of these if you wanted to right now.
You can: https://www.exxactcorp.com/Exxact-TS4-149591758-E149591758 . You can get thousands of tps of GLM 5.3 output out of this thing, which grades around Opus 4.8. Payoff is around 1 year vs. spot prices on these GPUs, including power.
Ah thanks for the solid info, too bad. I'd seen them come up as a pretty good price for 6000 RTX's in the past, which seem generally pretty available, good source for those?
Yeah, they're good source. But the price for those GPUs is 5 figs even with the nvidia startup program nowadays. Also, I went back and looked. Most of my GPUs are actually from Central Computers who were great, but Exxact is real too. So "lots of" was inaccurate.
Also, the lead time I quoted was for individual 8x nodes.
I can't tell from the ad -- it says "supports" 8x MI350X GPUs, but does that mean "includes" 8x MI350X GPUs? For $300K I'd certainly hope so, but I'm assuming not.
A system with 4x RTX 6000s costs about $60K these days, and can (as you note) trade blows with Opus 4.8 if not Fable. In fact, it'll give you a better pelican than Fable 5.1, and in less time.
Well, they are if you're into animating pelicans. :-P But yes, in the general case Opus is a better match.
And Opus is no slouch. I'm satisfied that GLM 5.3 is just as strong as Opus. Z.AI has promised/bragged that they will be at Fable 5.0 level by the end of the year or early next year, and I don't see any reason to doubt them.
Baseline yeah. But part of the reason you run open models is how much nicer fine tuning them is. Granted, you probably don't want to try and make LoRAs on a 4x RTX6000 setup, but you could if you really wanted to and there are other ways to modify models. And yes, if you're good at it, you can turn a piddly mid-range model that's only good at benchmarks into a heavyweight clanker (for a specific domain).
Ha fair, I'd definitely confirm with a salesperson before wiring them $300k. But most of the signs on the configurator seem to point to it including the GPUs? Not going to make 30k BTUs/hr of heat without the 8kw of GPUs.
Here’s an experiment: purchase an anthropic pro max subscription for $200/m. Now go buy the hardware to run DeepSeek’s equivalent. In a year, who spent more?
In normal times in which hardware used to depreciate (lately that's not the case and HW even appreciates, but let's not get distracted), if you calculate only with depreciation costs, plus the fact that when you have such a setup, it'd take many 200$ subs to cover your lack of limits in the other, I think it'd not be a clear victory for any side.
If you just ask "who spent more in the first year" (100% depreciation) then even with 5-6 max accounts, buying HW will be a couple of times more expensive. But when does it make sense to ask that question?
Maybe the SotA models will need better hardware so your investment will not be useful after a year or you'd need very expensive upgrades? But then (as in Fable case) subscribers need to spend more too.
It’s not so clear after 5 years that you’ll come out ahead. You’ll have spent $20k. The apple computer owner will probably be running local models that are better than today’s frontier on the same hardware.
Idk where you live, but where I am running the M5 Ultra Mac Studio at max rated power 24/7 for a month costs C$42.
The considerations against Apple hardware are 1) hardware advancements 2) early access to the best models. But it’s really not that clear.
(The other guy who thought hosted models on openrouter are cheap has spent $100k in 5 years.)
Open weight models have been getting better/smaller every year.
Also, from what I can tell, MLX inference is not as well optimized as CUDA, and the M5 Ultra has additional kinds of AI compute which is unavailable on other M models. With the massive 1.2 TB/s 512GB Mac studios coming out, I think MLX will get a lot more attention.
In short: Todays models should run faster next year, and next year's models should also be more efficient.
It does make me wonder how the hosted stuff is so cheap. For pretty much everything else, hosted/rented is more expensive but offers better convenience and flexibility. But for AI, even if you consider the total lifetime cost and are utilizing it heavily. You never break even by buying.
They're not cheap at all. I did one xhigh Qwen 3.8 27B agentic coding task last week via OpenRouter and it cost me like $10.
99% of the cost was in input tokens, I only used like 100k ish output tokens. It was a one shot task asking the agent to implement proxy injection to Guice. It did a pretty amazing job.
If you were to use hosted LLMs for a lot of agentic coding, a maxed out M5 Ultra Mac Studio would pay for itself in under a year.
I've been hosting Qwen3.8-27B myself. On my endpoint it's $0.30/1M in, $0.10 cache, $2.03 out - so those agent turns that re-send the same prefix get a lot cheaper when cache hits. UI at inference.tiyuvta.ai/app if you want to try it. Hosted is up to 210 tok/s and 280ms TTFT with reasoning off.
Qwen is weirdly expensive. Deepseek v4 flash is dirt cheap. You'd need at least 128gb of ram to run this model and in my experience, a days work with it costs around 80 cents.
So I ran the math, assuming the agent takes 75 turns per 200k context, with deepseek v4 flash it costs around $2.57 to reach 1M context in 375 turns. Cached input costs scale quadratically with # of agent turns.
Considering that I hit the 1M compaction multiple times per day with codex, it would definitely cost at least $5-8/day to use deepseek how I normally use codex.
Here's the thing that's a little different about data centers; we can tell from Anthropic and OpenAI that they're capacity constrained. Inference demand is there. I notice Cerebras doesn't offer much directly any more, all their capacity is getting completely sucked up by B2B sales. Grok did overbuild, but Anthropic was so desperate for more compute they ate their pride and leased the excess capacity.
That means all these data centers are being heavily utilized by actual end user inference demand. Well, some is research on new models, but a lot is actual end user demand. No one has given an explanation of why peoples usage would decline.
On top of that, margin on inference appears to be decent. It's model training that's a serious financial burden.
And maybe that's where there will be a slowdown, maybe the market doesn't justify spending as much on R&D as it does, but the end demand for inference is there.
Does that justify these stock prices? That's a different question. But the housing boom left behind endless rows of empty homes because demand disappeared. The 'dot com' boom left behind thousands of miles of dark fiber that'd been built out well ahead of demand for bandwidth. I can see the stock market having a giant sell off, but I don't see data centers sitting idle in that same fashion.
The number of planned data centers and the scale is pretty nuts.
Here's what you have to believe:
- AI demand is at least several times larger than what can currently be satisfied, or will grow. (This one I can buy, but...)
- AI chips (GPUs, TPUs, compute-in-memory, whatever else is being studied) will not get significantly more efficient than they are now. It will not be possible in, say, 5-10 years, to do 2X or 4X or 10X more AI requests per rack than is possible now. I think this one's the single most likely thing to be false, since all computing history contradicts it.
- Edge devices (PCs, laptops, specialized but smaller scale AI compute nodes) will never be powerful enough to run frontier models at a reasonable price that's appealing for professionals, enthusiasts, or businesses, and there will never be a market for this. None of the demand will be served on-device or near-edge. AI must all go in giant data centers.
- AI models will not become significantly more efficient than they are now. There are no large gains on the table from better model architectures, better training, more efficient quantizations, better harnesses, etc.
If all those things are true, than the current planned like 4X-10X increase in data center capacity makes sense. If even one or two of them are not true, then the planned data center build-outs start looking excessive. If all four are not true, it's a total bubble that will crash and burn. Answer is probably somewhere between, but how far toward bubble? That's why I picked a number like "only 20% ever gets built." It might be as high as 50%. It ain't gonna be 100%. The planned built-out is batty.
Oh I forgot two more...
- Data center capacity currently serving non-AI work loads does not shrink through either reduced demand, more efficient software, or (most likely) faster chips and denser RAM. If that happens, more pre-existing DC space can serve AI work loads.
- Orbital solar powered compute nodes never happen. If this happens (free power! much less political opposition!) then terrestrial data centers have significant competition.
Honest question. Who got to decide to accept this decision on behalf of state and federal citizens? This seems performative rather than punitive and doesn’t seem to match the harm.
My father was a plumber in the 80s and bought our family an Amiga 1000 for $1,200. Today thats about $3800 in inflation adjusted money. I just bought my son a top of the line pc for $2,000. I get that it used to be cheaper, but value is still relatively good for these machines.
In 2011 I bought a fully loaded 27 inch iMac paid 3700 (used it for 9-1/2 years) at today’s prices that would be about $5400, bought a M2 Mac studio M2 ultra that cost about $7200 2-1/2 years ago the price for the new Mac studio M5 ultra just announced by Apple is $11,400.
Former owner of Amiga, 1000, 2000, 3000, and 4000 I would have gotten a Next computer at the time if I could’ve afforded it, unfortunately, I was tied to a PC at work using AutoCAD, Revit and Navis Managed a few years later.
The Games back in the day on the Amiga computers were ahead of the PC for about 9-10 years, coincidentally Sony bought a British gaming company called Psychosis for their games and gaming technology. They were pretty big back in the day on the Amiga.
Except we've gotten so much better at making computers since the Amiga, its a travesty that things have gotten so bad for consumers. You know the ones that bail out the bubbles
They have a vision which consists of computer users being tied back into mainframe computing with them as the gatekeeper. That is the wet dream of Microsoft, OpenAI, Anthropic, Nvidia, Google along with Meta.
The thought of local AI is also disgraceful, Nvidia didn’t buy for Hugging Face for 12-13 billion for nothing.
Well said. I hope some day to make it and start parent.capital a place where parents can get funding. I believe there are thousands of people out there with great ideas and immense talent who are unable to leave their salary and benefits behind to pursue them.
When you become a parent, your risk profile changes immensely. I can hire some of the world’s best people tomorrow if i can promise them equal salary, benefits and retirement accounts for a set number of years. Plus the benefit of a large upside if it works. Most startups offer lower salaries and abysmal benefits. Security matters.
I still don’t get it. It sounds like the business plan is to fund companies so they can offer compelling employment. That’s great but that’s no different than any other VC.
Thanks for all your open source work. You changed the industry for the better. Are you thinking of building a Github competitor? I would find that very appealing.
If you’re going this far, why even have a cli? Just have your agent grab the openapi spec from the server and set envvars. It can curl its way around happily. Security? I don’t see how this is any more or less secure than any other claude code workflow.
It seems we've round about rediscovered apis.
I tried this with datadog and it can build high quality monitoring notebooks for the pr im working on in a single shot.
We’re looking for a Senior Product Engineer to join Hipcamp and help build the core product experiences that connect people with the outdoors. This is a hands-on, end-to-end role working across the stack—shipping user-facing features, collaborating closely with product, design, and data, and iterating based on real customer behavior.
This is a challenging and rewarding role at the intersection of marketplaces, real-world impact, and modern product engineering. If you enjoy owning outcomes, working in a remote-first environment, and using thoughtful tooling (including AI) to move fast without cutting corners, we’d love to talk.
As I'm on vacation, I took a couple days to put together an idea I've had bumping around in the back of my head for a while. What if you could write pure ruby methods and ask an llm to figure out all the implementation details to return the response to any method call?
The library leans hard on ruby's magical (hence the name) `method_missing` which has always impressed me with its simple cleverness.
This is my first open source project, so please leave feedback and be kind.
I mostly agree, but why stop at tests? Shouldn’t it be spec driven development? Then neither the code or the language matter. Wouldn’t user stories and requirements à la bdd (see cucumber) be the right abstraction?
Natural language is too ambiguous for this, which makes it impossible to automatically verify
What you need is indeed spec-driven development, but specs need to be written in some kind of language that allows for more formal verification. Something like https://en.wikipedia.org/wiki/Design_by_contract, basically.
It is extremely ironic that, instead, the two languages that LLMs are the most proficient in - and thus the ones most heavily used for AI coding - are JavaScript and Python...
I don't think you're wrong but I feel like there's a big bridge between the spec and the code. I think the tests are the part that will be able to give the AI enough context to "get it right" quicker.
It's sort of like a director telling an AI the high level plot of a movie, vs giving an AI the actual storyboards. The storyboards will better capture the vision of the director vs just a high level plot description, in my opinion.
You can. It’s called multimodal and basically is just picking a shipping container off a train car with a crane and putting it on a truck platform or a ship. 1 platform supports all modes.
This works great for finished consumer goods, but bulk cargo that makes up a significant portion of both rail and truck traffic like grain, liquid products (crude oil, gasoline, vinyl chloride), ore, etc. have very specialized transports that don't work well with the existing multimodal system.
