The Office of Weights and Measures exists because, long before any of us were born, in 1836, companies were up to shady shit and consumers were paying for inconsistent products. I.E. Being scammed.
AI companies should be subject to the OWM like any other company that sells a product that varies in weight. Perhaps when a sane administration is re-elected; one that can read history books and comprehend why our regulations exist in the first place. Or have even a semblance of respect for its citizenry.
Fluctuations of a few percentage points are to be expected and should not surprise anyone who knows how LLMs work.
This Twitter analysis of Fable 5 is not that at all. They analyzed their coding sessions and blamed all of the fluctuations on Fable changing. They then compared to ARC-AGI-2 questions as the benchmark for thinking tokens and tried to stir up anger that coding turns don't produce as many thinking tokens as the ARC-AGI-2 problems.
This page has been in 'New model — collecting baseline data. Degradation detection paused.' state for months now. It seems to never say 'degraded'.
If you look at the graphs, the latest benchmarks are showing a pretty significant dip, and they match pretty well with some horrible experiences I've had in recent weeks. You can see token usage steadily going down, matching exactly what the author measured on his own.
> This page has been in 'New model — collecting baseline data. Degradation detection paused.' state for months now. It seems to never say 'degraded'.
Click the part at the end that says "View historical performance". They wait to collect more data about a new model before adding it to the overall charts.
The overall solution rate continues to climb when new models are considered.
> If you look at the graphs, the latest benchmarks are showing a pretty significant dip, and they match pretty well with some horrible experiences I've had in recent weeks. You can see token usage steadily going down, matching exactly what the author measured on his own.
The y-axis is amplified to make differences look larger than they are.
Hover over the dots to see the confidence interval. A 1-2% change means nothing.
Fable 5.1 was released 20 days ago. That's a lot of time to gather data. Since they're still tracking Opus 5 anyway, there is no obvious reason the perf delta section would be disabled. Are you involved with the project?
The Anthropic post points to the latest fix on Sept 12, and the issues mentioned also only affected Sonnet 4 and Haiku. Opus was misbehaving just last week. You are choosing to not see the evidence of degration, 85% -> 75% is a generational dip in intelligence.
Not involved, no. Just guessing. It doesn't look frequently updated.
A lot of these daily-benchmarking sites popped up earlier this year. Most of them have faded away after they all failed to produce the smoking gun that everyone expected. This site survived because it kind of caught a dip one day, maybe.
The results are really rather flat and daily benchmark runs are expensive, so most of these projects give up after a while.
It would change my perception but only if there were a competent and stringent administration in place. I didn't used to have to wonder if the ground beef I was buying was actually 1lb because there were inspections and repercussions, but stuff is kind of chronically underweight these days.
A properly run OWM enables you to stop wondering if you're being ripped off and that's what AI needs because I think it's incredibly easy to just assume we're being ripped off because these companies are all built on a foundation of wonton theft. (Not that I really care about that — I think all information should be free, but still.)
If you go to https://marginlab.ai/trackers/claude-code-historical-perform... there is a very clear downwards trend in the two weeks before Opus 4.7 release. Then a sudden and dramatic drop seven days before Opus 4.8. And now we seem to have entered another decline in the last ten days, beyond the usual noise of Opus 5 scores
> Changes to the Services. Our Services are novel and will change. We may sometimes add or remove features, increase or decrease capacity limits, offer new Services, or stop offering certain Services.
> Unless we specifically agree otherwise in a separate agreement with you, we reserve the right to modify, suspend, or discontinue the Services or your access to the Services, in whole or in part, at any time without notice to you. Although we will strive to provide you with reasonable advance notice if we stop offering a Service, there may be urgent situations—such as preventing abuse, responding to legal requirements, or addressing security and operability issues—where providing advance notice is not feasible. We will not be liable for any change to or any suspension or discontinuation of the Services or your access to them.
You're not buying a gallon of milk or a pound of flour. You're buying hosted software that the host reserves the right to modify.
My comment is what is on the tin. As a consumer, yeah, I find this annoying. But do I want to bring the full force of government regulation on it? That's quite a strong reaction
It will be interesting to see if this changes because presumably AI is using very predictable historical models, but it seems like the climate is shifting into something unseen that we won't have models for?
Climate as a pattern of weather over a long period of time. If the climate is increasingly unpredictable, I would think that it wouldn't really effect our ability to make short-term predictions, like a few days out.
But our ability to forecast weather on a longer timeline, like for industrial forecasting, is calibrated on historical weather patterns. But with weather being more erratic and unusual, I don't understand how AI will be forecasting with the models they have now.
I think you're misunderstanding my point. The headline's usage of the term "forecasting" is not referring to weather forecasting, or climate forecasting. It's referring to forecasting a wide range of possible future events.
For example, predicting which party will win an election, or if there will be a major cyber event in the next year, or the price of Gold in 6 months. Presumably a few of the questions could be related to climate as you're thinking of.
Nobody at OpenAI is closely monitoring token usage or egress, eh? Alright. They should potentially fire a whole team of engineers if that's the case. I monitor egress from VMs that don't have shit on them.
AI companies should be subject to the OWM like any other company that sells a product that varies in weight. Perhaps when a sane administration is re-elected; one that can read history books and comprehend why our regulations exist in the first place. Or have even a semblance of respect for its citizenry.
reply