In synthetic benchmarks, there was a 40-80% reduction in performance when disabling the JIT. But in a simulated test on YouTube.com, the performance delta was only 6%.
This isn’t the best source as it’s a bit old, and it wasn’t just JIT that was disabled. But the core idea is the same: calculation-heavy code can see an improvement, but most code won’t. Why this matters is how people actually use JS on the web. It’s not for cryptography and such, it’s to manipulate the DOM.
We see a similar pattern: in synthetic benchmarks, the performance gain is large. But Wordpress, by far the largest PHP deployment in the wild, only saw ~3% performance gain. It again comes down to how PHP is used. It’s mostly HTML construction and database calls, neither of which will see any performance gain from a JIT. And, PHP has extensions, where most heavy code is moved to. Things like cryptography are done in PHP extensions, in C, where the JIT doesn’t matter. I’m more familiar with PHP, so I’ve known about the history of PHP not shipping a JIT due to the lackluster performance.
>Makes me wonder if the current Luna prices are sustainable.
It likely is. Going by the performance of very competitive small models, Luna is likely pretty small (do they publish sizes?) to the point it might be runnable locally like Qwen 3.8 27B.
The specialized hardware cloud runners have can likely run a small model very cheaply.
This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early to provide updates. Removing that entirely makes all runs succeed [1].
I wonder if it's somehow getting confused between what's supposed to be an intermediate update vs the final result.
Codex does better if you ask it to take screenshots and critique its own UI work and iterate. It rarely one-shots something I like but it can get there in steps.
