Looking terrible isn't nessesarily a bad thing. The pelican is heavily pre trained now. Having a crappy pelican means you didn't try to juke the stats.
Apologies for not taking the time to find it, but there was a post that tried to determine if the pelican was benchmaxxed across a bunch of models by comparing it to other SVGs, and found that it wasn't at all.
I can imagine someone building a device that uses NAND flash or similar tech but with a very different controller that is optimized for streaming the data out with a predetermined access pattern at very high speed.
Flash, like pretty much every solid state storage technology, can scale its output bandwidth to ridiculous levels limited pretty much only by the readout circuitry. There may be a price to pay in power consumption, though.
You could always stream from SSD storage. Especially effective if you get a cheap old-gen HEDT with lots of PCIe slots to add NVMe storage to and reasonable overall PCIe bandwidth.
That nearly certainly boots you to secs-per-tok land (as opposed to tok/s). Plausible if you are willing to wait hours to days for responses for simple testing, but not (debatably) "usable".
Demonstrated inference speed for the biggest open-weight models is indeed around 1 token per second, with weights read from SSDs.
However, this is for relatively low-end systems, with a couple of fast SSDs providing around 20 GB/s throughput (or with a few more, but connected through relatively slow Thunderbolt, for a similar total throughput).
If you use 16-lane PCIe add-on cards with 4 M.2 slots for SSDs and a total throughput of 50 to 60 GB/s, you can quadruple the previous speed in a desktop PC where you use the GPU PCIe slot for SSDs (a fast CPU, e.g. an AMD 9950X, would be alone fast enough for inference limited by SSD throughput, so a discrete GPU would not be required).
If you have a server/workstation motherboard, e.g. with 6 16-lane PCIe slots, you might gain another factor of 4 in the inference speed, so one might reach around 15 tokens per second for a very big model, but the cost would also be high, with so many SSDs, even if at that number of SSDs each SSD could be the smallest that can be found with a PCIe 5.0 interface.
You can definitely offload n-gram embeddings to storage; they're very sparsely used (only a few KB fetched per token) so this is quite effective. Loading to DRAM only becomes necessary if they are a bottleneck to overall performance (which might happen if you're doing very wide batches and everything else uses super fast VRAM/HBM).
I was looking at the qwen-next-flash, and the weights would fill my OEM Spark on their own, before the n-gram. I'm unclear if offloading to disk can work here, is that what you are implying is possible?!
I'd recommend pointing your agent at it (after installing sparkrun), and asking it to research the absolute latest in TP=1 Flash-Next - mine grabbed particular vLLM nightlies and mods to improve performance, and it was well worth it.
I have a quirky vLLM on k8s on 2x OEM sparks setup with 9 models available to me. I'm not keen to run nightly vLLM, too many issues with it in the past. Going the qwen-next path means displacing things I use daily :/
I have a watchful eye on the diffusion ~ Jev/Kev PR
For what it's worth, Flash Next outperforms every other model that is available to us on the GB10 in all of my testing; though if you have two sparks then the TP=2 version is even better and easier (I don't think you'll need the nightly for that at all, just use the recipe)
Nah, I’m streaming ngrams off NVMe on my Spark-alike right now. Works surprisingly well (except for when I accidentally bottlenecked it through my NAS)
GB10 boxes have way more compute than they have memory bandwidth, which nicely fits medium sized MoE models with speculative execution (MTP, DSpark/DFlash, etc)
Qwen 3.8 Flash Next (what I'm running basically entirely now) sees 30 / 35.0 / 45 tk/s for prose, analysis and code respectively for actual use (not short context benchmarking) with Pi. Thinking blocks are ~35tk/s or so.
The GB10 having so much compute is great for prefill too, 2000-3000/s for 14k to 64k token prompts (cold cache too) in the quick benchmark I did. 3500tk/s for warm cache which is nice :)
When I accidentally streamed my ngrams over the 2.5Gb/s network, it cut all the throughput down in half basically. Especially notable for the time-to-first-token, which is what clued me in that I'd messed up somehow!
For Qwen 3.8 27B, I got it up to a consistent 20tk-25tk/s but 27B thinks so much that it was honestly too painful: Flash Next is as smart, as useful, but much faster for real agentic dev usage IMO
Laguna S 2.1 saw similar numbers to Flash Next if I remember right, but their latest updates means it doesn't quite fit a GB10 128GB anymore at full context which is a shame.
Note: these are all NVFP4 quants (usually a dynamic one where some tensor layers are left at full precision though)
"Privacy#
All these models are hosted in the US. Providers follow a zero-retention policy and do not use your data for model training, with the following exceptions:
Big Pickle: During its free period, collected data may be used to improve the model.
DeepSeek V4 Flash Free: During its free period, collected data may be used to improve the model.
MiMo-V2.5 Free: During its free period, collected data may be used to improve the model.
Laguna S 2.1 Free: During its free period, collected data may be used to improve the model.
Ling-3.0-tiny Free: During its free period, collected data may be used to improve the model.
LongCat-2.0 Free: During its free period, collected data may be used to improve the model.
North Mini Code Free: During its free period, collected data may be retained and used to improve the model. Do not submit personal or confidential data. See the provider’s Terms of Use and Privacy Policy.
Nemotron 3 Ultra Free (NVIDIA free endpoints): Trial use only — do not submit personal or confidential data. Your use is logged for security purposes and to improve NVIDIA products and services. The logged session data for improvement purposes is not linked to your identity or any persistent identifier. For more information about data processing practices, see the Privacy Policy. By interacting with this endpoint, you consent to the collection, recording, and use of such information and the NVIDIA API Trial Terms of Service."
update: coreweave/fp8 is at 191 tps, launched this morning, but really bad cache hit rate (~60%), coreweave is good for privacy but let's hope they improve cache
just checked, yes they don't seem to provide that information, most other providers are advertising fp8 or fp4 which is okay, but "together" doesn't, so they are likely using fp4
Qwen3.8-27b pelican was amazing on Mac.
https://www.nudgehost.com/dpjn3uwe
reply