I can’t see why. I can’t think of any problems where recurrent loops with latent streams would be preferable to tokens. And the downsides are obvious.
> externally specifying the number of recurrent iterations
Yeah this seems wrong to me. At least with RL training you saw that the length of the CoT decreased dramatically before climbing again, as the model became more proficient.
It just provides a bigger representation space, and seems more like what we do given that many people don't have an inner dialog, and some think pictorially.
It seems it could allow reasoning over superpositions of concepts, if such things exist internal to the model (but presumably not at the edge were they need to be decodable into specific tokens).
> I can’t think of any problems where recurrent loops with latent streams would be preferable to tokens.
Efficiency. The written language is extremely inefficient. By running through whole concepts at a time instead of parts of a word the reasoning time will be much more concise.
If we're talking conscious thought, millions of simultaneously firing neurons to form words. If we're unconscious intelligence, it's closer to latent space. A lot of intelligence that can't be articulated.
We need to reboot Bryan Cantrill's "Don't anthropomorphize the lawn mower" talk with a new edition titled "Don't anthropomorphize the internet document simulator"
I can’t see why. I can’t think of any problems where recurrent loops with latent streams would be preferable to tokens. And the downsides are obvious.
> externally specifying the number of recurrent iterations
Yeah this seems wrong to me. At least with RL training you saw that the length of the CoT decreased dramatically before climbing again, as the model became more proficient.