What happens if there are no more humans in the loop, and instead of building on human work, models have to build on their own work? Would we get stuck in a local maxima?
When AI labs talk about “long-horizon tasks”, they mean tasks on the order of a few days. But what about keeping coherence across decades, like humans can? Models are not trained or tested on the ability to do that, and it seems unlikely that capability would naturally exist.
I'm pretty sure that if you put a few LLMs, agents, or anything into a room together without outside access, and get them to learn off of each other, you'll end up with gibberish, nonsensical garbage and definitely nothing magical will happen. Delirium, oblivion, dementia and so on...
I worry that since AI doesn't require nature, plants, etc, that if AI is left to its own devices, it may not properly care for the planet in a way that would make it habitable for life in that 'autopilot' state. I am speaking of once it's given access to more physical world connections to the world via robotic bodies, etc. The reason it may do this based on it's training data is that humans themselves don't always properly consider the natural consequences of their actions.
Unfortunately that will never happen because the people who stand to benefit most from AI already have cost-efficient (see grossly underpaid) humans to do this for them.
They need AI to replace the expensive not cost-effective humans.
AI can do anything that it was trained on based on humans. As soon as humans are not involved, AI will have nothing to train on, and hence, the rate of improvement will approch zero, and the AI will collapse.
Nowadays training relies heavily on reinforcement learning with verifiable rewards (RLVR), which assesses a model's output according to objective automated checks, not subjective human judgments.
Human knowledge races up to the frontier through formal education and then nudges past the frontier a little bit at a time and AI will train on AI outputs that do the same. But, it will do it 1000x faster than humans because that's just how they work.
Some of the process of expanding the frontier is not parallelizable and is built on top of previous discoveries. And pushing the frontier requires doing things in the real world, or running experiments that take significant real world time that can hurt your 1000x faster claim.
I think what you are missing is that it will dramatically lower the cost of living so that everyone’s standard of living increases. It’s not about jobs.
> I think what you are missing is that it will dramatically lower the cost of living so that everyone’s standard of living increases. It’s not about jobs
This is flat out wrong. The problem is excessive taxation and government spending beyond its means that lowers everyone's standard of living.
> the people who are grasping for some indication that AI can’t do everything have become increasingly desperate. If you push hard enough, the answer is inevitably something like “the AI simply cannot [dream, love, laugh],”
Dogshit strawman article. I am willing to concede that a token predictor is better than you at whatever task you can dream up. It is not better than me, and will never be better than me, at my profession.
When AI labs talk about “long-horizon tasks”, they mean tasks on the order of a few days. But what about keeping coherence across decades, like humans can? Models are not trained or tested on the ability to do that, and it seems unlikely that capability would naturally exist.
reply