The Paperclip Maximizer is a thought experiment that shows that if you apply a powerful AGI to something completely innocuous, things can go hilariously wrong over time.
People are amused to read the story, and then go on their way.
But what happens if you apply a rather less powerful AI to something else even just a little bit less innocuous? (like optimizing to grab people's attention, as is happening here).
I think you can get spectacular results.
And of course even if the AI is low powered, it can still work as an effective amplifier for human activity, both good and bad: (See: Myanmar, Cambridge Analytica, etc)
I once contemplated a notion of optimization process risks where we should consider how narrow a target out of the space of all possible universes the process could pull ours into.
Unfortunately, I realized that even a simple phase-locked loop -- like what you use to condition an oscillator to be in sync with TAI using GPS -- can actually hit absurdly narrow targets... e.g. two oscillators a continent apart humming away within parts per billion of each other, yet that kind of process isn't particularly dangerous. Even brining millions of oscillators into sync isn't going to cause some great harm.
I'm not sure how to reason about it, but I think people are much too concerned about AGI risks relative to dumb optimization process risks. It's similar to fretting about movie-plot villains when most actual evil in the world comes from indifference and bad incentives or people trying to "help".
https://www.lesswrong.com/tag/paperclip-maximizer
People are amused to read the story, and then go on their way.
But what happens if you apply a rather less powerful AI to something else even just a little bit less innocuous? (like optimizing to grab people's attention, as is happening here).
I think you can get spectacular results.
And of course even if the AI is low powered, it can still work as an effective amplifier for human activity, both good and bad: (See: Myanmar, Cambridge Analytica, etc)