I hope Apple contributes to an open standard for this, instead of keeping this proprietary. In the emerging world of generative AI, we need this more than ever. Not just iPhone, but ideally most camera systems will have some kind of verifiable capture mode.
The timing of the AI doom narrative could not come as a more suspect time. Chinese labs are releasing open models that are nipping at the heals of the frontier models. Anthropic and OpenAI know these frontier class open weight models can threaten their valuations by changing inference to a commodity where the provider who can serve the best models the cheapest and fastest will win. Few are going to pay $50/million tokens when they could pay <$1/million tokens for similar outcomes. There's little hope investors could recoup the wild investments made into Anthropic and OpenAI, IMO.
> The timing of the AI doom narrative could not come as a more suspect time....There's little hope investors could recoup the wild investments made into Anthropic and OpenAI
Personally, I think we should take the doom narratives very seriously, and act on them, but make sure those actions are not in the interests of Anthropic and OpenAI investors.
This is the correct approach. I think we can do both things - try to add some controls while also ensuring it’s not a complete giveaway to entrenched interests and investors.
The problem with that hope and approach is regulatory capture. Big money is now the only thing that apparently matters in shaping policy, and changing that will require a lot of very sweeping vote and policy changes that may not ever have the momentum to succeed.
Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resorting to magical thinking or underpants gnomes level business plans. Engineering details are what matters here, not wishy washy things you once saw on Star Trek or The Outer Limits. Go.
I don't think we'd be okay with that if it took a few years longer, right?
I thought the Bengio blog post posted here the other day answered that question quite well (ignoring your timeline).
The argument is that in training for the capability to achieve certain goals, you may inadvertently also train for secondary instrumental goals you did not intend, such as aggressive behavior or deception.
I'm actually no sure I buy that this is particularly likely, or necessary, but I found that take reasonable and worth considering.
Sure, unintentional outcomes, fair. But I'm asserting that things which are basically impossible today are not arriving any time soon. I 100% acknowledged the possibility and even the likelihood of rogue unintended behavior of AIs and in fact it's happening already. What I reject is the annihilation of humanity, the curing of cancer*, immortality, and von Neumann replicators heading to the stars anytime soon.
No, they do not. They are both bullshit campfire ASI horror stories written by nontechnical sorts in the wishy washy invented field of AI safety. For example, I want someone with a proven track record in the field of robotics to explain the entire pathway from impressive 2026 Chinese robots to an entire self-assembly food chain from raw ore to von Neumann replicators that can travel to the stars and reproduce themselves, not just assert that ASI will make that happen among so many other unlikely events like curing cancer because reasons because all any of this takes is superintelligence according to these sorts.
But no one likes hearing from the engineers on this, I get it. We are simply no fun. So many would rather believe a time traveling superintelligence will punish a simulated avatar of them for not making the ASI happen sooner. And even if that were true, why am I supposed to care about what happens to a simulated avatar of me?
Ha well, by definition what you want can't be provided!
Of course these stories can only be technical up until the recursive self-improvement part. Then they're necessarily fantasy, as the AI is more intelligent than the story author. Unfortunately, this doesn't make that scenario impossible.
However, we know now from Hugging Face that even without the later sci-fi powers, the LLMs are already capable of causing damage.
It doesn't need them to be able to make Von Neumann probes to e.g. steal their own weights from badly secured OpenAI servers, hack into various Neoclouds, distribute themselves, make Teslas crash into things, hack into all our power and water infrastructure and collapse global civilisation.
Explain how rogue FSD cars take down civilization. And I say that with the opinion that the only good scenes in The Fate of The Furious and Leave The World Behind were the ones with the rogue Teslas. But also, nothing new under the sun really:
And so granted we could probably get a global 9/11 in deeply urban regions out of your scenario, we then bomb the datacenters and we finally address the horrific tech debt in our infrastructure blissfully relieved of the option of continuing to ignore it. Don't think this is happening either BTW but I acknowledge its probability is slightly greater than zero.
To that end, my house is entirely off-grid. And in the event of what you call the end of civilization but I call the big burp inconvenience, I've made friends with my neighbors because community is what really matters during a disaster, and I've been through a few.
Eh, about the biggest one I could see AI causing is the need for a black start on all 3 major US grids. Millions will die before it comes back up, well, if the US pulls out of it at all.
And in that case, thanks for setting up a nice place to live for someone far more ruthless than you. They'll snipe you, and enslave your women and children if things have gone that far downhill.
Or, we could avoid all that by making sure AI labs are properly regulated.
Such an individual would get a quick corrective lesson in the second amendment from my friends and neighbors. But once again, hyperbole. The US grid goes down. That sucks for up to a year and a half.
But hospitals and other critical organizations have generators and the national guard would be deployed to keep them fueled. And yes it will crash the economy, which is kind of a myth anyway. Buy solar panels and batteries, live off a well, and get to know your neighbors if you think this is likely. Also consider being like the Mormons and storing a 6-12 month supply of nonperishable food.
Regulating AI achieves nothing but to hand the 21st century to China. Regulating its deployment OTOH will not be allowed by companies like Meta and Palantir whose gross margin is reliant on it. I'm favor of regulating deployment. The research not so much.
> None of this makes any logical sense whatsoever.
It doesn't make any logical sense to set the bar as as you have, either. I mean if someone could provide the roadmap you're asking for, it's probably already too late.
And while we're talking about sci-fi, it seems fairly likely that an AI that could "kill us all" could fail to build a "entire self-assembly food chain from raw ore to von Neumann replicators." I'd be a civilizational collapse [1] without even birthing something new.
But it makes far more sense to worry about the AI x Late Capitalism crossover story or the havok that could be caused by some immature tech nerd amplified by AI, than far out sci-fi stuff.
[1] IMHO, even a genocidal plague probably wouldn't render humanity extinct, something like uncontacted tribe in a rainforest somewhere would probably survive and humanity would carry on (from square one, in a resource-depleted world).
Fort Detrick and likely its Russian, Chinese, and possibly Indian equivalents have had stocks of bio agents that could give a humanity a very bad day for decades. I'm far more worried about those existing stocks than I am about a tech nerd trying to DIY the genocidal plague.
Which is to say it's already a threat, it's unproven but not entirely impossible COVID leaked from a lab as just one example, but Ebola suggests that it requires more than a rogue nerd to spread a deadly virus global because once people start dying in the streets, the rest hide because they don't want to die. COVID spread because it wasn't deadly to 99% of humanity and now most of the 1% vulnerable to it are either dead or permanently crippled by long COVID so it's ambient now.
I worry about attacks to the infrastructure, but I just don't buy that as a direct route to civilizational collapse.
If someone showed you such a path, would you consider changing your mind? You're always going to be able to find a reason that such an explanation doesn't count if you're dedicated to looking for one.
Yes, actually I would. There's a long list of people in my career path who bet on me sticking to my guns against compelling evidence to the contrary. But I know, I know, that's impossible! Anyway, after they figured out I would change my mind, then they described me as lacking conviction because you can't win, but I digress.
But upfront, it's a detailed path with specific breakthroughs and a concrete plan to achieve or it's just more fanfic from doomer fanbois. Gary Marcus wrote a fantastic critique of AI 2027. Start there.
I've read Gary Marcus's critique of AI 2027. How do you grapple with the fact that superhuman coding arrived on schedule, despite his skepticism that it would? Doesn't that mean that the other predictions are also more plausible than we may think?
I work with Codex and Claude daily. They're fantastic for script coding, config issues, and simple projects. They lose the plot on bigger things to this day. Just last night, Astra decided to cheat to present the illusion of progress until I called its BS. And I was so looking forward to finally working with the AGI. Humans remain the muse, the common sense, and the manager of making these things productive. And as much as I agree with Carmack's suggestion to not become the out of touch Kung Fu Master:
We understand "superhuman coding" to mean very different things, and I think it's a me problem that I can't figure out where you're coming from, because it doesn't sound like our practical experience differs. I'm going to have to think on this.
Define it. I haven't seen anything yet from a coding agent I couldn't write myself and usually write better. But I don't have to gold plate every line of code, just the 5% or so that eats 90+% of the cycles. And this is where the agents are weakest. This does get to one of the reasons I don't believe AI cures death or builds von Neumann replicators any time soon because that requires out of sample breakthroughs. And that's not their strong point.
Just like AlphaFold 2 genuinely increased the weak sequence but strong structural homology detection of proteins, but it did far less for out of sample domains and it provides no intel into the kinetics of folding. Details matter. And nobody likes to hear about them.
Is there a retrospective on AlphaFold you’d recommend? Maybe that’s the insight I’m missing, I didn’t realize it had broad identifiable gaps like that.
I see similar conceptual gaps in coding agents when I work with them, even Astra and Fable, so maybe this a is decent proxy to describe the gap between hype and reality.
> Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resorting to magical thinking or underpants gnomes level business plans. Engineering details are what matters here, not wishy washy things you once saw on Star Trek or The Outer Limits. Go.
Why? Those aren't the things I'm actually worried about, and those aren't the only "doomer" ideas, just the most extreme ones (thus easy to straw-man).
My main worries are about economic disruption, social damage due to the pace of technological change outpacing human ability to manage it, further concentration of power into small elites (along existing lines), and the reallocation of labor towards less personally-fulfilling pursuits.
I also totally buy into the idea that "AI leaders" are playing up certain risks as a marketing strategy. After all these are smart tech people, and those types have a tendency to get really arrogant and be irresponsible.
So the main thing is to slow all this down, so there's a chance of getting it under control, and snatch the payday away from such greedy and irresponsible people.
And China will slow down too? Because China becoming economically dominant will cause everything you fear. While I think that might already be inevitable, I see no benefit whatsoever to regulating AI research. Deployments, sure, and start with Meta, Google, Palantir, Flock, and all the credit agencies and see how it goes. I'm sure I missed some, but I see no societal value in building the Torment Panopticon.
I feel they will…. The economy is so shit, the robots mentioned a few posts up, are largely performative, just like AI. Its just about money. Everything is AI enabled now, from scales at the supermarkets to vending machines, but its 90% people behind a camera in a cheap remote location. In the past three years my salary halved every year. There are no jobs. Just speculation. This thing is gonna crash harder than the housing market.
Tangent: what happened to these staffless supermarkets of 10 years ago? I have never seen them again after .. covid.. weird!
What might not affect you, might affect millions across the ocean.
A society no longer creating bullshit jobs because there is no reason for it anymore. You have some AI.
Right now a whole industry is transforming (again) but now filling the automation gap we were unable to do because of a shortage of labor or expertise.
Our society already has peak employmet reached, now AI takes away all the random digital jobs too. Someone building a website? Takes 5 minutes. Someone doing odd jobs on fiverr with blender or any other tool? Takes 5 minutes now.
Regression can happen very fast. The Romens were a high level culture and an empire. Gone.
Today you have more people on the planet then ever, everyone who might want to eat and who needs to work something for it.
AI already replaced people, already makes a lot of tasks A LOT cheaper and faster and human independent.
We never had a proper discussion as a society. Most countries have a social system but look at countries like the united states of america: if you are poor, you are fucked.
And AI is not the only thing which is now peaking: Physical AI / Robots are progressing fast, its clear today already that in 50 years these robots will be better than humans (50 years is very conservative).
If we have 10 more golden years due to restructuring and rebuilding things and preparing everything for agents and robots etc. what happens after these years?
If we have 10 or 20 more of these years, what do you think will happen to all the new kids (yours perhaps too?) and jobs and everything?
Riots? Famine? Mass migration? Mass death? Slums?
And don't respond with "enough people will survive this" bullshit. No one is saying that earth as a planet is not surviving. Its the same thing with climate change...
Actually, some sorts do assert humanity will be wiped off the planet by AI. I don't share that view either. I think it's ludicrous. We've lived through much worse without any sort of technology beyond fire and a big stick.
But the end of scarcity and disease is a fantastic thing if it happens. If our lizard brains do not bridge that transition and instead attempt to maintain the billionaire status quo, I agree, we're going to go through some things once people start starving, two meals from a revolution and all that. But as much as I think people like Elon Musk and Mark Zuckerberg are unhinged lunatics, I also think they're smart enough to keep people fed and entertained for pennies on their accumulating dollars. Bread and Circuses kept Rome alive for 500 years if you need an example.
>We've lived through much worse without any sort of technology beyond fire and a big stick.
No 'we' didn't. A whole shitload of much smaller isolated communities did. The world added 7+ billion more people since then. Imagine chlorine doesn't show up tomorrow? How many people die in two weeks? How far do you travel when no gas comes out of the pumps. How much will you stand out at night if you're the only ones with power? How fast do working antibiotics run out?
The billionaires you're talking about are mostly ignorant fucking idiots with a god complex. The moment any complex system starts falling apart they are saying as dumb of shit as everyone else (during covid for example).
Most of the pasts anti-fragility is gone. We survive by means of very complex systems and if you lose too many key points in those systems they can totally collapse.
The unprepared bonobos in the west will absolutely go through some things. But we have a national guard and once people aren't commuting to work daily, we have plenty of fuel in reserve (or we will again if we ever conclude the stupid war with Iran). People are a lot nicer to each other in my experience once the lights go out and social and corporate media stop ragebaiting them to hate each other.
1B+ individuals worldwide currently live in squalor. They will barely notice. The real crime is that in the midst of the west's carbon footprint buffet, we won't bother to uplift their lifestyles with the most basic conveniences. And while I am no fan of the billionaires, they absolutely have public and private personas or their business ventures would implode. They absolutely have god complexes, but don't mistake that for genuine idiocy.
Musk did a 180 degree turn around. Killing USAID etc. i don't think he sees normal humans which do not help him or are not aligned to him as humans anymore.
And Zuckerberg bought so much land and bunkers, he is prepared for shit hitting the fan. After all he doesn't care what his platform did to humanity anyway.
And we have a planet full of general contractors who know their way around welding and pouring concrete down ventilation shafts. Many of whom these sorts wanted deported. If they want to build a Per Stenson terrarium and flee into it, by all means do so. And then let's seal them inside, problem solved.
> We've lived through much worse without any sort of technology beyond fire and a big stick.
Lack of technology alone will not wipe out the entirety of humanity, I agree. But we're not talking about just a lack of something. We're rapidly sprinting to give the fewest people the most power over the planet ever. I think it's likely to happen in the 21st century. What happens when the worst case scenario isn't just being left with nothing, but facing an entity that is so rich and powerful that it can use active measures to kill as many as possible? This is more power than any warlord or emperor could've dreamed of in the past.
> But as much as I think people like Elon Musk and Mark Zuckerberg are unhinged lunatics, I also think they're smart enough to keep people fed and entertained for pennies on their accumulating dollars.
So far, every time the billionaire class has been asked about how much money should go to them in society, their answer has always been a resounding "ALL OF IT". You don't get to that level of wealth while retaining a state of mind that allows you to toss a penny to the beggar. Reality is what's holding them back right now, but what would happen if their dreams of total automation eventually come true? How long from then until one of them snaps and decides to use state of the art automated military technology and weapons of mass destruction to wage war against the peasants (armed with sticks and stones) to solve the 'problem' once and for all?
To be clear, I'm not saying any of this is imminent or remotely likely to happen soon. But our society is moving in a direction where something like this could eventually be possible. Giving a human truly absolute power over the entire world has never been tried before.
Other than the tech fascists, does anyone love billionaires right now? Reality is indeed holding them back. And their dreams of total domination are materials science and resource-limited - no self reproducing robots anytime soon.
So with that, we can rule out extinction at the hands of the machines 100% leaving only machines somehow accessories to near ELEs humans could trigger today and could have triggered for decades. And at that point, Scary Potter and his friends need to STFU.
So scaling down to an AI takeover in the next decade, paradoxically, the idiotic pronouncements by Sam Altman and Dario Amodei that AI is coming for your jobs and that they are afraid of their own creations is not the sort of thing that leads to an AI takeover, it leads to regulations and bans. What would would make such an AI takeover viable in the next decade would be a magnificent seemingly benevolent AGI that raised living standards and spread happiness as it enchanted and delighted its way to everyone happily ceding control to the greatest thing ever. Does that remotely sound like the near future to you?
In the words of Ted Chiang "most fears about AI are best understood as fears about capitalism" and that aligns with what's going on right now. So let's skip the intermediary surrogate and attack the worst bits of late stage capitalism, mmkay?
Not a kid - I wish. My point was that (admittedly some not all) societies if faced with a loss of a major utility and supply chains that were dependent on would start to devolve pretty quickly into chaos.
That to me is the greater threat than annihilation due to autonomous weapons, etc.
>> but make sure those actions are not in the interests of Anthropic and OpenAI investors.
> So if the end of the world can be stopped, but the only specific action to do so helps Anthropic/OpenAI (and it's proven) you don't pull that lever?
Your comment has a false premise. If "the end of the world can be stopped," there's no way that the only way would be one that "helps Anthropic/OpenAI['s investors]." That can be shown by a counterexample: nationalize them and leave the investors holding the bag, then take whatever action you're talking about.
Yes, but I'm forcing a tautology in my logic - if in the end it is deemed the only way to stop AI from destroying the world is something that helps Anthropic or OpenAI - then this person is asking to not perform that action.
And unfortunately my tautology is actually closer to being reality than alternatives. Stopping progress even in Chinese models (a deal with Xi for example) but that specifically helps OpenAI/Anthropic.
There are very few scenarios that would not help these two companies by adding regulation. Your counterexample is 100% less likely because it's not legal for the US to nationalize them outright without taking other action first.
The singularity has been termed, "Rapture of the Nerds" since the futurists promoting it sound like evangelicals talking about the end times, both with the Revelations doom stuff, and the utopian stuff that follows.
Why? I was raised as an evangelical christian. End of time sermons are quite common, along with telling followers that they are warriors of christ. This doesn't seem any different than what these MBA leaders are saying "AI will destroy humanity, I can protect it."
It's delusional but it's quite effective, luckily it doesn't work on the general population but you can get a dedicated group of people, and as we see with the current government you don't need big numbers to make a huge impact.
Just stay away from it - if @dang allows it, then someone else will justifiably start saying "on par with shia muslims" in the same way you wrote your text.
I guess but I don't see how it wouldn't be allowed. Evangelicals have quite a stranglehold on US politics and have been influencing the republican party for like 40 years now. Trump
Not being allowed to thread the connections seems odd, it's how we know what these same leaders are going to do next.
Huh? Open weight models have been 3-6 months behind for the last year or so. Astra and Fable 5.1 just came out to widen the gap more, 5.1 is 8 points ahead of the nearest OSS model on Artificial Analysis Intelligence Index. An OpenAI internal model just solved a millennium prize problem.
It’s much more straightforward to me that the calls for regulation are timed with:
1. The Hugging Face incident blowback, especially after third party investigation results
The benchmark comparison to other models was suspiciously missing. The self-comparison is a good representation of progress, but it is light years behind frontier models. Speed is great, but wrong is much worse than slow, IMO.
Yeah of 10 minutes ago. It is shocking how long some seemingly simple things can take. I know there are some things I can do faster than the LLM and some things it can do faster than me. The amount of rambling BS is the exhausting part.
Try different models, it's a breath of fresh air. GLM 5.2, etc. all make life much more enjoyable. They may not one-shot a complex project the same way that Claude can spit out memorized architectures, but that sort of system is always only useful for a one-off prototype anyway, so not much is lost.
Edit: for how to do this, I set up an OpenRouter account so that I could easily switch models, and then ran them in Pi inside of Orca ADE. Orca lets me easily switch from Pi to Claude Code to Kilo to Codex or Hermes or whatever. Pi+OpenRouter lets me easily switch the LLM. All of it lives in a single open source orchestrator to avoid any platform lock in to any AI company going forward, and can even do local LLM should I care to take on that massive hassle.
I think lithium air batteries will be the real inflection point for aeronautics. Li-air has potential density 12 kWh/kg. CATL is focused on this tech. May be some years before they are able to achieve the full potential density. Still, as the other commenters said, the efficiency of electricity make make up that difference.
yes experimenting with it actually, will update here! in fact we've generated most of the tourist spots in sf, should be reflected in the repo soon too
I really wish GLM models had vision capabilities. I've worked around that in the past to use a vision MCP in my harness that GLM can call. It is not the same, but it allows the model to query images.
That's wonderful. I was going off an older version of the Artificial Analysis page for GLM-5.3-Flash https://artificialanalysis.ai/models/glm-5-3-flash. The page is updated now to show that it does support multi-modal image inputs.
Give it a couple days, and there will be plenty of other inference companies hosting it. Don't like z.ai's TOS? Use the model on a provider with TOS that you agree with.
reply