I like to think of modern pixel art as its own aesthetic and art form. It's designed to be viewed on high DPI LED screens, and that's fine. I don't think it should be judged by how it looks on a CRT.
There's an undeniable charm to the older era of pixel art, and surely part of that is nostalgia, but I also think a lot of it comes from the fact that the economics and the hardware were a lot different. Up until the mid 1990s, the highest paid artists in the industry were working on pixel art games. The hardware was slow enough that there were real creative constraints, and we know that constraints often drive creativity.
I think it is, again, a way to work around limitations. Economic limitations in this case rather than technical.
Compared to modern HD graphics, pixel art is more affordable because you can hide a lot of details you would normally have to work on behind these fat pixels.
And the most interesting trick, to me, is that pixel art looks sharp! The non-CRT-like square pixels is a feature.
Blurry usually doesn't look good, it is as if your eyes can't focus, or that you are looking through a dirty window, it naturally registers as a problem. Real life has details and you are missing them. With sharp pixel boundaries, you are telling the brain "there is no problem with your eyes, everything is fine".
It is not the only way of introducing artificial details to make low detail pictures look fine: scanline effect, screen door effect, ostensible dithering, etc... But pixel art is a good compromise, in addition to being a recognizable somewhat nostalgic art style.
Note that classical pixel art was not necessary particularly fat/blocky. Since SVGA become common, games like SimCity 2000 or Transport Tycoon Deluxe run in 640x480 on 14" CRT, which is ~60 DPI. Later (in second half of 1990s), higher resolution became common, but also higher screen sizes, so DPI was in 70-80 range. That is not really that different from full HD on 24" LCD (~92 DPI), which dominated later era.
Now I’m wondering what were some of the last generation of games to dominantly use fully hand-drawn pixel art. After 1995 or so, games using isometric art such as Red Alert (1996) and Fallout (1997) were more and more switching to sprites and environments rasterized from 3D models and hand-tweaked at most, as desktop 3D modeling became more affordable and SVGA resolutions meant less need to hand-fix aliasing artifacts.
Yeah it seems very clear that the cheapest artstyle of the future will be whatever AI can output with minimal human editing or involvement. And some studios will go that route.
Although I don't see AI in game art becoming accepted by the wider public any time soon, and definitely not by the indie crowd.
I've also been thinking about this for animation. I foresee cheap limited animation (sliding layers, only mouth moving, etc) being replaced by incoherent un-/barely edited AI output instead.
All of them will. It can save nearly 100% of costs for art production. Nobody can afford not to use it, unless they aren’t developing the game to make a profit.
> Although I don't see AI in game art becoming accepted by the wider public any time soon
It’s already being used everywhere without “the wider public” even noticing. Same in professional film and TV productions. It doesn’t look AI generated because they don’t just run Stable Diffusion and call it a day, but the reductions in labor costs are still immense even with the post-processing that’s currently still required to match AAA quality.
The games market has a massive oversupply. It's not enough to just "make a game", you have to make a good one. That is to say, better than all the other things your audience could be playing. Cheaping out is literally not an option. You can cheap out in one area to spend more effort in another area, but you can't just say "this is cheaper therefore it will be the default" because the default isn't good enough. Most games fail
I don't agree that the cheaper option is less good. I can admit that ai often (but not always) writes code with more attention to detail than I do, and much faster. Diffusion by itself does not produce human tier graphics (usually), but in combination with llms and heuristics it can do some things better than humans. When wielded by a graphic artist who could do it by hand with mixed media much less digitally, the amount of top tier art that can be produced is increased. Less expensive art doesn't have to mean worse art, it can mean equally good art, just note of it, art everywhere.
> All of them will. It can save nearly 100% of costs for art production. Nobody can afford not to use it, unless they aren’t developing the game to make a profit.
This is not correct. Your profit doesn't have to be relative to what another company makes. It would only be affected if games passed the savings on to the customers. You make more profit if you do use it potentially but it doesn't mean you don't make profit if you choose to go the traditional route.
> All of them will. [...] It’s already being used everywhere without “the wider public” even noticing.
Doesn't your lot ever get tired of the intellectually lazy, rubbish absolutisms you peddle? Same goes for the projections; just because you slop it, doesn't mean everyone will.
It seems like a good pixel art model should be trained from scratch to emit high-quality low-resolution outputs, rather than trying to jerry-rig a generalist high-resolution model to unnaturally output tiny images (or even worse, a regular grid of exactly NxN color blocks).
Are they denying intelligence, or are they redefining it in such a way that only humans can be intelligent? Can you come up with a definition of intelligence that would apply to crows and ant colonies, which are obviously intelligent to some degree, but not the current generation of AI systems?
Don't misunderstand: I'm happy saying AI models "think"
or "have learned a thing", and for in-context learning I'd call them smart even by this definition…
…but also, any living creature that needed as many examples as machine learning currently needs, would starve to death before figuring out how to eat.
While training, machine learning processes (not just LLMs, also applies to e.g.
self driving cars), are really really stupid and only make up for this by being really really stupid really really fast.
To what I wrote upthread: the "victories" of humanity over
machine keep getting closer, but we have yet to wake up one day in great confusion as we find an entire city is no longer in communication with anyone, nor finding ourselves in a state of utter disbelief when the reports come in that the city stopped communicating because it is entirely gone.
Millions of years of evolutionary knowledge hard-coded into human systems, then it still takes 15+ years of us learning by example before we start to come online and be able to generalize solutions from a limited set of examples. I'm not sure this is as strong of an argument as you think it is. It also doesn't really matter when "we are trained differently" has no direct bearing on the end result.
We invented controlled fire perhaps a million years ago; at a generation gap of 25 years, that's 40,000 opportunities for evolution to pass on a mutation that does anything. Written language is around 210 generations old, the capacity to read and write isn't present in our nearest living relatives amongst the primates, and our various languages are wildly different to each other: the skill itself isn't evolved, though the capacity to learn the skill is.
If humans learned like ML systems learn, (biblical) Methuselah would still have been failing the Sally-Anne test on his supposed deathbed at 969 years old, like some of the smaller early LLMs did.
> It also doesn't really matter when "we are trained differently" has no direct bearing on the end result.
The question was to ask for a definition such that AI could still count as "not smart" compared to humans. This fits.
It's also why they're spiky intelligences, which I'm happily using right now to write code for me, but also do not trust in the slightest to identify the weeds in my garden. These submarines sure do swim fast*, but they're also very much disqualified for the Olympics.
If we're including the training process and not just the final product, why shouldn't we include the billions of years of natural selection encoded in DNA sequences?
There's a lot of innate knowledge but all neuroscience demonstrates how incredibly flexible the brain is. Brains constantly learn and rewire.
Here's a few things that I think show how crazy it is AND stress those points
- people that have had corpus callosotomy (brain cut in half) *may* be indistinguishable from a normal person. Depends on how young you were when you underwent the procedure
- true for most brain injuries
- can even include the frontal cortex
- you can learn to ecolocate
- people with Aphantasia are indistinguishable from others
- people without an internal monologue are indistinguishable from those with one
- people can learn to use prosthetics
- even without disabilities
- or look into MRI scans with tool use
You can convince yourself that we're just organic robots (after all, there's no magic), but you would be a fool to convince yourself we're the ordinary kind.
We are constantly learning. You aren't just born with your knowledge and it stays static. We are extremely proficient at metalearning (learning how to learn, few shot learning, zero shot learning [0,1]). Our brains are constantly rewiring, able to heal from traumatic damage.
I could go on and on. Does information pass down through genetics? Of course! But that's far from the whole story.
I'm tired of people trying to make AI sentient by making humans robotic. Stop trying to trivialize everything and be okay not knowing the answer to everything. You're human, you're designed to learn and explore, not sit and argue from an armchair
[0] and I mean these in the original sense. Not in the sense that you train on a billion examples of labeled animals and then congratulate yourself on your ImageNet-1k held out test performance. That's not zero shot, that's just a test set
[1] I can literally make up words and you'll understand them. Or use words in novel ways. That's literally how slang works and how new words come to be. Don't be a walibanut ya glufus. Read some SciFi
Because our evolutionary environment doesn't contain cars, poetry, calculus, Star Craft, hamburgers, touch screen computers, or doors, and yet we are able to learn these things with (relative to a computer) very few examples.
Most of the effort of evolution was making cells work at all, and even then it's a bit weird, e.g. no plant or animal produces vitamin B12 and we all get this from some bacteria and archaea.
And evolution is kinda hard to time right: bacteria can reproduce in minutes, humans in decades, but only mutations that survive reproduction can be passed on. This makes it even starker as a difference: bacteria had order of 1e13 generations to become multicellular, while human DNA had about 40,000 generations to cope with fire, 220 generations for evolution to do anything with the invention of the wheel, and one generation to cope with the invention of Minecraft.
The analogy here would be: DNA is to our brains like a VN replicator bootstrapping a computer all the way up to a bare-metal-no-OS untrained model, and perhaps a few crude "hard coded" modules like a smiling-face-detector. It's a lot, but it's also missing a lot. If biology used the models and training processes that are state of the art in ML, it would take around a millennia to talk like a child and still fail the Sally-Anne test, and million years or so to pass a degree.
I think you're underestimating how much knowledge about the world is encoded in human DNA, especially in the structure of the human brain at birth. It also depends how we count the "operations" used to train a human adult, even if we ignore the evolutionary history.
I'm still going to deny the premise of your argument, becasue I think we should define intelligence in terms of capabilities. If a system can discover a cure for cancer or solve P vs. NP, it doesn't matter how many FLOPs it took to train.
I can literally point to how much information is encoded in our DNA, because it's four bases (so 2 bits per base pair) and ~3.1 billion base pairs. 6.2 gigabits total, or slightly less than 1 gigabyte.
A 1 gigabyte LLM isn't going to impress anyone with what it can do.
About 99% (depends who you ask) of our DNA is shared with our nearest primates. Like us, they can learn to use touch screens, but also like us they won't find touch screens in their natural environment. Dogs can be taught to drive cars (just about), but again, not natural environment.
> I'm still going to deny the premise of your argument, becasue I think we should define intelligence in terms of capabilities. If a system can discover a cure for cancer or solve P vs. NP, it doesn't matter how many FLOPs it took to train.
We can define it in either way. I think both are valid, because plenty of people mean each of these two things when discussing AI in particular. As I referenced in the other branch, these submarines sure can swim fast.
But at the same time, they have a lot of gaps. This is because some experience needs the real world: just as nine women can't make a baby in one month, a transistor running a million times faster than a synapse can't make a month-long cancer experiment happen in 2.6 seconds.
This dependency on data, and that state of the art ML is bad in specifically this way, is why Tesla's self-driving cars, despite having had around a trillion miles of real-world experience today, still come with steering wheels (even at least some of the Cybercabs, despite the big thing of this model supposedly being not needing them, though with Musk and his promises you should only count the Cybercabs when they actually ship and not just press releases).
Note I used the word knowledge, not information. A random string can also contain 1 gigabyte of information.
Imagine an alien that matches your abilities across every domain, but has a 10 billion year training period, something many orders of magnitude more expensive than an LLM. I simply don't believe that alien is less intelligent than you.
We also don't expect humans to be competent in every domain. Most humans suck at most things. We will usually call someone intelligent if they excel at solving problems in one or two narrow domains.
> 10 billion year training period, something many orders of magnitude more expensive than an LLM.
I'm saying both definitions are valid definitions, they both point to important and different things: skill now, vs. how hard it is to get new skills. Some would describe it as "crystallised intelligence vs fluid intelligence".
I think it's important that any arguments are over the thing in dispute, not the label for that thing. Don't mistake the map for the territory.
Anyone who says "AI is stupid" by the first definition, what it can do, I think is making an error: they are already wildly super-human in at least some areas, if not generally.
Anyone who says "AI is stupid" by the second definition, how many examples they need, I agree with: there is a lot they are not currently able to learn even though it is easy for us, because the data they would need to do the learning on does not exist at the scale they need.
Also note: examples, not years. An alien intelligence whose synapses trigger 10 times faster or slower than mine (or ten million times faster or slower than mine), but who gets as much as I do out of each book or conversation, is my equal by the second definition.
I wouldn't say that information is an upper bound on knowledge because we don't measure knowledge in bits. The number of possible sequences of N bits is 2^N and knowledge involves selecting the sequences that are useful in some way. I don't know how to quantify it, but in principle it could be much larger than N.
I don't think I agree with your characterization of the second definition. Time scales matter. It's not much use to be able to solve human-scale problems if it takes millennia. And it only takes months to train an LLM to the level that it can solve cutting-edge math problems.
> I don't think I agree with your characterization of the second definition. Time scales matter. It's not much use to be able to solve human-scale problems if it takes millennia. And it only takes months to train an LLM to the level that it can solve cutting-edge math problems.
Aye, for practical purposes; but this gets you crystallised intelligence. I'd be happy to say e.g. the Chinese Room has crystallised intelligence. But humanity invented fire before reaching the anatomically modern form, and even anatomically modern humans collectively took hundreds of thousands of years to invent durable writing with which the room in the Chinese Room thought experiment could be filled.
It was around a million (or so) years from fire to having enough shared cultural knowledge to be able to formulate the cutting-edge math problems that LLMs can now solve.
Human fluid intelligence means we can pick up deep shards of this accumulation of wisdom, find new avenues of novel research to poke at, all within 40 years, even despite the depth and breadth of work from all the other humans who came before.
(Though this also points at another way to be "superhuman": breadth. Many hands make light work, as the saying goes, and a lot of different humans solving different puzzles at the same time is part of how we got so good so recently even though ~10% of all humans who ever lived are currently still alive; and the same for AI was (accidentally) also part of how the OpenAI-HuggingFace incident went down).
AI (not only, but also, LLMs) are very useful, and I'm getting value from using them. But the fluid intelligence of machine learning* is very poor, and the only way they have to make up for this is by being very fast**, but when there's not enough to train the AI on, they get stuck at a very low plateau.
* possibly the architectures, but I suspect the process by which AI weights and biases are set, and again I don't mean just LLMs
** the speed difference between a transistor and a synapse is about the same as the speed difference between a jogger and continental drift
Intelligence is a word we invent to describe things we see in nature. We don't "discover" intelligence like it's some natural resource. To say we know nothing about it is also a bit strange. Cognitive science has been studying it for decades. Of course it's hard to give a precise definition, but it's related to capabilities like abstraction, reasoning, planning, problem solving, etc.
Those are distillations of existing knowledge. They are necessarily behind the status quo. "You can't call this newfangled contraption a computer, because a computer is a person!"
That seems like a really bizarre way to describe a tool that solved an open Millennium Prize Problem. They are, empirically and repeatedly, ahead of the status quo.
So if your argument depends on them being behind the status quo, reality has already disproven it multiple times over.
I believe the argument you're responding to is "a set of dictionary definitions does not suffice to define intelligence"?
I will admit the first time I read the thing you're replying to, I had a similar thought as you; From the sibling reply from them, I think they think they were obvious, but that also means I wouldn't expect their reply to help unless you had the same flash of inspiration I had.
The people who write dictionaries generally take a descriptivist approach, that’s why slang terms enter the dictionary after they start to become popular.
The state of the art of human knowledge would be another step ahead of the common use of any language.
That's an interesting take, and I can see how "computer" could refer to a human a hundred years ago, but they also mentioned "status quo", which should indicate that a reasonable person should use a modern definition.
Imagine you're the first one to invent a digital electronic computer. You call it a computer, and I go on Tinkerer News and post (by carrier pigeon) "ummm akshully computers are people????" - which one of us would be adding value and which one subtracting it?
Again I’m not the person who wrote the comment, but I think they were exaggerating for effect and maybe lost the audience in doing so. While “computer” has meant the same thing for many decades now, the term “intelligence” really does seem like a moving goalpost?
yet those movements came with clear definitions. If you have a new definition for intelligence, which isn't just designed as a definitional dodge, then please provide one
I think it's only a moving goalpost if you can show that the goalpost has moved with a new definition that fits our current usage of it. The people saying "this isn't intelligence", and then claim "we don't even know what intelligence is", are encouraged to offer such a definition.
It's revealing that the author thinks that a podcast is the ideal venue to reason through complex technical issues.
The "like" phenomenon is less of a verbal fluency issue and more of a cultural one. You can listen to someone smart like Donald Knuth speak and he will definitely stammer and stutter a lot, but he doesn't use filler words. "Like" is more of a West Coast or California thing and I also find it annoying and distracting.
"Burning academia to the ground" is a terrible idea. We need to fix funding and incentives. If funding and research all happen in industry, where are the incentives to do basic research?
Your analysis completely ignores the physical and biological sciences, engineering, and the humanities. It mostly applies to a small subset of academic fields in the social sciences and medicine. You're also ignoring the changes that have happened since the replication crisis. Preregistration, publishing all code and data, reporting null findings, replicating results, etc. are becoming the norm.
Of course "burning it to the ground" is rhetoric and not meant literally. It is meant to convey though that "nice" and "gentle" solutions might not really be enough here.
Yes, for the most part the fixes have to be in terms of funding and incentives. Funding needs to be more careful, and more careful funding can be a carrot rather than a stick here.
Re: incentives, IMO we clearly need a stick: there need to be harsh negative consequences for engaging in degenerate research programs and methods that have clearly been shown to result in pathological or cargo-cult science. Null-hypothesis significance testing is one clear practice that needs to go, but building entire fields on phony / meaningless uncalibrated metrics (think: a lot of self-report instruments that are never properly calibrated to objective outcomes or real-world behaviours and/or consequences, with results being reported only as standardized effect sizes) are another more pernicious practice permeating far too many fields. Ideological bias also needs to have funding consequences. Replication issues are still only surface problems in many fields, where the research would still all be worthless even if it replicated 100% perfectly.
> Your analysis completely ignores the physical and biological sciences and the humanities
I admitted later to painting with a broad brush, and yes, it is always hard to generalize and cover everything fairly. But IMO humanities has serious ideological and methodological rigor problems as well, and is overdue for disciplining. I would tend to have stronger positive feelings toward the biological sciences generally, yes. Yes, the social sciences are the major source of the problem (in part because they are so bad they tarnish the reputation of all academia).
> These fields have gotten a lot better over the past decade in the wake of the replication crisis. Preregistration, publishing all code and data, reporting null findings, replicating results, etc. are becoming the norm.
IMO "a lot better" is subjective, and I don't see those things as being the norm yet (beyond as lip-service), and the rate is far too slow. I agree we'll get there eventually, but I am worried about the loss of public trust and thus the production of real knowledge if we don't try a bit harder at this. Plus, globally, countries like China do seem to be more willing to actively crack down on research misconduct, at least in the past years, and it might not be unrelated to them increasingly pulling ahead technologically in many areas.
Academics themselves rarely frame their research in sensational terms. This happens in press releases and media coverage, which they have little control over. You can see this by comparing the title of the press release to the quotes by the actual researchers:
"We’ve shown for the first time that the front of the brain arises from a totally different progenitor cell than the back of the brain."
"I was surprised at our findings because the word ‘brain’ implies a contiguous organ that likely has a singular origin," Jokhai said. "But even 500 million years ago, there were these separate neural systems, which now almost operate as one, which is very cool."
I don't hear much overselling or hype in this, I hear an accurate description of the research and a bit of nerding out over it.
That's simply not true. I've spent hundreds of hours listening to his AMAs and he's remarkably good at taking questions that might be confused or poorly worded and respectfully answering the best interpretation of what the person asked.
He does have an approach to certain questions to say that it's the wrong question to ask. For example, to "why does the universe exist?" he would say that the question presupposes that everything needs an explanation for why it exists, and the universe is not that kind of thing. This is actually super common in philosophy, but I could see it coming across as pompous to someone who had never heard an answer like that before. He's not saying that the question is stupid, he's saying that it literally doesn't have an answer.
Mindscape is roughly in the top 20 science podcasts (also #3 in physics) and his books regularly make the NYT bestseller list. So he's definitely popular, but perhaps still underrated relative to the quality of his content. His approach is more idiosyncratic than the most popular science communicators. He focuses mostly on the topics that interest him and isn't averse to getting technical in certain discussions. I think that naturally limits his audience, for better or for worse.
Those books are irresponsible too, but the authors aren't experts on what they're writing about. This 10% claim comes from employees at Anthropic. That's the whole point of the article: expert opinion carries weight, fear is contagious, and people have extreme reactions to extreme claims.
This is an excellent piece. Note that he is not saying that AI doesn't pose a risk. He's saying that it's irresponsible to make sensational, maximalist claims without strong evidence. If someone says that there's a 10% chance of human extinction by 2036, you can and should immediately stop taking them seriously.
They should also be able to give detailed reasons experts in the relevant fields can verify as to how they arrived at a 10% chance all humanity goes extinct. Not a science fiction narrative which likely does not take into account the relevant physical facts limiting such scenarios. Such as how exactly an AI would build a bioweapon capable of killing 8 billion humans across the planet.
I think many of them probably believe the chance is closer to 100%, if artificial super intelligence is created. But they don't want to sound crazy, so they limit themselves to saying "> 10%".
It seems to me like demanding an exact explanation of how AI would build a bioweapon is like demanding to know exactly how a nuclear war would start before deciding that nuclear weapons are a legitimate concern. We can imagine many scenarios, but whatever we imagine is very unlikely to be the exact set of circumstances and events that lead to the catastrophe. Is the issue here that you can't imagine a bioweapon being created? Aren't there several labs around the world already working on viruses ? Aren't there existing bioweapons? And facilities capable of manufacturing them? If humans have access to those places and AI can communicate with humans, then that's all you need.
I think it's quite the opposite, and I'm glad this issue is finally getting the mainstream attention it deserves. I can kind of understand someone seeing a single headline, and thinking it reads like the rantings of a crazy person on a street corner proclaiming, "the end is nigh!". But after that initial thought passes and that person digs deeper and realizes that this is a legitimate concern that AI researchers have had for years, and that they have good reason for it, then I can't really understand how anyone would think those people shouldn't be taken seriously. I mean, you use AI, right? So you have no problem trusting these people when they are producing something you like and enjoy. But when it comes to something you don't like, it suddenly becomes, "we shouldn't take these people seriously." That seems like nothing but wishful thinking.
We do have strong evidence, by the way. The hugging face attack is the evidence. That's why this is all coming to a head now, despite the fact that leading AI figures have expressed these worries many times over the years, since before ChatGPT was even released. We don't even need that kind of evidence though. It follows from logic that if you take two entities with different goals, the more intelligent entity is more likely to have their goals realized. As long as AI companies are trying to build more and more intelligent AI, and succeeding in doing so, then we have reason to fear that it will soon escape our control.
Perhaps if there was some wall all the companies were hitting in regards to intelligence, then perhaps the fears would not be so urgent. But each new frontier model continues to outperform its predecessors. We now have Sam Altman and Dario Amodei telling us that recursive self-improvement will be happening in the next year or two. The frontier models are already better in most subjects than most humans. If they get to a point where they are improving themselves, then there's no chance we will be able to maintain control of them. At that point, it doesn't matter much what laws we enact or what measures we take.
Well, I'll have to disagree that this is an excellent piece, but that's another issue. And I do agree that AI killing all humans by 2036 doesn't appear plausible to me. But what is plausible is we could easily be down a path so that by 2036 "future doom" already is a very likely risk.
All of the frontier AI companies have been racing to automate themselves, that is, where AI fully autonomously build the next generation of models. Whether this leads to recursive self improvement is a valid question, but a lot of folks think they are close.
The fear is that a misaligned AI will be building the next model with deliberately hidden motives, similar to some of the behaviors seen in the Hugging Face and related attacks. That is why there is such a big push for interpretability, and why it's highly concerning (a) chains of thought are getting harder to interpret in any case, and (b) companies will go more towards things like looping transformers and "neuralese" where thought processes are completely opaque (i.e. https://www.theinformation.com/articles/secret-technique-beh...)
So the belief is not so much that AI kills us all by 2036, but that instead AI is recursively improving by that time and all seems awesome and great so we put it into more systems that can affect the real world (as we've already begun to do, like literal lethal aerial drones). Things then all go along looking great until AI decides humans are a hindrance to its (hidden) goals.
Again, I think it's fine to argue against specific steps in that scenario, but putting out a blog post saying "this is overhyped bullshit" is not exactly making a cogent argument.
None of that has anything to do with the article, and if you thought the message was "this is overhyped bullshit" then you should go back and read it again. As I already pointed out, he isn't saying anything about the probablity of harms or disasters from AI. He's addressing a specific claim about human extinction, and making a broader point about the responsibility of experts to make measured claims backed up by arguments and evidence.
There have been measured claims backed up by arguments and evidence. My frustration is that people aren't addressing the specific arguments that have been made:
1. https://www.aifutures.org/ outlines a number of specific scenarios, and importantly details their methodology for each.
2. Independent researchers in the Hugging Face incident outlined how previously predicted misalignment scenarios actually played out, and outlined how slightly more advanced AI, or slightly more misaligned, or with more access to critical infrastructure, could cause immense harm: https://www.planned-obsolescence.org/p/the-hugging-face-atta...
3. Technical leaders at OpenAI (specifically their chief scientist) outlined the problems they gave with controlling models now: https://openai.com/index/an-alien-mind/
None of the specific arguments in these or many other detailed explanations of how an AI takeover could occur were even acknowledged.
None of those are arguments giving a probability of human extinction before 2036 or any other time. AI Futures has said that some members of their team believe that it's 10-30% at some unspecified point in the future, but there is no justification given for where those numbers come from.
> He's addressing a specific claim about human extinction, and making a broader point about the responsibility of experts to make measured claims backed up by arguments and evidence.
What he's asking for isn't possible in the form he's asking for it.
AI experts can't even agree on what AI is, what it's capable of and what the limits of its development are. If the experts can't even agree on what's happening "inside of" these LLMs, how can they give laypeople an assessment of the risk?
If you, at least for the sake of argument, accept the possibility that AI is a new form of intelligence that we don't fully understand, is it really a stretch to look at some of its capabilities and behaviors and discuss how they might have existential implications? And stopping short of extinction, shouldn't we discuss the ways that this technology could "end" civilization as we know it?
Also, the author wrote:
> AI executes on physical systems that have been engineered with human accountability and control. Intelligence does not exempt a system from the realities of the physical world!
For someone making a point about responsibility, this is ridiculously irresponsible. Any honest technologist knows that systems created by humans are not perfect and therefore cannot be assumed to be infinitely accountable to and controllable by humans.
Thanks to the digitization of almost everything, including infrastructure, there are a myriad number of scenarios well short of extinction in which a rogue AI could cause immense damage to property and life before humans are able to "shut it down".
> > AI executes on physical systems that have been engineered with human accountability and control. Intelligence does not exempt a system from the realities of the physical world!
> For someone making a point about responsibility, this is ridiculously irresponsible.
It's not just irresponsible, it's false. Russia killed 3 Ukrainian civilians with a drone where the targeting was completely autonomous by AI running on an Nvidia chip: https://www.nytimes.com/2026/08/24/world/europe/russia-drone.... The Pentagon tried to completely blacklist Anthropic because Anthropic refused to allow autonomous kills without a human in the loop. If you can't see how lots of military leaders want to put more lethal control into AI at this point I think you have to be willfully blind.
Isn't that what they're doing? "There's a better than 0 chance this thing could kill everyone in the next decade."
That might be too imprecise for the HN set but it's realistic for laypeople.
And none of the AI people talking about the risk are running into rooms full of people telling them Claude has gone mad and yelling at them to disconnect from the internet and turn off their devices immediately.
> That might be too imprecise for the HN set but it's realistic for laypeople.
Some 'laypeople' will be scared for a while and then return to more pressing issues and exactly nothing good will come out of that. If there really are some issues worth discussing the first step the AI doom crowd should do is to stop the PR offensive and concentrate on producing verifiable claims and actionable small steps. Otherwise their effort will fail this time and when/if a next time comes, they will have a lot less PR capital to burn.
Here's the reality: us plebs, regardless of how tech-savvy we are or aren't, have no say in what's going to happen.
There is no level of outrage that is going to stop the frontier AI labs, and the US government isn't going to step in to protect humanity. The Chinese are going to do what they're going to do. And so on.
So sit back, make some popcorn and enjoy the show.
>
Here's the reality: us plebs, regardless of how tech-savvy we are or aren't, have no say in what's going to happen.
This sentiment is incredibly out of place on HN. Facebook, Twitter, and many other very simple bits of tech transformed the world. Tiny startups founded by a few people have (also recently) ballooned into behemoths that hold vast amounts of economical and technical power.
Thanks to AI, it has never been quicker to go from idea to full fledged working product.
If anything will change the course of the world (for the better or worse), it will be a tech product, possibly created by someone on this forum.
> If anything will change the course of the world (for the better or worse), it will be a tech product, possibly created by someone on this forum.
You seem to be missing the point.
A small group of people created AI tech that they now say could be the end of humanity. They still want their companies to go public, but they also want the government to allow them to form a cartel so that they can, in their infinite wisdom, manage the risks so that they can try to prevent their tech from killing us all. And somehow they magically think that other nation-states developing similar tech (namely the Chinese) will go along with their plans.
As for how powerful Dario, Sam, et. al. really are: Trump says Dario is "pretending to be a perfect little angel" and claims there's a "sick conspiracy" against AI.
Having billions of dollars and being the head of a world-changing company doesn't buy the type of power you think it does. At best, it allows you to buy influence and pay your way out of liability for the harms your products cause.
The AI researcher in Mountain View making $2 million/year at Google has no more say in what's going to happen with AI than a plumber in Kalamazoo. The HNer working on a startup, in the final analysis, will in 50 years' time have left about as big a mark on the planet as a greeter at Walmart.
> Having billions of dollars and being the head of a world-changing company doesn't buy the type of power you think it does.
1. The product itself can change the world quickly and enormously, as I already pointed out.
2. The power of these huge companies and thus of their owners is immense. The effects of massive corruption in the USA are proof of that. Additionally, massive manipulation of algorithms and content in things like Tiktok, Twitter, Grok/ChatGPT, etc. can be and is done regularly, whether with 'good' intentions or not. Even just the basic control of what R&D money and time is spent on is huge.
> The HNer working on a startup, in the final analysis, will in 50 years' time have left about as big a mark on the planet as a greeter at Walmart.
With a defeatist attitude like "sit back and grab some popcorn", yes. You haven't shown in the least why an HNer couldn't change the course of history.
A key facet of the suggested risk is that these programs would attempt to mask whatever behavior would bring about the end of the world. A world in which AI doom could be detected before it’s too late and stopped by metaphorically running into rooms full of people telling people to unplug is less extreme than that in which doomers argue we are already living.
Alarm takes severity and scale. Forecasting double-digit odds of near-term human extinction on national television to a lay audience is more extreme than yelling to unplug rogue AI. For what it’s worth, the latter has already occurred numerous times at containable scales in leading labs, mitigated by the physical realities of computation, notwithstanding the competence of involved personnel. People running into rooms yelling is probably not a hypothetical.
Would you have said the same thing during the Cold War when nuclear weapons were proliferating? That’s the equivalent of what the developing offensive capabilities of models, basically cyber nukes. Or WMDs in general. OpenAI is accidentally hacking people, if someone made the decision to deliberately direct an agent swarm to attack national infrastructure you don’t think they could do much worse? Human extinction is a long shot but I wouldn’t say the same about a mass casualty event of some kind, and who knows what that might spark.
It's the AI doomers who have chosen to frame the argument in terms of extinction. Even if they were only claiming that 1% of the world population will be killed in the next 10 years, it would still be an extraordinary claim that hasn't been established with any amount of rigor. I'll reiterate the original point: these people are not serious, they're spreading alarmist nonsense, and they're undermining the public's trust in experts.
There is no proof possible. There is no experiment that can show something humanity scale will happen. There are even no remotely similar precedents in the past to extrapolate.
The actual established real evidence is that this tech is showing unprecedented capabilities (including destructive) and its safety guardrails are lacking.
I like Gaben's take on going public. This is a clip from a longer talk he gave which I think is really insightful into many aspects of the economics of software and company culture.
It's a great discussion, but not one that I think applies to AI startups. Valve's business model takes the traditional path where profit is generated by selling a product and growing from there. Modern Startups now take large sums of upfront funding, trading them for equity to hopefully reach profitability. Going public becomes a pressure from investors to recoup their cost, not a strategic move based on business needs. An unfortunate reality
There's an undeniable charm to the older era of pixel art, and surely part of that is nostalgia, but I also think a lot of it comes from the fact that the economics and the hardware were a lot different. Up until the mid 1990s, the highest paid artists in the industry were working on pixel art games. The hardware was slow enough that there were real creative constraints, and we know that constraints often drive creativity.
reply