Maybe I'm slightly projecting my European sensitivities here, but I think there is more going on there than just the emptiness. There is a common theme that the spaces depicted are, on their own, quite unpleasant (barren, made from flimsy materials, probably emitting some amount of toxic fumes); it's just that this unpleasantness is normally masked by their human use and interpretation, which invariably some form of commercialism and hype. (Hence the abundance of hastily erected American mall architecture.)
A historical European town devoid of people does not work as a liminal space picture at all, because it still looks nice; and neither do the postapocalyptic settings that Japan is so fond of (YKK etc.). Eastern European commieblock and UK Brutalist hellscapes are actually quite similar in terms of the feeling they evoke, and have their own fandoms, but are considered their own genre - so I would conclude that "liminal space porn" is spaces only made tolerable by commercialism with the commercialism taken away, and the related "/r/UrbanHell" material is spaces only made tolerable by human habitation with that taken away or suppressed (e.g. if the humans are so bereft of vitality that they can no longer overcome the space's badness).
This is more or less my thoughts as well. Walking around the exterior of Nymphenburg Palace even when empty would not be very emotional at all.
Walking around a modern suburban development devoid of people, houses, pets, etc. would be at least unusual in the feeling. The spaces are intentionally designed to put people and their things within obvious boundaries. With the boundaries still there but lacking the things within them it becomes quite a different experience.
There are layers to that suburban setting and I wonder which ones you imagine.
There actually used to be this "empty suburb" feeling at many times per day when a typical bedroom community had sent its kids to school and parent to work. Particularly when they were not wealthy enough to have paid laborers around doing things during their work day. If anything, they got busier since COVID as people have more varied schedules.
Then there is the new but incomplete development, e.g. with graded lots and some subset of streets and walkways. If work is suspended for some reason, it may be decorated with idled earth movers, piles of building materials, or partial foundations or framing.
Or it might turn into the next type, which is an aborted subdivision build or after a severe wildfire, which is basically a moonscape of graded lots with no buildings nor vegetation.
Then there are the abandoned neighborhoods that were once vibrant. Old, decrepit buildings, and wild vegetation, e.g. around dead industrial towns.
I think that's narrowing it too much. Edward Hopper's "Sunday Morning" evokes the same feeling with me - the feeling of something that just was there, that we passed through without paying much attention to it at the time, but which we now remember.
Maybe you wouldn't have that feeling about a beautiful old European town, but I bet some of the people living there would.
I think classic liminal esthetic would also include empty swimming pools, resorts etc. It doesn't need to be barren or cheap, could be relatively luxurious, but needs to be late Modernist as a base - i.e. mass produced and devoid of conscious symbolism or ornamentation.
Its the combination of capitalism and lowest bidder architecture that is alienating to humans. And if humans are present, makes the space tolerable.
Remove the humans, and the space is off-putting, eerie and has a sense of dread.
And yeah, brutalism (usage of concrete as primary architecture) is also seen in the USA. And frankly, most of the buildings feel like prisons, and not where you want to go. Look no further than Indiana Univerity Hermann Wells library. https://maps.app.goo.gl/6FDvKR9sHSk3z8v56
Brutalist buildings feel directly hostile to humans, and not a vague sense of dread.
However I had the pleasure of seeing a brutalist hotel in Iceland (Fosshotel Vatnajokull) which combines concrete with wood. It felt sturdy and powerful, but also soft and welcoming. https://maps.app.goo.gl/wnXEawh2wptmQ3Rd7
But yeah, I do think youre on the right track. It really is related to capitalism and making horrible spaces (malls, etc). And without people, they are alienating and mild dread-inducing.
It's unfashionable but I still love brutalist architecture. It feels stable, like it would still be there whether it had people in it or not. Like living in a cave.
Maybe it feels that way, but that's an illusion, then. The things are often haunted by poor decisions taken when they were built and take a ton of maintenance. Concrete isn't as maintenance-free as it seems!
After reading your first paragraph, I was already drafting a slightly standoffish response in my head, to the effect that the children who grew up on our sanitised fairy tales seem to be blowing each other's heads off on the battlefields of 2026 with undiminished enthusiasm and sadism, and the only difference is that now more of them need a Xanax prescription afterwards. I appreciate that you actually addressed this view with less snark than I was able to.
The problem here is that there is more than two metaphorical people involved: there is the developer, the would-be user, and the evangelist who harangues the developer with "rewrite it in Rust brah" drive-by comments or blog posts about how nobody sane would use memory-unsafe languages/ecosystems without a vibrant community package management ecosystem in the year of our lord 2026.
The last person, I think, most clearly, does "owe" you supply-chain security, in the sense that he bears moral (and ought to be made to bear professional) responsibility for any adverse consequences you may suffer from its lack, though in practice he will probably often protest that he couldn't do anything about it because it's not like he is developer. Whether the developer also owes it is a more interesting question, and I think it greatly depends on what attitude he takes towards the evangelist (does he consider him a nuisance who makes implicit promises the developer is uninterested in delivering, or an ally who raises the dev's profile?).
Long ago, I remember seeing a cartoon which involved a tag-team of two people robbing a third, with A pointing a gun at C and saying "give your money to B", while B comments "I'm really just standing here, but I figure it's best if you do as he says". I'm not sure what exact piece of day-to-day politics this was made to comment on (though it was probably some or another flavour of political violence), but it seems somewhat applicable here as well. The lines just become "accept the supply chain, or suffer my public ridicule" and "I'm just providing the software 'as-is', but you probably should do as he says".
Taking a step back, I think "search for outliers" doesn't quite get to the heart of the issue. Why are we searching for the outliers, and why are we so particular about the base distributions that we are searching for outliers of - why are there women's sports at all (if the outliers they find are not outliers on the same metric in the whole population), and why is boxing, for example, divided into weight classes?
It seems to me that a big part of the point of competitive spectator sports is to send, to the spectator, a message along the lines of "this could have been you". It is hard to argue that the ability to throw a 1kg+ discus exceptionally far is otherwise so useful that would justify all the expense of finding and showcasing the outlier. Therefore, the point of the competition stands and falls with whether the spectator buys this message.
When do spectators tend to believe in it? When should they? Arguably, there is a plethora of reasons why the median American spectator looking at a clip of Usain Bolt running could not in any meaningful sense have been him. Yet, somehow, the "could-have-been-me sense" that people are endowed with transcends these reasons and results in men commonly looking at him and getting some of that could-have-been-me sense that gives the sport meaning, and women looking at him and getting much less of it. To solve this, we maintain a separate women's category. The winner there is not as much of an outlier relative to the distribution of the whole population. Most likely, she is still every bit as dissimilar to the spectators as Usain Bolt is. Yet, the women watching, and the ones merely learning about this event happening through osmosis, get their heart warmed by the dubious sense that this could have been them, and perhaps encouraged to try harder and hold more hope for some other pursuit of their own, in a way that they never would have due to Usain Bolt. Would they or would they not get the feeling for a transwoman sprinter? How would we even measure this?
> and why is boxing, for example, divided into weight classes?
Entertainment value. Put a flyweight against a heavyweight and the audience are not going to care. No audience means no money for the show runners, and the Olympics is, when you get down to the brass tacks, all about money.
Assuming the benchmarks are sound (rather than capturing a fluke), the provided explanation still does not pass the smell test. As far as I can tell, there is nothing about the training process of these models that would encourage them to make the output of any layer apart from (n-1) meaningful as the input of layer n, unless perhaps these layers were initialised as identity and the training process did not get to change them much. (Plausible for middle layers?)
Considering this, I think (again, assuming the benchmarks themselves are sound) the most plausible explanation for the observations is (1) the layers being duplicated are close to the identity function on most inputs; (2) something happened to the model in training (RLHF?) that forcefully degraded its reasoning performance; (3) the mechanism causing the degradation involves the duplicated layers, so their duplication has the effect of breaking the reasoning-degrading mechanism (e.g. by clobbering a "refusal" "circuit" that emerged in post-training).
More concisely, I'm positing that this is an approach that can only ever break things, and rather than boosting reasoning, it is selectively breaking things deleterious to reasoning.
Empirical findings tell a very different tale: all LLM layers use vaguely compatible internal representations. And middle layers in particular can be almost interchangeable - a lot of what they seems to be "iterative refinement of the same representations". Proven by various probes and ablations, but the most obvious one is probably the good old logit lens.
This is likely to be shaped by tied embeddings and skips on one end, and maybe training pressures on the other.
The very top of FF stack and the very bottom of FF stack both reflect the same token embeddings - and this propagates through the model, setting up a shared identity space. Skip connections propagate that through the layers. No explicit shared identity imposed, but there is an implicit one set by the architecture. Fairly well established.
(Now: highly speculative! Attention over past tokens creates an implicit "robustness/convergence" pressure? The model can't be "certain" if it'll have access to the right representations at a given layer, because representations depend not just on the past layers, but also on the highly uncertain contents of previous tokens as passed through attention. Which in turn depends on more of the same, increasing variance further. So the training causes: "each layer can't be certain of what it will have access to, so it develops to refine anything it currently has access to in a convergent fashion, because that's what's useful under pressure of attention-induced uncertainty".)
LLMs are notoriously nonfragile, and robust to perturbations. Far more so if you anneal with SFT/distillation after your model surgery, although this wasn't done here. Plenty of weird franken-LLM experiments prove that empirically.
So I'm not too surprised to find that someone has managed to improve benchmark performance on a few narrow tasks by duplicating a few middle layers. "Duplicating a few layers that were doing convergent iterative refinement benefits a few tasks that suffered from insufficient depth of convergent iterative refinement" is a fairly reasonable hypothesis, in my eyes.
The chances of duplication "breaking something somewhere" are high, and I would expect the capability profile of an unannealed franken-LLM like this to have a few gaps in it if evaluated extensively against the original. But "franken-LLM layer duplication can actually improve some things" is far too plausible with what we know to be dismissed pre-emptively.
That's interesting, could you point me to some source on these findings?
It seems to me that the difference between "iterative improvement" as you put it and "close to the identity" (as in the output is close to the input for most of the volume of the input space) as I put it is fairly subtle, anyway. One experiment I would like to see is what happens to the reasoning performance if rather than duplicating the selected layers, they are deleted/skipped entirely. If the layers improve reasoning by iterative improvement, this should make the performance worse; but if they contain a mechanism that degrades reasoning and is not robust against unannealed self-composition, it should make the performance similarly better.
These, other papers, and the lottery ticket phenomenon; what it boils down to is that any neural network like system which encodes some common mapping of a phenomenon in the context of the world - not necessarily a world model, but some "real-world thing" - will tend to map to a limited number of permutations of some archetypal representation, which will resemble other mappings of the same thing.
The lottery ticket phenomenon is a bit like the birthday paradox; there will be some number of structures in a large, random initialization of neural network weights that coincide with one or more archetypal mappings of complex objects. Some sub-networks are also useful mappings to features of one or more complex objects, which makes learning hierarchical nested networks of feature mappings easier; it's also why interpretability is so damned difficult.
> As far as I can tell, there is nothing about the training process of these models that would encourage them to make the output of any layer apart from (n-1) meaningful as the input of layer n
Right, I had the same thought.
Even if the output was in the same "format", does the LLM even have any way to know which order the outputs will go in? The ordering of the nodes is part of our representation of the network, it's not fundamental to it.
It would be like shuffling the bytes in a PNG file and expecting the program still to understand it as a PNG file.
The more I think about this, the more I don't get this at all.
> there is nothing about the training process of these models that would encourage them to make the output of any layer apart from (n-1) meaningful as the input of layer n
There is something that does exactly that - the residual connections. Each layer adds a delta to it, but that means they share a common space. There are papers showing the correlation across layers, of course it is not uniform across depth, but consecutive layers tend to be correlated.
> far as I can tell, there is nothing about the training process of these models that would encourage them to make the output of any layer apart from (n-1) meaningful as the input of layer n
Wouldn't "pass-through" identity connections have exactly that effect? These are quite common in transformer models.
> something happened to the model in training (RLHF?) that forcefully degraded its reasoning performance
I've been seeing more people speculating like this and I don't understand why. What evidence do we have for RLHF degrading performance on a key metric like reasoning? Why would this be tolerated by model developers?
Can someone point to an example of an AI researcher saying "oops, RLHF forcefully degrades reasoning capabilities, oh well, nothing we can do"?
It strikes me as conspiracist reasoning, like "there's a car that runs on water but they won't sell it because it would destroy oil profits".
The most obvious way would simply be excessive agreeableness. Users rate responses more highly if they affirm the user's thinking, but a general tendency to affirm would presumably result in the model being more inclined to affirm its own mistakes in a reasoning chain.
There was some research about it early on that was shared widely and shaped the folklore perception around it, such as the graph in https://static.wixstatic.com/media/be436c_84a7dceb0d834a37b3... from the GPT-4 whitepaper which shows that RLHF destroyed its calibration (ability to accurately estimate the likelihood that its guesses are correct). Of course the field may have moved on in the 2+ years that have passed since then.
If you are so allergic to using terms previously reserved for animal behaviour, you can instead unpack the definition and say that they produce outputs which make human and algorithmic observers conclude that they did not instantiate some undesirable pattern in other parts of their output, while actually instantiating those undesirable patterns. Does this seem any less problematic than deception to you?
> Does this seem any less problematic than deception to you?
Yes. This sounds a lot more like a bug of sorts.
So many times when using language models I have seem answers contradicting answers previously given. The implication is simple - They have no memory.
They operate upon the tokens available at any given time, including previous output, and as information gets drowned those contradictions pop up. No sane person should presume intent to deceive, because that's not how those systems operate.
By calling it "deception" you are actually ascribing intentionality to something incapable of such. This is marketing talk.
"These systems are so intelligent they can try to deceive you" sounds a lot fancier than "Yeah, those systems have some odd bugs"
Environmentalism has always been a "weight of our sins" sort of issue. Plastic straws are a rounding error relative to all the capricious uses of plastic and fossil fuels in our economy, but few things feel as frivolous as using once and then throwing away a piece of plastic for personal convenience while engaging in an already-kinda-sinful feeling activity like indulging in a soft drink, while simultaneously the paper straw that turns to cardboard mash in your mouth is perfectly calibrated to make you feel like you are doing real penance without encumbering anything economically important.
So plastic straw bans (instead of plastic slipper bans, plastic food packaging bans, taxes on plastic clothes fibres...) are what we get. And because the structure of the cause/problem is the same, the language of environmentalism naturally attaches itself and gives form to the vague sense of moral unease surrounding AI. Governments are surely already building tomorrow's tightly integrated thought police drone swarm complexes, but a crusade against those who simulate a zoo of programming weasels in our midst is much easier and morally no less fulfilling.
Unfortunately, all it will take is an appropriate choice of story about "Nazis"/"child predators"/"pirates"/"terrorists"/"Russian bots" sideloading unregulated apps or disabling the GPS trackers on their cars, and every prospective member of Doctorow's great new coalition (including most everyone in attendance when the talk was given) can be peeled away with ease.
Do you not think that trying to malign your opposition by putting a comical misspelling in their mouths is a bit infantile as a rhetorical tactic? The same thing being done to you would look something like an insinuation that what is being banned is "hurting someone's widdle fee-fees"; surely the discussion here would not benefit if everyone stooped down to that level.
> surely the discussion here would not benefit if everyone stooped down to that level.
Oh we were already at that level by that time: the comment mine responds to makes the claim that "it is really difficult to define what hate speech is" (untrue); that "more often than not it's used as a cudgel to silence the opposition" (unsubstantiated); and claims that the UK government's intentions match that of Iran and Russia (untrue).
For some reason, so many people seem to tolerate outright disinformation but draw the line at mild childishness. It's bewildering.
Do you think that the people who made those remarks you cite considered them untrue themselves? If yes, you are suggesting bad faith (which should be grounds to extricate yourself from the discussion and/or call it out, not add fuel to the fire); if not, you are suggesting that factual disagreement is appropriately answered by childishness, which basically is saying that you think every discussion worth the name should devolve into childishness.
Often, it seems like this concept of "disinformation" you invoke just serves as a way people give themselves moral license to suspend normal rules of debate conduct in the face of disagreement. Being charitable to your opponents and having to engage with their claims is tiring and difficult, and sometimes they even come better prepared - how much easier if you can just frame dissent as dangerous enemy action and shut it down.
Do you also insist that we treat with proper decorum those who throw out assertions that jetfuel cannot melt steel beams? I notice you have yet to criticise them for posting what is at best misguided and unsubstantiated misinformation, and at worst disinformation. Hardly decorum on their part, is it? Instead, you are hyperfocusing on my "freeze peach", disregarding everything else I said in my comment. I find this to be a boring distraction from the topic at hand.
Well, I don't see anything obvious to criticise about what your interlocutors posted; their statements seem plausible enough to me, and if there is actually a knockout argument against them, I don't know it, because the person who seemed to disagree (you) was busy making childish noises instead of making it!
> jet fuel/steel beams
This debate was carried out sufficiently publicly that I got the sense people actually ran experiments confirming the pro-beam softening/structural failure/whatever case; certainly the "truther" case should have been taken seriously before that, and with decorum always because there is no situation in which any debate in a moderatable forum benefits from playground behaviour.
A historical European town devoid of people does not work as a liminal space picture at all, because it still looks nice; and neither do the postapocalyptic settings that Japan is so fond of (YKK etc.). Eastern European commieblock and UK Brutalist hellscapes are actually quite similar in terms of the feeling they evoke, and have their own fandoms, but are considered their own genre - so I would conclude that "liminal space porn" is spaces only made tolerable by commercialism with the commercialism taken away, and the related "/r/UrbanHell" material is spaces only made tolerable by human habitation with that taken away or suppressed (e.g. if the humans are so bereft of vitality that they can no longer overcome the space's badness).
reply