That's why I suspect it's something new and might be related to the DDoS. The main spam and abuse problems are handles by the system I mentioned, which is not the reason here. So this is likely a defense against something else.
You're looking at simple, static pages. Unless something is horribly wrong with their infrastructure, these should take milliseconds to generate, if even that.
Stackoverflow pages have many dynamic components like vote counts, reputation points, sidebar related-questions-links, new comments, etc. An excerpt from their blog explains they can't cache the output :
https://nickcraver.com/blog/2019/08/06/stack-overflow-how-we...
Even though Stackoverflow's website io access pattern has higher reads than writes, the resultant generated html is still not as static as Wikipedia pages. Even if they're efficiently using cpu to generate the dynamic elements, the cost of egress traffic may also be a factor.
All that said, I don't have any insight into what heuristics they use to block certain ip addresses.
> variants of cache... anonymous, or not? mobile, or not? deflate, gzip, or no compression?
I don't understand why the markup would vary between mobile/desktop, or why they would even consider compression a varying factor to account for in cache. Maybe that's because their backends produce such specific variants that they have a hard time caching in the first place?
If 80% of pages are only requested every two weeks, then it doesn't make sense to cache those. There's still probably lots of stuff you can cache, such user profile/stats and question/answer scores for several minutes. You can also cache many parts of the markup that are not going to change often. I mean it's always even better when you cache the entire markup, but there can still be lots to gain if you cache only small bits that are expensive to acquire/template.
> But the cost of memory to store those strings (most large enough to go directly on the large object heap) is very non-trivial. And the cost of the garbage collector cleaning them up is also non-trivial.
It looks like their cache implementation could (should?) have been based on better foundations. First, i think the cache doesn't have to reside in memory: disk accesses are fast, certainly much faster than running a database query over the network which will need to access several files and cross-reference data with extra latency on top. Then, because if you're gonna store long-lived stuff in memory you should probably use a garbage collector based that's tailored for this usecase, not your language's (.Net) default GC... maybe Redis? Don't get me wrong, i find it pretty cool if some engineers want to develop a homebrew cache, but that sounds like a huge project in itself.
> the cost of egress traffic
I'm not aware of SO tech stack, but i'd be surprised if they have much egress fees. They're a very big site so they probably run off unmetered dedicated servers if not their own hardware on cheap transit. Who knows, they may even have their own AS and peer with other providers in some locations? From a quick request, it looks like stackoverflow.com is served from Fastly AS but i personally don't understand why: i don't remember seeing any heavy content (video/images) on SO so in that kind of situation a CDN would hurt more than help on slow links. Maybe that's because they bundle megabytes of javascript crap? Now that they're blocking tor users i can't check for myself :-)
That makes sense. From what i remember of using redis a few years back, it doesn't do file-based cache, only in-memory. Something like nginx is very good at file caching, though.
OK, that's true, they're not static, but we're still talking about a few database queries which quickly hit indexed columns. If you're running a public website like this, it's not like each page takes several seconds to load.
The egress traffic is also trivial - a page seems to clock in at below 50k. If they're paying $0.02/GB, that's $1 / million requests.
More importantly, if it really were a DoS attack, there's way less obtrusive methods, such as a CAPTCHA or similar verification screen.
They're not static pages, the lists of questions, users and everything else are dynamic. Some of them are obviously heavily cached, but you can't cache everything. It's certainly possible to find more expensive pages if you put in some effort.
It potentially could be related to all of the "knock-off" websites that scrape StackExchange data. Maybe they are going outbound on various Tor nodes and getting the IPs blacklisted as a result of reading thousands of pages too rapidly.
Given my experience with the network quality of Tor, I'd be surprised if scraping was A) efficient to do over Tor and B) that Stack Overflow would even notice it because as I said, the network speed is too slow, so can't add that much traffic compared to the absolutely staggering amount of traffic they get from non-Tor.
Interesting, but doesn't fit the context of Stack Exchange blocking Tor. Your example there is regarding a mobile app hotlinking a image of a flower, which seems easy enough to block/fix, while Stack Exchange blocking all Tor users from even reading Stack Overflow doesn't make so much sense.
