Update on the ongoing DDoS attacks and blocking Tor exit nodes

Question

For the past month, Stack Overflow has been hit by weekly DDoS attacks that progressively grew in size and scope. In each incident, the attacker(s) have been changing their methodology and responding to our countermeasures. Initially, we were able to detect and mitigate the attacks before any performance degradation could be noticed, but the latest attacks ramped up very quickly and the site was brought down before we could react.

While we cannot go into specifics on each attack in order to maintain OPSEC and not tip off the attackers, we can say that each individual attack has been using different IP addresses and targeting different aspects of the site. During an outage, our top priority is always getting the site back up and running. After traffic has been stabilized, we perform a post-mortem for the incident where we assess and improve upon the actions we have taken.

During the outage last Sunday, we noticed that a large amount of the DDoS traffic originated from Tor exit nodes. The decision to block Tor exit nodes did not come lightly; in fact, Teresa, our CTO was on the call when we discussed remediation methods. Due to the persistent nature of the attack and our desire to bring the site back up as fast as possible, we made the decision to block all DDoS traffic endpoints, including these Tor exit nodes.

We did not target, nor set out to block all traffic from Tor; that’s not something Stack has ever done. However, due to the shared nature of Tor exit nodes, some of them were also routing DDoS attacks to our sites and were blocked. We have tried removing these blocks between attacks, but this action has resulted in further site outages as DDoS efforts continue to originate from these exit nodes. Unfortunately, blocking the Tor exit nodes also blocks legitimate users from using them. An immediate solution for users who find themselves blocked is to access our site from other IP addresses, via home internet, work internet, or other VPN services.

We are continuing to evaluate the situation and will keep our community updated. Thank you for your patience and understanding.

Update 2022年02月10日:

You may see Tor access improve over the next few days as we do some testing. The changes may be temporary depending on how everything goes. We will continue to keep everyone updated as the situation evolves.

Update 2022年02月11日:

There was another DDoS attack that briefly affected the site last night; we were able to test some of the changes we have made, and we are happy to report that none of the DDoS traffic originated from Tor.

Update 2022年05月19日:

I wrote a blog post sharing some lessons learned: https://stackoverflow.blog/2022/05/16/stack-under-attack-what-we-learned-about-handling-ddos-attacks/

• Makoto: Not that I'm explicitly disagreeing or even impacted by this decision, but your mitigation for legitimate users on Tor using Stack Overflow is...diametrically opposed to why users would leverage Tor in the first place...
• Haney: @Makoto that's definitely true, and we feel that way also. However the options right now are: block Tor exit nodes, or unblock them and we get DDoS'd / go down every few days. So the answer (at least for now) is obvious unfortunately. A bad actor or actors on the Tor network. :(
• Michael Kohne: @Makoto - unfortunately, if Tor is going to be abused like this, then it's going to suffer the same issue (and potentially the same fate) as open mail relays and many other projects over the years: the legitimate users are going to get shafted as the rest of the world tries to protect themselves from a bunch of bad actors. You see this in most 'open to everyone with no restrictions' projects - the bad actors abuse the heck out of a thing to the point that it gets modified till they can't use it (making it less useful) or it goes away entirely.
• Andras Deak -- Слава Україні: As they say: that's why we can't have nice things.
• Josh Zhang: @NickODell correct.
• davidbak: I had no idea Tor exit nodes aggregated enough bandwidth to make it possible for a serious DDOS attack through them.
• Josh Zhang: We do not want to permanently block Tor exit nodes, there are 1,246 published endpoints: check.torproject.org/torbulkexitlist. We're actively working on a solution but can't make any commitments.
• Makoto: @Haney: No no, it's like I said - I don't disagree with it. Blocking Tor exit nodes is a sensible mitigation to prevent DDoS attacks. It's just that you're saying to users who use Tor as a part of their OPSEC discipline to do the things that they would never consider doing. No objections at all to the mitigation of DDoS, but let's be realistic - if someone legitimate only accessed Stack Overflow through Tor, they're not going to magically shift around their OPSEC just to access this site.
• bta: Do you have any data about how much legitimate traffic you've historically received from Tor users? I'd be curious to know whether this would impact 1% of your users or 0.0001%.
• Andras Deak -- Слава Україні: @JanPokorný they can't counteract nation-state hacking attempts if Stack Overflow is down i.sstatic.net/lcNXA.gif
• wizzwizz4: @cocomac Using Cloudflare would have almost the same effect on Tor users as just blocking Tor entirely; their anti-DDOS wall is a usability nightmare (despite their apparent attempts to mitigate it) and it's made more than one website completely inaccessible to me. That's when I'm not using Tor; I hear it's worse for Tor users (and completely unusable for Tor users who aren't using Tor Browser).
• Example person: What happened to Fastly? They are in front of the Stack Overflow servers right? Won't they stop it?
• Dan P: Just a thought, but would it be possible to block Tor traffic to the .com pages as you are now, but then operate your own .onion Tor link, much like Facebook does, where Tor users and only Tor users can access Stack pages. Would take some reengineering to get it all to work I imagine, but would be a long term solution to filter Tor traffic so the only damage Tor nodes could do were to itself. Not sure how plausible this actually is in practice though.
• Journeyman Geek: I know at least one real user who's using SE and contributes - he's someone very knowledgeable in his field, and uses tor to remain anonymous. He annoys me, but I think it annoys me more that he got affected as collateral damage
• O. Jones: Hey, Stack ops people: Thanks for all your hard work on this freakin' mess.

Answer 1

During the last outage when Fastly's servers were down, Stack Overflow and many other websites were down. During that time, in Twitter, I saw a tweet that said that they are going to temporarily route traffic directly to the backend Stack Overflow servers. I guess it was a situation where they didn't have any other choice.

Fastly themselves say that they mitigate DDoS attacks in here. (Maybe it requires a separate plan?). They do not seem to have mitigated the attack to Stack Overflow.

All these things seem to say that the attack was directly launched onto Stack Overflow's servers' IP addresses (without Fastly being in between).

Note: AFAIK, it seems that Stack Overflow is still protected using Fastly.

• loops: If you override your DNS resolution to go directly to the Stack Overflow origin IP, it works and you can use the site normally, albeit with a TLS certificate for teststackoverflow.com. While obscuring the origin IP would be difficult (there are only 512 IPs delegated to SE's network, AS25791), dropping connections from non-Fastly sources would probably go a long way
• Example person: @Smitop oh no, it seems Stack Exchange's servers' IP addresses are very exposed. And even some of them can be used to visit the site by changing the "Host:" header... who knows what will happen if the attacker reads your comment
• You Old Fool: if the attacker reading that comment endangers to the site, something is terribly amiss
• Example person: @EatenbyaGrue obviously. Because Stack Exchange is exposing their IP address of their backend servers. They do not even try to block connections from non-fastly sources. They should fix this
• Largato: Just checked on bgpview.io - I really can't think anything useful except allowing fastly (and some other known) IPs only, but I believe they have a proper team dealing with it. Thanks for all the clarification, best regards (removed my obsolete questions here to keep comments tidy).
• Jason C: Perhaps Fastly did mitigate some of it, but the attack was aggressive enough that it still affected SE, and it would have been much worth without them.
• Example person: @JasonC if that was the case, who knows what will happen if the attacker attacks directly....
• Example person: Maybe some attacks on: 1. '5:33 AM 11 Feb 2022', 2. '~10:30 PM 14 Feb 2022', 3. '2:19 PM 14 Feb 2022'
• Clonkex: It seems to me that Fastly probably are mitigating attacks (and probably the vast majority of them), but we just don't see it. It's the same with multiplayer games. Companies spend vast sums to try to keep up with cheaters but some always slip through. Players complain that "there's cheaters in my games therefore the companies aren't doing anything to prevent cheaters" but what they don't seem to grasp is that if the companies weren't doing anything, the number of cheaters would be many orders of magnitude higher.

Answer 2

For the past 3 weeks we have been working to further strengthen our DDoS mitigation while also accounting for Tor traffic. At this point, we feel that we have been able to strike a good balance between protecting the site from DDoS attacks without blocking Tor traffic. Although we cannot rule out the possibility of blocking Tor exit nodes in the future if they are used as part of a DDoS attack, we will work to unblock them as soon as possible after we are able to mitigate the attack.

Unfortunately, we’re still unable to go into specifics about the nature of the DDoS attacks and what we have implemented to mitigate them. Our monitoring shows that we are still being attacked, but fortunately, we have been able to mitigate them before any major disruption occurs.

Again, we’d like to thank the community for your patience and understanding.

• Sonic the Anonymous Hedgehog: So Tor users aren't blocked at this time, to be clear?
• Josh Zhang: @SonictheAnti-NewVariant-hog correct, we are not blocking any Tor exit nodes that are published here check.torproject.org/torbulkexitlist
• forest distrusts StackExchange: @JoshZhang I use Tor exclusively for access and although I'm not always blocked, I do frequently get rate limiting pages blocking my way.
• Brian Drake: @forest I have been getting a lot of those rate limit messages recently, including some today.