The attack detailed below has stopped (for the time being) and almost all network access for almost all customers have been restored. We're keeping this post and the timeline intact for posterity. Unless the attack resumes, we'll post a complete postmortem within 48 hours (so before Wednesday, March 26 at 11:00am central time).
Criminals have laid siege to our networks using what's called a distributed denial-of-service attack (DDoS) starting at 8:46 central time, March 24 2014. The goal is to make Basecamp, and the rest of our services, unavailable by flooding the network with bogus requests, so nothing legitimate can come through. This attack was launched together with a blackmail attempt that sought to have us pay to avoid this assault.
Note that this attack targets the network link between our servers and the internet. All the data is safe and sound, but nobody is able to get to it as long as the attack is being successfully executed. This is like a bunch of people blocking the front door and not letting you into your house. The contents of your house are safe -- you just can’t get in until they get out of the way.
We're doing everything we can with the help of our network providers to mitigate this attack and halt the interruption of service. We're also contacting law enforcement to track down the criminals responsible. But in the mean time, it might be a rough ride, and for that we're deeply sorry.
DDoS criminals have attacked and tried to extort many services lately. Just a few weeks ago, Meetup was attacked, and it took a whole weekend of fire fighting before they were out of the woods. There is unfortunately no single, quick fix to these attacks, so we regretfully ask for your patience in advance. As said, we're doing everything we can, and will work as quickly as possible, but it's impossible to give a clear timeline for ultimate resolution.
The only thing we're certain of of is that, like Meetup, we will never negotiate by criminals, and we will not succumb to blackmail. That would only set us up as an easy target for future attacks.
We'll keep everyone updated through http://status.basecamp.com and Twitter (@37signals). Again, terribly sorry about this lousy way to start the week.
--
UPDATE: Attacker identified as being responsible for similar attacks (9:55am central time, March 2014)
We've learned that the very same criminals currently attacking and trying to extort us hit others just last week. We're comparing notes with everyone affected who have been in touch. The blackmail came from an address matching this pattern: dari***@gmail.com. If you have been extorted by this person, please get in contact so we can compare notes on both technical defenses and the law enforcement effort to hunt them down.
--
UPDATE: Law enforcement efforts pooled, attack currently waning (10:21am central time, March 2014)
We've pooled our law enforcement efforts with the other victims now, and are working with the same agent on the case. While tracking down these criminals is notoriously hard, we'll do our very best to bring them to justice.
At the moment it seems that the attack has also let up a bit. Our network providers have been doing a good job dealing with up to a 20Gbps attack. But from what we've heard from the other victims, the criminals are capable of even more than that, so we're not out of the woods yet.
--
UPDATE: Main attack has stopped, but still some network issues (10:41am central time, March 2014)
The main attack seems to have stopped now, but we're still dealing with a variety of network issues. Basecamp and the other services should currently be accessible to most customers, but not all. We're working fast and diligently to resolve all lingering issues. We may well still be attacked again, but for now it's about cleaning up the damage. We again thank everyone for their patience. This has been a horrible morning.
--
UPDATE: Service restored for 95% of customers, still working on last 5% (10:56am central time, March 2014)
With the main attack stopped, we've been able to restore service for about 95% of all customers. We're still working on restoring everything for everyone everywhere, though. When these attacks happen, the rest of the internet will sometimes put you in quarentine to prevent the fire from spreading. So even after an attack has stopped, it can take a while before you're allowed to leave quarentine. That's the phase we're currently in.
Reminder: The attack has stopped for now, but there's no guarentee it will not resume. Other victims have told us about how the attacker would take a break, and then try again later with a different method. Hopefully that will not be the case, but we remain on the highest alert for now.
@xnljfr 20Gbits of malicious traffic rushing in per second can potentially have catastrophic effects on almost any network, however, it's all dependant on how that traffic is used.
Reading through this gist it seems the information regarding the attack is very vague, so there's no real telling what type of attack was used. Additionally, they may not receive attacks very often, if not at all, so spending thousands upon thousands of dollars protecting against such things may not of been on their mind or something they felt the need to do.
Also attacks ranging from 20Gbps and upwards were not reported until 2007 (according to the below source), so if they were living in 2004, they'd have something quite nice to go in the history books :).
Here is an (outdated) graph of some of the largest reported DDoS attacks:
The full page/pdf can be found here.
In regards to NTP attacks; they have been proclaimed to be dead from the lack of vulnerable servers available to execute the attack, so it's doubtful we'll be hearing much more in regards of future attacks. This doesn't mean we won't be seeing attacks as large if not even larger in the future however; there are still many attack vectors to be used that have quite the potential if used correctly.
On an almost final note, I'd wish you the best of luck with the attack issues and hope they come to a halt!