Basecamp is under network attack (DDoS)

Basecamp-DDOS.md

Basecamp was under network attack

The attack detailed below has stopped (for the time being) and almost all network access for almost all customers have been restored. We're keeping this post and the timeline intact for posterity. Unless the attack resumes, we'll post a complete postmortem within 48 hours (so before Wednesday, March 26 at 11:00am central time).

Criminals have laid siege to our networks using what's called a distributed denial-of-service attack (DDoS) starting at 8:46 central time, March 24 2014. The goal is to make Basecamp, and the rest of our services, unavailable by flooding the network with bogus requests, so nothing legitimate can come through. This attack was launched together with a blackmail attempt that sought to have us pay to avoid this assault.

Note that this attack targets the network link between our servers and the internet. All the data is safe and sound, but nobody is able to get to it as long as the attack is being successfully executed. This is like a bunch of people blocking the front door and not letting you into your house. The contents of your house are safe -- you just can’t get in until they get out of the way.

We're doing everything we can with the help of our network providers to mitigate this attack and halt the interruption of service. We're also contacting law enforcement to track down the criminals responsible. But in the mean time, it might be a rough ride, and for that we're deeply sorry.

DDoS criminals have attacked and tried to extort many services lately. Just a few weeks ago, Meetup was attacked, and it took a whole weekend of fire fighting before they were out of the woods. There is unfortunately no single, quick fix to these attacks, so we regretfully ask for your patience in advance. As said, we're doing everything we can, and will work as quickly as possible, but it's impossible to give a clear timeline for ultimate resolution.

The only thing we're certain of of is that, like Meetup, we will never negotiate by criminals, and we will not succumb to blackmail. That would only set us up as an easy target for future attacks.

We'll keep everyone updated through http://status.basecamp.com and Twitter (@37signals). Again, terribly sorry about this lousy way to start the week.

--

UPDATE: Attacker identified as being responsible for similar attacks (9:55am central time, March 2014)

We've learned that the very same criminals currently attacking and trying to extort us hit others just last week. We're comparing notes with everyone affected who have been in touch. The blackmail came from an address matching this pattern: dari***@gmail.com. If you have been extorted by this person, please get in contact so we can compare notes on both technical defenses and the law enforcement effort to hunt them down.

--

UPDATE: Law enforcement efforts pooled, attack currently waning (10:21am central time, March 2014)

We've pooled our law enforcement efforts with the other victims now, and are working with the same agent on the case. While tracking down these criminals is notoriously hard, we'll do our very best to bring them to justice.

At the moment it seems that the attack has also let up a bit. Our network providers have been doing a good job dealing with up to a 20Gbps attack. But from what we've heard from the other victims, the criminals are capable of even more than that, so we're not out of the woods yet.

--

UPDATE: Main attack has stopped, but still some network issues (10:41am central time, March 2014)

The main attack seems to have stopped now, but we're still dealing with a variety of network issues. Basecamp and the other services should currently be accessible to most customers, but not all. We're working fast and diligently to resolve all lingering issues. We may well still be attacked again, but for now it's about cleaning up the damage. We again thank everyone for their patience. This has been a horrible morning.

--

UPDATE: Service restored for 95% of customers, still working on last 5% (10:56am central time, March 2014)

With the main attack stopped, we've been able to restore service for about 95% of all customers. We're still working on restoring everything for everyone everywhere, though. When these attacks happen, the rest of the internet will sometimes put you in quarentine to prevent the fire from spreading. So even after an attack has stopped, it can take a while before you're allowed to leave quarentine. That's the phase we're currently in.

Reminder: The attack has stopped for now, but there's no guarentee it will not resume. Other victims have told us about how the attacker would take a break, and then try again later with a different method. Hopefully that will not be the case, but we remain on the highest alert for now.

Good luck guys! Hope you can cease their fire and catch them after that.

Best of luck guys! My company just went through that recently as well. Nowhere near as bad as 20Gbps, but enough to put extra security and proxies in place to lighten the attack. Ours lasted for about 3 days, although the first 24 hours was the worse and affected our site along with our client sites so it was something that had to be resolved asap.

@inspectorfegter Believe what you want. Not once in my responses did I say that I personally wrote it. It was my fault for not citing, regardless of the fact that I forgot and I apologize to anyone who was greatly affected by the notion that I had originally wrote it. But trying to help explain my initial comment which was so badly taken out of context was not meant to make me some beacon of knowledge that everyone should look to, but to help raise awareness that this attack should concern others of the safety of their proprietary information.

Your comments are negative, malicious and no better than the actions of these criminals attacking 37Signals and their Basecamp product website. How about adding something constructive and positive to the conversation, rather than trying to prove some irrelevant point to make yourself look cool?

^ just stop? ^

@SaintIsaiah I have not attacked you personally nor will I. In two posts, you have called me a child and a criminal. I was pointing out the fact that what you posted was absolutely not original to you and that you are just adding a layer of paranoia to this thread that shouldn't exist in the first place. Why drum up additional concerns when there is no evidence of anything other than a denial-of-service attack? In fact, your arguments can only serve to harm 37 signals by calling into question their integrity of data storage at a time like this. It also was a bit foolish to expose your organization's practice of storing plaintext user/passwords in Basecamp publicly. This serves only to make your account a rich target for script kiddies around the globe. - Signed The Cool, but Childish Criminal

give us the person's email address, we can assign hundreds of todo's to them! see how they like it!

JOKE

It is great to see a open post like this. Good luck !

Can you tell what mitigation services/ methods you are using it will be helpful for the community sorry if i missed some online link briefing about this.

woow this is what I call a GOOD way to communicate a serious problem, Good job 37Signals team!

@inspectorfegter You accused me of theft, which is a personal attack. Though your claim was valid, which I promptly corrected when it was pointed out, it could have been pointed out in a more mature way, hence why I told you to grow up. I did not say you were a criminal, but that your comments were no better than the actions against Basecamp today, in that they were purely negative and destructive.

My team and I have already changed all of our passwords and stored them in a different service specifically for passwords, so I'm not worried about it.

And regardless of having additional evidence or not, it's not unreasonable to ask 37Signals (which I assumed they are monitoring since they have consistently updated the post) how secure my company's information really is. What should I do? Send a contact request through their website and sit idly by while I wait into oblivion for a response? If it's a concern for me, it's undoubtedly a concern for many others, and I'd rather try to get an answer now rather than sweep it under the rug and wait for a more serious security compromise in the future.

I'll just leave it at that and walk away, as this conversation is not meant for us to bicker back and forth, but for customers to be heard and updated throughout this issue.

@inspectorfegter +1. I feel so sad that they found themselves in a tough situation like this. Please support them and help them if you would. Please mind that this is not the time to criticise. Good luck 37s and DHH.

In an age of the 400gbps+ ntp attacks they took you down with 20gbits and you were not prepared ? are you kidding me right now ? Are you living in 2004 ?

@xnljfr 20Gbits of malicious traffic rushing in per second can potentially have catastrophic effects on almost any network, however, it's all dependant on how that traffic is used.
Reading through this gist it seems the information regarding the attack is very vague, so there's no real telling what type of attack was used. Additionally, they may not receive attacks very often, if not at all, so spending thousands upon thousands of dollars protecting against such things may not of been on their mind or something they felt the need to do.

Also attacks ranging from 20Gbps and upwards were not reported until 2007 (according to the below source), so if they were living in 2004, they'd have something quite nice to go in the history books :).

Here is an (outdated) graph of some of the largest reported DDoS attacks:

The full page/pdf can be found here.

In regards to NTP attacks; they have been proclaimed to be dead from the lack of vulnerable servers available to execute the attack, so it's doubtful we'll be hearing much more in regards of future attacks. This doesn't mean we won't be seeing attacks as large if not even larger in the future however; there are still many attack vectors to be used that have quite the potential if used correctly.

On an almost final note, I'd wish you the best of luck with the attack issues and hope they come to a halt!

All the best in the fight and thanks for the transparency and commitment.