DDOS attack on deepthought, and Speakeasy's response

[ network ] Message 14924 Sun Oct 22, 2006 2:44pm
From: John DuBois
Subject: deepthought & Speakeasy

On Saturday night at 6:05PM, someone initiated a DDOS attack against deepthought for unknown reasons, involving at least 20,000 hosts. The response of our ISP (Speakeasy Broadband), once they noticed, was to create a /32 null route for deepthought, making deepthought immediately inaccessible to the rest of the world. This was at about 7:25PM. I called them up that evening, and the tech I talked to had no idea what was going on. He could see the null route, but didn't have any idea what would cause such a route to exist - meaning that their network technicians didn't even bother to put a note in my account when they created the null route. They also didn't bother to call me, despite having reliable contact information for me since I have a “business class” account.

The support person did eventually get hold of someone who told him it was an “abuse” issue, leaving him and me both thinking it was something that Speakeasy's technicians had thought originated on deepthought. He told me that the abuse desk isn't open on weekends, but if it was urgent he could have someone look at it the next day. I told him it was, and then spent the evening wasting my time trying to find any signature of an attack being carried out from deepthought, and trying to limit the damage by changing deepthought to a new address, though what I can do is constrained by the fact that the root servers know deepthought's (nullrouted) IP address.

Today I called them up and found that the abuse person had been contacted and was looking into the situation. He got back to me and explained the DDOS situation. I asked them to remove the null route, to see whether the attack had subsided. He told me that they don't have the authority to do that; it can only be done by a network technician. And Speakeasy doesn't have network technicians on duty on weekends. Yes, seriously, that's what he told me. Presumably the DDOS attack was serious enough for someone to be called on to duty. But once such an action has been taken, it can't be undone throughout the entire weekend - giving up to about 60 hours of downtime. Of course, the script kiddies know that ISPs have such negligent policies, which is why they carry out their attacks on weekends.

So, I spent the earlier part of today making further changes to the various services that deepthought runs, but things are going to remain essentially broken until tomorrow at the earliest. And I can no longer recommend Speakeasy to anyone who actually needs reliable, “business” class service, because it turns out that they don't offer such a service. No attempt whatsoever to contact a customer when they take an action that will cause an indefinite outage, and no ability to correct a network problem over a weekend, are both grossly incompatible with a “business-class” service.

Addendum: I've been generally impressed with the service and support (in some cases highly proactive) that I've received from Speakeasy in dealing with other issues since this event, though it still leaves me concerned about Speakeasy's response should anything similar happen in the future.