Most customers by now are fully aware that we experienced an email issue lasting a little over one full day this week. As much as it pains me to admit we had a failure, we owe it to clients to let them know what happened and why.
As painful as the loss of email in our daily lives is due to our reliance on it as a primary method of communication internally, with clients and vendors alike, this is only the second outage we’ve experienced in the eleven years we’ve been using this mail host. From an uptime viewpoint, we’re not complaining! (Ok, we’re complaining a little, but really… it’s very little!)
DDOS Attack – What happened?
Since our users aren’t computer geeks, we’ll explain it in the geeky way, and the non-geeky way. A DDOS attack is a “distributed denial of service” attack and it exactly what it sound like. It’s a cyber attack in which rather than trying to steal someone’s information, they’re simply trying to disrupt your use of the service. They use hundreds, thousands, or hundreds of thousands of distributed computers around the world and basically flood your network with so much traffic that you can’t get to it yourself. Imagine if you were trying to make a phone call to someone and you KNOW the number is the right one but you keep getting a busy signal because there are one million other people calling that person non-stop all day just to prevent them from being able to use the phone. That’s the general premise of a DDOS attack.
I’ll use one more analogy for the non-technical people. You know the post office with your email is only 5 miles away down a two-lane road. You pull out and get on the road only to find bumper to bumper traffic for miles in both directions. Your post office is still open. You still have a box there. You just can’t get to it through all the traffic. That’s basically what happened yesterday.
The DDOS attack started yesterday, Monday March 25th at about 5 AM CST. The data center in Dallas experienced a massive attack. The firewalls, DDOS protection service, and some of the engineer’s own proprietery mitigation services were able to contain the initial attack with only a minor hit to network performance.
However, three hours later, while still dealing with the first attack, not to mention the additional strain on the network from legitimate customers trying repeatedly in vain to get to their email, a second attack was perpetrated, not on our host, but on the DNS provider itself. (That’s another company altogether suffereing, probably from the same kind of attack, at the same time).
Was any data taken? Was my account ever at risk?
No. Nobody lost anything and nothing was ever at risk. This wasn’t an attack to steal data. It was purely a malicious attack intended to disrupt service for no other reason then “because they can.”
What happens now?
Our host is staffed with some amazing people. I’ve known some of them on a first-name basis for almost ten years. They’ve been rolling out new features like crazy this entire last six months in a ramp up effort to provide all our customers with new and upgraded service. That’s coming as soon as I get the time to start migrating clients over one at a time.
As part of that , they already had a “upgrade DNS” item on the to-do list. It just wasn’t on the top of the agenda. They depend on their DNS providers to do a good job and for the most part they had done so in the past.
In this instance, the DNS provider itself was slow to respond, and didn’t really want to accept any blame for their own latency. Technically this was considered a performance issue, not an uptime issue from their perspective, so they don’t really have to admit their part in it, or offer any refunds, or open themselves up to any lawsuits. I get it. It’s their job to protect their company, but its still pretty unprofessional in my opinion.
With no support from that end, the engineers at our host came up with a workaround to route DNS traffic through different providers, both to solve this issue, and to prevent it from happening again.
Is service restored now?
Yes. Service has been restored for most of Tuesday but there were still some customers unable to get mail. I want to take a moment to tell you WHY that happened and how you can prevent it happening in the future.
Account Locking – what it does and why.
When Outlook started prompting people for passwords, and it usually doesn’t do that, users started putting in passwords they thought were correct. Maybe they were. Maybe they weren’t. Either way, if you start hitting our mail server with password requests rapid-fire from Outlook, Android, iPhone, iPad, Laptop, etc… our mail server has a safety feature that locks out the account. Even if you DO have the right password, you still can’t get in. You can hammer away at it all day long, even with the right password, and it wont’ let you access email. That’s by design to protect your account against brute-force attacks from someone trying to access your account.
When that happens, customers have to notify us and we can unlock the account for you. Under a normal momentary outage, this wouldn’t have posed any kind of problem. Since it was an extended outage, more and more customers simply started to doubt their memory of what their password was supposed to be and started trying all kinds of passwords. Account after account was locked out when the anti-intrusion system thought someone was trying to access something they shouldn’t.
I eventually logged into the server myself and unlocked EVERY account whether it needed it or not and that finally seemed to stem the flow and allow me enough time to call customers individually and verify they were back online with email access.
What if I still have problems?
If you are an existing customer and have problems accessing your email, simply use your customer support portal, call the office, or text me individually and I will be sure myself or one of the team work to get your issue handled. I think I can safely say that 99% of customers have email restored and the only issues that might remain are some newly created account-lockout issues that might need to be reset again.
We sincerely apologize for the impact this has had on customers. It was nothing on our end, and nothing on our hosts end that could have prevented an attack like this before it begins. We already have preventative measures designed to avoid the majority of these kinds of attacks, but if you’ve read the news any time in the last couple years you know the threat landscape is always evolving. No one can prevent every kind of attack until someone else suffers from it first and they figure out a plan to mitigate it.
If any customers have any questions, please do not hesitate to reach out. I’d be glad to discuss this with any of you at your convenience.
Warmest thanks for your patience this last 24 hours,
Tommy Jordan, CEO