G[F]ail (or “F*ck the Cloud, Day 2”)

Posted by T. Greg Doucette on Feb 28, 2011 in Technology | Subscribe

Another day on the internet, another day with no access to anything Google-affiliated :mad:

Now at least it says Google is trying to fix things...

After sporadically checking Gmail throughout Trial Practice and Business Associations this morning, I finally just gave up. Still no email. Still no docs.  Nothing. Nil. Nada.

At least Google was kind enough to put a new redirect in place when I tried to access my account. Now it tells me explicitly why I can’t access my account, instead of sending me to a page saying I violated their Terms of Service.

After winging AAJ trial team practice tonight since I didn’t have any of my docs, I made my way home from the law school around 11ish when I discovered I now at least (it appears) have access to my non-Gmail services.

I also noticed Google had posted an update on their blog making it out like there was never really an issue.  Here’s the copy/paste, but you can read the original at this URL:

Gmail back soon for everyone
Monday, February 28, 2011 | 6:30 PM
Posted by Ben Treynor, VP Engineering and Site Reliability Czar (24×7)

Imagine the sinking feeling of logging in to your Gmail account and finding it empty. That’s what happened to 0.02% of Gmail users yesterday, and we’re very sorry. The good news is that email was never lost and we’ve restored access for many of those affected. Though it may take longer than we originally expected, we’re making good progress and things should be back to normal for everyone soon.

I know what some of you are thinking: how could this happen if we have multiple copies of your data, in multiple data centers? Well, in some rare instances software bugs can affect several copies of the data. That’s what happened here. Some copies of mail were deleted, and we’ve been hard at work over the last 30 hours getting it back for the people affected by this issue.

To protect your information from these unusual bugs, we also back it up to tape. Since the tapes are offline, they’re protected from such software bugs. But restoring data from them also takes longer than transferring your requests to another data center, which is why it’s taken us hours to get the email back instead of milliseconds.

So what caused this problem? We released a storage software update that introduced the unexpected bug, which caused 0.02% of Gmail users to temporarily lose access to their email. When we discovered the problem, we immediately stopped the deployment of the new software and reverted to the old version.

As always, we’ll post a detailed incident report outlining what happened to the Apps Status Dashboard, as well as the corrective actions we’re taking to help prevent it from occurring again. If you were affected by this issue, it’s important to note that email sent to you between 6:00 PM PST on February 27 and 2:00 PM PST on February 28 was likely not delivered to your mailbox, and the senders would have received a notification that their messages weren’t delivered.

Thanks for bearing with us as we fix this, and sorry again for the scare.

A “scare”? Really??

I realize Google has a vested interest in downplaying this situation — after all, they’re trying to convince major corporations to entrust their networks to The Cloud — but I’m almost offended that they’re acting like (i) losing access to my email for 2 days now (potentially permanently), complete with (ii) redirects to a page saying I violated the Terms of Service, alongside (iii) cryptic and thoroughly uninformative status updates on the Google Apps Status Dashboard, and (iv) no substantive response from the company until after the media catches on to what’s going on, is all merely a “scare.” :roll:

Then there’s the rest of the content in this non-apology apology. They’re backing up my email from tape archives but it’s somehow still all there? I call bullsh*t: the sheer volume of tape cartridges they’d need to use to back up all their users is too huge to believe they do these backups daily. A more likely explanation is that they’ve got a monthly or quarterly tape archive that they’re going to use to restore my account (assuming they actually restore it), which would mean I’ll have permanently lost anything recent.

We’ll see what happens over these next 48 hours that they’re supposedly working. But for now I’m drifting toward becoming one of those anti-GOOG partisans…

[Postscript: And to those of you thinking “But TDot, Gmail is free! You shouldn’t complain!”: I’ll stop complaining about losing my access when Google decides to give me the advertising $$$ they made from ads targeted based on heuristic analysis of my email messages ;) All the “free” users are paying for Gmail service, exchanging the privacy of our email correspondence for cloud storage and reliable access.]

Tags: , , , , , , ,

2 Comments

Carlos
Mar 1, 2011 at 10:38 PM

I am a pretty big fan of Google and The Cloud and I am glad you are one of those people who is not naive and takes the cloud for granted and backs up their data. I love the idea of the cloud and its versatility, but we must also face the reality that not everything is 100% secure and that it is also our responsibility to make our own backups. Personally, I only back up the truly important emails from all three of my Gmail accounts which makes backup much less of big thing to do, you know?


 
TDot
Mar 2, 2011 at 9:28 AM

I’m with you in general — having docs in the cloud is certainly less risky than having all my backups in my apartment, which theoretically could go up in flames at any time :)

But situations like this make me seriously question the concept, especially when this particular bug was one of Google’s own creation, and their management of it from a PR standpoint has been horrible (since it “only affects 0.01x% of users” of course)


 

Copyright © 2019 law:/dev/null All rights reserved. Theme by Laptop Geek.
Find TDot on Twitter or on Google+.