Karlonia.com
For Gold, Peace, and Freedom

Karlonia.com

Entrecard Suffers Outage from Electrical Fire, Begins to Recover

June 2nd, 2008

entrecard-logo.jpgMembers of the Entrecard community experienced an unexpected interruption in their card dropping routines Saturday night when an electrical fire at The Planet, a major data center in Houston, Texas that houses the site’s servers, required all power sources (including generators) to be shut down according to instructions from the Houston Fire Department. This took the Entrecard site completely offline for a few hours while crews at The Planet worked to assess and repair the damage. Shortly after midnight on Sunday morning, the Entrecard staff emailed members to let us know what was happening:

Greetings,

Unfortunately, there has been an electrical fire at the data center housing Entrecard’s servers. The servers have not been damaged, but the site may be down for as long as 12 hours.

Entrecard is hosted with ThePlanet, and this fire has affected over 9000 websites, as many of the world’s most popular websites are hosted with ThePlanet. They are working around the clock to get their data center back online, and we are hoping that they get it back online fast.

Thank you for your patience and understanding with this rare and frustrating problem. Unfortunately, there is absolutely nothing we can do but wait for The Planet to get their datacenter back online.

Please check back with Entrecard in 12 to 24 hours.

Thanks,
Entrecard Team

Fortunately, the Entrecard staff was able to respond fairly quickly and managed to get a basic splash page up that informed visitors about the outage and restored the widget graphic on Entrecard-enabled blogs, which prevented us from having to worry about displaying server error messages on our sites where the widgets would normally be located. There was also a link on the splash page that directed us to The Planet’s forum, where we were able to learn more specific information about the extent and actual cause of the outage from their management:

This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.

We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.

There is no impact in any of our other five data centers.

I am sorry that this accident has occurred and I apologize for the impact.

By late Sunday morning, they had begun migrating some of the DNS infrastructure to other locations, and the Entrecard site was back up and functioning, but had not yet fully recovered as the overall page loading speeds were much slower than normal. However, in spite of the slowness I managed to get a few drops in and checked the Campaign page to see if it was still possible to advertise. Then I realized that because the outage had lasted a little over 24 hours, there were many blogs that had dropped one or two price levels during this period. However, because the site had been down, there were no advertisers around to snap up the relatively inexpensive ad deals.

gatling-gun-1864.jpgAt this point, my opportunistic marketing sensors began to tingle. In classic Gatling gun fashion, I began firing off ad requests in rapid succession into the target-rich environment of 2ec and 4ec advertising offers, dipping below my normal credit reserve of 20,000 and battling slow servers during the page loads. In the end, I wound up getting in over 800 ad requests, most of which have now been accepted:

entrecard-ad-blitz.jpg

By 7 PM Sunday evening, Entrecard had another outage, but this one was actually planned so that the team could work on “server optimization issues”. By Monday, the site was back up and functioning at close to its normal speed, although there have still been some occasional periods of slowness. Overall, it looks like the trend is toward recovery, and most Entrecarders seem to have resumed their normal dropping routines. It remains to be seen whether or not my latest advertising blitz will bear any significant fruit, but at least this outage was able to provide me with a few moments of fun in this otherwise humdrum existence.


8 Responses to “Entrecard Suffers Outage from Electrical Fire, Begins to Recover”

  1. comment number 1 by: Binary Ant

    Hi! Thanks for visit my blog. Hopefully this technical issue has been solved. I’m starting with Entrecard and my first English blog :) , my natural language is Spanish. There is a little difference in prices between you and me ;) but I’ll work to fill that gap.

    See you!

  2. comment number 2 by: Jude

    The last year I have been totally devoid of any marketing capabilities so I missed the oppotune time to pounce. Thanks for stopping by have a good evening….Jude

  3. comment number 3 by: SEO Man

    This was huge problem that affected a lot of sites. Hopefully all the problems are fixed.

    Good job on the quick pounce and getting the good deals.

  4. comment number 4 by: James

    Thanks to EntreCard team for their effort to bring the EntreCard back online. But the lesson learn for EntreCard or the Planet is to think about better Disaster Recovery plan for their data center..


  5. Hi, Thanks for dropping by my blog.

    It seems that evereything is running fine now. Regarding the 2ec and 4ec advertising offers, I don’t know what’s your experience, but in mine’s I’ve never managed to get more than a couple of visits (if ever) from those such low priced sites.

    Now I’m targeting to the most expensive ones, trying to find an opportunity window in which ad prices fall to a reasonable (in my budget frame) one.

  6. comment number 6 by: Mobile J Games

    Nice blog… Entrecard is back and working fine now. Bad it was down for the whole day..

  7. comment number 7 by: Sonnie

    It’s seem most serious droppers had some withdrawal pangs the other day, as most missed visiting their favorite e-buddies without the drop inbox, and most also missed the traffic.

  8. comment number 8 by: Karlonia

    @Binary Ant:

    Your blog is coming along nicely - lots of interesting topics there. If you ever need help with English, you can check out my English usage category from the sidebar. I am currently posting weekly English lessons on various usage and grammar issues. You can also contact me through the Contact page if you need help with something specific.

    @Jude:

    Marketing capabilities will take time to develop - it seems to be one of those things that is an “acquired taste”. I never knew anything about Internet marketing or advertising until I discovered GPT programs and traffic exchanges back in 2004. Although I’m not exactly rolling around in money piles yet, I’ve definitely come a long way since then.

    @SEO Man:

    Thanks - nice keyword placement there, although I can tell you from experience that your field is very competitive!

    @James:

    Yes, the Entrecard team actually seemed to do a pretty good job considering the circumstances. As their site and traffic continues to grow, they will probably put better disaster recovery and contingency plans in place.

    @Anthony:

    In general, it is true that the low-priced sites do not bring in as much traffic. However, my reason for advertising with these “newbie” sites is to attract clicks and drops from the other webmasters themselves even in cases where the sites have no other visitors. When other site owners click through to my beloved blog here to check out who is wanting to advertise with them, most of them will go ahead and drop on my strategically placed Entrecard widget and may even read some of the content or subscribe to the feed.

    These drops are a major factor in determining the popularity rating of my site at Entrecard. More drops leads to higher placement in the Entrecard rankings, which in turn leads to more drops and visits from other Entrecard members. It is a self-reinforcing cycle that can work in my favor over time, especially when I can fire off hundreds of cheap ads within a relatively short period of time.

    @Mobile:

    Actually the downtime was a nice little break for some of us. We had the chance to rest our clicking fingers and catch up on other things such as blog commenting and advertising.

    @Sonnie:

    Yes, we did have a small drop in traffic because of the outage and even some early withdrawal symptoms, but it did give us a nice vacation and reminded us that there is actually a life outside of Entrecard, which is difficult to imagine for members who become sufficiently addicted.

Post Your Comments, Opinions, or Suggestions Here:

Name

Email (optional)

Website (optional)