Technogory

Okay, at first glance, you may say “What in the world does that mean?” Well, as many of you know who have been regular visitors, the site was going very well. However, I ran into a huge wall out of my control, and have been MIA!

I am back; ready to take on technology!

Lesson Learned The Hard Way

Okay, I have to admit to you what the issues were. I have contracts with various data centers across North America. I had a primary site in Philadelphia where a majority of my web business was run from. In early September, there was a catastrophic failure with hardware that was clustered together. Or so I thought…

You see, whenever you sign a contract with a data center, you have to be very specific on what hardware and service is being delivered. I thought I was. I thought I had purchased large multi-core servers with mirrored drives and off-site backups. Everything, up and until this catastrophe, had been running smoothly. I had no reason to doubt that there were any hidden issues.

When the failure occurred, the first thing I did was contact the service center to have the systems brought back on line. After waiting several hours, I get a cryptic message that the drive is corrupt. So, I send back a message and say, break the mirror and boot the system. Should be simple, right?

Wrong, after several more hours, I get a message that says, there is no mirror! “No Mirror!?!” I exclaim! So, I start going through the backup logs. I find that the backups that have been run are also corrupted. Apparently, the jobs put in place were going through the motions, but nothing was getting written.

Lesson 1 Learned

Check your backups. Why on earth I trusted that these services were happening beats the heck out of me. I practically demand that my large companies with in-house data centers perform regular checks and verifications of the backup data. As a matter of fact, most large companies will actually perform a disaster recover exercise just to test the very integrity of the backups.

Lesson 2 Learned

Don’t trust that anything is happening. Most data centers that you contract with provide some level of control. But nothing beats a trusted pair of eyes on the actual physical server(s) you are leasing.

Lesson 3 Learned

Spread your liability. In other words, in my case, I should have not relied on one data center as the primary, and the others as secondary. As it turns out, I had more problems with the primary then I have ever had with my secondaries. So, I have done just that. I leased a new data center, and have now spread my sites evenly amongst all of them.

Who Was The Data Center

It doesn’t really matter. Several weeks ago, I may have been tempted to name names, drop all kinds of bombs on them, but not now. Just know, that regardless of the data center, your service is only as good as the technicians and sales people behind the operation. If you don’t find a service provider that gives you 99% of the control, move on.