Downtime: Database Hard Disk Failure

Post by **Themicles** » Tue Jun 10, 2008 11:14 am

Our database has experienced a Primary Master Hard Disk Failure and is currently down for diagnosis and repair.

We take daily backups of the database, so at worst it may get rolled back by 24 hours.

This down time may be extensive as the hard drive may need to be replaced (we have a spare or two to switch to).

During this downtime of the database, the servers will be down as well as they generally cannot function properly without it.

Post by **Themicles** » Tue Jun 10, 2008 3:16 pm

After some shuffling of hard disks, writing a new MBR on a hard disk that was previously a secondary slave and is now a primary master, Lazarus is back up.

The database was completely safe from the hard disk failure. The servers are back up.

Post by **Arkon** » Tue Jun 10, 2008 3:19 pm

I have said it before, and I will say it again. You Roxorz Themicles

Rali'vinee · Post by **Rali'vinee** » Tue Jun 10, 2008 3:23 pm

+1

*encourages teh clicky to DONATE:*

portal.php

edit: I've replaced the long page widening link with a link to the front page where donation links for both the hosting of the game servers, and the website hosting reside. - Themicles

Tremayne7 · Post by **Tremayne7** » Tue Jun 10, 2008 3:37 pm

Themi and the Hala Team all Rock!

You guys are the best.

To my fellow players....

I think we can find a few gold coins to toss Themi's way.

Thanks and love as always,

Tremayne

Post by **Themicles** » Tue Jun 10, 2008 5:09 pm

I was about ready to pass back out when I made my last post... just realized the first sentence is incomplete. Fixing it now.

EDIT:
And now I'm editing this post too...

Some more details: Apparently Lazarus' power cord was mistaken for another one, and Lazarus ended up not plugged into the UPS, while something else that didn't need backup power was. This morning our power started flicking on and off rather rapidly and we woke to the sound of computers turning off and back on along with the warning beeps from the UPS telling us it was switching to backup power.

Many people know that rapidly powering up and then powering down a PC can really trash hard disks. Most cases, you may just end up with a corrupt Master Boot Record, or with Windows 2000 and higher you may end up with a corrupt hard disk controller driver (hint, run a "Recovery Install" from your Windows install disc. DO NOT format! Do not start a fresh install).

This killed the Primary Master hard disk that had the Master Boot Record for starting the computer. It was a Maxtor 30 GB hard disk that came out in 2003. Right around the time Maxtor hard drive longevity started going down hill. Luckily, the Linux OS install along with the database were on another hard drive. Ironically, it was the oldest hard drive in the computer at 13 GBs and a production date in 1999! It still passes health tests even.

I had to write a new MBR to the 13 GB disk so that the system would start again, and rearranged all of the drives with the 30 GB one now missing. This meant I also had to rearrange some config files in Linux, but it's all good now.

The 30 GB hard drive only had a Win2k AS install and some files backed up to it that mostly had copies elsewhere on the network anyway.

For anyone concerned, we do have daily complete database backups that are saved to a whole other computer on the network with much newer and (hopefully) more reliable hardware. Lazarus is next on the block to be rebuilt, and I'll be going with a redundant RAID setup for backups (though the database will be on a single disk to keep write performance up).

whirlin_merlin · Post by **whirlin_merlin** » Tue Jun 10, 2008 11:56 pm

Themicles wrote:redundant RAID

A points if anyone can figure out why this phrase is amusing and how many points you are getting.

Also, Themi is 31337.

Post by **Themicles** » Wed Jun 11, 2008 2:19 am

whirlin_merlin wrote:
Themicles wrote:redundant RAID
A points if anyone can figure out why this phrase is amusing and how many points you are getting.

Too true, but not all "Redundant Array of Inexpensive Drives" configurations have any sort of data security (redundancy). Raid 0 for instance. ;P

Sable · Post by **Sable** » Wed Jun 11, 2008 8:05 am

neither are they always inexpensive

btw, do you actually suffer much in the way of write latency? (RAID 5 can cause this in high transaction DBs, but I'm not sure how many transactions per second a NWN server could create).

Its a daft question I know, but I assume RAID 10 is not an option?

whirlin_merlin · Post by **whirlin_merlin** » Wed Jun 11, 2008 12:22 pm

Themicles wrote:
whirlin_merlin wrote:
Themicles wrote:redundant RAID
A points if anyone can figure out why this phrase is amusing and how many points you are getting.
Too true, but not all "Redundant Array of Inexpensive Drives" configurations have any sort of data security (redundancy). Raid 0 for instance. ;P

Right, but with the acronym being what it is the phrase is still redundant.

Sable wrote:neither are they always inexpensive

Indeed not. RAID is sometimes said to stand for Redundant Array of Independent Disks.

Rudiki · Post by **Rudiki** » Fri Jun 13, 2008 1:15 pm

*looks at all the pretty words and is glad there are people here who understand them*

I'm also glad I was unable to log in anyway for RL reasons when the servers were down; so painless; please always arrange things that way!

Druid523 · Post by **Druid523** » Fri Jun 13, 2008 6:58 pm

Rali'vinee wrote:*encourages teh clicky to DONATE:*
portal.php

Done!