Explanation for Server Downtime and Apology

Talks that may or may not have anything to do with Hala or NWN

Moderator: Top Team

Post Reply
Themicles
Game Server Admin and Web Monkey
Posts: 306
Joined: Thu Dec 29, 2005 1:46 pm
Location: Michigan
Contact:

Explanation for Server Downtime and Apology

Post by Themicles » Fri Aug 14, 2009 2:13 pm

Sunday, the house leaked and got water on the battery backup. Fried the battery backup and in turn fried the motherboard on the main network server. In the process of getting the network back up, I find out that the hardware that made up my desktop PC that I thought has been dead for months was good, but it needed to go into the server to save our network. (In other words... Damn! I could have been using my own computer these last several months!)

4:30 in the morning on Tuesday. I was actually sleeping, believe it or not. DSL dropped for an unknown reason and was down until I woke up. (In other words... Why do things have to wait til I'm asleep to break!?)

Then around about the same time yesterday, the DSL dropped again... while I was sleeping. While investigating the issue linked in the last sentence, I also notice that the power supply fan is slowly dying. By 1pm, I had swapped out the network card, the power supply, video card and the CPU. The last two items were parts meant to go with the hardware from my computer. The CPU being 3.0ghz with hyperthreading versus 2.4ghz without. The power supply being a 550watt being more than enough to handle everything in there with room to spare. Reboot, troubleshoot the DSL issue. It's not the network card's fault as the issue continues. I finally move the PPPoE login to the DSL modem itself and just setup the network connection between Zeus and the DSL as the NAT interface.

Satisfied that everything was running well, I didn't have to worry about a dying fan, and the DSL login issues were solved, I started the rest of my day. Kessa and I own two Dodge Neons. We know them well, and have worked on nearly every part of the cars. When a friend was looking at buying a Neon, I offered to work on it if they needed anything, free of charge. Sure enough, the car needed (and still needs!) work. The oil pan was rusted to the point that there was a steady leak. I start work on draining the oil pan for an extended time to make sure there wont be any significant drips when I take the pan off and am lying under the engine. Get the pan drained, put the plug back in to catch any remaining oil in the pan while I start disassembling. To get at all the bolts for the oil pan, and to get the pan out at all, I have to remove the front motor mount. This is Michigan... Steel at the lower front of a car is going to rust. Bad. I spent most of the first couple hours just getting rusted bolts and nuts off the car.

Take a break to get something to eat, because I hadn't eaten all day. By this point I was starting to get shaky given all the effort I had to put into those rusty parts. I notice we don't have internet. :x So I check, and Zeus is off. I press the power button... nothing. This is the old behavior of my PC that made me think that hardware was dead. After turning off the power supply, unplugging it, and replugging it... it works. It stays up for about 10 minutes before I figure it's good and go back to work on the car.

I get back out there, finish fighting with some rusty bolts, as well as taking out the oil covered bolts on the engine side bracket for this motor mount (one of which was insanely long for it's application). With the bracket off, and the oil pan bolts slightly loosened, we go pick up the oil, filter and gasket. Get back, take out the oil pan bolts, and try to drop the pan. But, it's getting stuck no matter which way I move it. I realize I need to take a trans to engine bracket off the back of the engine to allow me to take the inspection plate off the side of the trans so that I can have that extra inch of movement towards that side to get the pan down. No big deal. While the pan is off, I check the rod bearings to make sure they're not so loose as to shake them by hand, all seem good. I go to put the gasket up to make sure it's a proper fit and find there's a tab sticking off one side of the gasket that hits the bracket for the AC clutch. Fix that issue, put the gasket on the pan and the pan on the car.

Start the car, idle it for a couple minutes. Shut it down, check for leaks. Damn, it's leaking on the driver's side on the back corner of the block. Clean it thoroughly, start the car, crawl under it while it's running. Huh, no leak! So I clean up, get all the tools set aside and out of the way, clear out from under the car and put it back on the ground. Take it around a couple blocks to test it, pull it back in, jack it up and check for that leak again. It's back :evil: I am not happy at this point. I did absolutely nothing wrong, yet the damn thing still leaks. Now we have to replace oil again, and probably the gasket too, to re-do this job and fix the leak. If I had a place for them to stay, I would have asked them to stay the night while I stayed out in the garage re-doing the job through the night. But, we don't. So I recommended to them, since the leak isn't too bad, that they go home and get some rest, then come back in the morning so I can do the job all over again. (In other words: Why the hell can't things ever work right the first bloody time!)

While finishing up the car, I started suspecting that the power supply was to blame for the problems with Zeus not staying turned on. The list of blame at this point and in order of likelihood went; power supply, CPU, video card. In order of ease of checking; video card, power supply, CPU. Given the chances that the power supply was at fault, I swap the power supply with the dying fan back in. Sure enough, Zeus has stayed on since then. We'll be grabbing a power supply this weekend that doesn't have a dying fan, and isn't killing the system every ten minutes.

And to top it all off... I'm itchy all over today. Damn mosquitoes started swarming me when I was finishing off the car. I have 8 bites in close proximity on my left hip where my shirt had come up a bit and exposed skin while I was lying on my back on the creeper under the car. And more bites spread around such as my elbow, and my legs. Buggers bit me through oil and grime, and even my jeans.
World Leader: Tairis'nàdur
Administrator: Confederation of Planes and Planets
Server Host & Web Monkey: Hala The Heroic Domain of Ysgard

_JM_
Honor Guard: Church of Pants
Posts: 1059
Joined: Tue Jan 09, 2007 8:23 am

Post by _JM_ » Fri Aug 14, 2009 2:56 pm

Ouch. Sounds like with both the car and the computer you went about things in a methodical sensible manner. Checked for the leaks thoroughly but it came back on the car. Put in a new PSU (with more than enough power) but found the old dying-fan one worked better…

Sympathies and I hope your luck improves, that nothing else breaks or at the least that things fix quickly and first time for a while.

Gladenfar
Whelp of the Unholy Church of Newbieism
Posts: 37
Joined: Mon Jul 05, 2004 2:42 pm
Location: Pennsylvania (USA)

Post by Gladenfar » Fri Aug 14, 2009 5:26 pm

no need to apologize. All your time and effort is greatly appreciated!
*smiles*

ambrosia
Honor Guard: Church of Pants
Posts: 1466
Joined: Fri Apr 18, 2008 10:43 pm
Location: Usually Zvidureth

Post by ambrosia » Fri Aug 14, 2009 5:38 pm

Gladenfar wrote:no need to apologize. All your time and effort is greatly appreciated!
+10

Absolutely agree!

Thank you Themi! :mrgreen:

Chemical-Burn
Knight of the Holy Church of Big Mouths
Posts: 213
Joined: Thu Aug 24, 2006 12:29 am
Location: GMT-8(-7)

Post by Chemical-Burn » Fri Aug 14, 2009 5:54 pm

What they said. I just tried to help by saying what time it went down but I guess you troubleshot it well enouph already. Thanks again Themi.

Moonyju
Minor DM
Posts: 1450
Joined: Mon May 07, 2007 12:37 am
Location: *points up* That's Yana and Zunya (GMT-5/-4)

Post by Moonyju » Mon Aug 17, 2009 3:31 am

Indeed! No need to apalogize!

We appreciate everything you do!

*hugs*

Thanks Themi! :)

Post Reply