Servers down (They're back up now -Themi)

The latest announcements from your Hala Team

Moderators: Arkon, Top Team

Post Reply
LadyPhoenix77
Knight: Church of Pants
Posts: 973
Joined: Mon Dec 27, 2004 5:11 am
Location: Whats the use of have names if you lose all your friends. That's where I'm from.

Servers down (They're back up now -Themi)

Post by LadyPhoenix77 »

Servers may be down for an unknown amount of time due to storms and issues related with storms.
Themicles
Game Server Admin and Web Monkey
Posts: 313
Joined: Thu Dec 29, 2005 1:46 pm
Location: Michigan
Contact:

Post by Themicles »

So, around 9pm last night during all the storms, I was chatting in IRC about paintball.

At 9:02pm, I heard a loud and deep zap come from the office room. At that time, our connection dropped.

I immediately suspected something fried in the office, and went to look. Zeus was dark and cold (off), the wireless AP was off, and I was hearing a quiet splashing sound. I rush over to Zeus to start trying to figure things out and the first thing I notice is the water on the UPS.

Yes, water, on the top of the UPS which is full of vent slots.

The heavy rains yesterday got into the wall, and seeped out the top of the window frame in the office for a long fall to splash on anything below. That anything was the Belkin UPS that serviced the network's domain controller (Zeus), the DSL modem and the wireless AP that also serves as the wired switch between that room and this room.

I immediately yanked all the power cords for the devices from the UPS, and crawled under the desk to unplug it from the wall. After moving it to the middle of the desk, I pulled the battery cover and pulled the batteries. It's cooked, but I didn't want any juice left in the batteries to continue arcing on water still dripping inside the thing.

After making sure there was no longer a threat of electrical fire from internal arcing, I set about getting power back to the rest of the room. After plugging Zeus, the DSL modem, and the wireless AP into a surge protector and that into the wall, I tried to power on Zeus. The fans spun up, it sounded like all was okay... and then it shut down again. This isn't Zeus' first encounter with power surges. It was already down one of two Athlon MPs and a chipset fan from roughly 5 years ago. It even exhibited this behavior before. So, we spent several hours thoroughly cleaning the computer in the hopes that tearing it down and rebuilding it would bring it back to life. It was disgusting. I'm amazed it hadn't already fried just from dust build up causing excessive heat.

Alas, it was all just time spent covering myself in a thick layer of dust. After clearing the CMOS, Zeus powered on consistently... even without a power switch. It would just cycle the fans like it was rebooting constantly, no video output, no beep codes. Absolutely nothing but fans revving up and down. We're out of whole spare computers that have gone unused due to upgrades. It was looking like an extended downtime. But I still had a last ditch plan before we gave in to giving up our plans for the next few months to build a cheap server.

Enter the Beast. For those of you who haven't been around long enough to remember, Beast was my desktop PC. Late last year it stopped working, and all diagnostics pointed towards a dead motherboard. Some suggested replacing the CMOS battery, but that never worked for me in past scenarios so we put it off. We did finally pick up a CMOS battery a couple months ago, even put it in, but didn't try powering on the computer. So given the growing desperation last night, we decided to try powering it on. It turned on, there was video... POST... and an error that there was no hard disk connected (which there wasn't).

We set about swapping the guts of the Beast into the Zeus' chassis. Did a quick power on test with Zeus' power supply connected to the scavenged internals. Connected Zeus' hard drive and routed cables. Turned it on for a full boot into Windows and spent an hour or so installing drivers. Luckily didn't have to do a repair install on Windows as often is required when doing an extensive hardware swap. Sometimes Windows doesn't know how to talk to the hard drive after that and needs a quick repair.

After installing all but one device driver, I started trying to figure out what was wrong with the secondary network card. It wasn't showing in device manager, but it's light would still turn on if a network cable was plugged in. This card was in the previous hardware, so Windows still had the driver. I couldn't figure out why it wouldn't see the card. So, I figure I should download the latest driver installer... but how? Without Zeus, we don't have internet. I tried connecting the DSL through the wireless AP, but something's wrong with the DHCP server in the firmware, and nothing wanted to work right. Then I had an idea...

I have an unlimited data plan on my phone. Try to download it on my phone, copy it to the network over bluetooth... Except my phone is a cheap one with a small screen and a mobile browser. Few websites work with it, and with so many manufacturers are using PHP scripts for their downloads rather than direct links, it was a no go. So, I grabbed the laptop, pulled the network card, and drove the 5 minutes to my sister's place at 4:30 in the morning. Sat outside in the car, hopped on her network and downloaded the drivers from the manufacturer's website. Came home, put the network card back in, but this time in a different slot. Booted Zeus expecting to install drivers, but wait... there's the card, and it's supposedly installed already. :evil: A trip to download drivers wasted.

I spent the next few hours fighting with the network connection between that card and the DSL modem. Tried reinstalling the drivers, the program froze. Tried uninstalling the driver, device manager froze. Tried rebooting, Windows froze. I pulled the card and moved it to yet another slot just on a hunch. Booted it at around 7am. The card showed up, was installed, was connected to the DSL modem, and the DSL interface was already logging in. Great!

After asking a few #hala users to ping the network, I found that all the port forwards were still there, but disabled. I fixed those and another routing issue and now everything is back up and running.

We're minus a $200 UPS that took a shower, minus an old and very aged dual Athlon MP system, but have regained a 3ghz P4 system that was used to rebuild Zeus. Now I really do have to upgrade to get a new computer, since now that mine is working, it's the domain controller.

I apologize for the extended down time. The money just isn't there to keep whole backup computers around for events such as this. In the event one happens, we have to go through every possible test and fix before finally resorting to scavenging off old computers or parts laying around from upgrades. Luckily, we had a whole one that ended up being in good shape.
World Leader: Tairis'nàdur
Administrator: Confederation of Planes and Planets
Server Host & Web Monkey: Hala The Heroic Domain of Ysgard
Arkon
World Leader
Posts: 2902
Joined: Fri Jul 02, 2004 11:28 pm
Location: Ironton, MO

Post by Arkon »

Thank you Themicles, once again, for all the hard work. At this time I would like to remind people about the donate button found when you first come to www.ysgard.org.

Servers are expensive to build and can be expensive to run. If any has the means and the desire, donations would be greatly appreciated as these donations are what help keep the servers running so everyone can enjoy our little world.
Themicles
Game Server Admin and Web Monkey
Posts: 313
Joined: Thu Dec 29, 2005 1:46 pm
Location: Michigan
Contact:

Post by Themicles »

Another short outage around 4:30am. DSL dropped for an unknown reason and took about 20 minutes to reconnect. :roll:
World Leader: Tairis'nàdur
Administrator: Confederation of Planes and Planets
Server Host & Web Monkey: Hala The Heroic Domain of Ysgard
darkfire
Squire of the Holy Church of Annoyance
Posts: 56
Joined: Tue Nov 07, 2006 6:54 pm

Post by darkfire »

This is not at all a form of bragging, but Avlis has gotten some generous donations and have done a huge upgrade to what we (also!) call the Beast. The difference a new server can make is huge. Money won't be wasted if you do decide to donate. Since we've upgraded, we've experienced a significant boost in reliability and server performance.

Just wanted to say that if there is anyone who isn't sure your donations would help, or that upgrading wouldn't really be worth it... it definitely is.
[quote]WrathOG777: This is a roleplaying game. There is no such thing as winning or losing. Only playing.[/quote]
Themicles
Game Server Admin and Web Monkey
Posts: 313
Joined: Thu Dec 29, 2005 1:46 pm
Location: Michigan
Contact:

Post by Themicles »

darkfire wrote:This is not at all a form of bragging, but Avlis has gotten some generous donations and have done a huge upgrade to what we (also!) call the Beast. The difference a new server can make is huge. Money won't be wasted if you do decide to donate. Since we've upgraded, we've experienced a significant boost in reliability and server performance.

Just wanted to say that if there is anyone who isn't sure your donations would help, or that upgrading wouldn't really be worth it... it definitely is.
While The Beast is nice, it wont solve any of the problems we recently experienced. The most benefit we'd get out of it is lower electricity cost, and that's it. If that single server has a hardware failure, everything would go down instead of just one. Considering our problems have almost always been hardware failures...

EDIT:
That's not to say that upgrading the individual boxes wouldn't be a good idea. NWserver can be a real resource hog.
World Leader: Tairis'nàdur
Administrator: Confederation of Planes and Planets
Server Host & Web Monkey: Hala The Heroic Domain of Ysgard
darkfire
Squire of the Holy Church of Annoyance
Posts: 56
Joined: Tue Nov 07, 2006 6:54 pm

Post by darkfire »

The money just isn't there to keep whole backup computers around for events such as this. In the event one happens, we have to go through every possible test and fix before finally resorting to scavenging off old computers or parts laying around from upgrades. Luckily, we had a whole one that ended up being in good shape.
I was mostly referring to this bit.

More donations equal new stuff. Old parts aren't as reliable (not saying new stuff can't break too though). If you are out of old parts and back up, then that would be bad. Getting donations in would allow for a new system instead of scavenging old parts. An upgrade would also entail some of the things I mentioned in the first post.

Not trying to compare Avlis' Beast or saying get that setup too, just saying that if people donate, having to not rely on older parts (assuming older parts can be scavenged for) is not the only benefit that can come from it.

(trying to encourage donations :P)
[quote]WrathOG777: This is a roleplaying game. There is no such thing as winning or losing. Only playing.[/quote]
Post Reply