Recently, one of my DNN sites had the following rendered for the home page:
My first thought of course was a drastic one, "Oh my god! The site was hacked!" I QUICKLY logged in to the server to investigate. It was immediately apparrent that none of the files on the site had been tampered with, and there didn't appear to be any additional files in the site either. (Some of you may know, that one of the most common exploits of prior versions of DotNetNuke® involved a malicious user dropping files into various directories on your DNN portal installation.)
Well, the next step I took involved looking at the database connection. I opened up SQL Server Management Studio, and attempted to connect to the database. I got nothing but a
connection error. So I pinged the server. The ping failed. UGH! That meant that I had to go to the data center and physically visit the server. :( Luckily for my employer this happened on a weekend - but that was not lucky for me!
Upon arriving, I brought the server up on the screen. I saw the login prompt, so I attempted to login. I entered my credentials and the login prompt changed the status prompt like we are all used to. However, it appeared to be hanging. After a few minutes, the server rebooted on its own.
I was able to duplicate this behavior over and over. In addition, while I was looking up information online about the current symptoms, I noticed that even if I didn't attempt to login, the server would reboot on its own.
During the POST, I noticed that the RAID controller appeared to be failing. Although I cannot remember the exact text right now, it was not at all clear as to what the problem was. When I visited the support web site, it described the error as being related to not having enough disk space or memory.
Great! I can fix that! I'll just boot into DOS mode and clear out some dump files or something and we'll be back in business, right? Wrong. I couldn't get to DOS mode either.
Now, I guess I should tell you that this is a Dell Server. I know, I know. I have a different preference for desktop and servers as well... One thing that is cool though, is that Dell servers come with a handy little software utility that runs a diagnostic on all of the hardware.
I ran the diagnostic tool, and came up with only one "error". The DVD ROM drive was not accessible. Once again, I was happy! I'll just go into the BIOS and make sure that the DVD ROM is not in the boot sequence. That must be the problem. Then I can complete the troubleshooting in Windows. Nope. It wasn't the problem.
Since I initially thought that this was a problem that I created (because this particular server was able to chew up disk space if I didn't monitor it), I spent about 2 or 3 hours just trying to get into Windows or DOS mode so I could clear out disk space.
Well, I exhausted all of the ways I knew of to get into Windows. I even used a Linux boot disk that I use to get into Windows when the logins become inaccessible or the Administrator password is forgotten. Nothing I tried worked.
That left Dell Support. I called Dell Support, and spoke with a very nice and very American-sounding fellow. (Thank goodness!!!) He obviously didn't know very much about computing or system administration, but his "script" that he had to go through was VERY thorough. I was highly impressed. They must have an excellent documentation team there.
However, the troubleshooting process did take up about another hour. At the end of the call, I had a dead cell phone, and it was fully charged when I arrived. We finally determined that the RAID controller card had gone bad. After reseating it, it was determined that a new one was in order. That is indeed great news! Let's set it up.
Unfortunately, this was the Saturday during the Memorial Day weekend, so under the current service contract, the part wouldn't be available until the following Wednesday. Ugh! The
good news was that they offered an expedited service that guaranteed the part and installation in 4 hours. Woo hoo! Let's do it!
I went back to my office to get authorization for the expedited service, and to get the server software. The support person I was speaking to seemed to think that Windows might have been corrupted because his documentation says that we should have been able to at least get past the login. While I was picking up the software, I received authorization to expedite the service call.
I called Dell Support back and asked for the part to be delivered to my current location which they had on file, but to have the service technician meet me at our data center. Simple enough, right? Wrong. The courier that was delivering the part called me when they arrived at the data center an hour later. Grrr... This was getting exhausting!
The courier and I agreed to meet halfway between the data center and my office. When I arrived at our meeting location (a convenience store parking lot), I was surprised to see that
the courier was a husband and wife team in a beat up old (probably 1980's) vehicle. It had for sale signs on it, and looked like it was about to die. Also, they were not wearing any uniforms or identification of any kind. It seemed pretty unprofessional to me. I signed for the part, and while having a bit of small talk found out that their next delivery was in Fort Lauderdale. I wished them good luck. I hope they made it. :)
I am sure that they are just private contractors that get paid per mile or per delivery. With fuel costs as high as they are now, and they obviously are not making a whole lot of money at this, I hope they do okay. I really feel for them and anyone else in a similar situation. This economy and fuel costs both really suck right now and are putting everyone in a bind. :(
I arrived back at the data center, and the Dell technician arrived moments later. I had found out that he only lived a couple miles from the data center, so I just called him on my way from picking up the part.
In less than five minutes at the server, we had the part installed and the server up and running. Windows had no issues and continues to not have any. (You were wrong Mr. Dell Support Phone Specialist.) The technician confided in me that he changes out that same card all the time. He speculated that just during the past month, there were at least 20.
I am sorry, but that is A LOT! There has to be some systems engineer at Dell getting a butt-chewing, because that number is just for the local area. That means that there are similar numbers for every metropolitan area. What a loss of revenue!
We have been up and running strong since. Overall, I was impressed and very pleased with the Dell Support staff and service. I am disheartened at the frequency of RAID controller
failures with the PowerEdge 1950. I have more of them, so I am concerned about the RAID controller problem happening again. After mentioning this problem to a colleague, I found out that he also had the same exact issue recently with the same model server. Not good.
I also hope that companies like Dell can find a way to better take care of and support the folks that they have delivering their parts. At least supply a vehicle and a uniform. Brand the vehicle with a wrap of Dell product information like some other companies do. Why not?
On a scale of 5, I would rate the Dell Support experience a 3. The low rating is only because after finding out about the frequency of the problem, I do not feel that we should have gone through this at all. Also, a more experienced phone support person would have cut down the troubleshooting time by at least half, also cutting down on the server down-time. I also included in that score, my experience and uncertainty of dealing with the unprofessional courier service. That is not about how they spoke, or treat me, or anything, just the overall appearance of them and their car. And the final thing that came into that decision is that they mis-routed the delivery location of the part.
I would rate the PowerEdge 1950 at a 3 as well. If not for the part going bad and the fact that it is apparently a common problem, I would give it a 5 because I have not had any other problems with it.
I hope this helps someone out there. When all is said and done, I am still happy with the Dell product. It is getting the job done.