That was NOT FUN

Early Sunday afternoon, there was apparently a power and/or UPS failure in the datacenter at my office. By 5pm, most of the systems were back up except for one that I’m responsible for.

Fire up the VPN, hit the ILO remote console (one of the few things that makes using x86 systems as servers bearable) and see “Cannot find boot.bin” Uh-oh. I head to the office at 6pm.

It seems that a certain Sun patch turns non-GRUB systems into non-bootable systems after application. There’s a specific set of steps to follow when this patch is installed and a system is “upgraded” to the GRUB bootloader, but apparently Sun’s “smpatch” utility does not follow these steps. The patch had been applied months ago, but the system didn’t get rebooted until the power outage.

I figured “Okay, the system was running Solaris 10 FCS, so its time to do an upgrade install of S10u4 anyway”. After some other problems and workarounds, four hours later, I watch in resignation as the install hangs and locks up (not accepting keyboard input at a Y/N prompt) while trying to install the CPQary package. This package is the drivers that Solaris x86 needs in order to use the hardware RAID built into the Compaq DL360.

I bite the bullet and do a “nuke from orbit” fresh install of S10u4, planning to restore from backups. I ended up having to rebuild most of the services on the box by hand (which was better in the long run, as things needed cleaning up) as our backup system had also been affected by the power outage and it wasn’t available until Monday morning.

To make a long story short, I went to the office at 6pm Sunday, and finally walked out of my office to go home and get some sleep at 10:30am Monday. 16 hours is the longest single shift I’ve ever pulled anywhere, and certainly the longest after-hours session.

I’ve got one more service to restore onto the box on Tuesday, but it’s non-critical and could wait until I got some rest.

I can’t complain – I might have incidents like these once or twice a year, and it’s a lot better than getting called or paged every other day like I was used to at my last job. I really like my job and my managers and coworkers.

Sirius-ly Persistent?

I was a happy Sirius Satellite Radio customer for over a year, using it mainly to pick up the 80s channel because I couldn’t get decent local FM reception in the office building where I work. I was finally able to improve my local reception and decided to cancel the service and sell the radio to a friend.

I had no problems cancelling the account; my radio stopped working a few minutes after I got off the phone with Sirius (even though it should have been paid up to the end of the billing cycle). I figured that everything was done with, and sent my friend his “new” radio.

About a week later, we started getting calls to my home phone that caller ID identified as being from Sirius. After two or three missed calls, I picked up the phone and called Sirius customer service at 1-888-539-SIRIUS on August 22nd. I spoke to a “Gabrielle”, who said that my account was paid up in full, closed, and she didn’t know why I was still getting phone calls. I told her “If you wanted to keep me as a customer, you should have done the save when I called to cancel, not two weeks later after I already got rid of everything.”

Gabrielle said that she would put a note in my account, and if I got any further calls, to tell them to not call me again. I asked her, “If I’m already on the phone with you right now, why should I have to tell anyone who calls that they shouldn’t call me again?” Her answer was “Well, in case they DO call you again, even though I’m putting a note in your account…”

Last week, my wife called me at work. “Sirius called again.” I called them on September 14th, and spoke to a “Cheryl”. I explained that my account was closed and that I did not wish to receive any calls from anyone at Sirius as I was no longer a customer. She said “I don’t know why we’re still calling you, but I’ll put your number on the Do Not Call list.” She didn’t have an answer as to why this had not been done the first time I called.

This morning, my wife called again. “Sirius called, AGAIN, and when I tried to tell her that we wanted them to stop calling, she rattled something off a script and hung up on me.”

I just finished making my third call to Sirius. This time, on September 17th, I spoke to a “James” and explained the situation. He once again verified that my account was closed, that I was fully paid up with them, and that I had notes on my account from the previous two calls. James first tried to remove my phone number from my account records, but when that failed his solution to this (according to “what my manager told me”) was to put a “privacy flag” on my account. “This should stop any emails or calls from us.”

I can only hope that the third time is the charm; otherwise I’ll have to call them again and stop being polite. The only way I’ll ever have their service again is if they want to give it to me for free; an overzealous customer retention department has soured me on what was otherwise a good experience with their company.

Houston 1940 Air Terminal Museum

Amy and I visited Houston’s 1940 Air Terminal Museum today, and I took a lot of pictures. I really love the architecture and look of the place, and am glad that it wasn’t bulldozed and is instead being restored for people to enjoy.

The museum is housed in The Houston Municipal Airport Terminal, an airport terminal that was constructed in 1940. The terminal building is an example of classic art deco airport architecture from the 1940s. Designed by architect Joseph Finger (who also designed Houston’s City Hall), the terminal was built to meet Houston’s growing role as a center for air commerce in the 1930s. The terminal served as the primary commercial air terminal for Houston until 1954.

Edit: here’s some of my favorite pictures. Click a thumbnail for the Flickr page.

Art Deco ChandelierThe 1940 Air Terminal MuseumThe 1940 Air Terminal MuseumThe 1940 Air Terminal Museum
The 1940 Air Terminal MuseumThe 1940 Air Terminal MuseumThe 1940 Air Terminal MuseumThe 1940 Air Terminal Museum

Giving up on the Mighty Mouse

I’ve given up on the Apple Mighty Mouse, having gone through two of them since 2005. I finally got fed up with having to completely dissasemble it to properly clean the ball rollers every couple of months, and then either glue or tape the “retaining ring” back onto the bottom of the mouse.

In my opinion, Apple really should have made the scroll ball optical instead of mechanical rollers and Hall-effect sensors, and they should have designed it to be easily cleanable. Apple’s suggestion to “Roll the ball vigorously while cleaning with water” while holding the mouse upside down” is NOT an acceptable solution. More than once, I had to completely take apart the Mighty Mouse, unscrew the “ball module” from the top shell, and then take THAT module apart to be able to clean the gunk from the tiny rollers with magnetic ends.

I tried every solution in the book to avoid dissasembly, but none of them worked for long. Saliva, Windex, tiny adhesive tape strips wrapped around the ball, etc. I had moderate success with Hoppe’s #9 gun-cleaning solvent, but even that only worked for a week or two before the rollers started gunking up again.

If the Mighty Mouse wasn’t $49, it might be a different story – I’d just go buy a new one every six months. For now, I’ve replaced it with a $25 Logitech LX3 optical mouse. The only problems I’ve ever had with Logitech mice were their microswitches wearing out after four or five years of heavy use.

As a final note, I’ll say that the Mighty Mouse is only the second piece of Apple hardware that I’ve “given up” on – the first was the original FireWire iSight camera. I bought one in 2004, and returned it for a refund the next day – for $150, I expected much better audio and video quality than I got from the camera.