Well, that was fun.

e89eb2739d317a3add03de4d2230566c_viewI’m still here! Just been very, very busy with a massive relocation project at work that started in April (although the planning began in 2013) and finally finished in early September.

We moved around 500 users, their associated desktop systems, multiple computer labs, and most of an entire-floor datacenter/server room from one facility (in multiple buildings) to a newly-remodeled/built-out area in another building about four miles down the road.

This all had to be done with a minimum of disruption to production systems, and at the same time I had to move our DNS and LDAP infrastructure from systems housed in that datacenter to VMs running at our corporate datacenters – so that it wouldn’t be affected by the move.

Lots of long hours, and then the final move phases itself – the first weekend I spent 32 hours at the office, and the weekend after that I was here for 22 hours. Then three days later my mother had surprise emergency heart bypass surgery and I had to spend a week in a hotel in Oklahoma City while she pulled through and started recovery.

However, all that’s done, and my insane stress level is slowly, slowly coming back down to normal now that the holiday season is here.

Trying to learn to relax

It’s been a busy three months at the office.

A coworker had an extended two-month absence that increased my workload. Right after he finally returned, I was the primary contact for a big project that just wrapped up this past Friday after three weeks of preempting everything else I was working on.

I’m finally starting to be able to relax; I was wound up, stressed, and tense for the past three months and went from “happy go lucky guy” to “grouchy jerk”. Those people that know me well realize that it takes a LOT for me to start getting visibly angry, and I was so stressed that I was starting to snap and yell at my house pets.

Back to work..

I’ve been on vacation since Thanksgiving, but am really in the mood to return to work on Monday. I can’t stand being off work for more than a week or so – I tend to get bored. Bad habits of a work-a-holic I guess.

I’ll also be taking every Friday between now and the end of the year off, since my employer gave me an extra week of vacation (now four weeks total per year) this year as it was my fifth anniversary with them. I didn’t find out about the extra week until last month, so now I’ve got to use it up by 12/31 or I’ll lose it.

That was NOT FUN

Early Sunday afternoon, there was apparently a power and/or UPS failure in the datacenter at my office. By 5pm, most of the systems were back up except for one that I’m responsible for.

Fire up the VPN, hit the ILO remote console (one of the few things that makes using x86 systems as servers bearable) and see “Cannot find boot.bin” Uh-oh. I head to the office at 6pm.

It seems that a certain Sun patch turns non-GRUB systems into non-bootable systems after application. There’s a specific set of steps to follow when this patch is installed and a system is “upgraded” to the GRUB bootloader, but apparently Sun’s “smpatch” utility does not follow these steps. The patch had been applied months ago, but the system didn’t get rebooted until the power outage.

I figured “Okay, the system was running Solaris 10 FCS, so its time to do an upgrade install of S10u4 anyway”. After some other problems and workarounds, four hours later, I watch in resignation as the install hangs and locks up (not accepting keyboard input at a Y/N prompt) while trying to install the CPQary package. This package is the drivers that Solaris x86 needs in order to use the hardware RAID built into the Compaq DL360.

I bite the bullet and do a “nuke from orbit” fresh install of S10u4, planning to restore from backups. I ended up having to rebuild most of the services on the box by hand (which was better in the long run, as things needed cleaning up) as our backup system had also been affected by the power outage and it wasn’t available until Monday morning.

To make a long story short, I went to the office at 6pm Sunday, and finally walked out of my office to go home and get some sleep at 10:30am Monday. 16 hours is the longest single shift I’ve ever pulled anywhere, and certainly the longest after-hours session.

I’ve got one more service to restore onto the box on Tuesday, but it’s non-critical and could wait until I got some rest.

I can’t complain – I might have incidents like these once or twice a year, and it’s a lot better than getting called or paged every other day like I was used to at my last job. I really like my job and my managers and coworkers.

Another day, another dollar

My boss walked in this morning and handed me a letter detailing this year’s raise. I’m certainly not complaining!

In addition, someone from another group offered an open position with their team to me this morning, but I had to turn them down as I’m really happy where I’m at right now. I’ll still help them with their particular area of work, but I much prefer being a “general purpose” sysadmin versus concentrating on one single aspect of the services we provide.

Programming projects..

I’m hacking on a system at work to automatically generate network accounts based on employee requests. So far I’ve gotten it authenticating against Active Directory (using adLDAP), and have rebuilt PHP with SQLite support.

Tomorrow I build the majority of the guts of the application and a basic proof-of-concept user interface. Wednesday I’ll automate the actual account creation, by either firing off an Expect script or by using the built-in Expect functions in PHP.

After all that works, I’ll throw the pieces together into a first revision of the production application and put it behind a secure webserver (so passwords don’t get passed in cleartext).