Windows of Change

By a virtual show of hands, how many of you open change records for changes that occur between the hours of 23:00 and 06:00? That's almost all of you. No, I didn't actually have to see you to know how many of you either raised your hands virtually or physically. Don't forget that I live this life of trips to the data center, change windows, change records, service requests, backout plans, and root cause analysis. I've lived it for a very long time – so long in fact, I find myself asking a good question: Why don't we perform changes when we're actually awake and when everyone who can help, if things inevitably go wrong, is awake, too?

Spoiler alert: I am under no delusion that I will change anyone's behavior by stating my observations and opinions here, but at least you'll have considered them as an alternative to the long-established behaviors.

After having already spent at least eight hours on the job, I often have to plan for a change that begins at or near midnight. For mission-critical business applications, databases, and storage, this doesn't appear to be the wisest decision. Wouldn't it make more sense to perform these changes during the day, when you have a fresh and awake staff to do the work? It makes sense to me, especially when data centers are built with redundancy in mind. Isn't that exactly what the purpose of redundancy is?

Supposedly, every system has a failover option – at least from A to B. Routers, switches, storage arrays, power supplies, services, databases, and applications all have redundancy that protects the business from failure should a system fail. Systems fail during the day, too, although it seems like they fail more often in the wee hours of the morning or over a long holiday weekend than they do during the day. We fix them during the day, too, don't we?

I once presented this case to a project manager and an account executive who couldn't argue with the logic, but the answer was still the same from both: "We're not going to do that." OK, thanks for that elaborate explanation. The problem is that we're not seeing clearly enough to realize that tasking people with highly technical changes late at night, after working a full day, is just not a great idea.

If an emergency security patch came over the wire, the patch would be applied without hesitation and during the day, so why can't it be that way all the time?

Personally, I think it's a fear of things going too well. Certain people can't justify their jobs if they don't stay up all night directing technical folk in their collective journey toward a successful change. Frankly, we just don't need that kind of help. It is far better to conduct changes during the day, so when something goes bump in the night, everyone can jump on the outage call and take care of it with a fresher mind, rather than working exhausted from an all-nighter two day ago.

I've never heard a good reason not to perform changes during the day. I've heard lots of reasons, just none of them good – or not good enough to convince me that working people around the clock is good for them or business-critical services. As I said, I don't expect my logic to change a time-honored tradition of performing changes at night, but I can dream, can't I?

I think part of the problem is that people generally fear change. They fear change to the extent that they want something to change when they're asleep rather than facing it head-on during the light of day. Yes, if you gathered that data center work is a metaphor for life, you have read well. Facing changes with fresh minds, plenty of light, and ample support is good for the soul. There's no reason to do everything in the dark and at the point of exhaustion. Because you're probably reading this during a lull in a change window, put it down, pay attention to what's going on, and read this again later when you have a better attitude and more sleep. Things are much clearer in the light of day.

Ken Hess * ADMIN Senior Editor