Three Simple Steps to Avert Deployment Fiasco

A couple of weeks ago, I had to deal with the frustrated members of my payroll team. They were unable to process payroll as they could not log into our outsourced payroll system. The vendor’s customer service lines were jammed and it took a couple of hours before our team could speak to their rep.

Later our representative filled us in on the details of the day’s mess.

On that day, HR operations of numerous corporations who use their service were impacted when the payroll company deployed a new version of the software. The authentication and security features of the software were upgraded which resulted in all users going through activation of their accounts once again.

Apparently, the deployment happened at midnight and at day break when users started to log in, their email system could not handle the load to send new activation codes. Naturally, this resulted in customer service phone lines getting jammed. Consumers were getting their activation codes several hours after they had requested the code due to email backlog. By then, they had made multiple requests and the code that was received from an earlier request was no longer valid.

Deployment is like the finishing lap of a race. A project that is ON track can go OFF track towards the very end if the deployment phase is not planned and executed properly.

With my project management and technology cap on, I started to think how I would have handled this deployment differently if I was running this project. I came up with three fundamental steps that the payroll firm failed to take before their deployment which could have averted this deployment fiasco for them.

1. Plan and Conduct “Mock” Go-Live

Mock Go-Live is like a “dress rehearsal” for the deployment event prior to go live to test the implementation plan and identify gaps in the plan. Mock Go-Lives are sometimes used to simulate the change in processes and to have the first team perform multiple cycle counts so that they may train others in the organization.

For large deployments, multiple mock go lives should be planned to iteratively identify the gaps, make adjustments, perform additional QA and educate teams.

2. Perform Load Testing

Listing the impacted parts of the application and clearly identifying the effect this deployment will have on the user’s workflow will highlight the areas load testing should be focused on. In this project, where the upgrade focused on authentication and security features, the login and activation functionality will be affected. So the primary focus of load testing should have been these two features.

3. Proactively Communicate with all the Stakeholders

Proper timely communication to all the stakeholders is critical before the deployment. Give your stakeholders (users, sponsors, customer support, help desk, call center) enough heads up by informing them about the deployment timeline and service interruptions that might occur. By sending this communication ahead of time, your stakeholders will have the ability to plan their activities around your deployment schedule.

The payroll processing firm could have avoided the egg on their face if they followed established best practices in large systems rollout that would always include these three steps I outlined. Hundreds of thousands of corporate customers could not process payroll in a timely fashion due to this failure.

I quickly wanted to share what was going on in my mind based on all of our large systems experience. Have any horror stories like these? Please share them – so we can all learn from them.

Keep Reading: Assuring Development Quality: 3 Types of QA Teams