A couple of weeks ago, I had to deal with the frustrated members of my payroll team. They were unable to process payroll as they could not log into our outsourced payroll system. The vendor’s customer service lines were jammed and it took a couple of hours before our team could speak to their rep.
Later our representative filled us in on the details of the day’s mess.
On that day, HR operations of numerous corporations who use their service were impacted when the payroll company deployed a new version of the software. The authentication and security features of the software were upgraded which resulted in all users going through activation of their accounts once again.
Apparently, the deployment happened at midnight and at day break when users started to log in, their email system could not handle the load to send new activation codes. Naturally, this resulted in customer service phone lines getting jammed. Consumers were getting their activation codes several hours after they had requested the code due to email backlog. By then, they had made multiple requests and the code that was received from an earlier request was no longer valid.
Deployment is like the finishing lap of a race. A project that is ON track can go OFF track towards the very end if the deployment phase is not planned and executed properly.
With my project management and technology cap on, I started to think how I would have handled this deployment differently if I was running this project. I came up with three fundamental steps that the payroll firm failed to take before their deployment which could have averted this deployment fiasco for them.
1. Plan and Conduct “Mock” Go-Live
Mock Go-Live is like a “dress rehearsal” for the deployment event prior to go live to test the implementation plan and identify gaps in the plan. Mock Go-Lives are sometimes used to simulate the change in processes and to have the first team perform multiple cycle counts so that they may train others in the organization.
For large deployments, multiple mock go lives should be planned to iteratively identify the gaps, make adjustments, perform additional QA and educate teams.