In a past role, I joined a test team that was in a User Acceptance Testing (UAT) “Death Spiral” that had caused the user base to lose confidence in the integration testing team. Based on conversations that I’ve had with others, I believe that the UAT Death Spiral is a common scenario that people encounter, and it can destroy a test team. It took some work, but we were able to pull out of the downward trajectory and regain a functional, productive partnership between the business folks and the integration testing team.
Our company had an understaffed integration testing team, aggressive deadlines, and a culture that valued meeting deadlines above all other goals. This meant that software often had known major bugs when UAT started. Even worse, there were typically areas of code that hadn’t been tested at all by the integration testing team before UAT started. It was common for the UAT team to find major, obvious bugs.
Because major bugs made it past the integration test team, the business felt a need to create a more formal, robust UAT team that would catch the numerous errors missed by the integration test team. The business folks assumed that the bugs were missed because the integration team wasn’t very good at their job.
As UAT became more robust, they realized that they needed more time to complete their testing. The company culture of tight deadlines meant that release dates could not be extended to accommodate in-depth UAT. Instead, the business insisted that the integration test cycle be shortened and that UAT start earlier.
I think you can see where this is going…
When UAT started earlier, they found even more bugs and the business lost even more confidence in the integration test team. The business then insisted on testing more of the functionality in UAT and starting UAT even earlier. This death spiral continued until the business had lost complete confidence in the integration test team. However, because the UAT team was not made up of experienced testers, the business was not finding all of the bugs either.
Basically, the integration team didn’t have time to do their job, the business was spending a huge amount of resources to test everything, even features already tested by the integration test team, and the quality of software in production was as poor as ever.
Breaking free of this downward spiral was more than just a logistical problem. It was a political problem as well. The integration test team feared that the UAT team was trying to take over their jobs and the UAT team felt that the integration team wasn’t competent. Rebuilding trust was critical for any new process to be successful.
I worked with both the integration and UAT test teams to plan a new strategy. The strategy was that the integration test team would first test anything that did not have a user interface. In addition, the integration test team would write, maintain, and run automated regression tests. Basically, they would test the areas that required their expertise. Only after these areas were tested and any major bugs were fixed would the UAT team start their work. We would divide up the test cases to reduce any overlap of testing between the two teams as much as possible.
Even though the UAT team agreed that this plan made sense theoretically, they feared that removing the redundant testing would mean that bugs were missed and worried that starting UAT later would mean that they wouldn’t have time to complete their work. I convinced them to give the plan a try on a smaller project. If the advantages to the new plan didn’t materialize, it would be easier to adjust for the lost time on a small project.
Fortunately, on the small project, everything fell into place. The integration test team was able to adequately test their portion of the plan before UAT started, and the UAT team knew exactly what parts were and weren’t tested by the integration team. The UAT team had a shorter test cycle both because they didn’t run as many redundant tests and because the initial quality of code was much better. Each bug takes time to find, fix, and retest.
Best of all, the software went live and had no problems in production.
For the rest of my time with this team, we followed the new process. This resulted in higher quality code with lower cost, and it had the added benefit of greatly improving the working relationship between the two teams.
I’d like to hear your experiences with User Acceptance Testing issues. Have you been in a situation where the business users lost confidence in the development team or integration test team? If so, what do you think were the root causes of the issues?
A key aspect of building quality software is ensuring that it does what the users need it to do. In my experience, the backlog of feature request (whether written or held in the stakeholders’ heads) is always much larger than what the development team can build in a short period of time. However, prioritizing these features seems to be difficult for people. Everything seems important, so everything gets priority 1.
Of course, if everything has the same priority, the stakeholders are de facto allowing the development team to prioritize the features. This can be a problem because the development team often doesn’t have the visibility to all of the factors that may determine the importance of the feature to the company’s success.
Here’s one example of a client who had an issue with prioritization, and how we arrived at a working solution.
A client had a new system replacing an existing business critical system. Unfortunately, their existing system had reached its technical limits before the new system was fully tested, and management made the decision to go live without much testing. Of course, the results were predictable. There were many errors in a production system that had to be fixed right away.
The “prioritization” method initially was that end users would come into the room of developers and tell them that they needed to drop everything and work on whatever issue the end user mentioned. The problem was that many different users were coming in each hour, and the developers didn’t get a chance to finish any task before being told to drop it and work on something else.
The first thing we did was set up a SharePoint list where users could report their issues. We created a process where the users would report their issues in SharePoint. Then, I would triage the list and assign the work to developers. This simple improvement resulted in a huge increase in productivity for the development team because they could complete tasks without interruption.
However, we weren’t always working on the most important issues. Users were choosing the priority, and every issue was the highest priority to that user. Even when we met with representatives from all departments together and set definitions for priorities, every issue was priority 1 on a 3-priority scale.
Our original 3 priority levels were called “High”, “Medium”, and “Low”. Because all issues were production issues, people didn’t want to minimize any by calling them “Medium” or “Low”. Everyone agreed that the issues were not all the same priority, but they weren’t willing to prioritize using those names.
First, the client came up with a category called “Priority 1 – Urgent”. This was higher than “Priority 1” and the client felt comfortable putting some items in Priority 1, and some in Priority 1 – Urgent. Still, way too many items were in Priority 1 – Urgent, so the development team was still choosing the priority.
Then, the client decided that the most critical items would be in a new priority called “Priority 0”. This was reserved for the top 5-7 issues to be worked on immediately by our 5-person development team.
This worked! The client was completely willing to prioritize into “Priority 0”, “Priority 1 – Urgent”, and “Priority 1” even though they were not willing to prioritize into “Priority 1”, “Priority 2”, and “Priority 3”. Just by changing the names of the priority levels, we were able to accomplish the goal of dividing the issues into 3 different levels.
We could then focus on the issues that brought the most value to the system.
What challenges have you had with prioritizing features? What actions have you tried to address the challenges? I’d love to hear your stories from the trek.