Category: User Acceptance Testing
Too Busy for User Acceptance Testing (UAT)?
I Love UAT
One of my favorite parts of testing is getting the opportunity to coordinate UAT. I love it because we’re getting feedback from the people who are actually going to use the system once it goes live. Even if a team is great about getting user input, everything up to this point is theoretical. All of us can think we know what we want, but once we actually see it, touch it, and live with it, we don’t know if what we asked for is what we really wanted or needed.
I’m planning to write a few articles about what I’ve learned during my UAT experiences, and I hope it will help others with their efforts.
Who Should Perform UAT?
The obvious answer to who should perform UAT is “users”. What users should perform UAT, though? Like many things, “it depends.” For a public-facing application, you may want people from the general public who don’t have a lot of deep knowledge about the system or industry. For an internal application, you likely want experienced users. However, those experienced users will often, understandably, plead that they are “too busy” to help test.
At one of my past clients, I requested the time of experienced users to test a back-end system for processing college applications. Like most companies, they were a bit understaffed and perpetually a bit behind at getting the work done. To the team, and to the manager who was trying to guard their time, my request was just one more bit of unnecessary busywork added to their plate.
The manager offered to hire a temporary worker to perform UAT so their most experienced workers wouldn’t be pulled away from their important work.
The Users Will Live with the Results
Given what I’ve seen others do with UAT in the past, this team may have believed that UAT was simply going to be formality where they would run tests that I had already verified met the written requirements. UAT is sometimes simply a step done to mark a checkbox in the few days before going live. However, when I run a UAT effort, it is much more than that.
I knew that having a temp worker wouldn’t give our team the insight we needed. A temp worker could certainly execute a pre-written test script, but that is not a useful way to perform UAT in my experience. Our team needed to know if this software truly met the needs of the team. We genuinely wanted to know, and we planned to incorporate the feedback into the final product.
I had a heart-to-heart with the manager and the team lead. I explained to them that their team would be using this application to do all of their work in the not-too-distant future. The team, in order to make the most of their investment, would likely be using the software for years. Shortly after the software went live, I’d be gone, but their team would be living with the results.
I explained that we truly were going to take their input into account to ensure the best software for doing their work. Then I asked if they still wanted a temp to be responsible for giving that input and making those decisions that would affect their team for years.
Once they understood the true impact of UAT, the answer was obvious. The manager and lead both made time for UAT, and they even had another employee participate. They brought on a temporary worker to handle some of the more routine clerical tasks instead of having them perform UAT.
We did incorporate their feedback into the final product, and the client was very happy with the end product. It was a true improvement to their old system.
If you could use some guidance with user acceptance testing, please reach out to me at [email protected]
User Acceptance Testing Tales from the Trek – Writing a Perfect Bug
My early experiences running User Acceptance Testing (UAT) typically involved dealing with just 1 or 2 users. When they encountered an error, they’d show it to me, and I’d record it. However, as I gained experience in Quality Assurance, I worked on larger projects.
Training – How to write a bug
On my first larger project, I needed to train a group of 20 users who were going to perform User Acceptance Testing. Having worked with several different users in the past, I knew several of the pitfalls that users may encounter. However, I never really had asked users to record errors themselves. Still, I thought it would be easy to get the point across.
During the training session, I said that it was critical to record all necessary steps so that the developer could recreate the bug. I told the users that we often hear from users who simply say, “It doesn’t work,” and explained that the developers will not be able to fix a problem that was described so generically. The users said that they understood perfectly and were ready to start testing.
Bug Writing – Frustration
With 20 inexperienced testers working independently, it was important to filter the reported bugs before assigning them to the developers. I was responsible for triaging all bugs reported by the users. I verified that I could recreate bugs and eliminated duplicate bugs before assigning them to the developers. If I could not recreate a bug based on the instructions, I would reassign it to the user who reported it and ask for more detail. I assumed that the users would quickly realize when they were not providing sufficient detail and start writing perfect bugs after a day or two.
I assumed wrong.
Although a few users were writing clear, reproducible bugs, most were writing ones that were way too generic. For example:
- I was on the “Choose a Course” screen.
- When I chose a course, I got an error.
In this case, when I tried to reproduce the bug, I did not receive an error. So, I sent the bug back to the user requesting more detail. In most cases, the first time a bug was recorded, I had to reject it. In many cases, I had to reject the rewrite as well. Pretty quickly, the users got fed up with me and complained to their manager that I was rejecting their bugs.
Enlightenment
I realized that I needed to find a way to illustrate to the users exactly what was needed in a clear, reproducible bug report. I developed the following simple exercise that was very effective. I have used it on all subsequent large-scale UAT efforts with the same success.
I ask the users to do the following with their first 5 bug reports:
- Record the bug as clearly as you can and assign it to yourself.
- Wait 5 hours.
- Try to recreate the bug following only the information that you recorded.
- If you are able to recreate the bug, assign it to me. Otherwise, go back to step 1.
It is important that the users wait some amount of time before trying to recreate the bug. I found that when they try to recreate the bugs immediately, the steps are too fresh in their mind to truly follow only what they wrote. They seem to fill in the gaps without realizing it.
I have found that once users complete this exercise, they are able to consistently write bugs that have enough detail to be reproducible. They do not become expert testers, of course, but they are able to write a useful bug.
I’m interested in hearing about other people’s experiences as well.
What concepts have you found especially difficult to get across to people performing UAT for the first time?
Were you able to find a technique that helped the users better understand the concept?
User Acceptance Testing – Planning for Value
Throughout the years, I’ve led many user acceptance testing (UAT) efforts. There are a couple of common ways to plan the testing that I’ve found to be problematic. The methods allow someone to say they’ve planned UAT but add very little value. The end result is that unhappy users report problems after the software is live, and the project manager asks why they didn’t find the problems during UAT.
After years of experimentation, I found a method that adds a lot of value to the testing process with less effort than the other methods.
Method 1 – Users write UAT scripts
I’ve been involved with quite a few projects where the client wanted the users of the software to write the test scripts for UAT. This was often done to reduce development costs. The folks planning the project believed that they would save money by having business employees write the test scripts for UAT rather than have someone from the development team work on them. They also believed, often correctly, that the business folks understood the goals of the system best, so they were the most qualified to write the UAT scripts.
The problem with this is that most business users have more than a full time job already. In addition, most business users have never planned a testing effort or have written a test script. It’s difficult enough for business users to find time to execute tests, but nearly impossible for them to find time to plan testing. Even if they do find the time to plan it, their lack of experience results in test scripts that test very little.
Method 2 – Test Analyst writes UAT scripts
On other projects, I’ve been asked to write test scripts that the users can run during UAT. Usually, I’m asked for the same scripts that the test team ran during system testing. Under this scenario, it’s unlikely that the users will find any issues that the test team didn’t already find. This method doesn’t uncover functional issues that the test team missed. In addition, I’ve found that the business users rarely take the time to follow the test script closely and end up not doing much testing at all.
UAT Success Method – Guided Functional Checklists
The UAT planning method that I’ve found adds the most value with the least effort is Guided Functional Checklists. This method is quite simple:
- A member of the test team meets with each key user and asks what business goals they need to accomplish with the software.
- The test team member compiles the list into a spreadsheet that will be used for recording the test results.
That’s all there is to the planning. The result is an organized list of what should be tested based on the business goals of the users rather than based on the development team’s view of the project. Because this takes little of the users’ time, they are able to make time to complete the planning. By not having a detailed script, users are able to ensure they can meet their business goals in the way that they will in the field, which is often a bit different than the development team expects. This allows everyone to identify programming or training issues before going live.
Feedback
What UAT planning challenges have you faced? How did you address the challenges? I’d love to hear your tales from the trek.
The User Acceptance Testing Death Spiral
In a past role, I joined a test team that was in a User Acceptance Testing (UAT) “Death Spiral” that had caused the user base to lose confidence in the integration testing team. Based on conversations that I’ve had with others, I believe that the UAT Death Spiral is a common scenario that people encounter, and it can destroy a test team. It took some work, but we were able to pull out of the downward trajectory and regain a functional, productive partnership between the business folks and the integration testing team.
What is the User Acceptance Testing Death Spiral?
Our company had an understaffed integration testing team, aggressive deadlines, and a culture that valued meeting deadlines above all other goals. This meant that software often had known major bugs when UAT started. Even worse, there were typically areas of code that hadn’t been tested at all by the integration testing team before UAT started. It was common for the UAT team to find major, obvious bugs.
Because major bugs made it past the integration test team, the business felt a need to create a more formal, robust UAT team that would catch the numerous errors missed by the integration test team. The business folks assumed that the bugs were missed because the integration team wasn’t very good at their job.
As UAT became more robust, they realized that they needed more time to complete their testing. The company culture of tight deadlines meant that release dates could not be extended to accommodate in-depth UAT. Instead, the business insisted that the integration test cycle be shortened and that UAT start earlier.
I think you can see where this is going…
When UAT started earlier, they found even more bugs and the business lost even more confidence in the integration test team. The business then insisted on testing more of the functionality in UAT and starting UAT even earlier. This death spiral continued until the business had lost complete confidence in the integration test team. However, because the UAT team was not made up of experienced testers, the business was not finding all of the bugs either.
Basically, the integration team didn’t have time to do their job, the business was spending a huge amount of resources to test everything, even features already tested by the integration test team, and the quality of software in production was as poor as ever.
Breaking Free
Breaking free of this downward spiral was more than just a logistical problem. It was a political problem as well. The integration test team feared that the UAT team was trying to take over their jobs and the UAT team felt that the integration team wasn’t competent. Rebuilding trust was critical for any new process to be successful.
I worked with both the integration and UAT test teams to plan a new strategy. The strategy was that the integration test team would first test anything that did not have a user interface. In addition, the integration test team would write, maintain, and run automated regression tests. Basically, they would test the areas that required their expertise. Only after these areas were tested and any major bugs were fixed would the UAT team start their work. We would divide up the test cases to reduce any overlap of testing between the two teams as much as possible.
Even though the UAT team agreed that this plan made sense theoretically, they feared that removing the redundant testing would mean that bugs were missed and worried that starting UAT later would mean that they wouldn’t have time to complete their work. I convinced them to give the plan a try on a smaller project. If the advantages to the new plan didn’t materialize, it would be easier to adjust for the lost time on a small project.
Fortunately, on the small project, everything fell into place. The integration test team was able to adequately test their portion of the plan before UAT started, and the UAT team knew exactly what parts were and weren’t tested by the integration team. The UAT team had a shorter test cycle both because they didn’t run as many redundant tests and because the initial quality of code was much better. Each bug takes time to find, fix, and retest.
Best of all, the software went live and had no problems in production.
For the rest of my time with this team, we followed the new process. This resulted in higher quality code with lower cost, and it had the added benefit of greatly improving the working relationship between the two teams.
Your Experiences
I’d like to hear your experiences with User Acceptance Testing issues. Have you been in a situation where the business users lost confidence in the development team or integration test team? If so, what do you think were the root causes of the issues?