Author: Kevin Rutkowski
Tester Traits – Comfort with Cognitive Dissonance
I believe that software testers need to be comfortable with two conflicting ideas in order to best contribute to the success of a project. They need to be able to find and point out every defect in a given software application; however, they also need to keep in mind that the software can meet the business’ needs even if it is not perfect. The best software testers that I’ve worked with are comfortable with the cognitive dissonance caused by these two conflicting ideas.
Goal of Perfection
I’ve worked with some software testers who are great at finding defects, but they get so focused on the defects that they lose sight of the big picture. They seem to believe that the ultimate purpose of software is to be perfect rather than to meet a business goal. These software testers can cause a software project to be unnecessarily delayed by arguing that minor issues should prevent software from going live. They can also reduce end-user acceptance of a product by complaining to anyone who will listen that the software is subpar and by pointing out every known issue to end-users.
I believe that testers do need to point out every bug possible to the internal development team and ensure that the team knows the business implications of any bugs. However, good testers need to be comfortable with the fact that perfection is not ultimate goal for business software. Instead, software has some business purpose that does not require perfection. Once software goes live, I believe that a tester should champion the software to the end-users.
When I encounter a tester who has trouble accepting imperfect software, I remind them that most useful commercial software has known bugs. Often, some of these bugs are even listed in a “Read Me” file for end users to see. There is a real value to having less than perfect software available rather than having no software at all.
Goal of Delivery
Although I have seen testers who are too focused on perfection, I have also seen testers who are too focused on delivery of the software. These testers either are not detail-oriented enough to find critical bugs or they ignore the real business impact of bugs. These testers do not want to stand in the way of the delivery date under any circumstances. Project Managers often put enormous pressure on testers to sign-off on software that was not properly tested or has major bugs in order to meet a delivery deadline. However, the best testers will have the backbone to stand up to any pressures and point out the specific risks of going live.
In my opinion, it’s important that the tester understand the business goals well enough to point out the specific risks to key decision makers; however, I believe that it should not be up to the tester to decide whether or not software goes live. The ultimate decision should be made by management who understands the full scope of business needs and can weigh the quality risks against every other concern.
The Ponds
There’s a poem by Mary Oliver called The Ponds that I think captures the essence of this trait that software testers need to contribute to a successful project. Testers need to notice imperfections while still being “willing to be dazzled”. This version of the poem is from a book called New and Selected Poems by Mary Oliver published by Beacon Press in 1992. I hope you enjoy it and find it relevant.
Every year
the lilies
are so perfect
I can hardly believe
their lapped light crowding
the black,
mid-summer ponds.
Nobody could count all of them –
the muskrats swimming
among the pads and the grasses
can reach out
their muscular arms and touch
only so many, they are that
rife and wild.
But what in this world
is perfect?
I bend closer and see
how this one is clearly lopsided –
and that on wears an orange blight –
and this one is a glossy cheek
half nibbled away –
and that one is a slumped purse
full of its own
unstoppable decay.
Still, what I want in my life
is to be willing
to be dazzled –
to cast aside the weight of facts
and maybe even
to float a little
above this difficult world
I want to believe I am looking
into the white fire of a great mystery.
I want to believe that the imperfections are nothing –
that the light is everything – that it is more than the sum
of each flawed blossom rising and fading. And I do.
User Acceptance Testing Tales from the Trek – Writing a Perfect Bug
My early experiences running User Acceptance Testing (UAT) typically involved dealing with just 1 or 2 users. When they encountered an error, they’d show it to me, and I’d record it. However, as I gained experience in Quality Assurance, I worked on larger projects.
Training – How to write a bug
On my first larger project, I needed to train a group of 20 users who were going to perform User Acceptance Testing. Having worked with several different users in the past, I knew several of the pitfalls that users may encounter. However, I never really had asked users to record errors themselves. Still, I thought it would be easy to get the point across.
During the training session, I said that it was critical to record all necessary steps so that the developer could recreate the bug. I told the users that we often hear from users who simply say, “It doesn’t work,” and explained that the developers will not be able to fix a problem that was described so generically. The users said that they understood perfectly and were ready to start testing.
Bug Writing – Frustration
With 20 inexperienced testers working independently, it was important to filter the reported bugs before assigning them to the developers. I was responsible for triaging all bugs reported by the users. I verified that I could recreate bugs and eliminated duplicate bugs before assigning them to the developers. If I could not recreate a bug based on the instructions, I would reassign it to the user who reported it and ask for more detail. I assumed that the users would quickly realize when they were not providing sufficient detail and start writing perfect bugs after a day or two.
I assumed wrong.
Although a few users were writing clear, reproducible bugs, most were writing ones that were way too generic. For example:
- I was on the “Choose a Course” screen.
- When I chose a course, I got an error.
In this case, when I tried to reproduce the bug, I did not receive an error. So, I sent the bug back to the user requesting more detail. In most cases, the first time a bug was recorded, I had to reject it. In many cases, I had to reject the rewrite as well. Pretty quickly, the users got fed up with me and complained to their manager that I was rejecting their bugs.
Enlightenment
I realized that I needed to find a way to illustrate to the users exactly what was needed in a clear, reproducible bug report. I developed the following simple exercise that was very effective. I have used it on all subsequent large-scale UAT efforts with the same success.
I ask the users to do the following with their first 5 bug reports:
- Record the bug as clearly as you can and assign it to yourself.
- Wait 5 hours.
- Try to recreate the bug following only the information that you recorded.
- If you are able to recreate the bug, assign it to me. Otherwise, go back to step 1.
It is important that the users wait some amount of time before trying to recreate the bug. I found that when they try to recreate the bugs immediately, the steps are too fresh in their mind to truly follow only what they wrote. They seem to fill in the gaps without realizing it.
I have found that once users complete this exercise, they are able to consistently write bugs that have enough detail to be reproducible. They do not become expert testers, of course, but they are able to write a useful bug.
I’m interested in hearing about other people’s experiences as well.
What concepts have you found especially difficult to get across to people performing UAT for the first time?
Were you able to find a technique that helped the users better understand the concept?
User Acceptance Testing – Planning for Value
Throughout the years, I’ve led many user acceptance testing (UAT) efforts. There are a couple of common ways to plan the testing that I’ve found to be problematic. The methods allow someone to say they’ve planned UAT but add very little value. The end result is that unhappy users report problems after the software is live, and the project manager asks why they didn’t find the problems during UAT.
After years of experimentation, I found a method that adds a lot of value to the testing process with less effort than the other methods.
Method 1 – Users write UAT scripts
I’ve been involved with quite a few projects where the client wanted the users of the software to write the test scripts for UAT. This was often done to reduce development costs. The folks planning the project believed that they would save money by having business employees write the test scripts for UAT rather than have someone from the development team work on them. They also believed, often correctly, that the business folks understood the goals of the system best, so they were the most qualified to write the UAT scripts.
The problem with this is that most business users have more than a full time job already. In addition, most business users have never planned a testing effort or have written a test script. It’s difficult enough for business users to find time to execute tests, but nearly impossible for them to find time to plan testing. Even if they do find the time to plan it, their lack of experience results in test scripts that test very little.
Method 2 – Test Analyst writes UAT scripts
On other projects, I’ve been asked to write test scripts that the users can run during UAT. Usually, I’m asked for the same scripts that the test team ran during system testing. Under this scenario, it’s unlikely that the users will find any issues that the test team didn’t already find. This method doesn’t uncover functional issues that the test team missed. In addition, I’ve found that the business users rarely take the time to follow the test script closely and end up not doing much testing at all.
UAT Success Method – Guided Functional Checklists
The UAT planning method that I’ve found adds the most value with the least effort is Guided Functional Checklists. This method is quite simple:
- A member of the test team meets with each key user and asks what business goals they need to accomplish with the software.
- The test team member compiles the list into a spreadsheet that will be used for recording the test results.
That’s all there is to the planning. The result is an organized list of what should be tested based on the business goals of the users rather than based on the development team’s view of the project. Because this takes little of the users’ time, they are able to make time to complete the planning. By not having a detailed script, users are able to ensure they can meet their business goals in the way that they will in the field, which is often a bit different than the development team expects. This allows everyone to identify programming or training issues before going live.
Feedback
What UAT planning challenges have you faced? How did you address the challenges? I’d love to hear your tales from the trek.
Metric Misuse – Quality Assurance Metrics Gone Awry
I was reading through some posts from Bob Sutton, one of my favorite management gurus, and I ran across a post that contains one of my favorite Dilbert comic strips.
Bob Sutton’s post, as well as the comments that I made on his blog, reminded me of one of my favorite topics: misused Quality Assurance metrics.
Tying Quality Assurance Metrics to Financial Rewards – A Dangerous Game
“Treat monetary rewards like explosives, because they will have a powerful impact whether you intend it or not.” –Mary and Tom Poppendieck, authors of Implementing Lean Software Development: From Concept to Cash
Over the years, many people have asked me what Quality Assurance metrics they should use to evaluate employee performance. My advice is that Quality Assurance metrics should not be used directly to evaluate employee performance. The Dilbert comic strip may seem a bit extreme, but it’s exactly what happens when employee performance is based strictly on metrics. This is true regardless of whether monetary rewards are explicitly tied to the metrics or not.
In my comments on Bob Sutton’s blog, I mentioned three specific metrics that had unintended effects when used for evaluating employee performance:
- rewarding testers for the number of test cases they wrote resulted in poorly written test cases;
- rewarding testers for the number of bugs they found resulted in a high number of unimportant or duplicate bugs reported; and
- penalizing testers for bugs rejected by the test lead or development staff resulted in important bugs going unreported.
Many people think that they have the ability to write a set of metrics that can be used to unequivocally gauge the performance of a Quality Assurance professional, but I have not yet encountered a metric that couldn’t be manipulated to favor the employees.
(If the metric can’t be gamed, it probably isn’t under the control of the employees, so it wouldn’t be effective at driving behavior anyhow.)
Are Metrics Worthless Then?
Actually, metrics are a great tool for identifying coaching opportunities and potential problems. However, in order to get honest metrics, they shouldn’t be used directly for employee evaluations or employee rewards.
When I’ve looked at the metrics that I mentioned earlier with an eye towards coaching, I had excellent results.
- Reviewing the number of test cases written helped me identify a tester on my team who was putting much more detail than I wanted into his test cases. After some coaching, he was able to consistently meet my expectations.
- Reviewing the number of bugs found by each tester helped me identify a tester who was digging into the root cause of the most difficult to reproduce bugs. She didn’t report as many bugs as others, but her work was critical to getting a great product out the door in a timely manner. It turned out that she was the most skilled tester even though she reported the fewest bugs.
- Reviewing the number of bugs rejected by the development staff helped me identify a manager who was evaluating his programmers based solely on the number of valid bugs found in their code. The developers were motivated to simply mark bugs as invalid rather than fix the bugs. This insight allowed me to address the problem directly with that manager.
Good Quality Assurance metrics provide powerful tools for managing a Quality Assurance team when used properly. However, they shouldn’t be used in a vacuum. They should just be considered one data point among many.
I was only able to scratch the surface of this topic in this blog post. I plan to discuss specific metrics in future blog posts. In the meantime, if you want to read a much more in-depth review of the pros and cons of employee incentives, you can find one paper here.
Your Experiences
I know that a lot of people feel passionately about Quality Assurance metrics, both pro and con. I’m very interested to hear about your experiences with Quality Assurance metrics. Have you found any that were particularly useful? Have you found any that had unintended consequences?
Communication – Tales from the Trek – Perception is Reality
Based on my conversations, I believe that most people perceive that they do more work than others. It seems common that all members of a household think they do more than their fair share of work, and most members of teams at work feel the same way. I believe that a big driver in this perception is that people know exactly what work they’ve done, but they are not aware of the work that others have done.
IT is Lazy!
At one place where I worked, the business side of the department often expressed to me that IT was lazy. They said that IT was never getting to any of their requests, and they weren’t sure what IT was doing all day. Not surprisingly, the IT department expressed that they were overworked and accomplishing a lot. Why was there this extreme difference in perception?
The IT department met with the business leadership each year to determine the priorities for the year based on the hours budgeted. IT was excellent at delivering on the priorities; however, the business had many more requests for changes and improvements that were above and beyond the priorities for the year. So, most individuals on the business side didn’t have their personal priorities addressed even though the overall strategic priorities for IT were accomplished.
Changing Perceptions
A director of the IT department came up with an idea that was very simple but incredibly effective. He created a giant checklist of the scheduled IT projects and posted them in a few prominent places around the office. As each project was completed, he put a checkmark next to the item.
The comments that I heard from the business completely changed. Although their personal priorities may not have been addressed, they could see the importance of the priorities posted on the checklist. In addition, they could see clearly the progress that had been made. The checklist had greatly improved the business’s perception of IT.
Feedback
What perception challenges have you faced? How did you address the challenges? I’d love to hear your tales from the trek.
Value of QA – People Do Not Find Their Own Errors
I’ve noticed, through the years, that people often dismiss the need for Quality Assurance by saying, “These are good developers. We don’t need to test their code.”
Of course, anyone in testing knows from experience that even the best developers have errors in their code. I’ve also found that it seems easier for me to see errors in other people’s work than in my own.
It turns out that I’m not the only one who experiences that phenomenon. According to this blog post by Dorothy Graham, some research shows that people tend to find only about 1/3 of their own errors on initial inspection.
That’s a powerful justification for an investment in Quality Assurance.
Are you familiar with this research or statistic? I’m planning to look more into Capers Jones’ research and see how it applies to the value of Quality Assurance.
Influencers – James Bach
There are a number of blogs that I enjoy reading about all areas of the software development process. The software development process includes project management, business analysis, development, and testing, of course. In addition, delivering software that works requires consideration of concepts around management, sales, and business organization.
I’d like to share some of the blogs that I find influential when thinking about how to build software that works.
James Bach and Exploratory Testing
One of my favorite blogs is James Bach’s Blog. He is the creator of Rapid Software Testing. I first became interested in his blog because he wrote about exploratory testing in a way that made sense to me. I knew from experience that exploratory testing was one of the best ways to find important bugs, but I didn’t have a great way to communicate the high value of exploratory testing to teams that were focused heavily on mechanizing the testing process. James Bach has written a lot on the topic of exploratory testing. This particular post about the history of Exploratory Testing captures many of the concepts that drew me to the blog.
I particularly like the concepts of testing vs checking and the “responsible tester.”
Feedback
What do you think of James Bach’s blog and the concepts he writes about? What blogs and resources do you find particularly interesting and useful for building software that works?
Prioritization and Quality – Tales from the Trek – The Hoarders
Impact of Prioritization on Quality
As I’ve mentioned before, a key aspect of building quality software is ensuring that it does what the users need it to do. In my experience, the backlog of feature request (whether written or held in the stakeholders’ heads) is always much larger than what the development team can build in a short period of time.
When the backlog gets too big, people could spend more time managing the backlog than actually building anything. What is more likely, though, is that most of the backlog is ignored, and the clutter causes great ideas to get lost. I have seen cases where key customer issues ended up unaddressed for months until the customer complained a third or fourth time.
Idea Hoarding
I sat down with one of my clients to look at their backlog, and we found that they had over 400 backlog items that had not even been viewed for more than a year. They had more new, high-priority work coming in than they could deliver, so their backlog was growing. Clearly, nobody was ever going to review, let alone work on, the items that were over a year old. I suggested simply closing the backlog items that hadn’t been touched for over a year, but the client didn’t want to remove any items from the backlog without first reviewing them in a meeting with a team of key people, which was not going to happen.
The discussion reminded me of an episode of Hoarders where they were trying to convince someone to sell most of his 27 tool boxes. He agreed that 27 might be overkill, but he still didn’t want to sell them and insisted that the average person, who wasn’t a handyman like him, would need at least 7 toolboxes.
Time to Declutter
When a backlog gets hopelessly large, you may want to consider declaring backlog bankruptcy (based on the concept of email bankruptcy) and simply close all items that haven’t been looked at in over a year. If that sounds scary, I can understand. I tend towards hoarding myself, and I hate the thought of getting rid of something that might come in handy later. If declaring backlog bankruptcy, it may help to keep these ideas in mind:
- When there are too many backlog items, they are all ignored. The best ideas can’t break through the clutter.
- Business changes so quickly that most ideas more than a year old aren’t relevant any more.
- If an idea in the backlog is truly that important, it will get entered as a new entry again. In fact, if it’s that important, it probably was entered multiple times and has already been implemented.
- You can always search on closed items if you really, really want to!
Keeping the Backlog Under Control
Once you cleaned up the backlog, you want to try to keep it manageable. It helps to have a weekly triage process where the items are reviewed and prioritized. Some decisions that should be made during the triage process are:
- Is there any realistic chance this item will get resources in the next year? If not, close it.
- If the item is going to remain open, assign it to an owner who will take responsibility for getting the item onto the schedule.
- Enter the date the issue was last reviewed.
- Assign a priority and effort to the item.
I’ve found that it’s easier to identify which issues to review if you create a report that shows the priority of each item, the date it was entered, and the date it was last reviewed. This type of report helps ensure that older items are addressed.
Feedback
What challenges have you had with backlog clutter? What actions have you tried to address the challenges? I’d love to hear your stories from the trek.
The User Acceptance Testing Death Spiral
In a past role, I joined a test team that was in a User Acceptance Testing (UAT) “Death Spiral” that had caused the user base to lose confidence in the integration testing team. Based on conversations that I’ve had with others, I believe that the UAT Death Spiral is a common scenario that people encounter, and it can destroy a test team. It took some work, but we were able to pull out of the downward trajectory and regain a functional, productive partnership between the business folks and the integration testing team.
What is the User Acceptance Testing Death Spiral?
Our company had an understaffed integration testing team, aggressive deadlines, and a culture that valued meeting deadlines above all other goals. This meant that software often had known major bugs when UAT started. Even worse, there were typically areas of code that hadn’t been tested at all by the integration testing team before UAT started. It was common for the UAT team to find major, obvious bugs.
Because major bugs made it past the integration test team, the business felt a need to create a more formal, robust UAT team that would catch the numerous errors missed by the integration test team. The business folks assumed that the bugs were missed because the integration team wasn’t very good at their job.
As UAT became more robust, they realized that they needed more time to complete their testing. The company culture of tight deadlines meant that release dates could not be extended to accommodate in-depth UAT. Instead, the business insisted that the integration test cycle be shortened and that UAT start earlier.
I think you can see where this is going…
When UAT started earlier, they found even more bugs and the business lost even more confidence in the integration test team. The business then insisted on testing more of the functionality in UAT and starting UAT even earlier. This death spiral continued until the business had lost complete confidence in the integration test team. However, because the UAT team was not made up of experienced testers, the business was not finding all of the bugs either.
Basically, the integration team didn’t have time to do their job, the business was spending a huge amount of resources to test everything, even features already tested by the integration test team, and the quality of software in production was as poor as ever.
Breaking Free
Breaking free of this downward spiral was more than just a logistical problem. It was a political problem as well. The integration test team feared that the UAT team was trying to take over their jobs and the UAT team felt that the integration team wasn’t competent. Rebuilding trust was critical for any new process to be successful.
I worked with both the integration and UAT test teams to plan a new strategy. The strategy was that the integration test team would first test anything that did not have a user interface. In addition, the integration test team would write, maintain, and run automated regression tests. Basically, they would test the areas that required their expertise. Only after these areas were tested and any major bugs were fixed would the UAT team start their work. We would divide up the test cases to reduce any overlap of testing between the two teams as much as possible.
Even though the UAT team agreed that this plan made sense theoretically, they feared that removing the redundant testing would mean that bugs were missed and worried that starting UAT later would mean that they wouldn’t have time to complete their work. I convinced them to give the plan a try on a smaller project. If the advantages to the new plan didn’t materialize, it would be easier to adjust for the lost time on a small project.
Fortunately, on the small project, everything fell into place. The integration test team was able to adequately test their portion of the plan before UAT started, and the UAT team knew exactly what parts were and weren’t tested by the integration team. The UAT team had a shorter test cycle both because they didn’t run as many redundant tests and because the initial quality of code was much better. Each bug takes time to find, fix, and retest.
Best of all, the software went live and had no problems in production.
For the rest of my time with this team, we followed the new process. This resulted in higher quality code with lower cost, and it had the added benefit of greatly improving the working relationship between the two teams.
Your Experiences
I’d like to hear your experiences with User Acceptance Testing issues. Have you been in a situation where the business users lost confidence in the development team or integration test team? If so, what do you think were the root causes of the issues?
Prioritization and Quality – Tales from the trek – Priority 0
Impact of Prioritization on Quality
A key aspect of building quality software is ensuring that it does what the users need it to do. In my experience, the backlog of feature request (whether written or held in the stakeholders’ heads) is always much larger than what the development team can build in a short period of time. However, prioritizing these features seems to be difficult for people. Everything seems important, so everything gets priority 1.
Of course, if everything has the same priority, the stakeholders are de facto allowing the development team to prioritize the features. This can be a problem because the development team often doesn’t have the visibility to all of the factors that may determine the importance of the feature to the company’s success.
Here’s one example of a client who had an issue with prioritization, and how we arrived at a working solution.
Going Live too Early
A client had a new system replacing an existing business critical system. Unfortunately, their existing system had reached its technical limits before the new system was fully tested, and management made the decision to go live without much testing. Of course, the results were predictable. There were many errors in a production system that had to be fixed right away.
The “prioritization” method initially was that end users would come into the room of developers and tell them that they needed to drop everything and work on whatever issue the end user mentioned. The problem was that many different users were coming in each hour, and the developers didn’t get a chance to finish any task before being told to drop it and work on something else.
Getting Organized for Quality
The first thing we did was set up a SharePoint list where users could report their issues. We created a process where the users would report their issues in SharePoint. Then, I would triage the list and assign the work to developers. This simple improvement resulted in a huge increase in productivity for the development team because they could complete tasks without interruption.
However, we weren’t always working on the most important issues. Users were choosing the priority, and every issue was the highest priority to that user. Even when we met with representatives from all departments together and set definitions for priorities, every issue was priority 1 on a 3-priority scale.
Priority Names Matter
Our original 3 priority levels were called “High”, “Medium”, and “Low”. Because all issues were production issues, people didn’t want to minimize any by calling them “Medium” or “Low”. Everyone agreed that the issues were not all the same priority, but they weren’t willing to prioritize using those names.
First, the client came up with a category called “Priority 1 – Urgent”. This was higher than “Priority 1” and the client felt comfortable putting some items in Priority 1, and some in Priority 1 – Urgent. Still, way too many items were in Priority 1 – Urgent, so the development team was still choosing the priority.
Then, the client decided that the most critical items would be in a new priority called “Priority 0”. This was reserved for the top 5-7 issues to be worked on immediately by our 5-person development team.
This worked! The client was completely willing to prioritize into “Priority 0”, “Priority 1 – Urgent”, and “Priority 1” even though they were not willing to prioritize into “Priority 1”, “Priority 2”, and “Priority 3”. Just by changing the names of the priority levels, we were able to accomplish the goal of dividing the issues into 3 different levels.
We could then focus on the issues that brought the most value to the system.
Feedback
What challenges have you had with prioritizing features? What actions have you tried to address the challenges? I’d love to hear your stories from the trek.
- ‹ Previous
- 1
- 2
- 3
- Next ›