Category: Quality Software
Too Busy for User Acceptance Testing (UAT)?
I Love UAT
One of my favorite parts of testing is getting the opportunity to coordinate UAT. I love it because we’re getting feedback from the people who are actually going to use the system once it goes live. Even if a team is great about getting user input, everything up to this point is theoretical. All of us can think we know what we want, but once we actually see it, touch it, and live with it, we don’t know if what we asked for is what we really wanted or needed.
I’m planning to write a few articles about what I’ve learned during my UAT experiences, and I hope it will help others with their efforts.
Who Should Perform UAT?
The obvious answer to who should perform UAT is “users”. What users should perform UAT, though? Like many things, “it depends.” For a public-facing application, you may want people from the general public who don’t have a lot of deep knowledge about the system or industry. For an internal application, you likely want experienced users. However, those experienced users will often, understandably, plead that they are “too busy” to help test.
At one of my past clients, I requested the time of experienced users to test a back-end system for processing college applications. Like most companies, they were a bit understaffed and perpetually a bit behind at getting the work done. To the team, and to the manager who was trying to guard their time, my request was just one more bit of unnecessary busywork added to their plate.
The manager offered to hire a temporary worker to perform UAT so their most experienced workers wouldn’t be pulled away from their important work.
The Users Will Live with the Results
Given what I’ve seen others do with UAT in the past, this team may have believed that UAT was simply going to be formality where they would run tests that I had already verified met the written requirements. UAT is sometimes simply a step done to mark a checkbox in the few days before going live. However, when I run a UAT effort, it is much more than that.
I knew that having a temp worker wouldn’t give our team the insight we needed. A temp worker could certainly execute a pre-written test script, but that is not a useful way to perform UAT in my experience. Our team needed to know if this software truly met the needs of the team. We genuinely wanted to know, and we planned to incorporate the feedback into the final product.
I had a heart-to-heart with the manager and the team lead. I explained to them that their team would be using this application to do all of their work in the not-too-distant future. The team, in order to make the most of their investment, would likely be using the software for years. Shortly after the software went live, I’d be gone, but their team would be living with the results.
I explained that we truly were going to take their input into account to ensure the best software for doing their work. Then I asked if they still wanted a temp to be responsible for giving that input and making those decisions that would affect their team for years.
Once they understood the true impact of UAT, the answer was obvious. The manager and lead both made time for UAT, and they even had another employee participate. They brought on a temporary worker to handle some of the more routine clerical tasks instead of having them perform UAT.
We did incorporate their feedback into the final product, and the client was very happy with the end product. It was a true improvement to their old system.
If you could use some guidance with user acceptance testing, please reach out to me at [email protected]
Those Pesky Bugs – Special Characters
Pesky Bugs
Some bugs keep popping up in project after project. It’s important to catch these sneaky bugs before they get into a production system.
In the past, in a one month period, I caught three separate bugs that occurred due to improper handling of special characters. This type of bug can be easily missed if the test team isn’t specifically looking for it.
Culprit 1 – <
The innocent less-than sign is usually used for good. However, it can be a menace when it falls into the wrong hands. It can be used to send malicious executable scripts into systems from web pages.
Our program screened out HTML characters from the main entry screen by design. So, we didn’t expect to have any problems. However, it turned out that a portion of data in the system was imported from a spreadsheet rather than entered through the data screens. When the user tried to use a feature that created a new record from an existing record, the system failed. With a little digging and changing the settings that determine how the error messages are displayed, we identified the error.
Of course, this brings up an additional issue of how to handle data imports to screen out data that would fail the validation rules, but that’s a discussion for another time.
Culprit 2 – ñ
Of course, any site that accepts names should be sure to accept the letter ñ. In my experience, this letter typically hasn’t caused a problem, but it did cause a problem on a recent project. We built a system where people could set up a secure profile that included their name. The name and some other information were then sent to a third party that would send a welcome e-mail. Because we utilized Embedded Quality, we tested profile creation early, which included testing names containing ñ. Everything worked fine early on.
In a later round of testing, the ñ started causing an error. As we dug into the problem, we found that the issue was that the 3rd party e-mail system could not understand the Unicode representation of ñ. Instead, the 3rd party expected this character to be in UTF format. Once we realized that, we were able to convert the format of the characters before sending them to the e-mail vendor.
Culprit 3 – : )
On the same project, we encountered an error that became known as the “smiley face” bug. This bug was not found until user acceptance testing, which is much later than I like to find bugs. In this case, we had tested online profile IDs that contained every acceptable special character without a problem. However, we did not specifically test the situation where the profile ID ended with a smiley face emoticon. When one user created an ID of janed:), the profile creation failed.
It turned out that the expression in code that checked for a duplicate profile ID before creation could handle any special character except for a single, right parenthesis. It interpreted that character as part of the expression, and caused a failure.
Summary
Encountering these 3 special character errors within a one month period reminded me of how important it is to test for every special character before going live. The experience validated my general rule of “if the system allows it, the users will do it.”
Have any of you found bugs due to special characters? Have any made it into production? If so, what was the consequence?
Embedded Quality – Development Team
When I’ve introduced Embedded Quality to the developers on a project, a common reaction is fear and loathing. They often have a visceral, negative reaction to the words “Quality Assurance,” and the idea of working more closely with the Quality Assurance team instills a feeling of dread in them.
As a reminder, I originally outlined the basic concepts of Embedded Quality in this blog post. The concepts are:
- Quality Assurance starts on Day 1
- Quality Assurance is part of the Core Project Team
- Quality Assurance is performed by qualified experts
- General User Acceptance Testing does not begin until Core Project Team QA is complete
- Strong Foundation – No code is “Complete” until it is tested and works correctly
Although the developers I’ve worked with generally have a negative first reaction to the concept of Embedded Quality, once they’ve experienced it, they almost always request it on their next project. The question is, “how can we convince developers to try Embedded Quality the first time?”
Overcoming Common Concerns
I’ve found that it’s typically harder to overcome the developers’ concerns than the Quality Assurance team’s concerns because the developers usually start with the mindset that they don’t want to use Embedded Quality. However, there are ways to address their concerns.
The Code Isn’t Ready to Test
The developers will often say that the code really isn’t ready to test until very close to the end of the project. However, there are some good reasons why this should not be the case.
- The project schedule generally shows sections of code being completed before the end of the project. If a developer says that the code is complete, it’s difficult to imagine why it can’t be tested.
- It’s much easier to identify and fix bugs in small portions of code.
- Finding and fixing bugs early means that new code is built on a solid foundation.
Having the Quality Assurance team test code early in the process is analogous to having a building inspector view various portions of a building before it is complete. It makes a lot of sense for the inspector to check the foundation, the electrical, the plumbing, and the HVAC system before the building is anywhere near complete. Fixing these items after the building is complete is much more costly and difficult than fixing them earlier.
It Will Take Too Much of My Time
Another concern that developers have is that they will have to spend too much time explaining things to the testers. Because the code isn’t complete, they are afraid that the testers will report bugs for items that are simply not yet finished. Also, they will have to spend more time explaining how to test certain items when the complete UI may not be available.
These concerns are legitimate and show the need for having a test team of qualified experts. The testers should have a mindset of collaboration with the developers rather than antagonism. The testers will work closely with the developers to understand what code can be tested and what code is not ready. Of course, any code that is not ready cannot be considered complete for project planning purposes.
It is also true that the developer will have to spend more time explaining how to test some incomplete code. However, if the Quality Assurance team is qualified and experienced, they should be able to quickly learn any tools needed for testing. In my experience, any extra time that the developers spend helping the Quality Assurance team is saved many times over by finding the problems early.
I Don’t Need the Extra Pressure of Someone Looking Over My Shoulder
Many developers have had bad experiences working with testers who take delight in pointing out the flaws of others. These types of testers tend to make comments like, “I don’t know what those developers were thinking,” when they chat with each other in the break-room. This attitude really does put unnecessary pressure on the developers and contributes to a hostile relationship between developers and testers.
This is why it’s critical to ensure that the Quality Assurance team is made up of true professionals. I always remind the developers that Quality Assurance is part of the core team, which means that our goal is to ensure that the project gives a good impression to everyone outside of the team. I tell the developers that I know that they’re good developers and that bugs are a normal part of the process. The developers shouldn’t feel bad if the Quality Assurance team finds bugs. In fact, they should be happy because every bug that is found by the Quality Assurance team is one not found by the project sponsor or the end users. The project sponsor and the end users are the people who are generally the harshest critics of the development team.
Other Concerns?
I’d like to know what concerns you would have introducing Embedded Quality into your organization. What concerns would you anticipate from others? I’ll do my best to address those concerns in future posts.
Offshore Agile Testing…a Request from Management
A friend of mine asked me for some advice about how to run an Agile software development team with the developers and business users working in the US and the testers working for an outsourcing company in India.
My short answer was, “You can’t”.
The Agile Manifesto and Offshore Testing
I’ve actually had similar requests a number of times over the past few years. Managers want to be “Agile” and want to use “Offshore resources” so they ask their project teams to implement the combination of the two hot concepts. In the cases that I’ve seen, management has tried to send only the testing portion of development offshore. They wanted the onshore, local team to be “Agile”, but they also want to send code “over the wall” to an offshore test team. Unfortunately, these two concepts do not work together.
The Agile Manifesto principles include “Business people and developers must work together daily throughout the project;” “Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done;” “The most efficient and effective method of conveying information to and within a development team is face-to-face conversation;” and “The best architectures, requirements, and designs emerge from self-organizing teams.”
These 4 (out of 12) principles are very difficult, if not impossible, to implement in an environment where the testers are separated from the rest of the team by geography and time zones. In addition, when only the test team is outsourced to another company, the testers often are required to communicate with the rest of the team via a single point of contact rather than directly with developers and business people.
To make things even more difficult, I’ve found that offshore, outsourced teams are typically set up to run a full, lengthy regression test for each build. In one case, I worked with a team that was trying to do Agile development with a test team that insisted on running a 6 week regression test for every build. This ran completely counter to the Agile Manifesto principle of “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.”
Can Agile Work with Offshore Teams?
I believe that Agile development can work with offshore teams if the teams are structured correctly and significant effort is put forth in building trust. Here are some techniques that I’ve used effectively to help build an Agile, globally distributed team.
- Have a full team consisting of all roles in each geographic location. This means that developers, testers, and business experts must be physically located in each location. The business expert may be a Product Owner Advocate rather than the actual Product Owner.
- Ensure that everyone on the global team has time to work together in person early in the project. This means flying people around the world which can be costly, but it makes a huge difference in the ability for team members to trust each other and work together.
- If the team is divided into drastically different time zones, schedule specific weeks throughout the project where the global team will work at the same time regardless of time zone. During these weeks, the team should make a concerted effort to communicate in real time with their counterparts around the world.
In a future post, I’ll talk about some additional techniques that are not specific to Agile that I’ve used to make global teams more effective.
Further Reading
Over the years, I’ve found these articles helpful in understanding the issues related to Agile distributed teams and how to address those issues. Maybe you’ll find them useful as well. To me, the two articles available for purchase from IEEE Computer Society were definitely worth the small cost.
Agile Strategies for Geographically Distributed Teams
Global Development and Delivery in Practice
Agile Offshore Techniques – A Case Study (Available for purchase)
Using and Agile Software Process with Offshore Development
Follow the Sun: Distributed Extreme Programming Development (Available for purchase)
Your Experiences
I’d like to hear any of your experiences with global Agile software development. What worked and what didn’t?
Contact me if you’d like help improving the effectiveness of your global team.
When the Firefighter is the Arsonist – Tales from the Trek
A number of years ago, I read a headline that said, “Ex-Forest Lake fire fighter pleads guilty to arson“. The story went on to say,
“Once considered a hero as a firefighter, John Berken was accused of starting a fire that burned 2,000 acres of nature area.”
This story got me thinking about a common pattern that I have seen on software development projects.
Heroic Measures
Many teams glorify the team members who take heroic actions to save a project. These team members, who work nights and weekends for weeks, or even months, at a time to pull a project success from the jaws of defeat, are held up as positive examples for the company and lavished with praise.
However, too often, the very same people who perform the heroic deeds are the people who created the near-disaster in the first place.
As a QA Sherpa, one of my top goals is to prevent the need for heroic actions at the end of a project. I feel happiest when we successfully implemented Embedded Quality and find no important bugs during the last few weeks before going live. In this role, I’m particularly attuned to those “heroes” who actually created the problems in the first place.
These are the people who dramatically underestimated their work, skipped unit testing, refused to work with the testing team to practice Embedded Quality, and generally eschew any form of teamwork or foresight. Despite all these negative traits, they end up being celebrated for “going above and beyond the call of duty”.
True Firefighters
True firefighters do need to be heroic at times, and it’s important that they are ready and able to be heroes when necessary. However, their bigger goal is to prevent fires in the first place. In addition to preventing fires, they work hard to put measures in place to limit the damage of fires when they do occur.
The bulk of a fire department’s efforts are not spent putting out fires. The bulk of the time is spent educating people about how to prevent fires, ensuring that the fire code is effective and that buildings follow the fire code by implementing fire walls, fire doors, sprinkler systems, alarms, and fire extinguishers.
Similarly, a solid development team should be spending their time implementing automated unit testing, working with the test team to implement Embedded Quality, performing peer code reviews, and getting feedback from key stakeholders throughout the project. All of these actions work both to reduce the odds of a disaster and to limit the damage when a disaster occurs.
As I said earlier, heroes are still needed at times, but you should take a second look at those heroes when the disasters are too frequent. If a city has way more fires than the norm, the fire department probably isn’t doing their job well. If every project requires heroic measures, it’s time to look a little closer at why that’s the case.
Feedback
I’m interested in hearing about your experiences with fire fighters who may also be arsonists. Have you worked on a project where team members who seemed to sabotage the project early on were held up as heroes at the end of the project? If so, do you think it was due to the individuals on the project or due to a broken system?
Metric Misuse – Quality Assurance Metrics Gone Awry
I was reading through some posts from Bob Sutton, one of my favorite management gurus, and I ran across a post that contains one of my favorite Dilbert comic strips.
Bob Sutton’s post, as well as the comments that I made on his blog, reminded me of one of my favorite topics: misused Quality Assurance metrics.
Tying Quality Assurance Metrics to Financial Rewards – A Dangerous Game
“Treat monetary rewards like explosives, because they will have a powerful impact whether you intend it or not.” –Mary and Tom Poppendieck, authors of Implementing Lean Software Development: From Concept to Cash
Over the years, many people have asked me what Quality Assurance metrics they should use to evaluate employee performance. My advice is that Quality Assurance metrics should not be used directly to evaluate employee performance. The Dilbert comic strip may seem a bit extreme, but it’s exactly what happens when employee performance is based strictly on metrics. This is true regardless of whether monetary rewards are explicitly tied to the metrics or not.
In my comments on Bob Sutton’s blog, I mentioned three specific metrics that had unintended effects when used for evaluating employee performance:
- rewarding testers for the number of test cases they wrote resulted in poorly written test cases;
- rewarding testers for the number of bugs they found resulted in a high number of unimportant or duplicate bugs reported; and
- penalizing testers for bugs rejected by the test lead or development staff resulted in important bugs going unreported.
Many people think that they have the ability to write a set of metrics that can be used to unequivocally gauge the performance of a Quality Assurance professional, but I have not yet encountered a metric that couldn’t be manipulated to favor the employees.
(If the metric can’t be gamed, it probably isn’t under the control of the employees, so it wouldn’t be effective at driving behavior anyhow.)
Are Metrics Worthless Then?
Actually, metrics are a great tool for identifying coaching opportunities and potential problems. However, in order to get honest metrics, they shouldn’t be used directly for employee evaluations or employee rewards.
When I’ve looked at the metrics that I mentioned earlier with an eye towards coaching, I had excellent results.
- Reviewing the number of test cases written helped me identify a tester on my team who was putting much more detail than I wanted into his test cases. After some coaching, he was able to consistently meet my expectations.
- Reviewing the number of bugs found by each tester helped me identify a tester who was digging into the root cause of the most difficult to reproduce bugs. She didn’t report as many bugs as others, but her work was critical to getting a great product out the door in a timely manner. It turned out that she was the most skilled tester even though she reported the fewest bugs.
- Reviewing the number of bugs rejected by the development staff helped me identify a manager who was evaluating his programmers based solely on the number of valid bugs found in their code. The developers were motivated to simply mark bugs as invalid rather than fix the bugs. This insight allowed me to address the problem directly with that manager.
Good Quality Assurance metrics provide powerful tools for managing a Quality Assurance team when used properly. However, they shouldn’t be used in a vacuum. They should just be considered one data point among many.
I was only able to scratch the surface of this topic in this blog post. I plan to discuss specific metrics in future blog posts. In the meantime, if you want to read a much more in-depth review of the pros and cons of employee incentives, you can find one paper here.
Your Experiences
I know that a lot of people feel passionately about Quality Assurance metrics, both pro and con. I’m very interested to hear about your experiences with Quality Assurance metrics. Have you found any that were particularly useful? Have you found any that had unintended consequences?
Communication – Tales from the Trek – Perception is Reality
Based on my conversations, I believe that most people perceive that they do more work than others. It seems common that all members of a household think they do more than their fair share of work, and most members of teams at work feel the same way. I believe that a big driver in this perception is that people know exactly what work they’ve done, but they are not aware of the work that others have done.
IT is Lazy!
At one place where I worked, the business side of the department often expressed to me that IT was lazy. They said that IT was never getting to any of their requests, and they weren’t sure what IT was doing all day. Not surprisingly, the IT department expressed that they were overworked and accomplishing a lot. Why was there this extreme difference in perception?
The IT department met with the business leadership each year to determine the priorities for the year based on the hours budgeted. IT was excellent at delivering on the priorities; however, the business had many more requests for changes and improvements that were above and beyond the priorities for the year. So, most individuals on the business side didn’t have their personal priorities addressed even though the overall strategic priorities for IT were accomplished.
Changing Perceptions
A director of the IT department came up with an idea that was very simple but incredibly effective. He created a giant checklist of the scheduled IT projects and posted them in a few prominent places around the office. As each project was completed, he put a checkmark next to the item.
The comments that I heard from the business completely changed. Although their personal priorities may not have been addressed, they could see the importance of the priorities posted on the checklist. In addition, they could see clearly the progress that had been made. The checklist had greatly improved the business’s perception of IT.
Feedback
What perception challenges have you faced? How did you address the challenges? I’d love to hear your tales from the trek.
Value of QA – People Do Not Find Their Own Errors
I’ve noticed, through the years, that people often dismiss the need for Quality Assurance by saying, “These are good developers. We don’t need to test their code.”
Of course, anyone in testing knows from experience that even the best developers have errors in their code. I’ve also found that it seems easier for me to see errors in other people’s work than in my own.
It turns out that I’m not the only one who experiences that phenomenon. According to this blog post by Dorothy Graham, some research shows that people tend to find only about 1/3 of their own errors on initial inspection.
That’s a powerful justification for an investment in Quality Assurance.
Are you familiar with this research or statistic? I’m planning to look more into Capers Jones’ research and see how it applies to the value of Quality Assurance.
Influencers – James Bach
There are a number of blogs that I enjoy reading about all areas of the software development process. The software development process includes project management, business analysis, development, and testing, of course. In addition, delivering software that works requires consideration of concepts around management, sales, and business organization.
I’d like to share some of the blogs that I find influential when thinking about how to build software that works.
James Bach and Exploratory Testing
One of my favorite blogs is James Bach’s Blog. He is the creator of Rapid Software Testing. I first became interested in his blog because he wrote about exploratory testing in a way that made sense to me. I knew from experience that exploratory testing was one of the best ways to find important bugs, but I didn’t have a great way to communicate the high value of exploratory testing to teams that were focused heavily on mechanizing the testing process. James Bach has written a lot on the topic of exploratory testing. This particular post about the history of Exploratory Testing captures many of the concepts that drew me to the blog.
I particularly like the concepts of testing vs checking and the “responsible tester.”
Feedback
What do you think of James Bach’s blog and the concepts he writes about? What blogs and resources do you find particularly interesting and useful for building software that works?
Prioritization and Quality – Tales from the Trek – The Hoarders
Impact of Prioritization on Quality
As I’ve mentioned before, a key aspect of building quality software is ensuring that it does what the users need it to do. In my experience, the backlog of feature request (whether written or held in the stakeholders’ heads) is always much larger than what the development team can build in a short period of time.
When the backlog gets too big, people could spend more time managing the backlog than actually building anything. What is more likely, though, is that most of the backlog is ignored, and the clutter causes great ideas to get lost. I have seen cases where key customer issues ended up unaddressed for months until the customer complained a third or fourth time.
Idea Hoarding
I sat down with one of my clients to look at their backlog, and we found that they had over 400 backlog items that had not even been viewed for more than a year. They had more new, high-priority work coming in than they could deliver, so their backlog was growing. Clearly, nobody was ever going to review, let alone work on, the items that were over a year old. I suggested simply closing the backlog items that hadn’t been touched for over a year, but the client didn’t want to remove any items from the backlog without first reviewing them in a meeting with a team of key people, which was not going to happen.
The discussion reminded me of an episode of Hoarders where they were trying to convince someone to sell most of his 27 tool boxes. He agreed that 27 might be overkill, but he still didn’t want to sell them and insisted that the average person, who wasn’t a handyman like him, would need at least 7 toolboxes.
Time to Declutter
When a backlog gets hopelessly large, you may want to consider declaring backlog bankruptcy (based on the concept of email bankruptcy) and simply close all items that haven’t been looked at in over a year. If that sounds scary, I can understand. I tend towards hoarding myself, and I hate the thought of getting rid of something that might come in handy later. If declaring backlog bankruptcy, it may help to keep these ideas in mind:
- When there are too many backlog items, they are all ignored. The best ideas can’t break through the clutter.
- Business changes so quickly that most ideas more than a year old aren’t relevant any more.
- If an idea in the backlog is truly that important, it will get entered as a new entry again. In fact, if it’s that important, it probably was entered multiple times and has already been implemented.
- You can always search on closed items if you really, really want to!
Keeping the Backlog Under Control
Once you cleaned up the backlog, you want to try to keep it manageable. It helps to have a weekly triage process where the items are reviewed and prioritized. Some decisions that should be made during the triage process are:
- Is there any realistic chance this item will get resources in the next year? If not, close it.
- If the item is going to remain open, assign it to an owner who will take responsibility for getting the item onto the schedule.
- Enter the date the issue was last reviewed.
- Assign a priority and effort to the item.
I’ve found that it’s easier to identify which issues to review if you create a report that shows the priority of each item, the date it was entered, and the date it was last reviewed. This type of report helps ensure that older items are addressed.
Feedback
What challenges have you had with backlog clutter? What actions have you tried to address the challenges? I’d love to hear your stories from the trek.