Category: Quality Assurance

When the Firefighter is the Arsonist – Tales from the Trek

A number of years ago, I read a headline that said, “Ex-Forest Lake fire fighter pleads guilty to arson“. The story went on to say,

“Once considered a hero as a firefighter, John Berken was accused of starting a fire that burned 2,000 acres of nature area.”

This story got me thinking about a common pattern that I have seen on software development projects.

Heroic Measures

Many teams glorify the team members who take heroic actions to save a project. These team members, who work nights and weekends for weeks, or even months, at a time to pull a project success from the jaws of defeat, are held up as positive examples for the company and lavished with praise.

However, too often, the very same people who perform the heroic deeds are the people who created the near-disaster in the first place.

As a QA Sherpa, one of my top goals is to prevent the need for heroic actions at the end of a project. I feel happiest when we successfully implemented Embedded Quality and find no important bugs during the last few weeks before going live. In this role, I’m particularly attuned to those “heroes” who actually created the problems in the first place.

These are the people who dramatically underestimated their work, skipped unit testing, refused to work with the testing team to practice Embedded Quality, and generally eschew any form of teamwork or foresight. Despite all these negative traits, they end up being celebrated for “going above and beyond the call of duty”.

True Firefighters

True firefighters do need to be heroic at times, and it’s important that they are ready and able to be heroes when necessary. However, their bigger goal is to prevent fires in the first place. In addition to preventing fires, they work hard to put measures in place to limit the damage of fires when they do occur.

The bulk of a fire department’s efforts are not spent putting out fires. The bulk of the time is spent educating people about how to prevent fires, ensuring that the fire code is effective and that buildings follow the fire code by implementing fire walls, fire doors, sprinkler systems, alarms, and fire extinguishers.

Similarly, a solid development team should be spending their time implementing automated unit testing, working with the test team to implement Embedded Quality, performing peer code reviews, and getting feedback from key stakeholders throughout the project. All of these actions work both to reduce the odds of a disaster and to limit the damage when a disaster occurs.

As I said earlier, heroes are still needed at times, but you should take a second look at those heroes when the disasters are too frequent. If a city has way more fires than the norm, the fire department probably isn’t doing their job well. If every project requires heroic measures, it’s time to look a little closer at why that’s the case.

Feedback

I’m interested in hearing about your experiences with fire fighters who may also be arsonists. Have you worked on a project where team members who seemed to sabotage the project early on were held up as heroes at the end of the project? If so, do you think it was due to the individuals on the project or due to a broken system?

Setting Expectations – Lessons From a Little League Umpire

I’ve found through the years that projects go more smoothly when expectations are set at the beginning. Whether I’m in a Business Analyst or Quality Assurance role, I’ve found that the project goes more smoothly when the processes are laid out clearly up-front and the known limitations of the project are called out and addressed before the project starts. When people are not aware of the time that they need to dedicate to the project or the limits of the project scope until the project is well underway, they can get quite upset.

My most memorable lesson on the importance of setting expectations is still the one I learned while umpiring little league in high school.

Strike One!

I umpired my first game when I was 14 years old. The league was a city-run little league program, and some of the divisions had kids as old as me. Other than watching a lot of baseball, the only preparation the city gave me was handing me an armful of equipment and a copy of the rulebook. I read the rulebook several times and felt that I was as ready as I could be.

I set up the bases and put on my equipment for the first game. I said hi to the two managers and nervously took my spot behind the plate. My calls were a little shaky, but I was surviving. Surviving until the third batter, that is.

I lost my concentration on a pitch and uncertainly called, “Strike?” The batter’s manager yelled at me, “How can that be a strike?!? It hit the dirt before it even crossed the plate!”

I sheepishly said, “It did? I guess it was a ball then.”

Needless to say, the managers, players, and parents argued with me vehemently on nearly every call after that. I wasn’t sure I’d ever want umpire another game. However, the manager of the umpires was desperate for warm bodies to call the games and convinced me to finish out the season. I never had a game as bad as the first one, but every game was stressful and the managers, parents, and players argued with me regularly.

Year Two – Setting Expectations

In the next year, I read a book called “Strike Two” by former Major League umpire Ron Luciano. His stories about handling some of the toughest personalities in baseball gave me the insight and confidence that I needed to try umpiring again. The difference this year was that I was going to set some expectations before every game.

I met with both managers before every game and explained the following:

  • I’m the only umpire, and I only see each play once. Don’t bother arguing any judgment call such as ball vs strike or safe vs out. I won’t change my call.
  • I call a big strike zone. If any part of the ball is over any part of the plate from the top of the batter’s shoulders to the bottom of their knees, I’m calling it a strike.
  • I’m not a professional umpire. I may make an error in the rules. If either manager believes that I made an error in the rules, let me know. I will call a time-out and meet with both managers to review the rule book and make an adjustment to the call if warranted.

Taking the time to set these expectations made a huge difference in my ability to manage the game. The managers would tell their players that I have a big strike zone and that arguing with me was pointless. The players swung a lot more, put the ball in play a lot more, and I got very few arguments.

In addition, after the games, parents would tell me that it was one of the best games they went to all year, especially for the younger age groups. Apparently, in many other games, the batters for each team would simply draw walks until they reached the 10-run mercy rule for the inning. This was boring for everyone involved. Because players were swinging more when I umpired, the ball was put into play more, and the kids and parents had a lot more fun.

Your Experiences

I’d like to hear about your experiences with setting expectations. Have you found that setting expectations up front helps your projects? Have you received any resistance to setting expectations at the start of a project?

Tester Traits – Comfort with Cognitive Dissonance

I believe that software testers need to be comfortable with two conflicting ideas in order to best contribute to the success of a project. They need to be able to find and point out every defect in a given software application; however, they also need to keep in mind that the software can meet the business’ needs even if it is not perfect. The best software testers that I’ve worked with are comfortable with the cognitive dissonance caused by these two conflicting ideas.

Goal of Perfection

I’ve worked with some software testers who are great at finding defects, but they get so focused on the defects that they lose sight of the big picture. They seem to believe that the ultimate purpose of software is to be perfect rather than to meet a business goal. These software testers can cause a software project to be unnecessarily delayed by arguing that minor issues should prevent software from going live. They can also reduce end-user acceptance of a product by complaining to anyone who will listen that the software is subpar and by pointing out every known issue to end-users.

I believe that testers do need to point out every bug possible to the internal development team and ensure that the team knows the business implications of any bugs. However, good testers need to be comfortable with the fact that perfection is not ultimate goal for business software. Instead, software has some business purpose that does not require perfection. Once software goes live, I believe that a tester should champion the software to the end-users.

When I encounter a tester who has trouble accepting imperfect software, I remind them that most useful commercial software has known bugs. Often, some of these bugs are even listed in a “Read Me” file for end users to see. There is a real value to having less than perfect software available rather than having no software at all.

Goal of Delivery

Although I have seen testers who are too focused on perfection, I have also seen testers who are too focused on delivery of the software. These testers either are not detail-oriented enough to find critical bugs or they ignore the real business impact of bugs. These testers do not want to stand in the way of the delivery date under any circumstances. Project Managers often put enormous pressure on testers to sign-off on software that was not properly tested or has major bugs in order to meet a delivery deadline. However, the best testers will have the backbone to stand up to any pressures and point out the specific risks of going live.

In my opinion, it’s important that the tester understand the business goals well enough to point out the specific risks to key decision makers; however, I believe that it should not be up to the tester to decide whether or not software goes live. The ultimate decision should be made by management who understands the full scope of business needs and can weigh the quality risks against every other concern.

The Ponds

There’s a poem by Mary Oliver called The Ponds that I think captures the essence of this trait that software testers need to contribute to a successful project. Testers need to notice imperfections while still being “willing to be dazzled”. This version of the poem is from a book called New and Selected Poems by Mary Oliver published by Beacon Press in 1992. I hope you enjoy it and find it relevant.

Every year

the lilies

are so perfect

I can hardly believe

their lapped light crowding

the black,

mid-summer ponds.

Nobody could count all of them –

the muskrats swimming

among the pads and the grasses

can reach out

their muscular arms and touch

only so many, they are that

rife and wild.

But what in this world

is perfect?

I bend closer and see

how this one is clearly lopsided –

and that on wears an orange blight –

and this one is a glossy cheek

half nibbled away –

and that one is a slumped purse

full of its own

unstoppable decay.

Still, what I want in my life

is to be willing

to be dazzled –

to cast aside the weight of facts

and maybe even

to float a little

above this difficult world

I want to believe I am looking

into the white fire of a great mystery.

I want to believe that the imperfections are nothing –

that the light is everything – that it is more than the sum

of each flawed blossom rising and fading. And I do.

User Acceptance Testing Tales from the Trek – Writing a Perfect Bug

My early experiences running User Acceptance Testing (UAT) typically involved dealing with just 1 or 2 users. When they encountered an error, they’d show it to me, and I’d record it. However, as I gained experience in Quality Assurance, I worked on larger projects.

Training – How to write a bug

On my first larger project, I needed to train a group of 20 users who were going to perform User Acceptance Testing. Having worked with several different users in the past, I knew several of the pitfalls that users may encounter. However, I never really had asked users to record errors themselves. Still, I thought it would be easy to get the point across.

During the training session, I said that it was critical to record all necessary steps so that the developer could recreate the bug. I told the users that we often hear from users who simply say, “It doesn’t work,” and explained that the developers will not be able to fix a problem that was described so generically. The users said that they understood perfectly and were ready to start testing.

Bug Writing – Frustration

With 20 inexperienced testers working independently, it was important to filter the reported bugs before assigning them to the developers. I was responsible for triaging all bugs reported by the users. I verified that I could recreate bugs and eliminated duplicate bugs before assigning them to the developers. If I could not recreate a bug based on the instructions, I would reassign it to the user who reported it and ask for more detail. I assumed that the users would quickly realize when they were not providing sufficient detail and start writing perfect bugs after a day or two.

I assumed wrong.

Although a few users were writing clear, reproducible bugs, most were writing ones that were way too generic. For example:

  1. I was on the “Choose a Course” screen.
  2. When I chose a course, I got an error.

In this case, when I tried to reproduce the bug, I did not receive an error. So, I sent the bug back to the user requesting more detail. In most cases, the first time a bug was recorded, I had to reject it. In many cases, I had to reject the rewrite as well. Pretty quickly, the users got fed up with me and complained to their manager that I was rejecting their bugs.

Enlightenment

I realized that I needed to find a way to illustrate to the users exactly what was needed in a clear, reproducible bug report. I developed the following simple exercise that was very effective. I have used it on all subsequent large-scale UAT efforts with the same success.

I ask the users to do the following with their first 5 bug reports:

  1. Record the bug as clearly as you can and assign it to yourself.
  2. Wait 5 hours.
  3. Try to recreate the bug following only the information that you recorded.
  4. If you are able to recreate the bug, assign it to me. Otherwise, go back to step 1.

It is important that the users wait some amount of time before trying to recreate the bug. I found that when they try to recreate the bugs immediately, the steps are too fresh in their mind to truly follow only what they wrote. They seem to fill in the gaps without realizing it.

I have found that once users complete this exercise, they are able to consistently write bugs that have enough detail to be reproducible. They do not become expert testers, of course, but they are able to write a useful bug.

I’m interested in hearing about other people’s experiences as well.

What concepts have you found especially difficult to get across to people performing UAT for the first time?

Were you able to find a technique that helped the users better understand the concept?

User Acceptance Testing – Planning for Value

Throughout the years, I’ve led many user acceptance testing (UAT) efforts. There are a couple of common ways to plan the testing that I’ve found to be problematic. The methods allow someone to say they’ve planned UAT but add very little value. The end result is that unhappy users report problems after the software is live, and the project manager asks why they didn’t find the problems during UAT.

After years of experimentation, I found a method that adds a lot of value to the testing process with less effort than the other methods.

Method 1 – Users write UAT scripts

I’ve been involved with quite a few projects where the client wanted the users of the software to write the test scripts for UAT. This was often done to reduce development costs. The folks planning the project believed that they would save money by having business employees write the test scripts for UAT rather than have someone from the development team work on them. They also believed, often correctly, that the business folks understood the goals of the system best, so they were the most qualified to write the UAT scripts.

The problem with this is that most business users have more than a full time job already. In addition, most business users have never planned a testing effort or have written a test script. It’s difficult enough for business users to find time to execute tests, but nearly impossible for them to find time to plan testing. Even if they do find the time to plan it, their lack of experience results in test scripts that test very little.

Method 2 – Test Analyst writes UAT scripts

On other projects, I’ve been asked to write test scripts that the users can run during UAT. Usually, I’m asked for the same scripts that the test team ran during system testing. Under this scenario, it’s unlikely that the users will find any issues that the test team didn’t already find. This method doesn’t uncover functional issues that the test team missed. In addition, I’ve found that the business users rarely take the time to follow the test script closely and end up not doing much testing at all.

UAT Success Method – Guided Functional Checklists

The UAT planning method that I’ve found adds the most value with the least effort is Guided Functional Checklists. This method is quite simple:

  • A member of the test team meets with each key user and asks what business goals they need to accomplish with the software.
  • The test team member compiles the list into a spreadsheet that will be used for recording the test results.

That’s all there is to the planning. The result is an organized list of what should be tested based on the business goals of the users rather than based on the development team’s view of the project. Because this takes little of the users’ time, they are able to make time to complete the planning. By not having a detailed script, users are able to ensure they can meet their business goals in the way that they will in the field, which is often a bit different than the development team expects. This allows everyone to identify programming or training issues before going live.

Feedback

What UAT planning challenges have you faced? How did you address the challenges? I’d love to hear your tales from the trek.

Metric Misuse – Quality Assurance Metrics Gone Awry

I was reading through some posts from Bob Sutton, one of my favorite management gurus, and I ran across a post that contains one of my favorite Dilbert comic strips.

Bob Sutton’s post, as well as the comments that I made on his blog, reminded me of one of my favorite topics: misused Quality Assurance metrics.

Tying Quality Assurance Metrics to Financial Rewards – A Dangerous Game

“Treat monetary rewards like explosives, because they will have a powerful impact whether you intend it or not.” –Mary and Tom Poppendieck, authors of Implementing Lean Software Development: From Concept to Cash

Over the years, many people have asked me what Quality Assurance metrics they should use to evaluate employee performance. My advice is that Quality Assurance metrics should not be used directly to evaluate employee performance. The Dilbert comic strip may seem a bit extreme, but it’s exactly what happens when employee performance is based strictly on metrics. This is true regardless of whether monetary rewards are explicitly tied to the metrics or not.

In my comments on Bob Sutton’s blog, I mentioned three specific metrics that had unintended effects when used for evaluating employee performance:

  1. rewarding testers for the number of test cases they wrote resulted in poorly written test cases;
  2. rewarding testers for the number of bugs they found resulted in a high number of unimportant or duplicate bugs reported; and
  3. penalizing testers for bugs rejected by the test lead or development staff resulted in important bugs going unreported.

Many people think that they have the ability to write a set of metrics that can be used to unequivocally gauge the performance of a Quality Assurance professional, but I have not yet encountered a metric that couldn’t be manipulated to favor the employees.

(If the metric can’t be gamed, it probably isn’t under the control of the employees, so it wouldn’t be effective at driving behavior anyhow.)

Are Metrics Worthless Then?

Actually, metrics are a great tool for identifying coaching opportunities and potential problems. However, in order to get honest metrics, they shouldn’t be used directly for employee evaluations or employee rewards.

When I’ve looked at the metrics that I mentioned earlier with an eye towards coaching, I had excellent results.

  1. Reviewing the number of test cases written helped me identify a tester on my team who was putting much more detail than I wanted into his test cases. After some coaching, he was able to consistently meet my expectations.
  2. Reviewing the number of bugs found by each tester helped me identify a tester who was digging into the root cause of the most difficult to reproduce bugs. She didn’t report as many bugs as others, but her work was critical to getting a great product out the door in a timely manner. It turned out that she was the most skilled tester even though she reported the fewest bugs.
  3. Reviewing the number of bugs rejected by the development staff helped me identify a manager who was evaluating his programmers based solely on the number of valid bugs found in their code. The developers were motivated to simply mark bugs as invalid rather than fix the bugs. This insight allowed me to address the problem directly with that manager.

Good Quality Assurance metrics provide powerful tools for managing a Quality Assurance team when used properly. However, they shouldn’t be used in a vacuum. They should just be considered one data point among many.

I was only able to scratch the surface of this topic in this blog post. I plan to discuss specific metrics in future blog posts. In the meantime, if you want to read a much more in-depth review of the pros and cons of employee incentives, you can find one paper here.

Your Experiences

I know that a lot of people feel passionately about Quality Assurance metrics, both pro and con. I’m very interested to hear about your experiences with Quality Assurance metrics. Have you found any that were particularly useful? Have you found any that had unintended consequences?

Value of QA – People Do Not Find Their Own Errors

I’ve noticed, through the years, that people often dismiss the need for Quality Assurance by saying, “These are good developers. We don’t need to test their code.”

Of course, anyone in testing knows from experience that even the best developers have errors in their code. I’ve also found that it seems easier for me to see errors in other people’s work than in my own.

It turns out that I’m not the only one who experiences that phenomenon. According to this blog post by Dorothy Graham, some research shows that people tend to find only about 1/3 of their own errors on initial inspection.

That’s a powerful justification for an investment in Quality Assurance.

Are you familiar with this research or statistic? I’m planning to look more into Capers Jones’ research and see how it applies to the value of Quality Assurance.

Influencers – James Bach

There are a number of blogs that I enjoy reading about all areas of the software development process. The software development process includes project management, business analysis, development, and testing, of course. In addition, delivering software that works requires consideration of concepts around management, sales, and business organization.

I’d like to share some of the blogs that I find influential when thinking about how to build software that works.

James Bach and Exploratory Testing

One of my favorite blogs is James Bach’s Blog. He is the creator of Rapid Software Testing. I first became interested in his blog because he wrote about exploratory testing in a way that made sense to me. I knew from experience that exploratory testing was one of the best ways to find important bugs, but I didn’t have a great way to communicate the high value of exploratory testing to teams that were focused heavily on mechanizing the testing process. James Bach has written a lot on the topic of exploratory testing. This particular post about the history of Exploratory Testing captures many of the concepts that drew me to the blog.

I particularly like the concepts of testing vs checking and the “responsible tester.”

Feedback

What do you think of James Bach’s blog and the concepts he writes about? What blogs and resources do you find particularly interesting and useful for building software that works?

Embedded Quality – Building on a Solid Foundation

Embedded Quality is a Quality Assurance (QA) technique that I’ve introduced on many projects with great success. Embedded Quality has some of the following benefits:

  • Avoids long test/fix cycles at the end of projects
  • Ensures that projects finish strong
  • Lowers cost
  • Shortens timeline
  • Improves quality
  • Increases customer confidence in IT department
  • Easily applied to any software development methodology

Basic Concepts

The basic concepts of Embedded Quality are nothing Earth-shattering. The concepts draw from standard software development techniques and seem like they should be common sense. However, in the 20+ years that I’ve been working in software development, I’ve found that these concepts are not commonly practiced.

QA starts on Day 1

I have found that on many projects, Quality Assurance is an after-thought. Involving QA as part of the core project team helps prevent nasty surprises at the end of the project.

QA is part of the Core Project Team

Too often, the development team and QA team are completely separate entities who send their work “over the wall” to each other. Quality and speed increase exponentially when we break down the barriers between these groups and work as a team.

QA is performed by qualified experts

One of my clients used to hire people out of high school to test their software. If they did well, they got promoted to doing phone support. Eventually, they realized that they could drastically decrease phone support and increase customer satisfaction by hiring qualified people to do their Quality Assurance work.

Just as building inspectors need special knowledge and experience to do their jobs well, software QA experts need special knowledge and experience to do their jobs well.

General User Acceptance Testing (UAT) does not begin until Core Project Team QA is complete

A common practice on projects is to rely on UAT as the only form of testing. This results in the end users seeing a large number of the flaws in the software. Invariably, I have seen the end users permanently lose confidence in the development team when using this QA strategy.

I liken this to a builder turning over a new house to a buyer without having any inspections and saying, “We don’t really know if the plumbing is solid, all the electricity works, or the roof leaks. Just try everything out and let us know what the problems are.” (In fact, this does happen: http://abcnews.go.com/GMA/Consumer/story?id=2630414&page=1.)

Strong Foundation – No code is “Complete” until it is tested and works correctly

Many project schedules track code as complete when the developer completes their initial delivery. This can give a false impression of the project status.

More importantly, this ensures that new code is built on a strong foundation. When developers build new code on untested code, it is very similar to building an office tower on a foundation that has not been inspected. If a problem exists in the foundation and is not identified early, the building will fail.

http://blogs.wsj.com/chinarealtime/2009/06/29/shanghai-building-collapses-nearly-intact/

Embedded Quality – Next Steps

Over the next few months, I will post tips on how to implement Embedded Quality on your next project. I’ll share stories of my experiences over the past 20+ years, and I would like to hear your experiences as well.

Scroll to top