Category: Communication
Offshore Agile Testing…a Request from Management
A friend of mine asked me for some advice about how to run an Agile software development team with the developers and business users working in the US and the testers working for an outsourcing company in India.
My short answer was, “You can’t”.
The Agile Manifesto and Offshore Testing
I’ve actually had similar requests a number of times over the past few years. Managers want to be “Agile” and want to use “Offshore resources” so they ask their project teams to implement the combination of the two hot concepts. In the cases that I’ve seen, management has tried to send only the testing portion of development offshore. They wanted the onshore, local team to be “Agile”, but they also want to send code “over the wall” to an offshore test team. Unfortunately, these two concepts do not work together.
The Agile Manifesto principles include “Business people and developers must work together daily throughout the project;” “Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done;” “The most efficient and effective method of conveying information to and within a development team is face-to-face conversation;” and “The best architectures, requirements, and designs emerge from self-organizing teams.”
These 4 (out of 12) principles are very difficult, if not impossible, to implement in an environment where the testers are separated from the rest of the team by geography and time zones. In addition, when only the test team is outsourced to another company, the testers often are required to communicate with the rest of the team via a single point of contact rather than directly with developers and business people.
To make things even more difficult, I’ve found that offshore, outsourced teams are typically set up to run a full, lengthy regression test for each build. In one case, I worked with a team that was trying to do Agile development with a test team that insisted on running a 6 week regression test for every build. This ran completely counter to the Agile Manifesto principle of “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.”
Can Agile Work with Offshore Teams?
I believe that Agile development can work with offshore teams if the teams are structured correctly and significant effort is put forth in building trust. Here are some techniques that I’ve used effectively to help build an Agile, globally distributed team.
- Have a full team consisting of all roles in each geographic location. This means that developers, testers, and business experts must be physically located in each location. The business expert may be a Product Owner Advocate rather than the actual Product Owner.
- Ensure that everyone on the global team has time to work together in person early in the project. This means flying people around the world which can be costly, but it makes a huge difference in the ability for team members to trust each other and work together.
- If the team is divided into drastically different time zones, schedule specific weeks throughout the project where the global team will work at the same time regardless of time zone. During these weeks, the team should make a concerted effort to communicate in real time with their counterparts around the world.
In a future post, I’ll talk about some additional techniques that are not specific to Agile that I’ve used to make global teams more effective.
Further Reading
Over the years, I’ve found these articles helpful in understanding the issues related to Agile distributed teams and how to address those issues. Maybe you’ll find them useful as well. To me, the two articles available for purchase from IEEE Computer Society were definitely worth the small cost.
Agile Strategies for Geographically Distributed Teams
Global Development and Delivery in Practice
Agile Offshore Techniques – A Case Study (Available for purchase)
Using and Agile Software Process with Offshore Development
Follow the Sun: Distributed Extreme Programming Development (Available for purchase)
Your Experiences
I’d like to hear any of your experiences with global Agile software development. What worked and what didn’t?
Contact me if you’d like help improving the effectiveness of your global team.
Drama Addiction
In a previous post, I lamented people who were rewarded for heroic efforts despite their earlier actions causing the problems in the first place.
A couple of people commented that these “heroes” usually didn’t intentionally cause the near-disaster situation. More often, they caused the situation by neglecting to follow a proven process or by simply not being proactive. I agree with these comments, and I believe that this points to a common problem. People tend to celebrate the visible, novel, and dramatic rather than the smaller, more important actions that keep things running smoothly.
Visible, Novel, and Dramatic
When I spoke at Agile Austin a number of years ago, one of the panelists mentioned that his team’s agile processes were working smoothly and his team was more productive than ever. However, several of his team members were becoming bored and unmotivated because everything ran too smoothly.
Around the same time, a friend on Facebook expressed frustration with the fact that people who lose weight “can brag about weight loss and exercise regimens and get praised by all, but a person talks about her workout and expresses pride, and they are met with sarcasm or daggers.”
This made me realize that people are rewarded more for the visible, novel, and dramatic in many areas of life beyond software projects.
- Politicians rarely brag about money spent on maintaining existing roads and bridges; however, they’re quick to take credit for new roads and bridges.
- A new, entry-level employee can impress co-workers by showing up on time and performing basic tasks competently; however, excellent performance is simply expected from a 15-year veteran employee.
- The first responders to a terrorist attack are celebrated as heroes, but the multitudes of people who worked to prevent many other attacks live in anonymity.
- A pilot who safely lands a plane with mechanical trouble is revered, but the pilots and other people who ensure thousands of safe flights per day are ignored.
This addiction to drama seems to undermine efforts to implement processes such as Embedded Quality that help projects run smoothly. Personally, I prefer to keep drama in the movies and out of my projects.
Overcoming Addiction
Is there any way to supply teams with the excitement and adrenalin rush associated with the visible, novel, and dramatic while still implementing processes and procedures that keep things running smoothly? It’s definitely a challenge, but there are some techniques that can help.
Celebrate Success
When a project goes smoothly because talented, dedicated people adhered to a good process, make that success visible. Celebrate the success publically and enthusiastically, and point out that the reasons for the success.
Create Positive Challenges
Talented, dedicated people crave novel challenges, so it’s important to provide productive challenges so people don’t fall back on the challenges of heroic efforts. The challenge might be to improve on the success of a previous project or it might be to learn a new skill, technical or otherwise. Any extra time or risk invested in team members learning a new skill on a project will pay dividends in employee motivation and skillset in the future.
Don’t Reward Unnecessary Drama
If a team is forced to work ridiculous hours to save a project due to failure to follow a process, thank people for their efforts and point out how the drama should have been avoided. Don’t celebrate the efforts as something to be emulated in the future.
You do need to be careful to determine the true cause of the drama, though. Every project can’t be perfect, and you do want to reward heroic efforts when they’re truly justified.
Feedback
I’m interested in hearing about other ideas for keeping teams enthusiastic and motivated when processes are running smoothly and efficiently. I’d love to hear about any techniques that you’ve tried and how they turned out.
Just Rebuild It – Tales from the Trek – Understanding Existing Functionality
On projects that replace existing systems, project sponsors have often insisted that we didn’t need any business analysis because the existing system was “fully documented” in the form of the old system. I wrote about one problem with that mindset, which is that the new system will have all the issues that the old system had.
Another problem is that it takes a lot of time to figure out how the existing system work. The development team will end up taking time to analyze the existing system regardless of whether analysis time is in the budget or not. Contrary to what project sponsors often believe, users of the system don’t know all the details of how a system works, and code is not “self-documenting”.
Multiple Users
Most complex systems have a variety of users. A relatively simple sales system has different users who place orders, manage orders, ship orders, manage the financials of the orders, report on orders, manage inventory, and handle returns. It’s time consuming for an analyst to research, understand, and record how all the parts of the system work from the users’ points of view.
User Knowledge is the Tip of the Iceberg
Even after talking to all the users, an analyst won’t know the full functionality of the system. In my experience, it’s rare to find users who know the details of every calculation, decision, workflow, and automated interface the system has behind the scenes. In the few cases where I’ve found that a system was well documented, I’ve been told that the documentation was outdated, so I couldn’t trust it.
Why not Just Look at the Code?
I’ve had project sponsors ask why the developers can’t just look at the code and rebuild the same system in a new technology. They assume that the rebuild is almost a copy and paste of the existing code. Although code examination is one of the techniques for understanding an existing system, developers are not able to quickly reverse engineer functionality from code in a complex system, even if the code is well organized. Any developer who’s made even a small change to a complex, unfamiliar codebase will tell you that understanding the code is time consuming.
Interviewing users, reviewing the exitisting user interface, and reviewing the existing code base are all important ways to understand the existing sytem. However, these actions clearly take time, and need to be budgeted for.
Feedback
What challenges have you had getting budget approved for business analysis? What have you done to address those challenges? I’d love to hear your tales from the trek.
When the Firefighter is the Arsonist – Tales from the Trek
A number of years ago, I read a headline that said, “Ex-Forest Lake fire fighter pleads guilty to arson“. The story went on to say,
“Once considered a hero as a firefighter, John Berken was accused of starting a fire that burned 2,000 acres of nature area.”
This story got me thinking about a common pattern that I have seen on software development projects.
Heroic Measures
Many teams glorify the team members who take heroic actions to save a project. These team members, who work nights and weekends for weeks, or even months, at a time to pull a project success from the jaws of defeat, are held up as positive examples for the company and lavished with praise.
However, too often, the very same people who perform the heroic deeds are the people who created the near-disaster in the first place.
As a QA Sherpa, one of my top goals is to prevent the need for heroic actions at the end of a project. I feel happiest when we successfully implemented Embedded Quality and find no important bugs during the last few weeks before going live. In this role, I’m particularly attuned to those “heroes” who actually created the problems in the first place.
These are the people who dramatically underestimated their work, skipped unit testing, refused to work with the testing team to practice Embedded Quality, and generally eschew any form of teamwork or foresight. Despite all these negative traits, they end up being celebrated for “going above and beyond the call of duty”.
True Firefighters
True firefighters do need to be heroic at times, and it’s important that they are ready and able to be heroes when necessary. However, their bigger goal is to prevent fires in the first place. In addition to preventing fires, they work hard to put measures in place to limit the damage of fires when they do occur.
The bulk of a fire department’s efforts are not spent putting out fires. The bulk of the time is spent educating people about how to prevent fires, ensuring that the fire code is effective and that buildings follow the fire code by implementing fire walls, fire doors, sprinkler systems, alarms, and fire extinguishers.
Similarly, a solid development team should be spending their time implementing automated unit testing, working with the test team to implement Embedded Quality, performing peer code reviews, and getting feedback from key stakeholders throughout the project. All of these actions work both to reduce the odds of a disaster and to limit the damage when a disaster occurs.
As I said earlier, heroes are still needed at times, but you should take a second look at those heroes when the disasters are too frequent. If a city has way more fires than the norm, the fire department probably isn’t doing their job well. If every project requires heroic measures, it’s time to look a little closer at why that’s the case.
Feedback
I’m interested in hearing about your experiences with fire fighters who may also be arsonists. Have you worked on a project where team members who seemed to sabotage the project early on were held up as heroes at the end of the project? If so, do you think it was due to the individuals on the project or due to a broken system?
Just Rebuild It – Tales from the Trek – We Don’t Need Requirements
Technology Upgrades – Self Documenting (?)
I’ve been on several projects where the goal is to replace an existing, mature system with a brand new system in a new technology. This is generally a huge investment in time and money, fraught with risk, and results in almost the same functionality. Still, it can be necessary due to the old software being dependent on unsupported or non-secure technology and exponentially increasing maintenance costs. In a way, it’s like making the decision to replace an old car rather than repair it.
On nearly every project of this type that I have experienced, the project sponsors have insisted that we could cut out, or at least drastically reduce, business analysis. Their reasoning was that because the new system simply had to work exactly like the old system, the new system was already “fully documented” in the form of the old system.
There are a number of reasons why assuming that the old system “fully documents” the new system is a mistake. I’ll address one of them here.
Rebuilding Bugs and Poor Functionality
Often, software that’s running on outdated, unsupported technology has a lot of bugs and poor functionality that haven’t been addressed. Any changes to the system have become risky because the system is so fragile. If the only direction given to the development team is to rebuild the existing system as is with no additional analysis, the team will literally be building the same bugs and functionality into the new system. There’s no good way for the development team to know if something should be changed without at least some investment in business analysis.
The end result is that the users and project sponsors are even more frustrated with their investment in new software only to end up with the same problems. Even worse, sometimes, building the bugs into the new system costs more money. Imagine how much more it would cost to try to literally rebuild an old car exactly as it is rather than build a new car with the same basic functionality. It would take effort to make the brakes squeak just like the old car, have it shake at high speeds just like the old car, and randomly stall in the middle of intersections just like the old car.
The money spent rebuilding old bugs would be much better spent on some business analysis.
Feedback
What challenges have you had with generating requirements when replacing existing systems? What actions have you tried to address the challenges? I’d love to hear your stories from the trek.
Setting Expectations – Lessons From a Little League Umpire
I’ve found through the years that projects go more smoothly when expectations are set at the beginning. Whether I’m in a Business Analyst or Quality Assurance role, I’ve found that the project goes more smoothly when the processes are laid out clearly up-front and the known limitations of the project are called out and addressed before the project starts. When people are not aware of the time that they need to dedicate to the project or the limits of the project scope until the project is well underway, they can get quite upset.
My most memorable lesson on the importance of setting expectations is still the one I learned while umpiring little league in high school.
Strike One!
I umpired my first game when I was 14 years old. The league was a city-run little league program, and some of the divisions had kids as old as me. Other than watching a lot of baseball, the only preparation the city gave me was handing me an armful of equipment and a copy of the rulebook. I read the rulebook several times and felt that I was as ready as I could be.
I set up the bases and put on my equipment for the first game. I said hi to the two managers and nervously took my spot behind the plate. My calls were a little shaky, but I was surviving. Surviving until the third batter, that is.
I lost my concentration on a pitch and uncertainly called, “Strike?” The batter’s manager yelled at me, “How can that be a strike?!? It hit the dirt before it even crossed the plate!”
I sheepishly said, “It did? I guess it was a ball then.”
Needless to say, the managers, players, and parents argued with me vehemently on nearly every call after that. I wasn’t sure I’d ever want umpire another game. However, the manager of the umpires was desperate for warm bodies to call the games and convinced me to finish out the season. I never had a game as bad as the first one, but every game was stressful and the managers, parents, and players argued with me regularly.
Year Two – Setting Expectations
In the next year, I read a book called “Strike Two” by former Major League umpire Ron Luciano. His stories about handling some of the toughest personalities in baseball gave me the insight and confidence that I needed to try umpiring again. The difference this year was that I was going to set some expectations before every game.
I met with both managers before every game and explained the following:
- I’m the only umpire, and I only see each play once. Don’t bother arguing any judgment call such as ball vs strike or safe vs out. I won’t change my call.
- I call a big strike zone. If any part of the ball is over any part of the plate from the top of the batter’s shoulders to the bottom of their knees, I’m calling it a strike.
- I’m not a professional umpire. I may make an error in the rules. If either manager believes that I made an error in the rules, let me know. I will call a time-out and meet with both managers to review the rule book and make an adjustment to the call if warranted.
Taking the time to set these expectations made a huge difference in my ability to manage the game. The managers would tell their players that I have a big strike zone and that arguing with me was pointless. The players swung a lot more, put the ball in play a lot more, and I got very few arguments.
In addition, after the games, parents would tell me that it was one of the best games they went to all year, especially for the younger age groups. Apparently, in many other games, the batters for each team would simply draw walks until they reached the 10-run mercy rule for the inning. This was boring for everyone involved. Because players were swinging more when I umpired, the ball was put into play more, and the kids and parents had a lot more fun.
Your Experiences
I’d like to hear about your experiences with setting expectations. Have you found that setting expectations up front helps your projects? Have you received any resistance to setting expectations at the start of a project?
Influencers – Jayme Edwards
There are a number of blogs that I enjoy reading about all areas of the software development process. The software development process includes project management, business analysis, development, and testing, of course. In addition, delivering software that works requires consideration of concepts around management, sales, and business organization.
I’d like to share some of the blogs that I find influential when thinking about how to build software that works.
Jayme Edwards – Consulting and Continuous Delivery Expert
Jayme Edwards is someone who I’ve known for a while. We’ve had many great discussions over the years about consulting and continuous delivery. I’ve gained a lot of insight from him directly and from his blog posts.
One blog post that recently resonated with me was titled Establishing Trust to Make IT Development Process Changes. His points about authenticity and setting realistic expectations up front are keys to a successful client/consultant relationship. In addition, he referenced Peter Block’s Book, Flawless Consulting which lays out three roles that a consultant can have when interacting with a customer: Expert, Pair of Hands, or Collaborator. I have been in all three roles, and although I enjoy them all, I especially enjoy working with a client in the Collaborator role.
Jayme Edwards’s is also incredibly knowledgeable in the area of Continuous Delivery. He is intimately familiar with the theory of Continuous Delivery. More importantly, I have seen him successfully move multiple clients into a successful Continuous Delivery model. Because Jayme Edwards has put the theory into practice many times, I find his posts on the topic to be particularly insightful and credible. If you’re interested in learning about Continuous Delivery, you should check out his posts.
Feedback
What do you think of Jayme Edwards’s blog and the concepts he writes about? What blogs and resources do you find particularly interesting and useful for building software that works?
User Acceptance Testing Tales from the Trek – Writing a Perfect Bug
My early experiences running User Acceptance Testing (UAT) typically involved dealing with just 1 or 2 users. When they encountered an error, they’d show it to me, and I’d record it. However, as I gained experience in Quality Assurance, I worked on larger projects.
Training – How to write a bug
On my first larger project, I needed to train a group of 20 users who were going to perform User Acceptance Testing. Having worked with several different users in the past, I knew several of the pitfalls that users may encounter. However, I never really had asked users to record errors themselves. Still, I thought it would be easy to get the point across.
During the training session, I said that it was critical to record all necessary steps so that the developer could recreate the bug. I told the users that we often hear from users who simply say, “It doesn’t work,” and explained that the developers will not be able to fix a problem that was described so generically. The users said that they understood perfectly and were ready to start testing.
Bug Writing – Frustration
With 20 inexperienced testers working independently, it was important to filter the reported bugs before assigning them to the developers. I was responsible for triaging all bugs reported by the users. I verified that I could recreate bugs and eliminated duplicate bugs before assigning them to the developers. If I could not recreate a bug based on the instructions, I would reassign it to the user who reported it and ask for more detail. I assumed that the users would quickly realize when they were not providing sufficient detail and start writing perfect bugs after a day or two.
I assumed wrong.
Although a few users were writing clear, reproducible bugs, most were writing ones that were way too generic. For example:
- I was on the “Choose a Course” screen.
- When I chose a course, I got an error.
In this case, when I tried to reproduce the bug, I did not receive an error. So, I sent the bug back to the user requesting more detail. In most cases, the first time a bug was recorded, I had to reject it. In many cases, I had to reject the rewrite as well. Pretty quickly, the users got fed up with me and complained to their manager that I was rejecting their bugs.
Enlightenment
I realized that I needed to find a way to illustrate to the users exactly what was needed in a clear, reproducible bug report. I developed the following simple exercise that was very effective. I have used it on all subsequent large-scale UAT efforts with the same success.
I ask the users to do the following with their first 5 bug reports:
- Record the bug as clearly as you can and assign it to yourself.
- Wait 5 hours.
- Try to recreate the bug following only the information that you recorded.
- If you are able to recreate the bug, assign it to me. Otherwise, go back to step 1.
It is important that the users wait some amount of time before trying to recreate the bug. I found that when they try to recreate the bugs immediately, the steps are too fresh in their mind to truly follow only what they wrote. They seem to fill in the gaps without realizing it.
I have found that once users complete this exercise, they are able to consistently write bugs that have enough detail to be reproducible. They do not become expert testers, of course, but they are able to write a useful bug.
I’m interested in hearing about other people’s experiences as well.
What concepts have you found especially difficult to get across to people performing UAT for the first time?
Were you able to find a technique that helped the users better understand the concept?
Metric Misuse – Quality Assurance Metrics Gone Awry
I was reading through some posts from Bob Sutton, one of my favorite management gurus, and I ran across a post that contains one of my favorite Dilbert comic strips.
Bob Sutton’s post, as well as the comments that I made on his blog, reminded me of one of my favorite topics: misused Quality Assurance metrics.
Tying Quality Assurance Metrics to Financial Rewards – A Dangerous Game
“Treat monetary rewards like explosives, because they will have a powerful impact whether you intend it or not.” –Mary and Tom Poppendieck, authors of Implementing Lean Software Development: From Concept to Cash
Over the years, many people have asked me what Quality Assurance metrics they should use to evaluate employee performance. My advice is that Quality Assurance metrics should not be used directly to evaluate employee performance. The Dilbert comic strip may seem a bit extreme, but it’s exactly what happens when employee performance is based strictly on metrics. This is true regardless of whether monetary rewards are explicitly tied to the metrics or not.
In my comments on Bob Sutton’s blog, I mentioned three specific metrics that had unintended effects when used for evaluating employee performance:
- rewarding testers for the number of test cases they wrote resulted in poorly written test cases;
- rewarding testers for the number of bugs they found resulted in a high number of unimportant or duplicate bugs reported; and
- penalizing testers for bugs rejected by the test lead or development staff resulted in important bugs going unreported.
Many people think that they have the ability to write a set of metrics that can be used to unequivocally gauge the performance of a Quality Assurance professional, but I have not yet encountered a metric that couldn’t be manipulated to favor the employees.
(If the metric can’t be gamed, it probably isn’t under the control of the employees, so it wouldn’t be effective at driving behavior anyhow.)
Are Metrics Worthless Then?
Actually, metrics are a great tool for identifying coaching opportunities and potential problems. However, in order to get honest metrics, they shouldn’t be used directly for employee evaluations or employee rewards.
When I’ve looked at the metrics that I mentioned earlier with an eye towards coaching, I had excellent results.
- Reviewing the number of test cases written helped me identify a tester on my team who was putting much more detail than I wanted into his test cases. After some coaching, he was able to consistently meet my expectations.
- Reviewing the number of bugs found by each tester helped me identify a tester who was digging into the root cause of the most difficult to reproduce bugs. She didn’t report as many bugs as others, but her work was critical to getting a great product out the door in a timely manner. It turned out that she was the most skilled tester even though she reported the fewest bugs.
- Reviewing the number of bugs rejected by the development staff helped me identify a manager who was evaluating his programmers based solely on the number of valid bugs found in their code. The developers were motivated to simply mark bugs as invalid rather than fix the bugs. This insight allowed me to address the problem directly with that manager.
Good Quality Assurance metrics provide powerful tools for managing a Quality Assurance team when used properly. However, they shouldn’t be used in a vacuum. They should just be considered one data point among many.
I was only able to scratch the surface of this topic in this blog post. I plan to discuss specific metrics in future blog posts. In the meantime, if you want to read a much more in-depth review of the pros and cons of employee incentives, you can find one paper here.
Your Experiences
I know that a lot of people feel passionately about Quality Assurance metrics, both pro and con. I’m very interested to hear about your experiences with Quality Assurance metrics. Have you found any that were particularly useful? Have you found any that had unintended consequences?
Communication – Tales from the Trek – Perception is Reality
Based on my conversations, I believe that most people perceive that they do more work than others. It seems common that all members of a household think they do more than their fair share of work, and most members of teams at work feel the same way. I believe that a big driver in this perception is that people know exactly what work they’ve done, but they are not aware of the work that others have done.
IT is Lazy!
At one place where I worked, the business side of the department often expressed to me that IT was lazy. They said that IT was never getting to any of their requests, and they weren’t sure what IT was doing all day. Not surprisingly, the IT department expressed that they were overworked and accomplishing a lot. Why was there this extreme difference in perception?
The IT department met with the business leadership each year to determine the priorities for the year based on the hours budgeted. IT was excellent at delivering on the priorities; however, the business had many more requests for changes and improvements that were above and beyond the priorities for the year. So, most individuals on the business side didn’t have their personal priorities addressed even though the overall strategic priorities for IT were accomplished.
Changing Perceptions
A director of the IT department came up with an idea that was very simple but incredibly effective. He created a giant checklist of the scheduled IT projects and posted them in a few prominent places around the office. As each project was completed, he put a checkmark next to the item.
The comments that I heard from the business completely changed. Although their personal priorities may not have been addressed, they could see the importance of the priorities posted on the checklist. In addition, they could see clearly the progress that had been made. The checklist had greatly improved the business’s perception of IT.
Feedback
What perception challenges have you faced? How did you address the challenges? I’d love to hear your tales from the trek.