Archive for the ‘Processes’ Category

Test Escape Reports

We just had a big launch at work, which of course leaves me thinking about test escape reports.

I have mixed feelings on test escape reports, and for a long time I didn’t want to use them here. My reason being that if you have a small test team, and you know everyone is really dedicated and working hard, then the occasional test escape is to be excepted and we shouldn’t harp on them.

However, I think over time I’ve seen the other side of this issue. Test Escape reports ensure stakeholders that you are on top of any issues both before launch and after. They hold the team accountable, not just for the issue but also for the resolution and prevention of it in the future.

 

The format we use is simple:

Issue:

Severity:

Reason missed:

Steps taken to cover functionality:

Test cases numbers for verification:

 

The point being not to look backward, or to blame someone for missing a bug- but to look forward into how we can test that in the future and how we can avoid over looking similar areas on other projects. The goal is to improve visibility- we admit our mistakes, take steps to correct them and let everyone take comfort in/participate in the improved coverage.

How do you handle missed bugs? Are test escapes’ a whole team problem or an individual tester’s problem? Is there a formal process surrounding them?

Advertisements

Moving to Agile: What the QA team needed to learn.

When my company moved to  Agile, the QA team had to make some adjustments to integrate and fit in with the new process. We’ve been working at it for a good three months now, and I think we are all starting to feel more comfortable with it.

Staying in Step:


Firstly, QA is used to being just a little behind development. We were focused on projects, not features. Although we tried to get in as early as possible, we were usually still an iteration or a milestone behind. It wasn’t terrible, but there were complaints that by the time we had questions about features or stories, the dev team had already moved on and answering our questions meant that they had to switch gears. We had a hard time because it seemed like our bugs stuck around longer and we were just a little out of the loop. Playing catch up on projects when we came in and ramped up was hard and time consuming. We wanted more communication.

Now, we attend each development team’s daily stand up – so we not only are held accountable for our tasks but we hear what the team is working on from day to day. We are part of the sprint planning meeting and we work along side the developers so our questions usually come up early enough to save time and make clarifications right when the feature is being built.

Divide and Conquer:

We have a few teams each working on different projects. Its a lot for one QA to keep up with.  I guess one of the challenges of a small QA team is finding a balance between giving each project the focus and dedication needed, while still getting all the work done on other projects. To handle this,  I split up to cover more. We can each specialize in one specific team, work with them, attend their meetings, and stay up to speed on all their projects. Our own, informal standups just allow the QA team to give eachotehr a quick overview.  It works because each QA gets to be an expert on a few projects and has time to get more indepth when they need, but if they need to switch projects to help out, there is already an expert on that project ramped up and ready to help.

 

Accept Early:

My favorite thing about the switch to Agile is acceptance tests. Once each story is defined, QA works with developers to come up with acceptance steps that become part of our story. This has been great, because I get to talk to the team before they start working to clear up any ambiguities about the feature. I also get a heads up so that when they finish it, I already know what to expect and how I want to test. It has increased our team work and our communication and it helps QA stay out in front of the game.  Once all the features have been looked at individually, we give the whole product a run through, of course. But it gives everyone piece of mind to have the features tested individually first.

Sauce Labs

We got set up with Sauce Labs last week and are starting to play around with it. It seems like it will be a great addition to our automated testing. We are in the process of redefining how we use Selenium here, so that it is used more effectively and strategically.

Really, our greatest challenge has been getting everything smoothly integrated through Hudson, which I think will make a huge difference. The whole team worked through it together and we learned a ton.

At Agile 2010 I heard Jason Huggins speak about using selenium to make demos of functionality for shareholders. I loved this idea. What a great plan  to give people an easy way to keep track of a project’s current status and to nail down the functionality for some high level tests. We are trying a similar idea now for a widely used site to help users get a good handle on the features. We are hoping it is our first finalized example of the new automation process with sauce labs integrated. I am really excited!

Time for QA to fully embrace Agile

My favorite idea from Agile 2010 was that “QA should be the headlights for a project” A quality guide, not a quality police.

We can sit at the end of the schedule, run tests and file bugs. Or, we can jump out in front, help define requirements, learn from the programmers as they work on the project, offer advice and keep the user experience and project quality on everyone’s mind from beginning to end. Which helps the project more? Actually, which is more fun? Because personally I’d much rather get involved and get people excited about turning out a good product in the end, than just force quality on them right before release.

So how do you change your company culture to embrace that? I am presenting at a lunch and learn next week, with the hopes of kicking off this next phase of the QA process. I am so inspired and excited for the change after a week of being surrounded with agile talks… but how will the company take it? I think I have gotten my department on board, but the hard part will be the other departments who have to put up with us a lot more now. No one likes change, and organizations are harder to change than people.

So how do you get everyone excited about it? How do you show everyone how important this will be? I haven’t finished my presentation yet, but I’m trying to think back to those really inspiring sessions at the Agile Conference and give it a go. Heres hoping!

Agile Conference 2010

I arrived in Orlando with my team last night. We are looking forward to a week at the Agile Conference 2010. Our company has been moving more and more into true agile all year, and we are learning as we go, so this conference will be a great chance for us to pick up new ideas.

Finding out a testers place on the team has been an ongoing and ever changing process. Which is why I was so excited when I learned that there is an amazing testing track with classes I can’t wait to take and speakers I am looking forward to hearing. I hope to learn a lot to bring back to the team at home and try out for ourselves!

The Tester’s Scientific Method

We all remember the scientific method from jr. high science labs.  But as a tester, we still use it every day. The steps may entail different things, but the underlying philosophy remains the same. Testing is an interesting blend between scientific and creative, between destructing and constructing. It requires a flexible yet ordered mind and, as with any scientific approach, it requires curiosity.

1. Ask a Question: The most valuable thing a QA can do is to ask the right questions. Is this feature supposed to work this way? Could this functionality be broken if I do this?  How will this affect existing functionality? What does this mean for the user experience?

2. Do research: Talk to the engineers, look at the documentation, discuss with your fellow testers. Stand up meetings are a great time to hear what developers are up to and to ask any questions you have run into on those features.

3. Construct Hypothesis: Determine how the feature should work and all the ways you need to verify that it passes when it is supposed to pass and fails when it is supposed to fail. Write your automated and manual test cases

4. Test: Run test cases

5. Analyze Results: Evaluate what is failing and why. It is not enough to know that a feature has an issue, you need to know what exactly reproduces it? If you spend a moment looking at it, perhaps you will find related issues or other ways it can be reproduced.

6. Report results:  Carefully fill out your bug report. Stick to a bug filing standard. Spell check. Make sure that anyone could understand your bug and the information you have gathered from it.

7. Be prepared to test again. Bug fixes always bring with them the risk of inserting more bugs. Don’t just verify a bug fix, test the bug fix. It should get the same attention that the original feature did. Be aware that you may have uncovered a cluster of bugs or a series of new bugs that have appeared since the fix, so look at it with fresh eyes.

The “Mythical” QA schedule

I just finished reading The Mythical Man Month by Frederick Brooks. This classic was still amazingly relevant, even thirty some years later.

Some of the tactics he suggests have just been made obsolete by the advances in technology – however, his use of a surgical team as a model for software teams seemed surprisingly agile and modern (not that agile and modern are synonymous). What was most interesting to me was the importance he placed on software testing. In his planning for software schedules, he allots half the project time to testing and bug fixes.

What could you do with that kind of time? Could better fixes be implemented for bugs? Would more bugs be fixed pre-launch? I find that as development schedules get pushed out, it is testing that get shortened.

Squished between milestones that can’t move or running up against a launch date that has already been announced, testing is often a race against time. Sometimes – new projects are hard to estimate for. Sure, you know how long it will take you to run through your test cases, but what about the debugging time? That varies greatly depending on the bugs found. Having half the schedule devoted to testing (or even just a greater percentage) would certainly be a change.

Would such a schedule just stretch things out or get shortened itself? Would it hinder the creativity of the engineers by shortening their time? Or would it work to allow a team with greater importance and visibility placed on testing? A place where quality was not just a “QA” thing, but a collaborative time of testing and bug fixing that could bring the importance of quality at every step of the development process to the forefront.