Archive for the ‘Test Plan’ Category

What I’ve learned about Agile Testing

I started on an agile team last August. In the past few months, I’ve be exposed to a wide range of agile-ness as we have learned to adapt it for different projects and struggled to make it work for us. I have found great worth in some of its practices, and unfinished mess in others. As a team, the company has had to switch practices and courses a few times. The cool thing is that we are able to. No one is so stuck on procedure as to not admit that it could use tweaking. And where procedure was too lax, people have asked for more.

Here is what I’ve learned as the QA department has grown up and around the development teams:

– Agile doesn’t mean “No Processes” it just means flexible processes

Documentation is more a struggle. Shortcuts made in the name of agileness and flexiblity have been huge road blocks in my testing effort. We are still sorting out the right place for them. Something that allows for creativity and adjustments, but still leaves us with a solid document to test against.

– Trying to keep meetings short by cutting out discussion is pointless and unhelpful. Structuring meetings to allow for quick agendas and then discussion time is priceless.

Transparency between departments and code reviews are QA’s best friend, even if it takes more time at first.

– Short iterations mean quick test cycles. This means you need to have as much ready before had as possible – test plans, test suites – both automated an manual, report templates etc can all help keep you on track.

– Managing an agile team means knowing when to step back. The team is responsible to each other, not to you.

– Be flexible. This should be obvious, and maybe it is to some. But in my experience, learning agile meant learning to contort your testing efforts to the time and space available. That doesnt mean you should let your team get pushed around, but it means that you have to be willing to do the agile dance along with everyone else.

– QA needs to make its needs known. Don’t let yourself get deadlines pushed back and back until you are the one left struggling to pick up all the slack. Just because deadlines can change mid cycle now, doesn’t mean that your team has to sacrifice thoroughness in the name of changing requirements and speed.

– Communication is even more important – with things changing and people having so much freedom with their work, if you have a bad communicator, your whole project is at risk.

I am sure this list will grow in the coming months. For those of you who are new to Agile testing- what have you learned? Was it what you expected? Have there been any roadblocks on your project?



Now that choosing a new team member is complete, it is time to formalize goals for the QA department moving forward. When I started here, QA was a footnote to the development process. The engineers ran unit tests and the production team “clicked through” some functionality at the end. But no one was really focused on what a QA department should look like or what that role should be. I think one of the biggest challenges I faced when I came on was carving out a bigger spot for QA within the existing environment. It is an ongoing process, but in the past six months, I’ve seen vast improvement. It is amazing to see processes start to take shape and make an impact. This position has been incredibly rewarding, in a large part, because I can see improvement every day. The whole team here has been flexible, even when I tried something that didn’t really fit, everyone gave me time to try it and then quickly adapted to something new until we found the processes that worked for us. It is nice to reflect on that progress. But no matter how much further we are now than when I started, there is still so much that can be improved upon.

What I hope to accomplish next is opening up the QA process more to the development team. There is a review process now,  but it is loosely followed. I’d like to finish a test plan and have a round table time for people to review, critique and discuss risk areas and test cases. I think it could help in so many facets. Engineers can think more about risk areas when programming and QA can hear about trouble spots or areas of confusion and revise test cases to be more complete in those areas. The production team will have a more complete picture of what needs to go on before a deploy and most importantly, we all get a chance to sit and talk about quality together. I think its really important for such a discussion to occur before the official testing period.

I keep coming back to where QA teams can add value to the whole process, not just the end product. In an interview, I had one candidate define QA’s role as “Reactionary” to the development process. I have to say, I really don’t want that in this department. I want the QA team to be integrated and active throughout the development process – proactive about reviews, discussions, risk areas and testing. There is no reason to wait until the end of a cycle for that, especially not on an agile team. So here is to a larger team, new goals, and hopefully some more great progress like we have seen in the past six months.

Document Standards

Recently, I have read some conflicting reviews on using IEEE standards for documentation and test plans. I’ve heard people say it is a crutch, and claim that if you need the document to do your job, then using it is actually harming you. This makes sense in a lot of ways.

If you approach the template like you would approach a worksheet in school: fill the blanks in with word for word answers from the book and don’t give it much thought, of course it is a crutch. It gives you a false sense that you know things and are learning, when really, you are just being prompted.  I’d hardly call that the template’s fault though. Likely, a tester who put so little thought into their test plans would have trouble adapting with or without a template. Does this mean they are not viable tools for test documentation or requirements specification?

The way I see it, IEEE templates are just that: templates. Of course no one template is going to work for every company or even every project.  The template must simply guide a person to create a living, meaningful document for your project. If used in that way, it is not a crutch but a standard to hold yourself to. You just need to remember to apply some rational thought to it instead of using it blindly.

I have found IEEE 829 (test plan) and 830 (requirements document) to be very useful on my project. They have been a good starting point for building documentation and test plans as well as providing a cohesive feel and expected deliverables for the team. I find them very flexible and easy to adapt to every project we’ve tried so far.

Some great detail on IEEE 829 can be found here – I’ve included the basic outline below.

1. Test Plan Identifier
2. References
3. Introduction
4. Test Items
5. Software Risk Issues
6. Features to be Tested
7. Features not to be Tested
8. Approach
9. Item Pass/Fail Criteria
10. Suspension Criteria and Resumption Requirements
11. Test Deliverables
12. Remaining Test Tasks
13. Environmental Needs
14. Staffing and Training Needs
15. Responsibilities
16. Schedule
17. Planning Risks and Contingencies
18. Approvals
19. Glossary

A nice overview of IEEE 830 for Requirements documents can be found here, the format is below:

1. Introduction
1.1 Purpose
1.2 Document conventions
1.3 Intended audience
1.4 Additional information
1.5 Contact information/SRS team members
1.6 References

2. Overall Description
2.1 Product perspective
2.2 Product functions
2.3 User classes and characteristics
2.4 Operating environment
2.5 User environment
2.6 Design/implementation constraints
2.7 Assumptions and dependencies

3. External Interface Requirements
3.1 User interfaces
3.2 Hardware interfaces
3.3 Software interfaces
3.4 Communication protocols and interfaces

4. System Features
4.1 System feature A
4.1.1 Description and priority
4.1.2 Action/result
4.1.3 Functional requirements
4.2 System feature B

5. Other Nonfunctional Requirements
5.1 Performance requirements
5.2 Safety requirements
5.3 Security requirements
5.4 Software quality attributes
5.5 Project documentation
5.6 User documentation

6. Other Requirements
Appendix A: Terminology/Glossary/Definitions list
Appendix B: To be determined

Agile vs. Waterfall Development

Being new to the Agile world, I am trying to figure out exactly where testing should fit in this model.

Waterfall testing is clear – you test the final product at the end of the development cycle, report bugs, test bug fixes and sign off on testing.  Agile approaches things a little differently. Documents aren’t always finished before the project starts, the product isn’t finished before you start testing, and the very things you are testing can change quickly. Agile Development is in vogue – but where does good testing fit in that schedule?

I am learning that effective agile testing has the same principal as waterfall: testing quickly and early. The difference is in how you arrange your test schedule around development sprints, instead of grouped together at the end. Adjust your test plan into mini-test plans based on milestones. Treat each milestone as a whole project that needs to be tested. At the end, you can run through all the test cases again to test the complete functionality, but each step along the way has been tested as completely as if it were a stand alone product.

Get involved in the process at the kick off meeting, talk to the developers, pop into daily stand ups when you have time, ask questions and listen to how they talk about what they are working on. Does it seem like certain parts of functionality are giving them a hard time? Take note. If you listen to your team, they will tell you the problem areas, the parts that were rushed, the developers who weren’t talking very much while they were working. It gives you some red flags to look at when designing your test plans.

Agile Development promises more transparency in the development process – we can include QA in that. Open up test plans and test cases for review. Ask developers to sign off your coverage, give them a chance to make suggestions. It gives you a chance to learn from them, but it also makes them aware of how you test and what you look for. Quality becomes a group goal.

This is a great video from youtube on the differences between agile and waterfall development, which is a great overview. I initially found the video on this blog, which also has lots of info on Agile Testing.